Hey @alittlebitlostandunsure thank you for opening up about this memory you have, it sounds like it's been weighing on you for awhile.
I'm sorry to hear that after this encounter you feel vulnerable and nervous being alone with a guy after what happened. I don't think you're making a big deal of what happened because not being able to remember something like that would leave most people feeling confused and taken advantage of.
There's more and more education coming out about consent and we have a really good article about it here, if you'd like to have a read. An important point is that consent should be very clear and that being under the influence definitely impacts consent.
I'm just wondering if you've told anyone in your life about this? A friend or relative that you trust? Please know we are here to chat with you about this, I can imagine it's a lot to process.
I also just wanted to let you know I edited some of your post. When you have a chance give our guidelines a read here. If you have any questions at all about them, feel free to ask 
