My first time I really smoked some good stuff (not the ditch weed I grew up with in high school) and listened to "Turned on Bach" - yeah, totally psychedelic. I took a 27 year hiatus, and still have never had that experience again.
I hate to say this, but it may very well be that with time and experience, the brain acclimates - which is to say, you are not the same person who had those psychedelic experiences. Now the ground you walk on is "known" and "ordinary" - your brain doesn't recognize it as "novel" anymore.
That is not to say you couldn't, with mediation and mindfulness still notice those things, they are just not "new" anymore, so you don't think of it as "WOW" like you did the first time.
Just a possible explanation here, nothing to back it up.
I would have to agree that there is surely a familiarity aspect going on but maybe it also has something to do with the fact that most of us have our first marijuana experience during development or puberty?