AITA for not wanting to go on vacation with my family?

I, 26M have always had a decent relationship with my family. Some time in mid 2010s my grandparents moved to our city and began to live with us. Before that, we would take 1 trip a year, whether that was a road trip to another city for a few weeks or overseas for a week and sometimes camping. I actually really enjoyed those times. However, since my grandparents moving in we've stopped unless my mom can accommodate them.

We stopped going out, not even to a bar or a restaurant. We stopped taking road trips. We stopped going to neighboring cities for weekends. No more camping. We genuinely stopped doing everything, unless my grandparents could be part of it, when most of the time they b*** and complain about everything.

My last straw was when we went to Spain. It wasn't a good time for myself and my little brother as we wanted to go out and do our own thing. My mom refused to let us unless we could bring grandma and grandpa. So the entire time we spent kinda just doing nothing, playing cards, and sitting at the beach (which to me wasn't too bad but for my little bro was really frustrating as he wanted to go do certain kid friendly activities).

Once we got back from the top, I pulled my mom aside and explained that this is a pattern and it's been years where she puts everyone else's interests to the side (me, little bro, and dad) all so that she can be with my grandparents every second of the day. I explained that I'm not really interested in going on a vacation with them ever again if grandma and grandpa are there. So. We haven't been on a vacation with just the 4 of us in around 8 years. I'm on my own now and I don't really care about family vacations anymore.

I get a call from my mom and dad begging me to go on vacation with them in a few months. I'm not really interested and I refused to go. I made up an excuse that I don't have any vacation time left and left it at that. I then received multiple phone calls from them explaining why I should go but I'm not interested. So after the n^th phone call I just told them flat out:

1. If my grandparents are going to be there, then don't bother inviting me.

2. That I'm just not interested and would much rather not miss work.

3. Based on our previous experiences, I don't see how it's going to be any different this time around and I'd rather go on vacation with my girlfriend instead.

They just said okay and ended the call. AITA? Should I have just rolled the dice, that maybe it would be different this time around?

P.S. I like spending time with all of them, I go over for dinner and I invite them over for dinner every week. I just don't want to be treated like a 26 year old child.