AITA for hanging out with my biological father after my grandmother and mother warned me not to

I (28m) grew up with my grandparents, and didn't' find out about my actual parents until I was 11 or 12. I loved my grand"parents" anyway, and didn't let it bother me. When my grand"father" passed away a few years later, I took it upon myself to attempt to contact my biofather just to get an answer on why he never attempted contact. In college, after pestering my grand"mother" and mother for months, I finally got a lead and a phone number. After sending a text and explaining who I was and what I was looking for, he texted me his home address and invited me over for coffee, as he only lived about 40 minutes away from me, and had my whole life. I made it to his place and knocked, he answered and hugged me for a good two minutes. We chatted for hours, telling stories, I guess catching up. I had been enjoying myself, but before the visit ended, I asked why he'd not try to contact me all this time? He explained that my mother and grandmother hated him, and in an effort to keep the peace, he complied when they told him to stay away. That was two years ago, and my bio-father and I have a wonderful relationship now. Recently, my mother and grand"mother" have been voicing their opinion on our visits, saying he only wants to hang me with me so that he can use me for something. I don't believe he is, as money or favors have never been brought up between us, if anything he's offered me money almost every time we hang out. They're telling me I should limit contact because "You are starting to turn into him and we don't like hanging out with you anymore."

AITA?