AITA by not giving a refund for a beach trip?

My girlfriend and I have organized a beach trip for 10 people over Memorial Day weekend. Everyone agreed to it, paid up, and we booked an Airbnb 2 months ago. However, now one of the couples (Couple 1) is not associating with another couple (Couple 2) in the group because Couple 2 has been commenting racist things in general since we booked the trip. Couple 1 wants us to cancel the Airbnb booking which is fully refundable until the end of this month and give them their money back. They have convinced another person in the group to want to bail as well (saying racism from Couple 2). They are offering to help look for replacement Airbnb’s for the 7 of us that still want to go to a beach somewhere, but there's only a month and a half left. The Airbnb listings they've showed us are similar but not as nice for the same price per person. We booked so early hoping to get a good Airbnb (which we got).

We want to go on the trip and have already made plans based on everyone’s agreement. While I understand that Couple 1 is uncomfortable with Couple 2's behavior, should it be our problem? If I don't give them a refund, AITA?