My friend is accusing me of being a buzzkill and he says I never want to do anything outside of my comfort zone.

There's a Halloween party tomorrow night that he wants to go to, but it's in a town outside of the city, so it's going to be a long drive, especially if everyone is drinking. I don't want to put myself or anyone at risk and get drunk at a Halloween party to drive 45 minutes back. He doesn't want to pay for a cab or Uber (I agree, it's expensive), and is now mad that I don't want to go to this party. We don't have a designated driver and I don't want to go to a Halloween party and stay sober all night with a bunch of people I've never met.

Am I being a jerk here or should I hold my ground?