What do you say to non-Christians when they ask why you don't join them in their party or activity or gathering?

Simply tell them to read 1 Cor. 15.33 and a few verses above and a few vereses below it to really understand the passage.