Comment on
Received a red packet after giving a gift - how to proceed?
I'd pass it on to someone younger than you. You didn't benefit, the tradition/luck was upheld, seems like the best answer here.
Morally that makes you all squared up, although legally you still accepted it, so this advice is only good if you know you could get away with accepting money, since this sounds like you visit this guy as part of a job.