Never a good idea to approach a cat from above, which is exactly what your hand did to make the cat hiss and swat. This is extremely basic cat behavior.
I did that once, to a cat I was helping to look after. I did it because she had unexpectedly scratched the top of my head and then hissed at me after I'd been behaving quite hyperactively near her. I only hissed to keep myself safe, but the poor thing was scared. Still feel really bad about it to this day, many decades on. 😢 I wouldn't advise hissing back. The hissing is only done because the cat's unsettled with the situation for whatever reason - never out of spite. Don't do it to spite them, or really for any other reason if it can't be avoided. They'll be affected by it, trust me.
You can't just stroke him as though he's automatically going to be agreeable with it! 🙂 Perhaps that's not - *_entirely_* - what you meant by the context of your comment?? The man was in a position whereby stroking him wasn't straightforward, and the cat wasn't warmly reciprocating the man's willingness to do so. If there was doubt (it was written on the cat's face), then he shouldn't have pressed the issue. Hopefully he won't get that wrong in the future.
Kinda creepy seeing comments from people who think animals are just toys and they're "being bad" when they don't want some goon 10 times their size shoving his hand in their face. A person wouldn't like that, and animals often like being left alone too. Animals aren't toys, especially small animals whose instincts make them defensive.
