As someone who's 6'8, my worst fear is the short guys that are even remotely stocky. They're strong as hell and once they grab you they ain't letting go LOL.
Most short dudes I have known are WAY more aggressive than the biggest dudes. That said, height and weight does matter. My Dad is 5'4 though, and his best friend was 7 foot. They had fought so much over the years that my Dad learned how to deal with tall ass motherfuckers. He was a ninja in his old days lol.