This film can be found for free online. I'd tell you where, but it's "hush, hush" because they will take it down if they find it. Old movie fans might figure out where to look because the site in question has thousands of old films.
Zatoichi might come across as just a harmless buffoon, but it's all an act. He's killed many people and is far more clever than he lets on. If those two knew who he actually was, they wouldn't have been so quick to dismiss him in this scene.
Unlike the others Zatoichi is blind so is actually disabled and needs every advantage to survive so that means being a crook did not come naturally for him unlike the other 2 people.