Imgine the yiga are so well disguised they are links clothes and have witnessed him torturing koroks and killing ganondorf so they are to scared to show themselves again
Yo I seen a lone bokoblin, and I was like "yo this is gonna be easy". Now imagine me and my brothers surprises when at like 1:30 at night a tree raises up and beats me up.
This makes sense because you can find these trees in the depths and are strengthened by the gloom. The yiga clan also states that the gloom will weaken them in a journal.