Stanford CS234: Reinforcement Learning | Winter 2019 | Lecture 4 - Model Free Control 

This proof around 33:03, to a certain extent, is very cumbersome since all these transformations did not take you anywhere to be honest. Essentially, what is really important is that for V^{pi_{i+1}}, there is a taking max{Q} over action, which is certain always larger or equal to Q over action, which is what V^{pi_{i}} gives you in the previous iteration. On average, it is the greedy action that ensures the monotonic improvement.

Interestingly, they didn't fix the proof in the 2023 class either

the (1 - eps) / (1 - eps) is just to show that we did not changed the value of the sum by multiplying by this quantity and then we can write 1 - eps another way i.e with the sum of pi(a|s) - eps. Then, sum pi(a|s) - eps = (1 - eps + eps/card(A) + [(card(A) -1)/card(A) * eps)] - eps. The expression between parentheses corresponds to the sum of probas by definition of the epsilon soft policy. Be careful that they put the epsilon inside the sum over a but it is not correct, we do not sum epsilon over all possible actions. Then with the simplification and the fact that max_a(Q_pi(s, a)) >= Q_pi(s, a) you get the result and what is interesting with this "cumbersome" writing is that the simplification it gives you corresponds to V_pi(s) and so you have shown that there's a policy improvement. You can also check the book Sutton & Barto Aabout RL the proof is phrased a little bit differently but it's the same idea (p.101 of the 2018 edition)