Thanks for posting this! Really odd setup for these handles. Also, I ended up holding the handle out with a hook tool in the door. Then pushed the handle in and out.
Actually, the best way I realized was to pull the door handle out when you push in the part through the hole. It pushes that piece under the handle and is much easier to remove.
Do not press inside the oval! Aim for the edge! Inside the oval is thinner than the walls and your screw driver might blow thru........... Just trust me.....