I have played tenor and soprano for a long period of my life, but recently decided to get a baritone. I thought it might be difficult to play, but to my surprise it was very easy, almost easier than any other kind of sax.
As a brass player, learning saxophone, I’d say that starting on baritone was probably the easiest thing I did considering it’s pretty much in the same range as a trombone
As a saxophone player the soprano is definitely the most difficult. Tenor would be the easiest imo. Bari is the second easiest (cuz u need good air control)