Actually what we're seeing is the remains of the palace that was residence from 1530 until 1698. A fire in 1698 destroyed the majority of it. This is what remains. This explains the vast area behind that we now know as the Parade Grounds. There have been renovations done over the years, of course to put in electricity and such. I would assume that the content creator was referring to the horse piles about the ground in the courtyard and not to the historical buildings themselves.
There's a two-word term for this procedure... Horse Guards was built in the mid 18th century. For a building that age, it looks OK to me. "Neglected" is a bit of a stretch.
