David was very publicly telling Amanda that she brought him nothing but sorrow during their relationship and expected her to respond but she disappointingly says nothing and that's why he doesn't grab her outreached hand at the end of the song. He is highlighting her misery and in the end when 'she' says "Oh David, its gone all black!" He tells her "Don't Start!" because she can do nothing but moan about everything. Sorrow indeed.
