He pretty much speaking to black folks at power who share tables with powerful white men, who are aware of the issues going on in the black/urban communities but refuse to speak or give any resources. Because if they do they(black people) will lose there seat at the table.
It's mostly affiliated with house slaves those who had better living conditions then of the field slaves.. so he's saying it as he's the savior of African Americans. Who at the doorstep of a white slave owner. So he says that to the house slave... But the main message is African American who are lobbyist and are rich and in power not help thier community and instead ignoring same as their white counterparts....