I say go for it. My fraternity had no black and brown people before I rushed and I was nervous as well, but my brothers accepted me. I found that by me being a member of a white organization I broke barriers for other men of color to see that yes, you can join this and you'll be fine. I guess it really depends on the school too. My school is pretty diverse and accepting. If you rush and an organization rejects you because of your skin, then that's not an org for you or anyone, really. Good Luck! Representation Matters!