I have a problem whit....Fighting?

It's not like I can't take a punch or two, can't fight back or that I'm fat or anything or that I didn't use to fight from time to time if someone really asked for it. Up until 2 years ago I could do that no problem, then this guy sweared at me, I hit him and I broke his nose. It was the first time a fight ever went that far for me. I never actually hurt someone so bad. Since then I just fell afraid to fight, afraid that maybe It'l happen like then again. It's just stupid, I can't even defend myself now. Please tell me, Am I crazy? how to get rid of this felling? I just can't let myself beaten every time someone hits me.