Is this kid bluffing?

I'm a freshman and I accidentally bumped into a junior and he fell over (hes a small guy). He got back up and called me a dumb ***. Pissed me off cause I apologized and he responds with that? I sort of want to fight him cause he looks weak and the fact that he pissed me off. I went around asking some freshman who he was, and it turns out hes an but people love him. They said if I started a fight with him all the jocks he hangs out with would ruin my high school years. Rumors also say that he is a black belt in karate and tae kwon do but hes never fought in high school yet. Do you think they're bluffing or does it sound risky?