Somebody wants a fight with me, but I’m smart enough

So, a co-worker is should I say "harassing" me to fist with him. Unfortunately, no digital evidence or conversation rather than in-person talks.

I'm smart and continue to ignore but he still keeps on barking at me.

I am avoiding as much as possible any trouble to the company and around me. I can report him to the police, but what are your thoughts on this?

Can anyone help me on some legal advice I can take?