Running five minutes late may not seem like a big deal to you, but for others it clearly sends the message that you are not a good time manager. Punctuality is harder for some than it is for others, but it's a skill that can be learned.
Your relationship to time is one of your most important relationships. Consider that time is the currency of your life. All the choices you make about how to spend your time add up to the kind of life you have.
I realize there are cultural differences in expectations -- in some places, it's a major faux pas to ring the doorbell at the time a dinner party is nominally scheduled to begin -- but let's put that aside.
Feeling as though you're always running twenty minutes behind schedule is an unhappy feeling. Having to rush, forgetting things in your haste, dealing with annoyed people when you arrive...it's no fun.