Simply put, it is not the presence of conflict that stresses the relationship; it is the manner in which the couple responds. Positive, respectful communication about differences helps keep a marriage thriving.
Appreciate your partner's strengths and work around the limitations, and your partner will be more likely to do the same for you. If your relationship is basically healthy, you are not settling in the sense of accepting less than you deserve.
There seems to be one common thread running through the behavior of chronically late individuals that is probably the most shared reason for them being perpetually tardy, and yet it is consistently overlooked. You are late because you don't want to be early.
Running five minutes late may not seem like a big deal to you, but for others it clearly sends the message that you are not a good time manager. Punctuality is harder for some than it is for others, but it's a skill that can be learned.
Your relationship to time is one of your most important relationships. Consider that time is the currency of your life. All the choices you make about how to spend your time add up to the kind of life you have.
I realize there are cultural differences in expectations -- in some places, it's a major faux pas to ring the doorbell at the time a dinner party is nominally scheduled to begin -- but let's put that aside.
Feeling as though you're always running twenty minutes behind schedule is an unhappy feeling. Having to rush, forgetting things in your haste, dealing with annoyed people when you arrive...it's no fun.