I suggest that you work on a list of ways that you can enhance your life, and goals and values that you have for your life, that don't revolve around your husband. If sex is not on that list, then fine. But if you do want a sex life, and I think you do since you have had an affair due to sexual rejection before this, then your husband is right: you two want different things.
When beginning to venture out into dating life again, it's important to ask oneself these five questions to help clarify what you're looking for in a new relationship. These aren't resolutions, but rather internal conversations that will hopefully fortify your resolve to be connected to the most important elements of the life you want to have.