Just because the term of the lease is over does not mean that the landlord automatically gets their seasonal rental property back. In many States, the tenant must surrender possession of the property prior to the landlord retaking possession.
Condos don't work in sag harbor. That's why I was very surprised to see one going up on a recent visit. To be an apartment building in prime sag harbor sounds good in theory, but I haven't seen it work yet.