A century ago, in 1912, summer arrived here in Massachusetts at 1:17 p.m. on June 21. This week it arrives almost a day earlier, at 5:09 p.m. on June 20. Both years are leap years (which by inserting an extra day makes summer arrive earlier), so that doesn't explain the difference. What's going on?
The problem is that inserting a leap year every four years makes for an average calendar year of 365.2500 days, whereas the actual "tropical" or seasonal year is 365.2422 days long. After 100 years, summer would arrive 100(.2500-.2422) ~ .78 days earlier. So we should eliminate about 3/4 of a leap day every century, or about three leap days every 400 years. And we do! Only one in four century years is a leap year: 2000 was a leap year, so 2100, 2200, and 2300 will not be leap years. By 2092, summer will come very early: at 1:14 a.m. on June 20, but the omission of a leap year in 2100 will mean that 100 years from now, in 2112, summer will be back at 9:14 p.m. June 20. And then in another hundred years, after the omission of a leap year in 2200, it will make it back to June 21. (I'm comparing just leap years, since summer comes a bit later on non-leap years.)
These century adjustments give our current calendar 365.2425 days per year, still a tiny bit longer than the actual 365.2422 days. This suggests that we should skip an additional leap day in about 1/.0003 years, or about 3000 years.
But there is much more to the story. The length of the year is changing! First of all, the earth's revolution is gradually slowing down, about one day every ten million years. (Then will the year become 364 days?) Amazingly enough, there is a much bigger effect in the opposite direction. Like a spinning top running down, the wobbling or precession of the earth's axis speeds up. Since our seasons are caused by whether the axis tilts towards or away from the sun, this increasing precession causes the year to speed up, currently at about one day every 167,000 years. This would cause an accumulated error in the calendar of one day after about 600 years, around the year 2600. Note that this effect is much greater than that due to the current mismatch between the calendar and the year.
There is another big effect, often overlooked: The day is getting longer, and longer days mean fewer days per year. The lengthening day requires the occasional addition of a "leap second" and the readjustment of all accurate clocks worldwide. Why is the day getting longer? Friction with the tides for example is causing the earth's rotation to slow, although there seem to be other complicating influences and this effect seems irregular and hard to measure. Ancient rocks that recorded daily lunar tides indicate that about a billion years ago the day lasted only about eighteen hours. If in a billion years the day lengthens by a factor of 4/3, the number of days in a year goes down by 3/4. In a billion years from now the year would lose about 91 days, or one day in about 11 million years. Actually, current controversial measurements suggest we are now losing about one day per 150,000 years, causing an accumulated error in the calendar of one day after about 550 years. One possible cause is the increased inertia of the earth due to huge water reservoirs behind dams. Combined with the previous effect of increasing precession, this yields a one-day error in about 400 years.
Meanwhile a controversy has raged over a peculiar definition usually used by astronomers. They mark the year by the beginning of spring, while they probably should average over all the seasons. It makes a difference because the earth's orbit is not perfectly round, but a bit elliptical. Precession has a larger effect on the summer solstice, when we are farther from the sun, and a smaller effect on the other seasons. As far as I can tell, the effects of this peculiar definition currently may be canceling out the other effects and keeping our calendar in almost perfect agreement with the tropical year, at least for the coming millennia. So nothing to worry about.