As the world knows, on July 4 it was announced that the Higgs boson, or a reasonable facsimile, has been seen by two independent experiments at CERN. The statistical significance reported was expressed as "5-sigma." Let's look at what this means.
When subatomic particles are smashed together at high energy, they create a complex mix of secondary particles. Before you can claim a new discovery in that mix, you have to show, among other things, that the effect upon which you base the claim is very unlikely to be simply a statistical artifact.
The effect reported from CERN is of a type that particle physicists have been exploring for over fifty years. Basically, they looked for evidence of a particle with such a short lifetime that it would not leave any measurable track in the detector. Instead, it decays into secondary particles after travelling only a few nuclear diameters.
When I was a graduate student at UCLA in the early 1960s, bubble chambers and other detectors at the Lawrence Radiation Laboratory in Berkeley (now Lawrence Berkeley Lab), Brookhaven National Laboratory on Long Island, and CERN in Geneva were finding signs of many such short-lived particles that had never before been seen or even anticipated. They were clearly not composed of the well-known particles such as protons, neutrons, and electrons but seemed elementary in their own right.
We experimenters did then just as the Higgs-searchers are doing now, measured the energies and momenta of all the outgoing particles produced in high-energy particle collisions. Using these data, we formed a quantity called the "invariant mass" for each of the various particle combinations. Accumulating a large number of collision events, we then looked at the statistical distribution of the various invariant masses. When an unexpected "bump" appeared above what was the expected background, we had a candidate for a new particle.
Of course, everyone wanted to discover a new elementary particle and we all got excited whenever even the smallest bump appeared. At first, it seemed that a bump of three standard deviations, that is, "3-sigma," above the background was sufficient for a discovery. At the time, most physicists were not experts in statistics (and still aren't) and this struck us as reasonable. If the statistical fluctuations were given by a normal distribution (bell curve), then in only one in every 740 times you look at such a distribution will you get a 3-sigma bump or larger from a statistical fluctuation. That is, what is called the "p-value" was 1/740 = 0.00135.
We had a simple rule of thumb that drove statistics experts crazy, and still does. If the background under the bump, estimated by looking at either side, or calculated from some model, contained N events, then sigma was set equal to the square root of N-1.
Now, here my fifty-year old memory gets hazy, and I have not been able to dig up any documentation. (If anyone has any, I would greatly appreciate getting a copy). As I recall, at first the journals were publishing 3-sigma results. But many were not being independently replicated. So, again according to what I remember, the primary physics journal for rapid publication, Physical Review Letters, asked Art Rosenfeld at Berkeley to come up with a criterion for publication. He used frequentist probability arguments, which advocates of Bayesian statistics despise but have served us particle physicists well over time.
Art counted up all the experiments being done, all the plots being looked at, all the bins on the plots, all the combinations of particles for which invariant masses were being measured, and came up with a rule that has been at least informally in use since: the probability of the bump being a statistical fluctuation must be less than 1 in 10,000. For a normal distribution, only one in 31,574 times will you get an upward statistical fluctuation of 4-sigma or greater. The observed 5-sigma fluctuation for the Higgs, or a larger one, would result only once in 3.5 million trials.
However, this method of analysis is open to question. Several observers have pointed out a flaw, which is known in the literature as "sampling to a foregone conclusion." That is, the experimenters keep collecting data until the reach the level, in this case 5-sigma, where they then can reject the null hypothesis. The proper method according to the experts is to decide ahead of time what criterion you will use and also how much data you will take before rejecting the null hypothesis. Since that is not generally done, it is technically illegitimate to interpret the result as a probability.
But it's the method we have used in particle physics for half-century and, so far, it has not resulted in any major discovery claim being later proven to be in error. Furthermore, in my experience I saw many 3-sigma bumps go away as more data were accumulated. In any case, physicists no longer leave it just at that. They perform sophisticated Monte Carlo computer simulations of the experiment using their best available models and compare results with (signal plus background) and without (background only) the assumed signal. This was a major activity of mine when I was in research.
The assumption of a normal distribution of fluctuations may not be a good one. In today's experiments, the events are cut in so many different ways that biases away from normal statistics can occur. The Monte Carlo analyses can avoid this by calculating the relative probabilities for the data fitting to signal plus background and background alone.
Of course, statistical significance is a major concern in all experimental sciences, and for a long time I have been critical of what I regard as unacceptably low publication standards used in some fields.
I still remember going to the World Skeptics Conference in Buffalo in the 1996, which featured many prominent speakers including Stephen Jay Gould and Chris Carter, the creator of X-Files. One speaker was Jessica Utts, a professor of statistics at UC Davis. She argued that the standard that was used for publication in medicine and psychology, p-value = 0.05, was adequate to show that ESP exists. She said that evidence for ESP was just as good as the evidence that aspirin helps avoid heart attacks.
I stood up and protested that this implied that one out of every twenty reports of some positive effect was a statistical fluctuation. Furthermore, since negative results are often not published, one can only wonder how many reports in these fields are trustworthy, if any.
Note the difference between an extraordinary claim (ESP) and an ordinary one (aspirin). In the case of aspirin, we can provide a simple explanation: aspirin thins the blood and makes arterial blockages less likely. We have no explanation for ESP within existing knowledge. Claims that it is supported by quantum mechanics are total nonsense.
And there's more. Here again I must rely on memory, since to my knowledge no documentation exists. When in the 1980s I was working on very high-energy gamma ray astronomy on Mt Hopkins in Arizona, using the atmospheric Cherenkov technique, a collaborator reported to the rest of us that he thought he saw a signal from a certain pulsar. We all rushed to a meeting at his university and spent the better part of a week going over the data. His original estimate of the probability that the observation was a statistical fluctuation was one in a billion. After we counted all the various combinations he had looked at, the probability dropped to one in a thousand. This would have been more than adequate for a parapsychology or other pseudoscientific journal, but not a physics or astronomy one. We didn't publish. No one else since has reported a gamma ray signal from that pulsar.
I have looked at the results just reported by the two experiments at CERN, Atlas and CMS. Both show 5-sigma signals for a range of secondary particles at a mass of 125-126 GeV. The standard model of elementary particles and forces predicted a 4.6-sigma effect at that mass, although the value of mass itself was not predicted. Not only is each individual result significant, rejecting the null hypothesis at a probability of one in over three million, the fact that two independent experiments agree surely makes the case for a previously unknown particle at 125 GeV proven beyond a reasonable doubt. Whether it is the long sought-after Higgs boson or a composite of known particles is yet to be definitively established.
Your righteous indignation and personal incredulity are not on any level valid arguments, scientific or otherwise.
Disliking something does not mean you know nothing about it.
Electronics technicians and engineers, even someone at home in electrodynamics, should understand; that the model of the particle physicists is more advanced, and thus complete... than yours.
Yeah it is.
“Classical electrodynamics is, for all practical purposes, complete for the technology level.”
I'm speaking of our various and sundry models of the universe, the particle physicist's, is more complete than that used in electrodynamics; which thus lags in its conceptions of the universe. And though I speak of models, I would ask you to define what the “technological level.” meant, even if I wasn't.
“You will not be able to find any shred of quantum effects at 1kHz, 1MHz or even 1GHz with electronic means, alone. Unless you can couple to atomic physics, good old Maxwell is enough.”
Your source charge, comes from... where? What's not quantum about it?
And sometimes people mean Maxwell but Maxwell is, or rather was... 20 quaternion and quaternion-like equations in twenty unknowns. After Maxwell himself watered them down (by the time of his death about 80% of the original was gone); the priesthood wasn't ready for the higher group symmetry of quaternions, it seems.
Then after his death Heavyside, Hertz and Gibbs, with re-gauging by Lorentz “simplified” Maxwell even further. The quaternion equations were, in a final act of mutilation... recast in the (newly formed; the priests can move when they want...) lower symmetry, vector algebra.
I'd bet dollars to doughnuts that's what you mean(t) by Maxwell.
These concepts of statistical certainty carry to all parts of life, such as deciding how much benefit one gets from a preventative nutrient or deciding what kind and degree of cancer treatment one decides to undergo. Numbers mean nothing in isolation in these cases, but the relationships between them mean everything.
The most important concept I carry from college chemistry is that of significant digits, or exactly what degree of numerical accuracy is relevant? These brainiacs live in this world. Be ready for a verbose answer when you ask about anything, "Are you sure?"
My quest to understand continues...
The media attention given to this story is great for sparking interest in science in general, might even result in a few more young people exploring to option of a science career.
Secondly, regarding significance: in my field of ecology, data are often messy with large variation. Significant results are great, but it is usually necessary to run several independent tests if possible to see if results are consistent, results must make biological sense, and theory should usually be consistent with results. Also, the methods used for testing data must be appropriate (reviewers jump all over this), and any assumptions of the testing methods must be addressed to take into account possible biases.
For these reasons, a significance threshold of 0.05 is standard.
Ecology is simply orders of magnitude more complicated than physics and it is much more difficult to design clear tests of phenomena.
Not more complicated than high energy physics. Maybe more convoluted. But I am not even sure about that. When was the last time ecologists spent 28 years and $10 billion with collaborations of tens of thousands of people on a single ecosystem, after having investigated the very same ecosystem with similar efforts for the sixty years before that?
As for your suggestions, all of this, and much more, is being done in physics. Much more, actually. We simulate our detectors down to the wire (literally). Theoreticians have thought virtually every key aspect of these experiments thousands of times over.
either that, or you're talking to an alien from another planet, in which case you should probably punch it in the face while yelling, "welcome to earf!" or something equally catchphrase-worthy.
It's interesting but incomplete. We only have part of the elephant, one day someone will follow in Higg's footsteps, take a different feel of the "mammal," and we will then have a more complete understanding.
Also is a Homing Pigeon, "psychic?" That faculty in a human becomes (after interaction with human intelligence) what? More nerve dendrite (according to pigeons) than quantum mechanics.
Rigorously though; accelerators, microscopes (no resolution smaller than half wavelength of the light used for observation), and telescopes are all means of extending our senses, and its only because we are at the limit of those, means (for now...) that the use of statistics is so, prevalent, lets say.
Perhaps different metaphors contexts (non esp, at the moment..), for looking into those realms,would be fruitful.
I understand.
So, half wavelength of light? No, not that? Or looking at the need for greater resolution in our instruments of measurement and indeed extended perception? Mk the pigeon thingy; homing pigeons don't, “home?”
I always say that I don't debate ignorance, so... And who cares, anyway? There are more important things, batter fish to fry.
See that about as far as one can take the conversation; some people go deeper into the bench some less so. It could have been interesting, well, lol, more, interesting... one must always be graced or damned it seems to hope for such when a reply happens.
“Whacky” means (at the least) that I am on point as it were., so... thank you (…::bowing::...), in any case.
My perception of scientific advance is that we have periods of information gathering followed by consolidation by people like Newton, Maxwell and Einstein. It took centuries of observation to get from Eratosthenes to Newton, maybe not so long to go from alchemists to Maxwell.
Today we have more people working, budgets and instruments, but the periods of gathering and recognition are still there, even if shorter. Now we also have new tools, both physical instruments such as colliders but also the tools of math such as the FFT and statistics, enabled by the data processing power of computing.
So we stand on a new threshold today with the Higgs, maybe several others in other fields as well. The data is coming in so fast that we have more data than people who can interpret it - astronomy, particle physics, biology - what a ride. My background is physics and computing, applied to instruments and controls. But if I were a youngster today, I would be drawn to biology and neurology. We have some tools now to open the door to really complex phenomena like evolution and the brain.
Hopefully today's tools will soon be as laughable as Galileo's telescope, but look what Galileo caused when he recognized what he saw. Now we see the Higgs. Can we recognize what it means?