BOSTON -- Gallup, the company that has faced intense criticism after its polling consistently understated President Barack Obama's support during the 2012 campaign, shared details of its ongoing review of its election polling methods on Saturday.
Over a four-week period before the November 2012 election, Gallup's daily tracking poll showed Republican opponent Mitt Romney leading by margins ranging from 1 to 7 percentage points, including a 4-point Romney lead just 10 days before the election. Obama defeated Romney by a 51 to 47 percent margin.
Shortly after the election Gallup's editor-in-chief Frank Newport pledged to conduct an internal review of Gallup's telephone survey methodology. As promised, Newport reviewed on Saturday the kinds of studies conducted on issues including drawing samples, interviewing voters, and how to weight data and select the likely electorate. The company pledges to make the findings of its ongoing review publicly available.
Although much analysis is now complete and set to be unveiled June 4, the investigation awaits "a major experiment" in conjunction with the gubernatorial campaigns in the fall.
"We take it seriously" when polls misfire, Newport explained on Saturday. "We've been doing presidential polling since 1936 which is what put George Gallup on the map ...The results [in 2012] certainly were not what we wanted them to be from Gallup's perspective."
Newport's remarks came during a private, on-the-record briefing in Boston attended by The Huffington Post, and a handful of other pollsters and academic researchers at the annual American Association for Public Opinion Research (AAPOR) conference.
Michael Traugott, a University of Michigan political scientist and survey methodologist retained by the company for the review, said he recommended broadening the scope and timing of the inquiry.
"Early on," Traugott said, "Frank and I had a conversation, about trying to learn what we could from an investigation of the data" collected during the 2012 campaign, "but then using that information to design a series of experiments going forward."
Traugott also said he recommended reviewing all of Gallup's methods and assessing the data collected before the 2012 campaign. "We have been assembling data on presidential approval back to 2002 as a benchmark to look at the Gallup data in relations to other data and also party identification data going back."
His analysis confirms, for example, that by "a small but consistent amount," ratings of President George W. Bush "were a little bit higher than the average" of other pollsters "and the Obama ratings were a little bit lower than the average." Those findings are consistent with a Huffington Post investigation of Gallup's results published in June 2012.
Traugott also clarified that Gallup "has agreed to make all of this information publicly available." He stressed that both he and Newport, as former presidents of AAPOR "are firmly committed to transparency. And this is an ongoing part of our relationship as well."
When pressed on whether that included releasing raw respondent-level data, Newport said that although they are "still making the final decisions, we think we'll try to make the data available as well." Such data will allow other scholars and researchers to attempt to replicate Traugott's findings and test other theories. On Saturday, Newport also asked assembled AAPOR researchers for further input on a list of 20 specific parts -- virtually every aspect -- of Gallup's methodology included in the review.
While Newport declined to reveal specific findings from the review ahead of the June 4 announcement, he shared some new information.
One of the experiments, for example, checked on potential bias associated with the Gallup brand name by running a full "shadow" sample to a recent Gallup poll in which respondents were told the polster was "Selection Research Incorporated," rather than Gallup. "The Gallup name significantly increased the response rate," Newport said, but without producing differences in the demographics of the resulting sample.
Newport also explained that earlier this year, the company changed how it asks respondents to describe their race, replacing yes/no questions critiqued in last year's Huffington Post investigation with a question in which respondents could select more than one in a list of potential answers. That change allowed for a related modification to Gallup's weighting procedure. "We think that's already had some impact on our data," Newport said.
It is difficult, Traugott explained, to reconstruct Gallup's problems "retrospectively" using the data collected last fall. Instead, he convinced Gallup to conduct experiments to determine if alternative methods would have produced different and more accurate results.
Because such experiments only make sense "in the context of a campaign," Traugott explained, they decided to undertake them during the upcoming gubernatorial campaigns in Virginia and possibly New Jersey. He stressed that the experiments would only be used for further analysis, and would not be released publicly before the election.
Three academic survey methodologists are joining Traugott in the investigation: Chris Wlezien from Temple University, an expert in likely voter models; James Wagner from the University of Michigan, an authority in telephone survey operations; and Frauke Kreuter from the University of Maryland, an expert on sampling and weighting.
Shortly after the election, Newport speculated that "it is likely that we could see significantly fewer polls conducted in the 2016 election." Asked Saturday if he was signaling Gallup's intention to withdraw from polling in the next presidential race, Newport replied, "Well, check back with us in 2016. I don't know what any polling organization is doing in 2016," given the increasing challenges to traditional pollster methods.
However, despite the months of criticism, Newport was upbeat about the potential for improvements with advances in technology. He spoke favorably about a future in which "polling is augmented by non-probability data, administrative data, social media data."
"I think it is an exciting time," he said later. "Rather than doing things the same way in any business, it's always exciting when technology in particular opens up all these other opportunities."
Also on HuffPost:
2012 -- Barack Obama
U.S. President Barack Obama waves to supporters following his victory speech on election night in Chicago, Illinois on November 6, 2012. (JEWEL SAMAD/AFP/Getty Images)
2008 -- Barack Obama
Nov. 4, 2008: U.S. president-elect Barack Obama waves at his supporters during his election night victory rally at Grant Park in Chicago. (JEWEL SAMAD/AFP/Getty Images)
2004 -- George W. Bush
In this Nov. 3, 2004 file photo, President George W. Bush and first lady Laura Bush salute and wave during an election victory rally at the Ronald Reagan Building and International Trade Center in Washington. (AP Photo/Charles Dharapak, File)
2000 -- George W. Bush
U.S. Republican presidential candidate and Texas Governor George W. Bush casts his vote in Austin, Texas on November 7, 2000. (PAUL RICHARDS/AFP/Getty Images)
1996 -- Bill Clinton
President Bill Clinton, wife Hillary and daughter Chelsea wave to supporters in front of the Old State House during an election night celebration in Little Rock, Ark. on Tuesday, Nov. 5, 1996. (AP Photo/David Longstreath)
1992 -- Bill Clinton
Bill Clinton and Al Gore celebrate in Little Rock, Arkansas after winning in a landslide election on November 3, 1992. (AP Photo)
1988 -- George H. W. Bush
President-elect George Bush and his family celebrate his victory on November 8,1988 at the Brown Convention Center in Houston. (WALT FRERCK/AFP/Getty Images) <em><strong>CORRECTION:</strong> An earlier version of this slide was titled "George W. Bush." It has been fixed.</em>
1984 -- Ronald Reagan
President Ronald Reagan gives a thumbs-up to supporters at the Century Plaza Hotel in Los Angeles as he celebrates his re-election, Nov. 6, 1984, with first lady Nancy Reagan at his side. (AP Photo/File)
1980 -- Ronald Reagan
President-elect Ronald Reagan and wife Nancy wave to well-wishers on Tuesday, Nov. 4, 1980 at Century Plaza Hotel in Los Angeles after his election victory. (AP Photo)
1976 -- Jimmy Carter
Democratic presidential candidate Jimmy Carter embraces his wife Rosalynn after receiving the final news of his victory in the national general election on November 2, 1976. (Photo by Hulton Archive/Getty Images)
1972 -- Richard Nixon
U.S. President Richard M. Nixon meets at Camp David, Maryland, on November 13, 1972 to discuss the Vietnam situation with Secretary of State Henry A. Kissinger (L) and Maj. Gen. Alexander M. Haig Jr.(R), Deputy Assistant to the President for National Security Affairs. (Photo by AFP PHOTO/NATIONAL ARCHIVE/Getty Images)
1968 -- Richard Nixon
President-elect Richard M. Nixon and his wife, Pat, were a picture of joy at the Waldorf-Astoria Hotel in New York, Nov. 6, 1968, as he thanked campaign workers. At left are David Eisenhower, Julie Nixon's fiance, Julie and her sister Tricia at center. (AP Photo)
1964 -- Lyndon Johnson
President Lyndon Johnson proves he's a pretty good cowhand as he puts his horse, Lady B, through the paces of rounding up a Hereford yearling on his LBJ Ranch near Stonewall, Texas, on November 4, 1964. (AP Photo/Bill Hudson)
1960 -- John F. Kennedy
Caroline Kennedy peeps over the shoulder of her father, Senator John F. Kennedy, as he gave her a piggy-back ride November 9, 1960 at the Kennedy residence in Hyannis Port, Mass. It was the first chance president-elect Kennedy had to relax with his daughter in weeks. (AP Photo)
1956 -- Dwight D. Eisenhower
President Dwight D. Eisenhower and Vice President Richard Nixon salute cheering workers and Republicans at GOP election headquarters in Washington, November 7, 1956, after Adlai Stevenson conceded. (AP Photo)
1952 -- Dwight D. Eisenhower
President-elect Dwight Eisenhower and first lady-elect Mamie Eisenhower wave to the cheering, singing crowd in the Grand Ballroom of the Hotel Commodore in New York City on Nov. 5, 1952 after Gov. Adlai Stevenson conceded defeat. (AP Photo/Matty Zimmerman)
1948 -- Harry S. Truman
U.S. President Harry S. Truman holds up an Election Day edition of the Chicago Daily Tribune, which, based on early results, mistakenly announced "Dewey Defeats Truman" on November 4, 1948. The president told well-wishers at St. Louis' Union Station, "That is one for the books!" (AP Photo/Byron Rollins)
1944 -- Franklin D. Roosevelt
President Franklin Roosevelt greets a young admirer as he sits outside his home in Hyde Park, N.Y., on election night, November 7, 1944. Behind him stands his daughter, Mrs. Anna Roosevelt Boettinger and the first lady, Eleanor Roosevelt. (AP Photo)
1940 -- Franklin D. Roosevelt
American President Franklin Delano Roosevelt (1882 - 1945) speaking to a crowd of 25,000 at Madison Square Garden in New York on Nov. 8, 1940, before his sweeping re-election for a third term. (Photo by Fox Photos/Getty Images)
1936 -- Franklin D. Roosevelt
The Republican Governor of Kansas and presidential candidate, Alfred Landon (1887 - 1987) greeting the American President Franklin Delano Roosevelt (1882 - 1945) (seated) prior to the presidential elections. Future United States President Harry S. Truman can been seen in the background. (Photo by Keystone/Getty Images)
1932 -- Franklin D. Roosevelt
Governor Franklin D. Roosevelt of New York at his Hyde Park, N.Y. home November 6, 1932, seen at the conclusion of the arduous months of campaigning following his presidential nomination in Chicago. (AP Photo)
1928 -- Herbert Hoover
President-elect Herbert Hoover is seated at a table with wife, Lou, and joined by other family members on Nov. 9, 1928. Standing from left: Allan Hoover; son; Margaret Hoover, with husband, Herbert Hoover, Jr.,at right. Peggy Ann Hoover, daughter of Herbert Hoover Jr., sits with her grandmother. (AP Photo)
1924 -- Calvin Coolidge
U.S. President Calvin Coolidge and first lady Grace Coolidge are shown with their dog at the White House portico in Washington, D.C., on Nov. 5, 1924. (AP Photo)
1920 -- Warren Harding
Senator Warren Harding, with wife Florence and his father George, shown on Aug. 27, 1920. (AP Photo)
1916 -- Woodrow Wilson
Surrounded by crowds, President Woodrow Wilson throws out the first ball at a baseball game in Washington in this 1916 photo. (AP Photo)
1912 -- Woodrow Wilson
Woodrow Wilson (1856 - 1924), the future American president, casts his vote while Governor of New Jersey, on Nov. 14, 1912. (Photo by Topical Press Agency/Getty Images)