By Alexei Oreskovic
An ambitious plan to prevent "Fail Whales," the cartoon icon that greets frustrated Twitter users during network outages, has turned into a fail whale of its own.
A new, custom-built facility in Utah meant to house computers that power the popular messaging service by the end of 2010 has been plagued with everything from leaky roofs to insufficient power capacity, people familiar with the plans told Reuters.
The botched move threatened new product development and forced Twitter -- whose user accounts have burgeoned to 200 million in just five years -- to seek another location despite committing significant investment to the facility.
The data center move reflects growing pains faced by red hot tech start-ups as they aim to justify soaring valuations and transform into reliable channels for advertising and commerce.
Twitter said last month its data center had moved to a new home at an undisclosed location -- a feat that industry insiders say is impressive by any measure.
People familiar with the matter said the move was to an existing Sacramento facility more than 600 miles away, owned by co-location company Raging Wire, rather than into the custom-built data center in Salt Lake City, Utah.
Twitter had signed a $24 million, 4-year minimum commitment lease with C7 Data Centers which was building the Utah center. It was not immediately clear how much, if any, of the fees would be returned or if it planned to pay out the full term of the contract.
Those plans "spiraled out of control," one source said. "They were in over their heads and they did not recognize that until far too late."
Twitter declined to comment about its data centers or finances. Twitter Vice President of Engineering Michael Abbott said the company has done more to upgrade its infrastructure in the last six months than it did in the previous 4.5 years.
"Twitter now has the team and infrastructure in place to capitalize on the tremendous interest in Twitter and continue our record growth," Abbott said in an email.
Large Internet companies such as Google Inc and Yahoo spend anywhere from $50 million to $2.5 billion to construct and equip such data centers, according to Forrester Research analyst James Staten.
WATER HAZARDS
Last month Twitter hired eBay's Mazen Rawashdeh as vice president of operations -- a job that sources say didn't exist at Twitter in July 2010 when it announced plans to move its operations infrastructure.
For Twitter, which had been renting servers from hosting provider NTT America for the past several years, moving into its own data center marked a significant milestone.
Within a couple of months of Twitter's July announcement though, it was quietly moving many of its servers out of Salt Lake City and instead shipping all new gear from Sacramento, where server capacity and space was initially very tight.
Twitter concluded that the Utah facility failed to meet its needs, these sources said. The center initially lacked key features such as a second fiber network connection, and less than half of the electricity was actually available.
The roof leaked water onto the top of Twitter's server cabinets with every rain, forcing staffers to move equipment out of harm's way, sources said.
C7 President Wes Swenson declined to confirm Twitter as a customer but defended his company's work at the center. He said 5 megawatts of power were available from Day One and the building had a brand new roof once construction was over in October.
"We experience less than 1 percent churn in our customer base," Swenson said in an interview. "But sometimes we deal with companies that are not very sophisticated; often times a lot of customers that have never owned their own equipment may go through a learning curve."
Twitter's change of plans to move its data center to Sacramento affected product plans as management enforced a several month moratorium on launching new major features.
"Minor stuff was OK," said the person familiar with the matter, but "if you developed a new feature which needed 20 servers, you just weren't going to get the 20 servers."
For now, Twitter is using the remaining servers for limited roles, such as providing analytics functions for its service.
"Nobody wants to put any part of the site that's mission critical in Salt Lake City," said another source familiar with the matter. "That would be a huge battle royale inside the company if they were told they had to."
(Reporting by Alexei Oreskovic; Editing by Kenneth Li and Richard Chang)
Copyright 2011 Thomson Reuters. Click for Restrictions.