ESPN.com's Fantasy Games servers finally came off the disabled list Thursday morning, after four days on the sidelines.
It was later than promised, though. On Wednesday, a top ESPN Internet Ventures official promised that ESPN.com's fantasy baseball and golf games would be back online before Thursday. But the entire Fantasy Games index page was downsite was still offline as of 8:15 AM PT, before finally coming back online.
"We are literally rebuilding database tables here as we speak," said Geoff Reiss, senior vice president of ESPN Internet ventures in Bristol, Conn., in an interview with ZDNN on Wednesday.
Both ESPN.com's fantasy baseball and fantasy golf leagues have been down since Sunday morning, when the system crashed. Its NASCAR fantasy leagues were unaffected, as were other portions of ESPN.com.
Reiss said that both leagues would be up and running before 9 PM PT, but an update posted on the site after the interview said that loading data on a production server was taking longer than expected, so the site would now be up no later than 10 PM PT.
ESPN.com has updated the Fantasy sports several times today, in marked contrast to its behavior during the first few days of the outage, which began early Sunday morning. Until Wednesday, ESPN.com had simply posted a message saying that they were sorry the site was down.
Reiss acknowledged that ESPN.com was caught flatfooted by the crash, which occurred during a server update. He didn't provide details, other than to say that "fundamental combination of hardware and software failures" during a SQL Server update.
"We don't have a lot of DEF CON 4 messaging scramble plans out there," he said. "We generally speaking have had a high enough amount of confidence in our system that we weren't terribly well prepared for this."
Reiss said ESPN.com had suffered worse events on the fantasy front, but not since it began fantasy sports with a fantasy football league in 1995 that was "a shockingly bad product."
'Wishful thinking'
A grim Reiss said ESPN.com was probably guilty of wishful thinking in the early stages of the crash. After almost four days, he said the staff on the fantasy site was demoralized.
"When something like this happens, you get a large group of people walking around like their dogs have died," he said.
"I think we were probably thinking more from a wishful perspective than a purely analytical one," he said. "You know, if Lucy can find a way not to tell Ricky she smashed up the car, she'll find a way not to do it. I think early on there were hopes that we were going to get this behind us a little bit quicker."
Lucky break
As for the impact, Reiss said that traffic loss could have been much worse.
"Fantasy-driven traffic on ESPN.com is going to hit an ebb during the Monday-Tuesday-Wednesday of the All- Star break anyways, but it hasn't had a zero impact," he said. Reiss said the site was fortunate that the upgrade happened to come during Major League Baseball's annual All-Star hiatus, when no regular games are played.
The bigger potential loss is in customers not signing up for future ESPN.com fantasy leagues. Leagues are made up of 10 members, each of whom pay between $19.95 and $29.95 for teams. One customer, Mike DiLorenzo, told ZDNN on Tuesday his league had voted to find a different host site for next baseball season.
But Reiss said in general, ESPN.com had received "a surprisingly modest amount of feedback," much of it seeking information more than expressing outrage. That could change as members get access to their leagues again, only to find that many of their recent transactions went unrecorded. Reiss said ESPN.com was not sure how many player trades and other movements could be recreated.
Reparations likely
Another customer e-mailed ZDNN a copy of the note he had sent to ESPN.com, which warned of a switch if service did not improve.
Reiss said the site was taking such notes seriously. "The only positive to take out of an event like this is that it potentially shakes some what might have been complacent approaches to what are still some really high-wire operations," he said. "If we don't run a pretty damn tight service for the rest of the season."
He said ESPN would make some kind of reparation to affected fantasy sports players, though he declined to specify these, saying he preferred users to find out from the site. Sources close to the company said that users would probably be given complimentary access to ESPN.com's post season baseball fantasy challenge.
