In the belly of the beast: The PokerStars Data Center
It is arguably the most important building in all of online poker. Hidden away behind walls of bleachers built to watch the TT motorcycle race, tucked on the other side of an ancient cemetery, behind a nondescript but foreboding green gate sits the PokerStars Data Center. Inside, behind the kind of security reserved for nuclear warheads and heads of state, sit the machines that deal every hand you get on PokerStars. Nearby sit hundreds of hard drives that hold the history of every hand of poker ever dealt by PokerStars' ethereal dealers.
There is also a fastidious Manx man who has very particular ideas about how to run hundreds of kilometers of cables and another guy who takes a particular amount of joy in describing the sheer power of the explosions and fires that could erupt in the complex. It's a science fiction movie made all the more off-putting by the fact that it's all real.
I am let in only after my passport has been checked, my chest outfitted with a badge, and the direst of instructions given by Manx Telecom's Data Center Director Stuart Paul.
"Please don't touch anything," he says.
Paul is a well-dressed man with a slick head and cautious gaze, but behind his eyes is lit a twinkle. He's almost childlike in his excitement at the sheer technological marvel he's showing me.
"This site is a bit like a cruise liner," he says with glee, "but we can never be in dry dock, and we have to do maintenance at sea."
Everywhere we walk, cameras watch us. Quiet, stoic engineers sit in the Operations Room glancing back and forth between eight large television screens on the walls and the laptops in front of them. The screens show the status of everything happening in the building. They also show a view of every camera on the property. If a ninja tried to get in the building, these guys would stop it from happening. I'm convinced of that, despite never having met a ninja. These guys are that serious.
Before long, we reach a room where two walls are full of nothing but batteries. There are dozens of them. They cost 500 pounds apiece, and they only last for four years before they have to be replaced. What's more, they are dangerous.
Paul leans in. "If you touch two together, you lose your eyesight."
I can't help but think this fact makes him very happy.
Inside the PokerStars shuffle
If that all sounds a little more romantic that you'd expect a data center to be, you'll have to forgive me, because for the two hours I was in the building, I was geeked out as much as your average guy can be. Why? Because, having known and loved PokerStars for the better part of a decade, I was literally standing in the place where the magic happens. The magician was none other than Gary Hill.
"When players connect to PokerStars," Hill says, "that game is actually running right here on one of these servers."
Six years ago, PokerStars recruited the Isle of Man native from a local telecom company. He's since overseen the monstrous task of building and maintaining the technology required to keep PokerStars running. Every bit of the infrastructure for the game play technology, security technology, and hand storage falls under his watch.
Oh, and the random number generator that controls the PokerStars shuffle? We're standing within spitting distance of it, although I'm sure if I spit, an armed guard will appear from the floor and detain me.
I wanted to know how the cards get decided. Hill surprised me. It remarkably involves mirrors.
"It fires photons at a small mirror, and the direction of those photons reflecting off the mirror is actually what decides what cards the player gets. That all happens in this room," he said.
Without putting too fine a point on it, I sort of want to hug Hill and ask him to fix my iPhone. Instead, we move on. There is a lot to see. What's more, I've leaned against a machine, and it's clearly making Hill nervous.
Keeping the lights on
Everywhere I look, there are lights. The important thing to remember is that if someone ever turns out the lights here, the world as you know it on PokerStars ceases to exist. That's why Hill and Paul have made sure it's virtually impossible for the power to ever go out. That in itself is a task, because it takes a lot of electricity to run the world's biggest online poker site. At any given time, the data center is pulling enough energy to power 70 to 120 homes running with all their lights on. (And if you're an advocate for green energy like me, you can take heart that the center is using and developing some very environmentally clean ways of running).
But the point of all of this is, despite the notoriously bad Isle of Man weather, there can never be a loss of power here. And that's where redundancy comes in.
"Everything you see, there's two of it," Paul confides.
And he's not kidding. Every air conditioning system has a back up air conditioning system. In the room where high voltage currents are turned into low voltage currents, there are huge circuit breakers. And then there are duplicates of those.
But all of those things are dependent on the power coming in the building, and I want to know what happens when the lights go out in Douglas.
That's where the UPS comes in. It's what Paul calls his "box of tricks." Inside that box is a super duper computer than can immediately sense if the power is going out. The UPS (Uninterrupted Power Supply) detects it and sends word to the generators to get to work. Those generators are the beasts in the machine.
In one room, two of them sit (yes, there is another room with two more). They are gigantic diesel engines that make 200dBs of noise when both are running. They have a giant muffler as big as a room that makes sure the neighbors barely hear a sound. With the fuel they have on hand, they can power the center for 72 hours. If that gas runs out, there is a special fuel line that can be tapped to keep things on indefinitely.
As much as I am impressed by the sheer size of the engines (seriously, we could fit the entire blog team inside one if it were hollow), I'm wary of things like this. I mean, if you have ever tried to start any kind of engine, you know there are days that they can be temperamental. Imagine that happening when the fate of PokerStars hangs in the balance.
Well, of course there is a back-up.
That is the battery room (and, of course, its duplicate room) where in the event the generators take a little too long firing up, the batteries can keep things going for 15 minutes. It doesn't seem like a lot of time, but it's enough to make sure everything stays afloat until the big mama engines get running.
I think back to Paul's candid admission about those batteries.
"They're very dangerous," he had said.
If they explode, the PokerStars servers could suffer a fate far worse than a minute without power. What happens then?
Paul smiles as if I'm a child.
"We have a VESDA," he says.
That's a Very Early Warning Aspirating Smoke Detection machine. It can sniff the finest of dust particles from the air to determine if something is amiss. Put another way, it knows if there is a fire in the room before the fire knows there is a fire in the room.
And if there is?
Argonite. That's what. Big red tanks of it that will release a gas that sucks all the oxygen out of the room and snuffs out the fire.
Gary Hill lives by a simple motto: "The site just cannot go down."
The only thing left to chance at the PokerStars Data Center is how you decide to play your inside straight draw.
When the tour is over, I don't really want it to be. I feel like I've seen something special. The people who work inside are immensely proud of what they do and how they do it. PokerStars' Gary Hill speaks about the data center like he would a child.
"I can honestly say, hand on heart, the facility that we have here on the Isle of Man, it beats all the data centers that I've seen in the UK and Europe in terms in the amount of investment in infrastructure."
In the end, all the work that happens inside is with one goal: Players get to play their poker game in a secure, fair, and protected environment. It happens with such regularity that you might take it for granted. That's exactly how Gary Hill and PokerStars want it.
I'm taken to the door where cameras watch me prepare to leave and each room through which I pass has a lock clicked behind me. I make for the exit when an employee stops me, almost embarrassed. It must be done.
"Your badge, sir," he says.
I take it off, give it to him, and watch him store it away.
As you might expect, he puts it somewhere safe.
Brad Willis is the PokerStars Head of Blogging