As IT pros, we often feel like we’re racing from one problem to another or we never have enough time to deal with all the demands of the job. So it feels great when efforts pay off and .
One such instance took place when I was working for a company that produced free-to-play browser games for large numbers of players. I was part of a team that constructed a fictitious world where players could roam around continents and oceans and come across many quests to solve. They could also battle each other or band together to go on quests.
Several times each week, our team released updates to this gaming world, so each working day was a busy race toward the next due date.
Our players were very vocal about changes, whether they liked them or not, and after the updates were released, the in-game chats and forums would be especially busy.
: load tests, renting more cloud servers, networking with CDN suppliers, and even borrowing servers from other games (because our supplier would’ve taken too long to deliver new ones).
All the while, the players bragged on the forums how they would get back at us for the rule changes.
Eventually, the appointed time for the protest rolled around and we’d be able to see in real time if our preparations would work out. Our team gathered around as one of us logged into the game and went to the place where the protesters had agreed to meet: a deserted void in the center of the world. On another floor, the devops team watche server performance even more closely than usual.
The moment of truth
The appointed time came, and within seconds the place was occupied by tens of thousands of protesters. We could tell from statistics that some protesters had even been playing the game for days and nights on end to have their game avatar traverse seas and continents in order to make it. Very quickly, the entire screen turned to pitch black, as the software wasn’t able to render all those characters individually. The in-game chat was flooded with so many protest messages that it flew by in a blur.
Then something happened that no one had expected: The massive gathering attracted more players who arrived on scene. They had been roaming the world as usual, looking to challenge, battle, and plunder. Immediately, they attacked the protesters. The protesters, caught up in the action and not able to tell friend from foe, hit back.
A massive battle broke out—the largest ever witnessed in this game. Within moments, the screen turned white with smoke and dust, then black again with defeated avatars decaying into the ground and into eventual oblivion. The avatars left standing quickly fled the scene.
Shortly after it had started, the protest was over and had vanished. The in-game chat turned to silence as the players went quiet from the shock of what had happened.
The onlooking team members were also shocked—faces turned pale and jaws dropped to the ground. Then someone reached for a phone and called devops. Our preparations had paid off, and our system had balanced the server load so well that it had easily handled both the expected protest and the unexpected battle without any problems ensuing.
We went back to our regular duties. And as time would show, that was the end of anybody debating that particular rule change.
The lesson learned? It will pay in time to have your proverbial tech haystacks tied down thoroughly as there’s always a storm coming—though when and where nobody can tell in advance. And when the solid preparations can be beefed up to handle a known, tougher scenario, it’s very satisfying when the efforts pay off.