Customer Success Story
DeNA speeds provisioning 30x faster and gives developers freedom with operational control
In each location like San Francisco or Santiago, game development is organized into multiple teams. Each one is independent, self-contained with game designers, back-end engineers, and front-end engineers. Each game needs a full-stack environment. For us in the operations team, the question is always how to get environments quickly to the game teams so they can chart their creative goals and change as they proceed.
Scaling game development
In the world of game development where things change quickly, we need to launch games quickly too. Projects have several checkpoints before release. Games run separately as different entities where teams want to choose their own hardware and environment. DeNA values game developer creative freedom. That means although the architectural team reviews all the production game environment stacks, the operations team doesn’t force development teams to follow these configurations strictly. Our IT operations team in San Francisco is small. Given the volume of game titles we push, providing game environment stacks custom fitted for every project checkpoint just doesn’t scale.
Innovation without compromising governance
Also, the game teams include many junior developers who aren’t operations-savvy. We wanted a more efficient way of giving deployment freedom in the cloud without that going completely uncontrolled and unsupervised.
A better provisioning solution
Traditionally, we had used RightScale and AWS EC2. We hosted on some of our hardware nodes in our datacenters. And we continue to use Amazon Elastic Beanstalk for particular games that use PaaS. Elastic Beanstalk presented challenges. It locked us into a lot of the nuances like multiple user accounts on the machine, which is not good for audit trails and security. With RightScale, the operations team would set up the provisioning, but developers didn’t have any insight into that process. We wanted a solution that did more than spin machines up or down because AWS already subsumed this functionality in its console. What we mainly wanted was to create a self-service environment
Integrating with existing tools
Our instances are AMIs with Puppet based infrastructure configuration. We produce a vanilla Ubuntu 12.04 base image with Puppet classes on top of it. We required an easy way to build a library of scripts in a service catalog using Puppet. What we needed was to operationalize and create the blueprint so that game teams can then run them on their desktops as well as in staging or production.
Game teams have varying needs and, therefore, need different technology stacks. Our typical game environment stack runs the game code apart from the runtime libraries on Linux boxes. These run as Node.js based applications with the mobile phone application as the front-end and MongoDB as the database.
Before ElasticBox, if game teams needed a new environment, they opened a ticket with the operations team and waited a week or more to get it. When we have so many people who interact with us, we can’t possibly have everyone come through our pipeline for every need. The operations team has top priority tasks to scale systems and keep the production systems running. Our focus is heavily on improving game environments’ reliability and functionality. That’s why we need to give development teams control. Our goal was to allow game teams to self-service their environment needs quickly and also minimize the number of interactions with the small operations team in San Francisco. Now with ElasticBox, they self-service custom virtual machines and environments for test, staging, and production in 5-8 minutes. That’s 30 times faster provisioning!
Developers can readily use the parameters of a box to set up custom environments. They can enter the version of Node.js they want or enter keys to SSH into the machine. Or choose between Redis or Memcached for caching, or integrate pieces of the stack into one box. ElasticBox offers more flexibility for reuse. Game developers can modify production style environment boxes and reuse those same boxes to develop new game environments.
We don’t have the time to figure out new technologies all the time. It’s a perfect opportunity for using a product like ElasticBox. It helped us ramp up teams with our expansion in Santiago. It’s hard to create an atmosphere where people can grow while building the products, or grow from junior to senior engineers. A lot of the learning happens on their own time so creating an organic atmosphere for learning is vital. ElasticBox allows for an organic way to transfer knowledge. We can point developers to an article on how to produce boxes and say, now go out and build your technologies.
When we considered a RightScale alternative, collaboration ranked as an important consideration. In the end, the collaboration features (team workspaces and sharing) in ElasticBox helped educate the junior software engineers faster. In less than six months of signing up with ElasticBox, we had most teams adopt the selfservice model of provisioning game environments. This adoption happened organically without us mandating a push.
“ElasticBox helps the operations team stay in control of the bottom of the stack. We give game developers the keys to choosing their destinies, but we hold the skeleton key to everything.” — Jesse Sanford, Senior Systems Engineer and Lead Architect, DeNA
We were spending a quarter of our IT costs on RightScale because they charged by the number of instances. On the other hand, the per-user pricing in ElasticBox made more cost-effective sense. To forecast user cost and relate that to game developers’ success was easier to predict and measure.
DeNa is a company with a DevOps perspective. ElasticBox is a great DevOps tool for empowering developers. What we like most about it is that it’s super friendly. As an operations person, I don’t have to think much about what the developers are doing and in turn they don’t have to ask us. Rolling out self-service for easy access across independent teams was the main reason we went with ElasticBox in the end.
World-class enterprise support
Another factor was the ElasticBox team. They were very open to suggestions and made changes quickly in response to our challenges and concerns.
Policy-based control and flexiblity
ElasticBox helps the operations team stay in control of the bottom of the stack. We give game developers the keys to choosing their destinies, but we hold the skeleton key to everything. We do that with common elements like creating sudoers files, which store root privileges. Or we work with things like systemd, a Linux service manager or we lock down parts of the system not appropriate for access, and so on.