devopsy.kinsta.cloud

Author: Tony Bradley

  • Netflix, the Simian Army, and the culture of freedom and responsibility

    Netflix, the Simian Army, and the culture of freedom and responsibility

    What do you do if you’re entire business relies on delivering streaming video content over the Web to millions of customers, and there are no examples to follow or tools to use? If you’re Netflix, you set the example and build the tools.

    Netflix had tremendous success with its original business model shipping movies on DVD to customers in trademark red envelopes. The precision of the shipping logistics, and the efficiency with which Netflix was able to meet user demand and ensure speedy delivery is a case study of its own. But, Netflix also saw the writing on the wall and became one of the first to embrace delivering movies via streaming video over the Internet.

    Networking protocols are fairly resilient, and able to get data from Point A to Point B eventually, but streaming content like audio or video is less forgiving. The data needs to be transferred at high speed, and the packets need to arrive in order to avoid buffering or glitchy playback.

    Netflix chose Amazon Web Services (AWS) to provide the cloud server infrastructure for delivering its streaming content. Amazon is reliable and resilient, but it is not infallible. When Amazon experiences a massive outage—as it has a few times in the past—it can mean down time for Netflix customers. Netflix, however, works very hard to guard against being affected by issues like that, and—thanks to the Simian Army—it is quite successful.

    Simian Army

    I spoke recently with Ruslan Meshenberg, director of platform engineering and Josh Evans, director of operations engineering for Netflix to learn more about how Netflix has embraced both DevOps and open source to develop tools that help it deliver high quality streaming video content to customers around the world.

    Simian Army is the name Netflix has given its suite of automated tools engineered to stress test the Netflix infrastructure and proactively identify weaknesses so Netflix can resolve them. The Simian Army is comprised of tools like ChaosMonkey, ChaosGorilla, and ChaosKong.

    Josh explained that when you’re dealing with a massively distributed system like the Netflix cloud infrastructure, it’s inevitable that you will encounter bugs. Rather than waiting for issues to surface, Netflix uses the Simian Army to intentionally induce failure on its own terms, so that issues can be identified and fixed before they impact customers.

    ChaosMonkey wreaks havoc on an individual server or cluster level—randomly shutting down servers to make sure that automated resilience works, and that redundant systems pick up the slack so content delivery is not interrupted. ChaosGorilla works on a larger scale by taking out entire AWS zones, and ChaosKong takes the concept to a national level by randomly shutting down the East Coast or West Coast regions of AWS to ensure traffic is automatically redirected to the remaining region.

    There are also other tools in the Simian Army. JanitorMonkey keeps everything tidy by cleaning up unused EC2 resources. As the name implies, its job is to automatically clean up after developers. Then there’s ConformityMonkey, which checks for EC2 instances that are not conforming to predefined rules for best practices.

    Netflix lets the Simian Army roam free causing random chaos, but only Monday through Friday between the hours of 9am and 3pm when managers and developers are present to address any urgent situations that might be caused inadvertently. In some cases the tools take automated corrective action, and in some they simply generate an alert and escalate the issue to the appropriate group or individual.

    Embracing Open Source

    One of the things that makes the Simian Army unique is that Netflix created these tools out of necessity. Netflix was blazing a trail on the cutting edge of delivering content from the cloud, and there simply were no commercial tools available to do the things Netflix needed to do.

    In the early stages of scaling cloud services to meet demand, Netflix frequently stumbled and hit a variety of walls. Netflix was a pioneer in delivering content through the cloud at this scale, but the open source community has experience, and an established understanding of working with a distributed environment.

    By marshaling the resources of the open source community, Netflix was able to get more eyes on the code, and enlist the volunteer support of hundreds or thousands of developers to help refine and improve the tools in the Simian Army, and other tools Netflix has developed. Ruslan and Josh also explained that there is an unintended benefit of cooperating with the open source community: in-house developers are more diligent when they know that the code they write will be exposed to the general public.

    Netflix doesn’t just take advantage of the open source community; it also makes its tools available to the general public through the Netflix Open Source Software Center on Github. There are tools for availability (like the Simian Army), cloud management, persistent systems, infrastructure services, developer productivity, and more—all available for free for anyone who wishes to use them.

    Culture of Freedom and Responsibility

    Lots of organizations talk a good game. It’s easy to come up with a profound mission statement, or an insightful set of company values. Actually living up to them, however, is a whole different story. Netflix prides itself on building a culture of freedom and responsibility.

    The Netflix culture is part of what makes the company so successful, and why Netflix has been able to achieve what it has in the realms of DevOps and open source. Netflix believes in finding highly skilled, seasoned engineers and developers, and allowing them the autonomy to get their jobs done and own responsibility for what they produce.

    Other organizations can learn a lot from Netflix. Netflix has paved new paths for DevOps, set the bar for collaborating with the open source community, and established a culture of success that any company could benefit from adopting.

  • Let the Code do the Heavy Lifting

    Let the Code do the Heavy Lifting

    One of the first jobs I had when I started in IT was as the IT administrator for a dot.com startup. IT admin at a smaller company like that was typically an umbrella term used to describe a one-man IT department, which is essentially what I was.

    The company was growing exponentially—both in terms of customers and revenue, as well as traffic to our website. What that meant for me was a steady stream of UPS shipments from Dell, and a virtually full-time job of mounting servers on racks in the data center, getting them properly configured, and deploying them to the network.

    Dealing with 50 or 100 hundred servers in 2000 was a daunting task, but dealing with hundreds or thousands of virtual servers in 2014 is—for all practical purposes—impossible. That is, it would be all but impossible doing things the old-fashioned way. However, the DevOps revolution changes the rules of the game because it uses code to do the heavy lifting, and automate those tasks which can be automated so the IT admins of today are free to engage in more productive tasks.

    If we were still dealing in physical servers, the benefits of automation that you get in a DevOps environment would still not be possible. What allows so much of the traditional “racking and stacking” today to be automated is that we are dealing primarily with virtual servers. Although a virtual server may act like a server, and fulfill the same duties as a physical server, it is essentially just a software program. The fact that a virtual server is comprised of nothing more than code means that virtual servers can be created, configured, and deployed automatically through code as well.

    It’s a good thing, too. The fast pace of a DevOps culture would be challenging to maintain without the efficiency of automation. The ability to create, provision, destroy, or move virtual servers around on a whim is part of what facilitates agile development and accelerated execution by DevOps teams.

    There are also some adverse consequences of automation. People have lost their jobs in manufacturing plants as machines have come along to automate tasks they once performed. There is a risk that automating many of the routine tasks normally delegated to IT personnel could result in fewer people being necessary, but that is not a reason to not embrace automation. Automation enables people to develop more advanced skills and focus on more important tasks—those functions that can’t be easily automated.

    One of the primary aspects of functioning in a DevOps environment is a faster development cycle, and more frequent deployments. Traditional organizations might deploy updates once a quarter, or once a month, but DevOps teams often deploy updates at least daily—sometimes multiple times per day. Automation reduces the possibility of error—it is a simple, repeatable process that works reliably and predictably.

    A survey of DevOps Conference attendees published by JumpCloud and SoftLayer found that about 75 percent of the respondents use a configuration automation tool such as Puppet or Chef. More than half use scripts of some sort to execute tasks, and about one-third reported using a patching tool of some sort. Many of those surveyed admitted that they have the tools they need, but there simply isn’t enough time to get things done.

    Perhaps that explains why 82 percent of survey respondents plan to automate more. Breaking the data down a little farther, nearly a third are looking to increase automation of configuration and deployment, and about 10 percent want to automate user management, and a variety of tasks like log management, scripting, testing, patching, and monitoring all came in around five percent or below.

    It’s still possible to create virtual servers, and configure them manually one at a time. That may work if you’re bored, or nostalgic for the “good ole days” of IT, but it’s impractical, and frankly just dumb. It’s also possible to cut your lawn with a pair of scissors, or paint your walls with a Q-tip, but it is highly inefficient and ill-advised.

    Don’t waste time and resources performing tasks that can be automated. Identify the functions you routinely perform that are consistent and repeatable, and embrace automation to take care of those things for you so you can focus on more important things.