devopsy.kinsta.cloud

Tag: Netflix

  • Netflix, the Simian Army, and the culture of freedom and responsibility

    Netflix, the Simian Army, and the culture of freedom and responsibility

    What do you do if you’re entire business relies on delivering streaming video content over the Web to millions of customers, and there are no examples to follow or tools to use? If you’re Netflix, you set the example and build the tools.

    Netflix had tremendous success with its original business model shipping movies on DVD to customers in trademark red envelopes. The precision of the shipping logistics, and the efficiency with which Netflix was able to meet user demand and ensure speedy delivery is a case study of its own. But, Netflix also saw the writing on the wall and became one of the first to embrace delivering movies via streaming video over the Internet.

    Networking protocols are fairly resilient, and able to get data from Point A to Point B eventually, but streaming content like audio or video is less forgiving. The data needs to be transferred at high speed, and the packets need to arrive in order to avoid buffering or glitchy playback.

    Netflix chose Amazon Web Services (AWS) to provide the cloud server infrastructure for delivering its streaming content. Amazon is reliable and resilient, but it is not infallible. When Amazon experiences a massive outage—as it has a few times in the past—it can mean down time for Netflix customers. Netflix, however, works very hard to guard against being affected by issues like that, and—thanks to the Simian Army—it is quite successful.

    Simian Army

    I spoke recently with Ruslan Meshenberg, director of platform engineering and Josh Evans, director of operations engineering for Netflix to learn more about how Netflix has embraced both DevOps and open source to develop tools that help it deliver high quality streaming video content to customers around the world.

    Simian Army is the name Netflix has given its suite of automated tools engineered to stress test the Netflix infrastructure and proactively identify weaknesses so Netflix can resolve them. The Simian Army is comprised of tools like ChaosMonkey, ChaosGorilla, and ChaosKong.

    Josh explained that when you’re dealing with a massively distributed system like the Netflix cloud infrastructure, it’s inevitable that you will encounter bugs. Rather than waiting for issues to surface, Netflix uses the Simian Army to intentionally induce failure on its own terms, so that issues can be identified and fixed before they impact customers.

    ChaosMonkey wreaks havoc on an individual server or cluster level—randomly shutting down servers to make sure that automated resilience works, and that redundant systems pick up the slack so content delivery is not interrupted. ChaosGorilla works on a larger scale by taking out entire AWS zones, and ChaosKong takes the concept to a national level by randomly shutting down the East Coast or West Coast regions of AWS to ensure traffic is automatically redirected to the remaining region.

    There are also other tools in the Simian Army. JanitorMonkey keeps everything tidy by cleaning up unused EC2 resources. As the name implies, its job is to automatically clean up after developers. Then there’s ConformityMonkey, which checks for EC2 instances that are not conforming to predefined rules for best practices.

    Netflix lets the Simian Army roam free causing random chaos, but only Monday through Friday between the hours of 9am and 3pm when managers and developers are present to address any urgent situations that might be caused inadvertently. In some cases the tools take automated corrective action, and in some they simply generate an alert and escalate the issue to the appropriate group or individual.

    Embracing Open Source

    One of the things that makes the Simian Army unique is that Netflix created these tools out of necessity. Netflix was blazing a trail on the cutting edge of delivering content from the cloud, and there simply were no commercial tools available to do the things Netflix needed to do.

    In the early stages of scaling cloud services to meet demand, Netflix frequently stumbled and hit a variety of walls. Netflix was a pioneer in delivering content through the cloud at this scale, but the open source community has experience, and an established understanding of working with a distributed environment.

    By marshaling the resources of the open source community, Netflix was able to get more eyes on the code, and enlist the volunteer support of hundreds or thousands of developers to help refine and improve the tools in the Simian Army, and other tools Netflix has developed. Ruslan and Josh also explained that there is an unintended benefit of cooperating with the open source community: in-house developers are more diligent when they know that the code they write will be exposed to the general public.

    Netflix doesn’t just take advantage of the open source community; it also makes its tools available to the general public through the Netflix Open Source Software Center on Github. There are tools for availability (like the Simian Army), cloud management, persistent systems, infrastructure services, developer productivity, and more—all available for free for anyone who wishes to use them.

    Culture of Freedom and Responsibility

    Lots of organizations talk a good game. It’s easy to come up with a profound mission statement, or an insightful set of company values. Actually living up to them, however, is a whole different story. Netflix prides itself on building a culture of freedom and responsibility.

    The Netflix culture is part of what makes the company so successful, and why Netflix has been able to achieve what it has in the realms of DevOps and open source. Netflix believes in finding highly skilled, seasoned engineers and developers, and allowing them the autonomy to get their jobs done and own responsibility for what they produce.

    Other organizations can learn a lot from Netflix. Netflix has paved new paths for DevOps, set the bar for collaborating with the open source community, and established a culture of success that any company could benefit from adopting.

  • Step One Of DevOps Ignores Technology

    Step One Of DevOps Ignores Technology

    I’ve spent a bunch of time talking with Adrian Cockcroft, former cloud tzar at Neflix, and listening to his various talks and presentations. I’ve also spent a bunch of time looking around the various rooms within which he presents and seen the dazed look of attendees who have rightly assumed that just seen a picture of the way organizations should be. Netflix is unquestionably an exemplar for IT reinvented – where innovation, agility and developer enablement are natural parts of what they do.

    But then I’ve seen these new disciples of the “Netflix way” write a list of the technologies that Cockcroft detailed and attempt to shoehorn those same approaches into their own organization. All to often the result is disappointment, failed projects and a swing back to the traditional rigid ways of doing things.

    These thoughts struck me then I hear about this new masthead and its aim of creating a hub for DevLops thought leaders – surely giving IT practitioners a good place to find the latest tools and techniques will help spread the DevOps gospel?

    Well yes, and no. You see I’m a firm believer in the viewpoint that DevOps has very little to do with products, processes and policies, and instead is a direct result of a particular type of culture. On the one hand you have Netflix, clearly a cultural outlier with its openness, willingness to experiment, embracing of failure-driven iteration and aim of planning for failure. On the other hand you have traditional IT shops that are top-down, driven by a rigid policy framework and consider the “operations department” and the “developer team” as two distinct entities.

    This was a topic that Christian Reilly (he of the sartorial elegance and funny accent tendencies) and I discussed a couple of years ago, before the DevOps hype had really started to hit. In the video, he reflected directly on his organization, Bechtel, as an outlier at one end of the spectrum, while Netflix is an outlier at the other. It’s a somewhat dated video, and talks about public/private cloud, but the themes in terms of organizational readiness hold true for DevOps as well.

    [youtube http://www.youtube.com/watch?v=gVG1Pj4cW-g&hd=1]

    So my aim in my regularly irregular DevOps column in here is to talk about everything but the technology. And where better to start that the first step towards DevOps which is, I believe, all about building cultural bridges rather than making technology choices. It should be a natural concluding that, since DevOps is all about bridging the developer/operations span, that perhaps getting people in a room to talk about their particular drivers and metrics would be a good idea. But alas most people go from talking about DevOps to deploying their choice of automation tool.

    There’s so many things to think about in embarking on this process – over on the ScriptRock blog Alan points out that visibility is a key first step before automation is embarked upon. While I agree with him that visibility is key, for me the money quote comes in his post where he writes:

    Don’t automate what you don’t understand

    But to then suggest that the first step should instead be a visibility tool does little to improve the outcomes – unless both sides fo the organizational divide are in the same room and accepting a common lingua franca and viewpoint on the data – visibility achieves little. So takeaway #1 focus on the culture. There’s lots of ways to build the bridges needed to change the way an organization looks – I’m going to cover some of that stuff over time.