devopsy.kinsta.cloud

Author: Evan Powell

Enough monitoring. Act!
Or How I learned to stop worrying and love the auto-remediation

These days StackStorm – and other approaches to auto-remediation and event driven automation more generally – are getting broadly adopted.

Everyone from Facebook through Netflix and WebEx (yeah users!) are not just better “managing” their 2am pages (although that’s cool too PagerDuty) – they are exterminating them.

And as they adopt StackStorm for remediation inevitably a discussion emerges about monitoring vs. fixing. Why?

At some level it is because of the excellent work that has been done recently in response to monitoring sucks. Thanks Sensu, thanks SignalFX, NewRelic, DataDog, AppDynamic, Splunk, the entire ELK stack, and the other >150 monitoring solutions I’ve left out.

But at another level it is because of OODA tells us that all that observing and orienting is only as good as the decisions and actions you take.

The thing is, even though folks gradually are coming to realize that good event-driven automation INCLUDES the D in it – and does not purely rely on the smarts of monitoring since with event driven automation it is possible to run a routine that validates events before acting upon them, most people still miss the fundamental insight of ODDA. And it is an insight that is crucial especially if you, like those users above and many folks in the world of active cyber-defense, are going hand to hand with smart actors and otherwise are in a world in which what goes wrong is changing.

Here is that perhaps non-intuitive insight that, again, I’ve learned from folks that live the OODA against tricky actors (human and the entropy of scale and creeping complexity) everyday:

Make your false positives and the creation and control of your actions cheap and you can move fast to fix, way before a root cause analysis (RCA) would typically have completed.

Some of the tips that even the wikipedia on ODDA repeats for actual war fighters utilizing the ODDA is “embrace the chaos” – don’t wait to know what is happening before your act because the pace of your actions is itself a key determinate in whether you triumph or not.

One way to think of it is that you have a budget from O through A and the total of those steps is your Mean Time To Recover (MTTR).

If you spend all of your MTTR budget on O and O then your D and A will be too little, too late.

Returning to the world of infrastructure and DevOps – here are some drivers pushing us all to focus more on the D and the A and to obsess less over the O and the other O:
- Docker and containers and cloud itself. Now compute can be ephemeral (sometimes). Same deal with lots else that can be software defined. Even storage! (hi Nexenta ;))
- Imperative stuff exposed via APIs and declarative systems. Yes, this is a StackStorm plug. U R surprise? Check it – when Netflix knows that there is a happy API there that will do the care and feeding of their Cassandra reliably even when AWS is doing an AWS entropy dance, they can obsess slightly less over RCA and more over making sure StackStorm has the resources it needs to fix and fix and fix and fix again before any of us start noticing Netflix acting strange. (You want to see how Netflix auto-remediates their Cassandra environment? Hands on tutorial with code samples here. )
- Infrastructure as code. Without infrastructure as code the cost of changing the integrations and automations that live inside of event driven automation can be astronomical. In the world of legacy run book, needing to tweak an automation to respond to RIAK burps instead of Cassandra burps is a bunch of perfectly performed mouse clicks (intentional irony) OR a nice set of proprietary code. In our world – all the glue, the rules, and the workflows are themselves just code.
- And – sorry to extend the point but this is crucial – the code is YAML or maybe just whatever scripts you have lying around or maybe the configuration management system you have already deployed. A system like StackStorm can start with the bunch of scripts you already have – those that, along with a lot of human activity comprise your D and A – and ingest them. (yum, yum, Java, Python, Bash and Ruby) And then those scripts, those configuration management systems, all of it plus your monitoring and more are there for you to tie into a system that automates the entire OODA loop. And then the good stuff happens because you have incredibly fast MTTR (and even proactive MTTR, preventing the outage before it occurs).
- Workflow. With workflow you can see what is happening. Especially if exposed via FLOW (another pitch, I know). But not just FLOW and not just StackStorm. The return of workflows is real and widespread. Read a good overview of many of the projects published this Spring on devopsy.kinsta.cloud here.
- Feedback loop Maybe goes without saying – but once you automate your entire OODA and have it as code – you can iterate better on it. And that’s especially true if you take every single transaction from O through A and log it. How could you do it? Don’t worry, works out of the box with StackStorm. Or just set up a unique execution ID on every event and pass that through to each interpretation of the event via your rules engine and then keep that ID as you get the multiple workers in your highly resilient workflow system to start executing and then use that ID as well when doing post-hoc reporting; and don’t forget the APIs, GUIs and CLI as well. Or just grab StackStorm – it is open source after all.
In conclusion – OODA is a great frame of reference. However we in DevOps seem to be overly obsessed with the O and the O whereas those that first articulated OODA – those folks using OODA in combat – focus at least as much on D and A.

To me this not only implies do the D and A faster and better (via automation) – it also implies that we may not need yet another monitoring project. The world can stop reinventing monitoring for now, we have 157 projects at least.

Instead – let’s turn to figuring out how to lower the cost of false positives and the cost of automating ever changing environments – so we can iterate faster and faster – staying farther ahead of the entropy monster and the real monsters trying to steal and corrupt what we build.

If you like this sort of discussion and want to learn how Facebook, WebEx, LinkedIn, and coming up in a few weeks – Netflix is doing it – you may want to join the Event Driven Automation Meet-up which I help organize with my friend Brian Sherwin, LinkedIn SRE and author of the remediation system Nurse. Netflix has been kind enough to agree to host the next one at their new offices nearby to their old offices in the South Bay here in the Bay Area. Even if you don’t think you can make it, we try to share content via this meet-up and whenever possible we also stream the sessions.

Also, at the risk of being “too vendory”, StackStorm is the leading open source project and company dedicated to event driven automation and specifically auto-remediation – we stormers and the broader StackStorm community are here to work with you, to support you, to get you good code and to pick your brain and to ask for your PRs, feedback, integrations and automations (aka packs). So grab some code, join our Slack based community, and let’s get going.
October 7, 2015
AWS Lambda – Useful for DevOps Automation?

One of the highlights of AWS re:Invent, judging by the twitter stream and the points emphasized in Amazon CTO Werner Vogel’s Day 2 keynote and blog post, was the launch of Lambda.

In his blog announcing Lambda, Vogel calls the solution “the easiest way to compute in the cloud.” He goes on to describe it as both a way to run arbitrary code on AWS with complete abstraction away from the underlying infrastructure including EC2 instances, AND as a fundamentally better way to compose applications through the use of Lambda as an event-driven approach to intra-application behavior.

What struck me as interesting was Vogel calling Lambda “an event-driven computing service for dynamic applications.”

In his twitter feed, he has this to say about it:

Most heard comment at #reinvent “dude, lambda is the coolest shit, ever” http://t.co/b1DBbV8NMZ

— Werner Vogels (@Werner) November 15, 2014

So Lambda is cool, and is getting a massive amount of interest. And it has a lot to do with being event-driven.

A natural question that has arisen is could AWS Lambda be used for the automation of operations? And how does Lambda compare to the home-built scripts that are used by many operators to tie events to actions or to broader automation projects like StackStorm?

So, what is similar and what is different?

Deployment model:

The easy and obvious comparison is that scripts and projects like StackStorm or Chef for example or others are open source software used on premise or by users on their public clouds, whereas Lambda is of course an AWS service.

What events:

AWS Lambda comes already able to listen to a few different publishers of events from within AWS – “either a particular Amazon S3 bucket, Amazon DynamoDB table, or Amazon Kinesis stream.” By comparison, your own scripts or other automation software are of course completely agnostic: you can integrate with anything you want, you just need to write that software.

Scheduling:

AWS Lambda does not currently enable you to run events on a particular schedule, whereas some solutions like StackStorm come with such an ability built-in. Or you can leverage chron itself or, if you are Mesos aficionado, you might select chronos as did AirBnB for example.

What code:

Lambda today needs JavaScript / node.js; this is a real restriction however AWS says they will likely expand the range of supported software.

By comparison, Python is extremely common in home-made automation systems. Also many solutions including Ansible and StackStorm support the use of YAML to define workflows and other automation.

Method of execution: Workflow included or abstract?

Speaking of workflow, some automation solutions include workflow to tie together underlying atomic events. Like the use of YAML, workflow can simplify the process of authoring and then maintaining automation.

You could do the same with Lambda; however, it would require you to write your own workflow within JavaScript. Corner cases in workflows are unfortunately often exciting resulting in a lot of important code to write and maintain so the lack of workflow would seem to be an important difference; presumably AWS will eventually address this by integrating their own simple workflow into Lambda.

Abstraction of underlying resources:

Lambda enables you to auto scale and to not worry about what the underlying resources are when you run it: “AWS Lambda seamlessly deploys your code, does all the administration, maintenance, and security patches, and provides built-in logging and monitoring through Amazon CloudWatch.”

By comparison, scripts and other automation projects are just code. You can run them in an autoscaling group, but then you have to manage that. Lambda is also free up to a certain volume as AWS is betting on it further increasing the stickiness of your usage of AWS.

Community:

It is pretty easy to argue that open source projects have larger communities than AWS. And also easy to argue that by being dependent on AWS and Lambda you are locking yourself in ever tighter. On the other hand, Amazon continues to cut prices and their development pace is incredibly fast. I tend to believe in the power of open innovation however especially as Amazon regularly strip mines open source projects – exploiting those advances and adding their own – it could well be that Amazon is going to be able to progress at least at the pace of open source projects.

Summary:

On the whole, AWS Lambda is pretty clearly intended to be the internal wiring for AWS based software by tying together many loosely coupled services into an overall application. By comparison, automation software including your own scripts are used in operations to improve efficiency and frankly to eliminate some of the annoying manual work that otherwise would need to be done; also, home-built automations and automation projects are increasingly built to be easy to author and maintain whereas you have to be a java script coder to use Lambda; last but not least, enterprises are extremely heterogeneous so they must have a broad range of integrations available or easy to add including multiple public cloud providers and on premise technologies as well.

That’s not to say that AWS Lambda and open source automation projects will not converge. It seems likely that there is enough demand on the part of users to support multiple approaches to automation as automation is increasingly seen as one of the fundamental pillars of DevOps, along with all important cultural and skills improvements.

November 24, 2014
DevOps > Cloud, SaaS, SDDC, and apple pie

Over the weekend I was chatting with a really successful venture investor who recently published a top trends list.

That made me think about DevOps vs. Cloud and vs. SaaS and vs. Mobile and other disruptors. I’m convinced more than ever than DevOps is the right focus for professional investors and for all of us investing our time in developing, adopting and operating IT. Examining and understanding DevOps is more useful than understanding cloud, SaaS, SDDC (software defined data centers) and more – and here’s a few reasons why.

1. DevOps is a common denominator

Keep peeling back the layers at the top operators and, underneath, you find the organizational structure and the set of tools that is DevOps. Some of these operators use containers, but some use bare metal, and some use virtualization – so there is quite a bit of variance there. Yes, there are other common patterns like developing applications that are stateless themselves and able to aggregate underlying resources – however DevOps in organization and in technology is a common denominator in a way that, for example, virtualization alone or technologies X, Y or Z are not.

2. DevOps is more open (and hence generalizable) and more powerful than SDDC

This might be a little ironic coming from me as I did a bit to help popularize the concept of software defined storage and of software defined data centers while helping lead Nexenta. And we tried hard there to be specific in what we meant.

However, the discourse about SDDC was and is really controlled by EMC and by VMware. And, as such, SDDC as commonly understood is well aligned with VMware’s vision. In that vision, VMware software becomes the orchestration or control system for the data center. Their APIs, such as VASA for storage, pass along information about what applications need to have provisioned and so forth and the rest of the data center infrastructure dances to that tune.

The SDDC is a powerful vision. However – because it is VMware centric it is by definition one and only one way that an application centric view of the data center could be built. For example the VMware vision of SDDC assumes that virtualization, as opposed to containers or aggregation via Mesos for example, are the fundamental building blocks in compute. And building a SDDC in that way does not mean that users will necessarily achieve the productivity boost of 10-100x that are being achieved by adopters of DevOps.

As I’ve mentioned in a previous blog post, the current VMware product line up does not fit nicely into a DevOps workflow. So to the extent that SDDC really is a VMware led narrative, it is not quite compatible with DevOps.

I could make a similar case about DevOps being at the core of – and more important in some ways – than SaaS. SaaS that only scales up and that uses expensive legacy technology is not long for this world thanks to competition from DevOps centric SaaS shops. Because of this, again, as an investor, pulling on the DevOps thread is more valuable than pattern matching for SaaS.

3. DevOps is more about people

Most importantly, DevOps is people plus technology – done right, to move fast, to build and operate code. By including how people work in its very core, DevOps as an approach is more likely to actually deliver its promise. No technology is of any value if people don’t actually use it. And DevOps as a movement, as an idea, as an approach, again, to building and operating code has the importance of people built right into even its name.

Who knows, maybe in 2015 the discourse will shift away from DevOps. But I don’t think so. I think DevOps is a more important and more valuable concept than SaaS, or SDDC, or “cloud” for that matter. And I think that value will continue to be demonstrated for years to come.

June 13, 2014
From Orwellian Nightmares To Unicorns: Why I’m All In On DevOps

Over the years I’ve helped literally thousands of customers gain efficiencies in their data centers. I’ve met and become friendly with a lot of IT pros but I’ve got to admit – I never envied them. Well, maybe I envied some of the views in NY and Jersey City. And some of the toys they got to play with over the years were amazing.

Nonetheless, I didn’t envy them because many of my friends in director and VP roles spent a huge percentage of their time fighting politics that resulted in Orwellian distortions of the truth. A couple of stories from the early 2000s might help illustrate my point:

1. Two guys pulling all-nighters delivering a $1.5B roll out of IP communications while 25+ project managers debate progress from 9-5

The headline pretty much captures it. This now too-big-to-fail bank, as well as their vendor, had brought in several project and program managers to ensure groups from facilities through supply chain through security and operations were “aligned.” I spent a couple of meetings getting to know all of these project managers before realizing that there were two guys working every night, all night, to manually, using fingers and spreadsheets, push and validate changes. Eventually those operators quit – and the project ground to a halt.

All the alignment, of course, was pretty much BS. At one point I mentioned in a meeting that our software would not run on the same server as the IP Communications System (Call Manager for those curious). I was corrected – “it isn’t a server.”

I responded with something like, “it is an HP DL something or other. Isn’t that a server?“

Answer: “No, if we call it a server then the server team will need to get involved and we don’t want the server team getting close to this project.”

You cannot make this stuff up. Servers = / servers. Got it?

2. Similar story, different bank – 1% of achievement of goal = “on plan” to outsourcer

The story starts when I was brought in by a huge vendor to a bank that is also too-big-to-fail. This bank also had a high profile project to get the bank over to IP communications. Keep in mind that this was after 9/11 and all the banks were looking up to the teams at Lehman that had been able to get Lehman back trading significantly faster in part because of their use of IP communications. So the project to get the bank onto a new approach to real time communications had CIO visibility.

One of the most surreal meetings of my career occurred when after having been vetted, tested, recommended, and designed into the solution I was granted an audience with the SVP handling the bank at EDS; I feel ok with naming EDS since they no longer exist and I’m sure this executive has retired. Like many folks at EDS he was former military and as an account executive he owned a north of $1B outsourcing contract.

I started off by saying that we can help you get back to schedule, as I know you have only done about 1% of the deployments originally planned. He looked me right in the eyes and said: “EDS is on plan, and we always will be.”

After some back and forth I got it – no matter what the facts on the ground were, he was never going to accept any culpability at all for the project delay. So missing the project time line by 99% was officially “on plan.” That’s Orwellian.

DevOps restores faith in humanity / IT

Compare any decent “DevOps” shop to the above stories and you can see why I’m so enthusiastic about doing my small part to help bring DevOps approaches to operations to a broader market.

The teamwork between not just Dev and Ops, but between everyone responsible for delivering capabilities to the organization, is so refreshing. Just last week, Google engineering blogged about using machine learning to improve operational efficiency of their data centers, with a focus on power consumption and so forth. The technology I get – we are headed down a similar path with deep learning and control theory. But what struck me more than the technology was the data they are using to draw inferences would often be siloed and tightly controlled by groups within IT at legacy companies precisely because it could be used to score points.

So DevOps entails more than infrastructure and code and collaboration. With DevOps, IT organizations move from fighting over terms like “server” and “on plan” that are often filled with well meaning and intensely frustrated teams towards an environment where cool stuff happens all the time. And IT done right – making sense of the world with software and hardware – is incredibly cool.

I’m betting on DevOps because I want the world to be filled with DevOps unicorns and because, like my friends that are operators within legacy environments, I cannot stomach more Orwellian obfuscation of the facts on the ground for political expediency. DevOps makes the world smarter and IT teams a heck of a lot happier. What’s not to like?

June 3, 2014
User pragmatism is stronger than ever

This is the third of a series of posts about why we thought the world needed another DevOps oriented company and so invested in and started StackStorm. I hope we are touching on general themes that can help others understand the rapid disruptive shift towards DevOps approaches to building and operating technology.

When starting StackStorm we were inspired by OpenStack. This week is the OpenStack summit and we are contributing heavily to some pieces of OpenStack, so we remain enthusiastic community members. However, our third reason for StackStorm has to do with how we lost our naive perspective on OpenStack and understood it more as a symptom of a new, more enlightened approach to technology selection, development and operation.

OpenStack is important – however, users are more pragmatic and agnostic than ever.

OpenStack is the full stack plus, plus, plus; it is very different than almost every other open source community thanks to its breadth, from storage through networking, security, compute, management and more. Not surprisingly, OpenStack is a massively diverse community. And it has unstoppable momentum. However, when we went to interview practitioners we consistently saw that the DevOps savvy see OpenStack pragmatically. The operators we know see OpenStack as a means to an end.

To that point, a group of about 30 operators of OpenStack were recently asked at the OpenCompute 2014 summit in San Jose – what distribution do you use? There are of course many, many distributions and there has been talk of a distribution war. So we were really interested in the answer – who are these operators using?

The answer – NO ONE. Top operators are taking the bits they need. While RedHat and presumably others will succeed over time in selling support for a distribution, I don’t think most commentators really understand to what extent the DevOps mindset alters the equation for legacy open source models as well as, of course, legacy proprietary models.

A DevOps operator, generally speaking, takes the pieces that make sense and then ties them together so that their operations environment, their hybrid cloud, their software defined data center, are truly theirs and operate together as one system. To avoid accidental lock in, maintenance burden and technical debt, each of the many, many, many pieces that together comprise a DevOps environment are themselves almost always open source and broadly deployed by others. This way the operator avoids lock in while also having what they need for their business.

So – open source and specifically OpenStack pieces are fundamental to the operators we know. However, through formal and informal communities these operators are now empowered to take the best of breed components and integrate them into a whole. And what does this get them?

Refer back to an earlier post – this approach is delivering 10x-100x productivity improvements, which means a chance to stay in business because if you don’t make this transition and your competitors do, they will cycle faster than you closer and closer and closer to the customers.

This brings me to a discussion of DevOps architectures. How can you tie together so many loosely coupled pieces? Does the Legos analogy stand up? Yes it does. More on that soon.

May 13, 2014
Let’s keep the countdown going: Reason #2 for StackStorm to join the fray

Stackstorm is a new company focused on automation, operations and DevOps. It’s co-founder and CEO Evan Powell has been keeping a running journal as the company made ready to emerge from stealth. This is the 2nd in the series of posts from Powell’s thoughts and experiences as a new DevOps company comes to market.f

In yesterday’s hello devopsy.kinsta.cloud – and “hello world”- post we put the 10-100x boost in productivity of the top DevOps operators in historical context. It’s like software is catching up with Moore’s Law’s relentless beat of exponential progress all at once. The shift is changing the world, has changed the world.

However, there are barriers. Like many technology life cycles, a fundamental limitation is us – the people involved in this transition.

And that’s where driver #2 for us starting StackStorm comes in:

There are not enough DevOps unicorns in the world – and there never will be.

The ideal DevOps engineer is a full stack engineer who can code at least in python and is really good at cross-functional collaboration. We are a magnet for this type of unicorn – we have hired a few and have a handful in our utterly kick-ass advisory board.

However:

a) Being a full stack engineer is really, really hard; in my time leading Nexenta, we were happy to find someone who knew something about virtualization and could go deep on storage and while that is rare even that is not a full stack perspective.

b) To add to the job spec of being truly full stack to also know how to code well enough that your scripts work reliably and are trusted is extremely rare.

c) To then combine that with the personality types that seem to thrive in the “anti-siloed” world of DevOps is really, really rare.

Folks that have attributes 1, 2 and 3 – and can lead teams to deliver cohesively – are worth their weight in gold. Software needs to step in here – to help extend and automate these full stack unicorn automators.

As an aside, I may be biased because I have had the pleasure of leveraging DTRACE in Nexenta products over the years, but this has got to be the most awesome Unicorn image going:

Automating all things is great. Automating the automators is even better.

And that’s where we are headed. That’s where the massive operators already are. Amazon, Facebook, Google, PayPal and others have built systems and applied some level of machine learning to the operations systems to enable them to operate at scale and to capture improvements. They’ve entered into a virtuous cycle. Not surprisingly, the IP behind these operations systems is not freely shared with the outside world. Components developed at Amazon and elsewhere may be shared, but the operations automation platforms and intelligence are rarely shared.

That’s where we and the broader community come in. Which leads to the next driver: the community. I’ll discuss how OpenStack and the success of related communities led us to found StackStorm tomorrow.

If you are interested in being invited to StackStorm’s private beta of our operations automation software – please register here: www.stackstorm.com/beta

May 8, 2014
Automation, Operations, DevOps – What We Thought Was Missing

Stackstorm is a new company focused on automation, operations and DevOps. It’s co-founder and CEO Evan Powell has been keeping a running journal as the company made ready to emerge from stealth. Over the next few days we will share Powell’s thoughts and experiences as a new DevOps company comes to market.

Over the next few posts I’ll be sharing my thoughts on why StackStorm – a new company focused on DevOps based automation – was still needed. My first comment is that the users will decide how we overlap or complement existing automation solutions. So far they are confirming our belief – that we are strongly complementary to trail blazers like Chef, Puppet and of course CF engine and more recently Salt. Our approach is to focus on operations automation that wires together the data center and that – in the case of systems like Facebook’s FBAR and our own beta level software – learns over time. With that as some context, over the next five days I’ll unpack our key drivers for us starting StackStorm – starting with something I think all readers of devopsy.kinsta.cloud should agree with:

DevOps and the shift to the cloud is much bigger than generally assumed. Over the last few years, my co-founder Dmitri and I have met with thousands of enterprises, service providers, government agencies, integrators and sundry analysts and experts. And I’ve been speaking about the inevitability of the shift to Software Defined Data Centers for many years while helping to create the software defined storage market as founding CEO of Nexenta Systems. More recently, as StackStormers, Dmitri and I have met with approximately 100 DevOps practitioners with OpenStack projects either ongoing or kicking off. The shift to DevOps – sometimes lumped in with the shift to Software Defined Data Centers and also the shift to the cloud – is the biggest shift in IT we’ve seen in our careers. The top DevOps practitioners are 10-100x more productive than your legacy siloed IT operations department. Software is “eating everything” AND the approach to building and operating software has improved radically. Nearly every company and government in the world must either leverage this step function improvement in productivity of software development and operations or face disruption. It might be worth discussing the 10-100x productivity improvement:

a. Is it real? Organizations like PayPal and eBay have published their own data showing at least a 7x improvement in development velocity. And of course the DevOps surveys by folks like Puppet Labs have shown even higher improvements in the speed and quality of development. And, again, our own experience indicates it is very real. One operator tells us that they went from building and deploying 1 new service a year before DevOps to 54 in the 12 months after they got up to speed with a DevOps approach. Revolutionary.

b. Is 10x a lot? Software getting built and operated 10x better is changing the world. A comparison that Dmitri made in his talk at SCALE12x this past February was to the tractor. As I think any casual student of the 20th century knows, the tractor changed the world by enabling urbanization and more. According to most economic historians, the tractor was about 4x more productive as measured by crops produced vs. input in human and machine energy. IT fairly suddenly figuring out how to develop and operate software 10-100x more productively compares favorably with the most important improvements in human history.

So – GREAT. The world is changing. What do we and other DevOps solution providers have to do with this? Ask yourself – why don’t we ALL just jump on the DevOps bandwagon and start building and operating software in this new way? I’ll take on that question tomorrow as the second driver that led us to start StackStorm.

Feedback welcome. Comment below please.

May 7, 2014