devopsy.kinsta.cloud

Tag: devops

Five Top Tips for DevOps At Scale
Since I published my blog explaining the topic of ‘Big DevOps’ late last year, the concept of Enterprise DevOps has exploded, with commentary, experience, and advice (some good, some awful) popping up from many different quarters. Fortunately, many pundits have now come to agree that application delivery in a traditional enterprise is very different from in a typical web-scale/startup environment.

As a result, a growing body of work is filling a significant and important gap with unique insight into Enterprise DevOps, from many excellent commentators and practitioners who understand how large enterprises are different, and how to address those different challenges.

As Enterprise DevOps takes off, I seem to field questions about it on an almost daily basis, so I decided to contribute to the body of work with a webinar explaining my answers to five of the most frequent challenges I hear about DevOps at scale. It was titled ‘Big DevOps: Five Top Tips When Implementing at Scale’ and you can catch the full replay on YouTube.

The top five issues I chose to address are below:

My five ‘Big DevOps’ tips for overcoming these issues center mainly on cultural change, because DevOps is primarily a cultural and attitudinal shift. However, it is also undoubtedly true that certain technologies can play a large part in enabling that change, especially in the more complex world of larger enterprises.

With that in mind, my top five tips to address these challenges are as follows:
- Facilitate a culture of teaming and collaboration – this is at the core of devops in any environment, large or small, but especially for large organizations with bright lines between teams, this is essential to overcoming the tyranny of complex and dispersed silos
- Establish agile development as a shared discipline – agile development is brings code to market faster, but can result in just filling the clogged pipes of PMO, HR, training, security, and other departments faster. ‘Agile’ needs to be common throughout the SDLC – in dev, ops, and beyond
- Automate relentlessly to enable rapid devops response – automation allows faster execution throughout the system, enabling downstream processes to keep up with speed of DevOps. Automation also adds audit and governance, especially important to larger organizations
- Push smaller releases faster, measure and remediate impact – large organizations especially are can avoid big errors by avoiding big releases. Adopt faster, smaller cycles with closer scrutiny to achieve a DevOps approach while managing the risk of faster cycles and fewer controls
- Manage user identities to enable shared access – in smaller organizations it might be acceptable to give out root access like candy, but not for large enterprises. Identity and access management can enable a new DevOps approach that is safe, secure, auditable, and well-governed
While these tips may not be the perfect answer for every organization, I have seen how these approaches have made the journey to DevOps easier, safer, and more effective for big enterprises. I hope they help you on your journey.

If so, you will want to stay tuned to DevOps.com. Since starting work on Enterprise DevOps concepts, I have collected many questions on how large organizations can best succeed in their DevOps transformation. I will be sharing these questions along with my answers here on DevOps.com in a regular series, to give everyone the chance to understand how large enterprises can overcome the challenges of adopting DevOps at scale.

So keep your eyes out for my posts over the coming days, weeks, and months in the brand new occasional series, ‘Enterprise DevOps Q&A’. I hope you will keep reading, and maybe even help me out by discussing your experiences and ideas in the comments below.

Finally if you have any questions about Big DevOps, Enterprise DevOps, or anything to do with adopting a DevOps approach at scale, let me know in the comments below too. I would love to hear from you, and answer your questions if I can, or source guest opinions from the DevOps community.

Meanwhile, please stay tuned for more Enterprise DevOps Q&A. I am excited to continue the discussion on what it takes to transform application delivery at scale.
July 24, 2014
ChatOps: Communicating at the speed of DevOps

DevOps is about knocking down barriers between developers and operations teams, and in doing so reducing organizational frictions wherever possible. And that’s certainly the spirit behind ChatOps – the term for the hyper-collaborative way of running DevOps, including operating aspects of systems and infrastructure, through online chat.

Proponents of ChatOps say it streamlines communication in a centralized and powerful way. Born within Github, ChatOps evolved from its open source chat bot, Hubot, which has changed the way the web-based Git repository runs operations through the automation of deployment, graphing, monitoring, provisioning, and even mitigating security events.

Think of Hubot, a Node.js app that is commonly run on Heroku, as a robot that helps keep everyone informed in a centralized place and in real-time, or in a near real-time, way. There are many benefits to ChatOps, says David Mortman, David Mortman, contributing analyst at Securosis, including the integration of ops activities with notifications into one place, making it easier for everyone to know what’s going on across the enterprise, as well as everyone throughout the organization having the ability to see error messages on events as they occur, he says.

Github, as a highly distributed organization, with its several hundred employees spread across 103 cities, can use all of the communication efficiencies it can muster. As Mark Imbriaco, Ops engineer at Github, in his presentation ChatOps, Augmented Reality for Ops, uses Github, email, internal apps, other web systems that they’ve built to help teams communicate asynchronously – and yet they needed an option for the real-time or “semi”-real time communication. “I say semi-real time because, when we talk about chat, the social contract at GitHub around chat doesn’t require you to be immediately responsive to everything you see. Chat is still asynchronous, but it’s more synchronous than GitHub or email or other sorts of notification mechanisms,” said Imbriaco.

Without enhancement, Hubot doesn’t do a whole lot right away. But when added to chat with one of the dozen or so connectors that include access to Google Talk, IRC, Skype, and Hubot monitors for events that come from within chat or from one’s systems and applications. “The magic happens when you build additional functionality into him to automate, well, everything,” Imbriaco said.

Hubot will do certain things, such as notify people not present in a chatroom that they were mentioned, phrases can be registered so that notifications to those interested in certain events will be notified. Github has also plugged Hubot into the GitHub service itself. Today, when anyone creates an issue or pull request in GitHub, according to Imbriaco, a link and definition of the event is sent to chat. While Github already has ways to send notifications, by entering events real-time into the chat stream, others can jump in and comment, help, or otherwise advise on events and interact as they’re occurring.

As for the adoption of ChatOps, Github isn’t the only one: but it’s not yet widespread among DevOps practicing organizations, either, based on our reaching out to numerous organizations. Nor, is the concept especially well known.

That’s not slowing Owen Zanzal a developer at systems management provider VividCortex down from adopting ChatOps. Zanzal says in his blog post, ChatOps at VividCortex, that, despite ChatOps being off to a slow rise, “ChatOps has really revolutionized how we operate,” he wrote.

The first Hubot mission at VividCortex was to help improve how code is deployed. The problem Zanzal highlights is that encryption keys and new developers apprehension around code deployment, was slowing down production.

To streamline deployment, Zanzal moved the execution of Capistrano commands to a Jenkins (a continuous integration server), and created Hubot commands so that jobs could be created using a Jenkins exposed REST API. When done, developers needed only run a chatroom command, such as “hubot stage webapp” or to deploy “hubot release webapp,” Zanzal explained. “Now, no one is blocked by the gatekeeper from pushing code into staging or production, and every developer can push code when they are ready. The unfamiliar Capistrano commands were simplified into much easier to understand Hubot commands,” he wrote.

VividCortex is also using ChatOps to fetch logs and more quickly remediate issues as they arise, Zanal wrote.

Chat-as-Ops
GitHub has also pushed ChatOps much further than simply as a place to stage and deploy code. It’s using ChatOps as a way to help actually run ops. For instance, Imbriaco spoke to how Hubot and ChatOps is used for efforts such as rebalancing shards across its file servers. “Honestly, it took me a little while to get used to some of these. We have some [Hubot actions] that do things like rebalance shards across our file server. And that’s horrifying to me, because the consequences of failure are so bad. But the reality is, I’m either going to run it from a chat room, or I’m going to run a shell script from a terminal. If I ran it here, at least everyone saw it. So at the end of the day, this is repeatable, it’s tested, it’s commonly used,” Imbriaco said. “And if it stopped working we would notice,” he said.

And one time when Github was fending off a distributed denial of service attack, Imbriaco needed only enter the name of their service provider to direct traffic and the attack was successfully mitigated.

Moving to ChatOps isn’t without risk, however, Imbriaco explained. A lessoned learned when it became a little too easy to deploy to Amazon Web Services. “We went from zero to way too much spend on Amazon in about two weeks, when I turned up the ability for us to provision EC2 hosts from Chat,” he said. But that issue was resolved by giving users information at the time of provisioning, such as including metadata into the instance that details who created it, as well as informing users how much it’s likely to cost,” Imbriaco said.

Despite the efficiencies of ChatOps, don’t expect a widespread embracing of ChatOps, but more likely a slow adoption. “GitHub is a flying unicorn. They’re like Netflix,” says Mortman. “People will pick up some of what they are doing but it’s too advanced for most,” he says.

July 16, 2014
Meet Infrastructure as Code

Maybe you have already heard the terms “Programmable Infrastructure” and “Infrastructure as code”. But even if you have not, if you are interested in DevOps it’s something you need to know. Most new lingo causes me to roll my eyes. But in this case, it’s not just new lingo, it’s an entirely new concept. Besides making the nerdy feel cool when they say it, these terms describe a new idea of what an application consists of. (more…)

May 5, 2014
An open letter to Jeff Knupp
Dear Jeff,

First off, nice recovery. I commend your gesture to return to positivity and make what to my eyes, looks like a kind and altruistic offer to an underserved class of engineers. I personally sent your offer to all the female engineers on my team and got very good feedback from several of them.

Some people will try to tear you down and criticize your offer to make a difference in the lives of female engineers as patronizing or sly self promotion, but I take you at your word and applaud your offer as both poised and graceful.

Given your offer to one underserved population in IT, I thought you would be OK with some constructive feedback from one dev guy concerned with the welfare of IT workers to another and hope you take this feedback as it is intended, with the same eye towards a desire to be a force for positive change to the plight of all IT workers, everywhere. Please remember that this is just my perspective. I could be wrong, and often am. So, here goes:

DevOps does not mean everyone has to do everything

DevOps is not just for startups.

I work in a big shop and we have all the sorts of disciplines you call out in your posting on how DevOps is killing the developer. We are also embracing the DevOps movement not to force anyone to take on roles they aren’t optimized for, but rather to inspire them to expand the boundaries they have set for themselves and others in an effort to make things better for everyone.

We want QA professionals to know how to script and automate tests. This is good.

We want operations staff to know how to script and automate changes. This is good.

We want release managers who know how to triage problems, automate configuration management and instrument the throughput of the IT machine. This is good.

We want developers and architects who know how to go deep when required to help solve performance problems. This is good.

You may disagree and I respect your right to do so. I just am not that interested in hiring an IT professional of any type who says “That ain’t my job”. That’s not a guy or gal who gets things done or makes things better.

I’m not saying that everyone has to be able to do every role or even that developers must branch out. I am saying that making others wrong for increasing their value prop is not likely to do anyone any good.

Injustice anywhere is a threat to justice everywhere

Using narratives of which type of IT worker is more important than another type of IT worker does us all harm. All of us. All men. All women. All IT workers. All Developers. All operations staff. All employees from any department of any company no matter the size.

Now, I realize that this is a bold statement (even from the guy who wrote “Sharepoint is Crack and Microsoft is the Pusher”), and I have no problem admitting that you probably have no desire to harm any employee anywhere (Except for the guy or girl who invented IVRs. I can understand wanting to do bodily harm to whoever did that). I would ask that you consider a few ideas before writing me off as a troll.

1) The schism upstream is the enemy to be fought, not the people downstream from you

You may not be aware of the schism to which I refer, but I would argue that you are probably very familiar with it’s effects. The deep and distrusting divide that exists between enterprise IT and the customers of enterprise IT across the industry is the single most destructive and demoralizing artifact in the lives of IT employees everywhere.

Have you ever worked in a shop where IT employees are casually referred to as “technical” while marketing, product and design employees are referred to as “creative”. Have you ever been exposed to a shop where the developers were treated and talked about like they were worker insects who’s job it was to keep their mouths shut and code what they’ve been told to?

If you have somehow managed to have an IT career and avoided exposure to this phenomena that seems to pervade so many (but not all) IT shops around the world, I would ask you to pull back and reflect for minute. Spend some time and have some gratitude for the fact that you have not been either witness to, or been the subject of the message: “we are the so and so team and you are not on our team so you have no say IT boy, go back to your nerd closet and build what I tell you to build”.

If you are familiar with this idea that robs developers everywhere of the right to claim their creativity and basic need for self realization through recognized craft, then I ask you this: Aren’t you all sick and tired of the completely loathsome ideas that we are not “creative” because we are IT? Doesn’t that make you want to throw up? Don’t you want to go postal when you hear people push developers down the collectively agreed to illusion, referred to by you as the “totem pole”, that creates a veritable corporate caste system?

I would run out of my word limit if I were to try and cite the publications that point out that silos are the prime roadblock to unlocking business potential, but I will point you to a few modern classics which make the case much better than I ever could:
- The Advantage: Why Organizational Health Trumps Everything Else In Business by Patrick Lencioni, best known for The Five Dysfunctions of a Team: A Leadership Fable
- Silos, Politics and Turf Wars: A Leadership Fable About Destroying the Barriers That Turn Colleagues Into Competitors also by Lencioni
- Orbiting the Giant Hairball: A Corporate Fool’s Guide to Surviving with Grace by Gordon MacKenzie
This isn’t just about IT workers. This is about every employee everywhere who has their role demeaned and belittled by another. There are three basic roles inside of most companies in the products and services business. Sell the work, do the work, support the work. One thing in common to them all is that if any of the three roles don’t get done well, then everyone is at risk of not being paid.

We are all necessary and none of us is sufficient at any real corporate scale. The moment we allow division (i.e. the totem pole) between ourselves, we have ceded the ground that other people upstream from us or up the hierarchy from us have the right to demean whomever they want to.

You can’t possibly eliminate injustice via injustice

2) The schism will always exist if we reinforce it

Being of service to another is an honor and all who claim to want to improve the lives of workers anywhere, should hold themselves accountable to not belittle the contributions of others.

Every single employee has the potential to be innovative and creative in their work. We constantly create and reinforce the concept of commodity items and jobs and then are paradoxically shocked and surprised by:
- how the unmotivated and unengaged masses of people in jobs frustrate us with their lack of passion for doing a job well
- how new people constantly create a “new and improved” version of something perceived to be commoditized forever and then dominate the market because they believed it was possible to reinvent a new level of quality
Innovation is not a department. Creative is not a department. Aren’t you all sick and tired of narratives of ownership and control from other teams who claim to own these abstract qualities when they are delivered at you?

When we the development community adopt the same basic ideas, we allow it to continue.

If you want to world to be different, than you must have the courage to be willing to be one of the first ones who fully live in that new different world.

What world are you working towards?

3) How will you know your ideas and contributions are better if you set up barriers that make it harder for others to contribute?

I am privileged to lead one of the best dev teams in the world. I am privileged to work in a shop that is committed to breaking down the barriers to collaboration and cross-pollinating innovation. To keep us all all at our best, we have to be brave enough to say that our dev team and our creations can more than likely withstand any criticism or contribution from any team up or down stream and if they don’t because some other idea was better, so be it.

I won’t take the cheap and easy shot and ask “Are you afraid that the outside idea is better and that you will seem weak for having a lesser idea?”, because I doubt that this is true. I honestly think this kind of resistance has roots in the human condition itself. It probably has more to do with fear of the clock running out or the possibility of lesser ideas winning the day than of fear of being shown either as “weak” or “not as smart”. With these dreadful possibilities in mind, I would ask you consider this: The way to maximize the likelihood of these undesirable outcomes is by allowing others to say we are closed off to outside ideas. This is what leads to goat-rodeos, witch hunts and ultimately those with more authority than we have removing the little authority and control we actually have.

Let the best ideas win, not the ones made by those with the authority backed up by a title.

If you truly are looking for the best and most game changing ideas to help your enterprise, be aware that the richest source of these ideas will inevitably come from people who’s blinders have not been firmly affixed by the discipline they are embedded in. This is a widely accepted historical truth laid out by Thomas Kuhn in the timeless masterpiece The Structure of Scientific Revolutions and if you doubt me, remember that Albert Einstein was a patent clerk when he gave humanity the theory of relativity.

I’m not trying to make you out to be a bad guy or anything close to one. In fact I praise you for having the guts to articulate a fairly controversial point of view that you believe in.

When I write this letter, I’m not fighting for or against DevOps in particular (even though I am DevOps fan). I’m fighting for the end to the schism between Business and IT.

I’m fighting for the end of narratives of control and authority.

I’m fighting for the end of the un-empowering messages that authority gives to individuals every day.

If you want to know why enterprise IT employees everywhere are asked questions from their business partners that display lack of trust over and over and over; This is why…because the schism exists.

If you want to know why enterprise IT can’t complete things it starts; This is why…because the schism exists.

If you want a world where people trust IT employees expertise and allow IT shops to complete work without questions of minutia that spawn from lack of trust, we must not embody what we oppose – narratives of authority and ownership combined with an “us vs them” mentality.

I don’t say any of this to scold you Jeff. I say this as an attempt to call upon you to join us . Help us tear down the walls. We need more help. We need you.

Respectfully,

Stephen Fishman – Creative IT Professional
May 2, 2014
Automation versus Orchestration

Yes, Virginia, there is a difference.

One of the things devops practitioners are tasked with is the provisioning and configuration of all sorts of infrastructure. Application servers, web servers, load balancers, proxies and database servers are among the lengthy (and no doubt growing) list of “boxes” devops needs to get up and running to support just about any given application today. One of the key value propositions of a devops approach to operations is that it can reduce the time it takes to get applications to market by getting them up and running in production faster. That’s increasingly important as we move into the Era of Things, driven by extreme connectivity of everything, because all those “things” need are going to need to be talking to applications on the back side.

So speed is of the essence, but not at the cost of accuracy. We’re encouraged, then, to automate tasks and orchestrate processes to achieve that speed. The side effect of codification of tasks is higher accuracy (aka fewer mistakes).

So far so good.

But you may notice that I have (and will continue to) clearly delineate between “automate” and “orchestrate”, between “task” and “process”.

The reason for that is that, simply, they are not the same thing. We should strive not to confuse these two because while the two share some benefits, there are other benefits that are achievable only by one or the other – or the combination thereof.

Process optimization, for example, can’t be achieved by simple automation. Automation is concerned with a single task – launching a web server, configuring a web server, stopping a service. Orchestration, however, is concerned with automating, if you will, the execution of a workflow – of a process. A provisioning process may be comprised of multiple tasks and involve multiple systems. An “application” is not just a single server, after all, it’s likely several servers – web, app and database in a traditional three-tier architecture.

The goal of orchestration is not just to automatically execute a service, which brings speed to the deployment process and gets applications into production faster. It also affords an opportunity to streamline – to optimize – those processes for even greater gains in deployment velocity.

One of the simplest ways to optimize a process is to eliminate repetitive steps. This entry on Vagrant explains it fairly well :

“We could just SSH in and install a webserver and be on our way, but then every person who used Vagrant would have to do the same thing. Instead, Vagrant has built-in support for automated provisioning. Using this feature, Vagrant will automatically install software when you vagrant up so that the guest machine can be repeatably created and ready-to-use.” [emphasis mine]

— Vagrant Docs “Getting Started | Provisioning”

Now certainly the step of installing a web server on a new box with vagrant is hardly all that time consuming, it’s just a single line of text. But every line of text must be manually typed, and that introduces the possibility of fat fingering the command, which either consumes time in fixing it or introduces an error into the system. Neither are desirable results.

Ensuring that the process is repeatable and orchestrated by automating all the tasks that comprise that process an optimize it by eliminating redundancy. That results in greater deployment velocity.

Recognizing that processes should be comprised of composable tasks offers a higher degree of reuse across operations. The task of provisioning and configuration a web server with the appropriate base software means consistency, aka standardization, across all applications. Processes that take advantage of standardization result in greater infrastructure stability, another benefit often cited by organizations that have adopted a devops approach to operations.

The key here is to recognize the difference between automation and orchestration and how they relate to one another. Automation is about codifying tasks and orchestration is about codifying processes. Orchestration takes advantage of automation by reusing these basic building blocks.

There is a difference, and done right, the two play off one another to result in streamlined, more accurate execution of operational processes that speed up successful application deployments.

April 10, 2014
DevOps Needs a Tsunami To Jump The Chasm

All start up business endeavors must go through the phase of crossing the chasm. Most of those business endeavors usually fall prey to the chasm. The chasm is this black hole that everyone claims to understand but no one truly does. We have all seen businesses cross the chasm or fail to cross the chasm and fall into the black hole. But no one seems to understand why and how that crossing happens. The bestselling book by Geoffrey A. Moore titled “Crossing the Chasm,” is all about the heart and soul required to get early stage technology across the chasm, from early adopters to mainstream customers.

There is a big difference between people who are enthusiastic to try leading edge technologies and the rest of the inhabitants, who tend to be much more conservative. The reasons that people spend time trying to figure out how to cross or jump, if you will, the chasm is because that black hole is what stands between them and making their fortune. The early adopters are just a small sample of the market and there is no money in building a business or technology around them. To make real money, you need to cross over into the mainstream.

In Geoff’s model, the innovators are all about technology. You know those people, when a new iphone is released, they spend days standing on line to get latest and greatest new technology. The early adopters are the visionaries that are always on the bleeding edge. The mainstream are the pragmatists, the people who practice risk avoidance. These pragmatists are the key to jumping the black hole, they move the needle.

DevOps first order of business was getting on the curve and to that end DevOps has evangelists. But, understand that what worked in converting these early adopters probably won’t work for the pragmatists. These DevOps evangelists and visionaries want DevOps power tools, as opposed to the pragmatists who simply want to push the button and say “that was easy”.

By all appearances DevOps has been accepted by the early adoption set. Just how does DevOps cross the chasm? Jumping the black hole after all can be quite a daunting task. To do so DevOps has to create what I refer to as the tsunami effect.

The Tsunami effect is where DevOps gains acceptance in multiple industries or markets in what seems like a gold rush, it’s here that fortunes are made. More accurately, it’s where cult communities transform overnight into mainstream communities. Many technologies simply create a single tornado which remains in a silo forever never gaining enough momentum to really emerge. Another common path what I call an earthquake, there is a lot of hype but it tends to fizzle out quickly, if not tended to properly. Right now DevOps has all the appearances of an earthquake, Devops evangelists have created a lot of buzz, but acceptance beyond a set of power tools is still a huge question.

Considering DevOps is in the earthquake phase, as opposed to the tornado phase, DevOps cannot be successful converting one group at a time. The strategy in the tornado phase is convert industries and markets one at a time, via tornados forming within industries or markets. The tactical execution is to get smaller industries or markets committed and therefore the larger industries or markets will then embrace the movement. That is simply crossing the chasm. DevOps has too many cultural implications to convert believers over a long period of time; DevOps requires reorganization, not a subtle shift.

Since I consider DevOps to be in the earthquake phase, crossing the chasm is not an option. DevOps must jump the chasm in a surge of overwhelming momentum, hence creating a tsunami. DevOps has created a very closely knit culture, DevOps needs to push hard to create a tsunami and flood the pragmatic population. But, be advised, DevOps has to be all but universally accepted, there is no almost in bringing together Dev and Ops. But if DevOps wins, it wins big, DevOps will go from being a niche culture to a truly mainstream practice.

So what’s the strategy to create a tsunami? And more importantly, how do you move beyond the earthquake phase? Here’s how I see it according to Geoff, with my DevOps spin: First, DevOps must clearly establish that the mission-critical software development process is broken and is causing management enough pain that it would gamble on unproven methodology that is populated with a bunch of prima donnas. I am not convinced the DevOps cult has clearly established this as fact. Is the development and operations process broken or have just new power tools been created to improve speed and accuracy? Second, DevOps needs to present the total solution, not just tools. Overwhelm the community; each new DevOps tool should leverage the success and technology of the last hence creating a methodology. Even more importantly, because the DevOps tools are closely related, credibility can cross industries and markets and create the momentum required for a tsunami. Once Devops has reached critical mass by providing the total solution in a few industries or markets simultaneously the tsunami is created and DevOps has jumped the chasm.

The tsunami will emerge suddenly with everyone seeming to adopt DevOps all at once. This will result in explosive growth and will pull, willingly or unwillingly, everyone into its path. It’s driven and sustained by the pragmatists. Here, everybody has to get the latest thing all at once; the masses have been converted to DevOps. DevOps needs mass-market success, DevOps needs to become a tsunami and everyone will want to play DevOps.

Now that you know the formula, go forward and create the perfect storm.

If you would like to say something; email parker@devops.com and don’t forget to comment here.

April 10, 2014
Hacking Your Auditor
No, not that kind of hacking, give me a little credit…

Recently I was having a conversation with a consulting friend about DevOps, and he found himself in a bit of a tough situation. The organization he was working with was a pure cloud/DevOps shop, but one that was stepping into having the meet compliance obligations. Needless to say, things weren’t going well with the auditor.

For those of you who haven’t had to deal with the miasma of audits, one of the strongest requirements, for nearly any regulation/contract/agreement, is strong change management. This is especially true after those Enron guys screwed everything up for the rest of us.

Change management makes a ton of sense from both an operational and compliance standpoint. I, myself, have blown things up in production but not being smart about my change management. For compliance, it is absolutely essential. If you are dealing with anything with financial or privacy implications, you sure as heck want to get it into a stable, secure, compliant state and ensure that state is kept over time.

There is a big misconception out there that DevOps-style organizations don’t have change management. That when you push multiple changes per developer per day into production, there is no accountability or control. Now, while you can certainly build things this way, that sure isn’t how I see successful teams doing it. Not even close.

The problem isn’t that DevOps necessarily skips change management, it’s that it looks different. Much of change management becomes automated, and the fundamental nature of changes themselves is different. But think about how you describe DevOps to your traditional IT-focused friends. Now imagine trying to explain this to an auditor working off set standards that are still figuring out if cloud includes a co-hosted Mac Mini duct taped to a rack.

Here are some tips on how to both adjust your process, and explain it to the auditors:
- First, figure out which pieces of your application and operations are in-scope for the regulation. Odds are only financial and privacy related bits are, and you can then apply more-granular controls or logging to these if you need.
- Make sure your continuous integration tool (if you have one) includes some security checks and logs all updates/changes. If you push right from your code repository to production, get a CI tool.
- Keep records of all changes as they flow from dev to test to prod (or whatever you use). Again, this should be trivial to automate and dump the records to a secure log repository someplace.
- Unless your cloud provider tracks all API calls (like CloudTrail does on AWS, although that only works for some services and regions) use an API or web UI proxy like Netflix’s Asgaard. This really is a good security practice anyway, and can actually speed up infrastructure and platform management processes anyway. Keep the logs from this to make the auditor happy.
- If you SSH into instances, you need to log that activity and dump the logs to a central repository.
- Make sure you keep revisions of both all code changes and infrastructure automation changes (e.g. your CloudFormation templates, Chef/Puppet scripts/etc.). This is often built-in, but make sure you configure it correctly.
- Those suggestions now give you logs of all application changes, infrastructure changes, and direct access to instances. These are the things the auditor wants to know exist.
- Filter them based on your in-scope compliance requirements. Such as payment systems components only.
- Explain to your auditor that all changes go through an automated approval system that checks the code, tests it (detail the tests), tracks the change, and then moves it into production. That all server logins and infrastructure changes are logged, as is the complete history of the structure of your entire environment. And that you can provide all the documentation, filtered for the pieces within compliance scope.
This might look like a lot, but odds are very high you are already doing most of this. You keep code revisions, run changes through test suites before they go into production, and track all infrastructure changes. You might not have your own Ops portal/proxy, but you really should for a lot of reasons that have nothing to do with compliance.

The trick is explaining this to your auditor in their terms. They expect manual checks and records. You need to show them that you still have these checks and balances, but they are automated. You still might need to add some pieces, such as an alert if certain parts of your code are changed (e.g. the file with payment processing or account reconciliation), but these will hopefully be minimal.

A good DevOps environment actually has a much better compliance infrastructure in place than traditional infrastructure. The automation and template-driven models give us much better control and insight as compared to manually fixing everything, even when we push more day to day changes. The trick is to document this, add a couple pieces if you need it, and then explain it all to someone with… less than refined technical skills.
March 24, 2014
Don’t let automation be the tail wagging the dog

There is quite a bit of talk right now about DevOps and how important automation is. This could be compared to GM in the 1980s when the company tried to transform the way they built cars through automation and robot. In the early 1980’s GM spent $90B dollars in an attempt to speed up the way cars were delivered to the market. This failed with famous stories of robots painting each other and others welding doors shut. Ultimately this project failed and the robots were scraped. It was later reported that the money lost on the project could have been spent to acquire both Toyota and Nissan at the time.

Automation dominates the conversation when it comes to DevOps. If you do not learn how to build cookbooks you are not doing DevOps. I fear this is going to set companies trying to transform IT off on the wrong track, focused on the outcome and not on the scientific process to get there. Alan Sharp-Paul recently wrote that you cannot automate what you don’t understand. The best way to understand it is in two key ways, change and performance.

Let Science set you free

This was the secret that Toyota used to dominate the automobile industry. They were able to transform all of their employees into little scientist. Constantly tinkering to squeeze our inefficiencies in the system. Every worker was empowered to think how things could be done better.

At the heart of the agile and DevOps processes is the practical application of the scientific method. Think back to your 8th grade science fairs. No not the volcanos but the ones that won best of show. These had a clear hypothesis that stated what they were going to try and prove. They were able to understand what they were going to change and measure the difference between the before and after. They could clearly state what changed and if it was a successful experiment or a failure.

This thinking is key for companies trying to transform to a DevOps IT organization. If a company does not understand the experiments that are running and what the outcomes should be then automation will just make the problem worse. That is why change management and performance management are foundational for transforming to a DevOps culture.

Tracking your experiments

Any good scientist knows exactly what he is working on. He knows every experiment that is running anticipating the outcome. He knows all the details about each change that has been made to see how they impact the outcome. This is the same way that IT should be tracking every change happening.

Every change done in IT is a scientific experiment running. Tracking every change allows for understanding of the outcome of changes within the application. Developers and Operations will be more informed about the cause and effect of their changes. There are multiple different ways that you can track changes but the key is that you are doing it very well before you start to apply automation.

Measuring the outcome

There is no need to track the experiments if there are not being measured. A scientist is very deliberate in how they measure the outcomes. Good enough just does not get this done. This is beyond just monitoring. Monitoring in a lot of cases is after the fact, in production. This is about measuring the difference for every experiment regardless of where it is being tests. This creates the need to test changes all the way to when the first functional tests are done, measuring the changes of every code update. Following it through all of testing and to production. Understanding how each experiment impacts the system as a whole.

Efficiency through automation

Once there is good understanding of all the experiments and a standard platform for measuring the outcome then automation starts to return exponential benefit. Because every experiment it known the lower priority ones can be identified and automated. Risk is better assessed. Processes can start to speed up because there is visibility and accountability. Applications are no longer classified as high risk or low risk only change is classified that way. Determining what can be automated just becomes another experiment that is tried and measured for success and and failure. Automation becomes just another outcome of DevOps to the business and not the route to achieve it.

March 21, 2014
Why Baking Security Into the DevOps Framework is Vital to Its Success

I attended my first official DevOps conference in Israel in September 2013. But I was initially introduced to DevOps by my customers – firewall and network managers who, through their efforts to automate security policy management, found themselves dealing with a new and like-minded set of IT stakeholders – DevOps teams.

They said DevOps was also all about automation – automation that allowed IT to increase their business contributions by enabling new applications and other IT services to be rolled out faster with greater control and efficiency.

Conceptually, it was clear to me that the DevOps value proposition is just as relevant to Security as it is to Operations and Development groups, especially since Security teams apply automation with the same fervor as DevOps.

In fact, using automation to accelerate IT has caught hold in the network security world to fuel a new movement and class of solutions, called Security Policy Orchestration. As the CTO of Tufin, a Security Policy Orchestration company, I could not help but recognize the synergies. I was (and still am) intrigued by DevOps, so when I heard about the conference, I decided to go.

And I am so glad I did.

The alignment between DevOps and Security Policy Orchestration is obvious. They both champion the use of automation to accelerate the pace of business. We Security Policy Orchestration folks are just coming at it from a different area within the organization. Kindred spirits, right?

In theory, absolutely, but in reality…not so much.

The conference as a whole was great. It completely validated my theories regarding the synergies between Security Policy Orchestration and DevOp. But what surprised me was that security was a topic that was not being discussed at all at the conference.   Maybe a hat-tip to application security, but network security was not even on the DevOps radar, at least not at this particular conference.

It needs to be. For DevOps, to really be able to move the needle, we have to work together. If history has taught us anything, it’s that when new applications and services are not created or rolled out with security in mind, at some point we will have to go back and retrofit it back in.

Envision the typical experience:   the development team builds a new application, brings IT Operations in to implement it. Once that is done, IT Ops prides itself on all the corkscrewing it took to make the application work on the actual infrastructure. Except for once the app is in production, it doesn’t work. The troubleshooting begins. Invariably, issues of network connectivity come up. Solving them requires app, dev and security managers to break down the cultural and organizational barriers that exist within their organizations to work together to ensure the application is architected correctly into the network. This starts and the process of change requests between AppDev and IT to get the app up and running.

If that process is not automated, it can take months longer than it needs to resolve everything.   Not to mention any company subject to PCI has to consider terms like secure zoning, a concept that is not second nature to DevOps folks.   If PCI is not an issue, there are plenty of other internal and regulatory mandates and security –related considerations that, if for no other reason than common sense, can exponentially reduce the chances of a company ending up in the headlines due to an APT.

These are just a couple of many scenarios that can be avoided if development, IT and security work together from the get go.

The net/net is that Dev and ops have the same goal but they are siloed, which is slowing down the pace of business – inhibiting it. Collaboration is not some warm and fuzzy team-building thing – it is a fundamental business requirement.

And that’s why I am so excited to have the opportunity to do this blog. I think I can add a unique perspective on how DevOps and Security can work together to reach our common goals.

My next post will explain will dig into what Security Policy Orchestration actually is, so we can begin to tease out the similarities and determine how to build on that to true alignment and partnership.

Stay tuned as the drama unfolds! This should be fun….

March 11, 2014