Maybe you have already heard the terms “Programmable Infrastructure” and “Infrastructure as code”. But even if you have not, if you are interested in DevOps it’s something you need to know. Most new lingo causes me to roll my eyes. But in this case, it’s not just new lingo, it’s an entirely new concept. Besides making the nerdy feel cool when they say it, these terms describe a new idea of what an application consists of. (more…)
Category: Blogs
-
An open letter to Jeff Knupp
Dear Jeff,
First off, nice recovery. I commend your gesture to return to positivity and make what to my eyes, looks like a kind and altruistic offer to an underserved class of engineers. I personally sent your offer to all the female engineers on my team and got very good feedback from several of them.
Some people will try to tear you down and criticize your offer to make a difference in the lives of female engineers as patronizing or sly self promotion, but I take you at your word and applaud your offer as both poised and graceful.
Given your offer to one underserved population in IT, I thought you would be OK with some constructive feedback from one dev guy concerned with the welfare of IT workers to another and hope you take this feedback as it is intended, with the same eye towards a desire to be a force for positive change to the plight of all IT workers, everywhere. Please remember that this is just my perspective. I could be wrong, and often am. So, here goes:
DevOps does not mean everyone has to do everything
DevOps is not just for startups.
I work in a big shop and we have all the sorts of disciplines you call out in your posting on how DevOps is killing the developer. We are also embracing the DevOps movement not to force anyone to take on roles they aren’t optimized for, but rather to inspire them to expand the boundaries they have set for themselves and others in an effort to make things better for everyone.
We want QA professionals to know how to script and automate tests. This is good.
We want operations staff to know how to script and automate changes. This is good.
We want release managers who know how to triage problems, automate configuration management and instrument the throughput of the IT machine. This is good.
We want developers and architects who know how to go deep when required to help solve performance problems. This is good.
You may disagree and I respect your right to do so. I just am not that interested in hiring an IT professional of any type who says “That ain’t my job”. That’s not a guy or gal who gets things done or makes things better.
I’m not saying that everyone has to be able to do every role or even that developers must branch out. I am saying that making others wrong for increasing their value prop is not likely to do anyone any good.
Injustice anywhere is a threat to justice everywhere
Using narratives of which type of IT worker is more important than another type of IT worker does us all harm. All of us. All men. All women. All IT workers. All Developers. All operations staff. All employees from any department of any company no matter the size.
Now, I realize that this is a bold statement (even from the guy who wrote “Sharepoint is Crack and Microsoft is the Pusher”), and I have no problem admitting that you probably have no desire to harm any employee anywhere (Except for the guy or girl who invented IVRs. I can understand wanting to do bodily harm to whoever did that). I would ask that you consider a few ideas before writing me off as a troll.
1) The schism upstream is the enemy to be fought, not the people downstream from you
You may not be aware of the schism to which I refer, but I would argue that you are probably very familiar with it’s effects. The deep and distrusting divide that exists between enterprise IT and the customers of enterprise IT across the industry is the single most destructive and demoralizing artifact in the lives of IT employees everywhere.
Have you ever worked in a shop where IT employees are casually referred to as “technical” while marketing, product and design employees are referred to as “creative”. Have you ever been exposed to a shop where the developers were treated and talked about like they were worker insects who’s job it was to keep their mouths shut and code what they’ve been told to?
If you have somehow managed to have an IT career and avoided exposure to this phenomena that seems to pervade so many (but not all) IT shops around the world, I would ask you to pull back and reflect for minute. Spend some time and have some gratitude for the fact that you have not been either witness to, or been the subject of the message: “we are the so and so team and you are not on our team so you have no say IT boy, go back to your nerd closet and build what I tell you to build”.
If you are familiar with this idea that robs developers everywhere of the right to claim their creativity and basic need for self realization through recognized craft, then I ask you this: Aren’t you all sick and tired of the completely loathsome ideas that we are not “creative” because we are IT? Doesn’t that make you want to throw up? Don’t you want to go postal when you hear people push developers down the collectively agreed to illusion, referred to by you as the “totem pole”, that creates a veritable corporate caste system?
I would run out of my word limit if I were to try and cite the publications that point out that silos are the prime roadblock to unlocking business potential, but I will point you to a few modern classics which make the case much better than I ever could:
- The Advantage: Why Organizational Health Trumps Everything Else In Business by Patrick Lencioni, best known for The Five Dysfunctions of a Team: A Leadership Fable
- Silos, Politics and Turf Wars: A Leadership Fable About Destroying the Barriers That Turn Colleagues Into Competitors also by Lencioni
- Orbiting the Giant Hairball: A Corporate Fool’s Guide to Surviving with Grace by Gordon MacKenzie
This isn’t just about IT workers. This is about every employee everywhere who has their role demeaned and belittled by another. There are three basic roles inside of most companies in the products and services business. Sell the work, do the work, support the work. One thing in common to them all is that if any of the three roles don’t get done well, then everyone is at risk of not being paid.
We are all necessary and none of us is sufficient at any real corporate scale. The moment we allow division (i.e. the totem pole) between ourselves, we have ceded the ground that other people upstream from us or up the hierarchy from us have the right to demean whomever they want to.
You can’t possibly eliminate injustice via injustice
2) The schism will always exist if we reinforce it
Being of service to another is an honor and all who claim to want to improve the lives of workers anywhere, should hold themselves accountable to not belittle the contributions of others.
Every single employee has the potential to be innovative and creative in their work. We constantly create and reinforce the concept of commodity items and jobs and then are paradoxically shocked and surprised by:
- how the unmotivated and unengaged masses of people in jobs frustrate us with their lack of passion for doing a job well
- how new people constantly create a “new and improved” version of something perceived to be commoditized forever and then dominate the market because they believed it was possible to reinvent a new level of quality
Innovation is not a department. Creative is not a department. Aren’t you all sick and tired of narratives of ownership and control from other teams who claim to own these abstract qualities when they are delivered at you?
When we the development community adopt the same basic ideas, we allow it to continue.
If you want to world to be different, than you must have the courage to be willing to be one of the first ones who fully live in that new different world.
What world are you working towards?
3) How will you know your ideas and contributions are better if you set up barriers that make it harder for others to contribute?
I am privileged to lead one of the best dev teams in the world. I am privileged to work in a shop that is committed to breaking down the barriers to collaboration and cross-pollinating innovation. To keep us all all at our best, we have to be brave enough to say that our dev team and our creations can more than likely withstand any criticism or contribution from any team up or down stream and if they don’t because some other idea was better, so be it.
I won’t take the cheap and easy shot and ask “Are you afraid that the outside idea is better and that you will seem weak for having a lesser idea?”, because I doubt that this is true. I honestly think this kind of resistance has roots in the human condition itself. It probably has more to do with fear of the clock running out or the possibility of lesser ideas winning the day than of fear of being shown either as “weak” or “not as smart”. With these dreadful possibilities in mind, I would ask you consider this: The way to maximize the likelihood of these undesirable outcomes is by allowing others to say we are closed off to outside ideas. This is what leads to goat-rodeos, witch hunts and ultimately those with more authority than we have removing the little authority and control we actually have.
Let the best ideas win, not the ones made by those with the authority backed up by a title.
If you truly are looking for the best and most game changing ideas to help your enterprise, be aware that the richest source of these ideas will inevitably come from people who’s blinders have not been firmly affixed by the discipline they are embedded in. This is a widely accepted historical truth laid out by Thomas Kuhn in the timeless masterpiece The Structure of Scientific Revolutions and if you doubt me, remember that Albert Einstein was a patent clerk when he gave humanity the theory of relativity.
I’m not trying to make you out to be a bad guy or anything close to one. In fact I praise you for having the guts to articulate a fairly controversial point of view that you believe in.
When I write this letter, I’m not fighting for or against DevOps in particular (even though I am DevOps fan). I’m fighting for the end to the schism between Business and IT.
I’m fighting for the end of narratives of control and authority.
I’m fighting for the end of the un-empowering messages that authority gives to individuals every day.
If you want to know why enterprise IT employees everywhere are asked questions from their business partners that display lack of trust over and over and over; This is why…because the schism exists.
If you want to know why enterprise IT can’t complete things it starts; This is why…because the schism exists.
If you want a world where people trust IT employees expertise and allow IT shops to complete work without questions of minutia that spawn from lack of trust, we must not embody what we oppose – narratives of authority and ownership combined with an “us vs them” mentality.
I don’t say any of this to scold you Jeff. I say this as an attempt to call upon you to join us . Help us tear down the walls. We need more help. We need you.
Respectfully,
Stephen Fishman – Creative IT Professional
-
DevOps makes coding more social, social coding makes better code
One of the main principles of DevOps is to break down silos. What’s more like a silo than the image of a lone developer, shutting himself off from the world in order to code more and interact with people less? “I’m just here to write code and that’s all I do.” There’s nothing collaborative or DevOps about that scenario and unfortunately, it’s one that is all too frequent within organizations.
Enter social coding. This is a concept that didn’t exist twenty, or even ten, years ago and one that continues to grow in popularity. At its essence, social coding is an approach to software development that puts the focus on formal and informal collaboration. There are many benefits to social coding but the most important is probably the opportunity it presents for mutual growth and learning. Developers are allowed to collaborate in an environment that fosters discussion, creation and sharing.
When talking social coding, you can’t avoid a mention of GitHub. Take everything you love about your favorite social network and then apply those same ideas to a code repository. That’s basically what GitHub did, and continues to do, with much success. You can create a profile, follow or message someone, see what others are doing with their code (or yours) and even go so far as to subscribe to RSS feeds of activity streams. GitHub started to take off a mere five years ago and is now held up as one of the more successful examples of how social coding can be commercialized.
Companies like GitHub and SourceForge are leading the way in revolutionizing how developers commit and share their code but the foundation of social coding still rests with code review in a public forum. As Mark Chu-Carroll puts it…
“If you’re programming and you know that your coworkers are going to look at your code, you program differently. You’ll write code that’s neater, better documented and better organized — because you’ll know that people whose opinions you care about will be looking at your code. Without review, you know that people will look at code eventually. But because it’s not immediate, it doesn’t have that same sense of urgency, and it doesn’t have the same feeling of personal judgement.”
What’s next for social coding? After it’s start in the open source community, this is one idea (like many others) that is heading towards the enterprise market. Companies are beginning to recognize that even internal projects can be managed and treated as open source efforts. From wikis to crowdsourcing, there are many ways that the enterprise market can take advantage of social coding. The end user is the one who stands to benefit the most from such a change.
Dave West, a Principal Analyst at Forrester Research, has this to say about social coding:
“We’re social animals, and the development of stuff tends to be social in nature. We think a better way of thinking about it is as social development. It’s not just about coding; it’s about enabling business change and delivering business value in a social way. It will be interesting to see how companies broaden the idea from social coding to social development. Social coding is a fundamental change.”
-
Nagios is not a monitoring strategy
When I visit clients to talk about DevOps, I usually ask them what their monitoring strategy is. Too often, the answer I hear is “We use Nagios”. I think Nagios is a great tool, but it sure is not a strategy. Nagios does a good job of monitoring infrastructure. It will alert you when you are running out of disk, CPU, or memory. I call this reactive monitoring. In other words, Nagios is telling you that your resources are getting maxed out and you are about to have issues. Proactive monitoring focuses more on the behavior of the applications and attempts to detect when metrics are starting to stray away from their normal baseline numbers. Proactive monitoring alerts you that the system is starting to experience symptoms that can lead to a degradation of performance or capacity issues which is more preferable than Nagios telling you are about to be screwed. With reactive monitoring, it is not uncommon that customers start complaining about the same time that the Nagios alerts start going off. The goal of proactive monitoring is to head off issues so that customers don’t even notice.
The next question I ask is “What things are you monitoring?” A typical answer usually revolves around various infrastructure assets and databases. That’s a good start but there is much more to consider. But first, let’s talk about why proactive monitoring is so critical. In the pre-cloud days we used to ship software to our customers where they would install the software, perform capacity planning tasks, manage the infrastructure, and operate the day-to-day activities. Once we shipped the code we were done. In today’s world, we are no longer shipping product. Instead we are delivering services that are always on. The customer no longer owns and operates the infrastructure and the software. Instead they pay for a service and expect that service to run reliably all the time. To meet those expectations, we need a more robust monitoring strategy. We need to monitor more than just the infrastructure.
A good monitoring strategy starts by identifying all of the actors who needs access to data and all of the categories of data that needs to be tracked. Some metrics are monitored in real-time while others are mined from log data. Every good monitoring strategy is accompanied with a sound logging solution. In order to perform analytics to predict trends within the data, one must collect various data points ranging from customer usage activity, security controls, deployment activities, and much more. The following presentation goes into much more detail about the different areas that should be monitored and why different actors need these data points to perform their jobs.
[slideshare id=33320431&doc=monitoringstrategies-0408014v1-140409081614-phpapp01]
The bottom line is, before building in the cloud, it pays to invest some time into a sound monitoring strategy. I have seen too often where teams don’t think through how to support these highly distributed, always on SaaS solutions and end up delivering software that does not meet the reliability and quality expectations of customers. Monitoring provides feedback to developers, product owners, operators, and even customers so that systems can continuously be improved. Nagios is great, but there is no single monitoring solution that can implemented to effectively operate today’s always on services. -
Not all collaboration types are equal
Talking about collaboration in general terms is easy. I’ve done it and so has everyone else. It’s time now to dig deeper into the specifics of collaboration, meaning the different types – asynchronous vs. synchronous, vertical vs. horizontal – and how different companies fit into the magic collaboration quadrant. And yes, the quadrant is indeed magical.
Let’s start with the two types of collaboration that are time-relevant, asynchronous and synchronous. Asynchronous collaboration takes place outside of real-time. The best example of a ubiquitous asynchronous collaboration tool is email. You send an email and the receiver has time to digest the information. The drawback to asynchronous communication is that it’s inadequate as a means of communicating a sense of urgency. You have to wait for someone to respond to an email and that lack of immediacy can cause damage in troubleshooting or problem solving situations. Email replies tend to include the original message for context because you can’t rely on the timing of the message to keep the context.
On the other side of that collaboration coin, you have synchronous collaboration. Synchronous communication takes place like a conversation and more closely resembles a face-to-face interaction. A chat session is the ideal illustration of this concept. The obvious benefit to synchronous collaboration is the ability to send and receive information right away. This adds to the feeling of immediacy and tends to provide a more effective means of working together to solve issues in real-time.
Now that we have that covered, it’s time to move to the other defining aspect of collaboration: whether it’s vertical or horizontal. Vertical collaboration consists of problem solving tools that are domain-specific. An example of a vertical collaboration tool is one built for certain parts of the business like Salesforce for the Sales & Marketing teams or Rally for the development team. Horizontal collaboration consists of non-domain specific tools, meaning that they have potential use across all teams of an organization. Project management tools like Campfire or remote meeting solutions like Join.me or WebEx are examples of horizontal collaboration tools which tend to be widely adopted but lack the ability to really go very deep in helping in a domain-specific way.
Check out the quadrant I created to better show where different companies currently fit into the ever-growing collaboration landscape, based on the definitions provided above.
When looking at the state of collaboration tools, it’s interesting to note where most companies land within the quadrant. Why so few vertical synchronous ones? I’ll answer that next time.
-
Automation versus Orchestration
Yes, Virginia, there is a difference.
One of the things devops practitioners are tasked with is the provisioning and configuration of all sorts of infrastructure. Application servers, web servers, load balancers, proxies and database servers are among the lengthy (and no doubt growing) list of “boxes” devops needs to get up and running to support just about any given application today. One of the key value propositions of a devops approach to operations is that it can reduce the time it takes to get applications to market by getting them up and running in production faster. That’s increasingly important as we move into the Era of Things, driven by extreme connectivity of everything, because all those “things” need are going to need to be talking to applications on the back side.
So speed is of the essence, but not at the cost of accuracy. We’re encouraged, then, to automate tasks and orchestrate processes to achieve that speed. The side effect of codification of tasks is higher accuracy (aka fewer mistakes).
So far so good.
But you may notice that I have (and will continue to) clearly delineate between “automate” and “orchestrate”, between “task” and “process”.
The reason for that is that, simply, they are not the same thing. We should strive not to confuse these two because while the two share some benefits, there are other benefits that are achievable only by one or the other – or the combination thereof.
Process optimization, for example, can’t be achieved by simple automation. Automation is concerned with a single task – launching a web server, configuring a web server, stopping a service. Orchestration, however, is concerned with automating, if you will, the execution of a workflow – of a process. A provisioning process may be comprised of multiple tasks and involve multiple systems. An “application” is not just a single server, after all, it’s likely several servers – web, app and database in a traditional three-tier architecture.
The goal of orchestration is not just to automatically execute a service, which brings speed to the deployment process and gets applications into production faster. It also affords an opportunity to streamline – to optimize – those processes for even greater gains in deployment velocity.
One of the simplest ways to optimize a process is to eliminate repetitive steps. This entry on Vagrant explains it fairly well :
“We could just SSH in and install a webserver and be on our way, but then every person who used Vagrant would have to do the same thing. Instead, Vagrant has built-in support for automated provisioning. Using this feature, Vagrant will automatically install software when you
vagrant upso that the guest machine can be repeatably created and ready-to-use.” [emphasis mine]— Vagrant Docs “Getting Started | Provisioning”
Now certainly the step of installing a web server on a new box with vagrant is hardly all that time consuming, it’s just a single line of text. But every line of text must be manually typed, and that introduces the possibility of fat fingering the command, which either consumes time in fixing it or introduces an error into the system. Neither are desirable results.
Ensuring that the process is repeatable and orchestrated by automating all the tasks that comprise that process an optimize it by eliminating redundancy. That results in greater deployment velocity.
Recognizing that processes should be comprised of composable tasks offers a higher degree of reuse across operations. The task of provisioning and configuration a web server with the appropriate base software means consistency, aka standardization, across all applications. Processes that take advantage of standardization result in greater infrastructure stability, another benefit often cited by organizations that have adopted a devops approach to operations.
The key here is to recognize the difference between automation and orchestration and how they relate to one another. Automation is about codifying tasks and orchestration is about codifying processes. Orchestration takes advantage of automation by reusing these basic building blocks.
There is a difference, and done right, the two play off one another to result in streamlined, more accurate execution of operational processes that speed up successful application deployments.
-
DevOps Needs a Tsunami To Jump The Chasm
All start up business endeavors must go through the phase of crossing the chasm. Most of those business endeavors usually fall prey to the chasm. The chasm is this black hole that everyone claims to understand but no one truly does. We have all seen businesses cross the chasm or fail to cross the chasm and fall into the black hole. But no one seems to understand why and how that crossing happens. The bestselling book by Geoffrey A. Moore titled “Crossing the Chasm,” is all about the heart and soul required to get early stage technology across the chasm, from early adopters to mainstream customers.
There is a big difference between people who are enthusiastic to try leading edge technologies and the rest of the inhabitants, who tend to be much more conservative. The reasons that people spend time trying to figure out how to cross or jump, if you will, the chasm is because that black hole is what stands between them and making their fortune. The early adopters are just a small sample of the market and there is no money in building a business or technology around them. To make real money, you need to cross over into the mainstream.
In Geoff’s model, the innovators are all about technology. You know those people, when a new iphone is released, they spend days standing on line to get latest and greatest new technology. The early adopters are the visionaries that are always on the bleeding edge. The mainstream are the pragmatists, the people who practice risk avoidance. These pragmatists are the key to jumping the black hole, they move the needle.
DevOps first order of business was getting on the curve and to that end DevOps has evangelists. But, understand that what worked in converting these early adopters probably won’t work for the pragmatists. These DevOps evangelists and visionaries want DevOps power tools, as opposed to the pragmatists who simply want to push the button and say “that was easy”.
By all appearances DevOps has been accepted by the early adoption set. Just how does DevOps cross the chasm? Jumping the black hole after all can be quite a daunting task. To do so DevOps has to create what I refer to as the tsunami effect.
The Tsunami effect is where DevOps gains acceptance in multiple industries or markets in what seems like a gold rush, it’s here that fortunes are made. More accurately, it’s where cult communities transform overnight into mainstream communities. Many technologies simply create a single tornado which remains in a silo forever never gaining enough momentum to really emerge. Another common path what I call an earthquake, there is a lot of hype but it tends to fizzle out quickly, if not tended to properly. Right now DevOps has all the appearances of an earthquake, Devops evangelists have created a lot of buzz, but acceptance beyond a set of power tools is still a huge question.
Considering DevOps is in the earthquake phase, as opposed to the tornado phase, DevOps cannot be successful converting one group at a time. The strategy in the tornado phase is convert industries and markets one at a time, via tornados forming within industries or markets. The tactical execution is to get smaller industries or markets committed and therefore the larger industries or markets will then embrace the movement. That is simply crossing the chasm. DevOps has too many cultural implications to convert believers over a long period of time; DevOps requires reorganization, not a subtle shift.
Since I consider DevOps to be in the earthquake phase, crossing the chasm is not an option. DevOps must jump the chasm in a surge of overwhelming momentum, hence creating a tsunami. DevOps has created a very closely knit culture, DevOps needs to push hard to create a tsunami and flood the pragmatic population. But, be advised, DevOps has to be all but universally accepted, there is no almost in bringing together Dev and Ops. But if DevOps wins, it wins big, DevOps will go from being a niche culture to a truly mainstream practice.
So what’s the strategy to create a tsunami? And more importantly, how do you move beyond the earthquake phase? Here’s how I see it according to Geoff, with my DevOps spin: First, DevOps must clearly establish that the mission-critical software development process is broken and is causing management enough pain that it would gamble on unproven methodology that is populated with a bunch of prima donnas. I am not convinced the DevOps cult has clearly established this as fact. Is the development and operations process broken or have just new power tools been created to improve speed and accuracy? Second, DevOps needs to present the total solution, not just tools. Overwhelm the community; each new DevOps tool should leverage the success and technology of the last hence creating a methodology. Even more importantly, because the DevOps tools are closely related, credibility can cross industries and markets and create the momentum required for a tsunami. Once Devops has reached critical mass by providing the total solution in a few industries or markets simultaneously the tsunami is created and DevOps has jumped the chasm.
The tsunami will emerge suddenly with everyone seeming to adopt DevOps all at once. This will result in explosive growth and will pull, willingly or unwillingly, everyone into its path. It’s driven and sustained by the pragmatists. Here, everybody has to get the latest thing all at once; the masses have been converted to DevOps. DevOps needs mass-market success, DevOps needs to become a tsunami and everyone will want to play DevOps.
Now that you know the formula, go forward and create the perfect storm.
If you would like to say something; email parker@devops.com and don’t forget to comment here.
-
A Winning DevOps Approach for Established Enterprises: Pace-layered
Does your bank offer the ability to deposit a check by taking its picture with your smartphone? When that feature debuted a few years back, I remember thinking it was interesting and convenient enough that if my bank didn’t offer it soon, I’d be willing to switch to one that did.
I bring this example up for two reasons: one, it’s a microcosm of the larger race to innovation that is happening in every company and industry as we speak; and two, it involves banks, which are good touch-points for a discussion about the unique challenges established enterprises face as they attempt to implement DevOps practices in their IT organizations.
We know that DevOps means incorporating best-practice application delivery methods that improve communication, integration and collaboration between development and operations, but what is its ultimate goal? Quite simply, getting high-quality services to market faster than competitors – at a cost that aligns with both IT budgets and business goals. And as my check-depositing story illustrates, if you’re not first in the market, you at least need to be nimble enough to catch up, or you risk losing customers.
For long-time enterprises, this can be easier said than done. In my previous blog about DevOps culture, I used Etsy as an example of an organization that has had great success with its DevOps transformation, but the e-commerce company has been around for less than a decade. Its young IT architecture is apples to the oranges of established enterprises, whose complex environments often comprise a mix of legacy and modern technologies that have been cobbled together over time.
As a result, the DevOps journey is typically longer and more difficult for such organizations – but that doesn’t mean they can afford not to pursue it.
Improving Deployment Practices at Every Infrastructure Layer
When you look at the IT infrastructure of an established enterprise, you normally find three layers of technology. Let’s call the first “systems of record.” These are the legacy transactional systems (often mainframes) that have been with a company since the beginning. They do their jobs well, but their size and inflexibility make them resistant to the type of Agile development that DevOps typically embraces.
The next layer is “systems of differentiation,” or the various hardware, software and middleware components an organization accumulates as it grows, merges with/acquires companies and enters new markets. While these systems are often more Agile-ready than the systems of record, their decoupled/distributed nature can complicate and slow down development and deployment processes.
Finally, there is the “systems of innovation” layer, which is made up of mobile and Web applications and leverages the latest technologies. Because of its light footprint and flexibility, this layer is the most adaptable to DevOps practices. In fact, as you move up the layers from systems of record to systems of innovation, the holy grail of high-quality, fast and low-cost deployment becomes more and more attainable.
What does this mean from a macro perspective? Basically that established enterprises need to take a pace-layered approach, where application delivery occurs at different paces within the different areas of their architectures. The point, though, is to improve in each layer.
So if an organization pushes out a mainframe update once per year, perhaps it can move to a bi-annual schedule (with support from the right people, processes and technology, of course). On the other hand, a Web team that publishes updates once or twice a week could leverage DevOps tactics and technologies to adopt a daily update cadence. Bottom line: Different deployment paces are fine across layers, as long as improvements are occurring across the board.
Kick-starting the Journey with Service Virtualization
Now that we know why the DevOps journey can be difficult for established enterprises, let’s look at how one such organization, namely Union Bank, overcame these challenges and achieved success.
According to an article on Bank Systems & Technology, Union Bank was looking for ways to compete with top-tier banks, so it began “acquiring other institutions, updating legacy systems and launching new front-end applications.” In order to support these efforts, the bank implemented a service virtualization solution, which simulates constrained or unavailable systems across the software development lifecycle (SDLC). This allowed its developers, testers and performance teams to work in parallel for faster delivery and higher application quality and reliability.
Here are some of the results Union Bank has achieved since:
- Cut testing environment setup time from 42 days to 3 days
- Slashed three and a half months from mobile and Web application development time
- Reduced the cost of testing cycles by 65 to 85 percent
Obviously, a service virtualization solution is only one piece of the DevOps puzzle, but Union Bank stands as a powerful example of how a long-standing enterprise can leverage such a tool to be more innovative and nimble in the marketplace.
-
Dev, Ops and Security Collaboration: Bring the body and the mind will follow
Complexity has a way of muddying even the clearest of waters, and this has certainly been the case with IT Operations. While Dev, Ops and Security teams share a common purpose, the silos between them are so deeply entrenched, it’s easy to see how they have forgotten that they are working towards the same end. Fortunately, the rise of DevOps has provided the opportunity to break down those silos and leverage automation to increase agility, efficiency AND security. But accomplishing that requires an alignment that is only possible through a sustained, collaborative effort. So, I wanted to dig into more detail into the stakes Security, Dev and Ops all share, as I am certain the benefits of working together will become startlingly apparent.
First, let’s take a look at where each group is coming from:
- Developers want fast, automated application deployments. They only care about infrastructure as it relates to their ability to roll out applications.
- Operations folks (network managers) live, eat and breathe infrastructure. When it comes to application deployment, they need clear technical requirements to reduce “redos.”
- Security folks need to make sure policy & process are enforced with auditable security controls that also can ensure compliance.
A big part of the problem is that until now, security has been introduced at the tail end of the development lifecycle, after risky practices have been introduced into application delivery architecture. Security comes in after the fact, implementing fixes and controls the best they can. Obviously this slows things down, and creates a dynamic in which security is perceived as an obstacle, and security teams are pigeonholed as “the folks who are always saying no.”
The good news is that automation can ensure that security is woven into Dev and Ops processes from the get go, and to great benefit. But technology alone won’t change the underlying problem. Breaking down silos requires cultural shifts that don’t happen overnight. The first step to changing these dynamics is to better understand the relationship that exists between applications and infrastructure. This does not mean security folks need to become domain experts in agile development, or that developers need to become network security experts. In fact, even a basic understanding of the collective impact of these trends should create the common ground needed to move towards a more collaborative model.
As the DevOps movement reaches critical mass, the increased agility, greater efficiency and substantial security improvements it can deliver are being talked, written and Tweeted about in depth, and with increasing frequency. And most importantly, these benefits are not just marketing buzzwords – they are tangible and multifaceted, as is illustrated in Steve Hall’s Jan 2014 blog post:
“Business people completely understand ‘release my product faster’, ‘time to market’ and ‘make more money’ which are some of the outcomes that DevOps pontificates on. This is why DevOps as a movement has a higher probability of succeeding than other investment choices because there is a direct correlation between it and top/bottom line revenue. So if you’re an InfoSec leader, a good bet is to align yourself to the DevOps initiative (or spearhead it yourself for that matter) and help the business understand the value of security in a way that doesn’t measure things by # of incidents, time lapse from vulnerability to patch, or compliance score.”
As for the shared stakes between Dev and Ops, check out John Allspaw and Paul Hammond’s Velocity 2009 talk on how cooperation between Dev and Ops enabled 10+ deploys per day at Flicker. If that doesn’t demonstrate what “increased agility” looks like, I don’t know what does.
These are just a sampling of the myriad ways Security, Dev and Ops can reap greater benefits by working together. But…. it takes time and effort to get the ball rolling. Perhaps the best way to get started is to make sure Dev, Ops and Security start talking, literally. Once a week. Every week. No matter what! Get the bodies there, and I promise you, their minds will follow.
Based on our customers’ experience, a few quick wins is all that is needed to get the ball rolling. Once Dev, Ops and Security folks experience the upside for themselves, those silos will start to crumble. In my next post, I’ll provide some examples where automation can enable Dev, Ops, and Security to gain those quick wins and build momentum for further cultural change.
-
Hacking Your Auditor
No, not that kind of hacking, give me a little credit…
Recently I was having a conversation with a consulting friend about DevOps, and he found himself in a bit of a tough situation. The organization he was working with was a pure cloud/DevOps shop, but one that was stepping into having the meet compliance obligations. Needless to say, things weren’t going well with the auditor.
For those of you who haven’t had to deal with the miasma of audits, one of the strongest requirements, for nearly any regulation/contract/agreement, is strong change management. This is especially true after those Enron guys screwed everything up for the rest of us.
Change management makes a ton of sense from both an operational and compliance standpoint. I, myself, have blown things up in production but not being smart about my change management. For compliance, it is absolutely essential. If you are dealing with anything with financial or privacy implications, you sure as heck want to get it into a stable, secure, compliant state and ensure that state is kept over time.
There is a big misconception out there that DevOps-style organizations don’t have change management. That when you push multiple changes per developer per day into production, there is no accountability or control. Now, while you can certainly build things this way, that sure isn’t how I see successful teams doing it. Not even close.
The problem isn’t that DevOps necessarily skips change management, it’s that it looks different. Much of change management becomes automated, and the fundamental nature of changes themselves is different. But think about how you describe DevOps to your traditional IT-focused friends. Now imagine trying to explain this to an auditor working off set standards that are still figuring out if cloud includes a co-hosted Mac Mini duct taped to a rack.
Here are some tips on how to both adjust your process, and explain it to the auditors:
- First, figure out which pieces of your application and operations are in-scope for the regulation. Odds are only financial and privacy related bits are, and you can then apply more-granular controls or logging to these if you need.
- Make sure your continuous integration tool (if you have one) includes some security checks and logs all updates/changes. If you push right from your code repository to production, get a CI tool.
- Keep records of all changes as they flow from dev to test to prod (or whatever you use). Again, this should be trivial to automate and dump the records to a secure log repository someplace.
- Unless your cloud provider tracks all API calls (like CloudTrail does on AWS, although that only works for some services and regions) use an API or web UI proxy like Netflix’s Asgaard. This really is a good security practice anyway, and can actually speed up infrastructure and platform management processes anyway. Keep the logs from this to make the auditor happy.
- If you SSH into instances, you need to log that activity and dump the logs to a central repository.
- Make sure you keep revisions of both all code changes and infrastructure automation changes (e.g. your CloudFormation templates, Chef/Puppet scripts/etc.). This is often built-in, but make sure you configure it correctly.
- Those suggestions now give you logs of all application changes, infrastructure changes, and direct access to instances. These are the things the auditor wants to know exist.
- Filter them based on your in-scope compliance requirements. Such as payment systems components only.
- Explain to your auditor that all changes go through an automated approval system that checks the code, tests it (detail the tests), tracks the change, and then moves it into production. That all server logins and infrastructure changes are logged, as is the complete history of the structure of your entire environment. And that you can provide all the documentation, filtered for the pieces within compliance scope.
This might look like a lot, but odds are very high you are already doing most of this. You keep code revisions, run changes through test suites before they go into production, and track all infrastructure changes. You might not have your own Ops portal/proxy, but you really should for a lot of reasons that have nothing to do with compliance.
The trick is explaining this to your auditor in their terms. They expect manual checks and records. You need to show them that you still have these checks and balances, but they are automated. You still might need to add some pieces, such as an alert if certain parts of your code are changed (e.g. the file with payment processing or account reconciliation), but these will hopefully be minimal.
A good DevOps environment actually has a much better compliance infrastructure in place than traditional infrastructure. The automation and template-driven models give us much better control and insight as compared to manually fixing everything, even when we push more day to day changes. The trick is to document this, add a couple pieces if you need it, and then explain it all to someone with… less than refined technical skills.
