devopsy.kinsta.cloud

Category: Blogs

SecDevOps: Security Automation By Example – The Firewall Change

Security Automation By Example

The Firewall Change

Just when you thought DevOps was the new black, along comes SecDevOps. Yes folks, like most things in life, the new cool is already here. Before I move on to trying to explain SecDevOps, please accept my mea culpa because for many people DevOps is yet to be clearly defined. I can imagine your frustration that I’m going to try and explain a new term based on a loosely defined term.

I’m not going to try and provide a complete definition of SecDevOps, but instead I’d like to provide what I believe to be important attributes of the SecDevOps movement. In this post, lets discuss security automation and use a firewall change process as an example.

One important attribute of SecDevOps is heavy use of security automation. In particular when I think security automation, I tell people to use IFTTT thinking. Stolen from the service whose tagline is “put the Internet to work for you”, I explain security automation simply as “if this, then that”. Its a highly simplistic, but poignant way to imagine ways in which one would automate workflow.

A simple act of changing a firewall has huge potential for security automation. The SecDevOps firewall change process goes beyond changing a few lines in a config, but also provides an independent and automated verification process that most organizations don’t perform today.

Consider what happens when a firewall policy is changed. First the policy change request is reviewed and approved. Then the policy change request is assigned and acted upon. Many times, but not always, the change is verified by another person through configuration review.

Security automation takes this process one step further. Imagine when a firewall is changed that a security audit event is automatically logged. Next, an automation component notices the firewall change and automatically spins up a remote scanning instance in a cloud provider. Next, that scanning instance is contacted using APIs to perform a remote network scan. The scan completes, results are exported and compared with prior scan results automatically. Finally, the automation component automatically updates the firewall change request in a ticketing system showing the scan results. We have now completed an automatic, independent and autonomous firewall change verification. That simplified yet powerful workflow is security automation and completes what many people refer to as a closed loop.

The new SecDevOps method for a firewall change not only provides a closed loop process, but does so by automating the verification steps. Joining traditional security processes with new API driven cloud services is a true force multiplier that many organizations will soon wish they had.

In upcoming posts, I’ll also discuss some of the other important components of what I believe make up the new black of SecDevOps.

April 30, 2014
Does DevOps spell death for Line-of-Business (LOB) Applications?

Perhaps wrongly, “DevOps” has been associated with “new” and “that thing for the cool kids” and the “hipster developers”. This is used both as an excuse and as a source of animosity against the movement. But more subtly, I sometimes observe IT departments that acknowledge DevOps as a thing, but claim it wont work for their applications. I always pause and ask: What are the applications they are referring to, and are they correct?

(more…)

April 30, 2014
You have to crawl before you walk…

In previous posts I have explained what Security Policy Orchestration is, why DevOps folks should care, and how it can help facilitate the cultural change necessary for organizations to reap the long-term benefits of the cloud and virtualization. In this post, I’ll provide a few examples of how Security Policy Orchestration can create some “quick wins” – wins that create common ground between Security, Dev and Ops, and set the stage for higher profile wins with significant business impact.

As infrastructure becomes increasingly complex it gets harder to decipher. In software we call this Readability – it determines our ability to support the code over time and to ensure security. The same concept applies to networks – the more complex they are, the harder it is to use them, maintain them and ensure security.

The primary value of Security Policy Orchestration is providing a holistic view of the infrastructure. It “sees” the entire network and creates an index, which enables network administrators to perform quick queries across their environment providing “needle-in-a-haystack” search capabilities. While it might seem obvious, gaining visibility into the environment is a very powerful tool that can deliver huge benefits, especially when the environment is complex and subject to constant changes.

Let me give you a few use cases:

Audits: As my security brethren know, preparing for an audit can take weeks (if not months).   However, once the network infrastructure is indexed, meaning you have structured records of the network elements (devices, interfaces, routes, security rules…), the kind of documentation required for security audits for PCI, SOX, NERC, and other regulations is literally a few mouse clicks away. Imagine finding all applications flows involved in a PCI flow in one click, or, even better, through a REST API call.

Many of Tufin’s large enterprise customers report that their internal audit requirements are more stringent and time consuming than external compliance mandates, and require customized reporting, which they often have to script themselves. Policy Orchestration solutions enable them to be much more agile – to the point where security no longer becomes a “showstopper.“

A successful and effective audit (one that actually improves security) requires a collaborative approach across Dev and infrastructure – having the needed data at hand ensures agility, which is crucial for embedding this into the continuous process.

Server Migration: Many enterprises and service providers are currently tasked with server migrations to virtual data centers or cloud environments. The challenge they face is identifying the talkers, connections and protocols that are needed to maintain connectivity and business continuity. Firewall policies are often the best place to find this information because their important role in “attaching applications to the network”. However, it’s a labor intense task. Even if you have only 10 firewalls, with a few hundred rules and a thousand objects, identifying the flows is going to take a while. Now multiply that by 100 migrations per month – it’s virtually impossible.

However, once you automate the process, it becomes quite easy. With change automation in place, all you have to do is click “Find”, or, even better, call an API, and you will know the flows immediately.

Application Deployment: As most devopsy.kinsta.cloud readers are (likely) well aware, when an Application team needs a connection opened to say, an LDAP database, they open a ticket with a business request: “I need to connect server A to LDAP database B.”

It’s the network guy (or gals) job to translate that request into a set of IP addresses that plot a secure, efficient network path that connects the database to the application. A week or so later, the requisite changes are made, and voilà, the application can leverage the LDAP database. Or not, and then the network team needs to re-trace its steps and figure out what the problem is.

The above process, in many organizations, is a loose one, usually conducted via email, and between people who don’t often traverse the same orbit.

Add human error into the equation – a mistyped or incorrectly documented IP address, and you end up with re-dos, which everyone hates because they are time consuming and in complex network environments can be impossible to trouble shoot manually.

Ask any network manager – reducing (if not eliminating) re-do’s is an impressive feat that offers substantial efficiency gains.   It’s also worth noting that without automation it is really, really hard to properly risk assess proposed changes. In worst-case scenarios, an insecure network connection becomes apparent after something bad has happened. But why wait for something bad to happen before innovating? Bring security into your next network virtualization initiative at its onset – I promise you, you won’t be sorry.

DevOps folks are likely familiar with the idea of abstraction, but it is still a relatively new concept in network security circles.   When applied to network change management, Abstraction separates business logic from the technology underpinnings of network infrastructure, and allows Operations teams to translate higher-level application language into terms they can use to enable underlying network services.   In scenarios like the above, the ability to automate this process can not only deliver huge efficiencies and create a much more collaborative dynamic between application, network and security teams.

Hopefully, these examples build the case for how Security Policy Orchestration can be leveraged to deliver some quick wins. Of course, organizations that commit to deeper and wider change reap greater benefits. However, as my Co-founder, Tufin CEO Ruvi Kitov likes to say, you have to crawl before you walk, and walk before you run. So in my next post, I’m going to explore why security people are so weary of automation, and how to get around that, in hopes that Security, Dev and Ops teams can all walk together.

April 29, 2014
DevOps storytime: MTTR vs. Goodheart’s Law

I am not a fan of metrics. Wait. That’s not right. Let me try again. I am not a fan of singular metrics that stand by themselves. Nope. That’s still not right. I loathe singular metrics that stand by themselves as if they themselves are solutions to problems. Ok. That works. Let me explain.

Last week, I told a tale of a secret passageway into DevOps called Mean Time To Resolution (MTTR) and explained how the introduction of this one tiny metric can magically open the door to all things DevOps. Given that MTTR and cycle time are inextricably linked, any shop that is serious about MTTR will sooner or later find itself using DevOps practices.

Metrics + Crafty Humans = Doom

This is very different than two weeks ago when I told a different tale. A tale of how the very origins and roots of DevOps gives the DevOps movement an opportunity to do what ITIL could not; Change the relationship between business and IT. In the telling of this tale, rife with philosophical terms like reductionism and formalism, I quoted a brilliant English economist, Charles Goodheart, who noted that “when a measure becomes a target, it ceases to be a good measure”. Goodheart discovered that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” The layman’s explanation is that humans are crafty little buggers and will find ways to artificially hit any target because the quantitative systems can always be rigged.

So what is a DevOps voyager to do? Introduce the metric, get the change, and then proceed to watch devops die a slow death? No, my fellow DevOps devotees, that shall not be our fate. We can avoid this fate by knowing that halfway down the path of the secret passageway known as MTTR, there is a fork!

More Metrics to the Rescue!

Instead of introducing and presenting a flat and un-nuanced view of MTTR by itself, we can get a little crafty ourselves and find an appropriate hedge against Goodheart’s lay and instead present a weighted view of MTTR with a few other key metrics to allow for a more sophisticated view of the health of the IT factory floor.

Apply Weighted Averages – Weight the MTTR of defects and their associated resolutions by priority to intentionally skew the measurement towards the highest priority defects. Clinging to an absolute view of defect free software doesn’t work for a large code base with limited funding and time (aside from being scientifically impossible to achieve via godels proof)

Track Inbound Defect Counts – Track and present the inbound defect count, keeping the inbound defect count low is important, but pushing it to zero is more likely to be indicative of a shop that is more concerned with perfection than throughput

Track Outbound Defect Counts – Track and present the defect close count. keeping a negative inbound-outbound differential is a sign of a shop that is arcing in the right direction. Several weeks or months of positive differential is a warning sign that a quality mindset is slipping into dangerous territory.

Track Backlog Defect Counts – Track and present the count of defects in the backlog. A backlog of zero can be incredibly hard to maintain in a large enterprise with a large code base. In my experience 20 and 50 have been ominous numbers. Any more than 20 defects in an application and the backlog starts to become a fearsome beast that can only be tamed by stopping the factory. Any more than 50 and the incremental additions of any new defect seem trivial.

Track The Age Defects – Tracking the average age of defects in the backlog is critical for any presentation of MTTR. A spike in MTTR can actually be a good thing if it represents a wave of old defects being cleared from the back of the backlog.

Resist The Urge to Simplify

A five part metric begs the question of an balanced score card approach with a grading formula and despite my belief in the value of this tactic for availability and performance dashboards, I advise you to strongly resist the temptation to apply it here (otherwise, we are right back where we started with Goodheart’s Law. Damn you crafty humans!).

If you decide to bring the magical metric of MTTR into your shop, this type of reporting may be one area where starting small is quite possibly worse than doing nothing. Without the nuanced view, true visibility into the health of the shop isn’t possible. Worse than that, a flat view of just MTTR has the potential to mask real problems and raise false ones.

When you pull all of the metrics together in a single view, like in the above graphic, the real message becomes clear. The health of the factory floor isn’t tied up in a neat little bow by a single number. It isn’t even made perfectly clear by three, four or five numbers. The real health becomes clear in the trend. Is your shop doing better week over week? Month over month? Getting better every day is the name of the game, and MTTR (with a few other metrics) can be a lever that moves your IT world.

April 28, 2014
DevOps makes coding more social, social coding makes better code

One of the main principles of DevOps is to break down silos. What’s more like a silo than the image of a lone developer, shutting himself off from the world in order to code more and interact with people less? “I’m just here to write code and that’s all I do.” There’s nothing collaborative or DevOps about that scenario and unfortunately, it’s one that is all too frequent within organizations.

Enter social coding. This is a concept that didn’t exist twenty, or even ten, years ago and one that continues to grow in popularity. At its essence, social coding is an approach to software development that puts the focus on formal and informal collaboration. There are many benefits to social coding but the most important is probably the opportunity it presents for mutual growth and learning. Developers are allowed to collaborate in an environment that fosters discussion, creation and sharing.

When talking social coding, you can’t avoid a mention of GitHub. Take everything you love about your favorite social network and then apply those same ideas to a code repository. That’s basically what GitHub did, and continues to do, with much success. You can create a profile, follow or message someone, see what others are doing with their code (or yours) and even go so far as to subscribe to RSS feeds of activity streams. GitHub started to take off a mere five years ago and is now held up as one of the more successful examples of how social coding can be commercialized.

Companies like GitHub and SourceForge are leading the way in revolutionizing how developers commit and share their code but the foundation of social coding still rests with code review in a public forum. As Mark Chu-Carroll puts it…

“If you’re programming and you know that your coworkers are going to look at your code, you program differently. You’ll write code that’s neater, better documented and better organized — because you’ll know that people whose opinions you care about will be looking at your code. Without review, you know that people will look at code eventually. But because it’s not immediate, it doesn’t have that same sense of urgency, and it doesn’t have the same feeling of personal judgement.”

What’s next for social coding? After it’s start in the open source community, this is one idea (like many others) that is heading towards the enterprise market. Companies are beginning to recognize that even internal projects can be managed and treated as open source efforts. From wikis to crowdsourcing, there are many ways that the enterprise market can take advantage of social coding. The end user is the one who stands to benefit the most from such a change.

Dave West, a Principal Analyst at Forrester Research, has this to say about social coding:

“We’re social animals, and the development of stuff tends to be social in nature. We think a better way of thinking about it is as social development. It’s not just about coding; it’s about enabling business change and delivering business value in a social way. It will be interesting to see how companies broaden the idea from social coding to social development. Social coding is a fundamental change.”

April 25, 2014
Gene Kim interviewed by JumpCloud CEO Rajat Bhargava

Two members of the devopsy.kinsta.cloud Board of Advisors, Gene Kim and Rajat Bhargava filmed an interview that is on YouTube and on JumpCloud’s blog.

We are putting it up here and hope you enjoy it.

April 25, 2014
DevOps: Security’s last best hope

Help us Obi Wan! The fact is that DevOps is security’s last best hope. The sooner the security industry realizes it the better it will be for everyone.

I read George Hulme’s story today about proceeding with caution and going slow with DevOps. Of course it was no surprise to me that those cautioning to go slow were my fellow security industry professionals. What else do you expect, after all we are the “people who say no”. At least the three folks George interviewed didn’t flat out say no to DevOps. They just advised caution. The fact is that DevOps is security’s last best hope. The sooner the security industry realizes it, the better it will be for everyone.

Let’s be honest. The security industry for the most part has not been a stunning success in keeping our data, apps and infrastructure safe and secure. Some may even say it has been an abject failure. What do you expect though? The game is stacked against us. Almost by definition the entire security industry is by definition a reactive model. We are always one step behind, reacting to the newest moves of the bad guys. To be a security professional is to know failure.

What is worse, we are generally ignored until we fail and then the harsh, bright light of a security breach shines brightly in our eyes. Is it any wonder that so many security folks feel that they are not successful? Our budgets are never enough, we never have enough resources, no one ever does what we ask and not enough people take security seriously enough until after the doo doo hits the fan. Yes it takes a special kind of masochist to be a security person.

It is not by happenstance that in Gene Kim’s book, The Phoenix Project, the poor bastard of the security guy goes out on a bender and comes back at the end having fallen off the wagon?

For all of the aggravation of fighting an impossible to win fight how are security people viewed by the rest of the IT folks and others in the organization. I am sorry to say that all too often we are though of as the crazy person in the back of the room yelling no. Again as in Gene’s book, it gets so bad that people stop inviting us to their meetings. They know what we are going to say, NO, so what’s the point.

There are more enlightened or maybe smarter security folks who don’t say no any longer. Now they say yes do it, but let’s go slow and make sure we understand it. When asked why, these security folks have a built in scape goat. They blame compliance. Yes, f**king compliance. Yeah, that’s the ticket. We have to go slow because our compliance obligations require it. I laughed out loud when I saw this raggedy old flag being hoisted in George’s article.

Compliance is always the fallback of the security guys who say no. I have seen many examples of this over the years. Two of my favorites are:

1. The security guy who asks the IT manager or C-level guy “how do you look in stripes”, because if you do this without going slow and making sure we are compliant, we could all wind up in the slammer.

Hell we didn’t send anyone to jail for bringing this country to the brink of economic ruin. Do you really think we are going to send someone to jail because by adopting DevOps we might blow a PCI requirement? Can we really do any worse than Target?

2. Walking around a US Army base talking to the security officer I asked him how he was dealing with the compliance issues around securing wireless networks on base. He replied that was easy, he forbid wireless networks on base, so they were in full compliance. Of course as he said this I was watching people unplug their WAPs and throw them under their desks as we walked by.

Yes, security folk can fall back on compliance as the reason for our no, but it is tired and old. The same for the security person who are always promising the next magic bullet. If only they had the latest shiny, new budget busting trinket they could finally sleep well at night. Even if they get their new toy, it only turns up that the bad guys have found a new vector and we need the next silver bullet.

No, I am afraid being the guys who says no or even the people who say OK, but let’s go slow is just not going to cut it. The speed of business today at the enterprise or the startup is anathema to the “go slow” mantra.

On the other hand, DevOps offers a new way. A way to change the way the game is played. For so long I have heard security folk wax poetic about the possibility that security could be built in to the process. Instead of being bolted on after the fact, wouldn’t it be great if it were built in by design? Wow, what a great thing that could be.

DevOps has the potential to deliver on the built in dream. DevOps can make the developer more aware of security and even compliance issues while the code is being written. It can give the security team insight and input into the development planning and process.

DevOps has the potential to break down the silo that separates security from the rest of IT. Just as it brings Dev and Ops together, DevOps can give security a seat at the table. Security can move from being the crazy madman in the back to a respectable member of the team.

This security transformation won’t be easy though. It will demand a change in security’s attitude. First and foremost, we have to change from being the people who say no or go slow, to the people who say “yes we can”. We need to understand that many of these projects and initiatives are going to move forward with or without us. Better to get on the bus than to get run over by the bus.

Next we have to do a better job of communicating our concerns or issues. Falling back on FUD or compliance is not cutting it. We need to communicate the risks involved, how we can mitigate the risk and understand that a business decision will be made as to what level of risk is acceptable and how that risk is going to be managed. At that point we have given enough information for the business people to make a business decision. They make the decision, we get behind it, we don’t bitch bite and we move forward.

Security can’t be the people who say no or go slow. When security starts to say yes we can and this is how we do it, security will be given the seat at the IT table that we deserve.

April 24, 2014
DevOps developers; don’t be a DevGoof

There was a lot of brush back last week over Jeff Knupp’s post about how DevOps is killing the developer. Frankly I wasn’t shocked by Knupp’s opinion that developers “are the dentists in the dental office”. In my 30+ years of involvement in the IT industry my experience is that developers often think of themselves as the only smart person in the room. Often times that is the reason they are are the only person in the room, lol. The fact is many developers relate better to code than to people. For me all this talk about DevOps killing the developer is akin to saying power steering killed the driver. Just because you can do more with new technology, does not mean one dies, one just has to pivot and in this case become business social.

Now the sociable part may be the problem. Sometimes I feel that DevOps is trying to create a love fest between Ops, Developers, QA, security and the rest of the IT stack. While they may not have to love each other, I do think they all need to be nice to each other as opposed to ignoring or just going around each other, as was all too often done in the past. A common scenario I have seen for example is for developers to rollover Ops by stating this is mission critical for the business. These tactics will no longer work to let developers have their way.

Developers need to realize they need to become a normal part of the business social structure. They will be required to talk and work with people, not throw tantrums when they don’t get their way. Left unchecked, certain developer personality types can sabotage your project, or worse, make themselves no fun to work with. I am sure we have all met and unfortunately had to work with people like that. In my career, I’ve encountered virtually every personality type. In fact, I’ve also knowingly hired all of them. They know who they are; they are the developer that if challenged or worse accurately taken to task for lack of being part of the team; quit cause they can’t manage the confrontation, thus interpersonal stress. In the old days they would not come out of their house for weeks continually playing Leisure Suit Larry, if your not familiar with it Google it

I remember one of my good friends telling me this story that sums this whole developer persona issue perfectly. This company, we will call DevGoof, was a well funded start up. Every quarter they held a meeting with all the employees to let them know how the company was doing. Most employees had some kind of equity play in the company, so everyone was incented and theoretically interested in company performance. Several quarterly meetings came and went and not one developer showed up. Eventually DevGoof management got concerned and called a mandatory meeting for all developers’.

DevGoof management put out some pastries, juice and coffee for the brain trust. Then proceeded to tell them why these meetings were important to them and their fellow employees. Notably pointing out how it is not just about them but also about the rest of the DevGoof team. This meeting went on for about one hour and not one developer asked a question. So eventually this meeting was over and everyone went back to developing. The next morning, about 50% of the developers quit and some in a not so professional fashion.

The moral of this blog is the DevOps is going to force the development community to become more interpersonal within the business environment. This will lead to personal success within the new world of DevOps. The DevOps Developer/Engineer that can demonstrate leadership and compassion for team will be the one leading the DevOps way within the organization. Don’t be a Devgoof!

If you would like say something; email parker@devops.com and don’t forget to comment here.

April 23, 2014
What Cisco’s OpFlex means for DevOps

Some big news in networking land at Interop this year revolved around Cisco’s latest contribution to open source and the Internet, its proposed “OpFlex” control plane protocol.

The protocol itself is focused on communicating with network elements and specifies encoding in a variety of formats, including JSON, that are developer-friendly. That’s nice, of course, but the big question as someone interested in operations and devOps is why should you care about what appears to be a networking protocol? After all, beauty is only interface deep. What’s in it for you?

To see where DevOps fits into the picture, consider the description of OpFlex as specified in the IETF draft of the OpFlex protocol:

The OpFlex architecture provides a distributed control system based on a declarative policy information model. The policies are defined at a logically centralized policy repository (PR) and enforced within a set of distributed policy elements (PE). The PR communicates with the subordinate PEs using the OpFlex Control protocol. This protocol allows for bidirectional communication of policy, events, statistics, and faults. This document defines the OpFlex Control Protocol. [emphasis mine]

What’s being offered is not just a protocol for bidirectional communication, but an entire architecture for sharing policies and actionable data. What’s more important to those who for whom “networking” is not their native technical tongue is that policies are designed to be declarative and abstract. That means instead of issuing a set of commands, often specific to the device or service, a policy declares (describes) what’s needed and each component participating in the architecture interprets those needs according to their specific function and capabilities.

You might be thinking at this point (I know I certainly was) “What is this policy and who is defining it?” because that seems pretty central to the value proposition of this whole endeavor.

For that, we need to look to the OpenDaylight (ODL) project which has created a new project called the ODL Group Policy plug-in. As explained by Cisco in an OpFlex white paper, “the goal of this project is to provide a policy-based API that can serve, in practice, as a standard information model in OpFlex implementations.”

Jumping over to the ODL Group Policy Plug In wiki, then, the intention of the Group Policy Plugin is to introduce “the notion of groups of endpoints and policy abstractions that govern communication between these groups. The goal is to support a northbound API that accepts abstract policy based on application requirements while offering numerous concrete southbound implementations for programming network element with configuration/programming stanzas rendered from application policies.”

Okay, so that’s a whole lot of .. words. What it boils down to is that instead of a policy describing a set of rules governing routing with the appropriate ACLs, one might simply declare that “Web Server A can talk to Database Server A to perform database transactions” while “Administrative Console B can talk to Database Server A about administrative tasks.” If you were to translate that into specifics you would likely have multiple imperative command lines, each describing VLAN participation and routing as well as entries allowing traffic between specific VLANs or IP addresses on the ports appropriate to the application layer protocols. It would be long and prone to misconfiguration simply due to the number of commands required. Using a declarative (abstract) policy language simplifies not only the policy but also addresses the disparity inherent in competing products across a given market by eliminating it from being problematic.

The problem with traditional proposals to solve the “different definitions and object models” across a given market space (like ADCs or IPSs or switches) is that they’ve all relied on finding the lowest common model, which inevitably ended up with all offerings needing to dumb down (commoditize) their value in order to ensure consistent support for all “XYZs”. By abstracting to the policy – and not the object model – level, the OpFlex approach effectively eliminates this issue and emphasizes defining what is desired by the application, but not how.

Now, the actual policy declaration is unlikely to take the form of natural language (that’d be nice, but we’re not quite ready for that… yet, though Watson might disagree with me on that) but the general premise is to provide a far more, shall we say, friendly policy language.

What this means is the operations folks most likely to be familiar with the application architecture and who understand what it needs and how it needs to work, could be the ones to actually codify that into policies. You’ll note that the intention for OpenDaylight, at least, is to “support a northbound API that accepts abstract policy based on application requirements.” That means someone – hint, hint, that’s you – might be able to integrate into the application deployment lifecycle provisioning of the network services and topological constraints required based on an understanding of the complex web of connections between each piece of an application. Someone who knows from experience that a “three tier architecture” means more than just three components, and can more effectively describe how they should be interacting.

But that’s just the beginning. You’ll note that in the RFC summary above it specifies “bidirectional communication of policy, events, statistics, and faults.” That means actionable data is being exchanged between participating components. Actionable data enables, well, action. Actions that might address a fault that occurs between end-points. Actions that might address performance issues. Actions that might mitigate attacks in real-time.

Ultimately this level of capability will drive new solutions – new workflows and processes – that take advantage of event-based systems and policy-driven networking. OpFlex should enable statements like “When X happens, do Y” to be executed upon, even if it requires coordination of three or four or more network components.

What OpFlex means for the devops practitioner is a way to influence (or, one hopes, specify directly) what services the network provide for the applications you’re tasked with deploying and delivering, and how the network responds to the things that happen on any network that impact the performance, security or stability of those applications.

As networking continues to recognize and address the need to provide application-driven services, including provisioning and real-time adaptation, it’s going to continue to become more programmable and integrated with the systems driving a more automated, orchestrated data center. And that means devops practitioners are going to have more and more options as to how they manage the entire application (and network) service lifecycle.

April 23, 2014
Advanced Automation – Getting Your Systems to Work for You

In my previous post I discussed how to take your DevOps to the next level by taking it beyond infrastructure automation, to the automation of your deployments and code pushes, through patches and updates.

And then I promised to make it interesting…so here is the next stage – actually using the extracted data to get your systems to work for you.

Monitoring Like IT’S YOUR BUSINESS

Let’s start by discussing what it means to monitor your application, and what kind of data you can extract from it.

At the most basic level, you want to know whether your application is available for your users. It sounds very basic and simple, but setting it up properly is actually not an easy task. It is intended to give you an answer to the most important question to your business: can my users use the application? Think of it as a big red or green light on your dashboard. Although the answer may seem obvious and simple, but it is not at all. There are all kinds of parameters to take into account when answering this question, for example, what if the system is apparently up and running but the response time is very slow, or what if only parts of the system are actually running. At the end of day, you want to have that kind of an alert “traffic light” that tells you whether your system is functioning – yes/no, up/down.

System, Application, and Business Metrics

The green-red traffic light is an aggregation of multiple levels of monitoring. The most basic one is system related. It contains indicators like process availability, CPU and memory utilization, etc.

The next level of monitoring is application related and is composed of application level KPIs (key performance indicators). KPIs can be anything from the average response time for a user for a certain request, to the number of concurrent database connections. In other words, anything that’s specific to the application and its architecture.

And then above these, comes the highest level of metrics, which are business metrics. Ultimately these are what’s really important. If your business metrics are ok, then you can assume everything is running fine. Sample business metrics include the rate of failure to register to your website, how many users did a certain operation in the website or how many users executed a specific transaction. If we want to be really specific, then take Google hangout for example, a business metric of theirs would be how many people in a certain timeframe joined or started a hangout.

Logs to the Rescue

Logs are often used here as means to collect your metrics and alert about erroneous conditions. Logs usually contain a lot of useful information, but you need to be able to extract it and make sense of it. This means gathering all of the logs emitted from all of your servers, parsing them and looking for specific patterns that will help you generate the KPIs over time. For example, you look for a log message that says “User X signed in”, and by counting these messages you can produce a business level metric for how many users register over time. If you add some more data to the log message, such as the user’s location, or operating system, or anything else for that matter, you can slice and dice data and get deeper insights into how your application is behaving. When logs are emitted in a structured and consistent way (e.g. in a JSON format), it becomes very easy to analyze them and produce the relevant KPIs. There are many toolsthat can help with that.

From Simple Automation to Orchestration

That’s where orchestration comes into the picture. At the most basic level, orchestration is a higher form of automation, which helps you setup all the pieces that are related to your application, starting from the infrastructure (VMs, networks, block storage volumes, security groups, etc.), to the platforms your app runs on (database, web server, etc.), and all the way up to the application modules and code. This entire setup is often referred to as a topology. The role of an orchestration framework is to materialize a certain topology. More advanced orchestrators go beyond materializing the topology, and change it to meet the current workloads and needs of the application.

As you probably gathered by now, monitoring and log gathering are an essential part of any running app, so they should be an integral part of the orchestrated application topology. Since the orchestration process is topology aware, it can wire and configure monitoring for your application components very easily, which is one of its greatest benefits. Going through these process without a global view of the topology can often be time consuming and error prone. Moreover, as the topology changes , you need to reconfigure and rewire your monitoring tools, and a good orchestrator will that for you as well.

Next Up: Reactive and Proactive Orchestration, The Devops Holy Grail

Hopefully by now you see the value of orchestration as a higher form of automation. In the next post I’ll dive deeper into the post-setup phase, and discuss how orchestration tools can react to events and monitoring data and adjust the application’s runtime topology to best fit the current workloads.

April 23, 2014