devopsy.kinsta.cloud

Category: Features

Q&A with Gene Kim: Bringing the auditors to DevOps

As more enterprises embrace DevOps, organizational disconnects often are created between what controls DevOps teams have in place and what IT controls auditors believe need to be in place. And if the right controls are, in fact, in place they absolutely need to be communicated to the auditing teams. Bridging these gaps is often painful for both IT teams and the auditors.

Helping enterprises to better understand what auditors need from them to do their work is the top-line goal of the DevOps Audit Defense Toolkit, the aim of which is to “define the authoritative guidance of how management and auditors should conduct audits where DevOps practices are in place,” according to the homepage for the effort. “By doing this, the DevOps Audit Defense Toolkit will elevate the state of the management practice, defining how to understand risks to business objectives, correctly scope and substantiate the of effectiveness of controls, which reduces the costs of audits and increases effectiveness of audits.”

The review draft of the DevOps Audit Defense Toolkit was release earlier this month for review and comment. The authors, James DeLuccia, Jeff Gallimore, Gene Kim, and Byron Miller, expect the final version to be available in coming months.

To get a deeper understanding of the Toolkit, we grabbed a few minutes of time with Gene Kim, author of The Phoenix Project.

Why do you think something like the DevOps Audit Defense Toolkit is necessary?
I think at its heart DevOps is transformative. It’s the way to increase the flow of work through the DevOps value stream. DevOps increases reliability, stability, and security. And it also helps the organization win in the marketplace.

Yet, when organizations actually embark upon this journey, probably the number one obstacle they encounter is that their “compliance guys will never let us do this.”

I think there’s some truth to this. Because of so much that’s being proposed when you embark upon the DevOps journey, you actually take away some of the key controls that traditional IT organizations have used – like separation of duty, like change approval processes.

So when they hear, “Hey, I’m going to be letting our developers deploys whenever they want, without necessarily getting a change approval order from the change advisory board,” auditors, compliance managers, and security people often freak out.

And you have first-hand experience of seeing this in action, right?
It’s why I think it’s a very important effort. The more I talk to people, the more I hear about the friction between the audit teams and audit groups in organizations trying to move this way.

How do you see the DevOps Audit Defense Toolkit achieving this?

The DevOps Audit Defense Toolkit was designed to train. Since we can’t bring DevOps to the auditors, we must bring auditors to DevOps. What I mean is the goal is to train practitioners in DevOps how to think like an auditor and actually walk through what an auditor will do when examining a process and systems. That’s starting from the very highest level of the top-down risk assessment into the details, just as an auditor would go through the list asking what can go wrong.

The DevOps Audit Defense Toolkit shows how an enterprise has the environment that can prevent bad things from happening, and if we can’t prevent it, we can detect and correct for it. Then, it shows how we evidenced it. I think by doing all of that, one is doing exactly what the best auditors who examine DevOps organizations do themselves.

By training people in in the DevOps organization to think like auditors, we can actually bring the auditors along, show our work, connect the dots, and hopefully have the auditor become DevOps’ best friend along the way.

So, this is training IT how to think like an auditor and view the systems and processes the way that auditor would?

In many ways, yes. This is kind of a structured thinking process that any world-class auditor would do to actually create an audit plan and then audit fieldwork.

The toolkit asks questions like: Why is this being audited? Describe the environment Walk through the process of how code moves to production, and where the development environment comes from. It also asks: How do you prevent developers from introducing backdoor code into production; how do we ensure that untended code doesn’t make it into production?

I’ve learned a lot just working with the other team members, in terms of showing how we have a basis to say that we know what we’re doing, and that we’re really thinking through how to have an effective control environment.

That takes it from sort of the high-level platitudes and claims down to well, no, really, this is how we enforce peer reviews of code. We ensure that all deploys are put into the build systems. We run all the automated tests. Every time someone commits code, we run all the security tests before it can ever get into the staging environment. We can confirm that what’s deployed actually passed all the tests.

We communicate most effectively when we write down and show our work. And we’re exposing all that is required for an auditor to be able to look at it and say, “You know what? This is actually better than my best auditors would’ve done in the field, and you know what you’re doing.”

I would imagine that simply learning the language of auditors and how they speak is very valuable itself.

Yes, we’re now using a language that auditors use among themselves. Even just exposing what that language looks like, I think, is helpful to connect our own dots to be able to say, “Oh, I know why it’s being phrased this way. Tons of value, just by providing that.

Did you learn anything along the way as you created the DevOps Audit Defense Toolkit that surprised you?
It’s funny how much I learned by working on this. You would think the person who co-wrote the Phoenix Project would know what this would look like? But I learned quite a bit when I was looking at what were the actual business flows, and where that regulated data could reside, and what specifically are the applications in-scope for the compliance domains.

The second thing was going through the management risk analysis – to walk through the structured thinking process and look at how a company is really doing the diligence that is required, deciding what are its top business risks and if it really has a control environment that can prevent, detect, and correct.

Of course, we all know that DevOps makes everything better, but we need to know why and how to prove it, By what standards are you holding developers accountable? You talk about peer reviewing, but where is that documented? What is the standard? How is it enforced?

You want to be in compliance, and that’s important. But you also don’t want people to insert backdoors. You don’t want systems to crash when changes are deployed. So you really need to name all the dots, connect them, and then have that processed challenged. I have learned so much in this project.

June 30, 2014
DevOps doesn’t end at deployment

If you ask five IT admins the question “What is DevOps?” you maybe get eight different answers. We’ve attempted to define and explain the term DevOps here on devopsy.kinsta.cloud a few different ways, but a precise definition is still elusive. One thing is certain, though, DevOps is about more than just automating virtual server deployment and configuration.

At Velocity 2009, John Allspaw and Paul Hammond presented a session titled “10 Deploys Per Day: Dev & Ops Cooperation at Flickr”, which some credit with launching the concept of integrating development and operations—at least to mainstream IT. For many, though—especially those who are still trying to understand what DevOps is, and what it means for their organization—DevOps amounts to tools that simplify the process of implementing tens or hundreds of virtual server instances, and automating the initial configuration.

Automating IT

Tools like those offered by Chef and Puppet Labs are useful for automating routine tasks associated with deploying and configuring the IT infrastructure, as well as deploying and updating applications. These vendors can also help monitor and maintain a consistent, stable environment by enforcing a designated configuration.

To be honest, that sort of automation is more or less table stakes for DevOps. It’s just one small facet of a much larger concept that involves the entire scope of development and operations—agile development and rapid iteration of new versions and features combined with an automated and easily-scalable infrastructure.

Once the virtual server environment is up and running, though, and once the applications are deployed, the work is really just beginning. Part of merging development and operations, and keeping up with a more rapid pace of IT involves ensuring that everything is running smoothly. With frequent updates and rollouts, it becomes even more important to have tools in place to automate monitoring and alert IT admins when something goes awry.

DevOps also involves tools like those from Stackify—tools that keep an eye on the apps running in your environment and let you know when things don’t quite seem to be firing on all cylinders. When something goes wrong, you want to know about it in a few seconds, not a few hours or days. Stackify can monitor and trend important metrics like CPU, memory, and disk utilization on your servers, and alert via email or SMS text when problems arise.

Automating the creation, deployment, and initial configuration of servers is awesome—but you also need tools to address the challenges of maintaining and updating thousands of server instances as well. You need to be able to add or change user privileges, apply patches and updates to the operating systems or applications, change security configurations, enforce compliance policies, and more. When you’re talking about hundreds, thousands, or tens of thousands of servers, it’s imperative that you have tools like JumpCloud to help automate those tasks across the entire environment.

More than just tools

As great as all of the various DevOps platforms and tools are, though, DevOps is not just a matter of buying or implementing an array of tools to manage your IT environment. As the name implies, and as Allspaw and Hammond spelled out in 2009, it’s about instilling a culture that breaks down traditional barriers and enables developers and operations to work cooperatively and collaboratively.

The traditional roles of developers and ops impose unnecessary friction that gets in the way of being effective. Developers blame problems on the implementation or IT infrastructure. Operations personnel blame problems on poorly developed code. Creating an environment where all appropriate parties work together, and where everyone takes responsibility rather than assigning blame creates an entirely different dynamic for productivity.

When it comes down to it, it’s the culture of DevOps that accelerates the speed of changes and updates that necessitates tools and platforms to automate deployment and infrastructure management. The culture drives the tools, not the other way around.

So, yes, one of the most visible components of DevOps may be tools that automate the deployment and configuration of servers, but that is just one element of DevOps, and it’s just the beginning. Everything that happens after that to monitor, maintain, and update that environment once it’s deployed, and everything that happens outside of the IT infrastructure to streamline productivity and increase efficiency is all part of DevOps as well.

June 23, 2014
National Renewable Energy Laboratory Uses DevOps For Better Compliance

One of the big stumbling blocks naysayers claim DevOps can’t overcome is the perceived incompatibility of DevOps with high levels of compliance regulations. But forward-looking IT practitioners at financial institutions, healthcare organizations and government agencies alike have already started to show how untrue that perception really is.

The National Renewable Energy Laboratory (NREL) recently offered a perfect case study in this phenomenon. Not only do the key IT staffers in the trenches at NREL believe that DevOps is compatible with DevOps, but they actually used DevOps and continuous delivery principles to help the agency deliver compliance with a key piece of regulation around cloud integration within the NREL infrastructure.

The success came from a year-long project designed to garner the lab a needed update to its ‘Authority to Operate’ (ATO) designation under the Federal Risk and Authorization Management Program (FedRAMP) for the new integration of Amazon Web Services across its IT infrastructure. The project successfully doubled as a pilot program for DevOps and continuous monitoring, says Ryan Kelley, systems engineer at NREL. He explained that the agency wanted to take lessons learned from staffers in a very limited separate department who had previously dabbled with AWS and DevOps practices and apply them for a fresh approach in a broader project within NREL.

“The approach we took was, we’re not just going to try to jam all of our legacy toolsets into this and try to get them to work with the cloud,” he says. “We wanted to take a step back and say, ‘What’s the best approach for policy, what’s the best approach for toolsets and things like that for this new architecture?’”

Part of the impetus behind this fresh approach was the lingering memory of how painful it was for NREL to gain its ATO five years ago. At that time the lab used outside consultants to shepherd it through the process and it turned out to be “extremely painful, drawn out and very, very expensive,” Kelley says, explaining that the ATO governs

This time around, NREL hoped to find a way to use the expertise and resources it had in-house to streamline the process and not only comply with regulations but see a true return on its investment through a much improved infrastructure. According to Kelley, the project forced NREL to approach infrastructure almost like it was starting a new business.

“We had a fresh slate and we kind of had to build almost everything policy-wise and everything toolset-wise,” he says, explaining that in order to comply with the security requirements around achieving an ATO, NREL was most concerned about what’s called customer-responsible controls, which are often very configuration-centric. “That’s where having a centralized configuration management system pushes you towards compliance because it’s just – it gives us that central place to audit all of our configurations. Out of the gate, to get cloud underneath our authority to operate, we needed the ability to centrally manage and have our configurations approved.”

By approaching it as almost a new business, the IT team was able to make the cultural shifts necessary to achieve that through the improved teamwork and continuous delivery patterns that are the hallmark of DevOps.

“Enterprise IT tends to think that they have to deliver these big, huge projects all at once and tied up in a nice bow, whereas we’re trying to take the approach of getting a minimally viable product and then rapidly iterating on it,” he says, explaining that it took about a year to get the ATO, but that NREL is still iterating to improve the cloud integration overall.

Kelley says DevOps is providing a much needed shift for the organization due to the highly distributed nature of its programming resources. While the lab has a small cadre of IT staffers to run the backbone of IT infrastructure, the lab itself actually employs many programmers that actually work directly under what are essentially the lines of business for NREL, basically smaller sub-groups and departments working on individualized research projects. This ATO project showed that it is possible for the main IT group to interact with these smaller groups of developers to deliver quickly and effectively on projects and NREL plans to build on this success.

“You know, we’re dealing with the same problems everybody else is: complaints that we’re too slow to deliver, not responsive enough, and not working with the business enough,” he says, explaining that past pundits’ warnings for IT to ‘align itself to the business’ never before offered concrete advice on how to do that. “This was kind of one way that we did was just trying to find a project, develop a partnership and then sort of create a grassroots type of DevOps mentality. It definitely wasn’t top-down. And we’re continuing to work with these groups in this particular manner.”

Meanwhile, Kelley explains that the project also showed how continuous delivery toolsets can be instrumental in helping to with compliance demands.

“We feel that these new modern toolsets that let you treat your infrastructure as code actually gives our cybersecurity team more insight into what we’re doing because once you do that and start treating it like software, then it can be audited a lot easier,” he says. “It’s a lot easier than building each server on its own and then waiting for security to scan it and then waiting for them to get us a report back and then we deploy it.”

June 19, 2014
The web-scale avalanche

Old-school businesses may not like it, but they’re competing with Google, Facebook, Amazon and other Internet powerhouses because those players have grown from day one with an eye on rapid, scalable, cloud-based and continuous IT operations.

Companies have been running application delivery aligned to an aging model based on low frequency release cycles. For decades, the major IT suppliers like Microsoft, Oracle, IBM, SAP and others have released software at the predictable, plodding cadence of annual or bi-annual releases. IT departments have adapted to the vendor release cycles with mirror-like processes: highly structured in the ITIL fashion with processes in place for working around lots of bugs that don’t get readily fixed by the vendors. Yet this delivery management model is now out of pace in today’s always-connected, Internet-based world in which apps and sites morph on a weekly or even daily basis.

Suddenly, companies and vendors alike must change what they’ve been doing for years. They need to transform operations and products from the legacy-based delivery model to one which is faster, dynamic and proactive. But that’s hard for most companies. It’s like driving a 1979 Chevy pickup on the Autobahn. You can’t keep up with the pace of traffic, no matter how hard you try.

Companies such as Facebook, Twitter and Netflix, born as Internet-age companies, don’t have the same problems adapting to modern times in which consumers expect instantaneous response times. They were built from the ground up to handle the relentless pace of technological change: their infrastructure and applications automatically shift according to network and CPU conditions and workload spikes. Such companies are masters at Web-Scale IT, and rely upon a multitude of open source and web-based technologies designed for distributed computing. More than just technology it also takes radically new processes and mindsets to become a Web-Scale IT company.

Operating in the Web-Scale World

Web-Scale IT consists of technologies and methodologies that enable developers to build, change and scale applications 10 to 100 times faster than in the past. Web-Scale IT relies upon many modern concepts, including cloud computing, peer-to-peer architecture, NoSQL, SOA, Agile development, DevOps, Continuous Delivery and Deployment and more. Gartner describes Web-Scale IT as “a pattern of global-class computing that delivers the capabilities of large cloud service providers within an enterprise IT setting.” The research firm predicts that by 2017, Web Scale IT will be an architectural approach found in 50 percent of global enterprises, up from less than 10 percent in 2013. Below are a few areas of change that IT organizations must navigate to make the shift.

1. Reevaluate IT partners. It’s not just CIOs that need to shift operations for Web-scale computing. The legacy IT vendors are struggling to walk away from profitable business models based on licensing to embrace the on-demand, services-based revenue that just a few years ago, catapulted startups like Salesforce.com into billion-dollar enterprises. The legacy vendors are unable to innovate fast enough and are having to rely on serial acquisitions to keep their technology relevant. Yet rapid acquisition comes at a great price and is not sustainable. IT departments will need to look far and wide to find the best partners that can support their Web-Scale IT efforts moving forward.

2. Acquire diverse, open-source based IT operations tools: CIOs will likely need to constantly review their technology sourcing and have a more critical method of choosing their technology partners. Relying on a few trusted long-term partners will likely not achieve the Web-scale IT goal. Consider that Netflix, which now runs its business almost exclusively on Amazon AWS, only a few years ago relied on its own datacenters full of legacy vendor equipment. Today, the company manages its high-availability video streaming business using a collection of inhouse-developed open source tools (coined The Simian Army) that test the fault tolerance of the company’s operations. Being flexible and open to change allows Web Scale companies to keep ahead of the competition. Standing still, even for a short while, is no longer an option.

3. Foster iterative, collaborative processes: Strong DevOps cultures, where developers and IT operations personnel exchange information frequently to stay in step with changing business requirements, is also pivotal to making this shift. Companies with advanced DevOps practices are usually committed to running their businesses differently: more efficiently, more customer-focused and with the keen ability to respond quickly to change. CIOs and CTOs need people with strong Web technology skills yet also an unflinching commitment to collaboration and complex problem-solving. IT people today need to think on their feet and adapt as they go. Twelve-month planning cycles are out the window.

The building of a storm

Just a few years ago, it was considered evolutionary, if not revolutionary, to purchase CRM or HR as a service. That’s commonplace today. The next area for innovation is within core applications such as a retailer’s pricing application or a bank’s market forecasting tool. Those applications must now also run at Web Scale to be responsive, customer focused and speedy. Don’t count on your Oracle or SAP database to shift so easily or quickly—at least not yet. That’s why Internet leaders are managing IT infrastructure using the newer generation of Web Scale technologies including Puppet, Chef, AppDynamics, New Relic and yours truly, Boundary.

If running at Web Scale sounds overly complex, it’s not. Since most of the latest, open source-based operational tools are highly automated, using them is actually simpler. Crowdsourcing is another strategy that companies are exploring for Agile development projects. The bigger hurdle is in moving from the old-fashioned software development style of over-engineering everything to the more intuitive, iterative style that characterizes modern Web application development. With Web-Scale IT, it’s okay to cut corners and take more risks, because the infrastructure will stand behind you.

There are plenty of people who’ll disagree with me, but I can envision a world in which the Oracles and IBMs no longer run most of the companies in the world. — one that is adaptive, business-responsive and always forward-looking. The future is closer than we think. Will your business evolve?

June 17, 2014
Five Top Tips for Cloud and DevOps Automation

A while back, I overheard someone asking why many of the ‘cloudy people’ have moved on to DevOps.

One reason is certainly that cloud is at the heart of DevOps. Many of the grounding principles of DevOps – rapid iteration, agile development, automated testing, continuous delivery and continuous integration – are barely even possible without cloud. However, I think it runs even deeper than that. DevOps and Cloud are both fundamentally enabled by the same technology I have been working with my entire career – automation.

When IT leaders get excited about cloud, it is mainly because of automation. Sure, cloud providers get excited about essential characteristics like broad network access and resource pooling, because that is how they maximize reach and minimize costs. Similarly, CFOs get excited about the measured service, because that means they only pay for what they use.

However, according to a recent Luth Research study, most IT leaders prioritize the higher-order values enabled by on-demand self-service and rapid elasticity – faster innovation, superior performance, better scalability and higher resiliency. These capabilities are deeply rooted in automation – of provisioning, deployment, configuration, scale-up/scale-out and other processes.

Similarly, DevOps – beyond a certain point – also builds on a foundation of automation. As Rackspace CTO John Engates noted in his recent column in this journal, “DevOps is more than a set of automation tools.” He is right, of course. You can achieve a great deal through cultural and structural changes – better collaboration, shared goals, team building, process improvements, etc.

However, tools for release automation, configuration automation, process automation, test automation, service virtualization and provisioning – just to name a few – are adding tremendous value with a rapidly maturing DevOps automation toolchain. Indeed, in recent research from Vanson-Bourne, over half of senior IT decision-makers worldwide rate automation as the most important DevOps component – above such putative stalwarts as agile development, collaborative teaming and process alignment.

Looking at the history of enterprise automation, and considering my own experience, I believe there are many common principles that you can apply to both cloud and DevOps. Here are just five of my top tips:

Don’t Automate Bad Processes

Automating bad process just makes bad things happen faster. This seems obvious, but I often see automation that only encodes (and accelerates) a bad existing practice. To avoid a high-speed train wreck, first identify, catalog, document, review and refine key manual processes to make sure you are automating the right activities, at the right time, in the right way. Solve for simplicity, speed and throughput, even before you apply automation. Remove friction across silo boundaries – departmental, technological and disciplinary. Start simple, be inquisitive, and iterate methodically. Then make sure you automate deeply, thoroughly and efficiently.

Cross-skill Existing Resources

Great people are hard to find and salaries for cloud- and devops-relevant skills are amongst the highest in IT. This explains why, according to the same Vanson Bourne research, over 70 percent of DevOps leaders are retraining existing personnel, while only 50 percent are hiring new resources. Look for staff with a history of automation in their resume, through multiple waves of technical change, rather than specialists in one automation technology or product. Find the IT equivalent of a “good athlete” rather than a “position player”.

Security Cannot Be an Afterthought

Automation systems and the staff responsible for them have always needed highly privileged (and frequently shared) user IDs, whether automating a mainframe IPL or scripting a production code release. This is an audit red flag, and can result in high-impact incidents, whether accidental or malicious. Identity and access management provides essential audit and control, while giving users (including automation tools and APIs) easy access to the resources they need (locally and in the cloud), when they need them.

Document, Measure, and Backup

If you do it right, automation quickly becomes pervasive and mission-critical. Without documentation, you create inscrutable systems, tribal knowledge, the hero and other ‘anti-patterns‘ in your Cloud or DevOps machine. Without measurement, you cannot know if (or when) complex autonomous systems run amok or stop working. Without backups, critical systems are prone to irrecoverable failure. Cloud and DevOps teams must value and prove traditional disciplines like documentation, measurement and backup – painful as they may be.

Build Awareness into Automation

As automation becomes mission-critical, it must be resilient to errors, failures, exceptions and other unexpected conditions, and still deliver a correct result. Remember when ‘autonomic computing’ was a thing, when we dreamt of ‘self-aware’ systems? Similarly, concepts like ‘desired state’ and ‘idempotence’ go back at least as far as mainframe and server automation. Similar concepts apply to cloud computing and DevOps too. In both, it is critical that automation routines are aware – of their own state and of the surrounding environment – so that they can test for and accommodate unexpected conditions, and still deliver the expected results.

Given the depth of history in automation, and the many parallels with modern computing, some may claim that cloud is nothing new, that we have been doing that since the mainframe; while others will say something similar about DevOps. I think they are wrong, but there are definitely parallels, and George Santayana has a point – we should be at least informed by history, if we are going to try to write the future, while avoiding the mistakes of the past.

June 16, 2014
The US Air Force was doing ‘DevOps’ before it was cool

One of the pillars of DevOps is the obliteration of traditional silos, and the ability for cross-functional teams to work together to just get things done. While the concept may be new for developers and businesses, the United States Air Force already demonstrated the power of this philosophy during Operation Desert Storm more than 20 years ago.

“DevOps” – Air Force Style

The US Air Force is configured primarily in separate units, or silos. Generally, a base or wing is comprised of a single type of aircraft. C-130 transports, KC-135 refueling planes, F-15 fighter jets, A-10 ground engagement aircraft, and F-111 bombers each had their own base of operations, and separate command and control infrastructures. During Desert Storm, though, a new Composite Wing was formed, which included a cross-section of all of these aircraft and Air Force resources tasked to work closely together with one chain of command.

Air Force Brigadier General Lee Downer wrote about the formation of, and experience with the Composite Wing in an Airpower Journal article in late 1991. In essence, by removing layers of red tape and bureaucracy, and putting all of the required assets and resources under one command, the 7440th Composite Wing was able to operate with unprecedented efficiency and effectiveness.

Downer concluded with these thoughts: “The 7440th Composite Wing’s achievements are now a part of history. The efforts to get the wing into the war are a special success story that should be told in more detail in the future. As is usually the case, once the decision to go was made, the success of the effort was up to the people who were tasked to make it work. Their accomplishments clearly show that they assumed the challenge with a great deal of enthusiasm and skill. They knew their jobs, did not hesitate to make decisions, and kept improving the operation as they became seasoned and experienced.”

I can speak personally about the success of the 7440th Composite Wing as well. I was serving in the Air Force at the time, and I was deployed to Incirlik AB as a part of the Composite Wing effort for both Operation Desert Shield and Operation Desert Storm. While my role was traditionally focused on, and restricted to working with F-111 bombers, the exact same technology was also used with other aircraft from other bases.

During Operation Desert Shield, while working as a part of the 7440th Composite Wing, those barriers were removed and I was given the freedom to do what I did best without regard for the aircraft or what base it originated from. It was a much more logical and effective use of resources—in this case the resource being me.

Fast Forward 20 Years

The same concepts and philosophy the US military embraced during Desert Shield form the foundation of the DevOps movement today. The goal is to minimize friction, and empower people to do what they do best with minimal obstacles.

By removing the barriers between different teams and departments, and allowing people to work together in ways that make sense, organizations are able to get more done, in less time, and get it done more efficiently.

You don’t have to create a formal merger of assets and resources like the US Air Force did with the Composite Wing. While it did facilitate better integration, and greater effectiveness, defining the resources of the merged team is its own silo in a way, and still puts limitations on what the new team can accomplish.

You should, however, embrace the essence of the concept—which is to remove obstacles and allow individuals and teams the freedom to share resources, and collaborate together to achieve goals without unnecessary red tape or bureaucracy.

June 12, 2014
Log Data as a Valuable Tool in the DevOps Lifecycle (and Beyond)
As many of us embrace DevOps, developers have to think beyond delivering code that is taken by “operations” to be deployed, run and monitored in production environments. DevOps instead focuses on breaking down silos within organizations and sharing the responsibility of development and operations such that developers are now more conscious of how their code will run in a production environment. They also play a critical role in the process of deploying, monitoring and operating the applications they write.

Thus the responsibilities of developers can now span from development and testing, to deployment and monitoring, to gathering business metrics from production environments – i.e. they now follow a DevOps lifecycle rather than a development lifecycle. The focus of these responsibilities can change slightly depending on which stage an application is at in its maturity model. In this post, we first outline three high-level stages of an application’s maturity and then look at how logs can serve as a valuable tool at each of the different stages and in particular for teams following a DevOps methodology.

The Stages of Application Maturity:
1. Stage 1 – Initial system development and prototyping: When organizations embark on new projects, they usually have a small team iteratively develop an initial prototype before throwing the weight of a large development team behind it..This prototype is used to test any hypothesis and initial functionality; it limits risk and allows for an iterative feedback loop from early users. More and more often, prototypes are being quickly developed on cloud infrastructures where the initial development team plays the role of developing the system, selecting and managing the infrastructure, deploying the application, as well as putting in place initial monitoring for the running system. Only after an initial user group has used the system, provided feedback and helped prove a hypothesis will the system will move into a “go-live” state in the real world.
1. Stage 2 – System going live: When an application moves from being a prototype to “going live,” an organization has made the decision that the project is worth pursuing and there is potential value in extending the system to a wider audience. This can involve investing in a larger development team, a larger infrastructure to handle real-user load, as well as marketing budgets to get the word out on this new offering. At this stage, focus can shift from initial concepts to building out more advanced and robust capabilities – ensuring service uptime, performance and reliability.
2. Stage 3 – System generating value: If a system moves into a stage of creating value for the business, a number of new stakeholders can arrive on the scene with new requirements. Typically product managers and marketing teams will have requirements on how a new application’s features are being used, how an application’s marketing campaigns are tracking and how cohorts of users are flowing through the sales funnel. These new stakeholders are largely concerned with different business metrics and how they can be obtained from the running system.
So where and why log data?

While log data has traditionally been used by developers for debugging purposes (i.e. investigating exceptions stack traces), more recently log data is being applied for a much wider set of use cases. A recent survey carried out across a sample of 25,000 log management users shows a breakdown of the following use cases:

Results from a recent survey from a sample set across 25k users who were asked: “What are the top use cases for your logs?”

As applications move across the different stages of maturity above, logs act as a valuable tool to assist with the insights required by the different stages of the DevOps lifecycle – e.g., during development activities, testing, production monitoring, and web and business analytics. In fact, logs provide a single data stream that can act as a common language across these different tasks.

Because of this shift, it is important to look at how logs can be applied to assist with DevOps-related responsibilities during the different stages of application maturity:

How logs are applied during application maturity stage 1

The ability to apply agile development and testing are key during this phase such that early prototypes are iterated upon and hypothesis proved out:

Logs as a debugging tool: Logs have always been a valuable debugging tool and continue to be so. They can capture events at different severity levels from every layer of the software stack (operating systems, web servers, databases, your software) and from across all devices (mobile devices, servers, routers, firewalls). During development, verbose logging can be enabled to give incredibly detailed information on the system as it executes. Today’s log management technologies are particularly useful as they allow you to utilize the benefits of verbose logging to easily cut through the noise and identify key events when exceptions occur. They also all allow for fast searching of key events in an extremely large volume of log data.

Logs for system load and performance testing: Before a system goes live, system load and performance testing is often performed to give a level of confidence that real-world loads can be handled by the application. Load testing tools will often measure system response times to make sure that the system is performing well even under heavy loads. However, log data should also be analyzed during such tests to make sure high levels of exceptions are not being thrown and that the system is behaving “under the hood” as expected.

How logs are applied during application maturity stage 2

Production monitoring and troubleshooting can be a key focus during this stage of a system’s maturity given that any downtime can be particularly costly from a monetary and reputational perspective once a system goes live.

Log for production monitoring: Production monitoring is fast becoming a leading use case for log data. Real-time processing capabilities of log management solutions now mean that log events can be parsed in real time, and important metrics can be extracted and visualized in high-level monitoring dashboards. Furthermore, real-time alerting can be applied to give notifications when important system events occur or particular thresholds are breached.

Logs for production troubleshooting: Logs no longer simply contain exceptions and stack traces but are instead a blend of information from a range of different sources that can be correlated to allow for quick troubleshooting. This is especially important when there are issues in production. For example, logs will regularly now contain: server resource usage information, application performance metrics, feature usage information, exceptions and errors. This blend of information, which was traditionally only available by utilizing a number of other tools, is now available in a single location via a standard log management solution.

How logs are applied during application maturity stage 3

When an application begins to create value, product and marketing stakeholders will want to focus on usage and business metrics. While this information may go beyond the typical DevOps lifecycle, it is becoming more and more common for developers following a DevOps methodology to be tasked with gathering such information.

Logs for web analytics: Analyzing website trends is not something typically associated with someone in a DevOps role and is usually the mainstay of marketing folks. However, developers are often engaged to collect this data for different parts of the organization. Logs provide a lightweight way to understand who is accessing an application, how often and from where. For starters, a lot of this information is already contained in web server logs without any additional instrumentation required. Furthermore, logs can give not only an aggregate view of web trends, but also allow the drill down to a per-user view to analyze behaviors at an individual user or account level.

Logs for business metrics: Today’s log management technologies allow you to extract field-level values and can aggregate them and roll them up into high-level metrics dashboards. Thus,,logs can be used to not only understand a systems performance and usage trends, but also to aggregate business metrics such as signups per day, number of transactions, the value of those transactions, etc. However, the real benefit of using logs for metrics dashboards, in my opinion, is that they maintain the evidence. In other words, they allow you to easily validate any trends or changes in your system or business’ behavior. This has traditionally been a painful task, as it often involves engineering cycles/checking database tables, etc. Now, logs allow you to immediately drill down from a high-level view to the log-level to quickly view and understand the individual events that may have led to a spike in a key performance indicator.

After spending many years talking to thousands of log management users, I am continuing to see logs become more and more critical to the different stages of an application’s maturity. The DevOps movement has greatly assisted with this because the same team or individual responsible for initial prototype development may also be responsible for production monitoring or gathering business metrics for that matter. The value of logs is now being recognized early in the DevOps lifecycle, and the power of log data is quickly expanding across a range of different functions.

This article is by Trevor Parsons, PhD, Co-founder and Chief Scientist, Logentries
June 11, 2014
The 7 skills Ops pros need to succeed with DevOps

There’s a lot of confusion about what DevOps exactly is, and what it takes for enterprises and individuals alike to succeed. Some argue it’s about successfully using collaboration toolsets. It’s not. DevOps is about collaboration between software developers and IT operations teams. DevOps grew from the need for a more agile organization, and to knock down the barriers between development and operations so that applications and IT resources could be delivered more quickly, and improved through steady iteration.

What does all of this mean, specifically, to operations professionals – and the skills they need sharp to thrive? The walls coming down are a bit of a culture clash for most large enterprise operations and development teams. And both developers and operations professionals need to ensure that they have the right skills in place to succeed in this new environment. In this post, we’re focusing on those skills ops members need in the years ahead.

The skills that are important may surprise some. Consider the findings from CA Technologies’ recently commissioned a study, What Smart Businesses Know About DevOps. “The most in-demand skill requirements for success with DevOps were actually soft skills,” says Andi Mann vice president of strategic solutions at CA Technologies. “I expected to see skills like programming or scripting; experience with specific tools for provisioning, configuration management, release automation, or shared monitoring; or even an understanding of Agile or Lean practices. In fact, those skills were much lower,” Mann says.

More on the results from the CA study below, but first we want to share the results of our informal anecdotal survey of a handful of DevOps experts and practitioners, well known here at devopsy.kinsta.cloud. Here they are:

First Up: The Soft Skills
Interestingly, it is these soft skills that are both needed and often derided. “Everyone doing DevOps, whether Dev, Ops, security, QA, project management all need [soft skills],” says David Mortman, chief security architect and distinguished engineer at Dell, Enstratius.

1. The ability to communicate clearly and concisely
“Sharing is a key tenet of DevOps and if you can’t share what you know and what you’ve learned, then having that knowledge is effectively useless. You’ve just made yourself a bottleneck,” Mortman says. The other skill Mortman cited along these lines is striving to actually understand the shoes in which your fellow enterprise collaborators stand. “I can’t emphasize enough the value of having empathy for everyone else and what they need to get done. Every role has its challenges and empathy for those challenges goes a long way to a more congenial workplace,” he says.

2. Collaboration techniques
New collaboration techniques are essential, explains Donnie Berkholz
IT industry analyst at RedMonk, and council member and Sr. developer at Gentoo Linux. “Ops will need to learn effective, distributed collaboration techniques such as those used in open-source software development. Leading-edge DevOps organizations make heavy use of online chat and shared tooling to log both communication as well as action. In some pioneers, this even involves so-called “ChatOps” where you can make changes to infrastructure through calls to an automated bot in an online chat. Typically this includes asynchronous communication in preference to phone calls, both long-form (e.g. email) and short-form (e.g. IRC), in an archived and searchable form,” he says.

This enables, Berkholz says, on-call ops during any crises or fires that arise, as well as eases the ability to identify and brainstorm infrastructure changes, where everyone can easily contribute their ideas.

3. A flexible mindset
We’re not necessarily talking about advanced forms of mental Yoga, but certainly the ability to change as knowledge and conditions change, whether those changes are technical or business driven. “The willingness to make changes and to evaluate alternate options is key to DevOps. Things are going to fail, you or a decision you’ve made are going to be the cause of that failure. Or things just won’t work as well as you’d hoped or thought. Be willing to change your mind and look for better options,” says Mortman.

4. Good enough is never enough
It’s about consistently seeking continuous improvement. “This mindset is laser focused on continuous improvement daily,” says BEHR. “It is the ability to unlearn and acquire new more relevant knowledge in light of complexity and emergent information – then disconfirm it with the scientific method and move on to application only where relevant,” says Kevin Behr chief science officer at CXO advisory and board advisory practice Assemblage Pointe, Inc.

Next up: The Technical Skills

Understanding of modern software engineering
5. “To move to a world of “infrastructure as code,” where all systems are maintained using configuration-management tools such as Puppet or Chef, including version control, unit and integration testing, agile software development, as well as continuous delivery and deployment models,” says Berkholz. “Traditional sysadmins may still live in a world of login prompts and manual edits of configuration files, so this will be perhaps the largest hurdle in terms of knowledge,” Berkholz estimates.

“The technical mindset I look for are skills which allow a team to treat servers more like cattle and less like cats,” says Andrew Storms Sr. director DevOps at CloudPassage. “To work effectively at that kind of systems mentality, DevOps engineers need programming and systems architectural skills. Whether that means Ruby or Python or whatever is needed to effectively work with orchestration tools, APIs, and automation. It’s less about needing to know a specific programing language more about understanding programing algorithms and robust systems design,” Storms says.

6. Modern software architecture
This includes cloud deployment and administration as well as distributed systems. “As we move to a cloudy world of software, ops will need to deploy, maintain, monitor, and troubleshoot production problems with this infrastructure. And yet in many cases it’s much more complex than the previous equivalent, although it brings with it significant benefits in agility and uptime if done right. Ops will need to familiarize themselves with cloud instances and services, as well as lower-level software infrastructure such as Linux distributions and databases when deployed in a cloud,” says Berkholz.

That also includes taking a step back from the trees to consistently stay aware of the forest. “Seeing the system and its boundaries rather than just the tasks and their due dates. This understanding of the interactions between the various actors in a system and [understanding] just how those interactions are making value occur is fundamental,” he says. “Without this understanding we will not know when to push harder versus be silent,” he says.

7. Understanding that security incident response are continuous efforts
Security and safety in the new DevOps environment needs to embrace new ways of addressing security and QA. “This requires continually emergent solutions rather than using ‘tried and true best practices.’ The effective team will evolve through relentless self evaluation, study, and application of more relevant and effective methods to work in complex infrastructure environments,” says Behr.

That’s all great advice, and certainly not all inclusive of the ideal skillsets. We also found it interesting that CA Technologies’ survey of 1,300 senior IT decision makers echoed the need for broader business and so-called soft communication skills when it came to DevOps success.

These included: knowledge of business priorities/strategies/metrics (47%), knowledge of current business processes (42%), communication skills (36%), and project management and people skills both scoring 30%. In fact, experience with specific tools was dead last in the survey at 12%.

June 9, 2014
Citrix puts DevOps front and center

I had the privilege of attending Citrix Synergy 2014 in Anaheim at the beginning of May. Citrix unveiled a variety of new and updated products, but one announcement in particular surprised me a little. Citrix talked about new capabilities for automation with a very DevOps-ish approach.

During a keynote presentation at Citrix Synergy 2014, Sudhakar Ramakrishna, senior vice president and general manager for the enterprise and service provider division at Citrix talked about automatic provisioning through Xen Desktop, and creating blueprints for deployment. He didn’t overtly call it DevOps during the keynote, but it all seemed very similar to what other DevOps tools and services accomplish.

I followed up with Citrix, and had a conversation with Steve Shah. Shah told me that DevOps is absolutely front and center in the thought process for Citrix when it comes to developing products and services for its customers, as well as for mapping out its future strategies.

Shah was at one point a bit of a DevOps naysayer. In an article in 2011, Shah questioned the credibility of DevOps and wondered whether it would survive as an actual thing, or if it was really just a flash-in-the-pan buzzword. Three years later and still gaining momentum, I think we can safely say that Shah’s doom and gloom assessment was off base.

Citrix understands the need and the value for automating routine aspects of provisioning and managing virtual systems. According to Shah, though, one of the things Citrix has noted about the broader landscape is that there is no clear platform for doing DevOps. There are a wide variety of tools and services with a few leaders, but no clear “winner”.

Shah told me that Citrix is only one part of a much larger solution, and it respects that it has to play nice with a wide variety of other products and services, or customers will find other solutions. There are too many unique, custom applications out there for Citrix to individually address them all, so Citrix has instead focused on creating open APIs that enable vendors or customers to develop their own methods of integrating with Citrix.

Another area where Citrix has embraced DevOps—whether it is openly called that or not—is through the use of Blueprints. Citrix determined that there are certain deployment scenarios that are very common for customers across all industries. Rather than having each one of them struggle to develop their own solutions, Citrix designed Blueprints of common scenarios that customers can use as a foundation for their own environments—similar in many respects to the Recipes and Cookbooks used with Chef, a popular DevOps platform.

My revelations from the Citrix Synergy keynote, and my subsequent conversation with Citrix made two things clear. First, many organizations may be embracing elements of DevOps without even realizing or calling it that. There are things that just make sense when managing a large, rapidly evolving environment that naturally lead to implementing aspects of DevOps. Flipping that point upside down, those same needs are more or less the origin of the DevOps movement.

Second, DevOps is gaining momentum and becoming more mainstream as time goes on. While many still struggle to define DevOps, it shows up in day-to-day operations, and in the platforms and tools organizations use whether they understand it or not, and whether they call it DevOps or not.

June 5, 2014
Two Ways DevOps Unlocks the True Potential of Agile

In a recent post on InformationWeek, Erik Weber, a professional Scrum trainer and Agile expert, explores four signs that you’re doing Agile development wrong. Upon finishing the article, I was reminded that Agile in theory is often very different from Agile in practice, and when that gap exists within IT, it can have massive consequences on both development operations and the business.

At the risk of stealing some of Mr. Weber’s thunder, I’m going to spoil the first two signs he discusses – namely, “failure to ‘get done’” and “failure to launch” – because I believe they are common challenges for many IT groups. The good news is, organizations can close their gaps and realize the true value of Agile development by incorporating DevOps processes and tools into their systems development life cycle (SDLC).

Getting to “Done” Faster

“If it takes you longer than a few weeks to turn an idea into a production-ready feature, you are not Agile – full stop,” says Weber, and I would agree. The whole point of Agile was to move away from the slow, monolithic development methodologies of the past by breaking the process into smaller chunks and having developers, testers and business development people working concurrently to speed new features to market. Most Agile practitioners would say they’re doing just that, but many are still not “getting to done” as fast as they would like. Why is that?

According to Weber, the biggest problem is that many IT groups leverage Agile techniques for development, but then do all of the testing at the end. Obviously, this creates numerous problems that can delay project completion. As we all know, the later bugs and glitches appear in the SDLC, the longer it takes to fix them. When that happens, costs go up, delivery pushes out and customers are unhappy.

The DevOps principle to employ here is getting the operations team involved in testing earlier in the SDLC – essentially shifting their work left – and the tool that can help is service virtualization. By simulating constrained or unavailable systems across the SDLC, service virtualization enables developers, testers and performance teams to work in parallel for faster delivery and higher application quality and reliability. With DevOps, they can realize truly parallel Agile development without constraints – and ultimately get to “done” faster.

From “Failure to Launch” to Continuous Delivery

“If you haven’t actually released your software to production – and on to customers – for three or more months, you are missing the big picture,” argues Weber. He believes that many product managers and executives hold back official product releases because they “must have all the features,” and when this happens, they forfeit the ability to both get new features into customers’ hands on a regular basis and react more nimbly to their wants and needs.

There are numerous reasons why the “all features” mindset persists, but a good way to prove its obsolescence is by incorporating the DevOps principle of continuous delivery into the development process. Continuous delivery means always being in a ready state for pushing applications and services through the SDLC and into customers’ hands – and release automation is the tool that helps organizations reach that state.

Release automation enables continuous delivery by automating complex, multi-tier release deployments through orchestration and promotion of applications from development through production. As a result, visibility improves across an organization’s deployment chain, giving it the flexibility and agility to continually release new features to market and react quickly to changing market demands. Once an IT organization reaches a state of continuous delivery, it becomes easier for executives and product managers to pull the trigger on a given release – because they know that the next one isn’t far behind.

Agile is as Agile Does

Weber offers four signs of ineffective Agile development in his article, but I would argue that there is a unified idea running through them all: It’s not enough to employ Agile practices; you have to follow through until you meet your Agile goals. Ask yourself, “Are we really delivering releases to market faster, at a higher quality and at lower cost to the business?” If the answer is “no,” the right DevOps methodologies and tools might be just what you need to put these failures behind you and finally realize the true value of Agile development.

June 4, 2014