devopsy.kinsta.cloud

Category: Features

Eight critical DevOps practices: innovate, deliver, repeat
DevOps has been garnering a lot of interest for some time. Discussions and debates on forums like devopsy.kinsta.cloud are improving our collective understanding of the right approach and now we are seeing concrete proof points of fast-movers applying DevOps practices to disrupt entire industries with rapid-fire, digital-innovation based on Cloud, Big Data and Mobile technologies. I have personally been part of this transformation at IBM for several years and would like to reflect on what we have learned, and also share with you results of a recent study from Forrester Research.

DevOps helps organizations address a critical business challenge – that of continuously capturing and responding to feedback (both internal and external) so we can rapidly and repeatedly turn innovative ideas into highly relevant, and desirable, products and services. This involves not only development and operations, but all aspects of the organization, as it entails the collaboration across the entire organization including business and supply chain. Compared to single-direction, traditional models, DevOps is bi-directional and is based on not just pushing something out to the customer but also on getting their feedback to make improvements.

In a recent webcast, I discussed eight DevOps practices that speed up software delivery with Kurt Bittner, Principal Analyst at Forrester and a leading expert on agile and iterative software development. This webcast highlighted a recent Forrester survey on DevOps practices, cycle times and where enterprises are in terms of the maturity of DevOps practices adoption.
You can watch the webcast replay here to know more – ibm.biz/Forrester-talk

Their survey of 600 IT professionals with application development responsibilities found that about a third of teams consistently deliver at cadences of one to three weeks and that the fastest teams generated higher business satisfaction than slower teams, an indication they don’t sacrifice quality to deliver fast. The study reveals how these teams did it: by adopting the following eight DevOps practices:
1. Minimum Viable Product – Deliver the smallest viable product or application and then improve it in subsequent rapid releases, based on feedback, and delivering value incrementally.
2. Dedicated Teams – Form dedicated, cross-functional teams, avoid over-specialization and ensure they are not interrupted by side projects.
3. Loosely Coupled Architecture – Use loose architectural coupling within and between applications to reduce complexity and enable delivery in small increments.
4. Minimize Hand-offs, Maximize Flow – Eliminate unnecessary steps, delays and friction between steps to increase the flow of work.
5. Deliver in Small Batches – Deliver in smaller batches helps to expose the most uncertain issues first and enables feedback on the most valuable usage models earlier.
6. Transparency – Transparency into progress helps everybody to see where you are at any point in time and also visibility into risk without decreasing productivity.
7. Eliminate Overhead –Eliminate the overhead of periodic and manual status reports and status meetings which absorbs valuable time and distracts focus from delivering value.
8. Automate Testing using APIs – Shift away from heavy manual testing to improve delivery speed, quality, and testing accuracy, and dramatically reduce cost.
The study refutes the myth that releasing fast requires you to sacrifice quality and accept higher risk. Adopting these eight DevOps practices, especially automated testing and delivering in small batches, does exactly the opposite: small incremental releases are more thoroughly tested, more stable and lower risk.

The best part about DevOps adoption is the flexibility to choose which practices to adopt first, based on each organization’s priorities, and then to incrementally adopt the rest of the practices as needed to optimize the software delivery lifecycle and accelerate innovation.

To learn more about the survey and the eight DevOps practices, download the Forrester study along with the approach that you can follow to adopt these as per your organization’s current state of delivery cycles.

About the Author

Ashok Reddy is responsible for Product Management & Strategy, product Design/user experience and Business Development for Rational Software Division of IBM Software Group and for the IBM DevOps portfolio. He is based in Raleigh, NC, USA. Ashok’s role involves all aspects of global product delivery including setting product strategy and for product line management, establishing product roadmaps and partnerships for IBM DevOps portfolio.
October 22, 2014
4 themes to expect from DevOps Enterprise Conference

The next few weeks are busy ones for the DevOps conference circuit. In just a week, devopsy.kinsta.cloud will kick off another one of its intimate Camp DevOps gatherings in Houston, with talks lined up on everything from the evolution of DevOps to mythbusting DevOps fallacies. Meanwhile, another worthy show kicks off from San Francisco today: DevOps Enterprise. A product of collaboration between IT Revolution’s Gene Kim and Electric Cloud’s Steve Brodie, this week’s confab sold out with 500-attendees, and its own program is jammed with a slate of speakers passionate about agile development, continuous delivery and DevOps. devopsy.kinsta.cloud caught up with Kim and Brodie before the show to learn a little bit about what’s on tap. According to the pair, there are four key trends and themes that seem to weave themselves throughout the conference’s programming:

1. DevOps Is About Normal Organizations Doing Heroic Things

According to Kim, the stories being told by the program speakers are as “heroic and amazing” as anything you’d hear from the outlier “unicorn” companies like Google, Amazon, Etsy who are setting the pace for the DevOps movement.

“And they are not only replicating the results you’d typically see from these unicorn Internet companies, they’re doing it inside organizations that have perhaps decades of low-trust, command-and-control bureaucracy,” he says.

2. Chief Architects Are The Newest Actors in the DevOps Cast Of Characters

Not only will the storytellers include ops leadership, dev leadership and executive leadership, there’ll also be a strong contingent of chief architects stepping in to discuss continuous delivery.

“Architects are often the first people to see the problem because they see the end-to-end BLANK stream,” Kim says, “and architects are in a very powerful place to affect change that spans dev, test and operations.”

3. DevOps Spans Across More Industries Than You’d Think

While there will be speakers from the type of usual suspect companies one would expect to be practicing DevOps—Web, IT, software and so-on—the range of types of companies represented by the speakers shows the surprising reach of the movement across industries, Brodie says.

“DevOps is relevant across sectors,” Brodie says. “We have presentation from companies you’d expect, but also, for instance, government, and organizations doing embedded systems development. Sometimes people think of DevOps as big data center-centric, but it is relevant with the Internet of Things and in this mobile world.”

4.Security Is Possible Through DevOps

“The number one objection that comes up over and over again in terms of why we can’t do DevOps is information security and compliance,” Kim says. But more than one speaker will explain why infosec is not only possible with DevOps, but actually stands to benefit. And that includes not only risk management, but compliance in highly regulated industries as well. “We have a whole panel on security and compliance, including James DeLuccia from Ernst & Young, describing how publicly traded, heavily regulated organizations are putting DevOps systems into place and still maintain compliance.”

Even though the conference is sold out, the entire DevOps community can benefit from the programming. The show is offering a live stream starting at 9 A.M. PT for those interested in viewing talks remotely.And if you’re still craving the benefits of networking in a conference hallway track, there’s still time to sign up for Camp DevOps Houston. Registration is still open and it is just $5.

October 21, 2014
Embedding security testing to avoid headline-inducing configuration fails

When news broke earlier this month about the nation’s largest bond insurer exposing sensitive account information to Google’s indexing spiders due to a server misconfiguration, many an IT professional wondered how such a massive fail could have been neglected for so long. Security experts called it out-and-out negligence by the company, MBIA. But the truth is that plenty of other businesses with traditional IT departments could have just as easily suffered the same fate. MBIA was just unlucky enough to have caught the eye of security researchers and inquiring journalists.

The fact of the matter is that most IT shops today still stink at configuration management. According to a survey out by Avaya earlier this year, 82 percent of organizations today still experience network downtime due to human error around configuration changes. In MBIA’s case, it wasn’t downtime but a data exposure that MBIA suffered as the result of a misconfigured Oracle Reports server tied to a web application. The issue allowed back-end data such as account numbers, routing numbers and other sensitive data to be indexed by Google, along with administrative credentials that would let attackers access any other data not already out there for the world to see. Avoiding these kind of blunders stands as one of the big value propositions of DevOps processes and the orchestration toolsets that support them.

“If they’d used DevOps, they’d probably have caught this issue much sooner,” says Rajat Bhargava, CEO of JumpCloud. “They should have controlled configs and change management, and DevOps is generally useful for those things. You make a change, you make it in code and theoretically people ought to be able to document these changes and then apply them across the board. So it is a little more controlled than someone going in manually and changing a setting and forgetting about documenting it.”

However, when it comes to reducing the overall risk posture of infrastructure managed by the constant flux of DevOps’ fast iterations, Bhargava believes many organizations could stand to take their practices to the next level. He says they can do that by better embedding security testing into the DevOps cycles.

“Clearly, DevOps shops are already doing automated testing around their code,” he says. “So there’s no reason why we can’t embed automated security testing just as we’ve automated QA and so on.”

This means not only testing the security of the code itself, but also of the configurations and overall system security once that code goes live. Doing so acts as a backstop to the already automated configuration management process, which can be huge since a human error at the top of the orchestration stack could prove even more disastrous than the kind of one-off configuration error that hurt MBIA.

“With great orchestration and configuration management tools, comes great responsibility. A simple misconfiguration of a system is only going to be compounded to the nth degree when you introduce DevOps configuration tools,” says Andrew Storms, a security and DevOps expert.

Just as DevOps can speed up update cycles for software and configuration, adding security automation and testing into the equation can speed up response to problems—whether they come from internal human error or news of new threats that need immediate attention from both dev and ops teams.

“For example, I know of companies that patched their entire DevOps enterprise for Heartbleed in a few hours,” Storms says. “Security best practices dictate that the security of systems be continuously checked in all environments — from early in the development process all the way into the production environment. Security is struggling to retool their processes and mold their staff to fit into the new ever changing DevOps world. It’s not just DevOps shops that need to do a better job at incorporating security into their processes. However, combining DevOps automation tools with security products is a true force multiplier that is our next best hope to get security right.”

Rich Mogull, CEO for analyst firm Securosis, agrees. He sees the kind of embedded security testing that Bhargava is advocating for as a huge opportunity for DevOps organizations.

“If you’re using the DevOps pipeline, to be able to go ahead and add security testing to that is a massive opportunity, and there are a bunch of tools out there to do some awesome testing (whose effect) would be extremely difficult to replicate using non-DevOps, traditional deployment testing,” Mogull says. “Not only is there an opportunity to get security engaged with automated testing, but you combine that with the building of a secure DevOps pipeline and I think we can wipe out entire categories of security issues.”
For example, at Black Hat Europe this week, Mogull says he talked to a company that no longer has to deal with the traditional firefighting exercise of patching machines.

“When they have to patch for something they actually just push the updated instances through their automation,” Mogull says. “So they’ll rebuild a new image and when they’re done with that, Jenkins will push it out into production and terminate the old instances.”

Mogull also believes that security testing of code is a big part of the value proposition of embedding security into DevOps. Security vendors like Veracode are now starting to tie into Jenkins to let developers know code has security issues and where to find those issues before pushing them into production. And tools like Gauntlt are making it possible to spin up cloud environments and automate vulnerability assessment, network scans and configurations and flag policies through a Jenkins plug-in. Additionally, in the same vein as the patching example, incident response to viruses can be completely rethought, because “if an instance gets infected or something along those lines, they blow them away,” Mogull says.

On a different security front, Mogull says the early adopter he spoke with at the conference also disabled SSH for admin log-ins to their production systems.

“Because, with DevOps, nobody should ever log into a production system,” Mogull says. “Any changes you make, you make at the beginning of your pipeline and then that gets pushed out and flows into production.”

October 20, 2014
Microsoft partners with Docker to bring containers to Windows Server

Docker has taken the DevOps world by storm, and that has caught the attention of Microsoft. Now, Microsoft and Docker are teaming up to bring native support for the container technology to the next generation(s) of Windows Server.

Docker is an open platform that enables organizations to run any application as a Docker container. The Docker containers can be implemented across virtually any infrastructure—delivering tremendous agility and flexibility for managing virtual applications. The container concept accelerates software development timeframes, and allows organizations to reduce costs and improve efficiency at the same time. It’s easy to understand why Docker is so popular.

There have been more than 21 million downloads of the Docker platform, and there are over 45,000 “Dockerized” applications available on the Docker Hub Registry, and 13,000-plus Docker-related projects on GitHub. Just last month Docker took in $40 million in another round of investment.

Major names in tech, like Rackspace, Google, and IBM have already pledged their support for Docker. Now Microsoft wants to bring the hot technology to Windows.

Docker Engine for Windows Server will allow customers to create Docker container applications that can run locally on Windows Server, or on virtual Windows Server instances in the cloud on Azure. In a blog post announcing the Docker partnership, Ross Gardler of Microsoft Open Technologies explains, “As part of this announcement, MS Open Tech is contributing code to the Docker client that supports the provisioning of multi-container Docker applications on Azure. This code removes the need for our cross-platform CLI to bootstrap the Docker host. In other words, we have taken a simple process and made it even simpler.

“This development is interesting because Docker has been a really fast moving technology having come out of almost nowhere a year ago to where now it is an incredibly widely supported technology,” stated Al Hilwa, an IDC analyst. “It is a bit of engineering but Microsoft wisely decided that they did not want to leave Windows developers out of the Docker party. This is good for Microsoft because those 45K ‘Dockerized’ applications will be able to run on Windows.”

To be fair, Microsoft isn’t completely new to Docker. Earlier this year it added support for Docker in Azure, but only on Linux. This new partnership launches an open source project to develop a Docker Engine runtime for building, running, and managing Docker containers natively within Windows Server. Docker Engine for Windows Server will be managed as a Docker open source project, but Microsoft will be a major contributor to the community.

The Microsoft blog post describes why Docker’s container technology is so important for the fast-paved development of a DevOps environment. “Containers do not include a full operating system, consequently rapid development and scaling of container-based applications is possible through very quick boot and restart operations. Furthermore, highly efficient creation of modified container images, by virtue of only capturing the differences between the original and new containers, enables improved management and distribution of containerized applications; the resulting images are both small and highly portable across almost any platform.”

Hilwa summed up with some thoughts about the new strategic vision of Microsoft. “What is even more interesting is that the work on the Docker engine in Windows is being done in open source—really showcasing the new Microsoft attitude towards open source (e.g. the open sourcing of Roslyn a few months ago). I really think we are seeing a faster and more open Microsoft, one that is more willing to integrate and collaborate with competing technologies without a lot of hesitation.”

October 16, 2014
How to talk to your CFO about DevOps
How to talk to your CFO about DevOps

As DevOps practitioners, we understand the value of DevOps: It helps keep our infrastructures automated, allows us to push out code regularly, and reduces the frequency of breaks. But, does our CFO understand its value as well as we do? Probably not. The question is: How do you explain the benefits of DevOps to your CFO in order to justify the cost?

The quickest way to a CFO’s heart is through his wallet. So, the key here is that as DevOps practitioners we need to talk in the CFO’s language, not our own. We need to speak in dollars and cents rather than bits and bytes. This is not dissimilar to many other instances when IT departments have to work with C-level teams that are business, if not technology focused. But sometimes this is incredibly hard for IT folks to do. It’s just not how we are wired, and similarly, CFOs aren’t wired to think like IT folks. While every CFO and organization is different, there are some common things they all care about.

For example, most CFOs love to count things, especially money. Where is it going? Why? What is the benefit to the company? What’s the ROI? You know typical bean counter type of questions that make a lot of technical people squirm.

Your CFO can be one of two types and you can have a lot to do with how your CFO views DevOps. The two types of CFO are:
1. A CFO that understands the why behind DevOps and is a valuable ally to an IT organization.
2. A CFO that doesn’t understand the benefits and will stifle even the best, most productive programs.
So the question is how do you wind up with CFO number 1? Well, part of it is that you need to educate your CFO in the value of DevOps. This is exactly a situation described in “The Phoenix Project” by Gene Kim et al, but it bears repeating here. The other part is that you need to talk to your CFO about a few key topics that will be central to winning them over.

Let’s dig in.
- Generating cash (and by consequence revenue) – the lifeblood of any company, cash, is a top priority for CFOs. A strong DevOps program can tie directly to generating cash for an organization. Here are some examples of how DevOps can help in this area:
o   New products come out to market quicker and more tightly aligned to customer needs. Translation: revenue comes quicker from new products because they are more on the mark.

o   Products are more stable which means that there are fewer customer issues and less downtime. If the organization is subject to an SLA, there should be less money going back to customers. If they aren’t subject to an SLA, with a more stable product, customers are less likely to churn.

o   Back-office systems such as billing and provisioning will have better uptime and reliability. Fewer errors means that there is less churn and more opportunities to sell happy customers more products and services.
- Reducing expenses – the other half of the equation for a CFO is how to keep costs down. DevOps can be a huge win in this area. A fundamental tenet of DevOps is automation. By automating rote tasks, your team can accomplish more. They don’t need to hire more people and/or they can take on more projects. Either way, it is a significant opportunity for IT organizations to do more with less by having resources better aligned on the most critical problems. IT organization should walk CFOs through how investments in automation can pay off.
- Growth – for CFOs, it’s not just about managing the revenue and the costs, it’s also about how the company is going to grow. How can DevOps help a business grow? Growth comes from the increased sales of existing products/services and/or the introduction of new solutions. Building a better product is core to the DevOps mission. In my opinion, the definition of DevOps is actually the increased probability that you will find product/market fit. This means, if you can quickly iterate to find what customers need, you will sell more of it. By building DevOps into your organization, you will also find what customers need so you can introduce new solutions as well.
- Long-term stability – CFOs tend to feel more comfortable with long-term stability solutions. Part of the mission of DevOps is to increase stability of the product and infrastructure. By turning around quick iterations, product solutions should continually be more reliable as well. If the product performs well, customers rely on it. The result? Loyal customer bases and steadier revenue streams. Another aspect of long-term stability is predictability of resources and expenses. DevOps can drive that predictability by closely tying demand for solutions to the expenses. A DevOps program can deliver on a regular cadence of product which can translate into increased visibility and stability for organizations.
The conversation with a CFO about why an organization should invest in DevOps is crucial. If IT leaders can have that conversation using a CFO’s key concerns, then the outcome is very likely to be more positive. Our suggestion is incorporate the four key concerns CFOs have – cash, expenses, growth, and stability – into the conversation.
October 14, 2014
Electric Cloud’s ship.io tackles mobile continuous delivery

As increasingly more organizations run into unique challenges around extending the continuous delivery development model to mobile applications, vendors are coming out of the woodwork to start helping organizations revamp their mobile build and test infrastructures accordingly. This week, Electric Cloud tossed its hat into the ring with the beta release of ship.io, a new SaaS service that it’s launching to help users build, test and deploy native iOS and Android apps.

In spite of the fast-paced demands of the mobile marketplace, according to studies from Forrester, 64 percent of organizations release mobile apps four or fewer times per year per device. Many organizations struggle with the fragmented nature of the mobile device and operating system landscape, which can make testing a nightmare.

“Testing typically needs to cover hundreds of different devices operating across multiple mobile platforms, including mobile devices that are no longer being sold or even supported but that are stil used in the marketplace,” writes Mark Buenen, an expert in managed testing services for consulting firm Capgemini.

As a result, mobile testing lags significantly, with stats out from Capgemini stating that only approximately 55 percent of organizations test their mobile apps.
At the same time, commercial app store demands for testing prior to release raises the stakes even higher for testing, as a buggy release could take days to fix due to the time it takes for app store approval. In just those few short days, negative reviews about the bugs could scuttle the app’s future success. And within the enterprise, mobile apps are gaining importance as mobile applications gain a higher profile within core business processes.

“Mobile app development is increasingly important to a growing range of organizations, and it represents a challenge due to the range of operating systems and devices that each application must support,” said Steve Brodie, CEO for Electric Cloud.

Ship.io automatically detects and configures iOS and Android dev projects to simplify the process of creating code across multiple operating systems, versions and device screen densities. Made possible by the acquisition of intellectual property from AngelPad-backed Cisimple, an early start-up in the mobile continuous delivery space, the service is powered by integrations with GitHub, BitBucket and other Git repositories. Similarly, ship.io centralizes the automation of testing and distribution through integrations with AppThwack, HockeyApp and TestFlight. These capabilities are all condensed into a centralized management platform meant to offer a dashboard into build and test status levels across all apps and development platforms.

The ship.io release is a logical extension of services for Electric Cloud, which already focusing on automating processes for continuous delivery within the enterprise. Last week the company released a new version of its ElectricFlow Deploy product, which now, among other features, automatically detects the differences between what’s about to be deployed and what is already in an environment so that only the changed code is deployed in order to reduce risk and deployment time.

While products like ship.io could stand to make DevOps teams more effective in bringing continuous delivery to their development and deployment of mobile applications, process will still play a huge role. According to Capgemini, in its annual World Quality Report focusing on testing trends, its study showed there was a marked shift in worry about mobile testing challenges from previously lacking tools to do a good job to now lacking appropriate testing processes and methods, which 56 percent of organizations stated as a stumbling block. Similarly, lack of mobile testing expertise was named by 48 percent of organizations as one of the biggest challenges.

October 13, 2014
Task Organization for Enterprise DevOps

In 2007 I served as a captain in the U.S. Army during a 15-month tour in support of the “Surge” in Iraq. I deployed with the 4^th Stryker Brigade Combat Team to Baqubah, where our unit’s mission was, literally, to neutralize Al Qaeda in Iraq and unseat them from their position of consolidated power, in order to deny them the ability to influence events in the capital of Baghdad. You wear many hats during a combat deployment, but my primary roles were that of lead planner and battalion Fire Support Officer (FSO).

As a planner I formulated the maneuver strategy and led the battle staff’s planning efforts in over a dozen battalion-sized operations, and over 30 company operations. Once the planning was complete, I would switch hats and go out on mission as FSO, responsible for integrating all manner of indirect fires to support the ground commander’s fight. This came mostly in the form of calling in airstrikes on targets ranging from suspected deep-buried improvised explosive devices (IEDs) to high-level meetings of Al Qaeda leadership.

I have since transitioned into the private sector and the world of software, and am constantly amazed by how many shared concepts exist between the military and business, particularly in the areas of strategy, project management, and operations. Having said that, there is one concept I found to be extremely powerful in planning military operations which seems to have no civilian counterpart, and may be useful in the discussion on implementing DevOps in the enterprise. It is the concept of “Task Organization.”

Army doctrine defines task organization as “the process of allocating available assets to subordinate commanders and establishing their command and support relationships.”

I know, I know. Right now you’re sitting in rapt jubilation over reading that statement, thinking to yourself, “What the heck is this goober talking about, and what does any of it have to do with software?!?” Stick with me for a minute – I’ll get you there.

Military operations are mission-oriented, and by their very nature extremely dynamic and fluid situations. Task organization is a construct to assist commanders in choosing the right “tools” to accomplish the mission. Back in the homeland, units are homogeneous to facilitate training and administration. But when deployed, units become heterogeneous as commanders mix and match capabilities, forcing collaboration among disparate skill sets in order to achieve a common goal – in military parlance, this is known as the “combined arms fight.” This is how task organization relates to DevOps.

To illustrate, consider the following real-world example. In 2007 my unit’s area of operations (AO) was nestled right in the Fertile Crescent, dominated by irrigated farmland with villages scattered throughout, and a handful of urban centers. Many of those villages had never seen Coalition Forces before, but it was my commander’s intent to visit each, link up with the leadership, assess the security situation, and provide any help that we could. The gambit was intended to co-opt and partner with the local populace to root out terrorist cells.

However, at the time Al Qaeda was very adept at placing IEDs on highly trafficked routes to disrupt our freedom of maneuver, making it extremely risky for us to conduct vehicle movement. Like Waterfall, it was difficult for us to go anywhere due to the extra time, effort, and resources required to travel safely. We simply would not have been able to reach every village in our AO using this method.

To combat this deadly tactic, we first took a look at our task organization and asked ourselves, “What combination of capabilities will give us the most advantage and the least risk in achieving our stated goals?” Actually, we used a lot more swear words when speaking to each other, but I’m paraphrasing to keep this thing PG. After our analysis, we decided the best course of action was to take to the skies, by requesting and task organizing a unit of Blackhawks under our headquarters. We reverted to old-school air assault tactics, delivering units by helicopter – and completely circumvented the deadly IEDs Al Qaeda had laid out for us. Goal achieved – this method worked wonderfully, and the locals were so appreciative of receiving some support that they partnered with us in the fight against Al Qaeda.

Incidentally, it was during this time that I invented a new method for bomb disposal. Our dismounted patrols would travel with an Explosive Ordnance Disposal team (again, task org), but the EOD guys could only hump so much C-4 in their rucks, and there were more IEDs scattered throughout than they could dispose of. Having acquired a lot of experience dropping JDAMs, I knew how deadly accurate they were, and suggested that when they found a bomb buried in the road, I could detonate it by dropping another bomb on top of it. This method worked so well that I started to joke that I’d figure out how to open a can of soda with a JDAM by the end of the tour – again using a lot of swear words. But I digress…

That’s what DevOps is about – not cussing, but collaboration. By discussing our problems with units who had completely different capabilities, we were able to find new and innovative solutions to our problems together. We were able to turn a perceived weakness into a strength, and gained additional speed and agility by using air assault tactics.

Imagine for a minute that the Blackhawk unit in this example is like IT operations – they were responsible for providing the infrastructure and capability to deliver our “application” to the correct “market segment” in a timely manner, so that the “business” could achieve its desired objectives. The dismounted patrol is like the development organization, translating “business requirements” into tactics on the ground. By collaborating together, we were able to refine our process over multiple iterations so that we could Continuously Deliver combat power anywhere on the battlefield, at any time and place of our choosing. This also enabled a sustainable advantage over our “competitors,” allowing us to beat them to market time and again.

Gene Kim offers two DevOps patterns which dovetail nicely with the concept of task organization – embedding development in operations, and embedding operations in development. By task organizing an individual or team from operations into development, operations becomes aware of proposed changes before they come down the pipe, allowing them to better prepare to support those changes. By task organizing a section from development into operations, developers become more aware of the effects of their changes on downstream environments, and can adjust their code or provide more information to make the lives of operations easier.

For those exploring DevOps in the enterprise, I’d suggest an exercise in task organization before you begin implementation. What capabilities and skill sets do you have available to you? What can you request from “higher?” How can you mix and match these capabilities to achieve your stated objectives with the most expedience and the least amount of waste? After you’ve mixed capabilities, who will report to whom, and how will they be supported?

And don’t forget to use a lot of swear words.

About the Author

Rex Morrow is the Marketing Director at Datical, an enterprise software company whose solution, Datical DB, manages and simplifies database schema change management in support of complex, high velocity application releases across any environment, for any database. Prior to Datical, he co-founded Texas Venture Labs, a startup accelerator at the University of Texas, and received his MBA from the McCombs School of Business. Before graduate school, Rex served as a Company Commander in the U.S. Army in the rank of Captain, and was awarded two bronze stars during combat deployments in Iraq. Website: http://www.datical.com/ Twitter: @Datical Linkedin: https://www.linkedin.com/company/datical

October 10, 2014
devopsy.kinsta.cloud Examines DevOps-as-a-Service

Should enterprises buy DevOps or build their own? devopsy.kinsta.cloud looks at the benefits and detractors of DevOps-as-a-Service.

To -As-A-Service or Not To -As-A-Service

Vendors are moving enterprise technologies to the cloud, adding “–as-a-Service” at the end of their titles. Such was the case when Disaster Recovery became Disaster-Recovery-as-a-Service. But not everything that can be done should be. Some say that DevOps-as-a-Service is a good example.

“If DevOps is a growth engine for your company, then outsourcing it is like building a car with the motor located elsewhere,” says Michael Riley, Co-Founder, Boxter http://contentboxter.com. Optimizing DevOps requires tight integration between differing and complex entities. If a service can wrap this system into an external package for an enterprise, then either that customer doesn’t have a real need for DevOps or this approach is going to hurt the business, concludes Riley.

But TriNimbus is forming partnerships rather than selling packages. The DevOps-as-a-Service provider educates enterprise customers, brainstorms with them, and develops a shared vision of DevOps, according to Goran Kimovski, Principal Cloud Architect, TriNimbus http://www.trinimbus.com/. The final product is external to the enterprise in varying degrees, depending on the customer’s comfort level and expertise.

TriNimbus’ Take on DevOps

On the process side, TriNimbus takes the stance that DevOps is not primarily about continuous deployment. “The way we see it, DevOps encompasses many aspects including performance monitoring, management, and being able to recognize anomalies and patterns,” says Kimovski. DevOps includes solid backup and restore and disaster recovery processes for all aspects of the deployment. “We essentially try to match requirements with processes,” says Kimovski.

On the tooling side of DevOps, TriNimbus introduces automation tooling, different services that support automated deployment, and the implementation of automated workflows. The enterprise may have homegrown tools or be running continuous integration. “We try to architect a solution that involves the tools that they are already using and take advantage of those by using the skillsets that exist in their organization. So, we architect a solution that is going to scale and that we can maintain,” says Kimovski.

But taking these kinds of home grown tools, services, and workflows into the cloud with a DevOps-as-a-Service provider is risky. “It can sound like a cost effective solution, but end up killing your company’s growth in the long run. You are taking critical infrastructure and handing it over to a third party. If they care more about your business’s growth than you do, then you have a whole other problem to solve,” says Riley.

Software Development Expertise

Creating DevOps-as-a-Service requires expertise in software development. “Our background is software development,” says Kimovski. TriNimbus uses solutions architects and software engineers with 10 to 15 years of experience. TriNimbus has a background in continuous integration and build and release management.

TriNimbus’ experience enables it to work with the customer to break down barriers. In one instance, the customer’s obstacle is patching. “We have a current project that we are working with a client. It’s fairly complex. Each key area has its own piece of infrastructure. Patching it all together is a problem of integration and registration,” says Kimovski. It’s not a single problem of pushing a build onto a machine. TriNimbus built an orchestration workflow that patches things individually and supports rolling patches back if anything goes wrong.

The Cloud Advantage

TriNimbus uses the cloud (Amazon’s AWS http://aws.amazon.com/) to give developers greater access to provision things like compute and storage. “AWS as a software arm seems to understand the need for embedding tools into the platform itself,” says Kimovski. Developers may want to increase the frequency of automated testing. They may want to do load testing. TriNimbus works with developers to enable them to clone a production environment to do that.

But simply cloning a production environment in the cloud for automated testing using a DevOps-as-a-Service provider is more complex than it sounds and carries some concerns. “Cloning a production environment for testing can help you solve problems, but does not help you create a production environment in the first place. It could also cause problems if it’s not directly comparable for performance,” says Riley.

Increasing Release Frequency

Though continuous deployment isn’t the only thing DevOps has to offer, an increase in release frequency is something most TriNimbus customers want. But even TriNimbus admits there are no guarantees. “Just because we have the tooling automation in place doesn’t mean that magically things will start to happen faster,” says Kimovski. A lot of it has to do with being able to test more frequently and get confidence sooner that what the customer has built so far is not going to break things. As a tool in the right hands, that can raise release frequency.

October 9, 2014
Q&A: Gene Kim on the pain of accruing technical debt

Last week, we ran the story Paying Down Technical Debt. This week, we’re discussing that very topic with Gene Kim (@realgenekim). As a follower of DevOps trends, you probably are already aware of his most recent book project, The DevOps Cookbook, and most certainly his previous works, The Visible Ops Handbook and The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.

Gene also founded the security firm Tripwire and was the CTO there for 13 years. Gene has made a career of studying high performance companies and we’re please he took the time to share with us his thoughts on technical debt.

devopsy.kinsta.cloud: Tell us what has been keeping Gene Kim busy lately?

Gene: There is a lot going on. Right now, I’m very concentrated on the DevOps Enterprise Summit taking place in San Francisco October 21 through October 23. The summit will focus on Lean IT, DevOps, and Continuous Delivery. We’re pretty excited about it. We have 50 speakers from large enterprises such as GE, Disney, and Macy’s, and other incredible speakers.

devopsy.kinsta.cloud: Sounds like a great event, Gene. I’m curious to get your thoughts on technical debt. Some people argue that it doesn’t exist. Others contend that it does. Does technical debt exist, and if so how do you define it?

Gene: There is a great definition of technical debt that was given by Ward Cunningham: “Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite. … The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a standstill under the debt load of an unconsolidated implementation, object-oriented or otherwise.”

My simple take on that quote is that technical debt is what you feel the next time you want to make a change. It accrues from not only all the shortcuts you make in a project when it’s rushed, but even every time the developers don’t write an automated test – every time that you don’t do data code analysis. When you skip these things, the debt builds up every day. It’s basically all the variants, from the right way to do things versus the way we actually do things. The more of those that accumulate create this ever-increasing amount of technical debt.

devopsy.kinsta.cloud: What does technical debt commonly look like in an enterprise?

Gene: Randy Shoup gave a great talk on this, The Virtuous Cycle of Velocity: What I Learned About Going Fast at eBay and Google, at FlowCon San Francisco 2013, when he was CTO at KIXEYE. In that talk, Shoup provided the countermeasure to technical debt. He stated that any sort of thing that affected performance and quality, as defined by reliability of scaling, was treated as a priority zero defect.

Essentially, what he’s saying is whenever there’s a reliability or a scaling issue, it’s treated as a priority zero defect. In other words, it is as bad as a site down issue. What it means is that it’s like Toyota and when they have a serious issue, everyone stops work in order to swarm the problem until it is resolved. If they can contribute to helping the problem, they do. This is the opposite of what happens in most organizations, where it’s “never my problem” and when technical debt is allowed to accrue. Where you put problems on to the defect backlog, you put it into the security remediation queue, but it will never get done.

It goes there to die. It’s the landfill of good ideas that never get implemented. What Randy Shoup tells is that investing pays off. I would say that’s exactly the culture that enables high performance. High performance is where you’re getting 30 or fewer code deploys, very short lead times, high success rates when they deploy, and short lead time to prepare. That’s as true for operations as it is for information security.

So, technical debt is bad for ops; it’s bad for security. It’s also bad for development, because it slows down future flow.

Shoup nails the difference between the virtuous versus the vicious cycle. The virtuous cycle is described as one having a solid testing cycle. You have good testing so you have a solid foundation, which lets you be more confident. This lets you go faster and faster. The vicious cycle is you don’t have testing. You have a very shaky foundation. You’re afraid to make changes. And you go slower and slower.

devopsy.kinsta.cloud: In your experience of watching high performance organizations, what is the challenge of getting from the slide that discussed the vicious cycle and the technical debt world to one with the virtuous cycle? How do you implement what’s detailed on slide 13, I think it is?

Gene: It really is policy. We have to have a policy that elevates paying down technical debt and enables teams to focus on the nonfunctional requirements – a policy that one’s definition of done is whenever the developer says it’s done, then it’s done. It’s not done until there’s an automated test, an automatic deployment process, so that anyone can push the button and it goes through the test cycle and gets deployed into production.

Deliverables also have to operate in production as designed. That’s a good definition of done. That prevents us from going from sprint to sprint and accumulating technical debt behind us. I would say the countermeasure to technical debt is the right definition of done. It’s like in the agile philosophy quote: The definition of done is at the end of each sprint interval we have working and potentially shippable code.

I would add that it is integrated into chunks. There’s a checking depository. There’s automated testing that anybody can run before the code can be deployed. I think these are good countermeasures.

devopsy.kinsta.cloud: That’s excellent. Now, to get from the vicious to the virtuous cycle, I imagine it takes significant investment upfront before the payoff?

Gene: Unfortunately, this is one of those things where you go slower before you can go faster. But the benefits are so compelling. We know that high performers are doing 30 times more frequent code deployments, and they can do them 8,000 times faster. That’s 8,000 times faster. So the payoff is huge.

Think about it: If you’re not doing a deploy a day, that means that you are 100 to 1,000 times slower than high performers. High performers can execute a deployment in minutes or hours, whereas lower performers need weeks, months, or quarters. Of course, we have to take investment to go from weeks, months, and quarters to on-demand, executing within minutes. But the value of doing that is too high to ignore.

October 7, 2014
Targeting software development to SoCs / NoCs using a DevOps approach

Without hardware to run it, software is useless. Hardware influences software development, since the software must fit the hardware platform and take advantage of its strengths, abilities, and components. Network-on-Chip / System-on-Chip hardware is a relatively new platform enabling communications between network and system elements that exist on the chip at the molecular level. This is a fascinating area. Learning to target software development for the NoC / SoC platforms using DevOps approaches should prove equally enthralling.

The Top Three Keys to a DevOps Approach to Developing Software to an NoC / SoC Platform

According to Aaron Cois, Researcher, the Software Engineering Institute, Carnegie Mellon University, the top three keys to a DevOps approach to software development for NoCs and SoCs are as follows. First, nurture a collaborative culture where collaboration between the parties involved in the software development process can flourish. Second, leverage Infrastructure as Code (e.g. simulation of hardware, EDA or ECAD applications, and programmable environmental configurations). Third, use automated creation / testing components for both hardware and software to create a fully automated DevOps pipeline.

The enterprise must foster equal collaboration and engagement among the necessary participants. “Getting technical staff with specific areas of expertise (Dev, Ops, Security) fully engaged in the project from the beginning ensures that their respective areas of expertise are first-class concerns in all system design and implementation decisions,” says Cois. This is correlates with Conway’s Law, which states that “two software modules A and B cannot interface correctly with each other unless the designer and implementer of A communicates with the designer and implementer of B. Thus the interface structure of a software system necessarily shows congruence with the social structure of the organization that produced it” (http://www.melconway.com/Home/Conways_Law.html).

Therefore, says Cois, a team of software developers will design a system primarily to meet software development concerns (features, performance). A team that also contains IT Ops professionals will design a system that also meets operational needs (deployment, maintenance). A team that also has Information Security professionals will produce a system designed to address security concerns. “All team members have to be equally tasked, engaged, and accountable for the project, however, for this effect to work,” says Cois; “this is where organizations commonly fail.” In real-world scenarios, tasking is unequal such as when IT operations attends a few meetings to consult, but the primary project team is made up entirely of developers. Or, in another example, InfoSec does a system audit before release and identifies issues that development should fix, but because InfoSec wasn’t strongly involved in the system inception and design, development may never complete the fixes.

The enterprise must focus simulation techniques on the development process. “Simulation and mocking are integral in situations where you are developing custom hardware as part of the overall process. This is the case with SoC/NoC systems,” says Cois. By simulating the chip hardware, developers can script/automate a number of things:

1) Testing hardware configurations
2) Testing hardware interfaces to software systems
3) Testing software/firmware deployment onto hardware
4) Security concerns
5) Possible networking scenarios on NoC chips
6) Other quality attributes such as response time and load testing

“Simulating the hardware platforms enables the development team to perform the same automated/continuous deployment techniques that are common to DevOps in general on SoC/NoC development projects as well,” says Cois. This gives developers advantages such as the ability to automatically deploy the system multiple times per day, which builds confidence in deployment success (through repetition). As with all automated deployment scenarios, the parity between the internal deployment platform (the hardware simulation) and the production deployment platform (real hardware) defines how much confidence development gains from repetitive deployment (e.g., if the simulation isn’t accurate, you won’t be very certain deployment will work on real hardware).

Challenges & Solutions

There are challenges to developing SoC/NoC software the DevOps way. First, the organizational culture needs to change so that communication and collaboration between various areas of expertise can dramatically increase. This is difficult. According to Cois, the solution is to hire generalists who are T-shaped professionals with a wide T. “They will have their specialty (Dev, Ops, Sec) and an understanding of other project needs. This understanding leads to empathy and effective collaboration–to the benefit of the project,” says Cois.

The second challenge lies in adopting automation, which is a significant technical shift, especially on legacy systems. It requires planning and technical expertise. “Knowing your goals before you start is key–a lot of organizations may jump into implementing automation without knowing the benefits they expect to see–or how to measure them,” says Cois.

October 6, 2014