devopsy.kinsta.cloud

Tag: CD

A 6-pack for building a business case for CD

There are all sorts of compelling business benefits awaiting those who adopt Continuous Delivery. It has the potential to reduce operating costs, reduce defects and redundant work, improve efficiency, and result in a much faster delivery speed. By releasing applications to market faster, you boost your chances of success and gain a competitive advantage. But how do you get from the theory to the practice?

If you really want to scale up and realize the benefits of CD, then you’re going to need money to retool, retrain, and change the existing culture. In order to secure the budget you need, all of the key players in your organization have to buy into the change. A strong business case is essential. Here are six key areas to look at when you build yours.

Production incidents

A natural place to start with building a business case for CD is with production incidents, because they’re very visible. When you perform a root cause analysis on an incident, you’ll often find that the software worked fine in the development environment. By standardizing the environment configurations you can reduce the number of incidents, free up resources from the provisioning process, and reduce wasted time.

It’s possible to calculate the cost of data center downtime for your company, and you can use root cause analysis to find out where provisioning errors were to blame. From that, you can extrapolate the potential savings made by reducing these outages with standardized environment configuration. You also have to consider the reduction in wasted time and effort, resources that can now be productively employed.

Deployment failures

Take a look at your past deployment failures and work out the average cost to the business of rolling back the deployment, diagnosing the cause of failure, and remediating. By automating deployment you can cut these costs completely.

Manual testing

To employ an API-based testing model that can be integrated into your delivery pipeline will pose an upfront cost, but that cost per build decreases with each new build. With manual testing, the cost rises with each new build. Automated testing gives you better coverage, earlier. If you can find defects earlier they tend to be quicker and easier to fix, which reduces the potential knock-on impact in terms of production incidents, and, ultimately, reduces costs. You’re also cutting manual labor costs and overhead.

Manual build process

If someone on your team is writing scripts to automate your builds, there’s a labor cost involved and they may cause a potential bottleneck. If you can switch to a centralized delivery pipeline where the build is automated and which development teams can customize for their needs, you can reduce the time and effort required. You also free up the developer who would have written the scripts to work on improving the product instead. At the same time, you’ll boost oversight, control, and reliability in the build process.

Better analytics

A lot of development time is wasted on building features that don’t add value. According to research by Forrester, only around a third of the ideas implemented result in a measurable improvement in the customer experience, a third have no obvious effect, and the final third actually have a negative impact. By building analytics into your code, and constructing a strong feedback loop, you can find out what really improves the customer experience and focus your efforts there. The earlier you can get feedback, the easier it is to prioritize and maximize the value you’re adding.

Focus on the big picture

Most business cases might focus on some of the benefits we mentioned above, and that’s because they’re easier to measure. But it’s the intangibles, such as improved competitiveness, goodwill, and revenue, which are harder to quantify, but which have the greatest impact.

You want to present the potential cost savings, but this is about more than cost cutting. The long term benefits filter through as you ramp Continuous Delivery up. Faster feedback loops and a greater understanding of your customer are laudable aims for any business. You’ll have to delve into specifics to generate the persuasive numbers you need, but it’s important that you keep those intangible benefits front and center as part of the business case you’re constructing.

About the Author/Andrew Phillips

Andrew Phillips heads up product management at XebiaLabs, a provider of continuous delivery automation. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.

November 10, 2015
To Be Continuous: Does your company have to be all in to do Continuous Delivery?

In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk about the spectrum of continuous delivery and where continuous delivery will and will not work.

Edith is CEO and co-founder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

Paul is founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

[soundcloud url=”https://api.soundcloud.com/tracks/231204216?secret_token=s-45TTy” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

Edith: So Paul, does one have to be all in to do continuous delivery?

Paul: So I think the obvious answer is no. And in particular, you know that the answer must be no because nobody starts at all-in, or at least if you’re starting a new project you can be all in, and many projects are not all-in.

So someone starts somewhere and they say, “Now we are going to do continuous delivery in some way,” and so they have to go from zero to full continuous delivery along some way.

Edith:

People have this conception that continuous delivery just means pushing any time you want, and a lot of very old school development shops did that. They would just hot patch and production constantly.

Paul: Right. So, I mean, are we calling that continuous delivery? I feel that the idea of SSH-ing into the server is not continuous delivery somehow.

Edith: But people have this misconception because they’re like, “Oh, I had a continuous delivery before this was a thing, I just patched it production.”

Paul: I’m going to put a lot of money on the line here and say that if you define continuous delivery as “I SSH into the production server,” then you are not doing continuous delivery right. In fact, you are doing it considerably wrong.

Edith: What exactly are you doing wrong then? What steps are you skipping?

Paul:

The whole point is that you have this automated process for releasing software. And SSH-into the production server is the very opposite of that.

That’s not to say you shouldn’t do it in the occasional emergency. And occasionally, if we have some sort of catastrophe, we will SSH into the production server, replace some code, execute some code on the server to keep things as they are and prevent customers from noticing or from things from going down.

But that isn’t the same as a continuous delivery pipeline or a release workflow, or an automated release workflow, which is essentially what continuous delivery is.

Edith: So continuous delivery isn’t just shipping stuff out to production, it’s more having all the processes behind it.

Paul: It’s very much the process. So when you think about can you be a little bit continuous delivery, I think what you’re looking at saying is there a lightweight version where we get some benefits of the continuous delivery as we go between all and nothing?

Edith: Can you be, do you have to be all in, or can you be a little bit continuous?

Paul:

I think the very obvious instance of a little bit continuous is where you deliver the code where you have the automated release process builds some sort of artifact and then someone manually deploys that by a single click.

Or, in fact, where a build happens and test happens and the remaining automated release process is triggered by a human saying, “Now we are going to release,” and then everything else is scripted, and so they just have one input, which is now.

Edith: Oh, that’s really interesting. So do you think to truly be continuous it has to all be completely automated with everything always pushed?

Paul: No, no. I don’t think so. I think it largely depends on the size of the team and that sort of thing. But I’ve seen a couple of examples where our customers were asking for particular things, and it indicated that they had sort of semi-continuous workflows.

And one example of that is a team that only wants to do continuous delivery during work hours.

Edith: Yeah.

Paul: And there was another team, which only wanted to do continuous delivery at midnight because that was the only time that they didn’t have enough customers on their site to be able to clear the caches.

Edith: Yeah.

Paul: Now if they wanted to go to complete continuous delivery they would have to say, “We are going to clear a subset of the caches,” or have some other caching policy that allows us to continue or that allows us to deploy at any time, but where they are is basically were semi-continuous.

Edith: I had a really interesting chat with Douglas Squirrel, who actually listened to our podcast, and he does a lot of consulting for companies in England, and he said he goes to a lot of companies that have all the tooling to be continuous.

Paul: Okay.

Edith: But they don’t actually use it.

Paul: And what does that mean exactly?

Edith: They have set-up continuous integration, they could, in theory, deploy whenever they wanted but they still, for whatever reason, still cling to their old schedules.

Paul: Okay. So, I was looking at this very interesting article on the First Round Capital blog.

Edith: Oh Jocelyn’s.

Paul: Yes, that sounds right.

Edith: Jocelyn Goldfein, who is at Facebook, right?

Paul: Yes, that’s right. Yes. So, what she talked about was this idea that delivery schedules are useful in certain contexts. And in some cases velocity matters, and that’s kind of the Facebook case, and in some cases predictability matters a lot more. And if you’re delivering enterprise software, for example.

An enterprise is not going to allow you to continuously deliver your product into their firewall or into their enterprise. What they’re going to want is every quarter a new thing.

Multiple times a year preferably, but each time you release to them it creates a new workflow, and they want to be able to audit it and see a change log and validate that things are, they might have a change process internally.

And so it’s not necessarily going to be a good idea to continuously deliver there. But a team that is building for that can do a continuous delivery process internally where they produce all of the artifacts and they do a complete release on every single push. Even if that is not released.

Edith:

There are a lot of reasons why an enterprise doesn’t want to take new features constantly, and a lot of it comes down to training. If there are big new features that need to be rolled out to people then they have to take on the cost of training.

Paul: Right.

Edith: It was actually interesting. I chatted with a guy from Square and he talked about how they really try to limit the amount of pushes they put out to the actual cashiers.

Paul: Okay.

Edith: Because of the training cost. I mean, a cashier at a coffee shop is doing a gazillion other thing and they can’t just suddenly pop up some new functionality and expect them to grok it or even take a tour.

Paul: Right. That’s an interesting idea.

Edith: So, although they have all the processes in place to do a lot of changes, to do continuous delivery, they actually really limit what they push out to the true end users.

Paul: And it’s funny that the people who have far, far more users, someone like Facebook, can just do those changes because they don’t necessarily value an individual user on a…

Edith: You’re walking down a path.

Paul: No, I’m not saying that they don’t value their users. It is safe to piss off an individual user. Less so for someone like Square. And as you go up the value chain, the more they’re paying you or the more involvement they have to have in the product decisions, the less able you are to piss them off.

Edith: Facebook is actually a fascinating example because at any time they’re going to be pissing off somebody. It’s just a matter of, it’s kind of like Apple.

Ya know?

Paul: Right.

Edith: They’re always pissing off somebody, it’s just who.

Paul: Did you hear about today at Facebook where all the feature flags went on?

Edith: It was LinkedIn.

Paul: Oh, was it LinkedIn? Okay.

Edith: Ya. Well, it probably has happened to everybody that’s done feature flags. There’s the day that that happens.

Paul: Right.

Edith: And, it’s ugly, yet. Facebook is fascinating. I actually, I wrote a blog post about Facebook and how they use feature flags. They have a product called Gatekeeper.

Paul: Right.

Edith: That controls everything, everything. And after they had a huge PR disaster, i think it was in 2011, with their news feed, they actually do really care if they piss off a lot of people.

Paul: Right, right.

Edith: And that’s why they implemented Gatekeeper where they do a lot of 1% roll-outs.

Paul: So, I remember when I was setting up Circle I sat down and talked to my friend who worked at Facebook and talked about their workflow and that sort of thing, and their release process was that code got released on Tuesdays and everyone who was releasing code, or who had code in that bit that was going out, had to be available during the release process.

And the release process didn’t turn on any features because that was all done by Gatekeeper, but the code itself had to go out. And they had to validate that everything stayed the same and everyone had to be there. And so they had a fairly continuous delivery process but were still, manually there was a release going out.

Edith: Well ya. You want to have some sort of enforcement that people look at stuff and care.

Paul: Care in what sense?

Edith: So, the example I heard about also at Facebook is it’s so large that there’s just no way that all the groups can coordinate with each other. So literally what they do is when they roll stuff out with Gatekeeper they roll it out internally to Facebook-only employees and that’s how they find out if stuff breaks. It’s not like at a smaller company like CircleCI.

Paul: Right. But there’s a difference between the code being rolled out and the feature being rolled out.

And they don’t do, I don’t know what the story is nowadays, but at the time they certainly didn’t do continuous delivery on code. Or continuous releases or whatever you might call it.

Edith: I don’t actually work at Facebook, but what I did hear was that– that they just literally can’t coordinate. When you’re a small company you can still coordinate in a HipChat or a Slack room like, “Hey, we’re pushing this now, start looking for changes.”

Paul: Right.

Edith:

But with Facebook, the only way they know if something is broken is if somebody complains.

Paul: So there’s an interesting discussion about microservices at AWS. I guess they didn’t call them microservices at the time, but having fully service oriented architecture.

So, one of the things they talked about is how when you roll, or when you continuously deploy a service, you might break other services. So, the thing that actually breaks may not be in any way related to the people who’s alarms start going off. And, in fact, there may be multiple steps away from who’s alarms are going off.

Edith: Google has a similar system, and I heard that they monitor 200 plus different metrics. And one of the ones they monitor is just do people say, “Gmail sucks more?” on Twitter.

Paul: Oh. Interesting.

The thing that I’m getting at in terms of how little or much can you continuously deliver is, if you are Google or you’re Facebook and you have the monolithic code repos, the thing that you’re releasing is it’s not the same release process as someone like Amazon who has what I would describe as pure, continuous delivery.

Every service does it’s own continuous delivery and your dependencies may be changing at any time and you don’t actually know what’s going on. Versus Facebook, we are taking this multi-gigabyte binary and we’re uploading it to a set of servers, and sure, we’re slowly releasing and maybe we’re doing that multiple times a week or multiple times a day.

But it’s not quite the same as Amazon releasing every 11 seconds. And I think we can see, even between these multiple large companies who have incredibly advanced release process, that you could say that they are clearly at different spectrums along continuous delivery.

Edith: So, what do you think is suitable for, let’s say, a medium company with, let’s say, 10 developers?

Paul: I mean, there’s a lot of different things going in. Who is your customer base? What is the level of safety that you need? Let’s suppose that it’s a consumer web app. If it’s a consumer web app, I would expect that that company should be continuously delivering all the time. Especially if it’s slightly okay to break things.

Edith: Yeah.

Paul: Or at least, you break things you can kind of roll them back or whatever within a couple of minutes. A team of that size might not have the resources to make their continuous delivery process perfect. But it seems likely that they would have more success shipping with continuous delivery than shipping without it. So lower risks, higher velocity. That sort of thing.

Edith: It’s funny how continuous delivery has permeated even the VC world. Jason Lemkin, the guy behind SaaStr,

Paul: Right.

Edith: He tweeted yesterday that one of his due diligence questions was how,”How often can you roll back, and does it work?”

Paul: Right.

Edith: And it’s funny that I think of that as a very technical thing, like, “can you roll back?” But to the VC is seen as a sign of technical sophistication.

For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.

November 4, 2015
Continuous Delivery versus Continuous Deploy
Getting a solid grasp on the meaning of DevOps requires a clear understanding of the terms used to describe it. Over the last several years, we have seen a paradigm shift in the way software is developed, moving away from a waterfall approach, with monolithic releases, to an agile approach with incremental releases. This shift has changed the way in which the Application Lifecycle (ALM) is managed. Focus around what we normally would refer to as ALM is now more accurately described as Continuous Delivery. Continuous Delivery is not a set of tools, it is an emphasis on people (culture), tools and processes. The change from ALM to Continuous Delivery includes three very basic principles:
- The software is always deployable which means you prioritize keeping the software deployable over adding new features. To do this requires smaller, less risky changes released overtime instead of all at once.
- A feedback loop of the build, test and deploy steps expose the readiness of the software allowing anyone from development, test or production to determine that an immediate release of the software to any environment can be approved.
- Automation of the build, test and deploy process is critical in order for the process to be repeatable and to support a higher frequency of builds, deploys and test cycles.
So like ALM, there is an awareness of a lifecycle (Dev, Test, Prod). But unlike ALM the lifecycle should not be seen as linear. With Continuous Delivery, a software update may move from Development directly to Production without having entered the Test phase. Because Continuous Delivery principles support keeping the software in a deployable state, with a visible feedback loop to determine the readiness of the software, software can be confidently moved into any environment at any time.

This change between a linear, waterfall approach, to an non-linear agile approach is far more realistic in the way organizations manage their software. Over the course of my 20+ years’ experience working with both large complex enterprises and small software teams, the need to be able to perform a release, into any environment at the ‘drop of a hat’ has been consistent and problematic. The principles behind Continuous Delivery defines a more realistic approach to software development.

Moving from a linear application lifecycle approach to Continuous Delivery includes making changes at many levels. First, developers must begin embracing Agile practices, managing smaller, incremental releases. Because Agile increases the frequency of builds, testing and deployments, DevOps tools must be in place to support the automation of these tasks. In the waterfall approach, teams had the luxury of time to get build scripts to work, execute test manually, and write a deploy script for every environment. Continuous Delivery does not offer that luxury therefore automation of these tasks becomes critical. This is why Continuous Build, Continuous Deploy and Continuous Test tools are implemented. Continuous Build addresses the frequent building of code changes, updating binaries upon a check-into version control, and reporting on what artifacts were impacted providing feedback into the readiness of the build.

Continuous Deployment automates the movement of those binaries to any endpoint in any environment and supports server configuration management, all performed on an incremental basis with feedback showing the inventory and the changes. Continuous Test automates testing so developers and/or testers can run through critical functional test, and code analysis to provide feedback on the readiness of those binaries.

To maximize on automation, these steps are chained together using the workflow engine of the Continuous Integration Server. The workflow can be triggered upon a source code check-in, kicking off a compile/link (build), running code analysis, deploying to an endpoint and running test. In Continuous Delivery, each environment may have their own Continuous Integration workflow that performs the steps needed. For example, a Production CI workflow may just perform activities related to moving the code to hundreds or thousands of endpoints, with database updated, but not execute a build or test.

Not all is perfect in the Continuous Delivery approach. Automation continues to be a bottleneck. As with most paradigm shifts, changes in behavior and adoption is a major hurdle. In most companies, the use of manual scripts to perform builds, deployments, test automation and manage infrastructure are still called under the covers of the Continuous Integration workflow. These are the same scripts that were used in the waterfall approach, except now there is not the luxury of time to fix the scripts or update them to adjust to software changes. To support Continuous Delivery, the build, deploy, test activities must be elastic, automatically adjusting to code changes, database updates and server configurations.

To create a more dynamic Continuous Delivery process, DevOps tools are moving away from the need for coding one-off scripts and instead moving to a model driven approach with Playbooks and Plug-ins. Many developers still hold on to these scripts with the argument that they do not want to be at the mercy of a ‘vendor’ or ‘tool’ to manage these low level tasks. As Continuous Delivery pushes the frequency of releases, this final paradigm shift, moving away from the old approach to one-off scripting, will need to move into full automation. Continuous Build will be supported by Build Automation, Continuous Deploy will be supported by Application Release Automation and Continuous Test will be supported by Testing Automation and Virtualization. The automation tool chain will be orchestrated via a Continuous Integration Server and the entire process will become a reliable and fast Continuous Delivery process.

About the Author/Tracy Ragan

Ms. Ragan has had extensive experience in the development and implementation of DevOps for large organizations. She began her consulting career in 1989 on Wall Street specializing in build, testing and release management for the distributed platform. It was during her consulting experiences that Ms. Ragan recognized the lack of standardized development procedures on open systems that was common on the mainframe. In the years leading to the creation of OpenMake Software she worked with development teams in implementing a community driven standardized build and deploy process that enabled frequent builds and releases, automated around version control. Her knowledge and experience contributed to the creation of OpenMake Meister, the first commercial Build Automation solution. Ms. Ragan served on the Eclipse Foundation Board of Directors as an Add-in Provider Representative for 5 years. She has been published on numerous occasions and regularly speaks at conferences including CA World where she presented for 15 consecutive years. She holds a BS Degree in Business Administration, Computer Technology from California State University, Pomona.
October 30, 2015
The Sea Change that is DevOps – How it all started: Part 1
Over the last 20 years, software has slowly taken over the world. Businesses that historically operated on 8-hour business days are now 24×7, navigating a global economy that never sleeps. The Internet enablement of everything has thrust what was originally an important support role into the lifeline of a business.

A decade ago, Agile software development methodology forced more pressure into the IT system by deploying software at an ever-increasing rate to the point it’s at today – where innovative companies can deploy new software multiple times a day. DevOps is at the forefront of addressing the question of how IT deals with these problems, but like Agile before it, there’s a learning curve.

In this two-part series, I will discuss how the IT world worked before DevOps and how development and operations teams united to adopt more Agile practices.

How things were

Before this sea change, a company would hire IT professionals who would purchase and configure servers, place them in a data center, copy mission critical software onto them and with some help from software engineering, get things running. After some quality checks, the IT team would lock down the datacenter controlling both physical and virtual access and “let things run.” The common way to provide stability was simply to not let anyone touch anything, as most problems have been (and still are) self-inflicted wounds. Keeping the system “static” kept a lot of problems from surfacing. The software was probably not more stable; you just didn’t find the problems as fast.

In this world, IT professionals had a feast and famine existence. If a new service deployment was happening (usually every 3 to 6 months) it was an all hands on deck activity reminiscent of a rocket launch. The same manual deployment would happen, along with software engineering, database engineering, etc. Data migration would occur, new software would be installed, and the system would manually be brought back up. In the event of a software problem, or data migration problem, the team could elect to rollback and bring the old software back up. If things went well, then again some basic end-to-end quality assurance would take place and the new software release would be unleashed on the world. More progressive companies would do this for part of the system, diverting traffic to avoid perceived downtime. It would often take an entire night to do a new service release, and companies generally did them at night because downtime was not unusual.

Then software development changed

The competitive landscape that the Internet created along with the now ubiquitous SaaS business model made companies recognize that their ability to innovate the product was a competitive advantage. A lot of companies had adopted Agile software development practices in response to the increasingly high failure rate of larger efforts. Part of Agile is creating smaller development efforts that get code functional faster, reducing the size of the endeavor. This is commonly referred to as a “software sprint.” This promise of innovating the product faster was very alluring, especially to Web 1.0 companies that were now in feature battles with more nimble Web 2.0 companies who had more quickly adopted Agile. It was becoming more and more evident that the problem was shifting from an issue of creating software fast enough to deploying the necessary code in an expedient manner.
The big problem was the deployment stage of the process. IT teams were accustomed to changing production systems every three to six months. This combined with a trusted model that largely kept software developers out of production systems, effectively nullified any Agile gains in SaaS businesses. In other words, half the product development cycle had been fixed with Agile, but the other half deploying the software had not yet caught up.

Operations as a changing landscape

Major changes were also occurring for operations inside companies, many of which were driven by the development side of the house.
- Virtual server technology started to get major traction within production environments. This allowed teams to treat servers as software. You no longer had to run the operating system and application on a physical server, you could run them on a virtual server (that ran on physical servers). The configuration of servers that, by hand, took an incredible amount of time, could essentially be defined once and then cloned, the same way you would clone a file. Deploying a new server was much easier, and a corrupt server could simply be deleted and reborn in minutes. All these virtual servers could be managed by a single piece of software called a hypervisor, with which creating a new “server” would take a minute, when it used to take weeks.
- Puppet and Chef were born. At the same time development organizations, recognizing this major problem with deployment, started addressing the production configuration problem with tools such as Puppet and Chef that essentially allow software and IT teams to codify the configurations of virtual and physical servers and deployments of new code. This allowed for repeatable processes that could be rapidly done and redone. No longer did your IT team have to install software by hand, instead they could code it.
- Continuous integration and delivery enabled software teams to use newly developed tools like Jenkins to do nightly software builds, perform unit tests of parts of the software and automatically deploy software into production. All of these tasks had been highly manual operations historically and fraught with mistakes and inconsistencies. Codifying these steps would give way to deploying software within minutes or seconds rather than days or hours.
Most importantly, all of these efforts were largely driven by the development of more progressive operations in the company. The operations side of most companies was never conceived as a “maker” culture; the handwriting was on the wall – operations had to change or it will would have become a victim of this revolution. I’ll take a look at some of the resulting changes in the second part of this series.
September 3, 2015
American Psychological Association Finds CI & CD Therapeutic

The American Psychological Association (APA), the largest scientific and professional organization for psychology in the U.S. publishes the PsycINFO database with 4-million records and more than 90 peer reviewed journals, books, and other publications, starts Beverly Jamison, Sr. Director, IT Architecture, The APA. “We maintain a database of psychological tests and measures and psychotherapy demonstration videos,” says Jamison. The APA publishes member information and other data as well.

Because a broad audience including researchers, practitioners, members of other professions, students, the media, and the public use this data, the APA keeps a precise inventory of the data and its elements, checking it for quality and delivering it in a number of ways including as integrated and multimedia materials in B2B and B2C contexts.

APA Dev and Ops Before & After CI and CD

When Jamison joined the APA, some odd number of perl / shell-based coders like her used the VI editor as their IDE while others of the ColdFusion persuasion used Dreamweaver as their IDE. Developers deployed direct to the live system, at off hours, with the developers and perhaps one QA person testing AFTER the deployment was completed, says Jamison.

Today, the APA uses svn for version control for legacy development projects where they are accustomed to that and Git and BitBucket for newer projects that seem to benefit from that. “We use Ant for the scripts to deploy code from version control. For unit testing, we use X-Ray for xquery, JUnit for java, and mocha for javascript,” says Jamison.

The APA uses the Roxy community tool for deployments to MarkLogic to address MarkLogic database instances and configurations, which provides uniformity of the MarkLogic databases, explains Jamison.

The nature and purpose of each team inside Dev and Ops determines the usage of the tools. The work of the MarkLogic team dictates one application usage while the priorities of the identity and integration services team require another. The MarkLogic team deals with data services and data models that are often complex; as such they make a great deal of use of unit testing to ensure that a single change doesn’t lead to broken software throughout, explains Jamison. Identity and integration services uses Jenkins to script their builds and to ensure that each build is current; QA analysts test to ensure that their complex scenarios are reliable, says Jamison. Likewise, the UI team has still another set of priorities and usages.

A Panoramic View of the APA’s New Feature Development Process

First the developer will check out a branch from subversion to a local IDE, make and then test the altered code and run unit and regression tests on existing code, and update unit tests, configuration scripts, and any script changes needed, explains Jamison. The process would then include updating and tagging any JIRA tickets with the appropriate version number for the forthcoming release.

The code would deploy from svn to the given IDE for QA testing, which developers or engineers would run depending on the size and complexity of the system and the current system state with respect to release to the outside world, says Jamison.

“Then the code would deploy to the integrated UAT environment for testing by the business team. Once approved there, we would schedule a production release with the systems engineers handling deployment to live public facing servers,” says Jamison.

How CI & CD Impacts / Empowers This Development Process

CI & CD processes hand the developers and engineers greater freedom in adding new features, relieving their fears of doing extensive damage to existing software releases. A minor feature change or bug fix could see the steps in the above process happen in close succession as CI and CD processes avoid the risks of having to revert the code. “A major new feature or change to a data model would mean taking these steps on separate days with distinct and formal signoffs on each,” says Jamison.

Final Thoughts

In a word to the wise, Jamison reminds that while tools matter, it is how the teams commit to using them in disciplined CI and CD practices that counts.

August 26, 2015
Will DevOps Kill IT Outsourcing?: Part 2

Part 2: Continuous delivery tool chain efficiency > Outsourcing savings

(Check out Part 1 of the series Will DevOps Kill IT Outsourcing?)

Experts on digital disruption and DevOps practices are vocal in their belief that DevOps and traditional outsourcing models don’t mix well.

“Adopting DevOps requires organizations to erase the barriers between development and operations teams,” says Dan Cornell, CTO of Denim Group. “This is hard enough to do within a single organization and nearly impossible to do across organization boundaries.”

If push came to shove and an organization needed to justify an either-or decision for DevOps over outsourcing, the argument would likely revolve largely around business value. Continuous delivery makes it possible to move to market more quickly, develop products better aligned to customer sentiment and kick the pants out of competitors who don’t do the same. That was the impetus behind the shift at GE Capital. Last year the firm’s CIO Jim Fowler said he was looking to significantly reduce IT outsourcing and shift toward more Agile teams. Earlier this summer Matt Merchant, former global managing director of DevOps for GE Capital gave devopsy.kinsta.cloud a snapshot into how this is driving innovation. (Ed. Note: Merchant suddenly passed on this summer. Our interview was just prior to this tragic event.) He explained how the company was able shift customer-facing mobile app development from a six- to eight-week release cycle to twice weekly.

But the decision to dump outsourcing in favor of supporting DevOps practices within a more centralized IT group doesn’t just have topline revenue and innovation benefits. The upside presents itself in the bottom line as well.

In many ways, we can chalk that up to the fact that the automation and cloud technology that facilitates continuous delivery is light years ahead of where it was when belt-tightening outsourcing deals crescendoed following the 2008 financial crisis. As things stand, the cloud has enabled IT to decouple infrastructure outsourcing from operational and process outsourcing.

“Companies today realize they can achieve many of the cost efficiencies that outsourcing was originally intended to achieve by leveraging cloud technologies,” says Thomas Enochs, vice president of customer success at Chef . “There is a general trend towards leveraging cloud services rather than cheap labor to achieve the scale, speed, and flexibility they desire.”
The fact is that many an outsourcing deal was born in a day when the tools of development were slow to deploy and expensive.

“DevOps practices and modern software tools have drastically reduced the time and cost required to develop custom applications,” says Bernard Sanders, CTO of CloudBolt Software. “In the early 2000s, the step of just getting the servers and infrastructure set up for a new app could be a multi-week or -month chore.”

Virtualization vendors, cloud providers and self-service automation have all made it possible for developers to get easy and affordable access to the infrastructure they need when they need it, Sanders says. And then there’s the advances in deployment pipeline automation that are seriously streamlining hand-offs over the course of the software development lifecycle. It’s all adding up to added efficiency and reduced cost of development and deployment.

“In the future, the overhead required to build custom apps will be driven to near zero, and enterprises will be able to in-source this development, retaining this function that has become key to enterprises’ brand identity, product offerings, and customer experience,” Sanders says.

Even now, though, the DevOps toolchain is completely blowing up a lot of the financial models that initially drove IT departments to aggressively outsource. While many outsourcing deals continue to be renewed without reviewing old assumptions, this is starting to change.

“We have seen cases where outsourced functions like automating test or other processes has been brought back in house because equipped with the right approach and tooling, the internal team could achieve better results, but also because the automation work was a key building-block to a longer-term DevOps plan,” says Alex Henthorn-Iwane of QualiSystems.

August 5, 2015
An Open Letter to DevOps “enthusiast”

Dear DevOps “enthusiast”,

You are confused. But I am confused as well. For approximately two years now this DevOps thing has been both exciting and frustrating. And in ways that are totally different for each of us. At the intersection of all the disparate confusion is the “perfect storm”. What that means for now is that we are all hunkered down in the basement, reading books on “DevOps” to pass the time, and better understand the thing that is attempting to destroy us.

(more…)

July 9, 2015
Illumio Hopes To Cater Security To Continuous Delivery And DevOps Shops

Since coming out of two years of stealth development back in October, the data center security gurus at Illumio have been building momentum in the drive to break out of the confines of the traditional network-centric security model to make it easier to secure IT in the face of distributed computing, continuous delivery and DevOps patterns. This week, the firm got a big boost in the way of $100 million in Series C financing and continued forward movement with some new tweaks to its Adaptive Security Platform (ASP).

Designed with DevOps patterns in mind. Illumio’s ASP is meant to be a continuous delivery system for dynamic policy enforcement within application traffic across everything from bare metal servers to VMs and containers within the data center and public cloud infrastructure. According to Alan Cohen, chief commercial officer for Illumio, the idea is to build security attuned to the infrastructure as code movement in IT.

“Security has been tied to the infrastructure for the last twenty years–not just client-server, but also right back to the mainframe. And as we’ve created software infrastructure–software models for applications and compute processes, whether its storage or network or servers that can be spun up or spun down or moved very rapidly–the security industry hasn’t adapted well because its still very hardware, choke-point centric,” he says. “The industry hasn’t done anything to adapt to distributed compauting and continuous delivery. (When vendors do make adjustments) they’re just running something on software but it operates as it did before.

The announcement this week adds further capabilities for the platform to segment the continuous delivery of that enforcement down to the process level within workloads, so that as any part of the application changes, the platform automatically adapts security policies on all impacted workloads.

“So let’s say you have an application that’s a credit card processing application and it uses a certain database. You may have two instances of that database running on a single host. One is for PCI and one is not,” says Cohen. “Through our virtual enforcement node we can segment and separate those two workloads and effectively provide a layer three layer for firewall at the process level on an individual host.”

The added segmentation of dynamic security enforcement gives DevOps shops a greater degree of flexibility in securing microservices environments.

“When you talk about microservices there’s two things to think about,” says PJ Kirner, chief technology officer and founder of Illumio. “One is that they’re small and process-oriented, and the other one is they exist for a short amount of time. Both those aspects are important to consider for security. The fact that it needs to adapt quickly to all of these ephemeral workloads that are spinning up, spinning down to accomplish your goals.”

April 21, 2015
Chef defines the DevOps workflow with Chef Delivery
At ChefCon today Chef made a major announcement unveiling Chef Delivery a “new DevOps workflow product that for the first time enables the continuous delivery of infrastructure, runtime environments – including containers – and applications.” Chef has examined the use success patterns of many of its thousands of customers to distill what they believe is the formula for success in a DevOps environment.

I had a chance to speak with Alex Ethier, VP of Product and Jay Wampold, VP of Marketing at Chef about Delivery. They were both really jazzed as they think this is a real game changer for not only Chef but the entire DevOps market. Chef is already a leader in the DevOps community with a rapidly growing base of customers in the Global 2000 and a dominant adoption position among AWS customers. Delivery will extend the Chef platform into the entirety of the DevOps workflow.

Chef believes this is the time to expand the Chef platform to the entire DevOps workflow. Just about every analyst firm and survey data indicates that DevOps is on the way to wide spread adoption among IT teams of all sizes. While DevOps is about cultural change, once the organization recognizes the advantages to a DevOps culture, tools that leverage this culture are sure to expand their footprint.

I asked Alex and Jay if Delivery would bring Chef into competition with other DevOps tool vendors. For instance Jenkins and CloudBees in the CD and CI area would seem to be square in the cross hairs of Delivery. While on its surface this would seem to be the case, Delivery is actually a platform that is designed to leverage APIs and open source to to work with a large ecosystem of 3rd parties including Jenkins, CloudBees and other players in the DevOps ecosystem. I am still not 100% sure where the cooperation will end and the competition will begin, but I guess we will have to wait and see how that shakes out.

Based on Chef’s customer experience research they think this is what the flow of DevOps looks like:

DevOps lifecycle step by step

Some steps are and will remain manual. Chef is seeking to play in the automated areas. Yes this is an ambitious move by Chef. If they succeed they will leverage their already dominant position in the DevOps configuration management field into becoming a dominant player in the entire DevOps life cycle.

Powering this life cycle is the fact that with the move to the cloud today’s enterprises are moving from static systems to rapidly changing services. This is putting new pressures on IT departments to perform in a “high velocity IT” manner. The key to all of this is of course automation. Under it all Delivery seeks to automate CI, CD and continuous testing.

Chef’s CTO, Adam Jacob had this to say about Chef Delivery, ““The tools we use reinforce the behavior; the behavior reinforces the tool. Thus, if you want to change your behavior, change your tools.”

Other highlights of Chef Delivery from the Chef press release are:
- Robust Pipeline: Chef Delivery provides a shared pipeline and proven workflow for software development that safely takes code from a developer’s or operations engineer’s workstation through build, test, and production.
- Platform for Collaboration: Each step of the Delivery pipeline incorporates automated testing to provide developers, operations engineers, compliance and security officers, and IT architects rapid feedback on proposed changes. The common workflow allows personnel of each IT discipline to work together with full visibility into the flow of changes through the pipeline. Policies can be easily applied at each step to ensure maximum change control and governance.
- Sophisticated Analytics: Chef Delivery provides metrics for all stages of your development pipeline. With full audit capabilities, you can track both pipeline performance and activity, manage permissions and access comprehensive change history from easy to use dashboards.
- Scalable Architecture: Chef Delivery was designed from the ground up for performance and scalability. Its architecture is fully integrated with the Chef Server and designed to meet the demands of the largest and most complex enterprise IT environments.
- Full Ecosystem Integration: Chef Delivery integrates with an extensive array of operating systems and runtime environments, including public cloud platforms such as Amazon Web Services and Microsoft Azure, as well as runtime environments such as Docker containers.
- Availability: Chef Delivery is currently available through an invitation program
- Pricing: Chef Delivery will be sold in 2015 on a subscription basis.
This could indeed be a real game changer for already high flying Chef. While at ChefCon today I intend to ask many of the partners and vendors exhibiting here how they view Delivery and what kind of integrations we can expect to see from them for it.

One thing for sure is the DevOps tools market is evolving and maturing rapidly. Tool vendors will have to move quickly to allow high performing IT teams to move at the speed of DevOps.
April 1, 2015
Sofico Adds DevOps Approaches, Updates CD, Part Two

We recently geared up with Sofico’s existing CD process and the challenges it presented together with its search for a devops solution. Now read the conclusion as Sofico earns satisfaction through its own devops approaches.

Sofico Adopts DevOps Methods / Tools

Today, Sofico uses the Bamboo CI server to perform builds, deployments, and tests. Bamboo automates builds of the backend server, web interfaces, command line interfaces, and a Windows line. Sofico uses Bamboo to deploy its application builds. Sofico runs tests on software using tools such as Selenium for browser testing and functional tests.

Sofico runs Dynatrace monitoring together with the functional tests to gather performance data. “This gives us correlations between the functional outcomes, the performance outcomes, and the expected outcomes of the testing,” says Jan Swaelens, Software Architect and Member of Solution Management, Sofico.

“We actually created a custom Bamboo plug-in for that together with the people at Dynatrace in order to visualize the outputs of that correlation inside Bamboo,” says Swaelens. Sofico also uses JMeter to launch tests on servers and Dynatrace monitors those as well for performance metrics.

Sofico uses application tools to manage its Agile inner workings and to run reports on that. Sofico uses Stash to manage its code repositories where its developers do feature branching for new features. “Developers use the Dynatrace server to monitor their performance metrics for the features they’re building. It enables proactive monitoring of what they create, even before they commit the change to central testing,” says Swaelens.

So the developer does his regular testing and he also enables the agents that do the monitoring to identify errors. Sofico also does load testing using increasing loads for performance metrics, and performs tests for various use cases.

Results

Using DevOps to expedite builds, tests, and deployments significantly decreases the number of times that something will go into production and need to come back for a fix. “We actually manage to stop software with errors before it gets to our customer. This is very critical because in our sector when, for example, we make a major mistake in billing software and the customer actually has that in their production environment, their customers are impacted,” says Swaelens. This is very expensive for the customer and for Sofico because then they have to fix the issue in as little as a day, which is not always possible.

The bumps in the road that Sofico experienced were in rolling out devops and adjusting to the new way of working. “We found that the impact on systems and processes was even larger than we were expecting. We restructured our old company at every layer from management down to the developers and testers,” says Swaelens.

Sofico redesigned every process so everyone could work in devops teams. All of Sofico’s existing methods and processes were analyzed to determine how the company functioned, why it worked that way, and to make sure the end result was still providing the same or better results.

The changes and improvements that devops made were far-reaching. Rather than making devops come to Sofico, Sofico had to gravitate towards devops, change its business culture, and break down all its silos. “We went all in to get to the next level and do more business globally. Because we’re not just one company, we had to spread devops across all our sub-companies around the world,” says Swaelens.

Sofico can draw a straight line between adopting devops and the speed with which it can now grow globally to increase customer uptake of its offerings. The software quality and timeliness of delivery are where they should be. Sofico’s people are happier, too.

Sofico has relocated its teams together into the same space. Consultants, developers, testers, and operations staff are sitting physically together in the same areas. People are working together and talking with each other to derive the best solutions and to improve efficiency. “We actually have cases of people blossoming because they are seeing whole new parts of the business and working in new ways that we could never have imagined,” says Swaelens. Where Sofico’s people once saw only their piece of the development puzzle, they now see the entire process. They know the full value of their work.

March 17, 2015