devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

‘DevOoops’ Moves: SCM Conflicts
DevOps’ ultimate aim is to create efficiencies in the software delivery process. But it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call a “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. Following is a discussion submitted by solution architect Will Refvem about siloed departments.

Scenario:

One of the biggest challenges in DevOps is SCM. Early in my career, being given access to Git repositories was a bit like handing a belt-fed machine gun to a drunken toddler. My biggest DevOoops moment came when I was trying to master the basics of Git in a project that used Git submodules—don’t, just don’t—and had a series of bash scripts that implemented a primitive CI/CD workflow by running in the environments we were deploying to (seriously). I can’t tell you how many merge conflicts and detached HEADs I encountered, or how I got out of them. Fortunately, we were using the fork-PR model, so I did no real damage.

Software configuration management (SCM) is key not only for versioning, but also for providing a shared view of code and other assets, from development through to production. Giving proper consideration to your SCM structure, branching and merging model, access controls, etc., will support collaboration, reliability and traceability—all key for DevOps success.

In the 2019 DevOps and Jenkins Community Report, 66 percent of organizations respondents reported the use of modern solutions SCMs, specifically GIT-based solutions GitHub or BitBucket. But the older SCM solutions aren’t going away completely—45 percent of all organizations still use pre-GIT, pre-DVCS (distributed version control systems) such as Perforce, Subversion and CVS. While it’s possible to apply DevOps practices using these legacy solutions, it is important (and more challenging) to design a branching/merging model and surrounding workflow, which supports the key aspects of DevOps such as peer review, collaboration and short-lived branches.

Modern SCMs such as GitHub have features that inherently support collaboration, including the game-changing Pull Request, making it much easier to implement processes that best support CD and DevOps.

Poorly designed CD workflows or inadequate CD systems create friction multiple ways. If the interactions are too cumbersome, users will not commit code to the pipeline frequently enough. If it’s difficult to access the pipeline and create visibility into the process, it can lead to unresolved conflicts.

You need to pick the right tool. If you don’t have the option to get the modern tool, you need to take the time to design a legacy tool.

It’s also important to consider two other issues in relation to this DevOoops example: branching and acquiring the appropriate level of DevOps skills.
- Branching: A proper branching and merging model is required to ensure you are feeding the appropriate inputs into your DevOps pipeline, such that you are able to continuously integrate, test and ultimately deploy changes. A simple guide here is encapsulated in the KISS (keep it simple, stupid) principle. If you need a full-wall whiteboard or a spreadsheet fit for an accountant to list your active branches, you need to rethink things. Get as close to the ideal of mainline trunk development as possible, with short-lived feature branches, minimal application versions, etc.
- Skills: As organizations seek to implement DevOps, most likely new tools and practices will be implemented simultaneously. It’s important to understand and accommodate the fact that there will be bumps and bruises as you learn to use these new tools and practices. For instance, shifting from centralized version control such as Apache Subversion (SVN) to distributed version control such as Git is a paradigm shift that can be difficult to fully grasp and result in destructive mistakes, such as deletion of nodes from the change history, overwriting changes or misconfiguration access control. Consider hiring for these new skills and/or account for time and training.
Takeaways
- A proper branching and merging model are required to ensure that you are feeding the appropriate inputs into your DevOps pipeline to let you continuously integrate, test and ultimately deploy changes.
- If you are using a pre-Git legacy SCM solution, develop a branching and merging strategy within the legacy solution to achieve appropriate velocity in your DevOps model.
- Consider that you will need to develop new skills to adopt new SCM tools and practice and account for this in your plan.
— Brian Dawson
March 11, 2019
SauceCon Speaker Profile: Phil Wells

This is the seventh in a series of articles highlighting speakers at SauceCon 2019. SauceCon brings together the global community of Sauce Labs users and continuous testing experts to learn from each other and level up their automated testing and continuous delivery skills. This year’s event will be held April 23-25 in Austin, Texas.

Phil Wells is coming to SauceCon 2019 to talk about the New York Times Crossword.

No, really, Phil is a software engineer with the web group of the New York Times Games team whose job is to build quality into the online version of the famous puzzle done by thousands of people daily.

More than anything, Phil cares about mission. During his 12 years of web development experience, he has worked on software that educates parents of kids with autism, software that helps people perform better on standardized tests and now for the New York Times, whose mission is to help people understand the world around them.

“The puzzle brings so much joy into people’s days,” says Phil. “I’m really proud of my work.”

When Phil tells people what he does for a living, they want to know what it means. Does he just solve crosswords all day? Is he in web development or game development? Does he know Will Shortz? (The answer to that last one is no, but he is an emoji in their internal Slack instance!) People love the crossword, and they all seem to want to know more about how the sausage is made.

Phil’s session at SauceCon will be all about “Automating the New York Times Crossword.” He wants to show test practitioners that testing a web application as complex as the crossword involves the same best practices as testing anything else. During his talk, Phil will show attendees some of the tricks his team used to become fast and reliable.

“The main theme of my talk is about making software development feel better. We don’t want to feel the shame of failed builds or delayed releases,” he said. “So, we built these systems to help us ship the stuff we build quickly and safely.” The first half of his talk covers the things the team did to get cultural wins internally. In the second half, Phil will dive into his particular team’s bag of tricks; the tools they use to make sure the tests are making everyone do their job better.

When he’s not working on testing the crossword, Phil enjoys improv comedy. He is also an aficionado of “The Big Lebowski,” and has written a book called “Dudesong,” an epic poem adaptation of the movie, featuring 5,500 lines of poetry in rhyming couplets!

Whether or not you’re a fan of “The Big Lebowski,” you can hear from Phil and more than 30 other expert speakers at SauceCon 2019, the third-annual Sauce Labs user conference. To see a full list of presentations or to register, visit the SauceCon website.

— Rebecca Cramer

March 7, 2019
SoulCycle Flexes Its Muscles with DevOps and CI/CD
When SoulCycle wanted to improve application development and deployment, including improving services for customers across its 80-plus indoor cycling centers in the United States and Canada, the company turned to Mark Sost, a veteran developer and engineer who worked at several media companies, including Hearst.

Over the last two-plus years, Sost, who now holds the title of senior director of engineering at SoulCycle, has built-up the company’s engineering and DevOps teams, increasing the headcount from eight or nine developers to about 40 employees now.

The first major project was revamping a legacy, LAMP-based stack used to support applications to focus more on a modern platform that could take advantage of DevOps methodologies, as well as continuous integration and continuous delivery practices (CI/CD).

Sost and his team dubbed this new platform Podium. However, Podium is more than a development platform, Sost explained.

“Podium became the ethos and the methodology, as well as the architecture and application stack for re-platforming Soul’s technology stack in total,” Sost said in a telephone interview. “We moved from that monolithic to more of a service-oriented architecture. As such, we started to break out functionality on multiple applications.”

The re-engineering also meant investing in new infrastructure, as well as other cloud-based technologies. This includes a multi-cloud approach.

While most of the legacy applications have now been moved to Amazon Web Services, SoulCycle also uses Google Cloud Platform for Podium and all new developments. Sost also takes advantage of Kubernetes and Docker for container orchestrations, as well as investments in other managed services.

Other tools in the DevOps arsenal include Jenkins to run unit tests; DataDog and StackDriver for monitoring and analytics for workflows; combing Travis CI, Cloud Builder and GCR for CI; and technology from startup Harness, which provides CD-as-a-service. Sost called Harness the “last-mile” tool needed for application deployment.

“For Harness, that’s CD for us,” Sost said. “That’s continuous delivery to our various Podium environments. We’re leveraging it for delivery to all our Podium environments. Everywhere from UAC (user account control) during the code review cycle to QA after merge, to production, load test and everywhere in-between.”

This revamped approach to application development now means that the SoulCycle engineering team has created approximately 30 different apps over the last 24 months, this includes internal apps used within the company environment, and ultimately to customers using the facilities and equipment.

Additionally, Podium helps create a more unified architecture to help ensure that the entire team—whether it’s DevOps, QA or the production team—can paint a 360-degree view of the problem that needs to be solved and what the implications are for the business once the solution is put into place.

This is what Sost and his team call “Podium principles.”

“When we say that we’re building things with Podium principles in mind, we’re talking about the fact that everything is version-controlled,” Sost said. “We have full test coverage. We have everything in terms of Kubernetes manifests to infrastructure automation written down and version control checks it as part of source code. That applications are conforming to more of that 12-factor methodology and that we’re really focused really heavily on developer empowerment.

“I think for us, that really is a shift that Podium has introduced; a move from writing the code as an application developer and then throwing it over a wall to an operations person—a QA person—to test it and deliver it somewhere,” Sost added. “Instead, we provide a developer the tools and the patterns of principles to go about controlling more of their application development life cycle all the way through to delivery.”

The results have helped improve the overall SoulCycle experience for customers, which includes:
- Reducing deployment time by approximately 80 percent.
- Instead of one hour, deployments now take one developer 10 to 15 minutes.
- QA verification has been reduced by approximately 80 percent.
- It now takes three QA engineers one hour to verify, compared to three to six hours previously.
At the same time, Sost believes that the Podium approach gives developers more control over their projects and lets them focus on writing good code that creates apps people want to use.

“It’s more, ‘Now, I know what’s involved with getting my application to people,’” Sost said. “Because at the end of the day, engineers are just concerned with writing things that people want to use. … This is how we’re going to give you more control of the code and applications that you’re writing, so that you can see meaningful results quicker, without having to wait for someone.”

What’s next for Sost and his team at SoulCycle? The answer: Machine learning and a bit of automation to the DevOps process.

Here, SoulCycle is using Harness tools to log more data and infrastructure monitoring into the platform, which allows the algorithm to learn more about the various development projects. This can make the code better and eliminate errors that come with constant development and deployment.

“We’re letting it get smarter over time around how it can detect anomalies and understand whether our code was successful because there are going to be times where things slip through unit testing or slip through integration tests and there are even going to be times where they slip through load tests, where things like 99 percent are just not going to cut it,” Sost said.

— Scott Ferguson
March 1, 2019
Atlassian Adds Pre-Configured Pipelines to Bitbucket CI/CD

One of the primary reasons DevOps processes have not been widely adopted across the enterprise is that building pipelines across all the tools required is a painstaking effort. As a result, most pipelines today are constructed using manual pipelines. However, Atlassian has announced it is now making available pre-configured Bitbucket Pipelines that DevOps teams can access via a simple drag-and-drop interface.

Harpreet Singh, head of product for Bitbucket Cloud at Atlassian, said DevOps teams that rely on the Atlassian Bitbucket continuous integration/continuous deployment (CI/CD) platform will be able to employ a series of pre-configured Bitbucket Pipelines built by Atlassian or one of its third-party partners, rather than having to manually build every pipeline.

The goal is to make it much easier for DevOps teams to incorporate, for example, a new cloud platform into an existing set of DevOps processes without waiting for someone on the DevOps team to write, test and validate a script, said Singh. IT vendors that have worked with Atlassian to create Bitbucket Pipelines include Google Cloud, Microsoft, Slack and Amazon Web Services (AWS).

One of the ironies of DevOps is that while CI/CD platforms make it easier to build and deploy applications, constructing the CI/CD platform can be exceedingly complex. A recent survey conducted by Atlassian found DevOps teams rely on 4.3 tools to move code from development into production environments. A full 62 percent of respondents reported they resort to performing manual testing due to insufficient automated test coverage, additional manual processes or a lack of build/deployment pipeline automation.

While some organizations might have the engineering resources required to build pipelines, he noted, the average IT organization can’t afford to hire operations specialists that have the requisite programming skills required to build pipelines manually.

Best of all, noted Singh, the pre-configured Bitbucket Pipelines that Atlassian makes available will be maintained and updated by their original authors.

It’s too early to say to what impact pre-configured pipelines is likely to have on DevOps adoption in the enterprise. But most enterprise IT organizations won’t significantly alter their process until it becomes relatively easy to make that transition. Enterprise IT organizations have relied on administrators to manage IT for decades, and most of those administrators wouldn’t know where to begin if asked to manually construct a pipeline. The few enterprise developers who might be able to build those pipelines are simply to busy writing application code. The result is a paralysis that prevents organizations from modernizing their application development and deployment processes.

Of course, any meaningful change in the enterprise also involves a chicken-and-egg debate. Lacking skills and tools, enterprise IT organizations are unable to overcome internal inertia, which in turn results in conservative IT organizations preferring to rely on known processes no matter how inefficient they might be. The real challenge facing IT leaders trying to advance DevOps adoption is to remove as many excuses for resisting change whenever and wherever possible, starting with how pipelines are constructed and maintained.

— Mike Vizard

February 28, 2019
‘DevOoops’ Moves: Poorly Defined KPIs
DevOps’ ultimate aim is to create efficiencies in the software delivery process. But, it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call a “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. The following is a discussion submitted by product marketer Juni Mukherjee about poorly defined key performance indicators (KPIs).

Scenario:

How we define KPIs can change the game. For example, one quality engineering team defined the number of tests executed per sprint as a success metric. Humans are driven by incentives and the natural tendency of the team was to add more and more tests without even considering archival of outdated ones. Think of this: We could have a larger impact with fewer tests and, more importantly, fewer tests lower the test cycle time. So, instead of sheer quantity, we should focus on the coverage and effectiveness of tests. Similarly, a release engineering team defined the number of releases per sprint as a success metric. The number of releases reflects on velocity for sure. However, releases move bits from point A to point B without assessing the value added to business. It is critical to tie a business KPI (such as the number of new customers acquired, percentage of revenue increase, etc.) to velocity, such that we know we are investing in valuable speed and not suicidal speed.

Management consultant Peter Drucker boiled his whole philosophy down to the following few words: “If you can’t measure it, you can’t improve it.”

Successful DevOps leaders take this saying to heart. To understand where they’re going, how to get there and ultimately how far they’ve come, these leaders know how critical it is to establish KPIs that fit the overall objectives of their DevOps transformations. KPIs provide appropriate and continuous insight—and they plot a route to help you get to your destination.

The big question is: Are you measuring the right things? If not, you could be spinning their wheels.

To avoid this “DevOoops” scenario, you need to start by establishing KPIs that not only align to your objectives but can also be established so that they create a continuous feedback loop. As the project progresses, as you mature and learn more, you will likely need to maintain, monitor and possibly adjust and update these KPIs.

What would be an appropriate test coverage goal? Teams all want to improve quality. But by how much? Seventy percent? Eighty percent? One hundred percent? This is an example of a KPI that needs to be considered in the context of your application as well as measured against your real objective.

You can have 70 percent test or code coverage, but if that’s just treated as a measure to test as much code as possible, you may miss the mark. If you are covering 70 percent of the wrong code and the 30 percent that’s missing is critical code, you’ve put in significant effort with the risk of little or no return on quality level.

In terms of your goals, increasing and monitoring test coverage may be an indicator of quality, but it may not measure your actual quality of work. In this case you want a KPI that’s measuring the outcome rather than the activity. Look at what is severity 1 and severity 2 defect rate in production. You can see code coverage get to 90 percent, but if you’re not seeing a decrease in severity 2 defects, you won’t really be seeing an improvement in our code coverage metric.

Takeaways:
- You’re not implementing DevOps just to say you’re doing it; you need to develop goals and objectives for your initiative and align to them.
- Once goals are defined, you need to align KPIs and success metrics to those objectives. Properly defining these success metrics and KPIs will dictate what gets prioritized when building pipelines.
— Brian Dawson
February 22, 2019
Deploying into Azure App Service using Azure DevOps CI/CD Pipeline
This article demonstrates how to deploy an application into Azure App Service using Azure DevOps (VSTS) continuous integration/continuous delivery (CI/CD) pipeline.

In this article, you’ll create a sample environment to:
- New build based on code commit to your Azure DevOps Repo.
- Automatically deploy your app into Azure.
Prerequisites:
- Azure Portal Account, Azure subscription and Azure DevOps (VSTS) account.
- Azure Resource Group.
- Create a web app in Azure. Make note of the app service URL—you need it later.
What is Azure App Service?

Azure App Service enables you to build and host web apps, mobile backends and RESTful APIs in the programming language of your choice without managing infrastructure. It offers auto-scaling and high availability, supports both Windows and Linux, and enables automated deployments from GitHub, Azure DevOps or any Git repo.

Creating Resource Group in Azure

Once you have a subscription in Azure, you need to create a resource group to manage resources in Azure. The best practice is to maintain a separate resource group for each environment (DEV, INT, QA, STAG, PERF, PROD) and follow the proper naming conventions. To create a resource group, select Resource Group from left-side blade, click on +Add, then select Subscription, give a proper name to the resource group and click on Review + Create.

Creating App Service

Click on All resources from the left-side blade of the panel and click on Add, then select Web App and give a proper name to your app service. Then select Subscription and select the existing resource group. Keep the remaining option as default. and click on Create.

Once you have created the app service, click on that app service and go to Overview. There you can identify all the details that belong to this app service, including status, resource group, subscription and URL (please make a note of this; you’ll need it later).

Creating Service Principal

When an application needs access to deploy or configure resources through ARM or VSTS in Azure, you’ll need to create a service principal, which is a credential for your application.

Go to Azure Active Directory -> App registrations -> New application registration, then give a name to the service principle, select application type and give it a URL of your choice. After creating the app registration, go to Settings from that service and make note of application ID, then go to Keys create a key and copy the secret value. (Note: It doesn’t appear later, so please store it in safe place.)

Create a New Project in VSTS

The first step is to create an account in Azure DevOps (VSTS), then follow the steps below to start an application deployment. Here I’m taking sample open source java-based code from GitHub repository and importing it to Azure DevOps repo. (You can push your own source code into repo.) It is a simple multi-module Maven project. The application is a very simple online version of Conway’s “Game of Life.”

To create a new project, click on Create new project, give a proper name to your project and select create. Then go to Repos, click on Import, then select source type, type in the above URL in the Clone URL tab and select Import. The source code will import into your repository.

We need give service connections to our project, so go to project settings from the bottom of blade in our project home directory and then select Service Connections -> New Service connection -> Azure Resource Manager. Click on “Use the full version of the service connection dialog” from the pop-up, then paste the Application ID into the Service principal client ID tab and the secret key value into the Service principal key tab (which you saved during app registration creation). Then click OK.

Creating a Continuous Integration (CI) Build

Go to Pipelines -> Build -> New Build Pipeline, then click on “Use the visual designer.” It will ask source code repo details, so select Azure Repos Git and select Team project, which you created at the start of the project, select Repository and branch name, then click Continue.

Next, select a template Maven, type in the pipeline name and select Hosted VS2017 for the Agent pool, and click on pom.xml in the Maven POM file. There you can list the goal(s) as Clean Package and select Copy Files. At the contents section please enter **/*.war then go to Publish Artifact: drop, list the artifact name as Gameoflife and keep all remaining options as default values. Then, under Triggers, select Enable continuous integration and give branch filters listed at right, which enables the continuous integration feature. Once a developer commits their changes into master, the CI build will trigger automatically.

Once the configuration is complete, click on Save and Queue from top of pipeline to trigger a CI build. (Every commit into master will trigger a new build.)

Creating Continuous Delivery (CD) Pipeline

Go to Pipelines -> Releases -> New Release Pipeline, then select the template Azure App Service deployment, list the stage name as DEV and click on Add an artifact. Select project and source (build pipeline), then select default version as Latest, click on Add. Cick on Task (below the DEV), select Run on agent and select agent pool as Hosted VS2017, then click on the plus(+) symbol, search for template File Rename and select the .war file from source file and give it the new name ROOT.war.

In the Display name box, list Azure App Service Deploy. In the Azure subscription field, you need to select a service principal name to authorize the resources. Then choose the App type and App Service name. In the Package or folder selection, select war file and rename it ROOT.war, because for java-based applications we need to deploy our app into a proper directory structure.

Finally, select the template Azure App service manager and select the Restart option, making sure to select the same subscription and app service name you selected in the Azure app service deploy window. Click on Save, then click on Release -> Create a release -> select artifact build number -> create. Build will then trigger and deploy into the app service.

Now, your code is successfully deployed into Azure App Service.

You can access using app service URL from any browser (e.g., https://xyz-abc-webjob-01.azurewebsites.net) and you can check whether code is deployed successfully or not using below URL: https://xyz-abc-webjob-01.scm.azurewebsites.net.

— Srinivas Kudipudi
February 19, 2019
Death of the Release: The Move to Continuous Delivery and Experimentation

As we move from continuous delivery to continuous experimentation, we also must develop new practices

As software development processes have evolved, we have mostly said goodbye to the idea of defined product versions. Many modern product delivery teams are taking this a step further—even the concept of a “product release” is starting to fade. Instead, products are becoming a fluid, rapidly evolving set of features, assembled uniquely for any given user.

Twenty-five years ago, the new release of a software product was a notable event. The marketing department would even insist on a catchy version name for the release. Internet Explorer 6, Office 2000, Windows XP, OSX Mountain Lion. This is still the case for a few types of software—operating systems being the most notable—but for most of the software products we use daily the concept of a version has faded away. What version of Google Maps are you using these days? How about Facebook? Twitter? As far as the end user is concerned, there is no version, just “current.”

A few different forces have moved software products away from release versions. The rise of agile software methodologies and associated practices such as continuous delivery has allowed software teams to tighten up software delivery cycles dramatically. Products that once were released every few months are now updated every few days.

In parallel, the growing capabilities of the web as a platform has moved many products off of the desktop and into the browser. This has greatly reduced the friction involved in releasing a new version of a product. Pushing a code change to servers you manage is a lot easier than persuading your users to download and install an update.

Finally, there has been an architectural shift away from large monolithic systems toward service-oriented architectures and, in recent years, microservices. These modern architectures consist of many small, independently deployable units. This again reduces the friction around deploying a code change by reducing the scope of the deployment to one specific service.

These forces combined have led to software release cycles shortening dramatically, from multiple months or years between each release to a rate of weekly, then daily, then hourly software delivery for some teams. Delivery of a new software release has moved from a marketing event with fanfare to something that is happening almost continuously, even for large products. As far back as 2011, engineers at Amazon were releasing changes to production every 11.6 seconds.

As teams move from a monolith to microservices there is no longer one deployable unit, but rather an ecosystem of independently deployable services. It no longer makes sense to ask which version of a release is running in an environment. Instead, we must ask which version is running for each individual service.

And then there’s feature flagging. One can think of a feature-flagged service as existing in a sort of quantum superposition of potential code paths, resolving to a specific set separately for each request. This means that a service’s behavior in an environment can change radically without any code deployment and without any observable change to the release version.

Now, as we move from continuous delivery to continuous experimentation, we are reaching the point where it is no longer useful to attempt to describe the current running software in terms of a “release,” or even a set of releases. The situation has become a lot more fluid.

Coping in a Post-Release World

Modern approaches like microservices and continuous experimentation are a boon to product delivery teams. They allow us to move faster with more safety, thanks to the tight feedback loops they enable. But this new world also brings challenges.

It can be a lot more difficult to diagnose a bug when you are not sure what code was involved. The load on testers becomes greater as they need to consider the interaction between multiple releases of different services—each with different feature flags which can behave differently from request to request.

From a user’s perspective, a product’s rapid pace of change can be a little overwhelming. New features arriving frequently is nice, but having the UI change every time you log in can be disorienting. Even worse is when a feature is available one day and then gone the next. Users can also be wary of the idea that they are not getting the same experience as other people, particularly if there is any whiff of unfair business practices.

Ironically, while this accelerated feature delivery is partly enabled by continuous experimentation practices, that same speed of change can itself interfere with feature experiments. It can be difficult to statistically correlate an uptick in a KPI with a feature experiment when there are other changes happening over the course of the experiment.

Enhancing the observability of your software—the quality of logging and other instrumentation—is one key way to mitigate some of these challenges. Another is keeping the number of active feature flags reasonably low, which can help on a variety of fronts. It reduces the testing burden of multiple interacting flags, as well as the risk of experiments interfering with one another. Fewer active flags also mean fewer things changing at once for your users.

Managing all feature flags in a shared system can help in understanding the number of active flags and avoiding interfering experiments. Centralized flag management also means you can batch up a set of feature changes into a single release, which helps avoid overwhelming your users with many small changes. A centralized system makes it easier for testers to see which flags they should expect to be in play within different systems, and how those flags might interact.

We’re entering an era that will involve a massive explosion in the rate at which we release new experiences to our users. Embracing this next stage in the evolution of product delivery can enable teams to move faster and with greater safety. But entering this brave new world requires us to develop new practices if we are to avoid the pitfalls of a post-release world.

— Pete Hodgson

February 11, 2019
How to Automate HIPAA Compliance with DevOps

DevOps can help make HIPAA compliance more achievable

Automating the provisioning of HIPAA-compliant server infrastructure enables compliant hosting service providers with the ability to provision and deploy infrastructure-as-code with minimal human intervention. Providing the automated process is thoroughly tested, revised and updated, it can offer a healthcare organization huge flexibility when enforcing a HIPAA-compliant server or serverless infrastructure within a private, hybrid or public cloud offering.

Organizations that practice centralized operations, also known as DevOps, have a unique team who have the desirable skills of software development, IT operations and quality assurance (QA). The DevOps objective is to create a performance-based IT solution within a greatly reduced provisioning lead time using technical server blueprints within a continuous delivery environment.

DevOps within a healthcare organization must be able to meet the stringent administrative, physical and technical safeguards required by HIPAA. In all cases, the required safeguards must be met, but with DevOps, it can be significantly easier to meet the addressable requirements of HIPAA.

Planning

Each healthcare organization can present differing business needs which they need a HIPAA-compliant service provider to resolve. It is not always possible to provide a cookie-cutter solution for every healthcare organization, often one design does not fit all; however, it is possible to create a baseline of operations using automation.

A great deal of planning must be completed with all relevant stakeholders to not only understand the technical solution but to gather critical information about unique aspects of the architecture and then plan ways to automate the delivery process.

It may not be possible to automate every workstream, but getting the DevOps team involved in the early planning stage will allow thorough and accurate playbooks to be created.

With advanced planning, significant gains can be made in defining the networking security layout for the healthcare clients virtual private cloud, work out what isolated cloud resources can be defined and make decisions about potentially leveraging network layer protection services such as DDOS protection or a software-defined web application firewall.

Provisioning

The next step to creating the infrastructure-as-code is to write the automation playbooks that will provide the infrastructure. There are several provisioning applications that can be used, such as Terraform, Chef, Puppet, Google Deployment manager or AWS Cloud Formation, and playbooks will need to be written (or converted to) a compatible language such a python, YAML or JSON.

The machine-readable code interacts with the service provider’s API front end; this may be an on-premises VMware vSphere cluster or a public cloud provider such as AWS, GCP or Azure. The API interprets the code and uses the descriptor commands to provision and deploy infrastructure in a cloud environment.

A DevOps engineer will select the cloud provider they using and then define all the environmental variables to use to create the infrastructure-as-code. Connection variables are usually configured first and will define the cloud provider locations, regions and availability zones, as well as the encrypted account passwords or IAM configuration to interact with the provider API.

Once the connection variables have been initiated with the provisioning software of your choice, they are used throughout the project life cycle. The next stage is to define the HIPAA infrastructure environment, including storage, network and compute resources. This will include provisioning resources and setting private and public networking.

At all stages of the design it important to define where ePHI data will be stored and processed. The process must include step-by-step documentation and a customer responsibility matrix. At this stage, it is imperative to incorporate technical safeguards such as server access control lists (groups), introduce activity logging and provide auditable controls.

Server hardening policies must be created for servers, active directory, workstations or providing least privilege authorization accounts at multiple levels, such as user profiles, that enable or restrict access to individuals or third-party systems. Preapproved software is typically installed at this step as well.

Next, the inclusion of boot volume encryption and encryption of disk objects and storage volumes must be defined, this is easily achievable on mass provisioning of servers, simply providing a volume ID and an encrypted $TRUE statement.

Throughout the playbook designing phase, testing is conducted to close loopholes, refine automated network configuration and create shared VPN tunnels with passwords that automatically cycle. The playbooks set the properties which make an organization HIPAA-compliant, and can also be used to create secured bastion servers if VPN is not an option.

Playbooks allow DevOps teams to create rotating security keys, SSL certs and log aggregation and filtering. This can also be configured to trigger automated messages that can be emailed in case of any errors or failures in the automation process.

Continuous Delivery

When all the playbooks are streamlined and accurate, they can be used to define a healthcare clients service catalog, which enables the use of standardized templates to be used within the HIPAA environment. This may include server templates, container configuration, storage templates or any predefined software application.

Security tools can be used to track and inventory resource usage and any system changes, track all user activity and any API activity requests.

Monitoring tools can be used such as Cloud Watch, Nagios or Nimsoft to monitor server resources and application usage, these monitoring tools can also pick up errors or predictive errors and alert against predefined thresholds.

Many organizations choose to use serverless workloads such as Docker or Kubernetes. These massively scalable solutions also bring about a huge benefit that aids the DevOps continuous delivery approach of automated update rollouts—essentially, applications can be isolated over differing pods, allowing the underlying hardware to be updated or versioned with zero downtime.

DevOps is changing the automated delivery of HIPAA-compliant solutions. It is important to remember that DevOps does not have all the answers for HIPAA, especially many of the administrative regulation. However, it does empower managed service providers with the ability to provide tried and tested automated solutions that meet or exceed the technical and physical safeguards of HIPAA compliance.

— Moazzam Adnan Raja

January 11, 2019
Always-on Development: Why Continuous Delivery Relies on Security by Design

How continuous delivery can help organizations in their application security efforts

The old model of developing secure applications followed a structured, siloed, step-by-step process—build, check, release. Organizations’ desire for speed has forced development teams to ramp up their release cycles by any means necessary, even if it means bypassing incremental scanning for flaws that could increase application risk. This philosophy of rapid innovation also comes with increased risk exposure.

Security and development teams need to find ways to keep up with the rapid development life cycle without sacrificing security with every update. The continuous delivery model has transformed application development, but to last, it must also transform application security. It may take a little extra time and effort, but it is possible to develop efficiently and securely.

Continuous Communication

The key is to make sure that security and developer teams understand each other’s priorities and needs. As DevOps began to catch on, the immediate reaction for many organizations was to make their developer team into a jack-of-all-trades unit. This began by adding operationalization to the developer job description and has only grown from there. Soon after, security, testing and monitoring functions joined the list as well. Recruiting teams are now struggling to find people who are qualified for this multifaceted role.

Teams operating on a continuous delivery model need to find a middle ground between the developer solely focused on coding versus owning every step in the development process. Developers should be at the center of everything—keeping the priorities of each role in mind and making sure each is involved in the final product, but focusing on their main job of development.

This philosophy goes both ways. Security teams and other specialties can always learn more about the intricacies of the development process, which will enable them to better integrate their functions into a continuous delivery cycle without disrupting it.

Continuous Improvement

One way to encourage a unified team, specifically in regards to security, is to appoint a security champion for each development team. These advocates should be developers who have a special interest in or affinity for security. Their role is not necessarily to know how to solve every application security problem that arises in the development process, but to ask the right questions about security and interface with security teams.

Roles such as this create firms links between the different functions of DevSecOps practices, making it easier to realize opportunities for improvement under the new continuous release model.

One small change that can make a big impact is realigning review processes for continuous delivery by reviewing only the pieces that have changed instead of the whole application. Under traditional, linear development cycles, each new release requires a full assessment even if much of the code remains unchanged from previous versions. Applying the continuous delivery mentality to security testing can streamline and reduce the burden on security teams. Each new piece of code deployed in continuous release models is faster to scan, which makes flaws easier to find and fix.

The assembly line model of application development is no longer feasible. If security teams wait for the application to be complete before fixing vulnerabilities, they will never be able to keep up with the pace innovation and competition demands. Making the most of continuous delivery requires constant communication and collaboration across the full DevSecOps spectrum, along with a shift in thinking that allows developer and security teams to use the process to their advantage.

— Mark Curphey

January 9, 2019
Nullable Types

— ROELBOB

January 9, 2019