Category: Enterprise DevOps

Mainframe Computing and the API Economy

The API economy has come a long way since the days when I worked at a very well-known automotive website back in 2007. Back then, APIs were the thing that application programmers used at a very low level. Programming using web-based APIs was a rarity. In those days, we downloaded libraries into our local codebase and programmed against those binaries. I was programming in Java at the time, so the general practice was to use an application management framework such as Maven to get .jar files from the internet and incorporate those .jars into my code.

Yes, there were technologies such as RPC that allowed us to access functions running on other machines directly and there were shops that used the technique extensively, but in my little corner of the world we accessed services directly through in-memory binaries. Also, we—the automotive website—published our data as HTML in web pages.

Then something happened around 2010. It turns out that other developers outside the company wanted to use our automotive data. In fact, a lot of developers wanted that data. My employer saw the writing on the wall. Within a year, we were publishing an API that provided all the information we had on just about every car sold in the United States. We were in the API Economy big time!

It was a transformational undertaking. It was also incredibly taxing. The company had to alter a lot of its development processes to meet the demands API publishing. Not only did we need to support publishing the data under XML and JSON, we also had to put in a security layer to make sure only those developers who were entitled to use the API had access. And, we had to provide comprehensive documentation that provided the support necessary so that a developer could use the API to full efficiency. The work beneath the surface of the API to keep data flowing and current was even more daunting. And, this was only for V1. We had yet to develop any sort of road map that described future versions.

Was it hard work? Yes. Was it worth it? Yes. We were at the forefront of the API Economy. Moving our data out of HTML and into machine-readable formats such as XML and JSON provided a lot of commercial value, to the company and to other developers. It was a game-changer then. It’s game-changing now.

The good news is that technologies have emerged that make API publishing easy compared to those early days. Today, most of the major players in the technology landscape realize that to stay commercially viable, they need to represent their services as robust APIs in a way that is easy to publish and easy to consume at web-scale. This is particularly true in the realm of mainframe computing.

IBM Z Systems: A Significant Player in the API Economy

There’s a lot of mainframe technology in force in the world today. A very large number of banks, financial services, governments and large-scale transportation entities have major mainframe installations. Many of these installations are IBM Z Systems. IBM is major force in this landscape—always was, always will be. With a workforce of more than 300,000 people, how can it be otherwise?

One of the things that IBM has been very good at doing is devising technologies that allow services that reside in their Z System installations to be exposed as APIs. The way they do this is using a technology called z/OS Connect EE. z/OS Connect allows developers to expose Z System applications as APIs accessed via HTTP (See Figure 1.)

Figure 1: z/OS Connect EE serves as the intermediary between an HTTP based API and a IBM Z mainframe

Developers create an API using the z/OS Connect EE API Editor. The API Editor is an Eclipse-based IDE that allows developers to define the endpoint URLs for an API. (See Figure 2, below.)

Figure 2: The z/Connect EE API Editor allows developer to define the HTTP methods and associated endpoints to that make up an API

Then, once the endpoints are defined and HTTP methods (e.g., GET, PUT, POST and DELETE) are bound to services that reside in the Z System installation. (See Figure 3.). Each service maps to an application, such as a transaction in CICS, IMS, DB2, etc.

Figure 3: z/OS Connect EE provides a way to directly bind an HTTP request to mainframe resources

Not only does the z/Connect allow developers to represent services in a Z System as fully functional APIs, but it also has a feature that exports API definition as fully compliant Open API specifications.

This is a big deal.

Open API is one of the three most prominent API specification formats in use today. (The other two are RAML and API Blueprint.) A developer can use the Swagger specification that z/OS Connect generates to create interactive documentation for use by third-party developers. (See, Figure 4.)

Figure 4: z/OS Connect EE has the ability to generate an API specification in Open API format that can be used to create interactive, online documentation automatically

Again, this is a very big deal. One of the key factors for publishing a successful API is to provide comprehensive documentation. An API that is poorly documented rarely becomes a hit in the API Economy. The more comprehensive an API’s documentation is, the more widely used it will be.

z/OS Connector EE + Eclipse = A marriage made in heaven

I am a big supporter of using tools to reduce the labor of working with complex technologies, particularly with something as broad and multi-faceted as mainframe computing. The z/OS Connect EE API Editor fits the bill. It’s embedded within the Eclipse IDE. If you know how to use Eclipse to make a Java program, you’ll find using z/OS Connect EE API Editor to create Z System API to be an inviting experience. Yes, you might have to spend some time get up to speed on mainframe technologies such as CICS and COBOL. But, the good news is you won’t have to learn a new programming environment to do it. Just think of the API Editor as Eclipse wrapped around z/OS Connect EE and the IBM Z System computing environment.

This 10 minute video does a really good job demonstrating how to use the Eclipse based z/OS Connect EE API Editor to create a full blown REST API using services provided by a IBM Z System mainframe. It’s pretty amazing.

Once an API is published from a z/OS system, it can be used internally or exposed for public consumption. Representing Z System services as an API – is a significant step toward making mainframe technology a first-class contributor to the API Economy. In the API Economy, consumers care little for the underlying technology. What matters is the service. It’s striking in a way: What may look like a cloud-based API on the outside is really an IBM Z System underneath the covers.

The API Economy Moving Forward

The API Economy has come a long way since those first days almost 10 years ago when we wrote our first GET /make/mode/year request at the automotive website. The days of hard-coding routes and manually creating documentation have come to an end. Good riddance. To be honest, that type of development was not only a waste of labor, it also was questionable programming.

Today, modern tools such as z/OS Connect EE API Editor allow developers to focus on the challenges of designing APIs that make a difference. The focus on design is critical, particularly as we move into the era of the internet of things (IoT). In the not too distant future, most API activity will be conducted between machines—small devices, many pushing the boundaries of miniaturization, talking among themselves and to other larger systems. The volume of data these devices will create will be well beyond today’s norms. Interestingly enough, mainframe technology is geared to handle this extreme volume of data.

Mainframe technology will play a major role in the IoT ecosystem and the API Economy. If I were a gambling man and I wanted to position myself to be in the right place at the right time as the API Economy evolves, I would place a significant bet on getting involved in Z System development now. Technologies such as z/OS Connect EE and its API Editor open the doors to many lucrative opportunities. It’s up to us to make the decision to walk through them.

— Bob Reselman

March 6, 2018

Top 5 Considerations for DevOps Success in 2018

Top tips for maximizing DevOps effectiveness

Whether you are well into your DevOps journey or still thinking about getting started, there’s always room for improvement and an opportunity to learn from each other. Perhaps at the top of your to-do list this year is to implement DevOps as part of your product strategy. You can picture it in your mind: Increased software velocity fueled by frequent releases, better team collaboration and visibility, improved testing procedures and more robust and reliable products.

But picturing something in your mind and making it a reality are two very different things. Which is why I’m offering what I consider to be the Top 5 Considerations for DevOps Success in 2018. Keeping these five items in mind as you make DevOps part of your organization will help increase your chances of having a successful implementation.

However, before diving right into the top five, let’s take a minute to define what success looks like in this landscape. DevOps means releasing software every second of every day, right? No! While that may be a reality at companies such as Amazon and Netflix, it’s a common misconception that to be successful in DevOps you have to fire off releases constantly. In reality, what works fantastically for one company (say, Netflix) may not work at all for your organization. DevOps is a personal journey and realizing success is about being flexible and understanding your organizational requirements.

With all that said, across organizations and industries of all kinds, one constant about DevOps that remains is that it should focus on making continuous improvement through changes, both big and small. In doing so, you will create the ability to improve the flow of work through your value stream to change how you deliver value to your customers. The only way to realize that improvement in value is through change, and if DevOps is designed to do one thing, it’s to make change easier.

Top 5 Considerations for DevOps Success

With a loose definition of what success looks like in this context, let’s take a look at the top five considerations for DevOps success to drive your business forward in 2018.

Make the Work Visible

The goal of every company in the software development space is to deliver value to your users. It’s often incremental improvements over time, but ultimately, your focus is to deliver a better product and a better user experience. The challenge is that it is not always (or even rarely) straightforward, and that’s because the work is not visible. When work is not visible—which means progress is not visible—not only can the work take longer, but also there may be misalignment within the company. This can lead to more mistakes along the way, as issues don’t get properly addressed in a timely fashion.

Part of this issue of visibility stems from the fact that many companies don’t use tools or methods to visualize their entire value stream or the value flowing through it. While some companies use physical means to address this problem, there are modern platforms that are able to aggregate data from all the different sources, provide a digital representation of the value stream and help identify any value that is falling through it.

Using a Common Currency for Value Across the Enterprise

There are several parts of the software delivery life cycle teams might use to measure value for development—releases, versions, packages, binary artifacts, etc. When an organization is fundamentally measuring teams using different value currencies, it becomes very difficult to accurately understand whether it is successful. What it needs to do is find a common currency that can span the organization and give stakeholders and developers a real apples-to-apples way of comparing and looking at value-add. In DevOps implementations this common currency is work items. Then, when code commits are actually occurring, those code commits can be affiliated to those stories and work items, which is basically the business value. In that way, the business can use a common currency to track and measure progress.

Measurement

Over the past couple of years, we’ve seen many companies buying tools and changing processes and spending a lot of money in their DevOps initiatives. While these tools are great and can provide incredible insight into your workflow, it is necessary to dig deeper into the measurement to get a bigger picture of the entire pipeline.

So, while measurement is important, what will really drive increased value for an organization is to measure the entire flow of value across the full value stream, not only in one phase or in one area. This means measuring from the very beginning—when an idea is created—all the way until end-of-life.

Along this same line, it is absolutely imperative for a company to establish real baselines for its data. Even if the numbers aren’t pretty, the company must have truthful baselines not only to measure against future performance increases, but also to help identify bottlenecks and other areas of weakness that are opportunities to improve.

Configuration as Code

The goal here is to leverage version control in such a way that a company is creating ultimate efficiency in its build cycle. Versioning everything touched in the software development process will allow for speedier deployments. When there is an error, the team can roll back without much effort. Fundamentally, a company must establish the practice such that every code push becomes a snapshot of the configuration of the entire stack.

Doing this effectively lowers computing code and it becomes trivial to stand up entire environments for the sake of a single user story or a single feature. Having configuration as code will allow a company to perform rollbacks much easier and will enable faster and more frequent deployments.

Compliance

Compliance used to take place after the software was done. When it had gone through development and testing, and prior to going into production, it was passed to a compliance team or officer who was responsible for making sure everything checked out. Not only does this slow things down, it creates a single point of failure and puts the organization at greater risk for not realizing ever-increasing and changing compliance requirements.

In modern practices, teams are embedding the compliance aspect of software development into their value stream and that allows them to avoid rework and waste. Compliance activities should occur in parallel to the development and testing work, rather than when the release is marked done or code complete. Modern DevOps platforms allow builds of all the activities of the software development process into the different phases. The goal here is to have confidence that every software release that comes out of a company’s value stream will be fully compliant and can be deployed into production at any time.

Conclusion

Whether you are already knee-deep in DevOps or are just starting to consider it as a strategy to drive your business results forward and improve your software development, keeping these five considerations in mind will help you stay focused, generate better results and realize success in 2018. If your ultimate goal is to improve value for your customers through incremental and on-going change, it’s a great time to be part of the DevOps culture.

Want more? Check out this webinar for a more in-depth look into what drives DevOps success.

— Eric Robertson

March 5, 2018
DevSecOps: If You Build It, They Will Come
Spring training for Major League Baseball in the United States has begun. Millions of people share my love for baseball; however, the same can’t be said for security and compliance—well, at least not yet. Perhaps one day.

Much like in the immortal baseball movie, “Field of Dreams,” if you build a friendly security and compliance system, the developers and operators will come. At least, that is the contention of Julie Tsai (@446688), senior director of security operations at Box.

I first met Julie at the RSA conference’s DevSecOps Days three years ago and have been following her musings since. I recently watched Julie’s presentation, “Build It and They Will Come-pliant: DevSecOps in the Real World,” from All Day DevOps. Here is a summary of what I learned:

Good Architecture Sustains Sound Applications and Security

Julie’s contention is that you can build a system that might actually bring a little joy to developers and operators, and it starts with realizing that, at the end of the day, we are all looking for good architecture. Good architecture sustains sound applications and security. It makes everyone’s life easier, so we all have a little time for baseball (or football or board games or even curling).

The good news is that DevOps organizations are ahead of the game. Julie pointed out the old joke, “DevOps … isn’t that just where you give the keys to the developers?” Well, no, and it isn’t a specific tool or deployment. For security, it is about being lean on requirements, building compliance in from the start and integrating it across the development life cycle so you can build something that is secure and performant. Besides these being best practices for developers, they also allow you to use your code and policies to appease your auditors, as compliance is built-in. Bottom line: DevOps puts the rigor around security and compliance.

It Takes a DevSecOps Village

Julie pointed out a couple of takeaways from DevOps that help security and compliance:
1. You can own a problem and be an individual leader. You can think globally but act locally by seeing across the silos and bringing a message of empowerment.
2. It provides a path toward integration to internalize other groups’ values, bring them into your own words and ways and mutually reinforce and thrive. It also informs how people work together.
Remove Friction to Scale DevSecOps

The goal here is to ultimately get us something easier to scale and maintain and be compliant and secure. Julie outlines steps to get you closer to this idyllic system:
1. Use configuration management because it leads to precision and more unified and verifiable work.
2. Get rid of whatever you can to eliminate mistake factors. You will also get to a place to streamline workflow so there are fewer mistakes.
3. Extend infrastructure as code. It can become configuration as policy and your audit trail can connect the intent to the execution.
DevSecOps: Where Things Go Wrong

With these in place, Julie contends you need to realize the world isn’t perfect, so you need to build a system that injects security and compliance where it is most effective. To find the areas in the development life cycle that it is important to inject, ask these questions:
1. Where is someone intending to make a change?
2. When you are releasing it live into your production stack?
3. What is happening when something isn’t going as expected? Is it being changed back to where you need it to be, are you monitoring it, or are you doing nothing?
DevSecOps: String Together Wins

You also need to make it simple. She quoted Mark Burgess, “IT has a detail sickness,” noting that, “We are often burdened by complexity—we love it and dig right in, but it is important to understand the level of granularity you need. You need to look for the things that can make a critical uplift that gives you an incremental improvement. String together the wins.”

In the end, Julie said to “try to reach a goal of being visible and streamlined and leverage the automation and technology in a way that is joyful in how we use it. It is not about a rigid process that is going to die soon; it is about what we are trying to bring to the whole world so that we work more efficiently and more technical.”

The entirety of Julie’s presentation—including some live Q&A—is available here. If you missed any of the other 100 speakers at All Day DevOps, you can find their 30-minute presentations here.

— Derek E. Weeks
February 28, 2018
Standardization, De Facto Style

The first decade or so of DevOps had one truth to it: It was about change. Not just change in the way that IT worked, but change in the way DevOps worked—particularly in the tools space, where products and product preferences both are in flux. We’re still seeing a massive change, but increasingly the effect is more like the ripples created when a rock is thrown into a pond than a tsunami, dumping change upon change.

You could look at this one of two ways (and no doubt those who have more reason to dig into details than I have already done so): Either the slow but steady standardization typified by tools such as Jenkins and Puppet have enabled participation of enterprises in the DevOps life cycle, or the demands of enterprises for some amount of stability in their massively complex systems have driven standardization.

Either way, the result is the same. We are seeing a lot of change further out, but the core technologies for build/test/deploy are settling into a few products by a few vendors. That is good for adoption, even if you would argue that it is bad for innovation. It means that skill sets are, at least to a certain extent, portable across DevOps teams and organizations, which for a long while was not true. When I was knee-deep in deployment technology, knowing how to use product A did not mean you knew how to use product B. The more people using one or the other, the better for employee mobility and the pool of available resources, as both people and support tools like standardized scripts.

I recently was going through the offerings of a service provider in a space that doesn’t traditionally have DevOps enablement. The company’s offering included DevOps, so I drilled down. Seeing the company supported Jenkins, Jira and Xebia Labs made me feel better about the program it was putting together. It meant the company was adapting mainstream tools to a specialty market.

That’s what spurred this blog. It’s been a long time that these things were seen as “new and shiny”; now, for a service provider to include them as proof it is serious about customers’ problems shows just how far we have come.

There’s still a long way to go. The DevSecOps community is currently in the throes of growth, and lots of outer-edge DevOps tools are incompatible with each other and chosen per shop without a clear market leader. But we’re getting there. And at this point, a person going into DevOps who asks me what they should know will actually get an answer of “Learn X, Y, Z” rather than “You should learn X, but pick up some of X.1, X.2, and A too, then there’s Y …”

The thing is, though de facto standardization is happening, to many (particularly enterprise) shops, this is all still new. So if you don’t know the recognizable tools yet, don’t worry too much; there will be opportunity to learn as they are rolled out across a given org.

— Don Macvittie

February 27, 2018
Continuous Delivery: No Excuses
Jez Humble’s (@jezhumble) career has spanned roles through coding, infrastructure, and product development across three continents and organizations of varying sizes. To say he knows a lot about continuous delivery is a total understatement. In 2010, he and Dave Farley literally wrote the book on continuous delivery—and if you have not yet read it, I would suggest you pick up a copy. It’s a fascinating read.

I’ve heard Jez speak over the years at a number of DevOps conferences, and I was excited when he accepted our invitation to speak at All Day DevOps. At the conference, Jez presented “Continuous Delivery Sounds Great But It Won’t Work Here,” during which he addressed the four reasons he consistently hears from organizations on why continuous delivery won’t work in their organization:
- We’re regulated
- We’re not building websites
- Too much legacy
- Our people are too stupid
Jez also outlined the real reasons behind the stated reasons: Their culture sucks and/or their architecture sucks.

Jez spent his talk addressing each objection.

We’re Regulated

Jez noted it took Amazon four years to go from monolithic to service-oriented architecture (2001-2005) but the company did it. Do you think Amazon is the pillar of free enterprise, free of regulation? Not so. Being a publicly traded company, Amazon is heavily regulated; it is subject to Sarbanes-Oxley regulations; it handles personal and corporate financial and proprietary data; and it provides private cloud services to the U.S. government, including highly sensitive data for the Department of Defense.

Amazon Web Services’ GovCloud, in fact, provides the cloud platform for cloud.gov, the 18F project Jez worked on to address the authority to operate process within the government.

The old process had screened every software application before it was allowed on a federal IT system. It stifled innovation because it duplicated efforts multiple times over. There was no cross-agency assumption that if one agency approved a software package, it was okay at another agency. Cloud.gov moved controls from operations into infrastructure as a service. Now, agencies can receive software through cloud.gov to integrate into their applications.

The bottom line is continuous delivery is excellent for heavily regulated environments because it gives assurance to regulators by removing opportunities for human error.

We Aren’t Building Websites

Jez’s favorite example is HP LaserJet firmware. HP had a 400-person team spanning three continents. Obviously, the firmware was critical for new printers, but the team was moving slowly. Only 5 percent of their time was spent on innovation. HP tried hiring, firing, insourcing and outsourcing. With business school tried-and-true solutions exhausted, and knowing it had to eliminate non-value adding work to increase innovation, the company looked to continuous delivery with two business goals in mind: 1) Get firmware off the critical path; 2) Achieve 10X increase in development productivity, measured by percent spent on innovation.

HP had other goals:
- Reduce hardware variation
- Create a single package
- Implement continuous integration
- Implement comprehensive test automation
- Create a simulator
Note: the chart above doesn’t show the 23 percent of time spent managing the automation tests. Time well-spent.

In the end, between 2008 and 2011:
- Overall development costs were reduced by ~40 percent
- Programs under development increased by ~140 percent
- Development costs per program decreased 78 percent
- Resources now driving innovation increased by 8X
Too Much Legacy

Jez’s example here is Suncorp, one of Australia’s largest insurers (note: Also falls under the “We’re heavily regulated” excuse). Suncorp was burdened with loads of mainframes and legacy systems, but it built a set of test suites with open source tools to move into continuous delivery.

Our People Are Too Stupid

Jez looked outside of the software development industry into an old-line manufacturer: automotives. GM’s worst performing plant had been closed down. GM fired every one of the workers from the plant. The workers were notorious for problems ranging from sabotaging cars to drinking to gambling on the job.

Remarkably, GM’s union leaders convinced Toyota to hire the old GM employees to learn the Toyota processes, and Toyota hired the GM employees to run their new NUMMI plant. Toyota and the old GM employees showed it wasn’t the people that was the problem; it was the leadership, the management system and the use of obstacles. At the GM plant, an employee had a job and a certain amount of time to do it. If it didn’t get done, the assembly line kept running. At the Toyota NUMMI plant, employees were empowered to ask managers for help—without retribution. The car had to be in a good state before proceeded to the next stage of the production line.

In software development, this is continuous integration. Toyota’s NUMMI is the inspiration for Lean in the United States.

Jez’s take-home point for any organization is that results aren’t achieved when you adopt practices but don’t implement the culture. You need to:
- Agree and communicate measurable business goals
- Give teams support and resources to experiment
- Talk to other teams
- Achieve quick wins and share learning
- Keep going
Jez dives into more details in his talk, which you can watch here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.

— Derek E. Weeks
February 23, 2018
Application Discovery and the Value Proposition of IBM Z ADDI

Discovering how an application works can be a difficult, expensive undertaking. That’s why architectures based on REST are so popular. The architectural style provides an easy-to-implement interface that allows consumers to use an application productively without having to understand anything at all about its internals. All a developer needs to know to work an application is how to interact with the resource represented by the given endpoint. What’s going on behind that endpoint is another matter.

That’s the good news.

The bad news is that when it’s time to make changes to the application’s internals, things can get difficult. Many times a well-defined interface is hiding a pile of spaghetti code. Sometimes a developer might be able to find an analysis tool that will help make sense of things. Or, maybe there’s documentation that explains it all. But, it’s not unusual for developers to have to bite the bullet and go through the source code and deployment scripts line by line to figure out how the application works. Such work incurs significant expense.

Discovery: Understanding Dependencies in an Application

Mainframe developers have been solving the problems of application discovery for years. IBM has been at the forefront of addressing application discovery with a comprehensive solution called Application Discovery and Delivery Intelligence (ADDI). ADDI allows a developer to inspect a IBM Z mainframe application and analyze its performance.

Application discovery is accomplished using Application Discovery Intelligence (ADI), which is part of the ADDI toolset. Developers and system analysts use ADI, which integrates seamlessly into the Eclipse integrated development environment (IDE), to discover the assets used in an application and also to determine the dependencies between them once discovered.

Figure 1 below shows an example of the ADI graphical interface. The IDE provides a high level overview of the application (1) as well as a listing of its components (2). Developers can drill down to look at details in a central viewing pane (3). The IDE provides a detailed listing of a component’s connections and associations, such as subscriptions to message queues or SQL calls (4). Also, developers can view details about a particular assets, such as a SQL query (5).

Figure 1: Application discovery intelligence describes components in an application and describes interactions between them

Having immediate insight into the structure and operation of an application reduces the expense required to improve it. Developers save a lot of time not having to peruse code on a line-by-line basis. Yet, this is only part of the cost-saving. Understanding the composition of an application is an important first step. Understanding how an application performs is essential for implementing improvements. This is where delivery analysis comes in.

Delivery: Providing Quality Service 24/7

Ensuring quality service on a 24/7 requires that availability of up-to-date runtime information. A good performance monitoring tool provides the ability to to drill down to determine the cause of an issue. IBM Z provides such as array of tools. Remember, meeting service agreements to specification is and has been a critical activity in the mainframe computing culture. It’s part of a mature business process.

IBM Z Application Delivery Intelligence provides mission-critical insights on a drill down basis. As shown below in Figure 2, IBM Z Application Delivery Intelligence Analysis reports a dangerous condition (1). Then, engineers can drill down to view associated metrics (2) and then even deeper, to view a timeline that shows environmental conditions such as response time from the database and CPU time consumption (3).

Figure 2: IBM Z Application Delivery Intelligence provides drill-down reporting that allows engineers to identify root causes

One of the drawbacks of architectures that aggregate heterogeneous systems is that relationships between components become opaque. For example, an application that uses a database service to store and retrieve data has no direct insight into the underlying query behavior of the database service. All that can be accurately reported is the request and response time to and from the service. However, when both application logic and data storage reside inside the same location, reporting details becomes easier. But still, drill-down correlation between systems is still difficult when components are in a common location, such as a virtual machine, yet isolated from one another due to strict encapsulation. Mainframe systems, however, do not have the problem because all components reside in a single location in which access to components is transparent. Thus, cross-component drill-down is easier to accomplish.

Built to Scale in the Modern Cloud

When it comes to computing at scale, mainframes were always meant to handle multiple users in a high-volume, transactional environment. Mainframe computing does not lend itself to the type of latency bottlenecks that come about when there are racks of commodity PCs operating with with varying degrees of reliability. The hurdle that mainframe computing had to overcome was making it so it could be accessed using standard web-based standards, most notably REST. IBM addresses the problem with z/OS Connect, a technology that allows IBM Z application functionality to be represented by a REST API. Once a REST layer is added onto an IBM Z system, it becomes just like another resource in the cloud computing ecosystem. And, an IBM Z mainframe computer is designed to be fault-tolerant and highly available. There are IBM Z systems in production that have run without interruption or downtime for decades.

Making Discovery and Delivery Less Expensive

Discovering how an application works and ensuring that applications in production meet performance expectations are two of the most important activities in today’s modern enterprise. Application Discovery and Delivery Intelligence provides the tools to meet these needs in the IBM Z cloud computing ecosystem. Using ADDI provides the insights necessary to ensure that service level agreements are met cost-effectively to the satisfaction of all. When it to making applications that matter in a world of web-based computing, ADDI provides an essential value that is difficult to match.

— Bob Reselman

February 6, 2018
DevOps Heals at NIH
Cancer sucks. But with folks like Sarah, DevOps is helping make a difference in the race to a cure.

Sarah Elkins is not curing cancer herself, but she is employing DevOps practices to help those who are. Sarah supports the technology infrastructure for those who are trying to cure cancer at the National Institutes of Health (NIH).

Sarah Elkins (@configures) configures technology solutions at the National Cancer Institute (NCI), where she has worked for nine years. NCI is a federal agency, part of the National Institutes of Health. It supports more than 700 websites, from basic HTML to content management systems to complex bioinformatics systems that have multiple tiers and thousands of servers. It uses multiple operating systems, containers and physical and virtual machines (both on-premises and some AWS). Developers range from lone scientists to large teams.

Sarah and I have been following one another on Twitter for several years, but I recently caught the session she delivered in the “DevOps in Government” track at All Day DevOps. Sarah led a discussion about how the National Cancer Institute is automating its builds and deployments—showing it is possible even in a large bureaucracy.

Her talk covered the processes and technologies that enable software to move from source code repositories all the way to production servers at nci.nih.gov, including the use of GitHub, Jenkins, Nexus and other technologies, with a variety of teams involved.

We Need to Talk

Sarah’s cure begins with engagement: development, infrastructure and security all working together, just as teams of specialists work with patients to help them beat cancer.

To help speed development and approvals—a necessary evil that can grind development to a halt if not managed well—they agree that applications drawing on an existing technology catalog are approved as operational and security teams provide guidance to development on what is required for an Authority to Operate (ATO). In other words, they are trying to preapprove as much as possible and communicate earlier in the process.

DevOps Practices are Maturing

Source code is primarily kept on GitHub, including a public repository for non-proprietary source code, and NCI primarily uses Maven or Apache Ant for build scripts. Infrastructure teams provide XML templates to provide consistency.

Most NCI software relies on build dependencies on open source software components. The organization uses artifacts during builds and utilizes Sonatype’s Nexus as its repository manager.

All of this supports the automation of builds and deployments at NCI. For builds, most active projects are either on or migrating to Jenkins. Build artifacts may be .zip, .war/.ear or Docker images.

For application deployments, NCI uses development, quality assurance, stage and production stages. Most teams use some form of automation, from simple (copy content, stop/start container) to more complex scripts. Developers are allowed to perform deployments for robust applications, and some manual orchestration is required—for instance, for database timing or related applications.

As an organization, NCI is moving toward continuous integration (CI), with varying progress among teams.

Building In Security

NCI also now includes security early, often and automatically. The organization has deployed self-serve/on-demand Nessus scans, which allow developers to see, on their own, how they are doing as soon as the application is stable. Security teams run AppScan and Twistlock for Docker image scanning during Jenkins builds, just before deploying and frequently scanning the repository in between. For issues found, development and infrastructure teams work together to remediate security concerns.

At the end of her talk, Sarah offered up three takeaways:
1. DevOps at NCI is a work in progress (isn’t it everywhere?)
2. The wide-ranging needs at NCI require flexibility, communication and teamwork
3. There is no shortage of work
Since it is a work in progress, what opportunities are ahead at NCI?
- More automation for individual applications with Jenkins
  - Developers can perform container restarts on lower tiers
  - Database updates (some projects are using Liquibase already)
  - Some applications are still manual/batch scripted
- More Docker and improving Jenkins/Docker instances
- More orchestration automation and Puppet/Ansible integration
- Security improvements through
  - Scan dependency artifacts before building
  - More integration and automation
Sarah dives into more details in her talk, which you can watch here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. DevOps in Government is one of the five tracks.

— Derek E. Weeks
February 5, 2018
Dev vs. Ops Needs for Large-Scale Version Control Systems
Distributed version control systems (DVCS) such as Git (introduced in 2005 by Linus Torvalds, the creator of Linux) rapidly gained worldwide acceptance by the global software development community. DVCS offer very fast, lightweight and local software change control capabilities that support many parallel code branches across large project code bases. These are compelling benefits for individual developers and development teams. At their heart, DVCS accomplish this by tracking software changes as a series of file system snapshots in local repositories instead of via management of individual file changes in a central database. “A Short History of Git” explains this in more detail.

The DVCS approach is ideal for individuals and small agile teams because it puts control directly in their hands without the need to transact all file changes with burdensome systems that were designed using a central management paradigm.
However, the distributed approach taken by DVCS has historically presented challenges for operations, especially when large-scale enterprise software systems consisting of components from many separate repositories need to be managed and controlled. Enterprises with many developers, large source or binary files (e.g. graphics, audio or video), many artifacts or many large containers need granular end-to-end version traceability. Enterprise-class support requirements exceed the original design intentions of DVCS.

End-to-end software version management offerings that on one hand leverage efficiencies of distributed systems for developers yet still support scalable operations are required to address the needs of Dev and Ops communities in enterprises with large software version management needs. These complementary requirements have driven the evolution and best practices of large-scale hybrid version control systems:
- Simplify builds and manage complex projects by organizing multiple repositories
- Move code and build assets into the DevOps pipeline quickly
- Improve performance for remote sites by using a central system as a proxy/cache
- Speed up many activities at the system level, such as cloning and copying code
- Implement a seamless workflow that incorporates assets such as large graphics and binaries, in addition to source code
Related Content:

35+ Tools Every DevOps Expert Must Know

Automation: Achieving Faster Rollouts with Better Accuracy

As stated, historically, these were problems that could only be addressed by a centralized version control offering. DVCS alone simply couldn’t handle the large-scale continuous integration (CI) and operations requirements of large-scale software development communities. While there are still inherent issues in the way DVCS manages large binary assets, select vendors have begun to tackle the problem by mirroring DVCS repos in real time, which marries the developers’ need for easy branching and local control with the operations team’s needs for faster and more efficient builds.

For enterprises developing complex software incorporating both binary and text-based assets, large-scale enterprise hybrid version control systems with supporting both distributed and centralized structures offer a best practice solution. They provide capabilities for a global single source of truth, replication of all types of artifacts and manage versions of everything from planning to release. By resolving operational management problems resulting from sprawl of individual Git repositories, while at the same time preserving benefits of distributed systems, today’s enterprise-class centralized and/or hybrid version control systems can meet the needs of both Dev and Ops without sacrificing efficiency or operations control.

Summary

What do you think? Do you agree that the benefits of hybrid versions control systems listed above are important to achieving large-scale version control? Are there other solutions you recommend?

— Marc Hornbeek
February 1, 2018
Why DevOps Principles Fit Well in Highly Regulated Industries
DevOps is best suited to digital start-ups with minimal regulation and few restrictions, right?

Initially the DevOps movement stemmed from the agile, innovative, born-on-the-web companies that were able to change their way of working quickly. However, the principles and practices are completely transferrable to all organization types. This includes those at the other end of the spectrum, where processes are very tightly controlled and regulation dictates a significant part of the business investment.

Why Highly Regulated Companies Initially Resisted DevOps

When DevOps first emerged, the major concerns for companies following strict regulations were to maintain their security controls and governance processes. Deploying changes more frequently was viewed as a risk to security and the governance controls that were firmly in place.

Another reason for not moving toward a DevOps way of working was the “segregation of duties” requirement, making it impossible for developers to deploy into production—thereby not being able to “do” DevOps.

Related Content:

Break the Rules and Win at Digital Transformation

Understanding is Key to Achieving DevOps Culture

‘Is DevOps Really Worth It?’

There were established, formalized processes in place that satisfied all the regulatory needs. Some organizations saw making any changes to those processes as more work than what would be saved by automation. This meant that the processes stagnated and hindered future change.

Despite this, the majority of executives now believe that DevOps is a crucial ingredient in making sure that their company can adhere to new regulations and stay ahead of their competitors in compliance adoption.

Why DevOps is Actually Better for Highly Regulated Industries

DevOps can be your biggest ally when it comes to regulation and compliance—as long as you follow a few simple rules:
1. View auditors (regulators) as stakeholders in the DevOps journey.
By collaborating with the auditors in the same way as other stakeholders, you can engage with them early in the process and ensure details get agreed upon without costly change further down the line.

Getting buy-in from auditors on any technical solutions at an early stage reduces the likelihood of change requests, missed deadlines and non-compliance. By working closely with auditors, you will ensure that the company has a strong understanding of the level of compliance needed for each different aspect or component of their systems. This helps avoid the waste of being compliant in areas which are not necessary.

Sometimes you will only be able to clearly interpret a regulation through close collaboration with auditors. This avoids misunderstanding and unnecessary rework. In some cases, working closely with the regulatory body can highlight aspects that had not been addressed previously, developing a feedback loop with them. This evolving feedback means that the company will end up with a solution that is satisfactory.

Close collaboration with auditors increases trust in the processes which have been developed in your organization.
1. Codify compliance requirements and policies.
Good DevOps processes will increase the amount of audit trails and governance in the process, while enabling fine-grained traceability throughout the entire delivery process. Compliance documentation can be enforced more easily as part of projects. There is generally less need for process documentation, as so many tools now automatically generate any documentation required. By using collaborative sharing tools we can also reduce the problem of different document versions.
1. Automate your delivery pipeline end to end.
Automation gives reliability, repeatability and traceability. Automation means we have a predictable outcome and fewer manual processes, which is good for auditing and tight control.

By having fewer manual tasks, we can also reduce the possibility of manual error and missed processes. Automated environment provisioning improves the quality of testing and gives a high level of confidence in the change.

With end-to-end automation and process orchestration, controls are written in and embedded to processes and systems. This helps to reduce liability. Systems are designed to reduce the risk of individuals making errors, forgetting processes intricacies or even taking malicious action. This also helps to develop a blameless culture. Bullet-proof processes means that security protocols now become built-in. Rather than see DevOps as a threat to security, it is now being viewed as the best way to mitigate risk. It also becomes a way of enforcing security, audit and compliance requirements.

Automation in all areas allows you to quickly evolve with changing regulation requirements and last-minute additions, helping you to stay ahead of your competitors who aren’t able to do that. Future regulation is likely to require reporting on the lowest-level processes, and the only way to ensure that processes are being followed reliably every time is to introduce automation and DevOps—a culture of ongoing improvement.

What is Happening Now

Some of these companies in highly regulated industries are actually starting to lead the curve in terms of innovation and DevOps adoption now. They have seen just how powerful DevOps can be and are creating an agile business that can meet regulatory requirements and evolve with customer needs, making their business much more competitive.

— Bronwyn Davies
January 30, 2018
Top Challenges When Scaling DevOps in the Enterprise and How to Conquer Them

Preparing to adopt DevOps can be a challenge for organizations of any size. Scaling DevOps and bringing it to an enterprise demands a special strategy. Use these tips to build yours.

DevOps principles apply to all organizations. Everybody has business goals that can be met and exceeded by following DevOps practices. However, an organization’s size will inevitably impact the level of difficulty it will face during implementing and scaling DevOps. A large enterprise organization will encounter more daunting obstacles than a smaller organization, even if their problems are similar in nature.

Regardless of your organization’s size, you are probably wondering what to expect during your DevOps transformation and asking yourself which strategies you can use to overcome any complications. Luckily, you now have “The DevOps Handbook” to turn to for advice. This book is an excellent resource that distills the most important insights learned from difficult experiences into straightforward steps and solutions for overcoming common DevOps hurdles.

DevOps organizations move much faster than many other organizations today without sacrificing quality, performance or security. This blog post will cover some helpful strategies from “The DevOps Handbook” that will help you overcome challenges your organization might face during its shift to DevOps.

Changing Your Organization’s Culture

The first hurdle in the DevOps process is managing culture and expectations. Scaling DevOps throughout the organization requires changing how your business views itself. It also involves rethinking your process for delivering value to the customer. The goal is to get the changes you make in your products and services to the customer faster. In technical terms, this boils down to how often deployments happen. The DevOps process uses positive feedback loops to deliver value to the customer quickly.

Related Content:

What is an Agile IT Organization?

Everything Ops: Are DevOps Principles Being Applied Too Broadly?

To achieve this goal, “The DevOps Handbook” says that it is necessary to shift your focus from cost-optimization to speed-optimization. Optimizing speed does not require a massive, top-down reorganization. Rather, it means giving teams more autonomy. Different skills (such as development, operations and information security) should be embedded into existing teams. This will make teams become self-serving, which will enable them to develop, deploy and operate their software more safely and effectively. Majorly successful organizations such as Netflix and Amazon attribute their success to this approach.

However, building self-serving teams is just one part of the culture change. You also must change how your people think about themselves and their work. Let’s take Toyota, the pioneers of the lean manufacturing process, as an example. The “Toyota Way” has propelled the company’s success, as Mike Rother wrote about in “Toyota Kata.” In the book, he explains that while structural reorganization is an important step toward overall improvement, it is also critical to invest in the betterment of your employees. You have to build upon their technical work abilities and develop the habits of continual learning and experimentation. Toyota accomplished this by using what Rother called The Improvement Kata, which is a model for achieving goals that is based on coordinating theory and evidence.

DevOps organizations integrate continual learning and experimentation into their day-to-day work and decision-making. Verifiable experimentation, such as A/B tests and cohort analyses, advance business goals. The challenge is building the data-driven learning framework. Product owners and other decision makers need a variety of business telemetry. Accordingly, engineers must collaborate and implement whatever is necessary to provide it. Everyone can—and should—use this telemetry to understand what is happening with the business and IT at any point in time.

Etsy is a good example of an organization that has implemented telemetry well. According to “The DevOps Handbook,” in 2011, Etsy started tracking 200,000 business-layer metrics, displaying 30 of the most important metrics on its deployment dashboard. By 2014, the company had tracked more than 800,000 of these business metrics. This high level of dedication to telemetry made its engineers’ jobs easier with quick business and technical documentation, especially after deployments.

The next hurdle is combining telemetry with the practice of continuous learning and experimentation. The two practices work best together when the process is moving as quickly as possible.

Faster Iterations

Addressing speed-related challenges requires more of a technical effort than the others. DevOps requires fast feedback loops at all levels, which begins and ends with a new set of engineering practices.

Trunk-Based Development

First, you should start using Trunk-Based Development (TBD). TBD is a model for source-control branching, in which multiple developers work together on code from one, individual branch. With TBD, developers check in their code to trunk one or more times daily. Using this method has powerful knock-on effects. Automated tests run against the entire system, so it is immediately clear if something is broken. TBD also reduces long-running feature branches that may eventually create difficult merge conflicts or introduce unexpected bugs.

Another great thing about TBD is that it naturally keeps changes smaller, which means that they are easier to review, test and reason about, all while keeping your code in a releasable state. It can also help you reduce your total number of environments, which can easily save tens of thousands of dollars every month.

Automated Deployment

Next, you should automate your deployment process. This requires automating as many of the intermediate steps as possible, including developer workstation setup, test environment setup, environment promotion, the deployment itself and telemetry collection. This is a lofty goal, but taking it to the extreme creates powerful business outcomes. More people are able to deploy more often. Deployments become smaller, safer and more routine, and the number of production bugs is reduced.

Making this happen is no small task since everybody involved must adopt the “automation first” mindset. Start by automating the more error-prone and repetitive tasks. The effort will be worth your time, since automation increases throughput and reduces errors.

Etsy is a big fan of automated deployment. According to an Etsy case study from 2014 that can be found in “The DevOps Handbook,” “deployments are performed by anyone who wants to perform a deployment, such as Development, Operations, or Infosec. The deployment process at Etsy has become so safe and routine that new engineers will perform a production deployment on their first day at work — as have Etsy board members and even dogs!” You may not want actual dogs in your control room, but you certainly want that level of confidence and satisfaction.

Integrating Information Security

This amount of automation sounds fantastic for developers and operations, but what about information security? For them, it’s not necessarily true that topics such as compliance and security are treated as afterthoughts. DevOps is successful because it uses quick feedback loops to identify and fix problems early on in the software development process, but information security and compliance usually come into the picture at the very end, creating huge blockages and problems for everyone involved.

Information security is another skill such as QA or operations, so it can be embedded in cross-functional teams. Then, teams can work together to address known security issues, such as testing for package vulnerabilities as part of CI. Testing for proper secrets, such as a database connection string, can be handled with automation, too. Teams can also integrate static analysis to detect possible problems before the code is executed.

Continuous delivery practices also apply here. Security-related metrics such as total password resets, potential SQL injections, XSS attacks, core dumps and other strange events should be integrated into the deployment pipeline. Doing so provides visibility and immediately actionable information to the entire team during incidents.

Bottom-Up Scaling

Large enterprise organizations are structurally complicated. There are many leaders and smaller teams, each with their own mission and culture. As such, trying to change things top-down is a big mistake—it simply does not work. Scaling out from the bottom-up is a much better approach, and there are a few ways to accomplish this.

Starting small is the best advice. We recommend that you find a small group of like-minded individuals and give them technical autonomy, goals and your trust to improve the business. This will likely help them tackle technical debt issues that have hindered them in the past. They’ll try new approaches and learn down in the trenches.

Unfortunately, there is no such thing as DevOps boot camp—only principles and technical practices. It’s up to everyone in the team (and the organization in general) to figure out what DevOps means to them. Hopefully, their results will speak for themselves. Naturally, other teams will become interested in how they work and think. In time, some team members may even be ready to train their peers.

Ross Clanton at Target uses a “technological innovation center,” which is more commonly known as a DevOps Dojo. The Dojo coaches help teams across Target’s IT department elevate their states of practice. The “30-Day Challenge” is their most intensive training. Blocked development teams bring their work with them to work alongside dedicated coaches and engineers. The goal of the challenge is to unblock your team in less than 30 days. According to Clanton, taking part in this challenge has helped many teams finish work that would normally take months in just a matter of days. Completing these challenges has solved many internal problems, and has ultimately made the organization as a whole more productive.

Conclusion

There is more than one way to scale DevOps in your organization. In this blog post, we have reviewed three of the main DevOps principles that will help you along your journey. First, you need to begin changing your organization’s culture and organizational structure to allow for a “divide and conquer” strategy. Next, establish protocols that will help you find errors early. Doing so will increase the speed of feedback loops, which will make for quicker delivery of customer value. Third, invest in continuous learning and experimentation of new ideas.

Using this and other advice from “The DevOps Handbook,” you can work through your DevOps transformation with confidence. When in doubt, we suggest that you consult this excellent resource for more helpful tips.

— Rami Sass

January 30, 2018