devopsy.kinsta.cloud

Category: Enterprise DevOps

Focusing on the Real Value of AI for IT Ops

In all the hype over AI, let’s concentrate on the real value it can provide, especially for busy IT Ops teams.

2018 is already shaping up to be the year when AI and machine learning (ML) go mainstream—at least in terms of public awareness, if not yet of actual usable products. As I write this, the annual CES event—the show that is, Prince-style, “formerly known as “The International Consumer Electronics Show—is in full swing. The usual suspects are out in force, with Amazon’s Alexa getting embedded in more and more products and Google also showcasing its own consumer AI capabilities.

More unexpected AI adopters included Ferrari, but there were also some notable failed demos, with an AI-enabled suitcase repeatedly running away from its owner or AI refusing to answer questions during an on-stage demo. All of this led the BBC to question, “When will AI deliver for humans?”

What Happens in Vegas, Stays in Vegas

Of course, much of what comes out of CES is frivolous, but there are all sorts of other applications of AI and ML. The problem is that whenever there is this much hype, disappointment and disillusionment are not far behind. Gartner’s famous Hype Cycle model can help here.

By Jeremykemp at Wikipedia, CC BY-SA 3.0

The conventional concern is that overly optimistic projects initiated at the Peak of Inflated Expectations risk failing, leading to excessive caution and fewer projects starting during the Trough of Disillusionment, before a more realistic understanding sets in and the technology achieves true mainstream adoption. Due to the nature of AI and ML, though, there is a second failure mode that is far more insidious.

AI and ML generally operates as a black box. Results are provided, very quickly and without the need to develop deterministic rules, but it is generally not possible to determine why a specific result was generated or troubleshoot failures in detail. As these techniques are used in the real world, potential problems that might not have been obvious in the lab begin to appear. If your smart fridge misreads the expiry date on that chicken (and you don’t think to check yourself), you might be in for an uncomfortable few hours. If your smart thermostat decides for some reason that your house should be as hot as physically possible, you’re still on the hook for the heating bill.

Those are the “nice” failure modes. If a facial recognition algorithm was trained on a data set that didn’t have the right racial balance, you might wind up with some of your employees not being able to get in the door, or some of your customers not being able to unlock your device, or whatever.

It gets worse. Law enforcement has an obvious interest in facial recognition—but what if a faulty system places you at the scene of a riot? https://www.liberty-human-rights.org.uk/news/blog/misidentification-and-improvised-rules-we-lift-lid-mets-notting-hill-facial-recognition Even if you can prove your innocence, you might still have to deal with some difficult conversations with your family and your employer, and your neighbours may never forget the time you were hauled off in handcuffs.

Don’t Throw The AI Baby Out With The Hype Bathwater

So should we just forget about this AI nonsense? Far from it. In fact, I am personally convinced that AI is very much a key part of our future. It just needs to be used in the right way.

I mentioned training data above, and that is the key. AI systems need a lot of good data, and they need to be trained to make good assessments. Out there in the real world, both sides of that problem are actually very hard. Gathering enough data, making sure that the data gathered are actually representative of the problem space, dealing with privacy issues around personally-identifiable information, and formatting the data correctly for ingestion by your machine learning system are already not trivial issues. However, even assuming that you have successfully dealt with that hurdle, the initial results will not be very good or reliable. Intelligent assessment by humans who understand the domain that you are training for is key if your deep learning model is not to go off the rails.

Come Up to the IT Ops Lab

Seen in these terms, IT Ops is an ideal domain to which to apply these techniques. First of all, IT infrastructure already generates huge amounts of event data – Big Data, if you insist – and all of those events are already, by definition, formatted to be easily machine-readable. Sure, there are many different formats out there, from SNMP, via various proprietary product-specific formats, to modern REST APIs, but it turns out that computers are pretty good at translating information from one representation to another.

Once the event data are in the AI system, it can apply whatever algorithms to try to make sense of them: which events are even significant, which ones are related to each other, what other relationships might exist, and so on. When it comes to evaluating the results, again, IT Ops already has people, tools and processes working on precisely that question: is this a real issue, and if so, what to do about it — or if not, why not. This evaluation can be fed back to the learning system to improve its results for next time, iterating over time to provide ever more valuable and reliable results.

This learning can even be applied to the evaluation process itself, watching what users find most interesting and valuable, and extrapolating from that to present the most significant data first, or looking for patterns in human interaction to assemble the most effective teams to work on particular types of incident.

This is what AIOps is all about. Don’t assume that the sort of consumer-grade AI that you see at CES is all that is going on in this field. AI has huge potential to help streamline enterprise IT Ops, by presenting human specialists with actionable events, helping them collaborate more effectively, and learning and improving over time.

Once the CES hype has died down, take a good look at AIOps and start to think how it can fit into your plans for 2018.

— Dominic Wellington

January 29, 2018
DevOps Chat: Rosalind Radcliffe, IBM, on DevOps and Mainframe Systems

In this DevOps Chat we sit down with a frequent guest on DevOps Chat, Rosalind Radcliffe, distinguished engineer and mainframe expert from IBM. Rosalind talks to us about the latest generation of Big Iron from IBM, the z14 systems. Also, we discuss some of the new API and functionality that allows you to do just about anything with your mainframe that you do with your other computing platforms.

Rosalind is always an interesting interview with lots of great information.

Note: There are references to a video chat throughout the conversation. We tried to record this chat as a video, but unfortunately, the technology just didn’t work out well for us. But, as usual, the streaming audio of our conversation is below, followed by the transcript of our conversation.

Alan Shimel: Hey, everyone. This is Alan Shimel of devopsy.kinsta.cloud, and you’re listening to perhaps the first video chat or video DevOps Chat from devopsy.kinsta.cloud. And I’m happy to have as my first video chat guest none other than Rosalind Radcliffe of IBM. Rosalind, welcome.

Rosalind Radcliffe: Happy to be here, as always.

Rosalind Radcliffe

Shimel: Well, you’re not going to be able to sit and make faces while you’re talking because we can see you. But thanks for joining us. Let’s jump right into things. First of all for our audience who may not know Rosalind, Rosalind is a distinguished engineer with IBM and works within the IBM Enterprise Systems division, mainframes, e-systems, author, engineer, well-known speaker and just a great resource for anyone who is interested in DevOps, mainframe, systems of engagement, systems of records kind of stuff, and a frequent guest here of ours on devopsy.kinsta.cloud.

So, Rosalind, the last time I saw you was over at Gene Kim’s DevOps Enterprise Summit in San Francisco. And I thought it was a great show this year. Had a chance to do a bunch of interviews. I did a panel on DevSecOps. But more than that, I really enjoyed speaking to a lot of the people in attendance there, a lot of the attendees. Any key takeaways from you on it?

Radcliffe: Well, I think it was another really good conference, but it was one year next. So there was a lot of people who really have done the DevOps transformation or are a good long way on the DevOps transformation. And so there were a lot more stories about the value that they’re getting and how they’ve transformed, and more about the organizational transformation that happens after you start a DevOps transformation. So I thought that was very good.

We also had more mainframe customers talking about their transformations, and it was really good to hear the stories about mainframe is really just part of the DevOps pipeline, and more and more customers are starting to see that.

Shimel: I think so. And Rosalind, if you don’t mind, I actually wrote an article coming home from Gene’s event called “The DevOps Cha-Cha.” And it’s about this sort of undercurrent. I didn’t necessarily see it in the presentations, but I got it in speaking to people there. And that is this sort of whispered thing that hey, this DevOps thing isn’t that easy. Right? We thought it was going to be easy.

And you know, my DevOps transformation is not going as planned, and it’s not going as fast as we wanted. And it’s not—there’s a lot of work involved here. And I don’t know what gave people the impression that it’s a hockey stick. You know, it was straight up. Things seldom are. I think DevOps can be lumpy, or DevOps transformations can be lumpy. And it’s a little bit of three steps forward, two steps back, cha cha cha, and so forth. Your opinion?

Radcliffe: Well, I would absolutely agree. I don’t think people understand the cultural challenge to this transformation. Because it really is all about culture. And it’s not just bringing in tools, and a lot of people think, “I’m going to bring this toolchain in and it’ll all be fine.”

And maybe some of the teams that works for. But there are a lot of teams that that’s really just not going to work for. And there’s just no way that we can just have this continue in the environment. We really have to make sure we are understanding that this is culture, and that the culture is really, has to change. And it’s across the entire organization.

I think the other thing people didn’t realize is that it is a … what’s the best way to put it? It’s not just culture and you’re done, or it’s not just do it and you’re done. It really is a continuous improvement process. And we like to talk about the kicks team because they’ve been doing this for 15 years or something, and they know they’re not done.

Shimel: Mm-hmm.

Radcliffe: So it’s not something that ends. You learn your lessons. You get better. And you learn a new lesson, and maybe you didn’t get better. And so you learn the lessons. And it really is this continual improvement. And so it’s not easy. It’s not a piece of cake. But it delivers real value if you move down this pipeline.

Shimel: Yep. I mean, takeaways. Dev ops is never done, right? You’re never done with DevOps. And it’s only a failure if you stop trying, right? Anything else is just feedback loops. [Laughter] Everything else is feedback loops. So I think those are great takeaways from that.

So I went from Gene’s to another event to another event. And we’re rapidly, you know, Rosalind, approaching event season. And, of course, IBM’s big event this year is a little different than years past. It’s called IBM Think, right?

Radcliffe: Yes.

Shimel: It’s in March in Vegas. Can you –

Radcliffe: Yeah. It’s going to be an interesting event. We’re bringing everything together in one. So we used to have in our connect, and Watson’s event, and internet of things. You take your choice of names. Now it’s all one. So we’re going to have one big IBM event for all of our technology, all of our parts of the product. And they’re really trying to bring it in a focused way to make sure it’s a set of customer stories on what they’re doing and how they’re doing things.

Shimel: Do you know the exact dates, perhaps, off your head? Or if not we’ll –

Radcliffe: Yeah. See, it’s March 19th. It’s March 19th. The first day is an inner-circle day. And the next Tuesday, Wednesday and Thursday are the main days for the conference. And so it’s a shorter week this time, not making people come in on a weekend and giving you a chance to get home, which I think is a very good thing. But it’ll be a fully packed set of days.

Shimel: Fantastic. Hopefully we’ll see you out there. I’m not sure about devopsy.kinsta.cloud’s plans yet around it, but we will be making plans, for sure. So Rosalind, you spoke a little bit about Gene Kim’s event. We spoke a little bit about the upcoming IBM Think event. Let’s focus in now on Z systems, enterprise systems. I know you guys are, you released a new generation. What was it, around September or August?

Radcliffe: It announced in July, July 16th or 17th, and became available in September. And yes, the Z14 with the latest announce.

Shimel: Yep. And it’s brought a whole bunch of stuff to the forefront, probably some of the biggest might be around security and encryption from my point—humble point of view—anyway. But you’re excited about some new stuff coming up with the Z systems we were talking. Why don’t you share it with our audience?

Radcliffe: Yeah. I’m really happy. We’ve been doing a lot of work to make the dev ops pipeline work for z/OS software. And a lot of customers are turning to open source. And so they want to use Git. And so I got Rocket to port Git to z/OS, so that’s the first step. And that’s great. That’s been available this year. But we have to have a build system. And so we have made available something called dependency-based build, and it’s in beta, and it will be GA by Think. The idea is you can use Groovy Scripts to write your build definitions for z/OS.

So think about that. It’s not, you know, JCL. It’s not different. I can use Git just like any other language. I can use Groovy just like any other language. I could write my build scripts. And it’s not different. It can make our COBAL and PL/1 in assembler look just like all those other modern, fun languages.

And really they—okay, assembler might be a little hard. But, okay, other than that, COBOL is just English and PL/1 is a pretty good language. So why not? The fact that you have to do things so differently, we want to get rid of all that. And so this new capability, it’s in beta. Anybody’s well—it’s an open beta. Think about this. IBM open beta? Yep. There it is.

Shimel: IBM –

[Crosstalk]

Radcliffe: Go play with –

Shimel: That would be neat. [Laughter] Where can people go play with it? Did you have a URL or section?

Radcliffe: We have a URL. I’ll get it to you so you can post it. It’s not a friendly URL.

Shimel: You know, I can’t help but smile, Rosalind, and think about some of my friends and family who aren’t in the tech industry listening to this, or watching this, and hearing that we’re going to be playing with some Groovy scripts, what’s going through their head. But that is great news. And it’s part of this bigger movement.

You touched on it a little bit. We started at Gene’s event, and it does show. And that is initially with DevOps we’ve been through the binary IT mode, the multi mode, multi speed, binary speed, all this nonsense where let’s treat the mainframe folks as special, and all of that.

But I’m glad to see that—and Rosalind, I think I ran into you at CA World, perhaps, right, where IBM was a major sponsor. And that seems to be the message coming out of the enterprise world now, the mainframe world, which is that’s nonsense, that we’re not going to stand for being labeled special or anything like that. We move as fast as the next guy does, or as fast as the other systems do. We use the same languages in a lot of instances. And we’re even updating the languages that we do use to make them more relevant in today’s world.

So to me that’s a huge message, right? That we need to continue hitting home, though. Because I still hear people talking about multispeed and all of that stuff.

Radcliffe: Yeah. But bimodal is something that needs to go away.

Shimel: Uh –

Radcliffe: It just needs to go away. We’re done with it.

Shimel: Okay.

Radcliffe: The idea that things are going to go at different speeds is true, but it’s not based on platforms. If I build an API, I really hope I do a good job of that API. And it doesn’t matter if it’s running on a system of record or a system of engagement, it’s running in Java, or it’s running in Cobalt. If I write good code, I’m not going to be changing it 1,000 times a day. So the point is if you’re building APIs, you’re going to build them. But you need to be able to change them as fast as possible.

So it doesn’t matter the platform. I should be able to deliver that code today. I need to make a change, let me deliver it. And I really—I guess I want Z to be considered special for its qualities of service, i.e. its reliability, its scalability. Not because it’s different in the way you do work for it. So if it’s just another node in your pipeline, it’s just another node. It just happens to have a different set of qualities of service. And if you build your node, or your Java, whatever, and you end up needing the qualities of service of Z, run it on Z. Why not?

Shimel: Agree.

Radcliffe: And let’s get the differences because you think a system of record has to be slow, or the differences because you think Z has to be slow. I mean, it’s funny to me. We call the mainframe an old system, but really with the Z14 and the LinuxONE announcement, it’s the most modern platform that exists out there. It’s the most modern chip. So how is it old?

Shimel: Yeah. Yeah. I hear ya. A hundred percent agree. Rosalind, you know, we’re almost at the end of our time on this experiment and I do hope the video comes out. But if not, the audio will. And if you’re listening to this without video out there, it’s because the video didn’t work out so good. But at least we’ve got the conversation imparting the references to the video.

Rosalind, we hope to see you soon. Maybe between now and Think we can do something a little more in-depth, a webinar or something about what we’ll see from Enterprise Systems at Think and what else we have coming down the pike as we move closer to Think. And until then, though, have a merry Christmas, happy new year, happy holidays, and thank you for always being available to us here at devopsy.kinsta.cloud.

Radcliffe: Well, thank you. Thank you for your time. And you enjoy the holidays, as well as everyone who is listening if they hear this before the holidays. Merry Christmas.

Shimel: Yes, yes, yes. All right. Rosalind Radcliffe, IBM Enterprise Systems, distinguished engineer, and our guest on our very first video DevOps Chat. I guess they’re not DevOps Chats. All right, Rosalind. Thank you. Have a great day, everyone.

— Alan Shimel

January 26, 2018
Flexera Extends Scope of Open Source Scanning Tools

Now that many IT organizations have adopted an open source-first mentality to acquiring new software, tracking their dependency on all the various modules they are employing is becoming more challenging with each passing day. To help IT organizations track both usage and dependencies of open source software, Flexera has updated the FlexNet Code Insight software to continuously scan the network for open source software and highlight any known security and compliance issues.

Based on code Flexera gained when it acquired Palamida in 2016, FlexNet Code Insight is now easy to navigate using a single dashboard, says Jeff Luszcz, vice president of product management at Flexera. Its capabilities span high-level package analysis to more detailed analysis that detects when snippets of open source code are being used within a third-party application regardless of build dependencies, sub components, source code, modified source code and binaries and even Docker containers being used.

In addition, this release of FlexNet Code extends the integration Flexera provides to continuous integration servers and third-party build tools to automate scans during the software development process. IT organizations also can create their own custom plug-ins, says Luszcz.

FlexNet Code Insight can identify 13 million open source components and supports more than 70 extensions. That includes integration with data from the National Vulnerability Database and security vulnerability assessment software developed by Secunia Research, which Flexera acquired in 2015.

Luszcz says FlexNet Code Insight is a natural complement to the tools Flexera already provides for tracking use and dependencies between instances of commercial software. The difference now is a steady increase in the rate enterprise IT organizations are now depending on open source code, which changes more rapidly.

Flexera is not the only vendor these days focusing on identifying open source software (OSS) vulnerabilities and associated compliance issues. But it does uniquely provide tools spanning both commercial and open source software. That capability is especially appealing to senior IT executives who would prefer to rely on a single vendor to provide both classes of tools, says Luszcz.

While there’s currently a lot of enthusiasm for open source software in the enterprise, Luszcz says many IT leaders don’t appreciate some of challenges that lie ahead. Most open source software is made up of modules provided by different developers working within the confines of an open source project. Updates to that software to address, for example, a security flaw are not easily propagated throughout the ecosystem. In fact, Luszcz says that very issue lies at the heart of recent security breaches that occurred at Equifax.

Having a local repository to keep track of what software is being used simply isn’t enough, Luszcz says. IT organizations need an end-to-end approach for monitoring all the updates made to that software regardless of where it originates.

Of course, when it comes to open source software there really is no going back. Lusczs notes that as far as compliance and security are concerned, the proverbial genie is already out of the bottle. The only question now is how to marry OSS tools to DevOps processes to make living with that open source genie a lot easier for all concerned.

— Mike Vizard

January 23, 2018
A Definition of DevOps for the Masses

If you asked the average person in the street, “What does DevOps mean to you?” you’d probably get a blank stare. And there are those who aren’t directly involved but are vaguely aware of DevOps as a concept. While a vague understanding is better than nothing, it leaves room for misconceptions and misinformation, which can be problematic in the long term. An accurate definition of DevOps is necessary.

How can you provide an accurate definition of DevOps for a non-technical audience? Long-term DevOps practitioner Alex Honor echoed earlier agile software development principles when he coined the phrase, “People over process over tools.” This mantra provides a handy list of DevOps priorities, and as such offers a useful framework for an easily understandable definition of DevOps.

Start with the People

Everyone loves a good organizational chart, and discussing how DevOps teams organize themselves is a useful way to understand how DevOps should work in practice. DevOps is very much about bridging traditionally separate development and operations departments, but this involves more than simply increasing collaboration between teams.

Related Content:

5 Biggest Ways to Fail at DevOps

DevOps Gets More Exciting in 2018

In a traditional setup, developers build applications and fix them when necessary, while operations teams work to maintain, support and monitor those applications.

In DevOps, the wall dividing these two worlds disappears. But it’s not just about getting dev and ops teams to work more closely—it’s about creating an environment where developers and engineers work alongside each other at every stage of a product’s life cycle. In other words, there should be no separation between dev and ops, just a unified DevOps team.

By emphasizing the people factor and contrasting a DevOps team structure with more traditional arrangements, it’s easier to visualize DevOps playing out in the real world. This is a great place to start when it comes to demystifying DevOps for a less-technical audience.

Move on to the Process

When put into practice, DevOps is supposed to be faster, more efficient and more reliable than more traditional software engineering methodologies—once the organizational structure is in place, these benefits can be achieved via innovative DevOps processes.

In traditional development and ops, manual tasks have a habit of slowing down processes, and getting in the way of high-frequency deployments. But in DevOps processes based on infrastructure as code (more on the tech later), teams can automate the provision and management of resources.

Take the task of provisioning server resources, for example. In the past, the developer would email the infrastructure team and ask for a certain set of resources in a specific configuration. Then someone in the infrastructure team would look at a document and follow a checklist of tasks before finally deploying the server.

In a DevOps process, not only are developers and infrastructure engineers working alongside each other in the same team, they’re also implementing processes through infrastructure as code (IAC).

Within the context of IAC, the provisioning of IT resources is automated via code, so rather than provisioning servers manually, the developer simply clicks a button. The desired infrastructure then spins up automatically, accelerating the overall process and giving DevOps teams the means to deploy far more frequently and efficiently.

By describing the automated processes made possible by IAC, the differences between DevOps and more traditional practices become much clearer. For less-tech-savvy employees, this explanation provides some much-needed context in terms of the day-to-day working of DevOps teams.

Then Talk About the Tools

Technology, and the benefits it brings in terms of automation and management, is a key driver of DevOps. The maturation of cloud computing and virtualization, and the emergence of infrastructure as a service (IaaS) in particular, provides the flexibility and speed demanded by DevOps.

With IaaS, it’s possible to instantly provision computing resources for highly customized and constantly shifting IT requirements, without on-premises hardware or manual configuration. This in turn enables the infrastructure as code (IaC) processes detailed above, and the level of automation required for DevOps teams to function.

So cloud computing, in the form of IaaS and by extension IAC, lays the technological foundations of DevOps. That’s the big-picture stuff, but what about those brand-name tools that non-technical people might have actually heard of? Docker, Kubernetes, GitHub, Jenkins, Travis CI … the list goes on, but how vital are these to the core concept of DevOps?

Arguably, not very. While containerization, source control and continuous integration tools all can perform very important roles within a DevOps context, they’re hardly defining features of DevOps itself.

Continuous configuration automation solutions such as Puppet and Chef come more directly under the IAC umbrella, enabling the automatic configuration and scaling of data center resources. But like the others, these tools would be equally at home in a more traditional setup.

In this sense, there’s no such thing as a “DevOps tool,” just tools that can be used within a DevOps organization for DevOps processes. It’s how you use them that really counts.

So when offering a definition of DevOps for a novice audience, don’t fall into the trap of simply reeling off a list of tools. DevOps is far more about people, organization and process. And while it’s very reliant on IaaS and IAC technologies, many of the tools utilized by DevOps teams are more or less incidental.

And Finally, a Reminder that DevOps Works

The ultimate goal of DevOps isn’t simply to unify development and operations, but to result in shorter dev cycles, more frequent deployments, higher-quality products and happier customers. More bluntly, the ultimate goal of DevOps isn’t just to say, “We do DevOps,” but to increase productivity and make the business more profitable.

Just look at sprawling organizations such as Google and Amazon. The success of these companies hinges on their ability to deploy new software multiple times every day, or even every hour. Amazon and Google may not agree on a precise definition of the term, but what they do agree on is that DevOps, as a set of unified software engineering practices, is crucial to their profitability.

With a consensus on the broad strokes of DevOps emerging over the last few years, lack of definition of DevOps will hopefully become less of an issue as its culture crystallizes. In the meantime, the good news is that the big ideas behind DevOps are also the most straightforward to explain to a non-technical audience.

By breaking DevOps down into people first, process second and tools third, an accurate and relatively simple definition of DevOps is possible. DevOps is fundamentally about people, how developers and engineers organize themselves and work as a unified team to deliver better products faster. It’s on top of this structure that DevOps processes are built, with a heavy focus on automation and innovative tools—always within the context of the core DevOps principles and practices.

— Neal Thoms

January 23, 2018
IBM Z: It’s Not the Dog. It’s the Trick
There’s a piece of conventional wisdom that runs rampant in the tech-o-sphere: You can’t teach an old dog new tricks. Supposedly, once people become enamored with a technology and use it for a long period, they are loathe to adopt new ones. We’ve all heard it, and some believe it. Me? I not buying it any more than I believe that you can’t teach a new dog old tricks. From where I stand, a good trick is a good trick, period!

So let’s talk about a really good trick: mainframe computing. It’s technology that is used by many business sectors—airlines, insurance companies and banks. And, it’s been designed from inception to support interaction from hundreds—if not thousands—of users. Yet, many perceive the trick to be old dog, that’s it’s not applicable to modern computing.

Actually, the opposite is true. It’s an important trick that should be used by any dog, new or old. It’s a question of adaptability: how to make the trick useful in any circumstance.

Let’s put the trick-and-dog analogy aside and talk directly. Companies are sitting on a boatload of mainframe assets that can and should be leveraged for use in this world of mobile devices, APIs, microservices and speed-of-light DevOps deployment practices. The question is, how?

To make mainframe technology viable in modern cloud-based paradigms, it must satisfy four conditions. The technology must:
- Support information exchange using modern API standards;
- Support coding in modern programming environments;
- Support automated testing throughout the deployment pipeline; and
- Be deployable according DevOps best practices.
IBM Z, a mainframe-based technology, has done a great job of meeting these conditions. The various products and toolsets that IBM provides for IBM Z technology make working with IBM Z in today’s cloud computing environment surprisingly easy. Let’s take a look at the details.

Supporting REST APIs for Information Exchange

REST is fasting becoming the standard by which information is exchanged between systems over the internet. As mobile devices and the internet of things grow as the means by which users interact with applications, the data these applications require is accessed in real time using REST. Today, REST is a first-class citizen in IBM Z technology. A key feature of IBM Z Application Discovery and Delivery Intelligence (ADDI) is the capability to discover applications running on IBM Z hardware and identify them as potential REST API candidates. Support for REST makes IBM Z applications consumable to a wide variety of computing devices. Also , supporting REST means that IBM Z applications integrate easily with standard DevOps testing and deployment practices.

ADDI provides the support for REST that allows IBM Z to be an agnostic resource on the internet

In the world of modern, cloud-based computing REST is the lingua franca of information exchange. Supporting REST makes IBM Z data and transactions available to the modern digital ecosystem.

Support for COBOL and PL/1 Programming Using the Eclipse IDE

The IBM Explorer for z/OS, which is part of the Application Delivery Foundation for z Systems (ADFz) tool suite, allows COBOL and PL/1 developers to code in the Eclipse Integrated Development Environment (IDE). Eclipse is well-known to a broad range of developers working in languages such as Java, PHP, Node and Python. Using Eclipse to do IBM Z programming under COBOL and PL/1 allows developers to have a shared programming experience and increase their productivity by an average of 15 percent.

Although the actual lines of code the developer creates will be particular to the developer’s programming language of choice, the unit testing and debugging experience Eclipse provides is similar regardless of language. A developer uses Eclipse to unit test code as it’s being developed. If the test fails, the developer sets breakpoints and starts the debugging session to inspect the code at runtime. The way code-test-code-debug is conducted in Eclipse is consistent. Thus, the skills acquired using the Eclipse IDE are easily transferable between languages.

Having all developers work in a common development environment creates a shared experience—an important step toward breaking down organizational silos. Reducing the silos that isolate knowledge workers from each other so they can collaborate better and faster across teams and geographies, combined with deploying code in an automated CI/CD pipeline, significantly speeds a company’s ability to get code to market while also making collaboration easier across teams and across geographies.

Support for a Wide Range of Testing Techniques

For a software development framework to be viable in a CI/CD infrastructure, it must support a wide variety of testing capabilities. In addition, testing needs to be verified via automation. As mentioned above, IBM Explorer for z/OS allows developers to create unit tests on COBOL and PL/1 code. Once unit tests are created, they can be automated for use throughout the development pipeline. Test reuse via automation speeds up the deployment process significantly.

However, testing at the code level is not enough. As code moves through the modern deployment pipeline, it must be subjected to load and performance testing. Test engineers accustomed to using mainstream tools such as Apache JMeter or Rational Performance Tester will have little problem using IBM Z Workload Simulator for z/OS to do load and performance testing. The concepts behind load and performance testing are similar, regardless of the language and environment driving the development process. For example, Workload Simulator for z/OS allows test engineers to emulate hundreds, if not thousands, of users to exercise the software under test. Being able to create stress on a system in a controlled, reportable manner is critical for getting good performance testing metrics.

Supporting the DevOps Deployment Practices

Basic to DevOps best practices is the notion of releasing quality software quickly and incrementally. Instead of waiting months to do a release that has many features, the DevOps practice is to release one feature at a time, as quickly as possible. In the DevOps world, release cycles are no longer a matter of weeks and months, but rather days, if not hours. To achieve the fast release cycles emblematic of DevOps, management of the deployment pipeline must be automated.

However, when it comes to mainframe assets there is a hitch in terms of deployment units. In the microcomputing software environment, deployment units typically take the form of an executable file, supporting dependency file or script file such as .exe, .jar, .js or .py. Also, given the rise of container technology, the deployment unit can be the container itself. In terms of IBM Z, the deployment units are the file and the job.

Engineers at IBM Z have taken the clever approach to managing asset deployment. File and jobs have become abstracted deployment units accessible via the IBM Explorer for z/OS Atlas APIs. Thus, IBM Z components can be treated as any other artifact in the deployment pipeline. DevOps engineers use the APIs exposed by Atlas to cancel a job or delete an associated file. Standardizing direct access to the IBM Z deployment pipeline using REST APIs gives DevOps engineers the flexibility required to get working code released quickly.

DevOps engineers not only can automate the pipeline, but should ad hoc intervention be required, they also can use the Atlas APIs to make the changes required at a moment’s notice. Combining the Atlas APIs with ADDI’s ability to collect, correlate, calculate and produce mission-critical application and testing health metrics, as well as trend analytics, makes IBM Z a first-order participant in the world of cognitive DevOps.

Putting It All Together

One of the great things that the internet has done is to create a standardized way by which any computing system can interact with another. As long the system supports REST, regardless of whether it’s a smartwatch, a cellphone or a large-scale computing environment such as IBM Z, it’s a leveragable asset on the internet. And, if the system meets the conditions that define DevOps best practices, it can be managed with the degree of control and predictability required to participate in the breakneck release cycles that are part of modern software development. It really doesn’t matter whether the system is new or old, as long as it can adapt to present demands. Adaptability is key to longevity, no matter the age of the dog or the trick.

— Bob Reselman
January 23, 2018
IBM Z: An Architecture That Scales for Modern Computing

The architectures I like best are the ones that are built to scale. I can’t tell you how many time I’ve run across applications that have hit the wall when the request count goes up from hundreds to thousands of simultaneous users. The next thing you know the CPUs start to peg, memory get maxed or overall availability just magically goes away. All that’s left is the dreaded status code: 500 Internal Server Error.

Those of us who have made distributed computing a way of life spend a lot of time making sure that our applications scale up. We’ve found the pursuit to be costly. As result, some of us have turned the entire problem over to managed service providers. Paying a bill at the end of the month to ensure 24/7 availability is a whole lot better than taking a call at 3 a.m. demanding to know why the inventory system in Toledo has gone down. Whether we choose to take care of our own metal or use an MSP, we understand that an architecture that cannot scale has little value.

Surprisingly, one of the best architectures around is one that is used by 90 percent of the world’s largest airlines, 70 percent of big retailers, most of the banks worldwide and all of the largest insurance companies. Where is this architecture to be found? In mainframe computing. Yep, you got it: Big Iron.

Big Iron: The Gift that Keeps on Giving

Mainframe computing in general and IBM Z in particular have a computing architecture that is built to support usage by thousands, if not millions, of clients. Since their inception, these systems have been designed to provide high-availability, large-scale, multi-processor computing in a cost-effective manner. Whereas a typical data center filled with racks of commodity hardware will suck thousands of dollars a month in electricity, a single IBM z System that processes 2.5 billion transactions a day uses about as much electricity as the clothes dryer down in your laundry room. (2.5 billion daily transaction is the equivalent of 100 cyber Mondays every day of the year!)

In terms of high availability, mainframe uptime is measured in years. There are systems out there that have been running continuously for 10 years and are still up today. This is no small accomplishment. Think about it: Imagine what it takes to have enough redundancy built into an architecture to never have to reboot, no matter how weird any of the applications get. Memory and application isolation have been built into the technology since the start. In a way, we’re talking cloud before the cloud.

Memory isolation and application redundancy have been key features of mainframe technology since inception

It’s an amazing technology, one that more developers, particularly those of us working on the forefront of application development should leverage. The question becomes how to benefit from its capabilities and drive business value from this asset with its treasure trove of data. The first step is to understand the difference between a system of interaction and a business critical system.

Understanding Systems of Interaction and Business Critical Systems

In the world of modern, distributed computing, application logic can be represented in a variety of ways. For example, it’s quite possible to check on the status of flight information by using a mobile app, going to an airline’s website or by calling the carrier’s toll-free number. The logic needed to find and deliver flight information is in a central location; the means to access that information is distributed. There are two systems in play—one is called the system of interaction and the other is the business critical system. The system of interaction is the system that interacts with the end user, collecting data to send as output. The business critical system is the system that serves as the repository of common application intelligence. The concept is akin to the thinking behind client-server architecture, but it’s a bit more in that it’s system based. Think of a phone system, cable TV system or online games as a system of interaction. Think of a rack of servers, the cloud or a mainframe as a business critical system. The overall digital experience is the aggregation of its components.

This may seem like an obvious separation, but there’s a bit more to it, particularly when considering large hardware such as IBM Z and the like. Being a business critical system is more than being a simple data store. A business critical system can contain all the data necessary to perform complex machine learning activity upon a particular dataset submitted to it, weather prediction or medical diagnosis, for example. Of course, the business critical system has data, but the system also has algorithmic and parallel processing capability far beyond a standalone PC or cell phone, the components that are typical in a system of interaction.

Business critical systems, particularly those that have been in service for decades, become very powerful when they support common standards of digital interaction—APIs based on REST, for example. Back when distributed computing was limited to a small part of the population, a proprietary approach to system interaction was acceptable. But, the rise of the internet changed it all. Today we have billions of people online, on some part of a global network. Thus, common standards such as HTTP and REST have emerged to make interacting with software systems worldwide easier. Companies such as IBM understand the importance of supporting the ways of the internet. Thus, a product such as z/OS Connect EE allows developers to connect to a Z System installation and design an API around the code in that installation. Using z/OS a developer can wrap a REST API around some COBOL code. Having this type of capability opens up a world of possibilities.

Going to the Next Level

As the world of mainframes gets more attention from those making mobile apps and aggregating services into the cloud, a whole new level of software development emerges. Using a product such as z/OS Connect EE to create REST APIs combines the world of Big Metal with the world of big data. Those of us in DevOps understand that breaking down silos and fostering collaboration is what our work is about. Bringing together the skills and experience of all those who are dedicated to making quality software, from COBOL programmer to engineers who embrace infrastructure as code, is essential in the world of modern computing. The more people and systems we have working together effectively, the better will be both our software and the world we make.

— Bob Reselman

January 16, 2018
How Blockchain Tech Can Make DevOps Better

DevOps and blockchain are among the hottest topics in today’s technology world. However, you rarely hear the two discussed in tandem. Going forward, that may change. Here’s why.

On the surface, the two may not seem to have much to offer each other. DevOps is a set of cultural principles and practices designed to improve software delivery. Blockchain is a way of storing data in a decentralized fashion.

Bitcoin, a digital currency whose transaction records are stored using a blockchain, is the most well-known example, but the world is much bigger than Bitcoin.

So, besides using DevOps practices to create blockchain-based applications, how can they be used together?

DevOps on the Blockchain

The answer centers on making software delivery chains more transparent and reliable via blockchain technology.

What if, instead of relying on tools such as continuous integration servers and automated test suites to determine that code is moving down the pipeline successfully, you instead recorded that data on the blockchain?

This would make software delivery more transparent, which would in turn improve software quality and reliability. Application delivery status and information about software problems could be viewed instantly by anyone over the blockchain, providing a much greater deal of visibility into the continuous delivery process.

In addition, smart contracts could be used to regulate how code moves down the pipeline. They could also ensure that software deliveries meet user expectations.

Rather than relying on tools within the pipeline to catch problems such as a continuous integration error, information could be independently and automatically verified via smart contracts on the blockchain. If a problem occurs, the smart contract could require that it be fixed before it moves down the pipeline and grows into a bigger problem.

SLAs and Smart Contracts

At the same time, software functionality could be guaranteed using smart contracts. Currently, DevOps teams use service level agreements (SLAs) to promise their application users a certain degree of functionality and reliability in an application.

SLAs are not a great solution. They are inefficient to negotiate and costly to dispute. If an application fails to meet SLA terms, it’s on the user to report the problem and demand a solution.

If service agreements were instead recorded on the blockchain and enforced via smart contracts, users would have a guarantee that an application’s failure to provide the guaranteed functionality would be detected automatically. It could also result in an instant solution, such as the refund by the smart contract of a contractually determined amount of money.

Traditional SLAs, and all of the inefficient legal and contractual trappings that surround them, would become a thing of the past.

Conclusion

For now, no one is actually applying it to improve DevOps and continuous delivery. Like most blockchain-enabled innovations, this remains a theory, not a reality.

Yet, its potential to improve the work of DevOps teams, and the experiences of their users, is clear. Blockchain may not be a part of the DevOps conversation today, but it is poised to become a part in the future.

— Chris Tozzi

December 12, 2017
The Role of Operations in a No-Ops World

It seems that every once in a while, we in high tech get wrapped up in the next thing that will eliminate work and/or jobs. Sometimes, as is the case with DevOps, we get something that reduces redundant work and actually makes our life easier in the long run. Most of the time, however, we don’t. XML was going to end programming, Java would make everyone a programmer, SOAP meant non-programmers could piece together chunks of code and create complex systems … the list goes on and on.

Today’s trip on the merry-go-round is for Ops and DevOps. The love in some quarters is “No-Ops”—no operations necessary at all! The data center will run itself! But where is there meat to these claims, and where is it just not possible with today’s technology? Let’s take a look.

SaaS is Today’s No-Ops Option

If you want a nearly complete no-ops solution today, sign up for software as a service (SaaS). You’ll still have to manage users and access—the normal AAA stuff—but the application is built and managed by someone else. That’s as close to no-ops as we have in today’s world.

Any other solution requires a fair amount of ops, no matter what we are told. Installing servers—be they in the data center on hardware, in VMs or in cloud instances—still requires work. Even if you have a tool such as Ansible or Kubernetes cranking hard for you and doing most of the repetitive heavy lifting, you are still tying the pieces together and making things work correctly. Networking is often left to operations/DevOps teams, and making servers into cattle requires operations heavy lifting, even if you are that automated. Apps break and troubleshooting is still a largely manual exercise, with automated help in figuring out what went wrong, but the fix still heavily in the hands of operations.

Hints at an Future Sans Operations

As is usually the case with these claims of ending some portion of the IT workload, there is some underlying benefit if you don’t get caught up in the hype. Remember that early in cloud some were telling us that there was no longer any reason for a local data center. Of course, there are a billion reasons, and each environment is different. But there is no denying cloud has opened up how we look at server deployment. The same is true for no-ops. There are a lot of reasons that it will not actually be no-ops for an entire project (let alone an entire data center), but there are huge benefits to gain from the idea.

Serverless

In serverless, you develop the code behind and API, then expose the API. The system is designed to run the code only when requests come in. Application firewalling is generally mandatory and built into the system. This partially a security feature, and partially the location used to determine what code needs to be called to service a given request. The idea in the cloud is that you only pay while your API is running, and all of the operations parts of the call are taken care of for you. That’s true, if you accept that, in serverless, developers are doing operations as part of development—app firewall configuration, even the simplified version of it that serverless uses, is definitely operations, not development. And troubleshooting between serverless and server-based apps that call the API is going to fall—at least to some degree—on operations. It must, since operations are the people who have network and internetwork knowledge.

Serverless offers some great options for deployment, though, that undeniably reduce operations overhead. For things such as cloud-bursting, the benefit is pretty undeniable. It makes pay-as-you-grow cloud into a seriously granular proposition, and can eliminate the kind of bottlenecks that all code has by spinning up a new instance for each connection.

Container Management

With an increasing amount of functionality folded into container management tools, the ability to auto-scale applications (or, at least, instances of a part of an application) is a big step toward no-ops. Without manual intervention, the container management tool can expand or contract the application’s footprint in response to connections, response times, etc. The built-in routing, when it fits the needs of the organization, limit the amount of networking required to facilitate this scaling also. As time goes on, expect more automation to be folded into container management to make at least the maintenance of applications easier. The advertised ability to perform rolling upgrades on applications while keeping them live is a huge operational bonus too. It’s not precisely no-ops, but it certainly is less-grey-hair-ops.

Dev/Test/Deploy

Application release automation, the growth of continuous test/delivery/deployment and configuration automation tools all reduce ops as well. They don’t make for no-ops, but they do facilitate the concepts, reducing operations overhead while advancing the concepts of DevOps. Expect these tools to continue to grow. Most configuration management/application deployment tools work with a selection of networking devices/appliances already, expect that trend to continue. Again, it’s a reduction in ops—not quite no-ops, but a great step in that direction.

And More

We’ve looked at a couple of really cool technologies that forward the reduction in operations theme beyond what is listed here. Automation of auditing and full-on automation of security testing (combining code analysis with real time data to get better coverage than today) are just two examples. Keep an eye on the market as these toolsets mature.

Keep Kicking It

Meanwhile, Ops isn’t going away. Stay the expert while expanding your skills. As I (and others) have said before, a tool that reduces the hands-on aspects of operations is not a threat, it’s an opportunity. That freed-up time will be useful for all those projects you wish you could do, but didn’t have the time. Or, even better (we can debate that statement later), all of those things the business wishes you could do but you don’t have time.

In the end, you are keeping the org running, and connected to the world. Embracing the parts of no-ops that make your job easier just makes sense. Don’t fall for the hype; just pick up the good bits, and keep rocking it.

— Don Macvittie

November 28, 2017
DevOps in the Age of Digital Transformation
Digital transformation is one of the hottest buzzwords in the technology industry today. While it tends to be overused, the term does represent a widespread, ongoing movement that will set a standard for the next generation of enterprises.

Digital transformation is all about keeping up with the pace of innovation and using technology to achieve business goals. A key differentiator in the age of digital transformation is customer experience. Consumers are increasingly engaging with brands on digital platforms, whether it be a website, social media profile or mobile app. The foundation of all these platforms is, of course, software.

Agile software delivery is key because it helps organizations innovate faster and gets new software in the hands of consumers sooner. This requires new, more productive ways of working for enterprise IT teams. Development teams have already undergone several “transformations” in recent years, from the siloed waterfall approach to agile methodology and now DevOps. Software is the true differentiator in the age of digital transformation, and many companies are using DevOps to ensure their software meets the standard.

Here are a few best practices for using DevOps to keep up with the pace of digital transformation:
- Automate wherever possible. We no longer collect the work of multiple engineers over multiple sprints into a release. To out-innovate the competition in the digital transformation era, a team must release the work of a single engineer as soon as it is ready. This is usually accomplished through a CI/CD pipeline directly from the source repository through automated testing and then deployment into production, preferably without any human intervention. Choosing the right tools for your IT environment will cut costs, time and room for error.
- Align the goals of development and security. DevOps isn’t limited to just the development and operations teams. Without security, all progress made in the digital transformation era is at risk to cybercriminals. The accelerating software development life cycle means little time for security analysis or penetration testing when the release candidate is ready to deploy. To ensure a piece of software is secure before pushing it to market, security standards must be included in the definition of “done.” Preventative scanning will not only train developers to write secure code, but will allow the software to pass the assurance tests included in any good CI/CD pipeline.
- Measure ROI. Measuring the impact of DevOps can be extremely difficult and up to interpretation because of the focus on culture and processes. However, it’s important for DevOps teams to gain continued executive buy-in and dedicated resources by proving their value. The value of DevOps is determined in terms of achieving improved business outcomes. There are specific questions to ask to help determine ROI: How long does it take to release a code change? How long does it take for a product to reach a customer’s hands? The ability to continually deliver quality software faster results in a positive customer experience, ultimately improving brand reputation and revenue.
While keeping up with the competition is necessary in the digital transformation era, putting customer experience first is how the true leaders succeed. By shortening the software development cycle, DevOps speeds the pace of innovation for an organization and helps earn customer trust and retention. Implementing these best practices can help a business use DevOps to place software and customer engagement at the heart of its digital transformation.

— Pete Chestna
November 6, 2017
Digital Transformation Survey: Help Us, Help You

The term digital transformation is probably one of the most overused, if not abused, terms in the tech/DevOps world today. Everyone is doing it, but what is it? Is it a synonym for adopting DevOps? What are its effects on you, your job and organization? What does it mean for software development and deployment? So many questions. Actually, they’re only a few minutes’ worth of questions in our new survey, Digital Transformation and Your Organization.

These surveys really help us understand where the market is, what our readers and community members are dealing with and what we can do to serve them better. You helping us by taking the survey allows us to help you—not only you, but the wider community as well. Your opinion is important and it counts. So please take a few minutes and help us. Take this survey now.

You still here? OK, if our words didn’t persuade you, let me go one better: We are willing to pay you to take this survey. We will give one in every 10 people who take the survey a $50 Amazon gift card. To be eligible for the gift card, you need to enter your email address in the optional last question of the survey. Otherwise, we don’t know who to send it to. So that is another reason to take the survey.

We will make the report of the survey findings available as well, and everyone who registers will receive a copy. The survey is open to anyone, so please feel free to share the link with your colleagues.

Beyond taking the survey, I don’t want to minimize the importance of digital transformation. While for some—especially in our DevOps audience—digital transformation means DevOps transformation, that is not its only use or meaning. For many businesses, just adopting a digital footprint, using the internet as an active part of their business or adopting modern methods, software, Agile methodologies, etc., opens up unimagined new ways of business. At the end of the day, digital transformation may, in fact, be one of the biggest agents of change in our world over the last 20 years.

We’re looking forward to analyzing the responses and seeing your digital transformation experiences. Please help us help you by taking the survey.

— Alan Shimel

November 2, 2017