As we close out 2023, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2023.
Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Scrum sucks, sources say; and here comes the Linux 6.5 kernel. (more…)
Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: VS Code drops support for Python 3.7, Windows drops VBScript, and Europe plans the fastest ARM supercomputer. (more…)
As we close out 2022, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.
Welcome to The Long View—where we peruse the news and strip it to the essentials. Let’s work out what really matters.
This week, something a bit different: I’m devoting my entire column to Agile and Scrum. They’re increasingly getting a bad reputation, being associated with the worst aspects of toxic workplace culture: Sexism, racism, burnout, micromanagement. (more…)
Think! Welcome to Big Blue Week here at The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: IBM’s employment practices scrutinized, IBM z/OSaaS breaks cover, and IBM encourages staff back to the office.
First up this week: IBM’s defense in an employment age discrimination lawsuit looks to be shredded. Internal emails “appear to show high-level discussion about paring the ranks of older employees.”
According to one of the plaintiff’s lawyers, unnamed “top IBM executives were explicitly plotting with one another to oust older workers … to make room for millennial employees.” On the face of it, they seemed to be doing the polar opposite of the Age Discrimination in Employment Act.
Noam Scheiber: Making ‘Dinobabies’ Extinct: IBM’s Push for a Younger Work Force
Top IBM executives were directly involved in discussions about the need to reduce the portion of older employees … sometimes disparaging them with terms of art like “dinobabies.” … Documents made public by a Federal District Court … show executives discussing plans to phase out older employees.
…
The newly unsealed documents, which quote from internal company emails … show top IBM executives specifically emphasizing the need to thin the ranks of older workers and hire more younger ones. [One] email refers to IBM’s “dated maternal workforce,” an apparent allusion to older women, and says: “This is what must change. They really don’t understand social or engagement.”
…
Adam Pratt, an IBM spokesman, [claimed,] “IBM never engaged in systemic age discrimination.” [He] said that some of the language in the emails “is not consistent with the respect IBM has for its employees” and “does not reflect company practices or policies.”
Yikes, ageism and sexism? Experienced staff are valuable, says MrBanana:
I did some time at IBM [and] remember being invited to a meeting with a new group, headed by a new “thinker,” who had something to present to myself and a colleague. We walked past the foosball table to the meeting room. Naturally the room had no chairs, and was covered in multi-coloured Post-it Notes. Once they had gone through their mood boards and the non-linear presentation of this new idea, I turned to the other old geezer in the room. “Didn’t we look at something like this 8 years ago?”
“Yeah, we worked out that compute power just to run the UI for one session wouldn’t be available for a couple of decades.” The children were somewhat disappointed, and (fortunately) we were never asked back to their romper room.
But experience can be both a blessing and a curse, learned aahortwwy:
I experienced both sides of this in five years at a company in hypergrowth. … I learned so much at that job: … In year 1, I was determined to accomplish X and ignored the skeptical comments of some colleagues who’d been at the company longer than me. I failed and wasted some time. In years 4 and 5, new colleagues were determined to accomplish X and ignored my skeptical comments.
“Mainframe” is not a dirty word. It’s secure, vertically scalable, and it put the VM in VMware. Now, what IBM Wazi did for on-prem LPARs, Wazi aaS promises to extend to hosting z/OS instances.
Mainframes and cloud-native DevOps need not be incompatible. To modernize legacy apps using hybrid cloud instances for dev/test—and even prod—can be a big productivity boost. But isn’t it weird that IBM spun out its mainframe-as-a-service zCloud offering only a few months ago?
Mike Vizard: IBM Shifts Mainframe App Testing to the Cloud
While many workloads over the years have been migrated off mainframes or simply retired, the applications that run on z/OS today are still among the most mission-critical that organizations run. … IBM this week revealed it has added the ability to set up testing environments on the IBM Cloud for z/OS mainframe applications.
…
IBM plans to make IBM Wazi as-a-Service (Wazi aaS) for z/OS generally available in the second half of this year, [which] will enable DevOps teams to self-provision an instance of a z/OS virtual server on a secure private space running on IBM Cloud. … At the same time, it will become easier to modernize mainframe applications by making them more accessible to other applications via a set of secure APIs … using a cloud-native set of DevOps tools that run in a Kubernetes environment.
…
Modernizing mainframe applications is a less complicated endeavor than migrating them. The challenge is that building, deploying, maintaining and securing all the APIs required to do so represents a significant amount of effort.
OK, boomer, why are we talking about 1960s mainframes on devopsy.kinsta.cloud? Here’s ArchieBunker:
Mainframes are different beasts altogether, almost alien when you compare them to PC hardware. You have binaries written for the S/360 system in the 1960s? Good news you can still run them natively without a recompile.
You think virtualization and containers are a hot topic? IBM was running multiple OS instances inside a VM in the 1970s. … The engineering is incredible.
But z/OSaaS? It’s déjà vu time for Beleagured Greybeard:
“Cloud Managed Services for z” AKA “zCloud” has been around for over 10 years. … IBM offloaded it with the Kyndryl spin, so its odd that they now want to re-invent it.
…
If it’s a strategic platform, why didn’t they hold onto it instead of spinning it off to Kyndryl? Sounds like yet more daft leadership decisions.
IBM wants to “keep employees safe” as they return to office work after being fully remote. In practice, this means a vaccine mandate.
Although there are exemptions on medical or religious grounds, IBM has promised to place dissenters on unpaid leave. As you might expect, the policy has a few employees up in arms.
Cassie Werber: How Salesforce, IBM, and HSBC are approaching return to work
When the pandemic hit, and IBM moved 95% of its workforce to fully remote, it was pleasantly surprised to discover no drop in productivity. … Still, they planned to return to offices.
…
The company saw this as a way to keep employees safe. … One example … is its development of the IBM Digital Health Pass as an alternative to vaccine cards or paper vaccine certificates, which they use to make sure all the staff that come into their offices are vaccinated.
…
Arvind Krishna, IBM’s CEO, meanwhile came up with what he called the IBM Work From Home Pledge. [It] included commitments to flexibility, kindness, sensitivity to family, setting boundaries, and acknowledged video fatigue. It concluded with a commitment to “be connected.”
Tricky to “be connected” if you’re not allowed back in the building. Anne D’Innocenzio, Heather Hollingsworth, Tali Arbel, Mae Anderson, Ken Sweet, Matt O’Brien and Dee-Ann Durbin:
IBM software engineer Justin Albano said his company recently notified employees that it would be revoking badge access to workers who aren’t vaccinated or haven’t proved that they are. “We’re not able to enter an IBM office at this time,” said Albano.
…
He said some workers have successfully sought religious exemptions but “my thought process was I shouldn’t have to explain to my company my faith” to get out of a policy that’s “immoral in its foundation.”
What’s the alternative? With a perspective, it’s Chris Haney:
Or, get the vaccine. Protect yourself and the people around you. You people really are detached from reality.
You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.
Image: Marcin Wichary—cc:by; leveled and cropped (ask your parents)
As we close out 2018, we at devopsy.kinsta.cloud wanted to highlight the five most popular articles of the year. Following is the first in our weeklong series of the Best of 2018.
Well-written code is a work of art. Always has been, always will be. A programmer pulls a thought pretty much out of nowhere and transforms it into a working idea that can be used by others. It’s abstract expression made real. Computer programming requires a depth of creativity and discipline of logic that is hard to find elsewhere. Maybe architecture and the theoretical sciences come close, but computer programming stands apart. Computer programming is special and I love it!
Thus, I am always interested to learn a new programming language. Every programming language offers a new way to expand my thinking. It’s always time well-spent.
Recently I decided to learn COBOL, for no other reason than the fact that there are a lot of mainframe installations out there and it’s the mainstay language for many of them. Mainframes are critical to the operations of many banks, insurance companies, transportation systems and governmental agencies. Learning COBOL been on my Bucket List for while. So I took the plunge.
| What does the term, COBOL stand for? Common Business-Oriented Language |
The first thing I needed in my journey to learn COBOL was an IDE. I am a big supporter of coding in an integrated development environment (IDE). I like being able to write, test and run code all in one place. Also, I find the support features that an IDE provides, such as visual code structure analysis, code completion and inline syntax checking, allow me to program and debug efficiently.
The IDE I found is an open source product, OpenCobolIDE, as shown below in Figure 1.
Figure 1: OpenCobolIDE provides many tools to make COBOL programming easier
OpenCobolIDE allows me to write, compile and run code all in one place, without having to go out to the command line, which is very good because I am the world’s worst typist!
| Programming COBOL in the Real World
When it comes time to do real world, mainframe based COBOL programming, you will do well to look at the IDEs that IBM provides. The tools are designed to go about against its Z Systems environment seamlessly as a set of Eclipse plugins. These tools allow COBOL developers working in the mainframe environment to code, debug, unit test and do problem determination. |
Having set up my development environment, the next thing to do was design my first project.
Typically when learning a new language I like to start small and work my way up into more complex tasks. I did the typical Hello World program. You write the code to display string, Hello World, to standard output. Every developer does one. It’s a trivial program, as shown below in Listing 1
Listing 1: A Hello World program written COBOL
While the Hello World required that I learn about the concept of a DIVISION, which I will discuss in detail below, in the scheme of things, I needed more. I wanted to write a program that would force me to learn how the following:
Thus, I create the program, RESEL-WORLD, as shown in Listing 2, below.
Listing 2: The COBOL program, RESEL-WORLD accepts and manipulates users’ input
The user input and program output is shown below in Figure 2.
Figure 2: The RESEL-WORLD COBOL program asks a user for first name, last name and age, then manipulates the input
The RESEL-WORLD program taught me a lot about COBOL and how to program in the language. In the spirit of giving, I am going to share what I learned from writing the program. However, before I delve into the details, please be advised that the information I’m presenting is but a high-level overview. As with any programming language, it takes about a year of consistent coding to become an entry-level professional. My sincere hope is that the information I provide gives the reader a good sense of the language and enough motivation to want to learn more. As I mentioned early, learning COBOL is worth the effort.
The most important thing to understand when learning COBOL is that is very strict in terms of code layout. The layout rules relate to the use of columns and characters. Also, the format uses a hierarchical outline structure. The following sections describe the details of layout specification.
In COBOL a line of code can be no longer than 80 columns in length. You can also think of a column as a character. Columns are segmented into groups, with each group serving a particular purpose. The columns groups are as follows:
Columns 1-6 is the group in which a programmer defines a sequence number. A sequence number is similar to a line number.
Column 7 is reserved for special characters. Asterisk (*) starts a line of comment. Hyphen (-) indicates line continuation and slash (/) is form feed.
Columns 8-11 is called Area A. Area A is the group of columns in which you start a DIVISION, PARAGRAPH and SECTION. We’ll talk more about these below in the next section, The COBOL Structural Hierarchy.
Columns 12-72, also known as Area B, is where you write code statements.
Columns 73-80 is reserved for developer use. You can write poetry in there, if you so desire. Just make sure you don’t go past column 80.
Figure 3, below illustrates the columns grouping described above.
Figure 3: Each column in a COBOL file is specified to serve a specific purpose
COBOL is a compiled language. When it’s time to run your code, the compiler will check to make sure that the code layout adheres to the column grouping specification. If there is a violation, the compiler will error.
| Understanding Performance in Terms of Compilation and Compilers
COBOL is a compiled language, as are others such as Java, C# and C++. Compilation is the process of taking textual source code and converting into a binary format that the computer can understand. There’s a lot of variety in the COBOL world when it comes to compilers, particularly when it comes to cost and performance efficiency. For example, how a compiler orchestrates the way numbers are loaded and then computed in memory matters a lot! A good compiler will make code run really fast. IBM has been in the COBOL business for a long time and has a keen understanding about making compilers that are fast and cost-effective. For example, the z14 System compilers improve efficiency by taking advantage of almost 24 new low-level instructions. As a result, computation speeds improve dramatically. As you work more with COBOL, you’ll come to appreciate the power that an industry-leading compiler such the IBM Z Systems brings to the coding experience. |
The concept behind the file format of a COBOL program is based on the structure of a document outline, with a single top-level heading followed by subordinate levels. The organizational units that make up the hierarchy are PROGRAM, DIVISION, SECTION, PARAGRAPH, SENTENCE, STATEMENT and CHARACTER. Figure 4 illustrates the hierarchy.
Figure 4: The structural hierarchy of COBOL program entities
PROGRAM is the root level of the COBOL code hierarchy. PROGRAM represents the unit of code that the mainframe job scheduler, the JCL, loads into memory to run. The program is identified by the PROGRAM ID statement in the IDENTIFICATION DIVISION. The IDENTIFICATION DIVISION is part of the next level of hierarchy that descends from PROGRAM. PROGRAM must contain the IDENTIFICATION DIVISION.
There are other DIVISIONs that can be included. These other DIVISIONs are: ENVIRONMENT DIVISION, DATA DIVISION and PROCEDURE DIVISION, which occur after IDENTIFICATION. Also, these subsequent DIVISIONs must appear in the order defined: ENVIRONMENT, DATA and then PROCEDURE. You can read a detailed description of each DIVISION here. DIVISION names are terminated with a period, for example,
DATA DIVISION.
The next organizational level down from DIVISION is SECTION. (Please see Figure 5.) Each DIVISION will contain SECTIONs that are special to it. For example, the DATA DIVISION can contain FILE SECTION, a, WORKING-STORAGE SECTION and/or the LINKAGE SECTION. You can read the details about DIVISIONs and SECTIONs here.
Figure 5: A COBOL program is structured in a hierarchical manner
A SECTION name is terminated with a period, for example:
WORKING-STORAGE SECTION.
A SECTION contains zero or many PARAGRAPHs. (Typically a SECTION will have at least one PARAGRAPH.) A PARAGRAPH name is terminated with a period, similar to a DIVISION and a SECTION.
A PARAGRAPH contains SENTENCEs or STATEMENTs. A SENTENCE is a group of STATEMENTs. Usually a PARAGRAPH contains one or more STATEMENTs. A STATEMENT is a line of execution. A STATEMENT is made up of CHARACTERs. A CHARACTER can be an alphanumeric symbol or a special character. CHARACTER is at the bottom of the COBOL code format hierarchy.
COBOL allows you to declare variables in a variety of data types. Special to COBOL is the concept of a variable level as represented by a level number. A level number defines a variable in terms of being or having a parent variable. (You’ll see more about this in the section, COBOL Supports Hierarchical Data, later on.)
The following statement, which will be declared in the WORKING-STORAGE SECTION of the DATA DIVISION, is the declaration of a variable, WS-QUANTITY. WS-QUANTITY will hold a numeric value.
01 WS-QUANTITY PIC 9(2) VALUE 12
The expression above above declares a variable , WS-QUANTITY to be a two-digit value with an initial value of 12. Also, the variable is declared to be at level 01, which is the highest level possible. The variable has no parent. The way we know that WS-QUANTITY is a variable for a two-digit value is due to the PIC (Picture) clause. You can think of a PIC clause as a type declaration. Table 1 provides a high-level description of the various PIC clause symbols.
| Symbol | Description | Example Declaration | Sample Value |
| 9 | A numeric value where each occurance of 9 represents a digit | 99 or 9(2) | 35 |
| a | Alphabetic | aaa or a(3) | “Bob” |
| x | Alphanumeric | xxxx or x(4) | “R2D2?” |
| v | Implicit decimal* | v(3) | .175 |
| s | Sign | s9(2) | -76 |
| p | Assumed decimal* | p9 | .6 |
Table 1: Symbols for a PIC Clause
*COBOL has a special way of storing numbers such that the numeric conversion take place when the characters are loaded into memory. For example, a variable with a PIC of p9(2) will be stored as 56 and loaded into memory as .56. For a more detailed discussion of usage and the PIC clause, go here.
Storing data in a named group of variables makes programing easier. For example, in the C programming language, we can use a struct to name a group of variables, as shown below in Listing 3.
struct User {
char first_name[10];
char last_name[10]
int user_id;
};
Listing 3: A simple structure in C
One of the real nice things about COBOL is that it allows you to organize data according to a named group of values similar to that of a C struct. What’s interesting in a historical sense is that COBOL has had this capability well before Kernighan and Ritchie created the C programming language.
The way COBOL accomplishes named data grouping is with construct called a record. Figure 6 below shows a graphical representation of a record named, WS-USER.
Figure 6: The record, WS-USER contains subordinate variable, WS-FIRST-NAME, WS-LAST-NAME, WS-AGE
The code snippet in Listing 4, which is from the RESEL-WORLD program at the start of this article, shows the COBOL code that declares the record, WS-USER.
Listing 4: The record, WS-USER contains 3 subordinate elements, WS-FIRST-NAME, WS-LAST-NAME, WS-AGE
Notice in Listing 4 above the there are two levels of data variables in play: 01 WS-USER and 05 WS-FIRST-NAME, 05 WS-LAST-NAME, 05 WS-AGE. The variable, WS-USER is a parent to the subordinate, level 05 variables, WS-FIRST-NAME, WS-LAST-NAME, and WS-AGE.
Assigning variable to a numeric level is special to COBOL. The declaration logic is that a record variable (the root of the record) has a level number of 01. Levels 02 to 49 are used to declare subordinates elements. (You can think of a element as a member of the record.)
You use the reserved word OF to access an element of a record. Listing 5 below shows two statements. The first asks a user to input their first name. The second statement takes the value entered and assigns it to the element, WS-FIRST-NAME of the record, WS-USER.
DISPLAY "What is your first name?". ACCEPT WS-FIRST-NAME OF WS-USER.
Listing 5: Accessing an element of a record in COBOL
It make sense that records were part of COBOL from the start. COBOL was intended to be a language used for business application. Businesses have been organizing values according to named groups since before the customer form was created. COBOL was designed to reflect business needs. It’s interesting that the fundamental needs of businesses in terms of logical data structures have been surprisingly consistent over time.
Expressing a statement in COBOL is very similar to the way a person speaks naturally. Listing 6 below shows the statement that adds together the values of the two variables, WS-AGE-DELTA and WS-AGE (from the record WS-USER), and stores the result in the variable, WS-NEW-AGE.
ADD WS-AGE-DELTA WS-AGE OF WS-USER TO WS-NEW-AGE.
Listing 6: Adding two integers, WS-AGE-and DELTA WS-AGE, then storing the sum in the variable, WS-NEW-AGE
Notice that the statement in Listing 6 is very close to saying, “Add WS-AGE-DELTA and WS-AGE from WS-USER to WS-NEW-AGE.”
What I find pretty astounding is that many modern programming paradigms try to capture the ease of natural language expression that COBOL has had for years. What comes to mind immediately is the use of natural language expressions in the NodeJS Chai package. Chai is used in NodeJS unit testing to express BDD assertions. Here’s a Chai expression that checks a variable, myVar to assert that it’s a string.
expect(myVar).to.be.a('string');
| FAST FACT: Is COBOL a case-sensitive language?
No. COBOL will consider a variable named WS-FIRST-NAME to be the same as one named ws-first-name. |
For programming language to be really useful, it needs to support DRY. DRY is an acronym for Don’t Repeat Yourself. Most programmers don’t want to write the same code over and over again, nor should they. What you want to do is to encapsulate code in to a single area of execution that can be called repeatedly. Being able to program by DRY is common for most languages—BASIC, Java, C#, Javascript, Python, to name a few and … you guessed it: COBOL.
COBOL supports code segmentation and reuse in a variety of ways. One way is through linkage, calling routines between programs. Another way is to segment code inline under a PARAGRAPH in the PROCEDURE DIVISION and then call that PARAGRAPH. (See Figure 7.)
Figure 7: You can make code reusable in COBOL PARAGRAPHs
The little RESEL-WORLD program I wrote uses the inline PARAGRAPH technique. Before I go in to the details of calling code encapsulated into a PARAGRAPH, it’s important that you know that in COBOL the entry point of execution into a program is the first line of code after the declaration of the PROCEDURE DIVISION, as shown below in Listing 7.
Listing 7: COBOL supports reusable code by using PARAGRAPHS to segment execution
Notice the PERFORM statements right after PROCEDURE DIVISION. These four statements are calling PARAGRAPHs. (For those of you familiar with JavaScript, you can think of each PARAGRAPH as a function definition.) Essentially what the code is saying is, “Execute the code at the paragraph named, GET-DATA, then perform the code at CALC-DATA, SHOW-DATA and then FINISH-UP.”
Encapsulating code into STATEMENTs within a PARAGRAPH prevents code from become spaghetti that is hard to maintain. Also, the encapsulation enforces the sensibility of DRY. Any language worth its salt needs to support some sort of encapsulation. COBOL meets the need with room to spare.
The more I learn about COBOL, the more I like it. The language continues to evolve to meet the needs of our fast-changing times, with revisions as recent as 2014. Since its inception there have been a dozen enhancements to COBOL including a continuing stream of formal standards.
Today’s COBOL supports modern programming paradigms such as object orientation. The IDEs have grown to keep pace with the demands of modern users. And, given the immense installation base out there, there is a lot of money to be made doing COBOL programming. To quote Wikipedia:
In 2006 and 2012, Computerworld surveys found that over 60% of organizations used COBOL (more than C++ and Visual Basic .NET) and that for half of those, COBOL was used for the majority of their internal software.
What’s even more amazing to me is the cleverness of engineering the language has promoted. COBOL developers addressed problems in the past that still vex many today. We modern developers can fall prey to thinking that before we came along, there was no cool stuff. It’s like the young, aspiring guitarist who thinks thinks he has his chops down and can shred to the top of the heap. Then one day his music teacher gives him a 1936 recording of Django Reinhardt playing guitar, and playing with only two functional fingers on his left hand! At that point the young artist realizes that virtuosity transcends era and that creativity has no bounds. This is what it’s like for me as I learn COBOL. It’s a beautiful, expressive language that was cool then and is very cool now. Learning it is making me appreciate how much amazing thinking went on back then and continues to emerge. There’s a lot of rich opportunity at hand to make great code for mainframes using COBOL. It’s only a matter of mastery, creativity and discovery.
If I have seen further it is by standing on the shoulders of giants.
—Isaac Newton, Letter to Robert Hooke, February 5, 1675
Learning any programming language is hard work. Years ago, when I wanted to learn a new language, I would get a few books on the topic and then hunker down to spend the hours necessary to absorb the required knowledge. If I had the time, I might take a course. Maybe I had an expert friend I could call up when I hit a wall or needed real-world guidance.
We’ve come a long way since that time. The internet is a game-changer. Online tutorials, videos and interactive, digital books make things a whole lot easier than in earlier times. My experience learning COBOL testifies to the value of these modern benefits. But still, as I’ve been learning, I’ve hit more than one wall that required professional guidance and review. Allan Kielstra and Roland Koo at IBM (who also wrote this great article) provided the expertise that made my learning COBOL not only less difficult, but also fun. I am in their debt. Their enthusiasm for COBOL was infectious. Their commitment to technical excellence is inspiring.
There’s a piece of conventional wisdom that runs rampant in the tech-o-sphere: You can’t teach an old dog new tricks. Supposedly, once people become enamored with a technology and use it for a long period, they are loathe to adopt new ones. We’ve all heard it, and some believe it. Me? I not buying it any more than I believe that you can’t teach a new dog old tricks. From where I stand, a good trick is a good trick, period!
So let’s talk about a really good trick: mainframe computing. It’s technology that is used by many business sectors—airlines, insurance companies and banks. And, it’s been designed from inception to support interaction from hundreds—if not thousands—of users. Yet, many perceive the trick to be old dog, that’s it’s not applicable to modern computing.
Actually, the opposite is true. It’s an important trick that should be used by any dog, new or old. It’s a question of adaptability: how to make the trick useful in any circumstance.
Let’s put the trick-and-dog analogy aside and talk directly. Companies are sitting on a boatload of mainframe assets that can and should be leveraged for use in this world of mobile devices, APIs, microservices and speed-of-light DevOps deployment practices. The question is, how?
To make mainframe technology viable in modern cloud-based paradigms, it must satisfy four conditions. The technology must:
IBM Z, a mainframe-based technology, has done a great job of meeting these conditions. The various products and toolsets that IBM provides for IBM Z technology make working with IBM Z in today’s cloud computing environment surprisingly easy. Let’s take a look at the details.
REST is fasting becoming the standard by which information is exchanged between systems over the internet. As mobile devices and the internet of things grow as the means by which users interact with applications, the data these applications require is accessed in real time using REST. Today, REST is a first-class citizen in IBM Z technology. A key feature of IBM Z Application Discovery and Delivery Intelligence (ADDI) is the capability to discover applications running on IBM Z hardware and identify them as potential REST API candidates. Support for REST makes IBM Z applications consumable to a wide variety of computing devices. Also , supporting REST means that IBM Z applications integrate easily with standard DevOps testing and deployment practices.
ADDI provides the support for REST that allows IBM Z to be an agnostic resource on the internet
In the world of modern, cloud-based computing REST is the lingua franca of information exchange. Supporting REST makes IBM Z data and transactions available to the modern digital ecosystem.
The IBM Explorer for z/OS, which is part of the Application Delivery Foundation for z Systems (ADFz) tool suite, allows COBOL and PL/1 developers to code in the Eclipse Integrated Development Environment (IDE). Eclipse is well-known to a broad range of developers working in languages such as Java, PHP, Node and Python. Using Eclipse to do IBM Z programming under COBOL and PL/1 allows developers to have a shared programming experience and increase their productivity by an average of 15 percent.
Although the actual lines of code the developer creates will be particular to the developer’s programming language of choice, the unit testing and debugging experience Eclipse provides is similar regardless of language. A developer uses Eclipse to unit test code as it’s being developed. If the test fails, the developer sets breakpoints and starts the debugging session to inspect the code at runtime. The way code-test-code-debug is conducted in Eclipse is consistent. Thus, the skills acquired using the Eclipse IDE are easily transferable between languages.
Having all developers work in a common development environment creates a shared experience—an important step toward breaking down organizational silos. Reducing the silos that isolate knowledge workers from each other so they can collaborate better and faster across teams and geographies, combined with deploying code in an automated CI/CD pipeline, significantly speeds a company’s ability to get code to market while also making collaboration easier across teams and across geographies.
For a software development framework to be viable in a CI/CD infrastructure, it must support a wide variety of testing capabilities. In addition, testing needs to be verified via automation. As mentioned above, IBM Explorer for z/OS allows developers to create unit tests on COBOL and PL/1 code. Once unit tests are created, they can be automated for use throughout the development pipeline. Test reuse via automation speeds up the deployment process significantly.
However, testing at the code level is not enough. As code moves through the modern deployment pipeline, it must be subjected to load and performance testing. Test engineers accustomed to using mainstream tools such as Apache JMeter or Rational Performance Tester will have little problem using IBM Z Workload Simulator for z/OS to do load and performance testing. The concepts behind load and performance testing are similar, regardless of the language and environment driving the development process. For example, Workload Simulator for z/OS allows test engineers to emulate hundreds, if not thousands, of users to exercise the software under test. Being able to create stress on a system in a controlled, reportable manner is critical for getting good performance testing metrics.
Basic to DevOps best practices is the notion of releasing quality software quickly and incrementally. Instead of waiting months to do a release that has many features, the DevOps practice is to release one feature at a time, as quickly as possible. In the DevOps world, release cycles are no longer a matter of weeks and months, but rather days, if not hours. To achieve the fast release cycles emblematic of DevOps, management of the deployment pipeline must be automated.
However, when it comes to mainframe assets there is a hitch in terms of deployment units. In the microcomputing software environment, deployment units typically take the form of an executable file, supporting dependency file or script file such as .exe, .jar, .js or .py. Also, given the rise of container technology, the deployment unit can be the container itself. In terms of IBM Z, the deployment units are the file and the job.
Engineers at IBM Z have taken the clever approach to managing asset deployment. File and jobs have become abstracted deployment units accessible via the IBM Explorer for z/OS Atlas APIs. Thus, IBM Z components can be treated as any other artifact in the deployment pipeline. DevOps engineers use the APIs exposed by Atlas to cancel a job or delete an associated file. Standardizing direct access to the IBM Z deployment pipeline using REST APIs gives DevOps engineers the flexibility required to get working code released quickly.
DevOps engineers not only can automate the pipeline, but should ad hoc intervention be required, they also can use the Atlas APIs to make the changes required at a moment’s notice. Combining the Atlas APIs with ADDI’s ability to collect, correlate, calculate and produce mission-critical application and testing health metrics, as well as trend analytics, makes IBM Z a first-order participant in the world of cognitive DevOps.
One of the great things that the internet has done is to create a standardized way by which any computing system can interact with another. As long the system supports REST, regardless of whether it’s a smartwatch, a cellphone or a large-scale computing environment such as IBM Z, it’s a leveragable asset on the internet. And, if the system meets the conditions that define DevOps best practices, it can be managed with the degree of control and predictability required to participate in the breakneck release cycles that are part of modern software development. It really doesn’t matter whether the system is new or old, as long as it can adapt to present demands. Adaptability is key to longevity, no matter the age of the dog or the trick.
The architectures I like best are the ones that are built to scale. I can’t tell you how many time I’ve run across applications that have hit the wall when the request count goes up from hundreds to thousands of simultaneous users. The next thing you know the CPUs start to peg, memory get maxed or overall availability just magically goes away. All that’s left is the dreaded status code: 500 Internal Server Error.
Those of us who have made distributed computing a way of life spend a lot of time making sure that our applications scale up. We’ve found the pursuit to be costly. As result, some of us have turned the entire problem over to managed service providers. Paying a bill at the end of the month to ensure 24/7 availability is a whole lot better than taking a call at 3 a.m. demanding to know why the inventory system in Toledo has gone down. Whether we choose to take care of our own metal or use an MSP, we understand that an architecture that cannot scale has little value.
Surprisingly, one of the best architectures around is one that is used by 90 percent of the world’s largest airlines, 70 percent of big retailers, most of the banks worldwide and all of the largest insurance companies. Where is this architecture to be found? In mainframe computing. Yep, you got it: Big Iron.
Mainframe computing in general and IBM Z in particular have a computing architecture that is built to support usage by thousands, if not millions, of clients. Since their inception, these systems have been designed to provide high-availability, large-scale, multi-processor computing in a cost-effective manner. Whereas a typical data center filled with racks of commodity hardware will suck thousands of dollars a month in electricity, a single IBM z System that processes 2.5 billion transactions a day uses about as much electricity as the clothes dryer down in your laundry room. (2.5 billion daily transaction is the equivalent of 100 cyber Mondays every day of the year!)
In terms of high availability, mainframe uptime is measured in years. There are systems out there that have been running continuously for 10 years and are still up today. This is no small accomplishment. Think about it: Imagine what it takes to have enough redundancy built into an architecture to never have to reboot, no matter how weird any of the applications get. Memory and application isolation have been built into the technology since the start. In a way, we’re talking cloud before the cloud.
Memory isolation and application redundancy have been key features of mainframe technology since inception
It’s an amazing technology, one that more developers, particularly those of us working on the forefront of application development should leverage. The question becomes how to benefit from its capabilities and drive business value from this asset with its treasure trove of data. The first step is to understand the difference between a system of interaction and a business critical system.
In the world of modern, distributed computing, application logic can be represented in a variety of ways. For example, it’s quite possible to check on the status of flight information by using a mobile app, going to an airline’s website or by calling the carrier’s toll-free number. The logic needed to find and deliver flight information is in a central location; the means to access that information is distributed. There are two systems in play—one is called the system of interaction and the other is the business critical system. The system of interaction is the system that interacts with the end user, collecting data to send as output. The business critical system is the system that serves as the repository of common application intelligence. The concept is akin to the thinking behind client-server architecture, but it’s a bit more in that it’s system based. Think of a phone system, cable TV system or online games as a system of interaction. Think of a rack of servers, the cloud or a mainframe as a business critical system. The overall digital experience is the aggregation of its components.
This may seem like an obvious separation, but there’s a bit more to it, particularly when considering large hardware such as IBM Z and the like. Being a business critical system is more than being a simple data store. A business critical system can contain all the data necessary to perform complex machine learning activity upon a particular dataset submitted to it, weather prediction or medical diagnosis, for example. Of course, the business critical system has data, but the system also has algorithmic and parallel processing capability far beyond a standalone PC or cell phone, the components that are typical in a system of interaction.
Business critical systems, particularly those that have been in service for decades, become very powerful when they support common standards of digital interaction—APIs based on REST, for example. Back when distributed computing was limited to a small part of the population, a proprietary approach to system interaction was acceptable. But, the rise of the internet changed it all. Today we have billions of people online, on some part of a global network. Thus, common standards such as HTTP and REST have emerged to make interacting with software systems worldwide easier. Companies such as IBM understand the importance of supporting the ways of the internet. Thus, a product such as z/OS Connect EE allows developers to connect to a Z System installation and design an API around the code in that installation. Using z/OS a developer can wrap a REST API around some COBOL code. Having this type of capability opens up a world of possibilities.
As the world of mainframes gets more attention from those making mobile apps and aggregating services into the cloud, a whole new level of software development emerges. Using a product such as z/OS Connect EE to create REST APIs combines the world of Big Metal with the world of big data. Those of us in DevOps understand that breaking down silos and fostering collaboration is what our work is about. Bringing together the skills and experience of all those who are dedicated to making quality software, from COBOL programmer to engineers who embrace infrastructure as code, is essential in the world of modern computing. The more people and systems we have working together effectively, the better will be both our software and the world we make.
IBM recently GA’ed perhaps its biggest zSystems mainframe release in the last decade or longer. The z14 mainframe from IBM has also been designed from the ground up to be DevOps-empowered. From APIs and other integrations to language support, the z14 is IBM’s most advanced mainframe ever. I had a chance to speak with Hayden Lindsey, VP of DevOps Enterprise Systems for IBM, about the z14.
Hayden gives us the low-down on how the z14 has been designed to work in hybrid environments. It also works well with modern CI/CD tools such as Urban Code and others. Additionally, APIs give it the ability to integrate with many different environments, tools and apps. It is truly a mainframe for today and beyond. Hayden does a great job of describing all of this.
As usual, the streaming audio of our conversation is below, followed by the transcript of our conversation. Enjoy!
[su_pullquote align=”right”] [su_custom_gallery source=”media: 117632″ width=”130″ height=”130″] Hayden Lindsey
Hayden Lindsey, Vice President & Distinguished Engineer – DevOps for Enterprise Systems. Hayden is responsible for the development, product management, sales enablement, and services of this portfolio. He has worked at IBM for 28 years and has held a variety of positions, including software developer, product architect, product manager, co-leader of the Eclipse tools platform, and director of the WebSphere Studio/Rational modeling and construction tools. He has a bachelor’s degree in Mathematical Sciences from the University of North Carolina at Chapel Hill.[/su_pullquote]
Alan Shimel: Hi, everyone, this is Alan Shimel, devopsy.kinsta.cloud, here for another DevOps Chat. Our guest on this episode of the chat is a good friend of ours, Hayden Lindsey, VP, Enterprise DevOps at IBM. Hi, Hayden, welcome to DevOps Chat.
Hayden Lindsey: Hi, Alan. Good to be with you again.
Shimel: Good to have you here. Hayden, it’s a big day—big day for the folks over at the IBM Enterprise team. It’s the general availability of z14 today, isn’t it?
Lindsey: It sure is. Yes, we announced this back in the middle of July and we’ve now been making our products available. A lot of mine were available the latter part of last week, and now the box itself, GA, today. So—a big day.
Shimel: Fantastic. And Hayden, just—you know, our audience being so focused on DevOps, they may not realize the full scale of what the z14 release is about. What it is, it’s a combination of hardware—as you mentioned, a new box. There’s a ton of software, there’s a tone of integration—it’s just chockfull of all kinds of goodies. Can you give us kind of a—at a high level, the scope of z14?
Lindsey: Sure. And I think what you see is the result of pulling software, both middleware and tools, from the various divisions that we used to have like Rational and WebSphere and moving those into a single z or mainframe focused brand.
And so, in contrast to most prior hardware releases where you’d be hearing about the speeds and feeds and, you know, the z13 had the fastest processor of any box on the planet—this time, the focus is more on the entire solution that involves both hardware as well as software and tools. And so I think—we worked on that, we did design thinking, we had sponsor users, we did the whole nine yards form the hardware up through the operating system, middleware, and tools, and I think that made a big difference.
And so this time, for example, we have something called pervasive encryption. We all know how important security is, we all hear about these breaches that are happening—Experion being the most, or Equifax being the latest one. And, of all of the records that have been stolen, and I think it’s something like 9,000,000,000 since 2013—only 4 percent are encrypted. And this new mainframe lets you encrypt 100 percent of the data with no impact to performance. So, that’s just—that’s one example. Clearly, there’s hardware stuff there, but there’s also middleware that had to be involved in making this solution. So, that’s an example—it’s really cool.
Shimel: That’s actually a big example, Hayden, because you know what? Encryption has been around—I mean, I spent 20 years in the security space and encryption has been around for most of that. The problem was the hit you took on performance when you encrypted and then had to decrypt and re-encrypt just the latency and—
Lindsey: And the programmer had to do it. [Laughter] Or the database administrator had to set it up for the database, but then what about when it was being used in the application? And so that was—this is what is being solved, where you don’t have the time inhibitor of people doing it, and then you also don’t have the performance impact where it makes a system unusable.
So, I think this could be really groundbreaking for the companies and the government agencies around the world that deal with sensitive data.
Shimel: Sure—and we all do now. Hayden, you said something else, though, that I wanted to key in on, and that is sort of moving away from the speeds and feeds, if you will, metrics of—you know, to measure the value of anything. And I worked with a woman once, and she used to call that bag diving. Right? You know, when you gotta start getting into the speeds and feeds to show the value of your product, you’re diving into your bag instead of talking the true value.
Lindsey: Right.
Shimel: So, I’m happy to see that that’s kind of the path that IBM is taking, talking about the value, but I also think it’s a result, as you said, of so much being added into here, right, from the design thinking around the entire release, if you will, to the integration with things like cloud, with the DevOps entire tool set and methodologies that IBM and others have developed.
This is so much more than just, “Oh, we put—this is the next iPhone with the faster processor.” This isn’t the next z with the faster processor, it’s much more than that. As I think I—maybe you said it earlier or maybe we talked about it off-mic. This is probably the most substantive release for z in at least a decade.
Lindsey: I think it is, and of course, the market will tell us if our aspirations come to bear or not, but I certainly think the encryption element is just—well, I think it’s phenomenal, and it could be very, very, very significant for virtually any company out there.
And then we’ve got more on machine learning and analytics so that you can do in transaction analytics rather than doing it after the fact and having to try to back out or just write off fraudulent transactions. And, of course, we’ve done a lot with our DevOps portfolio compilers. But I really do think, maybe, since we put Linux on the platform, it’s that significant.
Shimel: Wow. Huge. So, Hayden, we are about DevOps here, so let’s focus in a little bit. I mean, one of the nice things is, I’ve seen some briefings around how you can use the new z14s with the IBM Enterprise DevOps solution and really sort of leverage—I don’t wanna use the term bimodal multi-speed nonsense. Leverage integrating your z system with cloud systems into one coherent business strategy. How’s that sound? [Laughter] So, talk about it.
Lindsey: Yeah, so one of the three themes for the release is open and connected, and that obviously ties into a hybrid cloud strategy. It ties into, you know, a digital economy or digital transformation. It ties into API economy—whichever angle you want to take, you have to be able to make the data that resides on the mainframe, which is estimated that roughly 80 percent of the Enterprise data in the world resides on and is managed by the mainframe, and about 70 percent of the business transactions.
And so, if you want to, though, make that available to your new iPhone X app, as we saw introduced yesterday—you know, or 10 or whatever they’re gonna call it.
Shimel: Mm-hmm. Yeah, they’re calling it 10.
Lindsey: They’re calling it 10 using a Roman numeral.
Shimel: Yeah.
Lindsey: But, you know, if you’re gonna do that, you have to make it easy to access the data and transactions, find those transactions, understand them, because very likely, they were written by somebody who’s not around any more or had moved on to other jobs. And so, you need to have tools that make API enabling and opening up and connecting to the mainframe easy and efficient, and just as easy, just as efficient as you would if you were opening up something on the Linux platform—although, of course, you have Linux on z—or a Windows box or anything else. So, that’s—it’s essential that we allow z to participate in the broader digital economy, API economy, and that’s what my team has been focusing on, along with other teams in IBM.
Shimel: Sure. So, now, Hayden, in that same vein, you know, there’s still that mainframe attitude where, you know, we don’t open our mainframe up to all of these APIs, and maybe we don’t want to hook into our systems of engagement, you know, up in the cloud or what have you.
For clients listening out there—and a lot of our listeners are Enterprise class, they work in Enterprise class organizations—how important is it that they really consider opening up, you know, their mainframes to this API economy, as you called it?
Lindsey: Well, I think increasingly, certainly, the IT and line of business executives, the clients with whom I work, are realizing they have tremendous assets that they could leverage if they were to make them available. Nobody—at least no companies I know of, or government agencies, have the resources to just recreate everything from scratch. And, I think if you look back at what the analysts were saying back in the SOA era, they said, you know, it was five times faster to create a service leveraging something that exists that already does the same thing or something very similar than it is to create it from scratch—not to mention the fact that, if it already works, you’re likely to have higher quality as well. And I think the exact same analysis would prove to be true now that we’re talking more about RESTful APIs.
So, I think more and more clients realize they need to expose that value. The key is, just, it’s sometimes daunting when you’ve got—you know, the estimate is like 250,000,000,000 lines of COBOL, not to mention PL/I and Assembler and 4GLs and so forth. It’s getting your arms around applications that were probably created in a more monolithic, less well-structured fashion. You need to do refactoring. You need to discover what is there, and then put together a plan for doing that API enablement.
But I think people know they need to do it, it’s just, sometimes it’s viewed as a daunting task and they have to figure out how to get started, and we have something called the digital transformation model that we’re rolling out to help our clients put a plan in place.
Shimel: Got it, and I agree. So, you’re just setting me up for my next things here, but—so, that’s the biggest question we hear at devopsy.kinsta.cloud, Hayden, which is—hey, this is all great. How do we get going? How do we roll this? How do I get executive buy-in? How do we—is there a one, two, three? Is there a blueprint I can follow.
So, z14, let me throw it right back to you—where do we start? How do they get moving with this? How do they—and specifically, how do they start incorporating the DevOps and API stuff into this?
Lindsey: Yeah, it’s the question that I hear from virtually every customer I talk to, and I’m getting ready to do a tour in Europe next week, and I’m sure it will come up in every single meeting. And the fact is, every customer is different, so there’s no single answer. And, within any given client, they have applications that span the gamut from nearing end of life to ones that are under active or brand new development. And so, what we’ve—but, of course, there are patterns and similarities across clients and application types, and there are best practices that we have gleaned by working with clients through the years, and that’s where I mentioned this thing called the digital transformation model.
We’ve shifted to this way of working with our clients to focus on applications rather than one answer for the Enterprise—hey, you should start with deployment automation, or you should start with test automation. That answer should not be an Enterprise answer, it should be for this suite of applications because of what their current state is, the as-is state, and what you desire the to-be state to be. If you just need something to run efficiently and you’re gonna replace it in a few years or retire it, all you want it to do is run efficiently.
So, use our latest compilers, but you know, the fact of the matter is, putting in a fully automated pipeline is likely a wasteful effort. Whereas, on the other end of the spectrum, for the things under very active development, that’s where you want to do DevOps in its fullest, and really, it’s just a matter of what steps do you take to get there.
And so, this model that we’ve come up with has four levels—the first one I described about, just do your compiling, is run—just run it. Just run it efficiently. The next is maintain, the next is expose, the next is evolve. So—I mean, and I don’t know if we’ve got the best names in the world, but the fact is, you really need to think about, for the business, what degree of DevOps adoption do you need for a given application or application suite. So, that’s the more, I think, realistic answer that we’ve been using with clients this year and a tool that we’re using that’s different from what we’ve done in prior years.
Shimel: Excellent. So, Hayden, I mean, just to illustrate this, I remember back at Interconnect last year, you know, we had the whole little DevOps TV booth set up, and I was interviewing some IBM clients. And one of them, it was a bank, and I forget if they were from Poland or the Netherlands now, or maybe even Australia. They were IBM’s z System customer and they had just updated, I wanna say, from COBOL 4 to 5—maybe it was 5 to 6, I don’t remember now.
But they had just updated their version of COBOL, and it made a huge, huge difference for them in terms of performance and—
Lindsey: Yeah. Yeah, exactly. You know, we’ve—one of the things that our compiler team does every single release, and this starts from the very earliest days of, we were just imagining what the new mainframe or new power platform is going to be. We were working hand in hand with the hardware folks.
And, you know, with COBOL specifically, where something like 70 percent of the world’s business transactions are running COBOL, obviously, performance does matter, and so this is where you can get into a little bit of feeds and speeds. Mainly because, not because—some of the numbers aren’t eye popping, but it’s because you can be more responsive for your clients. But with some compute intensive apps, they run 16 times—not percent, times—faster on the latest compiler, which is 6.2, versus 4.2, which may be where that client was coming from. So, that’s skipping the version 5 things, but we still have the majority of our clients on the 4.2 version.
So, it’s tremendously faster, but this is by doing this joint development. You know, if you put new instructions or new vector support in the hardware and the compiler doesn’t generate these new instructions, well, then, obviously, no client is gonna be able to take advantage of it unless they want to write their own assembler. So, you know, our compilers are delivering tremendous value, and I gave you a COBOL example, but you get similar results with PL/I and with our automatic binary optimizer that basically just does the, tries to do the work that the compilers do, but just working on binaries rather than having you go back to the source.
So, it’s a big deal. You know, and the less—it’s maybe a bit more mundane. Yeah, the stuff runs faster and, as you said, diving into the bag. [Laughter] But, you know, when you’ve got numbers like 16X or 5X or what have you, that’s huge.
Shimel: It sure is, it sure is. So, Hayden, you know, I mentioned Interconnect, too, and as we’re running low here on time, I just wanted to quickly prompt you on that and let our listeners now.
So, this year, Interconnect is part of an even bigger kind of unified conference that IBM has put together called Think—Think 2018, I believe it’s called, and if I’m not mistaken, it’s March.
Lindsey: Yes.
Shimel: Go ahead.
Lindsey: Well, you might see a trend here. [Laughter] Because, you know, Interconnect was the combination of Impact and Innovate. Innovate—I think that’s the name it was. I’ve kinda forgotten, but anyway, we had a Rational conference, we had a WebSphere conference, we had IoT stuff and all that. And now we combine those into Interconnect—at least the WebSphere and Rational stuff. And it got bigger, and now we’re doing something that is more or less IBM-wide and it’s called Think. It’s gonna be the week of March the 19th at the Mandalay Bay in Vegas, so still in Vegas like most of our other conferences have been of late.
And so, I think there’s pros and cons to having a more broad, where we can tell the full IBM story, and that’s the focus, and you’ve got SMEs from all the different areas of the country, be it cloud, cognitive, Zpower, what have you. But I think that’s great, but because you do have those experts, and we will have the breakout sessions or keynote sessions on specific areas, be it DevOps or z or compilers or deployment automation like UrbanCode.
So, there will be still those specialty sessions, and yet you get the big picture. And the one thing that I would like to ask your listeners is, the call for papers is open, and you know, we are shooting for having 100 percent of the talks in the various tracks either be a customer or partner only, or being customer and/or partner jointly with an IBM-er, but we’re just really trying to get away from IBM-er talking head telling a story without a partner or a customer there saying what they’ve done in real life with a given solution in a given area.
So, we’d really love to have as many partner and client nominations as possible.
Shimel: I love that idea. You know, I shared a panel the last two years, Hayden, on Dev—I think they were on, well, it was on DevOps and then SecOps. But, you know, I tried really hard to balance it with—yeah, we had one person from IBM, but we also had an analyst and then we’d have a partner and then a practitioner. And I think that sort of balance gives it, it makes it—
Lindsey: It adds some legitimacy to the whole thing and different perspectives.
Shimel: Yep. Just real quick, for anyone listening, if you go to IBM.com/events/Think, you can get all the scoop there and follow the links to submit. And as Hayden said, if you’ve got something to say and share with people, it—having been to three or four of these, or more, it’s a great platform. So, I’d encourage everyone who would like to participate to do so. And even if you don’t want to speak, but if you want to go really understand what’s happening, whether it be with Watson or z Systems or hybrid cloud or Bluemix or any of those things. You know, if any of those are important to you, I think it’s a great show to attend.
Hayden, we’re about out of time—we’re actually over time, but that’s okay. Thanks so much for taking the time on what is a busy day with the general GA of z14. Best of luck with it. Let’s check in, I don’t know, maybe before the end of the year, and see how—you know, what kind of market reception a lot of these features and capabilities are receiving and we’ll see where the rubber is meeting the road.
Lindsey: Sounds good, Alan, and thanks, again, for your time.
Shimel: Thank you for your time. Hayden Lindsey, VP, Enterprise DevOps with IBM on GA Day for z Systems z14. Thanks, Hayden, have a great.
Lindsey: Okay. Bye for now.
Shimel: This is Alan—thank you. This is Alan Shimel for devopsy.kinsta.cloud. We’ll see you soon on another DevOps Chat.
The word “legacy” evokes mixed reactions from the IT and business crowds alike. For those who depend on legacy systems such as the zSeries and iSeries platforms, it is a way to deliver stable and reliable applications into production. Business stakeholders who may not know the intricacies of the technology often take a view of, “Why fix it when it ain’t broken?”
On the other hand, for those who haven’t experienced the joys and pains of legacy systems , legacy world will continue to defy their logic and assumptions because their view of the world is shaped by the open system world which has completely different characteristics: being more malleable to change and easy to maintain etc. Hence there is this great divide between the legacy and new worlds. Add Agile and DevOps to the equation with the lofty goals of increased release frequency and reduced lead times, we end up with a divide that is too far to bridge.
The likes of Gartner have come up with constructs such as two-speed delivery to help deal with this legacy challenge. In my view, there is a problem with the idea of dual-speed because it is based on a wrong analogy to start with — that of allowing vehicles to move at different speeds depending on whether you are on the motorway or a city road. That analogy works well for traffic management scenarios but not for complex IT architectures we see in the enterprise. The dependencies across the systems in the enterprise landscape are just too intricate and too tightly coupled to be defined using such a binary logic. In my experience, I find that many Agile and DevOps transformations hit the roadblock and never achieve the desired levels of productivity and effectiveness because of an oversimplified view of these complex dependencies across legacy and open systems.
The three biggest challenges that prevent a legacy DevOps transformation are mindset problems, architectural constraints and tooling limitations.
“There are no legacy systems, just legacy thinking.” That’s a great quote I heard recently from an engineer in a large organization trying to adopt DevOps at enterprise scale. The initial resistance from teams to move into DevOps ways is due to reluctance to change, emerging from an inertia of doing things a certain way for years if not decades. The mindset problem is as much a problem for those outside the legacy world as those inside. The “outsiders”—typically leadership, management team and consultants—tend to oversimplify the problem and, too often, rely on examples from the open systems world. This just alienates the incumbent legacy teams and makes them even more skeptical of DevOps driven change. The outsiders need to have a genuine appreciation of the challenges with legacy architecture and need to work in collaboration with the legacy teams if they really want to transform their organization.
Legacy systems have certain characteristics such as being static, tightly coupled and monolithic blocks of procedural code. These systems were not built for the modern Agile workflows that focus on incremental and iterative deliveries. A significant majority of these systems are written to operate in batch mode.Testing the systems requires cycling through multiple business days, which simply limits the level of agility you want to introduce in the organization. I am currently working with a legacy engineering team in the financial services space where you have to cycle through a minimum of 20 business days to simulate even 70 percent of the test scenarios. However, there are a number of solution options to overcome the architectural challenges; they range from killing applications to migrating to modern platforms or re-engineering legacy code. Not surprisingly, the right approach depends very much on your context.
Tooling in the legacy world to support a DevOps ways of delivery has always been a challenge. But it is fast changing now, thanks to a number of startups and even old-school IT product companies investing in this space. If you consider zSeries as an example, there are better IDEs (integrated development environments) such as RDz that can replace traditional code editors typically based on “green screens.” These IDEs provide great features such as better navigation, debugging, split views and searches on the legacy code. One of the CIOs I worked with made it clear to his team that he wanted to get rid of “green” screens from the IT floor. There are tools such as SmartBear Collaborator that support a great peer review experience and help bring one of the original XP practice of pair programming to the legacy world. There are tools such as endeavor from CA that promise automated build and deployments in the z-series world. Similarly, there are tools that support test automation of mainframe applications and tools for code coverage and code quality checks on legacy programming languages like COBOL. XATester looks like a promising tool for unit test automation from a Danish startup called Xact.
While this write-up is not intended to be a compendium of all DevOps and engineering tools, the central message is that there are a number of DevOps tools emerging on the legacy landscape which can help your legacy delivery teams get on to the DevOps and Agile journey.
What is more important is for organizations to realize that many of these legacy systems will remain part of their IT landscape for the next foreseeable future and it is time to invest in these core systems. They may not look as appealing as the digital platforms, but they are fundamental for businesses and you cannot ignore them on your DevOps journey. DevOps in the legacy world is not going to be smooth, but it is certainly not a mission impossible!
There are two fundamental approaches of how to design automation processes, declarative (also often known as model) and imperative (also often known as workflow or procedural).
The purpose of this article is to explain the principles of each, explore a little of how they work, try to highlight some of the strengths and weaknesses and explore how they can be used together. I don’t think one size fits all—based on your requirements and some of what is explored below you may use a combination of the two, even if some of the tooling you use leans toward one way or the other.
The fundamental of the declarative model is that of desired state. The principle is that we declare, using a model, what a system should look like. The model is “data driven,” which allows data, in the form of attributes or variables, to be injected into the model at runtime to bring it to a desired state. A declarative process should not require the user to be aware of the current state; it will usually bring it to a required state using a concept known as idempotent.
What this basically means is that if you deploy version 10 of a component to a development environment and it is currently at version 8, changes 9 and 10 will be applied. If you deploy the same release to a test environment where version 5 is installed, changes 6 through 10 will be applied, and if you deploy it to a production system where it has never been deployed, it will apply changes 1 through 10. Each of the deployment processes brings it to the same state regardless of where they were initially. The user, therefore, does not need to be aware of the current state.
Once we have a model for our infrastructure or applications, any future changes required are made to the model. This means that as a model changes over time—and may have a current, past or future state. This raises the question of how can a model be applied to a target system, how can it know what changes to make? This is not always easy to answer and is not the same for every platform and runtime you are managing.
It’s hard to cover a broad subject like this without making some generalizations—and there are some in this section. There are broadly three methods used to keep an environment or system and its desired state models in line:
Maintain an inventory of what has been deployed. This where we maintain the state of what has been deployed, and what is in the desired state of a release, and only apply the difference.
For example:
Use Case: This type of state management is particularly useful for things such as databases or systems that requires updates from a propriety API where the state of the target cannot always be easily determined.
Validate/compare the desired state with what has been been deployed. As a simple illustration, we may want to determine if important_file.xml from my desired state deployment exists on the target system and whether it looks the same as the one in my package.
Use Case: Probably the most widely used implementation is to simply manage the files that many cloud-native runtimes use to store configuration definitions, such as ssh keys for a server or xml files to manage a java runtime such as tomcat. This also can be used to maintain runtimes that are managed via an API; for instance, What is my heap or config item for my WebSphere Server? Is this what it should be? If it is doing nothing, set it to correct desired state value.
Just make it so. There are occasions when we don’t really care about what exists. The most obvious example here is anything that is stateless, such as a container. There are probably very few example of why you would want to run any automation within a container to change its state; you would just want to instantiate a new one that has the new configuration you require.
The fundamental of what often is referred to as workflow, procedural or imperative approach is that a series of actions are executed in a specific order to achieve an outcome. For an application deployment, this is where the process of “how the application needs to be deployed” is defined, and a series of steps in the workflow are executed to deploy the entire application.
A standard example might include:
The often-cited criticism of this approach is that we end up with lots of separate workflows that layer changes onto our infrastructure and applications—and the relationship between these procedures are not maintained. The user, therefore, needs to be aware of the current state of a target before they know which workflow to execute. That means the principle of desired state/idempotent is hard to maintain—and each of the workflows are tightly coupled to the applications.
The reality here is that the situation is not black and white. Puppet is an example of what is seen as a Declarative automation tool, while Chef is said to be Imperative. Do they both support concepts of desired state and are idempotent? The answer is, of course, Yes. Is it possible to use a workflow tool to design a tightly coupled release that is not idempotent? The answer is, again, yes.
The benefit of using a workflow is that we are able to define relationship and dependencies for our units of automation—and orchestrate them together. In the example above, we can create a workflow to perform pre-install, installation and post-install steps, perhaps adding additional conditional steps if it’s in a particular environment such as production (disable a monitoring system, additional security configurations, etc.).
Procedural workflows also allow us to, for example, deploy component A & B on Server1, then deploy component C or Server2 and then continue to deploy D on Server1. This give us much greater control in orchestrating multi-component releases.
It is often suggested that the choice is between Imperative- and Declarative-Based Deployment. However, these two approaches are not mutually exclusive.
One of the issues in the world of DevOps is the lack of a consistent lexicon or terminology. Just ask someone who works in the industry what their definition of an environment is or what the term application means to them. My guess is you will get differing answers.
To me, an application is made up of components and a release is generally in the form of an application that is created from multiple versioned components. Now, depending on the organization you work in, you may or may not recognize the definition I have given—and if everything in your world is a stateless loosely coupled microservice on a cloud native platform, then this may be the case. If you work in a more traditional enterprise organization, you will more likely recognize the requirement for multi-component release orchestration, requiring procedural or workflow processes to handle the coordination and dependencies between the different components.
When it comes to deploying a specific component, I think it’s hard to argue against the benefits of using a Declarative or model-based approach. The concept of something being Idempotent—I can deploy any version to any target system if it has never been deployed to before, if we want to introduce a new change to an already existing environment or even if we want to roll a unit of automation back—is hard to argue against for its consistency and simplicity.
The future of IT Operations is doubtless based on things such as stateless microservices running in fungible containers, using Kubernetes watchers to trigger event-driven automation, much as the future of retail is for Amazon to deliver milk to my IoT-connected refrigerator using a drone.
That being said, most of us still live in the present and have the need, for some time to come, at least, to address some of the automation and orchestration challenges that exist in today’s IT landscape. With this in mind, a combination of Declarative (or model-driven) units of automation, coordinated by Imperative (or workflow-based) Orchestration can be used to achieve this. An imperative/workflow-based orchestrator also will allow you to executive not only declarative automation, but also autonomous imperative units of automation should you need to.
My recommendation is that an application workflow defines the order and dependencies of components being deployed. The Declarative Model for a component determines what action needs to be taken to bring a target system into compliance with the model.
It is hard to underestimate the importance of desired state—it has changed the way we look at system and application updates and drastically improved the visibility, audit and control of IT systems. Any progressive organization looking to move to a continuous delivery/deployment model or implement things such as autoscaling or “deploy and destroy” for development or testing purposes will benefit from the use of declarative/model-based automation.
However, a model-based approach does not answer all the difficult questions around orchestration. This, in my opinion (for the moment, at least) is best achieved using Imperative, Procedural or Workflows processes.
Dave Sayers is a co-founder and technical services director of MidVision, a provider of Release Automation and Continuous Delivery Software. He works closely with customers and partners on DevOps and Automation initiaitves throughout Europe and North America and is responsible for devising and managing the partner and channel strategy. Connect with him on LinkedIn.