devopsy.kinsta.cloud

Author: Bob Reselman

  • Using IBM ADDI and VTP to Enhance Observability in Mainframe Development

    Using IBM ADDI and VTP to Enhance Observability in Mainframe Development

    Observability has always been a key factor in software development – not so much on the mainframe. However, the rise of the internet as the go-to platform for modern commerce has caused an explosion of transactional activities. There’s much more complexity behind the scenes than the request/response activity that used to be the usual interaction between a client and a single server. That necessitates ubiquitous observability as a mission-critical necessity for any modern application. As computational activities increase, so, too, does the risk of failure. One way to avoid disaster is to rely upon observation technologies that monitor not only runtime behavior, but design-time behavior, as well.

    Analytical Insights are Key to Application Modernization

    IBM observabilityModern computing activity goes well beyond simple request/response interactions. Today, one request can traverse any number of computing resources before resolving to a completed response. These resources range from application logic that’s encapsulated and then exposed to the network at a number of endpoints, to a diverse mixture of data storage mechanisms. And, if the application is part of an event-driven architecture in which messages are passed around to a large number of recipients, ancillary logic can be executed in ways that are unknown to the originating request. This can cause significant side-effects, nonetheless.

    Given the complexities inherent in modern applications running at web-scale, without ubiquitous observability, developers are flying blind. This is true not only for what we have come to know as commodity computing, in which applications are hosted on racks of X86 servers housed in private and public data centers but also for mainframe computing.

    The Emergence of Observation Technologies

    Over the last decade, a growing number of observation and monitoring technologies have emerged for network applications and database environments. Open source projects such as OpenTracing, OpenTelemetry, and Prometheus give developers insights into the runtime behavior of applications. Also, there are a plethora of commercial observability products that companies can buy to meet their specific monitoring and observability needs.

    At the development level, there are open source tools such as JSHint for JavaScript developers, PyLint for Python developers and FindBugs for Java developers, just to name a few. These tools can inspect static code and point out shortcomings. Some tools, such as .NET analyzers, will go as far as to correct code as the developer is writing it. The easiest bugs to fix are those that are never created. Correcting code as it’s being written brings a whole new dimension to design-time observability.

    However, while the X86 development community has benefited from observability tools for a while, their mainframe counterparts haven’t had it so easy. Commodity developers require nothing more than a click on a web page link to access any number of powerful observability tools, but mainframe developers have been starved for these types of tools.

    Until now.

    Applying the Principles of Observability to Modern Mainframe Development

    One of the big challenges of mainframe programming is getting a clear picture of application dependencies. Unlike programming under Java, C# or Node.JS, where external dependencies are defined in a central configuration file, or internal dependencies can be discovered with a code analyst tool, mainframe programming makes knowledge of dependencies a difficult undertaking. This type of design-time observation hadn’t really existed until technologies like IBM’s Application Delivery and Discovery Intelligence (ADDI) came along.

    An important thing to understand about dependencies in most mainframe computing is that they exist at both the function and executable levels. A developer can create a function, which, in mainframe parlance, is called a routine. That one routine might be used by dozens, if not hundreds, of other routines. Sharing code from a common resource is not unusual in X86 development, where binary libraries get shared all the time. However, in mainframe computing, these shared routines, along with file layouts and resource definitions, are physically copied in with the developer’s code using a technology called a copybook. Then, the code is compiled and linked to create an executable file. Once turned into an executable, the code becomes opaque. As a result, observing what’s going at runtime and design-time within the executable can be a herculean task. For many years, it was.

    Fortunately, you can use a tool like ADDI to overcome this obstacle. IBM’s ADDI provides important insights and analysis into the code that makes up a mainframe application. Both developers and system analysts can use ADDI to determine not only application dependencies but also to gather information about the execution environment; for example, scheduler settings and database configuration.

    As with X86 development where ancillary application intelligence, well beyond information about the source code, can be displayed in the developer’s integrated development environment, ADDI also makes it possible to report information in the developer’s IDE. Also, systems analysts can use the additional information provided by ADDI, such as the structure and segmentation of an application, to make decisions about how to refactor code when it comes time to make improvements.
    Providing a broader scope of observability into design-time activity is but one aspect of enhanced observability in the modern mainframe environment. ADDI also reports test coverage metrics. Knowing what and how much code has been tested is an important part of observability, even more so when used with tools that are focused on mainframe testing. One such tool is Virtual Test Platform (VTP).

    Leveraging Observability in the Mainframe Testing Process

    Until VTP came along, mainframe developers were flying blind – they had limited visibility into the transactions that their code used and the dependencies that their code relied upon. They had an enormous amount of work to do to figure out transactional behavior throughout the application. They spent a lot of time doing archeological digging in legacy codebases to get a clear idea of what was happening. And, it didn’t get any easier as testing demands on all types of applications, both X86 and mainframe, increased due to the rise in commercial activities on the internet at a global scale.

    According to Chris Trobridge, senior offering manager at IBM, “The testing load for developers has been increasing tremendously, and with the introduction of agile development processes, developers just can’t keep up with the testing load that they have.”

    VTP addresses many of these testing issues, both in terms of observability and quality assurance. “VTP really is observing what goes on at runtime, but without knowing anything about the application,” Trobridge said.

    VTP can monitor an application and capture and record all its interactions. This type of observability enables developers to make modifications and then run those changes against observed behavior. It’s a novel approach to mainframe development in which the benefits of both design-time and runtime observability are leveraged in a synergistic manner. In order to understand how groundbreaking this capability is, you need to remember that only five years ago, while X86 developers were enjoying a maturing set of monitoring and tracing tools; for mainframe developers, this type of observability wasn’t possible. Now it is. It’s a dramatic step forward.

    “VTP is a tool that enables developers to automate a lot of the testing, and, in turn, enables the recording of a transaction environment without having to know a lot about the environment,” Trobridge explained. “The recordings themselves can be played back rapidly without compromising security. This is an important aspect of VTP. This level of observability and interactivity really is something that mainframe developers have not had access to before. It’s a transformational technology in the mainframe world.”

    Technologies such as ADDI and VTP are a transformational approach to enhancing observability in mainframe environments. In many ways, the focus on exposing the granularity of computing at the routine level is a harbinger of things to come for all developers, in both the X86 and mainframe worlds.

    Taking a Function-Centric Approach to Observability

    Historically, for commodity developers, observing request/response behavior between services at runtime, along with identifying code quality and complexity issues at design time, allows for identification and remediation of most issues in software development.

    For the mainframe developer, request and response behavior is only the tip of the iceberg when analyzing application behavior. In a mainframe environment, one sub-routine might be used by thousands of other functions and subroutines at any given time. Commodity developers have been spared the burden of accommodating the function as the fundamental deployment unit because, until recently, functions were grouped into well-encapsulated deployment units, such as Java’s .jar file, .NET DLLs or, more recently, containers. If something went wrong, the error would generally appear in one of the deployment units.

    However, with the current rise of serverless computing, where the function becomes the primary deployment unit, X86 developers are in the same boat as their mainframe counterparts. In the past, for the X86 developer, one application might use a limited number of components that could have dozens, maybe hundreds of functions. Now in the serverless world, all these functions are exposed as standalone units, in which each function is the responsibility of a distinct team. We’ve gone from an ecosystem that has a limited number of deployment units to an ecosystem that can contain thousands. Fixing bugs can require hours of poring over logs and examining minute details just to determine the root cause of an issue.

    As mentioned above, this depth of dependency at the function level is something that mainframe developers have been dealing with for years. Fortunately, tools such as ADDI and VTP provide a high degree of observability for very complex mainframe applications at both runtime and design-time.

    It’s strange to say, but the large, function-based ecosystem issues that mainframe developers are solving today will become a growing concern for X86 developers in the very near future. In many ways, tools such as ADDI and VTP are setting an example for X86 tool development moving forward. The mantra, “You can’t fix what you can’t see” is as true today as it has ever been. That the mainframe community is at the vanguard of creating new ways to work in massive computing environments is a testament to the power of rejuvenation in the most unexpected ways.

  • Wazi VTP Test Recording Streamlines Mainframe Testing

    Wazi VTP Test Recording Streamlines Mainframe Testing

    Mainframe developers have an easier time testing with the Wazi Virtual Testing Platform

    Working with test data is tricky. If you’re a developer working in a distributed environment, things are difficult enough. It seems as though you’re forever replenishing test data on a database in your local system or one that’s dedicated to you out on the network.

    You run tests that twiddle with that data and when things go wrong that data gets so corrupted. It’s useless. So you have to go through the whole replenishment process again. This is no big deal if you’re working with a few gigabytes of data, but when you’re running a test against a database that is really big—on the order of hundreds of gigabytes or a few terabytes of data—the replenishment process can take hours or, in some cases, days. It’s a lot of time-consuming work, even when a good deal of the process is automated.

    As if this isn’t challenging enough, things get even more difficult when you’re executing tests that run in a multi-user scenario. You have a whole bunch of users making a whole bunch of changes to shared datasets that, in turn, affect the application’s overall state. When tests fail not only do you have to figure out if the problem is with the code, the data or both, you also have to determine where the failure occurred.

    Remember, a distributed application is intended to run multiple instances of application logic that gets distributed over a cluster of machines. Figuring things out requires a lot of very detailed monitoring of all activity in all the instances of the application running over the network. And, determining the root cause of each failure requires a good deal of expertise to do the analysis successfully.

    It’s a lot of work despite the fact that developers working on distributed applications have a lot of direct control over their environment. For mainframe developers, working with test data can be more than tricky; it can be a staggering challenge that tests the mettle of even the most gifted developer.

    Understanding the Problem of Data Sharing on a Mainframe

    There is a fundamental difference between the way data is managed in mainframe testing and distributed testing. Whereas a distributed developer “owns” the data in their environment, mainframe developers really don’t have such luxury unless they are working in dedicated subsystems. Typically a mainframe programmer’s programs are compiled binaries that are usually part of a larger unit of execution. An example of a larger unit of execution is the work that sets up a worker to receive unemployment insurance.

    Let’s say that Sarah is a COBOL developer who has written a program that is part of the process that determines a worker’s unemployment benefit. Sarah’s program discovers all the worker’s employers over the last five years and adds the income from each employer to a table in a database that is shared with that other programs that are part of the unemployment benefits process.

    Of course, Sarah will unit test her code. If she is working on a z/OS mainframe, she might use a product such as zUnit to do the testing. But, just testing Sarah’s code is only part of the work of the entire process; a broader range of testing is needed.

    Now here’s where things get tricky: Sarah’s code is but one of many programs doing the work needed. So, while it’s entirely possible that her unit testing might be fine, once her program changes data and those changes are applied to the bigger picture things might go haywire. In such a case, if Sarah’s data corrupts the overall state of the unemployment benefits process, the data of all the other programs that interact with Sarah’s data might be corrupted, too. Thus, the entire database will need to be replenished. Remember, Sarah does not “own” that shared data; she is one of many users.

    Sarah’s example is but one instance of a real problem that is a common, time-consuming headache for mainframe developers. If the data transformations made by Sarah’s program could be isolated and made reusable on demand, that capability would have far-reaching benefits that could change the way mainframe testing is conducted in the enterprise.

    Fortunately, the capability exists. It’s the test recording feature of the Wazi Virtual Testing Platform.

    Understanding the Mechanics of Test Recording

    Test recording as implemented by the Wazi Virtual Test Platform (VTP) addresses two fundamental problems around managing data in a mainframe testing environment. The first problem is creating a way to isolate and make reusable the test data that is the result of the actions of one or many mainframe users. The second problem is creating a way to update that test data should it go stale.

    Wazi VTP allows a developer or test practitioner to record the terminals of one or many users executing data query commands against a database. Wazi VTP saves the results of those queries for reuse as test data.

    Wazi VTP has the capability to record CICS transactions or JCL batch jobs. CICS transactions can use any combination of CICS commands or Db2, DL/1 as well as MQSeries calls. Batch jobs can use dB2 or DL/1. Recording can be started from CICS by using the BZUS transaction command or from batch by running the application. All of this is done without any change to the user’s application programs.

    Once the recording has completed, the CICS transactions or the batch job can be replayed entirely in batch, completely standalone and free from any subsystem. Since the data was originally recorded and then fed back to the replay process, it maintains the state of the data for all subsequent testing.

    To stop recording using CICS, the developer enters the command, BZUE. Figure 1 below shows the command sequence from a conceptual point of view:

    Figure 1: Running the Wazi VTP Test Recorder

    The commands to start and stop recording can be entered from a mainframe terminal as shown in Figure 2, below. The developer enters the command BZUS without any parameters to record only the queries executed from the terminal in which the command is invoked. This is shown in the upper-left portion of Figure 2 below. If the developer wants to record the activities of many terminals, they add a wildcard parameter as shown in the lower right of Figure 2. In this case, BZUS AB* is the command invoked to record the activities of terminals that have an ID starting with the characters AB. Also, the developer can record activity coming from a specific terminal by using the name of the specific terminal instead of a wildcard parameter.

    Figure 2: Wazi VTP can record activity from either single or multiple terminals

    When a developer wants to stop the record they enter the command BZUE.

    Using Wazi VTP is not confined to 3270 terminals or terminal emulators. The Wazi VTP also supports an API that can be used to perform integration testing within a CI/CD workflow.

    The important thing to remember is Wazi VTP does a lot more than capture screen activity. In fact, screen capture is but one tiny aspect of the data collected. Wazi VTP captures all of the arguments on all of the CICS, DB2 or DL/1 subsystem calls. And, as mentioned above, the data it collects can be used as test data.

    Figure 3 below shows an example of recorded data and program call arguments. Notice that Wazi VTP has recorded both CICS and programs (PGMS) that are written in COBOL, PL/1 and Assembler, as indicated on the first column on the left of the screenshot shown below.

    Figure 3: The data that is a result of a Wazi VTP recording session

    In addition to simply recording data that is then available as reusable mock data, Wazi VTP detects and then reports when data is changed. Detection is based on changes made by program logic. Wazi VTP allows those changes to the programming logic to continue to execute, until either a new, unrecorded path is attempted or until the program’s end.

    This is a big difference from test data in the world of distributed computing. Unless precautionary measures are taken, once cloned instances of the test databases have no real insight into the state of those databases from which they were derived. Things might go well in unit, integration and performance testing only to blow up in production because the test data is completely out of sync with the real-world production data. Mainframe developers running Wazi VTP don’t have this problem.

    Tracing Behavior and Gathering Statistics

    One other advantage that Wazi VTP brings to mainframe testing is the ability to trace query behavior and gather statistics about the query behavior executed.

    Figure 4 below shows how to invoke tracing under 3270 emulation. The image on the upper left shows invoking a standard trace. The image on the lower right of the figure shows the call that invokes a deeper internal trace.

    Figure 4: Wazi VTP supports standard tracing as well as lower, internal tracing

    Figure 5 below shows the details that are the result of a standard trace.

    Figure 5: Standard tracing shows a broad range of information including information from recorded tests

    Figure 6 below shows the richer set of details reported when an internal trace is invoked.

    Figure 6: Wazi VTP Internal tracing provides a deeper level on reporting, including the address and memory usage of calls to a program

    As mentioned above in addition to tracing, Wazi VTP will report statistics around test activities. Figure 7 below shows a set of statistics from a typical testing session conducted using Wazi VTP. Wazi VTP can report not only the number of times a program has been called but also the identifier of the caller. In addition, Wazi VTP reports status code that goes with different aspects of a CICS transaction.

    Figure 7: Collecting program statistics using Wazi VTP

    Wazi VTP recording is a powerful feature. Being able to record and reuse test data reduces a significant amount of the burden that goes with data management for mainframe testing. Tracing and statistics-gathering capabilities add even more power—so much so, that mainframe testing using Wazi VTP in many ways surpasses the capabilities present in current distributed computing paradigms. This is no longer the “coders code, testers test,” waterfall-driven approach to testing that was common only decades ago. It’s a modern approach to the way that enterprise testing works in today’s world. In short, it’s very cool!

    Mainframes Development Offers Opportunities for Everyone

    I am a big fan of the opportunities presented by mainframe programming. The machines are powerhouses. They’ve been designed from the get-go to handle thousands of users simultaneously. Mainframe computers still handle over 80% of corporate data.

    There remains a real need for mainframe programmers now and well into the foreseeable future, particularly as the existing workforce continues to age into retirement. I’ve counseled more than a few younger programmers to consider spending a year or two doing mainframe development. Mainframe programming experience provides a great fallback to get through the ups and downs of an increasingly volatile job market.

    As the mainframe and distributed communities continue to cross-fertilize, it’s entirely possible for developers from all walks of life to work in either environment provided they have a clear, competent understanding of the general principles of software development as well as a solid appreciation of the best practices that are general to any type of programming environment. Once the basics are in place, all that remains is taking the time to learn the particulars of the programming landscape. In the past, this might take years. Today, with tools such as Wazi VTP, the learning curve can be reduced to a matter of months provided the developer is committed to doing the work necessary to get the skills required to contribute effectively to the work at hand.

    Wazi VTP is a compelling, state-of-the-art testing platform for mainframe developers. But it’s potential is much broader: Wazi VTP brings mainframe development into a common development landscape. In many ways, Wazi VTP is a door that opens into a world of opportunities for all developers from any computing environment.

    Check out IBM® Wazi Virtual Test Platform Demo:  https://mediacenter.ibm.com/media/t/1_tbcimobz

  • DRY Comes to COBOL in IBM Z Development

    DRY Comes to COBOL in IBM Z Development

    Don’t Repeat Yourself (DRY) is a fundamental tenet of modern computer programming. Just about every CS student is taught that duplicating code in an application is an accident waiting to happen. The hazard is that you might alter a piece of code in one place, only to leave another identical instance of that code unchanged. Then when you run the code all sorts of strange things happen. In some cases, the changed behavior is in force. In other cases, it’s not. Maintenance becomes a nightmare.

    These days adhering to the principle of DRY is not that difficult. Object-oriented languages such as Java, C# and Objective-C make it easy to encapsulate a set of functions into a class that can be reused throughout an application. Also, technologies such as RPC, XML-RPC and gRPC make it possible to expose distinct functions to the network in a sharable manner. When a single piece of code can be easily shared at runtime, the risk of writing redundant code goes down. Why reinvent the wheel when you can just use one that is pre-existing and well known?

    However, this has not always been the case. In the early days of programming, lines of code executed sequentially. You started at line 1 and ended at line 200. There were no side trips in the middle.

    Language designers were aware of this shortcoming and made it so programs could jump around the lines of code. You could start at line 1 and run to line 20, then jump to line 190 and run to line 220, coming back to line 21 and then jump again to exit out at line 240. It was definitely hopscotching, but it worked.

    Procedural programming arrived soon after and with it came the early forays into segmenting code into a discrete unit of execution, the function. Functions introduced the concept of code sharing within an application. Still, a framework of sharing a single unit of code among an entire system, let alone many systems, on the network took a while to develop. As a result, in the early days of programming code redundancy was rampant.

    The thing to remember is that the proliferation of redundant code was not so much a reflection of professional incompetence. Rather, it was due to the absence of a programming framework by which to avoid code redundancy.

    But, that was then and this is now. Today, DRY is a standard operational mantra. This is good. On the other hand, there’s still a lot of redundant code out there, particularly in shops that have a lot of legacy COBOL. Remember, COBOL just celebrated its 60th birthday. It’s been around for a while.

    Fortunately, the world of redundant code in the COBOL is about to become a whole lot smaller with the introduction of Duplicate Code Detection Technology Preview that’s built into IBM Developer for z/OS.

    Understanding Duplicate Code Detection Technology Preview

    The Duplicate Code Detection Technology Preview (DCD) is a feature supported by IBM Developer for z/OS (IDz) that automatically discovers duplicate code within a given codebase. It’s an optional feature, available as of version 14.2.2, that needs to be added during IDz installation. DCD is an enterprise-level tool that’s intended to be used with very large amounts of COBOL source code, at the level of a million lines of code or more (See Figure 1).

    Duplicate Code Detection Technology Preview for IBM Developer for z/OS
    Figure 1: Duplicate Code Detection Technology Preview for IBM Developer for z/OS exposes duplicate code for elimination or refactoring.

    Once duplicates are identified, developers can use DCD to justify whether the given redundancy is appropriate or if it’s a duplication that needs to be removed. Thus, DCD can be used as a tool to assess code quality and as a tool to refactor the codebase.

    See Duplicate Code Detection Technology Preview in Action

    At IBM’s upcoming webinar, “Duplicate Code Detection — Simpler & Faster Thru’ Out the Lifecycle Webcast,” on May 20, 2020, from 9 a.m. to 10 a.m. (ET) you’ll get an in-depth look at Duplicate Code Detection Technology Preview for IBM Developer for z/OS. Architects Herve Le Bars and Jean-Yves Rigolet will be on hand to go over the details for using DCD to eliminate duplicate code and improve code quality.

    The webinar is free and it’s filled with important information that’s sure to enhance the productivity of developers using IBM Developer for z/OS. You can register for the DCD webinar here.

    How Duplicate Code Detection Technology Works

    Duplicate Code Detection Technology is an optional feature that can be fully integrated into IBM Developer for z/OS. Developers can use DCD as a standalone application to search for duplicates in his or her workspace. Also, developers have the added benefit of sharing duplicate code detection results among a team. In order to have a collaborative experience with Duplicate Code Detection, three additional components need to be installed on the server-side. These components are:

    •     IBM Websphere Liberty Server
    •     Duplicate Code Detection Feeding Agent
    •     Source Code Management System

    Let’s take a look at how it all fits together.

    As mentioned above, developers can use DCD with IBM Developer for z/OS to examine the individual developer’s workspace for code duplication. In this case, DCD is essentially a stand-alone tool (See Figure 2, callout 1).

    Duplicate Code Detection
    Figure 2: Duplicate Code Detection works standalone and as a collaborative tool.

    While it is beneficial to have an individual developer detecting duplicate code in his/her workspace, it’s limited in scope. There is only so much work a single developer can do. Where things get really powerful is when the result of searches for code duplication can be shared among the team. This is where using DCD in a client-server architecture comes into play.

    Central to the DCD client-server architecture is the DCD Server which is an IBM Websphere Liberty Server and the DCD Web Application.

    The DCD Server is used to share the results of duplicate code detection across members of the development team. Developers interact with the DCD Server using the DCD component running within IBM Developer for z/OS (See Figure 2, callout 1, above).

    On the server-side, there is an independent client, the Duplicate Code Detection Feeding Agent that acts as a bridge between the source code stored under a source control management system and the DCD Server.

    The feeding agent asks the SCM for the changed COBOL files in the source code management system (See Figure 2, callout 2). Then the DCD Feeding Agent sends the changes to the DCD server (See Figure 2, callout 3).

    The DCD server keeps track of duplicate code (See Figure 2, callout 4). As mentioned above, team members access the DCD server using IBM Developer of z/OS to visualize the duplicate code information. The DCD feeding agent works incrementally: Only changed COBOL files are transferred for examination, thus reducing operational overhead.

    You can read the details for installing and configuring Duplicate Code Detection by downloading the DCD Installation Guide.

    The Benefits of Using Duplicate Code Detection Technology

    There are many benefits to using Duplicate Code Detection Technology. Automated duplication detection simplifies refactoring. Duplicate Code Detection Technology identifies redundancies that occur throughout millions of lines of code automatically while leaving it up to the developer to judge if the code duplication is justified or not. If duplicates do need to be removed, DCD allows developers to make the changes at once, thus saving time while improving productivity. Remember, Duplicate Code Detections works at lightning-fast speeds. It takes on average 100 milliseconds to search 100,000 lines of code.

    Also, when distributed collaboration is enabled using the DCD Server, searches can be shared with team members and stored for later use, thus improving collaboration efforts among teams and team members.

    Duplicate Code Detection enhances search efficiency by examining code stored directly in source control management repositories. DCD improves code readability while reducing complexity. In addition, it supports versioning using standard source control management frameworks, such as GitHub.

    Putting It All Together

    The principle of DRY is commonplace these days. While it’s been a principle that’s easier to implement in younger companies, those businesses with codebases that go back decades have found adopting DRY to be difficult. Refactoring can be risky. Many companies think that unless there’s a compelling reason to refactor the old code, most times it’s better to leave well enough alone, particularly if that code is written in COBOL.

    But as the dramatic need for COBOL programming talent at the New Jersey Department of Labor recently revealed, COBOL matters. And, when that COBOL code is refactored, which is sure to happen, applying DRY in the refactoring effort will be a driving principle. Hence, the opportunity exists for DCD to play a key role. It’s a compelling technology. Given the power that DCD brings to the COBOL programming experience, implementing the DRY principle against both current and legacy code requires no more labor than the work that goes with spell checking a large document.

    Duplicate code has been the bane of developers since the first Hello World was written over half a century ago. Over time the situation has improved, but for many companies it’s still a problem, particularly companies with a large legacy of COBOL programs. Companies that make Duplicate Code Detection Technology part of their COBOL programming toolbox will not only have the benefit of creating code that is more elegant, more readable and of higher quality, they will be well on their way to placing the DRY principle at the forefront of their software development activities.

  • Metropolis Now: The Two Worlds of Employment in the Age of Automation

    Metropolis Now: The Two Worlds of Employment in the Age of Automation

    Current events have revealed that there are two worlds in the employment landscape. One is digital, the other analog. For the most part, those in IT—the digital workers—have been unaffected by the spike in unemployment due to the economic side-effects of the COVID-19 pandemic. But, it doesn’t mean they’ll be untouched in the long-term. As current events demonstrate, when it comes to the essentials in life, we’re all connected. Yet, the divide will continue to separate us to the detriment of all unless we do something about it.

    When observing current events, I can’t help but think about Fritz Lang’s 1927 silent movie, Metropolis. It’s a science fiction film that depicts a world in which a small number of people live a privileged life in skyscrapers that tower high above the city. The rest of the population toils away below ground, doing the mundane work required to keep the city running. It’s a stunning piece of cinematography. The sets and special effects are breathtaking. What’s also impressive is that, for better or worse, so much of what Lang presented nearly 100 years ago has come to be.

    You can view the full-length version of Metropolis here.

    metropolis-scenes
    Figure 1: Scenes from the 1927 silent movie, Metropolis by Fritz Lang.

    Lang’s vision of the future has a strange, almost eerie prescience for me, particularly as someone who has spent most of his adult life working in IT. It’s one that’s entirely possible, yet one I hope we avoid.

    Allow me to elaborate.

    Life Is A-OK in the Cloud

    There’s a good argument to be made that today we live in Lang’s Metropolis, a world in which those who are doing really well live in the light of society while those who are just getting by live below ground in darkness.

    The widening gap between prosperity and economic hardship is a trend that’s been unfolding over the last 30 years. We’ve become accustomed to it. Sadly, it’s no longer newsworthy.

    What is newsworthy is the dramatic spike in unemployment due to the business shutdowns caused by the COVID-19 pandemic. The uptick in unemployment reveals a troubling set of circumstances: If you’re a digital worker chances are you’re still working. If you work in the analog world, that place where factories, restaurants, airlines, movie theaters, hotels, cruise ships and warehouses exist, there’s a good chance you or someone you know is out of a job, for now anyway.

    Take a look at the graph from the US Private Sector Job Quality Index, a project from Cornell University Law School and others, by way of vox.com. It lists the jobs that are most vulnerable to layoffs due to the COVID-19 pandemic. IT workers don’t even make the list. It’s as if we live in another world.

    at_risk_jobs_by_industry
    Figure 2: Information workers are absent from the list of jobs at risk due to COVID-19.

    The long and the short of it is that for developers, sysadmins, SREs and all the other people working in information technology, the recent economic downturn is but a blip in the daily news feed. For those manning a cash register, taking a ticket at the local cineplex, cutting hair or driving for a rideshare service, it’s a life-altering event. Or, to put it terms of Lang’s film, those of us who live in the clouds are doing well, and for those on the ground, it’s a different story.

    Now, consider this: What if all the people who are presently laid off from their jobs find out there are no jobs to go back to? What if the economic recovery is slower than anticipated or society gets so accustomed to doing without that consumption doesn’t resume? Is such a scenario possible? Yes. Is it probable? Dunno. Will those of us in IT who work remotely all over the planet, from the comfort of our wired office, suffer? I doubt it. But, as for the rest, what then?

    Let’s go back to the movie.

    A Workforce Divided

    There’s a point in the film when the protagonist, Freder (the son of the industrialist who built Metropolis), goes to his father to inquire about a catastrophic underground explosion shown earlier in the film.

    The son asked, “Your magnificent city, Father, and you, the brain of this city, and all of us in this city’s light … and where are the people, Father, whose hands built your city?”

    The father responded, “Where they belong.”

    The son asked, “Where they belong? In the depths?”

    His father nodded his head yes. (See the scene here).

    There you have it. With that one physical gesture, Lang foreshadows a future which might very well be at our doorstep. As I mentioned earlier, it’s eerie, it’s prescient and it looks as if it’s coming to be.

    What Does This All Have to Do with DevOps?

    Those of us in DevOps have made valuable contributions to the world. We’ve done dramatically more good than harm. We’ve also been well compensated. The average salary for a SRE in the USA is $131,340, with a low of $60K and a high of $192K. Yet, the top salary for grocery clerk at Ralph’s supermarket is $22 an hour, which comes out to $44K a year over a 50 week year.

    Do SREs deserve such a high salary? Of course, they do. Does a grocery clerk deserve only $44K a year in comparison? At the end of 2019, it would have been easy to say, yes. It’s a low skill, low education job. But that was then, and this is now. As of this writing, toilet paper is a premium commodity and the shelves need to be stocked with it. I’ll wager that in some circles, paying someone $50 an hour to get the rolls on the shelf is reasonable compensation. It all depends on how much people want toilet paper.

    Am I being a bit dramatic?  A little, but still, it’s worth thinking about.

    So, what’s the important take away? Let’s go back to Metropolis one last time.

    At the beginning of the film, there’s a scene in which the protagonist, Freder, was frolicking about in a lush arboretum-like greenhouse atop a skyscraper. He was surrounded by pretty maidens dancing. An elevator door opened into the garden. Freder observed as a group of raggedy waifs from beneath the city exited out of the elevator. A woman leading the waifs was at the head of the group. The waifs looked around in amazement at such luxury. Then, Freder’s servants ushered the group back into the elevator to exit. Freder was obviously affected by the waifs. He went into the depths of the city to investigate.

    Shortly thereafter, Freder went to see his father to inquire about the explosion I described earlier. His father asked, “What were you doing in the machine halls, Freder?” 

    The son responded, “I wanted to look into the faces of the people whose little children are my brother, my sisters…”

    It’s a powerful scene. It makes me wonder how many of the faces I’ve seen of those who were working in February are now sitting home, unemployed and unable to pay the rent in April.

    The takeaway is this: We’re all connected at a level much deeper than the trivial interactions of the given social network. Current events have demonstrated the fact in spades. COVID-19 was first reported in China around November 17, 2019. On April 4, 2020, less than six months later, there were 240 cases reported in the state of South Dakota in the United States. If that’s not connection, I don’t know what is.

    Those of us who work in the cloud are doing quite well, despite the setbacks caused by the global pandemic. But, there’s a growing number of us who are not doing so well and won’t be for the foreseeable future. It’s easy when everything is peaches and cream to be unaware of a world in which six-figure salaries are not the norm. But, when things go south and there’s no toilet paper to be had, the realization that we’re all in this together smacks us square in the face, like it or not.

    The question then becomes, given that we’re all connected, what type of world do we want to share, the one in Lang’s Metropolis or something better?

    It’s something worth thinking about. The pandemic will pass, but the underlying disparities will remain unless we do something about them.

     

    The Impact of Automation

    Don’t forget to check out my new, free book, “The Impact of Automation.”

    This book takes a serious examination of the history, impact and results from automation on our society and culture. Hope you enjoy it.

     

  • Social Distancing in the Age of Automation

    Social Distancing in the Age of Automation

     

    An unintended consequence of COVID-19 is that the practice of social distancing might remain long after the threat posed by the virus subsides.

    It seems as if COVID-19 (aka the Coronavirus) has appeared out of nowhere. At the end of 2019, we were all unwrapping out holiday gifts and looking hopefully into a new decade. Now, in March of 2020, schools are closed for weeks, if not months, major tech conferences are canceled, the NBA has suspended the season and the shelves at Walmart and Costco are bare of daily essentials.

    We’ve gone from zero reported cases at the end of January 2020 to over 120,000 worldwide six weeks later. (See Figure 1, below).

    Total-cases-covid-19-who

    Figure 1: Total confirmed cases of COVID-19 as of March 12, 2020. Source:  By Our World in Data – OWID, CC BY 4.0.

    The World Health Organization (WHO) has declared that we are in a pandemic. COVID-19 is spreading around the globe and its spreading fast. People are dying from it. As of March 13, 2020, sources report that Italy had 1,016 deaths among the 15,113 cases of infection. Another source reports that in mainland China, at least 3,158 people died from the 80,778 infected. The number keeps growing. What’s amazing is that in November of 2019 there were no reports of deaths from COVID-19. To say that this is the stuff of science fiction made real is no understatement.

    Social Distancing as a Tactical Response to COVID-19

    COVID-19 is a contagious disease. The rate of infection is pretty high when people gather together in large groups. To address the risks, gatherings of over 250 people have been discouraged, if not banned outright by business and government. This tactic has been given a name, “social distancing.”

    Social distancing means that we keep our distance from large groups of people. As with any change in social behavior, there are economic impacts. Most people go to a workplace to do work. Take away access to the workplace and work stops, the economy comes to a halt. At least that’s how it was in the old days.

    Today we live in a wired world that makes online interactions as easy as a stroll down Main Street. As a result, many of us work online thus allowing commerce to continue even when access to the physical workplace is restricted. Yet, most of us—working online or not—are still tethered to a common, physical workplace, at least we were until COVID-19 kicked in. Now we’re going home and telecommuting. Once the threats posed by the virus have passed we’ll all go back to business as usual. This current episode of social distancing will be but a memory.

    Or will it?

    What if social distancing becomes a way of life that stays with us for the foreseeable future? What if we all just get so accustomed to social distancing—employer and employee alike—that we just stay home after this all blows over? What then? Think about it. It could very well happen and the implications are significant.

    Social Distancing Is Not New

    Social distancing has been a trend even before COVID-19 came along. Before the introduction of mass-produced private homes, people either lived in cities or in the rural countryside. Urban dwellers congregated in public spaces, Venice Beach, Boston Common or Central Park for example. They ran into each other walking down the street. They rideshared on mass transit.

    Then the automobile came along and social distancing set in. More people started to prefer riding in the isolation of the privately-owned automobile than on a crowded city bus or subway. The interstate highway system emerged to support the millions of new cars on the road. These highways and the 30-year fixed mortgage gave birth to suburban living. Also, television reduced movie-going to a predominantly entertainment experience. Who needs to see a newsreel in a movie theater when the news is getting piped into your television set every day?

    The suburban house became the castle of any man or woman who could cough up a relatively small down payment to buy one. The next thing you know the house becomes the social center: barbecues, cocktail parties, Super Bowl Sunday and legions of teenagers wreaking havoc at alcohol-fueled backyard pool-parties while their parents are out of town.

    Eventually, the introduction of video streaming (neé cable-based HBO) big-screen TVs, and movies on the demand made the family entertainment center a permanent fixture in residential house design. Finally, along comes high-speed internet and dad’s den gets replaced by dad’s home office and mom’s home office.

    But, we still had to go to the store to get stuff. Hence the rise of the shopping mall as a common ground in which to socialize. And, we still had to go to work to pay for it all … for a while, anyway, until Amazon comes along and brings social distancing to the shopping experience and telecommuting gizmos such as Slack, Zoom and a plethora of mobile apps make working from home a feasible way to conduct business. The only catch is that employers needed to get on board with telecommuting. Some did and some didn’t until COVID-19 started killing people. Now, they’re on board.

    Living in a Social Distancing World

    COVID-19 is forcing everyone to stay put. Every day an increasing number of companies are mandating that if you can work at home, you are to work at home. Late-night television shows are broadcasting empty theaters. There’s even talk of canceling the Democratic National Convention. It seems as if we’re all going home.

    The risk is that we just might stay there.

    Let’s face it, we’ve created a digital infrastructure that can support a situation in which most of the population of the developed world stays in a state of social distancing permanently. All the things required to make it happen are in place. Today, we can buy just about all we need online and have it delivered to our doorstep. An increasing number of the ordinary activities of daily life happens on the internet, everything from a casual chat on Skype to a game of cards with our friends on Trickster Cards, are typical. And yes, people have been having romantic interludes online since personal computers could support video conferencing.

    In the not-too-distant-future, home healthcare devices will do away with most visits to the doctor’s office. Soon, if we need to get somewhere, we’ll call a driverless vehicle to get us there. Manufacturing automation will continue to improve to the point where we won’t work in factories directly anymore. We’ll just control the robots that do work in the factories from computers in our home office.

    As finance and services continue to become a growing part of the economy, the work that industry requires can be done from home, too—quite easily in fact. Yes, we’ll still need human plumbers, electricians and nurses. But these workers will occupy a special place in a socially distanced world. They’ll be well vetted and well protected. 

    So, the long and short of it is that current events might very well be a harbinger of things to come. And, if anything, the hazards of COVID-19 are pushing forward the propensity to make social distancing the norm. When that happens the really big question to ask is this: If all of us are socially distanced by default and more of our interaction with, and understanding of, the world comes from that which we see and hear on cell phones, computers and television sets, how will we know what is real and what is well-contrived fiction? It’s an important question that deserves serious consideration. I hope we can answer it. Then, once answered, the even bigger question to ask is: Will we care?

    — Bob Reselman

  • Commercial Objectives in the Age of Automation

    Commercial Objectives in the Age of Automation

    When I was 17 years old, back in 1971, I sold orange juice near a busy street corner in New York City. My job was to stand out in front of a fruit stand and dispense quarts of OJ from a machine. It was pretty simple: keep the hopper on the top of the machine filled with oranges and keep the supply of quart jars steady. Then, when a customer came by to make a purchase, I filled a quart jar with juice, took the customer’s money and rang up the sale in the cash register.

    That was my job, rain or shine, in warm weather or cold: keep the machine operational, give the customer the goods and collect the money. Other people had this job, too. Selling fresh-squeezed orange juice to the masses by way of a rather simple piece of automation technology kept us busy.

    But, here’s the thing. If my employer had to choose between me or the machine, he’d have chosen the machine every time. The machine was a gold mine. It made more orange juice in a minute than 10 human squeezers could have done in 10 times the amount of time. In fact, if my employer used humans, he would have lost money. The machine was key to his commercial success selling orange juice.

    Now, hold this thought while I move forward to recent times.

    In May of 2019, the rideshare service Uber went public. In order to issue shares on a public exchange, the company, as prescribed by law, needed to publish a prospectus describing its business model as well as how it intends to use the money it raised through the IPO.

    Every company that’s listed on a public stock exchange publishes a prospectus. Facebook has one, and so does Microsoft. Even World Wrestling Entertainment, the folks that produce action-packed events such as the WWE Super ShowDown and WrestleMania, has one. The prospectus is the way the SEC creates transparency into the finances and commercial strategies of a  company. If you take the time to read one, a prospectus will tell you a lot.

    So I did. I read the Uber prospectus and I found something very interesting on Page 10: “We are investing in technology to power the next generation of transportation. Our Advanced Technologies Group (‘ATG’) focuses on developing autonomous vehicle technologies, which we believe have the long-term potential to provide safer and more efficient rides and deliveries to consumers, as well as lower prices,” stated Uber.

    In other words, Uber is actively pursuing driverless technology as a key commercial objective. Why? Well, to answer this question let’s go back to my days in New York City playing trusty sidekick to the orange juice machine.

    Back then, in order for my employer to make money selling fresh-squeezed orange juice, he needed both me and the machine working in tandem to fulfill the commercial objective to turn some oranges and a glass jar into money. In terms of his commercial objective, I was the weak link in the chain. I got sick. I had bouts of adolescent unreliability. I moved on to greener pastures when better opportunities appeared. If he could have replaced me with a vending machine that sold fresh orange juice he would have. If I were in his shoes, I’d do the same thing. But at that time, orange juice vending technology was not as advanced as it is today. I was lucky, I guess.

    But, that was then and this is now. So let’s go back to Uber.

    Remember, the commercial objective of rideshare is to turn a moving vehicle into money. Right now this involves a cellphone, a human and a car. The cellphone provides the driver with passenger pickup information and routing instructions to get to a destination. The car provides the transportation. The human operates the car in response to and with the assistance of information coming from the cellphone.

    Right now humans are really good at operating motor vehicles and interacting with the cellphone and passengers very much in the same way I was really good at keeping the orange juice machine full of oranges and dispensing the product to customers. But, just as I was the weak link with the orange juice transaction, so is the human in a rideshare. The cost of a human can vary and things can get sticky in terms of legal matters that go with human employment. Yet, the risk is offset by profit the human produces, for now anyway. But, that can and will change. Take the human out of the equation and the profits go up and the risk goes down, provided the underlying automation technology is advanced enough to meet the demands of the situation.

    If you have any doubt that driverless technology will advance to the point where humans can be replaced, think back to how crappy voice recognition was in 1989. It was really, really bad. Now, voice recognition is really, really good, even in a variety of languages. Getting to that level of technical maturity only took 20 years. Now, imagine how good a driverless vehicle will be 10 years from today. I’ll wager it will be really, really good.

    Yes, some people will point to the current fatalities created by driverless automation to assert that human displacement won’t happen. But, if you have any doubt that it will, keep in mind that the were a lot of crashes early on when the Wright brothers took to the skies in 1903. Yet, by 1919 KLM was flying commercially. That’s a pretty dramatic accomplishment in terms of safety improvement for a technology.

    So the long and the short of it is, just as the owner of the fruit stand would have gladly chosen an orange juice vending machine over me, so too does Uber want to take the human driver out of the car. For all intents and purposes, a rideshare vending machine will be the preferred way to meet the commercial objective at hand.

    So, where does that leave humans?

    The conventional wisdom is that as human labor continues to get replaced by machine automation, there will be new, unimagined jobs for those displaced workers. I think otherwise.

    Consider this: around the 1880’s there were almost 200,000 horses in the City of New York. Now, if you can find 400, you’re lucky. Why? Because that much horse labor isn’t needed anymore. Some have gone onto other jobs, such as being a thoroughbred competitor at Aqueduct Racetrack or a tourist amusement pulling a carriage through Central Park, but as far as being the engine the moves street-cars, carriages and mounted pedestrians, that time has long passed. Horse labor no longer meets the commercial objectives of transportation.

    So too will it be with humans who earn a living driving a motor vehicle. They will be displaced. Only this time there is a very good chance that there will not be the number of new, unimagined jobs created that will meet the needs of the broad segment of the population being displaced. Yes, there will be new specialized work, but not at the scale necessary to absorb the number of those out of work. After all, once the automobile came along, the 200,000 horses left behind in New York City did not go on to become thoroughbred racers.

    The point of the analogy is this: the more specialized the work becomes, the more difficult it is to quickly acquire the skills necessary to do the work. For example, some displaced drivers might be able to become data scientists quickly, but a significant majority will not. To assume otherwise is folly.

    Addressing the issue of worker displacement due to advanced automation is a serious problem, one that deserves significant attention. Hopefully, we’ll figure out some solutions that have long term viability. Otherwise, we’re in for some big trouble because no matter what, businesses will continue to advance and embrace automation to meet their commercial objectives. As the saying goes, “an ounce of prevention is worth a pound of cure.”

    — Bob Reselman

  • IBM Z Open Editor Support for Language Server Protocol is a Game Changer

    IBM Z Open Editor Support for Language Server Protocol is a Game Changer

    The integrated development environment (IDE) is an indispensable tool for software developers. Before it came along, coding was a laborious, detail ladened undertaking. We’ve become accustomed to the syntax checking and code completion features than even the most basic IDEs provide. These days we tend to forget how hard it was programming with nothing but a rudimentary text editor. Something as simple as finding a missing comma or a misplaced curly bracket that was causing a compilation error could take hours, maybe days should the codebase be big enough. When it came to tracing your way through a seemingly endless chain of functions and classes in order to find the culprit of a runtime error, well…fuggedaboutit! Without the modern IDE, we’d be sunk.

    Well, that was then and this is now. A few years back, a new technology emerged on the technical landscape that’s boosting the power of the IDE, and subsequently the scope of work a developer does and the decisions he or she makes by an order of magnitude. This new technology is Language Server Protocol (LSP).

    LSP Takes the IDE to a Whole New Level

    LSP is a specification developed by Microsoft that standardizes the way certain typical features are implemented within a programming IDE. LSP attempts to consolidate the implementation of IDE commonplace features such as code completion, hover-over tips, jump-to-definition, find-references and syntax checking into a standard format that can be applied to any IDE in an agnostic manner.

    LSP is a technology designed to address the following problem. Imagine that a tools vendor has an idea for implementing a new plugin that will automatically do code completion for programs written in the COBOL programming language. Such a feature might be useful to any number of COBOL programmers. However, not every COBOL programmer is programming using the same tool. Some might be using an IDE such as IBM Developer for z/OS. Another might be using a text editor such as Sublime Text, Atom or even something more basic such as vim. In earlier times, in order to get the feature to market the vendor needed to implement different versions of the plugin in a way that was special to the given development environment. There’d be a team working on a Sublime Text version of the plugin, another team working on the vim plugin so forth and so on. Not only did the tool developer need to accommodate different operating systems, but each team might come up with its own, particular way to implement the feature. As team members left the company and new members arrived to replace them, those new developers needed to learn each particular way the feature was realized. As you can see, not only did the variety breed complexity, but things got very expensive very quickly.

    LSP does away with the need for such specialized tools development. LSP provides a standard protocol that can be implemented in any tool. The protocol defines a set of standard methods, objects and communication methods that provide a common set of services in any IDE.

    Thus, the only work that needs to be done to get the previously described COBOL code complete feature to market is to express it in a way that is compatible with the LSP. The actual work of realizing the feature then involves writing a client that supports the standard. It’s a “one ring to rule them all” process so to speak.

    The LSP specification was originally published by Microsoft specific. But the standard has been embraced as an industry-wide. There are implementations in Java, C++, C#, Python, Scala, Swift and PHP, to name a few. Also, there is an implementation of LSP for COBOL under z/OS, which brings us to IBM Z Open Editor.

    Bringing the Use of LSP to the Mainframe

    IBM Z Open Editor is a free Visual Studio Code extension that provides a state of the art programming experience for COBOL and PL/1 developers. Also, IBM Z Open Editor integrates with Eclipse Che and Eclipse Theia. Integration with Eclipse Che and Theia is particularly interesting because it enables browser-based editing. It’s a powerful tool, even more so given that historically COBOL and PL/1 programmers have lived in the backwaters of modern IDE development. IBM Z Open Editor makes IDE-based software development a general programming experience. Developers can use the same tool to program COBOL as they do in Java or even HTML for that matter.

    LSP Enabled Features in IBM Z Open Editor

    The following is a list of tool features available to mainframe programmers using Visual Studio Code with IBM Z Open Editor enabled:

    • Code completion.
    • Hover over instructions and snippets.
    • Jump to definitions.
    • Workspace symbols.
    • Find references.
    • Diagnostics and syntax error detection.
    • Outline view.
    • Previewing of included copybooks on hover.
    • Navigate to copybooks.
    • Code template snippets.
    • Refactoring such as “rename symbol.”
    • Variable completion.

     Using a single IDE to program in a variety of languages is a trend that’s been evolving for a while. However, adding LSP into the mix allows any IDE to present a consistent set of design time features. For example, one IDE is now able to do code-completion, hover-over reporting and variable tracing regardless of the programming language in play.

    For example, Figure 1 below shows the COBOL code completion feature of IBM Z Open Editor plugin for Visual Studio Code. IBM Z Open Editor uses LSP to facilitate code completion.

    Figure 1: IBM Z Open Editor uses the Language Server Protocol to facilitate context-sensitive code completion.

    Figure 2 shows the search and sort features of IBM Z Open Editor, which again implemented according to the LSP.

    Figure 2: Using LSP to implement IBM Z Open Editor’s sortable and searchable outline view for navigation.

    Finally, Figure 3 shows how the IBM Z Open Editor takes advantage of the extensibility capabilities of LSP to implement a feature that resolves and loads copybooks for preview. A copybook is code that can be copied from a master source and inserted into several different programs or into a variety of places in a single program. A copybook is used to define the physical layout of program data, variables for example, as well as instances of procedural code.

    Figure 3: IBM Z Open Editor extends LSP to review and load a copybook within Visual Studio Code.

    Clearly, LSP brings a lot of power to the mainframe programming experience. Now tools developers can create a unified programming experience using a common methodology. But where things really get interesting is when LSP is used to report data that goes beyond standard programming and code management activities, to a place where the real-time behavior of those using deployed code is being reported. In other words, LSP makes it possible not only to get information about code that is local to the developer’s desktop but also about the code as it is being used at runtime out on the internet.

    Beyond the Mainframe: The Long Term Significance of LSP

    LSP is extensible by design. As you read above, IBM Z Open Editor extends LSP to allow programmers to preview and load copybooks. This is done by extending the protocol.

    The typical IDE to server interaction under LSP is that the LSP client reports to the LSP server when a code document has been opened, changed, saved and closed. Then, once the LSP server is aware that a code document is in play, the server can inspect the document and report back events that are relevant to the programming language (Figure 4).

    Figure 4: LSP uses a predefined set of commands and request/response objects to support programming IDE is a language and operating system agnostic manner.

    For example, the LSP server can inspect the document to make sure the code is syntactically correct. If an error is encountered the server reports back the starting and ending location of the error, along with a message. In the case of the copybook review feature, the client reports back the location of the cursor in the code window and the server determines an opportunity for a copybook review and informs the client accordingly.

    While the copybook preview feature is powerful, it’s confined in scope to the immediate document. This is good. But, imagine what it would be like if the server had knowledge of all instances of the code deployed and running on computers connected to the internet?

    Suppose for example, that your code was powering an API running on the internet and you had the ability to learn about the number of calls to a particular function backing the API or better yet, imagine that you had the ability to learn about function calls that encountered particular problems, such as a pattern of mistyped parameters or high latency calls. Then imagine if that global runtime behavior was reported right in the IDE? Such information would change the very nature of how a programmer approaches his or her work. More importantly, it would change the way an enterprise manages programmers.

    What if a programmer could decide what to work on based on global runtime behavior reported into the IDE via an LSP extension? For example, going back to the imagined pattern of mistyped parameters, let’s say that when a developer hovers over the related function in IDE a warning icon appeared informing the developer of 1000 instances of mistyped called were made at the global level. The developer decides that such an error rate is unacceptable. Then, he/she examines the developer documentation and discovers that the parameter type definitions for the function are poorly described and that the document needs to be corrected. The developer then makes updating that developer documentation a critical path, part of the day’s work that needs to be delivered immediately.

    Given the power that LSP brings to the programming experience, such a scenario is entirely possible. In fact, there are companies out there such as Apollo Graph Manager that presently allow developers and third parties to register their production APIs in order to collect the sort of global information described above and have it reported in an IDE via LSP. As those of us who have been in the software business for a while have come to learn, once a feature such as global reporting catches on, it’s only a matter of time until it becomes an industry-wide practice. It’s only a matter of time before globally collected runtime metrics delivered to a programmers LSP become commonplace.

    Again, the ramifications are significant. Once a programmer has the ability to make fundamental decisions about feature development and deployment schedules, what then becomes the role of the project manager and the product manager?

    To take the issue one step further, how much power should a programmer be given? And in terms of general business practices, how much should the programmer know about a company’s products? These questions might seem fantastic, but keep in mind at one time companies guarded their source code as a trade secret. Today, more code than not is shared between companies as part of the open-source movement. As a result, the way code is made and a developers work has changed. The same possibility exists due to the growing trend to extend the capabilities of the LSP. The ramifications of LSP go well beyond simply reporting syntax errors and doing code completion. LSP will change the way the entire industry works, from mainframe programming to developers working in the cloud or on commodity hardware.

    Putting it All Together

    Given current trends, there’s little doubt that LSP will make even the most rudimentary code editor a powerhouse tool for software development. Standardizing IDE features into a common format allows new features for any development tool to be implemented faster and at a reduced cost. In no time at all we might even figure out a way to use LSP to make is so large bodies of code can be written automatically with just a few guiding keystrokes from the developer’s keyboard no matter the programming language. It’s a real possibility.

    Not only will LSP making coding easier and faster, but it’s also going to change the role of the software developer. Once all the code running on the internet becomes LSP aware and accessible, developers will get runtime information about how their code is performing at runtime on a global level. The trend is already in play. As history has shown, the further down a technology goes into a given population, the more powerful that population becomes and the greater the ramifications when the power is exerted. There’s a very real possibility the information that LSP enabled IDEs provides will make the day-to-day programmer more prominent in the product decision-making processes that are presently the responsibilities of program and project managers. Once that shift takes place we’re likely to see another industry-wide transformation take place in the way software is made and delivered. History has shown that such transformations have long term benefit, but in the short term are disruptive and disturbing. We’ll do well to be prepared.

    — Bob Reselman

  • TDD, Unit Testing and the Mainframe

    TDD, Unit Testing and the Mainframe

    Unit testing is like exercise: Companies know it’s good for you and you should do it, but given their druthers, they’d rather avoid the effort and just get to the benefit.

    I wish I could say it’s otherwise. After all, making unit testing a standard practice in a company’s software development process is probably the first, best thing a business can do to significantly boost code quality. To go one step further, adopting the best practices of Test Driven Development (TDD) can not only speed up release velocity in the long term but also has the benefit of making code maintenance less costly. For example, if you want to know what a piece of code is supposed to do, all you need to do is review the unit tests. This reduces the need for volumes of support documentation.

    It’s almost a no brainer. Yet, there’s still a lot of perception out there that TDD’s dictum to write the test first slows down the development process. The thinking goes that if a developer is going to write a test for the code before writing the code, just save time and write the code anyway and then write the tests only for critical parts of the code.

    This might be a sound rebuttal of the TDD process, except for two things. First, can you name a part of your codebase that should not be tested? And second, as those of us out on the terrain have discovered, when release time approaches and the corners start to get cut–and they do get cut–many times the first thing to go is unit testing. More than one development manager has uttered the words “We’ll catch it in QA” when faced with an approaching deadline and an understaffed development team. Sadly, when given the choice between getting so-so code out the door on time and delaying the release in order to create great code, it’s been my experience that in companies that have not embraced unit testing to an acceptable code-coverage threshold, so-so code will win every time. Those who have taken a swig of the TDD Kool-Aid don’t have to make that choice. The tests are already written.

    It’s true that all the unit testing in the world will not make poorly engineered code better. But, if the engineering is sound, unit testing works. Still, companies resist, even those who program in languages from which unit testing as a way of life emerged. There are a lot of businesses out there programming Java, C#, Python, etc., that want the benefit of unit testing and yet don’t want to pay for the effort. When it comes to the mainframe community, resistance be as strong as that found in the French Underground in WWII.

    Why? Well, when it comes to mainframe development, unit testing is much harder to implement than in the x86 world. There’s the problem of language compatibility. There is still a lot of COBOL out there supporting the 80% of the world’s data that still resides on a mainframe. Unit testing as a way of life is a fairly recent occurrence in the history of computing. There’s a boatload of testing libraries out there for Java, C/C++, .NET, Python. But until recently, there weren’t any unit testing frameworks for COBOL. Also, in traditional mainframe programming, you can have situations in which many developers work on the same body of code. Test isolation can be hard.

    Finally, there’s the matter of culture. Remember, the thing that makes unit testing and TDD work is that designing and executing tests are the responsibility of the developer. Pushing testing onto developers is a notion at odds with traditional mainframe sensibilities. In traditional mainframe shops in days of old, developers wrote the code and QA tested it. It’s a process that worked for years as long as those years meant you could release code every six months. Today it’s all about getting code revisions out weekly, if not daily. Still, the Dev/QA segmentation is there in the older companies that are mainframe driven.

    But that was then and this is now. I am happy to report it’s a new day in the land of software development of the mainframe. Adopting TDD in general and unit testing in particular as a way of life in mainframe software development is not only possible but it borders on being, as I describe above, a no brainer. The tools and processes are there. Let’s take a look.

    Unit Testing Mainframe Code Using IBM Z Open Unit Test

    Writing unit tests for the mainframe has had a difficult history. If you’ve started your professional career in software development within the last ten years, you’ve had the pleasure of enjoying the benefit of TDD and CI/CD as a standard way of doing business. Tools such as Jenkins, JUnit, Selenium and the like have always been around. But in the mainframe world, for many, this is all new stuff.

    As a result, for many mainframe developers, the notion of writing a unit test let alone integrating one into a CI/CD process means entering a strange new world. In the pre-tool world, writing any test, let alone a unit test, meant spending days doing tedious, laborious work in some sort of jury-rigged testing environment. Or, in some cases, the developer might not be responsible for testing the code at all. Rather, the developer writes the code and gets it working to his or her satisfaction by any means necessary. Then he or she gives the code the old “toss over the wall” to the QA department for testing. What those testing processes are might be known to outsiders, but in many cases they’re not. So, the development process becomes at best a slow, grinding exchange of test results and remedies between groups that have limited insights into the processes and priorities of the other.

    The good news is those in the known have come to understand there’s a better way. The better way involves three things. First, making the developer responsible for testing the code he or she writes. Second, removing the cultural, and in many cases, the organizational wall between development and QA. And third, putting tools at the forefront of testing activity.

    In other words, the new thinking is to understand testing must be done as early as possible in the software development cycle (hence, the Shift Left Movement), everybody is responsible for the quality of the software and tool adoption is critical in the TDD process.

    The construction industry caught onto this notion a long time ago. A bulldozer driven by a competent operator is a lot better for moving earth than a legion of human beings with shovels. The same is true in software development. It’s better to give a single developer a tool to create and apply tests to his or her code than have a legion of other employees sit around with text editors writing one-off units tests for their code and applying them by hand.

    Which brings us to IBM Unit Test.

    IBM Z Open Unit Test

    IBM Z Open Unit Test is a tool designed to make writing and executing unit tests on the mainframe fast and efficient. Also, IBM Z Open Unit Test facilitates integrating unit tests directly into a company’s CI/CD processes under a tool such as Jenkins.

    The tools support automated unit testing for batch and CICS COBOL programs. Also, IBM Z Open Unit Test provides recording and playback functionality for CICS COBOL programs, as shown in Figure 1. This tooling allows developers to easily record external calls to CICS and import that recorded data for modification in playback. The ability to record, modify and playback test data at execution time means developers can run their automated unit tests as part of a build (or, as a batch job). There is no need to deploy to CICS, as it is now stubbed out.

    IBM Z Open Unit Test aligns with TDD methodology in that the tool allows developers to record, view and modify the input and expected output as assertions, within an easy to use interface. This can be done before making a change to a program, and developers can understand what kind of data is flowing in and out.

    Figure 1: IBM Z Open Unit Test has data recording and playback capabilities for Batch and CICS COBOL programs.

    As mentioned above, IBM Z Open Unit Test can be run from within an IDE or executed as part of the enterprise’s CI/CD process. Also, the product supports test debugging and importing tests written in mainstream formats such as JUnit and SonarQube. These are good and important features. But the real kicker is support for code coverage.

    Why Code Coverage Counts, Even for Mainframe Testing

    Code coverage is the elephant in the room in terms of testing administration and management, particularly around unit testing. One of the key principles of test driven development is that every line of code that gets written gets tested. Yet this is the dream rather than the practice. I’ve been in more than a few companies that say they support TDD, yet when asked if they test every line of code written, which translates into 100% code coverage, the answer usually involves something along the lines of “we’re getting there.”

    Many companies are tickled pink that their developers are doing any unit tests at all. In such companies, asking a developer to get to 100% coverage is akin to asking him or her to go to the moon and back during lunch.

    Some companies that have matured a bit set the coverage standard at 80%. This means that as long as eight out of every 10 lines of code written get touched by a unit test, things are just fine. Or are they? What if the untested two lines of code is a short else clause in an if-then statement that eats memory and blocks I/O? Are these the two lines of code you want to leave untouched in a test? Remember this: Typically it’s not the tested code that wreaks havoc, it’s the code that’s untested.

    Thus, the mission critical importance of 100% code coverage.

    Mainframe development has lagged a bit embracing the imperative to ensure 100% code coverage. This is mostly due to the culture that went with legacy software development practices and the immaturity of tools at the time. The good news is both culture and tools are evolving.

    Getting back to IBM Z Open Unit Test, the tool not only provides test coverage reporting, which is critical for providing the metrics necessary to get to 100% coverage, but also a feature that supports negative testing.

    Negative testing is also known as an unhappy path testing. It’s an important, yet often overlooked aspect of unit testing. Just getting a piece of code to work under the right conditions is hard enough work. To subject that code to every bad condition (unhappy path) that can happen and make sure it still works, requires significantly more work. Thus, when crunch time comes, and it always comes, typically the developer will let negative, unhappy path, testing go. It’s not malicious intent. It’s about meeting a deadline.

    Fortunately, this dynamic is a known problem in the industry. IBM Z Open Unit Test addresses the problem by providing a feature that automatically generates negative tests. This is a big deal. In fact, if this is all the tool did, it would be a boon to the practice of TDD on the mainframe. Writing negative tests is a tedious, laborious activity. Using a tool such as IBM Unit Test not only saves time but helps achieve the ultimate goal: 100% code coverage.

    Putting It All Together

    Unit testing is an essential part of the practice of TDD. While TDD has taken hold throughout the x86 based distributed computing, there has been a lag to embrace the discipline in mainframe computing.

    Mainframe computing is becoming very cool again, if for no other reasons that it presently controls 80% of the world’s data and in many ways was the cloud before the cloud. In the past, TDD has been hard to implement in the mainframe world due to a lack of tools to support the discipline. Fortunately, with modern tools such as IBM Z Open Unit Test, the benefits of TDD can be easily enjoyed in the mainframe development community.

    — Bob Reselman

  • CAP Theorem No Match for Mainframes

    CAP Theorem No Match for Mainframes

    One of the biggest problems in modern distributed computing is data management. These days data are spread out all over the place. As a result, getting the data you need, exactly when you need it, becomes a problem. Wider distribution creates more network latency. If you have a system in which some machines are in California and others are in New York, logic dictates it’s going to take more time for data to travel coast to coast than from within the same geolocation. It’s a matter of common sense.

    The data management problems created by distributed systems became more pronounced with the growth of the internet, which is, by nature, highly distributed. Eric Brewer, a computer scientist at the University of California, Berkeley, published a paper in 1999 that describes the dynamics of data management in a distributed system as a set of immutable constraints. Brewer published the work as the CAP principal. Three years later Seth Gilbert and Nancy Lynch at MIT proved the principal to be a theorem. Today, the CAP theorem is a central consideration when designing and implementing applications at web-scale.

    Simply put, the CAP theorem asserts that when your data is distributed over a variety of machines and geolocations, if you want the latest, most accurate data, you’re going to have to wait. And, if you don’t want to wait, the data you get will probably be stale. However, if data resides on a single machine, then everything changes. In such a case, you can have the latest data on demand, at any time.

    Of course, having a single data source implies having a machine that supports the demands of web-scale. Sadly, the x86 machine has yet to be made that can fit the bill. But a mainframe can. And, not only can it handle an enormous operational load, under the right set of circumstances it also can be a cost-effective way to meet the need at hand.

    Allow me to elaborate.

    The CAP Theorem Explained

    Before we go into the details as to why a mainframe computer such as an IBM Z or LinuxONE can make the problems presented by the CAP theorem go away at a reasonable price, let’s look at the theorem itself.

    Essentially, the CAP theorem asserts that of the three aspects of data management in a distributed system—consistency (C), availability (A) and partition tolerance (P)—you can have only two at any point in time.

    Figure 1: The CAP theorem dictates that of the three aspects of data management in a distributed system, only two are available at any point in time.

    In other words, if you want the latest information (consistency) from your data source immediately (availability), that information needs to be stored in a single, fail-safe location (partition tolerance). If you want to spread the data out over many machines (partition tolerance), either the data will be accurate at some point in time but not immediately (consistency) or you can have data immediately but it might be stale, awaiting an update to the most recent state (availability).

    The CAP theorem is more than theoretical. It has an actual operational impact on companies. A telling example is a problem that Netflix experienced when implementing its Monet system.

    Netflix Gets CAP-Ped

    Back around 2018, Netflix decided to implement an internal ad management tool under design using GraphQL, an open source API specification originated at Facebook that was becoming popular in many companies that operate at web-scale. In fact, GraphQL is still growing in popularity. So Netflix decided to take the plunge toward adopting GraphQL with an internal project named Monet.

    The way Netflix decided to implement Monet was to have the GraphQL API server act as a facade for existing RESTful APIs. The thinking was to reap the benefits provided by GraphQL, yet reuse as much existing code as possible. The idea was good, the actual deployment fell short. Once Monet was put into production, it ran dog slow.

    It turns out the REST APIs were distributed among a variety of data centers (see Figure 2). Network latency was killing them.

    Figure 2: Initially Netflix’s Monet system queried data from a variety of data centers.

    The company’s solution was to bring all the REST APIs into a single data center (see Figure 3).

    Figure 3: Netflix improved Monet’s performance by moving all RESTful APIs into a single data center.

    The result? Once all data resided in the same data center, performance improved dramatically. It makes sense. Shorter distance to travel means less latency in the system overall.

    Now, how this relates to the CAP theorem is interesting. Remember, according to CAP, as partitioning goes down (P), consistency (C) and availability (A) go up. So, while it’s true that Netflix was not using a single machine as a data source, moving everything into a single data center created what one could consider a single virtual data source. Yes, there might be latency issues within the data center. But, rack-to-rack latency is nowhere near as high as the latency between geolocations. We’re talking the difference of 1 to 10 milliseconds as opposed to hundreds, if not thousands, of milliseconds.

    In Netflix’s case, the company addressed the constraints imposed by the CAP theorem by reducing the breadth of partition at the geolocation level. All the data lived in the same data center. Imagine if partition size could be reduced at the machine level. What if instead of having data distributed among many machines within the same data center, data actually resided within a single machine? Partition size would be reduced to one. Then, applications could have the latest data immediately. All that’s needed is a single machine that can handle the load. This is where using a mainframe comes into play.

    Mainframes Are Built for Big Data and Agile DevOps

    Mainframes since their inception were intended to be able to store and process enormous amounts of data. Today 80% of the world’s data still resides on mainframes. As mentioned in a previous article about this topic, one mainframe can process 2.5 billion transactions daily. That’s the equivalent of 100 Cyber Mondays in a single day. That’s a lot of computing power easily equivalent to more than a few racks of commodity hardware in a data center. In fact, in some situations, one mainframe can provide the computing power found in an entire data center.

    So, let’s go back to the Netflix problem. What if instead refactoring those distributed RESTful APIs into a single data center, the time and money spent was used to refactor those RESTful APIs into a single mainframe? Yes, there are some technical hurdle to overcome, but considering the running Linux on a mainframe is standard these days, it’s not as if we’re talking transforming a bunch of Python code into COBOL. Databases from standard RDBMS such as DB2, Oracle to the NoSQL MongoDB can run perfectly well on a mainframe. There’s work to be done, but reinventing the wheel is not one of the tasks. Using a mainframe is a viable solution except for one sticky little problem: The conventional thinking is that using a mainframe is prohibitively expensive. Thus, the bias toward racks of commodity hardware or simply going with a cloud service.

    Mainframes as a Cost-Effective Solution

    The general perception in most IT shops is only big banks, insurance companies, governments and large-scale manufacturers use mainframe computers. They’re just too expensive for small to medium-size business. While many mainframes come with a price tag that can exceed $1 million, there are still surprisingly affordable models. For example, an entry-level IBM z114 will run about $75,000.

    If you’re a company spending $50,000 a month on AWS or Google Cloud computing resources, then for the $600,000 a year cost you’re spending on a cloud provider and the latency that comes with it, your company can well afford not only an entry-level IBM Z system to handle a production load, but also a second one to serve as backup, should failure occur.

    In terms of system administration, just about all the tools commonly used to support modern CI/CD processes today in the DevOps world have corresponding equivalents for mainframe implementation. As with any adoption of unfamiliar technology, there is a learning curve. But, again, we’re not reinventing the wheel. The tools, resources and expertise exist. Adoption is just a matter of following standard change management practices typical in medium-to-large-size companies that are well-positioned and use a mainframe to take advantage of implementing a cost-effective, no-partition data source strategy.

    Integrating Mainframe Technology into a DevOps-Driven Ecosystem

    Mainframes are capable of doing DevOps in an open, agile and collaborative fashion. The tools, resources and expertise required to support a mainframe’s participation in CI/CD processes exist today. For example, Zowe is an open source software framework that provides solutions that allow development and operations teams to securely manage, control and develop on the mainframe like any other platform. DB2 DevOps Experience for z/OS is one of the first commercial products built on Zowe. It allows IT the control over the mainframe database, its business rules and standards, while allowing development teams the ability to work independently using self-services and integration with modern CI/CD tools to speed their application release time with better quality and development practices.

    When you think about it, in certain mission-critical situations, using a mainframe to have the latest data at any time not only makes sense from a technological point of view, but it’s also attractive in terms of return on investment. The trick is to be able to identify the right use case. Of course, if you have a situation in which 90% of your users are in California and you have a mainframe situated in Bangalore, the laws of physics will apply. Moving the bits from the mainframe to user will take time. But, if you have situations in which user and data are in continental proximity, using a mainframe as a standard data source begins to look very attractive.

    Again, having the right use case is key. But, everything else being equal, using a mainframe for sensitive, data-driven applications warrants significant consideration.

    Putting It All Together

    The constraints on distributed computing imposed by the CAP theorem remain a challenge to those who design and implement large applications at web-scale. The “pick two” limitation CAP imposes can be an annoyance for simpler websites that provide content intended for the general consumer. But, for large scale, data-intensive applications, having accurate data immediately is not optional, it’s essential. Thus, using mainframe technology to create a no-partition data infrastructure becomes a viable way to address the limitations imposed by the CAP theorem. Under the right conditions, making a mainframe the central repository for your application’s data provides a significant return on investment hard to match when compared to commodity-based cloud providers.

    The nice thing about using a mainframe is when it comes to the CAP theorem, you still only get to pick two, but the two you get to pick are consistency and availability. In the scheme of things, those are the two that count.

    — Bob Reselman

  • Technology: Isolation in the Age of Automation

    Technology: Isolation in the Age of Automation

    Dependence on technology makes us more independent overall, but it also makes us more isolated. So, what happens if things come to the point where we depend on machines more than we depend on other human beings?

    Earlier this month I attended a presentation about machine learning on mobile phones at Google IO 2019. Right now Google is spending a lot of time and money developing machine learning models and capabilities that operate solely within the mobile device. No connection to the internet is required. It’s a big deal.

    During the presentation, the company showed a video that featured an illiterate woman in India who uses her phone’s text recognition features to read to her. All she needs to do is point the phone’s camera at a recipe or newspaper and the phone converts the text to speech.

    I found the story inspiring as a technologist and as a human being. But it’s not all peaches and cream. At the end of the video, the woman said something that gave me pause. When talking about using the technology to book a train ticket, she said, “I can book it myself. I don’t need anyone’s help.”

    There’s no doubt the phone liberated her, very much in the same way that the light bulb liberated millions of other people in times past, bringing human activity forward beyond the setting sun. Dependence on a particular technology can create greater independence overall. However, I do wonder what happens when things evolve to the point where we depend on machines more than we depend on others, when we no longer need “anyone’s help.”

    To my thinking, human interdependence is the thread that binds the fabric of a society together. The thousands of little interdependencies that make up the human experience have beneficial side effects. To get along, we learn the basic rules of civility. We learn to say “please” and “thank you.” We learn how to wait in line at the DMV. We pick up after our dog. We open the door for others. It’s all part of the implicit contract that makes living with other people possible.

    But, these days it seems we depend more on machines than the kindness of strangers. The fact is, you don’t have to be nice to a machine, even for a self-serving purpose. You can yell at your cell phone. You stick your tongue out at the ATM as it accepts your deposit. Whether you’re nice to it or not doesn’t matter. Your deposit is not going to be processed any faster or slower. Your cell phone isn’t going to disconnect you because you were rude to it.

    Take away these simple rules of civility and strange things happen. It’s interesting that road rage is common, yet sidewalk rage is rare. Maybe wrapping ourselves up in a piece of silicon-powered transportation hardware predisposes us to a hostile attitude when interacting with others in the same situation.

    As I stated earlier, depending on technology makes us more independent in the big picture. But, it also makes us more isolated. Whereas in the past, talking about the weather with a stranger might be a prelude to another conversation that’s more meaningful, today the stranger next to me usually has his ears plugged up listening to his own interests. And, if I really want to know about the weather, I can ask Alexa.

    Modern technology gives me help on demand. I no longer ask a stranger for directions to the train station, I use my phone. It seems as if I need the help of others less during those small, random moments of vulnerability. A machine is but a voice prompt away. And, thus, the thread that binds me to the world at large gets a little bit looser.

    Now don’t get me wrong—isolation is nothing new. Five hundred years ago most people never traveled far from where they were born except on very special occasions. Village life is a case study in group isolation. But, while villages were isolated from the outside world, inside the village it was a different story. The social fabric was dense; the threads were tight. Being shunned by the group was a matter of life and death.

    Today we’re all over the place, literally. But we move in our own domains in which we end up wishing happy birthdays to people we’ve never met but who somehow have meaning to us by way of LinkedIn, Facebook and the plethora of other social platforms brought to life and sustained by something other than human labor.

    Is this such a bad thing? Dunno. I like it that my phone is now my personal stenographer. It helps me think better. But, I’m not sure that Google Assistant will ever be a good best friend. What’s even stranger is that a time might come when I don’t even need one. It will be just me and my mobile device. Then I won’t need anyone’s help, ever.

    — Bob Reselman