devopsy.kinsta.cloud

Category: Features

Atlassian Builds a DevOps Stack for the Enterprise

One of the major challenges associated with DevOps is that it’s not something an enterprise IT organization can buy. Because of that issue, it’s often challenging for IT organizations to forecast what types of DevOps tools they are going to need. To make it a lot easier to cope with that issue, Atlassian has announced Atlassian Stack, a suite of software that makes Atlassian DevOps tools available in a single package.

Sean Regan, head of product marketing for Atlassian, says this approach will make it easier for enterprise IT organizations to embark on a DevOps journey without having to worry about what type of tool they may need now versus in the future. For example, an IT organization may opt to get started using the Atlassian Jira software to track software development projects and then work its way up to deploying Atlassian Bamboo continuous integration/continuous deployment (CI/CD) software. From there, Regan says, a DevOps team may opt to embrace collaboration software to streamline communications across their various teams.

The move to launch Atlassian Stack comes after the company updated a slew of its DevOps offerings, including the release of Bitbucket Server and Data Center 5.0 and Bamboo 6.0. In addition to tightening the integration between both DevOps tools, the company added capabilities for verifying commits, smart mirroring and the ability to define a Bamboo configuration as code.

Rather than making DevOps teams acquire each tool individually, he says, many senior IT leaders now request a means through which the entire Atlassian portfolio can be made more readily available. That shift in purchasing behavior suggests that the level of DevOps maturity inside many organizations is advancing considerably, he notes.

Of course, there are many who would contend that level of DevOps maturity has been a long time in coming. Web-scale companies have been employing DevOps tools for years to accelerate the rate at which new applications are released and updated. In contrast, traditional enterprise IT organizations have been slow to implement DevOps processes. In fact, it’s taken the pressure arising from digital business transformation initiatives that have accelerated application deployment timetables to get enterprise IT organizations to fundamentally rethink IT operations. Business leaders are making it painfully obvious that their patience with extended rollouts for new applications is at an end.

Naturally, the rate at which those transformations are occurring within organizations will still vary considerably. However, the silos that exist within enterprise IT organizations need to be consolidated. The next issue is determining to what degree altering those processes needs to come in the form of a DevOps edict from above versus letting teams come together of their own accord to meet a specific application deadline. Whatever the approach, the way enterprise IT has been managed historically is no longer up to the digital business transformation tasks at hand.

— Mike Vizard

June 15, 2017
Node.js Becoming More DevOps-Friendly

Over the last several years use of Node.js has exploded across the enterprise, as developers moved to embrace a variant of Javascript that made it easier to build both client and server-side applications. However, every time it is upgraded, all the dependencies between it and application modules written in C or C++ breaks.

To solve that problem, the Node.js Foundation is making available on an experimental basis a Node.js API (N-API), developed collaboratively by Google, IBM, Intel, Microsoft, nearFrom, NodeSource and other contributors. The API will be in Node.js 8, and promises to eliminate the need to update all those dependencies. Pending additional testing, it is expected to be given Long Term Support (LTS) status later this year.

At the same time, The Linux Foundation is adding new features to experimental tracing and async tracking features. A diagnostics API will make it easier to monitor the operation of the Node.js event loop across the complete life cycle of a service request. Finally, an inspector core module, which allows developers to leverage the debug protocol used by the Google Chrome inspector to inspect currently running JavaScript code, is also being added.

Each of these new capabilities are nice to have on their own. But collectively, they signal the beginning of a maturation process in the Node.js community. Increased use of Javascript in general—and Node.js especially—have arguably caused many enterprise IT organizations re-evaluate their DevOps processes. Developers using Node.js have been able to increase the number of applications they could develop considerably. Since then, the number of languages being employed within the enterprise has exploded.

Mark Hinkle, executive director of the Node.js Foundation, says moving forward there will be more focus on reducing the friction surrounding upgrades of Node.js applications as part of an effort to further expand the ecosystem. Many organizations commonly end up supporting multiple versions running across a variety of applications. Hinkle says N-API is a step in helping to reduce the maintenance costs associated with supporting those applications. In an ideal world, the foundation would be gathering additional input from DevOps experts pursuant to achieving that goal. In most IT organizations, the care and feeding of Node.js applications is still left in the hands of IT operations teams after the apps are deployed in production.

In the meantime, Hinkle says its performance continues to improve. Naturally, there’s a corresponding relationship between better performance and the number of use cases for which it will be considered. Of course, that doesn’t mean Java and other legacy programming languages won’t be used to develop new applications. But it does means that in terms of the number of programming languages employed across the enterprise, it’s now a polyglot universe.

— Mike Vizard

June 13, 2017
Survey: DevOps Divide Grows a Little Wider

The gap between the number of organizations that have invested in DevOps is starting to narrow, and IT organizations that made the transition to DevOps earlier are starting to see accelerated returns on that investment, according to the latest survey by Puppet.

The annual “State of DevOps Report” finds that out of 3,200 IT professionals surveyed, only 11 percent were part of organizations that could be considered high DevOps performers. High performers are deemed to have higher deployment frequencies, lower lead times for changes, better mean time to recovery (MTTR) and better change failure rates. The higher performers in this year’s survey are deploying applications 46 times faster than the same designated group of higher performers were last year.

The survey also finds that higher-performing organizations have automated 72 percent of all configuration management processes. Those same high performers spend much less time (28 percent) employing manual configuration processes. Lower performers are spending almost half of their time (46 percent) on manual configuration.

Just over a quarter (27 percent) of the respondents self-identified themselves as being part of a DevOps team, according to the survey. In contrast, only 16 percent said the same last year. Puppet CTO Nigel Kersten says those results suggest that understanding of the power of DevOps is increasing. It’s also apparent the early implementers of DevOps processes are enjoying a level of IT agility that is much greater than the average IT organization.

In general, Kersten says that while much DevOps progress has been made, most IT organizations have automated only various pockets of silos. Developers, for example, may have automated some processes on their own, while IT operations specialists may have developed a custom script to automate the configuration of a handful of servers. Kersten notes IT leaders should be cognizant of the technical debt their organizations are incurring every time different elements of the IT organization employ different tools to automate the same basic processes time and again.

Very few have automated their operations within the context of a continuous integration/continuous deployment (CI/CD) pipeline, says Kersten. That becomes more challenging to accomplish as new computing models based on containers, for example, are added to the enterprise mix. On a more positive note, Kersten says more processes are being well-defined, as they are bounded by an application programming interface (APIs). Those APIs then become the core technology over which IT automation frameworks are then laid.

The issue many IT organizations have yet to confront is the degree to which they want to stumble toward DevOps versus implementing a concerted plan of action. Much of the DevOps progress to date has been driven from the bottom up inside many organizations. The time is now approaching for IT leaders to imbue DevOps as a part of the IT culture that permeates the entire organization.

To one degree or another it’s a safe bet every organization will implement some form of DevOps going forward. The challenge will be figuring out how to transform all those isolated instances into something more akin to an actual IT strategy.

— Mike Vizard

June 12, 2017
Q&A: Michael Beckley, CTO/CCO, Appian

Recently I had the chance to speak with Michael Beckley, chief technical officer and chief customer officer at Appian, about his company’s move to go public and the DevOps space in general. For those of you who don’t know Michael, he is a founder of Appian and leads the company’s technology vision and oversees customer initiatives worldwide. He also serves on the Industrial Advisory Board for the University of Virginia’s Computer Science Department and is a member of the University of Maryland Advisory Board in Computing. He has an interesting take on the IT space and DevOps in particular. Read on for our discussion.

Alan Shimel: Michael, what do you think has been the “special sauce” that has led to Appian’s success to date?

Michael Beckley: With Appian, companies build unique business applications up to 20x faster and mobile-enable and modernize existing systems. For example, a company like Ryder was able to build a web and native Android tablet application that cut the time it took to rent a truck in half.

But the “special sauce”? That’s Appian’s culture of innovation. Our engineers think long-term and invest in breakthrough tech like our patented SAIL UI technology for zero code native mobile apps. Thanks to SAIL, the web apps you configure in Appian automatically deploy to iOS and Android as native, full-featured apps or specialized micro apps we call “Sites.” As long as we get the culture right, we expect to continue to deliver enduring value for our clients.

Shimel: With becoming a publicly traded entity, some companies have found “managing to the street” does not match their own long-term business plans and their customers needs, how does Appian avoid this?

Beckley: Companies, like people, are an aggregation of their habits. Appian’s habit has always been to hold to our long-term vision of enabling every company to be a software company. We are not your typical Silicon Valley tech company. Our roots are in DC, we largely bootstrapped this business, and learned to be good stewards of capital while helping to support the core missions of federal agencies and some of the world’s largest brands. The proceeds following this IPO will be invested back into our business to benefit our customers.

Shimel: What do you see post-IPO being the biggest areas for growth as it relates to new products and services?

Beckley: We plan to invest in innovating and enhancing our platform. While our platform is industry-agnostic, we have recently made, and plan to continue to make, investments to enhance the expertise of our sales and marketing organization within our key industry verticals of financial services, health care and government. We have made, and will continue to make, investments in research and development to strengthen our platform and expand the number of features available to our customers. As we continue to increase the functionality of our platform and further reduce the amount of developer skill that is required to build robust applications on our platform, we believe that we have the potential to expand the use of our platform.

Shimel: Digital transformation is a term that is being used a lot these days. How far along the adoption curve are we on digital transformation?

Beckley: While many speculate that digital transformation is just a tech industry buzz word, in reality, digital transformation is a living, breathing effort. According to a recent survey, digital transformation is a C-level priority and is considered a top focus for almost 50 percent of all companies this year. From Appian’s perspective, we believe that the utilization of low-code software to drive digital transformation is still relatively new and our customers will be able to succeed in their digital transformation efforts by incorporating low-code development.

Shimel: Where does the DevOps market intersect with the Appian business model?

Beckley: Organizations are turning to DevOps to accelerate development and bring an application’s capabilities to a user quicker. The Appian low-code development platform fits into the DevOps model and supports the seven major DevOps categories including code, build, test, package, release, configure and monitor.

Overall, our business model supports the DevOps market; we’re living in a world where citizen developers are playing such an important role in the enterprise and our platform empowers the citizen developer while alleviating developer resources to focus on the more challenging, business-critical developer work.

Shimel: What about containerization and microservices? How do these technologies affect Appian?

Beckley: Appian already uses containers and microservices. Appian’s enterprise customers often require a mix of cloud, hybrid cloud, and on-premise installations. We view containers and microservices as enabling technologies to support the diverse workloads and deployment models required by our enterprise clients.

Shimel: What about security? What challenges does cybersec pose to Appian’s development of their platform?

Beckley: Appian works with some of the nation’s most important and highly regulated industries, therefore we have built a comprehensive security and compliance structure that meets several industry standards. We frequently undergo third-party audits to ensure that we are always operating effectively to protect our customer data.

— Alan Shimel

June 7, 2017
VMware Moves to Automate Management of Hybrid Clouds

Moving to expand the reach of its portfolio of IT management software well beyond virtual machines, VMware today announced that the vRealize suite of IT automation tools can be employed across containers and virtual machines running on-premises or in a public cloud.

In addition to providing support for VMware’s Admiral container management portal, version 7.3 of vRealize Automation adds much tighter integration with both Amazon Web Services (AWS) and Microsoft Azure public clouds. For Microsoft Azure, vRealize now can deploy and manage middleware and application services. For AWS, vRealize Network Insight now can microsegment networks in an AWS Virtual Private Cloud as well as troubleshoot network traffic issues in AWS.

From a DevOps perspective, however, the most significant new capability is being added via an update to the vRealize Operations module: the ability to automatically deploy workloads across multiple clusters and datastores based on business policies defined by the IT organization, says Mahesh Kumar, senior director of product marketing for VMware. This update also adds a predictive Distributed Resource Scheduler (pDRS) that leverages analytics provided via vRealize Operations to predict anomalies and identify potential I/O contention issues before they occur.

At the same time, the latest update to VRealize Operations will incorporate a tab for vRealize Business for Cloud 7.3, which has been extended to provide cost and capacity utilization metrics for AWS and Microsoft Azure alongside VMware-based private cloud costs.

Other additions to the VMware portfolio include tighter integration between VRealize Operations and VMware vSAN storage software, NSX network virtualization software, enhanced blueprints for sizing deployments and integration with configuration management tools from Puppet. Kumar says that latter capability will be extended to include support for Chef and Ansible configuration management tools as well.

With these additions to the IT automation portfolio, VMware is clearly signaling its ambitions lie beyond its own hypervisor. Within enterprise IT environments VMware is dominant. That position has enabled it to gain significant traction in terms of driving adoption of its management tools. Now, VMware is moving to extend the reach of those tools into public clouds and containers. At the recent Dell EMC World 2017 conference last month, VMware debuted instances of its portfolio of software running on AWS. Most containers today are deployed either on top of a virtual machine or in a platform-as-a-service environment. But VMware is also aware of the fact that over time more containers will be deployed on bare-metal servers that are not running any type of virtual machine software at all.

Of course, just about every other provider of an IT management framework knows this as well. In addition, there’s no shortage of management tools designed specifically for containers. But as DevOps continues to evolve, most IT organizations are naturally expected to favor approaches that enable them to manage the broadest number of platforms possible.

— Mike Vizard

June 6, 2017
IBM: DevOps, AI to Drive Digital Business Transformation
Conventional wisdom holds that one of the biggest challenges organizations face today is a chronic shortage of application developers. But as artificial intelligence (AI) advances, the process of writing code will become more automated. But rather than there being not enough developers to write all the code necessary, the shortage will be in software engineers who can develop innovative systems.

Bob Lord, chief digital officer for the Digital Business Group at IBM, says a glimpse of the future of application development is already being provided today. Instead of writing low-level code, most developers today simply combine modules to create new applications. Anytime new functionality is required, another module gets dropped into the application. The number of developers writing the low-level code needed to create those reusable modules is relatively few, says Lord.

Over time, however, AI models will learn how to write that code. People will add value by providing the engineering expertise needed to combine modules to create an innovative application, Lord says. Naturally, most of the processes employed to deploy that software also will be automated. But IT operations teams will be needed to continuously train the AI systems being used to deploy that software. This is in much the same way that we need human pilots to fly a plane whenever something new or different occurs, despite the existence of autopilot technology.

In general, Lord says, IT organizations would be well-advised to focus on three things:
- They need to decide what architecture strategy is going to be employed to accelerate DevOps integration.
- They need to develop a data strategy that drives digital business transformation.
- They need to layer a cognitive computing strategy that employs advanced algorithms to maximize the business value of the data they’ve accumulated.
While some organizations approach these issues in isolation, Lord says the reality is they’re all closely connected. Organizations are investing in DevOps not just to reduce costs—DevOps is about attaining a level of IT agility capable of driving digital business transformations. In fact, Lord believes the way IT organizations are measured will change: Rather than being rated on traditional IT metrics, they will be measured on how effective they were in attaining business goals—especially when to comes to velocity at which applications are rolled out and updated.

In the meantime, Lord says, developers are starting to exercise more influence over the technologies organizations are employing to achieve those goals. Because of that issue, IBM is making an effort to get closer to the developer community. But as application development advances, many of those developers will be software engineers rather than people who write code, while IT operations teams will become teachers who train intelligent systems how to respond to the needs of deployed code.

— Mike Vizard
May 30, 2017
Chef Applies IT Automation to Meet Compliance Mandates

Much of the productivity waste associated with compliance requirements stems from the fact that most developers and IT operations teams aren’t aware how an application might be non-compliant until it’s too late. Aiming to eliminate that fundamentally inefficient approach to meeting compliance mandates, Chef this week tightened the integration between its Automate continuous automation platform and InSpec, an open-source project lead by Chef for specifying compliance and security policies.

At the same time, Chef enhanced support for Docker within InSpec and released InSpec-AWS, InSpec-Azure and InSpec-vSphere as incubation projects that IT operations teams can employ to test and audit for compliance. Chef also pledged to extend InSpec beyond operating systems to include support for middleware to address compliance issues across an entire platform. The announcements were made at the ChefConf 2017 conference.

Chef CEO says Barry Crist says that in much the same way Chef automation software turns infrastructure into code, InSpec turns compliance into code. Tighter integration between the two platforms should enable IT operations teams to automate compliance testing before code gets deployed in a production environment. In effect, Crist says, compliance becomes just another continuous process automated by Chef. This approach not only leads to faster application deployments, it also reduces the amount of time auditors must spend checking on where production applications comply with a particular mandate, as all the documentation has already been generated.

Compliance mandates are, in many ways, the enemy of innovation. In fact, it often takes longer these days to navigate compliance issues than it does to write a piece of code. Rewriting code to comply with multiple compliance requirements over several different manual tests is a waste of time and money. InSpec makes it possible to make compliance testing a part of an integrated DevOps process in much the same way security testing is fueling DevSecOps.

As the number of platforms that IT operations teams now need to support has expanded, the compliance testing process has become even more complicated. Crist notes that for DevOps to be truly effective, it needs to encompass not just the deployment of code but also making it possible to make compliance and security testing an integrated component of the application deployment process. Of course, as an open-source project there’s no reason developers can’t incorporate InSpec into any continuous integration/continuous deployment (CI/CD) framework they choose. Chef is just moving to make InSpec a more natural extension of its IT automation framework.

There’s a lot of commonality in the controls implemented across various compliance specifications. That creates opportunity to automate controls to reduce the need to rework an application, either before it gets deployed or, worse, after it has been deployed into production. Longer-term, the level of commonality across compliance controls also creates an opportunity to leverage machine-learning algorithms to automate implementing those controls.

Of course, there will also be controls that will be unique to one compliance standard or another. But the days when IT organizations need to manually address those compliance controls one application at a time are coming to an end.

— Mike Vizard

May 25, 2017
The Time Is Now for an IT Operations Revolution

There’s a level of frustration starting to build inside IT operations teams that’s about to boil over. Lucas Carlson, vice president of strategy for Automic, a unit of CA Technologies that provides an IT automation platform, says developers are bullying IT operations teams by programmatically invoking application programming interfaces (APIs) exposed by public cloud service providers. As the number of application workloads running in those clouds increases, IT operations professionals are losing control.

To ensure their continued relevance, IT operations teams need to leverage automation to inject levels of agility into local data center environments, he says. That would make it much less likely developers would feel the need to programmatically invoke IT infrastructure resources.

Carlson says achieving that level of agility will require IT operations teams to rely more on machine- and deep-learning algorithms to automate low-level tasks. Instead of thinking in terms of managing silos of compute, storage and networking resources, Carlson contends IT operations professionals need to reinvent themselves.

The artificial intelligence (AI) models based on advanced algorithms must be trained continuously. IT operation professionals could become the trainers of those AI models. In fact, Carlson says IT operations teams are approaching a critical moment in history: They can either wait for AI models to eventually replace them or they can leverage them to take back control of IT infrastructure.

However, developers tend to make decisions based on what they prefer versus what’s best for the organization. Before too long, multiple duplicate operations management frameworks are in place. Rather than incur all those management headaches and associated technical debt, Carlson says organizations need a single framework that can more adroitly accommodate the multiple roles within the DevOps team. It’s already long past the time when IT teams should have modernized their operations, he says.

The relationship between developers and IT operations teams has always been contentious. The more belligerent members of the developer community are even making a case for “NoOps.” In that context, all the IT dollars spent on operations are moved into developing applications because the management of IT infrastructure is completely automated. Given the amount of legacy infrastructure that isn’t likely changing anytime soon, achieving that level of automation inside the enterprise won’t happen for a while. But there’s no doubt that IT operations needs to get more agile. Because of that issue, investment in software-defined data center (SDDC) technologies is growing rapidly.

Instead of resisting that trend, Carlson says teams need to demand that internal IT operations be brought into the 21st century. It’s true that SDDCs infused with AI models will automate most of the lower-level tasks associated with DevOps. But those members of that IT operations teams who become the masters of building and training those AI models will be among the most highly sought-after individuals in IT.

— Mike Vizard

May 22, 2017
Automation: A Few Decades of Pain? Or, What Does Jack Ma Know That I Don’t?
Recently I came across three articles that gave me pause. Each article was a bit disconcerting on its own merit, but when I considered them together, the result caused more than a momentary worry. One article was report in Bloomberg Technology quoting Jack Ma, head of Alibaba, the second richest man in China and a major player in the e-commerce space. He said:

“In the next 30 years, the world will see much more pain than happiness.”

Ma was referring the impending effects of automation and e-commerce on the economies and societies of the world .

The second was from a piece written by Martin Neil Baily and published by the Brookings Institution in 2015. The article is titled, “U.S. manufacturing may depend on automation to survive and prosper.” Here is the quote, referring to U.S. manufacturing:

“The manufacturing sector is still very much alive and reports of its demise are not just premature but wrong. … The sector will be revived not by blocking new technologies with restrictive labor practices or over-regulation but by installing them—even if that means putting robots in place instead of workers.”

The last piece is by Joe McKendrick over at ZDNet. The subtitle to his article titled, “Containers, DevOps, IT automation seen as antidotes to complexity,” says it all:

“Seven in 10 IT professionals agree: if we don’t automate, we die.”

When I put the ideas in the articles together, I get a thought that goes like this: In the coming decades more automation will be put in place, a good many workers are going to be displaced by robots, and it’s going to be a world of pain.

Do these notions seem far-fetched? Maybe not. The need is there, the will is there and the money has been allocated. The result is going to be great for the bottom line. For humans and workers, not so much.

But, a thought is not reality. There are many people out there saying that automation is a blessing and in the short and long term, most of us will do well. Automation will create more jobs than it destroys.

Predicating the Future is a Risky Business

Predicating the future is a risky business. Could anybody have predicted that the underlying technology that powered Netscape back in the 1990s would produce today’s internet of things? Who would have thought that the first cell phones, selling for around $4,000 in 1984, would lead to mobile devices becoming the dominant computing device today? Not many. To quote Yogi Berra:

“It’s tough to make predictions, especially about the future.”

Yet, the three predictions I describe above bothered me. I wondered, Do these guys know something I don’t?

So I decided to do take a look at some data.

What the Data Says

One of the metrics that I have found particularly interesting when considering automation is the relationship between the number of employees a company has and its revenue. This number is sometimes referred to as Revenue Per Employee (RPE). The simple assumption is that companies that have high revenue per employee are more productive because that productivity is realized via tools and machines; in other words, automation.

Using data in both the NYSE and NASDAQ stock exchanges, I determined the Top 100 companies with the highest RPE in a given year. I excluded from consideration companies that had fewer than 1,000 employees. (A publicly listed trading company that deals in precious metals, with 30 employees generating revenue of $120 million will create a distortion that is atypical to the norm.) I looked at each year between 2007 and 2017. The result is shown in Figure 1, below.

Figure 1: Top 100 Companies by Revenue Per Employee excluding companies with less than 1.000 employers

Given the large representation of the Energy sector in RPE, I decided to look at the Top 10 companies in terms of revenue from 2007 to 2016 to see if there was any correlation. I grouped the Top 10 revenue earners into sectors. The result is shown below in Figure 2:

Figure 2: Grouping of Top 10 Revenue Earners according to sector 2007 – 2016

Notice that companies in the Energy sector have the largest representation in the Top 10 revenue earners throughout the years. This pattern is somewhat similar to that of representation of Energy shown Figure 1, in that Energy has the largest representation. There is a correlation.

An Assumption Proved False

My initial thinking was that there is something going on in the Energy sector that might be relevant to my simple assumption about RPE: The greater the RPE, the more automation in force. I looked that the individual RPE performance of Royal Dutch Shell (RDS.A), ExxonMobil (XOM) and Chevron (CHV), three companies in the Energy sector that always appeared in the Top 10 Revenue Earners from 2007-2017. I compared each company RPE to its Total Employees. Figure 3 below shows my findings:

Figure 3: Revenue Per Employee and Total Employees for ExxonMobil (XOM), Royal Dutch Shell (RDS.A) and Chevron (CHV)

As mentioned above, my initial assumption is that as automation increases, RPE should increase, while employee count remains steady or decreases. Figure 4 below illustrates the assumption.

Figure 4: One might assume that as automation becomes more prevalent, fewer employees should produce more revenue per employee

Yet, as you can see in Figure 3, the numbers did not confirm my assumption. We see a significant dip in RPE from all three companies around 2010, while ExxonMobil acquired more than 20,000 employees. Also, we see that Royal Dutch Shell and Chevron have a decline in the number of employees over the number of years. If the efficiency that automation might be attributed to RPE were in force, revenue numbers would at the least mirror employee numbers, or hopefully increase due to more productivity achieved through automation. But revenue does not continue to map to increased revenue versus the number of employees. Revenue is all over the place.

I was stumped until I considered the impact of price of oil. When I compare revenue to the historical price of oil according to Brent Crude Spot Price to oil company revenue during the years in question, a similarity occurs, as shown in Figure 5, below.

Figure 5: Oil Company Revenue compared to Price of Oil, 2007-2015

The shape of the lines in the lower chart in Figure 5, Revenue in Billions and the shape of the line for oil price, in the top chart look similar.

A reasonable inference can be made that the while oil companies enjoy a sizeable return per employee due to capital investments in tools, technology and automation (ExxonMobil returned more than $3 million per employee in 2016, for example), market price conditions have a more significant effect on revenue.

So it seems that in terms of the Energy sector, for this admittedly small sample size, automation does allow for significant revenue, but does not increase revenue by way of increased efficiency in terms of operational production. I went back to being stumped. When Jack Ma was talking about decades of pain, was he talking about continuing slump in revenue from the Energy sector? Or was he talking about something else? I decided to look at some data about the world’s largest commercial employer, Wal-Mart.

All Roads Lead to Wal-Mart

Wal-Mart employs more than 2 million people and is the leading revenue earner for the years 2014-2017. Yet, when we look at its RPE compared to other companies, Wal-Mart is at the bottom of the stack, as shown in Figure 6, below. As mentioned above, in 2016 Exxon earned more than $3 million per employee. Amazon had an RPE of ~$475,000. JPMorgan Chase came in at ~$401,000. Wal-Mart? $211,000. To put it in perspective, ExxonMobil did more than ten times as much in terms of RPE.

Figure 6: Wal-Mart’s Revenue Per Employee of $211,000 in 201y puts it well below others

I could not help but wonder, why does ExxonMobil have a higher RPE than Wal-Mart? Or, for that matter, why so to for JPMorgan Chase and Amazon? I have a theory to propose to answer the question.

Energy, Finance and Amazon have historically been heavily automated industries. The size of a land-based crew drilling oil is between four to six employees per eight-hour shift. These people are not out there with picks and shovels. They are operating big machines with each machine having some amount of automation built in.

Once the well is drilled, there is some maintenance labor, but revenue is derived from what comes out of the ground. A lot of machinery is used to get the oil out of the ground, processed and delivered for end use. Hence, a large automation factor. Human labor is secondary.

Same with Finance. Computers have been at the heart of Finance industry—banking, investments and insurance—since the first mainframes were put into action. Is the revenue of some companies in the Finance sector subject to the whims of the market? Most likely, yes, particularly as one considers the events of 2008. But again, one trader with a single Bloomberg terminal can do a lot of work.

And Amazon? The company has been the poster child for automation since Jeff Bezos sold the first book online.

Still, as noteworthy as the facts were, there seemed to be no historical precedent to justify Jack Ma’s prediction doom. But, the dreadfully low RPE of Amazon might give us a clue. So went back to the drawing board and spent another afternoon making charts using data from NYSE and NASDAQ.I came up with the two charts you see in Figure 7 below. Then, the plausible became possible.

The Plausible Becomes Possible

The chart on on the top of Figure 7 below shows the annual revenue for Amazon, ExxonMobil and Wal-Mart. The chart on the bottom of the illustration shows the total employees for each company. The time span of each chart is 10 years. Wal-Mart shows a steady rise in revenue until 2016 when things start to go flat. ExxonMobile starts to lose revenue in 2012, while its total employee count remains relatively even. Amazon, on the other hand, shows a steady rise in revenue, as well as a rise in employee count. Both Amazon’s total employee count and revenue grows at a steeper growth trajectory than Wal-Mart.

Figure 7: Total Employees and Revenue: As Wal-Mart hires more employees revenue begin to flatten; Amazon revenue increases

Is there something going on?

Well, when we look at net income, there very well might be. Figure 8, below shows the net income of ExxonMobil, Wal-Mart and Amazon. This is where the rubber hits the road. Net income is the money that’s left over after all expenses and taxes have been paid. It all comes out in the wash when you look at net income. You can be a company doing $1 billion in revenue, but if your expenses are $999 million, your company is doing no better than one doing $10 million in revenue with $9 million in expenses. In terms of Amazon, Wal-Mart and ExxonMobil, as we move to the later years, 2016-2017, the numbers come into the same ballpark. And, in that ballpark Wal-Mart is showing some decline, ExxonMobil is showing a continuing big decline. Amazon is on the rise.

Figure 8: Net Income: Wal-Mart and ExxonMobil show declining net income while Amazon is on on the rise

So now is a good time to go back and review the assertions made at the beginning of the piece:
- Jack Ma: “In the next 30 years, the world will see much more pain than happiness.”
- Brookings Institution: “even if that means putting robots in place instead of workers.”
- Joe McKendrick: “Seven in 10 IT professionals agree: if we don’t automate, we die.”
Given that data presented, we see that industries that have a good deal of automation, Energy and Finance for example, have a high degree of employee efficiency as reflected in RPE. Also, we see that Energy is subject to the whims of market pricing. And lastly, we can see the country’s largest employer, Walmart, has poor RPE numbers, while Amazon, in many ways analogous to Wal-Mart in that Amazon has a significant presence in the retail space, albeit the online retail space, has very good RPE and increasing net income.

Will Amazon continue to hire? A reasonable assumption is yes. Will Exxon continue to hire? If you go back to Figure 7, you will see that ExxonMobil employee count has been declining since 2010. Couple this fact with a continuing decline in revenue, we’re not going to get fired if we say that ExxonMobil is probably not going to increase employee count unless some sort of miraculous business pivot happens. But what about Wal-Mart?

Wal-Mart has 2 million employees, a dreadful RPE number and declining net income. What will happen should Wal-Mart’s CxOs decide to more fully embrace automation? After all, hiring more people does not seem to help. It hired more and revenue declined. It’s a pretty good bet that Wal-Mart will go more toward the full automation end of things. How else will the company survive? So let’s play out this scenario.

A Closing Hypothetical Scenario

Let’s close with a hypothetical scenario. Let’s say that somehow Wal-Mart magically figures out a way to increase its RPE from the current $241,000 to $475,000 within the next five years. ($475,000 is the current RPE of Amazon.) And, let’s say that Wal-Mart’s revenues continue to stay level at the current $482 billion a year. Thus, to calculate the number of employees required to achieve the desired RPE, we use the formula:

$475K (RPE) = $482 bln/Number of Employees

When we do the math, the number of employees that Wal-Mart needs to achieve $482 billion in revenue with an RPE of $475,000 is 101,4736,842—about half the size of its current workforce. Think about this. The implications are dramatic.

Most likely, Wal-Mart will continue to automate. It must to survive. The question becomes, given its current dismal RPE, does the company somehow double its revenue to maintain the size of its current workforce. Or does, it accept current revenue trends and let 1 million people go.

It seems that Jack Ma is predicting the latter. This might very well be but one of many examples in decades of pain he predicts, an outcome that is the result analysis offered by the Brookings Institution and 7 out of 10 IT professionals. Or, who knows? Maybe the future is one in which the Wal-Marts of the world double revenue and manage to keep the human workforce intact.

As we have learned on the terrain, it’s easy to predict the future when it involves describing the outcome of writing a check for $100 when you have only $10 in the checking account. Predicting the impact of automation on future human employment is a more complicated terrain to travel. But no matter what, there will be a future and most likely, it will be more automated. What will happen to the employees at Wal-Mart? I’ll let you be the judge.

A Special Thanks to YCharts.com

Please allow me to extend a special thanks for the folks at YCharts.com for providing access to the data and charting tools used in this article. They were extremely helpful throughout the development of the ideas expressed herein which, by the way, are my own. — Bob Reselman

— Bob Reselman
May 19, 2017
DevOps Chat: Value Stream Mapping with CollabNet’s Eric Robertson

This DevOps Chat features Eric Robertson, VP of product and engineering at CollabNet. We discuss value stream mapping, which comes from Lean methodology and has been around for a while. But do you actually know what value stream mapping is? How can it help your DevOps transformation? Eric does a good job of explaining.

As usual, the streaming audio of our conversation is immediately below, followed by a transcript of our chat. Enjoy!

Streaming Audio

Transcript of our Chat

Alan Shimel: Hey, everyone. This is Alan Shimel of DevOps.com here for another DevOps Chat and our guest on this chat is Eric Robertson, VP of product management and engineering at CollabNet. Eric, welcome to DevOps Chat.

Eric Robertson: Thank you. Thank you, Alan. Glad to be here.

Shimel: Yeah. We’re glad to have you here, Eric. I don’t think we’ve had anyone from CollabNet, actually, on DevOps Chat before, though I think we’ve done some video interviews over at DevOps Enterprise Summit—maybe it was San Francisco in October and November—but we’re—so we’re glad to have you here. We’re eager to find out about what’s going on with CollabNet and about some of the stuff you’ve been doing. Eric, if you don’t mind, though, there might be some people out in our audience who are not familiar with CollabNet, so, if you can, just—doesn’t have to be long, but maybe a quick elevator pitch on who CollabNet is, what you guys do.

Robertson: Sure. Sure. So CollabNet’s been around since, I think, 1999, and it was one of the foundational on the open source, with our Subversion version control software. We since then moved into the ALM, application lifecycle management, platform. It’s our product offering called “TeamForge,” where we started linking in that planning, development, and testing life cycle, with integration to tools. And now, with me, we actually extended that life cycle all the way down into the operations—thus, the concept DevOps. And now we are launching—have launched, actually, a series of products and partnerships around DevOps for our customers.

Shimel: Fantastic. Fantastic. So, Eric, one of the things that, when we spoke off mic earlier, we spoke about value stream mapping, and I know that’s something kinda, if not near and dear to your heart, something that you’ve been involved in with CollabNet. Why don’t you give our audience—let’s start off with this. There may be people out here who, just as they didn’t know CollabNet, maybe they’re not familiar with the term “value stream mapping.”

Robertson: Yeah. It’s very interesting. Value stream mapping is actually an older concept. It’s part of the Lean management framework. For example, many may have heard about Toyota. It was started there, as far as best practices around mapping material and information flow and how could we apply to, for example, the value chain, and pretty predominant in the manufacturing domain. And it’s kinda of interesting because I have a background in software engineering and I have degrees in computer science and engineering, but my first job was actually in architecture—not in software architecture, but actually designing building. CAD design.

And it was interesting that, during that time, they started talking about design patterns and best practices, as far as architecting and putting together systems and designing. And I said, “Wow, this is very similar concept to what I learned back in college, when I was doing software engineering, software design.” And, very soon, that started translating into object-oriented and design patterns and things of that nature. So it’s interesting how patterns and techniques that were very popular and used in other domains, like manufacturing and architecture, now being applied to software. In other words, software is maturing, right? The software development process is maturing. It’s starting to utilize these type of practices.

And the value stream map is just an example of that, right? It’s Lean management method for really analyzing that current state and designing a future state, based upon a series of events that take a product, a service, from its beginning through to its customer. Right? And, again, like I mentioned, Toyota—this is known as that material and information flow mapping, and it can be applied to any value chain, for example, right? And what is a value chain? It’s just a set of activities that take, for example—that delivers a valuable product or service to the market, right? So you got, in the manufacturing world, this was machines that did it; in the software world, it’s now tools that we use to deliver a product from a planning all the way to its operations stage.

Shimel: Yep. Got it. So, Eric, this is something, obviously, that you have written and you’re talking a lot about it. Can you tell our audience, if they want—maybe if they wanted to find out a little more, where can they go? Before we dive in more than that, maybe where can they get more information on this?

Robertson: Yes. Actually, they could come out to our website, collab.net, but, also, I recently published an article around how value stream mapping delivers a better DevOps tool chain, and we can post those links—I believe it was also on DevOps.com and we have one that was delivered through TEDxBeacon, so there’s a series of outlets, and we’ll post those links there on where they can get more information about this.

Shimel: Excellent. You know what? We’ll, for sure, do that. And then, Eric, you had also mentioned that you guys were doing some sort of roadshow or a little city tour, where you’ll be talking about this, as well as other areas of interest around DevOps and what CollabNet’s doing. What’s going on with that?

Robertson: Yeah, so we’re actually having a series—we’re also doing it through webinars, but we’re also doing it through seminars, and we’re gonna be having these seminars throughout the different series. And, basically, what we’re doing there, what’s really critical there, is that we’re going to be not only talking about the value stream as far as more of the how this applies into the DevOps—in DevOps world, but also we’ll have customers there talking about how they are leveraging value stream mapping and how they’re mapping that to their tool chains and how they’re able to, from bare, start applying measurement, which, to them, is what they’re calling “the last mile” of DevOps, right? And it’s really—they’re gonna talk about how that provides a window to allow them to monitor, measure, improve their workflow processes, all around a series of KPIs, both velocity KPIs and also quality KPIs.

Shimel: Mm-hmm. Excellent. So, again, I think, Eric, what we wanna do is post a—maybe if you have a schedule of cities that we can post with the notes from today’s show.

Robertson: Yes.

Shimel: I think that’s the way to do that. So—

Robertson: Yeah.

Shimel: —let’s return, though, for a moment, to value stream mapping and kinda dig in here a bit. One of the things—and, again, as we spoke about off mic, I’ve had some experience with value stream mapping—one of the things, though, in speaking to potential customers of value stream mapping, is it’s easy to say, “Oh, yeah, that sounds good.” Right? “Oh, that’s something we wanna do.” Okay, well, let’s schedule some time and I’m gonna need the right people at the table because, if you don’t have the right people there, it’s very hard, number one, to truly map out the value stream and, number two—

Robertson: Exactly.

Shimel: Right? And harder than that is let’s get some consensus on what we’re gonna do to make it happen.

Robertson: Exactly.

Shimel: You know, unfortunately, it can’t be some developer who raises his hand and says, “I really wanna make a digital transformation here and I need to convince my CIO. Can you come in and give me a value stream mapping kind of session or workshop or can we discuss it?” It’s not a one-man show.

Robertson: Exactly. Exactly. And it is very interesting you said that because, because value stream looks at the lifecycle from planning to operations, there’re different stakeholders, right? And what’s happening in, especially in enterprise, is these stakeholders now—they have implemented the tools or they have guys that work for them or folks that work for them that implemented these tools, and they’re starting to ask questions or they already started asking questions around value. What I mean by that—you have the executive and he’ll say, “You know, my folks are telling me we did four deployments today.” Great. What does that really mean for the business?

Shimel: Yeah.

Robertson: Did we get little features out to the business, where they get utilized? Are they utilizing that? Did I actually place the right emphasis on those features? Right? The last release that I deployed on time, at least from a developer perspective, said I deployed this, I deployed it on time, but I’m having this increase of service desk tickets. What’s going on here? Why am I being called? Right?

Shimel: Yeah.

Robertson: And I _____ –

Shimel: Oh, yeah. Exactly.

Robertson: Right? [Chuckles] And it means that the IT office guys, right? They have their own set of—type of metrics that they’re looking at, right, and they wanna understand, right, “Is there anything impacting this release?” and without having to chase down and talk to every other person or look into every tool, to try to gather this information, right? And then you’ve got the engineers, like the QA engineers, and now compliance, security’s coming into play as well, right?

Shimel: Absolutely.

Robertson: So now folks wanna know, “Before I deploy, did all this pass through the quality gates? Was this sufficient?” Right? “What about security and compliance checks? Can I automate that feedback? Can I get that feedback to the appropriate stakeholders?” Right? So they’re already involved in asking these questions, but they really don’t have any direction there on how to get that information. And this is where—this is the impetus to say, “Well, you know what? Now’s the time to start looking at these tools, where you’re gathering this information, what’s generating this information flow, and how we bring this together.” Right? So it kinda starts to endow that value stream mapping conversation pretty well.

Shimel: Yep. Excellent. Excellent point. Because, like DevOps itself, value stream mapping is a team sport, right? And—

Robertson: Yep.

Shimel: And it’s cultural and you need that group kind of involvement to make it happen. So, Eric, I’m sure you probably have a few war stories around this, though. Can you share with us sort of—you know what I always say? You learn more from your failures than you do from your successes.

Robertson: Oh, my God.

Shimel: Can you share with us maybe some value stream mapping engagements that didn’t work out and, of course, only if you’ve learned something from it, and what was the valuable lesson you learned?

Robertson: Exactly. I mean, that’s—it’s very interesting that you said that because we had one customer—and I can even use it from my experience. I mean, I worked at a company and I used to dread Wednesday mornings because Wednesday mornings was—wasn’t really a scrum meeting, but it was more of a meeting as far as, “What is the latest status of this release that went out the previous day?” because we did weekly deployments, weekly deployments. And everybody would dread that because it was really no place to go to get a full understanding that, “Well, if this release actually was deployed, what was in that release? If it wasn’t deployed, what was the bottleneck in there?” Right? “Was it that it failed a certain testing that it didn’t get through or did we discover that, in this environment, we did some monitoring and it didn’t pass this stage?” We had no idea. I mean, did the builds fail? Why? There was no insight to anywhere in that chain, right, from the planning, ’cause it’s all kinds of activities that are happening in the coding, the building, the testing, the deploying a release.

And so, if you don’t really have an idea, as far as that information flow in a place where you can go and view that information and quickly understand “What are my exceptions, right, that I need to quickly identify and quickly address?” everybody’s chasing their tails there. And I’ve been on so many projects where that transpired and happened. And so this was really impetus of why we wanted to start having—start putting—leveraging this best practice, leveraging these practices, and then the tooling becomes more secondary type of aspect, rather than primary goal for driving processes.

Shimel: Yeah, I agree with you. The tooling then winds up taking sort of a backseat to it. So let’s—well, we’re almost out of time, unfortunately, Eric. As I mentioned to you, the 15 minutes goes quick. We do the best we can.

Robertson: No worries.

Shimel: Let’s put—

Robertson: I’m enjoying it.

Shimel: —value stream mapping to the side a little bit. What else is—

Robertson: Sure.

Shimel: What else is going on with CollabNet?

Robertson: So CollabNet is also partnered with a company called “Clarive”—it’s release automation software—and what we’re doing is is that we’re linking the release automation capabilities into our value stream management platform. Now what’s nice about this is that this allows us, very quickly, to start getting metrics across different type of applications and projects. And what I mean by that is, if you go into a larger enterprise, for example, they – of course, they’re experimenting with container type technology and Web-scale IT technologies, but they’ve also got some legacy applications, right? There may be Java, right? There’s—or on WebSphere or a traditional type of applications or architecture. They may even have mainframes or they may even have package apps, right?

So one of the things that we’re doing is that we’re getting—we’re creating this abstract _____ layer around all those tools, so customers can have, enterprises can have a common user experience around release automation, whether or not—so, again, we’re abstracting the tools, so you’re focused on the process, and it doesn’t matter what tooling is utilized underneath. If it’s Webscale IT technologies, container-based technology, it does not matter. You can basically have a common release process across your different type of applications—package, legacy, and traditional—so we’re very excited about that.

Shimel: Excellent. Sounds very cool. Eric, we’d love to talk more to you about this, but we kind of are over time. What I’d like to do, though, is maybe invite you back and we can talk. You know, we’ll leave value stream mapping for a moment and we’ll talk about other stuff next time, but I think we’d like to hear what’s going on in the world of DevOps, obviously, and with the players in it, and CollabNet has certainly been on the scene here and lots to tell. Until then, though, it looks like we’ll call it a wrap on this version of DevOps Chat and we’ll speak to you soon, and I—Eric, we’ve gotta get those links over. There’ll be in the notes for people who are listening in. But until then—

Robertson: Great _____.

Shimel: Eric Robertson, VP of product management and engineering at CollabNet, thanks for being our guest on this episode of DevOps Chats.

Robertson: Thank you, Alan. Glad to be part of this.

Shimel: My pleasure. This is Alan Shimel for DevOps.com and DevOps Chat. Thanks for listening in today and we hope to see you soon on another DevOps Chat. Buh-bye.

— Alan Shimel

May 16, 2017