The rate at which IT operations will become automated is about to accelerate significantly over the next two years. A global survey of 1,500 senior IT leaders conducted by Cisco Systems finds that 64 percent of respondents today would describe their approach to IT operations as either reactive (26 percent) or proactive (38 percent), but within two years more than two-thirds expect their approach to be either become predictive (33 percent) or preemptive (33 percent). Only 14 percent of respondents describe their IT environments as being preemptive today.
In general, the survey finds 88 percent of IT leaders said their investment in IT operations over the past 12 months had improved external customer satisfaction, while 89 percent said there have been improvements in innovation. But only 26 percent collect data on an ongoing basis and even fewer (17 percent) employ any kind of real-time automated analysis, the survey finds.
Nevertheless, 42 percent of respondents said they expect artificial intelligence (AI) to have the biggest impact on their ability to automate. More than half (51 percent) said they already rely on AI to some extent today to automate IT operations.
A full 82 percent also already collect operational and performance data about their security infrastructure.
Joseph Bradley, global vice president for IoT, Blockchain, AI and Incubation Business at Cisco, said much of the investment in automating IT operations is being driven by the realization that IT has become a profit center rather than cost center. As organizations increasingly realize they are a software company that happens to make something, they are looking to maximize the return on their investments in IT wherever possible, said Bradley, adding as a result, line of business (LOB) executives are exercising more influence over what applications get built and deployed within the context of an overall digital business transformation strategy.
Naturally, all that increased appreciation for the value of IT is putting more pressure on IT organizations to modernize their processes. As the number of applications that need to be built, deployed and maintained continue to increase, the need to embrace DevOps processes becomes a greater imperative. To facilitate that shift across enterprise IT environments that now routinely span multiple clouds, Cisco has been making a two-fold case for centralizing the management of highly distributed IT environments and upgrading to next-generation software-defined infrastructure that makes use of machine learning algorithms to automate IT management.
Cisco is not the only IT vendor pursuing a similar strategy. The success of those strategies, however, depends as much on the ability of the IT organization to implement the processes required to modernize their IT environments as much as it does the technology platforms. That creates something of a chicken-and-egg relationship between processes and platforms. It is difficult, but not impossible, to implement some level of DevOps without having a lot of automation. But applying DevOps at any meaningful scale within an enterprise undoubtedly will require a lot more automation than currently exists within most IT environments.
CircleCI has unveiled what it describes as the first package manager designed specifically to address configuration tasks involving software delivery automation.
Matt Wyman, vice president of product for CircleCI, said the Orbs package manager condenses commands, executors and jobs into single, reusable lines of code that automate tasks on the continuous integration/continuous delivery (CI/CD) platform the company provides as a cloud service. The initial set of Orbs being made available by CircleCI are focused on automating more than 25 different cloud deployment scenarios to enable organizations to employ DevOps processes at scale, he said, noting those Orbs can then be validated and shared via an Orbs registry that CircleCI is making available.
CircleCI is also launching a technology alliance program through which Jfrog, Twistlock, Anchore, Aqua Security, Cypress, Postman, Coveralls, datree, Rollbar and Codacy have committed to building Orbs.
Wyman said shareable Orbs that require only a few lines of code to develop makes it possible to share configurations across multiple CI/CD workflows. Those Orbs then can be integrated with any number of third-party tools.
CircleCI is making a case for a CI/CD platform delivered as a managed cloud service that eliminates the need for DevOps teams to set up and manage on their own. Open source CI/CD platforms can be too complex for the average IT organization to set up. Alternatively, CircleCI makes available a version of its CI/CD platform that organizations can deploy behind their own firewall. The CircleCI approach is designed to enable DevOps teams to flexibly create workflows that can be executed sequentially or in parallel.
There are 12 million builds a month being generated by more than 300,000 developers taking advantage of the CircleCI cloud service to build applications, Wyman said. One of the primary drivers of that success is a cloud service that allows organizations to apply more of their limited resources to developing applications versus managing CI/CD infrastructure, he noted.
Most IT organizations have yet to standardize on a specific CI/CD platform. Competition between providers of CI/CD platforms has never been more fierce as more enterprise IT organizations embrace DevOps. As DevOps processes continue to mature, it’s also clear that DevOps processes will need to be extended across multiple clouds in addition to on-premises IT environments.
In general, CI/CD platforms are initially set up by developers who have embraced DevOps processes to accelerate development and deployment of applications without sacrificing quality. The most widely used CI/CD platform is the open source instance of Jenkins. Developers often can adopt Jenkins on their own without requiring anyone to sign off on a purchase order. But as organizations move to scale up their DevOps processes, CircleCI is betting interest in a commercial service that can be easily accessed across an entire enterprise will increase as organizations move to standardize their CI/CD environments.
In the meantime, pressure to increase the rate at which applications are built and deployed is only going to increase as organizations become more aggressive about digital business process transformation. The issue now isn’t so much whether they will have a CI/CD platform, but rather to what extent—and, in some cases, how many.
Has automation changed the way we perceive interaction with others?
A few weeks back I got an email from an old friend, Jim. Typically, that’s no big deal; I get emails from old friends all the time. Except, in this case, it was different. According to a posting on Facebook, Jim passed away about a year ago.
Another friend who is still among us reports having a similar experience of receiving emails from those who are dead. It’s an eerie event that’s becoming more common. Bad actors hack into a data store and pull up some email addresses, which they blast about nefariously to unsuspecting recipients. It’s a strange, almost morbid experience when you’re on the receiving end.
But it’s gotten me thinking: What does it mean to be alive on the internet? And, more importantly, how much of what we perceive on the internet as a human really is? Jeepers—for all I know, the Facebook post could have been a ruse. Maybe Jim uploaded his consciousness into eternity and he really is trying to make contact. How would I know it otherwise?
Allow me to elaborate.
Technology that Captures the Moment
Mankind has been using technology to represent reality since early caveman figured out how to draw pictures of hunters and animals on the walls of caves. The more real a depiction seemed, the more valuable it was. A sign of significant wealth was having the wherewithal to pay an artist to paint your portrait. The painting passed your likeness onto future generations. Your posterity knew what you looked like. The poor just drifted in away into imageless anonymity.
However, whether you were rich or poor, there was one thing that technology couldn’t do: It couldn’t capture the moment. A painting took weeks, maybe months to complete. And, no matter what, the rendering was an interpretation.
Photography changed all that. It captured the moment, which was a first in human history. And it democratized portraiture. In practically no time at all, it seemed that just about every town and city had a photography studio. Affordable photography made it so images of mothers and fathers could be passed on—future generations had a very clear idea of what great-great-grandma Mary and grandpa William looked like.
Portraits of George Washington left a lot to the imagination. Photographs of Abraham Lincoln left no doubt.
Figure 1: Portraits of George Washington are renderings of the artist’s perception created over time, not in the moment.
Figure 2: Photographs capture a moment in time. Abraham Lincoln in 1858 (left). Abraham Lincoln in 1861 (right).
The first time that 19th century former slave, abolitionist and author Frederick Douglass had his photograph taken was in 1841. By the time of his death, he was the most photographed man in America. Douglass understood the power of the technology both in terms of information representation and dissemination. Photographs of Douglass made his humanity real at a time when a good portion of the population of the United States was deemed subhuman.
Figure 3: Frederick Douglass, born a slave, was the most photographed man in 19th century America. The reality that the technology revealed confirmed his humanity.
The power that photography brought the technological and cultural landscape created trust in the image. The veracity of information in a newspaper article could be argued, but the image the camera captured represented reality beyond a doubt—at least until the practice of doctoring photos to misrepresent reality came into practice.
Reincarnation as Seen on TV
And then came television, which made it possible to represent reality as a stream. Whereas a photograph captured the moment, television captured as many moments as the viewer had to spare. But, it was a trade-off. Television distorted perceptions of life and death. Before TV, no child ever had the experience of viewing an actor portraying a character who died in an episode of “Gunsmoke” one week, only to reappear as another character on another TV show the following week. Death became impermanent. Also, the line between fact and fantasy blurred. The same television screen that brought the nightly news also delivered episodes of “Star Trek.”
In terms of time and space, the only “real” thing is the TV set. Everything else is content. Sometimes you could tag the content as “fact” and other times as “fiction.” Or, you could tag it to a third category, “I’m not sure,” which gave rise to the harebrained idea that the 1969 moon landing really happened in a TV studio at a secret government location.
Now, here’s where it gets really interesting. In the past, most people interacted with others most of the time in close proximity. You went to Grandma’s house for the holidays. You bought your groceries at the neighborhood supermarket. You sat in classrooms under the supervision of one or many teachers for at least 12 years, maybe more.
That was then and this is now.
Today, visiting Grandma for the holiday might be nothing more than a call on Skype. More people are making essential purchases online. The online class is replacing the brick-and-mortar school building.
The fact is, we’re spending a lot of time in an environment that is essentially representational. Every day, more of our interactions with the “world” take place on the screen of a smartphone, tablet or a desktop computer. In some cases, it’s nothing more than talking to a device by name, as in “Alexa, what time is it?”
Is it Human? Does it Matter?
And thus we have the problem of distinction. We’re at the stage now where we’ve come to accept that some of the interactions we have online might be with a human and some interactions might be with a machine powered by AI. Sometimes it’s easy to tell. After all, I know there’s is no miniature person living inside my Echo device. And I know there is no little man inside my refrigerator who turns on the light as I open the door. This is low-hanging fruit; it’s obvious what the machine is. Still, in the long run, how can we tell? And, even if we can’t, does it matter?
These days I have a lot of interactions with people on video conference and on the phone. I estimate that of all the people I interact with regularly, I’ve shaken the hand of about half. Twenty years ago, I physically touched nearly all. Such is the way of progress.
I’ve had the benefit of living in a time in which the person came before the photograph. Now, we live in a time in which the photograph can come before the person. We think up the ideal actor and send it over to the special effects department for rendering. Today, we interact with images that have no human origin and yet behave in a way that is indistinguishable from a human. For those born today, it’s entirely possible that most of the meaningful relationships they have throughout their lifetime will be with non-human intelligence. Yes, there is a good argument to be made that people will still gather together in real time for concerts, religious services and sporting events. But consider this fact: 73,000 people attended the 2018 Super Bowl in a stadium, in real time. However, 103 million watched the event on TV. The “attendance” at the digital representation of the event exceeded the real-time experience by more than a factor of 10. And, companies paid millions of dollars for advertising time during the televised game. Today, that which we perceive as real is just as meaningful as that which actually is real.
Which brings us to life and death in the age of automation. Unless Kurzweil proves right and uploading ourselves to a computerized host becomes possible as the Singularity approaches, all of us will die. And yet the images, both the ones we make and the ones that were made for us, will live on forever. Will these images take on a life of their own? Will we be able to use artificial intelligence to inject these images with the behavior that made us us, using information about us that was gleaned by observing every aspect of the online interactions we had over our lifetime?
I’m not sure. But the possibility has stopped me from replying to that latest email from my friend, Jim. To be honest, I’m afraid of what would happen if I did.
Dan Lahl, global vice president of product marketing for SAP Cloud Platform, said SAP is now adding a Content Advisor service that makes use of machine learning algorithms and other forms of artificial intelligence (AI) to recommend what integrations should be created next based on those that have been previously deployed on the platform.
In addition, SAP announced the general availability of SAP Cloud Integration Automation Service, which provides deeper integration with a standard set of metadata SAP is in the process of defining for all its packaged application software, and SAP Cloud Platform Open Connectors, a set of application programming interfaces (APIs) based on code SAP has licensed from Cloud Elements.
SAP also revealed that it will start making an on-premises edition of SAP Cloud Platform available via partners including IBM and Atos beginning in 2019. Lahl said that decision was driven by customers that for either regulatory or intellectual property concerns did not want to deploy custom code on any cloud service. Those efforts become more feasible once SAP finishes porting its distribution of Cloud Foundry to run on top of Kubernetes, which presents a more efficient approach to deploying an application development platform using a cloud-native platform that can run anywhere. While SAP remains committed to Cloud Foundry, the cost and effort of standing up instances of a Cloud Foundry PaaS in its original form are considerable.
Finally, a new SAP Business Partner service on SAP Cloud Platform, planned for general availability in November, will provide access to a set of master data services that simplify access to and orchestration of enterprise business applications. The master data services can be shared and reused for any LOB-specific intelligent business processes to ensures a consistent and comprehensive view of customers, sales and products. In effect, SAP is now providing compliance as a service, said Lahl.
The integration additions to the SAP Cloud Platform should improve performance and help drive down costs, he said, because collectively they eliminate the need to rely on an external integration service when building and deploying applications on SAP Cloud Platform.
SAP has put considerable effort into building out SAP Cloud Platform as part of a strategy that provides a layer of isolation between SAP applications running on the HANA database and custom applications running on Cloud Foundry. That approach enables SAP to deliver a quarterly cadence of updates to both its packaged applications and HANA running on a cloud service without having to worry about the impact those updates might have on custom applications. One of the reasons why so many SAP customers are several releases behind on SAP packaged applications is because they have all been customized extensively in on-premises IT environments.
As part of that effort, SAP is now also making available a series of best practices for implementing DevOps processes across its platforms. It may be a while before organizations that have standardized on SAP platforms master those processes. But modernization of those platforms is now more a matter of when rather than if.
The DevOps Revolution started with a handful of simple principles. A key one was that systems should be managed by code, not by humans, to eliminate complexity and ensure better security, compliance and scalability. To manage compute infrastructure at any scale, the process should be automated with logic captured in code. And, ideally, automation also would apply as you move up to the stack to middleware and applications layers.
Unfortunately, it’s not that simple. In fact, automating applications turns out to be way more complex than automating server deployments. For this reason, we need a new approach to managing applications and infrastructure. Managing servers and lower-level environments could be simpler, but existing bottom-up methods work fine. Managing applications, however, must come from the top down. That is, applications should come with everything they need to be successful: configuration, packaging, discovery, scaling requirements and more. And that just isn’t how we have historically built software. Applications are built on layers and layers of dependencies that connect all the way down to the system level. So how do we break the mold and make an application “self contained?
Thousands of Apps with Millions of Dependency Permutations
By automating the Linux and Windows server ecosystem and a small number of derivative operating systems we can cover roughly 95 percent of operating systems, and the series of functions that are common across them. The complexity arises when we try to lay that matrix upon the array of applications that exist in the real world of deployment and production. In smaller companies, you usually have at least a dozen applications and often more. In larger companies, you are dealing with hundreds or thousands of applications.
This melange spans bespoke apps running in legacy languages, proprietary third-party software and more modern open source software. There are literally millions of permutations that must be automated across dependencies, along with security and compliance regimes. In the world of Kubernetes, for example, each container requires its own Sidecar that contains configuration and management information. Kubernetes Sidecars, however, don’t have robust automation capabilities and quickly sprawl out into a mass of human-defined management instructions that can easily conflict or break down under stress.
The upshot of this? Today we can write a small amount of code that can effectively manage hundreds of thousands of servers for Facebook. This is true to basically infinite scale. But to automate management of Facebook’s software has injected orders of magnitude more complexity.
Fear of Application Complexity Killing Cloud Migrations
This dichotomy has led infrastructure automation to go sideways; organizations have adopted inauthentic automation coupled with departures from the real ideals of DevOps and agile methodologies. Many companies just aren’t realizing the true benefits of infrastructure automation because they are doing something that is needlessly complicated and may have characteristics of automation but in fact is even more complicated. Thus, they have lost the value of the word and merely exchanged one complexity for another.
Similarly, many application development teams are doing what they call “ad hoc” infrastructure automation as DevOps teams avert their eyes and pray. Translation? A roll-your-own mishmash of tools to manage containers or VMs, applications and all the dependencies from the top to the bottom of the application runtime environment. What this really translates into is a human executing an ever-changing set of narrow paths against an individual server—a recipe for chaos. This chaos is why the majority of our customers believe infrastructure automation is a critical step to achieve agility but a smaller subset have fully embraced it.
It’s also a major reason why many companies that would love to migrate to the cloud have not yet done so. Managing the repackaging and movement—the “lift-and-shift”—of all their applications is a daunting prospect that they rightly believe will be far more difficult than simply switching from bare-metal or VMs to instances in AWS or Azure or Google Compute Engine.
Now, we don’t need to throw the baby out with the bathwater. The automation of operating systems and VMs and containers works just fine and is actually a tremendous improvement on past efforts. In fact, we could easily operate there with a smaller set of features and capabilities than we have to date. But applications must be managed from the top-down to accommodate the inherent complexity of managing so many more moving parts.
Enter The New Paradigm: Top-Down Application Management
Top-down application management works in a fundamentally different way, Rather than rigid configurations and choices about what an application should do, modern application management tools for DevOps teams should ask what an application needs to succeed and do its job well across the entire life cycle, from development to production to retirement. These tools should automatically discover changes in dependencies of an application stack, from front end to middleware to database and load-balancing and security.
If there are changes in these dependencies or other critical factors, then those tools need to automatically rebuild and test the applications, and have them ready to deploy either as a canary or directly into production if the changes are minor. For example, if a new SSL patch is published and a DevOps team updates its underlying infrastructure, then the application management toolchain should recognize which applications need to make which changes, and how that might affect both upstream and downstream dependencies. Ultimately, the toolchain should quietly and seamlessly fix all affected applications with the patch but avoid service interruption.
Once the success criteria of the applications are defined by a human DevOps person, then the application configuration and management tools’ job is ensure that criteria is always met. Collectively, this is a new type of packaging that captures the best of all possible worlds, merging configuration and packaging into intelligent application management that can scale and flex as needed.
By capturing the dependencies of configuration and the strictures of traditional packaging, top-down application management methodologies and tools allow a CIO to move an aged C or C++ back-office application into a cloud-native environment just as easily as a passel of Node.js microservices backstopping a mobile application, and have them both run with the same level of continuity and flexibility. Which is really what everyone wants—a safer, easier and better way to run applications without having to worry about the underlying infrastructure.
Conclusion
In a sense, then, the DevOps Revolution is entering a new phase. As microservices continue to slice and dice the old functionality of monolithic apps into numerous smaller apps, the challenge of managing this app proliferation will only grow—and become more complicated.
The way we thought about DevOps tools in the days when cloud computing was relatively young was understandably naïve and not yet informed by the vast application cornucopia we would face in subsequent years magnified by the reality that legacy applications decades old would continue to run in corporate data centers until their code keels over. Now we know and we can design an automation methodology and practices that map to our modern realities. We can do this by giving infrastructure the simple rules and tools it needs, while giving apps more leeway to be themselves and succeed regardless of what is happening at lower levels of the stack.
Like any industry, DevOps fosters its own unique culture. And like any culture it resists change, it has inertia. “This is how it has always been done” is an incredible force to reckon with.
Even though the realities of DevOps have changed dramatically in recent years with the advent of new tools and services, many ideas that have been ingrained in the culture will probably still be with us for ages. Some of these ideas are not only wrong, they border on being dangerous. One such fallacy is the sacred rule cherished by development teams and system administrators: “NEVER deploy on a Friday afternoon.” That is, unless you want to spend your entire weekend fixing bugs. A more pernicious example is the division of labor between us, humans with “root access” to machines, and the robots, the tools teams create to make their processes less manual. These days you’ll still find many development groups that delegate the menial tasks to automation, but keep the most dangerous, infrequent tasks to human operators. These are two prime examples of “thinking software wrong.”
These ideas are not only incorrect, they can be crippling to an organization, causing delays in software development and leaving the company vulnerable to data breaches, for no good reason. These fallacies are what we refer to as “broken culture”—what happens when culture is deeply misaligned with the interests of the organization, the individuals who compose it and its stakeholders. It usually manifests when historic cultural norms prevent members from embracing new and better ways of doing things.
Fear of Friday deployments is our first example of broken culture. This idea has taken a strong hold in DevOps culture because developers haven’t trusted their own software and systems enough, and they don’t want to spend their Friday nights (or the whole weekend) debugging a production problem. Fair enough, but there is a better way.
Let’s look at another example: the Equifax breach. This incident—which cost the company billions of dollars and continues to haunt consumers to this day—happened because a certain procedure mandated that a specific piece of software must always be updated by a human. Since humans are prone to error, this lone simple procedure (designed and supported by years of a broken DevOps culture) resulted in a devastating data breach. Once again, there is a better way.
Deploying is not the same activity it was a decade ago. Advanced automation tools are drastically streamlining deployments, significantly reducing the chance of error and speeding the time it takes to identify and correct any errors that do occur. This means companies can deploy any day of the week and any time of the day, worry free—and can let the deployment date be dictated by what is best for the company and its customers, not the weekend plans of developers.
The same concept applies when it comes to updating software. A better way is to leverage the automation tools that are now readily available, and allow organizations to have robust management of compliance requirements without human error or delay.
In addition to embracing better tools, here are a few other ways companies and developers can rethink software for the better:
Software must be valued as more than just the cost it takes to create it, or to replace it. Code should be an asset, not a liability.
Software should be thought of as constantly changing—a dynamic system with which people, machines and processes interact continuously. When this never-ending process is supported by the right tools, deployments are easier and managing and updating software is seamless and secure.
Code by itself is useless. It needs infrastructure to build it, and data to create something meaningful. Therefore, if you only test the code, but are not managing your infrastructure in the same way, you’re not really testing the software system.
Companies can dictate the cadence at which they create new software, but they do not have this same luxury when it comes to fixing broken software. Identifying and fixing problems must be done as soon as possible.
Every new update or fix must be version controlled so that everything is auditable. That includes changes to infrastructure configurations!
Once companies truly understand these fundamental truths about software and embrace the modern tools available to them, they can start to shed the aspects of DevOps culture that are limiting or damaging.
Companies can deploy new apps and features on Fridays, confidently and without issue. Companies can even rebuild an entire infrastructure from scratch, in minutes, while applying all security updates for every single service and library—and be completely sure nothing will break. If you can’t do that—or don’t believe you can—you’re thinking software wrong.
Akamai is extending its campaign to make content delivery networks (CDNs) a natural extension of DevOps processes by adding more application programming interfaces (APIs) and other automated functions to simplify a variety of tasks.
Specifically, Akamai has added an Akamai Sandbox that makes it simpler to conduct tests within the context of the Akamai continuous integration/continuous deployment (CI/CD) framework.
In addition, the company now makes it possible to cache GraphQL responses using the Akamai application programming interface (API) gateway, which provides a more efficient mechanism for caching REST APIs using the Akamai Intelligent Edge service.
Akamai is also making it possible to run a series of what-if analyses using the mPulse metrics it exposes now all the way down to specific resource levels. That real-time and historical data can be employed within the context of A/B testing on the CDN platform. Akamai has also made available Single Point of Failure (SPOF) protection within the Ion Script Management tool, which makes use of real user performance data to identify problematic first and third-party scripts on a user’s site. SPOF can automatically defers or blocks any scripts that don’t meet standards for performance.
Finally, Akamai has added an API for securing entire web application footprints within a Bot Manager tools, in addition to support for OAuth protocols for securing API calls.
Ari Weil, vice president of product marketing at Akamai, said the CI/CD framework that Akamai provides makes it possible to push updates to multiple CDNs. Akamai obviously would prefer all those CDNs to be provided by Akamai, but it recognizes that, in many instances, organizations will be taking advantage of CDNs provided by cloud service providers alongside Akamai, he said.
Weil noted that Akamai is trying to position its CDN as the first tier on which a wide variety of applications will be deployed and secured. Rather than deploying those applications on a local web server, the Akamai CDN provides a layer of isolation between cybersecurity attacks aimed at web applications and the infrastructure the rest of the enterprise relies on, said Weil. As part of that effort, Akamai has been exposing its CI/CD platform with the CDN, which organizations can employ to apply DevOps processes to applications that are deployed and updated on the Akamai CDN.
Many of the DevOps motions these days assume the application will be deployed either on a public cloud or in an on-premises IT environment. Akamai contends that a CDN not only provides a better platform for deploying and managing distributed applications, but those applications will be inherently more secure as Akamai absorbs the brunt of a distributed denial of service (DDoS) attack, for example, and less costly as the amount of data that needs to be moved in and out of a public cloud sharply declines.
Naturally, not every developer may fully appreciate the role CDNs can play within the context of a larger set of DevOps processes. But as applications become more distributed, there may come a day when the CDN service chosen winds up being a lot more relevant than any of the platforms being connected to the web.
The use of scripts as a key tool in the average developer’s bag of tricks has altered the development pipeline in significant ways. It has even led some organizations to outsource their entire team to manage the glut of scripts that have been developed over the course of time.
To help companies move beyond the “scriptocalypse,” we recently participated in a webinar with our friends at Tricentis. Sunil Mavadia, global head of customer journey at Electric Cloud, and Thomas Stocker, director of product management at Tricentis, walked through best practices to follow to avoid the scriptocalypse. Here are some takeaways from their discussion.
Improve Process Robustness and Scalability
Of course, everyone wants to improve their process robustness and scalability, but how? One of the most effective ways of doing this is to employ a tool that helps orchestrate your releases and provides reusable objects. This reduces risk because you no longer have to write scripts for every single application, which creates a cascading effect—not only do you write fewer scripts, but also you don’t have to maintain these scripts across the board. As a result, you are assuming much less risk across many levels of your technology and pipeline. Additionally, system-level visibility across all of your processes will increase—which has many benefits for your entire development life cycle.
Don’t Just Automate, Orchestrate
Many developers make the mistake of thinking that because they have scripts, they’re automated. But to be truly automated, every action needs to be automated and driven by your pipeline. Your approvals, your handoffs … everything. Oftentimes throughout the release process, team members find themselves having to stop in the middle of their tasks to ask for approval. This only creates a process that interrupts the entire pipeline. But what if you could automate all of that?
A great quote from the “2018 Accelerate State of DevOps Report” that speaks to this says, “High performers automate significantly more of their configuration management, testing, deployments and change processes than other teams.” If you look at it purely from a team perspective, the high performers are those that have fully automated all their processes. While this can certainly sound like a daunting task, it’s best to look at it with an agile mindset—this is an iterative process. You have to start somewhere, while keeping the goal of 100 percent automation in view.
Not only should you be automating, you also should orchestrate your pipeline. This is critical. By automating and creating reusable objects, you can get to the point of using your pipeline as a service. Imagine reusing this particular pipeline for every application within your process, so you could have hundreds, even thousands, all using this pipeline and all these objects underneath that are all reusable objects. It’s a pretty picture, isn’t it?
Create a Value Stream Map
When you create a value stream map for a pipeline, you want to be very inclusive about who you invite in the room to create the value stream. Include representatives from release management, development, deployment engineers, team leads, executives and infrastructure engineers. Basically, anyone who has “skin in the game” for your release should be present.
Once you feel you have a solid representation of who is impacted by your releases, sit down and draw out the entire process. This includes figuring out what each script does. While that may be a big task depending on how your pipeline is currently structured, there’s actually a lot of opportunity there. For example, you might find there is one application for which you use one script to do that deployment. Typically, that script is duplicated for another application and so on. Imagine doing that 100 times. You’ll have 100 scripts or variations of that one script for your 100 applications. That’s an obvious and beneficial place to deploy a bit of automation.
This is also a great time to understand what tools you are using for each step. When you’re jumping from toolset to toolset, it can be difficult to get a holistic view of how this is impacting your overall performance without a complete picture of the value stream. When you have that picture, you can start to ask yourself questions such as, “What are all these tools that we interact with that we can integrate into the pipeline? How are we handling approvals?” By including detailed sequencing timelines and identifying bottlenecks, you can realize some impactful efficiencies.
Finally, creating a value stream map gives you an opportunity to organize your scripts. Laying out the entire value stream can give you a picture of the variety and quantity of scripts you’re running—whether they are CI scripts, deployment scripts or test automation scripts—and bucket them in a logical way.
Identify Redundancy Dependencies and Bottlenecks
Of course, identifying (and hopefully rectifying) dependencies and bottlenecks is a good best practice for any aspect of your development pipeline. But it becomes even more important in the context of using scripts. If you have one application dependent on another application, it’s almost impossible to script. For example, imagine you’re trying to do some testing with two dependent apps. If you don’t have the right versions across both, you can’t test one because the other one is not up to speed or it hasn’t been updated in an environment. Something even as simple as that can be really disruptive and make scripting very difficult. By understanding where your dependencies and redundancies exist, you can take the first step toward solving the issues they present.
Scripts are one of the most effective ways of streamlining your pipeline, but they need to be used in a way that will allow your organization’s applications to be scalable and sustainable. By following some of the best practices outlined above, your organization can use scripts like pros—and avoid the dreaded “scriptocalypse.”
Do you have an application marked with cryptic warning signs and a wealth of cobwebs that is running on legacy hardware hidden away in the back corner of your data center? If you’re in enterprise IT, chances are high that you do. These old platforms are often considered a bane to IT. More importantly, legacy applications can present a real headache when attempting to uplift the people, processes and tooling needed to embrace a hybrid cloud model.
Providing self-service, orchestrated workflows and delivery pipelines are just a few signature attributes to constructing a hybrid cloud. Imaging trying to apply these ideas to legacy technology, where the interface requires using an archaic console and software that looks like it was written for Windows 3.1. It’s not fun and often derails any efforts to deliver services living in an on-premises software-defined data center (SDDC) or public cloud environment. It also splits up a team focused on working within the realm of a DevOps cultural model, because concepts such as infrastructure as code, stateless configuration and collaborative code check-ins crumble. For most folks I speak with, legacy applications remain with a small skeleton crew that keeps the lights running and the old hardware humming.
Hybridity is a reality for most infrastructures rather than being purely in the public cloud or purely on-premises. This is where the importance of DevOps and embracing an API-first mentality is multiplied. Having an API-first approach isn’t necessarily about having a single code repository or a single application that does “all the things.” It is about leveraging APIs across multiple repositories and multiple applications to weave together a single, programmatic software fabric in which everything can communicate and integrate regardless of whether legacy or cloud-native.
The way to end chaos induced by the complexity with hybridity is a progressive, policy-driven state with proper implementation.
Here are four tips on how to integrate your legacy crew with your DevOps team to create a high-functioning hybrid cloud model:
Find tools that play well with others: Legacy applications do not have to be the anchor holding you back from adopting a DevOps strategy. Find solutions that treat legacy applications, modern applications and cloud-native applications as if they were all first-class citizens. Legacy applications tend to run on old OSes, including some that are no longer supported. Eventually this configuration becomes increasingly fragile, requiring manual care and feeding. Adopting an automation strategy can reduce risk associated with build, testing, deployment, remediation and monitoring.
Strive for simplicity; don’t create silos: Make sure the workflows that deliver services are applicable to the vast majority of use cases. Historically there has been little to no incentive to building mechanisms that are shared across the enterprise. Systems thinking advises that there are dependencies, but if you only improve one application in the cluster, there will be no benefit. The entire cluster must be addressed for successful progress. The use of RESTful APIs is turning tables, allowing for a single set of tooling across many applications, platforms and services. Share data across the organization; leverage APIs to streamline how the team interacts with legacy workloads by extracting data from the legacy architecture.
Embrace a one-team mentality: Don’t form multiple teams or tiers of teams. If the decision is made to adopt DevOps, then it must be embraced by the entire organization. As the old adage goes: “There are no legacy systems, just legacy thinking.” DevOps isn’t about developers and operations folks collaborating; rather, it’s about two separate silos becoming one team. Focus on improving communication and set aside time for learning.
Avoid infrastructure-specific solutions: Abstract storage and compute; work instead on the application layer and how to deliver services for those applications. Legacy applications often are coupled tightly to the underlying hardware, making it challenging and risky to manage any application component individually. This means maintenance and upgrades are incredibly time-consuming, difficult and even expensive. The adoption of infrastructure-as-code, whether on-premises or in the cloud, gives teams the permissions and tooling provision infrastructure on-demand. As you can imagine, not every legacy application can be easily migrated to this type of infrastructure, but many can.
Using automation increases the organization’s agility by reducing human interaction and orchestrating dependencies across the organization. Self-service consumption of infrastructure further eliminates silos. DevOps is not incompatible with legacy applications, but it requires an organization to evaluate what this implementation actually means and how to embark upon this transformation. Never fear: You can still depend on a legacy application and embrace DevOps. It requires additional design considerations, elbow grease and a little bit of creativity. The fundamental element is to apply these principles in a consistent and efficient manner in the context of all applications, both legacy and modern.
Company founder and CEO Rod Johnson said with its Software Delivery Machine (SDM) project, Atomist seeks to eliminate much of the complexity developers currently must navigate as they build code, including making changes that require updates to dependencies across projects.
That capability makes it easier to keep multiple development projects consistent and up to date automatically, Johnson said.
Atomist is applying many of the concepts Johnson used to create the widely employed Spring framework as an alternative to Java Enterprise Edition (J2EE), which is now being developed under the auspices of Pivotal Software, a unit of Dell Technologies. Atomist created a domain model for software delivery based on an API server and GraphQL, an open source data query and manipulation language.
This approach eliminates the need to work within the construct of more cumbersome framework based on, for example, YAML files, Johnson said. Rather, Atomist is making use of code to develop an alternative framework that is optimized around a more familiar API construct, which frees developers to extend the SDM in any direction that best reflects how they want to work.
Atomist currently provides a service based on SDM to large enterprises that have aggressively embraced DevOps. Johnson said that service enables organizations to allow developers a maximum amount of flexibility within the context of policies that can be defined by an IT operations team. The local instance of SDM is intended as an open source tool for increasing individual developer productivity.
Most organizations are becoming increasingly obsessesed with developer productivity at a time when companies recognize how dependent they are on software to differentiate themselves. The rate at which new and existing applications are being rolled out and updated has expanded considerably. For developers face to rise to that challenge, many of them embraced microservices that make it easier to add and replace new functionality using Docker containers. Those microservices, however, create a massive number of dependencies across different code bases that must be tracked. SDM can be employed to not only spin up containers images and Kubernetes clusters, but also update all the interdependent microservices spanning multiple clusters.
Asking developers to keep track of and manually update all those dependencies involving multiple projects is not the best use of their time, Johnson said. Real DevOps progress will be achieved only when organizations provide tools that make it easier for developers to do the right thing in a way that doesn’t try to impose an inflexible framework.
Most of the processes that organizations rely on to build code today are not nearly as automated as they should be. In fact, DevOps adoption arguably would be much further along than it is today if it didn’t require so much manual effort.
