A funny thing happened on the way to being agile. We adopted a ton of tools to make us more responsive. They really do help us deliver software quickly and efficiently, and they have improved IT processes along with our specific process improvements.
They also created a toolchain that resembles areas of responsibility of old. That makes perfect sense; we still need to store source code, build applications, deploy somewhere, worry about security, etc. And we need to do all of that faster, which is where automation and the toolchain really shine.
But you need to keep in mind the overall impact of the toolchain upon the development process. Now the organization is dependent upon those tools. We went from slow and steady to fast and with a large series of steps along the toolchain. Things are better, we’re delivering quality software at a faster pace, but in some ways, the process risks becoming more fragile. If a tool in the chain shows signs of trouble, you need to be aware of it and on top of the issue almost immediately. The price of ignoring issues in the process can be catastrophic disruptions in software delivery. Losing a CI tool is akin to shutting down customs at an international airport: The source of the problem is big, and the downstream problems are show-stopping.
So while you are tooling your applications, make certain you are tooling the build chain, also. Know when a build that always takes ten minutes suddenly spikes to triple that. Know when a test suite delivers non-breaking errors at an increased rate. Know when space is low on a shared drive that the build tools use. Tool everything, because the software delivery toolchain is now the core of application development and deployment.
Tooling these systems can also offer a warning of problems in a given application. It is possible that the increase in build time is because there are a lot of previously unused libraries being included … and that those libraries have not been properly vetted by security. It is possible that the disk space on that shared drive has PII test data that needs to be cleansed. There is a lot of good that can come from over-tooling the build chain, and unless you have a bit of tooling that slows the process, there is no downside. It won’t take too long to build a list of what to watch, and modern DevOps tools have enough reporting and APIs that it won’t take long to begin watching those items, either.
That is where the people part comes in. Don’t ignore the tooling once it is in place. Build a process to regularly review results and take action when necessary. No tooling is useful if the results are ignored; sadly, we have a long history of spectacular failures that started with, “We saw indications, but didn’t think it was important at the time …” So act on results, even if that just means flagging them for review in the next reporting cycle—just don’t ignore them.
And keep rocking it. The organization thrives on the software you build, deploy and support. It is a massively complex system that has IT staff at its heart, so keep it thriving and watch for indicators of illness in the core toolchain.
In our industry’s attempts to follow best practices that marketers assure us we must have or we are going to lose to our biggest competitor … No, wait, we’ll lose to that startup no one ever heard of … No, hold on, we’ll lose to foreign competition … No, wait … (okay, I’ll stop. You get the picture), many of us have gotten completely unreasonable in our IT approaches.
Like asking a vendor, “Where can I run your application?” Well, the more “where” options the better, because every organization has different needs. But a vendor has to start somewhere, and you can tell when they get to a certain size because they don’t want to talk about it. “We’re SaaS,” was the en vogue thing to say six months ago; now we’re back to “containers” or even “cloud,” thanks to a whole slew of cloud-first offerings out there. But what all of these vendors really need to be able to say (and a disappointingly small number of them actually can) is, “We run on containers, so we don’t have to care where you deploy us.”
Those are the types of scenarios that forward-looking enterprises need to start anticipating—the broadest, easiest-to-implement, longest-lasting solutions that can be found. The days of “devs picked this DB no one ever heard of because it was 0.0000001 microseconds faster on a hundred-line query,” need to be gone. Standardize on highly usable tools across the DevOps toolchain from libraries approved for use in apps to deployment technology.
A big part of this approach right now is my pretty common refrain: Advanced automation. The more a system can do, the more time staff has to do other important things. All forms of testing are finally within our grasp because of advanced automation technology ranging from Kubernetes programmability and controllers to ‘AI’ filtering of results (Side note: What most IT folks currently call AI is not; we need to stop calling it that and give it a more accurate name. Machine learning is better; advanced automation is more to the point, IMO). But whatever you call it, it enables massive pre-release testing. From functional to security, it offers the ability to make these tests useful by speeding up testing and results processing so staff need only look at issues considered critical. No one cares where the test engine runs, just that it can run tests wherever the application resides.
The same is true for a whole lot of other ‘whats.’ Vulnerabilities are probably the best example. Tools are starting to look for them in an automated fashion. Very few organizations have a comprehensive list of which vulnerability databases a tool must support. Any given organization might have one or two databases in their list, but most just require that the functionality be available.
So don’t get bogged down in the details, and don’t select five different tools to do the same job on five different teams. Start standardizing again. Oh, standardization won’t last when something evolutionary comes along in the space, but it will leave you less clutter to clean up and more specific expertise in-house.
And that leaves you more time to keep wowing the business. Keep those systems running smoothly and keep cranking out more useful stuff for the organization. Most orgs show their appreciation these days, but some still do not. In those cases, we thank you for them. We know what you’re going through—we are going through it, too, and yet the lights are still on. Keep rocking it.
One thing Agile and DevOps definitely brought IT was a more accepting view of the whole “mistakes happen” mantra. Crashing systems is no longer a guarantee of a free ticket out the door. Indeed, off the top of my head, I can think of at least two cases where a CIO themselves checked in sloppy code and it trashed the system—CIOs at orgs big enough that they probably should not have been coding in the first place.
And that brings us to today’s blog. The standard response to “But what if there is a bad update?” is “We’ll just roll out a new version!” That works in some instances. In a lot of instances—most that involve GitOps as part of DevOps—it doesn’t. Implosions can be huge and take out chunks of infrastructure. So you need a better plan than just assuming you can fix it with a forward update. Do you have rollback capability? Are you making use of it, assuring all is set to roll back if needed? Testing to make sure rollback works with the other changes made across the system in this update?
That is part of the problem. Fixing with a new update is the best option, simply because in the age of massively distributed, microservices-based solutions, rolling back comes with a ton of baggage—enough that it may not be viable for you. Okay. So you can’t quickly roll forward. You can’t quickly roll back. Quick! What do you do?!
And I don’t have the answer to that question, because I’m not in your organization working on your systems. At some point, all IT is personal. And this is one of those points. You need to know what your best options are if an update spirals everything, and you need to have a plan to implement it. But what your best options are is not going to be the same as the next org. This is that point.
One option is to keep the ability to build the entire system. That’s a large setup, but every minute systems are down is hurting the company. And that’s what you need to plan for. Think of it as disaster planning for DevOps. A man-made disaster that destroys systems and infrastructure.
So, like any other disaster planning scenario, walk through the chain that makes the system work, identify weaknesses and list ways to address them. Then test those to make certain they do what you hope they will. Then, set up an automated system to keep this whole plan up-to-date.
In short, we’re in a new automated landscape, and we need a new automated tool to pull our rears out of the fire when the inevitable happens. Be it from dev error or malicious attacker, it is a safe bet that, sooner or later, you will have a massive systems outage that your DevOps toolchain can’t adequately address. Know what you are going to do. Or at least have thought about it, so you’re not just starting to think it through while coworkers and customers can’t access systems.
And keep rocking it. This is just another layer of protection for all the hard work you’re doing. Take the extra step. Like insurance, if you ever need it, you will absolutely be glad you did.
Perforce Software has added a universal package manager, dubbed Helix Artifacts, to its portfolio that enables users of its Helix Core version control platform to store and track source code.
Gerhard Krüger, product manager for Helix Core digital asset management at Perforce Software, said the goal is to enable DevOps teams that employ Helix Core to access a package manager without having to exit Helix Core.
That approach also provides a more secure and cost-effective way to store, consume and distribute artifacts that also serves to improve developer productivity, he added. The ability to manage artifacts on top of a version control platform also means DevOps teams will not need to set up additional storage infrastructure or configure separate security permissions just to distribute software packages, noted Krüger.
Helix Artifacts is a self-hosted package manager that functions as a translator for packages such as NuGet, Maven or Docker that enables them to communicate directly with Helix Core as well as build systems and integrated developer environments (IDEs). Helix Artifacts is made available via a per-server subscription model that costs substantially less than rival approaches to managing artifacts, Perforce Software claims.
Most DevOps teams are trying to strike a balance between employing best-of-breed tools to build applications and the need to reduce complexity and, ultimately, cost. As such, any time a tool becomes a feature of another platform that a DevOps team uses, there will be a review of the processes being employed. Many DevOps teams are trying to reduce the amount of time they spent maintaining tool integrations.
There’s also now a lot more focus on software supply chain security. Each new tool generally creates yet another set of credentials that need to be protected as cybercriminals continue to aggressively attempt to inset malware into applications before they are deployed in a production environment. It’s not clear to what degree DevOps teams are looking to consolidate toolchains, but given those rising security concerns, it’s only a matter of time before security teams start to ask why the DevOps toolchain attack surface is so wide.
In the meantime, the number of organizations that have widely embraced DevOps best practices across the enterprise continues to steadily grow. Over the past decade, most organizations have had at least one application development team embrace DevOps. However, as the pressure to build software to drive digital business transformation initiatives has steadily increased since the onslaught of the COVID-19 pandemic, the need to apply DevOps best practices to accelerate application development has become more pronounced. As a result, more organizations than ever are looking to standardize their application development efforts on a smaller number of platforms to reduce friction.
It’s too early to say whether the rate at which applications are being developed and deployed since the start of the pandemic has actually increased. The one thing that is certain, however, is the bulk of organizations are now trying to accelerate like never before.
Imagine if you backed out of your driveway, tapped the gas and BAM! You are at work. Nothing to worry about along the way, no traffic congestion, no other cars to worry about, etc.
That is, unfortunately, how many of us approach automation. “Fire and forget” is real, until a problem crops up. I could talk about some complex environments, such as ARA, in a containerized environment with multiple languages, accessing mobility-limited databases and targeting several client platforms, for example.
But we’ll keep it simple, and I’ll make an admission. Backup was one area where we at Ingrained Tech fell into the fire-and-forget trap. One part of our backup system runs desktops out to a server, the server (and other servers) out to a NAS and the NAS out to the cloud. I don’t think our security peeps will freak out if I mention we use duplicity for this backup and Backblaze for our backup cloud. Backblaze is a dream to use compared to others we’ve looked at, and the pricing is competitive, so it suits our needs well.
A few weeks ago, we realized that while backups were completing successfully and verification was fine, it was taking nearly the entire week to do our weeklies. That’s bad news. We back up terabytes of data, but this is 2020, and our drive IO/Network can handle it. We started by looking at both of those things, and sure enough, it was drive IO causing our problems.
Unsurprisingly, reading the disk and creating a collection of encrypted tarballs (a super-simplification of what duplicity does, but gets the point across) on those same disks was causing problems. After investigating, the one build that occurs on that server was suffering from backups stretching out, too. We had seen occasional issues with the build, but nothing jumped out as the source, and it was only one night every month or two. It was on our list. You know, the same prioritized list you have. And this build wasn’t nightly-critical. You get the idea.
This was a relatively urgent problem, as soon as last week’s backup would not be complete when this week’s tried to start. Not good.
We took our DevOps mentality to the problem and discovered a simple solution that solves the issues. The backup NAS is RAID 50, so it can handle a lot more abuse than the server that was slowing down backups. So, we took the simple expedient of rsyncing the source server to the NAS, then using duplicity to tar and encrypt before sending it on to the cloud buckets. Internally, we do not believe we need to encrypt backups because if someone is in our network, all of that data is laying around. By putting duplicity and both the tar function and the encryption function onto the backup NAS, we moved it to a far less busy system. Backup times dropped from days to hours instantly, and our unrelated build problems went away.
We have to rework the automation somewhat to make sure validation is, well, actually validating the data is correct. But we’re well on our way to having a stable, faster system.
How does this fit with DevOps? We see the same thing happening often in the automation rush of DevOps. Yesterday’s best solution is not necessarily tomorrow’s best solution. “Automate and watch” should be the catch-phrase. Improvement comes from both DevOps culture and paying attention to changes in the tools market.
As Alan pointed out, someone will eventually win the end-to-end DevOps rush—since I feel the urge to inject my opinion on who stands the best chances, expect a blog on this topic soon. Until then, knowing what your options are and how to best achieve your goals–even on problems that were “solved” last year–is your best path for continued improvement. Yes, it means rework, but just like reworking your code, the point is to streamline things, get closer to your organization’s goals and keep improving after the low-hanging fruits of DevOps have been gathered. The rate of improvement in several areas of DevOps is stunning, so you may find you actually like a new tool better than the one you chose a couple years ago. And remember, knowledge is power.
Innovation and environment are only two factors in determining which DevOps tools are best for your organization
Any time there is a movement in IT that involves a new space and new toolsets, there is sudden growth, then inevitably consolidation and, finally, a selection of tools that suit the needs of most customers. Then the cycle starts over. Usually by this point, users have developed a defense mechanism about their choices regarding which solutions to implement and discussions can get heated.
I’m happy to say that the last bit—heated discussions—has not occurred with DevOps the way it has with other technologies (think the browser wars, OS dominance or even cloud service provider preferences). There is the occasional “You’re doing it wrong,” but not the wild-eyed flame-wars so far.
But the other two? Yes, yes indeed. DevOps has been through a couple of cycles, and the end is nowhere in sight. I don’t necessarily see that as a bad thing; in fact, as long as innovation is occurring, you have the opportunity to look at changes and decide what can best help your DevOps efforts.
My mantra since the language wars was that C, C++, VB, etc., were the “right tools for the job.” I have kept that mantra for all of IT. It is easy to adapt the tools you are familiar with—to do more than they were designed to—and most shops have fallen into this trap at some point. But a serious look at what is available and where the market seems to be evolving usually stops this kind of adaptation.
Some shops use Jenkins for far more than it was designed for. It’s a great tool, but at some point, the benefits of using tools aimed at a given piece of the DevOps toolchain outweighs the fact that Jenkins could handle that bit.
The popularity of Puppet is starting to cause the same issues, if anecdotal evidence is to be believed. In words I use to my children, “Just because you can, doesn’t mean you should.”
The environment always plays a role in the “right tool for the job,” but it should not be the major factor. Having a DevOps tool that you already know that could do the job is one answer—but quite often not the best answer.
So, use resources like us here at devopsy.kinsta.cloud, and your favorite site for IT ratings and reviews. Do some feature research, think about what it would take to make your existing tools do the same job, and consider bringing in new tools to achieve a given goal. We see so much change these days that sticking with what you know on this one tiny bit of the toolchain, may seem reasonable, but make sure you are not missing out on additional functionality or suitability to task over the long term.
On March 15, I gave four keynote speeches on DevOps in Toulouse, France. As the main speaker of Econocom Club MEP 2018, an annual think tank summit organized by the European business-to-business digital services provider, I shared the day with 15 firms in industries as prestigious as financial services, energy, aerospace, aviation, and telecommunications and heard all about their implementation projects.
The lessons learned from this year’s summit are troubling, they tell a story the vast majority of DevOps vendors aren’t prepared to hear: They miss their client expectations. There’s a severe disconnect between what’s expected and what’s delivered.
Participant visions and approaches didn’t surprise me; they simply confirmed my point that DevOps is misunderstood. By stressing the implementation of Continuous Delivery (CD) toolchains—selling technology is more lucrative than transforming a business—IT service providers aren’t helping.
What do these lessons tell, exactly? if you’re an open-minded leader who understands that a new IT service delivery paradigm more focused on business concerns is needed, this is for you.
The Untold Frustration
In his opening speech, facilitator Fabrice Berne was inspired to ask participants while they introduced themselves to share the purpose and challenges of their DevOps implementation project.
What I heard didn’t surprise me; it was a mix of fascination and frustration—participants were at the same time excited to work on a DevOps project and frustrated because they weren’t getting the promised business benefits.
“The reason is, you all tackled your DevOps project the wrong way,” I said. “You started by implementing the CD toolchain, thinking tools on their own will deliver the expected business benefits. That illusion is widespread in the IT community!”
So, What is DevOps?
As my claim turned the atmosphere icy, I added, “DevOps can’t be narrowed to these tools—not even to agile practices and automation. It’s more than that. DevOps seeks to make the business competitive, wealthy and healthy by continuously delivering high-quality software.” And concluding my introduction, I made it clear: “The fundamental question is, How is business value created? Many experts, particularly techies, struggle to answer it.
“Let me say this straight: Most DevOps projects fail because certain experts erroneously spread the idea that the most important piece of DevOps is the CD infrastructure,” I said, adding, “The fact of the matter is these experts and the IT solution vendors they represent are creating confusion; through massive marketing hypes they force CIOs to address DevOps through the distorted lenses of the so-called DevOps tools.”
The Inconsistent Disparate Visions
The following chart results from a survey we conducted at ITaaSNow last year. It involved 2,681 businesses that were unhappy with their cloud and DevOps migration projects; we wanted to show how poor DevOps approaches lead businesses to failure:
Figure 1 – Disparate and erroneous visions that cause implementation failure
“As you can see, there’s no homogeneous perspective right now,” I told the audience. “It varies according to the business culture, objective and strategy or the DevOps architect’s understanding, experience and expertise. In most cases, these customized visions are incomplete and generate no business value at all.”
I picked two failure causes in the chart to demonstrate my point.
Starting with the notion that, “automation is the core of DevOps,” I argued, “DevOps primarily seeks to help generate business value. Generating business value is about identifying market and customer expectations, translating expectations into profitable service ideas, developing, deploying, promoting, selling them and continuously improving customer experience.
“Where did techies get the idea that to make a business profitable the only thing to do is to implement Git, Jenkins, Cucumber and Chef into a CD toolchain?” I asked, somewhat ironically.
Picking a second cause, “DevOps strategy should only stress software development activity,” I continued. “Thinking that isolatedly software development activities will generate business benefits without involving marketing, sales, testing and operations is a pure illusion. Once again, generating business value is a complex process that covers identifying market and customer expectations, translating expectations into profitable service ideas, developing, deploying, promoting, selling them, and continuously improving customer experience.
“Where did solution vendors get the belief that, software development alone is enough to generate business value?,” I asked. “That’s another myth they nurture!”
I then warned: “Make no mistake about it: All the visions represented on the chart are either incomplete or wrong. They do not reflect what DevOps is. You must acknowledge once and for all thatit is a business transformation effort, not only an IT capability implementation. Period.”
The Contrast with the Founding Fathers’ Vision
“Who better than its founders can provide the right vision and definition of DevOps?” I asked.
In his article, Adam reminded us, “Becoming better at building and delivering software is the stake. It’s not an optional piece of their strategy: It’s the future of how their customers want to work with them. Software is table stakes for survival.”
He added, “But the real answer lies in looking at what John, Paul, and Jesse were doing in 2009—not in the specific technical choices they made, but the style in which they worked, the essence of what they believed made them high functioning and successful. … Then the challenge begins: how to apply this new style to businesses trying to become better at building and delivering software.
“DevOps is about bringing together all the people you need to build and run your business effectively, and empowering them to move as quickly as possible toward their goals. Tools matter. Make no mistake: Trying to change the way you work without changing the mechanisms by which you do that work is a futile exercise in excruciating failure.”
He concluded, “Fundamentally, it is about taking the behaviors and beliefs that draw us together as people, combining them with a deep understanding of our customers’ needs, and using that knowledge to ship better products to our customers.”
With Respect to Techies, DevOps is Primarily a Business Operational Model
“If you paid attention to what Adam Jacob reported about Jesse, John and Paul’s vision, you probably noticed that it emphasizes two fundamental elements of high-performing businesses: the well-known IT infrastructure and the unknown operational model,” I told the audience.
Moving to the next slide, I provided evidences to my point, “As the picture shows, DevOps, according to its founding fathers, isn’t the CD platform, automation toolchain or the platform-as-a-service (PaaS) infrastructure solution vendors and certain consultants would like you to believe.”
Figure 2 – The Founding Fathers’ Vision
The slide seemed to clear up a doubt about what the audience had been hearing. I confirmed, “With all due respect to my fellow techies, as I’ve been implementing it, DevOps transcends software development issues; it’s a business operational model supporting your company’s competitiveness.”
Figure 3 – DevOps as the Business Value Chain
I told the audience I chose to represent it as a two-component system “because I wanted to make sure IT professionals and business leaders understand that implementing DevOps is primarily a business transformation effort for optimizing the organization’s operational model and IT infrastructure.” I added, “The operational model is the subset of interactions, staff and skills, processes and practices, values and beliefs, and tools that’s repeatedly mobilized across your organization to generate the expected business benefits. It’s where all its value propositions and competitive advantages—organizational agility, flexibility and velocity, innovation culture, more like it—are created. As (fellow speaker) Stephanie Carton puts it, the operational model is where DevOps culture is implemented.”
Anticipating the audience’s reaction to that point, I said, “It doesn’t mean CD toolchains are useless or doesn’t bring value, without them we’re dead. The point here is, business benefits result from an optimized operational model enabled by a well-engineered CD toolchain.”
I’m not the first one to go against the techie perspective of DevOps, I warned. “Ernest Mueller in his fantastic, “DevOps: It’s Not Chef and Puppet,” raised the red flag.” His message is clear: “If someone has a systems problem, and you say that the answer to that problem is Chef or Puppet, you understand neither the problem nor the tools.”
The Near Future of DevOps
In the first part of this article, I wanted to share with you the clarifications I made and more importantly, re-establish the truth on messages as erroneous as “DevOps means CD toolchain and Agile practices,” and “CD infrastructure or software development on its own creates business value.” They’re misleading CIOs and are taking hundreds of projects straight into the wall!
Figure 4 – DevOps Big Picture
In part 2 of this article, next month, I’ll tell you what we debated about the near future:
Why Agile and Lean practices will boost business competitiveness?
What is ITaaSNow DevOps Capability Maturity Model (CMM)? How does it enable continuous business performance improvement?
What is Artificial Intelligence for Operations (AIOps)? Why will it make DevOps the cornerstone of businesses’ digital strategy?
Stay tuned!
The Key Takeaways
This what I wanted to tell you about DevOps as we discussed it. The key takeaways are:
DevOps isn’t narrowed to agile practices and CD toolchains. Don’t fall into the trap of certain solution vendors who tell you that Git, Jenkins, Cucumber, Chef, Containers and microservices will boost your business performance. It’s false; things aren’t that simplistic.
DevOps combines an optimized operational model and a CD platform to bring together marketing, sales, post sales, software development, testing and operations into an application delivery capability that boosts your organization’s competitiveness.
Starting the CD toolchain implementation without optimizing the operational model is the surest way to take your DevOps implementation project into the wall.
DevOps implementation is primarily a business transformation effort, not just an IT capability deployment effort.
What matters isn’t applications and digital services delivery at the speed of light, but the timely delivery of added-value applications and digital services to the right market segments and business users.
According to Econocom’s feedback, most participants are reviewing their DevOps implementation strategy; they will make their operational model optimization the priority. Mission accomplished!
