devopsy.kinsta.cloud

Category: DevOps Practice

How to Calculate ROI on Infrastructure Automation
Infrastructure and cloud automation. You know you want it. Infrastructure as code and programmable infrastructure are hot topics these days. The possibilities opened by applying software development practices to infrastructure make the vision of transparent infrastructure closer than ever.

But before you start automating, you may want to look at the value you get from your investment and make sure your plan makes sense. After working in automation for over a decade, I’ve seen the importance of having a clear understanding of the value of automation time and again. It’s about time I write about calculating the return on investment (ROI) on infrastructure automation. But before getting to the specifics of infrastructure and cloud automation, it may be wise to start with the value of automation in general.

The Value of Automation

The equation is simple. You have a long, manual process. You figure out a way to automate it. Ta-da! What once took two hours now takes two minutes. And you save sweet 118 minutes. If you run this lovely piece of automation very frequently, the value is multiplied. Saving 118 minutes 10 times a day is very significant. Like magic.

The Value Formula

The formula for the value from automation can be very simple:

Time or Money Saved by Automating a Manual Task

Time or money saved by automating a manual task should include any benefit you can attribute to the automation. Here are some examples:
1. The time saved by automation to accomplish a complex task (this process took eight hours. Automation does it in eight minutes. Now we can do it more often).
2. The time saved for the person who used to do it manually (the person who used to do it manually now has eight free hours, while automation does the work).
3. The time saved for a person who was waiting for the task to be completed (the person who used to wait eight hours for the task to be done, now waits eight minutes; they can do more).
4. The time saved by not making human errors (the person who used to spend 10 hours troubleshooting after the eight-hour person finished saves 10 hours).
5. The time saved by not having to document a manual process (the two people who spend five hours a month teaching other people how to do the eight-hour thing …).
Investment in Automation

Investment in automation is often naively calculated as the investment in building automation, but the real investment in automation also includes maintaining and supporting what you have built. So, if someone tells you it took five minutes to whip up this little Bash script, ask yourself what it would look like to update it and support it forever (and when there are 300 more of these scripts). Your calculation of the investment in automation must include building, maintaining and supporting your automation.

What Should I Automate?

The formula makes it easy to decide what your automation targets should be—or, in other words, where you would find the most value in automation.

A good start would be to look for tasks that take a long time to do manually or tasks that are done very often, as automating them will save more time and money. But you need to make sure the investment in automating them would justify the gain.

I wish I had a dollar for every time I talked to someone who wanted to automate the “most difficult task” to prove the value of automation. This follows the logic, “If you can do the hardest thing, you can also do the simplest thing.”—a bit like making it in New York City. But this is not how automation works. Why? Because automation is very much like software development.

In the software development world, one of the most common answers product teams gets from developers when they ask, “Is it possible to implement feature X?” is: “Everything is possible. It’s just a matter of effort.”

Well, automation works the same (and that’s not the only similarity to software product development).

Pretty much everything can be automated. The question is whether it’s worth it.

If you automate something that takes a whole day to do manually, but you only need to do it once a year , it may not be worth automating.

You could be automating the stupidest thing, and only save 20 minutes, but this thing happens 2,000 times every day, so that may very well be worth it. And the more stupid it is to automate, the better.

Real Life Infrastructure Automation

Back to the value formula. In real life, there are more facets to this formula. One of the factors that affect the value you get from automation is how many people have access to it.

You can automate something that can potentially run 2,000 times a day, every day; this could be a game-changer in terms of value. But if this is something that 2,000 different people need to do, there is also the question of how accessible your automation is.

Getting your automation to run smoothly by other people is not always a piece of cake (“What’s your problem?! It’s in git! Yes, you just get it from there. I’ll send you the link. You don’t have a user? Get a user! You can’t run it? Of course, you can’t, you need a runtime. Just get the runtime. It’s all in the readme! Oh, wait, the version is not in the readme. Get 3.0, it only works with 3.0. Oh, and you edited the config file, right?”).

Your calculation of the time or money saved by automating a manual task needs to consider the value derived from automation accessibility:

The more people involved, and the more distributed they are, the more difficult it is to achieve accessibility. Your calculation of the investment in automation should therefore also include the investment in self-service:

Adding Self-Service

The self-service part of the equation is often neglected when looking at the value of automation, especially when the focus is very technical. It’s very easy to show a promise of value if you can automate a task, and by the time the question of self-service surfaces, it may be too late.

This is very common with grassroots automation. In my field, I see this time after time, with teams that adopt free infrastructure automation tools (Ansible, Terraform, Cloudformation, ARM, you name it). In 2020, most people understand that there are no free lunches and that it takes precious time to develop and maintain this automation. Most already get that supporting it over time results in an automation team that is often too swamped to do much else.

But I dare say that few properly estimate how difficult it is to provide this automation as a service to other teams. I don’t blame them. For an engineer, the thought “This is too slow! I will automate it” comes very naturally. It’s much more difficult to imagine as a developer a user who doesn’t understand the beautiful capabilities of Terraform; someone who freezes when they hear, “It’s in GitHub”; someone who isn’t as fascinated by the elegance of the code and has no #$$%$@ idea what to do with code.

In a company with multiple teams, the gap between the people who can automate to the people who need the automation can be huge. And if you ignore it, your automation value formula can be very skewed.

Can Everyone Run This Automation?

When you consider automation, the technical question, “Can it be automated?” is not enough. How efficient the code will be, or how much time it would take to build the automation, is also not enough.

The question, “Can all relevant users run it as a seamless part of their workflow?” has to be asked and weighed in. You may be surprised, but highly accessible yet technically mediocre automation can actually win in some cases.

How the Value Equation Works in RPA

Robotic process automation (RPA) is a great example. RPA systems typically provide users with lists of automated actions by recording tasks that the user performs on a graphical user interface.

For many coders and automation professionals, this approach is almost sacrilegious. You can automate fast, but scaling the technology is extremely limited. It gives a wide audience access to automating tasks without enforcing coding best practices (we already know the masses can’t be automation experts). But RPA can offer high ROI in some cases. If we return to the value formula, this quick-and-dirty type of automation is highly accessible, and therefore the investment in self-service is low. If using RPA means x100 or x1000 people can benefit from automation, compared to high-code tools that only a few can use, we may be able to get impressive ROI.

Does this mean we should use RPA for DevOps automation? Not really. RPA is not always the right tool for the job, and it won’t always provide the best value in automation. RPA works well when you want to eliminate toil in small tasks (e.g. it is very popular for accounting tasks that a person triggers manually), but the value is lower when you have strict scale, efficiency and reliability requirements, which is typical in DevOps.

In the case of DevOps, with machine-to-machine heavy interfaces, the investment you will have to make in scaling and maintaining RPA can be very high and the ROI would eventually be very low. RPA may therefore not provide the greatest value in this field, and it makes sense to opt for more scalable options. But this doesn’t mean you should focus solely on good engineering and the question of accessibility and self-service must be addressed in your ROI calculation.

Infrastructure Automation in DevOps

In the past years, there has been a growing awareness of self-service in traditional infrastructure automation. Often led by IT and Ops, some of it started with simple means, such as connecting ticketing systems (JIRA, Service Now) to automation tools.

However, in DevOps infrastructure automation, accessibility of automation is often an afterthought. DevOps automation initiatives usually start with developers. The natural tendency is to invest in high-code automation and to focus on value derived by the time saved on the execution of a task rather than the multiplication of the value through the enablement of a broader user base. Existing self-service mechanisms are thrown aside, and for good reasons: they are too slow and they don’t seamlessly fit into the DevOps process.

Ignoring self-service is one of the reasons why DevOps initiatives show great value-promise in small-scale and fail in larger scales.

Moreover, since DevOps spans so many areas of subject matter expertise, the chances of leaving critical expertise outside the coders’ silo is very high—especially expertise in operations, security and finance. Ignoring these experts’ accessibility to automation leaves them outside the DevOps loop.

I was very happy to see Gartner’s “Market Guide for Infrastructure Automation Tools,” published in April. If you don’t have access to the gated content, I recommend taking a look at Manju Bhat’s post on the topic. Gartner finally highlights self-service as a critical capability for infrastructure automation:

“Infrastructure automation tools provide DevOps teams with on-demand, self-service access to standardized environments. Adopting these tools enhances their ability to deliver customer-focused agility and velocity improvements while consuming new technology platforms with limited disruption.”

I see this acknowledgment in the importance of customer-facing accessibility as a crucial milestone on the way to realizing value in larger-scale DevOps initiatives.

Additional Value Factors in Infrastructure Automation

Unlike general automation, infrastructure automation is a field in which there are additional factors in the value formula, which fall under the Infrastructure Governance category:
1. Cloud security.
2. Cloud cost.
3. Compliance with industry standards.
These elements need to be taken into consideration when addressing the value derived from automation. Infrastructure automation, especially in public cloud, involves sensitive assets: compute, network, storage, data and security elements. These elements can be expensive, and they could be very vulnerable to attacks.

If ignored, these three factors can diminish value. But handling them properly in automation can also generate value.

The “time/money saved by automating a manual task” could, therefore, include an additional value segment:

And the investment calculation should also include embedded governance:

Build Your Value Formula

If you are working on an automation initiative, don’t forget your value formula. It will help you build a business case, get management buy-in and achieve a better understanding of value factors you should focus on.

The TLDR version:
July 1, 2020
Businesses Modernizing Apps in Wake of Pandemic

A survey of 5,000 business decision-makers, IT decision-makers and app developers published suggests organizations are taking advantage of the crisis created by the global COVID-19 pandemic to accelerate the modernizing of apps using DevOps practices.

Conducted by the market research firm Vanson Bourne, the survey finds 80% of respondents said organizations must modernize the technologies they use to develop and deploy software to be more successful. Nearly as many (79%) said their organization will not be able to deliver a best-in-class end user experience without successfully modernizing software development processes.

In addition, 80% of respondents said embracing developer processes such as test and learn would enable organizations to work more effectively and efficiently.

The survey also finds some organizations are not letting a proverbial good crisis go to waste. A total of 70% said they have good or excellent alignment between application development and IT teams, up from 64% prior to the pandemic. Alignment between business teams and application development (67%) and IT teams (67%) increased by 10% and 12% respectively since the start of the pandemic, according to the survey results.

Sumit Dhawan, senior vice president and chief customer experience officer for VMware, said the need to make applications more accessible and reliable is accelerating a transition to the cloud. The survey finds the primary benefits of modernizing apps are enabling remote workforces (54%); pushing quick updates in response to changing landscape (42%); and maintaining reliable uptime (41%).

The scope of application modernization initiatives varies widely. Some organizations are simply lifting and shifting existing application workloads along with the virtual machines they run on into the cloud. Others are refactoring applications to run natively on a different virtual machine platform provider by a cloud service provider. Others are wrapping existing applications in containers so they don’t have to be refactored when deployed on another virtual machine. Still others are employing containers and Kubernetes to develop entirely new microservices-based applications that tend to be more flexible and reliable than existing monolithic applications.

Regardless of the path forward, the inability to physically access local data centers during the pandemic has accelerated a shift to the cloud that was occurring even before the pandemic hit. There still will be many applications that for performance, security and compliance reasons may need to remain in on-premises IT environments. However, it might not be long before IT organizations also modernize those applications to make them both more accessible, flexible and resilient.

Most IT teams will find themselves managing microservices-based applications alongside modernized monolithic applications for years to come. Over time, however, many of those monolithic applications will be deconstructed into a core set of microservices.

At the core of all those initiatives will be a set of best DevOps processes that each IT team defines for itself based on the nature of the application portfolio that needs to be developed, deployed, secured and maintained. The challenge now is determining which applications should be modernized versus simply replaced by either a software-as-a-service (SaaS) application or a custom cloud-native application that is a better fit for the purpose.

June 30, 2020
Why is Site Reliability Engineering Important?
Site reliability engineering (SRE) is one of the fastest-growing enterprise roles and set of operational practices for managing services at scale.

During the Site Reliability Engineering SKILup Day June 18, I asked a group of the DevOps Institute Ambassadors why SRE was important to them.

Below you will find some insightful responses, tips and calls to action. If you missed SRE SKILup Day or wish to rewatch any of the sessions on-demand, visit the conference landing page and access the videos for free: https://devopsinstitute.com/site-reliability-engineering-skilup-day/.

Happy Reading!

Anshul Lalit
DevOps Institute Ambassador from Kristiansand, Norway

“The term SRE surely has been introduced by Google, but directly or indirectly several companies have been doing stuff related to SRE for a long time, though I must say that Google gave it a new direction after coining the term ‘SRE.’ I have a clear view on SRE as I believe it walks hand-in-hand with DevOps. All your infrastructure, operations, monitoring, performance, scalability and reliability factors are accounted for in a nice, lean and automated system (preferably); however this is not enough. Culture is an important aspect driving the SRE aspects, along with business needs. As the norm ‘to each, his own’ goes, SRE is no different. It is easy to get inspired from pioneer companies, but it’s impossible to copy their culture and means to replicate the success, especially with your ‘anti-patterns’ and ‘traditional’ remedial baggage. Do you have similar infrastructure and business needs as the company showcasing brilliant success with SRE? No. Can it help you? Absolutely. The key factor here is to recognize what is important to your success blueprint after understanding the fundamentals of it and find your own success factors considering your cultural needs. Your strategy and culture need to walk together, just like your guiding (strategy) and driving (culture) factors.”

Maciej Jarosz
DevOps Institute Ambassador from Warsaw, Poland

“With all of those methodologies and frameworks out there, one can only ponder: ‘Which one should I choose?’ Well, all of those frameworks and methodologies provide something of value, as long as you have an opportunity to actually test some of those concepts in your own local environment. Consider SRE for example: It worked for Google, sure. If anyone doubts it there are at least three books on the topic with the materials, proof and philosophy behind SRE. Even if your company is not the scale of Google you still should take a look at best practices that worked in 2017, 2018, 2019 and that are still working now. Considering the crazy times that we live in right now and the plethora of various free materials, conferences, blog posts, research whitepapers and videos on the topic, what are we waiting for? The knowledge is right here, ripe for the taking. Advancing the humanity of DevOps is just a matter of experimentation, trust and continuous improvement. And with events such as the SKILup Day: Site Reliability Engineering where you can network, exchange field stories, brag about your ‘battle scars’ and participate in some splendid speeches, it’s only easier to jump into the fascinating world of SRE discipline.

Helen Beal
DevOps Institute Ambassador from Chichester, England

“I might be a bit biased since I’ve been focused on DevOps for nearly a decade as a consultant, coach and learning facilitator, but for me there are a number of systems of thought that contribute to the DevOps body of knowledge. I stole (with permission!) a phrase I heard from DevOps Institute’s CEO, Jayne Groll, many years ago that DevOps is the ‘harmonious, polygamous marriage of ITSM, lean and agile’ and I’ve also presented at several events over years around the correlations between DevOps and Holacracy. At an event in Munich a few years back, John Willis and I started a conversation about a DevOps Superpattern as he also added his ‘three-legged stool’ of The Theory of Constraints, Safety Culture and Learning Organizations’. For me, SRE is another valuable system of thought that we can learn from and use to make practical changes to the way teams and organizations work to drive higher organizational performance.”

Mark Peters
DevOps Institute Ambassador from Texas, USA

“I recently saw a social media image for a shirt which broke out SRE as ‘Simply Restart Everything.’ While not entirely true, the concept of a site reliability engineer is important as another way to link Ops and Dev in the DevOps construct. SREs formalize a position whose job is to connect, to link folks together, and to streamline processes. Increasing reliability means to increase simplicity across multiple workflows. The bigger your organization, the more simple the process needs to be. Unfortunately, many organizations increase complexity, reduce transparency, and create wasted effort as they add more individuals. SREs are important in trying to understand processes from end to end and emphasizing outcomes rather than any particular process stage.”

Sean Davis
DevOps Institute Ambassador from Georgia, USA

“Site reliability engineering bridges the gaps between platform design, development and operational execution by providing new perspectives on system reliability. When you link software engineering and systems engineering mindsets with operational engineering, you streamline the support of a product-focused on business outcomes not ticket output. It can be a valuable gambit for the traditional approach that pays off big for companies. By trading off specializations in focusing on operational burden, you gain a larger view of the platform and reduce the efforts required to support the product and spend more time automating and innovating to move towards self-healing and autoscaling products instead of manual, reactionary troubleshooting.”

MIke D. Kail
DevOps Institute Ambassador from California, USA

“Instead of the word reliability, I prefer to use and approach it as resiliency, which then includes approaches and contributing members of the chaos engineering group that are hyper-focused on continually increasing the ability to recover from any and all failures, which we all know happen in both software and the underlying hardware. Resiliency also should resonate with the core Development teams as they should strive to continuously improve and collaborate with the SRE teams.”

James Wasson
DevOps Institute Ambassador from Oklahoma, USA

“We create pipelines, version control, and infrastructure as code like breathing now. That means we are done, right? The beauty of SRE is that it takes us right to where the value is made. It requires us to go back to the beginning and ask questions about decisions we need to make regarding a system. Site reliability engineers help us remove “resulting” from our processes. With the help of these principles, we begin to make near-real-time decisions based on the impacts to our customers, rather than the results of the past. We spend so much time creating amazing features, determining the velocity of our teams, and implementing the newest framework from that great book, but one of the most helpful questions we get from site reliability engineering is ‘How fast is too fast?’ If our organizations can answer this question early, we are able to make more informed decisions that create extended and meaningful value to our customers.“

Biswajit Mohapatra
DevOps Institute Ambassador from Pune, India

“The ever-changing digital and cloud landscape brings in an unprecedented need for collaboration, transparency, resiliency, stability, performance, reliability and correctness. SRE sets measurable engineering goals mapping to service level objectives (SLOs) and enables monitoring and tracking of these quality of service (QoS) parameters. SRE is a set of practices focused on reducing silos by shared ownership, planning for failures using error budgets, small-batch changes with focus on stability, automation of manual tasks and introducing a culture of measurement, monitoring and tracking. The fundamental goal of SRE is aimed at depicting a prescriptive approach to plan, build, implement, measure and achieve DevOps objectives with a focus on reliability and automation at every opportunity. SRE covers several critical areas of cloud platform architecture such as orchestrated automation, responsive operation, optimized performance, just-in-time scaling, modernized environment and predictive event management. It’s time to look at operations through the lens of software engineering practices. Leveraging SRE to design, build, operate and enhance software systems is critical to building the culture of risk-managed and reliable digital footprint for enterprises.”

Mitesh Soni
DevOps Institute Ambassador from Mumbai, India

“What vs How
Interface vs Class
Abstract Class vs Class
DevOps vs SRE

Similarity and difference lie in their positions. Ryan Foland has given a wonderful example to describe What and How: ‘What’ begins at the surface level, and usually describes things in a general sense while ‘How’ describes the process of something or the underlying steps and actions that are done to get results. I find this example a perfect fit for the relationship between DevOps and site reliability engineering (SRE). Both DevOps and SRE have created buzz and confusion and will continue to do so. DevOps is cultural change to transform existing practices with people, processes, and tools for a better quality outcome with speed. SRE is a concrete implementation of DevOps practices or principles. I would like to leave one question here. Let me form the base first: Agile and DevOps complement each other. Agile is like a train compartment and DevOps is like an engine. Is SRE like a train track?”

Marc Hornbeek
DevOps Institute Ambassador from Puerto Vallarta, Mexico

“Site reliability engineering (SRE) compliments DevOps by measuring and achieving reliability of applications and services working on production and DevOps infrastructure in a prescribed manner using error budgets, team relationships brokered by an error budget, Ops-as-code, and the use of reliability to control deployments. SRE emphasizes the following objectives that are important to DevOps engineering:
- Assure availability of applications and services.
- Assure error rate and latency meet Service Level Agreements (SLAs).
- Enable large-scale systems while controlling risk.
- Create operational cost savings by automation and policies.
- Improve skills of developers and SREs.
- Resolve conflicts between Dev and SRE.
- Improve capacity planning and provisioning.”
Suma Puligella
DevOps Institute Ambassador from Virginia, USA

“SRE encompasses both software engineering and IT operations and bridges the gap between them. It ensures that the organization’s systems are scalable, stable, predictable and automated. SRE and DevOps seem like two sides of the same coin with a unified goal of enhancing the release cycle without any compromises. However, there are some differences between SRE and DevOps:
- SRE covers architectural and design patterns along with reliability implementation techniques such as reactive programming. Whereas DevOps is a set of practices that combines software development (Dev) and IT Operations (Ops). DevOps also aims at continuous software delivery with high quality, reliability and shared ownership among software stakeholders.
- DevOps raises the risks for the development team(s) to solve. Whereas SRE’s approach is to find problems and solve some of them themselves. SRE teams set service-level agreements (SLA) that define how reliable the systems need to be to end users. Their main goal is to improve performance and operational efficiency.
- DevOps is all about the “What” needs to be done, SRE talks about “How” this can be done.
The bottom line is that it’s all about the data. You need information in order to understand how to measure success and failure and how to gain continuous reliability across the applications.”

Tiffany Jachja
DevOps Institute Ambassador from Maryland, USA

“Organizations are finding the best practices for delivering faster and meeting customer demands. Site reliability engineering is about building confidence that our systems work as expected. SRE defines the methods, concerns, and responsibilities needed to maintain or increase the reliability of a system. It improves our ability not only to deliver a product to end customers but also to do so in a sustainable way as an organization. Many workloads can be costly, complicated, or time-consuming for different reasons. These practices provide a foundation for the root analysis of our systems, at the end of the day we want to be able to set key goals, measure systems, and ensure a product or service fulfills its objectives. This is important for anyone with a stake in software development and delivery.”

Uchit Vyas
DevOps Institute Ambassador from Singapore

“Site reliability engineering is a discipline that seeks to create reliable IT systems by managing the rate and quality of change, ensuring clients protect themselves from any break in service and avoid being subject to fines and penalties.

As per my experience, SRE practices are mainly based around the following elements:
- SLIs, SLOs and SLAs.
- Monitoring.
- Toil Definition.
- Error Budgets.
- Test Automation.
- Capacity planning and measuring risk factors.
SRE implements a lot of the same practices as DevOps, but from an operations perspective focuses predominantly on operational stability and resilience of the production estate.”

Zuzana Homolyova
DevOps Institute Ambassador from Slovakia

“SRE works as a kind of connection bridge between development and operations as it cares about every process from the source code to deployment. It really focuses on how to bring scalable and reliable software systems into the real infrastructure, way, application, performance and security. To describe the difference between DevOps and SRE I would say: ‘DevOps asks what needs to be done but SRE asks how it can be done.’

What is really important is that DevOps and SRE do overlap each other. They share goals to understand how to measure success or failure and how to gain continuous reliability across every application.”

Craig Pearson
DevOps Institute Ambassador from Newcastle, England

“I’ve walked the line between DevOps and SRE in organizations and witnessed first-hand both good and bad practices. While there are a lot of SRE success stories, I’d like to focus on the occasional use of SRE as a justification to rebuild the walls around operations/production that DevOps strived for so long to take down. While adopting SLOs and focusing effort on toil reduction are obviously beneficial, taking a defensive position over the production environment is not. The trigger for this is often an issue impacting production—sometimes major, sometimes minor—or the perception that more audit rigor is needed in this new continuous delivery world. Whatever the trigger, sometimes SRE is proposed as the answer (R stands for reliability, right?) SRE has to shift-left and spread more of the wisdom of production to teams delivering value-adding products and services to that production environment. Deploying value rapidly is crucial. Doing this in a reliable manner is key. Both are valid reasons for introducing site reliability engineering—the future. However, keeping the wisdom of production in a separate (silo?) SRE team and withholding access to production is a backward step. A touch of back to the future maybe …”

Nadeem Augustine
DevOps Institute Ambassador from Cape Town, South Africa

“Having been exposed to DevOps and SRE, I do see a difference between the two roles. I often find the DevOps way of working focuses on how to bring the issues into development and a way to process it between teams. This creates a collaborative way of working in the organization. That is a pivotal part in rolling out DevOps. DevOps is the ‘what‘ that needs to be done in order to bridge the gap between development. Having played both roles as a ‘developer’ and ‘operations,’ you now see how SRE plays a deeper role in assisting the organization in achieving the end goal. As an SRE you focus on the ‘how‘ to resolve issues end to end with the quickest most stable resolution. Having resolved the issues you create a more reliable and repeatable process.”

Daniel Oh
DevOps Institute Ambassador and CNCF Ambassador from Boston, USA

“Who would be the most valuable resource to stabilize the cloud platform built on an immutable infrastructure (i.e. Kubernetes) for the Ops team? Here is the evidence that many recruiters continue to seek the roles that are named site reliability engineers (SRE) but also more IT Ops folks want to have SRE responsibilities in their careers.“

Hugo Lourenço
DevOps Institute Ambassador from Lisbon, Portugal

“How long does it take to find the product you need? How long does it take to get the product delivered? Does the product/service require training or expert assistance? Do you need other calls +, products +, services +, to make this product / service work? Does the product require external maintenance and you will be accountable to remember the provider when it’s needed? To get support assistance or any other contact with the provider consumes your valuable time. Your time? All these reasons were the reasons why I embraced agile, lean and automation = DevOps, DevOps is the only weapon in a digital era. I’m passionate about building products and services where consumers will pay to keep your company to exist and deliver such amazing products/services!”

Felipe Duenas
DevOps Institute Ambassador from Quito, Ecuador

“SRE is more than buzzwords and trends. It is far more about the people and process than the technology that underlies them. While technology certainly can (and likely will) adapt to the concept as it matures and more teams adopt it, you don’t need new tools to align your development and operations organizations around the principles of Site Reliability Engineering. SRE ’embodies the philosophies of DevOps with a greater focus on measuring and achieving reliability through engineering and operations work.’ SREs also ensure that everyone in the company agrees on how to measure reliability, and what to do when availability falls out of specification. This includes contributors at every level, from developers, through team managers and all the way up to VPs and executives.”

Suresh GP
DevOps Institute Ambassador from Singapore

“Site reliability engineering is redefining the value of infrastructure operations and its relevance in the wake of DevOps toolchain and deployment pipeline. After a long time, we see that this area of discipline is creating a profound impact on the holistic value chain by building world-class resilience and reliability of products and services with an end-end view. It also focuses on creating a sense of shared ownership and experimentation in challenging the status quo and delivering streamlined customer experience. I see a lot of synergy with Agile, DevOps, Lean, ITSM with SRE that compliments the dimension brilliantly. Last but not the least, cultural change is of phenomenal importance to make this transition happen and be sustainable. The principles and practices are so sound that the legacy environments can still take advantage of it and contribute to velocity increase and improved reliability.”

BMK
DevOps Institute Ambassador and CNCF Ambassador from Wellington, New Zealand

“Enterprises who run or want to run a cloud application at scale, need to adapt to SRE practices and disciplines. Merely stating a 99.99% SLA does not help, if you care about your customers and your customers’ experiences with your systems then, yes, we need to level-up our game, we need to understand the nitty-gritty of reliability of our systems, operations and more importantly understand the limitations and constraints within which we need to operate. Some useful pointers: 1) Caution: Do not simply rename your ‘system admins’ to ‘SRE’ 2) Upskill your ops folks to code 3) Upskill your dev folks to learn and know more about infrastructure and operations.”

Aparna Balasundar
DevOps Institute Ambassador, India

“If DevOps is the extension of agile to build continuous delivery pipelines, SRE is a natural extension of DevOps to build resilient, repeatable, reliable, scalable production systems. Shift left to include the SRE as part of the architecture design, build, deploy favoring long term viable systems. With the cultural mind shift, if SREs have to provide safe and reliable systems, developers have to dedicate building such systems. These joint working partnerships will benefit the overall business goal.”

Simone Jo Moore
DevOps Institute Ambassador from Occitanie, France

“Site Reliability Engineering has as many touchpoints across the service value stream as any other practice. I challenge you to map your interactions across your enterprise so the impacts of SRE to the value outcome are clearly visible. Those companies that treat these touchpoints as part of their beating heart competencies versus peripheral consideration, will continue to leap forward. The socio aspect is important here. Technology is a catalyst—people’s interaction with the tech has shifted, is shifting and continuing to shift. Security is embedded; there is no point at which practices (human and automated) do not exist in SRE enterprise-wide. The resources we apply depends on how big and complicated the system is.

It goes beyond technical feedback to being an advantageous feedback loop for improvement across business objectives with engagement and financial benefit. Think of SRE as a movie, yes I’m a sci-fi fan but think about movies like “Mission to Mars” or “The Martian” and even “Ad Astra.” In each of these, reliability also includes our trust in something being resilient. Even when things don’t go as expected, there are ways in which the system can be adapted to solve problems. Resiliency is about humans rather than technology, but think about the progression of AI. We want AI to become resilient in not about coming back to a baseline but move forward to a new baseline of its own accord. Its point of stasis is never the same yet reliable based on the quality of the engineering. Our point of stasis is never the same either, and the cycle continues.”
June 30, 2020
OpenLegacy, Boomi Simplify Legacy Application Integration

OpenLegacy, a provider of a platform through which application programming interfaces (APIs) are layered on top of legacy application processes, announced today an alliance with Boomi, an arm of Dell Technologies that provides organizations with access to an integration platform-as-service (iPaaS) that is made available as a cloud service.

Zeev Avidan, chief product officer for OpenLegacy, said the company’s core platform enables IT teams to convert legacy application processes into microservices that can be involved by tools such as Boomi using an API. Rather than merely providing access to an application, OpenLegacy is designed to capture all the business logic, metadata and associated middleware associated with a specific business process executing on, for example, an IMS or AS/400 platform, he said.

The Boomi alliance extends that capability to include processes created by either citizen integrators or developers via the Boomi cloud service. That’s critical because most mission-critical applications in the enterprise still run on legacy platforms. Rather than having to rewrite those applications from scratch, Avidan said OpenLegacy makes those applications more easily accessible to other applications and platforms via a software development kit (SDK) based on Java.

At a time when many organizations are looking to accelerate digital business transformation initiatives, Avidan said being able to programmatically invoke an entire business process accelerates the rate at which those initiatives can be completed because business processes don’t necessarily have to be re-engineered.

In the case of Boomi, that capability is now being extended all the way to end users to make it feasible to tap into existing business processes using a set of tools that already have been adopted by more than 11,000 organizations.

OpenLegacy is making a case for DevOps teams to think in terms of legacy business processes rather than individual applications. Historically, many organizations have relied on enterprise service bus (ESB) platforms to integrate applications. However, a DevOps team still has to manually integrate all the software components that make up a business process once application environments are connected.

Legacy platform integration in the absence of OpenLegacy becomes the digital equivalent of an archaeological dig, said Avidan, noting many organizations wind up hiring expensive systems integrators to accomplish the task.

In the wake of the COVID-19 pandemic, there is clearly a lot more interest in modernizing business processes. Many employees are working from home for the foreseeable future. Cloud services such as Boomi provide a means of enabling end users to craft workflows on their own. The challenge has always been finding a way to easily connect those workflows to transaction processing systems and other systems of record around which the business depends.

It’s not clear to what degree providing that level of access will drive a burst of business process innovation. However, DevOps teams that spend an inordinate amount of their time integrating new applications with legacy business processes might be keen to leverage an approach the promises to turn those business processes into a reusable microservice.

June 29, 2020
But Wait, There’s More! The Trouble With DevOps Tools

DevOps tools today increasingly are no longer out-of-the-box ready for use. And that’s an aggravating issue

Imagine you bought a new power tool—a sweet battery-powered drill/screwdriver that did everything you needed.

Imagine it looked as though it ran on its own, but when you got it out of the box, you discovered that it didn’t come with a battery.

Imagine you got the battery, and only after the battery was installed you discovered that it needed a specialized wrench/chuck to install and remove bits. And you had to go buy that.

Imagine you got the chuck and were all set to go, only to discover that it would slip and normal drill bits wouldn’t turn unless you had a specialized adapter.

That’s what our tools environment in DevOps has become. And we need to demand better.

If your software requires X and Y and Z to be useful—whether you are commercial or open source, whether X and Y and Z are a million bucks or zero—state it before install.

We’re setting up an entirely new environment, stretching things a bit for an internal project, and honestly, the amount of “Oh yeah, this is required, too” that we’re finding out after install is insane. We have done our research; we should not be getting surprised by “But wait! There’s more!” type installs.

For an industry that insists on plug-and-play for all OSes and every odd bit of hardware ever dreamed up, we sure let software steamroll us. Vendors need to be clear about what will be required to run the tool and what will be required to run it effectively before we waste our time. Because there are some dependencies we’re not willing to accept, for usability reasons and/or security reasons. And if, under the covers, the tool does exactly the same thing as another tool … then what, exactly, is the value-add? It better be good.

We’re just one DevOps group—and a small one at that—but we have decided for this project not to use tools that dupe us into thinking they’re usable out of the box, then immediately require us to install something else. Or at least we’re trying. This has gotten so prevalent that in a couple of areas, we’re struggling. After all, the software still has to be built.

It’s a good thing drill manufacturers don’t think that way. Maybe there is hope.

Meanwhile, we’ll keep trying to do what you all are doing: kicking rear and keeping systems running.

June 29, 2020
How to Minimize Latency and Its Impact on UX
Latency can have an enormous impact on user experience. Here’s what you need to know to mitigate latency before it impacts your customer experience

Businesses have been digitizing for decades, but recent events have accelerated the process beyond all expectations. Customers are now using more digital services than ever, and their switching costs are often extremely low. That puts enormous pressure on businesses to deliver not just online services but also consistently high performance. Businesses that once measured performance in seconds now measure it down to the millisecond and even the microsecond. For that reason, they’re eager to minimize latency and its impact on their bottom line.

Latency can be defined as the delay before a transfer of data begins following an instruction for its transfer. However we define it, its effects are clear: A study by Amazon found that 100 milliseconds (ms) of latency cost 1% in sales. A similar study by Google found that a half-second (500 ms) lag in returning search results reduced traffic by 20%. A study by Loadstorm shows that latency can have similar effects on any business’s conversion rates. Long latencies lead to abandonment of shopping carts and frustrated users leaving your site to find the same information, product or service elsewhere.

The moral: As businesses try to convert users and move them through their pipeline, latency can be the greatest enemy. But how do you measure it, and what can you do about it?

Measuring Latency

Given the impact of latency on the user experience, it’s surprising how many businesses don’t give it enough attention—or even measure it properly. A common mistake is to focus on average performance, which is ultimately a theoretical measurement that might not reflect actual end user experience.

A better, more meaningful, way to measure real-world latency is with percentiles.

Latency is typically calculated in the 50th, 90th, 95th, and 99th percentiles, commonly referred to as p50, p90, p95, and p99. Imagine 10 latency measurements: 1, 2, 5, 5, 18, 25, 33, 36, 122, and 1000 milliseconds (ms). A p50 measurement represents the median performance of the system. In this case, the p50 measurement is 18 ms, meaning 50% of users experienced that latency or less. The p90 measurement is 122, meaning that 9 of the 10 latencies measured less than 122.

Each measurement within a given percentile reflects a real latency that actually impacted the experience of at least one, but likely many more end users.

By contrast, the mean of our 10 measurements comes to roughly 125 ms, not even close to the midpoint in the observed data–in fact, slightly above the 90th percentile.

It is vital to keep in mind the outliers, what are called “long-tail latencies.” Under real-world conditions, p99 (and higher) latencies are often the ones you need to worry about most.

Gil Tene, the CTO of Azul Systems, addresses this problem in his presentation, “How NOT to Measure Latency.” He notes that p99 latencies can easily impact a far larger than expected proportion of users. If a typical user session involves five page-loads, averaging 40 resources per page, about 18% of users will experience at least one response longer than even the p99.9. He also points out that the average response percentile is experienced by more than 95% of users will be the p99.97 latency.

As Google engineer Luiz André Barroso put it:

“When a request is implemented by work done in parallel, as is common with today’s service-oriented systems, the overall response time is dominated by the long tail distribution of the parallel operations. Every response must have a consistent and low latency or the overall operation response time will be tragically slow.”

In other words, the slowest operation in a set of parallel operations usually defines the user experience. It doesn’t matter that 99% of the operations finish quickly if the other 1% holds everything up. This explains why it’s not enough to keep p95 latencies blazing fast. You have to keep long-tail latencies (p99 and up) consistently low as well.

Causes of Long-tail Latency

To fix the problem of long-tail latency, we need to understand why it occurs. Long-tail latency has many causes, but, significantly, it is not usually caused by application-specific problems or normal network lag. Instead, long-tail latency has systemic causes, usually related to problems in the underlying infrastructure. These can include pauses due to Java Virtual Machine (JVM) “garbage collection,” context switches, database repair, cache flushes and so on. This makes long-tail latency very tricky to diagnose and fix, as it’s often a “whack-a-mole” exercise.

Disk I/O and Network Bottlenecks

The main causes of long-tail latency are network and system bottlenecks. IT commonly invests heavily in the best components available: great solid-state drives (SSDs, and associated NVMes), CPUs and memory. But it often shoots itself in the foot by configuring a “narrow” network that bottlenecks all that highly performant hardware.

SSD settings are also commonly misconfigured. Cloud SSDs have two associated throttles: burst and sustained. Background operations such as streaming and compaction can chew up the sustained throughput, which may be 1/10 of burst. While the backend process hogs I/O, there is no capacity left for frontend, user-induced operations. While some transactions will be able to use the burst capacity, the steady-state frontend traffic will start backlogging and eventually time out.

Speed Mismatch

Database performance is strongly influenced by the combined speed of CPU and memory on the one hand and disk speed on the other. CPU tends to be faster, while disk tends to be slower. SSDs are catching up with CPUs, but the rule still generally holds.

Speed mismatch is a more common problem with write workloads bounded by disk speed. This can happen either because the disk is slow or because the payloads are large. Large payloads shift the bottleneck to the disk. When the time comes to write to disk, any relative disk slowness will cause queries to back up in the buffer. This will result in outlier p99 latencies.

Exceeding Your Latency Budget

Every system is designed with a “latency” budget, which is the ratio of latency to throughput. p95 and p99 latency are a function of the throughput a given system is designed to support. You can think of this ratio in terms of highway traffic and throughput as the number of lanes on the highway. More lanes mean that the highway can support more vehicles, and hence more trips. That’s throughput. Latency is defined by the speed limit. Having many lanes combined with a slow top-speed reduces the overall number of trips. The ratio between the two defines the latency budget.

Ideally, if you just add more throughput you should be able to reduce latencies, right? Well, not necessarily. Just like traffic on freeways can expand to fill an extra lane, still leaving you stuck in a traffic jam, you can easily overwhelm systems that weren’t designed well to begin with, even if you try to add more throughput. In fact, trying to add more throughput can degrade all of your latencies. In the routing world, that sort of poor architectural consideration is called bufferbloat.

If a system is trying to add too much throughput while other bottlenecks still exist in a system, then you only end up with longer latencies, even to the point of transactions timing out. That can exacerbate a bad situation since all failed transactions will attempt one or more retries.

Systems that handle big data have several considerations. These include the most obvious hardware-oriented factors; storage (the reason why NVMe SSDs are widely prevalent these days, for example), CPU cores, memory (RAM) and network interfaces.

Modern servers running on Intel Xeon Scalable Processor Platinum chips are common across AWS, Azure and Google Cloud. Server needs are based upon how much throughput each of these multicore chips can support.

Additional considerations get into how your database system was designed to handle data distribution and replication for high availability, peer-to-peer versus leader/follower designs, how it minimizes hops and redirects between servers, effective use of caching, how it efficiently it utilizes all the hardware available to it and so on.

These are all components that define a system’s latency budget. It’s important to understand and stay within those limitations. When they are exceeded, outlier latencies are virtually assured to occur.

Latency and the Database

So far we’ve treated long-tail latency as a systemwide phenomenon. Now let’s consider the role of the database. Most HTTP and API requests result in at least one, if not more, database calls, often a mix of reads and writes. Hence, the database plays a central role in the end user experience of latency.

NoSQL databases were invented specifically to address the need for low-latency, globally distributed data stores. The basic premise of the NoSQL database is the ability to “scale out” on cheap, commodity hardware. When more capacity is needed, administrators simply add servers to database clusters. This simplicity, however, has morphed into a vulnerability. Clusters often grow out of control, resulting in a phenomenon called “node sprawl.” The architecture of the first-gen NoSQL databases actively encourages larger clusters of less powerful machines. From the perspective of latency, this architecture virtually ensures inconsistent p99 performance due to more internode traffic, context switches, disk failures and network hiccups, all of which create latency outliers.

One database that is particularly susceptible to this problem is Apache Cassandra. First, Cassandra is implemented in Java, making it vulnerable to pauses caused by garbage collection (GC). Some monitoring teams even have a dedicated metric devoted to GC stall percentage. Compaction and repair operations, which become more onerous in larger clusters, add to the outliers that plague Cassandra deployments.

Second, Cassandra has limited ability to utilize modern hardware for compaction and streaming. Since Cassandra is an “append-only’ system, keeping data coherent for reads requires a compaction process. Servers with a large number of cores are readily available on public clouds, IaaS and for on-prem deployment. However, Cassandra can’t scale compaction linearly with the increased number of available cores, so it cannot effectively utilize those larger servers. This limitation results in higher read latency due to the need to deploy many more smaller servers in the clusters.

Since Cassandra exposes settings for storage media, JVM heap cache and concurrency on read and writes, the burden is placed on administrators to properly configure the system for specific workloads. When workloads are unpredictable and spiky, administrators must constantly tune the Cassandra cluster to optimize read and write latencies.

Many IT organizations turn to external caches as an aid to insulate end users from the poor long-tail performance of their databases. Caches are often used to enhance read performance but write latencies derive little or no benefit. Transactional systems, which depend on real-time performance, cannot drop mutations, so caches offer no help. Transactional systems depend on low long-tail latency for writes as well as reads. If spikes are too frequent, the client experience deteriorates, becoming very sluggish.

Still, front-end cache for reads is often a necessary evil, to the point where it is treated as obligatory. However, maintaining coherence between cache and persistent storage is a well-documented hassle that can easily bog down operations and negatively impact customers. The best approach is to identify a system that intrinsically combines cache and persistent storage in a way that optimizes performance and data integrity.

Winning the Battle for Consistently Low Long-Tail Latency

With these issues in mind, there are a number of guidelines your organization can follow to minimize long-tail latency.
- Design a system that ensures the lowest p99 latency based on your organization’s current and future throughput targets.
- Don’t ignore performance outliers. Embrace them and use them to optimize system performance.
- Reduce the number of components in your architecture, from redundant cache systems to bloated hardware infrastructure. Shrink database clusters by scaling vertically on more powerful hardware.
- Adopt database infrastructure that scales linearly with your resources: CPU, memory, storage, network and number of nodes in the cluster.
- When using database-as-a-service (DBaaS) offerings, look out for “scaling bloopers”—points at which you can’t scale linearly and have to increase spending beyond the marginal benefit you get from the system.
- Make sure your DBaaS vendor can offer multi-region deployments, so you can utilize the offering near your application, eliminating long-haul data traffic. While this will not reduce latency within a database, it will reduce end-to-end database latency between the database and the client application.
Latency often gets short-shrift in IT calculations because it’s so poorly understood and so badly measured. But you can’t ignore it. It does its damage whether it’s recognized or not (and the less you recognize it, the worse damage it does). Luckily, as we’ve seen, there are good ways to measure latency and squash it down to size. With the right metrics and an informed strategy, businesses can turn what used to be a fearsome adversary into a customer-experience advantage.
June 26, 2020
Survey Finds SRE Role Remains Largely Aspirational

Monitoring tools provider Catchpoint, in collaboration with the DevOps Institute, has published a survey that finds there is a significant gap between the tasks and duties that define the role of what a site reliability engineer (SRE) should be versus what is actually occurring within IT organizations that have hired SREs.

Based on a survey of 600 IT professionals who have SRE titles or individuals that identify themselves as performing SRE-like tasks, the “Catchpoint SRE 2020 Report” finds the bulk of respondents are still squarely focused on managing IT infrastructure.

Google, which is widely credited with defining the SRE role, maintains SREs should be spending as much time managing applications as they do infrastructure. Survey respondents on average are spending 75% of their time on operations. Only 14% of respondents said they are spending 50% or more of their time on application development processes. More than have of respondents (53%) also noted they are becoming involved too late in the application life cycle.

The Catchpoint survey finds the top three tasks survey respondents associated with their roles were monitoring and alerting (93%), dashboarding (73%) and managing infrastructure as code (71%).

Catchpoint CEO Mehdi Daoudi said one of the most telling aspects of the survey is only 53% of respondents identify observability as a task associated with their role, even though that’s a core tenet for best DevOps practices. Further down the list of DevOps tasks were testing (41%), telemetry (38%) and chaos engineering (26%).

Daoudi said it’s probable many IT organizations are adding SREs in anticipation of furthering their DevOps ambitions. Not every SRE has all the skills required, but as more of them gain hands-on experience it’s likely many will grow into those roles. In fact, 41% of respondents said half or more of their work revolves around mostly manual, repetitive tasks that could be automated. More than half of respondents (52%) said they spend too much time on debugging-related tasks.

Part of the challenges many IT organizations face derives from not putting enough focus on automating the management of legacy IT infrastructure, said Daoudi. Every time a new platform is adopted there’s a rush to operationalize it. However, IT teams frequently overlook the need to automate the management of rote tasks to free up more time to learn how to manage new platforms such as Kubernetes.

Going forward, the survey finds nearly 50% of respondents said they believe they will be working remotely post COVID-19. More than half of respondents also said personal challenges included staying focused and having a good work/life balance while working from home.

The degree to which SRE titles are being adopted to enable IT professionals to justify a raise or simply pad their resumes is unknown. However, the focus on the SRE role is likely to increase in the wake of the economic downturn brought on by the COVID-19 pandemic. In theory, a single SRE should be able to manage all the tasks that previously might have required as many as 10 IT administrators. At a time when many organizations are looking to reduce the total cost of IT, the number of full-time employees required to manage IT environments has clearly become a major concern.

June 25, 2020
Puppet Launches Relay to Simplify Automating DevOps Workflows

Puppet today made available a public beta of Relay, an event-driven platform through which DevOps teams can automatically invoke pre-configured workflows.

Company CTO Deepak Giridharagopal said Relay listens for DevOps events that can then trigger an automated workflow such as automatically deleting underutilized cloud resources, enforcing cybersecurity controls or remediating known issues.

Relay connects to dozens of DevOps tools, cloud services and application programming interfaces (APIs) to automate workflows across a wide range of platforms, including PagerDuty, GitHub, Datadog, Jira, Terraform, Slack and others.

Giridharagopal said Relay will also enable IT organizations to reuse workflows versus continually cobbling together one-off workflows for processes that address routine use cases that can be tackled using existing content.

At the root of that issue is tool sprawl within DevOps environments. Each DevOps team tends to create their own workflows using their preferred tools, even though they may be replicating a workflow process that has already been addressed using other tools. Relay provides a means to reuse workflow across disparate DevOps tools, which Giridharagopal said should significantly increase overall DevOps productivity.

The Relay platform is also designed to be extensible in that it also allows DevOps teams to create custom workflows that can then be reused by other members of DevOps teams, he added.

Giridharagopal said with the rise of microservices it’s apparent workflow automation within DevOps environments is about to become a more pressing issue. As the number of individual services that need to be managed increases, the need to automate workflows across more complex IT environments becomes more critical. DevOps teams can’t effectively manage modern cloud-native applications relying on manual workflow processes, he said.

Relay is part of a broader shift toward event-driven architectures, which are gaining increased traction within IT environments. While event-driven architectures have been around as a concept for decades, the emergence of platforms based on, for example, serverless computing frameworks are changing how IT resources are invoked. Via any number of APIs, it’s feasible to automate a wide range of IT functions using these types of platforms.

Of course, the biggest challenge for many organizations has been building automated workflows that work consistently across multiple platforms. The level of process expertise required to build those workflows is not inconsequential. Puppet, via Relay, is providing pre-configured workflows to address some of the more common use cases in a DevOps environment. DevOps teams will still need to build their own custom workflows to address more complex use cases. However, platforms such as Relay promise to sharply reduce the time spent creating scripts and associated workflows to address routine tasks.

In the meantime, interest in all forms of IT automation is on the rise in the wake of the economic downturn brought on by the COVID-19 pandemic. IT organizations are moving quickly to reduce the total cost of IT by automating as many processes as possible. In many cases, it’s not clear when some IT organizations might be able to add additional headcount anytime soon. Automation, as such, is now as much about survival as it is simply trying to increase productivity.

June 25, 2020
3 Reasons Why Traceability Should Be a DevOps Priority

Traceability in DevOps is about ensuring clarity, accountability and the best possible end product for customers

There’s a lot to think about when it comes to mobile DevOps. For a start, there’s the need for effective cooperation between members of different teams, perhaps even via team collaboration software for remote workers. With so many people working on DevOps, it’s easy for some to become siloed from their colleagues, hampering communication and undermining teamwork.

Then there’s the wider question of DevOps strategy—deciding on a vision, setting goals and working out how to measure your success (or otherwise, as the case may be). One thing that has to be central to your strategy is traceability. You may have come across the term a few times before. It’s commonly used elsewhere in the business world, especially with regard to supply chains. Basically, what it means is keeping track of a commodity or product at every stage of the production process.

Records of the product’s entire manufacturing and distribution history are kept so that the sources of any problems can later be determined and dealt with. Traceability thereby ensures that suppliers can act quickly and decisively in the event of a product recall, for example.

Another advantage of traceability is that it provides additional transparency, which helps to maintain consumer confidence. As consumers are becoming increasingly aware of how products are sourced and manufactured, this is now an important consideration. It reassures consumers that manufacturers and suppliers are aware of their concerns and that they’re looking out for their best interests.

You can see already, then, how much of this also applies to mobile DevOps. Traceability in DevOps is about ensuring clarity, accountability and the best possible end product for the consumer. But it’s especially important in DevOps because speed is of the essence—there’s constant pressure to get apps developed, into the hands of users and then incorporate feedback as soon as possible. It goes without saying that this can lead to oversights and slip-ups that might otherwise be avoided.

In addition, DevOps teams have sought to accelerate this process by allowing for greater decentralization and autonomy among their teams. The logic behind this is that it allows for quicker delivery and deployment. Screenshare solutions and other such technological fixes can help to facilitate closer collaboration. However, this easing of control can nevertheless create some potential for confusion in the development process. That may make it more difficult to maintain consistent coordination between developers and testers, and so on.

Here we’ll list three of the key reasons why traceability should be a leading DevOps priority. All are based on the experiences of development teams and the challenges they’ve encountered.

Cutting Down on Defects

As we’ve already discussed, the pressures for faster software rollout and feedback can make it more likely for certain bugs to not be addressed as quickly or as effectively as might otherwise be the case. We’ve noted further that buggy apps can do a lot of damage to companies’ reputations by providing a shoddy and unreliable user experience.

This can have serious repercussions—after all, word travels fast, especially on social media. It’s therefore in your interests to take whatever steps you can to make sure your software runs as smoothly and reliably as possible.

Experience strongly suggests that the more rigorous your traceability processes are, the less prone to faults your apps will be. The completeness of your captured traceability information makes all the difference here.

So, for example, EMR software developers will get great value out of more complete traceability info, as it will contribute to reducing the likely defect rate. This correlation between detailed traceability and defect rate can help to produce more reliable apps and improve efficiency by cutting out avoidable errors and bugs. You should actively solicit feedback from users where appropriate, and integrate this into your wider customer support package.

Ensuring Compliance

In other industries, thorough traceability is often a requirement set by regulatory bodies. This can be so that, in the event of a product needing to be recalled, the entire supply chain can be re-examined and the cause of the problem discovered. For example, food and feed businesses operating in the EU have been subject to compulsory traceability requirements since the General Food Law came into force in 2002. Similarly in DevOps, developers have to take regulatory compliance into account.

Thus, robust and detailed traceability helps DevOps firms working across a variety of sectors in their efforts to comply. There are numerous industries where this is relevant, whether it be financial services (a well-known regulatory minefield), HR software, insurance or public sector contractors and healthcare firms, any of which may be subject to stringent traceability requirements.

Tracing code is a common prerequisite insisted upon as part of compliance frameworks. For example, you won’t be able to complete the SOC 2 process integrity requirement unless you’re able to provide a complete, evidenced account for each code alteration in your application’s source code.

Improving Speed Without Sacrificing Quality

We’ve already discussed how there’s a delicate balancing act that needs to be struck in DevOps. That’s the balance between software rollout speed and the reliability of its performance. We’ve also noted how the increasing autonomy of different sections can hamper coordination between them; thus providing some benefits in terms of speed and rollout but often at the cost of the reliability and quality of the end product. The key here, then, is to find a way of threading the needle, ensuring that apps can be produced at a consistently quick rate while maintaining rigorous standards.

This is where traceability comes in. It helps to not just enhance speed but also drive up standards. It allows developers to execute tasks more quickly while simultaneously improving the standard to which they do so.

This also has a positive effect on overall efficiency levels, freeing up resources (i.e. members of the team) to take on other tasks for which they might be needed. Engineers working on contact center software, for instance, can benefit from better-maintained code by reusing or altering existing code depending on their needs.

It’s worth noting again here that requirement-to-code traceability may be mandatory from a regulatory standpoint. It will depend on the industry in which you’re working. But even if it isn’t, it’s something that’s still worth doing. It can lead to dramatic improvements across the board, leaving as its legacy sharpened and more efficient processes that enhance your team performance, benefits clients and consumers, and—most importantly—deliver real gains for your bottom line.

Having taken all this into consideration, then, you’ll need to think about exactly how to implement DevOps traceability. Certainly, when introducing traceability processes, you must do so with real care and consideration. Being too rash about implementation can either create bottlenecks, thus exacerbating or even causing problems which traceability otherwise ought to help resolve, and slowing down the overall speed of production.

Make sure, then, that traceability is implemented in a way that takes the needs of developers seriously. To start with, draw up a guideline document offering developers a clear and concise explanation of what’s expected of them. Outline the traceability process and how it ties in with key objectives. Furthermore, give developers some autonomy in the choice of tools they use for traceability purposes. You might, for instance, make astute use of automation (such as automatic code checks, for example). This should help everyone reap the rewards of traceability and avoid any unnecessary implementation hiccups.

June 25, 2020
JFrog Extends Reach and Scope of CI/CD Platform

At its online JFrog swampUP Conference today, JFrog announced it has extended the reach of its ability to distribute software to include options based on peer-to-peer and content delivery network (CDN) mechanisms.

In addition, JFrog expanded the reach of the JFrog Pipelines continuous integration/continuous delivery (CI/CD) platform, enabling DevOps teams to build their own library of reusable pipeline steps via an Extensions Development Kit (EDK). DevOps teams can encapsulate frequent and common actions in their pipelines that can run sequentially or in parallel. That approach also serves to reduce the need to rely on plugin modules to extend a JFrog delivery pipeline.

Finally, JFrog has added another free public artifact center that provides a means to manage Helm charts, which are widely employed to deploy applications on Kubernetes clusters. ChartCenter is intended to make it easier to deploy microservices based on containers.

JFrog CEO Shlomi Ben Haim said all three additions to the JFrog portfolio extends the ability for users to manage the entire DevOps workflow from integration to delivery regardless of what type of software is being deployed or where it is installed.

As DevOps teams look to deploy software on edge computing platforms within an internet of things (IoT) environment, Ben Haim said they will need to be able to employ either a peer-to-peer or CDN-based approach to manage that process efficiently. Existing mechanisms are not designed to handle large volumes of artifacts that need to be delivered in a specific order. The new mechanism added to the JFrog CI/CD platform enables DevOps teams to achieve that goal without having to acquire a separate platform, he said.

In fact, Ben Haim noted, efficiently delivering software to edge computing platforms represents the next great leap forward for DevOps. Achieving that goal, however, will only be practical if the entire CI/CD workflow process keeps track of all the dependencies and metadata end-to-end, he said, noting that approach will enable endpoints to pull only the delta code required to update the software after an application is initially deployed.

There is no doubt extending best DevOps practices to edge computing platforms will be a major challenge. Most DevOps teams already struggle with CD because each platform software is being deployed on is unique. Because of a general lack of standardization in terms of potential endpoints that might be deployed at the edge, that “snowflake” issue is likely to become even more exacerbated as IT environments become more distributed than ever.

In general, Ben Haim said far too much of the DevOps focus continues to be on source code management when in reality the biggest issue IT organizations face is how to efficiently manage binaries. At a time when organizations are making massive investments in digital business transformation, Ben Haim noted the rate at which software is deployed and updated is about to accelerate dramatically.

As such, Ben Haim said it’s now only a matter of time before DevOps workflow processes will need to be able to scale well beyond their current limits.

June 24, 2020