devopsy.kinsta.cloud

Category: DevOps in the Cloud

Accelerate State of DevOps Report: The Move to Multi-Cloud

Analysis of the 2021 Accelerate State of DevOps Report (SODR) data shows an increasing number of organizations are choosing multi-cloud and hybrid cloud solutions. In our survey, respondents were asked where their primary service or application was hosted, and the results showed that public cloud use is on the rise.

Of the 1,200 survey respondents this year, 55% indicated they’re using a public cloud, a 5% increase from 2019. Of those approximately 660 people, 21% indicate they use multiple cloud providers and 34% indicate using a hybrid cloud strategy. For clarification, multi-cloud refers to the use of two or more public cloud providers and hybrid cloud refers to the use of an on-premises data center and/or private cloud in combination with one or more public clouds.

Our data indicated that the No. 1 reason for using multiple providers is to leverage the unique benefits that each provider brings to the market. Every cloud provider provides basic compute and storage services, but building a best-in-class business can be best achieved by taking advantage of best-in-class cloud computing features and not being locked into a specific vendor. In fact, 26% of our respondents indicated this was their primary reason for their multi-cloud or hybrid cloud approach.

The second most-reported reason, at 22%, was availability. Companies around the world are increasingly identifying as software companies and relying on software to be their primary business driver. Software allows companies to deliver value to their customers, outperform their competition, respond to market events and bring innovation to the market. Availability of this core product is not a “nice to have”, it is essential. The truth is, however, that computers and networks are fragile—they fail. If we accept this fact, we can design and build systems that are prepared for this eventuality. Companies are addressing these risks with strategies that decrease the blast radius of changes and allow systems to continue operating even in the face of failure.

Speaking of worst-case scenarios, disaster recovery (DR) is the third most-reported reason for adopting multiple cloud providers. In fact, the cloud and the global proliferation of data centers has allowed companies to rethink what disaster recovery actually means. Gone are the days of systems going down and operators scrambling to rebuild systems and restore from tape backups—the cloud is the backup. The adoption of cloud plus fully automated pipelines creates a paradigm shift around DR that allows companies to design their disaster recovery strategy around zero downtime.

In addition to hybrid and multi-cloud strategies increasing, we also saw a rise in the use of the five essential characteristics of cloud computing. The National Institute of Standards and Technology (NIST) defines the five essential characteristics as:

These five characteristics must be embraced and used to realize the value of the cloud. A company that adopts cloud but does not employ these characteristics simply shifts their operating environment from one place to another but doesn’t gain anything. In fact, the unfamiliarity of a new environment, new pricing models and lack of optimization from years of running on-premises often leads to disappointment. Successful companies embrace these five characteristics.

We know from previous reports that these characteristics are essential for delivering software at a high or elite performance level. We see adoption rates between 73% and 78% across the five characteristics with the lowest being on-demand self-service (73%) and the highest being measured service (78%). There is no doubt that this increased adoption is a key driver for the jaw-dropping performance we saw this year from our elite performers.

Elite performers can now deploy 973x times more frequently, have a 6,570x faster lead time to deploy, have a 3x lower change failure rate and a whopping 6,570x faster time to recover from incidents when failure does happen. Go back and read that again—no, those are not typos nor an error in the data! An elite-performing company can bring a new feature to market 6,570x faster than a low-performing company. This is one of the reasons we see a direct link between software delivery performance and overall business performance.

We encourage you to read the entire 2021 Accelerate State of DevOps report. There’s lots of great information in the report including the addition of a fifth key metric—that’s right; the ‘Four Keys’ are now the ‘Five Keys.’ “What is the new metric?” you ask? Well, I guess you will have to read the report and find out. 🙂

September 21, 2021
JFrog Acquires Upswift to Push DevOps to the Edge

JFrog today revealed it has acquired Upswift as part of an effort to extend its DevOps reach to edge computing platforms. Terms of the acquisition were not disclosed.

Upswift has created agent software that is deployed as a binary file on an edge computing platform that makes it possible to continuously deliver software across a fleet of edge computing devices and platforms. That approach makes it possible for organizations to deploy software regardless of whether the target platform is only running Linux, containers or a virtual machine.

JFrog CEO Shlomi Ben Haim said the agent software developed by Upswift will be integrated with the continuous integration/continuous delivery (CI/CD) platform created by JFrog to make it easier for organizations to manage the full life cycle of application workloads deployed on edge computing platforms.

Organizations that deploy software on edge computing platforms and devices today typically have massive backlogs of updates that they are unable to deploy in a timely manner, said Ben Haim. By connecting the Upswift agent software to a (CI/CD) platform designed from the ground up to support binaries, it becomes possible to reduce that backlog as part of what JFrog describes as its ‘liquid vision’ for deploying software, he noted.

In addition, after acquiring Vdoo to gain access to a set of analytics tools that discover vulnerabilities in application binaries, JFrog will also be able to bring DevSecOps best practices to edge computing applications as well, noted Ben Haim.

The number of application workloads being deployed on edge computing platforms is rising sharply as organizations seek to process and analyze data closer to the point where it is being created and consumed. In fact, that shift is at the core of digital business transformation initiatives that depend on edge computing platforms and devices running, for example, internet of things (IoT) applications to process data in near-real-time.

Upswift CEO Amit Ezer said it has already become apparent that the existing processes of updating software on edge computing platforms and devices are capable of handling that influx of software. Upswift created its tool to enable developers to assume more control over those processes, as responsibility for managing and securing software continues to shift left, he added. The issue in many cases is that organizations lack the ability to consistently deploy software on edge computing platforms, which results in a hodgepodge of homegrown tools being created to deploy software on edge computing platforms using largely manual processes.

It’s too early to say just how much software will be deployed on the edge, but there may come a day when there is more software at the edge than there is in the cloud. Of course, the real challenge will be unifying the management of DevOps processes, regardless of where any given piece of software might be deployed. In fact, as edge computing continues to evolve, the line between edge computing and cloud computing will continue to blur until they are little more than natural extensions of each other.

September 13, 2021
Avoiding Budget Overruns During Cloud Migration

Many organizations have already started their migrations to the cloud, or soon plan to. It makes sense: The cloud is an attractive option for multiple reasons, including its potential for cost savings, scalability and support of continuous application development.

However, benefits aside, it can be very easy for enterprises to lose financial control of a cloud migration. The potential for costly mistakes when shifting to the cloud is significant, and organizations often end up getting hit with surprising bills that ultimately stop them in their tracks. In fact, in a 2020 survey, 94% of participants claim that it is a challenge for them to take full advantage of public cloud resources, with 40% citing the need to control cloud costs as their biggest obstacle.

These financial mistakes are typically made due to the pay-per-use model of cloud services and the growing cloud skills gap. Too often, IT team members are forced to work in cloud environments they are not proficient in, and as new features are released every week, it’s almost impossible for these individuals to keep up. One recent study claimed that 90% of enterprises found that their cloud skills gap doubled over a three-year period.

The Two Different Types of Cloud Migrations

There are two types of cloud migrations that typically occur, a lift and shift, or a replatform/rearchitecture migration. The lift and shift method is fast and relatively easy, but it doesn’t enable organizations to truly optimize their cloud usage. The rearchitecture method is the most popular as it gives an organization the opportunity to fix any outstanding issues by harnessing the benefits of cloud-native technologies and services.

However, once a rearchitecting migration begins, organizations often find it takes significantly more time, effort and resources to complete than they originally thought. This can become a real challenge, especially when facing critical deadlines or experiencing major events like the COVID-19 pandemic or the wild weather events experienced around the globe.

Understanding and Assessing Performance

For both types of cloud migrations, understanding and assessing performance and underlying architecture is an ongoing task, as cloud providers are always releasing new or improved services or lowering prices. To make sure they’re achieving the best performance and building a resilient architecture, application developers and IT teams must always be looking for ways to improve their applications. Within the cloud, this is typically focused on four key areas: Networking, computing, storage and application architecture.

From a network perspective, understanding where data is being used and where data is being accessed is critical and can help organizations cut costs exponentially. For example, cloud providers often have costly data ingress and egress charges, so assessing how best to access and use data when necessary can help a company navigate those charges.

Regarding computing, there are multiple resources that can help manage system load, make tweaks to computing infrastructure and find performance and cost benefits. For instance, leveraging the data and metrics from native cloud reporting tools can help organizations understand and predict demand, and taking advantage of services like auto-scaling enables them to scale up or down appropriately depending on the changes in application use and demand.

On the storage front, cloud providers offer a number of ways to save money and boost performance. From the release of GP3 volumes to tiered long-term storage options based on the frequency of access, storage is an easy way to find cloud savings opportunities. Additionally, cloud providers will usually make recommendations and set up tiering policies to help move data automatically into colder and deeper storage.

As cloud providers continue adding new services, looking at your application architecture is essential to innovation and cost savings—especially when including labor and effort—and truly achieving the promise of the cloud.

Solving Common Costly Mistakes

While there are numerous examples of common financial mistakes companies make when managing cloud instances, lack of visibility and understanding are the leading pain points for the developers who stand up and maintain their organization’s architecture. With the demand for cloud so high and with their attention directed elsewhere, developers and engineers often take the mentality of “If it isn’t broke, don’t fix it.” This hampers the company’s ability to assess and optimize their cloud operations. They don’t have time to consistently review their applications for ways to save money or even know how much their decisions cost.

This ubiquitous issue can be solved by bringing greater visibility into the cloud migration and management process and aligning real-time finances directly with established corporate budgets. In addition, organizations struggling with the cloud services’ pay-per-use model should take the necessary steps—whether internally or with a third-party vendor—to ensure cloud resource and service use is being effectively monitored. This makes it easier for developers to understand where and how their actions affect the financial posture of the organization before their boss comes knocking on the door.

If enterprises provide the resources necessary for IT teams to become more knowledgeable about how their applications are being used within the cloud and address the cloud skills gap in their workforce, they will be able to prevent cost overruns, experience cost savings and take back control of their budgets. This can take a variety of forms, but enterprises need to establish continual training programs to ensure team members are up-to-date on the latest cloud features and services, as well as leverage tools and automation to reduce manual intervention. Providing this critical visibility throughout a cloud migration can often mean the difference between a successful, on-budget migration and one that is stalled and costly.

September 13, 2021
4 Ways to Simplify Cloud Performance Monitoring

Over the past 18 months, technologists across the globe have played a crucial role in the survival of brands and organizations via digital strategies. Across all industries, their skill and dedication have enabled businesses to rapidly launch innovative new digital services to meet huge fluctuations in customer needs and allow entire workforces to operate from home during the pandemic.

This hasn’t been easy, even as end users’ expectations for seamless digital experiences are on the rise. In fact, the recent 2021 App Attention Index reaffirmed found 76% of consumers’ expectations of digital services have increased in the last year. And an overwhelming 85% noted that applications and digital services have become a critical part of their daily lives. This has placed even greater pressure on the technologists who are responsible for keeping applications and cloud experiences running smoothly.

Often IT teams are challenged with monitoring companies’ digital services, but they lack the visibility they need to monitor health and performance across an increasingly sprawling IT estate, exacerbated by a seismic shift toward cloud architectures over the last 18 months. IT departments don’t yet have the right tools and data to optimize health and performance in their cloud environments.

As a result, technologists are constantly addressing the same issues because they can’t easily isolate and identify them, and don’t know where to prioritize their actions. Hiccups can lead to brands losing customers—the same report revealed that over half of consumers indicated that brands have one shot to impress them, and that if their digital service does not perform, they won’t use them again.

After a year and a half since the start of the pandemic and with adoption of cloud technologies set to accelerate even further over the next few years, it’s time for technologists to get ahead. Here are four things technologists can do to take the stress out of managing performance across the cloud.

Implement Visibility from the Ground Up

Having visibility into applications and the underlying cloud infrastructures that support them can be challenging without the right tools, which has contributed to some of the pressure technologists are experiencing. Traditional monitoring has allowed technologists to identify and catch issues in certain areas of an IT stack, but this approach is of little use in dynamic, distributed and software-defined environments where organizations are continually scaling their use of IT and the cloud up and down.

As a result, the performance of the digital services can be impacted, potentially hurting end-user experience and eroding trust in the brand.

To keep up with the rising demands from consumers in the digital world, technologists need the ability to monitor the full IT estate, from cloud infrastructures to customer-facing applications through to core infrastructure such as compute, storage and network; including the open web and security. With access to tools like observability platforms, technologists can lessen the burden of individually monitoring the performance of each item supporting the digital service. Instead, they can achieve one holistic view into alerting, root cause and analysis of correlated metrics, events, logging and tracing (MELT) data from the cloud to enable early and easy troubleshooting.

Consider tools that link infrastructure (compute, storage and network) with the services and applications they execute. When there is an application-level issue, it is easy to take a closer look and identify problematic areas in what makes the application run or the application itself.

Align IT Performance to Business Results

After aggregating a full-stack view of applications and the cloud infrastructure behind them, technologists can then link performance data identified by the observability platform with real-time business metrics.

Linking the two provides technologists with the ability to identify where and how the digital services are impacting the business and end-users. Full-stack observability automatically provides technologists with these linked insights, enabling them to address the areas that matter most.

Educate Teams

In addition to having the right tools to help take advantage of the cloud, it is important that an IT team is well-versed in the technology itself. Managing the performance of the cloud is complicated and there is a finite pool of talent available to businesses. Therefore, business leaders should provide appropriate resources to ensure teams are educated and equipped to succeed.

To support a seamless implementation of cloud infrastructure to support a business’ digital services, business leaders must invest in upskilling their teams or recruit others who are knowledgeable about cloud monitoring.

This will ease the pressure on existing teams and give technologists the skills and confidence to progress in their careers.

Leverage Automation

Even with the best observability solutions in place and teams equipped with the necessary skills, there is still concern that there will be an overwhelming amount of data generated by accelerated digital transformation over the coming years, especially those involved in large-scale public cloud environments. It can become a challenge for technologists to dedicate the level of resources and skills required to manage and optimize performance across the IT estate.

To avoid these potential circumstances in the future, it’s worth considering how to minimize the impact of a data deluge by reducing an IT department’s reliance on manual interventions to monitor health and performance and resolve issues.

Two things that will play an increasingly vital role in tracking health and performance across the entire IT estate are automation and AI. This is particularly the case in large-scale public cloud environments, which are fast becoming the biggest headache for some IT teams.

With AI deployment to identify and fix issues in real-time, technologists can spend less time firefighting thanks to the AI/ML guiding hand. This will lead to increased job satisfaction and fulfillment and a better work-life balance.

September 8, 2021
Progress Expands Chef Automation Portfolio

In advance of its ChefCon 2021 conference, Progress today unveiled a series of enhancements to its Progress Chef automation platform that includes managed services and a software-as-a-service (SaaS) edition of its offerings as well as tighter integration with cloud services and other third-party tools.

Sudhir Reddy, vice president of engineering for the Chef business at Progress, said the SaaS platform, now available in beta, combined with the managed services provided by Progress will make the automation platform much more accessible to a wider range of organizations.

Progress is now also extending the reach of its Chef Compliance Automation for Cloud Resources offering across Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP) to include Chef premium content that aligns with best practices for each cloud environment in addition to providing integration with Azure Policy, a suite of tools from Microsoft that compares resources to a set of business rules that define how cloud infrastructure should be employed.

Finally, Progress has enhanced Chef Enterprise Automation Stack to provide tighter integration between compliance and infrastructure automation along with adding infrastructure operations dashboard, support for Windows patch management, integration with third-party secrets management tools, and improvements to Chef Workstation and Chef Test Kitchen to simplify testing within application development processes.

With the release of a SaaS platform and the availability of managed services, Chef is joining a long list of IT vendors that are removing responsibility for deploying and securing their platforms from internal IT operations teams. It’s clear more organizations now want to spend more time on building and deploying software than managing the platforms that automate those tasks on their behalf, said Reddy.

It’s not clear how many IT teams that have already deployed a platform might shift over to a SaaS platform or rely on an IT vendor to manage that platform on their behalf. However, in the wake of the COVID-19 pandemic, many IT teams continue to work from home to help combat the spread of the virus. Many of those organizations have either shifted platforms to the cloud themselves, adopted SaaS platforms or outsourced the management of IT platforms altogether. There are also many small organizations that simply don’t have the internal expertise and resources required to stand up an IT automation platform on their own.

That approach also makes it simpler to converge security, compliance and IT operations management within the context of a larger DevSecOps workflow, noted Reddy.

The great paradox of IT automation has always been the level of expertise required to achieve it. Organizations that lack the expertise required to create automated playbooks, for example, don’t derive much value from an IT automation platform. It’s only been larger companies that have mostly derived the most benefit from investments in IT automation. However, as IT automation becomes more accessible via SaaS platforms and managed services, the level of expertise required to benefit from it naturally drops. In fact, with the advent of artificial intelligence (AI) for IT operations (AIOps), many manual IT tasks are increasingly being automated.

It’s too early to say whether the IT industry as a whole is on the cusp of the next wave of automation, but the raw ingredients that make up the primordial soup needed to achieve that goal is, at this point, only awaiting a catalyst.

September 7, 2021
How to Improve Cloud Management

Cloud technology underpins emerging technologies like artificial intelligence (AI), IoT, edge computing and containerization. Organizations must prioritize implementing effective cloud management practices to avoid spiraling costs and potential security and compliance violations.

The COVID-19 pandemic has vastly accelerated cloud adoption. To support remote work, many businesses have moved workloads from on-premises servers to the public cloud. At the same time, thanks to the proliferation of technologies like AI and machine learning, organizations are generating more data than ever, much of which is also being stored in the cloud. It’s no surprise then, that the explosion in demand for cloud services is showing no sign of slowing down. Gartner predicts that worldwide end-user spending on public cloud services is forecast to grow by 23.1% in 2021 to total $332.3 billion.

But, it’s not all smooth sailing. As the average organization’s cloud infrastructure grows in complexity and size, it becomes increasingly difficult for teams to manage it effectively. This has significant security, compliance and cost implications. According to Accenture, an astronomical 30% of organizations’ cloud spend is currently wasted.

So, what can you do to ensure you avoid the most common cloud management pitfalls?

See the Big Picture

It’s surprising just how little visibility many businesses—even those with large teams of cloud architects—have into their cloud infrastructure. For instance, one recent survey found that over 25% of IT departments have no idea what their monthly cloud spend actually is.

Before taking steps to optimize your cloud management practices, it’s crucial that you gain full visibility into your current situation. You need to understand your cost history, audit trail and CPU utilization patterns to identify where you’re overspending.

Plus, make sure you’re using tagging consistently. A well-defined organization-wide tagging policy is essential, enabling you to accurately visualize expenditure by category. It’s also good practice to set up alerts which are triggered when a user launches instances without tags. This prevents any potential oversights.

Develop a Change Management Process

Gone are the days of rigid change management processes handled solely by IT teams. Nowadays, business units can acquire cloud services without even involving IT.

In the cloud, change management is a series of continuous integrations, deliveries and deployments. Given this fast-moving environment, it’s crucial to implement a robust change management process or risk spiraling costs and potential compliance violations.

It’s important that your change management process gives you visibility into and control of each infrastructure change without compromising on delivery speed. To achieve this while establishing strong policies and rules, it’s worth considering automation for at least part of the application delivery process. By auto-approving all routine and low-risk changes, you can free up time for your developers to focus on more complex projects.

In addition, it’s worth integrating your change management process with your DevOps tool stack. You can use collaboration tools like Slack, Teams and/or Jira to ensure everyone has visibility into changes and resolve any issues quickly.

Continuously Optimize your Cloud Infrastructure

Unfortunately, when it comes to effective cloud management, your work is never truly done. Conducting a one-off review of your existing cloud infrastructure will only get you so far.

Many businesses make this mistake and find themselves falling back into their old bad habits a few months down the line. To keep your cloud infrastructure in check, it’s best to continuously track a whole host of factors, including costs, changes, users, permissions and compliance breaches. After all, it’s much easier to solve an issue when you are able to spot it before it gets out of control.

Now you have an idea of where to start when it comes to improving your cloud management practices. However, manually juggling all of the above considerations is a significant undertaking. It’s prohibitively time-consuming for many organizations whose DevOps teams are already overstretched. This is why a whole host of SaaS-based cloud management tools have sprung up in recent years. These offer unparalleled visibility into changes, resource utilization, costs, security, compliance, performance and more. Yes, they come at a cost. But, most businesses find that the savings such tools enable them to identify significantly exceed any outlays.

September 3, 2021
How to Achieve Cloud Operational Excellence
Cloud operational excellence means delivering the right mix of cloud-based services at the optimal cost and quality to support your organization’s mission and strategy. You might be thinking, “Why go to the trouble? Isn’t ‘almost good enough’ sufficient?” It turns out that it matters—a lot. The difference between the top 10% and the bottom 10% in cost per relevant delivery metric can vary by an order of magnitude or more.

In the mid-1990s, Gartner acquired an IT metrics firm called Real Decisions. They offered benchmarking services so customers could compare their IT efficiency with similar organizations. With hundreds of “Global 2000”-sized customers, their database was rich. Over time, the Real Decisions team refined their catalog of metrics and enriched their data with repeat studies over time, to develop indices of efficiency.

Within their user population, the difference in cost per unit of productive work was 11 times better in the top 10% compared with the bottom 10%. That doesn’t mean 11% better, it means 1,000% better.

Note that the user base was self-selected. All participants wanted to get objective metrics of performance relevant to their business goals and paid for extensive studies involving questionnaires, financial audits and technical benchmarks. Notably, all participants had cost recovery programs (beyond chargeback) in place. The bottom 10% of this segment is still within the top 10% of the IT industry—and their score is 11 times worse than the best of the best.

Which raises the question: What is the industry average for IT efficiency? Is it possible that the hundreds of benchmarking users are all doing IT wrong, and the search for relevant metrics is misguided? I think not.

Choosing the Right Metrics

In 1911, Fredrick W. Taylor published The Principles of Scientific Management, which discussed approaches to optimizing two important variables: Output quality and worker compensation. Taylor recognized that successful firms work collaboratively, with management and workers jointly setting goals and developing methods and tools to achieve both profitability and proportionate compensation. Nowhere in this text—or in any of his recorded speeches or documents—does he say, “If you can’t measure it, you can’t manage it.” He never said that for two reasons: First, he did not believe it; second, it is not true. What he did emphasize was that if you do measure it, you will manage it. That was a warning: Pick the right metrics or you will exert effort pursuing a meaningless goal.

Here are some potential cloud performance metrics compiled from several cloud providers and users across various industries, not-for-profits and governmental agencies:
- Service Metrics
  - Reliability – mean time between failure (MTBF)
  - Availability – Uptime, expressed as a meaningful percentage of demand
  - Serviceability – Mean time to repair (MTTR)
- IT Metrics
  - Capacity
  - Latency
  - Bandwidth
  - Response time
- Strategic Metrics
  - Business agility
  - Customer engagement
  - Customer reach
  - Financial impact
  - Solution performance
The journey to cloud excellence starts with developing the metrics that are most relevant to your business goals. Picking the right metrics with the right scale matters. As a rule of thumb, generally right beats precisely wrong.

Sustaining Excellence

After you agree on a set of metrics that are both statistically reliable, repeatable, objective and aligned with your firm’s mission, how do you achieve and sustain excellence?

In the 1970s, the U.S. Department of Defense bought a lot of custom software. Sometimes it worked well, other times it didn’t. So, the DoD funded research on code quality. It turns out that the key difference between great and mediocre quality code flowed from how the organization managed problems. More specifically, how did the team react to an unexpected event? The spectrum runs from confusion and dismay through fire drill chaos to calm, rational assessment and remediation. That methodology gets baked into the code itself. This study produced the Capability Maturity Model (CMMi), which was created by the Software Engineering Institute at Carnegie Mellon University.

The CMMi framework identifies five levels of process maturity. A simplified assessment of an organization’s process maturity level comes down to two questions. First, is there current comprehensive documentation for the process, including how to deal with defects? If the answer is yes, the organization is at level three or higher. If the answer is no, the second question is, “Does anyone know what’s going on?” If the answer is yes, that’s level two; if no, level one.

A level one organization has no standard method in place to deal with problems. When something goes wrong, everybody grabs tools and tries to figure out what went wrong and how to fix it. Organizations like this do not spend much on training or analysis. Their focus is continuing to produce whatever they are trying to make. Over time, an individual may develop expertise in diagnosing a component, and when things go wrong, the call goes out to “Get Fred in here!” to troubleshoot the problem. Organizations with pockets of expertise are moving into level two. Most organizations fall within one of these two levels.

Organizational transformation is very difficult, and often requires significant changes in the management team, as well as funding different activities. Training and communications skills are crucial to proceeding beyond level two. Management rewards heroes who can shoot the most difficult bugs. They get the big bonus, the promotion, the better office, a parking space near the door. This behavior reinforces the culture of heroes. But moving forward requires the heroes take on a new role.

Once the organization creates documentation, it is on the path to level three. Note that these transformations are wrenching. It is not easy to tell the heroes that their greatest value to the organization is now how well they can write or teach. But with proper management attention, it can be done. And the benefits of moving forward are many:
- Dramatically fewer crises. Staff could make plans and keep them – no emergencies interrupting a family gathering, a school event, or a get-together with friends.
- High quality code. Maintenance tasks became much simpler, and customers experienced improved reliability. The documentation was helpful, and troubleshooting became routine rather than overwhelming.
- Reliable planning. In a mature organization, plans hold true because they are based in proven metrics, continuously validated processes and uniformly high competence within the team. Project estimates are accurate because the data stems from reliable, repeatable evidence.
Cloud excellence is not a phantom or an unachievable goal. It is the result of clear thinking and sound documentation. Over time, practices improve, skills build. To quote Macklemore, “The greats weren’t great because at birth they could paint, the greats were great because they painted a lot.” With practice and focus, your organization can achieve cloud excellence.
September 2, 2021
That Other Continuous: Data Protection

Looking around at things to add to our current project’s toolchain (that is one of the reasons we do these side projects; to check out all sorts of new ideas/tools/etc), and researching the company where a former coworker is the CMO, I was drawn into the resurgent world of continuous data protection (CDP). It is interesting that we could end up adding CDP to the list of ‘continuous’ activities, like CI/CD/… CDP, even though CDP was named long before any of our DevOps or Agile terms were born.

The whole idea of CDP is that it is always backed up, so you are continuously protected. It largely failed when first introduced because it massively increased both the cost and complexity of your storage environment. Essentially, it required a hot copy of all of your storage. Not something most enterprises were willing to invest in. I mean, it was friggin’ expensive. And it used virtualized storage, which no one wanted to mess with. Long story short, at the time, virtualized storage introduced issues no one wanted to deal with.

But several things have changed over the years, and it is interesting to see where CDP concepts have cropped up. You see, today it is being used to protect companies from ransomware. It’s still a bit of complexity, but we’ve become experts at hiding complexity. It still requires a lot of storage (I would argue that the current versions I’m seeing require more storage than the original CDP solutions did), but when your environment is taken over by ransomware, a clean machine can be used to restore data to a point before the ransomware took over. From a “continuous protection” standpoint—giving it a modern meaning along the lines of “continuous testing”—that is an astounding feat.

The way it works—at least the way that the current implementations I’ve looked at work—is pretty simple at heart, though as my former coworker likes to say, “Easy in concept, difficult in implementation.” It keeps every write ever made to storage, with time and currency markers. So it knows what the disk currently looks like, but it can also get to what the disk looked like at 12:33 am yesterday. Or January first, or a year ago… You get the idea.

It uses a ton of space because there is an implied truth here—nothing is destroyed. In traditional storage, if you change a document and hit save, the document is written over. In the new iteration of CDP, if you change a document and hit save, the changes are recorded, but the original is not altered. Call it “non-destructive writes,” if you will.

It is worth checking out. The concepts are huge, and offer protection that is otherwise in short supply. If you have a dedicated DevSecOps team, they’re the ones I’d point at it, since the big reason for using this is a security reason at heart (though there are other uses; I’m currently gushing about security because this is a big deal).

I’ll get back on track and report back about things directly in the toolchain for our newest DevOps project next week. I thought this was worth writing about, since most of you all could use the concept, even if only for peace of mind. You’re rocking it! Protecting your work against the newest range of attackers is just common sense. It’s worth seeing if the modern iteration of CDP is for you.

September 1, 2021
Majority of Organizations Now Adopt Multi-Cloud

As IT’s digital footprint grows, its cloud footprint is rising both in size and complexity. As a result, most organizations have matured into multi-cloud operators. A multi-cloud approach empowers different divisions with the autonomy to select the right computing environment for their needs. It also enables a lift-and-shift ability to port workloads across cloud service providers (CSPs) when needed. Adopting a multi-cloud model could bring performance benefits, cost savings or better scalability, all while encouraging a more vendor-neutral stance.

We’re in a multi-cloud world. A recent HashiCorp study found three out of every four organizations are already multi-cloud. This is a staggering increase from just a few years ago when mono-cloud was more accepted, and operators had zombie-like allegiance to AWS. The uptick in multi-cloud signals the maturity of offerings at other CSPs, like Azure and GCP. It also signals differentiation among clouds as each hones in their unique value propositions.

HashiCorp’s inaugural State of Cloud Strategy Survey offers some very clear results about the state of multi-cloud adoption. Below, I’ll review major takeaways from the report to better understand the state of multi-cloud adoption. To sum it up, large organizations tend to benefit most from multi-cloud adoption, as there are still significant roadblocks to truly reaping a sizable return from a multi-cloud status. Yet, more infrastructure automation may help companies of all sizes respond to multi-cloud complexities.

The Multi-Cloud Era

DevOps truly operates in a multi-cloud era. The report revealed 76% of organizations are already multi-cloud; that leaves only one in four companies leveraging a single cloud. And, that trend won’t stop—within two years, 86% predict they will be multi-cloud operators. The study also confirms that the larger a company is, the more likely it will adopt multi-cloud—94% of large enterprises with 5,000 or more employees are multi-cloud.

So, what’s driving this cloud fragmentation? Well, 34% of respondents cite digital transformation as a top driver for multi-cloud adoption. In the race to meet new digital innovation demands, multi-cloud has emerged as a strategic force. To enact this rapid change, many organizations are investing in a cloud center of excellence (CCoE) to guide cloud transformation efforts. A CCoE is most common within large companies with a cloud budget of $5 million or more.

Speaking of budgets, cloud spending is going through the roof. In 2020, only half of companies met their projected cloud spend, and 39% of companies overspent their allotted cloud budgets. As I’ve covered before, cloud-native costs are notoriously difficult to identify and trim. Without the proper guardrails to manage spending, surprise data egress fees or hidden server costs can strain the cloud checkbook. As it stands now, 40% of companies lie in the $100,000 to $2 million per year range, and 6% of enterprises are investing $50 million a year or more into multi-cloud. This spending will likely increase in the coming years as cloud reliance expands.

Enterprises Benefit Most

So, which clouds are companies using? If we compare cloud adoption, AWS is a clear market leader; 88% of companies use AWS, 74% use Azure and 61% use GCP. AWS’s hegemony is not expected to fade in the next two years. Yet, while AWS usage will remain the same, competing CSPs will experience modest increases. This is likely due to Azure and GCP making gains in geographical zones outside North America, such as Asia-Pacific and Africa.

In terms of technology used in cloud environments, open source software (OSS) is a popular choice. Cloud-native open source projects, such as Kubernetes, OpenStack and Terraform, have taken on a life of their own, with vibrant community support. However, not all shops build upon OSS and run it themselves. Somewhat ironically, we’re seeing more managed OSS emerge. One stand-out area is provisioning, where 42% of organizations build with open source and run it themselves.

On the whole, multi-cloud is working at most enterprises, with 53% of survey respondents saying multi-cloud has helped achieve their business goals. The top benefit of multi-cloud is digital transformation, at 34%. This is followed by avoiding lock-in (30%), reducing cost (28%) and scalability (25%). But multi-cloud does not treat all companies as equals; the likelihood of multi-cloud success increases as the company size increases. Less than half (42%) of small companies say they have multi-cloud success, which jumps to 64% for large enterprises. With fewer resources for support, multi-cloud may be less beneficial for startups and small to medium businesses. In fact, 41% of professionals say a lack of in-house skills inhibits their cloud success.

Challenges in a Hybrid Multi-Cloud World

An ongoing talent gap is just one potential challenge to cloud success. Just over half (51%) say cost concerns are a top inhibitor to their cloud program success. Hybrid cloud complexity is another prominent challenge, with 35% of organizations citing it as a major cloud obstacle. Many enterprises are still grappling with the remnants of legacy on-premises estates. But technical debt is not the only reason for hybrid modalities. Surprisingly, large platforms even have constructed their own data centers or used colocation as an intentional strategy.

Security is a top cloud inhibitor, according to 47% of IT professionals. The leading cloud security concerns include data protection (40%), data theft (33%) and regulatory compliance (31%). The myriad cybersecurity threats evolve each day, and cloud technologies are not immune; for example, web APIs are now a significant threat vector. A separate report found a 300% increase in malicious API traffic in the past six months. Other issues, such as exposed keys, insecure defaults and misconfigurations are known to hamper cloud-native infrastructure.

Contrary to popular belief, nearly half (46%) of companies reported that COVID-19 has not quickened their cloud adoption. This reality could indicate that cloud migration has been planned for years, and recent adoption is only part of an ongoing trend. The study found that the pandemic did, however, accelerate things like infrastructure-as-code (IaC), container orchestration, compliance and governance and other operational factors.

Infrastructure Automation Supports Multi-Cloud

Multi-cloud feels like an inevitable result of accelerating cloud adoption in a more diversified market. Rising multi-cloud use is also arguably in line with Conway’s Law. As organizations develop more and more digital experiences across new verticals, the platforms and backend processes to support these initiatives will inevitably grow more and more complex.

To mitigate some of the issues posed by multi-cloud, many organizations are turning toward increased infrastructure automation. In fact, 70% of those surveyed said infrastructure automation tools are extremely important to their organization. Of these tools, provisioning automation (75%) is the most popular kind in use. Automation for other areas, like networking (58%), security (50%) and app development (69%) follow.

Hashicorp’s State of Cloud Strategy Survey boldly announces, “Welcome to the Multi-Cloud Era,” and its findings back up that sentiment. For further insights, you can view the results freely on this website.

August 31, 2021
CloudBolt Survey Shows DevOps Frustration in the Enterprise

A survey of more than 200 senior IT leaders at organizations with more than 1,000 employees published today finds only 4% consider their organizations to be experts when it comes to continuous integration/continuous delivery (CI/CD).

Conducted by CloudBolt Software, a provider of a platform for many IT environments, the survey also found only 11% consider their CI/CD infrastructure to be reliable. Another 69% said their CI/CD infrastructure was somewhat reliable, while 21% said their platforms were somewhat unreliable or not reliable at all. A full 85% also report they regularly test the infrastructure they rely on to drive DevOps processes.

Overall, the CloudBolt survey finds only 5% of respondents report their organization is deploying code multiple times a day. Most respondents (85%) said their organization only deploys code on a weekly or monthly basis.

A full 70% said that, on average, it also takes multiple days or even weeks to set up a pipeline. Nearly two-thirds (63%) said there are too many manual processes involved when setting up pipeline infrastructure. More than half (56%) also noted they have no automated processes for post-deployment tasks.

More than half (53%) said it is also challenging to proactively detect infrastructure issues, while 55% said it is difficult to create a consistent CI/CD pipeline environment across multiple development teams. Improving those pipeline environments requires speeding up the process of provisioning infrastructure through automated processes (70%); continuously detecting infrastructure issues to reduce testing challenges and failures (62%) and simplifying remediation of infrastructure issues proactively (56%), according to survey respondents.

The CloudBolt Software survey specifically identifies issues organizations encounter when employing Terraform to manage infrastructure-as-code (IaC). Only 5% of respondents said they were “very satisfied” with using Terraform and another 75% reported they are somewhat satisfied. Top challenges include difficulty using it in on-premises infrastructure deployments (62%), lack of Day 2 management for deployed resources (53%) and governance (42%). Only 2% of respondents said they were employing Terraform to deploy 75% or more of their infrastructure. A full 90% noted that Terraform requires more custom-coded integrations with other critical tools and systems to work properly.

Finally, 29% said their organization lacks specific Terraform expertise. The most sought-after Terraform capabilities are enabling non-Terraform experts to easily deploy infrastructure using Terraform; ensuring that only Terraform plans with IT “guardrails” are executed and the ability to visualize and cost-optimize infrastructure deployed by Terraform. More than a third (35%) want to simplify deployment of on-premises infrastructure when using Terraform.

Jeff Kulkowski, CloudBolt CEO, said the survey makes it clear there simply isn’t enough expertise available to implement DevOps best practices at scale without enabling IT administrators—that often lack programming expertise—to manage infrastructure and DevOps platforms. That requirement creates the need for an IT management platform based on a set of visual tools that enables DevOps professionals and traditional IT managers to work more collaboratively, he added. DevOps teams can manage IaC whenever they desire, but it’s also crucial to enable IT operations staff to manage IT infrastructure alongside those teams.

Obviously, DevOps as an IT discipline still has a long way to go before it is pervasively adopted across enterprise IT organizations. The challenge now is to remove many of the low-level barriers that conspire to block IT modernization initiatives.

August 25, 2021