devopsy.kinsta.cloud

Category: DevOps Culture

Managing Technical Debt is a Balancing Act

Technical debt is a nightmare for product managers. It can slow down teams and inhibit response time for delivering quality products to market. Yet, there are times when technical debt can be beneficial; but turning it from a headache to a positive is a balancing act.

Nimble project managers and developers should strive to develop an equilibrium that balances creating, reducing and accepting technical debt. The good news is: That’s attainable.

The Shape of Technical Debt

It’s essential for PMs to understand how technical debt can arise so they can identify its characteristics and manage the amount to carry or avoid to the team’s advantage.

Laying too much brick and mortar will take software features too long to develop. It risks their rollout arriving well past due dates. But fixating on speed threatens to underdeliver on functionality. Circumstances like these will diminish the ability to deliver powerful results over time.

Similarly, ignoring the power of third-party solutions to slot in and satisfy customer needs and speed-to-market is outdated thinking. For example, software like embedded analytics can arrive off-the-shelf to complete an architecture task quickly and affordably while freeing up the team to address other challenges. Remaining agile with all options available to the team is critical.

Overall, the project manager’s goal should be to identify the indicators of technical debt and hone their skills to address each type.

Building Architecture

When a team is tasked with delivering new features or functionality, building upon the existing structure to support the new capability is not unusual. Lack of foresight or diligent planning about what’s necessary to enable this function can be a critical oversight that leads to a void in need resolution, i.e., incurring debt requiring payment.

Speed versus Quality

Delivering at the speed of modern business often means facing time constraints that can impact the quality of work. Faced with such limitations, teams may take shortcuts. Ultimately, those shortcuts can lead to an increase in technical debt.

Quantity versus Speed

On the opposite end of the spectrum, eagerly attacking a project that requires significant lead time can result in overbuilding. This leads to a different form of technical debt. Longer projects can be affected by technological change—an evolution that makes some previously completed structures outdated. Debt takes the form of teams forced to return upstream to make necessary changes when they should instead remain downstream.

Near-Sightedness

Unlike the oversight or overbuilding described above, if the team fails to consider future needs and focuses only on current tasks alone, it risks creating debt that arises by starting from scratch. If teams don’t anticipate the next feature or prepare for future functionality while in development, they will likely pay for this near-sightedness down the road.

To Build or Buy

Developers want to build. It’s what they do. But it’s not always in the organization’s best interests. When a team faces complicated requirements, is extended beyond its capabilities or is simply well outside its repertoire, debt can be avoided by looking toward existing platforms that fulfill its needs. In such instances, a third-party solution saves significant time and money. For example, an embedded analytics tool built with the latest technologies and deployed turnkey frees the development team to remain focused on its expertise and apply its talents there. Think: Does the team need to invent a new wheel, or does one already exist? While choosing the former risks taking on debt, selecting the latter holds the promise of reducing it.

Manageable Debt

Still, technical debt can be a benefit.

Consider this example: You want to roll out a platform that delivers to the customer and market, but it’s a ground-breaking product. You don’t yet know everything you want or need to know about how end users will engage with it and what they’ll ask it to do. In situations like these, it’s reasonable to get the functional framework of the software into the hands of the market so that you can absorb reams of feedback, return to the platform and then build out or fine-tune its features and functionality. Actions like these are wise. PMs haven’t mortgaged the farm for a bag of magic beans—they’ve taken on a reasonable amount of debt to fertilize team knowledge into a product that will ultimately deliver the most value.

Recognizing technical debt and its approach, avoiding or accepting it accordingly and then managing it with deliberate efforts to pay it down within acceptable limits is where PMs want to be. If you invest in paying it down or seek and accept third-party solutions to avoid it altogether, you’ll maintain team focus and velocity on the projects they excel in.

October 26, 2021
Mona Allies with New Relic to Converge MLOps and DevOps

New Relic and Mona, a provider of a platform for monitoring the models used to provide artificial intelligence (AI), announced today they have formed an alliance to help bridge the divide between DevOps teams and data scientists.

Most data science teams today build models based on machine learning algorithms using operations platforms that enable them to manage the life cycle of those models, otherwise known as MLOps. Mona provides a set of monitoring tools that are employed to detect drift that can occur as some of the underlying assumptions that were made when building the model are discovered to be less accurate or when new data sources become available.

New Relic, meanwhile, provides an observability platform that is widely employed by DevOps teams to monitor and observe an IT environment. IT teams that have access to New Relic One can now also view dashboards provided by Mona that detect anomalies involving data integrity issues and model performance issues.

The integration also enables users to create custom Mona alerts that are shared via the New Relic Alerts and Applied Intelligence service. That capability reduced the volume of alerts generated by correlating anomaly detection to specific IT events.

Yotam Oren, CEO of Mona, said that level of integration provides a bridge between DevOps and data science teams that typically have widely divergent cultures.

Incorporating AI models within applications is the next major DevOps challenge. There is almost no application being built today that won’t incorporate some level of AI. The challenge is AI models are built by data scientists that employ a wide range of machine learning operations (MLOps) platforms to construct those models. The pace at which those AI models are developed, deployed and updated varies widely. However, all organizations will eventually need to confront the problem of how to insert AI models within applications both before and after they are deployed in a production environment.

There are two schools of thought about how best to achieve that goal. The first is to assume an AI model is just another type of software artifact that can be managed as part of an existing DevOps process. In that context, there may be no need for a separate MLOps platform. Conversely, proponents of MLOps contend AI models are built using pipelines created by data engineering teams working in collaboration with data scientists. Developers only become part of the process when an AI model needs to be inserted into an application environment. AI models may ultimately need to be incorporated within an application, but the proponents of MLOps platforms maintain building models requires a unique platform to manage a software artifact that is built by training algorithms.

It’s only a matter of time before compliance issues force some form of convergence between DevOps and MLOps teams, noted Oren. The longer it takes to address both the cultural and technical issues that are required to meld DevOps and data science teams, the more challenging it will become as both teams are set in their respective ways.

October 25, 2021
Bridging Departmental Divides for Successful Automation Projects
For too long, IT has been largely isolated from an organization’s core business. Any division between IT and lines-of-business (LOB) is even more detrimental when speed is not just a competitive advantage but an existential necessity in today’s accelerating digital economy.

As business leaders streamline and turbocharge every facet of their operations, savvy organizations are rushing to eliminate inefficient and damaging gaps between departments by putting more software tools in the hands of more employees. Analysts predict 80% of technology products and services will be built by those who are not technology professionals by trade by 2024.

This hits home amidst the booming trend toward robotic process automation (RPA), low-code/no-code and other automated solutions, which have only continued to grow as more businesses realize the vast benefits of automation: Accelerating operations while also reducing costs; elevating customer experiences and shifting employees to higher-value positions and laying the foundation for new data-driven revenue streams.

Squandering Automation’s Advantages

But these advantages are quickly squandered if there’s a turf battle between the business unit looking to optimize processes and the IT department responsible for deploying the tools to make that happen.

When there isn’t collaboration and plenty of trust on both sides, automation initiatives are more prone to disappointments and delays.

This may happen if the IT department comes across as too smart for their own good, putting less value on feedback from their LOB colleagues and rushing to implement software with a focus on speed rather than outcomes. On the LOB side, leaders may assume their IT counterparts are too technical and don’t understand business objectives and choose to override important IT input about which vendors are best qualified when it comes to support, integration and security.

The problem here is not the technology; hundreds of thousands of businesses across the world have successfully deployed process automation solutions. It’s the execution.

When an automation initiative fails, there’s almost always blame to be shared on both sides of any divide. Likewise, success typically boils down to people across IT and every business department working better together to achieve a common goal.

To embrace a collaborative mindset and set up your automation projects for success from day one, I encourage you to follow these clear steps:
1. Reinforce a shared mission: Whether it’s automating the customer onboarding process or deploying RPA bots to automate a repetitive, time-consuming process, communicating a shared mission is the crucial first step to pulling off a successful project. On the IT side, that involves talking with the LOB about their goals, objectives and business strategy. LOB leaders, meanwhile, can emphasize that IT is an equal business partner that has valuable insights and expertise to contribute.
1. Establish visibility and transparency: Armed with a full understanding of business objectives, IT leaders can now evaluate tools built to give all stakeholders a full view of the processes that will be affected. By documenting and mapping out each process within an organization, leaders can better understand current processes and gain insight into areas that are ripe for standardization and automation.
1. Tailor solutions that match the problem: With heightened visibility across the organization, LOB and IT teams can then determine the scope of each problem and tailor solutions accordingly. Some processes can be kept and optimized; some can be tweaked with automation. Other processes may need to be reconfigured significantly before automating or shut down entirely because they are outdated or unnecessary.
A great example of this collaborative approach comes from Coca-Cola Beverages Florida, LLC (Coke Florida): Encumbered with a legacy process repository — a collection of PDFs, videos, pictures and spreadsheets outlining more than 600 processes — Coke Florida standardized and leveraged visual process mapping software to establish an easily accessible document repository that provides visibility and control over key processes, encourages collaboration and increases accountability.

Coke Florida continued an organization-wide initiative to digitize and automate its business processes with digital workflows, forms and RPA software, driving further efficiencies across its entire organization. As the company discovered firsthand, shared goals result in shared success.

When speed and high-quality outcomes become the norm, companies master the essential arts of urgency, resilience, visibility and collaboration. By bridging the divide between departments, organizations not only drive internal efficiencies but also gain a competitive advantage that is indispensable in today’s burgeoning digital economy.
October 22, 2021
Developer Challenges in Commercial Banking
Much like many other business sectors, the banking industry has had to accelerate their digital transformation efforts as a result of the COVID-19 pandemic.

Today’s retail banking industry is a far cry from where it was even just five years ago. Consumers no longer have to physically appear in their bank to process payments or open new accounts. Mobile and web apps feature intuitive experiences and sophisticated functionality. Overall, retail banks and their software development teams have done a tremendous job in creating tools that are integral parts of personal banking today.

However, while significant gains have been made in retail banking over the last few years, the commercial banking sector is lagging behind. For all of the innovation within the retail banking experience space, the commercial banking space is still as fragmented and “untechnical” as ever. For developers, this means they have their work cut out for them.

If you’re a developer who wants to delve into the banking space or if you’re just beginning to work on commercial banking products, here are a few things to keep in mind.

Be Prepared to Work with Outdated Tech

One of the biggest challenges developers will face is that, more often than not, commercial banks are relying on decades-old legacy infrastructure.

As banks have evolved, their IT teams, vendors and consultants have routinely built layers on top of and beside core banking systems. The problem is in many cases, these layers were designed and implemented when authentication delegation, microservices and event-driven architecture were either too novel or nonexistent.

Therefore, developers need to resign themselves to meeting the banks where they are in terms of technology. For example, they might see files instead of APIs, obscure XML schemas (never mind JSON), unexpected throttling and batching in addition to an overall lack of more advanced tools such as webhooks, events, streams and reporting.

Patience is Key

While elsewhere in the IT industry speed is king, many large organizations such as commercial banks tend to operate at a much slower pace. This goes beyond just using a “slower” methodology or having a “slower” development process with lengthier iterations. Unlike in a startup, the development process within larger organizations is a part of a bigger business process — and business processes in banks take a lot more time and effort and are subject to much greater scrutiny. Developers should expect a slower pace and must understand the need to be patient.

Embrace the Unknown and Experiment (Responsibly)

Banking, finance and accounting are very complex, intricate domains. Moreover, banks are large and complex organizations whose inner workings most developers are not generally familiar with. Therefore, developers may be afraid of making mistakes and opt for safety over creativity. This simply can’t be allowed to happen. Of course, no one likes to make mistakes. However, mistakes are often a key building block in innovation. Instead of letting the unknown stunt innovation, developers should embrace this unfamiliar terrain as an opportunity to experiment. Granted, there need to be quality control steps in place to make sure that these experiments don’t cause havoc or send critical functionality off the rails.

Here are a few best practices that each role can follow to strike the right balance between experimentation and objective achievement:
- Engineering managers:
  - Build a culture that encourages writing clean code followed by refactoring.
  - Ensure developers understand the business and prioritize addressing customers’ pain points.
  - Make sure the organization allocates sufficient capacity to accommodate the cost of innovation.
- Developers:
  - Don’t rush putting things into the core of your platform.
  - Move side effects to the edges of your application and decouple them from the business logic. Moreover, strive for keeping business logic as pure as possible when it comes to functionality; for example, make sure the same inputs produce the same outputs without creating side effects or depending on them.
- Everyone:
  - Use the Theory of Jobs-to-be-Done to better understand the customers’ needs. It takes much longer to understand how the banking industry works than to understand the customers’ jobs-to-be-done and their needs.
Having these guideposts in place will help developers become more comfortable with sparking innovation in commercial banking and allowing creativity to flourish. Understanding and accepting these challenges will enable developers to build the sophisticated tools and experiences that have been long overdue in the commercial banking space.
October 21, 2021
How do we Bring APIs Into Decentralized Web3?

Blockchain-based applications are growing in popularity and underscore the decentralized Web3 movement. Simultaneously, web APIs are helping port data and reusable functionality across application types and are driving microservices development practices. So, how do we bring these two paradigms together?

There are many cases where a blockchain-based smart contract may require data updates that only a web API request can fulfill. For example, perhaps the smart contract needs to access the latest price of a given stock ticker. Or, perhaps on-chain activities are contingent on the weather at a particular time. The ubiquity of APIs means there are thousands of helpful features available for the task at hand. Yet, integrating them into Web3 applications is tricky.

Interestingly, the API3 Alliance is making strides to introduce an industry-standard solution to bring APIs to Web3 applications. API3 provides a vendor-neutral standard gateway to apply on top of existing APIs that helps them appeal to decentralized apps. The vendor-neutral API Alliance already consists of many APIs across various verticals and appears to be gaining more traction.

I recently met with API3’s co-founder, Heikki Vänttinen, to learn more about the challenges of introducing traditional web APIs into blockchain development. According to Vänttinen, whereas previous solutions relied on a third-party middleware or ‘oracle,’ the best way to incorporate an API request and response system into smart contracts is through a node operated by the API provider themselves. Such a system could help blockchain-based applications leverage myriad software-as-a-service solutions, essentially uniting the two worlds.

The State of APIs in Web3

The number of APIs is growing exponentially. As companies reinvent themselves with digital transformation, APIs have become the glue for interconnecting the applications we use on a daily basis. When it comes to APIs, there are a growing number of potential use cases for blockchain—the technology is not just relegated to cryptocurrency; rather, it can power a number of scenarios that require smart ledgers.

Since APIs drive much of the functionality in modern software development, it makes sense that they may be useful in decentralized app development. For example, a blockchain-based decentralized finance (DeFi) app that handles lending and borrowing may want to call an API to collect raw data for a liquidation price, says Vänttinen. Or, a smart contract may wish to acquire the cost of an asset as a conditionality for making a purchase. Hooking into public APIs to check local weather or flight arrival information could affect how smart contracts respond.

There are many use cases for incorporating APIs into a distributed ledger system. However, “blockchain applications are not able to call APIs natively,” Vänttinen explained. The incompatibility is due to the nature of blockchain technology, said Vänttinen, as it requires consensus from multiple parties to maintain the smart contracts. For this reason, blockchain applications require a middleware piece to connect a blockchain node to an API, relaying data responses from outside of the chain.

Incorporating APIs Into Blockchain

As Vänttinen described, previous attempts to connect APIs to Web3 architectures have revolved around a third-party middleware node. Yet, externalizing integration responsibility in this way creates a centralized point of failure, which defeats the purpose of a decentralized environment altogether.

Instead of relying on such a dependency, Vänttinen advocates for a “first-party oracle approach,” in which API providers operate a layer in front of their APIs. This enables a request-response mechanism for smart contracts, allowing blockchain applications to initiate API requests from on-chain and incorporate the responses into the contracts. This method retains immutability and trustlessness, which is inherent to the distributed ledger.

One solution enabling such an architecture is Airnode, a Web3 middleware that connects any web API directly to any blockchain application. Companies that have adopted Airnode make up the API3 Alliance, which is “a collective of API providers who have signed up to enable this first-party use of their API for Web3 applications,” said Vänttinen. “Adopting Airnode is a pledge from API companies to make their API blockchain-compatible.”

The API3 Alliance is one significant initiative aiming to extend the capabilities of smart contracts and blockchain applications. This could enable such environments to call an API to send a text message or email, or grab sports scores, collect maps data and so on. Thus far, participating APIs in the API3 Alliance come from sectors such as agriculture, biotechnology, communications, open banking, insurance, supply chain management and many others. The initiative also accepts applications from API providers that want to join the marketplace.

Benefits of a Common Web3 Integration Layer

So, what are the benefits of embracing an industry-standard middleware for web-API-to-blockchain connectivity? Well, according to Vänttinen, it is the best way to enable complex off-chain computation. Enacting intricate computing on the chain is quite resource-intensive, resulting in high transaction fees. By extending particularly complicated calculations externally, blockchain applications could reduce operational overhead. “The industry is long overdue for connection between the real world and blockchain,” said Vänttinen. (By real-world, I assumed he meant other architectural styles that pre-date blockchain.)

Having a standard implementation could also reduce the development burden for the API provider to support nuanced technologies. Whereas other solutions would require someone in-house to operate blockchain nodes, Airnode is more of a plug-and-play API gateway that could be deployed on AWS as a Lambda function, Vänttinen said. This usability makes it easy for API-based services to appeal to Web3 applications, thus quickly opening a new potential growth segment.

Of course, Vänttinen admitted, the concept of syncing with off-chain processes is still at a very early stage. There are competing philosophies about how to address it on the market, and some foundational issues have not yet been resolved. For example, most efforts have focused on Read operations to acquire raw data. The next step will be having the middleware ‘oracle’ incorporate off-chain Write capabilities, too. Furthermore, more PubSub-like formats will be important to enable event listeners capable of reactive qualities.

Final Thoughts

The Web3 trend is bringing more decentralized, blockchain-based applications to fruition. But, such applications can’t operate in a closed-off world. To experience the full breadth of digital possibilities, they’ll likely need to figure out how to incorporate off-chain computations and mesh with alternative communication software libraries. Naturally, this will require integrations with web APIs, which have become an important method of cloud-based communication.

The API3 Alliance is “here to propose a solution to the API connectivity problem for Web3,” said Vänttinen. “When a data provider is blockchain-compatible, you can apply the reputation of the data provider in this decentralized data feed … The importance of this becomes more clear when you talk about connecting off-chain services and microservices with smart contracts.”

Airnode and the surrounding API3 Alliance is one initiative spearheading the connection of APIs and smart contracts. But, this is just one method in a maturing field. What do you think about bridging the two paradigms? Comment on your experience below!

October 21, 2021
Atlassian Survey Shows Lack of Collaboration in Software Development

A survey of 750 IT and engineering leaders found that while a full 83% of respondents agree that interdepartmental communication is critical for successful software development, only 41% of respondents view cross-collaboration to be a priority.

The survey gathered responses from within organizations that generated more than $50 million in annual revenue and was conducted by Forrester Consulting on behalf of Atlassian.

Overall, more than two-thirds of respondents (68%) said more departments than ever are involved with software development. Overall, 80% said they need to improve collaboration with other departments, but 82% don’t feel particularly confident about their ability to accomplish that goal. Three quarters (75%) also noted the need to improve collaboration within their own department.

Well under half of the respondents (42%) said their culture, processes and tools enable the real-time teamwork and free-flowing information that help leaders make sound decisions.

Among the subset of respondents that are highly collaborative, however, more than (53%) said they see greater customer satisfaction, with 59% reporting increased revenue because of that increased customer satisfaction.

Suzie Prince, head of product for DevOps at Atlassian, said the challenge organizations face is finding a way to successfully employ a mix of synchronous versus asynchronous communication mediums to better enable collaboration. In fact, collaboration as part of any DevOps culture needs to be a C-level discussion to consistently accelerate the rate at which software can be developed and deployed, she added. The core issue is that not every management team is equally comfortable with open communications between the various departments that make up an organization, said Prince.

It’s critical for C-level executives to set that tone and lead by example; DevOps best practices can’t only be adhered to by development teams. The entire business needs to embrace DevOps best practices as an operating model for the business to really attain the maximum benefit.

Unfortunately, DevOps practices within development teams have been unevenly adopted, so it’s often very difficult to get the rest of the business to embrace DevOps principles. Organizations that do embrace DevOps, however, tend to be much further down the path toward becoming a digital business and can more adroitly take advantage of business opportunities as they arise or shift strategy as changing business conditions may warrant. In fact, organizations that embraced DevOps best practices arguably fared better during the economic decline that followed the onset of the global COVID-19 pandemic that took every business and IT leader by surprise.

It’s not clear from the Atlassian survey results if a DevOps divide is starting to emerge that makes some organizations more competitive than their rivals. In theory, at least, if every organization is now a software company that happens to build something or deliver a service, then the faster they can reliably build and deliver applications the greater the competitive advantage they can attain and sustain.

There are, of course, a whole host of factors that determine how competitive any organization might be. However, in the age of digital business transformation, it’s clear that being able to build and deploy software at a fairly regular cadence is the new table stakes required for simple survival.

October 20, 2021
Pulumi Adds Registry to Share Secure IaC Code

Pulumi this week added a public registry to its portfolio that makes it easier to discover packages that have been created using its tools to manage infrastructure-as-code (IaC).

Joe Duffy, Pulumi CEO, said the Pulumi Registry will make it simpler for DevOps teams to share and reuse IaC tools that have been vetted versus always requiring each development team to write new code each time they want to programmatically provision infrastructure. Pulumi Packages provide modern cloud reference architectures in the form of software development kits (SDKs), code samples and how-to guides.

One of the primary reasons that security has become a major cloud issue is that developers often misconfigure infrastructure. The registry provides a way to reduce those configuration errors by making it easier for developers to reuse code written in TypeScript/JavaScript, Python, Go and .NET that they have stored in repositories such as npm.

In addition to the major cloud service providers, there are also Pulumi Provider Packages for Auth0, CloudFlare, Confluent Cloud, Datadog, DigitalOcean, Docker, GitHub, Kong, MinIO, MongoDB Atlas, PagerDuty, Snowflake, Spot by NetApp and others.

There are also Pulumi Component Packages for deploying container applications on instances of Kubernetes and other related platforms in addition to deploying applications on serverless computing frameworks.

Duffy said in addition to providing a “golden image” of their configurations the registry reduces the overall level of friction that often exists between developers, IT operations teams and cybersecurity professionals.

The latest Pulumi offering is part of an ongoing effort to democratize cloud computing in a way that is repeatable and reliable, noted Duffy. Pulumi, for example, just added support for a set of AWS Cloud Control application programming interfaces (APIs) that reduce the number of APIs that developers would otherwise have to master. The more APIs there are, the more likely it is a developer will make a mistake.

IT teams should expect that other cloud service providers will similarly streamline the APIs they present developers as part of an effort to make their services both more accessible and secure. In the meantime, frameworks provided by Pulumi provide a means to invoke those lower-level APIs at a higher level of abstraction for both developers and internal IT operations teams alike.

It’s not clear today how many developers routinely employ IaC tools, but as the number of workloads deployed in the cloud steadily increases, so too does the percentage of developers that employ these tools. While that shift represents a major boon in productivity for developers, it brings with it additional security risks that many organizations fail to appreciate until it’s too late.

The only way to really solve that issue is for organizations to adopt DevSecOps best practices that require a level of collaboration between DevOps and security teams that is still relatively rare. However, as the tooling provided continues to improve, the technical challenges associated with achieving that goal continue to decline. The next big issue, as always, is bringing together all the diverse cultures within an IT organization to actually implement those tools and define a set of best practices that work for all concerned.

October 20, 2021
Splunk Donates eBPF Telemetry Data Collector to CNCF

Splunk Inc. announced during the KubeCon + CloudNativeCon North America conference that it donated a data collector to the OpenTelemetry project run by the Cloud Native Computing Foundation (CNCF).

Splunk’s data collector runs as a sandbox-level program at the kernel of a Linux operating system and takes advantage of extended Berkeley Packet Filter (eBPF) technology to make it simpler to collect networking telemetry.

The OpenTelemetry initiative itself spans a range of open source tools, application programming interfaces (APIs) and software development kits (SDKs) that are used to instrument applications. Previously, Splunk contributed the SignalFx Smart Agent and Smart Gateway to the OpenTelemetry project along with more than 64,000 code contributions.

Morgan McLean, a director of product management for Splunk, said that while OpenTelemetry is officially in beta, some elements of the project are more mature than others. For example, tools for capturing metrics and traces are already being employed, while another set of tools for capturing log data will be ready sometime next year, he said.

The data collector contributed by Splunk is among the first that operate at the kernel level to collect data, added McLean. That approach makes it possible for some types of data to be captured by default rather than requiring developers to always instrument their applications using agents to enable an observability platform. It will also contribute toward the eventual convergence of network operations and DevOps processes.

Going forward, DevOps teams should expect to employ a mix of open source data collectors that operate at both the eBPF and application level, said McLean. The eventual goal is to instrument every application by default by making it much simpler to collect data. Today, most DevOps teams are employing a mix of proprietary and open source agent software to collect data that then must be deployed and then integrated with all the applications they build. Given the cost and level of effort required, the percentage of applications that are actually instrumented is, not surprisingly, fairly low.

However, as agent software becomes more readily accessible, the percentage of applications that are instrumented should increase considerably in the years ahead. For DevOps teams that depend on that data to optimize application performance, that capability should prove to be a boon for improving overall observability in IT environments.

Observability, in one form or another, has always been a core tenet of any DevOps best practice. Initially, DevOps teams focused on continuous monitoring as the most effective way to proactively manage application environments. However, it can still take days, sometimes weeks, to discover the root cause of an issue.

Monitoring focuses on predefined metrics to identify when a specific platform or application is performing within expectations. The metrics tracked generally focus on, for example, resource utilization. Observability platforms combine metrics, logs and traces—a specialized form of logging—to instrument applications in a way that makes it simpler to troubleshoot issues without relying on a limited set of predefined metrics focused on a specific process or function.

Those observability platforms then make it possible to correlate events so that it is easier to identify anomalous behavior indicative of an issue’s root cause. Armed with these insights, it becomes a lot simpler for IT teams to resolve issues faster.

It’s unclear when OpenTelemetry tools will become more widely employed, but as more tools for collecting data become available, the impact on DevOps will be nothing short of profound.

October 18, 2021
Why Observability and Monitoring Matter to SREs
Site reliability engineers, or SREs, do many things. They help developers build reliability into applications. They manage SLAs and SLOs. They play a leading role in incident management and incident response.

For all of these tasks, SREs draw heavily on observability and monitoring. Although other parts of the IT organization also typically help to manage observability and monitoring, it’s virtually impossible for SREs to do their jobs effectively in today’s cloud-native world without observability and monitoring tools and data.

Here’s what observability and monitoring mean, why they’re so important for SREs in particular and how SREs can systematically measure the success of their monitoring and observability practices.

Defining Observability vs. Monitoring

Monitoring is the collection of data from applications, infrastructure or other IT resources to track the status and performance of those resources. Monitoring has been a core IT practice since the late 1990s when the first modern monitoring tools, such as Nagios, appeared.

When compared to monitoring, observability is more intensive. Observability is the collection of discrete types of data from multiple systems and the correlation of those data sets to achieve actionable visibility into IT resources. One thing to note: Observability has a more technical definition in the field of control theory that has to do with using external outputs to make inferences about the internal state of a system; however, when people talk about observability in the context of IT and SRE, they usually mean the collection and correlation of broad sets of data in order to understand complex systems.

The debate about the differences between observability and monitoring is ongoing, but the consensus is that they boil down to:
- What versus why: Monitoring helps SREs determine that a problem exists. Observability goes further by allowing them to understand why it exists.
- Context: Observability provides context around what is wrong with a system. Monitoring, for the most part, offers surface-level information about the existence of a problem.
- Actionability: Observability empowers SREs to respond effectively to outages, performance degradations or other problems. Monitoring merely lets them know that something is wrong and that it may require intervention.
In most respects, monitoring is one step in the greater observability process. But observability involves additional practices, such as the correlation of various types of data, to achieve a level of context and actionability that monitoring alone can’t deliver.

Why SREs Should Understand Observability

Again, observability has become a core practice for engineers in a variety of roles. Developers can use observability tools to help measure and optimize application performance before deployment. IT engineers can leverage observability to gain visibility into issues that exist in a production environment. Observability tools can help QA engineers determine why an application failed a test.

But no group has more to gain from observability than SREs. Given that maximizing the reliability and performance of systems is the core mission of SREs, the ability to not just detect problems via monitoring but also to understand them through observability is critical for modern SRE teams.

That’s all the more true for SREs tasked with managing reliability for complex systems, like those based on microservices. In complex applications, identifying the root cause of a reliability issue—such as which specific microservice contains buggy code or which type of request triggers a performance bug—can be very difficult. By maximizing context into reliability problems, as well as allowing teams to trace requests across complex, distributed systems, observability places SREs in the strongest position to find and fix reliability problems quickly.

What’s more, beyond detecting issues in systems that have already been deployed, SREs can also leverage observability tools to help build reliability into applications before they are put into production. By observing applications in dev/test environments, SREs may be able to identify reliability risks and then use that insight to find ways to make the application inherently more reliable. Observability might reveal reliability weaknesses that stem from an application’s architectural design, for instance, or from the orchestration tool that manages the application.

How to Measure Observability and Monitoring

Getting the most from observability and monitoring requires systematic measurement of observability and monitoring initiatives. It’s only by collecting metrics about observability and monitoring outcomes that SREs can ensure they are improving reliability rather than merely collecting and analyzing data to no particular end.

While a full discussion of observability and monitoring measurement is beyond the scope of this article, consider the following types of metrics for assessing these practices:
- Incident frequency: How often do incidents (especially severe incidents) occur that impact production systems? Ideally, SREs will see a decrease in incident frequency that correlates with increased use of observability and monitoring tools.
- Rollback frequency: The less often an application release is rolled back to a previous version, the more effective the SRE team’s observability and monitoring practices likely are.
- Mean-time-to-remediate (MTTR): MTTR is a measure of the overall time it takes to resolve a reliability problem. It reflects how well you monitor (which determines how quickly you detect problems) and how well you observe (which is the key to resolving problems).
- Total data analyzed: Tracking the total volume of data that you collect and analyze through observability and monitoring tools helps you assess the overall scope and scalability of your practices.
- Total systems analyzed: In addition to tracking total data volume, consider measuring how many discrete applications, hosting environments, CI/CD tools and other systems you are managing as part of observability and monitoring workflows.
The ultimate goals of SREs extend beyond observability and monitoring. However, observability and monitoring serve as essential means to the broader ends that SREs are tasked with achieving: Maximizing uptime and performance. And while monitoring alone was once sufficient for supporting the goals of SREs, it’s very difficult to imagine an SRE team today that doesn’t also make extensive use of observability, which takes monitoring to the next level.
October 18, 2021
Trust in Cloud Providers is a Developer Issue
Earlier this summer, we worked with ClearPath Strategies to ask developers about the level of trust they have in their various cloud service providers. What we learned spoke volumes: While developers’ familiarity and comfort with Amazon, Microsoft and Google position these well-known cloud providers as market frontrunners, developers often do not agree with the companies’ values.

The research project gathered insights from more than 800 developers at small and mid-sized businesses, those with less than 500 full-time employees, across a wide variety of industries and geographies worldwide. You can download the summary report here (no signup required).

In ClearPath Strategies’ report, senior analyst Stephanie Fairchild said that “approximately 20% of developers believe that the big three hyperscalers will engage in behaviors that include leveraging their brands to stifle competition, focusing on shareholders more than customer needs, hosting or working with customers that developers find morally objectionable, monetizing user data for their own use and locking them in with proprietary tools. Criteria such as security and performance clearly take priority over these trust issues for developers, but it’s interesting to note that a sizable portion of developers are thinking about how their values align with those of their service providers.”
- Trust plays an important role in developers’ choice of a cloud hosting provider, but needs take precedence; when prioritizing needs or values, developers report they prioritize needs over their values 73% of the time.
- The top three reasons cited for developers’ preference of their chosen third-party cloud provider: Security, trust and quality of service.
- Survey respondents indicated that both hyperscale and alternative cloud providers deliver equally effective performance and security.
The report cited three top criteria for choosing a third-party cloud provider: Security, trust and quality of service. Survey respondents felt that hyperscale and alternative cloud providers deliver equally effective performance and security. Interestingly, although 73% of respondents said they prioritize needs over values when making a purchasing decision, approximately 20% said they don’t trust the big three providers (AWS, GCP and Azure) in what could be considered key moral areas.

It’s no secret that the extensive functionality and brand awareness of hyperscalers make them popular choices for developers, but an alternative cloud provider like Linode, DigitalOcean, OVHcloud or Hetzner can be a more trustworthy partner for innovative upstarts—one that likely offers better pricing and doesn’t threaten or compete with their business. And when an alternative cloud provider meets a developer’s functional needs—which is nearly always the case—developers can have the best of both worlds: A provider they can trust to meet their needs and one that shares their values.

To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021
October 15, 2021