devopsy.kinsta.cloud

Category: Doin’ DevOps

Moving Beyond Legacy API Management Tools and Strategies

What is legacy API management? That’s a question that many organizations haven’t even thought to ask because they don’t realize that the way they manage their APIs is outdated. On the contrary, they assume that if they have API gateways in place to provide security and load balancing for their APIs, they’re already ahead of the curve when it comes to managing and protecting their APIs.

But the fact is that traditional approaches to API management are fast becoming outdated. In a world where the ability to communicate in real-time is more critical than ever, legacy API management solutions can’t keep up, because they’re not designed for APIs that need to share large volumes of data using asynchronous paradigms.

That’s why forward-thinking organizations should be looking for ways to overhaul their API management tools and strategies. They need to embrace next-generation API management solutions, which can do much more than manage traditional, synchronous APIs.

Allow me to explain by walking through the shortcomings of traditional API management tools and explaining what next-generation API management entails.

Legacy API Management

Legacy API management solutions are tools designed to manage and secure synchronous APIs. Typically, these tools come in the form of gateways that sit between backend REST APIs and applications that connect to those APIs using the HTTP protocol. The gateways manage traffic flows and secure communications between the applications and the API backend.

That approach works well if two conditions are true:

1. All of your APIs are synchronous, which doesn’t always work well with continuous data streams.
2. All of your clients can consume data from APIs using HTTP requests.

Traditionally, these conditions were true for most API use cases at most organizations. That’s why the standard approach to API management is to deploy a gateway, configure some load balancing and security policies and call it a day.

Next-Generation API Management

Increasingly, however, not all data is shared synchronously, and not all applications can consume data using HTTP requests. Instead, businesses want to be able to stream data in real-time to multiple types of applications. They might use an open source event broker like Kafka to stream monitoring data from across their IT assets to the tools that IT engineers use to observe and analyze systems, for example. Or, they might need to stream sales transaction data to a fraud detection engine so it can catch fraudulent transactions in real-time.

For use cases like these, legacy API management doesn’t work well at all. Traditional API gateways aren’t very compatible with event brokers because the gateways are not designed to manage real-time data streams. Nor do they typically support protocols other than HTTP, which is a problem because event brokers like Kafka use their own protocols, not HTTP, to enable communication between servers and clients.

Next-generation API management solutions fill this gap. They operate as gateways that can not only mediate between traditional REST-based client and API endpoints, but also support event brokers, asynchronous APIs, and even apply policies at the message level to support message-based communication.

In addition, next-gen API management tools offer the ability to translate between different front and backend protocols. That means that you could have, for example, a Kafka stream on the backend that is exposed to clients using HTTP – effectively allowing apps that can only make requests via HTTP to connect to Kafka data streams, without having to build support for protocol translation into the applications themselves.

Thinking Beyond Internal APIs

At this point, you may be thinking, “We don’t expose data streams externally, so why would we need a next-gen API management solution to support them?”

That’s a fair point. The most basic use cases for data streaming platforms like Kafka involve streaming data internally – from one set of applications to another within the same IT environment, for example.

But this is certainly not always the case. There are plenty of circumstances where you might want to expose a data stream externally and require the security and reliability protections that next-gen API management tools deliver. Maybe you’re a manufacturer who wants to stream data to regulatory authorities so they can monitor your operations remotely, for example. Perhaps you use third-party monitoring or cybersecurity tools and need to stream data from your own systems to that software. Or, even if you are only exposing APIs and data to internal consumers, perhaps that data contains sensitive PII and additional security mechanisms are still necessary. And, for almost all use cases, being able to centrally expose, catalog, and make APIs discoverable is a major leg up.

Even if your business doesn’t currently expose data streams externally, by the way, there’s a good chance it will in the future. As Gartner notes, “Organizations are improving their decision intelligence and real-time applications by tapping the growing availability of streaming data.” The future for most organizations is likely to include more data streams and more complex use cases for those streams, making the functionality of next-gen API management increasingly critical to business success.

Conclusion

In short, it’s time to move beyond the confines of legacy API management – which means no longer being limited to supporting only REST API management. To capitalize fully on the opportunities enabled by streaming data, businesses need a more flexible and dynamic approach to API management, which requires a new breed of API management software.

October 5, 2023
GraphQL API Visibility Ensures Enterprises Can Escape Lock-In

While GraphQL is still early in its enterprise adoption curve, the runtime query language (and replacement for REST APIs) has already built an impressively robust ecosystem for jumpstarting major enterprise-grade deployments. Among the appeals of GraphQL are its open source implementations and breadth of available server types, which combine to offer enterprises flexibility and the benefits of best-fit solutions.

However, operating GraphQL in production, maintaining GraphQL APIs at enterprise scale and achieving the requisite GraphQL visibility and security require complementary tools and services. This is where enterprises can fall prey to vendor lock-in and lose the flexibility to change and control their own deployments if they aren’t careful.

Enterprises need vendor partners to manage their GraphQL deployments effectively, and vendors are glad to offer comprehensive SaaS solutions. But enterprises must understand that a vendor can seek to capture their long-term business by coercion—straitjacketing customers by hosting their entire GraphQL infrastructure and GraphQL gateway and making it, well, not exactly easy to let go. They can unleash particularly nefarious practices, using proprietary features to force customers to migrate to certain technologies or purchase expensive upgrades to keep their crucial GraphQL deployments afloat. Enterprises must be wary of allowing this vendor lock-in to happen for their own good.

Knowledge is essential to maintaining the balance of power in vendor relationships. Enterprises must understand their GraphQL graph and key measures to correctly assess current vendors and maintain agility and control over deployments by not entrusting their entire fate to any one partner.

These are the specific areas where enterprises should seek out intelligence and reliable metrics in order to choose their own future with GraphQL:

Understand Your Overall Graph (and Subgraph) Utilization

Establishing an accurate overview of your graph utilization—and harnessing actionable insights—are crucial, foundational steps that lay the groundwork for deeper analysis and better decision-making. You need to know what graph and subgraph objects and operations your most important API users heavily engage with and which objects and operations are underused or not used at all. You need visibility into who is using GraphQL APIs and how. Getting to actionable intelligence should also tell you where errors are most likely to occur and what’s causing them, as well as where major performance bottlenecks pop up during heavy load events.

With this overview available, the next steps call for building a depth of knowledge in each of these five areas:

1) Understand areas of heavy graph and subgraph usage

Knowing exactly where heavy usage occurs in your graph puts you in a position to upgrade or migrate your GraphQL API server or gateway on your terms while mitigating outage and error risks. Specifically, you want to know which operations in your graph are executed most and how often, which graph objects and fields are consumed most often, and which subgraphs support those most heavily burdened areas in your supergraph. This knowledge and some careful planning help to put successful GraphQL server or gateway changes within your power and also reduce risk when introducing changes to areas of the graph under heavy consumption.

2) Understand where your graph is underused

GraphQL migration and modernization efforts are simpler—and carry less risk—when you first remove any unused functionality and reduce your scope as much as possible. To do so, you need visibility into your graph’s least executed operations, objects and fields, and subgraphs, as well as any operations with zero recent usage. Then, flag low-use graph areas you can deprecate, replace or just remove to simplify your deployment before making changes.

3) Understand your GraphQL API users and usage

You want a seamless experience for your GraphQL API users throughout any changes you make, especially for the users and graph-usage patterns most crucial to your business. That means knowing the frequent users of the areas of your graph under the most heavy usage. You also want to recognize and bolster any GraphQL operations that your most important users frequently call, in addition to the objects and fields those users consume.

4) Understand graph errors and potential error hotspots

The right actionable intelligence will prepare enterprise teams to anticipate and address graph errors. This requires visibility into where, when, why, and how often graph errors occur and error hotspots (for example, high P95 latencies indicative of potential timeouts). Insights into which users feel the brunt of graph errors are also valuable in shaping mitigation strategies.

5) Understand graph performance bottlenecks

Bottlenecks caused by your GraphQL server or gateway itself may call for a shift in your graph’s underpinnings. Bottlenecks can also arise from costly queries of substantial depth and height, especially when these queries see intensive use.

To foresee and address bottlenecks, actionable insights include the average and P95 latencies of your graph’s most-called operations and the heights and depths of your graph queries with the worst performance metrics. You also want to know the number of times that bottlenecked operations get called each day, the subgraphs at the root of bottlenecks, the latency added by supergraph query planning, and which users are impacted the most by bottlenecks.

Graph Knowledge = The Power to Avoid Vendor Lock-In

Implementing the GraphQL API visibility and actionable intelligence strategies covered here help ensure that an enterprise has the means to modernize or migrate as necessary to preserve its strategic independence. Those unfortunate organizations already in the grip of vendor lock-in may need to bide their time but should still use these same techniques to improve and decouple their fundamental GraphQL services from their vendor-controlled GraphQL technology. There’s no acceptable reason why organizations relying on free and open source GraphQL servers and gateways should find themselves stuck in proprietary software arrangements. Thankfully, the right knowledge can deliver enterprises a more flexible future as they modernize API development with GraphQL.

October 4, 2023
ForAllSecure Streamlines Application Security Testing

ForAllSecure today provided early access to dynamic software bill of materials (SBOM) generation and software composition analysis (SCA) validation capabilities within its Mayhem Security automated code and application programming interface (API) testing tool.

Josh Thorngren, vice president of product at ForAllSecure, said the dynamically created SBOM lists only the components present at runtime. That’s a critical capability because, given the rate at which most code is developed today, most SBOMs are out of date shortly after they have been created, he noted.

In addition, Mayhem Security will also now filter results from SCA, static application security testing (SAST) and other similar tools to only surface vulnerabilities that are actually present in the code at runtime.

That approach will substantially reduce the cognitive load that developers experience today when reviewing the results these tools would otherwise generate when applied to the entire code base, said Thorngren.

There’s been an ongoing effort to shift more responsibility for application security left toward developers. The challenge is many of the security tools being provided to developers present an overwhelming number of alerts that, upon investigation, don’t impact code actually running in a production environment, noted Thorngren.

As a result, developers become less inclined to run these tools and miss the chance to discover vulnerabilities earlier in the application development life cycle, he added.

Discovering those vulnerabilities is becoming a more pressing issue as more developers employ generative artificial intelligence (AI) platforms trained using general-purpose large language models (LLMs). While those platforms make developers more productive, they also generate code snippets that have the same vulnerabilities as the code that was used to train them. The result can be an even greater number of vulnerabilities finding their way into production environments than there already are.

At the same time, governments around the world are debating legislation that would make organizations around the world more accountable for application security. As that legislation becomes law, it’s only a matter of time before an exploited software vulnerability results in heavier fines and penalties. It may be a while before those proposals become law, but organizations will need to start shoring up their DevSecOps practices today to address those requirements.

It’s not clear whether developers will ever fully embrace security, but identifying and remediating vulnerabilities can become a lot less painful than it is today. At a time when there is more focus than ever on developer productivity, the less time spent remediating vulnerabilities, the more time there should be to write code. In effect, organizations should be able to adopt secure-by-design principles for application development, noted Thorngren.

Ultimately, any effort to reduce the number of vulnerabilities in applications will go a long way toward bridging the current divide between cybersecurity teams that view developers’ lack of appreciation for security as the root cause of many of the issues they face. At the same time, if it becomes easier to identify and remediate vulnerabilities, fewer developers will view cybersecurity as an obstacle to be circumvented.

October 4, 2023
Avoiding Cloud Lock-In Amidst Potential Regulatory Change

At the end of my street, there’s a beautiful and much-admired garden with vibrant flowers and a well-manicured lawn. Next to this splendid garden is an abandoned vacant lot, long-neglected and overgrown with weeds. Every time I walk by, I think about how this neglected space could be transformed into something magnificent (like this garden), but no one wants to take responsibility for it. Similarly, many CIOs and CTOs have “discarded vacant lots” to deal with in their own work lives: The problem of cloud lock-in.

Cloud lock-in can be a significant headache for business leaders. Even the UK’s communications sector regulator, Ofcom, has expressed concerns about the issue. Earlier this year, Ofcom announced that it would refer the UK cloud market for investigation to the Competition and Markets Authority, highlighting barriers to interoperability and concerns about high egress fees charged by the cloud hyperscalers.

But organizations shouldn’t just wait (and hope) for regulators to solve this problem. As a true believer in harnessing the full benefits of the cloud, I propose two practical ways to combat cloud lock-in.

The Upside of a Technical Approach

The first option is straightforward: Have an honest conversation with your cloud service account leader. Express your concerns about lock-in and negotiate more favorable terms and conditions. This approach might yield some results, but it’s not a guaranteed success. So, there’s a more effective strategy: Adopt a technical approach that empowers you to avoid vendor lock-in altogether, without costing you an arm and a leg.

Imagine that, instead of trying to turn your neglected space into a garden alone, you could access a ready-made gardening community, where multiple people work together to create something amazing. The same principle applies to cloud databases. Rather than relying exclusively on one hyperscaler’s technology stack, you can opt for platforms that offer portability and flexibility across different cloud providers.

Let’s consider databases, a critical underlying component of many mission-critical business applications. Each cloud hyperscaler will offer a range of SQL, NoSQL, and graph database options, which cater to various types and scales of technical problems. The challenge is to avoid getting locked into a single cloud stack, which hinders your ability to switch between clouds if and when needed.

One viable solution is to choose “vanilla Postgres” – the basic open source version of PostgreSQL available from postgres.org or any cloud provider. This will certainly provide initial flexibility, but you are likely to encounter limitations as your application grows and requires enterprise-level features.

The real benefit of the cloud lies in modernizing your data layer to fully exploit cloud-native features like scalability and resilience.

To achieve this, you need databases that run across different public clouds and share common APIs. Born-in-the-cloud technologies, designed from the ground up for cloud environments, offer the most flexibility and benefits.

Examining your current cloud vendor’s offerings is natural, as you’re already on their platform. However, keep in mind that these services are customized to their specific cloud environment only, so this could limit your future growth. If you decide to change cloud providers or adopt a multi-cloud strategy, moving across the cloud database “matrix” can be cumbersome due to the inconsistent APIs.

If you were restricted to a single set of gardening tools when trying to transform that discarded space into a beautiful community garden, it would be nearly impossible and would not be fit for purpose.

You need to realize that these services and their functionality will always need to be customized in some way to the specific cloud environment. AWS’s fully managed relational database engine, Aurora, is compatible with MySQL and PostgreSQL, but it was specifically designed to run on Amazon. This means it can’t run on-prem, and it certainly won’t work in GCP (Google’s cloud).

As a consequence, you will have to make time-consuming and costly changes to your application to accommodate a bespoke version of PostgreSQL that will run efficiently in the cloud environment of your hyperscaler partner. You’re going to be locked into that version of the application because it can’t really work anywhere else. The same applies to the Google equivalent of vanilla Postgres (CloudSQL for Postgres), or the Microsoft version (Azure PostgreSQL).

If you truly want a multi-cloud strategy, you need to consider how you go, for example, from mid-range Google to low-end Amazon. Scaling is much discussed, but surely one of the main selling points of cloud is that you can go down as easily as you go up.

The Business Advantages of Multi-Cloud are in Reach

To achieve the advantages of multi-cloud, you need to consider databases that offer common APIs and portability across clouds. There are several open source PostgreSQL-based database options available, which enable you to keep your cloud database options open and avoid the trap of cloud vendor lock-in.

The wait for regulatory intervention might be long and uncertain. Taking proactive steps now to mitigate cloud lock-in is crucial.

Adopt a scalable open source PostgreSQL option to immediately enjoy many of the benefits of multi-cloud. Embracing this strategy now can spare you potential panic moves when and if regulatory changes come into play.

By freeing your business from the shackles of cloud lock-in, you can unleash the true potential of multi-cloud environments. Just like that “discarded vacant lot” waiting to be transformed into a beautiful garden, your applications can blossom in a multi-cloud ecosystem, which allows you to easily scale up or down and enjoy the best that each cloud provider has to offer.

So, don’t wait for someone else to fix the problem. Take action today and start cultivating your multi-cloud future.

October 3, 2023
The Role of Observability Engineers in Managing Complex IT Systems

The rise of microservices, the adoption of cloud-native architectures and the increasing shift to digital have led to the growing volume of telemetry data (logs, metrics, traces, events) contributing to the complexity of observing modern IT systems effectively and at the right cost. These challenges with processing the telemetry data on time and at the right cost make it difficult for organizations to maximize their observable surface area and ensure they perform at their best.

Enter observability engineers. These professionals are tasked with collecting, processing, analyzing and visualizing data from complex IT systems. They use this data to identify patterns, detect anomalies and understand system behavior. This awareness allows observability engineers to discover and address potential issues before they impact users.

Observability engineers play a critical role in ensuring the reliability, performance and security of complex IT systems, and are essential for organizations that want to stay ahead of the curve in today’s rapidly evolving technology landscape.

Let’s examine the essential characteristics and responsibilities of observability engineers to understand their role.

Characteristics of Observability Engineers

Observability engineers are problem solvers who specialize in optimizing system performance, ensuring reliability and driving actionable insights from telemetry data. They are the driving force behind data-driven solutions and operations. Their specialized skillset and unique focus on observability set them apart from other roles within IT operations. They may have previous backgrounds as system administrators, site reliability engineers (SREs) and IT operations members, who often react to incidents, but observability engineers are proactive and have adopted a shift-left approach to observability. They are adept at understanding the telemetry data and its formats, creating telemetry strategies, implementing comprehensive monitoring systems, and using advanced observability platforms to gain real-time insights.

Another important aspect of observability engineers’ expertise is their in-depth knowledge of telemetry data. They have the necessary skills to collect, analyze and process data such as metrics, events, logs and traces from various sources. They understand the transformations that the data needs for easy consumption by downstream SIEM, analytics, or visualization systems. They are well versed in the challenges posed by microservices architectures, cloud-native environments, and hybrid infrastructure setups and how to extract meaningful information from the system exhaust. This understanding enables them to design telemetry pipelines to move the right data, in the right format, to the right system; build monitoring systems; and implement observability practices that effectively capture, transform and analyze these complex systems.

Collaboration and cross-functional skills are other essential aspects of the observability engineers’ role. They break down the barriers between observability domains, including infrastructure, applications and networking. By coordinating and aligning the efforts of these domains, observability engineers create a more comprehensive understanding of the IT system’s behavior and performance. This approach allows for a holistic analysis beyond the limitations of individual domains and identifies cross-domain optimization and improvement opportunities.

Responsibilities

Observability engineers have many responsibilities focused on addressing problems within IT operations. Some of their key responsibilities are detecting anomalies, troubleshooting incidents, monitoring system health, optimizing resource allocation, enhancing user experiences, enabling data-driven decision-making and supporting compliance and security efforts. Through their skills and expertise, observability engineers empower organizations to maintain highly performant, reliable and secure IT systems.

They are needed throughout the lifecycle of IT systems. They play a vital role in system design and implementation, ongoing maintenance and monitoring, incident response and troubleshooting, optimization, performance enhancement and new feature development and releases. Their expertise ensures that observability is built into the system from the ground up, maintains optimal system performance, minimizes downtime and enhances the user experience.

How Observability Engineers Perform Their Tasks

Observability engineers use telemetry data to learn trends and patterns. Analyzing this data allows them to plan capacity, predict potential issues, and implement preventive measures. This approach enables organizations to continuously improve their IT systems, enhancing performance, reliability and user satisfaction.

To achieve their objectives, observability engineers use various tools for collecting and analyzing data. Telemetry pipelines are emerging as their primary tool for collecting data from various sources within an IT system, transforming it, and routing it to various destinations. These pipelines enable observability engineers to gather data from metrics, logs, events and traces. They process and transform this data to extract meaningful insights and send it to downstream analytics or visualization platforms or for long-term storage.

Various analytics and visualization APM and observability platforms allow observability engineers to analyze and visualize the data. Custom dashboards and visual representations offer them a comprehensive view of system behavior, performance trends and user experiences. With this information, observability engineers can troubleshoot and optimize system performance, finding root causes of issues and implementing effective solutions.

They also contribute to compliance and security efforts. By monitoring and analyzing telemetry data, observability engineers identify possible vulnerabilities and risks within the system. They help organizations maintain robust security practices and ensure compliance with regulations.

Investing in the Promise of Observability Engineering

Observability engineers are increasingly becoming essential for managing the complexity and unpredictability of modern IT systems. Their specialized skills and expertise enable them to address potential issues, optimize system performance and ensure reliability. As technology advances, the demand for observability engineers will only increase. Organizations striving to succeed in their IT operations must invest in these professionals.

October 2, 2023
Yes, Software Development is an Assembly Line, but not Like That

I grew up in a manufacturing town. Nearly everyone worked for (or previously worked for) the local textile factory. My father was no exception. He had a few jobs but eventually landed a management job at the factory and started working closely with the engineers and other plant operators. At the time, I was in high school and about to start my education in software development. At the time, I felt–maybe hoped–that my world and the world my father was steeped in were vastly different. I was working on cutting-edge technology in the software space while he was helping design and run decades-old manufacturing facilities. It took years to learn that wasn’t the case.

Agile Software Development

I was lucky to be able to learn about software engineering while also hearing my father learn about lean manufacturing at the same time. In the beginning, I failed to see the connection between them, but as I learned and read more about this relatively new “agile” thing, I started to see the big picture. It should come as no surprise to folks that agile software development grew out of manufacturing systems developed for exactly the same reasons–to build things more efficiently.

That same desire to deliver value faster at a lower cost drives much of the innovation and growth in every business, including software companies. It’s not surprising that key leaders in the industry found those impressive innovations in industrial engineering and began adapting them to software. We know that waterfall and agile were adopted from industrial engineering, and the next big software development innovation will likely be as well.

Improve Business Efficiency

We can see many of the agile processes directly map back to either lean philosophy or even older industrial processes. Just-in-time manufacturing, or the idea that you keep an extremely limited number of raw or intermediate materials, is a philosophy that focuses on narrowing your manufacturing process to the essentials. Most agile practitioners argue similarly that your stories and shipped features should be as small as possible to deliver only the value required by the user. Both systems aim to improve business efficiency and agility by reducing potential waste (material breakage, for one, and over-engineering for the other).

All of these parallels make perfect sense to me. However, there was one component that never really clicked. A ticket and a widget are incredibly different things. Let’s explore this difference in two ways.

Traditional manufactured goods are usually tangible things. Tangible things often provide value for a fixed amount of time to a single user. For example, a delicious baked snack I pick up on a road trip provides maybe 30 seconds of value when I get back in the car; however, that car may provide 30 years’ worth of value to multiple users. Features in a software platform similarly provide value to users that vary in scale and duration. Here’s where we find an interesting divergence. A good provides a set amount of value and is often produced in bulk; a feature in a SaaS platform is often produced once and can provide an unbounded amount of value.

Compare and Contrast

Another way to see the contrast between the types of products is to look at the differences between engineers in each discipline. In a traditional manufacturing environment, the engineers are not the people actually operating the machines to produce the widgets. The engineers design the manufacturing lines and the products that will be produced on them. In software engineering, the engineers are the ones producing the features. Curious.

However, if we look closely at the two roles, we see that they are actually producing similar things. Manufacturing engineers produce assembly lines and manufacturing processes that can produce those units of value. Software engineers are largely the same, also producing systems and processes that deliver units of value. The manufactured widget of software is actually the discrete user interactions with those features and pieces of software, not the features themselves. The assembly line in software engineering isn’t, as many think, the engineers producing features. Rather, it’s software producing value-creating experiences for users.

While this is interesting, how does it help us? I’ll propose two key ways.

Manufacturing processes are extremely adaptable to managing the technology and platform. Systems like Total Quality Management, which are focused on driving a cultural mindset of continuous improvement and an entire company focused on providing very low defect rates, easily translate to customer satisfaction in software organizations. Just to pick on TQM a bit, if we were to adapt it to software, we would focus on the number of times users are impacted by a defect more than the number of open bugs. Instead of tracking the number of defects and searching for more, we would be tracking the number of users who either failed to receive the promised value from the product or had severely diminished value. That would drive investment not only at the engineering level but would create buy-in from the whole company. Everyone in the company is interested in a conversation about how many users aren’t able to gain value from a feature, but many fewer are interested in how we go from 20 open bugs to 10 open bugs.

Secondly, this mindset shift can help bridge the expectation gap between engineering and the business. I’ve been a part of many conversations where engineering has discovered a potentially serious flaw, and the whole team is in agreement that this is potentially one of the most serious bugs we’ve found in a while. However, when they bring this issue to the business side, that passion gets quickly overruled if engineering can’t show that that bug is “impacting customers.” Those debates often fall into conversations of projected odds better left to the sportsbook room in Las Vegas. While there are obvious exceptions for security exploits, compliance issues, and other business risk-driven defects, the business is justified in not investing in fixing a defect that users aren’t impacted by.

Reporting Performance Issues

Conversely, this line of thinking can help engineering convince the business that something is a critical problem and needs to be addressed. While previously, I would have reported performance issues in terms of p99 times or average load times, that doesn’t describe performance in terms of our unit of value. If our unit of value is a customer using our app or a feature, then each negative interaction is a broken widget. As an example, I’ve since taken to reporting performance issues in terms of the number of sessions with a frustratingly long load time and the number of clients and users that are experiencing those.

Manufacturing is perhaps the most varied industry in existence, adapting itself to produce nearly everything from food to raw materials to consumer electronics to clothing. That long history, many contributions from a massive number of brilliant engineers, and significant innovation have led to dozens of amazing systems for delivering value. While I proposed two examples of how we should give these processes a second look as engineering and business leaders, it’s worth dusting off some manufacturing engineering books and giving them a read yourself. You may find the key to outpacing your competitors was drawn up by an industrialist decades ago.

My first college internship was at one of the factories my father worked at in the engineering department. I was able to see firsthand how deeply some of today’s engineers think about these types of production processes. They really know how to ship stuff, and software engineers may have a lot to gain by learning from them.

Now, go build great things!

September 28, 2023
Fortifying the Castle: A Quest to Secure the SDLC

Every business is a software business today. But as the pressure to churn out new applications and features grows, so do the chances of developers making mistakes. Tackling these challenges doesn’t just require more diligent developers, it requires continuous education across the software development life cycle (SDLC).

To borrow a metaphor from Arthurian legend, it’s a never-ending battle against hidden risks and formidable adversaries. It will require security champions to share their wisdom and bring order to the kingdom.

A Kingdom Under Siege

These are dark times for those fighting the good fight. The threat landscape is a wild and unruly place where risk is everywhere, and adversaries often have the upper hand. They have the advantage of surprise and are masters at hiding in the shadows. But the threat is real. One vendor blocked over 146 billion threats last year alone.

Many of these attacks take advantage of vulnerabilities in code. It’s a problem made more acute by the widespread use of open source components by today’s developer teams. One estimate suggested that last year alone, developers made 3.1 trillion requests for these components from the top four open source ecosystems. Yet many of these components contain vulnerabilities. According to the Linux Foundation, the average application development project features 49 vulnerabilities across 80 direct dependencies. Worse are the indirect dependencies that are far harder to track and, by some estimates, account for six in every seven project vulnerabilities.

That makes software supply chain attacks and breaches far more likely, especially as threat actors are actively placing buggy components in upstream libraries with the hope they’ll be downloaded en masse. It’s also getting more likely because developers are coming under increasing pressure to deliver. It’s claimed that over half of developers handle a hundred times more code than they did a decade ago. And most are under pressure to write code faster. This increases the chance of mistakes happening, which in turn can cause significant financial and reputational damage.

Fighting Back

The answer to these mounting challenges has traditionally been to focus on “shifting left” or moving testing and scanning earlier in the SDLC. To ensure products are designed with security from the outset, it’s essential to follow “secure-by-design” principles. However, this is only half the picture. Scanning for vulnerabilities is certainly important, but even better is to ensure teams are building more robust code from the outset. Prevention is always cheaper and more effective than cure.

This is where continuous secure coding education comes in. Continuous because technology and market demands are always changing. In this dynamic world full of risk, a one-and-done approach simply doesn’t cut it. Further, courses should be extended beyond the realm of the developer to every member of the SDLC team: From project management and UX specialists to QA, product management and beyond. All must share a common goal with a common responsibility to produce more secure code. Some will be security champions who share best practices with other stakeholders. But everyone must be pulling in the same direction.

Time to Fortify the Castle

So once the kingdom has assembled its diligent developers and security champions, what’s next? With the knowledge and skills imparted through continuous training, plus additional automated checks, real progress can be made to enhance security and resilience.

If you’re familiar with the OWASP Top 10, you’ll recognize concerns such as server-side request forgery, misconfiguration, using vulnerable components and shortcomings in security logging, cryptography and software integrity. A stronger focus on foundational security practices like the principle of least privilege, separation of concerns and layered defense can mitigate these risks. Among the most prevalent threats are injection attacks, which include SQL injection and cross-site scripting. However, by using tools and having developers well-versed in strategies like input validation, parameterized queries, data cleansing and secure LDAP queries, these threats can be effectively addressed.

Above all, it’s important to remember that everyone has a part to play in fortifying the castle, in protecting the kingdom from a mounting and formidable agglomeration of threats. That means, as a baseline, that everyone should be getting some form of continuous secure coding training. There’s no time to waste.

September 26, 2023
Why Data Silos Are Killing Your Developer Productivity

Every company is at risk of fostering data silos. In fact, a study conducted by Forrester Consulting for Airtable across employees in the U.S. and the UK revealed that, on average, knowledge workers spend 30% of their week looking for the information they require to do their job and the difficulty in finding and accessing data is the top reason why they feel disengaged from work. What causes essential data to be inaccessible in the workplace? We will explore the root of modern data silos, why they drain developers’ productivity and how to break away from these silos with better observability in your organization.

What are Data Silos?

A data silo is a collection of data held by a group that is not easily or fully accessible by other groups in the same organization.

Data silos can be a byproduct of organizational structure. When different departments or business units in an organization operate independently, they develop their own way of collecting, storing and managing their data. The processes to access that data from other business units can be cumbersome or even nonexistent. Sometimes, the individuals responsible for the data will even be against sharing it for fear that it would be misused or misunderstood outside of their particular unit. The difficulty in accessing information then leads to independent data collection from other business units and ultimately creates other data silos and multiple sources of truth.

Data silos can also happen during times of growth for an organization. New tools are necessary to support the growing needs of the business and are added to the list of tools already in use. Over time, this list grows, and systems that were designed to operate in isolation can create data silos by limiting the ability to integrate with other systems. A recent survey of marketing organizations by Emarsys showed that the surveyed organizations are using an average of 91 marketing cloud services. These 91 tools undoubtedly create data silos unless they are fully integrated together or into a central system.

Why are Data Silos Killing Your Developer Productivity?

Data silos aren’t only slowing your developers down by limiting their access to the data needed for development. They also put a lot of extra work on their plates. And it goes without saying that this additional time spent on data management reduces the time available for improving your product.

Management of Redundant Data
Multiple teams in the organization storing their data separately means increased workloads for developers, who need to update and maintain separate copies of the same data.

Inconsistent Data Quality
Data gaps or quality issues happen more frequently since different parts of the organization have their own standards for collecting, cleaning and validating data. This can lead to development delays or errors in the product.

Time-Consuming Integrations
Developers have to code a lot of integrations to connect all the different tools your organization uses and keep the data flowing between them.

Inefficient Collaboration
When different teams work in silos, the chances of misunderstanding are greater, leading to development delays and misaligned priorities.

Difficulty in Data Governance
Maintaining high standards of data privacy, security and compliance is time-consuming when the data is spread out in different silos and can create costly regulatory risks.

Steps Leaders Can Take to Break Down Data Silos

Fortunately, there are steps that you can take to break down the data silos in your organization, relieving your developers at the same time. Each step may have its own set of advantages and challenges, so the solutions you choose ultimately depend on your organization’s needs and objectives.

Create a Unified Data Strategy
Set clear goals and standards for data collection, storage, management and analysis at the company level. You should also implement data governance and security policies.

Adjust the Culture
Foster a culture of collaboration and data sharing across the company by encouraging teams to share data, insights, and best practices across different departments. You can achieve this through regular communication, training and incentives for cross-functional collaboration.

Establish Cross-Functional Teams
Bringing together members of different departments to collaborate on data-related projects can prove invaluable. These teams should have a shared understanding of the organization’s data strategy and should be empowered to make decisions about data collection, analysis and sharing.

Invest in Data Integration and Interoperability
Invest in tools and technologies that promote data integration and interoperability across different systems and departments. Adopt standard data formats, implement APIs and data connectors or use data integration platforms to make observability collaborative. Make sure that the tools you choose to add to your company have various native integration capabilities to avoid custom integrations as much as possible.

Congrats, you now know all about data silos and how to break them down! The next step for you is to identify the data silos in your company and put a plan in place to get rid of them. Your employees and coworkers will thank you!

September 25, 2023
There’s No Value in Observability Bloat. Let’s Focus on the Essentials

The observability market paired with cloud security has been predicted to reach $70B by 2026, but that market prediction is a myth, and I’ll tell you why.

If observability was a grand buffet and every company needed a feast of features and capabilities of this proportion, maybe this prediction would be valid. But most companies don’t need an over-the-top feast. Legacy APM and observability solutions are serving up an all-you-can-eat buffet when the reality is that most companies only need the essential ingredients to satisfy their observability strategy. And the cost for that can be much more modest. The $70B observability market prediction is, therefore, suspect, considering the true needs of most organizations.

Complex cloud ecosystems, the rise of applications built on microservices and Kubernetes, and the overall scale and complexity of applications have led to exponentially more data being created in the pursuit of observability. Storing and processing all that data is expensive, and as the volume of data balloons, so does the cost.

And while the complexity of business-critical systems and the volume of data have scaled, so have the overall cost of those legacy tools—in direct proportion. Partially, this is driven by the features and instrumentation added that only contribute to the bloat and valueless telemetry.

Customers aren’t seeing value in any of this bloat, and for the most part, all that’s been accomplished is increased complexity in deploying and operating the system, with a commensurate lengthening of time to value. Plus, pricing models haven’t significantly changed despite the advancements in the market. The fact is, organizations can’t really say that they have gotten more insight into the overall performance of their mission-critical systems. What they are saying, however, is that these tools are becoming overwhelming to manage and overwhelmingly expensive, and developer productivity is not increasing.

What organizations need and want are real-time insights into the health and performance of their cloud infrastructure and applications–specifically the confluence of logs, metrics and traces within the context of modern observability. There is little tolerance for software bugs, slow web experiences, crashed apps and other service interruptions, and the responsibility lies on developers, engineers and ITOps teams to quickly resolve production issues before they impact customer experience.

There is a call for observability, but rather than an explosion of features and noise, what’s being called for is a narrowing of focus.

Observability Has an Efficiency Problem

Telemetry data gathered from the distributed components of modern cloud architectures needs to be centralized and correlated for engineers to gain a complete picture of their environments.

Engineers need a solution with critical capabilities such as dashboarding, querying and alerting, and AI-based analysis and response, and they need the operation and management of the solution to be streamlined.

What’s important for them to know is that it’s not necessary to spend more to ensure peak performance and visibility as their environmental complexity grows.

Legacy players experiencing some maturity in the market may have conditioned them to believe this is true—they have set a standard for higher costs correlating with incremental amounts of data, no matter the quality of that data. Companies are being fed the lie that all this data may be relevant and they’ll be doing themselves a disservice by neglecting even one byte of it.

Data Does Not Inherently Equal Insights

No doubt, more data is being generated, but most of it is not relevant or valuable to an organization. Observability can be optimized to bring greater value to customers, and that’s where the market is headed.

Call it “essential observability.” It’s a disruptive vision to propose a re-architected approach to observability, but what engineers need is a new approach making it easier to surface insights from their telemetry data while deprioritizing low-value data. Costs can be reduced by consuming only the data that enables teams to maintain performance and drive smart business decisions.

Essential observability is cutting out the fluff and only focusing on—and paying for—the impactful data. It’s about arming engineers with meaningful insights so they can quickly resolve production issues and business disruptions. Organizational performance increases without inflating data complexity and resource consumption.

The Essential Observability Revolution

Observability tools have evolved beyond the model of linear price increases as complexity and data scales. For companies (especially growth-stage organizations) that want a partner to help them make more informed decisions and keep their costs more reasonable (i.e., stop recklessly spending for useless data), they need to focus on tools offering them the essentials.

Solutions are available, offering targeted features and automated capabilities out of the box, delivering the insights needed to understand and fine-tune data. These solutions provide continuous support, reduce complexity and accelerate time to value.

The next revolution in observability isn’t features and functionality but providing insights to make data-driven decisions that keep essential observability cost-effective while not compromising the value derived.

September 22, 2023
Why Enterprises Should Embrace Data-Driven Software Management

Many Fortune 500 companies have recently undergone extensive layoffs and now find themselves having to achieve more with fewer people—while grappling with how to integrate generative AI into their products. They must build while economizing, streamlining and increasing efficiency in every part of their company. Yet one of the most vital cogs of the machinery has struggled to evolve over the past 20 years.

In fact, until now, managing high-tech engineering teams at the world’s biggest companies has not been all that high-tech.

Back in the 1970s, the “waterfall” model was introduced: An inflexible process in which each phase of software development depended on the previous one’s completion. Then, around the 2000s, companies started moving towards “agile” models that tracked progress in two-week sprints.

Now, a third era of software development management is emerging. It’s not waterfall or agile; it’s data-driven.

Current management systems cannot give managers real-time, granular insights so they can make decisions on the spot and run projects in the most efficient way possible. However, data-driven models generate live data on how teams are progressing which has far-reaching effects on businesses’ productivity and future survival.

The emergence of DORA, a framework for tracking development metrics makes this possible. For the first time, new tools enable us to capture the software development life cycle automatically, without any manual input from the engineer, leading to fewer roadblocks, a 20% faster cycle time and greater ROI.

Despite this, only a handful of Fortune 500 companies have embraced data-driven metrics. Here are the reasons that shift should happen now.

Real-time Insights are Game-Changers

Until now, companies have been using inaccurate methods to track their software development teams. Project management reports on engineers’ output are reviewed every two weeks, monthly, or even quarterly. This means that impactful changes are made at a delayed point in time rather than when they can make the greatest difference to cycle time.

In contrast, DORA measures specific real-time performance indicators: Deployment frequency (DF), lead time for changes (LT), mean time to recovery (MTTR) and change failure rate (CFR) (For more information, see here). These far more data-driven metrics measure efficiency and velocity at every key point of day-to-day software development, with feedback to management happening continually.

Often, individual engineers won’t have enough capacity or oversight to inform managers of specific issues during a cycle. Using DORA metrics is like being able to detect that a team has an abnormally high BPM though they are in a rest phase. It’s a signal of illness; that more or different resources are needed. That then requires you to go in and inspect the micro factors that can be adapted.

For any Fortune 500 company, this quick turnaround is vital. Any small issue in a product cycle can easily escalate and become a critical problem in a matter of days. Instead, using DORA can accelerate software development and, ultimately, time to market.

DORA Gives you Internal Benchmarks

But, how can you tell if something is wrong while using DORA? In the first few months of adoption, it’s likely that companies will compare their internal data to industry benchmarks. Unprecedented data has recently been published that offers general performance benchmarks for each DORA metric based on industry research.

But the end game is to be able to compare your DORA metrics to your own internal benchmarks. With time, each company will develop its own personal baselines that better represent internal capabilities and limitations based on innumerable variables.

The purpose of DORA benchmarks, however, is not to draw a fixed red line. Think of it more like a speed limit that ensures that people are working at a steady pace while allowing for flexibility.

A clear sign that your team may be burnt out is a really low cycle time. At times it can be a technical bottleneck, but the beauty of DORA is that it can also point you towards well-being issues.

DORA Needs to be Coupled With Granular Insights

DORA is unparalleled when it comes to flagging managers that something is wrong. DORA metrics can flag that a team has a high cycle time – for example, an abnormally long deployment frequency suggests that engineers are working more slowly or that there may be a build-up of code that isn’t being uploaded to the repository quickly enough during the review process. At that point, you need to zoom in to get more information. There are multiple metrics that provide insights into the exact pain points in a team’s development, and you can develop these internally or via tools or engineering management software that provides more holistic oversight.

In this particular case, you can use a granular metric called “time to merge from first review” to see how fast engineers incorporate feedback from their peers during code reviews. If it is taking a week for the first review to be carried out, integrated into the code and finally uploaded to the general platform, then perhaps there’s a bottleneck occurring during the review process.

Perhaps the most important benefit of Engineering Leadership 3.0 is that it no longer has to be a guessing game. If they sense something’s wrong, they can check their DORA dashboard, receive confirmation of the problem, hone in with granular metrics, and take corrective actions immediately.

Teams that adapt their behavior as a result of these data-driven metrics will add incredible ROI to the company. But these changes can’t happen overnight. It will start with getting buy-in from engineering managers and the executive team. For Fortune 500 companies, that often requires elaborating action plans, integrating external tools to operate the tracking process and training engineering managers (in our case, this takes up to five weeks). But once they’re a part of the business, they will become indispensable in running smoothly without stacking up issues.

September 15, 2023