devopsy.kinsta.cloud

Author: Bill Doerrfeld

Does GraphQL Introduce New Security Risks?

The GraphQL query language is an excellent tool for increasing the ease of data sharing. The premise is that you request the fields you need in a single bundled request, avoiding multiple network calls. Due to its usability, GraphQL is a growing area of development. Yet, as with any new technology, it comes with caveats.

REST APIs function as a request-and-response paradigm in which the receiver accesses one resource at a time. On the other hand, GraphQL exposes many fields at once, requiring additional rate-limiting and time-outs to avoid nefarious queries. Additionally, the ability to traverse the entire schema could be abused by attackers to expose intricate data linkages. There are also some usability features with GraphQL that could be abused if misconfigured.

GraphQL is a friendly alternative to REST. But how do we handle the security implications of GraphQL? I recently chatted with Mike Benjamin, vice president of security research, Fastly, to dive deeper into the security repercussions of adopting GraphQL. Below, we’ll consider the unique insecurities associated with GraphQL and suggest specific steps to limit the impact of these risks.

Four GraphQL Security Vulnerabilities

1. Introspection

GraphQL provides built-in documentation that tells developers what the fields are, providing a schema of everything you can query. You’ve essentially given the entire dataset schema to a developer, meaning they can perform introspection to capture attack surface information. Hackers will most commonly look for injection or mutation capabilities that they can exploit, explained Benjamin. Or, they might seek to leverage the interlinkages between data to map a way through to classified information.

To mitigate this problem, Benjamin cautioned against over-exposing all documentation publicly. Instead, he suggested using a restrictive security model for external viewers. “Provide appropriate user documentation in a different method,” he says. This may end up looking more like the standard REST API view of documentation.

2. Field Suggestions

Another usability benefit in GraphQL could also be a potential security concern—field suggestions. Field suggestions occur when you don’t enter the correct field name or enter a partial input and the system returns a hint. You might type in 0 or user, and the system makes a suggestion to fill out the rest of the field.

This user interaction improves the developer experience, but, unfortunately, it could be abused by a hacker, said Benjamin. Thus, he recommended disabling hints after initial development. Or, if you decide to keep it in, he recommended placing a limit on its use. For example, if you try this more than five times in a second, the system could flag it as a bot.

Right now, this attack vector is pretty limited, but Benjamin anticipated that, over time, hackers could automate this introspection with mass scanners to divulge more information.

3. Denial-of-Service-Oriented Attacks

Without the proper guardrails, all web APIs are prone to denial-of-service (DoS) attacks. Such an attack may pummel a service with a million queries per second to shut it off completely. In the case of GraphQL, these queries can dive very deep in a nested data relationship. A social network, for example, might expose friends of users, friends of users’ friends and so on.

Or, hackers might utilize GraphQL’s bulk query ability to perform reconnaissance, brute-forcing the enumeration of data to query hundreds of user IDs to see which are valid. If you’re allowing unlimited data retrieval bundled within a single request, you could be inviting hundreds of bundled queries that could expose large swaths of the underlying database in one fell swoop.

To counter these actions, Benjamin recommended limiting the testing depth. This could involve placing limits on how long a query can run, which would cause a time-out for requests that require suspiciously high degrees of processing. Another option is to limit how deep the bundling can get, said Benjamin. For example, perhaps the system restricts going three levels deep into a graph database, as that could expose millions of interconnected endpoints.

4. Lack of Object-Level Authorization

Another vulnerability that GraphQL endpoints may be prone to is a lack of object-level authorization, which hackers could leverage to get at a datapoint through its edges, said Benjamin. Since the idea is that you can access any data from one all-encompassing endpoint, authorization rules are often too abstracted away from the actual data. Queries may be able to access nested data or query relationships between data, too, which may unintentionally divulge information.

Benjamin recommended placing authorization closer to your objects or data to rectify this issue. Use authorization-level controls that align permissions structure and roles with users. “Everyone needs a common authorization model,” said Benjamin.

Future Evolution: GraphQL Security Matures

As I’ve written before, the greatest strength of GraphQL is also its greatest weakness. Using GraphQL with default settings is a security trade-off in favor of usability. But according to Benjamin, we’re seeing more GraphQL servers disabling some introspection features, which he sees as a very positive move and a sign of maturity of the base configuration.

GraphQL might suffer from similar threats that all APIs are vulnerable to, such as XSS, injection, DDoS attacks and broken authorization. However, Benjamin stressed that the added complexity of GraphQL makes it a unique case. “The most important thing for folks to consider is that GraphQL is more complicated than a REST endpoint—much more complicated,” said Benjamin. “GraphQL is unique in that it has linkages of data which contributes to complexity. It takes a little more time to think about how that’s confirmed, how it’s presented and how it would be attacked.”

In other words, with great power comes great responsibility.

Although exploits in public GraphQL APIs are still rare, we should stay vigilant. Benjamin encouraged developers to continue reading about, understanding and thinking about these attack vectors when designing their interfaces, as these risks will become more prominent as GraphQL usage grows and hackers become more familiar with these specific frailties.

May 10, 2022
Smoothing the Transition From REST to GraphQL
Since its open source release in 2015, GraphQL has taken the development world by storm. GraphQL, the query language originally developed by Facebook, is a big boon for frontend developers, as it enables you to grab the exact data you need with a single request. This solves the problems of over-fetching and under-fetching, which classical REST APIs are prone to. GraphQL also offers a nice developer experience with a consistent schema and documentation. Some are even treating GraphQL as a convenient meta layer to aggregate multiple underlying services and abstract their interfaces with simple commands.

For these reasons and more, many development teams are considering switching from REST to GraphQL. Even enterprises are beginning to use GraphQL in large production environments. But, how can these teams make the switch without breaking existing integrations? How can we ensure that things like developer experience, security, and access control all remain stable?

I recently met with Anant Jhingran, the co-founder of StepZen, to discover tips for transitioning from REST to GraphQL. According to Jhingran, most organizations see the benefit in adopting GraphQL but are hesitant to forgo years of investment into RESTful architectures. But according to Jhingran, it’s rarely a case of “either/or”—more often, he sees projects running both styles simultaneously or layering one on top of the other. Below, we’ll explore some things to keep in mind as you invest in GraphQL.

The Decade of Developers

We’ve reached ubiquity of web APIs among companies big and small. “APIs have turned out to be phenomenal for software supply chains,” said Jhingran, the former CTO of Apigee. They help share data between partners and can enable platform ecosystems to thrive. Jhingran describes how within the last decade, APIs and API management have peaked to near-ubiquity in terms of adoption.

But not all APIs behave the same way. Although representational state transfer (REST) is the most popular style, it’s still quite a general architectural guideline for web APIs—how the data comes out of each service is still highly nuanced. The structure of requests and responses are different from service to service, causing developer experience hangups at scale.

GraphQL, on the other hand, “is a fantastic way of structuring data,” said Jhingran. With a single call, you can return the explicit information that the client requires in an expected format. For an e-commerce application, for example, this might be customer information, order numbers and delivery status. Displaying this all with one call is an awesome experience for frontend developers. And, GraphQL APIs are easy to explore and navigate since the schema is baked in. This improves upon REST APIs, where documentation may not always match the actual production implementation.

“This decade is the decade of developers,” said Jhingran. Satisfying developer experience has become more mission-critical as enterprises work to increase their footprint in the cloud marketplace and GraphQL is part of this movement. “For people who build apps, it’s just better,” said Jhingran.

But just because frontend teams clamor for the ease of use of GraphQL doesn’t mean backend teams can easily restructure their entire framework around it. Doing it right requires many skills, and the journey can be arduous, explained Jhingran.

It’s Rarely REST Vs. GraphQL

REST and GraphQL are often pitted against one another to the point that they seem like polar opposites. But in practice, teams often use GraphQL to build on top of APIs already in place. Jhingran describes the most common pattern as “GraphQL comes in as a layer that connects the dots between various REST APIs.” Either GraphQL is consumed directly or as a REST layer above GraphQL. “It could go either way,” he said.

GraphQL is a fantastic data infrastructure tier as it can mix and match database backends. It could, in essence, become the new plumbing upon which applications are constructed. But while GraphQL may be excellent for internal construction, the name of the game is stability when externalizing services to the outside world. With REST, the big goal is to decrease broken clients and ensure high-grade security for each resource. Therefore, Jhingran foresees certain use cases where using a REST wrapper around GraphQL services to deliver fixed JSON responses would be preferred.

Another point is that GraphQL is not only relegated to appeasing frontend developers—it has some backend potential, too. “GraphQL is an opportunity to reimagine frontend and GraphQL is an opportunity to reimagine backend infrastructure,” said Jhingran. This concept has weight nowadays, as Netflix engineers, for example, use GraphQL as a backend for microservices architecture.

Making the Transition From REST to GraphQL

Not all apps are going to be transformed on day one. Teams may encounter cultural resistance to new technologies. Interface redesigns, especially for those external apps which are exposed to third parties, must also require extra precaution to maintain app compatibility to avoid breaking change.

Thus, it may take time to transition effectively. “Getting it started versus getting it right are not the same thing,” said Jhingran. He said people who get started with GraphQL may struggle for months due to competing best practices and tools. For example, Jhingran identified two main perspectives when approaching GraphQL: Programming and declarative.
- Programming: Teams use GraphQL libraries such as Apollo to program their way out of your GraphQL endpoint. In this approach, you just start writing. It’s quick to get started, said Jhingran, but you end up describing everything you do and encountering many contingencies along the way.
- Declarative: The other option is a more declarative approach—instead of telling the system how, you tell it what to do. For example, this subsystem should do X, this subsystem should do Y and so on. According to Jhingran, adopting tooling with declarative snippets to create GraphQL endpoints is “better in the short and long run.”
GraphQL Security

A March 2022 report from Salt Labs found a shocking 681% increase in API-based cyberattacks over the previous 12 months. There are many concerns surrounding API security right now—these valuable endpoints are prone to broken access control, data overexposure and over-permissive states.

As I’ve covered previously, GraphQL security is a bit nuanced compared to REST. Within the REST paradigm, you can easily assign permissions and rate limit calls per resource. But with GraphQL, it’s not that simple. Since you can batch multiple queries, callers have much more power at their fingertips, and hackers might abuse such power if the proper authorization isn’t in place.

Companies may rightly worry about opening another can of worms with GraphQL. But according to Jhingran, these worries are all solvable. A comprehensive GraphQL endpoint will likely require another layer to uniquely segment authorization to retain a least-privilege model for all resources and backend endpoints.

Future of GraphQL

To date, GraphQL has gained modest prominence. But, Jhingran foresees an incoming inflection point in the next two to three years. “The dominoes are beginning to fall,” he said.

Now that the market is saturated with APIs, characteristics outside of functionality will be what sets a service apart and gives it a competitive edge. This includes things like performance, security and developer experience, the latter of which makes GraphQL an attractive proposition. This, paired with the explosion in GraphQL tools, will undoubtedly help developers perform their jobs better.

Looking to the future, Jhingran predicts increased usability and abstraction in how developers work with GraphQL. He compared it to how disjointed pages on the web became interconnected and searchable with links. We see similar thinking happening in the world of APIs, he explained, in which the disjointed APIs require a better connection. Similar to how Google simplified search, he foresees GraphQL and the tools that work with it helping to decrease cognitive load by abstracting complexity with declarative commands.
May 5, 2022
Shedding Light On Toil: Ways Engineers Can Reduce Toil

It comes in many shapes and sizes and is embraced as a necessary evil. It lurks in the shadows, emerging now and again to stealthily creep into our workflows, where it feasts on our perceived shame. If not pruned, it grows and wraps its tendrils around the workforce, draining the energy out of every digital worker across departments. We all have it and most of us believe we suffer alone.

I’m talking, of course, about toil.

Dramatics aside, toil is common throughout most programming and DevOps positions, whether we like it or not. And when it comes to platform engineering, the chances of encountering toil is high. You can think of toil as those tedious workarounds that should be automated but aren’t. This could be due to a lack of standard configurations for deployments. Or, perhaps engineers must copy and paste data from one module to the other—an integration that has not yet been automated.

One of the hallmarks of the SRE role is to spot and reduce such toil—hopefully before it gets out of hand. But, what exactly is toil? And, how can SREs reduce it?

I recently met with Jake Englund, senior site reliability engineer, Blameless, and Matt Davis, intuition engineer, Blameless, to uncover some tips on spotting and eliminating toil in the daily life of an SRE. Below, we’ll attempt to define toil and explore some techniques to defeat this ghoulish foe.

First, What is Toil?

Toil is tedious. Englund defines toil as repetitive manual tasks that are, ideally, automatable. “It’s not gratifying work; it lacks enduring value.” James Curtis similarly described toil as “tedious actions that really have no enduring value.” Toil is similar to technical debt in that, if left unchecked, it can hinder productivity and contribute to burnout.

Toil is invisible. Toil is the invisible work that goes on behind the scenes, says Davis. For example, at your team meeting or stand-up you might say something like, “I’m deploying this fix.” There’s usually no need to explain the nitty-gritty details of your exact step-by-step process, as it would be too far in the weeds. As a result, manual and repetitive tasks are often not given the spotlight.

Toil is persistent. Toil might be readily apparent, but it’s not as simple to eliminate; attempts at fixing toil could potentially create new toil. Tedious tasks may also be essential to keeping core infrastructure pieces up-to-date. According to Englund, what often ends up happening is that only the specific engineer knows the niche mechanics of what they’re maintaining, making them irreplaceable—a power that some employees might abuse. “If you’re the only one who knows how to do something and you’re hoarding that information, it leads to inertia against resolving toil,” said Englund.

Toil is pervasive. Toil is not limited to SRE or engineering. From UX to management or marketing, within most positions, you’ll encounter a degree of toil. This is because the ‘work as prescribed; is often much different than ‘work as done’, said Davis.

Four Tips To Reduce Toil

So, what can organizations do to conquer the nightmarish reality that is toil? Especially those that don’t have a dedicated engineering team to reduce drudgery?

1. Find It!

To reduce toil, you first have to find it. Sounds simple enough, right? Well, discovering toil might not be as obvious as it sounds. “You get into these toil situations where you might not even know you’re in it. How do you know that toil is encompassing what you’re doing?” asked Davis.

To find toil, the Blameless team has a weekly session open to the whole company to examine ‘work as done.’ According to Davis, this has been very beneficial for discovering where toil occurs. It also helps garner perspectives from roles throughout the entire company.

2. Standardize and Automate

Next, a proven method to reduce toil is standardizing your engineering processes. Standardization, said Davis, is “another word for removing ambiguity.” For example, SREs can benefit from having a common series of steps to follow when responding to alerts. Runbooks that aggregate data to respond to issues can reduce toil in the data collection process while also helping improve mean-time-to-discovery and recovery rates.

Standardizing change management is another way to reduce toil. If engineers are approaching change management in various ways, this could cause additional review and slow down the release process. Removing ambiguity from the change lifecycle can alleviate toil down the line.

When you find yourself having to retain mental models for configurations, it’s a good sign that standardization is required, explained Englund. For example, perhaps some configurations are needed in some environments but are commented out in others. Documenting the specific configurations for different development environments (and automatically populating configurations) is one way to reduce toil and the chance for human error.

3. Proactively Monitor

More proactive monitoring is another way to reduce toil, according to Englund and Davis. “Responding to a crash loop is responding too late,” added Davis. Instead, he advocated that SREs look toward leading indicators that suggest the potential for failure so that teams can make adjustments well before anything drastic occurs.

If SLIs like error rate and latency are getting bad, you must take reactive measures to fix them, causing more toil. Instead, proactive monitoring is best to see the cresting wave before the flood. Leading indicators could arise from following things like data queue operations connected to servers or the saturation of a particular resource. “If you can figure out when you’re about to fail, you can be prepared to adapt,” said Davis.

4. Practice How to Improvise

One major caveat of standardization is that you’re inevitably going to encounter edge cases that require flexibility. And when an outage or issue does arise, the remediation process is often very unique from case to case. As a result, not all investment into standardization pays out.

Alternatively, teams that know how to improvise together are proven to be better equipped for unforeseen incidents—one study on Arctic Sea emergencies found collective improvisation to be imperative for emergency response situations. Similarly, collaboration is vital for software teams. Davis calls this the “practice of practice.” If you don’t practice backup restores, for example, teams may flounder in the case of a database outage. Yet, since every remediation effort is unique, it really boils down to learning how to collaborate effectively as a team. “We can gain a lot from getting together to practice,” added Davis.

All the Small Things

Toil is like manually typing an email signature every time you are about to hit “Send.” Maybe you’ve just been putting off configuring your signature in your email client. Or, perhaps you like to write a custom sign-off for each recipient. Regardless of the reason, the longer this repetitive motion is required, it’s a stick in the spokes of productivity.

In the grand scope of things, this tedium can be a drain on time and resources. For every extra keystroke or click, every un-automated workflow or every copied-and-pasted code block, a small degree of toil is piled onto the daily workload of an engineer, dragging them away from focusing on what really matters.

Standardization, automation, proactive monitoring and collaboration are all helpful elements to consider on your journey toward reducing toil. But of course, there is the trap of increased software development automation—the time saved building automation must outweigh the upfront development cost (and ongoing maintenance effort). Otherwise, you might end up creating even more toil.

May 2, 2022
What’s the Difference Between SLI, SLA and SLO?

If you’re just getting into site reliability engineering (SRE) or platform engineering, you’ve probably come across a bunch of new terminologies, like SLI, SLA and SLO. These benchmarks are commonly referenced in the day-to-day life of an SRE but may seem foreign to outsiders. So, what are the differences between these abbreviations?

Well, one of the hallmarks of the SRE approach is to gauge performance and set targets for application metrics. These targets can be thought of as reliability thresholds. If a site’s latency reaches a certain point, for example, it may break the agreement the software provider has with its consumer.

SREs create and monitor service-level performance benchmarks for all sorts of metrics, such as uptime, latency, error count, mean-time-to-recovery and throughput, among others. But not all these performance goals are part of public agreements. Companies usually take a more proactive approach to setting more internal targets to stay far away from breaking external agreements and user trust. It’s become a best practice to set these baselines to retain functional, reliable systems.

As mentioned, the type of targets range from firm partner commitments to more internal goals. Below, we’ll attempt to define, in simple terms, the differences between service-level agreement (SLA), service-level objective (SLO) and service-level indicator (SLI). While these concepts share many similarities, it’s important to understand how they are applied in practice. Keep in mind that each company may have a nuanced understanding of these terms and may conceive and apply them differently.

Service-Level Indicator (SLI)

First off, service-level indicators (SLIs) refer to the actual metrics produced by software services. This is a direct measurement of a service’s behavior. These are the real numbers that indicate overall performances, such as error rate and latency over time.

One example of SLI would be the number of successful requests out of total requests over a one-month period. Say an engineer used an application monitoring tool that ingests data from production logs. This data showed that out of one million requests made, ten requests failed. An SLI for availability would thus be 99.999% uptime.

Service-Level Agreement (SLA)

Second, most technologists are familiar with service-level agreements (SLAs). Even if you’re not, you’ve likely agreed to many throughout your history as a digital user. SLAs are like a pact between the software provider and the software user or client. These binding commitments often note availability expectations that must be upheld. SLAs may also include responsiveness to incidents and bugs. It depends on the contract, but if an SLA is broken, some kind of penalty may be incurred such as a refund or a service subscription credit.

These days, the average business relies on many cloud-based SaaS, PaaS and IaaS. Just to initiate something as simple as an online payment might require hitting multiple remote servers. But if all these services just went offline whenever they felt like it, our digital operations would come to a grinding halt. SLAs are thus necessary for enterprise software contractual agreements to ensure both parties meet specific standards.

Once an engineer tracks SLIs and has an idea of typical behavior, they can then set an SLA that makes sense. If we take the example above, an SRE may consider guaranteeing a lower threshold than what is being actively monitored. This would ensure that application performance, at its current pace, doesn’t break any legal or contractual obligations. In this scenario, perhaps the SLA mentions an availability objective: No more than 100 requests fail per one million requests made. This would essentially equate to 99.99% uptime.

Service-Level Objective (SLO)

Lastly, service-level objectives (SLOs) are similar to SLAs but explicitly refer to the performance or reliability targets. An SLA may refer to specific SLOs. Or SLOs may be tracked just for internal purposes. As Google described, “the availability SLO in the SLA is normally a looser objective than the internal availability SLO.”

You don’t want the end users to be the first people clamoring about a 400%+ latency rise in your mobile web apps. Thus, SRE teams typically keep a close eye on performance to ensure they never even get close to breaking SLAs. Some do this by setting and monitoring internal baselines that are more ambitious than the SLA threshold.

If we take our example above, if an SLA guarantees a service uptime of 99.99%, the business may set an internal target of 99.995%. In other words, for every one million requests, no more than 50 should fail. If software systems aren’t hitting these marks, it’s a sign that the company must reevaluate designs and search for bottlenecks. Or, perhaps engineering teams have goals to improve the average downtime for the next quarter—an internal objective could be set at a higher standard than the currently perceived performance.

As the Google Cloud blog described, “every service should have an availability SLO—without it, your team and your stakeholders cannot make principled judgments.” It’s a good reminder to be modest in your reliability commitments, they added, as consumers will come to expect this level of performance. On the other hand, setting more ambitious internal performance targets has the benefit of delivering a better result than the agreement on paper, increasing a software service’s competitiveness.

Real-World Performance Benchmarks

Keep in mind that the above numbers are simply for demonstration purposes. One interesting resource for real-world figures is API.Expert, a service that queries popular APIs and posts weekly performance statistics. Since APIs are the heart of many UI-based platforms (and our digital economy at large), these benchmarks stand as a good indicator of average uptimes and latencies in the industry.

For example, at the time of writing, API.Expert’s Enterprise APIs collection ranked Microsoft Office365 and Pivotal Tracker at the top, both with a 100.00% pass rate and a 220 ms and 248 ms latency, respectively. On the other end of the spectrum, Docusign is at 99.93% with an 877 ms latency and Box is at 99.99% with 414 ms latency.

SLIs, SLAs and SLOs—Oh My!

Although it may sound good to an unpracticed ear, an SLA of 99.99% still equates to 52 minutes and 36 seconds of downtime per year. That’s nearly an hour of downtime in which customers are left scratching their heads or, worse, searching for other options. For traumatic health care situations, a loss of connectivity could be a matter of life and death.

Although creating SLAs and SLOs is important to gauge system health, the reality is that it can be challenging to track and enforce them. “These agreements—generally written by people who aren’t in the tech trenches themselves—often make promises that are difficult for teams to measure,” according to the Atlassian knowledge center.

In summary, SLIs demonstrate the real behavior of software systems. These metrics inform the creation of SLAs, which must be met to ensure B2B agreements. These SLAs often reference particular service-level objectives (SLOs) that must be met, which usually give more breathing room around SLIs. Lastly, in a digital economy with accelerating digital expectations, it makes sense to monitor internal SLOs and improve baselines over time.

April 21, 2022
Not Totally Buying Web3? You’re Not Alone

The interest in Web3 is heating up. We’ve seen major excitement around blockchain and new Web3 protocols that are poised to reinvent the web as we know it. But it’s hard to hide the skepticism and the suspicion that there are ulterior motives behind large cryptocurrency coin fundraisers. There’s no doubt that there are benefits to the Web3 movement — the decentralized nature of Web3 brings innovative forms of shared ownership. Yet, both the marketing hype and investment fervor could get in the way of actual progress in the industry.

I recently met with Netlify CEO Matt Biilmann to discuss the missing piece of Web3. For those who don’t know, Netlify has been a big proponent of the JAMstack movement, a design style incorporating JavaScript, APIs and Markdown (hence, JAM). The headless approach decouples the frontend and backend, enabling easier multi-platform support and other benefits.

According to Biilmann, the theory behind Web3 is exciting and sound and it fits into the headless movement quite well. Yet, in practice, Web3 will require more emphasis on interoperability and open standards, said Biilmann. Without inter-chain portability and better data control, the trend will likely fall victim to the same walled gardens and centralization that underpins the current web. There are some other concerns to keep in mind about Web3 and blockchain, which we’ll explore below.

Back to The Open Web

First off, what exactly is Web3? Well, Web3 is a broad term with loosely-related phenomena, but it can be summed up as a new web designed with decentralized technology and built on blockchains with distributed ownership. “There’s a general sense behind the Web3 movement that goes back to roots of the web as a non-centralized entity,” explained Biilmann.

Whereas in Web2 users were participating in someone else’s walled garden, the concept behind Web3 is that it’s fundamentally open. But how do we make more pieces of the stack a collaborative standard with no single owner? According to Biilmann, this is the aspect that the industry should expand upon.

Issues With Web3

There’s a lot of speculation. One big issue Biilmann foresees with Web3 is the rampant speculation. The long-term success of new Web3 projects founded on a blockchain is muddled by the fact that the creators wanted people to buy in so tokens rise in value. “How much of the underpinnings of Web3 are actually solving real problems and how much is pure speculative investment?” he asks. It’s not always apparent whether, behind their flashy, chic facades, blockchain solutions solve real problems.

A lack of inter-chain interoperability. A lack of interoperability between the many emerging flavors of blockchain could also pose a risk to this fledgling space. “To what degree are they actually building things around standards with the possibility to move from one player to the other?” asked Biilmann. “We need open standards where you can migrate from one to the other.”

Centralization still exists. No matter what the copy around decentralization said, there will still be players who hold more control and power in a capitalist economy. According to Biilmann, it will be important to make platforms fundamentally open and extendable to avoid creating more walled gardens. He admitted that having strong central providers is not a huge problem as long as these open platforms provided interoperability.

A lack of identity portability. When a person owns a web domain, they can move content around freely from server to server. In a similar vein, Biilmam believes that comparable freedom is necessary for identity within Web3. The ideal is to increase ownership and portability of one’s digital identity. Yet, Biilmann questioned whether blockchain is the best vehicle for that goal, especially as competing blockchain protocols could interfere with portability.

Blockchain will still hinge on the authority of an external source. Another issue is how blockchain will interface with reality. Often, a smart contract will point to a URL for an off-chain context without a strict guarantee of what’s happening outside of the chain. If blockchain must depend on the authority of an external source, this is somewhat counter to the ideals of an interdependent network. “Most of the time, you have to trust a provider between what happens in blockchain and what happens in the outside world,” explained Biilmann. In this scenario, the integrity of the blockchain could end up revolving around a single entity.

Connecting blockchain to the outside world is hard. On that note, the exact process for interfacing with off-chain constructs is still being developed. This usually relies on a third-party oracle model, although some headway has been made to eliminate the oracle concept in favor of a more modern standard for integrating off-chain computations.

Every operation incurs high processing costs. Blockchain requires excessive processing power to accomplish simple tasks, which can be wasteful at scale both in monetary and resource terms.

The benefits of NFTs are greatly exaggerated. NFTs won’t likely help the majority of creators. As Kimberly Parker explained, NFT platform sales data showed “an acutely minuscule number of artists making a vast amount of wealth off a small number of sales while the majority of artists are being sold a dream of immense profit that is horrifically exaggerated.”

NFT scams are common, and since minting an NFT requires exorbitant platform fees, the likelihood of an artist recouping their upfront investment is minimal. “I think that people who stand to gain the most are people like the Winklevoss twins, who are already mega-rich and own an NFT platform,” Cat Graffam said on Time.com. “The biggest losers are the vast majority of unestablished artists that are attempting to break into NFT sales.”

Keeping Alternative Options On The Table

“Let’s not jump to the conclusion that cryptocurrencies and blockchain are the solutions,” said Biilmann. As he noted, people have been tinkering with ways to open up identity and data for decades now. These thinkers may also have relevant insights and alternative solutions that shouldn’t be dismissed or ignored.

For example, Tim Berners-Lee, inventor of the World Wide Web and the HTML programming language, has been spearheading a new initiative called Solid. Solid is a “proposed set of conventions and tools for building decentralized social applications based on linked data.” The project’s goal is to enable true data ownership and better privacy. “If we’re after the goal of an open web, we should be just as curious to know what they’ve been studying for years,” said Biilmann.

We may be seeing Web3 projects excel simply due to the underlying speculative interest in cryptocurrency, Biilmann said, not that they’re technically the best solution or have the best open ownership model. Other solutions include IPFS for high-volume distributed storage. Or, take DNS, the decentralized naming system at the core of the web—”that part of the existing web is already very open and user-owned,” said Biilmann.

Goal: Plant Gardens Without Walls

“I’m excited about this—personally, I hold no grudge about the tech,” Biillmann clarified. “Making identity and underlying data fundamentally more open is awesome.”

But, at the same time, the space is too speculative and too far removed from building useful things for users, said Biilmann. There is also a pressing need for interoperability and portability in the blockchain industry. And with so much noise driven by the interest in tokens, it could be more beneficial for society to trust a more neutral party in architecting the fabric of our digital lives.

“It’d be a shame if people forget what we can build already,” said Biilmann, pointing to pre-existing technologies like IPFS and DNS. Although these open standards won’t make early stakeholders rich in the short term, they could help get us to that open state even faster.

Web3 is still in an early space. And if we separate currency from this ambition, it could deliver tremendous outcomes. “Let’s figure out how we make identity and data open and not tied to specific walled gardens,” said Biilmann. “That’s something we, as an industry, should work toward.”

April 19, 2022
How DevSecOps Teams Can Level Up

In 2019, I wrote a post detailing the traits of DevSecOps—the practice can be defined as the process of security automation in which IT and security are more de-siloed. DevSecOps introduces a more intentional culture wherein security-by-design principles are baked into the software development process early on. All in all, the aim is to have security enable development, not stall IT and slow down the SDLC.

So, years later, how is DevSecOps doing? Has it achieved this ideal? Well, I recently met with Jeff Williams, CTO and co-founder of Contrast Security, for insights into the state of modern DevSecOps. According to Williams, most large companies still have real problems with application security and software dependency providence is constantly in question. We’ve recently seen this reality rear its ugly head amid exploits in frameworks like Log4j and Spring4Shell.

Large organizations are likely carrying tens of thousands of vulnerabilities in their backlog. Yet, as consumers, we happily trust software for our critical applications without knowing what lurks behind it. “The industry has a real transparency problem,” said Williams. And as applications are getting more interconnected, they are getting more complicated and they’re getting more critical, Williams added. “These three Cs—connected, complicated and critical—make security harder.” These conditions are culminating in what he called “a perfect storm.”

6 Ways DevSecOps Teams Should Respond

So, how should DevSecOps teams respond to the seemingly endless barrage of new vulnerabilities and attacks? Williams shared some general ways teams can increase their security posture.

1. Practice Security in Sunshine

It’s easy to point fingers when something goes wrong, especially if it’s a serious vulnerability or attack. Instead, Williams encouraged teams to practice what he calls “security in sunshine.” This is an attitude that gives more transparency to security issues internally. Embracing a culture of blameless retrospectives can ensure employees are comfortable bringing up issues within their team.

Security in sunshine can be adopted in an external-facing way as well. Williams likened it to how, in product manufacturing, it’s standard practice to share a bill of materials (BOM) with trade partners. We may see similar labeling standards emerge for software in the form of software bills of materials (SBOMs) so that external consumers are better aware of dependencies and potential threats, Williams suggested.

Cybersecurity awareness certainly takes a communal effort that requires transparent communication. We see this realized daily in bug bounty programs, CVE reports and fixes to open source software. Bringing potential threats into the light of day is necessary to squash vulnerabilities before they are exploited.

2. Translate High-level Requirements

Security guidelines issued by bodies such as NIST and ISO are essential to keep in mind. Implementing the concepts these guidelines recommend, such as the principle of least privilege, multifactor authentication and zero-trust architecture, are considered best practices by most security professionals. However, these high-level requirements don’t often translate down to the developer on the ground floor. “The developer needs those requirements translated down four levels to specific technologies they’re working with,” said Williams.

A programmer needs more realistic advice on how these guidelines affect, say, Spring Boot applications or how they should configure permissions in AWS. According to Williams, teams must interpret these abstract recommendations with more concrete annotations that meet the nuances of the technology stack they’re working in. The same process is required to help developers create projects compliant with data regulations, as it’s difficult to take this high-level advice into practice. Translating these requirements into more readable terms is an excellent way to de-silo security and development teams.

3. Shift Smart

In recent years, there’s been a lot of talk about shift left. The idea is that to reduce security threats later on, organizations need a tighter grasp on security earlier in the software development life cycle. Shifting security left is often enabled with tools that hold code accountable for potential threats using automated analysis.

However, Williams sees this as an area where organizations need to be careful. Not all security tools are developer-friendly and might introduce inefficiencies at scale. Instead of shifting left, he advocates for “shifting smart” to introduce security at junctures where it’s most efficient. “You don’t want to shift all security left—you want to shift right too,” explained Williams. “Spread it out over the whole life cycle.”

4. Avoid Added Steps

There’s more than one way to bake a cake. Similarly, there’s more than one way to cook up a great software application. As such, most organizations are dealing with multiple tools, making it harder to apply security consistently across all stacks.

Security should be catered to the specific tooling and processes being used, Williams explained. But that requires answer certain questions like where is a software development team in their journey? Are they waterfall or Agile? How does each CI/CD pipeline behave?

Obviously, there are many unique factors to consider. Ideally, tooling should be flexible enough to attach itself to existing infrastructure, Williams said, instead of adding additional steps to an already sprawling tooling ecosystem.

5. Take Security Seriously

And, of course, there’s the cultural element. “It’s 2022, and companies still don’t take security seriously—they need to change,” said Williams. “It doesn’t matter how they get there—that they get there matters most.”

Admittedly, cultural change is a slow evolution and the best way to advance will be to add practices and technologies progressively over time. Thus, Williams recommended iteratively inserting pieces, measuring outcomes and proving value along each step of the journey.

6. Update Outdated Tools

WhiteHat Security found the average time it took to fix critical vulnerabilities in the spring of 2021 was 205 days. But such a long timeline is at odds with the pace of new threats and the need to patch them quickly.

The industry is changing the way they do AppSec, said Williams. Traditional DevSecOps security tools, he said, such as static analysis tools, dynamic analysis tools and web application firewalls, are outdated and less accurate compared to newer technologies like interactive application security testing (IAST) and runtime application self protection (RASP). Accuracy, speed and scalability will be important characteristics for security automation tools that don’t require additional human intervention or produce too many false positives.

DevSecOps Is Still Maturing

A healthier DevSecOps practice aims to decrease the “vulnerability escape rate,” as Williams called it. Hopefully, by providing instant feedback on designs, engineers will learn not to repeat the same mistakes, thus decreasing the number of vulnerabilities they introduce.

In summary, DevSecOps is still maturing. Looking to the future, a stronger DevSecOps approach will require a more open attitude to mitigate threats. It also requires practical guidance as not all top-down guidelines translate well into the active development environment. Applications will require proper code hygiene, and the software supply chain itself must be protected, too, in addition to conducting active monitoring of production environments.

April 18, 2022
Dagger: Standardizing CI/CD is the Holy Grail of DevOps
Continuous integration and continuous delivery (CI/CD) has become a hallmark of quick software release life cycles. Nowadays, many teams support CI/CD pipelines for their software, offering a repeatable pathway to build, test and deploy code. A CI/CD pipeline is necessary to support an iterative, rapid deployment approach—yet, maintaining one is not an easy task.

Enter Dagger. Dagger is a utility that aims to streamline how developers create and manage their CI/CD pipelines. Dagger, led by founders of Docker, recently unveiled their public beta launch along with a Series A $20 million funding raise led by Redpoint Ventures. While many open source CI/CD projects already exist on the market, Dagger is unique in offering easier configurations, pre-built components and more reusable actions.

Akin to traditional supply chain management, software supply chain management is an increasingly complex endeavor involving many third-party dependencies. Development teams truly require a set of standard building blocks to piece it all together. For this reason, Solomon Hykes, co-founder and chief executive of Dagger, told SiliconANGLE that “solving this problem is the holy grail of DevOps.”

Below, we’ll look at the common stressors inherent in creating and managing modern software delivery chains. We’ll also take a peek into the Dagger beta to see what it’s attempting to accomplish.

Issues Facing CI/CD

Software development supply chains are growing larger and larger to support a complex network of software architectures. But DevOps engineers, who must automate this chain, are drowning in tool sprawl—and stitching these components together to make custom pipelines can become a big bottleneck.

Especially as an organization expands its distributed microservices architecture into the cloud, it can be tricky to scale a repeatable CI/CD pipeline across various department and disparate tech stacks. This often results in many bespoke CI/CD pipelines. Even within the same team, nuances between codebases can complicate each CI/CD process and require engineers to rewrite scripts.

Simultaneously, organizations are under intense pressure to deliver to meet new digital innovation demands. But, until DevOps can make the release process more efficient, software development automation risks introducing more toil than it takes away.

Diving Into Dagger

So, how is Dagger alleviating these concerns? Well, Dagger creators described it as a “portable devkit for CI/CD pipelines.” Using Dagger, DevOps engineers can use a standard set of building blocks to assemble their CI/CD pipelines and then run them on their chosen cloud. It’s like a better developer experience for CI/CD pipeline creation.

So, how does Dagger work under the hood? Dagger is powered by Buildkit, the Dockerfile-agnostic builder toolkit. Developers then use Cue to specify how their code is run in a container and to link up actions. Cue is an open source configuration language developed at Google.

According to the docs, it takes about five minutes to install Dagger and the necessary dependencies. Dagger also provides boilerplate code to integrate with pre-existing CI environments. In essence, Dagger doesn’t need to replace your CI tool—it adds “a portable development layer on top of it.”

Then, there are the actions themselves. Dagger supports a growing library of actions that it calls Universe. At the time of writing, the Universe of actions is limited and is being staged in the main Dagger repository. These actions are reusable Cue packages. For example, this package details how to run and deploy the Nginx web server in Cue:
```
package nginx

import (
    "universe.dagger.io/docker"
)
```
And then,
```
#Build: {
    output: docker.#Image & _pull.image

    _pull: docker.#Pull
    *{
        flavor: "alpine"
        _pull: source: "index.docker.io/nginx:stable-alpine"
    } | {
        flavor: "debian"
        _pull: source: "index.docker.io/nginx:stable"
    }
}
```
Benefits

CI/CD can certainly benefit from standard building blocks and more fluid wiring between tools. There are several potential benefits from Dagger’s proposition:
- Unification: Unifying developer and CI environments could avoid headaches.
- Localization: Dagger enables testing a closed-loop pipeline locally. This helps for debugging.
- Reusability: Building blocks help avoid rewriting automation scripts repeatedly.
- Multi-cloud: Easier ability to migrate a pipeline from one cloud to the other, thus avoiding vendor lock-in.
- Compatability: Dagger can run in any environment compatible with Docker, meaning it’s probably workable with your current toolset.
Final Thoughts: Lego Blocks For CI/CD

Docker revolutionized the software industry by easing the process of managing containerized applications. And now, Docker founders Solomon Hykes, Sam Alba, Andrea Luzzardi and others at Dagger are tackling another meaningful problem within today’s complex software ecosystem. Given the success of their previous endeavors, it’s not too difficult to imagine a network of reusable CI configurations emerging with an excitement similar to Docker Hub.

Of course, Dagger is still in its early stages and there’s much development ahead. It will undoubtedly be an interesting project to follow in the years to come. Dagger is being built as an open source project on GitHub. The company has also opened a Discord channel to engage with the community.
April 5, 2022
Why the Built World Needs Standard APIs

Lately, new ESG initiatives have come front-and-center as companies seek to contextualize and limit their business’s impact on the environment. This is especially true for the built world—a term that refers to the human-made environment that provides the setting for human activity, including homes, buildings, zoning, streets, sidewalks, open spaces, transportation options and more—where monitoring energy efficiencies is becoming more commonplace and even required by law. For example, a New York City law now requires buildings of a certain size to post an energy-efficiency score at their entrances. Yet, calculating such a score isn’t that straightforward—there’s a lot of legacy technology in use within most large commercial and residential buildings, and pulling data from these systems is notoriously difficult.

A building manager may seek greater visibility into these systems to not only determine a building’s energy score, but to program optimizations, inform building security or enact new digitization. But the answer isn’t to hastily rip out and upgrade old hardware—old equipment like AC units and elevators were built to last 50 years or more and could be functioning seamlessly. Instead, this is another area where REST APIs are set to accelerate by uniting legacy integration software and disparate hardware with a standard layer.

I recently met with Shaun Cooley, CEO and founder of Mapped. According to Cooley, API-driven standards for built environments could open up transparency into hardware component behaviors and the interrelationships between systems within a building. Below, we’ll briefly consider the history of building protocols and look to standards like Brick Schema that aim to unite legacy and modern physical infrastructure. We’ll consider the competitive advantage APIs bring to the built world and how increasing data accessibility could assist operations.

The State of Physical Building Digital Protocols

The built world consists of various digital communications standards. Most buildings have many older on-premises protocols for lighting, HVAC, elevators and other systems. For example, many elevators and fire safety systems were installed by systems integrators in the 1970s and adopted connections using Modbus, a communication protocol based on serial connections. BACnet is also commonly used to control HVAC systems and is a step forward from Modbus in that it can more easily connect to IP networks.

Fast-forward a few decades to the dawn of the cloud; many companies overlaid sensors onto these old systems, explained Cooley. These power-over-ethernet (PoE) or battery-powered systems relay data immediately to the cloud and often speak in proprietary protocols. A single commercial building could easily be juggling a bundle of these various communication protocols, all originating from installations that occurred at different time periods. These nuances mean that you are “constantly dealing with chaos across your portfolio,” Cooley said. There’s so much data, but getting it normalized and usable is very challenging, he said. One must integrate with many sensors and protocols, which he describes as “the dirty plumbing” that no one wants to build.

There’s bound to be differentiation from building to building, too, which could place a further strain on managing multiple building operations. Without a window into building data, it isn’t easy to know things like the current occupancy of the lobby or how occupancy affects energy consumption. Unless managers can connect the dots between systems, it’s impossible to understand the relationships between components.

Making a Data Graph for a Building

For example, take an AC system in a commercial building. Such a unit isn’t as simple as a single-family home’s typical thermostat and furnace. It’s often a mix of a rooftop chilling unit, pipes, pumps and a VAV damper that controls the air. While protocols are good at describing how individual components are performing, the composition of components is really what defines a building’s system and its efficiency, said Cooley.

According to Cooley, instead of adopting a vertical data schema, buildings require a graph data model to adequately represent hardware interconnections. This is why he’s rallying behind Brick Schema, an open source uniform metadata schema for buildings. Instead of a vertical data schema, Brick Schema is built on a graph data model, which enables more descriptive relationships in a system.

End Benefits

Pulling unstructured data from 20 different data lakes into a clean API could allow easier access to new solutions. “We want to make it very simple to build applications for these sorts of environments,” said Cooley. Typically, system integrators must reverse engineer and connect to legacy building hardware, which can become very expensive very quickly, he said. He recommended using a sandbox based on open standards that can easily be rolled out to any building to reduce those hurdles. A more consumable layer with common endpoints could increase developer usability when creating innovative building applications.

Similar to how Twilio made SMS easy, Cooley said he hopes for a similar proliferation in the built world. Many applications could be built with such an abstraction layer. For example, managers could better understand energy usage to reduce consumption, meet regulations and save money. Or, such an abstraction layer could inform predictive maintenance to create automated alerts before failures occur. There are also plenty of ways to improve occupant experience or intelligently optimize the lighting and temperature in a space.

“Entirely new business models and concepts emerge that we hadn’t considered previously,” said Cooley. For example, he described a “tailgating app” that uses Brick Schema for a security use case. The app prevents building occupants from letting other people “tailgate” into the building behind them after someone swipes their badge. To do so, the app interfaces with many separate building systems (the elevator system, surveillance system, telepresence systems and other sensors) to ensure the number of people in the building matches the number of occupants that swiped in. A lingua franca helps to connect the disparate hardware involved.

The Future of Building Automation

This is by no means the first attempt to standardize building data. For example, the Green Button Initiative provides a standard API through which to download utility use data for electricity, gas and water and share it with solution providers. However, according to Cooley, this information only goes so far. Building owners will require more contextual data on-site.

For example, perhaps a building’s window glazing is causing specific rooms to overheat in the sunlight. Thermal sensors would spot this condition, but bulk energy consumption readings wouldn’t ascertain the root cause. Access to real-time granular data such as current occupancy count is also necessary for developing on-site applications that better refine exactly how human involvement impacts energy consumption.

Of course, as building systems become more intelligent, there is the growing possibility of privacy infringement. For example, systems that use facial monitoring and recognition could end up storing personally identifiable information. In that case, you “need to give data owners the control to decide what applications can’t or can’t access PII,” said Cooley.

While this field has many read-only applications, Cooley predicts future write operations will enable more automation with building hardware. This could allow applying ML/AI to anticipate elevator floor calls. Or, a predictive system could cool a room only when a meeting is scheduled. When such data is readily accessible and available for developers, it presents many innovative opportunities to optimize and interact with the built world.

March 3, 2022
How Stream Processing Goes Beyond Real-Time

How fast is real-time? According to Robert Miller’s 1968 seminal study on human-computer interaction, we perceive a response time of 100 milliseconds or less as instantaneous. This is much quicker than the average human reaction time, which one online test clocks at 273 milliseconds.

Nowadays, customer-facing applications need to respond in as close to real-time as possible to win our attention and meet standard requirements. For example, 100 milliseconds is a golden rule for credit card processors to meet SLAs. Especially in our new remote-first economy, other areas like media, health care, retail and manufacturing need to take full advantage of real-time data processing to enhance digital business and stay relevant. Yet, leveling up with real-time requires some technical nuances that might be challenging to implement with the current developer skills gap.

According to Manish Devgan, chief product officer at Hazelcast, the problem with most data management techniques is that data isn’t being processed in-stream at the edge. Instead, data gets stored and thrown into a data lake where it accumulates, unfulfilled. To make matters more complicated, decentralized data management means organizations may be overseeing many different data lakes, adding to the potential for unused data and increased latency.

I recently met with Devgan to learn how in-memory stream processing goes beyond real-time data. According to Devgan, the secret sauce is processing incoming data in-stream while tying it to persistent storage to create context. By doing so, companies could unlock more innovative real-time experiences, he said.

Use Cases For Real-Time

Real-time presents many opportunities for business operations and end-user customer engagements. This is especially true in the financial sector. For example, Devgan explains how one bank was able to use real-time data at the moment of customer interaction at their ATMs. The system was able to fetch a history of the user account and extend an offer for a low line of credit, increasing their total loans by 400%. Capturing value at the moment of user intent is a golden opportunity, said Devgan.

Capital One is another example of a financial institution leveraging advanced technology to improve the product experience. “They’re one of the leading banks in terms of technology,” said Devgan. Capital One’s threat detection system calls a machine learning model in real-time to detect and prevent fraud. Only by acting on incoming data in the moment of interaction can preventative responses like these be signaled.

Other scenarios are also beginning to incorporate increased real-time analysis. “This split-second preemptive decision-making is important for many other use cases,” said Devgan. Physical retail might loop into customer loyalty programs at point-of-sale systems to increase in-store spending. Or, automobile manufacturers may want to use real-time processing to detect a problem with their robotic arms sooner rather than later.

Real-time vs. Stream Processing

When an event comes in, it usually has very little information. But when you look up a customer profile and connect it to a greater context, you can direct further actions. “Correlating data in motion with data at rest is where the magic happens,” Devgan. This distinction uncovers the difference between real-time data and stream processing.

The real-time label tends to get thrown around a lot, and it’s quantified differently for different groups. Real-time might be five minutes for some applications but 10 milliseconds for others. Real-time could simply mean that a system accesses data at rest very quickly, delivering perceivably “real-time” results.

On the other hand, stream processing incorporates data at rest and data in motion. It computes incoming, in-stream data and correlates it with data at rest. “You don’t let the data land — you process it as it comes in,” Devgan said.

For years, enterprises have been collecting and storing exorbitant amounts of data. A survey by Seagate and IDC reveals that 43% of data collected is largely unleveraged and other studies estimate the amount of this “dark data” is over 50%. And, not acting upon data effectively makes it just another pile on top of the growing mountain of technical debt.

There is a significant storage expense for holding on to too much data, whether on-premises or in the cloud. But, the paradigm is shifting a bit, said Devgan. Organizations don’t want to hold onto everything. The attitude is beginning to shift toward “garbage in, garbage out.”

Technology Behind Real-Time Processing

So, what sort of technology does stream processing require? Processing data at the edge requires some fundamental changes to design thinking. For one, since data must be computed in-stream, there’s more emphasis on in-memory processing. Devgan explained this as “pushing compute around instead of pushing data around.” To enable this, Devgan pointed to open-source projects like Jet that provide powerful distributed batch and stream processing.

SQL has a storied history of acting upon data at rest. Now, Devgan said, he noticed the reemergence of SQL in a stream processing context. Though the industry is still defining how the SQL dialect will conform in a streaming world, he’s bullish on the prospect of SQL guiding this area. “Streaming SQL will be the lingua franca for streaming,” he insisted. We’re also in an early era of determining how to bridge the stateful and serverless worlds, Devgan added.

But regardless of computing location, it will come down to the net customer experience outcomes. “People want to build their business applications and are less worried about the operational management of the system,” said Devgan. Whether it’s building real-time offers for e-commerce or implementing fraud detection, it will come down to the numbers. There is also a role for stream processing in health care to optimize physical resources and enhance patient experiences with real-time data. In all these scenarios, rapid access to data and processing at the edge will determine much of their success.

February 8, 2022
Trends in the API Industry

The application programming interface (API) economy is chugging along full steam ahead. The API management market alone is expected to expand 35% by 2025, supported by the sheer number of web APIs coming to market. APIs have become ubiquitous across microservices architectures, public product initiatives, SaaS platform offerings, IoT and partner-partner integrations.

The industry has proliferated and continues to change, introducing new API use cases across all industries. This ubiquity creates a heightened need for an API program to maintain a quality developer experience and remain competitive. There are also different styles and specification options on the market, meaning designing the right API strategy is a moving target. And, amid peaking cyberattacks, developers now have a strong imperative to retain high security and reliability for these integration points.

I recently met with the CEO at Stoplight, Steve Rodda, to discuss the above trends and other forces pushing the API industry forward in 2022. Below, we’ll dive into these trends to see how they affect API strategies and what developers need to know to stay on top of their applications.

Non-Software Companies Embrace APIs

As software eats the world, more companies are becoming software companies. And, an API strategy is part and parcel of this digital transformation. A big trend that Rodda recognizes is increased API strategies among traditionally non-software companies.

For example, Rodda explained how a large beverage manufacturer evolved to adopt APIs company-wide to better use and scale their data. Standardizing the API development and design process helped avoid the “rat’s nest” of custom code, said Rodda. “APIs are no longer a byproduct; they’re a design artifact,” he said.

APIs are no longer reserved for startup cloud-native unicorns like Stripe or Twilio, either. We now see many industries and sectors like supply chain management, health care, shipping, built environments and mainframe modernization adopting API approaches. The financial services sector is especially keen to open up APIs as open banking is being driven by regional regulation or market pressures.

A Rise in Internal API-First Adoption

ProgrammableWeb, the most comprehensive API directory, lists over 24,000 APIs at the time of writing. There’s no doubt that public-facing APIs are very well documented. But this is only a sliver of total API adoption.

The 2021 Postman State of API report found that only 15% of APIs are publicly available. Most APIs are either partner-facing (27%) or private (58%), according to the report. In fact, the most popular use case for APIs is integration between internal applications, programs or systems. Along with the growth of internal API use, API-first thinking is gaining mind share. Postman defined API-first as “defining and designing APIs and underlying schema before developing dependent APIs, applications or integrations.”

Internal use cases are less frequently discussed but are pervasive for large enterprises, said Rodda. Whereas in the past, development teams paid less attention to style guides and documentation for internal services, he now notices that these practices are shifting. The zero-trust approach is influencing teams to take more care in defining and standardizing internal APIs as if they’re external-facing. This also eases consumption. “You need to take care of the folks next door to you and treat them with just as much respect,” said Rodda.

The Growing Need for Robust API Security

APIs have a security problem. Many don’t handle authorization correctly and expose far too much information. A hacker can often escalate their privileges or simply swap a different user ID into an API call to return loads of sensitive data or even make changes to the data. Many HTTP API endpoints that are intended to be private really aren’t. As a result, OWASP lists broken object-level authorization as a top API vulnerability.

In recent years, we’ve seen what can happen when APIs are improperly configured. For example, a Facebook API lacked an authorization check for unlisted posts, allowing a client to post on any user’s behalf. A similar authorization hiccup occurred with Peleton that allowed hackers to alter any user’s back-end account. APIs constantly work with sensitive data and must carefully navigate the various data privacy regulations around the globe.

According to Rodda, a design-first mindset is vital to ward off security woes early in the API life cycle. This means deciding on a structure with a common design and choosing appropriate style guides well before coding. “Good documentation, good design guides and good discipline around building something with a cross-functional nature will increase security and scalability,” said Rodda. When you know what’s going on, you have more control and things are inherently more secure, he added.

OpenAPI Carries The Day

The OpenAPI specification, formerly known as Swagger, is an industry-standard specification for describing REST web APIs. OpenAPI can help direct your API’s design and development. OpenAPI tooling can also help generate useful documentation, sandboxes and SDKs, making it a valuable specification for creating interoperability between partners.

“OpenAPI is going to carry the day for a while,” said Rodda. The OpenAPI Initiative has gotten the support of big names in the tech industry. Yet, Rodda estimated only 40% of companies have adopted OpenAPI in production. Rodda said he believes tooling will make it easier for more groups to maintain OpenAPI adoption across their organization, reducing the redundancy of writing “yet another YAML file.”

REST is still by far the most-adopted API design style—59.7% of developers use REST for production APIs, according to RapidAPI’s State of APIs Developer Survey 2021 report. Yet, asynchronous event-driven communication protocols are gaining traction, and those protocols tend to prefer alternative description formats like AsyncAPI. Websockets, gRPC and GraphQL are other options consistently used by a significant percentage of API developers.

Developer Experience Matches User Experience

Another major trend is the rising importance of developer experience (DX). DX is akin to user experience but is all about increasing usability for developer consumers and improving their ongoing relationship with software-as-a-service. “DX is as big a deal as UI/UX these days, if not bigger,” explained Rodda. “Developer experience directly translates into efficiency and reusability.”

In the context of APIs, increased consideration of developer experience means reducing the onboarding effort and maintaining more reliable connections. For example, users will likely look to other solutions if a third-party API has poor uptime and routinely introduces a breaking change. Better DX likely will also equate to increased abstraction layers and more code generation. Rodda pointed to GitHub’s CoPilot as an example of the use of AI and automation in the developer-of-tomorrow’s workflow.

“We’ve evolved as a developer society,” said Rodda. Just as consumers expect high-quality real-time applications, developers expect highly performant APIs. To help get there, one increasingly popular philosophy is the API-as-a-product perspective. “If you treat it all as a product, you’re building it with the same standards and quality that you would share externally,” he explains. “Then, there’s no attention to detail that goes missing.”

Final Thoughts: The Design-First Mentality

More industries are utilizing APIs than ever before, and APIs are seeping into traditionally non-technical companies to aid their internal integration efforts. But data privacy is a pressing concern when opening up data, requiring careful measures. Taking a design-first approach with standards like OpenAPI could help organizations stay alert and ensure consistency across their portfolio of services.

There are other trends at work in the API economy, too. For example, one interesting Web3 initiative is an attempt to combine the world of blockchain and APIs by enabling on-chain smart contracts to interact with traditional HTTP APIs. There is also a lot of development activity around GraphQL, which could act as a command line for consuming multiple APIs, thus making integration easier.

Rodda said the biggest takeaway is that APIs should no longer be treated as byproducts. He compared it to a construction project: When building a house, you must start with a blueprint before installing the foundation, wall, roof or electrical system. Similarly, APIs require solid forethought to ensure stability and scalability.

February 3, 2022