devopsy.kinsta.cloud

Category: DevOps Toolbox

Increasing Use of SLOs to Enable Observability
Observability is a growing discipline among most IT and operations departments. To release stable software faster, operators need continuous visibility into metrics like performance, uptime and availability. As a result, engineers are increasing their use of service-level objectives (SLOs) across the board—a recent study found that 82% of companies are increasing their use of SLOs.

SLOs enable deep visibility into specific applications’ performance and are often used by site reliability engineers (SREs) to ensure quality and avoid service interruptions. Not only that, but various teams are correlating SLOs to aspects of the business to reduce cost and help direct decision-making. But while many environments have visibility, there are still gaps present.

Nobl9 recently released a global survey of IT professionals and executives that tracked the state of service-level objectives in 2022. Below, we’ll look at the key findings from the report and consider what they mean for the state of SLOs and observability in general.

State of Observability and SRE

In general, site reliability engineering is still maturing across organizations. Though only 31% of companies have adopted SRE, it’s slated for much future growth, as 46% said they plan to embrace SRE in the future.

These operators are now faced with many cloud-native observability tools, which are producing a sea of data in the form of metrics, logs and traces. A full 39% of companies use anywhere from six to 10 observability and monitoring tools and 35% use more than 11.

With so many tools now deployed, who is using this observability and monitoring data? Namely, at 74% of companies, observability data support operational requirements. Operations teams are the ones most likely to use SLOs to monitor uptime, performance and overall efficiency. After operations teams, security teams also use this observability data (71%), which makes sense as SLOs can inform incident response. Other areas that follow are customer support, compliance and capacity planning.

One interesting finding is that only 42% of companies use SLOs for service-level agreement (SLA) adherence. This indicates that SLOs are most often applied to internal optimizations and decision-making.

Hybrid Environments Complicate Visibility

Companies are mostly tracking SLOs to increase their visibility into networks (83%), databases (76%) and applications (75%). Other top areas include private cloud environments and legacy computing arrangements. But although observability is trending, organizations still lack complete visibility across the entire stack.

Of note is that nearly half (46%) of respondents said their monitoring and observability tools don’t provide full visibility into all of their company’s IT assets. For example, only 45% of companies have visibility into their containers and just 35% have visibility into their microservices architecture.

A lack of full-stack visibility may be complicated due to rising hybrid and multi-cloud conditions, as 78% say a hybrid cloud environment makes monitoring infrastructure more difficult.

Benefits of Tracking SLOs

The report defined SLO as “a performance and availability target set for a given system, application [or] service over time.” And the benefits are readily apparent for organizations following such targets—they can help increase performances, direct decision-making and help avoid outages.

As a result, 70% of companies are currently using SLOs in some fashion. Here are some benefits of tracking SLOs:
- Increase microservice performance: 87% said using SLOs for microservices architecture would improve service performance.
- Enable full-stack observability: 58% said some of their company SLOs are mapped to business operations.
- Improve business decision-making: 91% agreed using SLOs can help drive better business decisions.
- Prevent service interruption: 67% said their company prevented business interruptions thanks to SLO thresholds alerts.
- Reduce expenses: 90% indicated SLOs saved their company money.
Final Thoughts

More teams are tracking service-level objectives than ever before. And most companies (71%) that aren’t using SLOs now plan on adopting them soon. The data demonstrated that the observability market is an evolving area with room to grow. And, the same can be said about the SRE role.

Yet, it’s good to point out that not all companies will adopt precisely the same roles or monitoring procedures. Therefore, there will likely continue to be variance in who is using SLOs and how they are applied. Regardless, the study suggested that investigating SLOs has the potential to benefit the software life cycle in many ways.

The study, “Adoption of SLOs Grow to Increase Visibility and Drive Business Improvements” surveyed 309 participants around the globe in various sectors. The study was sponsored by Nobl9 and conducted by Dimensional Research. For more information, you can download the report here.
May 17, 2022
Why Data Lineage Matters and Why it’s so Challenging

Change introduces risk. It’s one of those foundational principles of software development that most of us learned very early in our careers. Nevertheless, it always seems to keep cropping up in spite of those repeated life lessons. Our inability to foresee the impact of changes, even small ones, often leads to negative outcomes.

As the complexity of interconnected IT systems has increased, this problem has grown worse. It’s harder and harder to predict how even a small modification might impact systems upstream or downstream of the change. Data lineage accomplishes that, but it’s extraordinarily difficult to do data lineage well.

Why Data Lineage Matters

In the old world, data resided in silos with relatively few connection points between them. Today, data flows multi-directionally among multiple systems. Many of those systems share data with external organizations, which means that information subsequently flows to internal systems there, as well. Consequently, it’s easier than ever to overlook the impact of a single change on multiple other systems.

Imagine, for example, that your company provides a mobile app that enables customers to quickly and easily request a new service call. In the past, your customers could specify which product variant they owned. Your design team has determined that since you already have that information in your CRM database, you no longer need to collect it from users of the mobile app.

You’ve been tasked with simplifying the app, so you remove that input field and replace it with a query that pulls the customers’ product information from your CRM. Theoretically, the resulting work tickets should include all of the information that your field service personnel need to get the job done.

Unfortunately, there’s a problem; some of the customers’ product records in your CRM system contain null values. Your mobile app is querying that information, but it’s delivering invalid results to your ticketing system. Your field service personnel no longer know how to prepare for their next service call.

That might sound like a relatively simple problem of data quality—one that might have been foreseen if someone had been more thorough about vetting the CRM data. Even that might not have solved the problem, though, if null values in the CRM database were not introduced until later. If customer data from a newly acquired company is merged into your CRM system, for example, you might not have product information tied to those new records.

Sometimes these issues are more difficult to predict, and they may not be apparent right away. Imagine what happens when your marketing team decides to redefine its customer segmentation criteria. A new data field is added to designate the customer category; then it’s populated with values for all customers and the old field is deprecated.

Unbeknownst to the marketing department, though, the company’s top executives are still looking at the old data. They’re using dashboards that rely on data from the deprecated field. New customers are being added every day, but no one is entering any values in the deprecated field. As a result, the executive team is looking at distorted data. Suddenly, the C-suite is tracking very different KPIs than the rest of the organization. What’s worse, no one even notices that there’s an issue until weeks or months after the fact.

Next, imagine a similar scenario, but instead of populating an executive dashboard, the source system is feeding data to the AI algorithms that drive customer recommendations on your e-commerce site. When AI is trained on flawed data—or if the data changes materially in some way—it can dramatically impact the effectiveness of that technology. In this case, it distorts your customer recommendation engine.

Tackling the Challenge With Data Lineage

Data lineage tools provide a systematic means of understanding the impact of changes by presenting a complete roadmap of potential implications, both upstream and downstream.

This amounts to risk reduction and cost control. If we can predict how changes might impact other systems, we can avoid creating problems like the ones we described earlier. That leads to less wasted effort, faster time-to-market and lower costs. We’re all familiar with the principle that says it’s faster and cheaper to solve a problem when you catch it early in the development process; data lineage provides a critical missing link, potentially ferreting out problems before they even reach QA.

There are benefits to data governance and regulatory compliance as well. When the European Union passed the General Data Protection Regulation (GDPR) back in 2016, it set off a chain of new requirements for systems that house personally identifiable information (PII). If a customer asks you to delete their data, you’re legally obligated to comply with that request. But what happens upstream and downstream of that change? Will it break anything? Or will some of the customer’s PII remain elsewhere in your systems, rendering you non-compliant with the law?

Data lineage addresses these questions by automating the process of detecting upstream and downstream effects. It provides clear auditability that can help determine where changes originated and how the data arrived at its current state.

Column-level data lineage is especially difficult because it requires that SQL queries be parsed to discern exactly what changed and how. This means tapping into database logs, but because there are so many database vendors—each with their own unique SQL dialect—parsing that information for various databases can be particularly challenging. As each of those SQL dialects evolves over time, data lineage tools must constantly be refined to accommodate those changes.

An example column-level lineage view that shows all the tables and columns that are involved in computing the sales table used for reporting in an imaginary e-commerce business selling craft beers.

As interconnected systems extend their reach even further and the flow of information accelerates, column-level data lineage will become more important. Today, it’s an extraordinarily useful tool that saves time, effort and money. Soon, data lineage will be indispensable.

May 16, 2022
Observe, Inc. Dives Deeper Into Observability

Fresh from picking up an additional $70 million in funding, Observe, Inc. announced this week it has added multiple tools, dubbed Data Universe Maps, to its observability platform to visualize all datasets and surface underlying event data.

The company also added pre-built applications for Kubernetes, Amazon Web Services (AWS) and Jenkins environments with editions for Google Cloud Platform (GCP) and Microsoft Azure planned for later this year. Observe Apps are intended to make it easy for IT teams to ingest data that is surfaced via pre-built dashboards that will generate alerts when issues arise.

Finally, Observe, Inc. has added dashboards that provide transparency into its usage-based pricing model. IT teams can now more easily determine how many credits are consumed by both users and specific datasets to help contain costs.

Observe, Inc. CEO Jeremy Burton said those tools are crucial because in the absence of tools to control data costs, most organizations will soon discover that observability is unsustainable.

Of course, it’s not clear yet how many organizations are moving beyond monitoring metrics to embrace observability and launch queries to discover the root cause of performance issues. Many organizations don’t have the level of expertise required to structure such queries. However, as more organizations struggle to investigate and remediate issues impacting customer experience, interest in observability platforms is rising, noted Burton.

Burton said thus far, as a startup, Observe, Inc. has expanded its customer base by a factor of three to now include 50 organizations, which has resulted in a five-fold increase in the number of monthly active users of the platform. The platform is now ingesting more than 40TB of data each day, with more than 25 million queries each day launched across nearly one trillion rows of more than 10 petabytes of data. Capital One is included among the investors in Observe, Inc.

At the core of the Observe platform is a graph engine that makes it simpler to uncover the relationships between machine data that is ingested into the platform.

Going forward, Burton said Observe intends to add support for distributed tracing using the open source agent software created by the OpenTelemetry project that operates under the auspices of the Cloud Native Computing Foundation (CNCF).

Change within enterprise IT organizations happens slowly. Many will be using legacy monitoring tools alongside observability platforms for years to come. However, there is an opportunity to rationalize monitoring tools as the capabilities of observability platforms continue to expand.

In the meantime, IT environments will continue to become more complex as more modern applications based on microservices and constructed using containers are deployed. These applications, over time, are generally more resilient than monolithic applications; given all the dependencies between microservices, determining the root cause of a performance issue can be a major challenge. As a result, it may be just a matter of time before this new class of applications ultimately forces the observability issue in the enterprise.

May 13, 2022
Top 3 Requirements for Next-Gen ML Tools

As the machine learning market matures, new tools are evolving that better match data science and machine learning teams’ needs. Vendors, both open source and private, have been quick to introduce new products that better meet requirements making it easier and faster to develop models and enable collaboration.

These new offerings range from cloud-based platforms that make it easy to build and deploy models to specialized software that can speed up the training process. As an example, the existing field of ML experiment tracking is a multi-billion dollar market with a crowded amount of vendors: Comet ML, Weights & Biases, Neptune AI, Databricks’ MLflow and more. These tools will need to evolve to move beyond meeting only the basic requirements of ML teams like experiment tracking, visualizations and model training to address collaboration and technical requirements so these teams can get models to market faster.

The types of teams that are driving modern requirements for data science are the bet-the-farm types, where it’s all about finding new technology and solutions to drive efficiencies and get models to market faster for their organizations to maintain a competitive edge. Going forward, we’re seeing the following as requirements for ML teams as they evaluate tools to add to their MLOps stack.

An Open, Modular Approach

Data scientists and ML engineers use multiple tools to do their jobs, from Jupyter notebooks for model coding to DVC for experiment and data versioning, to automation tools like AirFlow to model visualization tools. Solutions will need to be open to integration with other tools instead of a walled garden that many ML tools, like those offered by cloud vendors (e.g. Amazon SageMaker, etc.) are now.

Familiar Developer Experience

ML developers and engineers are familiar with standard DevOps tools and workflows like GitOps (think GitHub or GitLab) and live in CLI-oriented solutions. ML tools should integrate tightly with these existing tools instead of complicating the developer experience with a separate ecosystem. And as a part of the developer experience, workflow is also important. Data scientists can run hundreds of experiments using their preferred CLI-based solutions but, often, all those experiments are streamed and posted into existing ML tools. A more optimal approach would be to offer data scientists the experience of running experiments locally and letting them commit only the most important experiments for further discussion and collaboration so the team isn’t overwhelmed.

Strong Connection to Source Code

This approach makes a stronger foundation for something like tracking ML experiments. As ML teams race to get models to market and continually improve them, being in lockstep with DevOps teams to deploy those ML models into production is critical. An example would be a developer tracking five experiments with changing hyperparameters. For these experiments, they may have a link to a Git repo for a particular version of source code. But what if the source code changes or data set changes are not committed or reported in their existing ML solution? Then, the connection is lost and existing metrics would be based on a previous version of source code or data that’s no longer accurate. That lost connection slows down time-to-market and makes it harder to optimize models.

As seen with the above examples, ML teams juggle myriad tools and workflows as they develop models for various applications and services. The requirements discussed will help these teams to collaborate better and build models faster in an increasingly heterogeneous MLOps landscape.

May 13, 2022
Container Storage for DevOps: New Options With NVMe
The DevOps world is ripe for a storage paradigm change, both in how it operates within the enterprise and how it implements applications.

NVMe, especially NVMe-over-fabrics (NVMeoF), is both a tool and a host target that can provide such a change—especially in the new world of microservices and containerization. NVMe can drastically improve application performance over classic implementations and, given the right tools, transform the process of writing those implementations to be faster, more robust and more flexible.

I/O-Intensive Work

A quick recap shows why this is so. The DevOps build toolchain, especially for large codebases, has always been heavily I/O-dependent and storage access has classically been the biggest bottleneck for building an image. The ability to cluster small, inexpensive Linux x86 servers connected to SANs twenty years ago was a first step in separating storage hardware from compute, but I/O latency was pretty awful. Build/link operations could actually take longer than the compilation of object files.

Virtualization helped make hardware utilization better and cheaper but didn’t really improve throughput. Cloud infrastructures such as OpenStack and CloudStack made those infrastructures simpler and more elastic to manage but, again, the IOPs bottleneck remained.

Containerization for DevOps made it even simpler from the applications level—just deploy the app and ramp up container instances as needed—no more dedicated infrastructure needed. And it’s cheaper for incremental functions.

However, not much was done to clear the I/O bottlenecks for largescale storage and increasing overall performance.

When the new nonvolatile memory technology of NVMe came to market in 2014 for the single server world, it delivered two game-changing capabilities.
1. An increase of 5x disk access I/O speed. This was, arguably, the biggest single performance improvement since multicore processors with onboard cache (think Opteron) emerged. With Linux adding support for standard, open source NVMe drivers in 2021, the options increased beyond Windows and Mac users.However, SANs and their protocol stack were still an issue. SATA- and SCSI-based SANS performance was gated to the 1.2GB limitations of those technologies and big iron to head-end the SAN still dictated high-cost infrastructure.
2. Unlike SATA or iSCSI, NVMe has the native capability for TCP/IP or fibre channel addressing. This allows NVMeoF to be an autonomous point for data storage in a normal network. A user can now keep control of their data but use network-attached compute as a utility with roughly 6Gb speed for I/O—greatly lessening the big penalty for disk I/O.
Applying Containers

The dramatic uptick in container deployments and the debut of container attached storage (CAS)—software that includes microservices-based storage controllers that are orchestrated by Kubernetes—represented another industry milestone. These storage controllers can run anywhere that Kubernetes can run, which means any cloud or even bare metal servers or on top of a traditional shared storage system. Critically, the data itself is also accessed via containers as opposed to being stored in an off-platform, shared scale-out storage system.

Since container-attached, cloud-native storage enables the use of small, disaggregated architecture without the need for large-scale SANs or storage controllers dovetails nicely with DevOps work.

Container-based applications can benefit enormously from NVMe-based storage. As experts have pointed out, since DevOps applications are all about scalability, the parallelism embedded in the NVMe spec fits nicely to support them.

While SANs are good for large-scale contiguous operations, they’re expensive chunks of capacity and not nimble to deploy. They’re also bound to the slower I/O storage protocols, which can negate performance gains possible by adding NVMe to that array (as opposed to discretely).

Container-based storage with NVMe gives DevOps teams the opportunity to add storage at low cost with deterministic management of that stateful storage—in theory, of course. In practice, the temptation to add like infrastructure to current infrastructure is powerful, even with all its limitations.

To make that theory a reality, a control plane with the fast data plane is necessary—and the market is beginning to address this need.

As one example, OpenEBS recently introduced an upgraded engine, MayaStor (now part of DataCore Software), which enables use of NVMeoF at impressive speed: Benchmarks with Intel SPDK architecture show a 5x read/write speed improvement over SATA.

OpenEBS debuted a few years back as a containerized storage solution for Kubernetes-based applications. With an optimized storage engine, Kubernetes has the opportunity to perform like other, more specialized hosting and development solutions.

Impact for DevOps

What does this mean for DevOps teams? Firstly, using NVMe in containerized DevOps applications speeds up the time required for large builds, thereby speeding iterations of writing software. It also removes the need for complex and expensive SANs to host build environments. This double whammy of benefits means that development is quicker and cheaper and, in turn, applications can deploy faster while using more flexible tools.

Second, NVMe greatly mitigates I/O concerns for how application code is designed and how environments are implemented. No more being beholden to the storage trunk—NVMeoF drops the performance penalty for I/O ops and distributes traffic across a normal TCP/IP network. This is now a triple whammy—faster, cheaper and more flexible containers built atop a fast, flexible virtualized infrastructure.

Backed by innovations in container-attached storage, DevOps teams can achieve the agility and flexibility that are paramount to its mission—and act as a proving ground for the rest of the IT team to consider alternative paths to greater efficiency and performance.

Join us for KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain (and virtual) from May 16-20—the first in-person European event in three years!
May 12, 2022
The COVID-19 Pandemic’s Lasting Impact on Tech

The COVID-19 pandemic created major opportunities for innovation and ushered in new technology solutions, but even as the danger subsides, this tech may still serve a purpose.

Whether it be face mask detection, temperature checks, facial recognition or AI surveillance, technology crafted to combat COVID-19 is here to stay. This may present a number of benefits, but there are ethical issues that persist.

Throughout the pandemic, school districts sought ways to mitigate the risk of outbreaks and ensure students were complying with safety regulations. AI-empowered cameras, for example, were one of the first pieces of technology implemented, allowing schools to monitor which students weren’t wearing masks.

“I think it’s helpful for COVID-19 stuff, but it seems a little intrusive,” said Noa Young, a 16-year-old junior from Atlanta’s Fulton County school district. “I think it’s strange that we were not aware of that.”

Although students knew cameras were monitoring them, they weren’t informed about the extent of the surveillance. The cameras have the capability to scan for body temperature and can also determine if a student is wearing their mask correctly.

In addition to monitoring for compliance, these cameras have the ability to assist with contact tracing. They can scan students’ faces and determine who they’ve been in contact with, enabling schools to retrace the steps of someone who contracts COVID-19. This is just one example of pandemic tech’s continued development, but as adoption and use spreads to other sectors and it’s used for other purposes, security and privacy concerns begin to pile up.

Courtenay Brown, 31, works in an Amazon warehouse and explained the scrutiny she is under throughout her workdays. Amazon uses similar AI surveillance to keep tabs on worker productivity, notifying management if someone deviates from their assigned task for too long. Many believe this level of surveillance creates an inhumane work environment.

“They basically can see everything you do, and it’s all to their benefit,” said Brown. “They don’t value you as a human being. It’s demeaning.”

The implementation of this technology is one of the factors pushing Amazon workers to take a stand through unionization efforts.

“There’s no way we’re going to stop or let this bring us down,” said Derrick Palmer, co-founder of the Amazon Labor Union. “It’s going to do the complete opposite. We’re going to go 10 times harder.”

Another byproduct of the pandemic was the shift to remote work in organizations across the country and in every industry. This shift created the need for remote access to an organization’s tools, information, data and assets, which didn’t come without inherent security risks—especially since such access often needed to happen literally overnight.

One way companies adjusted was through adoption of identity and access management (IAM) solutions. IAM can be used to identify, record and grant access in the cloud, an integral part of secure functionality in hybrid or remote workplaces.

“In the game of cybercrime cat and mouse, one could argue there is no such thing as being over-resourced. A major breach can bring down a company,” said World Wide Worx CEO Arthur Goldstuck.

Although IAM proved to be a workable solution to the challenge of remote access management, cybersecurity threats are ever-changing and now cyberattackers have shifted focus—they may be attacking IAM solutions.

“Sophisticated threat actors are actively targeting identity and access management infrastructure, and credential misuse is now a primary attack vector,” Gartner described in its Top Security and Risk Management Trends 2022 report.

To keep data safe, companies will have to be ready for an even wider range of cyberattacks, including threats to IAM.

These are just two examples of technological breakthroughs that were originally seen as innovative solutions but which have evolved to have their own set of problems.

May 11, 2022
Update Those Ops Tools, Too

We’re all pretty well settled on the idea that the rate of change in IT has been on the increase over time. From the advent of VMs to Agile to DevOps to containers, the movement has been faster—and the same is true for internal development.

I’m going to approach this topic from three perspectives instead of my usual two. Kind of like breaking the fourth wall in cinema. We’ll discuss this movement just a bit from the perspectives of my two preferred bits, DevOps and DevOps. But for this blog, we’ll also mention DevOps—the infrastructure, tools and processes that enable the other two.

For DevOps, languages—both updates to those currently in use and the addition of new ones—bring with them a nearly automatic update to tools used. Favorite libraries/modules, IDEs, build tools—these are all looked at and updated more frequently today than at any point in history, by my estimation. For teams that do not change languages or libraries much, source code analysis (SCA) scanning is increasingly helping to take a closer look at all those included tools and filter them based upon impact (normally security impact, but other impacts on occasion, too—like project activity and stability).

For DevOps, the toolchain is still new enough that evolution is causing regular change, and that change is bringing with it new tools and utilities to solve the problems of this existing infrastructure. In fact, I’ve long thought the DevOps part needed a bit of a cooldown on the change and new tools/utilities front, personally.

And finally, DevOps has gone a bit sideways in this regard. Every person—whether you call them ops, admin or DevOps—responsible for troubleshooting machines and apps has a collection of tools. But the divergence of platforms has created a divergence of tools. It is possible to work without a dedicated set of tools—and, indeed, many DevOps engineers do. Others have a folder. Meanwhile, there is another dedicated set of DevOps engineers that have a PCMCIA or USB full of useful tools that is always with them. Some have both. Of course, what tools are included in all of this varies widely also. It is an interesting artifact of our growth in IT that developers tend to share or borrow the best libraries/modules/APIs across an organization, but DevOps is more independent. As long as DevOps team members are getting the job done, no one is all that worried about which tools they used to troubleshoot. They do tend to share if asked, but with little physical evidence of what tools they used, it doesn’t come up often.

So, what am I getting at? Look for ways to enable DevOps tool sharing on an ongoing (or at least regular) basis. An internal social media channel dedicated to the vendor your teams use to communicate, brown bags, a maintained rating system “Why I think fdisk rocks” type of thing, etc. The more power there is at the fingertips of every DevOps member you have, the better. Given our history, building an “approved” list is likely a bad idea, but having that central communication location that security can also peruse to look for red flags is not a bad idea. Of course, they are likely too busy, but worst case, such conversations can help in forensic analysis later. Not being cynical, just living IT in the 21^st century.

And keep kicking it. More and better tools will help keep all those sweet apps and servers clipping along, so there is really only upside here. Give your teams the tools and revel in your considerable uptime.

May 11, 2022
Does GraphQL Introduce New Security Risks?

The GraphQL query language is an excellent tool for increasing the ease of data sharing. The premise is that you request the fields you need in a single bundled request, avoiding multiple network calls. Due to its usability, GraphQL is a growing area of development. Yet, as with any new technology, it comes with caveats.

REST APIs function as a request-and-response paradigm in which the receiver accesses one resource at a time. On the other hand, GraphQL exposes many fields at once, requiring additional rate-limiting and time-outs to avoid nefarious queries. Additionally, the ability to traverse the entire schema could be abused by attackers to expose intricate data linkages. There are also some usability features with GraphQL that could be abused if misconfigured.

GraphQL is a friendly alternative to REST. But how do we handle the security implications of GraphQL? I recently chatted with Mike Benjamin, vice president of security research, Fastly, to dive deeper into the security repercussions of adopting GraphQL. Below, we’ll consider the unique insecurities associated with GraphQL and suggest specific steps to limit the impact of these risks.

Four GraphQL Security Vulnerabilities

1. Introspection

GraphQL provides built-in documentation that tells developers what the fields are, providing a schema of everything you can query. You’ve essentially given the entire dataset schema to a developer, meaning they can perform introspection to capture attack surface information. Hackers will most commonly look for injection or mutation capabilities that they can exploit, explained Benjamin. Or, they might seek to leverage the interlinkages between data to map a way through to classified information.

To mitigate this problem, Benjamin cautioned against over-exposing all documentation publicly. Instead, he suggested using a restrictive security model for external viewers. “Provide appropriate user documentation in a different method,” he says. This may end up looking more like the standard REST API view of documentation.

2. Field Suggestions

Another usability benefit in GraphQL could also be a potential security concern—field suggestions. Field suggestions occur when you don’t enter the correct field name or enter a partial input and the system returns a hint. You might type in 0 or user, and the system makes a suggestion to fill out the rest of the field.

This user interaction improves the developer experience, but, unfortunately, it could be abused by a hacker, said Benjamin. Thus, he recommended disabling hints after initial development. Or, if you decide to keep it in, he recommended placing a limit on its use. For example, if you try this more than five times in a second, the system could flag it as a bot.

Right now, this attack vector is pretty limited, but Benjamin anticipated that, over time, hackers could automate this introspection with mass scanners to divulge more information.

3. Denial-of-Service-Oriented Attacks

Without the proper guardrails, all web APIs are prone to denial-of-service (DoS) attacks. Such an attack may pummel a service with a million queries per second to shut it off completely. In the case of GraphQL, these queries can dive very deep in a nested data relationship. A social network, for example, might expose friends of users, friends of users’ friends and so on.

Or, hackers might utilize GraphQL’s bulk query ability to perform reconnaissance, brute-forcing the enumeration of data to query hundreds of user IDs to see which are valid. If you’re allowing unlimited data retrieval bundled within a single request, you could be inviting hundreds of bundled queries that could expose large swaths of the underlying database in one fell swoop.

To counter these actions, Benjamin recommended limiting the testing depth. This could involve placing limits on how long a query can run, which would cause a time-out for requests that require suspiciously high degrees of processing. Another option is to limit how deep the bundling can get, said Benjamin. For example, perhaps the system restricts going three levels deep into a graph database, as that could expose millions of interconnected endpoints.

4. Lack of Object-Level Authorization

Another vulnerability that GraphQL endpoints may be prone to is a lack of object-level authorization, which hackers could leverage to get at a datapoint through its edges, said Benjamin. Since the idea is that you can access any data from one all-encompassing endpoint, authorization rules are often too abstracted away from the actual data. Queries may be able to access nested data or query relationships between data, too, which may unintentionally divulge information.

Benjamin recommended placing authorization closer to your objects or data to rectify this issue. Use authorization-level controls that align permissions structure and roles with users. “Everyone needs a common authorization model,” said Benjamin.

Future Evolution: GraphQL Security Matures

As I’ve written before, the greatest strength of GraphQL is also its greatest weakness. Using GraphQL with default settings is a security trade-off in favor of usability. But according to Benjamin, we’re seeing more GraphQL servers disabling some introspection features, which he sees as a very positive move and a sign of maturity of the base configuration.

GraphQL might suffer from similar threats that all APIs are vulnerable to, such as XSS, injection, DDoS attacks and broken authorization. However, Benjamin stressed that the added complexity of GraphQL makes it a unique case. “The most important thing for folks to consider is that GraphQL is more complicated than a REST endpoint—much more complicated,” said Benjamin. “GraphQL is unique in that it has linkages of data which contributes to complexity. It takes a little more time to think about how that’s confirmed, how it’s presented and how it would be attacked.”

In other words, with great power comes great responsibility.

Although exploits in public GraphQL APIs are still rare, we should stay vigilant. Benjamin encouraged developers to continue reading about, understanding and thinking about these attack vectors when designing their interfaces, as these risks will become more prominent as GraphQL usage grows and hackers become more familiar with these specific frailties.

May 10, 2022
OpenSSF Adds Open Source Package Analysis Tool Prototype

The Open Source Security Foundation (OpenSSF) has made available a prototype of a package analysis tool that has already identified more than 200 malicious packages uploaded to PyPI and npm software components.

Caleb Brown, an OpenSSF maintainer of the project, said the goal is to understand the behavior and capabilities of packages available on open source repositories. It also tracks how packages behave over time to make it easier to identify when previously safe software began to act in a suspicious manner.

Most of the malicious packages detected are employing dependency confusion and typosquatting techniques, Brown said. They usually contain a simple script that runs during installation and provides a few details about the host to an external command-and-control system. Most of these packages are not exfiltrating meaningful data except the name of the machine or a username, but they make no attempt to disguise their behavior. The issue is that any one of these packages could have done far more damage, noted Brown.

Future goals for the project include detecting differences in package behavior over time; automating the processing of the package analysis results; storing the packages themselves as they are processed for long-term analysis and improving the reliability of the pipeline. As such, the OpenSSF is looking for application developers and cybersecurity researchers to contribute to the project. The expectation is that many providers of application security tools will incorporate the package analysis tool within their offerings rather than requiring each vendor to create their own version of the same tool, said Brown.

The OpenSSF has launched a series of initiatives to improve the overall state of open source security. Most recently, it added an Alpha-Omega Project to automate security testing processes for open source software using a $5 million initial investment provided by Microsoft and Google. The core challenge is that many organizations are dependent on open source software projects created and maintained by just a handful of volunteer maintainers and contributors. The individuals that created those projects don’t always have a lot of cybersecurity expertise. In fact, many of them would argue that the onus for securing open source software is on the organizations that use what amounts to free software. It’s not the responsibility of the contributors and maintainers to, for example, drop everything and immediately create a patch to address a zero-day vulnerability.

It’s not clear how easily open source software could be compromised by cybercriminals that insert malware into downstream applications that rely on that code. However, in the wake of a series of high-profile breaches, it’s safe to assume that more cybercriminals than ever are trying to compromise open source software projects. While some organizations may opt to limit their dependency on open source code that has not been properly vetted, there is no way to entirely avoid using open source code. The challenge is finding a more efficient way to ensure the integrity of that code for all concerned.

May 9, 2022
Stytch Launches New, Flexibility-First SDK

Passwordless solutions have been a trend for a while now, improving user experience (UX) while reducing exposure to common attack vectors. Now, Stytch’s new JavaScript SDK aims to make password-free authentication a bit easier by increasing customization and flexibility while offering fully baked authentication out of the box.

Julianna Lamb, Stytch CTO, said the solution provides dual benefits by doing the heavy lifting of implementation while providing the flexibility to create a powerfully branded UX from the customer’s first sign-on experience. The Stytch JavaScript SDK can offer an all-in-one experience that handles the entire user auth journey with a pre-built frontend component. But it can also give teams complete control over designing a sophisticated and tailored authentication journey. In an era where tech companies chase the ultimate customer experience as their holy grail, this should appeal to teams wanting to improve a customer’s first impression of their brand, Lamb said.

Multiple Options for a Smarter Front Door

Like many authentication tools, Stytch’s product suite originally offered two options: One was a standard widget to handle frontend components while backend APIs provided logins. The other option provided direct API integration, in which the developers built their frontend, handled tasks like session management and setting cookies on the frontend and used the API to do things like send a one-time passcode or email a magic link.

“But we found people really want flexibility when it comes to any solution they’re using,” Lamb said. “They also don’t want to do the heavy lifting when it comes to things like session management. Those things can be complex to reason through and having those out of the box is really compelling.”

The new SDK decouples drop-in UIs from helper functions, giving teams multiple options. They can place Stytch’s UI components anywhere they want in the product, sculpting the authentication flow to their specifications. As a headless SDK, it can be embedded client-side in the application to handle all the calls to the API, do the authentication, create a session and check its validity. The result: Less backend code to write and less logic to implement.

Customizable UI flows mean that teams can fully own the UX experience and build a UI that is customized exactly how they want it to look—and they can do this without having to build an internal API to route authentication requests from client to server. Teams looking for ways to make a brand impact right out of the gate can capitalize on this ability by creating a unique user experience.

“Authentication is the front door to your application,” Lamb said. “Sign-up is the first impression the user has of your app—and the ability to customize it with your brand look and feel so the UI fits your aesthetic is super-critical.”

Frictionless Flexibility in Identity Verification

Because teams may have different authentication needs within one product, the SDK can be used in tandem with the core Stytch API to meet shifting needs. Teams could generate embedded magic links via the API while providing Google OAuth and Google One-Tap sign-in options via the SDK.

The solution also reduces friction by making it easier to not only authenticate users but verify them on return visits to continue the same session. By loading the SDK on the frontend, teams can automatically connect Stytch’s sessions product to the browser cookie, feeding user session data directly into the cookie.

The Stytch team is currently building out more SDK functionality and frontend components, Lamb said. Given the ongoing evolution of simple authentication tools that meet complex requirements, it will be interesting to see where innovative authentication goes next.

May 9, 2022