devopsy.kinsta.cloud

Category: DevOps Toolbox

  • Supergraph: One GraphQL Schema to Rule Them All

    Supergraph: One GraphQL Schema to Rule Them All

    GraphQL has been garnering much interest as a way to seamlessly interact with backend services. The query language is a boon for frontend developers, too, as it conveniently allows you to fetch the specific fields you require, thus eliminating the potential overfetching or underfetching concerns of REST APIs. And as GraphQL adoption matures within organizations, there is increasing interest (and investment) in how enterprises can use GraphQL to securely unify their growing catalog of disparate microservices and APIs.

    Using GraphQL as a meta layer could help unite many disparate systems under a single unified schema. This would enable the age-old promise of composability, in which software building blocks are seamlessly pulled together to assemble applications and unique experiences for different client types. However, managing GraphQL at scale can become tricky. And when a schema gets really large, it then may require a schema of its own! This is where the concept of the supergraph comes in.

    I recently met with Geoff Schmidt, Apollo GraphQL CEO, to discover what a supergraph is and how it may help platforms aggregate their backend microservices. According to Schmidt, “Developers want this composition layer for one abstract view of all their services.” Below, we’ll explore the concept of the supergraph and look at some new open source components that may help you enable GraphQL-based composability in your organization.

    Microservices Enable Composability

    First off, what do we mean by composability? Well, composable architecture is intrinsically linked with microservices. You can think of each microservice as a single lego piece that makes up one aspect of an application. For example, Uber uses 4,000 microservices to power its backend. These services span from maps, currency conversions, driver reviews, payments and other components. When combined, they ultimately create the convenience and the sleek user experience that is Uber.

    Historically, software developers were overseeing larger monolithic applications in which a single client was interconnected to a web server, Schmidt explained. On the other hand, microservices tend to be more domain-specific and used in tandem with other microservices. They’re also typically easier to iterate on and faster to ship when compared to monolithic codebases.

    The Need for Aggregation

    As a result of the microservices age, applications typically pull in far more data from various sources. For example, a banking application might loop in a mortgage service, investment service and other data sources to display a combined net worth on a screen. This explosion in microservices has created a “polygon environment” spanning databases and APIs, increasing service complexity, said Schmidt.

    Nowadays, many third-party components must be combined to render an experience. However, supporting this composability can be challenging for developers and backend API teams—not to mention that handwriting a backend-for-frontend could be burdensome. “App developers are slowed down by the need to integrate or compose these backend services,” said Schmidt. “Developers need a way to put all those pieces back together.”

    What is Supergraph?

    Supergraph can be thought of as a graph of graphs. The vision is to have one GraphQL schema and one ingress port to oversee many services. Creating a supergraph is becoming more accessible, too, as more open source projects emerge to enable this paradigm. For example, Apollo recently open sourced some components that can help to build and operate a supergraph. These include Federation, which helps to create a data graph across multiple services, and the Apollo Router, a tool written in Rust that decides how to route queries. Apollo has also open sourced a supergraph demo along with other resources.

    A supergraph architecture is already powering Expedia, Walmart, Netflix, Paypal, the New York Times and other large environments, said Schmidt. For example, Expedia has used a supergraph to combine flights, rental cars and other microservices to create integrated experiences. Hivee, an upstart modern supermarket chain, also uses a supergraph to unite its investment into APIs.

    According to Schmidt, the fundamental benefit is that less time is spent writing integration code which means faster development speed and the ability to quickly pivot your business. “It allows you to change business quickly to meet the needs of your customers.” A supergraph could also enable greater visibility into your API estate, making auditing easier.

    A key benefit of the above components is that they already have a proven track record running at a commercial scale. But outside of the Apollo projects, other groups are attempting to codify a similar GraphQL schema aggregation concept, as well. For example, WunderGraph utilizes GraphQL within a “virtual graph” that acts as a common language to call various underlying APIs.

    Repercussions of ‘One Graph to Rule Them All’

    Schmidt said the hope is that, at the end of the day, this abstraction can contribute to “making the world a little more human-centric.” This is because GraphQL can enable less rigid apps that conform to unique user tastes. But a robust composition layer will be necessary to allow such reconfigurations.

    Of course, as I’ve covered previously, there are significant security hurdles to address when supplying so much power to a single entryway. Increased GraphQL reliance will require forethought into new potential security risks, such as schema introspection and object-level authorization control.

    According to Schmidt, a GraphQL composition layer presents an opportunity to enforce security controls. He defines these as contracts that create access rules for a particular set of users. By declaratively managing GraphQL security, a development team, in essence, could expose a subgraph of the supergraph for a partner to consume. The graph could even be segmented for internal teams.

    A central point of ingress will require authorization rules to protect user privacy and avoid breaking the rule of least privilege. Schmidt also advocates for backend schemas to be built for app developers who understand their needs, instead of precisely mapping logic to business models.

    Supergraph: A Graph of Graphs

    Software is becoming more modular, which has resulted in an explosion of unique microservices and APIs. And we’re now at another tipping point in how software architecture is divided and combined, said Schmidt. He now sees GraphQL as a prime opportunity to create software architectures that call multiple backend services with a single GraphQL query.

    “GraphQL is the composition layer you need to build modern applications,” said Schmidt. “We’re super excited about this. Users have shown that GraphQL is the future. Supergraph is such a powerful technology, and we’re excited to bring this to everybody.”

  • Open Source Coder Tool Helps Devs Build Cloud Spaces

    Open Source Coder Tool Helps Devs Build Cloud Spaces

    Coder today launched an open source project that gives developers and data scientists a way to create cloud workspaces in a matter of minutes to make it easier to build applications and models faster.

    Currently, Coder provides a cloud application development platform that enables developers to build applications using any editor or integrated development environment (IDE) as an alternative to their local desktop. That capability is made possible by using the open source Terraform infrastructure provisioning tool to spin up a developer workspace. That approach provides developers with access to cloud resources that scale up and down as required versus, for example, being limited to the memory and storage resources that they can access on a laptop or desktop system.

    Ketan Gangatirkar, vice president of engineering for Coder, said that while individual cloud service providers offer ways of achieving that goal, those offerings lock organizations into a specific cloud service. The Coder approach allows developers to use their preferred tool across multiple cloud services, he added.

    Most developers are already building applications that run on multiple cloud platforms. As such, they require access to a set of tools that eliminate the overhead created when developers need to deploy and master a diverse range of tools to build those applications, said Gangatirkar.

    In contrast, with the Coder tool, developers can open a web browser, choose a workspace template and then start coding, he noted. That capability eliminates all the manual, error-prone processes typically required to get a developer workspace set up, said Gangatirkar.

    That approach also provides organizations with a more secure means of building applications compared to the time and effort that would otherwise be required to secure every local system used by a development team, he added.

    In general, there’s a lot more focus on developer productivity as organizations increase the number of applications they are simultaneously building and deploying. Hiring and retaining developers remains a major challenge as competition for talent intensifies in the race to accelerate digital transformation efforts. All things considered, many developers will prefer to work for organizations that allow them to use the tools they already know and prefer. That approach also makes it easier to onboard developers to a development project, noted Gangatirkar.

    It’s not clear how many application development initiatives are never begun simply because it takes too much time and effort to spin up a development environment whenever inspiration strikes. However, it’s clear that developers are wasting a lot of time maintaining multiple local development environments. Much of that time could be better spent writing actual application code.

    One way or another, open source tools—such as the one provided by Coder—should remove a lot of the friction that exists in application development today. The only issue that remains is whether those tools will be adopted by developers themselves versus the DevOps teams dedicated to serving them.

  • One Year Out: What Biden’s EO Means for Software Devs

    One Year Out: What Biden’s EO Means for Software Devs

    It has been just over a year since president Biden issued executive order 14028 (EO) to improve the nation’s cybersecurity posture. Despite the Log4j vulnerability and a worldwide increase in ransomware attacks, this EO signaled a major step in improving software security at federal agencies and establishing cybersecurity as a priority for the U.S. government. While the agencies tasked under this EO have made significant gains, it is important for industry to understand just how much progress has been made since its inception and what that progress means to the average producer of software.

    To understand the effects of the EO, it is important to remember that president Biden does not have the authority to order any government agency to purchase specific software, nor does the president have the authority to require any governmental agency to procure specific software. Instead, the EO instructs heads of agencies to perform specific tasks that will ultimately define more stringent cybersecurity requirements to protect government offices. This doesn’t mean the EO solely applies to U.S. government contractors; if your company is anywhere in the software supply chain leading to a U.S. government purchase, then you need to pay attention to the EO and plan accordingly.

    One of the first of those tasks was for the National Institute of Standards and Technology (NIST) to solicit input from the public and private industry surrounding the structure and management of software supply chains. That work resulted in an NIST publication on EO Critical Software in July 2021. That publication helped outline security measures for software use, including applying practices of least privilege, network segmentation and proper configuration. It also helped NIST to frame the Secure Software Development Framework (NIST sp800-218v1.1) (SSDF) published in February, which provides a set of fundamental secure software development practices.

    This guidance, along with the definition of a software bill of materials (SBOM) by NTIA in November and the updated Cyber Security Guidance for Supply Chain Risk Management (NIST sp800-161r1) released in early May now form part of the risk-based framework that OMB instructed agencies to follow.

    What it Means To Be “EO Compliant”

    Before beginning any discussion about EO compliance, it’s important to recognize that until there are contract clauses defined—and those contract clauses are incorporated into contracts—there isn’t anything to comply with. That process is part of the Federal Acquisition Regulation (FAR) and Defense Federal Acquisition Regulation (DFAR) effort, and the proposed clauses haven’t been published yet. Despite this, we do know enough to start preparing for future compliance and creating processes that will allow for EO compliance.

    The first step is to recognize whether your organization is a supplier within a supply chain of a U.S. government agency. In the context of the EO, that would mean any organization who creates, procures or embeds software into a component or assembly that is ultimately bought by a U.S. government agency. This includes open source software, low-code/no-code frameworks and DevSecOps tooling, to name just a few of the scenarios where U.S. government use might not be immediately apparent. 

    With that in mind, any provider of software within a supply chain should be prepared to answer questions about EO compliance if their software can be used in the U.S. government setting. For some, this could mean updating a few processes; for others, it may prompt a much larger set of changes in the development cycle. Those willing to change may be more likely to retain relationships or gain favor with government suppliers.

    How Software Providers Can Be Proactive

    There are three critical steps that can be performed today as part of preparing an organization for future EO compliance requirements. The first thing providers can do is generate a complete SBOM for their software in an NTIA compliance format and be prepared to attest to its accuracy. While it is not yet clear how the U.S. government will expect to receive an SBOM or what attestation format will be required, generating compliant SBOMs is one of the easiest tasks software producers will face as part of a post-EO world.

    Next, providers should baseline existing software development practices with the objective of aligning with the SSDF (NIST sp800-218v1.1). The SSDF defines a set of tasks that NIST views as a minimum requirement to securely create software and manage the software creation process. If the government procurement requirements include an attestation of compliance to the SSDF, then software producers who already know how well-aligned they are to the SSDF will be that much closer to compliance. 

    Lastly, providers must implement controls related to open source governance. While an SBOM helps identify open source use and the SSDF includes tasks related to its use, neither specifically defines how any software producer is expected to govern their use of open source software. Open source software is fundamentally different from commercial software in many ways; the most important differentiator is that it is freely downloadable from the internet. This free access means that creators likely do not know who their consumers are and thus cannot proactively provide updates or security information. Open source governance processes can help to fill that gap.

    Why It is Time to Embrace the EO

    These first steps can help software providers address the initial requirements that have come out of Biden’s EO and those who get a jump on implementation will have a leg up against their competition. Starting now can save providers a major headache in the long run, as federal agencies seek an understanding of how much risk they bear when operating the software you produce. 

    The rapidly changing cybersecurity landscape was a key driver behind Biden’s EO. Threat actors are evolving their tactics to keep pace with improving cybersecurity postures and it is in everyone’s best interest—from the federal government to the private sector—to put an emphasis on security. An important place to start—regardless of where you are in any software supply chain—is by taking the steps necessary to demonstrate that you understand the risks inherent in operating software and are actively working to reduce those risks for your customers.

  • Codenotary Adds SLSA Framework Support to Advance App Security

    Codenotary Adds SLSA Framework Support to Advance App Security

    Codenotary this week announced it has integrated support for the Supply-Chain Levels for Software Artifacts (SLSA) framework in its free notarization and verification service for ensuring the integrity of code.

    Moshe Bar, Code Notary CEO, said as the first application security platform to attain SLSA compliance, the company is making it easier for organizations to secure their software supply chains.

    SLSA is a framework defined by Google that it uses to attach binary authorization to code. In its current form, SLSA is a set of guidelines that identify how to attain four different levels of application security. The long-term goal is to create a framework that will automatically generate auditable metadata that can be fed into policy engines to create a SLSA certification.

    Codenotary provides tools for cataloging and verifying the authenticity of components used with a software development life cycle. At the core of that platform is an immutable open source immudb database that cryptographically attaches an identity to each software artifact. That capability can also be used to dynamically generate a software bill of materials (SBOM).

    The Codenotary Attestation Service can also be integrated with the continuous integration/continuous delivery (CI/CD) platform that DevOps teams use to build and deploy applications using those artifacts. Ideally, DevOps teams should construct a security pipeline that runs alongside their existing DevOps pipelines, noted Bar.

    In the wake of recent high-profile software supply chain breaches, the level of scrutiny being applied to how software is constructed has increased significantly. In theory, at least, developers have assumed more responsibility for application security as part of an overall shift left that provided them with more programmatic control over IT environments. Cybercriminals, however, are now becoming more adept at compromising developers’ credentials and insert malware as an app is developed.

    DevSecOps best practices are, of course, being adopted to teach developers how to better secure their application environments. The issue is that not enough developers are learning how to secure applications fast enough. The volume of attacks against the software supply chain continues to grow at a rate that is now much faster than organizations can thwart simply by educating developers.

    There is a now clear need to embed more cybersecurity guardrails within DevOps workflows, noted Bar. Rather than attempting to achieve that goal in a piecemeal fashion, Bar said Codenotary is making a case for an integrated platform that can be invoked as a natural extension of a DevOps workflow.

    In general, Bar said there’s now a greater sense of urgency when it comes to software supply chain security. Many of the reviews of those processes are, arguably, long overdue. One way or another, there will be a lot more questions asked about how software is actually developed. The organizations that consume this software are starting to appreciate how vulnerable they are as the overall dependency on software continues to increase. The only real question now, unfortunately, is how many more breaches there might be before those software security issues are thoroughly addressed.

  • Armory Aims to Turn Continuous Delivery Into a Service

    Armory Aims to Turn Continuous Delivery Into a Service

    Armory this week made generally available a continuous delivery-as-a-service (CDaaS) offering that promises to make it simpler for a much wider range of organizations to programmatically deploy applications.

    The Armory Continuous Deployment-as-a-Service platform has been available in beta since March. It differs from Armory’s existing open source Spinnaker continuous delivery (CD) platform in that it is based on a proprietary declarative platform that is designed to integrate with third-party continuous integration platforms.

    Armory CEO Jim Douglas said the goal is to meet organizations that have invested in CI where they are versus requiring them to rip and replace their entire DevOps platform. The company is still committed to providing a curated instance of Spinnaker, but it’s become apparent there are many organizations that want to experience the benefits of a more opinionated CD platform that is more accessible via a cloud service, he added.

    The Armory Continuous Deployment-as-a-Service offering applies many of the best practices that have emerged for the Spinnaker CD platform without requiring organizations to have a dedicated team in place to just manage the CD platform, noted Douglas. IT teams can start using the service to deploy applications in less than 30 minutes, he noted.

    The first edition of Armory Continuous Deployment-as-a-Service is optimized for deploying microservices-based applications on Kubernetes clusters, but Armory will soon extend the reach of the service to platforms running monolithic applications, said Douglas. The company also plans to make available an on-premises edition, he added.

    In either scenario, IT teams will be able to engage in everything from canary testing of deployments to automatic rollbacks in the event an application fails, said Douglas.

    The number of organizations that routinely make use of a CD platform is still relatively small. Historically, each platform used to run applications has been unique, so automating delivery has been problematic. CI platforms, in some cases, have been extended using custom scripts to automate application delivery, but those efforts don’t typically scale across an extended enterprise.

    Douglas said that as more IT teams start to build and deploy microservices-based applications, it’s only a matter of time before CD becomes a more distinct IT discipline. The challenge is enabling IT organizations to achieve that goal without necessarily having to hire a small army of site reliability engineers (SREs). There will still be plenty of DevOps teams that will prefer to deploy Spinnaker, but Douglas said there will also be plenty of instances where a less complex approach to CD will suffice.

    It’s hard to say whether there might soon be varying levels of DevOps platform services that organizations might invoke depending on the complexity of the project and their own internal cultural preferences. However, as the pace of application development continues to accelerate, the need to automate CD will become a more pressing issue. There may be many ways to achieve that goal, but if history is any indication, the path of least resistance is likely to be via the cloud.

  • Cloud Giants Shun Wind Power? | LaMDA not Sentient? | MS IE RIP?

    Cloud Giants Shun Wind Power? | LaMDA not Sentient? | MS IE RIP?

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Warren Buffett asks too much for wind energy, it’s “completely ridiculous” to say Google’s chatbot LaMDA is sentient, and Microsoft finally kills Internet Explorer. (more…)

  • Survey Surfaces Raft of Application Security Issues

    Survey Surfaces Raft of Application Security Issues

    A global survey from ReversingLabs found 87% of respondents agreed that software tampering has become a more frequently-used cybersecurity attack, but only 37% said they have any means to detect it.

    The survey, which polled 300 IT and security professionals, was conducted by Dimensional Research on behalf of ReversingLabs, a provider of a platform for analyzing any file, binary or object. Despite software tampering’s frequency, only 7% of survey respondents said they could detect software tampering at each phase of the software development life cycle.

    Just as troubling, only slightly more than half (51%) reported being able to protect their software supply chain with only a third being able to actually check for tampering once an application is final and deployed, the survey found.

    Overall, more than half of respondents (54%) said their organization knowingly released software with potential security risks. Nearly every respondent (98%) also acknowledged that reliance on third-party software use, including open source software, increases security risks.

    On the plus side, more than three-quarters of respondents (77%) said they appreciated the value of a software bill of materials (SBOM) that makes it possible to ascertain what software artifacts are being employed within an application. However, only 27% of respondents said their organization currently generates and reviews SBOMs. A full 90% said they are finding it increasingly difficult to create and review SBOMs. Nearly half of the survey respondents reported that SBOM processes today involved manual steps.

    Other SBOM hurdles respondents cited included lack of expertise (44%) and inadequate staffing to review and analyze SBOMs (44%).

    ReversingLabs CEO Mario Vuksan said that while it’s still early days as far as SBOM capabilities are concerned, a series of high-profile zero-day vulnerabilities has forced the issue into the spotlight. Organizations are now spending months looking for every possible instance of newly discovered vulnerabilities because no one is quite sure what application components are running across an extended enterprise, he noted.

    However, as DevSecOps best practices continue to evolve and mature, it’s also clear that SBOMs will play a critical role in securing software supply chains, Vuksan added. The challenge is that in the age of microservices, application environments are now much more dynamic as components are ripped and replaced with greater frequency, he added.

    SBOMs, of course, are just one element of what will be required to secure a software supply chain. Many organizations are now reviewing their software development processes and uncovering uncomfortable truths about how software is currently built and deployed. The ultimate goal is not to slow down the rate at which software is developed but to make sure there are guardrails in place that prevent insecure software from being deployed in the first place. Far too many developers today still view cybersecurity reviews as tasks to be avoided in the name of meeting an application deployment deadline.

    Hopefully, that won’t continue to be the case as more responsibility and accountability for applications security shifts left toward developers and DevOps teams. In the meantime, however, when it comes to application security, it’s apparent there’s still much room for improvement.

  • Jit Emerges to Address Cybersecurity Policy-as-Code Orchestration

    Jit Emerges to Address Cybersecurity Policy-as-Code Orchestration

    Jit today emerged from stealth with a free namesake orchestration platform beta that automatically implements security plans as code.

    Aviram Shmueli, chief product officer for Jit, said the company will both make security plans it developed available as code within GitHub. In addition, Jit will make available a service through which customers can submit documents and spreadsheets that Jit will convert into a custom security plan that can be executed as code.

    Fresh from raising $38.5 million in seed funding, Jit is aiming to advance the adoption of DevSecOps best practices by enabling cybersecurity policies to be more easily implemented as code via its orchestration platform, said Shmueli. The Jit platform enables developers and DevOps teams to take advantage of a graphical tool to automatically implement security policies within a DevOps workflow with a single click, he noted.

    The goal is to eliminate the need for separate DevOps teams to create security policies that each would need to individually manage, added Shmueli.

    In theory, DevSecOps best practices require shifting left the responsibility for security toward application development teams. The challenge is that development teams generally don’t have the expertise required to craft effective cybersecurity policies. Jit is making a case for a platform that makes it simpler to automatically implement vetted cybersecurity policies.

    Exactly how far responsibility for application security will shift left remains to be seen. Developers generally prefer to spend as much time as possible creating business logic rather than implementing cybersecurity policies. As such, it’s frequently left to a DevOps team to find a way to consistently implement cybersecurity policies across multiple applications. Each DevOps team will need to decide how much they can rely on the policies defined by Jit versus opting to implement a set of policies defined by their internal cybersecurity teams. The issue that often arises is cybersecurity teams lack programming skills; the only means available to define cybersecurity policies is some type of document or spreadsheet that, one way or another, needs to be converted into code.

    Given the ongoing chronic shortage of cybersecurity expertise, it’s apparent that more efficient methods for sharing cybersecurity best practices are required at a time when application environments continue to both rapidly expand and increase in overall complexity.

    Regardless of the DevSecOps approach, the pressure to address application security is mounting in the wake of a series of high-profile cybersecurity breaches. Organizations of all sizes are now reviewing software supply chains that are increasingly being targeted by cybercriminals. Security policy-as-code tools make it possible to ensure that applications don’t, for example, communicate with an external system without express permission.

    One way or another, the level of application security currently being attained will improve in the months ahead as more mandates are enforced. The only real issue remaining is the amount of disruption to existing processes that will be required to achieve that goal.

  • Splunk Adds Data Management Tools to Observability Platform

    Splunk Adds Data Management Tools to Observability Platform

    At its .conf22 event, Splunk today announced it is making it easier to both onboard data and then manage it across hybrid IT environments via the Splunk Cloud Platform.

    In addition, Splunk Enterprise is being extended to add support for Microsoft Azure with SmartStore for Azure to store cold data alongside existing support for Amazon Web Services (AWS) and Google Cloud Platform (GCP). There is also a developer edition of Splunk Cloud that enables developers to create and test their own applications, which is currently in preview.

    Garth Fort, senior vice president and chief product officer for Splunk, said the overall goal is to make it easier to automatically tune the parameters used to store data in the Splunk observability platform. As more organizations embrace observability as a DevOps best practice, many of them are encountering challenges managing the large volumes of data they are now collecting in the form of logs, metrics and traces.

    The Data Manager for Splunk Cloud Platform will initially be made available on AWS and Azure, with support for GCP to come later this summer. In effect, it provides IT teams with a control plane for data management that spans multiple clouds.

    Ingest Actions, meanwhile, makes it possible to apply granular controls that enable IT teams to filter, mask and route data in motion within the Splunk Platform or to external AWS S3 cloud storage services.

    Other capabilities Splunk unveiled at the conference include an expansion of Federated Search tools to enable a single pane of view across their entire Splunk ecosystem and a fully managed Splunk Assist service for Splunk Enterprise 9.0 that provides customers with deep insights into their security environment.

    Finally, the company has enhanced Splunkbase, a framework for accessing curated collections of more than 2,500 applications and integrations provided by Splunk and its third-party partners.

    As the pioneer of IT operations analytics, Splunk has a large footprint within enterprise IT environments. It is now looking to expand that base to become an observability platform provider. It’s still early days as far as the transition to observability is concerned, but the goal is to make it simpler for IT teams to query platforms and surface issues that could lead to an IT disruption before it happens.

    It’s not clear how many organizations have the skills and expertise required to formulate those queries, but over time it should become more feasible to augment an IT staff with machine learning algorithms capable of discovering those issues.

    In the meantime, IT organizations are collecting more data than ever about their IT environments. In theory, that data will provide the context needed to optimize application environments that get more complex with each passing day. The challenge, of course, then becomes finding a way to efficiently store and manage all that data both before and after it has been analyzed.

    Regardless of the approach to observability, the need to go beyond simple monitoring of IT environments has never been more apparent.

  • CDEvents Aims To Standardize CI/CD Interoperability

    CDEvents Aims To Standardize CI/CD Interoperability

    Companies are under mounting pressure to continuously release new applications and features. And as software development life cycles become more iterative, continuous integration and continuous delivery (CI/CD) has become a necessary ingredient to enable more rapid deployments.

    Many development teams now embrace DevOps, but fully automating a CI/CD pipeline from source code to production deployment is no easy task. For one, the number of tools in use continues to balloon, meaning CI/CD poses an ongoing maintenance hurdle. Furthermore, CI/CD tools often adopt varying formats to describe their events. A CI/CD pipeline, a form of event-driven architecture, may require a lot of special “glue” to patch events together if the output from each tool is not in a standardized format.

    Interestingly, CDEvents, an emerging open source standard from the Continous Delivery Foundation (CDF), is working to standardize the root semantics of how CI/CD tools describe the payloads of their events with more granularity than CloudEvents. The project aims to establish schemas for how common CI/CD events are described, easing integration and providing more actionable data surrounding the CI/CD pipeline.

    I recently met with Andrea Frittoli, open source developer advocate at IBM, to understand why the CDEvents project exists, how it works and what they’re hoping to achieve. According to Frittoli, who is also co-creator of the CDEvents project and a member of the CDF Technical Oversight Committee, CDEvents has the potential to bring much-needed interoperability to the sprawling CI/CD tooling ecosystem. Below, we’ll investigate where the project is and consider where it’s heading.

    The Problem With CI/CD

    At its core, the problem CDEvents is trying to solve is a lack of interoperability in the CI/CD field, explained Frittoli. As it stands today, DevOps engineers must routinely make point-to-point integrations between CI/CD tools. “No single tool will take you entirely from managing source code to production,” he said. A single development team could be using up to 15 unique components in one pipeline, and tooling preferences can vary dramatically across teams, he adds, further complicating CI/CD interoperability.

    Although CloudEvents has made inroads as a specification to describe event data, nothing has yet existed specifically for describing CI/CD events. According to Frittoli, CloudEvents falls short here, as it’s agnostic about what is inserted into the payload.

    Understanding CDEvents

    CDEvents is “a common specification for Continuous Delivery events.” Frittoli describes it as a shared dictionary or shared semantics for common events. These events could be a change in source code, running a test, or a code execution or deploy. CDEvents uses JSON object schemas to describe the event types and can work alongside CloudEvents as an attribute. “CDEvents is a standard format for the payload as well, not only for the envelope,” said Frittoli.

    As Frittoli explains, the CDF special interest group is still drafting the CDEvent specification’s exact semantics and common names. Currently, CDEvent organizes these objects into four main buckets: Core events, source code version control, CD events and CI events.

    While the group is still finalizing the format of events as they work toward the initial release, the example below is an approximation of what the final format will look like:

    "meta": {
  "version" : "draft",
  "event_id" : "A234-1234-1234",
  "source" : "/staging/tekton/",
  "type" : "dev.cdevents.taskrun.started",
  "timestamp" : "2018-04-05T17:31:00Z"
},
"subject": {
  "id": "/namespace/taskrun-123",
  "type": "taskrun",
  "content": {
    "URL": ...
    "pipelinerun": ...
  }
},
"extensions": [{
  "name": ,
  "content": ...}
],
"links": [{
...
}]

    Benefits of CI/CD Event Standardization

    So, why is it necessary to define the format of CI/CD event data? What benefits does this bring to developers and the business as a whole? Well, Frittoli sees two potential primary use cases. First off, CDEvents could reduce the effort spent writing and maintaining glue code between common DevOps tools like Spinnaker, Argo or Flux. “This will allow a smoother integration between all the parts,” said Frittoli. Alleviating this could help scale CI/CD initiatives and increase overall agility.

    Another positive side effect is collecting more detailed information about workloads and their performances. If all these components generate similarly-formatted events, Frittoli explained, then you can begin to create a true end-to-end visualization of the entire CI/CD pipeline. Then, you can start to measure things such as deployment frequency, failure rate, time to change and other metrics.

    If events begin to use a common language, you could do many interesting things with that data, says Frittoli. Event data could help ascertain the root cause of incidents and aid debugging. One could even foresee some AIOps potential to further automate operations and increase observability around CI/CD.

    Final Thoughts and Ways to Participate

    CDEvents emerged at a time when iterative software development has become commonplace. A slew of helpful CI/CD tools is at our disposal to enable niche operations. Yet, the space still requires a standard way to operate between them. CDEvents is an exciting prospect, yet the project is still in its early stages. It has yet to reach a full release or any production use and proofs-of-concept are still being developed.

    Of course, encouraging a vendor-neutral specification requires the tooling community to get on board. “It’s vital for the different tools to implement CDEvents for it to be successful,” said Frittoli. He encourages the tooling community to join the working group to collaborate in building proofs-of-concept. The SIG plans to continue working on finalizing the first version of the specification. In the meantime, the group has produced a proof-of-concept using Tekton and keptn and the project has its future sights set on Knative and Jenkins.

    The CDEvents project is in incubating mode and maintained by the Events Special Interest Group (SIG) at the CDF. For more information, you can peruse the CDEvents documentation here or become involved with the community here. The Events SIG also holds biweekly meetings and a CDEvents Community Summit will be part of the upcoming CDcon.