devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

  • Survey Surfaces Agile Development Challenges

    Survey Surfaces Agile Development Challenges

    A survey of 3,220 software development professionals published this week found that among the 80% of respondents that embraced Agile practices for building software, just under half are tracking on-time delivery (47%) and business objectives achieved (44%).

    The survey, conducted by Regina Corso Consulting on behalf of Digital.ai, a provider of a value stream management (VSM) platform. The survey also found a total of 19% of respondents were not sure what metrics were used to track software development progress.

    At the same time, 40% of respondents noted they are dissatisfied with Agile at their organization; a clash with company culture was listed as the leading cause (41%) of unsuccessful delivery. Half of respondents use a combination of Agile, waterfall and/or iterative processes, but 27% noted this combination of frameworks was a struggle for them.

    The survey also noted that just over a third of product and application owners (36%) that participated in the survey have adopted Agile.

    Derek Holt, general manager of Agile and DevOps at Digital.ai, said that percentage suggests that Agile principles are now being adopted beyond the engineering teams that construct software. As that appreciation for Agile principles continues to grow, the organizations these individuals work for also tend to start tracking value stream metrics, he added. That transition typically begins with a few pragmatic metrics before the organization matures enough to justify the adoption of a VSM platform, Holt noted.

    In time, however, that VSM platform not only helps organizations identify bottlenecks but also identifies potential quality issues and risks to the business, Holt added.

    Overall, the top three top Agile development priorities identified by respondents are to accelerate time-to-market (52%), delivery predictability (44%) and lower risk (31%). However, from a business perspective, the top three priorities are company goals (54%) and end customer satisfaction (43%) followed by time-to-deliver (35%).

    More than half (51%) of respondents, however, also noted that DevOps teams are not incorporating business feedback regularly. Well over half (58%) said the most valuable improvements to DevOps teams would be end-to-end visibility and traceability from business initiative through development, test and deployment to end users, while 55% said the ability to measure cycle time, wait time and bottlenecks would be an important improvement.

    Regarding Agile tools used, Scrum led the pack at 87%, followed by Kanban at 56% and Scrumban at 27%. The Scaled Agile Framework (SAFe) framework has been adopted by more than half of respondents (53%), while Scrum@Scale/Scrum of Scrums was adopted by 28% of respondents followed by Lean Management at 8%, according to survey respondents.

    The top benefit of Agile is increased collaboration (69%) followed by better alignment with the business (54%) and better work environment and increased visibility in the application development life cycle at 39% each.

    The debate over the merits of Agile frameworks, of course, remains fierce and no two organizations may ever have the same level of adoption. However, as the pressure to deliver quality software faster in the age of digital business transformation continues to mount, the more crucial it will become to align Agile and DevOps best practices.

  • Introduction to MACH Architecture

    Introduction to MACH Architecture

    In recent years, the need to construct end-to-end digital experiences has become increasingly important. But to deliver new features and services promptly, organizations must double down on agility and reusability. As part of this trend, applications have become increasingly modular, often constructed using pre-built microservices and integrating common SaaS to avoid reinventing the wheel.

    MACH is an architectural style that fits into these trends quite well. It stands for Microservices-based, API-first, Cloud-native, and Headless. The locus of the movement is The Mach Alliance, a group of API-first companies abiding by these principles. At the time of writing, MACH is gaining mindshare as more API-first companies enter the market and the appetite for cloud-native technology grows.

    I recently chatted with Gary Ballabio, VP of technology partnerships, Cloudinary, about the ongoing MACH trend. According to Ballabio, MACH delivers “best of breed” technologies that avoid the bloat of larger platforms. Yet, for software consumers, building in a MACH style will require stitching together individual components and orchestrating them effectively. Below, we’ll explore MACH and consider the benefits and drawbacks of utilizing tools that conform to this approach.

    What Is MACH?

    MACH is an emerging software architecture style adopted by a number of commerce and digital experience tools. To understand MACH a bit deeper, it’s necessary to understand the underlying concepts of microservices, API-first, cloud-native and headless. So, let’s first break apart these familiar terms:

    • Microservices: Unlike monolithic development, a microservices architecture organizes functionality into self-contained, specialized components. This improves reusability and allows for more frequent, targeted software updates.
    • API-first: API-first takes design-first one step further. It means putting the consumption model, the API, ahead of other concerns, like the application or UI. Web APIs have become ubiquitous for connecting a microservices ecosystem and exposing data and functionality to partners and public consumers.
    • Cloud-native: An umbrella term for projects purpose-built for the elasticity and scalability of the cloud.
    • Headless: Headless means separating the front end, or “head,” from the back end. By decoupling development this way, websites and apps are better equipped to cater to multiple platforms and devices.

    According to Ballabio, companies can leverage cloud-ready best-of-breed functionality by integrating tools that adopt MACH principles. Incorporating one single SaaS to fit a specialized need can also be considered an alternative to using large, comprehensive enterprise commerce suites like Salesforce, Adobe or SAP, said Ballabio.

    There are many examples of specialized SaaS services built in the MACH style. Some include Algolia for search and recommendations, Cloudinary for media management, Builder.io for mobile experiences, or Contentstack as a headless CMS. All these tools are members of The MACH Alliance, a group of over 70 vendors, partners, startups, systems integrators and other ambassadors dedicated to fostering the MACH approach. And although the emphasis has been on commerce and digital experiences, MACH is expanding beyond that scope to other areas.

    MACH technologies support a composable enterprise in which every component is pluggable, scalable, replaceable.

    Some readers may wonder, “How is this different than Jamstack (JavaScript, APIs and markdown)? In a way, MACH is similar to Jamstack, as both approaches aim for headless frontend experiences and a modular backend powered by APIs. But a key difference, said Ballabio, is that MACH doesn’t dictate the use of specific programming languages.

    Benefits and Drawbacks of MACH

    MACH combines many development trends we’re seeing in the software landscape. Digital applications are becoming more composable, cloud-first is rising, and headless models can deliver customization benefits. Yet, there are some potential drawbacks to introducing this style too. So, let’s weigh some of the positives and negatives of adopting MACH architecture.

    Benefits

    • Flexibility: The most significant benefit of MACH is the flexibility it offers, said Ballabio. “There’s value in being able to choose the services you want to use based on your own unique requirements.” Looping in a specific API-first tool for something like e-commerce, payments, CMS or email can enable a lot of customization.
    • Reduced waste: This can create less waste, too, as you create a leaner footprint by acquiring only what you need for each project, and freemium models enable consumers to pay for only what they use. This is helpful when you want a single component but don’t require the bells and whistles of a more holistic platform.
    • Best-of-breed: Niche API-first products are focused on doing one thing really well, and this particular focus can improve quality and agility. It also releases engineers from being at the whim of an extensive product’s development life cycle.
    • Developer-first: Lastly, most digital experiences share common functionalities that developers shouldn’t have to reinvent. MACH supports being developer-first, says Ballabio, which is an asset to companies investing in developer experience.

    Drawbacks

    • Increased maintenance: Consuming many third-party cloud services will inevitably bring new maintenance hurdles. These include tracking, monitoring and securing all integration points. It also requires responding to new version updates.
    • Stitching together: Naturally, when you add a cloud-based abstraction layer to the services you consume, the onus is on the consumer to then to stitch them together to make components talk to one another.
    • Startup costs: Due to the coordination between services, initial costs and engineering time could be more than implementing a holistic suite, admitted Ballabio. “But you get more long-term benefits,” he said.
    • Assistance may be required: Those unaccustomed to microservices development and API integration may need help. This may require the support of systems integrators or middleware to orchestrate connections.
    • Lack of standards: MACH is more of a general architecture rather than a standard specification. Thus, the precise technologies under the hood and the specific behaviors may differ from service to service.

    Future of MACH

    A recent MACH Alliance study found 79% of tech leaders plan to invest more in MACH technologies in the next year. Although the style has been primarily trending in Europe, Ballabio predicts the U.S. will soon pick it up. The MACH Alliance presents and advocates for an open and best-of-breed enterprise technology ecosystem.

    Along with the rise of composable architecture, the MACH approach is gaining interest among cloud-native leaders who see the advantage of assembling applications using pre-fabricated building blocks. MACH is especially a good fit for greenfield development, where the slate is clean to integrate new technology.

    MACH represents a new category of companies that are growing together. Ballabio remarks that he feels part of a movement, part of an industry—and, arguably, this movement can potentially disrupt larger commerce ecosystems.

    But although MACH gives a name to a face, most techies still aren’t aware of the acronym. While the concept may, for now, live in the domain of vendor-speak, Ballabio foresees it becoming a more common parlance for consumers and end developers.

  • Chainguard Adds Private Edition of Code Signing Platform

    Chainguard Adds Private Edition of Code Signing Platform

    Chainguard today added a private preview of a Chainguard Enforce Signing service, enabled by the open source Sigstore project, that allows developers to generate digital signatures for software artifacts using identities and one-time-use keys they create themselves.

    Kim Lewandowski, head of product for Chainguard, said Chainguard Enforce Signing provides an alternative to relying on a public service to generate those digital signatures. It also allows organizations to bring their own key and certificate so that key usage can be monitored and audited per compliance and privacy requirements.

    In addition, Chainguard added a library of policies that administrators can deploy directly from the Chainguard Enforce web console to their environments.

    The Chainguard Enforce Signing feature is based on Sigstore, an open source project that provides a protocol to simplify cryptographic certificate generation by removing the need to manage keys. At its core, Sigstore creates a set of short-lived ephemeral keys with a trust root based on an auditable public log. Because it creates an immutable log, everyone participating in the software development process, including auditors, can now transparently track the chain of custody for any software artifact.

    Key management, key compromise/revocation and the distribution of public keys and artifact digests are challenging to manage. Developers are also required to determine which keys to trust in addition to learning how to sign them when different platforms are employed.

    Chainguard’s alternative is enabling organizations to host their own immutable log rather than relying on a public service to eliminate the need to manage keys, noted Lewandowski.

    Historically, code signing platforms have been problematic because they were susceptible to tampering. Cybercriminals can, for example, swap out digests to gain access to compromised credentials. Digests and public keys are also often distributed across vulnerable websites susceptible to hacks or inadvertently stored on a public Git repository.

    Code signing—along with software bills of materials (SBOMs)—is one of two technological advances that will improve the security of software supply chains. SBOMs, of course, are going to be considered more reliable when the software components they describe are signed by the developers that created them.

    Less clear is whether the signing of code will improve its overall quality. Most developers take pride in their code, but the overall quality of the software deployed in production environments could always be improved. Developers may decide to review the code they contribute to a build more intently if they know they will be required to sign it.

    In the meantime, it’s only a matter of time before organizations require that developers sign code before it is deployed in a production environment. In addition to increasing confidence in the quality of the code employed, those signatures make it simpler to track down the development team that created code whenever there is an issue that needs to be resolved. After all, the most challenging issue organizations encounter when trying to update software is not so much the creation of a patch as much as it is identifying who should create that patch based on their knowledge of the software components used.

  • Looking to the Future of Developer Experience

    Looking to the Future of Developer Experience

    Developer experience is where it’s at! Developer experience (DX) means taking a user experience approach to the developer journey. DX (also abbreviated as DevEx) is commonly applied to areas like documentation, sandbox environments, SDK ergonomics and more. By improving a developer’s experience with a tool, software providers can increase stickiness and retain more users.

    Nowadays, having a sleek developer experience is a competitive advantage—especially since developers often have the discretion to purchase new tools. And amid an ongoing skills gap, maintaining quality DX for internal software ecosystems is also important to retaining happy engineers and avoiding burnout. All these factors mean developer experience is becoming a C-suite objective. So, where is DX heading? How will the developer workflow change in the near future?

    I recently met with Ash Arnwine, director of developer relations, Nylas, to ponder what’s on the near horizon regarding developer experience. According to Arnwine, DX is still a very new concept. Yet, it has become an integral aspect of positioning modern software, especially for public-facing SaaS, and will undoubtedly drive purchasing decisions in the year ahead. Below, we’ll share some predictions of how DX will evolve within four general categories: onboarding, building, deploying, and maintaining.

    State of Developer Experience

    First off, what is the state of developer experience within the tech world at large? Well, Arnwine is quick to note that the concept is still relatively nascent. “Developer experience was something that we didn’t talk about that long ago.” Yet, the idea is growing in mindshare as developers exercise greater control over the software life cycle.

    In fact, the 2022 StackOverflow survey found that 66% of professional developers have at least some influence over their organization’s purchases of new technologies, up from 56% in 2020. Furthermore, the way developers learn is changing. The new generation of programmers is primarily self-taught, relying on coding bootcamps or other online resources. An impressive 70% of developers learned how to code from online resources.

    What we end up with, said Arnwine, is a growing developer population with a much wider range of skills than in the past. And since no one is there to hand-hold them, the technology itself must be usable and intuitive—the onus is on the software provider to craft frictionless self-service workflows. And with developers’ growing influence in purchasing, DX becomes a competitive advantage and an executive talking point, said Arnwine.

    Onboarding

    Developer experience is a broad topic, but it’s most often associated with the time it takes to get started with a tool. This is often referred to as ‘Time to First Hello World.’ But although it’s an important step, the metric might doesn’t always carry much significance. Time to value, on the other hand, said Arnwine, takes this a step further and is a better gauge of success.

    For example, onboarding to a web API would likely involve the signup process, creating an account, getting API keys and perusing the documentation—all before a first call is sent. According to Arnwine, there are techniques to apply at this stage to help developers move faster throughout the onboarding process to reach value sooner.

    Here are some ways in which developer onboarding is becoming more streamlined:

    • Automatically generating keys upon signup.
    • Providing a terminal within the developer dashboard to test sample code.
    • Handing developers the code they need to be customized to their specific stack.
    • Using specifications as a single source of truth for documentation.

    Building

    What can SaaS companies do to help improve DX while building? Well, one way is to enable auto-complete within IDEs. When developers onboard with a new tool, they are often handed an SDK and reference documentation. But, it doesn’t add value to send programmers back and forth between a browser to look up method names. Instead, Arnwine believes developer tools should be “baked into the IDE in a way a modern developer would expect.”

    One example of auto-complete on steroids is Fig, a next-gen command line that supports auto-complete for over 500+ popular CLI tools. Taking this a step further is GitHub Copilot, an AI assistant that can take natural language prompts and generate pretty complex code suggestions. The industry appears to be heading toward more AI-fueled low-code software development—a trend that could significantly improve a developer’s experience during the construction process.

    However, one potential downside of increased software development automation is ensuring the AI suggests correct code snippets. Thus, Arnwine said he anticipated possible troubleshooting and debugging problems. Furthermore, SaaS providers will have to vet systems that generate glue code to ensure it’s accurate and up-to-date with the latest versions. His advice to SaaS providers is to meet developers where they are, but do their best to maintain a quality experience that they can control.

    Deploying

    For a long period of software development history, IT administrators were the only ones able to deploy software. The old-school method required getting a hosting provider, installing Ubuntu and plenty of other legwork. Nowadays, however, software deployment has become far more drag-and-drop. DevOps has become more democratized in recent years, and we will likely see greater usability emerge in the years ahead.

    One way to increase developer experience for deployments is through more custom automated routines via GitHub Actions or similar event-based systems. And plug-and-play pre-built actions could reduce the barrier to using such systems. Such connectivity is necessary to enable a docs-as-code approach, said Arnwine.

    Maintaining

    Another area that could use a serious developer experience upgrade is dependency management. Software dependencies go through a life cycle. They are versioned, updated, retired and sometimes replaced outright. As such, keeping track of an evolving surface area of third-party dependencies can be a headache. To make matters worse, software supply chain threats have complicated the integrity of many open source dependencies consumed by most applications.

    Although it’s imperative for organizations to maintain parity with dependencies, a single person is rarely tasked with continuously checking dependencies and monitoring for security threats. Therefore, more automation for dependency management and security monitoring is necessary. For example, Arnwine uses Snyk, which sends a weekly report on the state of his dependencies.

    Final Thoughts

    Assuming you have the killer functionality to back it up, an investment into developer experience can help a developer tool stick out in a crowded marketplace. But of course, improving the DX means many different things depending on your context. It could be a subtle user interface tweak, adding relevant code samples or making your documentation more intuitive.

    On the other hand, a negative developer experience can turn away potential users and cause frustration throughout the application support life cycle. This can instill a poor reputation among programmers.

    To review, in the near future, we’ll likely witness the developer experience evolve in the following areas:

    • Enhanced methods that streamline the signup process.
    • Auto-complete and AI-driven code suggestions that understand a developer’s stack and meet them where they work.
    • More intuitive, automated deployment models.
    • Increased automation around the maintenance of dependencies.

    While developer experience is a growing concern, it shouldn’t be the only measurement of a SaaS — the ongoing value it brings developer consumers and end users is just as (if not more) important to consider.

  • Three is the Magic Number: Planning and Executing a Successful DevOps Project

    Three is the Magic Number: Planning and Executing a Successful DevOps Project

    Diligent preparation, clearly defined workflows and watchful monitoring are key to delivering a high-performing DevOps project. DevOps is one of the hottest buzzwords in programming, but everyone seems to have their own ideas when it comes to structuring and deploying a DevOps team.

    In basic terms, DevOps is the integration of developers and IT operations workers into a single group, with the goal of increasing the speed and quality of software deployment. Its creation was a response to concerns over the segregation of teams, which led to communication breakdowns and a lack of cohesion.

    Since its inception, DevOps has grown in popularity, with 83% of IT decision-makers stating that they have implemented this style of working in some form.

    However, transitioning business focus in this way does not guarantee an immediate change in fortunes–this dynamic represents an IT culture shift, with increased emphasis placed on collaboration and interpersonal skills. For this model to yield returns, engineers should be open to new information, technology and ways of working.

    So, what vital steps must a DevOps team take to ensure the successful delivery of a project?

    Understanding the Needs of the Client

    The preliminary stages can make or break a DevOps project. If the team does not emerge with a vivid idea of what the client wants and how this can be delivered, the venture is doomed to failure.

    It is crucial that both the team and the client are willing to put the time in to understand each other’s goals, ensuring that their visions align for the steps to follow. This can be done through a series of meetings and workshops, where objectives are identified and both establish a good comprehension of how the final product should look.

    If executed correctly, the DevOps team will exit the project’s first phase with a well-defined brief and a crystal-clear understanding of what the client wants to see delivered. When this step is rushed, engineers will be held back by a lack of direction, increasing the chances that the finished product will not reflect the client’s hopes.

    Configuring the Environment

    This next phase is where the developmental process of the app begins, which is usually facilitated using a cloud-based solution. The team begins by preparing the environment’s aesthetic, working out the components it should contain and exploring how they should be configured to maximize efficiency.

    It is during this period that cybersecurity measures are added so that the final version of the app is as close to impenetrable as possible. This shows why a DevOps team needs a diverse range of skill sets: Cybersecurity experts are few and far between, but there should be at least one member of the group whose knowledge can be called upon to ensure protective measures are properly implemented.

    A successful developmental phase is underpinned by clearly structured workflows, with the project mapped out step by step and everyone involved understanding where they fit into the plan and how they can contribute.

    However, a DevOps project is rarely smooth sailing from start to finish–development is often disrupted by missed deadlines, bugs and conflicting duties pulling engineers away from their stations. For this reason, agile and multi-skilled team members, who can ably jump in and assist in various areas, are extremely important for ensuring continuity when workflows are derailed.

    Avoid Failing at the Final Stages of the Project

    Just because the development of a product is complete and the app is live, this does not mean it is time for the DevOps team to rest on their laurels and move on to the next project. It is essential that they continue to monitor the app in its early stages so that any bugs can be identified and immediately fixed.

    Engineers should be in place to see how much the environment is burdened and how it responds depending on the rate and number of new users. Any data obtained is a precious commodity and should be treated as such, with teams ordering and labeling results as soon as they are collected.

    Once the baseline data has been gathered, results can be analyzed to pinpoint which parts of the app may need recalibrating. After making these improvements, the final result should be a product performing at an optimal level while also meeting the client’s specifications.

    Don’t Rush the Project’s Process

    At the conclusion of all three phases, DevOps teams should have absolute confidence that the project is headed in the right direction and that they are fully equipped to execute the next procedural steps.

    A well-designed app is far easier to produce when a team is made up of specialists from an array of fields eager to collaborate and communicate with one another. Assembling the right personnel will aid the smooth transition between the stages of a DevOps project.

  • A History of Distributed Tracing

    A History of Distributed Tracing

    Organizations are increasingly using distributed tracing to monitor their complex, microservice-based architectures. Distributed tracing has become essential in microservice applications, cloud-native and distributed systems. Microservices and serverless applications can grow exponentially, which makes observing them at scale very challenging. The traditional logging method becomes expensive, and if there’s an issue, time-series data can reveal symptoms but not the root cause(s). With modern usage patterns of Kubernetes service mesh like Istio and Envoy, for example, there can be billions of microservices calls daily.

    What is Distributed Tracing?

    Traces describe the progress of a transaction or workflow through a system. Distributed tracing provides visibility into service dependencies and their interdependencies by providing end-to-end visibility. Visualizing transactions in their entirety allows you to compare non-normal traces with normal ones to determine differences in behavior, structure and timing. Through distributed tracing, you can identify bottlenecks in your system. As a traced request flows through a network, it can be tracked. 

    As requests flow from frontend to backend and to databases, traces provide diagnostic techniques that reveal how a set of services coordinate to handle an application request. Using distributed tracing, developers can troubleshoot requests with errors and high latency. A single trace typically shows the activity of a request or transaction within an application. A trace shows what has changed at each step. As you aggregate the traces, you can see which backend service or database has the most significant impact on performance. 

    The History of Distributed Tracing

    Dapper, a large-scale distributed systems tracing infrastructure, was introduced by Google in 2010. Two years after Dapper was made public, Twitter open sourced Zipkin, which was designed for application performance tuning. Zipkin was the first open source distributed tracing project. Zipkin trace data can be collected and visualized using a UI. In 2015, Uber launched Jaeger, which was inspired by Dapper and named after the German word for hunter. Later in 2017, Uber published a blog post, called Evolving Distributed Tracing at Uber Engineering, explaining the reasons for the architectural choices in Jaeger. In addition to creating Jaeger, its author, Yuri Shkuro, wrote a book about distributed tracing called Mastering Distributed Tracing. 

    In 2016 Ben Sigelman, founder of Lightstep, wrote a blog post called Toward Turnkey Distributed Tracing, describing OpenTracing as a single standard. Some people refer to this as the OpenTracing Manifesto. OpenTracing allows developers of application code, open source packages and open source services to instrument their code without binding themselves to any particular tracing solution. The goal of OpenTracing was to solve a standardization problem. Trace context must pass through all the components, including application code, dependent libraries, standalone open source services (Nginx, MySQL) and other vendor-specific libraries and services, to collect a complete distributed trace. Collecting a full path without a standard API to define the collection and passing of trace context is difficult. OpenTracing aims to solve this problem by defining a standard API that can be implemented by components from different tracing solutions, allowing the collection of end-to-end tracing data. The Cloud Native Computing Foundation (CNCF) accepted OpenTracing as its third project in October 2016. Two months later, OpenTracing 1.0 was released. After OpenTracing, Jaeger joined the CNCF. 

    W3C tracing context specification was proposed in November 2019, bringing distributed tracing closer to standardization. Over ten years, distributed tracing evolved from one paper to an active community. With standardization on all the layers, it is moving from just tracing to overall observability, ranging from latency optimization to root cause analysis and application performance management. It is moving from a single backend system to an end-to-end solution that spans multiple systems. 

    OpenTracing and OpenTelemetry merged in 2019. Using OpenTelemetry, distributed tracing can be implemented end-to-end. It released version 1.0 in 2021. CNCF’s OpenTelemetry project is one of the fast-growing projects, and OpenTelemetry has become the de facto standard for tracing, metrics and logging.

    How it Works

    Traces in distributed tracing consist of tagged time intervals called spans. It is possible to think of a span as a unit of work. Traces are directed acyclic graphs (DAGs) composed of segments known as spans. According to OpenTelemetry, There are four major components of a trace:

    • traceID
        • A unique 16-byte array to identify the trace that a span is associated with
    • spanID
        • Hex-encoded 8-byte array to identify the current span
    • Trace Flags
        • Provides more details about the trace, such as if it is sampled
    • Trace State
        • Provides more vendor-specific information for tracing across multiple distributed systems. Please refer to W3C Trace Context for further explanation.

    The spans in a trace consist of contiguous segments of work that are named and timed. There are start and end times for spans, as well as metadata such as tags or logs that help classify them. Parent-child relationships may be present between spans to show how particular transactions traverse the application’s numerous services and components. 

     

     

    A trace represents an end-to-end request; it can contain one or more spans. Spans indicate work performed by a single service with associated time intervals and metadata. The purpose of traces is to provide a request-centric view of spans through tags. Despite microservices enabling teams to work independently, distributed tracing offers a central resource that allows all teams to understand issues from a user’s perspective.

    Conclusion

    In distributed systems, response latency can have a significant commercial impact. Identifying bottlenecks and understanding how a request moves through a complex system is complicated. Other techniques, like logging and monitoring metrics, don’t provide insight into distributed applications such as those created with microservices architecture. Several open standards and tools are emerging within the distributed tracing space, like OpenTracing and commercial tools that could compete with existing APM solutions. Implementing distributed tracing for modern cloud-native services poses several challenges. OpenTelemetry tracing is necessary for processing high volumes of trace data while generating meaningful insight.

  • Why SaaS Vendors Need to Give Back to Open Source

    Why SaaS Vendors Need to Give Back to Open Source

    There aren’t many tech organizations that don’t leverage open source technology. Companies that refuse to import software will rely on a freely available library tool or codebase. Open source is one of the largest and most powerful mechanisms of software propagation. However, open source is in danger.

    The Two Sides of Open Source Development

    In recent years, the ubiquity and dominance of open source tooling have led companies to adopt them en masse. Products like Elasticsearch, Spring and Akka are incredibly popular as open source, freely available tools. But unfortunately, often only one-half of the open source philosophy is being honored.

    The Open Source Philosophy

    What does that mean? Well, open source depends on two things. Customers use the software and those same consumers contribute to the codebase. It’s a feedback loop of like-minded individuals who have a shared interest in building and using an outstanding product and are happy to donate their time to realize their collective goals. This agreement has driven some of the most exciting software achievements of the past two decades.

    Alas, while it became prevalent to use open source software, the popularity of contributions did not scale equally. Suddenly, small teams of dedicated developers were inundated with demands from huge enterprises that equated “open source” with “free.” This caused the owners of open source projects, like Elasticsearch, MongoDB and Akka, to reevaluate their position and change to the increasingly prevalent source available license.

    Growing Risk to the Open Source Community

    If users contribute to the codebases they leverage, then those codebases can thrive and no one needs to worry about subscription fees or fines. Moreover, users can maintain the certainty that they are not doing anything illegal when they depend on a piece of software. Without contributions from the engineering community, an open source project’s commercial and practical basis quickly disintegrates.

    So Why Should SaaS Vendors Give Back?

    SaaS solutions inhabit a particular corner of the market, assisting companies in solving a cross-cutting problem. For example, observability. This is an important detail because many open source tools also operate in this arena. This is exemplified by the fact that it’s common for SaaS vendors to regularly compete against an open source, homegrown alternative. Many SaaS solutions actually base their core infrastructure on the same open source software.

    This means that SaaS providers are often in direct competition with the open source solutions they leverage to build their product(s).

    SaaS Vendors Depend on Open Source Software

    The communities that support vital pieces of open source software, like Apache Kafka or OpenDistro, are helping to unlock innovation in enterprises worldwide. The continued pace of this innovation is inextricably linked to the continued growth and development of the open source tooling on which it depends.

    The only sensible course of action is for SaaS vendors to make every effort possible to contribute back to the open source space. They do not need to give up their unique features, nor do they need to release their proprietary software, but the simple act of maintaining patches, improving performance, increasing compatibility and even correcting poor documentation—all of this is a contribution that will benefit the entire SaaS ecosystem.

    Those Contributions Will Breed Specialization

    A big challenge of using open source software is finding engineers that understand how it works. The plug-and-play nature of many tools means that users never need to dive into the details of a library or application. This, however, comes at a cost. Users often don’t know why the software breaks or behaves unpredictably.

    If a SaaS vendor begins to contribute back into the core of the open source tools on which they depend, not only are they benefiting their product and the open source community, but they’re also seeding specialist knowledge in their teams. This specialist knowledge means they can utilize open source features in new and exciting ways, giving them insights and advantages over their competitors.

    Open Source Contributions Make Commercial Sense

    The SaaS community thrives when open source tooling works smoothly. Conversely, if fundamental open source tools like Kafka or OpenDistro begin to falter, then the success of the SaaS community is also at risk. They are highly interdependent. This means a contribution back to core open source tools is a contribution to the stability of SaaS products. Moreover, growing specialist skills within engineering teams means that when it comes time to leverage a new open-source feature, your developers may have skills and insights that other companies won’t have. This is a clear commercial advantage, and all for the cost of doing the right thing, and keeping the dream of free, open source software alive.

  • Securing Open Source Components in a World of Mixed Committer Motivations

    Securing Open Source Components in a World of Mixed Committer Motivations

    Our world runs on software that contains open source components. This places an increased burden on developers, as the primary consumers and deployers of those components, to use code that is fully up-to-date and secure. The vast majority of open source committers with write access to project repositories remain focused on maintaining and making positive improvements to that code, while also helping to benefit the open source community as a whole. However, the sheer complexity of many open source projects has opened the door to different, more insidious and concerted types of attacks which could cause widespread and serious harm.

    Developers Have to Move Fast and Reuse Gives Them That Speed

    One complicating factor is the prevalence of reuse of third-party open source elements across the majority of open source projects through two main types of dependencies. With direct dependencies, an open source component links straight to a third-party code library, while transitive dependencies occur when a component calls out to a library that is, in turn, communicating with another dependency.

    Dependencies enable developers, under ever-increasing corporate pressure to deliver new software rapidly, to move faster by building on top of existing work by their peers. The downside is that those dependency foundations are vulnerable to attack, with some libraries perhaps under the aegis of a single committer for whom the code collection may be a side project and perhaps not attended to on a regular basis.

    Sonatype research and that of other organizations tells us that developers working with a Java application may be dealing with a staggering 1,500 dependency changes every year, with the average Java application containing 148 dependencies and the average Java project being updated 10 times annually. While the number of open source projects isn’t ballooning, the amount of versions of those projects is skyrocketing. The majority of vulnerabilities today impacting open source projects result from transitive dependencies, the more complex type of dependency.

    In an area where multiple projects may exist effectively solving the same issues, developers need to be aware of which projects are being actively maintained versus those that are either poorly nurtured or no longer in operation. On the other hand, successful open source projects may have tens or hundreds of committers with write repository access, which address a whole different set of issues, including reaching agreement on what a vulnerability consists of and how to fix or patch it.

    Amid a Community for Good, Bad Actors and Incidents of Self-Sabotage Emerge

    The reasons why individuals choose to commit to open source software are wide-ranging; a strong desire to give back to the community to a passionate and, perhaps very single-minded, hobby to an integral and paid part of a developer’s role and responsibilities at work. A few years ago, the concept of a malicious open source committer was an abstract theory. The thought back then was that perhaps a handful of ‘bad apples’ might exist within a community of millions of developers. However, since 2019, researchers’ tools and those of other organizations have identified over 100,000 malicious, suspicious or proof-of-concept packages, pointing to a much higher degree of behavior by cybercriminals.

    Bad actors are deliberately taking advantage of the complexity of open source software to knowingly introduce exploitable vulnerabilities. These attacks are growing in sophistication particularly in relation to malicious code injections into the source code of an open source library. These attacks can often appear as harmless code changes and remain hidden before delivering a malware payload. There is the recent example of a crypto heist that stole over $3 million in cryptocurrency which leveraged compromised access to a private GitHub repository.

    Protestware, where a code maintainer chooses to self-sabotage their open source project often to draw attention to a cause, whether geopolitical or specific to that individual, has become an issue over the past year. In this case, although the stated goal is not malicious hacking, real harm, in terms of major disruption, to developers using that code can occur.

    To counter both the impact of bad actors and of self-sabotage on open source components, developers need to take advantage of release integrity features to help protect themselves and their organizations from harm. They must be able to have full visibility and control over the software they build and the open source components that they are sourcing as part of that development work. At the same time, governments around the world, including in the U.S., are weighing into issues related to the security of software supply chains, so increased regulations and penalties should also factor into how developers and their organizations behave moving forward.

    We’re only one step away from the next potential Log4j attack, and every developer and organization needs to up their game and be ready to move fast on vulnerability identification and remediation. The gathering and sharing of data is what will help to increase everyone’s safety.

  • AWS Previews Amazon CodeCatalyst to Automatically Create Dev Environments

    AWS Previews Amazon CodeCatalyst to Automatically Create Dev Environments

    Amazon Web Services (AWS) this week launched a preview release of the Amazon CodeCatalyst framework that leverages blueprints to automatically provision and configure developer environments in its cloud.

    Announced at the AWS re:Invent 2022 conference, Amazon CodeCatalyst takes advantage of a Project Blueprints tool to launch a source code repository and a project management tracker that are then used to create build and release pipelines. DevOps teams can use either the tools provided by AWS or substitute an instance of, for example, GitHub or the Jira project management software.

    Amazon CTO Werner Vogels told conference attendees the goal is to reduce the friction that developers currently encounter whenever they need to launch a new development project. As part of that effort, Amazon CodeCatalyst promises to make it simpler to consistently replicate development environments using scaffolds in addition to providing dashboards and search capabilities for tracking changes across commits, pull requests and deployments.

    Amazon CodeCatalyst uses a devfile to define the configuration of an environment that comes in four resizable instance options made up of two, four, eight or 16 virtual CPUs. The devfile defines and configures all the resources needed to code, test and debug a project. Devfiles, which are added to the source code repository by the selected blueprint, can also be modified.

    Collectively, those capabilities make it possible to deploy sample TypeScript, Python, Java, .NET, JavaScript, React, Angular and Vue code running in a few minutes. Developers can work with the AWS Cloud9 integrated development environment (IDE) or use tools such as JetBrains IntelliJ IDEA Ultimate, PyCharm Pro, GoLand and Visual Studio Code on their local machine. Developers, as a result, can simply switch between multiple codebases or collaborate with other developers as needed.

    The pipelines can be run on pre-provisioned builds in the cloud or organizations can opt to deploy their own container environments. Developers can incorporate build actions that are built in or provided by AWS partners in addition to incorporating GitHub Actions to compose fully automated pipelines that can be configured using either a visual editor or YAML files. Build and release pipelines enable deployment on Amazon Elastic Container Service (Amazon ECS), AWS Lambda and Amazon Elastic Compute Cloud (Amazon EC2) services.

    In general, Vogels advocated that IT teams should build event-driven applications that are loosely coupled to make it simpler to add new capabilities to existing applications running at scale without impacting the entire codebase. To make it simpler to build those types of applications, AWS added an Amazon EventBridge Pipes tool that creates point-to-point integrations between event producers and consumers in a way that eliminates the need to write that code. AWS also made available in preview a graphical AWS Application Composer tool that makes it simpler to design, configure and build serverless applications using infrastructure-as-code (IaC) definitions.

    It’s not clear to what degree developers will be embracing event-driven applications. As a programming model, event-driven architectures have been around for decades. The issue has been those applications also have tended to be among the most challenging to build and maintain. AWS is now clearly signaling its intent to make building those applications a lot easier.

  • AWS re:Invent — Top 4 Things We Learned This Week

    AWS re:Invent — Top 4 Things We Learned This Week

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week, four things that caught my eye from Amazon Web Services’ re:Invent 2022 conference: SnapStart turbocharges Lambda, Graviton3E ARM HPC SoC, AWS continues hiring, and AWS’s origin story. (more…)