devopsy.kinsta.cloud

Category: DevOps Culture

  • Energy Efficiency and the Data Center

    Energy Efficiency and the Data Center

    The IT industry has historically been a leader in the ongoing race to reduce CO2 emissions. Reducing emissions is good for the environment and for the corporate bottom line. The data center is well-known as a voracious data consumer. But the data center consumption findings keep pace with the trend. Even in the face of remarkable growth in data center capacity and performance, their energy consumption has increased only a few percentage points since the 2010s. This almost-flat rate of energy consumption is primarily the result of the shift to cloud computing as well as the shift to solar and wind energy, where it makes sense.

    Investments in renewable energy sources have helped tamp down criticism over the overall IT carbon footprint. Knowing that it takes a tremendous amount of energy to power a data center, owners and operators are always looking for ways to improve efficiency and reduce carbon expenditure. Today, there are newer and better ways for organizations to deliver the performance to properly power modern applications while keeping in mind the energy efficiency and carbon reduction the world requires.

    Minimizing Energy Use and CO2 Production

    How did we get here? Decades of semiconductor process miniaturization in concurrence with living out Moore’s Law has culminated in ongoing improvements in performance, storage capacity and device density. Today’s mobile phones are a great example of that ongoing progress. A secondary factor in the mobile phone revolution is a processor architecture optimized for efficiency and flexibility. This ARM architecture has been used to power almost all devices since the first iPhone. Over the years, ARM has also become a realistic platform for servers; often a better choice than an x86 system.

    Energy efficiency is one area where ARM systems win out over conventional server architecture. But it was not until recently that ARM processors could deliver the required performance for the data center. Today’s systems, based on Armv8 designs, have brought into play x86-class capabilities in a more energy-efficient way. The advent of the ARM-based AWS Graviton processors, which by some estimates have already captured 20% of the instance share on AWS, demonstrates that ARM can run production workloads while delivering what AWS calls “up to 40% better price performance over comparable current-generation x86-based instances for a wide variety of workloads.”

    To illustrate the impact on carbon production when a data center shifts from x86-based architecture to ARM servers, an analysis of the top elements contributing to data center energy use was undertaken. The energy savings was calculated and translated to reduced CO2 emissions based on the fuel mix used to power the electric grid in various locations. The results prove that an ARM-powered data center reduces CO2 production by 74%, equivalent to almost half a million barrels of oil.

    Power Consumption Reduced by Efficient Servers

    The premise is simple—innovative server design significantly reduces data center energy use. In this model, a server’s power draw and physical size are key factors. The energy needed for other data center subsystems was extrapolated by using ratios derived from experience and validated by estimates from other data center operators.

    Data center energy use can be divided into five categories of equipment and infrastructure: Servers, network communications, power supplies, conversion and UPS, cooling systems and storage. Servers and cooling comprise about 75% of the total.
    The calculations were based on the energy used by a medium-sized data center with 750 racks of conventional 1U servers (42 servers/rack; around 31,500 in total). Assumptions used included servers comprising two-thirds of the floor space and that the equipment space accounts for 65% of the total, translating to a 62,000 square-foot data center. For comparison, facilities operated by the major cloud vendors can be a million square feet or more. Knowing server power consumption and the ratios for the equipment categories, algebra can be used to calculate the energy consumption for each.

    The data center in this hypothetical example has, as we said, 31,500 servers. Using the 912.5W power draw of a Dell R640 as typical of 1U x86 servers, the total server usage in the ARM data center would be 251,795 MWh/year. Using this formula, the amount of energy saved by using ARM servers can be calculated. Each server requires about one-quarter the electricity of a standard server. In the formula, the assumption is that the ARM server installation would require half the equipment racks (375 instead of 750) and 52% of the power-per-system, comprising a data center using only 26% as much electricity (152,409 MWh/year) as one based on x86 systems.

    The math shows that switching one medium-sized data center to ARM servers can produce annual savings equivalent to 45,459 fewer cars on the road or 486,749 fewer barrels of oil used.

    The value in CO2 reduction is widely understood for preserving the environment. It is also advantageous from an economic standpoint. In fact, there are many technical and economic reasons to adopt energy-efficient ARM-powered servers in the next-generation cloud data center.

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021

  • Why You Need a Value Stream Delivery Platform

    Why You Need a Value Stream Delivery Platform

    While the pandemic was significantly challenging for businesses, it exposed the need to quickly adapt and change. Software products for internal or client use continue to provide an opportunity for a competitive edge in nearly any industry. Businesses will immerse themselves and compete in the digital economy by consistently producing better software products at an ever-increasing pace.

    In the past year, two new solutions with similar names have been gaining attention: value stream management platforms and value stream delivery platforms. Both are designed to help firms improve their software development process. The end goal for both is also similar—speed up development—but how each accomplishes this goal is important to differentiate the two.

    So which one should you use? We’ll cut to the chase—the value stream delivery platform provides more strategic long-term value. Here’s why.

    First: Value Stream Mapping

    To understand the need, you first need to understand value stream mapping. This involves visualizing the value stream processes and events from conception and creation to testing, production and, finally, release. By mapping the value stream, you can get a better understanding of the workflow and identify dependencies between the different activities, which will help you identify any blockages or impediments to delay delivering customer value.

    Staying abreast of your value stream is essential to delivering value to the customer as it connects all the multiple teams, tools and applications along the way. By gaining clearer visibility into the entire software delivery process, you can then scale your enterprise by bringing other teams on board for greater collaboration. Additionally, tracking your value streams can help reduce any risks to performance, as you can easily see the throughput and stability of your products.

    Value Stream Management Platforms (VSMP) Explained

    Simply put, value stream management helps solve poor visibility across the software delivery life cycle. You can measure and manage the performance of your software delivery team, identify any areas for improvement and leverage best practices that support collaboration through teams. Additionally, a value stream management platform helps connect applications, teams and tools through the collection of static data used to improve ongoing delivery performance.

    One benefit to using a value stream management platform is that existing toolchains remain undisturbed. The VSMP simply integrates with the different datasets across the existing process and provides the desired visibility and analytics to the different product value stream(s) across the organization.

    An Underlying Problem: Toolchains

    The lure of easy visibility that a value stream management platform provides is somewhat deceptive. The constant pressure to create and deliver better software faster using multiple tools and manually built toolchains has reached its breaking point:

    • Poor interoperability, manual handoffs and disparate data have created DevOps automation silos
    • The multiple test environments used through the process and the need for multi-cloud and hybrid deployments add tons of unnecessary complexity
    • Increased policy and security requirements simply cannot scale and remain manual

    Developers are caught in the middle—they must understand both a complex development and deployment process plus build a complex product for their customers—but being an expert in both is nearly impossible.

    The use of internal platform teams has been recognized as a way to solve this problem. The platform team is responsible for providing an easy-to-use solution; essentially treating developers as customers. In turn, the developers can focus their efforts on the software products they’re responsible for and on satisfying client needs.

    How is a Value Stream Delivery Platform Different?

    Many firms have learned the hard way that building an internal platform is risky, time-consuming and expensive. A value stream delivery platform replaces the traditional, manually built toolchains used today and sidesteps the risk, cost and development time involved in building a platform in-house.

    A VSDP should:

    • Cover the entire process of planning and creation, integration and verification, deployment and operations and, finally, monitoring and improvement
    •  Have both security and compliance capabilities built into the platform foundation
    •  Integrate DevOps point solutions, plus easily add or change tools over time
    •  Provide a centralized data model that creates one source of truth for the development process
    • Provide metrics and reports to accelerate process and value stream improvements
    • Improve software quality through continued enhancements based on best practices

    A value stream delivery platform cuts the administrative burden on DevOps teams, allowing them to refocus their efforts on providing significantly higher value. Instead of maintaining toolchains, they can now help identify process bottlenecks, expand automation and work directly with application teams to drive standards.

    Value stream management platforms provide value in understanding the process and where it can be improved. But they don’t fix the underlying toolchain problem. Instead, the improved visibility may tempt some to squeeze more productivity from developers over the short term without addressing the underlying toolchain problem. However, replacing toolchains with a value stream delivery platform can help address all these issues, and may prompt you to retire your value stream management platform.

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021

  • Security is About People, not Technology

    Security is About People, not Technology

    Just a random thought of the day: We have the world’s best example of how many of our security woes are actually people problems, not technology problems.

    We all know that. We’ve all discussed it—but let’s consider this example and have a quick discussion about 2FA.

    Epic Games had a problem not unlike the rest of us. But it was costing them money every day, in part because of the nature of their user base. The problem? Attackers taking over the accounts of legitimate Fortnite users. In the world of gaming, scammers are rife because the player base is younger. At least since Ultima Online (1997), online games have been growing and Fortnite is at the forefront of that movement, with user counts unheard of before it became popular. That many users combined with hackers and scammers taking advantage of the younger demographic attracted to games—over 60% of Fortnite’s massive user base is under 24 years old—they do not provide a further breakdown, but there are certainly kids as young as six playing; and possibly even younger players).

    That made for a problem. Investment of time and money and energy for supporting customers having account issues was no doubt huge. I do not have an insider feeding me information, but simple math—and the evidence of steps they’ve taken—suggest that this is the case.

    So they did with kids what far too many of us don’t have the guts to do with our adult workforce. They implemented 2FA. While they did not require it—and, like most game companies, are tight-lipped about any kind of player statistics—they implemented it with a carrot-or-stick approach that, no doubt, has been successful. My gaming group of half a dozen all use it—because we either wanted the carrot or were avoiding the stick. It’s not really a stick. Call it carrot and wall. The carrot comes in two forms for the two most popular versions of the game: A free dance for your character (toon, if you prefer) to perform in Fortnite: Battle Royale.

    Fortnite Dance

    And free gear in the paid version –  Fortnite: Save the World.

    Free gear

    For those of you who are not familiar, both of these are decent rewards; calculated by the Fortnite team to be compelling enough to warrant the steps, but not insane. The Save the World rewards are actually better in terms of usefulness, but the dance for Battle Royale is likely more popular.

    Then, they put up a wall. If you want to play competitive Fortnite, you must have 2FA turned on. It is possible that this was done to protect the sanctity of their tournaments, but it has the effect of providing the other half of the “reward and punish” equation. According to Compare Camp, 40 million players attempted to qualify for the game’s World Cup alone. So that is the minimum number of 2FA activated accounts—you cannot participate in competitive Fortnite without 2FA turned on.

    They then made turning 2FA on as easy as possible. You need to learn from this. It was harder for me to turn on 2FA and get it working for Discord than it was to turn it on and get it working for Fortnite. Strive to be as easy as Fortnite.

    At this point, we should all recognize that some form of 2FA, even the much-maligned ones like email (maligned because someone who compromises your email then has easier access to everything because 2FA is compromised), is almost essential. The cost of someone hijacking a valid user account—like the lead systems engineer or the CEO—is just too high to ignore. But some 2FA solutions are harder than others. So, we need to follow Epic Games’ lead, and make it as simple as possible, provide incentives to employees to turn it on and, if suitable, wield a stick. A minor ding on reviews until implementation is complete should be enough, if the incentives are nice. A free day off? A small bonus? You know your organization, both what management will agree to pay for and what users will find compelling.

    Get it done. The last thing you want is for your CEO to suddenly start saying/doing erratic things on corporate accounts. Unless they already do—then I suggest you consider looking for a new job. I know; there are 5,000 things to do and only a few of you. But this one is big. Don’t wait until you’ve lost accounts to turn on a pretty effective prevention mechanism.

  • 5 Steps to Succeed With Blue-Green Deployment

    5 Steps to Succeed With Blue-Green Deployment

    By allowing teams to maintain two production-ready environments at the same time, the blue-green deployment technique can significantly boost reliability. But blue-green deployment can also be difficult to execute and manage. Let’s unpack how blue-green deployment works, why it’s important and which best practices to follow for blue-green deployment success.

    Blue-Green Deployment, Defined

    Blue-green deployment is a software deployment technique in which teams maintain two identical but distinct environments that are capable of hosting a production application. One environment is “live” and actively serving users, while the other is kept in reserve.

    When it comes time to deploy a new version of your application, you deploy it to the reserve environment. You then test and validate it to make sure it meets performance and reliability requirements.

    If it does, you reconfigure your DNS or load balancers so that user traffic is directed to this environment, which becomes the new production environment. Meanwhile, the previous production environment is placed in reserve.

    Benefits of Blue-Green Deployment

    The blue-green deployment technique can boost reliability in several ways:

    • Environment parity: Because the reserve environment in which you test your application becomes the live production environment, blue-green deployment ensures parity between testing and production. In turn, it reduces the risk that configuration differences could cause unforeseen problems in production.
    • Failed deployments: If a deployment fails, it doesn’t impact your users because you deploy into the reserve environment. You can fix the issue while a stable user-facing version of your app continues to run in the other environment. 
    • Redundant environments: Although blue-green deployment is not a data protection or disaster recovery technique, having two production-ready environments available is useful in the event that one of your environments is wiped out by an event like a data center failure. In that case, you can restore operations quickly by using the other environment (provided it’s hosted in a separate data center, of course).

    How to Use Blue-Green Deployment: Steps and Best Practices

    While blue-green deployment is a great way to increase reliability, it does present some challenges. The following best practices can help you to handle them.

    Use Scalable Infrastructure

    One obvious potential downside of blue-green deployment is that it can essentially double the amount of resources you need to host your application since you have to maintain two identical environments. That can double your hosting bill.

    One way to reduce the cost impact of blue-green deployment is to create environments that can scale rapidly without changing their core configuration. You could do this using cloud-based virtual machines (VMs) and autoscaling or via containers and an orchestration engine (like Kubernetes) that can scale them.

    Using this approach, you can set up two environments that are effectively identical except that the production environment includes more VM or container instances than the reserve. Although you could argue that this is not quite a blue-green deployment because the environments are not perfectly identical, that semantic downside is outweighed by the benefit of not wasting a bunch of money on infrastructure you don’t need.

    Chaos Engineering

    Typically, chaos engineering entails experimenting with a production system to test its reliability. Chaos engineering offers many benefits, but it presents the obvious risk of creating a problem in production that impacts your end users.

    Blue-green deployment allows you to work around this risk by practicing chaos engineering in the reserve environment. Of course, because you won’t have real users, performing chaos engineering under these settings may not allow you to test every aspect of your production application. But you can still test many of the configuration variables by issuing requests yourself within the reserve environment—all without worrying about breaking things for actual users.

    Managing Database State

    Keeping databases in sync when you switch between environments is one of the biggest challenges of blue-green deployment. Even small discrepancies in database state could cause big reliability issues.

    The simplest way to address this issue is to turn off the database before switching and perform tests to ensure that databases in both environments are identical before they are reenabled. Of course, this approach may impact application functionality, because the app may not work properly if the database is turned off.

    Another solution is to sync databases between your two environments. This should ensure that database state remains identical even as you switch from one production environment to the other. But it also means that problems with the database will affect both environments.

    The bottom line: There is no simple way of managing database state in blue-green deployment. You should evaluate the different options available and consider which one best meets your needs. If you can tolerate a little downtime, stop the database and restart it during the switch. If you need total continuity, use syncing, but be sure to manage the associated risks.

    Change Load Balancers, not DNS

    There are two main ways to manage where traffic is directed, and hence which is the production environment: You can change your DNS configuration when you want to switch. That’s easy to do—you would simply change the mapping for yourapp.com from one IP address to another, depending on which environment is the live one.

    You can also use load balancers to manage traffic. Load balancer configuration is more complex than simply changing DNS entries, but it also enables more sophisticated and granular control.

    In general, it’s a best practice to manage blue-green deployment switches via load balancers, due to the extra control they provide.

    Prepare for the Unexpected

    While blue-green deployment can reduce the risk of user-impacting disruptions, it’s no silver bullet. You must also be sure that you have the reliability management tools you’ll need to detect and remediate issues within both of your environments.

    Blue-green deployment is a powerful reliability engineering technique. But to get the most out of it, you must optimize the way you handle requirements like infrastructure management, traffic routing and database state. 

  • Does Shift Left Matter to SREs?

    Does Shift Left Matter to SREs?

    If you’re a software engineer, you’ve likely heard all about shift left, a practice that can streamline certain aspects of software development.

    But shift left isn’t just for developers. It can be equally valuable for site reliability engineers (SREs). Although the main mission of SREs is ensuring the reliability of software after it has been deployed rather than actually developing software, encouraging shift left practices as part of a software development strategy can nonetheless make it easier to optimize reliability.

    To prove the point, here’s an overview of what shift left means and why it matters for SREs.

    What Does Shift Left Mean?

    Shift left is the detection of application problems early in the software development process. The core idea behind shift left is that instead of waiting until just before deployment to test software for bugs, security issues or other problems, teams should begin the testing process as early as possible. In other words, they should shift some testing to the “left” of the software development life cycle instead of testing only in the middle—hence the term “shift left.”

    In so doing, they can detect problems early in the development process, at which point they are typically easier to resolve. If you don’t catch a performance, reliability or security bug until just before you plan to push your application release into production, you may have to overhaul a large part of your codebase to fix the issue. In many cases, you’ll have to change not just the specific code that triggered the bug, but also other code that depends on or integrates with the buggy code. But by finding buggy code early, developers can often fix the problem with minimal adjustments to the application as a whole.

    Shift left is a high-level concept, and it can be implemented in multiple ways. In some cases, developers may start running performance tests on new code right after they write it, even if it has not yet been integrated into the main codebase. In other cases, shift left testing could mean compiling some parts of the codebase in order to run tests against it before the application as a whole is built and tested in a dev/testing environment. Either way, the result is a testing process that begins earlier than it would under a conventional approach.

    Importantly, shift left doesn’t mean that teams skip the tests that they would normally perform just prior to deployment. Those tests, which are important for catching bugs that materialize only when testing the application as a whole rather than individual parts of it, typically still happen as part of a shift left strategy. Thus, shift left means that testing begins earlier and coincides with a broader portion of the overall development pipeline, not that testing is simply shifted to the beginning of the pipeline at the expense of mid-pipeline tests.

    Shift Left and SREs

    Again, the main job of SREs is to manage the reliability of systems. They may spend part of their days working alongside software engineers to help devise architectures and coding strategies that maximize the resiliency of applications against reliability and performance issues. But SREs also devote much of their time to managing what happens post-deployment. They use monitoring and observability tools to detect problems that arise in production environments. They then take the lead in responding to them.

    In that sense, SREs may not seem to have much to gain from shift left. Shift left is a strategy that aligns with software development first and foremost, and software development is not the core focus of SREs.

    Nonetheless, a shift-left strategy can help SREs do their jobs better, for several reasons:

    • Fewer problems in production: Most obviously, shift left reduces the rate at which reliability issues reach production environments. In turn, it minimizes the number of incidents SREs need to respond to.
    • Optimizing software pre-deployment: By exposing reliability issues early in the development process, shift left is one way for SREs to acquire the insights they need to collaborate with developers on building reliability into applications. Shift left could highlight buggy dependencies or suboptimal architectural patterns, for example, which SREs can encourage developers to address.
    • Granular reliability insights: Because shift left is a great way to trace software problems to the specific code that triggers them, it provides highly granular visibility into reliability. It can help SREs find the weakest links within an application’s codebase or architecture. These weak spots can be harder to detect when monitoring or observing an application as a whole.

    How SREs can Follow the Shift Left Methodology

    Because the implementation of shift left practices is a task that ultimately falls to developers, SREs need to work closely with development teams (and DevOps engineers, if they exist in the organization) to make shift left happen.

    The first step in that process is to get buy-in for shift left among developers. SREs can do this by highlighting the ways in which shift left can streamline the work of developers and reduce the complexity of fixing bugs. That’s important to communicate because it ensures that developers understand that shift left benefits everyone, not just SREs.

    SREs should also work with developers to identify the best approaches and practices to implementing shift left. A key factor to consider here is where reliability problems most often arise, and how shift left can help to detect them earlier. If most performance problems stem from code written by individual developers, for instance, testing code as soon as it is generated may be the best way to detect problems early on. In contrast, in situations where performance issues are triggered most often by environment configurations, the ability to test applications early under different environment variables could be the best way to surface issues early in the development process.

    Finally, SREs should partner with developers to respond to the issues that shift left reveals. After all, while developers typically don’t pay close attention to what happens post-deployment, SREs have a special ability to understand how problems in development translate to problems in production. SREs may therefore be able to help developers identify the most effective way to fix bugs based on production environment requirements, even if it’s not the quickest or simplest way.

    Why SREs Need to Shift Left, Too

    Although the shift left methodology has not traditionally been closely associated with SRE, shift left is a crucial tool in the SRE toolbox. By helping teams to detect reliability issues when they are easier to resolve, shift left puts SREs in a stronger position to maximize the overall reliability of applications. At the same time, it reduces the number of production environment issues that SREs have to manage.

  • Snyk Extends Tools Portfolio to Drive DevSecOps Adoption

    Snyk Extends Tools Portfolio to Drive DevSecOps Adoption

    During its online SnykCon 2021 conference this week, Snyk extended Snyk Code, a static application security testing (SAST) tool that already supports the Java, JavaScript and Python programming languages to include support for C#, Ruby, PHP and Go.

    At the same time, Snyk Open Source, a platform for remediating open source vulnerabilities, has been extended to provide native integration with Atlassian BitBucket and AWS CodePipeline platforms for driving DevOps workflows. Snyk has also tightened integrations with platforms from DigitalOcean and HashiCorp.

    Snyk also added support for the Elixir programming language and package managers Yarn 2 and Poetry alongside integration with a C++ scanning tool from FossilID, a provider of a software composition analysis (SCA) tool for open source code that Snyk acquired earlier this year.

    The Snyk Container platform is now integrated with the open source Trivy container scanning tools and with Snyk’s vulnerability database in addition to adding support for container registries such as Quay, GitHub Container Registry, GitLab, Google Artifact Registry and Harbor.

    The Snyk infrastructure-as-code platform now also makes it possible to detect configuration issues in Kubernetes manifests in Terraform code in a way that is compatible with cloud platforms from Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP).

    Snyk also is adding support for version 3 of OpenAPI, which the company reports is three times faster than the previous version. That API is also the foundation for Snyk Apps that the company will continue to roll out.

    Finally, Snyk launched both a new free developer security education program, dubbed Snyk Learn, through which developers can attain and measure their level of DevSecOps expertise and Snyk Impact, an effort to foster collaboration among developers involving a wide range of socio-economic issues.

    Company president Guy Podjamy said as DevsecOps continues to evolve, the primary focus remains on developers and DevOps teams. However, as an issue, DevSecOps is also becoming more important at both the C-level within organizations as well as individual security operations teams. As such, and in the wake of recent high-profile security breaches, the sense of urgency surrounding DevSecOps is now a lot higher. The fundamental challenge organizations are trying to address is finding a way to improve overall application security without slowing down the pace at which applications are built, noted Podjamy.

    Snyk itself revealed it has secured an additional $530 million round of funding to address those issues. The company has now raised a total of $775 million to date with a valuation of $8.5 billion. Snyk also revealed it has increased annual recurring revenue (ARR) by 154% year-over-year from a customer base that now spans more than 1,200 organizations, including Asurion, Google, Intuit, MongoDB, New Relic, Revolut and Salesforce. Snyk customers and users collectively have run more than 300 million tests in the last 12 months to fix more than 30 million vulnerabilities in a 90-day period of time, the company claims.

    It may be a while before most organizations have embedded security tools within every DevOps workflow. However, given the resources being poured into DevSecOps, it may be more a question when DevSecOps will be finally achieved rather than if.

  • Microservices Adoption and the Software Supply Chain

    Microservices Adoption and the Software Supply Chain

    Software development is undergoing industrialization, with more and more software rapidly assembled out of components and an emphasis on building automation around software validation and release processes.

    Modern cloud-native software is no longer a monolithic application living in a single repo with the majority of its dependencies self-contained. It is integrated from third-party components provided by vendors, cloud providers and open source software (OSS) components, with as much as 90% of code coming from such dependencies. This allows developers to build applications much faster but makes maintenance much harder as it is no longer entirely under their control. If the third-party vendor makes a change to their API, developers are on the hook to update their applications before they break. 

    Each microservice interacts with hundreds of other microservices and each is built and released independently, making it hard to understand how to coordinate API changes across them. For example, if you want to make an API change to a microservice, what impact will this have on the consumers in the organization? Who will need to coordinate the change? When it was all within the same repository and released all at once, it was much easier to make such changes.

    The Changing Nature of Technical Debt

    What we continue to call technical debt is really the activities that are related to tending to and upgrading our software when third-party components are evolving or have common vulnerabilities and exposures (CVEs) and need to be upgraded.

    These are tedious, repetitive tasks that usually fall to the most experienced engineers as they require technical expertise to do correctly. Such activities can paralyze engineering organizations and are a tremendous burden on engineers; that often leads to burnout. Up to 30% of engineering time is spent on technical debt. The perception that somehow developers were responsible for accruing this technical debt and are doing something wrong that prevents them from keeping up is hugely demoralizing and demotivating. 

    However, if we reframe technical debt as software supply chain management and stop blaming engineering for it, we can make maintenance more predictable and consistent. By taking steps like inventorying third-party components and determining how pervasive they are in the application (frameworks take more effort to maintain than a third-party API that you just call from one part of the application), an organization can arrive at a maintenance estimate. 

    These activities are highly repetitive across organizations, as everyone is integrating some subset of the same third-party components to create business value. This high level of repeatability points us to automation. 

    Specialized Developer Tools for Software Maintenance Don’t Exist

    When developers write code, their hands often lag behind their thoughts. They think much faster than they are able to type the code. You often hear developers finishing a tedious task and complaining, “My fingers hurt.” When their IDE can autocomplete syntax, it helps reduce this lag to minutes.

    When a major framework version is released, developers contemplate the codebase. Maybe they need to change this type to that, reorder some method arguments or change dependencies. The types of changes they need to make are enumerable and must be made almost immediately; again, the ability to implement them lags far behind their recognition of the problem. Often, this type of lag can represent months or years of work when dealing with a large codebase. Accumulate enough of this lag and the codebase can grind to a halt.

    That lag kills the joy in development for most senior developers and leads to burnout. Upgrades/migrations amplify that lag by orders of magnitude.

    However, there are few (if any) tools that focus on helping developers automate these remediations/upgrades/migrations. The IDE’s focus is rightly on helping developers write new code and maintain their own code, but code maintenance needs to be coordinated across multiple places across the codebase, within the same repository or across the repository boundaries. Often, developers have a pattern they want to change in their repository, but the IDE only suggests this as an improvement in their current location (one file). Other times, changes need to be coordinated across repository boundaries if developers want to change APIs and their consumers. That’s why this technology needs to exist outside of the IDE.

    OpenRewrite

    OpenRewrite is an open source project that offers semantic analysis and refactoring of code as standalone operations so that everyone can use and contribute over time; composing more and more refactoring operations that make whole framework migrations possible. It is integrated with build tooling and can be plugged into different workflows, from CI integrations to mass refactoring of multiple microservices/repositories. Similar to an IDE or other developer tools, OpenRewrite manipulates an abstract syntax tree (AST) representation of code, but this AST has special characteristics that allow it to transform this AST and generate code back to the standard text representation. 

    The OpenRewrite AST is produced by guiding the compiler through the first two phases of compilation to generate compiler type-attributed AST. This is then mapped to the OpenRewrite AST, preserving formatting and breaking cycles in the AST. The three unique characteristics of OpenRewrite AST are:

    • Type aware, allowing 100% correct semantic code analysis and transformation.
    • Style-preserving, so transformations produced are idiomatic within the projects they are applied to. So, the same transformation applied to multiple projects will potentially look different.
    • Serializable, allowing to output the AST from the build and operate on it outside of the build en masse.

    OpenRewrite ASTs is full of metadata about code. Visualizing the code itself as a highly connected tree of various vertices and edges, we can start to get an idea of the density of this representation. Syntactic elements alone are blue, type attribution is yellow and formatting information is red. If we were to just represent syntactic elements, the tree might look something like this.

    Adding type attribution and formatting, the tree looks quite a bit denser.

    A spanning tree of the data involved in this AST can be laid out in three-dimensional hyperbolic space and then projected onto the unit sphere similar to the way that two-dimensional space can be projected onto a disk—the famous Poincaré disk model. This three-dimensional approach was pioneered by Tamara Munzner at Stanford.

    Every edge that isn’t part of the spanning tree is rendered as a lighter gray, and, as you can see, it is dense! In fact, to make this layout more tractable, we’ve omitted in this view an additional 600,000 vertices representing type information that make the syntax itself seem tiny.

    OpenRewrite calls a single code search or transformation operation a recipe. It provides a number of building block recipes such as find method, change method, find transitive dependency, upgrade or exclude dependency. These recipes, in turn, can be composed into more complex recipes by grouping them together into a composite recipe. When the building blocks are not enough, a recipe can be written as a program in the same language as the code we want to transform, allowing us to encapsulate complex logic with the full expressiveness of the language already familiar to developers. We don’t need to learn a new DSL or programming language.

    These building blocks abstract away many of the details to ensure that edits that we make to source code obey the original style of the project. This is possible for even complex changes like this automated migration from Spring Boot 1.x @ConditionalOnBean to Spring Boot 2.x AnyNestedCondition looks idiomatically consistent in the context of the project that it is inserted into:

    It is important to note that OpenRewrite is not a replacement for an IDE. With software now composed of so many independent components, we need to start to remediate and manage it using new approaches that go beyond single lines of code to looking across entire code repositories.

    Olga Kundzich, co-founder of Moderne, co-authored this article.

  • BMC Acquires StreamWeaver to Advance AIOps Strategy

    BMC Acquires StreamWeaver to Advance AIOps Strategy

    BMC has acquired StreamWeaver, a provider of a platform that makes it simpler to capture streaming data in real-time, as part of an effort to extend the BMC Helix IT operations platform infused with artificial intelligence (AIOps). Financial terms of the transaction were not disclosed.

    Margaret Lee, general manager and senior vice president for digital service and operations management for BMC, said StreamWeaver brings with it a wide range of integrations with multiple data sources that BMC will use to both train AI models and improve overall observability. While connecting to data sources is relatively straightforward, the challenge now is collecting that data in real-time in a way that is easily repeatable for IT operations teams, noted Lee.

    In general, AIOps continues to gain traction as IT environments become increasingly complex. Most IT operations teams are not going to be able to manage highly distributed computing environments without augmenting their IT staff with some AI, said Lee.

    Naturally, many rote tasks that were once performed manually by IT operations professionals will become increasingly automated. However, there will always be a need for IT personnel, noted Lee.

    Arguably, there may come a day when IT professionals will not want to work for organizations that don’t have an AIOps platform in place simply because the tasks assigned to them would otherwise be considered archaic, especially if they involve a series of tedious tasks that not many IT professionals really want to perform.

    At the same time, AIOps should reduce overall risk by making it easier for IT operations teams to proactively identify issues before they disrupt an application service. That’s especially critical as IT environments—increasingly with microservices-based applications constructed using containers—and serverless computing frameworks are being deployed with greater frequency alongside legacy monolithic applications.

    BMC has been making a case for AIOps as far back as 2017. However, since then, a wave of startups and IT operations incumbents have either built an AIOps platform from the ground up or updated an existing platform with AI capabilities. It’s difficult to assess which vendors are now ahead in terms of the AIOps race, but one thing that is certain is that IT organizations now have plenty of options.

    From a cultural perspective, however, not every organization is entirely comfortable with AIOps. While some IT professionals remain skeptical, others view AI as an existential threat to their existence. Most IT teams are also concerned about the potential impact any flawed AI model might have on IT operations—it’s one thing for a human to make a mistake, but an AI model making a mistake at scale could be catastrophic.

    There are few IT organizations that are embracing AIOps across the board, but most do recognize that AIOps platforms, in one form or another, are inevitably here to stay. After all, machine learning algorithms never forget anything they learn. Nor do they ever take a day off or decide to leave when offered higher pay. The challenge is figuring out exactly where the interface between IT staff and algorithms is going to be—not just today, but also in the near future.

  • Verica Launches Database for Sharing IT Incident Reports

    Verica Launches Database for Sharing IT Incident Reports

    Verica, a provider of an incident management platform, announced this week it is making available a database to make incident reports publicly available via a single repository.

    Courtney Nash, senior research analyst at Verica, said the Verica Open Incident Database (VOID) initiative is part of an effort to enable organizations to learn from outages and failures caused by software. Today, reports about such incidents tend to be difficult to find because they are scattered all across the internet, she added.

    That lack of transparency makes it difficult to compare and contrast incidents in a way that surfaces meaningful insights, noted Nash.

    Launch partners for the VOID include Indeed, NS1, Adaptive Capacity Labs, Security Scorecard and Auxon. Each will provide support to help fund the development, research and dissemination of knowledge generated by the project.

    The VOID contains more than just standard company postmortems or status updates, added Nash. Verica is committed to applying metadata to better understand how individuals, companies, media and others reacted during an incident.

    One early finding already derived from this initiative is that mean-time-to-resolve (MTTR) is proving to be an inaccurate metric due to the varied distribution of incident data. Other findings include the fact that only one-quarter of incident reports took the time to identify the root cause of an issue, while less than 1% took the time to study incident reports of near-misses.

    In general, there’s now more riding on the performance and availability of IT environments than ever. Digital business transformation became all the rage in the wake of the economic downturn brought on by the COVD-19 pandemic. As the economy continues to recover, businesses now expect IT to respond and pivot to changing conditions no matter how sudden.

    DevOps teams have, of course, learned to expect the unexpected by embracing modern incident management based on DevOps best practices that enable IT teams to adroitly respond to any outage or sudden degradation in application performance. The most important thing to remember about incident management is that practice makes perfect. While practice runs may never precisely mirror an actual crisis, organizations that have processes that are well-known by their IT incident teams and that practice ‘fire drills’ tend to respond both faster and better. After all, the more muscle memory there is to call on the more resilient the organization will be.

    It’s not clear to what degree IT organizations will be willing to share incident reports. However, there is an opportunity for IT professionals to come together that is similar to how engineers from different airlines collaborated to make flying safer for the general public, said Nash. In fact, with so many processes now dependent on software, it’s amazing there have not been more catastrophic events due to software issues. It’s even arguable that it’s only a matter of time before such levels of collaboration are mandated.

    In the meantime, the first step toward achieving that goal may simply be sharing a report with colleagues so they can learn from previous mistakes.

  • Can Web 3.0 Solve the Linkrot Crisis?

    Can Web 3.0 Solve the Linkrot Crisis?

    Data on the web is more ephemeral than you realize. A recent examination of linkrot and content drift by Harvard Law School found that as time progresses, web-based content is becoming more and more lost. By analyzing half a million links within New York Times articles, 6% of deep links from 2018 were found to be inaccessible, and 72% of links from 1998 were completely dead.

    The rotting web could have severe consequences that affect society’s collective understanding. Consider links used to substantiate pivotal arguments in the course of law. Studies now show that 49% of hyperlinks in Supreme Court decisions no longer function. The linkrot situation undermines humanity’s shared intelligence, which the Internet once promised.

    So much of the content on the web is outdated, inaccurate or has changed its URL destination, a problem known as content drift. Therefore, band-aids like the Wayback Machine internet archive have risen as a steward of the historical web archives. However, some proponents in the industry argue we need a more modern, decentralized, immutable reference for this historical data. You may have guessed where this article is heading—blockchain.

    I recently met with Jonathan Victor, Protocol Labs, to see how Web 3.0 and blockchain might be used to solve the linkrot endemic. According to Victor, smart contracts that reference canonical URLs could help create a more resilient web. This would, in effect, require a peer-to-peer model with shared ownership and incentive models to sustain content longevity.

    The State of Linkrot

    The longer data sits, the higher the likelihood of losing track of it. Forgotten databases and rising data storage costs can easily lead to whole networks of websites and user data being accidentally (or intentionally) destroyed. Putting so much faith in private companies to preserve data indefinitely is not viable, explained Victor.

    Public data isn’t really as public as you think it is. And older, user-generated content is often the first domino to fall. For example, a massive amount of media on MySpace was deleted in a botched server migration in 2015. In 2021, Yahoo Answers similarly erased 16 years of user-generated content in its shutdown. Surprisingly, users have little ownership or control of their data on forums and social media. A ban on Twitter, for example, could result in inactive links within posts.

    When an article points to a website, this imbues an implicit dependency on an external organization to keep the domain live, said Victor. The second dependency is on the domain owner to keep content unchanged. “If we can talk about content and how to reference it in a different way, maybe we can create references that are much more resilient,” suggested Victor.

    How Web 3.0 Can Help

    One way to solve the linkrot issue is to create a ‘fingerprint’ for web resources. This could be a canonical reference to specific pieces of content. Organizations would store and retrieve content based on a hash of that contract itself, Victor explained. Such a system would likely require an incentive-based, non-exploitative, decentralized network where web users own their data and control who profits from it.

    Some have coined this concept Web 3.0. Victor describes Web 3.0 as a broad classification of distributed technologies and tools that enable a peer-to-peer internet underpinned by blockchain.

    For example, one open source project spearheading progress on this front is IPFS, the InterPlanetary File System. IPFS is a peer-to-peer hypermedia protocol that uses blockchain to store cryptographic hashes. The project is “designed to preserve and grow humanity’s knowledge by making the web upgradeable, resilient and more open.” With IPFS, a client requests content using the hash and retrieves content. IPFS is based on Filecoin, a permissionless storage network.

    Benefits

    A decentralized nature fits the original goal of the open web, allowing creators to own and share content horizontally. If you can decouple data storage and retrieval from a specific location, it gives you an impressive ability to rearchitect modern communication, said Victor. This could have a number of benefits:

    • Redundancy: First off, a major benefit is the redundancy of data. Distributed nodes assure data is globally available and temporally accessible. Resources are in multiple locations, ensuring better global availability and temporal persistence.
    • Incentives: An incentive-based decentralized network could use block rewards to subsidize costs, helping fund storage initiatives. Building on the blockchain can also ease payment acceptance aspects — great for donation-based data hosting groups.
    • Efficiency: By using smart contracts, you can programmatically handle storage requests. “Blockchain gives you higher-level primitives to run this stuff automatically,” said Victor. This helps tie on-chain activities to real-world storage, Victor added.
    • Low-cost: It can be cheaper to store content on distributed networks than a typical cloud provider. Impressively, Victor estimated it would cost 1/1000 of the cost of Amazon. A distributed environment could also bring speed gains; similar to torrenting, seeding from distributed sources increases speed in orders of magnitude, said Victor.
    • Communal ownership: Such a model could spur more communal ownership for some content. This could benefit public-good assets that don’t have a clear owner. According to Victor, this aspect especially benefits universities and open source data initiatives that rely on public support to function.

    A Compelling Problem

    Constructing a new peer-to-peer Internet obviously has significant barriers. Namely, a decentralized structure requires a large group effort, and this community action may be difficult to encourage, let alone coordinate. It will also require consolidation around open standards and protocols, as many different decentralized projects crowd the scene. Shifting pre-existing architectures to this new style could also introduce roadblocks.

    “A lot of content on the internet becomes a public good,” says Victor. Yet, perpetual access to this content is fickle, even for well-established online communities. Some research has found the average lifespan of a website to only be 100 days. Hosting fees or accidental failures are often root causes for the loss of data.

    The fragility of the web is an issue that will take not only innovative development effort, but a radically new mindset to tackle. “It’s a really compelling problem,” added Victor. Using blockchain-based smart contracts to store canonical references for content is one proposed solution. But Web 3.0 is in a primitive stage, and to progress, it will take significant community action.

    Doing so in a way that is sustainable will be the trick. Open standards and blockchain-based protocols could allow the technical means for such a network to flourish. And a cryptocurrency backing could help incentivize participation and provide a means of funding. This decentralized non-ephemeral storage could help enable perpetual storage for the web’s most precious resources.