devopsy.kinsta.cloud

Category: DevOps Practice

Welcome to the New Field of Software Supply Chain Management

Supply chain management is the newest ‘shiny object’ in both the DevOps and DevSecOps communities. But what does it mean in relation to software development?

Historically, supply chain management is a commerce term that refers to tracking the logistics of goods and services moving between producer and consumer. This would include the storage and flow of raw materials with a focus on the channels that support the process.

When we look at this concept from the perspective of software development, we will soon see that it is a new form of software configuration management. Yes, the new SCM is … SCM. The core of supply chain management is the process of controlling changes in software. The new SCM expands those old practices to include all ‘raw materials’ which we manage as part of our software build and package steps. For instance, when companies purchase and download open source libraries used in their in-house software. Supply chain management includes the interrogation of these ‘raw materials’ as well as all of the internal code and libraries. We will soon be hearing more about software bill of materials (SBOM) reports and difference reports, as they are critical for understanding the objects that flow through the software we create.

The software supply chain or software configuration management practice hasn’t been talked about much until recently. This area of expertise has never been ‘sexy.’ That is, until the SolarWinds supply chain hack. The SolarWinds hack was a hard lesson to learn, but it taught us to take a closer look at how we consume our raw materials and to be more serious about how we build and package our software.

According to FireEye, the firm that discovered the backdoor supply chain hack:

“SolarWinds.Orion.Core.BusinessLayer.dll is a SolarWinds digitally-signed component of the Orion software framework that contains a backdoor that communicates via HTTP to third-party servers. We are tracking the Trojanized version of this SolarWinds Orion plug-in as SUNBURST.”

It was obvious, something went very wrong with the creation of that .dll. I don’t pretend to fully understand how this breach occurred. But I do understand the build process and have been talking about the potential for these types of breaches for over 25 years.

There have always been big security gaps and vulnerabilities in our most basic software development practice—the compile/link process. Most build processes are imperatively defined using an Ant, Make or Maven script. Each time the build runs, it rebuilds all the binaries, even if the corresponding source code has not changed.

This is a problem. Instead, we need a build that is incremental. Incremental builds are difficult to script because they need more advanced logic. Most developers are not given adequate time to create a better build process, one that only rebuilds the changes. This means it is next to impossible to audit why a binary was recompiled because they are always recompiled. If we only recompiled the binaries that had corresponding code updates, we could more carefully audit the code and make sure that only approved coding updates were included.

This may not have caused the SolarWinds attack, but it could have. Once hackers have gotten past the firewall they can find the build directory, read through the build scripts and identify the perfect piece of code to update with their bad function. The next time the build runs, a ‘clean all’ recompiles everything and, like dark magic, a SUNBURST is created.

In our not-too-distant past, the developer community had tools that controlled software builds beyond what an imperative script could do. OpenMake Software, for example, provided a build automation solution called Meister that ensured that incremental builds were easy. As part of the build, it generated SBOM reports and differences reports allowing teams to easily audit and compare the actual source changes to confirm that only approved code updates were included. Another example was Rational Software’s ClearCase which used a program called ClearMake. ClearMake used a process called ‘winkin’ to reuse ‘derived’ objects that had not changed. This process created an incremental build that could be easily audited.

I realize that I am simplifying the overall problem, but it is a good example of the basic supply chain best practices and how potentially vulnerable our processes are.

The time to carefully think about how we manage our raw materials and overall supply chain is now. I’m guessing that most monolithic practices will not change much; it is too costly and impactful to rewrite hundreds of CI pipelines. But that does not mean we can’t solve the supply chain puzzle as we shift into cloud-native development in a microservice architecture.

In a microservices architecture, we need to begin thinking about what the supply chain looks like. How do we track raw materials that are in the form of hundreds of small ‘bounded context’ functions? The good news is there is an emerging microservice catalog market that can become part of the supply chain solution. A microservices catalog can track versions of services to versions of their consuming applications. Your new SBOM report will have two levels: First, an SBOM for the microservice. Second, aggregated data at an application level. Remember, in this new architecture a full build is not done.

It’s important to understand that microservices are naturally incremental and are independently built, packaged and deployed. Managing these small services opens the door to more closely auditing their source code before they are released.

As we shift from monolithic development to microservices, we have an opportunity to incorporate supply chain and configuration management best practices. This includes being able to carefully audit the code before a release is done. And remember, we must do this as fast as possible. It sounds like a big task, but it’s well worth the effort. Collectively, we will find the answers as we explore new ways to manage our DevOps pipeline and incorporate configuration management and DevSecOps principles.

October 18, 2021
CNCF Adopts Meshery to Advance Service Mesh Management

The Cloud Native Computing Foundation (CNCF) announced this week during the ServiceMeshCon/KubeCon + CloudNativeCon conference that Meshery, a service mesh management plane created by Layer5, has become a sandbox-level project.

In addition, Layer5 has also donated Service Mesh Performance (SMP), a set of tools for measuring the efficiency of a service mesh, to the CNCF. SMP provides an open source framework to define standardized benchmarking practices, performance test configurations and measurements as part of an effort to create a MeshMark index for rating a service mesh. A set of service mesh performance methodologies will also be published by the IEEE later this month, developed in collaboration with engineers from Layer5, Intel, Red Hat and HashiCorp.

Layer5 CEO Lee Calcote said the goal is to make it simpler for IT teams to determine which service mesh to employ based on their specific use case requirements.

Several service mesh platforms are starting to gain traction as a means to both manage application programming interfaces (APIs) and provide a layer of abstraction between lower-level networking and security APIs that makes those services more programmatically accessible to developers.

Meshery itself is an implementation of SMP and the conformance tool created for the Service Mesh Interface (SMI), a separate CNCF project to promote interoperability between service meshes. Meshery is based on a set of microservices that are designed to make the platform easily extensible.

That approach makes it possible to manage multiple instances of service meshes that might be deployed on virtual machines as well as Kubernetes clusters using a range of management tools that are part of the Meshery project.

Meshery supports AWS App Mesh, Citrix Service Mesh, HashiCorp Consul, Istio, Kuma, Linkerd, Network Service Mesh, NGINX Service Mesh, Open Service Mesh and Traefik Mesh. Meshery also provides IT teams with more than 60+ best practices in the form of templates that can be used to deploy a service mesh.

It’s still early days as far as adoption of service mesh in the enterprise is concerned. Most IT teams still rely on either proxy software or a gateway to manage APIs. However, as the number of APIs that need to be managed steadily increases, it won’t be long before many IT teams start to at least evaluate a service mesh.

In the longer term, the impact those service meshes are likely to have on the management of IT could be profound. It is becoming apparent that a service mesh also provides a higher level of abstraction for invoking networking and security services. In fact, the service mesh may wind up playing a key role in driving the convergence of network and security operations with DevOps best practices.

In the meantime, IT teams will need to decide to what degree they may want to standardize on a single service mesh versus letting individual teams implement their own preferred platform that could one day be centrally managed via Mashery. One way or another, the way APIs are managed and invoked will soon fundamentally change forever.

October 14, 2021
Top 5 Must-Haves for IaC Automation Tools

These days, there are a lot of different DevOps tools to accomplish a lot of different jobs. Almost daily another startup comes out with a new and innovative product or a newer (maybe even better) version of existing tools. One of the biggest innovations has been infrastructure-as-code (IaC). Giving infrastructure admins and developers alike the ability to create, manipulate and destroy infrastructure using code files changes the way we administer our environments. Though these tools are revolutionary and can solve many problems, sometimes they come with a whole new set of issues—as can any type of tooling, really.

Generally, IaC tools are run locally at first. When an infrastructure admin or developer is just starting out with it, they store the files locally; maybe in a repository, then clone the files to execute locally. This is perfectly acceptable and a very common practice. The issues arise when it comes to scaling this practice to a team; for example, with regard to visibility and governance. If everyone is running their deployments locally, the team doesn’t have clear visibility into who is deploying what, when and with what variables. This is where IaC automation tools come into play.

There are many different types of tools to solve these problems. Some are better than others. CI/CD pipeline tools, custom-built automation platforms, purpose-built IaC automation platforms—each has their pros and cons. Getting into all of that can be overwhelming and we could go on for hours about each option. But, we’re here to talk a bit more generically today. So, we’ll save all of that for another time. For now, let’s take a step back and talk about the top five must-haves for any infrastructure-as-code automation platform you’re considering. Because each company and use case is different, we’ll talk about these considerations in no particular order.

Role-Based Access Control (RBAC)

This first factor is pretty important. I know I said I wasn’t discussing these in any real order, but this one should really be close to the top of the list for anyone looking into IaC automation. We talked about how automating IaC can cause visibility issues when you scale across a team. You may want to know who deployed what, when, where and why, for example. Usually, the reason you want that kind of visibility is that you would like to control all of these aspects. That way you can help limit waste, budget problems, security concerns and the like. You may want to provide self-service deployment access, but only within certain parameters; maybe only to certain people or teams. Having granular RBAC can help you design a security policy that makes the most sense for your organization and its needs. Some tools have this baked in, some take a bit of work to make it work. The point is that you should have some form of control over your deployment process.

Self-Hosted Agents/Runners

Following the security thread: A lot of these IaC automation tools are SaaS platforms. SaaS can be very secure. But sometimes, for compliance or regulatory reasons, you may want to retain a bit more control over your deployments. This is where self-hosted agents or runners come in. This type of design allows you to keep your “secrets”—such as your cloud credentials and other sensitive variable information—secure in your own way. You can use whatever secrets management solution you want, be it something from your cloud provider or another third-party solution. It also allows you to keep your code secure. All of these tools, when deploying your code, have to get a copy of that code to deploy. So, they’ll do a ‘Git clone’ or some kind of file copy process to get your files from where they are stored, to execute them. If this is a SaaS solution, then they will essentially have access to your code. Again, this may not be a huge issue for some, but it very much is for others. If you don’t want to give a third party this level of access to your code and secrets, a self-hosted agent or runner can help keep it all secure and under your control.

Plan on Pull Request

This feature is a big one when it comes to workflow. Some may not even know how important plan on pull request (PR) is until they realize how easy it can make their life. This feature may be called something different depending on the platform. Plan on PR, PR plan, speculative plan, etc. Essentially, this means that whenever you open a pull request against a branch, your automation platform will complete a deployment just up to the “plan” phase so that you can see exactly what this code change will do. For example, if you accidentally add an extra 0 to a variable or code file, you may end up with 100 instances instead of 10. Ideally, you’d like to know what was going to happen with a change before it is deployed, giving you the option to quickly fix a potential mistake before it actually goes through. In DevOps, you always want to ‘fail faster’ and close the loop so that big mistakes are caught before they happen. That is why this feature is so important. And if the automation platform you choose is doing this correctly, when it runs this plan phase it’ll automatically update the pull request directly with the possible changes as a comment. This way, your developers can see what is going to happen before they try to merge the deployment back to the main branch for deployment to production. And if they take it a step further by adding service checks to the pull request, you can even configure it so that if the deployment plan fails for any reason, developers are actually blocked entirely from merging the PR back until it is resolved. All of these features help to enable the ability to work in a GitOps methodology. Don’t get me started on the wildl misuse of the term GitOps—it is very specific. While one of the central pillars of GitOps is using your repository as your “single source of truth” for what should be deployed, it’s not just using Git to store your Terraform files; it is an infrastructure administration via pull request methodology. If you want more information about GitOps, you can read more here.

Continuous Deployment

This feature is essential for most folks. But, depending on the workflow, it can be less important. Even if it isn’t a must-have now, it may be down the road when you evolve and grow into new workflows. Continuous deployment is the ability for your automation platform to have a webhook or other connection to your source code repository so that it can automatically trigger a deployment process whenever you commit or push to the repository. This feature goes hand-in-hand with the plan pull request feature we just spoke about. To make that feature function properly, a CD feature would be needed, as well. Now, I want to make sure I clarify. Just because you have continuous deployment enabled doesn’t mean it is just going to push to production in the middle of the day every time a commit or push is done. You should still be given the ability to pause the deployment process after the plan phase. This way you can validate that the plan looks acceptable before you approve the deployment. For some use cases, like developer sandboxes, you may “auto-approve” the deployments because you don’t care about having to get validation first. It all depends on your workflow and processes. A good platform will give you the ability to control all of this.

Shift Left Extensibility

This one is a bit of a wild card. It can mean a lot of different things depending on the situation. Essentially, what I am talking about here is the ability to integrate—or at least be interoperable—with a multitude of other tools during the deployment process. There is a concept called continuous verification which means to shift some tools and processes left into the deployment process so that you can stay on top of problems before they start. In the PR plan feature section earlier, we talked about catching issues before deploying. This isn’t just for resource configuration issues. What about security issues? Compliance issues? Performance issues? Or even budget issues? If your IaC automation tool gives you the ability to integrate the processes and tools you are using to validate these things into the deployment process, they no longer have to be an afterthought. You never have to worry about going over budget if you are constantly validating the budget each time you deploy. You don’t have to worry about a security issue if you’re validating security during every deployment. This type of functionality can look and be named very differently between tools. But as long as you have the ability to get all of these other groups (security, finance, etc.) involved sooner, everyone will be better off. Like we talked about before, in DevOps, it’s always better to fail faster. And integrating these checks into your deployments can help you achieve this.

Conclusion

We’ve made it through all five! We’ve talked about some basic security features. We talked about some automation workflow must-haves to make your life easier. And we even scratched the surface on some next-step-type things to look into such as continuous verification. That was a lot—hopefully this has been helpful for anyone looking into IaC automation platforms and you now have an idea what to look for. Some features may be named differently, but the ideas behind them are the same. If your platform can’t do these, it may not be the right one for you. Or, maybe it is and that feature just isn’t that important to you. As I mentioned, your mileage may vary. Every organization is different.

To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021

October 14, 2021
Rethinking Your Approach to Ops Governance

The “You build it, you run it” approach has changed the mindset of modern organizations that are increasingly adopting DevOps practices. No longer completely beholden to the requirements of operations, engineering teams are continually acquiring more autonomy to deliver rapid innovation. At the same time, more organizations are adopting platform teams to accelerate development, empowering teams with easier access to services without requiring them to understand the underlying complexity. While this approach is helping teams build, scale and ship software faster, it’s also resurfacing previous areas of tension around operational governance.

Organizations are generally choosing between two governance approaches: Mandated or paved path. Each has a different perspective on the appropriate level of autonomy to grant development teams and where governance should sit within the development life cycle. But there’s a third approach—the “trust and verify” approach—that can provide both high levels of autonomy for development teams and the sophisticated governance that operations teams need to ensure compliance and security.

What is a Platform Team and Why are Organizations Adopting the Model?

A platform team provides a foundation of self-service APIs, tools, services, workflows and support that accelerates the delivery of digital services and products. Platform teams operate similarly to any R&D team; employing customer support, operations, development, SRE, product managers and other personnel. They use research, customer feedback and usage data to iteratively improve their services.

Rather than each development team reinventing the wheel, the platform team builds shared services that shield developers from the underlying complexity of infrastructure and enable self-serve access to services. As engineering resources become even more in-demand and scarce, organizations are using platform teams to optimize resources and share outputs.

Two Platform Governance Approaches: Mandated Vs. Paved Path

Organizations today typically are using one of two approaches to platform governance: Mandated or paved path.

A mandated approach solves the governance problem by mandating that all engineering teams use the platform. However, this approach puts the onus on the platform team to keep up with the changing demands of engineering. Organizations must invest in a platform team that can keep up with engineering requirements so they still feel like they can innovate and use the best tools.

The second approach is a paved path, popularized by Jeff Lawson of Twilio. This allows for engineers to go outside the platform if they are willing to take on the SLAs the platform provides and are responsible for all the costs of maintaining their own alternatives. Lawson said in this McKinsey interview, “What you do is you create the incentive structure for teams to take the paved path. But if they really have to go a different route, you make it possible for them to do that.”

While the paved path approach offers more autonomy for engineering teams, it requires teams that veer off the path to maintain the integrity of their software. Without guardrails, operations must put full trust in teams to keep their services secure and compliant. It can often be difficult for operations teams to maintain visibility into the services and infrastructure that fall outside the core platform.

So, given that a mandated approach restricts autonomy and a paved path can make compliance more difficult, what is the middle ground?

Bridging the Gap Between Mandated and Paved Path

First, it’s important to understand that governance is not synonymous with command and control. Instead, operations teams should think of governance as the practice of maintaining visibility and providing best practices. Rather than a mandated platform approach, organizations can choose an approach that offers more autonomy to engineering teams (similar to paved path) while also maintaining operational governance. This is a trust and verify approach.

Using the trust and verify approach, organizations should focus their efforts on technology, not people, to maintain compliance. The goal would be to have a platform that could easily connect to the entire toolchain any product team uses for complete visibility and context. In this way, operations can gain governance without adding friction for product teams. This requires a top-down approach within organizations: I&O leaders investing in engineering resources or an off-the-shelf platform that can enable this technology. Both engineering and operations can flourish in this modern environment. Approaching governance from a trust and verify perspective will expand access to innovation while ensuring compliance and security standards.

Governance Should be Over Technology, not People

Creating a bridge to a trust and verify approach means placing the responsibility for governance on technology rather than people to maintain detailed audit trails of human and machine actions tied to identity and reduce the potential for human error through automation. This approach provides autonomy for engineering teams while also ensuring governance through visibility and process controls embedded into workflows and tooling.

Shellie Miller, senior manager, technology governance and compliance at Hulu, believes technology can support DevOps practices. “To have a successful DevOps organization, we don’t need to have command and control or have this clear line of separation between dev and operations. It’s putting the responsibility on the infrastructure, the platform and the tooling and the policies and the processes that you have to enable those self-serve capabilities,” Miller said.

Mitigating Human Error Through Automation

This investment in technology should focus on automation and process controls that fit within teams’ current processes. “What auditors are really trying to do is mitigate human error,” said Miller. “Teams should really focus on automating controls, building those security practices and those best practices into the platform. That way, dev teams are just subscribing to that offering.”

Changes to applications, systems, data and code—one of the biggest culprits with regard to risk—is where automation and process controls embedded into platform offerings can make a huge impact. “From an engineering perspective, constant change is needed to support rapidly changing consumer demands,” Miller added. “However, from an audit perspective, these activities increase risk of material impact to viewer experience, revenue and/or financial reporting.”

Investment in Governance Comes From the Top

While technology holds the key to better governance and compliance standards, there’s an investment that must be made, whether that’s investing engineering resources in building automation or investing in an off-the-shelf platform that can enable automation.

“Leadership and organizational backing is absolutely key,” said Miller. “You can’t get to compliance or governance or any kind of operational maturity, for that matter, when you’re doing a bottom-up approach.”

Building a compliance and security program around technology and not people means having the proper skills and mindsets. A bottom-up approach could create friction, especially due to a lack of education and understanding of the why and how. A top-down approach ensures there are resources to both implement the technology and to foster a culture of understanding and education.

The most common question I get asked is, ‘Why can’t we trust our people?’” Miller said. “Compliance requirements don’t generally start with the thought that our people are being malicious. We should run an organization of trust, and through automated monitoring and governance, we can verify and govern without being intrusive and disruptive. Lack of training and the demand on teams is what drives the perception that there’s no trust. Security and compliance is intended to mitigate risk and prevent threats to the systems that enable the business.”

Building on a “Trust and Verify” Platform Approach

When it comes to governance, technology is our greatest asset. Of course, this investment can be substantial for many organizations that don’t have the engineering resources to build in-house. Today, new open source or off-the-shelf platforms enable operations teams to build self-service capabilities that provide guardrails against human error, an audit trail of human and machine actions and visibility across disparate tools and services.

No-code solutions that also provide a developer platform for deeper customization enable teams to build out these core requirements of governance without consuming considerable engineering resources. The door to a future where operations teams can provide more autonomy to development teams while maintaining governance is now open. Every organization has the ability to attain a “trust and verify” approach and meet the needs of both dev and ops.

To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021

October 14, 2021
5 Steps to Succeed With Blue-Green Deployment
By allowing teams to maintain two production-ready environments at the same time, the blue-green deployment technique can significantly boost reliability. But blue-green deployment can also be difficult to execute and manage. Let’s unpack how blue-green deployment works, why it’s important and which best practices to follow for blue-green deployment success.

Blue-Green Deployment, Defined

Blue-green deployment is a software deployment technique in which teams maintain two identical but distinct environments that are capable of hosting a production application. One environment is “live” and actively serving users, while the other is kept in reserve.

When it comes time to deploy a new version of your application, you deploy it to the reserve environment. You then test and validate it to make sure it meets performance and reliability requirements.

If it does, you reconfigure your DNS or load balancers so that user traffic is directed to this environment, which becomes the new production environment. Meanwhile, the previous production environment is placed in reserve.

Benefits of Blue-Green Deployment

The blue-green deployment technique can boost reliability in several ways:
- Environment parity: Because the reserve environment in which you test your application becomes the live production environment, blue-green deployment ensures parity between testing and production. In turn, it reduces the risk that configuration differences could cause unforeseen problems in production.
- Failed deployments: If a deployment fails, it doesn’t impact your users because you deploy into the reserve environment. You can fix the issue while a stable user-facing version of your app continues to run in the other environment.
- Redundant environments: Although blue-green deployment is not a data protection or disaster recovery technique, having two production-ready environments available is useful in the event that one of your environments is wiped out by an event like a data center failure. In that case, you can restore operations quickly by using the other environment (provided it’s hosted in a separate data center, of course).
How to Use Blue-Green Deployment: Steps and Best Practices

While blue-green deployment is a great way to increase reliability, it does present some challenges. The following best practices can help you to handle them.

Use Scalable Infrastructure

One obvious potential downside of blue-green deployment is that it can essentially double the amount of resources you need to host your application since you have to maintain two identical environments. That can double your hosting bill.

One way to reduce the cost impact of blue-green deployment is to create environments that can scale rapidly without changing their core configuration. You could do this using cloud-based virtual machines (VMs) and autoscaling or via containers and an orchestration engine (like Kubernetes) that can scale them.

Using this approach, you can set up two environments that are effectively identical except that the production environment includes more VM or container instances than the reserve. Although you could argue that this is not quite a blue-green deployment because the environments are not perfectly identical, that semantic downside is outweighed by the benefit of not wasting a bunch of money on infrastructure you don’t need.

Chaos Engineering

Typically, chaos engineering entails experimenting with a production system to test its reliability. Chaos engineering offers many benefits, but it presents the obvious risk of creating a problem in production that impacts your end users.

Blue-green deployment allows you to work around this risk by practicing chaos engineering in the reserve environment. Of course, because you won’t have real users, performing chaos engineering under these settings may not allow you to test every aspect of your production application. But you can still test many of the configuration variables by issuing requests yourself within the reserve environment—all without worrying about breaking things for actual users.

Managing Database State

Keeping databases in sync when you switch between environments is one of the biggest challenges of blue-green deployment. Even small discrepancies in database state could cause big reliability issues.

The simplest way to address this issue is to turn off the database before switching and perform tests to ensure that databases in both environments are identical before they are reenabled. Of course, this approach may impact application functionality, because the app may not work properly if the database is turned off.

Another solution is to sync databases between your two environments. This should ensure that database state remains identical even as you switch from one production environment to the other. But it also means that problems with the database will affect both environments.

The bottom line: There is no simple way of managing database state in blue-green deployment. You should evaluate the different options available and consider which one best meets your needs. If you can tolerate a little downtime, stop the database and restart it during the switch. If you need total continuity, use syncing, but be sure to manage the associated risks.

Change Load Balancers, not DNS

There are two main ways to manage where traffic is directed, and hence which is the production environment: You can change your DNS configuration when you want to switch. That’s easy to do—you would simply change the mapping for yourapp.com from one IP address to another, depending on which environment is the live one.

You can also use load balancers to manage traffic. Load balancer configuration is more complex than simply changing DNS entries, but it also enables more sophisticated and granular control.

In general, it’s a best practice to manage blue-green deployment switches via load balancers, due to the extra control they provide.

Prepare for the Unexpected

While blue-green deployment can reduce the risk of user-impacting disruptions, it’s no silver bullet. You must also be sure that you have the reliability management tools you’ll need to detect and remediate issues within both of your environments.

Blue-green deployment is a powerful reliability engineering technique. But to get the most out of it, you must optimize the way you handle requirements like infrastructure management, traffic routing and database state.
October 12, 2021
Mastering API Gateways for the Modern Cloud-Native Enterprise

As modern enterprises continue migrating to the cloud and the need for connected applications and devices dramatically increases, API-driven architectures are being widely adopted. In addition to the convenience of being able to efficiently connect and share resources, organizations benefit from the innovation and accelerated development speed that APIs bring in multi-cloud environments. APIs are not new, but as more enterprises adopt modern application architectures, they are increasingly occupying a large amount of a software development team’s delivery efforts.

When enterprises modernize and migrate to the cloud, it’s natural that challenges arise which need to be tackled before they become pain points—especially when it involves a complex microservices environment. API gateways are an essential ingredient that helps businesses overcome many of the challenges associated with APIs. API gateways are deployed at the edge of an infrastructure, responsible for routing client API requests to backend microservices. Modern API gateways also come with additional perks organizations can benefit from as teams and infrastructure grow simultaneously.

Let’s take an in-depth look at some of the major challenges organizations face around APIs, and how to maximize their benefits.

Rapid Adoption Meets Growing Pains

API-oriented architectures greatly accelerate the agility of an IT department, as they’ve become a well-known pattern for modern application architectures. However, this approach can be deceptive, as many critical issues and concerns around APIs aren’t always apparent from the onset. If they are not properly identified, they become growing pains teams have to face on a daily basis.

The first challenge is the need to authenticate users when they make requests. API gateways allow organizations to integrate and consolidate authentication providers using their protocol(s) of choice. With billing and analytics, organizations need to understand the number of resources consumed when processing a transaction and how their customers consume their APIs. API gateways become the natural point at which to collect this data.

In terms of DevOps initiatives, some API gateways use service discovery to configure themselves dynamically which greatly accelerates a development and ops team’s velocity when shipping new features or fixes. To help teams deliver a highly available and resilient platform, API gateways are enablers that provide them with the puzzle pieces for advanced load balancing, rate-limiting, circuit-breakers, mirroring and more. These components make the DevOps mindset a reality. Teams can scale them as needed to realize production-grade environments, continuously updating deployments with software changes and avoiding any unintentional single points of failure (SPOFs).

In recent years, the evolution of API gateways has solved growing pains in the cloud, helping organizations leverage cloud-native technologies and workflows. Combining cloud-native technologies with an API-led integration platform creates an effective architecture for a digital enterprise to increase productivity. Automating production, operation and services is no longer a goal but a must-have.

Modern API gateways features within this environment go well beyond what one would expect at first glance. API gateways can be the cornerstone of a deployment strategy transformation. With hundreds—if not thousands—of services to continuously deploy, the risk of introducing a regression to the platform soars, and being able to address this problem immediately is not enough. API gateways can help you transition to modern deployment strategies and blue/green or canary deployment becomes commonplace—helping deployments become an easily managed daily task.

API Gateways to the Rescue for Cloud-Native Architectures

As part of the shift to modern application architectures, enterprises should invest resources in implementing API strategies as part of their software development initiatives. In turn, they should leverage API gateways to alleviate growing pains and simplify delivery efforts. APIs should be secured, managed, observed and even monetized.

API gateways can also help with critical issues related to authentication, security and SSL, billing and analytics, life cycle management, high availability and resilience, and they can even help you initiate a culture shift toward a DevOps mindset. By combining API-driven architectures and cloud-native technologies, organizations emerge with an effective architecture for the digital era. API gateways help overcome the challenges and unfold the networking intricacies of today’s cloud-centered world.

October 12, 2021
Your Tech is Only as Good as Your Infrastructure

A few weeks ago, one of my neighbors dug a ditch in his yard, hit a fiber internet line and knocked out my internet for eight hours. There I was, ready to start working, and I was cut off from my colleagues for an entire day.

That experience highlighted the importance of a good digital infrastructure—both at the office and at home—particularly as the pandemic has ushered in a hybrid work model. When the pandemic hit, organizations that were agile and had a strong digital infrastructure (including processes) were better able to quickly shift to remote work. Those that were not as digitally mature were forced to adapt.

Eighteen months later, it’s clear that the hybrid model is here to stay, making it even more vital that organizations ensure their workers have the right tech tools to work effectively—and it all begins with a strong digital infrastructure.

Just Because it’s On a Screen Doesn’t Make it Digital

Let’s begin with a definition of digital. I recently had a conversation with a business owner who proudly proclaimed, “We’re digital now.” In actuality, all he had done was put a PDF form on an iPad. Once his clients filled out the PDF, he still printed it and put it in a paper folder. That’s what I call an immature vision of digital, and it’s an extreme example of the ambiguity that often exists around the concept of digital transformation.

It’s not that paper is anathema to digital processes; in some industries (health care, for example) we want paper as a backup. For instance, if I’m a physician at your bedside and the power and/or Wi-Fi goes out and I can’t access your digital records, I need to be able to look at your chart and treat you. So there is a need for paper. But when considering the hybrid or remote work environment, the goal should be to move processes online.

Digital Infrastructure is the Key to Agility

As noted earlier, agile businesses are better able to adapt to changes like the move to a hybrid work model. Agile simply means the ability to move quickly and easily, the ability to move fast and make decisions in a fluid way. To do this, we must remove unnecessary steps so we’re able to focus on the end goal and the essential parts of the process.

This is where a strong (and reliable) digital infrastructure comes in. When you think about Maslow’s hierarchy of needs for digital infrastructure, the foundation is electricity and the internet, and then you build on that. I need computers and hardware and a network, and then I need software platforms and then I need an end-user interface.

But we’re getting ahead of ourselves; let’s go back to the basics. Once we have reliable electricity and internet, we need to look at other technology that can benefit the remote worker, like upgraded microphones and cameras for all those Zoom meetings.

These upgrades shouldn’t be an afterthought on the part of organizations; they are at the center of their employees’ hybrid experience and can be another key to success. It’s not enough to send employees home with a laptop and a monitor anymore. Employers must focus on ergonomics (standing desks, good chairs) and the hardware their employees need to be effective at home. If the employer doesn’t provide them directly, they might consider offering a stipend for employees to make the upgrades themselves.

(Organizations might even consider providing generators to workers who live in rural areas. I have one at my house. And once satellite internet service comes to my town, I’ll have a much more reliable digital infrastructure.)

Now we can start building on that infrastructure. Start with audio and video communication (Zoom, Teams, Dialpad). Then add chat (Slack, Google Chat) and a good customer relationship management (CRM) system (Salesforce, HubSpot). Where does your data live? Having a great CRM system is critical.

Using Digital to Put People First

Organizations ignore the importance of digital infrastructure at their peril. Not only are the days long-gone when employees stayed with a company for 30 or 40 years, but we’re now in the midst of the Great Resignation, where workers are leaving their positions in droves. So it’s incumbent upon organizations to listen to their employees and give them what they need to be successful in a hybrid environment.

And we’re not just talking about tech. Organizations need to keep employees engaged to really understand the pulse that’s around the organization because, as we all know, the culture and tone of an organization can go ebb and flow. It’s critical for organizations to really listen to what employees are asking for and to be flexible so that they can react and adapt to changing times.

If they don’t, a ruptured internet cable may be the least of their worries.

October 8, 2021
Microservices Adoption and the Software Supply Chain
Software development is undergoing industrialization, with more and more software rapidly assembled out of components and an emphasis on building automation around software validation and release processes.

Modern cloud-native software is no longer a monolithic application living in a single repo with the majority of its dependencies self-contained. It is integrated from third-party components provided by vendors, cloud providers and open source software (OSS) components, with as much as 90% of code coming from such dependencies. This allows developers to build applications much faster but makes maintenance much harder as it is no longer entirely under their control. If the third-party vendor makes a change to their API, developers are on the hook to update their applications before they break.

Each microservice interacts with hundreds of other microservices and each is built and released independently, making it hard to understand how to coordinate API changes across them. For example, if you want to make an API change to a microservice, what impact will this have on the consumers in the organization? Who will need to coordinate the change? When it was all within the same repository and released all at once, it was much easier to make such changes.

The Changing Nature of Technical Debt

What we continue to call technical debt is really the activities that are related to tending to and upgrading our software when third-party components are evolving or have common vulnerabilities and exposures (CVEs) and need to be upgraded.

These are tedious, repetitive tasks that usually fall to the most experienced engineers as they require technical expertise to do correctly. Such activities can paralyze engineering organizations and are a tremendous burden on engineers; that often leads to burnout. Up to 30% of engineering time is spent on technical debt. The perception that somehow developers were responsible for accruing this technical debt and are doing something wrong that prevents them from keeping up is hugely demoralizing and demotivating.

However, if we reframe technical debt as software supply chain management and stop blaming engineering for it, we can make maintenance more predictable and consistent. By taking steps like inventorying third-party components and determining how pervasive they are in the application (frameworks take more effort to maintain than a third-party API that you just call from one part of the application), an organization can arrive at a maintenance estimate.

These activities are highly repetitive across organizations, as everyone is integrating some subset of the same third-party components to create business value. This high level of repeatability points us to automation.

Specialized Developer Tools for Software Maintenance Don’t Exist

When developers write code, their hands often lag behind their thoughts. They think much faster than they are able to type the code. You often hear developers finishing a tedious task and complaining, “My fingers hurt.” When their IDE can autocomplete syntax, it helps reduce this lag to minutes.

When a major framework version is released, developers contemplate the codebase. Maybe they need to change this type to that, reorder some method arguments or change dependencies. The types of changes they need to make are enumerable and must be made almost immediately; again, the ability to implement them lags far behind their recognition of the problem. Often, this type of lag can represent months or years of work when dealing with a large codebase. Accumulate enough of this lag and the codebase can grind to a halt.

That lag kills the joy in development for most senior developers and leads to burnout. Upgrades/migrations amplify that lag by orders of magnitude.

However, there are few (if any) tools that focus on helping developers automate these remediations/upgrades/migrations. The IDE’s focus is rightly on helping developers write new code and maintain their own code, but code maintenance needs to be coordinated across multiple places across the codebase, within the same repository or across the repository boundaries. Often, developers have a pattern they want to change in their repository, but the IDE only suggests this as an improvement in their current location (one file). Other times, changes need to be coordinated across repository boundaries if developers want to change APIs and their consumers. That’s why this technology needs to exist outside of the IDE.

OpenRewrite

OpenRewrite is an open source project that offers semantic analysis and refactoring of code as standalone operations so that everyone can use and contribute over time; composing more and more refactoring operations that make whole framework migrations possible. It is integrated with build tooling and can be plugged into different workflows, from CI integrations to mass refactoring of multiple microservices/repositories. Similar to an IDE or other developer tools, OpenRewrite manipulates an abstract syntax tree (AST) representation of code, but this AST has special characteristics that allow it to transform this AST and generate code back to the standard text representation.

The OpenRewrite AST is produced by guiding the compiler through the first two phases of compilation to generate compiler type-attributed AST. This is then mapped to the OpenRewrite AST, preserving formatting and breaking cycles in the AST. The three unique characteristics of OpenRewrite AST are:
- Type aware, allowing 100% correct semantic code analysis and transformation.
- Style-preserving, so transformations produced are idiomatic within the projects they are applied to. So, the same transformation applied to multiple projects will potentially look different.
- Serializable, allowing to output the AST from the build and operate on it outside of the build en masse.
OpenRewrite ASTs is full of metadata about code. Visualizing the code itself as a highly connected tree of various vertices and edges, we can start to get an idea of the density of this representation. Syntactic elements alone are blue, type attribution is yellow and formatting information is red. If we were to just represent syntactic elements, the tree might look something like this.

Adding type attribution and formatting, the tree looks quite a bit denser.

A spanning tree of the data involved in this AST can be laid out in three-dimensional hyperbolic space and then projected onto the unit sphere similar to the way that two-dimensional space can be projected onto a disk—the famous Poincaré disk model. This three-dimensional approach was pioneered by Tamara Munzner at Stanford.

Every edge that isn’t part of the spanning tree is rendered as a lighter gray, and, as you can see, it is dense! In fact, to make this layout more tractable, we’ve omitted in this view an additional 600,000 vertices representing type information that make the syntax itself seem tiny.

OpenRewrite calls a single code search or transformation operation a recipe. It provides a number of building block recipes such as find method, change method, find transitive dependency, upgrade or exclude dependency. These recipes, in turn, can be composed into more complex recipes by grouping them together into a composite recipe. When the building blocks are not enough, a recipe can be written as a program in the same language as the code we want to transform, allowing us to encapsulate complex logic with the full expressiveness of the language already familiar to developers. We don’t need to learn a new DSL or programming language.

These building blocks abstract away many of the details to ensure that edits that we make to source code obey the original style of the project. This is possible for even complex changes like this automated migration from Spring Boot 1.x @ConditionalOnBean to Spring Boot 2.x AnyNestedCondition looks idiomatically consistent in the context of the project that it is inserted into:

It is important to note that OpenRewrite is not a replacement for an IDE. With software now composed of so many independent components, we need to start to remediate and manage it using new approaches that go beyond single lines of code to looking across entire code repositories.

Olga Kundzich, co-founder of Moderne, co-authored this article.
October 8, 2021
BMC Acquires StreamWeaver to Advance AIOps Strategy

BMC has acquired StreamWeaver, a provider of a platform that makes it simpler to capture streaming data in real-time, as part of an effort to extend the BMC Helix IT operations platform infused with artificial intelligence (AIOps). Financial terms of the transaction were not disclosed.

Margaret Lee, general manager and senior vice president for digital service and operations management for BMC, said StreamWeaver brings with it a wide range of integrations with multiple data sources that BMC will use to both train AI models and improve overall observability. While connecting to data sources is relatively straightforward, the challenge now is collecting that data in real-time in a way that is easily repeatable for IT operations teams, noted Lee.

In general, AIOps continues to gain traction as IT environments become increasingly complex. Most IT operations teams are not going to be able to manage highly distributed computing environments without augmenting their IT staff with some AI, said Lee.

Naturally, many rote tasks that were once performed manually by IT operations professionals will become increasingly automated. However, there will always be a need for IT personnel, noted Lee.

Arguably, there may come a day when IT professionals will not want to work for organizations that don’t have an AIOps platform in place simply because the tasks assigned to them would otherwise be considered archaic, especially if they involve a series of tedious tasks that not many IT professionals really want to perform.

At the same time, AIOps should reduce overall risk by making it easier for IT operations teams to proactively identify issues before they disrupt an application service. That’s especially critical as IT environments—increasingly with microservices-based applications constructed using containers—and serverless computing frameworks are being deployed with greater frequency alongside legacy monolithic applications.

BMC has been making a case for AIOps as far back as 2017. However, since then, a wave of startups and IT operations incumbents have either built an AIOps platform from the ground up or updated an existing platform with AI capabilities. It’s difficult to assess which vendors are now ahead in terms of the AIOps race, but one thing that is certain is that IT organizations now have plenty of options.

From a cultural perspective, however, not every organization is entirely comfortable with AIOps. While some IT professionals remain skeptical, others view AI as an existential threat to their existence. Most IT teams are also concerned about the potential impact any flawed AI model might have on IT operations—it’s one thing for a human to make a mistake, but an AI model making a mistake at scale could be catastrophic.

There are few IT organizations that are embracing AIOps across the board, but most do recognize that AIOps platforms, in one form or another, are inevitably here to stay. After all, machine learning algorithms never forget anything they learn. Nor do they ever take a day off or decide to leave when offered higher pay. The challenge is figuring out exactly where the interface between IT staff and algorithms is going to be—not just today, but also in the near future.

October 7, 2021
Verica Launches Database for Sharing IT Incident Reports

Verica, a provider of an incident management platform, announced this week it is making available a database to make incident reports publicly available via a single repository.

Courtney Nash, senior research analyst at Verica, said the Verica Open Incident Database (VOID) initiative is part of an effort to enable organizations to learn from outages and failures caused by software. Today, reports about such incidents tend to be difficult to find because they are scattered all across the internet, she added.

That lack of transparency makes it difficult to compare and contrast incidents in a way that surfaces meaningful insights, noted Nash.

Launch partners for the VOID include Indeed, NS1, Adaptive Capacity Labs, Security Scorecard and Auxon. Each will provide support to help fund the development, research and dissemination of knowledge generated by the project.

The VOID contains more than just standard company postmortems or status updates, added Nash. Verica is committed to applying metadata to better understand how individuals, companies, media and others reacted during an incident.

One early finding already derived from this initiative is that mean-time-to-resolve (MTTR) is proving to be an inaccurate metric due to the varied distribution of incident data. Other findings include the fact that only one-quarter of incident reports took the time to identify the root cause of an issue, while less than 1% took the time to study incident reports of near-misses.

In general, there’s now more riding on the performance and availability of IT environments than ever. Digital business transformation became all the rage in the wake of the economic downturn brought on by the COVD-19 pandemic. As the economy continues to recover, businesses now expect IT to respond and pivot to changing conditions no matter how sudden.

DevOps teams have, of course, learned to expect the unexpected by embracing modern incident management based on DevOps best practices that enable IT teams to adroitly respond to any outage or sudden degradation in application performance. The most important thing to remember about incident management is that practice makes perfect. While practice runs may never precisely mirror an actual crisis, organizations that have processes that are well-known by their IT incident teams and that practice ‘fire drills’ tend to respond both faster and better. After all, the more muscle memory there is to call on the more resilient the organization will be.

It’s not clear to what degree IT organizations will be willing to share incident reports. However, there is an opportunity for IT professionals to come together that is similar to how engineers from different airlines collaborated to make flying safer for the general public, said Nash. In fact, with so many processes now dependent on software, it’s amazing there have not been more catastrophic events due to software issues. It’s even arguable that it’s only a matter of time before such levels of collaboration are mandated.

In the meantime, the first step toward achieving that goal may simply be sharing a report with colleagues so they can learn from previous mistakes.

October 7, 2021