devopsy.kinsta.cloud

Category: Features

  • HashiCorp Previews Updates to Consul Service Mesh

    HashiCorp Previews Updates to Consul Service Mesh

    During an online HashiConf Global conference, HashiCorp today made available a beta of an update to its open source Consul service mesh for virtual machines and Kubernetes clusters that makes it possible for multiple administrators to share the same instance of a multitenant service mesh.

    Version 1.11 of Consul will also add support for a command line interface (CLI) for Kubernetes that will provide an alternative to Helm charts for installing Consul on those clusters.

    Raymond Austin, director of product marketing for HashiCorp, said it’s become apparent that there is a need for a higher level of abstraction for managing application programming interfaces (APIs) that is easy to deploy across both legacy IT environments based on virtual machines and cloud-native environments based on Kubernetes clusters.

    Of course, in many cases, those Kubernetes clusters are deployed on top of virtual machines. However, there are also now more instances of Kubernetes being deployed on bare metal platforms at, for example, the network edge.

    As IT environments become more complex, a service mesh makes it simpler to discover, access and secure APIs that are now rapidly proliferating across the enterprise. That transition requires a more dynamic approach to not only managing APIs but also the network services those APIs depend on, added Austin.

    HashiCorp is betting that many IT organizations will prefer to rely on a lightweight open source platform supported by a single vendor rather than relying on more complex open source service mesh offerings being developed under the auspices of a consortium.

    It’s still not quite clear who within an IT organization is assuming responsibility for deploying a service mesh. Most often, it’s a development team that makes the initial decision. However, as IT operations teams become more aware that service meshes represent a capability that needs to be horizontally made available across multiple applications, it may be only a matter of time before some organizations decide to standardize on a single service mesh to reduce operational overhead, noted Austin.

    Service meshes will also play a critical role in simplifying management of networking and security services at a higher level of abstraction within a larger DevOps workflow.

    It’s still early days as far as service mesh adoption in the enterprise is concerned, but there are already a large number of service mesh options. In fact, there may never be a single dominant service mesh platform. Interoperability standards still need to be ratified that enable organizations that have standardized on different service meshes to apply policies to microservices spanning hybrid cloud computing environments.

    Of course, not every IT organization requires a service mesh yet, either. Many IT teams can still get by using a traditional proxy server or an API gateway until the number of microservices-based applications deployed starts to achieve critical mass. One way or another, however, the more APIs there are to manage, the more inevitable it becomes that an IT team will deploy a service mesh.

  • Pulumi Adds Registry to Share Secure IaC Code

    Pulumi Adds Registry to Share Secure IaC Code

    Pulumi this week added a public registry to its portfolio that makes it easier to discover packages that have been created using its tools to manage infrastructure-as-code (IaC).

    Joe Duffy, Pulumi CEO, said the Pulumi Registry will make it simpler for DevOps teams to share and reuse IaC tools that have been vetted versus always requiring each development team to write new code each time they want to programmatically provision infrastructure. Pulumi Packages provide modern cloud reference architectures in the form of software development kits (SDKs), code samples and how-to guides.

    One of the primary reasons that security has become a major cloud issue is that developers often misconfigure infrastructure. The registry provides a way to reduce those configuration errors by making it easier for developers to reuse code written in TypeScript/JavaScript, Python, Go and .NET that they have stored in repositories such as npm.

    In addition to the major cloud service providers, there are also Pulumi Provider Packages for Auth0, CloudFlare, Confluent Cloud, Datadog, DigitalOcean, Docker, GitHub, Kong, MinIO, MongoDB Atlas, PagerDuty, Snowflake, Spot by NetApp and others.

    There are also Pulumi Component Packages for deploying container applications on instances of Kubernetes and other related platforms in addition to deploying applications on serverless computing frameworks.

    Duffy said in addition to providing a “golden image” of their configurations the registry reduces the overall level of friction that often exists between developers, IT operations teams and cybersecurity professionals.

    The latest Pulumi offering is part of an ongoing effort to democratize cloud computing in a way that is repeatable and reliable, noted Duffy. Pulumi, for example, just added support for a set of AWS Cloud Control application programming interfaces (APIs) that reduce the number of APIs that developers would otherwise have to master. The more APIs there are, the more likely it is a developer will make a mistake.

    IT teams should expect that other cloud service providers will similarly streamline the APIs they present developers as part of an effort to make their services both more accessible and secure. In the meantime, frameworks provided by Pulumi provide a means to invoke those lower-level APIs at a higher level of abstraction for both developers and internal IT operations teams alike.

    It’s not clear today how many developers routinely employ IaC tools, but as the number of workloads deployed in the cloud steadily increases, so too does the percentage of developers that employ these tools. While that shift represents a major boon in productivity for developers, it brings with it additional security risks that many organizations fail to appreciate until it’s too late.

    The only way to really solve that issue is for organizations to adopt DevSecOps best practices that require a level of collaboration between DevOps and security teams that is still relatively rare. However, as the tooling provided continues to improve, the technical challenges associated with achieving that goal continue to decline. The next big issue, as always, is bringing together all the diverse cultures within an IT organization to actually implement those tools and define a set of best practices that work for all concerned.

  • Learn a Bit About AI

    Learn a Bit About AI

    Every once in a while, a trend becomes broad enough that I feel the need to offer career advice. This year it will be simple: Know enough about AI/ML to be dangerous. We are past the point where not having at least a basic understanding is a good idea. Looking around the data center, AI is in use from the earliest stages of development through production network performance evaluation. Somewhere in your environment, you are using AI and ML. More to the point, somewhere in the places you may interview going forward, they are using AI and ML.

    I’m not suggesting you become an AI/ML expert—though you certainly could; I have a relative that is a data scientist and that’s all he does—design and tweak algorithms. He’s learning new things pretty much daily, so if you like that (as I do), it’s an option. But for those of us doing DevOps or SRE or even regular old operations, we’re only going to have to understand a bit.

    Understand what tools like Jupyter are, and what the most commonly used ML algorithms in your space are. Then, learn about how AI and ML help with what you are doing. Just as knowing more about the compiler/interpreter of your chosen language helps you solve more odd problems, AI is going to have that same issue. Think about how those folks who understand the guts of routing are the ones we want solving our routing problems. Again, same scenario.

    Moving forward, AIOps, NetOps, DevSecOps … All intend to increase the amount of work offloaded onto the AI in question. In a lot of cases that is great. We have enough to do; if you can paw through those logs and give me relevant summary info, go for it. But it doesn’t end there—some AI engines are making the calls and implementing changes. In both network and application security, a fair number of vendors will tell you there is no time to wait for humans to stop an attack, and they encourage their customers to turn on automated responses. That’s the first of many applications for AI/ML. We will see an increasing number of AI/MLOps tools being used for full-on automation. We all do difficult jobs with a complex—sometimes bewildering—number of inputs. But we use pattern recognition and experience to make determinations and do what’s best for our organization. Our software has reached the point where sometimes, it can do that for us on the easier tasks so we can focus on the more complex ones. When the AI/ML fails—and it will, because pattern recognition and historical trends are not always indicative of normal usage—knowing what is going on under the covers can help you determine if it did the right thing in the wrong way, or simply did the wrong thing. And it can help you to quickly recover. I can’t speak for every organization, but I’ve never seen one that lamented, “Sure wish that outage had lasted another couple of hours!”

    And keep rocking it. Somewhere in the distant future, an  AI may be able to do your job as well as you. But that isn’t today. Even if it can handle one bit, it certainly isn’t rocking the entire breadth of your responsibilities like you are. Thank you, from all the users that don’t realize you’re keeping them live.

  • Splunk Expands Scope of Observability While Lowering Costs

    Splunk Expands Scope of Observability While Lowering Costs

    During its online .conf21 conference Splunk Inc. today revealed a bevy of updates that advance observability while at the same time reducing the total cost of deploying its platforms.

    As part of that effort, Splunk also announced it will now make workload-based pricing available to all customers.
    Previously, a select number of small customers had access to workload-based pricing while other customers licensed Splunk platforms based on how much data was loaded into the platform. That approach, however, tends to discourage IT teams from loading as much of their data as possible.

    CEO Doug Merritt said workload pricing was created to enable a value-oriented pricing model that customers could employ instead of a data-centric model that can discourage IT teams from analyzing as much data as possible to improve IT operations.

    To further lower total costs, the company is also previewing a Flex Index option for customers that choose workload-based pricing to enable longer retention for lower-value data at a lower cost. Splunk is also expanding its cloud storage to include a preview of SmartStore, which allows larger volumes of data to be stored more cost-effectively on Microsoft Azure Cloud initially with support for Amazon Web Services (AWS) and Google Cloud Platform (GCP) forthcoming.

    Other enhancements to the portfolio include a preview of a data manager, a revamped onboarding tool for data stored in Amazon Web Services (AWS) and Microsoft Office 365, and Ingest Actions, which enables IT teams to act on data in motion in addition to data at rest stored in Splunk or the S3 cloud service from AWS.

    The company is also now making it easier to find and explore data via a federated search tool and a Splunk Dashboard Studio offering, available via Splunk Mobile, that makes it easier to create dashboards.

    At the same time, the company is also making available Splunk Operator for Kubernetes to make it easier to install its platforms in cloud-native environments. There is also now Splunkbase, a curated catalog that makes it simpler to find or develop a Splunk application.

    In terms of observability, the company is providing a preview of AlwaysOn Profiling for its application performance management (APM) platform, dubbed Splunk APM. AlwaysOn Profiling extends existing tracing capabilities to provide continuous code-level visibility and analysis.

    In addition, there is now a preview of an enhanced version of Database Visibility for Splunk APM, which automatically uncovers slow queries that are causing transaction latency without having to instrument a database.

    The company also announced that its real user monitoring (RUM) for mobile applications, which makes use of open source, now is generally available alongside the Splunk Observability Cloud for Mobile platform. The company also revealed there are previews of integration between Splunk Log Observer and Splunk Enterprise and Splunk Infrastructure Monitoring AutoDetect, a tool that automatically discovers infrastructure anomalies. There are also now additional Content Packs for managing Microsoft365, third-party APM tools and synthetic monitoring.

    Finally, Splunk has expanded its security portfolio to include Splunk SOAR App Editor, which provides a way to edit, test and create security orchestration, automation and remediation (SOAR) applications. There are now more than 350 SOAR apps now available on Splunkbase. The company also launched Splunk Intelligence Management based on a platform it gained with the acquisition of TruSTAR earlier this year and that it is making available security services via a team of experts dubbed SURGe.

    It’s not clear to what degree these additions will enable Splunk to fend off what has become a larger number of rival providers of observability platforms. However, as one of the most widely used IT operations management platforms in the enterprise, it’s clear Splunk enjoys an incumbent leader status it is determined to maintain.

  • Splunk Donates eBPF Telemetry Data Collector to CNCF

    Splunk Donates eBPF Telemetry Data Collector to CNCF

    Splunk Inc. announced during the KubeCon + CloudNativeCon North America conference that it donated a data collector to the OpenTelemetry project run by the Cloud Native Computing Foundation (CNCF).

    Splunk’s data collector runs as a sandbox-level program at the kernel of a Linux operating system and takes advantage of extended Berkeley Packet Filter (eBPF) technology to make it simpler to collect networking telemetry.

    The OpenTelemetry initiative itself spans a range of open source tools, application programming interfaces (APIs) and software development kits (SDKs) that are used to instrument applications. Previously, Splunk contributed the SignalFx Smart Agent and Smart Gateway to the OpenTelemetry project along with more than 64,000 code contributions.

    Morgan McLean, a director of product management for Splunk, said that while OpenTelemetry is officially in beta, some elements of the project are more mature than others. For example, tools for capturing metrics and traces are already being employed, while another set of tools for capturing log data will be ready sometime next year, he said.

    The data collector contributed by Splunk is among the first that operate at the kernel level to collect data, added McLean. That approach makes it possible for some types of data to be captured by default rather than requiring developers to always instrument their applications using agents to enable an observability platform. It will also contribute toward the eventual convergence of network operations and DevOps processes.

    Going forward, DevOps teams should expect to employ a mix of open source data collectors that operate at both the eBPF and application level, said McLean. The eventual goal is to instrument every application by default by making it much simpler to collect data. Today, most DevOps teams are employing a mix of proprietary and open source agent software to collect data that then must be deployed and then integrated with all the applications they build. Given the cost and level of effort required, the percentage of applications that are actually instrumented is, not surprisingly, fairly low.

    However, as agent software becomes more readily accessible, the percentage of applications that are instrumented should increase considerably in the years ahead. For DevOps teams that depend on that data to optimize application performance, that capability should prove to be a boon for improving overall observability in IT environments.

    Observability, in one form or another, has always been a core tenet of any DevOps best practice. Initially, DevOps teams focused on continuous monitoring as the most effective way to proactively manage application environments. However, it can still take days, sometimes weeks, to discover the root cause of an issue.

    Monitoring focuses on predefined metrics to identify when a specific platform or application is performing within expectations. The metrics tracked generally focus on, for example, resource utilization. Observability platforms combine metrics, logs and traces—a specialized form of logging—to instrument applications in a way that makes it simpler to troubleshoot issues without relying on a limited set of predefined metrics focused on a specific process or function.

    Those observability platforms then make it possible to correlate events so that it is easier to identify anomalous behavior indicative of an issue’s root cause. Armed with these insights, it becomes a lot simpler for IT teams to resolve issues faster.

    It’s unclear when OpenTelemetry tools will become more widely employed, but as more tools for collecting data become available, the impact on DevOps will be nothing short of profound.

  • Welcome to the New Field of Software Supply Chain Management

    Welcome to the New Field of Software Supply Chain Management

    Supply chain management is the newest ‘shiny object’ in both the DevOps and DevSecOps communities. But what does it mean in relation to software development?

    Historically, supply chain management is a commerce term that refers to tracking the logistics of goods and services moving between producer and consumer. This would include the storage and flow of raw materials with a focus on the channels that support the process.

    When we look at this concept from the perspective of software development, we will soon see that it is a new form of software configuration management. Yes, the new SCM is … SCM. The core of supply chain management is the process of controlling changes in software. The new SCM expands those old practices to include all ‘raw materials’ which we manage as part of our software build and package steps. For instance, when companies purchase and download open source libraries used in their in-house software. Supply chain management includes the interrogation of these ‘raw materials’ as well as all of the internal code and libraries. We will soon be hearing more about software bill of materials (SBOM) reports and difference reports, as they are critical for understanding the objects that flow through the software we create.

    The software supply chain or software configuration management practice hasn’t been talked about much until recently. This area of expertise has never been ‘sexy.’ That is, until the SolarWinds supply chain hack. The SolarWinds hack was a hard lesson to learn, but it taught us to take a closer look at how we consume our raw materials and to be more serious about how we build and package our software.

    According to FireEye, the firm that discovered the backdoor supply chain hack:

    “SolarWinds.Orion.Core.BusinessLayer.dll is a SolarWinds digitally-signed component of the Orion software framework that contains a backdoor that communicates via HTTP to third-party servers. We are tracking the Trojanized version of this SolarWinds Orion plug-in as SUNBURST.”

    It was obvious, something went very wrong with the creation of that .dll. I don’t pretend to fully understand how this breach occurred. But I do understand the build process and have been talking about the potential for these types of breaches for over 25 years.

    There have always been big security gaps and vulnerabilities in our most basic software development practice—the compile/link process. Most build processes are imperatively defined using an Ant, Make or Maven script. Each time the build runs, it rebuilds all the binaries, even if the corresponding source code has not changed.

    This is a problem. Instead, we need a build that is incremental. Incremental builds are difficult to script because they need more advanced logic. Most developers are not given adequate time to create a better build process, one that only rebuilds the changes. This means it is next to impossible to audit why a binary was recompiled because they are always recompiled. If we only recompiled the binaries that had corresponding code updates, we could more carefully audit the code and make sure that only approved coding updates were included.

    This may not have caused the SolarWinds attack, but it could have. Once hackers have gotten past the firewall they can find the build directory, read through the build scripts and identify the perfect piece of code to update with their bad function. The next time the build runs, a ‘clean all’ recompiles everything and, like dark magic, a SUNBURST is created.

    In our not-too-distant past, the developer community had tools that controlled software builds beyond what an imperative script could do. OpenMake Software, for example, provided a build automation solution called Meister that ensured that incremental builds were easy. As part of the build, it generated SBOM reports and differences reports allowing teams to easily audit and compare the actual source changes to confirm that only approved code updates were included. Another example was Rational Software’s ClearCase which used a program called ClearMake. ClearMake used a process called ‘winkin’ to reuse ‘derived’ objects that had not changed. This process created an incremental build that could be easily audited.

    I realize that I am simplifying the overall problem, but it is a good example of the basic supply chain best practices and how potentially vulnerable our processes are.

    The time to carefully think about how we manage our raw materials and overall supply chain is now. I’m guessing that most monolithic practices will not change much; it is too costly and impactful to rewrite hundreds of CI pipelines. But that does not mean we can’t solve the supply chain puzzle as we shift into cloud-native development in a microservice architecture.

    In a microservices architecture, we need to begin thinking about what the supply chain looks like. How do we track raw materials that are in the form of hundreds of small ‘bounded context’ functions? The good news is there is an emerging microservice catalog market that can become part of the supply chain solution. A microservices catalog can track versions of services to versions of their consuming applications. Your new SBOM report will have two levels: First, an SBOM for the microservice. Second, aggregated data at an application level. Remember, in this new architecture a full build is not done.

    It’s important to understand that microservices are naturally incremental and are independently built, packaged and deployed. Managing these small services opens the door to more closely auditing their source code before they are released.

    As we shift from monolithic development to microservices, we have an opportunity to incorporate supply chain and configuration management best practices. This includes being able to carefully audit the code before a release is done. And remember, we must do this as fast as possible. It sounds like a big task, but it’s well worth the effort. Collectively, we will find the answers as we explore new ways to manage our DevOps pipeline and incorporate configuration management and DevSecOps principles.

  • Energy Efficiency and the Data Center

    Energy Efficiency and the Data Center

    The IT industry has historically been a leader in the ongoing race to reduce CO2 emissions. Reducing emissions is good for the environment and for the corporate bottom line. The data center is well-known as a voracious data consumer. But the data center consumption findings keep pace with the trend. Even in the face of remarkable growth in data center capacity and performance, their energy consumption has increased only a few percentage points since the 2010s. This almost-flat rate of energy consumption is primarily the result of the shift to cloud computing as well as the shift to solar and wind energy, where it makes sense.

    Investments in renewable energy sources have helped tamp down criticism over the overall IT carbon footprint. Knowing that it takes a tremendous amount of energy to power a data center, owners and operators are always looking for ways to improve efficiency and reduce carbon expenditure. Today, there are newer and better ways for organizations to deliver the performance to properly power modern applications while keeping in mind the energy efficiency and carbon reduction the world requires.

    Minimizing Energy Use and CO2 Production

    How did we get here? Decades of semiconductor process miniaturization in concurrence with living out Moore’s Law has culminated in ongoing improvements in performance, storage capacity and device density. Today’s mobile phones are a great example of that ongoing progress. A secondary factor in the mobile phone revolution is a processor architecture optimized for efficiency and flexibility. This ARM architecture has been used to power almost all devices since the first iPhone. Over the years, ARM has also become a realistic platform for servers; often a better choice than an x86 system.

    Energy efficiency is one area where ARM systems win out over conventional server architecture. But it was not until recently that ARM processors could deliver the required performance for the data center. Today’s systems, based on Armv8 designs, have brought into play x86-class capabilities in a more energy-efficient way. The advent of the ARM-based AWS Graviton processors, which by some estimates have already captured 20% of the instance share on AWS, demonstrates that ARM can run production workloads while delivering what AWS calls “up to 40% better price performance over comparable current-generation x86-based instances for a wide variety of workloads.”

    To illustrate the impact on carbon production when a data center shifts from x86-based architecture to ARM servers, an analysis of the top elements contributing to data center energy use was undertaken. The energy savings was calculated and translated to reduced CO2 emissions based on the fuel mix used to power the electric grid in various locations. The results prove that an ARM-powered data center reduces CO2 production by 74%, equivalent to almost half a million barrels of oil.

    Power Consumption Reduced by Efficient Servers

    The premise is simple—innovative server design significantly reduces data center energy use. In this model, a server’s power draw and physical size are key factors. The energy needed for other data center subsystems was extrapolated by using ratios derived from experience and validated by estimates from other data center operators.

    Data center energy use can be divided into five categories of equipment and infrastructure: Servers, network communications, power supplies, conversion and UPS, cooling systems and storage. Servers and cooling comprise about 75% of the total.
    The calculations were based on the energy used by a medium-sized data center with 750 racks of conventional 1U servers (42 servers/rack; around 31,500 in total). Assumptions used included servers comprising two-thirds of the floor space and that the equipment space accounts for 65% of the total, translating to a 62,000 square-foot data center. For comparison, facilities operated by the major cloud vendors can be a million square feet or more. Knowing server power consumption and the ratios for the equipment categories, algebra can be used to calculate the energy consumption for each.

    The data center in this hypothetical example has, as we said, 31,500 servers. Using the 912.5W power draw of a Dell R640 as typical of 1U x86 servers, the total server usage in the ARM data center would be 251,795 MWh/year. Using this formula, the amount of energy saved by using ARM servers can be calculated. Each server requires about one-quarter the electricity of a standard server. In the formula, the assumption is that the ARM server installation would require half the equipment racks (375 instead of 750) and 52% of the power-per-system, comprising a data center using only 26% as much electricity (152,409 MWh/year) as one based on x86 systems.

    The math shows that switching one medium-sized data center to ARM servers can produce annual savings equivalent to 45,459 fewer cars on the road or 486,749 fewer barrels of oil used.

    The value in CO2 reduction is widely understood for preserving the environment. It is also advantageous from an economic standpoint. In fact, there are many technical and economic reasons to adopt energy-efficient ARM-powered servers in the next-generation cloud data center.

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021

  • CNCF Adopts Meshery to Advance Service Mesh Management

    CNCF Adopts Meshery to Advance Service Mesh Management

    The Cloud Native Computing Foundation (CNCF) announced this week during the ServiceMeshCon/KubeCon + CloudNativeCon conference that Meshery, a service mesh management plane created by Layer5, has become a sandbox-level project.

    In addition, Layer5 has also donated Service Mesh Performance (SMP), a set of tools for measuring the efficiency of a service mesh, to the CNCF. SMP provides an open source framework to define standardized benchmarking practices, performance test configurations and measurements as part of an effort to create a MeshMark index for rating a service mesh. A set of service mesh performance methodologies will also be published by the IEEE later this month, developed in collaboration with engineers from Layer5, Intel, Red Hat and HashiCorp.

    Layer5 CEO Lee Calcote said the goal is to make it simpler for IT teams to determine which service mesh to employ based on their specific use case requirements.

    Several service mesh platforms are starting to gain traction as a means to both manage application programming interfaces (APIs) and provide a layer of abstraction between lower-level networking and security APIs that makes those services more programmatically accessible to developers.

    Meshery itself is an implementation of SMP and the conformance tool created for the Service Mesh Interface (SMI), a separate CNCF project to promote interoperability between service meshes. Meshery is based on a set of microservices that are designed to make the platform easily extensible.

    That approach makes it possible to manage multiple instances of service meshes that might be deployed on virtual machines as well as Kubernetes clusters using a range of management tools that are part of the Meshery project.

    Meshery supports AWS App Mesh, Citrix Service Mesh, HashiCorp Consul, Istio, Kuma, Linkerd, Network Service Mesh, NGINX Service Mesh, Open Service Mesh and Traefik Mesh. Meshery also provides IT teams with more than 60+ best practices in the form of templates that can be used to deploy a service mesh.

    It’s still early days as far as adoption of service mesh in the enterprise is concerned. Most IT teams still rely on either proxy software or a gateway to manage APIs. However, as the number of APIs that need to be managed steadily increases, it won’t be long before many IT teams start to at least evaluate a service mesh.

    In the longer term, the impact those service meshes are likely to have on the management of IT could be profound. It is becoming apparent that a service mesh also provides a higher level of abstraction for invoking networking and security services. In fact, the service mesh may wind up playing a key role in driving the convergence of network and security operations with DevOps best practices.

    In the meantime, IT teams will need to decide to what degree they may want to standardize on a single service mesh versus letting individual teams implement their own preferred platform that could one day be centrally managed via Mashery. One way or another, the way APIs are managed and invoked will soon fundamentally change forever.

  • Top 5 Must-Haves for IaC Automation Tools

    Top 5 Must-Haves for IaC Automation Tools

    These days, there are a lot of different DevOps tools to accomplish a lot of different jobs. Almost daily another startup comes out with a new and innovative product or a newer (maybe even better) version of existing tools. One of the biggest innovations has been infrastructure-as-code (IaC). Giving infrastructure admins and developers alike the ability to create, manipulate and destroy infrastructure using code files changes the way we administer our environments. Though these tools are revolutionary and can solve many problems, sometimes they come with a whole new set of issues—as can any type of tooling, really. 

    Generally, IaC tools are run locally at first. When an infrastructure admin or developer is just starting out with it, they store the files locally; maybe in a repository, then clone the files to execute locally. This is perfectly acceptable and a very common practice. The issues arise when it comes to scaling this practice to a team; for example, with regard to visibility and governance. If everyone is running their deployments locally, the team doesn’t have clear visibility into who is deploying what, when and with what variables. This is where IaC automation tools come into play.

    There are many different types of tools to solve these problems. Some are better than others. CI/CD pipeline tools, custom-built automation platforms, purpose-built IaC automation platforms—each has their pros and cons. Getting into all of that can be overwhelming and we could go on for hours about each option. But, we’re here to talk a bit more generically today. So, we’ll save all of that for another time. For now, let’s take a step back and talk about the top five must-haves for any infrastructure-as-code automation platform you’re considering. Because each company and use case is different, we’ll talk about these considerations in no particular order. 

    Role-Based Access Control (RBAC)

    This first factor is pretty important. I know I said I wasn’t discussing these in any real order, but this one should really be close to the top of the list for anyone looking into IaC automation. We talked about how automating IaC can cause visibility issues when you scale across a team. You may want to know who deployed what, when, where and why, for example. Usually, the reason you want that kind of visibility is that you would like to control all of these aspects. That way you can help limit waste, budget problems, security concerns and the like. You may want to provide self-service deployment access, but only within certain parameters; maybe only to certain people or teams. Having granular RBAC can help you design a security policy that makes the most sense for your organization and its needs. Some tools have this baked in, some take a bit of work to make it work. The point is that you should have some form of control over your deployment process. 

    Self-Hosted Agents/Runners

    Following the security thread: A lot of these IaC automation tools are SaaS platforms. SaaS can be very secure. But sometimes, for compliance or regulatory reasons, you may want to retain a bit more control over your deployments. This is where self-hosted agents or runners come in. This type of design allows you to keep your “secrets”—such as your cloud credentials and other sensitive variable information—secure in your own way. You can use whatever secrets management solution you want, be it something from your cloud provider or another third-party solution. It also allows you to keep your code secure. All of these tools, when deploying your code, have to get a copy of that code to deploy. So, they’ll do a ‘Git clone’ or some kind of file copy process to get your files from where they are stored, to execute them. If this is a SaaS solution, then they will essentially have access to your code. Again, this may not be a huge issue for some, but it very much is for others. If you don’t want to give a third party this level of access to your code and secrets, a self-hosted agent or runner can help keep it all secure and under your control. 

    Plan on Pull Request

    This feature is a big one when it comes to workflow. Some may not even know how important plan on pull request (PR) is until they realize how easy it can make their life. This feature may be called something different depending on the platform. Plan on PR, PR plan, speculative plan, etc. Essentially, this means that whenever you open a pull request against a branch, your automation platform will complete a deployment just up to the “plan” phase so that you can see exactly what this code change will do. For example, if you accidentally add an extra 0 to a variable or code file, you may end up with 100 instances instead of 10. Ideally, you’d like to know what was going to happen with a change before it is deployed, giving you the option to quickly fix a potential mistake before it actually goes through. In DevOps, you always want to ‘fail faster’ and close the loop so that big mistakes are caught before they happen. That is why this feature is so important. And if the automation platform you choose is doing this correctly, when it runs this plan phase it’ll automatically update the pull request directly with the possible changes as a comment. This way, your developers can see what is going to happen before they try to merge the deployment back to the main branch for deployment to production. And if they take it a step further by adding service checks to the pull request, you can even configure it so that if the deployment plan fails for any reason, developers are actually blocked entirely from merging the PR back until it is resolved. All of these features help to enable the ability to work in a GitOps methodology. Don’t get me started on the wildl misuse of the term GitOps—it is very specific. While one of the central pillars of GitOps is using your repository as your “single source of truth” for what should be deployed, it’s not just using Git to store your Terraform files; it is an infrastructure administration via pull request methodology. If you want more information about GitOps, you can read more here. 

    Continuous Deployment

    This feature is essential for most folks. But, depending on the workflow, it can be less important. Even if it isn’t a must-have now, it may be down the road when you evolve and grow into new workflows. Continuous deployment is the ability for your automation platform to have a webhook or other connection to your source code repository so that it can automatically trigger a deployment process whenever you commit or push to the repository. This feature goes hand-in-hand with the plan pull request feature we just spoke about. To make that feature function properly, a CD feature would be needed, as well. Now, I want to make sure I clarify. Just because you have continuous deployment enabled doesn’t mean it is just going to push to production in the middle of the day every time a commit or push is done. You should still be given the ability to pause the deployment process after the plan phase. This way you can validate that the plan looks acceptable before you approve the deployment. For some use cases, like developer sandboxes, you may “auto-approve” the deployments because you don’t care about having to get validation first. It all depends on your workflow and processes. A good platform will give you the ability to control all of this. 

    Shift Left Extensibility

    This one is a bit of a wild card. It can mean a lot of different things depending on the situation. Essentially, what I am talking about here is the ability to integrate—or at least be interoperable—with a multitude of other tools during the deployment process. There is a concept called continuous verification which means to shift some tools and processes left into the deployment process so that you can stay on top of problems before they start. In the PR plan feature section earlier, we talked about catching issues before deploying. This isn’t just for resource configuration issues. What about security issues? Compliance issues? Performance issues? Or even budget issues? If your IaC automation tool gives you the ability to integrate the processes and tools you are using to validate these things into the deployment process, they no longer have to be an afterthought. You never have to worry about going over budget if you are constantly validating the budget each time you deploy. You don’t have to worry about a security issue if you’re validating security during every deployment. This type of functionality can look and be named very differently between tools. But as long as you have the ability to get all of these other groups (security, finance, etc.) involved sooner, everyone will be better off. Like we talked about before, in DevOps, it’s always better to fail faster. And integrating these checks into your deployments can help you achieve this. 

    Conclusion

    We’ve made it through all five! We’ve talked about some basic security features. We talked about some automation workflow must-haves to make your life easier. And we even scratched the surface on some next-step-type things to look into such as continuous verification. That was a lot—hopefully this has been helpful for anyone looking into IaC automation platforms and you now have an idea what to look for. Some features may be named differently, but the ideas behind them are the same. If your platform can’t do these, it may not be the right one for you. Or, maybe it is and that feature just isn’t that important to you. As I mentioned, your mileage may vary. Every organization is different.

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021

  • Rethinking Your Approach to Ops Governance

    Rethinking Your Approach to Ops Governance

    The “You build it, you run it” approach has changed the mindset of modern organizations that are increasingly adopting DevOps practices. No longer completely beholden to the requirements of operations, engineering teams are continually acquiring more autonomy to deliver rapid innovation. At the same time, more organizations are adopting platform teams to accelerate development, empowering teams with easier access to services without requiring them to understand the underlying complexity. While this approach is helping teams build, scale and ship software faster, it’s also resurfacing previous areas of tension around operational governance. 

    Organizations are generally choosing between two governance approaches: Mandated or paved path. Each has a different perspective on the appropriate level of autonomy to grant development teams and where governance should sit within the development life cycle. But there’s a third approach—the “trust and verify” approach—that can provide both high levels of autonomy for development teams and the sophisticated governance that operations teams need to ensure compliance and security.

    What is a Platform Team and Why are Organizations Adopting the Model?

    A platform team provides a foundation of self-service APIs, tools, services, workflows and support that accelerates the delivery of digital services and products. Platform teams operate similarly to any R&D team; employing customer support, operations, development, SRE, product managers and other personnel. They use research, customer feedback and usage data to iteratively improve their services.

    Rather than each development team reinventing the wheel, the platform team builds shared services that shield developers from the underlying complexity of infrastructure and enable self-serve access to services. As engineering resources become even more in-demand and scarce, organizations are using platform teams to optimize resources and share outputs.

    Two Platform Governance Approaches: Mandated Vs. Paved Path

    Organizations today typically are using one of two approaches to platform governance: Mandated or paved path.

    A mandated approach solves the governance problem by mandating that all engineering teams use the platform. However, this approach puts the onus on the platform team to keep up with the changing demands of engineering. Organizations must invest in a platform team that can keep up with engineering requirements so they still feel like they can innovate and use the best tools.

    The second approach is a paved path, popularized by Jeff Lawson of Twilio. This allows for engineers to go outside the platform if they are willing to take on the SLAs the platform provides and are responsible for all the costs of maintaining their own alternatives. Lawson said in this McKinsey interview, “What you do is you create the incentive structure for teams to take the paved path. But if they really have to go a different route, you make it possible for them to do that.”  

    While the paved path approach offers more autonomy for engineering teams, it requires teams that veer off the path to maintain the integrity of their software. Without guardrails, operations must put full trust in teams to keep their services secure and compliant. It can often be difficult for operations teams to maintain visibility into the services and infrastructure that fall outside the core platform.

    So, given that a mandated approach restricts autonomy and a paved path can make compliance more difficult, what is the middle ground?

    Bridging the Gap Between Mandated and Paved Path

    First, it’s important to understand that governance is not synonymous with command and control. Instead, operations teams should think of governance as the practice of maintaining visibility and providing best practices. Rather than a mandated platform approach, organizations can choose an approach that offers more autonomy to engineering teams (similar to paved path) while also maintaining operational governance. This is a trust and verify approach. 

    Using the trust and verify approach, organizations should focus their efforts on technology, not people, to maintain compliance. The goal would be to have a platform that could easily connect to the entire toolchain any product team uses for complete visibility and context. In this way, operations can gain governance without adding friction for product teams. This requires a top-down approach within organizations: I&O leaders investing in engineering resources or an off-the-shelf platform that can enable this technology. Both engineering and operations can flourish in this modern environment. Approaching governance from a trust and verify perspective will expand access to innovation while ensuring compliance and security standards. 

    Governance Should be Over Technology, not People

    Creating a bridge to a trust and verify approach means placing the responsibility for governance on technology rather than people to maintain detailed audit trails of human and machine actions tied to identity and reduce the potential for human error through automation. This approach provides autonomy for engineering teams while also ensuring governance through visibility and process controls embedded into workflows and tooling. 

    Shellie Miller, senior manager, technology governance and compliance at Hulu, believes technology can support DevOps practices. “To have a successful DevOps organization, we don’t need to have command and control or have this clear line of separation between dev and operations. It’s putting the responsibility on the infrastructure, the platform and the tooling and the policies and the processes that you have to enable those self-serve capabilities,” Miller said. 

    Mitigating Human Error Through Automation

    This investment in technology should focus on automation and process controls that fit within teams’ current processes. “What auditors are really trying to do is mitigate human error,” said Miller. “Teams should really focus on automating controls, building those security practices and those best practices into the platform. That way, dev teams are just subscribing to that offering.” 

    Changes to applications, systems, data and code—one of the biggest culprits with regard to risk—is where automation and process controls embedded into platform offerings can make a huge impact. “From an engineering perspective, constant change is needed to support rapidly changing consumer demands,” Miller added. “However, from an audit perspective, these activities increase risk of material impact to viewer experience, revenue and/or financial reporting.”

    Investment in Governance Comes From the Top

    While technology holds the key to better governance and compliance standards, there’s an investment that must be made, whether that’s investing engineering resources in building automation or investing in an off-the-shelf platform that can enable automation.

    “Leadership and organizational backing is absolutely key,” said Miller. “You can’t get to compliance or governance or any kind of operational maturity, for that matter, when you’re doing a bottom-up approach.”

    Building a compliance and security program around technology and not people means having the proper skills and mindsets. A bottom-up approach could create friction, especially due to a lack of education and understanding of the why and how. A top-down approach ensures there are resources to both implement the technology and to foster a culture of understanding and education. 

    The most common question I get asked is, ‘Why can’t we trust our people?’” Miller said. “Compliance requirements don’t generally start with the thought that our people are being malicious. We should run an organization of trust, and through automated monitoring and governance, we can verify and govern without being intrusive and disruptive. Lack of training and the demand on teams is what drives the perception that there’s no trust. Security and compliance is intended to mitigate risk and prevent threats to the systems that enable the business.”

    Building on a “Trust and Verify” Platform Approach

    When it comes to governance, technology is our greatest asset. Of course, this investment can be substantial for many organizations that don’t have the engineering resources to build in-house. Today, new open source or off-the-shelf platforms enable operations teams to build self-service capabilities that provide guardrails against human error, an audit trail of human and machine actions and visibility across disparate tools and services. 

    No-code solutions that also provide a developer platform for deeper customization enable teams to build out these core requirements of governance without consuming considerable engineering resources. The door to a future where operations teams can provide more autonomy to development teams while maintaining governance is now open. Every organization has the ability to attain a “trust and verify” approach and meet the needs of both dev and ops.

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021