devopsy.kinsta.cloud

Tag: IaC

  • Fugue Marries Compliance-as-Code Tool to AWS Well-Architected Framework

    Fugue Marries Compliance-as-Code Tool to AWS Well-Architected Framework

    Fugue has infused the best practices defined by Amazon Web Services (AWS) within its software-as-a-service (SaaS) offering for provisioning infrastructure using its infrastructure-as-code (IaC) platform.

    Fugue CEO Josh Stella said that IT teams can now evaluate templates for provisioning AWS infrastructure that were created using AWS CloudFormation or Terraform tools to ensure they comply with the AWS Well-Architected Framework.

    The AWS Well-Architected Framework spans five pillars that AWS has defined for designing and operating reliable, secure, efficient and cost-effective systems in the cloud. AWS has defined a set of best practices for each pillar and rewards organizations for following those by making available additional cloud credits. The goal is to improve overall cloud security by providing guidance that ultimately reduces the number of misconfigurations so that cybercriminals can’t discover and exploit them.

    The Fugue platform complements that effort because it is based on Open Policy Agent (OPA), a general-purpose engine for managing compliance-as-code that is being advanced under the auspices of the Cloud Native Computing Foundation (CNCF). Fugue extended OPA by creating Regula, an open source tool that evaluates IaC files for potential security and compliance violations. Fugue then created a SaaS platform that makes extensive use of the AWS Lambda serverless computing platform to make it easier to enforce compliance policies.

    Those policies include turnkey coverage for specifications such as SOC 2, NIST 800-53, GDPR, PCI, HIPAA, ISO 27001, CSA CCM, CIS Controls, CIS Docker and CIS Foundations Benchmarks for AWS, Microsoft Azure, Google Cloud and Kubernetes.

    Stella said Fugue has now incorporated the technical aspects of the AWS Well-Architected Framework. However, it’s still up to each IT team to navigate the cultural aspects of the shared responsibility model of compliance and security that cloud service providers like AWS follow.

    Unfortunately, developers that employ IaC tools to provision cloud infrastructure often make some faulty assumptions about the level of security maintained by the cloud service provider. The result is often a raft of security and compliance issues that arise mainly because most developers don’t have a lot of domain knowledge expertise in either area.

    It’s not clear how many organizations have adopted the AWS Well-Architected Framework, but as various tools make it easier to implement these best practices defined by AWS the more secure and efficient cloud computing environments should become. In fact, in many cases, those tools are being embedded within a larger set of DevSecOps best practices that are being implemented across multiple clouds.

    Stella said he doubted there will ever be any standardization when it comes to security and compliance frameworks that could be applied to multiple clouds; however, it’s clear that cloud service providers are at least making available frameworks that borrow concepts from one another.

    Regardless of the approach, the overall state of cloud security and compliance will continue to improve as more guardrails are automatically implemented. The issue then becomes not only finding a way to ensure that cloud infrastructure is secure when deployed and also determining how much of the infrastructure being relied on today is less secure than it should be.

  • Building on Terraform: Evolution, not Revolution

    Building on Terraform: Evolution, not Revolution

    Terraform is widely adopted among both enterprises and service providers who embrace infrastructure-as-code (IaC) as their model for infrastructure provisioning automation in DevOps environments. And while the technology has limitations, innovations seeking to unseat Terraform are highly fragmented and have seen tepid adoption. This points to a market that’s settling on a de-facto standard, so rather than trying to replace Terraform with the next great innovation, infrastructure operators should instead find ways to extend and augment Terraform to suit their unique needs. Let’s explore several of the better-known shortcomings of Terraform, some drivers for the rise in new alternatives to the technology, and what users need to consider when contemplating a different path.

    Without a doubt, Terraform is big. Across the big three public clouds, the number of Terraform provider installs tells the story: At the end of August, there were 457 million provider installs for AWS, and around 72 million each for Azure and GCP. Yes, compared to longer-in-the-tooth infrastructure configuration management tools like Ansible, Chef and Puppet, IaC as a category is still early. But, the trend is clear: IaC is becoming widely used at scale, and Terraform is the IaC platform most enterprises go with.

    It’s also no secret that there are shortcomings to Terraform. A quick search on Reddit and Hackernews unearths the litany of complaints, some more valid than others: 

    • It’s tricky creating a Kubernetes cluster and then adding a resource to it
    • Challenges passing values through a config or dumping a pure representation of an elevated config
    • Cluster deletion is a pain
    • An overall lack of visibility into what’s happening during the provisioning process
    • The need for glue code to drive everything is a drain on developer productivity
    • Lingering challenges with backward compatibility
    • Continuous update challenges and the need to refactor
    • Scaling across multiple workspaces and multiple cloud accounts
    • Rollbacks can be tricky, especially when dealing with immutable infrastructure

    Terraform’s parent company, HashiCorp, of course, steadily works to address these and other pain points. But the whole thing highlights an interesting question: Why is Terraform so popular?

    Terraform is the only IaC solution that meets the needs of developers who want an IaC tool that preserves state, fits well into their CI/CD pipelines, is intent-based and gives them the right level of granularity. Terraform, even with its challenges, is better than the alternatives in these regards.

    Here’s why: 

    Terraform was, from the start, designed for and targeted to developers. It maintains state and supports iteration, making IaC a powerful model for automating infrastructure. PaaS solutions, by contrast, are not robust enough for most developers who do not need or want all of the infrastructure decisions abstracted away. Most cloud management platforms suffer from the least common denominator problem, and even though the technology has improved in this regard, it’s a solution that developers generally abhor for its granularity; it’s geared more to the needs of IT management. Configuration management tools, as an alternative, are not intent-based, so flow becomes too complicated.

    It’s interesting to note that state (maintaining the mapping between the resources that you’ve represented in your definition files and the actual physical resources that are in the cloud provider) was added into Terraform, helping resolve the issue of infrastructure drift. Terraform also realized that developers don’t necessarily need a UI. These tangible examples of the developer-centric design ethos at HashiCorp point to why it is so widely adopted.

    So, what’s driving the interest in new approaches to infrastructure automation? The proliferation of domain-specific use cases and an ever-present need to reduce time-to-value are the leading reasons. Domain-specific automation options are becoming more dominant over OOB solutions because they take too long to get to automation; one tool does not fit all. Domain-specific automation addresses issues such as security automation, AIOps, DevSecOps, security compliance, integration with container scanners—not just generic orchestration. People are looking for value very fast, which can be found only with domain-specific automation—and therefore, fragmentation happens as different solutions emerge to address those cases.

    There are several interesting alternatives to Terraform. Terragrant is used along with Terraform to expand its capabilities to multiple workspaces. It allows you to use the same template across many environments. Pulumi is another alternative that has achieved a measure of adoption and visibility. It provides native code support which addresses the problem with YAML file development complexity. Another more ambitious undertaking is the Crossplane open source project. It takes a Kubernetes-centric approach to creating a universal, cross-cloud control plane. It looks good conceptually, but it’s a fundamentally different approach, and adoption is nascent.

    Turning to one of the Terraform alternatives isn’t necessarily a bad idea, but is fraught with landmines. In the end, this choice is less about technology and more about adoption and ecosystem momentum. Vendor and professional investments in tools, training, and skill sets surrounding Terraform is a formidable moat, and it cannot be easily replicated. Tools that would win over Terraform would have to win adoption, and achieving adoption means delivering all the enterprise-grade features and support that Terraform users currently enjoy. Chef and Puppet are examples of how this could happen. Both were successful technologies, but they were slow to react to the challenge presented by Terraform. Ultimately, these projects suffered from a legacy mindset and were overtaken by Terraform’s more focused, simple solution.

    Most of the issues with Terraform lie in managing it at scale. When you have many workspaces and templates, it becomes hard to manage; it’s also a challenge that templates cannot be generalized among environments, resulting in a lot of template redundancy. Also, Terraform still needs help handling Day 2 operations and visibility into drift detection. 

    When considering ways to extend Terraform to address a broader set of use cases, you should first consider how well things will interoperate across your environment. If you have multiple clouds, multiple service layers and multiple cloud vendors, this is probably the most important factor. Also, investigate any new technology’s ability to support the things about Terraform that are critical to your Day 2 operations. Things like secrets and state come to mind. You’ll want enough management and monitoring in the new solution without it being a new burden to carry. You should be able to use your existing Terraform files, and you should be able to support all the versions of Terraform that you run. Compliance with governance and audit are key, and the new solution should support Terraform for both your on-premises/air-gapped environments and your cloud environments. Ideally, anything you add to Terraform would be open source, eliminating new complications from cross-vendor lock-in.

     

    Terraform use case 

    Architecture for how 8451 evolved its Terraform deployment

    For all its shortcomings, Terraform remains the best option on the market for DevOps organizations, and there are ways to manage it with fewer headaches and without developers wasting so much time writing glue code or reinventing the wheel over and over again. One way this happens is by integrating Terraform with meta-orchestration tools that allow Terraform to become more flexible in managing environments and workspaces at scale. The open source Cloudify project is one example.

    Yevgeniy Brikman gave a talk two years ago at HashiConf that has aged well. At the time, DevOps was still in the Stone Age. Today, it’s probably in the Bronze Age. Those of us building IaC today are being asked to deploy infrastructure that looks like modern cities, but the tools we have to work with remain relatively primitive. What we’re building takes too long to provision and it looks bespoke and temperamental at best or kludgy and fragile at worst.

    It’s a story repeated over and over: Teams try to build their own, homegrown solution first as a wrapper on top of Terraform. Ultimately, they realize that it’s unsustainable as a lack of engineering resources becomes a bottleneck. We need solutions that augment—not replace—what they’ve already built and what works. That’s almost always Terraform. With Terraform, DevOps organizations get a proven IaC option that’s targeted to the needs of developers, with a substantial ecosystem and market uptake. When considering its shortcomings, it’s best to embrace an evolution strategy, and not start from ground zero.

  • Pulumi Adds Registry to Share Secure IaC Code

    Pulumi Adds Registry to Share Secure IaC Code

    Pulumi this week added a public registry to its portfolio that makes it easier to discover packages that have been created using its tools to manage infrastructure-as-code (IaC).

    Joe Duffy, Pulumi CEO, said the Pulumi Registry will make it simpler for DevOps teams to share and reuse IaC tools that have been vetted versus always requiring each development team to write new code each time they want to programmatically provision infrastructure. Pulumi Packages provide modern cloud reference architectures in the form of software development kits (SDKs), code samples and how-to guides.

    One of the primary reasons that security has become a major cloud issue is that developers often misconfigure infrastructure. The registry provides a way to reduce those configuration errors by making it easier for developers to reuse code written in TypeScript/JavaScript, Python, Go and .NET that they have stored in repositories such as npm.

    In addition to the major cloud service providers, there are also Pulumi Provider Packages for Auth0, CloudFlare, Confluent Cloud, Datadog, DigitalOcean, Docker, GitHub, Kong, MinIO, MongoDB Atlas, PagerDuty, Snowflake, Spot by NetApp and others.

    There are also Pulumi Component Packages for deploying container applications on instances of Kubernetes and other related platforms in addition to deploying applications on serverless computing frameworks.

    Duffy said in addition to providing a “golden image” of their configurations the registry reduces the overall level of friction that often exists between developers, IT operations teams and cybersecurity professionals.

    The latest Pulumi offering is part of an ongoing effort to democratize cloud computing in a way that is repeatable and reliable, noted Duffy. Pulumi, for example, just added support for a set of AWS Cloud Control application programming interfaces (APIs) that reduce the number of APIs that developers would otherwise have to master. The more APIs there are, the more likely it is a developer will make a mistake.

    IT teams should expect that other cloud service providers will similarly streamline the APIs they present developers as part of an effort to make their services both more accessible and secure. In the meantime, frameworks provided by Pulumi provide a means to invoke those lower-level APIs at a higher level of abstraction for both developers and internal IT operations teams alike.

    It’s not clear today how many developers routinely employ IaC tools, but as the number of workloads deployed in the cloud steadily increases, so too does the percentage of developers that employ these tools. While that shift represents a major boon in productivity for developers, it brings with it additional security risks that many organizations fail to appreciate until it’s too late.

    The only way to really solve that issue is for organizations to adopt DevSecOps best practices that require a level of collaboration between DevOps and security teams that is still relatively rare. However, as the tooling provided continues to improve, the technical challenges associated with achieving that goal continue to decline. The next big issue, as always, is bringing together all the diverse cultures within an IT organization to actually implement those tools and define a set of best practices that work for all concerned.

  • Infrastructure-as-Code Security will Improve Your Cloud Security

    Infrastructure-as-Code Security will Improve Your Cloud Security

    Moving to the cloud creates a tremendous opportunity to get security right and reduce the risk of a data breach. Most organizations start their cloud security efforts by ‘shifting security left’ thereby addressing issues in the CI/CD pipeline. Fixing known vulnerabilities and risky configurations before pushing images to production makes sense. But there is an opportunity to shift security even further left—as you set up your cloud infrastructure—to reduce risk. Infrastructure-as-code (IaC) security allows you to define security controls as you configure your cloud infrastructure.

    Getting Started on Your Cloud Journey

    Before we dive into IaC security, let’s talk about why platform teams want to deploy infrastructure-as-code as they move through their cloud journey. Most organizations have not yet tackled IaC. They start their journey by experimenting with cloud configurations until they have defined a stack that works for their teams. In this early stage, application teams may be defining their own cloud stacks. They are manually configuring infrastructure or using scripts. Configurations may not be standardized across teams.

    Standardization of Infrastructure Saves Time

    Although experimentation is reasonable early on in the journey, standardization of infrastructure saves time. Cloud services are no different than applications that run on premises. When your teams are using different versions of tools in their stack, getting an application up and running can be a nightmare. Interoperability between services is challenging, and it can feel like you are fixing a different flavor of the same problem over and over.

    Mature cloud platform teams typically prefer to create a ‘golden image’ with a baseline set of services for development teams. They may create a menu of a few standard options to meet different teams’ needs. Because they are standardized, the platform team can take time to validate that they interoperate and are secure. For example, these golden images will be regularly scanned for vulnerabilities and updated to ensure they meet internal security standards and any relevant regulatory requirements.

    IaC security cloud

    Automating Infrastructure Deployment Reduces Errors

    Manually configuring the environment in the cloud for each team is tedious. Once these configurations have been standardized, the cloud infrastructure can be provisioned as code using templates. A declarative approach defines the desired end state, whereas an imperative approach defines the detailed steps to achieve the end state. The templates are normally stored in a Git repository and version controlled. Automating the configuration of infrastructure not only saves time but also reduces errors that can disrupt operations.

    IaC Security Shifts Security Further Left

    Once you have templates that define your infrastructure, standardizing permissions and access rights for services-as-code is a logical next step. The security team typically defines the guardrails or policies for accessing cloud services. These policies can then be defined as code along with the infrastructure configurations. This IaC security is a further ‘shift left’, allowing security to be embedded into infrastructure, even before an application can be deployed. Implementing policy-as-code minimizes the opportunity for human error in configuring security services. By defining these policies and enforcement guidelines together, the DevOps and security teams can establish a trusted partnership.

    Although some organizations may start their journey by implementing IaC security, for most organizations it happens later in the cloud journey. The order is less important than ensuring you take a DevSecOps approach and embed security at each stage to manage risk. Be sure to address the following areas as part of your cloud and container security program:

    Continuously Check Cloud Permissions and Configurations

    Even if you have IaC templates, you need a program to regularly take inventory of all cloud assets and confirm permissions and configurations. The Center for Internet Security (CIS) benchmarks are commonly used to check passwords and access rights. Beyond enforcing these best practices for security on cloud services, continuously checking for changes to permissions and suspicious activity is critical. The cloud vendors capture logs across all of their services, and your security tool should continually monitor and alert using those logs.

    Scan Images for Vulnerabilities and Risky Configurations

    Scanning for vulnerabilities and ensuring configurations use best practices allows you to shift security left into the development pipeline. Scan everywhere you have images—including within registries, pipelines and also check for new vulnerabilities identified for images running in production. Be sure to scan hosts as well as containers. Defining scanning policies and enforcing through the development process creates guardrails for the development team and clarifies expectations. Most regulatory compliance standards have image scanning requirements as well.

    Detect Threats and Respond to Alerts

    Prevention is fundamental but must be supplemented with threat detection and response. Alerting on unexpected behavior can identify the presence of a threat. Initially, teams may use out-of-the-box detection rules as a way to get started and then customize rules over time. Checking configs against dockerfile best practices is a good way to get started. Overly permissive access rights for containers is common in Kubernetes environments and should be allowed only when required.

    As your team develops more expertise with containers and Kubernetes, you can also implement zero-trust network security policies to block unexpected communication between pods, services and applications.

    Some DevOps teams handle triage of alerts, as they can quickly determine if the issue is a configuration issue or an indicator of compromise. Teams often map detection rules to the MITRE ATT&CK framework, allowing their security teams to manage alerts consistently across their environment. For any team to effectively respond to a security alert, it is important to capture a detailed record of exactly what happened before, during and after that event. This visibility can be particularly challenging with containers but it is critical because containers have short lives. The 2021 Sysdig Container Security and Usage Report revealed that about half of containers live less than five minutes.

    Complying with Regulatory Standards

    Complying with regulatory standards can be painful. Some standards were written with cloud computing in mind, like NIST 800-53 and SOC2, and others were not. With those broader standards, it is important to understand the controls that are relevant to containers and the cloud.

    As you consider your compliance program, be proactive—don’t wait until you are ready to roll into production. Start early by engaging your compliance team and helping them get up to speed. Your role is to provide evidence of compliance to pass an audit at any time using the documentation from your security platform.

    Getting Cloud and Container Security Right

    The move to the cloud creates a tremendous opportunity to get security right and reduce risk. The fact that physical security and a chunk of operations are delegated to cloud vendors frees up security and development teams to focus on their more narrow scope within the shared security model. By taking a secure DevOps approach and inserting security into every stage of the application life cycle, your team can confidently and securely run cloud and containers.

  • Cycloid Tool Can Reverse Engineer Provisioned IaC

    Cycloid Tool Can Reverse Engineer Provisioned IaC

    Cycloid this week unveiled a tool that makes it possible to reverse engineer code used to manually provision cloud infrastructure. The Infra Import tool is part of an effort to create a more consistent and reliable version of that code using open source Terraform software.

    Benjamin Brial, Cycloid CEO, said Infra Import is based on Cycloid’s open source TerraCognita software, which enables IT teams to reverse engineer infrastructure-as-code (IaC) created using open source Terraform tools.

    Infra Import extends that capability to cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform or an IT environment employing the open source OpenStack cloud platform that may have been manually provisioned. IT teams can then create a configuration based on that Terraform code that could be employed across multiple clouds.

    Infra Import is designed to connect to a cloud provider and then automatically reverse engineer Terraform files and IaC stacks based on the existing deployment. It also ensures that the latest edition of Terraform is being used to create that configuration code, noted Brial. That capability eliminates the need for DevOps teams to keep track of what version of Terraform might need to be updated as new versions of the IaC tool become available.

    Just as significantly, Infra Import also makes it a lot easier to onboard new members to a DevOps teams because the tools uses valid secure configurations they know can be relied on. Misconfiguration of cloud services is one of the biggest security issues that IT teams today are wrestling with, simply because most developers lack the security expertise required to configure cloud services properly.

    Cycloid is in the process of rolling out a comprehensive suite of management tools that complement a lightweight framework it created to make DevOps best practices more accessible. The challenge is many IT organizations have already manually provisioned cloud infrastructure in a way that is, at the very least, often suboptimal if not downright insecure.

    Misconfigurations are rife across cloud computing environments, but most organizations are not prepared to manually address the issue. Automating the reverse engineering of those configurations is the first step toward enabling IT teams to address the problem at scale.

    It’s not clear to what degree configuration concerns have impacted the number of applications that would otherwise have been deployed in the cloud. Security is always listed as the top concern IT leaders have when employing cloud platforms. The issue, however, is not the security of the platforms themselves but rather the processes employed to deploy applications on the platforms. Cloud service providers have promoted a shared responsibility model for security under which developers are accountable for both securely configuring services and ensuring the overall security of the software environment. The trouble is, not every developer understands the implications of that shared responsibility model much less how to actually secure an application workload.

    It’s not likely security concerns will slow down the rate at which applications are deployed in the cloud. However, as more cloud security issues are encountered, more scrutiny will be applied.

  • Fugue Aims to Simplify Securing Infrastructure-as-Code

    Fugue Aims to Simplify Securing Infrastructure-as-Code

    Fugue today unveiled a 1.0 release for Regula, an open source policy engine for infrastructure-as-code (IaC) security that comes with prebuilt libraries for implementing hundreds of policies that validate configurations on Amazon Web Services (AWS), Microsoft Azure and Google Cloud services.

    Regula is based on the Open Policy Agent (OPA) software being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) and is compatible with both Terraform and AWS CloudFormation tools for configuring cloud infrastructure.

    Developers can also build custom rules on top of OPA using libraries that Fugue created for the Rego programing language that is part of the OPA specification. Regula supports output formats such as JUnit, Test Anything Protocol (TAP) and JSON to make it easier to integrate Regula with other tools and frameworks that make up a DevOps workflow. Input formats supported include Terraform HCL, Terraform plan JSON, AWS CloudFormation and Serverless Application Model templates.

    Regula also provides out-of-the-box support for the CIS Foundations Benchmarks in addition to other policies defined by Fugue, such as dangerously permissive identity access management (IAM) policies, Lambda function policies allowing global access, volumes with encryption disabled and untagged cloud resources.

    It is deployed as a pre-packaged binary that includes a command line interface (CLI) through which Regula can be incorporated within a DevOps workflow and can be installed using tools such as Homebrew or deployed as a Docker image.

    Fugue CEO Josh Stella said Regula provides DevOps teams with an extensive library of rules that check for common security and compliance violations as well as advanced multi-resource misconfigurations. That latter issue is especially challenging because developers typically don’t have much visibility into dependencies that can inadvertently provide cybercriminals with access to, for example, an S3 cloud storage bucket that otherwise appears to be configured properly. Regula will also identify any required resources that might be missing, as well, added Stella.

    Stella said Regula 1.0 provides a simpler approach to securing cloud infrastructure. Cloud infrastructure is frequently misconfigured simply because a developer made a mistake. Unfortunately, cybercriminals have become especially adept at scanning cloud platforms for these misconfigurations, noted Stella.

    It’s not clear to what degree a backlash might be building against IaC in the wake of recent high-profile breaches of software supply chains. Organizations have embraced IaC tools to make developers more productive. However, each security incident involving a misconfiguration of cloud infrastructure resources makes it all too apparent that developers lack the expertise required to securely provision IT environments. Rather than creating a series of cybersecurity reviews that would reduce developer productivity, Stella said Fugue is making a case for a compliance-as-code approach that developers can readily understand and implement.

    One way or another, cloud infrastructure has to become more secure. Business leaders are growing weary of security incidents that, from their perspective, are caused by careless mistakes. Their first instinct is to empower cybersecurity teams to resolve those issues once and for all. The only issue that needs to be resolved now is just how cumbersome those security reviews need to be if developers are provided with the tooling required to secure cloud platforms themselves.

  • EP 9: The Road to GitOps

    EP 9: The Road to GitOps

    GitOps takes DevOps best practices for development and applies them to infrastructure automation. In short, GitOps delivers a hermetic, immutable deployment. So, is GitOps replacing DevOps?

    Managing infrastructure as code through GitOps can be a game changer for organizations. But, when is a GitOps approach best used or not a fit? In this DevOps Unbound episode, I’m joined by Tracy Ragan of DeployHub, Parag Doshi of Tricentis, Brendan O’Leary of GitLab and Mitch Ashley of Accelerated Strategies Group to discuss these questions, the impact of GitOps on the CI/CD market and where the future of GitOps is heading in 2021.

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript

    Alan Shimel: Hello, everyone. This is Alan Shimel, CEO at MediaOps. And you’re watching another episode of DevOps Unbound. Let me introduce you to our panel for this week’s episode, and we’ll jump right into our subject matter. First of all, we have the newcomer to DevOps Unbound, but he’s not a newcomer to our audience, my good friend Brendan O’Leary, senior DevOps evangelist at GitLab, a governing board member for CNCF. Hey, Brendan. Welcome.

    Brendan O’Leary: Alan, nice to see you, as always, and happy to be here.

    Shimel: Thank you for being here, Brendan. Next up we have our good friend and a semi regular here on many of our video segments at MediaOps, Tracy Ragan. Tracy is CEO of DeployHub, as well as the community-elected board member of the CD Foundation. Hey, Tracy. Welcome.

    Tracy Ragan: Thank you. Thank you so much, Alan, for having me here today. I always love doing these.

    Shimel: Absolutely. We love having you. Our third panel member today comes to us from Tricentis, who’s also the sponsor for DevOps Unbound, and it’s Dr. Parag Doshi. Parag runs engineering, VP engineering at Tricentis. Parag, welcome.

    Parag Doshi: Hello, Alan, everyone. Nice to be here. Thanks for having me.

    Shimel: Thank you. Thanks for being on. And then last but not least, my cohost for DevOps Unbound, Mitch Ashley. Mitchell is the CEO/founder of Accelerated Strategies Group as well as CTO at MediaOps. Hey, Mitchell, thanks for coming on today.

    Mitch Ashley: Absolutely, Alan. And thank you for all of our panelists for joining us. You are all fantastic. We enjoy having you.

    Shimel: You know, just before we jump into it, the continuing evolution of COVID times, we’ve done so much with our backgrounds, [Laughter] on our videos, right? Looking here at Tracy’s snazzy DeployHub background and Mitchell’s background and Parag’s – and Brendan’s old-school, man. Brendan’s giving us – if I’m not mistaken, he used to be down in the basement and now he moved up. I think he displaced one of the kids or something.

    O’Leary: Yeah, I moved upstairs.

    Shimel: Yeah, yeah. And that’s a real background, so good for Brendan.

    Ashley: And he’s got an MV7 microphone there. Nice job there, Brendan, on the mic.

    O’Leary: Oh, thanks. Yeah.

    Shimel: Yeah.

    O’Leary: All, you know, post COVID things to make it work, Alan. I wanted to have the background, the real background to make it work. But that was also a COVID change.

    Shimel: It’s authentic. Yeah. [Laughter] Crazy stuff. Anyway, guys, today’s show is around one of the hottest topics in DevOps – GitOps. Right? And I’ve actually heard from people, you know, that GitOps is the new DevOps or GitOps is somehow replacing DevOps, and I think DevOps has outgrown the – it’s going to be replaced by something.

    It’s transcended to a different level of existence, if you will. But certainly GitOps is on a lot of people’s lips. It’s not only on their lips, it’s in their development plans and how they are delivering software to the cloud, to cloud native, and all over. But is the right move for every situation? Is it a panacea?

    When is GitOps a good thing to use? When is it maybe not a good thing to use? Right? How much does it change sort of the classic CI/CD pipeline that we’ve started to finally wrap our heads around in DevOps? Stuff like that. So I am going to – Brendan, of course, GitOps, GitLab, they both have Git in it.

    So we’re going to make you – I’d like you to start off. Let’s start with some definitions. What’s GitOps mean to you?

    O’Leary: That’s a great question, and I think one that’s still probably [Laughter] unsettled, right? Much like defining DevOps can sometimes be seen as unsettled. GitOps came out of the cloud native community, and so a lot of folks maybe restrict the concepts of GitOps to say it only exists in a Kubernetes world. Because that’s kind of where this approach came from. But we’ve seen folks employ GitOps principles to all kinds of different infrastructure, automation, from old-school VMs to containers to Kubernetes clusters as well.

    So when I think of GitOps, I think it kind of requires three things. One, infrastructure as code. Right? You said that in the beginning. That’s kind of like table stakes to getting in. It’s something we’ve been working on for a long time. But really defining as code in Git – the Git part of GitOps – what our infrastructure’s going to look like, and then keeping that configuration stored in that code.

    And maybe the desired state is stored as code or not, but at least the way we get to that desired state is. Second, I think it’s merge request. So we call it merge request. You’ll hear other folks call them pull requests. But the concept of using the point where someone is proposing a change as the central place where teams collaborate on.

    Is this the change we want to do? Is this going to help solve the problem we’re trying to solve for? Right? Bringing that concept to the world of infrastructure is hugely beneficial, because it’s how our developers have already been working for years and years. And making that merge request, kind of that central source of truth for approvals and security checks and all those other things is really critical.

    And then last, I think it’s – the third thing is CI/CD, right? Of course. So the idea of GitOps is automating that infrastructure, update with a very simple Git workflow. And so you really do have to have embraced and adopted continuous integration and continuous delivery really in order for the whole package of GitOps to make sense for you.

    Shimel: Pretty darn good. Comments, add-ons, thoughts? Parag, Tracy?

    Ragan: I would only add one thing here. If you say what is GitOps, I like to shorten it down. I think what Brendan just defined is all accurate, but it’s operations by pull request. The goal of GitOps, what GitOps delivers is a hermetic, airtight, immutable deployment. And that is in essence what we’re trying to accomplish by using Git as the place where we declare and define what we’re going to push out to or have pulled out to the cluster.

    So it’s the concept of a hermetic deployment, something airtight that no human touched. Don’t you dare touch that and then deploy it out, because of we don’t know what you did. [Laughter] So the whole idea is to create a hermetic process.

    Shimel: I love that. That was concise and right on.

    Doshi: I have a similar viewpoint. I think of GitOps as a combination of infrastructure as code plus event-driven architecture. Right? And in this case we’re talking about merge requests or pull requests or whatever terminology you want to use, depending on the flavor of Git you love. But the way I think of this as a former physicist is entropy is there. It’s too hard to keep track of everything. Humans can’t remember everything.

    We need to be able to describe, as application developers have been doing for the past couple decades, in code the versions of things, the documentation associated with everything. And then the automation that we’ve had in CI/CD, which people think of, oh okay, that’s developers pushing their code out, well there’s no reason why we have to stop at application level. It needs to be the whole thing, because application and infrastructure must stay in sync at all times. So by putting a description of both the application, the infrastructure, and within the surrounding environment, all three of those into Git, you’re bringing the best practices of AppDev into the whole world together so they’re all synchronized.

    Ashley: Yeah. I like that description too, Parag, because in a way that’s breaking down silos that we still have around the code build and deployment process. And it’s almost like if you think back when we started using code repositories and, okay, everybody put their source code in one place, okay, let’s expand that to artifacts and other things. Well, now we’re doing the same thing of let’s abstract at one more level and make the whole thing as cradle to grave, if you will, for getting code out into product. Whether it’s any part of that stack, it’s a GitOps process.

    So to your point, Tracy, about it’s an immutable object, it’s something that we know from beginning to end, it’s all been in one environment and we happen to have manual processes that have just been automated that potentially are weak or could be broken.

    Doshi: I’ll add one more comment on top of Branden’s about – I think it was around – it came out of cloud native computing foundation, but really the way I look at containers is it doesn’t matter that’s a container or VM. We’ve just moved a control plane up to a higher level, but the same principle’s apply. So there’s no reason why you cannot use these same principles in a container type of environment or a non-container environment. The only difference I would say with containers is we can be very surgical, fine grain, down to each individual pot. So there’s a little advantage with cloud native architectures to be very fine grained.

    O’Leary: Yeah, I agree. And I think that’s important for folks to separate the concept of GitOps, the principles that we’ve all talked about. And I agree with the other panelists and how they’ve talked about it. But separate those principles from the technology, right? Because it’s not always true that Kubernetes is going to be the right technology for you to deploy every workload you have in, but still this concept of GitOps.

    And I really liked Tracy’s sealed container – well, overloading the word container there. [Laughter] But that sealed deployment that doesn’t have a lot of human – or doesn’t have any human interaction. That is what gets us to what Mitch was talking about, which is we want to take this isolated, people working in isolation, the kind of cave-like nature that operations typically used to have where you had to have this specialized knowledge that one person knew. Oh, wait. How do we deploy that service?

    Oh, we’ve got to call up Alan. He’s the one that knows how to do that. [Laughter] Right? You want to get rid of that and make it all, what Tracy said, which is like a hermetically-sealed deployment that we know we can reproduce every time.

    Ragan: Yeah. So GitOps was a brilliant way to leverage existing tools and habits to create Chef and Puppet. That is basically what was done. I mean, it has a Puppet kind of master architecture. You have a GitOps operator that’s running on your cluster. It does a pull based on what’s in Git. It very much took the basic concept of Chef and Puppet and leveraged existing tools that people already had, some open source, GitOps operators out there, and they were off and running. And that is the brilliance and beauty of GitOps. I mean, I have a lot of issues with GitOps, and I’m sure we’ll share them, but its brilliance is in that simplicity.

    Shimel: The irony though, Tracy, is that many people will tell you that Kubernetes killed Chef and Puppet, right? And we’ve seen Chef get sold maybe for less than money raises in valuation previously.

    Ragan: No, I’m just talking about the basic concepts.

    Shimel: No, no, I get it. But if we look at the concepts behind what Chef and Puppet did, which was automate that deployment, deployment automation like that, and then with Kubernetes it sort of negated the need for that, but yet we kind of snuck that in the back door. Except we put it in Git to use in Kubernetes. So the irony of that is pretty profound, right?

    Ragan: Well, it’s all about the container.

    Ashley: Right, right.

    Shimel: Yeah. You’re right. Maybe it’s because Chef and Puppet – [Crosstalk]

    Ragan: The containers simplified, yes.

    Shimel: – didn’t figure out how to work in the container. Look, for that matter Jenkins too. Right? We went to Jenkins X because of all of that and everything else.

    Ashley: That’s a really great example, yeah.

    Shimel: But what I’m reminded is of the old Who song, right? “Meet the new boss, same as the old boss.”

    Ragan: Exactly.

    Shimel: And you still have that old boss. With that being said, here’s another thing, and Parag, I’m going to ask you to kick off as I take us a direction here, and that is GitOps, yes, it’s for deployment. What about testing? I said this is all about testing. We’re doing our testing in Git. Is that the – that stops, right? Is that better? Worse? Is that the way it should be, always, sometimes, never?

    Doshi: So I don’t know if I’d call it testing in Git, but I would say now Git becomes a source of truth. So I think what Tracy said is right, that it’s just that same sort of automation of infrastructure but now moved into making Git the centerpiece of where you define this. So the Git part is unchanged, but the testing is now event driven. Right? So because you’ve merged in code, you’ve committed code or gone to do a pull request, that can trigger testing.

    And we call this continuous testing. And I think the evolution of this can even go as far as our test definitions themselves should also be put into Git. That’s the next evolution that we’ll have. I don’t believe it’s really commonplace just yet, but when that happens then, we are trying to describe as much as possible into Git as a repository and use the principles of version control added with the event-driven concepts.

    So when a developer does X or when an operator is in environment Y and something needs to change, that whole Git workflow can invite continuous testing to say do I allow that to merge back in? If you want to have an approval, you could do those kinds of things. So to me, incorporating continuous testing, and the day that we put test definitions into Git as well, we’ve gone quite a bit further of completing this vision and really making DevOps real. Right? Because DevOps isn’t just releasing really fast, deploying really fast. It’s deploying fast with confidence and ease. That’s what we need.

    Shimel: Thoughts on that, panel?

    Ashley: It sure seems like a natural evolution to include the test environment. And as you say, Parag, it’s probably not commonplace yet today. A question for you about that; do you think that there’s incentive with the folks that are taking care of the testing, whether they be the QA team or developers writing tests, things like that? Do you think they have the right incentives to move it into Git or will it be something else that causes it to happen?

    Doshi: I think as long as people have the incentive of delivering value as quickly as possible with the best meantime to recover, the lowest defect leakage rate, as long as we do break down those silos, as you said. If the testing team is, okay, well, I don’t understand what the developers do, just throw it over the wall and then I do my test, that’s not going to work. This is talking about dev testers, the full stack developers. And as long as everybody’s motivation is right, that we’re trying to deliver value faster and faster, that we will not have confidence – that a VP will not say let my applications go out – you know, they’re only going to look at portfolios of apps and say what amount of testing has been done? What amount of security checks have been done?

    That’s another thing. The environment. Okay? That’s the biggest entropy of all. We have the least control over the environment. The same code, the same infrastructure in different environments behaves differently. Until we get that environment description also into Git, coupled with the testing model themselves that says here’s how we test it in that environment or that application or this infrastructure pattern, then DevOps becomes real. Until then, this is all talk.

    O’Leary: Yeah, I agree. I think it’s critical, and it’s something that GitOps, again, this concept of GitOps, I think is exciting to a lot of people that have been trying to make DevOps real for so long, because it enables that in a way. Right? We see kind of the light at the end of the tunnel, I would say. And if that is all brought into Git and all made into something that can deploy independent of any human interaction, well now you have the ability to do continuous testing. You have the ability to stand up an identical environment, the golden goose or [Laughter] golden egg of a, hey, we want an environment that’s exactly like production to really test it and really hit with security tests and know that when we do deploy to production we’re going to have that confidence.

    So if you’re able to continue to iterate on the environment, the infrastructure, just like you’re iterating on the code, it enables you to then say, okay, now we can not only have the code to review and maybe some static code analysis that happens and some testing that happens, and then maybe some human testing that happens, now we’ve also got an entire environment spun up automatically that mirrors production, where we can have a high level of confidence that it’s going to work when it really ships.

    Shimel: So guys, you’re telling me DevOps hasn’t been real quite yet. I feel like I’m living in the Matrix, kind of living in a simulation. Tracy, I sense you’re Agent Smith here, though. You have a different view. Maybe GitOps isn’t the panacea it is or it’s been projected here. What’s your view?

    Ragan: So no hate mail, okay? No hate mail. [Laughter]

    Because I know there’s a lot of enthusiasts out there, and rightfully so. But let me just level set the way I see where we’re headed and where GitOps can’t take us. GitOps is imperative. It’s scripted. And it works really well – so think about how most people use GitOps.

    You have a code repository that you keep all your source code in. You make a change to your source code. You check that in just the same as you always did. You also have an environment repository, and what lives in your environment repository is your Kubernetes deployment YAML file. And that is where you define all of the stuff that we’ve been talking about and that’s what the GitOps operator looks at and pulls across.

    Now if you’re in a Kubernetes environment and you have containerized your application – this works great, it really does – and you don’t have a whole lot of environments – you have three environments dev, test, and prod – because remember, in a Kubernetes YAML file there’s a couple of things that are important to the GitOps operator. One is the tag of the container. So once you register your container image you get a SHA, and that makes it a unique identifier. And you have to update your YAML script with that SHA.

    Now dev, test and prod may have different values, configuration values. So you have to branch it. So you branch dev, test, and prod. So now you have three versions of that YAML file to manage your container deployment. And that’s pretty doable for most organizations today, but it is still imperative. Everything’s being defined by the script for each of those environments.

    Now let’s go and expand it and scale it. Let’s scale it to a microservice environment. Every microservice has its own YAML file because it should be independently deployed. And they should be shared, because that’s the beauty of a service-oriented architecture. They should be shared across teams. So what you do is as an application team, you’re going to make your coding changes in your coding repository and then you’re going to have different YAML files for the different services.

    Now let’s say you’re a store. I use the candy store and hipster store and the clothing store. They all use the card service. Well, they’re all going to put their card service YAML file into their environment repository, and that’s going to get deployed to their own clusters and name spaces. So you have a scaling issue, and it’s because of the imperative nature of GitOps.

    We are starting to move and have more conversations about declarative and events and catalogs, where this stuff is going to be stored and generated at runtime. So for example, instead of having all of these scripts, you’re going to have central places to go to say what’s the key value pair for this environment? I’m going to pull this registry information that has the SHA, I’m going to go get my key value pairs, and that way you don’t have to have so many scripts and you’re not managing – it’s onerous.

    It really can become onerous. We have customers who have 200 microservice and 20 clusters. That’s a lot of scripts. And if you’re an SRE and you want to update your microservice and you want to provide it to everybody, you’re going to have to figure out every single application environment repository that has it in there for every single environment and make those corrections. So it doesn’t scale as well as it should. Now I know that their CNCF has a group, a working group, around GitOps, and they’re starting to think about GitOps version two, and they’re going to have to address these kinds of core problems around an imperative process.

    Shimel: Love it. Brendan?

    O’Leary: Yeah. I mean, I like that concept and I think you’re right, Tracy, that the ability to manage the scale is definitely something you have to be prepared for if you’re thinking about GitOps, because if you just take it at its face value you end up with all those scripts and it could make the problems you have worse rather than fixing some of the problems you hope it would come into and solve. And so I think in a cloud native environment, again, it maybe pairs a little bit more simply, but I think that’s why again it’s great to think about the concept of focusing on that developer-centric approach and the experience that they’re going to have when it comes to shipping something out, to your point of if I want to ship an update to my microservice that impacts everyone, I don’t want to have to go through 78 repositories and update that everywhere.

    And so having GitOps do the infrastructure management in the same version control system as application development is great, but you also have to have a plan in place for how does that scale to the scale that you’re really going to build to? If that the scale of developers, if that’s the scale of services, if that’s the scale of environments. And so I think it’s important to have tooling and an approach in place that accounts for that as well.

    Shimel: Parag, you were going to say something I think before we –

    Doshi: Yeah. Sure. So I love what Tracy described there, and getting into the specifics of the proliferation of YAML files, and especially as we break down microservices. So this is the whole concern with microservices. What you took as monolith and you described as one big black box, now you’ve broken it down. And so now we have the proliferation of all these things. Right?

    So I like things like Helm to help to reduce that and parameterize all these things. But fundamentally with GitOps, the concepts are there. Like I said, infrastructure as code with an event-driven type of approach version control. What I like about CNCF and containerization is the purpose of containers inherently – and Branden can keep me honest on this – is really to make the environment completely separated from your solution, your application, your infrastructure. Right? So today’s industry, the best approach we have is containers or serverless, which is also part of CNCF, so that we have things like Docker files and YAMLs that allow us to reduce the entropy of the environment as we can as much as possible and describe them declaratively in things like GAML files.

    And I do agree with Tracy 100%, that now it becomes, okay, well now I’ve got all these different configs, pulled that out of my code. Even with service mesh technology and CNCF we’re pulling out things like security and rate limiters between every single microservice. All that’s taken out of the logic, the business logic, on everybody doing security different ways. Now it becomes a configuration issue. How do I – if I want to change a configuration, do I need to change 100 different YAML files or config maps? And I think the technology’s evolving.

    Help is just one example. And then I’ll associate that with all the continuous delivery. You all know Spinnaker and a bunch of different technologies in this area, Octopus and so forth, that can help.

    Shimel: Yep. Mitch, did you want to say something? Otherwise, I want to – I’ve got another –

    Ashley: All I can add to the conversation is version control across all of this also gets very complex. [Laughter] I think you all –

    Shimel: But isn’t that what Git and Repositories are supposed to help us with anyway?

    Ashley: It is, but now you’re versioning all of your scripts along with all the environments and all of the variables of what you’re trying to script.

    Ragan: And it’s not relational.

    Ashley: Yeah.

    Ragan: You know, when you have – what’s the blast radius of my microservice? I’m going to push it out because my hipster store needs the new card service. If you’re using an SOA architecture that means that you have one card service and everybody’s using that version and that’s the best way to move forward because you eliminate the sprawl. If you put it all in its own namespace, you have multiple versions of the card service. So now you have a problem with the card service and you have to go update all the namespaces that’s using that card service.

    So you have to have a relational model to be able to track what applications or what namespaces are using what microservices. And unfortunately, Git is not relational. We create relations using tags, using different repositories, but that becomes redundant.

    Ashley: Great example.

    Shimel: Fair enough. So let me –

    Doshi: Tracy, you talked about the structure.

    Shimel: Oh, go ahead, Parag.

    Doshi: I was just going to follow up and ask a question to Tracy. I think what you’re saying there is Git is just a big version control repository. If we can provide some extra structure and relationships around all these artifacts that we’re storing there we might have better control.

    Ragan: And that’s what you call catalog.

    Doshi: Yeah.

    Ragan: Right? So I think we’re starting – when we really look at a service-oriented architecture and we look at – let’s say, for example – let’s push ourselves farther into the future. And Brendan brought this up around service mesh. And I believe this can happen and will happen. I believe service mesh will be enhanced on.

    Right now you can say do a deployment and deploy out to 10% of my users. I think service mesh will start looking at active directory and say deploy out to my developers. So now we have one cluster and we have immutable services out there. We put a new service out there, it gets requests routed to the developers, they do their testing on it, once it’s good it’s rerouted to their testing community.

    Then it’s routed to a certain group of their production community. Which means that we don’t have deployments across multiple clusters. We’re going to start pushing into a one big cluster, which sounds a lot like the mainframe. [Laughter] But we’ve come full circle. And as Alan said, the song, right? Meet the new boss, same as the old boss. [Laughter]

    Shimel: Right. We won’t get fooled again. That is true. There is certainly some truth to that. I wanted to spend a little time talking about the impact of GitOps on the CI/CD market. We’ve written in devopsy.kinsta.cloud several articles, the gist of which was DevOps is the symptom of disease. CI/CD is the cure, if you will. Right? And we’ve built this whole DevOps industry around CI/CD solutions.

    And I mean look, GitLab’s a great example. They had a great Git product to begin with. But it was their CI/CD solution that kind of moved them into that end-to-end DevOps, if you will. Right? GitOps I think has profoundly changed that market because it’s profoundly changed the way we do CI/CD. Well, that and Kubernetes, right?

    Kubernetes I think was the initiator of it. What does it mean for the market? What does it mean for our audience out here? And we have a technical audience, 60% of which are practitioners, 30% are managers or higher, and 10% are just, I don’t know, vendors. But what does it mean for them out there?

    What does it mean? I mean, these CI/CD tools that they’ve gotten used to, that they’ve grown up with a bit over these last years. What does it mean for them?

    O’Leary: Well, Yeah. I think that we love to say CI/CD together. I mean, I love to say CI/CD together, right? That’s now we name our product. But those concerns have always really been separate and never really been completed merged and married. Like it’s great to try and think about them as a single unit – CI slash CD, right?

    But I think the reality is that CI is something that has been around for a really long time, probably 20-plus years if not more, but CD is something that a lot of organizations still struggle with for any number of reasons, whether that’s a technical reason, whether that’s an organizational structure reason, whether that’s a risk aversion reason. The idea of doing continuous deployment is something that really is very dependent on your organizational structure, who your end users are, and the, yes, to some degree like you said, Alan, the tooling that you choose. And what we’ve seen in traditional CI/CD tooling is this concept of applying what we do in CI, which is scripting based on an event, running through a number of scripts.

    That is what then becomes CD. Well, now we’re going to push with scripts out to the environment. And so GitOps, you noticed in my definition I didn’t say that a pull was necessary. I think you can do push-based GitOps if you have infrastructure as code, you have MRs at the point of change control, and you have the ability to do CI/CD. But I think most people when they hear GitOps think about this another new concept, which is this concept of pulling. Right? We’ve heard Tracy talk about that.

    And so having an agent, an active agent inside of your infrastructure that is constantly ensuring that your desired state is the actual state in the world – that’s what Kubernetes brought to us, that’s what – and then that’s what pull-based GitOps brings to us. And so that’s why for us we’ve been doing CD for years, like you said, Alan, and we’re doing mostly push-based. But as our customers’ needs expanded in GitOps we’ve adopted a Kubernetes operator and now we have our own Kubernetes operator, and you’re seeing this more and more in all the tools, that runs inside of the cluster and pulls changes in.

    And that methodology, while – again, credit Tracy – sounds a lot like Puppet, [Laughter] it still is something that was the alure of Kubernetes for a lot of people. Hey, I want to describe my desired state and have something else worry about that desired state always being true regardless of if I push something wrong once. I want this desired state to always be correct. And so I think that’s how we’re going to see a huge change in the tool space. Right?

    To Parag’s point, we’re talking about Spinnaker. That’s something that Netflix has been using for a long time, but no one ever thought that they would be deploying like Netflix. Well, now it’s a huge deal and lots of people are talking about it.

    Shimel: We are.

    O’Leary: And they are, right? And so it’s very interesting.

    Shimel: Mitchell, it sounds like an analysis question.

    Ashley: Well, I was thinking about that very much, and I appreciate Brendan’s perspective on it. Because any time you have a major paradigm change, you have early adopters and then you have a long tail of how that adopts out. And you’re talking about the Netflix. People are now starting – how long have we been talking about Netflix as the example? As long as we’ve had DevOps and continuous deployment.

    And I think we have a bit of the same challenge here of instead of still having layers of our stack and different responsibilities or processes or things that go into creating environments, whether it’s from code or not, when we say it’s all going to be based on a pull and it’s all going to happen in its own environment, if you will, that’s a bit of a big pill to swallow for a lot of organizations, especially in complex applications. I’m not saying it’s not going to happen, but I think it’s going to be slow to adopt. And I’m sorry if I’m going to get a bunch of hate mail.

    It’s a great concept, but I do think it’s going to be more difficult to adopt than even containers were for breaking down applications. Not that it’s not a good idea or the right idea, it’s just instead of software architectures, modern software architectures being something the developers now get their mind around with containers and microservices and fabrics, meshes, now you’re talking about different parts of the organizations going into infrastructure and ops. And that’s all getting compacted together and you’ve got multiple parts of the organization getting behind that paradigm shift. So that’s my view of it.

    Will we see things in the market around GitOps? Of course we will. But I don’t think we’re going to see this tipping point that will happen rapidly. That’s my assessment.

    Shimel: Okay. So it seems like you and Tracy have formed some sort of coalition here, and –

    Ragan: [Laughter]

    Ashley: I always go to Tracy when I need something _____.

    Shimel: Right. Send the hate mail that way.

    Ashley: _____ Brendan and Parag, too. [Laughter]

    Shimel: Right.

    Ragan: I want to say that the CI/CD pipeline can adopt GitOps quite efficiently. Think what Brendan pointed out. When you think about CI/CD you think everything’s a push. Right? And you add a script that you’ve executed that your CI/CD server, your orchestration server, your job scheduler said, “Hey, there’s a step here.” Then that step goes out and pushes an update to production environment.

    How that’s going to – because GitOps is a pull methodology it doesn’t have a promotion methodology. And what really we’re going to see is for folks using GitOps is we’re going to see the CI/CD process do the – create the pull request. And there will be an approval. If you want a pull request to go into production, there’s going to have to be probably a manual step to approve that pull request, and then once it gets there the CI/CD pipeline creates it. Now the challenge will be having the CI/CD pipeline automatically update all of those declarative scripts. [Laughter]

    Because there will have to be a different one based on the environment. But that is totally doable and it doesn’t change that. But what it does do, it says, hey, CI/CD. You need to have a hermetic process. It needs to be airtight. You need to have a single source of truth.

    What that is ultimately will still be called GitOps, even if it’s not Git that is that single source of truth. It’ll be like we reach for a Kleenex, not a tissue. [Laughter]

    Doshi: Let me back up what Tracy just said, and maybe take the view of – I think you said 30% of the listeners are looking at this from a business angle. And your question also sounded like what does this mean as our industry, to our industry? So today we’ve got, I don’t know, 170 different products, if you will, and tools around – and Brendan can probably say, no, it’s now 250 or something [Laughter] – around DevOps and all the new different concepts.

    Shimel: Closer to 700.

    Doshi: Oh, okay. My bad. [Laughter]

    I was a little behind. So we have so many things here, and there’s a reason for that too, right? So we can’t just say, hey, let me take a cloud provider’s full CI/CD DevOps vision and hope that we’re going to impose one way of doing things, because this is not all worked out. The moment you bring the developer mindset to other layers of the computer stack – infrastructure, environment, operations – it becomes so complex. And I think at an industry what it’s going to do is it’s going to actually spur a lot of opportunity, a lot more innovation.

    I don’t know if that means that 750 will go to 1,050, but there’s so much room for different companies that will do things differently – push, pull, I need an approval before production due to a segregation of duties. No, no, no. I want to approve before I even commit the code or at least run the security static analysis, the dynamic analysis, the container image scanning. Everybody will have different ways of doing things, and I think it’s going to spur on lots of opportunities for a lot of companies.

    Shimel: Fair enough. A giant petri dish. Guys, I’m going to need to pull the plug on us here because we’re up against our time limit. But what a great conversation. For our audience, in a couple of weeks – and I don’t have the date in front of me, I apologize – we’re actually going to do an interact roundtable on this subject. Some of our panel members will be back for that.

    And we’re going to allow you to come online and ask questions and comment and join the conversation on this. I think it’s a great conversation. So be on the lookout for that and come check us out. But in the meantime, Tracy, as always, you know, you bring so much to this. Thank you so much. Thank you so, so, much.

    Ragan: Oh, it’s my pleasure. Absolutely my pleasure.

    Shimel: Yes. Brendan, always a pleasure to have you on, my friend.

    O’Leary: Yeah.

    Shimel: Look forward – also, Brendan does a show with us here on TechStrong TV every month. It’s called The Lab. And he discusses – you know, the 22nd of every month is a new GitLab release and Brendan gives us the lowdown on each month’s release. It comes out once a month. It’s called The Lab, and it’s available here on TechStrong TV. So Brendan, thanks so much for that.

    O’Leary: Yeah. Thanks, Alan, for having me. I always learn so much from these panels. So hopefully we all learned something today, but I really enjoy doing them.

    Shimel: Always a pleasure. And you add as much as you learn, Brendan. Thank you. Parag, one question for you. You mentioned earlier that you were a former physicist. Is there such a thing as a former physicist? Isn’t it more like I think, therefore I am?

    Doshi: Yes.

    Shimel: Once a physicist, always a physicist?

    Doshi: It certainly is Hotel California. You’re absolutely right. [Laughter]

    Shimel: Yep. But thank you so much for joining us. Always great to have you here and bringing some great perspective to everything. I appreciate it.

    Doshi: Thank you.

    Shimel: Mitchell, what can I say? We wrapped another one, Mitch. Good stuff.

    Ashley: Yeah, we did. And I’m just so glad that Parag didn’t bring quantum physics into this, because I’d be in trouble.

    Shimel: That could be another show we’ll do.

    Doshi: [Inaudible comment]

    Shimel: Quantum computing. But, no, anyway, that’s going to wrap up this episode of DevOps Unbound. We hope you’ve enjoyed it. This is every other week we do a fresh DevOps Unbound, so be on the lookout. Every month we do the roundtable with the audience, so be on the lookout for that as well. This is Alan Shimel for MediaOps. You’ve – oh, before we go, many thanks to Tricentis for sponsoring DevOps Unbound. We couldn’t do it without them. So thank you all very much. We’ll see you on another episode.

  • DevOps Epitaph

    DevOps Epitaph

  • IaC and Kubernetes: A Natural Pairing

    IaC and Kubernetes: A Natural Pairing

    Using IaC with Kubernetes helps standardize Kubernetes cluster configuration and manage add-ons

    Infrastructure as code (IaC) is the ability to provision and manage infrastructure using a configuration language. It offers repeatability, transparency and the application of modern software development practices to the management of infrastructure including networks, load balancers, virtual machines, Kubernetes clusters and monitoring.

    IaC helps to reduce error and configuration drift while allowing engineers to spend time on higher-value tasks. IaC defines what the end state of infrastructure should look like, instead of defining a process to be executed.

    So Why Aren’t All Companies Using IaC?

    There are three main reasons why companies may not yet be embracing IaC.

    Many industries and engineers have not fully embraced or adapted to the paradigm shift of infrastructure APIs and code

    A lot of us think about infrastructure as touchable, involving physical installation and cabling. It’s only relatively recently in enterprise IT years that the industry has adopted APIs that allow borrowing and configuring infrastructure on-demand without the physicality. Engineers are good at standing up infrastructure using a time-tested, battle-hardened process. Incremental learning and integrating IaC into processes and culture helps to digest an otherwise daunting and challenging project.

    Industries are hard at work adapting security controls and regulations to incorporate the opportunity of cloud and automation without compromising high standards for privacy, safety and security. Engineers using IaC to improve their own efficiency can help champion its use in wider circles. One such example of this success is the 18F office which helps improve how the government serves the public through technology.

    IaC may feel like a loss of security or control

    Increasing the awareness of how and why infrastructure is configured can invite questions or feedback that initially may feel uncomfortable to those who have made infrastructure decisions. The ease with which others can make changes to infrastructure may feel dangerous or devaluing to engineers who have always pushed those buttons.

    IaC provides an opportunity for improving single human points of failure and knowledge bottlenecks without jeopardizing necessary controls. Access to infrastructure code and IaC tools can still match organizational structure, culture and approval processes. Infrastructure engineers remain integral to its evolution and success, and they are more available to tackle next-level problem-solving.

    ‘We move too quickly to take the time to ramp up with IaC, and we’re fast at clicking in user interfaces’

    The repeatability achieved by IaC means artisanally created infrastructure does not need to be kept around because it may be needed next month and parallel infrastructure can be created for testing and destroyed tomorrow. Start using IaC for something small such as spinning up that occasional temporary VM, then create a limited-access service account that someone else can use along with your IaC when they need that VM. Your IaC efficiency will grow with your organization’s body of IaC knowledge and code.

    Kubernetes and IaC

    Using IaC with Kubernetes helps standardize Kubernetes cluster configuration and manage add-ons. Infrastructure now can be versioned and committed to a Git repository, and it can more easily be “stood up” elsewhere when the company needs to expand to new regions or clouds. IaC benefits are vast and include:

    • Reduced human error and increased predictability: Create new test environments without impacting production and make changes across multiple environments using code review processes. Focus and attention to detail are less impacted by repetitive manual work.
    • Consistency: The repeatability of IaC with Kubernetes allows for the rapid creation of consistent infrastructure. This frees up time to address other challenges.
    • Improved changed tracking and auditability: When infrastructure is represented in a versioned repository such as Git, commits to the Git repository can reflect who, when and why changes were made. The code provides a record of how environments were built and evolved.
    • Faster speed to recovery: The reliability of an application is impacted by the ability to pivot and the speed to redeploy. Infrastructure as code can help you know what the recovery process looks like and practice that process more often.

    Start Using IaC and Kubernetes

    As a company moves into the world of containers, cloud-native apps and Kubernetes, IaC will help teams improve application reliability and stability, streamline development and operations and support a better user experience.

    In a Kubernetes environment, reliability becomes much easier to achieve with IaC because teams are deploying standard configurations. DevOps teams can use IaC to embrace the ephemeral nature of containers and Kubernetes pods, as well as avoid single points of failure.

  • 5 Capabilities Network Infrastructure Must Have to Support DevOps

    5 Capabilities Network Infrastructure Must Have to Support DevOps

    According to the Accelerate 2019 State of DevOps report, high-performing DevOps organizations achieve over 200 times more frequent deployments and over 2,600 times faster recovery times than low performers. These outcomes have been realized by both forward-leaning organizations such as Amazon, which deploys code every 11.6 seconds, and more traditional enterprises that have been in existence for a long time. For instance, Nordstrom, a well-known American retailer, increased the release frequency for its mobile app from twice per year to monthly.

    Embracing DevOps results in significant advantages, including improved team collaboration, cooperation and productivity. Companies can get digital products, applications and services to market faster because of the higher deployment and release frequency. Because development and operations teams are not operating in silos, unaware of each other’s constraints, there are lower failure rates within production environments. Cooperation leads to faster recovery times when there are issues with deployment.

    To achieve DevOps success, teams must engage in a new approach to network infrastructure, and the network infrastructure must have the following five attributes.

    Automation

    Automating repetitive tasks, such as configuring DNS or a firewall, saves a lot of time and effort. It eliminates manual errors by standardizing deployment and configuration of network infrastructure. Automation can be achieved by using Infrastructure as Code (IaC), which entails using an application programming interface (API) to create, modify and shut down infrastructure. This ensures that a repeatable, consistent configuration can be created. This code can be version-controlled, which serves as a form of documentation—it’s easy to detect what changes were made by whom in case there’s an issue—and rollback to a stable state is quick and easy.

    Standardization puts an end to “It worked in my environment” or “It worked on my machine” as any issues can be easily reproduced, resulting in faster resolution times. Support for IaC tools, such as Terraform, is essential for network infrastructure to advance DevOps practices.

    Continuous Integration/Continuous Delivery (CI/CD)

    CI is a coding best practice wherein code is checked into version control repositories frequently. It provides a consistent and automated way to build, package and test applications. CD picks up where CI ends and deploys applications in an automated fashion in a variety of environments, including production, development, test and sandbox.

    There are a variety of tools, such as Jenkins and Ansible, that DevOps teams use for CI/CD integration. It is imperative for network infrastructure to integrate with these tools to accelerate deployment velocity.

    Elasticity

    DevOps teams spin up and spin down test/development/sandbox environments routinely. Infrastructure may also be needed for short periods to account for seasonality or for handling sudden surges in traffic and demand. DevOps-friendly network infrastructure must be elastic to support these activities.

    Appliance-based vendors who provide network services are not scalable as they impose artificial limits on their usage (such as queries per second or leases per second) and require organizations to anticipate their needs in advance. When there is a need, organizations have to spend time procuring these appliances, waiting to receive them and, finally, deploying and managing them—a very inefficient way to manage demand.

    Support for Containers

    Containers are the foundation for developing cloud-native applications. Containers allow developers to package up application code and all its dependencies, including libraries and other binaries, as well as configuration files needed for running the application. This enables applications to be executed quickly and reliably across a variety of environments.

    Containers provide flexibility and portability of applications across public, private, hybrid cloud and even bare-metal environments. They are lightweight, as all containers share the underlying machine’s OS system kernel. They also lend themselves to greater modularity, which is essential for developing microservices-based applications.

    Instead of a monolithic application consisting of a three-tiered architecture, a microservices-based application consists of several smaller services that perform one function and one function only. It’s easier to scale, easier to manage and faster to develop such applications. Because containers have a small footprint, they are most suited for microservices architecture.

    In order to support containerized applications, network infrastructure must first be compatible with containers. Network infrastructure must be able to configure DNS, allocate IP addresses for containerized applications, route traffic intelligently between containers and balance loads for optimal performance. Network infrastructure must also support Kubernetes, the most widely deployed orchestration platform. To ensure that the needs of DevOps teams are met, it is important to inquire with network services vendors about compatibility with containers and Kubernetes, as many legacy, appliance-based solutions cannot support this technology. 

    Self-Service Access

    To move quickly, development teams may need to configure network infrastructure for test/development/sandbox environments. Having to submit tickets to network operations teams that use legacy appliances for delivering network services significantly slows down application development and deployment. Providing self-service access to network infrastructure, such as firewalls, load balancers, DNS and DHCP for development teams, eliminates the reliance of these teams on network operations teams. Self-service with robust role-based access controls reduces the deployment time and increases efficiency while freeing up network teams for more strategic projects.

    Conclusion

    DevOps attempts to minimize friction in application deployment by providing autonomy to development teams so that they don’t have to rely on network administrators to provide the necessary infrastructure needed for development and testing. But, the move to DevOps requires more than a consolidation of teams.

    Success requires best practices such as automation, CI/CD and self-service access to infrastructure, as well as support for containers and Kubernetes, which are key to building modern, cloud-native applications. Teams that incorporate these capabilities will see increased agility and speed that lead to shorter delivery time, higher-quality applications and improved communication and collaboration throughout a product’s lifecycle.