devopsy.kinsta.cloud

Tag: kubernetes

  • The New Era of Kubernetes, With StackRox

    The New Era of Kubernetes, With StackRox

    Orchestrators are becoming the operating systems of cloud for the future. Which one do you think has become the dominant one? Ali Golshan, co-founder and CTO of StackRox, placed his bet on Kubernetes right from the start.

    StackRox is a Kubernetes-native security platform that helps DevOps and security teams operationalize security, providing richer context, native enforcement and continuous hardening.

    During this interview with MediaOps founder and editor in chief Alan Shimel, Ali discusses the adoption of Kubernetes and its prevalence today. More companies are realizing they can’t use traditional security. The tooling has to be native to the infrastructure they’re using, it must integrate into the build and the deployment. And Kubernetes is the best solution available, he said.

    Check out the interview below to learn more about microservices, containers and Kubernetes.

    — Veronica Haggar

  • An Inside Look at GitOps

    An Inside Look at GitOps

    At a time when variations on DevOps abound, get ready for the latest: GitOps. This new term promises to bring some order to what can be operational chaos and empower operational teams to leverage best practices learned from their developer counterparts. It’s early days yet for GitOps, but here’s what we think you need to understand about this new facet of DevOps.

    What is GitOps?

    Everyone we spoke with has a slightly different perspective on GitOps, but this is our take: In its most basic form, GitOps is infrastructure and operational procedures defined as code, and living in a source control system, such as Git. Git becomes the single source of truth for what is running in production (or staging, or development).

    Declarative systems, such as Terraform, AWS CloudFormation and Kubernetes help enable this practice; but operations is much more than Infrastructure as Code. True operations requires a way to plan, review and approve moves, adds and changes to the infrastructure. It also requires well documented procedures, or runbooks, to be successful. This is where GitOps differentiates itself. We believe GitOps is the full operations lifecycle of infrastructure management, living as code in a git repository.

    GitOps may be used for managing both application deployments and base infrastructure made up of systems needed regardless of the applications deployed (i.e. networking, security, etc.). Both will follow the same procedures to plan, review, approve and release code into production. Done right, GitOps will let engineers constantly apply updates to infrastructure meet all the needs of the applications in a continuous delivery format. The version control system ensures everything is recorded and visible and an audit trail keeps teams compliant. GitOps will make it easy to revert problematic changes, becoming a single source of truth about what is happening in the system from both the software development and infrastructure perspective.

    What Technology is Required?

    In order to kick off GitOps, you’ll use a Git-based version control system. Then you’ll need the ability to create a declarative cloud infrastructure, using something such as Terraform and Kubernetes, with the help of Helm or Flux. Because both Terraform and Kubernetes are declarative systems, they make it simpler to create and deploy infrastructure as code in a manner that is both replicable and transparent. And you’ll also want to implement a solution such as GitLab that wraps operational procedures around these tools. GitLab, being an end-to-end DevOps solution can manage the backlog of requests, assign those requests to the appropriate subject matter expert to create the change in Git, facilitate the review and approval process, and kick off the execution of the changes using its flexible CI/CD system.

    While it is possible to achieve this sort of workflow by using different solutions to merge and ship code to Kubernetes, GitLab is certainly a straightforward path to GitOps. It allows developers and operations administrators to work from the same source of truth and with a common set of procedures and tools.

    What Are the Benefits?

    In the DevOps world, GitOps is perhaps unique in that it can potentially benefit both operations team members and developers. For starters, GitOps aims to address the sometimes cave-like nature of operations (a single person working largely in isolation) by taking all of the best practices from software development and letting them loose on the deployment side.

    Full oversight, architectural review, project management and all the other pieces of the process are brought together in GitOps in an effort to end ops versus everyone else and get all the stakeholders involved in the process. The basics don’t change–an ops engineer will still create an infrastructure–but that will just be one part of a much larger process that everyone can contribute to and collaborate on. No more worrying about an engineer creating a cloud infrastructure that can’t be replicated; with GitOps everything is tracked making repeatability easy. Suddenly it will also be simple to answer the “what changed?” question because no one will have to hunt for an answer. The bottom line is a successful GitOps program will benefit the actual bottom line because operational processes will be dramatically streamlined.

    Possible Challenges

    With any collaborative effort, change can be tricky and GitOps is no exception to that. It is usually easier to create one-offs and not follow a process, so the first thing to consider about GitOps is it will require discipline from all participants. It will be vital for teams to write everything down.

    At GitLab we live and die by this thanks to our culture of collaboration, but we know it’s not necessarily something that comes naturally to any individual or organization. It will be vital for everyone on the team to record what’s going on in merge requests and issues. The temptation to edit something directly in production or change something manually is going to be difficult to suppress, but the less “cowboy engineering” the better GitOps will work.

    If GitOps sounds like something your organization might benefit from, our best advice is to start small. Pick a tiny area to begin with, get a win and then build on that. At GitLab we call that iteration and it’s worked well for us.

     

    This article was co-authored by Priyanka Sharma, director of technical evangelism at GitLab.

    — Brad Downey

  • Survey Sees Cloud Foundry Momentum Gains

    Survey Sees Cloud Foundry Momentum Gains

    At a time when many organizations are wondering to what degree they should rely on legacy application development and deployment platform versus emerging container-based platforms, the Cloud Foundry Foundation (CFF) today released the results of a survey that shows reliance on the organization’s namesake open source platform-as-a-service (PaaS) environment continues to grow.

    Announced at the European Cloud Foundry Summit, the annual survey of 306 IT professionals finds 45% of respondents describe their use of the PaaS as “broad,” compared to 30% a year ago. The same survey finds nearly 20% started using Cloud Foundry in just the last 12 months and 39% of developers are deploying applications now in less than one day. More than three-quarters (77%) report application development cycles of fewer than three weeks.

    Nearly half of survey respondents report they have at least 100 developers using the PaaS, a significant jump from just one-third last year, and the share of companies with more than 50 applications on Cloud Foundry grew to 43% in 2019.
    The future of the PaaS has been called into question ever since the rise of Docker containers and Kubernetes. The CFF has been slow to adopt Kubernetes as a more efficient foundation for deploying a PaaS environment. Most recently, the CFF has provided organizations that adopt the PaaS the option of replacing the Diego container orchestration engine developed by the CFF with Kubernetes. At the same time, the CFF is working toward replacing much of the foundation layer for Cloud Foundry with Kubernetes to provide a more efficient means of standing up a Cloud Foundry environment.

    The CFF has also made it possible to deploy applications built on Cloud Foundry on a Kubernetes runtime. CFF CTO Chip Childers says that no one should be surprised when the rise of Kubernetes leads to more adoption of Cloud Foundry. Most of the application development environment available for Kubernetes today are not as robust as the Cloud Foundry PaaS, with the notable exception of the Red Hat OpenShift PaaS. Following the acquisition of Red Hat, IBM is now promoting Red Hat OpenShift as the cornerstone of a hybrid cloud strategy that has Kubernetes at its core. Meanwhile, last month Pivotal Software, provider of a curated instance of a Cloud Foundry PaaS, told financial analysts it expected to see a $40 million to $50 million decline in revenue in fiscal 2020. Shortly after, VMware moved to acquire Pivotal, which, like VMware, is a unit of Dell Technologies.

    An update to a previous report published today by source{d} shows there are now 879 repositories in the Cloud Foundry GitHub organization, with the total number of commits reaching approximately 50,000 per month. The report finds the highest levels of activity are centered around the Stratos UI project, Istio integrations and the core Cloud Foundry Application Runtime Cloud Controller.

    The CFF and Kubernetes, under the auspices the Cloud Native Computing Foundation (CNCF), fall under two separate arms of The Linux Foundation. It’s unclear if the Linux Foundation might one day merge these two arms or continue to keep them separate as a means of navigating conflicting vendor strategies. Whatever the outcome, Cloud Foundry is likely to remain a vibrant part of the open source community for many years to come.

    — Mike Vizard

  • Weighing the Cost of Improper DevSecOps

    Weighing the Cost of Improper DevSecOps

    Simply put, data breaches are terrible news for companies. And, the costs associated with such attacks continue to escalate. A recent IBM-sponsored report found an average price of 3.92 million per breach.

    Not adapting security automation and vulnerability scanning into development pipelines could have a drastic effect not only on cost but workload efficiency and team morale. With these sorts of headaches, it’s vital to consider the repercussions for not adopting secure (and lean) armaments and auditing procedures.

    Can organizations adopt DevSecOps beyond the buzzword it has become? In this article, we’ll uncover some startling evidence on the state of Kubernetes vulnerabilities and discuss what companies can do to rectify.

    Trends in Kubernetes and Istio Vulnerabilities

    It’s astounding how much in DevOps is left exploitable. Secrets are routinely unknowingly left in the open due to human error. These are the most common gateways for attackers. Plugging these gaps is the first, albeit obvious, step.

    There’s proof of an epidemic, too. A recent study conducted by Alcide ran a live production analysis of over 500 cloud-native environments across the globe. According to their findings, “89% of deployment scans show companies are not using Kubernetes’ secrets resources, with secrets wired in the open.”

    This means the majority of Kubernetes environments share susceptible to data breaches in some form. Among the 5,000 scans run, the Alcide report found secrets are commonly left open in a few areas, including:

    • Workloads that had to connect to cloud resources that exist outside the cluster
    • AWS access keys to connect to those resources
    • Tokens for workloads that require API access to external databases
    • Credentials for S3 buckets

    Gaps in Best Practices for Kubernetes and Istio

    Kubernetes brings many nuances to the table. It’s a sophisticated system, with a wide range of use cases and cloud-native experiences. These require extensive auditing and compliance checks to safeguard deployment.

    According to Gadi Naor, CTO and co-founder of Alcide, Kubernetes requires a good level of expertise. However, “there is a knowledge gap that DevOps doesn’t have,“ said Naor.

    Much of this gap lies in Kubernetes secrets handling and network policies. On top of Kubernetes, service meshes are a next evolutionary leap. Istio, an architecture layer for creating network policies is ten-fold more complex, according to Naor.

    On network policy creation, “75% of the scanned deployments use workloads, which mount high vulnerability host file systems; while none of the surveyed environments show segmentation implementation using Kubernetes’ network policies,” stated the Alcide study.

    These are compelling platforms, but best practices must meet the growing number of potential attack vectors.

    Why You Should Approach CI and CD Security Separately

    Many pundits now link continuous integration (CI) and continuous deployment (CD) together as if they’re synonymous. However, they are separate concepts. Securing them, in the same manner, could decrease overall efficiency.

    Scanning all resources created in the CI phase against security auditing could waste time and resources. For example, if you’re standing images for CVEs and only deploy 20% of those images, that is a lot of wasted exercises.

    Rather than scanning all environments in the CI phase, Naor believes you should focus on scanning 100% of images for CD. Therefore, security tooling is leaner, and anything deployed will be assuredly safe.

    Takeaways on the Cost of Poor DevSecOps

    Infrastructure components such as Kubernetes and Istio are powerful allies to the DevOps minded teams. These environments do come with their nuanced security concerns, which must be addressed. Otherwise, the data breach price tag is high.

    To intercept system frailties before the black hats do, you must consider the following:

    • Scan for the Obvious: Utilize automation to discover secrets, keys, tokens, left in public repositories.
    • Scan in a Lean Way: Maintain a lean security environment that retains your engineer’s attention.
    • CI vs CD: Recognize the security nuances between integration versus deployment.
    • Add Auditing Tooling Strategically: Adopt compliance and auditing tooling that doesn’t create friction with existing developer workflows.

    As Naor said, “Don’t ignore the obvious stuff.”

    — Bill Doerrfeld

  • DevOps Deeper Dive: VMware Transformed

    DevOps Deeper Dive: VMware Transformed

    At the VMworld 2019 conference this week, VMware in no uncertain teams made it clear it wants to become a major DevOps force.

    Following the acquisition of sister company Pivotal Software last week, VMware is moving to build out an application development framework for building and deploying cloud-native applications based on Kubernetes. That effort coincides with ongoing investments in its vRealize Suite of management tools, through which VMware is committed to automating IT operations across both on-premises IT environments and public cloud computing environments.

    To advance those twin goals, VMware this week launched a series of Tanzu management tools for managing any distribution of Kubernetes while at the same time launching VMware hybrid cloud platform, which combines everything—from vRealize automation tools to the monitoring tools from Wavefront by VMware—to create a central console through which all of IT can be managed.

    [su_box title=”The VMware DevOps Timeline”]

    • October 2015 – Dell Acquires VMware along with EMC
    • April 2017 – VMware acquires Wavefront monitoring service
    • August 2017 – VMware shifts public cloud strategy via Amazon Web Services (AWS) alliance; VMware and sister company Pivotal Software unveil Kubernetes distribution
    • November 2017 – VMware Launches VRealize Management Suite
    • February 2018 – VMware acquires CloudCoreo to gain cloud configuration tools
    • August 2018 – VMware acquires CloudHealth Technologies to gain governance tools; VMware launches AppDefense to provide a zero-trust framework
    • November 2018 – VMware acquires Heptio for Kubernetes expertise
    • December 2018 – Dell acquires all VMware outstanding shares
    • May 2019 – VMware Acquires Bitnami for application packaging tools
    • July 2019 – VMware Acquires Avi Networks to gain Web Application Firewall (WAF); VMware Acquires Bitfusion for machine learning algorithms and GPU virtualization software
    • August 2019 – VMware Acquires Pivotal Software; VMware Acquires Carbon Black; VMware Launches Tanzu Platform to manage Kubernetes; VMware Unveils Hybrid Cloud Platform

    [/su_box]

    After deciding to launch a series of services through which it manages instances of its software on public clouds such as Amazon Web Services (AWS) and local servers from sister company Dell EMC, VMware has been slowly expanding its reach into the realm of applications. The most significant element of that thrust to date has been its acquisition of Pivotal, a provider of the widely used Spring framework based on Java and a platform-as-a-service (PaaS) environment based on open source software from the Cloud Foundry Foundation (CFF).

    The CFF is in the process of migrating the application development environment created for Cloud Foundry to Kubernetes, a cloud-native container orchestration engine being developed under the auspices of the Cloud Native Computing Foundation (CNCF). Both the CFF and the CNCF are arms of the Linux Foundation. The rise of Kubernetes has resulted in many organizations shifting their application development strategies to a platform capable of supporting multiple application development environments.

    Prior to announcing the deal to be acquired by VMware, Pivotal cut its revenue forecast for fiscal 2020 by $40 to $50 million.

    VMware DevOps
    VMware CEO Pat Gelsinger

    Pivotal Software spun out of VMware in 2013, so to a certain degree VMware’s move to acquire Pivotal represents something of a homecoming that last week was valued at $2.7 billion. Pivotal currently has about 350 customers, many of which are hosting a distribution of the Cloud Foundry PaaS on VMware platforms. VMware also tapped Pivotal to jointly develop a curated instance of Kubernetes, dubbed Pivotal Container Service (PKS), that is optimized for VMware platform. As part of that effort, VMware is also developing an instance of VMware vSphere that will run Kubernetes natively.

    At the VMworld 2019 conference, VMware CEO Pat Gelsinger made it clear VMware plans to leverage Kubernetes to bridge the historical divide between developers and IT operations teams. In fact, he told attendees Kubernetes was the most important technology to emerge since the virtual machine was created. As part of that effort, Gelsinger noted that the acquisition of Pivotal coupled with the recent acquisition of Bitnami, a provider of tools for packaging applications, means there are now more 5 million developers participating in the VMware ecosystem.

    VMware is clearly hoping the developer ecosystem will expand as VMware makes it easier to launch virtual machines on every major public cloud as well as on-premises IT environments. VMware has faced a major challenge over the last several years as DevOps teams have bypassed internal IT teams to deploy applications on cloud services based on alternative virtual machines by AWS or Microsoft. Thanks to the inherent portability of Kubernetes, VMware is hoping organizations will now deploy cloud-native applications on VMware platforms. It remains to be seen, however, to what degree DevOps teams will embrace VMware on public clouds versus continuing to employ those virtual machines.

    When it comes to cloud-native computing, VMware is clearly playing catchup. VMware has 250 PKS customers today, while rivals such as Red Hat (now an arm of IBM) have already redeployed their application development and deployment platforms on Kubernetes as an alternative to the Cloud Foundry PaaS. On the plus side, VMware now claims to be the third-largest contributor to Kubernetes, thanks in part to its acquisition of Heptio, a provider of Kubernetes tools.

    In the meantime, there’s no question VMware is at the forefront of developing tools for automating the management and securing of IT infrastructure running its virtual machine software. Less clear is to what degree VMware will be able to translate that success to also unify application development and IT operations across multiple platforms a truly meaningful DevOps way, rather than merely bundling a suite of disparate tools together at a discount.

    — Mike Vizard

  • Morpheus Data Simplifies Management of Ansible and Kubernetes

    Morpheus Data Simplifies Management of Ansible and Kubernetes

    Morpheus Data is moving to make it easier to adopt and manage both Ansible and Kubernetes. Its Morpheus 4.0 update, announced today, includes a distribution of Kubernetes as well as tighter integration with the open source Ansible automation framework.

    Brad Parks, vice president of marketing and business development for Morpheus Data, said in the case of Ansible, Morpheus 4.0 will eliminate the need for some organizations to employ Ansible Tower, a web-based console and application programming interface (API) used to manage instances of Ansible. The goal is to make it easier for organizations to take advantage of the declarative Ansible automation framework without having to master all the low-level minutiae currently required to deploy and manage it, he said.

    Morpheus 4.0 also makes it possible to view real-time job status updates and detailed log output and execute phased-based and operational workflows using multiple playbooks and brings governance, auditing and reporting tools to Ansible. Morpheus 4.0 also makes it possible to configure Ansible to run over the Morpheus secure agent communication bus, allowing playbooks to be applied to instances where SSH/WinRM access may not be feasible due to security constraints. Available on both Linux and Windows platforms, Ansible also can employ that bus to query secrets securely from the Morpheus Cypher service for storing application secrets.

    In a similar vein, Morpheus Data is also moving to automate the deployment and management of Kubernetes, an open source container orchestration engine developed under the auspices of the Cloud Native Computing Foundation (CNCF).

    By including a distribution of Kubernetes in Morpheus 4.0 that has been certified by the CNCF, Parks said Morpheus Data now makes it possible to federate the management of Kubernetes clusters across a hybrid cloud computing environment. DevOps teams can employ either the distribution of Kubernetes provided by Morpheus Data or Morpheus to provision instances of Kubernetes services provided by public cloud service providers such as Amazon Web Services (AWS), Microsoft and Google.

    Morpheus 4.0 also makes it possible to securely share and govern resources using role-based access controls, optimize resource allocation using analytic tools and audit, monitor and manage logging. DevOps teams also have the choice of provisioning applications using either Morpheus or the Helm templates made available by the CNCF.

    As Kubernetes moves into the enterprise, Parks said IT organizations will need a management framework that can automate the provisioning of application stacks on any combination of legacy virtual machines, Kubernetes clusters and bare-metal servers. Rather than be required to acquire and master separate tools for each platform, Morpheus 4.0 provides a management platform that allows a single DevOps team to manage multiple platforms more easily, he said. As an enterprisewide orchestration and automation platform, Morpheus is designed from the ground up to eliminate the need for DevOps teams to create complex playbooks by providing out-of-the-box integration with wide a variety of IT platforms, Parks added, noting Morpheus 4.0 also now includes Morpheus Jobs, a tool through which jobs can be scheduled or run ad-hoc using Bash, PowerShell, HTTP, Ansible, Chef, Puppet, Groovy, Python, jRuby or JavaScript.

    The number of organizations attempting to unify the management of multiple platforms is expected to increase significantly in the months and years ahead as the number of cloud platforms being employed increases. The challenge those organizations will face involves not only what tools to employ to achieve that goal, but also defining the DevOps processes capable of spanning a hybrid cloud computing environment.

    — Mike Vizard

  • Pivotal Embraces Spinnaker for Continous Delivery

    Pivotal Embraces Spinnaker for Continous Delivery

    Pivotal Software has added support for the Spinnaker continuous delivery platform to its platform-as-a-service (PaaS) environment for the open source Cloud Foundry PaaS.

    Richard Seroter, vice president of product marketing at Pivotal, said version 2.6 of Pivotal Cloud Foundry (PCF) extends the reach of the Spinnaker CD platform developed by Netflix to its PaaS environment and Enterprise PKS, an instance of Kubernetes curated by Pivotal and its sister company, VMware.

    At the same time, the latest version of PAS adds support for a beta instance of custom sidecar processes, which enable developers to embed additional functions running in an isolated container directly within their applications. That capability makes it possible to deploy, for example, a service mesh within the application, said Seroter.

    Finally, Pivotal plans to add multicenter replication capabilities for MySQL databases and a permissions model to Concourse for PCF, a continuous integration tool for building application pipelines with a Pivotal CF environment. That latter capability provides a mechanism through which DevOps teams can automate application updates, including patches, end to end, Seroter noted. As more patches are released to address the growing number of critical flaws involving microprocessors that make use of hyperthreading, the ability to automate their deployment is now more critical than ever, he said.

    Pivotal is seeing the deployment of both traditional 12-factor applications and more modern containerized applications on its PaaS environment. At the same time, the Cloud Foundry Foundation, which oversees the development of the Cloud Foundry PaaS, has,  via Project Eirini, made it possible to deploy containers that rely on Kubernetes rather than the Diego orchestration platform developed for Cloud Foundry. However, Eirini is still an alpha project and Pivotal has not yet fully committed to a specific level of support for Kubernetes with its overall PaaS strategy.

    In the meantime, Seroter said PCF continues to gain momentum in organizations looking to build and deploy software much like a technology company. As more organizations realize how dependent they are on software to drive revenue, they want to rely on a platform that makes it easier for them to manage software development and deployment platforms at scale, he noted.

    As part of that effort, Pivotal Software is aggressively incorporating tools that will enable organizations using its PaaS to implement best DevOps process. Many have adopted various modern development platforms in isolation but continue to rely on the Cloud Foundry PaaS to automate the building and deployment of mission-critical applications. Now, two classes of development platforms are merging into a single entity that makes it easier to embrace agile development methodologies using a lightweight development platform.

    Whatever the outcome, PaaS environments are not going away anytime soon. However, what organizations perceive as the role a PaaS environment plays in any enterprise soon might expand considerably as more container technologies are infused into the platform.

    — Mike Vizard

  • DevOps Chat: Chaos Testing With xMatters

    DevOps Chat: Chaos Testing With xMatters

    Introducing chaotic, unpredictable test software into your methodical testing regime is a good idea, right? Yes, it’s a branch of testing called Chaos Engineering, or Chaos Testing. Netflix’s Chaos Monkey famously introduced many of us to the idea that resilient systems, networks and software become more resilient and less brittle if we use chaotic testing methods to find their weak points before customers do.

    An innovative engineer at xMatters launched a new open source chaos testing tool named after H.P. Lovecraft’s nightmarish character Cthulhu. Cthulhu is designed to test across multiple cloud providers, initially supporting Google Cloud with plans to support Amazon Web Services. It’s open source, free to use, looking for more contributors, and is available on GitHub.

    We are joined on this DevOps Chats by Tobias Dunn-Krahn, CTO, and Gabrielle Gasse, lead engineer on the Cthulhu open source project, both at xMatters.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley, with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Tobias Dunn-Krahn, who is CTO at xMatters, and also Gabrielle Gasse who is lead engineer of a software project we’re gonna talk about called Cthulhu, and that’s our topic today is Cthulhu.

    Now, if you’ve read the H. P. Lovecraft books, you know at least the fictional character that we’re talking about, but we’re gonna get into this software and find out a little bit more about what it does.

    Tobias, Gabrielle—welcome to DevOps Chat.

    Tobias Dunn-Krahn: Thank you.

    Ashley: Well, let’s start by introducing yourselves. How about you, Tobias, if you would start, just introduce—tell us a little bit about yourself, what xMatters does, what you do there as CTO.

    Dunn-Krahn: Sure. So, right, my name is Tobias Dunn-Krahn, CTO at xMatters. What that entails is purview for development operations, quality and product strategy. So, that’s my role.

    What does xMatters do? xMatters is a digital services availability platform. What that means in a practical sense is that xMatters helps teams that are responsible for digital services provide a high level of up time and reliability for those services. So, what that means is filtering unwanted signals for those services, engaging resolvers when those signals are relevant, facilitating collaboration during the resolution of an incident and, as well, integrating tools in a tool chain to eliminate any manual effort that’s involved in resolving an incident so that it can be done in a very timely fashion.

    So, that’s what xMatters does in a nutshell, and what I do there.

    Ashley: Interesting, yeah. I’m interested to see how Cthulhu fits into that. Gabrielle, would you introduce yourself and tell us a little bit about what you do at xMatters?

    Gabrielle Gasse: Yeah, of course. So, I’m the lead engineer working on Cthulhu, our chaos testing tool. On a day to day basis, I’m also just a Java developer working on the system itself.

    When I came into the company about a year ago, I was tasked to verify the resiliency of our system as we moved it to Google Cloud. And so, I did some research on existing tools and realized that we need something a bit different that was available already. And so, that’s how I came to start building Cthulhu.

    As—this is not the core product of xMatters, and we do want to promote the use of chaos engineering as a day to day practice for everyone, we decided to release the tool itself in open source.

    Ashley: Great. Maybe we should start with that, for folks that don’t know what chaos engineering or the theory behind it is, do you wanna say a little bit about that, Gabrielle?

    Gasse: Of course. So, the idea behind chaos engineering, if you think of a distributed system made of multiple microservices, you may end up with a fairly large amount of small VMs or containers running in parallel. And, as things are deployed in the cloud, sometimes failure happens. It always happens.

    And so, chaos engineering takes that as a premise, and it says that if you expect to fail all the time, then you will build your software with resilience in mind so that failure is not a problem. And so, this is where the idea of chaos engineering comes from. And so, a tool like Cthulhu is something that will run in our system that introduced outages so that we can verify that our system detects those issues and in some case—in most cases, ideally—recover, programmatically or thematically from those failures so that we don’t need to page one of our engineers at 3 a.m.

    Ashley: Tobias, is this a project or an idea that you came up with or you and Gabrielle did it together, Gabrielle came to you, or how did this all get started at xMatters?

    Dunn-Krahn: No, I will claim no credit for this project. [Laughter] I will hand that completely to Gabrielle. But another way of looking at that is, as we decomposed our monolith over time into microservices, what we really wanted to do was reduce the scope of subject matter expertise and knowledge amongst our teams to focus that on a smaller part of our system. So, that reduces complexity for individual teams and being able to keep those services up and running and highly reliable.

    But one of the things that it also introduces is, it chases that complexity into the space between the services—different failure modes of those services, different complexities around the dependencies.

    So, during our digital transformation we recognized that early on, and Gabrielle came up with the idea of introducing chaos testing for this and also that the existing chaos testing tools were not suited for our purposes, and I’ll let her explain how Cthulhu was different.

    Ashley: Okay, great. Gabrielle, why don’t you pick it up from there?

    Gasse: Yes. So, a year ago, when we started working on chaos engineering, there was a few tools that was available. Chaos Monkey is a very popular one. At the time, Chaos Monkey was working for Amazon Web Service primarily, and also it’s in deployment in Spinnaker. But it didn’t work for us, and so, we’re on Google Cloud.

    We came up with this tool that can connect to lots of different cloud services—currently, we support Google Cloud and Kubernetes, but we’re adding other services like Amazon Web Service, for example. And so, it can run those failure scenarios in concert together, and also, we wanted to have the ability to version control those tests so that when we find a particular scenario that does cause a failure, we could have a way to produce it as a file, if you like, that we could then attach to a bug that we could give to the engineer to work on.

    Ashley: Let me ask, then, you mentioned Chaos Monkey, which to me, was sort of a natural comparison for what you’re doing—does Cthulhu run in the background, kinda running all the time, creating its havoc, much like Chaos Monkey does of getting things to go down and see if they recover in a production environment? Is it similar for Cthulhu or are you taking a little bit of a different approach?

    Gasse: It is similar, although there is a different way to run it. In usual scenarios at xMatters, we are running it on a need basis. So, we have a small scenario that we’ll run immediately. It’s also possible to run Cthulhu, as you mentioned, in the background and it will pick up tests and run them, essentially.

    Ashley: Mm-hmm. So, run it in a targeted control manner, or let it run in the background and do its thing, it can go either way?

    Gasse: Yeah, exactly. The thought was very much to have this ability to run it constantly like Chaos Monkey, but also give a tool for an engineer to perform a scenario locally or in a development environment to try and reproduce and understand failures.

    Ashley: And you mentioned, of course, Chaos Monkey was created by Amazon Web Services, functions very well within an AWS environment—was multiple cloud services the main differentiator for Cthulhu, or was also getting into Kubernetes, Docker, Containers something that you kinda took on as a special added functionality that Cthulhu does that maybe another tool doesn’t? What are all of the differentiators that you’ve created Cthulhu for?

    Gasse: Chaos Monkey was actually created by Netflix, who have their stack on Amazon Web Services.

    Ashley: Oh, you’re right. Yeah, my mistake—thanks for correcting me.

    Gasse: But, as you mentioned, Chaos Monkey works on a virtual machine in Amazon Web Service exclusively. There are other tools like Pumba that works at the Docker Container level. Powerful Seal came out last year around the time that we were looking at that, and Powerful Seal works on Kubernetes deployment at the pod, but also at the node level, which is pretty interesting.

    But there’s no one system that will do—that will test across all of those platforms. You need to use Powerful Seal in concert with Pumba, and there was nothing that was running directly on Google Cloud at the time. Now, Chaos Monkey does support Google Cloud.

    So, the main value that Cthulhu brought for us was the ability to have a single tool that could perform complex failure scenarios across both our Kubernetes deployments and our VMs. And—yeah, and we wanted, also, the ability, as I mentioned earlier, to version control some of those tests so that we could pass them on to our developer where Chaos Monkey and those other tools, you run them with some parameters to filter out certain machines that you don’t want it to break. And then it just rolls from there. There’s nothing—there’s no way to build a scenario from it that is repeatable.

    Part of what we test with it is that our system is able to detect failures. So, as we run a scenario, we’re expecting errors to be logged in Splunk alerts to be sent out to us. And so, the tool itself simply introduced failures, and then it’s up to the engineer to look for what is broken.

    Ashley: Mm-hmm, mm-hmm. Very good. Tobias, how does Cthulhu fit into the strategy of xMatters?

    Dunn-Krahn: So, we would—you know, our personal experience or our corporate experience of going through our own digital transformation was—and a continuing process, I don’t wanna make it sound like we’re done, we’re never done. But it was very instructive, and we learned a lot, and we’d like to share that. And many of our customers are going through the same thing, so we would like to provide them the tools that they need to be successful in these transformations.

    So, Cthulhu in particular aids in the transition process more than what you would—the situation you’d be in if you were a Cloud Native company. So, we wanted to provide that support to our customers and to the community at large. We have long been  users of open source software, and we wanted to contribute back to the community as well as get all the advantages of having an extended developer community contributing to Cthulhu and we can all make it better together.

    Ashley: Great. So, Cthulhu is an open source project. Do you have it hosted at GitHub or your own servers? Where do people find it?

    Dunn-Krahn: Gabrielle, you can confirm, but I believe it’s on GitHub.

    Gasse: That’s correct, yes.

    Ashley: Great. You know, sometimes open source projects are open, but largely, most of the work is done by a lead developer or someone who created the idea. Sometimes, it’s a very vibrant community where people are testing or creating new features. You know, and part of that is also not only the style of the project but also the maturity of it.

    Where are you in that? Is it mostly you, Gabrielle, who’s doing the work? Do you have—what kinda activities do you have from the outside from other people?

    Gasse: Yeah, the publication of Cthulhu on GitHub is fairly recent. We’ve had a fair amount of people local to Victoria when we promoted it so far that are following the project, but we have yet to see people who are really picking up or starting the contributing modules for it.

    Ashley: Mm-hmm.

    Gasse: I think one of the next additions that will really help motivate people to use Cthulhu but also contribute additional modules will be us supporting Amazon Web Services in particular.

    Ashley: Mm-hmm. And how long has it been out? I should’ve asked that originally.

    Gasse: It’s been out for a few months, only.

    Ashley: Okay. So, still pretty early in its life. So, it’ll be interesting to see, as more people use it, that’s usually where someone will get interested and say, “Well, I wanna put some stuff in here. I wanna try to make some changes and contribute some things.” What’s Cthulhu written in?

    Gasse: Cthulhu is written in Java. We’re using the Spring Boot platform, and as part of that, there is an easy way to write modules that plugs into the platform that adds functionalities. So, it’s easy to add new ways to break things in Cthulhu without having to understand the entire code base.

    Ashley: What are other things that we should know about Cthulhu?

    Gasse: I can talk briefly about the next item on the roadmap, if—

    Ashley: That’d be great, yeah.

    Gasse: —yeah.

    Ashley: Sure—love to hear that.

    Gasse: Mm-hmm. So, after the support for AWS is done, my next task, if you like, is to start adding a functionality to run commands through SSH and sending files through SCP on target hosts. And that opens the door to a whole new range of tests. For example, one could pause a process on the target virtual machine to simulate a dead live or use tools like Traffic Control to introduce noise on the network, lost packets and all that.

    And so, once we have that support, we’ll really be able to have lots of interesting types of failure that go beyond simply just, a virtual machine has shut down and is no longer available.

    Ashley: Tobias, I appreciate very much you handing all the credit where credit is due to Gabrielle. Also appreciate your thoughts on where do you see this going from your perspective, either strategically or aligned with product plans, product strategy—what do you see as the future for Cthulhu?

    Dunn-Krahn: Sure. Well, as I mentioned earlier, the xMatters product is built to help teams support digital services, and part of that is making sure not only that you plan for an incident but that you can simulate real world incidents and do some form of practice or maneuvers.

    So, the way I see this being integrated into the product in the future is facilitating those types of practices and then evaluating the performance of the team or that all of the configuration in xMatters is correct to solve problems as quickly as possible, et cetera.

    Ashley: Excellent. Well, you know, we’ve probably just barely scratched the surface, and I’m excited to see what you all do and what the community does with Cthulhu. I’d love to thank you both for being on the podcast. I thank Tobias Dunn-Krahn and also Gabrielle Gasse from xMatters, any additional information might be available at xMatters, is that true, Tobias?

    Dunn-Krahn:  That’s a good question. Yes, there’s certainly some material on the website, or you can search for it on the web if you can figure out how to spell it.

    Ashley: [Laughter] Well, let me give everybody a head start, it’s C-T-H-U-L-H-U, so, just the way H. P. Lovecraft spelled it. Well, I’d like to thank both of you, Tobias and Gabrielle, for joining us today on the podcast. Time has flown by again, and hopefully, we can have you back another time as this evolves and you can tell us some more stories about where you’ve taken it and what people have done with Cthulhu.

    Dunn-Krahn: Alright. Thanks very much for having us.

    Ashley: You bet. Thank you. This is Mitch Ashley with devopsy.kinsta.cloud and you’ve listened to another DevOps Chat.

    — Mitchell Ashley

  • Agile Stacks Extends Reach of DevOps Automation Platform

    Agile Stacks Extends Reach of DevOps Automation Platform

    Agile Stacks this week announced it is extending the reach of its namesake DevOps automation platform to Kubernetes and additional cloud services.

    Company CEO John Mathon said that by extending the reach of the company’s platform to include Google Cloud Platform (GCP), Microsoft Azure and Red Hat’s OpenShift distribution of Kubernetes alongside existing support for Amazon Web Services (AWS), Agile Stacks will make it easier for organizations to implement best DevOps practices across hybrid cloud computing environments.

    Agile Stacks made the announcement at the HPE Discover 2019 conference. The company counts Hewlett-Packard Enterprise (HPE) among its investors, and at the conference this week, HPE this week outlined how Kubernetes is core to the company’s evolving hybrid cloud computing strategy.

    Mathon said the Agile Stacks platform provides a framework for automating the creation of DevOps pipelines across multiple tools and continuous integration/continuous deployment (CI/CD) platforms. In all, there are more than 80 DevOps tools and platforms available in the Agile Stacks Service Catalog, including Kubeflow, Jenkins, GitLab, Spinnaker, Harbor, Minio, Ceph and Portworx.

    At the core of the Agile Stacks platform is a SuperHub automation hub, which accelerates software delivery based using a set of custom composable stacks that can be configured by the DevOps team. Each component of a stack is auto-configured to deploy, upgrade, rollback, promote, clone, backup, generate test scripts, perform merges, conduct performance testing, scan for vulnerabilities and execute functional testing.

    In effect, Mathon said Agile Stacks is essentially automating the DevOps automation. Organizations that have embraced best DevOps practices tend to rely on multiple tools from different vendors, which inevitably leads to silos of automation that often work at cross purposes. Agile Stacks provides an extensible GitOps framework for integrating all those tools that enable organization to evolve more easily as their DevOps processes mature, added Mathon.

    In addition, he said Agile Stacks expects to play a role in making DevOps more generally accessible by reducing the complexity organizations initially encounter when trying to embrace DevOps practices.

    As hybrid cloud computing starts to become more commonplace, thanks in part to the rise of Kubernetes, Mathon said DevOps complexity is only likely to increase. Organizations have made it clear they want to leverage multiple clouds alongside their existing on-premises IT environment. Right now, however, the DevOps processes associated with each of those platforms is, at best, fragmented. In addition to making it easier to unify DevOps processes across multiple platforms, Mathon noted Agile Stacks can also enable organizations to keep better track of cloud spending by applying automated cloud tagging via Superhub.

    Of course, most organizations are challenged automating the development and delivery of software on a single platform. However, regardless of how far down the path any organization is toward DevOps, the business is moving forward. In many cases, the rise of hybrid cloud computing will finally force many organizations to implement a set of best DevOps process enterprisewide. As part of that process, organizations will look to standardize on a specific set of DevOps tools whenever possible, or at least until developers discover the latest DevOps tool to supplant all others.

    — Mike Vizard

  • HPE Extends Hybrid Cloud Computing Strategy

    HPE Extends Hybrid Cloud Computing Strategy

    Hewlett-Packard Enterprise (HPE) today advanced its hybrid cloud computing strategy by announcing at its HPE Discover 2019 conference an alliance with Google to embrace Kubernetes in hybrid cloud computing environments and then extending the reach of its software-defined infrastructure for building private clouds to legacy ProLiant servers.

    Lauren Whitehouse, director of marketing for the software-defined and cloud group at HPE, said the HPE Composable Cloud now extends beyond new ProLiant servers to include everything from legacy hardware to cloud computing platforms running virtual machines or as bare metal servers optimized for Kubernetes.

    To achieve that latter goal, HPE will make it possible to deploy Google Anthos, a previously announced framework for delivering cloud-native computing services across public clouds and on-premises IT environments, on HPE-managed platforms.

    HPE is giving customers the option of deploying configuring local servers with virtual machines with either a VMware vSAN storage software or the HPE SimpliVity hyperconverged infrastructure platform. Additionally, customers can choose to deploy Kubernetes on those platforms or run Kubernetes on a bare-metal server. On top of those stacks, HPE also embeds the HPE OneView IT infrastructure management and HPE Composable Fabric networking solution to create what the company calls a composable rack environment.

    Finally, HPE announced it will extend the reach of the HPE InfoSight artificial intelligence engine, which it gained for storage environments when it acquired Nimble Storage, to the rest of its data center portfolio.

    Whitehouse said that, by extending the reach of its composable computing strategy to include legacy servers, HPE seeks to make it easier to modernize on-premises IT environments within the context of a larger hybrid cloud computing strategy.

    One of the primary reasons so many workloads are being deployed in the cloud is because of the inherent flexibility of the IT environment. HPE is investing in higher levels of infrastructure abstractions to enable an equivalent amount of agility to be injected into on-premises IT environments. Those efforts will then make it easier for organizations to extend best DevOps practices across a hybrid cloud computing environment, said Whitehouse. The company is now removing the objection by organizations to have to implement a forklift upgrade across their entire on-premises IT environment to achieve that goal, she added.

    It’s not immediately clear who will manage what in this brave new world of hybrid cloud computing. Application developers have been exercising a lot more influence over platforms, while advances in automation appear to be heralding a consolidation of management functions within the local data center. As site reliability engineers, or so-called “super administrators” capable of managing compute, storage and networking infrastructure continue to emerge, the silos around job functions in the enterprise were built are breaking down. In fact, the more organizations embrace DevOps best practices, the faster consolidation of job functions is occurring.

    In the meantime, whoever provides the infrastructure that remains in an on-premises IT environment had better be able to enable local IT teams to be just as flexible as any cloud service provider an application developer is likely to encounter.

    — Mike Vizard