devopsy.kinsta.cloud

Tag: KubeCon

  • The Inevitability of Multi-Cloud-Native Apps

    The Inevitability of Multi-Cloud-Native Apps

    We all know the story: Enterprises moved to the cloud. They learned about data center virtualization and multi-tenancy and self-service developer provisioning. Along came Kubernetes, and they moved to cloud-native, where they learned about automation, DevOps and infrastructure-as-code (IaC). Now, even though most enterprises are still figuring out how to do cloud-native at scale, there’s a new imperative they must master: Multi-cloud-native applications. And for almost every enterprise with a global footprint, the move to multi-cloud-native applications is not optional—it’s inevitable. 

    But before we explore why this is the case, let’s first retrace our steps. For almost every enterprise out there, the journey to the cloud started with adopting a public or private self-service infrastructure provisioning and management model. These enterprises quickly realized that this new IT model required a new set of tools to solve challenges they’d never before faced—enabling their developers with self-service provisioning, managing and operating infrastructure. Enterprises figured it out, but as soon as they figured out the cloud, they quickly had to gear up for the second phase: Automating and operationalizing an enterprise-wide rollout of cloud operating models throughout their entire developer organization. 

    Of course, this phase has its own set of challenges, and a rich ecosystem of tools emerged to address them. Even so, we all can agree that the majority of mainstream enterprises are still in this second phase. They’ve adopted cloud, they’ve expanded its use, and they’re grappling with the challenges around multi-tenancy, security, DevSecOps and governance. Hand-in-hand with this goes a whole new set of challenges around cost management. Suddenly, enterprises are faced with managing the variable costs of public cloud infrastructure: Optimizing compute infrastructure, utilization and capacity to rein in cloud spend. It’s an issue Sarah Wang and Martin Cassado took head-on in their research on the Trillion-Dollar Paradox.

    So, enterprises moved to the cloud and are now, for the most part, successfully grappling with scaling it. But nothing stays the same for long, and businesses drive IT to innovate. This is where cloud-native entered the picture to help enterprise software development teams become more agile and help businesses transform to deliver new services and experiences faster. In fact, the consumer experience this model makes possible has rapidly become the norm.

    Consistently delivering rapid software iteration across a global footprint forces DevOps organizations to grapple with an entirely new set of technical challenges: Leveraging containerized applications and microservices architectures in production across multiple Kubernetes clusters running in multiple geographies. Customers want an on-demand experience. This third phase is what we call multi-cloud-native, and it was pioneered by hyperscale IaaS players like Google, AWS, Azure and Tencent.

    The reality is, of course, that hyperscalers aren’t the only ones who have figured out how to deliver multi-cloud-native apps. Webscale innovators like Doordash, Uber, Twitter and Netflix have done it, too. To get there, they had to make and share their multi-cloud-native apps across every geography where their customers live. And, in turn, to make that happen they had to tackle a new set of challenges: Develop new tools and techniques like geographically distributed, planet-scale databases and analytics engines, application architectures that run apps on the backend close to the consumer in a multi-cloud-native way. This enabled them to do arbitrage across fleets of infrastructure capacity across regions, all while giving their customers that instant experience they increasingly grew to expect.

    What enterprises must realize is that this experience consumers have grown to expect from webscalers is rapidly becoming the same experience they expect when accessing their bank account, their insurance apps and their tax records. For almost every planet-scale enterprise, this sets up a whole new set of challenges: To span multiple clouds and multiple geographies running multiple Kubernetes clusters, you now must worry about new ways to do autoscaling, self-healing, disaster recovery, operations management and cost optimization—all of which are things that you’re just now getting figured out for virtual machines (VMs) and clusters in single regions.

    Enterprises must face the reality that cloud-native application models are not the destination but rather a waypoint on a journey to becoming multi-cloud-native. Because the consumer experiences delivered by webscalers make this the inevitable destination.

    The Enterprise Journey: Cloud, Cloud-Native and Multi-Cloud-Native

    Don’t take my word for it.

    Uber and Twitter public disclosures describe how the multi-cloud-native model is an integral part of their strategic game plan. Hybrid and multi-cloud are part of this, too, leveraging Kubernetes as a cluster manager, but they still need tools to manage all those clusters across multiple environments in multiple regions at planet scale.

    The bottom line is that the architecture that webscale companies are using today is the same one enterprises will need to adopt tomorrow to remain competitive. Alternatively, these enterprises will cede market share and relevance to the next webscale innovator to disrupt an industry. In adopting this model of building multi-cloud-native apps, enterprises will have to deal with unified access management, controlling multiple clusters, resource management and more, all across multiple regions and clusters of capacity, aggregating versus disaggregating cluster resources, using logical services that span clouds versus namespaces that span a single cluster.

    The Enterprise Solution: Planet-Scale Cluster Manager

    They will need a planet-scale cluster manager to do this; something that can observe costs, operations, performance and SLAs across clusters, across public and private clouds, spanning VMs and Kubernetes through a single pane of glass so they can operate above the clouds in an automated, easy-to-use way. Band-aid solutions like running multiple spreadsheet models to figure out how to optimize are not scalable. What’s needed are tools like machine learning to automate and optimize at scale, bin packing technology to intelligently and automatically optimize across a planet-scale infrastructure fleet and high levels of automation either with humans in or out of the loop. That’s how hyperscalers and webscalers are already doing it.

    Every planet-scale enterprise is multi-cloud-native, running data-driven apps. It’s not a fiction: It’s already happening across webscale companies, not just hyperscalers. Multi-cloud-native is the path that every enterprise is going to need to follow if they want to transform themselves to compete with the next webscale upstart. The good news is, the pioneers have left us a map to follow.

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021

  • Why Browser and API Testing is Shifting Left

    Why Browser and API Testing is Shifting Left

    For developers working with cloud-native technologies, software testing and quality assurance is a key part of product development. For most teams, testing has “shifted left,” so the quality of the code can be assessed as it moves through the CI/CD pipeline. This approach to quality assurance reflects the rise of DevOps and other modern development practices – developers today own their applications from end-to-end, and conducting testing early in the product life cycle is how they find potential bugs, save their organizations time and money and build products that delight their customers.

    More recently, developers have added end-to-end API and browser tests into their CI/CD practices, extending the scope of their testing and quality assurance practices. By testing how code performs on different browsers and how it interacts with important data sources, developers have added another layer of quality testing to the early stages of their development processes. This has brought added benefits to other “shift left” practices, such as unit testing and security testing, and has led to a new understanding of quality assurance.

    Traditional Approaches to QA Can’t Support the Pace of the Cloud

    As companies have migrated to the cloud and adopted agile development methodologies that align with DevOps practices, they’ve encountered conflicting priorities between releasing code quickly and maintaining the highest level of quality on each release. Traditional manual testing techniques were not built for agile workflows, but instead supported older waterfall methodologies, in which testing could be conducted as a discreet instance before code went to production. With agile development, having a specific team in charge of all testing right before production is not the most sustainable approach, as it hinders the rapid pace of development that the cloud enables. This is why testing is “shifting left” to earlier stages of the development process, and why manual testing is being done by people embedded within each engineering team.

    Cloud-Native Teams Have Shifted Left Into CI/CD Pipelines

    As part of these agile methodologies, companies have moved toward leveraging CI/CD practices to ensure code is always in a state where it’s ready to be deployed. As code moves through these pipelines, developers want to ensure they are able to catch bugs earlier in the development cycle. To do this, they’ve taken to testing earlier and more often.

    In particular, developers have increasingly recognized the value of end-to-end testing throughout their CI builds. API tests allow teams to verify end-to-end workflows by chaining HTTP requests and API calls, so they can validate all the layers of their systems from worldwide locations. Browser tests capture the user’s view during critical transactions, so teams can understand how UI changes are impacting web applications. Shifting both left into CI/CD pipelines saves time and money, as fixing a bug earlier in the development lifecycle is much less expensive than trying to fix it at the last minute, when it has grown in complexity and scope.

    It also means many issues are caught by the developers, without the intervention of another engineer manually validating their work. This reduces the number of cycles required to find a bug, allowing the code to be circled back to developers, fixed and re-deployed to a testing environment. When regressions are caught early, developers can focus on the details that will make the product great.

    Visibility of Metrics, Traces and Logs Drives Value

    Running browser and API tests in your CI pipelines is beneficial, especially when teams can correlate these tests with data from across their stack – front end and back end, application and infrastructure. With correlated metrics, traces and logs that can be accessed easily without toggling between different products, teams will have the full context necessary for both detecting and troubleshooting failed tests. Detecting the failed tests allows for the rollback of bad code before it goes to production, and the correlated full stack data can be used to troubleshoot and fix any issues not caught during the development stage. This creates a new culture of quality assurance, in which maintaining quality is a holistic process that involves every part of the development cycle, and draws on data from every part of the stack.

    Consistent Quality Requires Collaboration

    This change in quality assurance goes beyond the practice of shifting browser and API testing left, and represents a fundamental change in how engineering teams collaborate within an organization. Testing happens before code goes to production, continues with robust monitoring once code is live, involves data from across the stack and is the responsibility of teams from various parts of the business. With this collaborative approach, for example, a front end engineer can easily identify an issue in the integration of their latest changes with the way the back end APIs are structured. They can then engage with the back end team owning this service to collaboratively iterate and find the best solution to resolve the issue. It may start with a shift left, but, ideally, it ends with the removal of silos and the opening of communication channels among teams.

    Automated QA Delivers Customer Value Faster

    A fully operational approach to quality assurance means that issues are resolved faster, or even resolved while code is still in development, so they never become issues at all. This creates a virtuous cycle – fewer regressions means less downtime, which makes customers happy, and frees up engineers to build new products, which makes customers happier still. That baseline of reliability and customer experience enabled by effective quality assurance is what allows for new innovation that delivers value to customers.

  • Mario’s Adventures in Tekton Land

    Mario’s Adventures in Tekton Land

    Tekton is an open source, cloud-native CI/CD pipelines engine. The project has its roots in the Knative project, where about a year ago it started under the name “Knative Pipeline,” with the aim of providing an extension to build Knative’s own “source to image” service. About eight months ago the project was reborn as Tekton and it moved out of the Knative space to its own GitHub organization and it joined the newly formed CD Foundation with a new mission: becoming the common denominator for CI/CD in the cloud-native space. But how is Tekton going to achieve its new aspirations?

    For it to be successful, we wanted Tekton to be intuitive, reliable and easy to operate and maintain. This was discussed in the Tekton working group, and we thought it would be useful if we started using Tekton ourselves as part of our day-to-day jobs, using it to provide CI/CD services to the community.

    Diving into Tekton

    Tekton pipeline aims to provide a set of components used to create a full-fledged cloud-native CI/CD system on top of Kubernetes.

    The core project of Tekton (tektoncd/pipeline) defines a set of Kubernetes custom resources (CRD) as standard constructs for creating CI/CD pipelines. The following is a brief introduction to these CRDs:

    • Task: A sequence of commands (steps) that are run in separate containers in a pod.
    • Pipeline: A collection of tasks executed in a defined order.
    • PipelineResource: Inputs (e.g. git repo) and outputs (e.g. image registry) to a pipeline.
    • TaskRun: Runtime representation of an execution of a task.
    • PipelineRun: Runtime representation of an execution of a pipeline.

    Those pipeline building blocks can be reused, version controlled and curated in a catalog (tektoncd/catalog) that embeds best practices.

    In addition, the Tekton community works on providing more components:

    • Tekton Triggers (tektoncd/triggers) to be able to act upon events (for example: GitHub webhooks, CloudEvents and more).
    • Tekton CLI (tektoncd/cli), which provides a command-line interface focused on the Tekton APIs and features (like getting the logs of your run, starting pipelines and more).
    • Tekton Dashboard (tektoncd/dashboard) which provides a simple dashboard (à-la-kube-dashboard) to view pipelines and logs from your browser.

    Re-Plumbing Test Infrastructure

    We designed Tekton to be intuitive, reliable and easy to operate and maintain. We started using Tekton ourselves as part of our day-to-day job, using it to provide CI/CD services to the community. But we didn’t start from a blank canvas.

    Tekton inherited a few features from the Knative ecosystem, including the CI setup. This was great as it allowed the project to relatively easily kick-start its CI using tools shared with several other projects in the Kubernetes ecosystem (including Kubernetes itself)–a combination of Prow, Tide, Boskos, a few tools from k8s.io/test-infra and several bash scripts contribute to tasks such as running E2E integration test.

    Deployment of the various CI services is manual and the same is true about their configuration. Releases are crafted mostly manually. Since Tekton is made of several projects, we set up tektoncd/plumbing repository to host the shared CI/CD infrastructure: prow configuration, shared test scripts and tools, test docker images. Mario adventure started!

    Mario’s Adventures

    We started to work on what we are calling the Dogfooding project, which is basically using Tekton to build Tekton. And like a ball of wool and a cat, the more we thought about it, the more questions we had. How is Tekton going to integrate with the existing prow-based CI infrastructure? How are we going to test changes to the plumbing repo in isolation? How are we going to deploy the instance of Tekton used to run Tekton CI pipelines? How are we going to do all that with no disruption to our CI service?

    Testing code continuously is essential infrastructure, even more so for cloud-native software, and re-plumbing all at once could leave us without running water. We decided to proceed with baby steps, reducing the area of impact as much as possible and starting our journey tackling the release automation work first.

    Here is an excerpt of the work we did on this journey:

    1. Run our releases using Tekton. That was the initial work, using Tekton to at least release itself, even if it gets triggered manually.
    2. Integrating Tekton and Prow. Prow is able to create Tekton resources since April of this year (#11888 if you are curious), although it took a few iterations to be able to use it on our own CI.
    3. Looking at solutions without using Prow at all. This one was (and still is) the trickiest:
      • We needed something that is able to react based on events (from GitHub), this is where the tektoncd/triggers project comes into place.
      • Prow comes with a UI and tooling to capture and store logs. This piece is missing from the Tetkon project as of today. We do have the tektoncd/dashboard project but it’s early days, there is no support for authentication yet.

    Mario’s Adventures at KubeCon

    Mario’s adventures will continue at KubeCon in San Diego. If you’re attending the conference  and you’re curious to know more, come and listen to our talk, or watch it on the CNCF youtube channel after the conference.

    This article was co-authored by Andrea Frittoli is a developer advocate at IBM and an open source enthusiast. He is a contributor to the Tekton project and he has contributed to OpenStack for several years. He’s a strong advocate for transparency in open source. Andrea has previously been a speaker at FOSSASIA, FOSS Backstage, OpenStack summits, Open Source Summits, Devoxx Days and various meetups.

    — Vincent Demeester

  • The NVMF Boogie: How Kubernetes and NVMF Will Rock

    The NVMF Boogie: How Kubernetes and NVMF Will Rock

    I recently wrote a blog about how Kubernetes creates a global computing grid that delivers unprecedented compute flexibility. Then I wrote a blog about the importance of going fully Kubernetes-native. In today’s blog I would like to drill down into the role that the NVMe over Fabrics (NVMe-oF or just NVMF) standard will play in enabling Kubernetes infrastructure flexibility.

    What is NVMe?

    NVMe is a protocol that dictates how a CPU, moves memory via the PCI bus to a storage device. NVMe communicates over a set of rings (per CPU) where commands get submitted from any CPU to the underlying NVMe device. The design of NVMe eliminates intermediate layers between the CPU and the storage device. NVMe devices consist of a controller, queues, namespaces, namespace IDs and the actual storage media with some form of an interface.

    Storage media can be grouped into sections called namespaces with an ID. In the context of NVMF, namespaces provide a way to enforce access control for the disks/consumers. Namespaces are analogous to an OS partition, except the partitioning is done in hardware by the controller and not the OS (you can still have OS partitions on namespaces). Some NVMe namespaces might be hidden from a user (e.g. for security isolation). A controller connects to a port through queues and a namespace through its namespace ID. A controller is allowed to connect to multiple namespaces and a namespace is allowed to be controlled by multiple controllers (and thus also, multiple ports).

    Imagine smearing out this NVMe device across multiple computers and you get to the next important concept, a storage fabric.

    What is NVMF?

    When you put a network between the PCI bus and the storage device, you use NVMe over Fabric (aka NVMe-oF or simply NVMF). NVMF enables fast access between hosts and storage systems over a network. Compared to iSCSI, NVMF has much lower access latency, in practice adding only a small latency difference between local and remote storage. NVMF delivers a breakthrough in throughput and seek time relative to traditional device attached storage:

    Source: Samsung.

    This is possible because NVMF uses RDMA to efficiently access the memory of another computer without using the CPU or operating system of either. RDMA is a transport protocol that bypasses the kernel stack, offloading the workload to network hardware. This means the transport layer takes place in the RDMA NIC itself, so data is transferred directly to or from application memory without having to involve the normal OS-level network stack, achieving low latency with low CPU consumption.

    Of course, NVMF performance is gated by the networking infrastructure used, whether Ethernet, Infiniband or Fibre Channel. The November 2018 final ratification of the NVMe/TCP transport specification has accelerated adoption of NVMF because it allows storage fabrics to be deployed on standard network hardware.

    NVMF can be used with white label storage systems (JBOD), feature-rich block storage systems (SAN), and storage systems built with previous generation (SATA) devices. This means data centers can simultaneously leverage low latency devices while also using more traditional storage subsystems. NVMF also supports using storage class memory as a cache in the data center compute hierarchy.

    Why is NVMF Important?

    Think of NVMe as a replacement for SCSI, and NVMF as a replacement for iSCSI and iSER. NVMF has several distinguishing technical attributes that add up to huge performance scalability at the storage infrastructure layer:

    • Protocol Efficiency: The NVMe-oF protocol is designed specifically for efficient memory transfers at scale. Designed from the ground up for scalability, the NVMe-oF protocol dramatically reduces the internal locking needed to serialize I/O and improves the efficiency of interrupt handling. NVMe-oF also supports MSI-X and interrupt steering, which prevents bottlenecking at the CPU level and enables massive scalability as systems scale. Combined with the large number of cores in modern processors, NVMe-oF delivers massive parallelism to connected devices which increases I/O throughput and lowers I/O latency.
    • Command Set Efficiency: NVMe-oF has a streamlined and simple command set that uses less than half the number of CPU instructions to process an I/O request that SAS or SATA does, providing higher IOPS per CPU instruction cycle and lower I/O latency in the host software stack. NVMe-oF also supports enterprise features such as reservations and client features such as power management, extending the improved efficiency beyond just I/O. NVMEOF devices can approach DDR4 performance levels and, with SCM, become an integral component of the computing memory hierarchy.
    • Queue Parallelism: With 65,535 parallel queues, each with a queue depth of 65,535 entries, NVMe-oF supports massive parallelism. Every CPU has a private queue to communicate with a data device and is capable of achieving high speeds because there are no locks between queues that live on separate cores. Because each controller has its own set of queues per CPU, I/O throughput increases linearly with the number of available cores.
    • Integrated Security: NVMe-oF includes a tunneling protocol that provides security features produced by the Trusted Computing Group (TCG) and other related communities. Features planned for NVMe-oF devices and systems include simple access control, data at rest protection, crypto-erase, purge-level erase and others. Additional information on NVMe-oF security is available in the Trusted Computing Group and NVM Express Joint White Paper: TCG Storage, Opal and NVMe.

    NVMe-oF is designed not only to solve the limitations of scale-out storage, but also to lay the foundation for a storage fabric that dynamically shifts as workloads demand.

    What Makes NVMF Important to Kubernetes?

    Kubernetes applications implicitly need the infrastructure flexibility enabled by NVMF. NVMF and Kubernetes makes distributed computing within sites easier.

    The key point of my Kubernetes-native blog was “(Kubernetes-native) applications break apart into their constituent elements and can deploy and scale across the latency pyramid where they fit best. The elements dance across the infrastructure following their constituents in what I imagine as a containerized ‘ballet’ focused on delivering the best possible user experience at the lowest possible latencies.”

    However, multi-cloud, multi-site and edge deployments introduce significant additional complexity for organizations operating Kubernetes at (or beyond) enterprise scale. Business needs are pushing IT teams to move workloads between sites, clouds, WAN edge and rapidly be able to clone, burst and then destroy workloads anywhere around the world as needed. Yet Kubernetes deployment across multiple regions, sites, clouds and WAN edge remains problematic.

    OpenEBS is the solution for latency sensitive stateful Kubernetes workloads that need universal data agility. A NVMF-based disaggregated storage physical fabric mates perfectly with the disaggregated data agility software infrastructure offered by OpenEBS. NVMF allows data to physically float across a single storage fabric while OpenEBS allows stateful workloads to be loosely coupled across multiple storage infrastructures. And the OpenEBS MayaStor project natively integrates to NVMF to simplify the developer experience.

    Together, NVMF and OpenEBS become the dance floor for those Kubernetes-native application components to do their infrastructure-aware (and infrastructure-intelligent) ballet. Given its performance and scalability, however, perhaps this is better called the NVMF Boogie.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Charles Piercey

  • The Challenge of Scaling Metrics in Distributed Systems

    The Challenge of Scaling Metrics in Distributed Systems

    Whether migrating from a monolithic architecture or building a distributed system from ground zero, there are many benefits to leverage from a microservices architecture–faster software deployment cycles, enhanced scalability, improved isolation of risk. However, distributed systems are complex and as such, accompanied by many challenges, particularly when it comes to the scalability of metrics.

    The Importance of Observability in Distributed Systems

    In order to realize the benefits of a distributed system, a strategy of data-driven optimization is essential. The paradox here is visibility simplifies complexity, while the complexity of visibility is dramatically increased in distributed systems.

    As New Relic points out in their eBook Cloud Native Is the New Normal, traditional monitoring approaches need to be adapted to meet the demands of decentralized architectures, including:

    • Overcoming the loss of end-to-end monitoring capabilities by learning how to connect the dots across a complex network of environments.
    • Developing tactics to quickly drill down into issues across interconnected parts to reduce mean time to resolution (MTTR).
    • Introducing proactive reliability strategies that focus on measurement first, followed by improvement.

    Finding the Right Balance Between Monitoring Nothing and Monitoring Everything

    Most application engineers leverage some sort of metrics tooling such as Grafana, Prometheus or Graphite to track vitals for their system health. While monitoring is essential, there is such a thing as bloat when it comes to metrics ingestion; and this overload translates to scalability challenges that affect performance, cost and reliability.

    So, how do you know what to monitor? In order to derive the most valuable insights from your system, you must first understand and identify the components, and more specifically which attributes within those components drive key metrics. Knowing this requires an intimate familiarity with how all the pieces work together in your distributed system.

    Google’s Site Reliability Engineering book provides a framework for determining what to monitor in distributed systems. They break signals down into four different categories:

    • Latency: Understand resource requirements and identify bottlenecks.
    • Traffic: Monitor system response to varying levels of load; most useful when paired with other indicators.
    • Errors: Track frequency and nature of interruptions in request/response flow.
    • Saturation: Surface capacity-related issues.

    Even when you’ve identified the most important metrics to track, scalability is still a challenge that must be addressed, particularly when dealing with applications that handle massive volumes of data across a complex application architecture.

    M3: Uber’s Open Source Large-Scale Metrics Platform

    A solution directly addressing the challenges of distributed metrics at scale is M3, an open source project developed by Uber’s engineering team.

    Before using M3, the Uber team sent metrics to a Graphite stack with storage handled using the Whisper file format in a shared Carbon cluster. From there, they created Grafana dashboards to visualize the data and used Nagios for alerting based on Graphite threshold checks via source-controlled scripts. As the volume of metrics continued to expand, manual processes to scale the solution became unsustainable and repeat failures resulting in permanent data loss led them to build a more fault-tolerant, scalable system.

    M3 is focused on providing a reliable storage solution for large-scale metrics over long retention periods, solving inherent single node-associated limitations with Prometheus by offering a multi-tenant store for Prometheus metrics. A key component of this solution is M3DB, the scalable storage backend for M3.

    M3DB, built in Go, is a distributed time series store and reverse index with configurable out-of-order writes. By enabling the storage of real-time metrics over long periods of retention, M3DB can replace a Prometheus cluster. In developing the M3DB solution, the Uber team essentially deconstructed Prometheus into microservices to enable the scaling of each component individually. By taking this approach, M3DB is able to offer better reliability, efficiency and performance.

    At a minimum, M3DB can be leveraged as remote storage for Prometheus (or used to replace any other remote storage solution). There are, however, other features the solution offers, including an external aggregation engine and a separate query engine.

    The M3DB project is still under development with a growing community of contributors.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Molly Wojcik

  • Learning Production Troubleshooting Techniques in Microarchitectures

    Learning Production Troubleshooting Techniques in Microarchitectures

    One of the great things about our industry is the ability for anyone to get started doing development with a minimal amount of tools. Anyone can learn to do development with just a computer of some sort, a text editor, a tutorial and their brain.

    In fact, the initial basics of how to think through a development problem don’t require a computer at all, reducing the cost of getting started even further. Having such an easy way to get started that doesn’t require a lot of tools lowers the bar for entry, enabling more people to join in on development teams and contribute new ideas and perspectives, which enriches the entire industry.

    Getting into containerization and building modern microarchitectures is a slightly higher bar, but only slightly higher. There are a number of wonderful tutorials out there for getting started with Kubernetes, Docker, OpenShift Origin and other free and open-source tools.

    The only differences in getting started between general software development and microarchitectures specifically are: needing a computer that has the processing power and memory to run containers, and enough of a grasp of a development language to have a basic application to play with. Even that last difference isn’t as crucial because there are so many sample applications people have built for the express purpose of teaching new folks to deploy microservices onto various systems.

    However, when it comes to learning from the DevOps, SRE or ops side of managing Kubernetes deployments or other modern architectures, there’s no solid free tutorial I could find that dumped a bunch of broken sandboxes in front of the student with the express purpose of building their troubleshooting knowledge.

    To gain insight into understanding how to bring microarchitectures back from failures or fall-overs requires access to larger applications and clusters and more senior, or more experienced, mentors–unless you are lucky enough to be able to build large interconnected applications and send traffic to them regularly enough to cause failures.

    This lack of strong tutorials and a requirement for more resources makes breaking into DevOps or SRE work on microarchitectures particularly difficult if no one is willing to take a chance on a less experienced candidate. I want to see that change. We are missing out on hearing from new voices, people who may look at a problem from a different perspective or who may have unique solutions to problems we all face.

    I decided to create a repository full of deliberately broken Kubernetes builds for new practitioners to build progressively more advanced troubleshooting skills. The basic idea is to build a pyramid of environments. The base level should be easy to understand and figure out. These environments will be bare-bones simple, with only one cause for failure.

    We need to build the baseline recognition of common causes in isolation. Many of us that have debugged production systems have an almost unconscious grasp of when a common cause has appeared simply because we recognize that faint signature in the pattern of the data we’re analyzing. If you’ve never seen that signature, though, and you don’t have a mentor with you on these debugging challenges, learning to find that signal in the haystack of your data is much harder. These initial environments should make it easier to learn what those signals look like.

    The next level of environments would have some additional factors unrelated to the actual problem causing additional logs, additional spikes in traffic or additional activity. In short, these environments would have noise. We can add another layer of complexity for some of these environments by adding another service or application that runs just fine on the same cluster. This level would train the ability to rule out sections of a system that aren’t broken, to identify systems that aren’t interconnected and to work on one system without disturbing the rest of the environment.

    Further environments up the pyramid would have multiple contributing factors, red herrings or complex structures. At this point, some of the environments will need to be hosted outside of the local system. I’m still debating how to do these environments so the tutorial could still be free and easy for new folks to work with–I’m open to suggestions.

    Ideally, there would also be a written guide with answers and tips a student wouldn’t look at until they’ve solved the problem on their own, but they certainly can look up and learn regardless of whether they solved the problem. After all, troubleshooting isn’t a closed-book exam by any stretch of the imagination.

    So how do you start? Find the repo at GitHub. It’s free–open source with an MIT license–and open to contributions. There’s a contribution section of the README on the repo that explains how to get started if you want to send commits. If you want to try using these environments yourself and practice or learn troubleshooting skills, the initial environments are built on Minikube, a tool to run small Kubernetes clusters on your local machine. Follow one of the many excellent tutorials for getting Mikube up and running on your local machine, then follow GitHub’s tutorial on cloning down a repo.

    Alternatively, you can download the repo directly. Once you’ve gotten Minikube up and running and you have a copy of the repo on your local machine, follow the instructions in each environment’s directory to deploy the environment to your local cluster, and start trying to figure out exactly what went wrong.

    Want to help make this project a reality? I would love to have more contributors! In addition to being available online for help, I’m going to be at KubeCon North America 2019 in San Diego. At the conference, I’ll have a Hackers’ Corner set up in LogDNA’s booth for most of the conference, and I’ll be sitting there working on this project, among others, waiting for you. Bring your ideas, your stories, your broken environments, your laptops, your willingness to write docs, your Kubernetes knowledge, yourself—anything that can help enrich this project is welcome.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Laura Santamaria

  • Four Steps to Implement an Observability Strategy for Microservices

    Four Steps to Implement an Observability Strategy for Microservices

    You’ve finally decided. You’re making the move to microservices and re-architecting your entire infrastructure. Some services will still be on-premises (those databases and their legacy apps are never moving), but a cloud-based microservices architecture will allow your teams to build and bring new products to market faster.

    But how do you handle the complexity that comes with cloud and microservices? You know you need a strategy for monitoring and managing the complex dependencies and ephemeral nature of your new architecture, but you don’t know where to start. What’s more, you don’t want to spend time and money now only to realize a few months down the line that you’re on the wrong road. Your investment needs to be future-proof.

    You need to implement a strategy for observability. Observability goes beyond monitoring to give you greater operational insight, leading to shorter incidents, fewer quality problems, a better product and happier customers. Your company has unique observability needs, which means how you approach your observability strategy will also be unique. As is usually the case, it’s important to start with a roadmap and goals. Based on my experience, the following four things are key to making a considered, comprehensive observability strategy that’s built for the long haul.

    Choose an Observability Platform that Fits Your Future

    Perhaps counter-intuitively, the right platform isn’t necessarily the one that’s the right fit for you today. If you’re investing in new technologies such as microservices, containers and Kubernetes, you’re going to see a significant increase in the scale and ephemerality of your environment. A monitoring platform that may suit your current infrastructure likely won’t be able to keep up and meet your future needs. Instead, consider how your infrastructure will look three, four or even five years down the road, and then select the platform that best fits that long-term strategy. It doesn’t matter if you choose a commercial tool or an open source one: What matters is that you give considered thought to your future infrastructure and choose a platform that can address the future needs of your environment—and protects your investment at the same time.

    Monitor the Metrics that Really Matter

    If you’re still in the early days of your shift from monitoring to observability, you’re likely taking a service-based approach, monitoring specific pieces of your ecosystem with a focus on individual incidents. The real value of observability is the ability to provide visibility into big-picture issues, such as measuring user experience, performance against SLOs and other key metrics important to the business. In modern environments, small upticks in a microservice can cascade into increased latency for an entire product, a specific feature, or even a specific customer. Having an observability platform that measures and collects data across multiple dimensions–e.g. customer/business-specific, application/service specific and infra/host/container specific–will allow you to understand the telemetry in many different ways. The platform should also aggregate the interactions between your services and unlock a wealth of data for engineers to investigate when issues do arise, and provides visibility into the metrics the business cares about most

    Embed Observability into Your Incident Management Process

    Incident management is a hot topic right now. I speak with CTOs across organizations and industries who have fantastic tools that provide detailed telemetry, but everything breaks down when an incident occurs. They need to better embed observability as a part of their incident management process. For example, ask your incident management team how observability helped in a given incident and record those results. This keeps track of the ongoing benefit. Next, ask how observability could be improved to get the team closer to the crucial clues they need during an incident. And finally, make sure to fold these items into your remediation plans so the team has a clear playbook when incidents do occur.

    Establish a Culture of Observability Across Your Organization

    Simply deploying a new technology of any type is not enough. You need to ensure you’ve built the appropriate processes and norms around the new platform to ensure its success—this is especially true with observability. Modern observability platforms can be incredibly powerful in providing insight into the performance of your ecosystem, but without an investment in your organization’s ability to use the platform, you’ll barely scratch the surface of its true value.

    Observability requires ongoing investment and improvement. Start by building a core group of champions, then fold training into onboarding, continuing education or brown-bag sessions. Using subject matter experts to create the content is a good way to get everyone off on the right foot. Use the output from observability tools in company meetings, team syncs or support settings to keep everyone on the same page. This sort of standardization raises the collective skill of the entire organization and demonstrates the value of observability to those who are not already on board.

    Following these four steps will set you off in the right direction on your observability journey. If your organization is like most, you keep an eye on what you think could go wrong. While that’s unquestionably important, a mature observability strategy will give you an insight into previous unknowns and help you more quickly understand why incidents occur. And as you continue on your observability journey and your understanding of what and why things break improves, you’ll be able to implement increasingly automated and effective performance improvements that impact your company’s bottom line.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Arijit Mukherji

  • From Pets to X-as-Code: Enabling Control and Predictability for ITOps at Scale

    From Pets to X-as-Code: Enabling Control and Predictability for ITOps at Scale

    Amy Wheelus, vice president of network cloud at AT&T, recently spoke about declarative predictability at the Open Networking Summit in Antwerp. AT&T is the world’s largest telecommunications company providing mission critical communication services where people’s jobs–and lives–can be on the line. As they move away from labor-intensive IT operations toward cloud-native automation, there is a need to trust the declared IT infrastructure state can be predictably controlled and reproduced. This requirement to build predictability, reproducibility and trust into IT systems is not new but has grown in both scale and complexity as IT departments went from a few servers blinking in the basement to the core around which every business rotates. 

    With every customer interaction and business function becoming more digital, companies must be able to rely upon their IT departments if they hope to continue to keep pace in today’s rapidly changing world. This article traces the evolution of IT operations from Pets towards X-as-code with each improvement bringing companies closer to declarative predictability. These best practices enable enterprises to reliably deliver core business value through their IT department and keep pace in the cloud native era.

    Yesterday: Pets Versus Cattle and Infrastructure as Code

    Back when the IT department was just servers in the back room, it was all hands on deck when anything went wrong because work ground to a halt. The CEO couldn’t get their emails and the sales department couldn’t book new orders. From these early days, it was learned that in order to trust IT with critical business services, high availability and reliability needed to be built into the system. This lead to the creation of the Pets vs. Cattle analogy where each infrastructure component is interchangeable and can quickly be replaced when anything goes wrong. This allows companies to quickly recover from or even avoid outages.  

    While treating servers as cattle solves some problems, as any rancher can tell you, handling a whole heard brings its own set of challenges. As the size of managed infrastructure grows, it can be difficult to keep track of where, when and how it is actually deployed. In addition, changes to production systems to keep them healthy can lead to configuration drift creating unique snowflake servers which are pets rather than cattle. It becomes a difficult task to predict how a system will react or rely upon it in critical situations when its actual state is unknown.

    To help eliminate this problem and control how infrastructure is configured and deployed, ITOps teams moved to infrastructure-as-code. In this way of working, applications and servers are never modified once deployed. If something needs to be changed, a new version is created and the old one is decommissioned. Each deployment can be versioned to create an overview of what is in production and how it is deployed.

    Managing cattle rather than pets and using infrastructure-as-code has helped companies move from ticket-based operations with long lived machines to more scalable and dynamic infrastructure. These practices build reusability, scalability, automation and versioning into IT systems to create a simpler, more predictable deployment and management process. By being able to declare the state of its infrastructure, companies are able to predict and trust the outcome.

    Today: Cloud-Native Infrastructure

    Looking at the definition of cloud native from the Cloud Native Computing Foundation, it quickly becomes clear why many companies are turning toward these best practices. They help companies build and run resilient and manageable infrastructure that–combined with robust automation–creates dynamic, yet predictable systems. Declarative APIs and immutable infrastructure will be used as two examples to understand how cloud native builds upon and expands the techniques covered above.

    Declarative APIs make infrastructure as code and automation possible. For example, with Kubernetes, operations teams are able to declare the desired state of the system and automate the reconciliation with the actual state until they match. It originally began with just container orchestration but has since expanded to cover many other types of cloud resources including the physical infrastructure itself through Custom Resources and Cluster API respectively. Declarative APIs make infrastructure management simpler and more reproducible by automating operations and eliminating errors in production. Declarative APIs can be leveraged to create immutable infrastructure.

    Immutable infrastructure goes beyond just servers as cattle and only treating the infrastructure as code. Immutable infrastructure covers every higher layer of the stack until each one is codified and managed like cattle. Only exact copies are deployed and they can be completely replaced at any time. If anything needs to be changed, a new deployment is made and the old ones are decommissioned. Immutable infrastructure works for the whole stack to eliminate variations leading to fewer deployment failures, consistency across environments, easy horizontal scaling and a simple rollback and recovery process.

    As these two examples show, cloud native enables companies to reliably scale out dynamic IT services without having to simultaneously scale out their operations team. Building declarative predictability into systems engenders trust in automatic processes, even when critical services are running on them.

    Tomorrow: X-as-Code

    While cloud-native best practices provide many benefits, applications and infrastructure are just the beginning of the story for enterprise IT teams. In today’s world, they are presented with the Herculean tasks to manage a dynamic and sprawling environment while still keeping in compliance with internal policy along with national and international laws. This challenge moves far beyond mere servers and treating them like cattle to compliance and governance across multiple clouds and hundreds or thousands of developers. With these challenges in mind, the question still remains on how to best deliver on these demands at the enterprise scale.

    Looking across architecture, configuration, blueprinting, policy and governance, if each of these is simply viewed as another layer of the stack above the application, their operations can be radically simplified. By applying the principles of immutable infrastructure and cattle to this further layer of the stack, each of these can be defined and managed using an X-as-code model. These types of management models can already be seen taking root across the cloud-native ecosystem.

    Kubernetes creates blueprints-as-code through the use of Admission Controllers while policy-as-code can be defined in Pod Security Policies and Network Policies. The CNCF even has a policy engine, OPA, as a sandbox project that gives operators fine-grained yet flexible policy control across their entire stack. By using code to decouple policy and other layers from the actual implementation, the resulting stack is easier to understand, flexible enough to handle future requirements and less expensive to maintain.

    At Loodse, our customers were struggling to manage configuration, services and policy across their clusters. Leveraging the Kubernetes controller paradigm in Kubermatic, we are able to automate the deployment and lifecycle of each of these add-ons. This gives users the ability to declare, enforce and automate the required state of not only the infrastructure but the configuration and policy, too. Kubernetes controllers bring predictability and control to dynamic and scalable cloud-native IT even as it begins to handle more and more mission critical services.

    Cloud Native in Production: Declarative Predictability

    With IT becoming the core of every business, bringing cloud-native best practices into production enables operators to predictably declare, run and manage their infrastructure at the enterprise scale. Being able to trust in these automated operations frees up resources and opens up many new possibilities. Developers can be given guardrails and become empowered to do their job rather than waiting in a ticket queue. Operators can easily create highly available self-healing infrastructure that meets governance standards allowing them to focus on delivering a platform with differentiated business value.

    At each step of the development of modern IT practices, from cattle to X-as-code, operational processes have become more streamlined allowing both developers and operators to drive more value to the bottom line. The future of IT infrastructure and operations will be a trusted platform that automates undifferentiated heavy lifting, allowing the whole business to focus on customer value.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Bill Mulligan

  • Datadog and Aqua Security Partner to Provide Seamless Visibility into Container-Based Application Security

    Aqua integrates with Datadog to give DevOps teams real-time security metrics and events

    Seattle, WA –December 11, 2018 – KubeCon/CloudNativeCon – Aqua Security, the leading platform provider for securing container-based and cloud native applications, today announced
    the integration of its platform with Datadog’s cloud monitoring and analytics platform. With this integration, Aqua provides real-time visibility into the security posture of cloud native applications to Datadog users, including information on vulnerable images, untrusted running containers, and security anomalies found by Aqua in the runtime environment.

    For DevOps teams that continuously monitor applications for operational parameters such as performance, bug tracking, and errors, security events are often a blind spot that is handled
    elsewhere although they may directly affect application uptime and resiliency. The integration of Aqua’s granular security information into Datadog’s comprehensive monitoring makes it possible to identify issues quickly and analyze their impact on application availability.
    As organizations shift to more dynamic infrastructure through cloud and container technologies, communication between application and security teams is more important than ever,” said Ilan Rabinovitch, VP Product and Community at Datadog. “By combining Datadog’s deep insights into containerized application performance with Aqua Security’s enforcement of
    security best practices, we are helping organizations bridge the gap between these traditionally siloed teams.

    The integration between Datadog and Aqua CSP features pre-built Datadog dashboards that display:
    – Container images currently in Aqua’s scan queue
    – Known vulnerabilities and security issues found in existing images
    – Containers running from unauthorized images
    – Aqua runtime policy violations and audit events

    Additionally, Datadog users can use the data provided in the Aqua dashboards to set up theirown alerts, aggregate data streams from different applications, and customize how data is
    displayed.
    “We are excited to be partnering with Datadog to deliver a more complete security view to DevOps teams,” said Amir Jerbi, CTO and co-founder of Aqua Security. “In the cloud native era, ensuring security can no longer the exclusive burden of security teams, and instead should be part of the overall operational soundness of applications throughout their lifecycle. Our integration with Datadog creates a valuable shortcut that allows security issues to be detected early and fixed quickly, preventing escalated security incidents in production.”

    About Aqua Security
    Aqua Security enables enterprises to secure their container and cloud-native applications from development to production, accelerating application deployment and bridging the gap between DevOps and IT security. Aqua’s Container Security Platform provides full visibility into container activity, allowing organizations to detect and prevent suspicious activity and attacks in real
    time. Integrated with container lifecycle and orchestration tools, the Aqua platform provides transparent, automated security while helping to enforce policy and simplify regulatory
    compliance. Aqua was founded in 2015 and is backed by Lightspeed Venture Partners, Microsoft Ventures, TLV Partners, and IT security leaders, and is based in Israel and Boston, MA.

    For more information, visit www.aquasec.com or follow us on twitter.com/AquaSecTeam

    About Datadog
    Datadog is a monitoring service for hybrid cloud applications, assisting organizations in improving agility, increasing efficiency, and providing end-to-end visibility across the application and organization. These capabilities are provided on a SaaS-based data analytics platform that enables Dev, Ops and other teams to accelerate go-to-market efforts, ensure application uptime, and successfully complete digital transformation initiatives. Since launching in 2010, Datadog has been adopted by more than 9,000 enterprises including companies like Activision,
    AT&T, Deloitte, Peloton, Samsung, Seamless, The Washington Post, T-Mobile, Turner Broadcasting, and Whole Foods.

  • Instana Extends Leading Kubernetes Monitoring Solution to Better Serve Developer Community

    Company’s Latest Release Provides Quickest and Easiest Developer Visibility of Kubernetes Application and System Performance

    Seattle at KubeCon – December 11, 2018 – Instana, the first APM solution provider to monitor kubernetes at the application layer, today announced further enhancements to the company’s automatic application monitoring solution for containerized applications. The new capabilities center around Instana’s Application Perspectives, a unique way to group and analyze all performance metrics and traces from specific application components.

    “While Kubernetes-based orchestration allows DevOps to keep applications operating properly most of the time, it’s quite challenging to understand what is happening,” said Pete Abrams, Instana co-founder and COO. “To help developers quickly and easily understand how their application code is behaving within a Kubernetes environment, Instana has connected Kubernetes and Kubernetes Service monitoring directly to application performance metrics and traces. This happens completely automatically.”

    Instana continues to make headlines at KubeCon conferences. Last December, in conjunction with KubeCon in Austin, Instana announced the expansion of their automatic application monitoring solution to include any applications running in Kubernetes or on Kubernetes services such as Oracle OKE and Google GKE. In May, at KubeCon in Copenhagen, the company announced the ability to correlate application performance metrics and traces with Kubernetes system metrics – becoming the first solution to monitoring Kubernetes, itself, from an application perspective.

    Instana’s Application Perspectives changes the way in which IT organizations, especially development teams, use APM solutions for monitoring and troubleshooting their distributed microservice applications. Using operational tags or allowing real-time ad hoc definitions, Instana’s Application Perspectives gather all the metrics and traces that are related to specific applications, filtering out the noise from other distributed systems.

    “DevOps and Site Reliability Engineers are important, but they aren’t the only application performance stakeholders,” Abrams continued. “Developers are also involved in performance tuning and troubleshooting containerized microservice applications. That’s why Instana has created an entire User Experience specifically for the developer community.”

    Instana’s automatic APM solution for dynamic applications is unique in its ability to automate every step of the application monitoring lifecycle, from monitoring deployment to application discovery and monitoring. The company continues to shake up the monitoring industry by automatically consuming any source of performance and tracing information, including popular open source solutions like Jaeger and Zipkin. Like other Instana APM capabilities, the new Kubernetes monitoring occurs automatically, making it the quickest and easiest way to monitor Kubernetes applications and assure their performance.