devopsy.kinsta.cloud

Tag: SRE

  • DevOps and AIOps: Better Together

    DevOps and AIOps: Better Together

    DevOps is an attempt to scale technology with humans but AIOps is the ultimate answer.

    In today’s world, where so many essential business tasks have become digitized, IT teams must deal with constant change while ensuring zero downtime.

    The irony is that, although IT has become business-critical, the productivity and agility of the people building and supporting the customer experiences has plummeted. This has serious consequences for businesses. When your apps and services malfunction, you lose customers and revenue drops. When IT teams focus mostly on fixing emergencies, agility suffers and innovation shrinks. You need seamless operations in order to grow, compete and thrive.

    Historically, IT teams grew along with technology, beginning with the mainframe through distributed computing. But, the advent of virtual computing led to a world of microservices and ephemeral logic. Now companies simply generate too much data for humans to monitor and understand manually and with legacy tools.

    This opened the floodgates for an AI-led operations evolution, known as Artificial Intelligence for IT Operations (AIOps). By streamlining and automating IT monitoring, AIOps has helped IT Ops teams regain control over IT environments, quickly detect and repair problems, and prevent outages.

    AIOps

    DevOps: Back to the Future?

    However,  there’s recently been a shift in technology operations to once again attempt to scale technology with humans. The DevOps movement has placed the ownership, support and success of their services on the developers that write the code. This has decentralized technology operations teams that primarily operated with little context and unclear escalation paths, creating several smaller teams of DevOps engineers. 

    DevOps teams work together, each on their own microservice, toward the company’s desired customer experience and business goals. Depending on the operations model, there can also be a dedicated site reliability engineering (SRE) team or individuals on the DevOps team with the primary SRE role of monitoring observability data to discover potential issues. In this scenario, SREs feed the insights they uncover back into the development cycle to correct and strengthen the reliability and scalability of the DevOps teams’ services.

    AIOps

    But, as incidents occur in real-time, the challenge remains with SREs and DevOps teams to achieve insights and awareness across their applications, infrastructure and, ultimately, business services. To understand where incidents are occurring and what impact they’re having on the services and customers, it’s necessary to surface important events from the noise, understand the relationships between alerts and obtain the context needed to engage the right teams and people.

    This leads to another challenge: Getting the right people to respond and resolve incidents before there’s any business impact. Each DevOps team has its own responsibilities and tools. Often, DevOps teams don’t communicate except through their APIs. When incidents occur that need immediate attention from multiple people from multiple geographically-dispersed teams, how do you contact them and get them to collaborate, considering they have complex and different on-call schedules and escalation processes?

    As companies continue to transform with a customer experience focus and a digital-first mentality, they realize the seriousness of these challenges, the magnitude of their monitoring data and its negative effect on their business. Part of transforming their business is transforming their technology operations: Enter AIOps with the light to shine on DevOps processes, leading to a manageable, efficient and profitable future. 

    What AIOps Brings to DevOps Teams

    AIOps provides a unique solution to address operational challenges and cover every aspect of your service assurance strategy and business.

    Bottom line: You need to free people to focus on mission-critical tasks and empower them to build better services for better customer experiences—not scale them along with technology in an attempt to operate it.

    AIOps allows you to scale by integrating with the tools and infrastructure you’ve invested in over the years and by adding critical layers of AIOps (intelligence) throughout. By applying patented AI and machine learning algorithms to your observability and monitoring data, AIOps learns your environment’s normal behaviors and generates alerts accordingly. This must be done at the edge, close to where digital services produce vast amounts of data.

    Based on the anomalies generated both locally and environmentally, AIOps will then surface and correlate important alerts from all sources into actionable, contextual insights. A true, comprehensive AIOps solution also identifies root causes and impacts, and prescribes potential solutions based on previous resolution steps and feedback. This all occurs in a virtual workspace, where team members can collaborate, visualize and provide feedback. 

    AIOps notifies and empowers the right people to take the right action. It simplifies and understands complex team structures, engagement methods, on-call schedules and escalation paths, ensuring the right people are engaged and collaborate, even when they’re located around the globe.

    When a situation or incident is surfaced, it is sent to users in real-time. Based on the insights derived from the underlying data and machine learning applied, AIOps knows the exact teams and people that need to take action, and the ones that need to be informed. These users have the context they need to respond, share a consistent view and stay in sync throughout the lifecycle of the incident. Once the incident is resolved, a streamlined post-mortem speeds up future response, by using similar incidents and predictive analysis to spot repetitive and future problems.

    AIOps: A Key Ingredient for Effective DevOps

    The only way to scale with the technology being created in today’s and tomorrow’s world by DevOps teams is with AI. It frees humans to focus on creating and improving the customer experience that drives maximum profitability for your business.

  • Continuous Security Through Developer Empowerment

    Continuous Security Through Developer Empowerment

    Every organization is embracing DevOps to one degree or another. The business impact of shipping software quickly and adapting to market needs is so immense that it has become a requirement—you’re either heading toward DevOps or heading toward bankruptcy. Yet, while our need for speed has increased, so has our need for security—and combining both is far from an easy task.

    The key to achieve the DevOps velocity is developer autonomy. Developers and their surrounding application teams need to make daily decisions about what to build and how to build it, and they need to do them on their own. Bottlenecks appear when teams are required to get external approval, slowing down the process and eliminating the dev team’s sense of ownership. The resulting effects are subpar outcomes in applications and attrition of top performing developers. 

    Yet, security practices, unfortunately, often depend on such reviews. Security audits and reviews are the bread and butter of most security organizations, requiring experts to determine if a decision is secure or not. For security to work at the speed of development, this practice has to change and give developers the power to make security decisions. The only alternative is insecure software and deceleration of software version development.

    How can we achieve such developer empowerment for continuous secure software development? Let’s review some key practices, and look at ops practices for inspiration. 

    Mindset Shift: From Super-Hero Operators to Multiplier Engineers

    Site reliability engineering (SRE) teams hold the same responsibility for a service’s uptime as the sysadmins that preceded them. The change in the role came from how they achieved this resilience. SREs don’t handle most incidents, nor do they review all code that affects reliability. Instead, they engineer the tools to empower developers to handle oncall and build operable software the right way. An SRE should only be pulled in for the extreme cases, be it an incident or architecture review. 

    Security teams must operate in the same manner. Their job must be tactical, working to reduce their involvement in the process and giving more responsibility to the developers. Cases will arise where their expertise is needed, but general inquiries should be kept to a minimum. 

    Tooling Change: From Security Tools to Developer Tools

    Before DevOps kicked in, app performance monitoring (APM) was owned by IT, who used synthetic measurements from many points around the world to assess and monitor how performant an application was. These solutions were powerful, but their developer experience was horrible. They were expensive, which limited tests developers could run. They excelled in explaining the state through aggregating tests, but offered little value to a developer trying to troubleshoot a performance problem. As a result, developers rarely used them. 

    Then, New Relic came on the scene, introducing a different approach to APM. Their tools were free or cheap to start with, making it accessible to all dev teams. They used instrumentation to offer rich results in developer terms (call stacks, lines of code), making them better for fixing problems. This new approach revolutionized the APM industry, embedded performance monitoring into dev practices and made the web faster.

    The same needs to happen for application security. If developers are the ones making impactful security decisions, then the most important users of security tools are no longer security folks—but rather developers themselves. Security tools should therefore focus on achieving developer adoption first and foremost, even when it comes at the expense of more effort by the security person involved. 

    Governance Change: From Blocking to Monitoring

    One of my favourite DevOps adages is, “If it moves, measure it. If it doesn’t move, measure it in case it moves”. It embodies just how critical it is to successfully monitor your running application. While CI/CD pipelines help you get software to production, monitoring gives you the confidence to do it at speed. You don’t need to worry so much about shipping a bug when you’re confident you’ll find out about it quickly and revert or roll out a solution.

    Security needs to embrace this approach. Finding out a security vulnerability has just been deployed and acting on it is almost as good as blocking it just before it was deployed—and far less disruptive. But to embrace this, you’ll need to invest in monitoring. 

    Developers are the first step in the application process. It makes sense that they are also the engineers of security solutions. Empowering developers to make security-first decisions allows IT teams and security professionals to address different parts of the business, saves time in the development process and ultimately creates stronger applications and outcomes. 

  • SRE (Part 2): A Practical Approach

    SRE (Part 2): A Practical Approach

    In my last piece on cultural SRE, I covered the basics of defining what SRE means for your team. However, some teams may not need dedicated SRE support. This is where the central organization comes in handy. Dedicated resources are focused on building guidelines, standards, services and platforms for teams to consume that can operate autonomously in your environment. Having a broadly communicated central team also allows for these other teams to know where to go when they need assistance or would like some recommendations on their technology stack. 

    Application/Service SRE Versus Infrastructure SRE

    Application SRE includes:

    • Embedded for application/service teams.
    • Architecture guidance for new services and infrastructure. 
    • Design and implementation for modern technologies.
    • Support within the application/service team.
    • Automation for apps and services for the team.
    • Automation for operational work.
    • Monitoring/metrics for application/service team.
    • Benchmarking and performance assistance for code.
    • Infrastructure deployment and architecture for applications and services.
    • Write documentation and runbooks for alerts issued by application/service stack.

    Infrastructure SRE includes: 

    • Build and management of “plumbing” technical infrastructure (provisioning, OS, dns, dhcp, networking, central auth, etc.).
    • Automation of infrastructure services (telemetry, monitoring, log aggregation, configuration management, anomaly detection, orchestration, etc.).
    • Build and management of consumable services and tools (message queues, databases, distributed compute farms, API services/integrations, containers, etc.).
    • Infrastructure as Code (IaC).
    • Implement Global IR&IM process.
    • Support for Application SRE teams.
    • Architectural guidelines and best practice documentation.

    Organizational Layout

    Organization structures, whether we like to believe it or not, really help in the flow of communication throughout the team. It allows for the team leaders to form a single mission and have that mission carried out by their team members. I mentioned in Part 1 of my SRE Narnia guide that I believe this organization must be centralized to be successful. Below is a high level overview that shows the separation between application SRE and infrastructure SRE.

    The leads can handle multiple teams and disciplines. This can shrink and scale as you see fit within your organization. Application SRE is straight forward. The infrastructure SRE could be embedded across existing infrastructure teams or form a new team focused on a specific project.

    Example SRE Interaction

    Below is an example interaction between an application team (this one runs a ruby app) and an SRE hierarchy. You will see a team called “Frontline Support” as the initial interaction. This team is not required but is a nice to have. It helps in any environment big or small, and can really offload a significant amount of the operational workload along with having a global view of issues coming in.

    The duties for frontline support are to follow runbooks for alarms flowing in from the monitoring systems. It is similar to a traditional NOC but with more expertise across the stack they are watching. It also allows for trend analysis and bringing data to conversations around reoccurring issues or bugs in the system.

    I do recommend using this as a starting point for junior level team members. It allows them to learn the system quickly and buddy up with more seasoned members that are either on this team or have the pager for the week. It is a great way to train and onboard your technical employees. Nothing works better than showing them what is broken. 

    Summary

    Overall, everyone’s journey will be different. There are quite a few books and examples that exist out there, but please only use them as ideas to form your own opinion on how these organizations should work. There is no prescription and nothing is ever perfect. I have left out quite a bit of information in this post but I am always interested in having conversations about this topic and others alike. There is one consistent theme you’ll see across the board and that’s shaping the culture. Focus on your culture and hire great talent along with great leaders. Good luck and happy hacking.

    — Anthony Caiafa

  • SRE (Part 1): A Modern Overview

    SRE (Part 1): A Modern Overview

    Site Reliability Engineering (SRE) is a topic that over the last several years has become a popular discussion across many companies I have interacted with. This article is the first of two that I’m passionate to share, as I’ve come across many organizations who are trying to build and implement this ideology, while all are asking the same questions—What is SRE? Who are SREs? How do we get it? Where do we start?

    I—just like everyone else—have opinions around this topic. However, there is a common ground between us all. SRE is not only about tooling and tech, it’s not only about the unicorn hires you can make, but it is primarily a cultural shift within companies. It changes the way developers, product owners, admins, general engineers, etc., all interact with one another. You have to shift your operating model and how you approach building in your organization.

    Now, just as a disclaimer these are just my opinions and experiences with building organizations like this and talking to other companies who have implemented or are currently implementing. There is no prescription for building SRE. Each and every company will implement at their own cadence and own internal philosophies. You will, over time, build it in a way that fits your organization and internal operating mode. Do not move too fast and attempt to force the model into your organization. Fail but fail gracefully and learn from those failure. You’ll get to your Narnia at some point.

    Definitions

    Let’s start with some quick definitions of the acronyms we will be using. These will be short explanations but give you the general outline. 

    • SRE: Site Reliability Engineer, an individual who possesses an interest in infrastructure and operations, with a software engineering background.
    • SWE: Software Engineer, someone who writes the application/service software. 
    • Infrastructure SRE: a variation of SRE that works on infrastructure related projects (monitoring, provisioning, IaaS, etc.).
    • Application/Service SRE: a variation of SRE that is dedicated to specific application/service group. These are the groups who are building the products that generate revenue for your company (hopefully).

    The definition of SRE is quite scattered and I don’t think anyone truly understands/knows what it means—and no, unicorn’s will not show up below.

    This is what SRE means to me:

    • SRE is a core group of individuals who have a wide array of skills. They are more of a technology generalist than specialist. These skill sets range from operations, infrastructure, networking, development, hardware, distributed systems, monitoring, stability, capacity planning, software engineering, etc.
    • SREs are responsible for architecture and implementation of technical infrastructure and supporting services, focusing on stability, security and scalability of those platforms.
    • SREs should be building standard best practices, platforms, services, infra, etc., that reach the global organization. 
    • SRE is not just about the technology. SRE is a mindset, thought process and cultural shift. 
    • SRE shouldn’t be made mandatory for everyone in your company. Teams will have the choice to fund SRE support if they need/want it.

    Areas of Responsibility

    Technology has so many different aspects that it may seem hard to narrow down what SREs should actually be spending their time and energy on. Since these individuals may not be directly building public facing products or features (depending on your company, of course), what exactly will they be doing?

    This by no means is limited to the list below, but should be the general areas in which you start to focus your SRE organization. Without these underlying infrastructure plumbing being properly built, monitored and scaled, it will be difficult to preach this new cultural mindset to the outside team members. The items below may seem very infrastructure heavy—and some may be. Many of these can be split between application SRE and infrastructure SRE. One is the primary producer, while the other is the primary consumer.

    • Monitoring.
    • Configuration management and automation.
    • Infrastructure services/networking/platforms/architecture (including hardware and gen compute).
    • Infrastructure tooling and capacity planning.
    • Big data/data warehousing/data analytics. 
    • Documentation and runbooks.
    • Incident response and incident management process.

    Is SRE a Single Team or Multiple Teams? 

    There is no prescription, and results will vary depending on multiple factors. However, I recommend that this is split logically into multiple teams if you have the staff to accommodate. You can start small and have a generalist team that covers multiple areas and eventually break into smaller, more manageable pieces. The areas are:

    • Infrastructure (hypervisor, storage, operating systems, containers, automation, SDN).
    • Observability (monitoring, telemetry, event correlation, trend analysis, IN&IM).
    • Tooling (custom tools, configuration management, developer experience tools).
    • Services (databases, message queues, orchestration, micro-services).
    • Apps (mostly application side knowledge and ability to support/troubleshoot).

    Even though they are split in some sense, the reporting chain must remain centralized throughout the organization. Decentralizing in my opinion will break the effectiveness and communication of the team and result in an unsuccessful SRE implementation. SREs should report to like-minded individuals whose mission statement is to push SRE agenda and philosophies. Agenda is not the only reason. A central hierarchy allows for governance around tooling, best practices and general technical/architectural guidance. From that, you can allow those folks to cross pollinate in other groups and teams and spread those standards across the board. A breakdown in structure and communication will inevitably cause a deterioration in the mission.

    Another reason for centralization is that it allows you to make sure your SRE hires do not become a dumping ground for operational work the app teams do not want to do or have pushed off. I have seen this happen in the past; it discourages the new hires and isolates them. They do not have a mission or a central team to work with and end up building things off on their own that only fit the needs of who they are supporting, instead of seeing the bigger picture.

    The application SWEs should still be doing operational related work, since they are the ones who created the need for it. The application SRE can help offload some of that work and find ways through the central chain on what tools others are using to offload. Operational issues can quickly turn into error budgeting for the application teams.

    Your SREs will likely be split into two disciplines: application SRE and infrastructure SRE. This allows you to cover both sides while pushing the same mission throughout all the teams. Infrastructure SREs are out there, and there are plenty of them. I have seen it come as a natural evolution of DevOps, system engineers and system administrators. You should focus on building your internal talent pool and then going outside for hires.

    Having some of the SWEs on the team volunteer and shift work over to SRE helps speed up the process for that specific application. You can leverage their knowledge of the code base and have them learn and understand the centralized mission and tooling to bring down to their stack. These individuals can also train the central team on the app stack.

    However, not every team in your company will need dedicated SRE support. Part two of this series will be a deeper dive into application and infrastructure SRE, and how to organize your teams from a management perspective.

    — Anthony Caiafa

  • Moving from NOC to the SRE Model

    Moving from NOC to the SRE Model

    Is Site Reliability Engineer (SRE) another case of “Do what Google does”? When viewed with a discerning lens, the practices and tooling high-performing engineering teams deliver to the market can be very beneficial. The SRE model is one of them and stands to benefit DevOps teams substantially.

    If I could change the name from Site Reliability to Service Reliability I would. Applications are increasingly more service oriented, architectures are based on services and the SRE model is generally more customer centric than previous application support approaches. SRE has become the production support companion for modern application development.

    While the Network Operations Center (NOC) and SRE primary objectives are to focus on application uptime, the responsibility of supporting applications is much greater. Those who support modern applications are under pressure from the business and development to respond fast and accurately. This has been done previously by a NOC only.

    In its most traditional form, the NOC was a 24x7x365 staffed room where all telemetry from various aspects of infrastructure and the application were presented on screens, usually with one screen dedicated to sports and video games. If you staffed the NOC, your job was to keep an eye on the application, spot any alarms and if there is an alarm, to find the right person—usually based on a spreadsheet—to fix it.

    The NOC has also classically been more tied to the infrastructure than the application. They would get alarms for the application, but they’d be unable to dig in and figure out what specifically in the application was causing the issue. The NOC model services waterfall development and monolithic architectures just fine. But now, with rapid release velocity and new application architectures, a new approach is needed. This is where SRE comes in.

    Customer Obsession

    The first big difference between the SRE model and the traditional NOC model is that the SRE role does not start when something breaks, it starts before. The most high-performing SRE teams I’ve seen always work backwards from the customer. They use the customer to decide what metrics to measure and collect, and how they will interpret the metrics.

    Google talks about RED (Request, Errors, Duration) metrics but these are in no way best practice for everyone, nor the only approach. They are the best practice for Google, perhaps, and how their users interact with the application. Things such as saturation, latency, etc., can be even more important. Also, it is not the individual metrics that matter, it is their relationship to each other and, ultimately, to the customer’s experience. The customer obsession bleeds into everything SREs do—from setting up automation to educating the rest of the technical teams.

    To be customer obsessed, SRE teams need to have good visibility into what is going on.

    Observability

    I’m sure you have heard the new popular term “observability.” You might have thought to yourself, “Isn’t that just monitoring?” Observability’s foundation is monitoring, but it’s purpose is to go from the static point-in-time view of applications that traditional monitoring has been used for. It also addresses new challenges that show up as application architectures become increasingly more fragmented with services, and more complex to monitor holistically.

    Observability in some ways is about removing the human observer. In the NOC, you have staff watching aspects of the application and infrastructure to decide when there is an issue. Platforms that are designed to be observable have the detectors to tell you when there is a problem. It also does not create an assumption that those on-call have existing underlying knowledge of how all layers of the application work in advance because with modern applications, this assumption is not possible.

    Observability is both a tooling approach, but also a strategy for building applications. Applications should be built to be observable, which means the application provides sufficient telemetry to support its observability. Monitoring is basically part of your application now.

    Additionally, application degradation (such as latency) is the new downtime, so the ability to measure depth and breadth of application activity is critical. Observability helps identify reduced performance of the application as well, or impending issues in services that could lead to catastrophic events.

    Automation

    There is one big assumption that comes with the SRE model: automation and the interest to automate. The SRE model enables greater context and visibility across the organization, but it has to do that without increasing human effort. Without automation that is not possible.

    Automation includes:

    • Going from insights from observability to incident response.
    • Gathering context from systems to report, to observability tools and incident response.
    • Mechanisms to support those who are on-call with self-service remediation.
    • Automation to understand similar incidents.
    • Automation to alert/page/call people based on on-call schedules and policies.
    • Automation to recommend responders based on their incident response history.

    Anyone Can Be On-Call

    One of the huge benefits of the SRE model is anyone can be on call. The practices and automation of the SRE model allow anyone to be added to a rotation and on-call schedule. Most companies have this as a dedicated team, and in those who are transitioning from the traditional NOC to modern, that is exactly the first change that takes place. They leave the dedicated physical room and are on-call virtually instead.

    Not just people who support the application are on-call, the SRE model allows for developers and quality engineering to be on-call as well. Arguably, you can’t embrace shift-left without embracing SRE.

    Context

    Automation without data does not create a truly observable and actionable environment. Leveraging data across your delivery chain that can support action is critical. For example, this means the ability to leverage tracing to pinpoint production issues down to a service, or embed more context as part of the payload of alerts to bring more clarity to those on-call.

    Data collection is one thing, but not knowing how you are going to leverage the data makes it useless. The SRE model is obsessed with leveraging data and doing controlled experiments to ensure the data collected supports the uptime and experience for the user.

    Stewardship and Strategy

    Finally, SREs spend more time on stewardship and strategy than the traditional NOC. Some SRE groups run DoJos for the organization to help spread the idea of quality and application support across all technical teams. They will collaborate with developers so they better understand aspects of the application the SRE processes can leverage to support it better. SREs also spend a lot of time creating the strategy of application production support for the organization. Choosing key metrics, deciding how automation will be setup and consumed and creating policies on how on-call should function.

    Before you get upset, SRE does not mean the NOC is dead. The modern NOC can be driven by an SRE model, and I’ve seen this in-place in several companies. The NOC is also necessary to support many on-prem applications.

    What happens is, instead of a staffed room, the NOC becomes a group of engineers who are building automation, setting policies and practices, and stewarding quality across all teams. Similar to DevOps, SRE has elements of culture, principles and practice. Organizations who have a team structure where the NOC is responsible for application support, oftentimes do not change this for SRE, they become the SRE.

    The NOC is changing, and has to change as modern applications become more complex, release velocity increases and users become more demanding. The SRE model is an approach to transform the NOC into a modern version of itself or a whole new organization and strategy. The goal is not only to support applications when something breaks, but to make sure customer’s expectations are met, and systems support current and future development efforts, without increasing the human workload.

    — Chris Riley

  • Human Transformation Breeds DevOps Skills and Success

    Human Transformation Breeds DevOps Skills and Success

    As part of its mission to advance the human elements of DevOps, DevOps Institute has released its 2020 Upskilling: Enterprise DevOps Skills Report. The year-over-year data builds on the inaugural 2019 project but it’s more insightful and more regional than last year. DevOps Institute is grateful to the 1,300 respondents this past year who took the time to share their insights that allowed us to present this important research to the community.

    One thing that is abundantly clear is that human transformation is the single most critical success factor to enable DevOps practices and patterns for enterprise IT organizations. Traditional upskilling and talent development approaches won’t soon be enough for large enterprises to remain competitive. This is because the increasing demand for IT professionals with core human skills is escalating to a point that business leaders have not yet seen in their lifetime. If there is but one call to action from the report this year is that we must update our humans through new skill sets as often—and with the same focus—as our technology.

    Thoughts on Skills

    What we continue to see is a well-recognized gap between the skills needed for digital transformation and the skills available in new and existing talent. According to 58% of the Upskilling report respondents, finding skilled DevOps individuals is a huge challenge, whereas 48% of respondents say the retention of skilled DevOps individuals is a challenge.  

    Unfortunately, the term “skill” is so broadly generic, which fuels the belief that there is a single stack of competencies that will guarantee enterprise success. That’s simply not true. The modern IT professional has to possess a set of skills across at least four categories: process, function, automation and core/human skills. Within each category, individuals must know which skills are considered must-have, nice-to-have or not as important—which leads to the question: How do you know? Seeking the answer to that question is the genesis of DevOps Institute’s Upskilling: Enterprise DevOps Skills Report project.

    Additional Takeaways from the 2020 Upskilling: Enterprise DevOps Skills Report

    • The DevOps journey is difficult for at least 50% of respondents since it changes the traditional structure of IT.
    • The DevOps human must have a hybrid set of skills since process skills and knowledge (69%), automation skills (67%) and human skills (61%) are the top three must-have skill categories in 2020.
    • Upskilling needs attention now since only 31% of respondents had a formal upskilling program while 38% do not,  21% said they were working on standing up a program.
    • Agile, DevOps and ITIL are getting strong competition from Site Reliability Engineering (SRE) which grew by 5% from 2019.  Value Stream Management and Systems Thinking are also gaining traction.

    To learn more, download the Upskilling 2020: Enterprise Skills Report.  The data is very granular and can serve as a waypoint for individuals and enterprises trying to plan their upskilling initiatives.

    — Jayne Groll

  • Top Six Open Source Tools for Monitoring Kubernetes and Docker

    Top Six Open Source Tools for Monitoring Kubernetes and Docker

    Kubernetes and Docker are two of the most commonly heard buzzwords in modern DevOps conversations. Docker is a tool that enables you to containerize and run your applications, and Kubernetes provides you with a platform to orchestrate or manage these containers—since managing thousands of containers manually using the Docker CLI would be a practical nightmare. 

    However, simply running thousands of containers and managing them through Kubernetes is not enough. You have to observe and analyze them properly to ensure that your services are up and running optimally. This process is called Site Reliability Engineering (SRE), a term initiated and popularized by Google. Observability and analysis are a major element of SRE. It can be subdivided into the following three areas:

    • Monitoring allows you to extract numerical metrics from your applications and resources that can later be visualized and analyzed to present the current state of your resources. Once the metrics are extracted, they can be used to set up alert rules, facilitate profiling and debugging, and make better SRE decisions.
    • Logging enables developers to debug their containers in case of a failure. Containers are short-lived, and so are logs. Kubernetes and Docker do provide a native way of browsing container logs, but it is very limited in its functionality. As a result, a centralized log plane is a must in any container-oriented environment.
    • Tracing allows you to debug services running on a network and follow a request trail until the source of a problem can be determined. In a microservices architecture, when multiple services/containers send requests to each other to perform one business task, a proper tracing solution is necessary. 

    This article will list and describe the most effective open source tools for monitoring and analyzing services running as containers. 

    Prometheus

    Prometheus is the first tool that comes to mind when discussing open source monitoring solutions. It is very popular in the development community, and it is a graduate of the CNCF project, which focuses on the monitoring side of SRE. Originally created and open sourced by SoundCloud, Prometheus simplifies the process of pulling numerical metrics based on a time series from a given metrics endpoint. It was built to monitor highly dynamic container environments.

    Prometheus is subdivided into three elements: Prometheus server, Alertmanager and exporters. Exporters are stand-alone, independent processes/containers that can be run on your target resource to generate and export metrics via a metrics API. The Prometheus server then takes care of service discovery as well as pulling metrics from the exporters to be stored in the Prometheus DB and later used for visualization or alerting. Alertmanager is responsible for setting up alert rules, analyzing the data in the Prometheus DB and sending alert messages to multiple receivers if a certain rule is triggered. There are a huge number of exporters, listed here, which are both officially supported by Prometheus and maintained by the community. 

    Prometheus has become the industry standard for monitoring cloud-native architectures. While it is known for its simplicity with service discovery, its ease of use, its alerting capacity and its integration with Kubernetes, its polling architecture is not ideal. Currently, the metrics endpoint has to be reachable by the Prometheus server. However, there is a pushgateway implemented in Prometheus that enables metric pushing rather than polling. Another downside of Prometheus is that it doesn’t scale well. This problem can be fixed in the Thanos adaptation of Prometheus.

    Related tools and technologies: Grafana, Cortex, Thanos, Prometheus Exporters, Alert Manager, Istio, Prometheus Operator.

    Figure 1: Prometheus Graphs. Source: SoundCloudDev.

    Grafana

    Grafana is an open source metric analytics and visualization suite. It allows you to create custom dashboards using data taken from multiple sources, such as Prometheus, Elasticsearch, MySQL, Postgres and Redis. Furthermore, Grafana has its own alerting system and a role-based access control (RBAC) system for software. As a data visualization tool, Grafana is famous among Prometheus users for enabling them to effectively visualize the metrics stored in Prometheus. There’s a wide array of official and community-built custom dashboards for various data sources in Grafana, allowing users to easily set up dashboards (found at this link) and get on with monitoring. Grafana offers another associated product called Loki, which aggregates logs in Kubernetes and integrates well with the Grafana UI.

    Related tools and technologies: Loki, Prometheus.

    Figure 2: Kubernetes Resources Dashboard in Grafana. Source: Replex.

    Elastic Stack

    Elastic Stack is a group of open source products from Elastic designed to help users search, analyze, and visualize data from any type of source, in any format, in real time. This product was previously recognized as the ELK stack, where each letter of the acronym stood for one of the company’s main products: Elasticsearch, Logstash and Kibana. Elastic Stack provides both monitoring and logging solutions with the help of its big data database, Elasticsearch. 

    In order to aggregate logs, people tend to use Elasticsearch for storage, Logstash or Fluentd for log streaming, and Kibana for visualization. Fluentd is not part of the Elastic Stack, but it is widely used with Kubernetes to stream logs in place of Logstash, the tool provided by Elastic Stack. Similarly, Metricbeat is used to scrape metrics and set up visualizations on Kibana. The enterprise version of Elastic Stack ships with X-Pack, a set of additional tools that enables reporting, alerting and role-based access control (RBAC), among other features. By default, the Elastic Stack GUI, Kibana, does not support RBAC. You have to use the previously mentioned enterprise version of Elastic Stack to enable it.

    Related tools and technologies: X-pack, Metricbeat, Logstash, Kibana.

    Figure 3: Discover View in Elasticsearch. Source: XpresServers.

    Sensu Go

    Sensu Go is a telemetry and service health checking solution for multi-cloud monitoring at scale. It allows you to gain visibility into servers, containers, services, applications, functions and connected devices across any public or private cloud. Sensu can be run side-by-side with Prometheus to get the best out of both solutions, or it can be run natively without Prometheus. Sensu works best alongside Prometheus, since exporting application level metrics to Prometheus requires loading the Prometheus SDK to your application codebase and exposing a metrics endpoint. The endpoint is then scraped and stored in the Prometheus Server. This may sound like a lot of work—and, sometimes, it is. Sensu avoids this complexity by using a sidecar concept. A Sensu agent is deployed alongside your application. The agent constantly collects and exposes metrics to the Prometheus server without your having to change the application codebase.

    Sensu works without Prometheus too. It can operate natively in Kubernetes, where it has its own server to store and visualize metric data exposed by the previously mentioned Sensu agent.

    Related tools and technologies: Prometheus.

    Figure 4: Events in Sensu. Source: becon.

    Sysdig Inspect

    Sysdig has two open source products: Sysdig Inspect and Falco. Here, we’ll focus on Inspect, which monitors and captures the container processes running in your system and allows you to drill into those processes for post-incident forensics. This enables you to analyze your application performance, troubleshoot errors and monitor any processors that could be misbehaving. Furthermore, if your system is breached, Sysdig allows you to understand how the breach occurred and what data was taken in the process. Sysdig Inspect is a very powerful tool that focuses on the performance tuning and security investigation of your system. 

    Related tools and technologies: Grafana, Sysdig, Sysdig Falco.

    Figure 5: Sysdig Inspect Overview. Source: Sysdig.

    Jaeger

    Jaeger is an end-to-end distributed tracing solution that was open sourced by Uber Engineering. It allows you to monitor and troubleshoot transactions in complex distributed systems. In modern microservice architectures, the majority of operational problems fall under the umbrellas of networking and observability. When a service failure occurs, you have no idea how the requests have gone from service to service over the network to complete a single business transaction. This makes debugging extremely difficult. Currently incubating under CNCF, Jaeger uses tracing to enable root cause analysis, performance and latency optimization, and distributed transaction monitoring. Jaeger works out of the box with Istio, a popular service mesh implementation open sourced by Google. 

    Related tools and technologies: Prometheus, Jaeger, Zipkin, Istio.

    Figure 6: Jaeger Find Traces. Source: Jaeger.

    Conclusion

    These tools are widely used in the tech industry, and they all have their benefits. Most of these solutions, however, require skilled implementation and ongoing manual maintenance that can be a burden for DevOps teams and a distraction from the business. There’s no one solution that can cater to all of your requirements, since each tool focuses on one or two specific aspects of observability and analysis. By mixing these tools together, you can derive a unique solution for your individual business needs. 

    For ease of comparison, the chart below outlines the features offered by each of the tools discussed in this article.

    — Ran Ribenzaft

  • SRE Is the Most Innovative Approach to ITSM Since ITIL

    SRE Is the Most Innovative Approach to ITSM Since ITIL

    For over a decade, ITIL has been the leading ITSM framework adopted by enterprises across the globe. So, what is driving a rapidly increasing interest in Site Reliability Engineering (SRE) as a service management alternative?

    In its own words, Google refers to SRE as its approach to service management: “The SRE team is responsible for the availability, latency, performance, efficiency, change management, monitoring, emergency response and capacity planning.”

    In traditional ITSM terms, the role of the SRE is responsible for service level, change, availability, event, incident, problem, capacity, performance, infrastructure and platform management. While the operational practice areas may be similar, there are significant differences in how the practices are approached.

    ITIL4 Framework Compared to SRE

    Released in 2019, the newest update to ITIL4 remains a complex governance model with four dimensions, seven guiding principles, a Service Value System and 34 processes (now renamed as practices). While ITIL4 pays homage to Agile and DevOps, there is little depth in the publications released to date—to be fair, several other publications in the series are in queue  for gradual release. The framework spans just about every aspect of software delivery and operations and seemingly is trying to be the single source of truth in both principle and practice for IT management.

    SRE Is More Closely Aligned to Agile and DevOps

    By comparison, SRE focuses specifically on the reliability and resilience of complex production system operations. As an engineering discipline, SRE more closely aligns with the Agile and DevOps patterns that are being adopted by product development teams including continuous integration, testing, delivery and deployment. SREs bring the wisdom of production into the teams thereby breaking down many of the silo walls that have impeded IT for so long.

    As a service management alternative, SRE also updates traditional ITSM activities with innovative and self-organizing concepts such as management to service level objectives, error budgets, toil reduction, release engineering, monitoring/observability and embracing risk as neutral approaches to service management. The core SRE book provides practical and actionable guidance for Site Reliability Engineers on managing incidents, learning from failure, testing for reliability, load balancing, handling different types of emergencies, software engineering and capacity planning. Related publications such as the Site Reliability Engineering Workbook and Seeking SRE provide additional insight.

    Most importantly, a Site Reliability Engineer is an actual hireable job with a defined role, set of responsibilities and skills. SREs and SRE teams are encouraged to be creative, accountable and must spend 50% of their time reducing toil by engineering automation in order to make tomorrow better than today. Like Agile and DevOps, SRE supports self-regulation with policies and consequences. In fact, some consider SRE to be the third piece of the Develop (Agile), Deploy (DevOps), Operate (SRE) feedback loop.

    What This Means for the “Humans of DevOps” in Operations and ITSM

    SRE is breathing new life and opening new career paths for operations and ITSM professionals who a few years ago were battling against the mantra of “NoOps.” According to Linkedin’s 2020 Emerging Jobs Report, SRE is the fifth fastest-growing job role with 34% growth.

    Services will always need to be managed. However, competing in a digital age requires new ways of working and thinking with speed and quality as key metrics. Agility must be instilled across the value stream spectrum in order to increase flow and deliver an exemplary customer experience. Your organization may not be as complex as Google’s, but the principles and practices of SRE are applicable to all environments.

    You can read the Site Reliability Engineering and Site Reliability Engineering Workbook publications for free from Google. For those wanting to learn more about the practices and patterns associated with SRE, DevOps Institute recently released its Site Reliability Foundation certification with accredited training being offered globally by its Global Registered Education Partner network. Either way, if you are an operations or ITSM professional, I would highly recommend learning more about SRE. It is the future of ITSM as we cross the digital divide.

    — Jayne Groll

  • Predictions 2020: Infrastructure and Ops Trends to Watch in 2020

    Predictions 2020: Infrastructure and Ops Trends to Watch in 2020

    Today, modernizing systems is more than simply moving technology to a new location. It requires an IT stack of a new generation of technologies and tools to work.

    As we head into the new year, let’s take a look at the top eight trends that will shape how we will develop and deliver software. 

    Kubernetes

    Yes, it’s big and getting bigger. The pace of enterprise adoption of this leading container orchestration solution will only gain momentum in 2020. Look no further than the 2,300% attendance increase from KubeCon 2015 (500) to KubeCon 2019 (12,000). As more companies begin to see the direct financial and operational benefits to the flexibility, scalability, automation, high availability and portability of Kubernetes, the demand for developers and engineering experts in this complex software will grow accordingly. Case in point: A 2019 national job search on LinkedIn turned up 16,744 open positions for Kubernetes-related roles. Given the already large and still developing Kubernetes community, we should also see an increase in the speed and volume of feature releases. 

    Microservices

    It’s predicted that by 2022, 90% of all apps will feature microservices architectures. The growing business imperative for software engineering teams to produce and release software in ever shorter cycles—continuous delivery—has paved the way for a huge surge in microservices, the ideal architecture for CD. As enterprises continue their accelerated move to the cloud in 2020 and beyond, they increasingly prize the flexibility to try out the new technology stacks that microservices provide. All businesses want to optimize resources, minimize downtime, reduce infrastructure costs and avoid technology or vendor lock-in. In 2020, microservices architectures will continue to offer up more and better of all these benefits as the migration from monoliths picks up speed. 

    Service meshes

    The rise of microservices has led to a parallel boom in service meshes. This will only accelerate, given how integral the current class of service meshes—Istio, Linkerd, Consul, Envoy—is to the successful functioning of microservices. As a software designed to enable the thousands of microservices that make up an application to talk to each other, it’s quickly become an essential component with a vendor ecosystem poised for substantial growth. Feature-rich Istio, backed by IBM, Google and Lyft, is the leading Kubernetes service mesh, and that shows no signs of changing in 2020, especially given the relative ease of setting up and running it. If your enterprise has embarked on the journey from monolith to microservices, it will soon be crossing paths with service meshes. 

    Observability

    In the recent past, the sheer complexity of modern distributed systems has outstripped the tools designed to allow teams to see into them. Enter observability—which isn’t monitoring. Like good helicopter parents, software engineers, application developers, SREs and DevOps teams want to know and understand what their systems are doing at all times. LightStep, Datadog and Honeycomb, to name a few, are vendors poised to offer logging, tracing and metrics solutions to the rapidly growing number of enterprise teams building cloud native systems. Keep a close eye out for OpenTelemetry, an open source observability framework currently a member of the CNCF Sandbox. 

    SRE

    One of the great challenges ahead for this still-emerging field is for enterprises of varying sizes and work cultures to find a form of SRE that works. What succeeds at Google may not be the recipe for SRE implementation at smaller tech enterprises or startups. The role of SRE and the expectations that go with it remain fluid. The SRE’s constant imperative to automate will only add to the profile of critical SRE and DevOps tools such as Terraform, Ansible and Jenkins. Ponder Amazon’s Prime Day 2018 losses of up to $99 million for a clear understanding of the business-critical importance of SREs in the years to come. The job market reflects this new reality. 

    Security

    The shift-left movement in the software development lifecycle is on. The days of bolted-on post-production security are rapidly fading. DevSecOps, though still a buzz term, is here to stay, with more practice and use cases added to the theory. There are huge opportunities here for automated application security tools that support the software development workflow. 

    Then there’s privacy. The GDPR took effect last May and the California Consumer Privacy Act takes effect January 1, 2020. On top of these, there’s HIPAA and the regulatory compliance pressures on businesses in the area of security, which are already enormous—plus the immense challenges of cloud security. A 2019 Symantec survey on cloud security threats reports that 65% of businesses in the cloud still aren’t using multi-factor authentication. The same survey also notes that an astounding 85% of Symantec’s cloud customers aren’t employing security best practices. Look for businesses to tend to security earlier in the process. 

    Serverless

    It’s a boom time in serverless, with Google’s Functions, Microsoft’s Azure Functions and IBM’s OpenWhisk all offering their serverless solutions following the 2014 rollout of AWS Lambda. Look for more businesses to adopt a serverless-first approach as they upgrade legacy systems and see the substantial cost saving and application development benefits. Startups, too, will see the cost-cutting upside and the agility that serverless brings. The changes in application development and architecture in the cloud as a result of serverless are hard to overstate. With the vendor solutions now on offer, developers are freed up to do what they most like to do: run code and add functionality to their application, faster, without worrying about scaling and availability. Like microservices, serverless will see a huge upsurge in usage as the age of the monolith fades. 

    DevOps

    With the demands of cloud computing and far greater enterprise investment in dedicated teams, DevOps in 2020 is poised to make the transition from policies and principles to actual practice. The addition of baked-in security to the DevOps realm—DevSecOps—adds urgency to the automation of building, provisioning and deploying secure software. ML- and AI-driven apps have arrived, and DevOps practices will play a large role in the development workflow. The goal of zero-touch automation looms large, and this adds urgency to the need for clean data, organizational buy-in and more integrated systems. AIOps, though getting plenty of attention, is still young and its benefits to IT operations are still more theoretical than actual–but stay tuned.

    As we head into 2020, it will be exciting to watch these trends unfold. DevOps teams can expect the way we develop and deploy applications–in any architecture–to continue to evolve. By keeping a pulse on new developments in Kubernetes, microservices and other new technologies, they’ll be well prepared to bring their businesses into the new decade.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit  featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Kelsie Pallanck

  • Predictions 2020: 7 DevOps Predictions Shaping 2020

    Predictions 2020: 7 DevOps Predictions Shaping 2020

    2019 was an exciting year for DevOps practitioners, with DevOps adoption continuing to grow at a fast pace. Most organizations were either executing or evaluating their DevOps strategies. So, what is 2020 going to bring in DevOps, infrastructure and application automation? Here are seven DevOps predictions for emerging trends in the new year.

    Customer Experience

    CX will become the guiding light of the DevOps journey, and not necessarily just technology. This is a big mindset shift from the early days of pure-technologist enthusiasm. As DevOps matures, we have to remind ourselves that it’s all about making business faster, not just the cool tech.

    Take advantage now:

    • Tie DevOps achievements to business needs and results to get executive buy-in and budgets.
    • Successful QA teams will start looking at their role as guardians of customer experience and not just testers.

    Trust

    Everyone will be talking about trust—not only in the context of zero trust networks but also in the context of making it possible for Dev and Ops to trust one another as the foundation for DevOps.

    Take advantage now:

    • Proactively invest in creating a culture of trust. Identify causes for lack of trust in your value stream and make trust a 2020 goal.
    • Encourage product-level conversation and define methodology and ownership when it comes to value stream management, application and infrastructure automation. It is the foundation that everyone is working on.
    • Invest in staging environments—Staging environments are needed to ensure things work before you put them in production. For many of us, jumping directly to automatically deploying to production is a big step. The risk can be mitigated by investing in staging environment automation, which will be the first step toward deployment to production.

    Value Stream

    The DevOps toolchain is going away. In 2020 we will think value stream. The industry is maturing beyond the notion of chaining together dozens of open source tools and trying to make it work together. In previous years many felt comfortable saying that if they have a CI system in place it means they “do DevOps.” This is no longer the case. DevOps is not just CI, and DevOps cannot be established in silos. Mature DevOps needs to take into consideration the entire value stream—from planning a new capability and the business requirement, all the way to the customer getting more value and a wonderful experience.

    In 2019, Gartner announced it’s the last year of the ARO MQ, as it’s not the right perspective. In 2020 we finally gain a system-level view on the value stream. More players are announcing end-to-end value stream management including Atlassian’s new Forge platform, GitLab and CloudBees. Companies are maturing and adopting system-level solutions. 

    Take advantage now: 

    • Plan for platform operations. System-level value stream perspective requires roles that take ownership of orchestration and automation throughout the value stream and not just in silos.

    The SRE Role Will Continue to Mature

    The SRE role that started with Google bringing software development practices into Operations will continue to evolve. Increasing volume, velocity and variety of data will make AIOps solutions thrive. Also, increasing levels of automation will require smart ways to handle dynamic infrastructure and applications without losing control, and being able to track changes all the way back to the coder. 

    Take advantage now:

    • If you haven’t implemented the SRE role yet, consider how SREs can be part of your teams in 2020. Define SRE areas of responsibility and KPIs, and how the SREs communicate with the teams. Is it a centralized position or are the SREs part of the team?
    • Make sure you provide SREs with the tools they need to succeed including:
      • What are the observability solutions you use? Who uses them today? How do you analyze it and how is it communicated?
      • What are the production environment orchestration, deployment and configuration tools that you use today?  How do you connect them to application and infrastructure automation that happens before production?

    Seamless Infrastructure

    We will have to make infrastructure seamless for our teams. Internal customers are also customers, and their experiences will be just as important. Self-service and empowering teams will be the main themes in the bridges that need to be built between Dev and ITOps. Public cloud, Kubernetes and serverless technologies will grow. And while it makes automation more feasible than ever, the emphasis will shift to self-service and to the end users. With more companies adopting similar approaches, the need to develop internal expertise will decrease, and we will see more off-the-shelf solutions.

    Take advantage now:

    • Map how infrastructure is used throughout your value stream, and work on unifying the approach in product-level.
    • Make sure you build self-service solutions with strong focus on the end-user in order to enable and balance both speed and control. Make sure developers and ITOps are equally represented when determining an approach to infrastructure automation.

    Infrastructure Automation

    Infrastructure automation is going to get smarter, with a lot of AI and ML. Making sure automation is properly governed will also increasingly be important. Take into consideration security and compliance, as well as cost.

    Take advantage now:

    • Assess infrastructure automation practices throughout your value stream. Who builds the automation? What teams and systems consume it? How do you enforce security best practices (e.g. no passwords in public repositories, using proper data sets for testing, compliance of automation blueprints)? How do you enforce cost control (e.g. tagging, reviewing cloud spend reports)?
    • Invest in control mechanisms before you lose control, especially in public cloud automation. If you don’t have governance in place, look at baking governance into automation, rather than adopting traditional cloud management approaches. Invest in automatic de-commissioning of infrastructure, automatic tagging, analysis of infrastructure consumption data and connecting it back to the business requirement, secrets and key management and access control.

    SaaS

    Open source is nice. SaaS is great. While big open source projects set the pace (K8S, Github) and are strongly backed up and funded by the industry’s giants, smaller projects are increasingly difficult to scale and glue together, especially when trying to assure enterprise readiness. The need to “do it yourself” is gradually fading, as more vendors offer products and services for supporting DevOps with clear business and support models and, as important, with OOB integrations with the ecosystem. Just Rest API is no longer enough.

    Take advantage now:

    • If your enterprise-readiness requirements make it difficult for you to scale DevOps initiatives, consider evaluating new SaaS offerings that eliminate much of the heavy-lifting and gluing that were typical to the skills-heavy early-days implementations. If you were concerned with security, cost and skill requirement, today you can find tools that are purpose-built to support DevOps for enterprise at scale. The good news is that most of the vendors are well-aware of the criticality of integrating to your ecosystem, making it easier to achieve your end-to-end value stream vision.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Maya Ber Lerner