Tag: tooling

  • Implementing SLOs-as-Code: A Case Study

    Implementing SLOs-as-Code: A Case Study

    This article was co-authored by Stephan Lips, Devin Cunningham and Justin Hoang of Procore and is a preview of their talk for SLOconf 2023 on May 15-18, 2023. To watch this talk and many more like it, register for free at sloconf.com.

    By managing service level objectives (SLOs) as code, we can co-locate SLO definitions and ownership with the product code and team. This supports horizontal scaling of SLO ownership while establishing a single source of truth and adding transparency, integrating with the code management process, and creating an audit trail for SLOs.

    When you measure the reliability of hundreds or even thousands of products and services across an enterprise, ownership of SLOs should not reside with a single team. There are a few aspects that make scaling SLO ownership horizontally across teams much more efficient: A standardized SLO management platform, process architecture and automation across the enterprise. Managing SLOs-as-code alongside product code is a step towards 360° product ownership and enables us to automate SLO updates via continuous integration.

    Objectives and Architecture

    The larger objective behind the SLOs-as-code approach is to move towards 360° product ownership. At Procore, teams own their product’s code and processes, including aspects like testing, performance, reliability, automated deployment pipelines and deployment configurations. Adding SLO ownership to a team’s portfolio will also help improve product reliability.

    Implementing SLOs-as-code involves three architectural components, which will be discussed in detail in their own respective sections. In summary, these components are:

    • SLO objects and definitions: Standardized SLOs and related resources facilitate centralized management and automation.
    • Automation: SLOs as code are our single and reproducible source of truth, which we automate via Continuous Integration.
    • A common platform: Nobl9 is the SLO management platform upon which SLOs, related resources, and automation are built. It aggregates the various data streams that power SLIs.

    SLO Objects and Definitions

    Procore’s Observability team has designed our SLO-as-code approach to scale with Procore’s growing number of teams and services. Choosing YAML as the source of truth allows Procore a scalable approach for the company through centralized automation. Following the examples put forth by openslo.com and embracing a ubiquitous language like YAML helps avoid adding the complexities of Terraform for development teams and is easier to embed in every team’s directories. We used a GitOps approach to infrastructure-as-code (IaC) to create and maintain our Nobl9 resources.

    The Nobl9 resources can be defined as YAML configuration (config) files. In particular, one can declaratively define a resource’s properties (in the config file) and have a tool read and process that into a live and running resource. It’s important to draw a distinction between the resource and its configuration, as we’ll be discussing both throughout this article. All resources, from projects (the primary grouping of resources in Nobl9) to SLOs, can be defined through YAML.

    Figure 2. A simple visualization of Nobl9 object relationships. A Project is a top-level object. Projects contain Services, SLOs are attached to Services, and SLOs can trigger alerts via project-scoped Alert Policies. Finally, Role Bindings define who has access to the Project contents.

    Procore adopted a hybrid approach to organizing our Nobl9 configuration so that the observability team can review systemic changes while teams still own changes to their service SLOs. A separate central repository is the source of truth for all other Nobl9 configurations, such as Project and Alert Policy definitions. As projects have a one-to-many relationship with their services, it would quickly become a game of “guess where” if the project configuration were defined within one of its service repositories. The central repository is owned by the observability team and allows us to manage permissions through pull request reviews submitted by product teams for their non-SLO Nobl9 resources. Once requests are merged, our automation applies the changes. The SLO definitions are co-located with the service’s code. Teams self-regulate these resources to maintain their SLOs internally.

    Automation

    Automating the creation and modification of Nobl9 resources is essential to Procore — it makes iterating on the deployment process quick and painless for our engineering teams. Automation removes human error and possible complications that usually come along with manually applying config files via CLI tools.

    With that in mind, the observability team created a CI job/workflow that our engineering teams copy into their project repo during their SLOs-as-code onboarding. Procore’s CI job uses the Nobl9 sloctl docker image, so we do not have to install the sloctl CLI tool in our CI containers. We configured this job to only apply the configs that have been added or updated, which helps to future-proof our pipeline as we scale the number of our SLO configs.

    Figure 3: SLOs-as-code Automation Workflow

    This requires the following steps in the workflow:

    1. An engineer creates or modifies a config and commits the changes to GitHub.
    2. The engineer opens a PR, gets reviews and merges their changes into the main branch.
    3. CI picks up the changes in GitHub and kicks off the following steps:
      1. Using git diff and regex pattern matching, we figure out what configs have been added or changed from the most recent merge to main:

    #!/bin/bash
    ADDED_OR_CHANGED_CONFIGS=$(git diff –name-only –diff-filter=d HEAD^ HEAD | { grep -E “^${NOBL9_RESOURCE_DIR}\/.*\.y[a]?ml$” || :; })

    2. We loop over the added/changed configs and run sloctl apply using the nobl9/sloctl docker image, which updates our resources in Nobl9:

    #!/bin/bash
    for file in “$ADDED_OR_CHANGED_CONFIGS”
    do
      docker run nobl9/sloctl:v0.0.80 apply -f “$file”
    done

    3. We use the same workflow in our centralized IaC repo that manages our projects, services, role-binding and alert policy configs.

    Common Platform

    Our platform requires another key feature for SLOs-as-code to work: We need to be able to consolidate different observability data sources and telemetry streams that power SLIs. This platform needs to support standardized data formats to define SLOs and related entities, such as alert policies for different conditions and trends, while allowing for the management of these formats via automation. We evaluated three vendors that offered SLO solutions, and only Nobl9 met all our requirements, which extended beyond the SLOs-as-code topic of this article.

    Observations

    Adopting SLOs-as-code can be impeded by three particular barriers of entry: New tooling (Nobl9 platform), data and format (SLO configs and resources) and process. To drive adoption, we have designed a self-service model supported by an open-door consulting practice. While this is an ongoing learning process for all involved, we are seeing early signs of success of an approach that uses:

    • Detailed, step-by-step instructions, including code snippets and templates.
    • Screencast tutorials supplementing the instructions.
    • Initial face-to-face, white-glove meetings to explain concepts and provide demos. The personal experience and conversation in our experience created considerable enthusiasm and goodwill among our teams. It proved to be a much better approach than just pointing them to documentation.
    • Fast feedback cycles with onboarding teams to improve instructions.

    Quite frequently, teams stated their primary challenges were not the technical aspects of SLOs-as-code discussed in this article, but how to design and improve on meaningful SLOs. We addressed this by authoring articles like Black Box SLIs and other internal resources for guidance. While posts can be a useful tool, we found that working with teams and discussing how their product impacts the user experience was more beneficial to modeling their SLOs using observability data.

    Last but not least, we are integrating with our embedded SRE (ESRE) team to help onboard more services. When a product team engages with ESRE on their deployment strategy and configuration, ESRE also discusses onboarding to SLOs-as-code and basic SLOs, such as error rate or duration.

  • DevOps and Automation Abstraction?

    DevOps and Automation Abstraction?

    Similarity breeds consistency, giving way to predictability and, thereby, dropping costs of operations and quality from a lower number of incidents and a shorter duration to resolve them. But at the heart of this chain of events is the recognition that we tend to do certain tasks the same way, over and over again, for a given type of technology. Variability then becomes our enemy.  Consistency becomes our ally.  So the question: How do we as DevOps service providers instill a level of consistency in what we do that is not engineer-specific (i.e. only one guy knows how it is supposed to work) and will still scale at the enterprise level?

    From a people perspective, we use teams, whether co-located in person or virtually using the same backbone of tooling over wide geographical separation. From a process perspective, we use governance and education to ensure the same processes and approaches are taken at a fundamental level across different types of technologies. And finally, from a tooling perspective, we use templates or frameworks (or pick your synonym) that allows us to construct a shell that guides the automation construction to be consistent from parent to child. The combination of these three fundamental elements leads to consistency that scales. But is that all?

    What Are We Missing?

    Could there be another fundamental element we have overlooked—namely, abstraction? Case in point: Consider the effect of a drag-and-drop interface versus a typed one to accomplish a given task. Both look to accomplish the same outcome. One could argue the typed text approach offers more flexibility and customization to get the task done. But the drag-and-drop method offers a consistent, simplistic way of doing a similar task, the same way, over and over again. Examining both methods objectively, the drag-and-drop method offers a level of consistency the typed method never could. For the typed one, you would always have to check to ensure punctuation and lack of changes (additions or deletions) to “know” it was the same script as last time. The drag-and-drop method on the other hand, is “canned” or largely unalterable; it may limit customization (by design), but it excels at consistency.

    The drag-and-drop method achieves this consistency by effectively abstracting the typed or scripted code within it from the user who employs it. The user only makes the choice to use it, when desired (or needed); the rest of the mechanics are abstracted from the user. The reason drag-and-drop methods exist is beyond just consistency, however; it is for simplicity. Apple computers would appear to be founded on this idea. For those of you old enough to remember DOS on PC computers, the parallel here immediately becomes clear. DOS was a typed-based command-line operating system (OS) fully capable of operating a PC many years ago. But Apple’s graphical interface made the same functions “intuitive” and forced the PC industry into Windows to stay competitive. Nobody chooses a command line OS like DOS when something as elegant as iOS exists, or Windows (or pick your graphical poison here).

    A graphical OS abstracts core functions from the user, and makes every attempt to make an elegant intuitive packaging of the same functions. Users “get” an iPhone, which accounts for its sales. You don’t need formal education classes or an eidetic memory to use an iPhone. Users effectively have been abstracted from all the typed code it takes to create an OS like this, millions of lines of such code packaged up into swipes, beeps and elegant graphical displays. Keep in mind, at its core this OS abstracts users from variability that might exist if hundreds of companies created their own versions of iOS. The simplicity is paired with consistency, and the quality of Apple is a result (for better or worse). But whatever the quality of Apple is, it is far better than if a wide degree of variability were permitted in the vendors of iOS or the construction of it.

    Extending Abstraction in DevOps

    Now enter DevOps. Containers are a level of abstraction widely popular in our discipline for keeping hardware particulars away from what are typically software engineers. The value of this abstraction is well-understood. My challenge is simply to extend this thinking into the next tier of DevOps tooling. What if the ability to include database DevOps automation was simply a product of drag-and-drop methods that every build tool understood perfectly? Granted, you may have to limit the expression of this idea to say Oracle or SQL Server at first to capture the nuances and particular capabilities of products and technologies in this space. But if someone built a “container” for Oracle, I would wager the choice to use Oracle would rise substantially over its competitors, at least until those containers or abstraction layers were just as effective with other database products.

    The abstraction phenomenon need not be limited to hardware, or as I have proposed to database platforms. It also could apply to .NET or Java development efforts at the top of the stack. The steps needed to take an idea, code it and put it through the build process are going to be largely similar when working in a .NET shop or a java shop (or pick your particular development tool of choice here). Even the top of the stack—that is to say, the development of applications itself—could be a viable candidate for abstraction from the mundane details as it relates to a build construction or a deployment to a given class of environment.

    Creating effective containers for the application layer in effect limits variability, and increases predictability. I realize it has to be technology-type sensitive. The .NET shops have their own container. The java shops have theirs, and so on. The only question is whether we are able to include enough of the common core steps we take to produce a build with a given technology type to form an abstraction construct of value. Over time, this would include every type of technology widely in use today.

    The effort to simplify through abstractions ultimately would result in our ability to include technologies such as WebSphere at our pleasure. It would also allow us to switch out WebSphere for a new competitor using the same abstraction constructs, universally, just by rebuilding all our apps one time. This kind of power is alluring to development teams. It drives application developers and operations engineers to begin to work together to look for the most cost effective technology that can perform a given spec. Being the best at something is no longer a cage from which there is no escape later.

    Technology components would be selected based on performance and cost of ownership (i.e. operations) later. Switching is ultrasimplistic if all we ever used to include it in our build routine was a drag-and-drop style method with universal, well-defined parameters. A customer’s ability to switch products to the best of breed through easy or simplistic means keeps pressure on vendors to actually be the best of breed. If the performance suffers or a new release is full of bugs, developers just switch it out through abstraction (not long, complicated scripts where everything has to go right, including a full moon on the third Tuesday of the month with a Gregorian chant in the background).

    Impacts to the Bottom Line

    The selection of DevOps tooling often is a cantankerous one in large organizations. Debates ensue about flexibility, extensibility and whether a given tool can be customized to accommodate the unique little snowflake of my development teams. Perhaps these debates are headed in the wrong direction. Perhaps instead of prizing and valuing the extensibility or customization a given tool affords, we should be looking closer at how that extensibility is achieved.

    For an enterprise to make an unwitting choice to get into the script business, is effectively stating: “I will be maintaining the scripts I use to construct a build with as much rigor as I maintain the code these scripts are designed to create builds out of.” So, we just doubled our programming staff without realizing it—one half working on the application code and the other half working on the scripting code that “automates” the delivery of a build during construction. While overall speed may be increased, it comes at the cost of doubling already expensive headcount.

    As an alternative, what if enterprise customers were asking tooling vendors how simplistic their tools are to use, and how standardized are their approaches. While we may not enjoy losing a degree of flexibility as we may be unable to customize a drag-and-drop method construct, the payoffs to the business are simplicity of use and scale at an enterprise level. Not only do I refrain from doubling headcount, I can actually free resources to focus on the disciplines and areas they enjoy most instead of performing script maintenance. Abstraction as a construct: From the hardware layer up through the application layer, it may be a driving fundamental that keeps DevOps scaling and lowering the cost of innovation for many years to come—but only if you begin to ask for it.

    To continue the conversation, feel free to contact me.

  • Delivering DevOps’ Business Value

    Delivering DevOps’ Business Value

    DevOps is an interesting space to be operating in right now. More than just culture or a set of tools and processes, DevOps delivers speed and agility that can be truly meaningful for the business. Unfortunately, for most large enterprises, the path to the benefits isn’t always well-known and isn’t assured.

    Over the last five to 10 years, a number of DevOps tools and solutions have emerged: Puppet, Chef, Jenkins, Git, SaltStack, Elasticsearch … and the list keeps growing. Originally, these tools were sought by developers and system engineers for their ease of use and low cost of ownership: You could download it, use it the same day and solve your specific problem. The issue for enterprises,  however, is that nature abhors a vacuum; the problem one developer addressed with Chef is being fixed with Puppet by another developer in a different business unit. Group A is using Jenkins, Group B is using Chef Delivery. In short, after five to 10 years of technology adoption by developers and system engineers, the tooling landscape in the DevOps space is packed to the rafters with competing technologies, all of which are solving similar problems with overlapping features.

    So why should this matter to you at all? Without careful planning, consideration and execution, your business will derive little value from this DevOps exercise.

    Here are a few common pitfalls we see far too often:

    Too Many DevOps Tools Doing Too Few Things

    Building Rube Goldberg devices looks neat on paper, but those mousetraps fail very often in real life. The more tooling and solution handoffs you have, the greater your likelihood of failure: the less likely you are to be able to scale horizontally or vertically, the more toolsets you need to maintain, and the more skills you need on your team. Convoluted messes do not yield speed and agility. If it feels wrong, it probably is. If it is overwrought, look to remove complexity from the landscape.

    Reintroducing Risk and OpEx Spent on Menial Tasks

    As an organization you’re onboarding risk, hidden costs and a lot of the things you just spent the last 10 years automating away: tribal knowledge, lack of governance, lack of compliance, lack of third-party auditability and reporting. Many enterprises spend millions of dollars making it so their admins and developers could spend time on more valuable workstreams, just to throw them back into the salt mines of making continuous integration (CI), continuous delivery (CD) and infrastructure-as-code “work” over umpteen toolsets. The goal isn’t for DevOps to “work”; the goals need to tie to some tangible (i.e. measurable) business benefits. Anything can be made to work, but far fewer IT programs deliver value. Be the people who deliver real value.

    Taxation Without Representation

    All of the stakeholders need a seat at the table. Whether that means forming a SWAT team from operations (all infrastructure domains and security), development, QA or a permanent DevOps matrix organization, all vested interests need a say in how the process and tooling develops for all impacted domains (which is just about everything) across all environments (i.e Dev, Test, QA, Production). Far too often we see one set of stakeholders acting as the service provider for another group of stakeholders playing the role of customer. This is a solution for and by Development and Operations.

    Congratulations, you’re having a baby. Let’s make sure it’s not ugly, okay?

    About the Author / Jesse Langhoff

    Web_Langhoff_Jesse-7633-EditJesse Langhoff is co-founder and CEO of Raindance, a cloud and automation consultancy serving enterprises nationwide. With over 15 years of experience in leveraging technology to address business challenges, Jesse works with customers to determine the optimal path to meeting their goals through the use of automation and solutions provided by the leading vendors in the space. Areas of expertise include strategy and assessments, private/hybrid cloud services, process automation and orchestration, DevOps and operations as a service. Jesse holds an MBA from University of St. Thomas – Opus College of Business.

  • Tool Development – In-House or Ready made

    Tool Development – In-House or Ready made

    Tool development and adoption

    The current pace of software development cannot manage its speed without the automation.

    Automation is the most important element in order to achieve agility in delivery of the product. Every part of the software development life cycle needs tools to perform their action automatically. These phases could be coding, code review, build & deployment, environment management, resource assignment, notification and many others.

    There could be 2 options, which a team can opt for:

    • Use a third party tool
      • One can select the tools, which are available in market, either open-sourced or free. You can even buy enterprise solution if needed.
    • Build an in-house tool
      • Tool can be built or developed in-house for the consumption as well.

    Lets take look different aspects of choosing options

    • Features

    Each organization and team may have different features for different use cases. Some time it makes more sense to build your own tool just for your need rather adaption of a third-party one for which most of the features are not going to be used.

    • Time for on-boarding

    Time is definitely a big factor here, if one thinks of developing in-house tool and use it then this will definitely take time. If, need of tool is not urgent or critical, in house development option could be considered.

    Third-party tool can be immediately on boarded and there won’t be much delay other then some logistics reason.

    • Effort and Resource

    If the tool is complex, its not easy to build, one will need resources and lots of time to build it and then next challenge is implementation.

    Above effort can be reduced by adaption the tools which is available in the market. There are chances that other users are aware of the tools and training time could be reduced.

    • Cost

    Lets discuss one more important part, which is the cost. Yes, money is also one of the biggest factors on deciding what to opt for. There may be good options to choose from free and open source tools. But if enterprise tools selected, then cost may incur.

    You will need a dedicated team and resources to build the tool with in organization.

    Its hard to decide if in-house tools will be costly or enterprise. Decision has be to taken by considering long term investment and return.

    • Support

    Enterprise tools usually come with a proper support. This is really important factor if tool is very critical. Open source tools may not have enough support, as it is community driven effort. In-house tools may need resource to provide support and training in the team/organization.  Even bug fix needs to be taken care under support category.

    • Adaption/Expertise

    In-House tools are highly customize as per use case and will need proper training for adaption with in organization. In case of thirdparty tool, as I mentioned earlier there are chances that other users are aware of the tools and training time could be reduced.

    So, Adaption of tool is not an easy task. Decision has to be taken with a very strategic thought process.

    If tool is complex then it is definitely not a part time task.

    One more important thing to consider is if the team is trying to build in-house tool, which is going to be major business for somebody else then it has to be developed with proper planning and priorities, otherwise there are high chances of failure.

    Please Note- I am not considering or referring to any tooling like script or some simple UI dashboard.

  • The Human Factor in DevOps

    The Human Factor in DevOps

    DevOps is about pinpointing and eliminating bottlenecks in order to deploy code often—shortening the time-to-market and increasing quality and effectiveness. One of the most common bottlenecks in the release process is the interaction between the developers and the integration services/operations group.

    Development (Dev) is entrusted with producing innovative software quickly while Operations (Ops) knows that service failures are often attributed to changes. This means Ops feels that if fewer changes are executed, fewer failures will occur and the system will be more stable and responsive. The natural conflict and opposing priorities between the two groups can create a productive equilibrium, but often results in communication difficulties, animosity and a financial loss for the company.

    (more…)