devopsy.kinsta.cloud

Category: Doin’ DevOps

  • The Hidden Configuration Tax Affecting Uptime SLO

    The Hidden Configuration Tax Affecting Uptime SLO

    This article is a preview of a talk by Greg Arnette for SLOconf 2023 on May 15 – 18. To watch this talk and many more like it, register for free at sloconf.com.

    Tax season was just a few weeks ago, and that got me thinking about how frustrating it is to get hit with a surprise tax bill and how that can relate to uptime: Configuration sprawl. DevOps practitioners constantly strive to maintain uptime service level objectives (SLO) while working with complex and ever-evolving infrastructure and applications. Configuration sprawl is a significant challenge many teams face, negatively impacting system performance and reliability.

    My company’s recent survey of 900 DevOps and platform engineering leaders revealed a startling (but not too surprising) conclusion: Most teams struggle managing secrets and configs at scale for infrastructure and applications.

    Where Does Configuration Sprawl Come From?

    As organizations scale their operations, the number of configuration files and settings grows, making it increasingly difficult to manage and maintain consistency across the board. They may have multiple environments, such as development, staging and production environments, often requiring different configurations, leading to duplicated settings and increased complexity.

    Configuration sprawl may also lack standardization as teams adopt different naming conventions, file formats and storage locations without enforced configuration standards, resulting in fragmentation and confusion. Relying on manual processes for configuration management increases the likelihood of errors and inconsistencies.

    Grinding to a Halt

    So you’ve got a lot of configuration–why should you care? Sure, it means more files to manage, but does it really impact your environment? There are a few places where configuration sprawl will negatively impact you. You might see an increase in error rates because as complexity grows, the likelihood of errors and misconfigurations increases, leading to system instability and downtime.

    Increased configuration complexity also means slower troubleshooting. Identifying the root cause of issues becomes more challenging and time-consuming when configurations are disorganized and fragmented. Ultimately this means less agility as you are forced to slow the deployment of new features and updates, impeding the organization’s ability to respond to market demands.

    How to Overcome Configuration Sprawl

    Don’t worry–you can tackle configuration sprawl, keep your SLOs on track and reduce the efforts required to manage and maintain configurations. To mitigate the harmful effects of configuration sprawl, consider adopting these best practices:

    Don’t repeat yourself (DRY)
    DRY config means variable names are declared once. Then flexible variable values are injected across build, deploy and runtime phases as defaults, inheritances and overrides. DRY avoids errors by allowing single changes to propagate where they are needed.

    DRY solves the common problem of forgetting to update critical secrets or parameters and failing to synchronize related changes (e.g., client and server port numbers or rotating a database password.)

    Decouple and abstract
    Decoupling and abstracting config means externalizing config from source code, separating how code and configuration interact, and isolating the specific values (log level) from the configuration interface (how the log level value is fetched.)

    This approach allows the same code to run in different environments using parameterized config. Abstracting config supports a best practice of implementing independent “code and config change” life cycles. Teams that frequently need to change config independently from code should decouple and abstract.

    Centralized config
    Consolidate configuration complexity into a single location because it’s easier to manage complexity within a single location versus multiple locations. Syncing config to the edge (where it is consumed) mitigates downtime. For example, it’s easier to fetch the configuration for service X versus coding all the logic needed to generate the configuration for service X.

    Standardize your configuration management by enforcing consistent naming conventions, file formats and storage locations across teams to reduce confusion and streamline configuration management. Centralizing the configuration and generating that configuration with transformations allows all the places that consume the configuration to be simple. Simple equals easy to understand and troubleshoot.

    Version control & monitoring
    Use a version control system like Git to track config changes to configuration files, making identifying and resolving issues easier. And continue to monitor changes and identify potential problems before they impact system performance.

    Train your team
    Ensure all team members know your best practices and patterns for configuration management. The proliferation of microservices and teams operating in parallel, establishing their unique naming schemes and patterns, exacerbates this problem for DevOps teams.

    Cutting the Uptime Tax

    By understanding the causes of configuration sprawl and implementing best practices to manage and reduce its impact, teams can improve their uptime SLOs and maintain a more reliable, high-performing infrastructure. Remember, investing in the proper tools and processes for configuration management is not only a best practice but also a critical factor in ensuring the long-term success of your DevOps initiatives. Those are tax cuts I can get behind.

  • Nixpkgs: How to Build Better, More Open Software

    Nixpkgs: How to Build Better, More Open Software

    The world we know today depends in fundamental ways on open source software. As providers of this software, package managers serve their respective ecosystems from their centralized, opaque hubs in ways that are sometimes difficult to reconcile with the open source nature of the software they’re serving. I want to talk about the extent to which that’s true and discuss Nixpkgs—the largest open source package repo in existence today—in this context.

    People sometimes focus on Nix the language or Nix the package manager, but it’s the Nixpkgs package collection (pronounced ‘nix packages’) that is core to the success of Nix. But it’s more than that: We can also understand Nixpkgs as a powerful tool to ensure the continued growth, transparency and resilience of open source software, first and foremost because it democratizes the process of curating this software and its build instructions. It is the living compendium of the efforts of thousands of contributors who both stand on the shoulders of giants (the software they’re packaging) and provide the ladder for others to climb further.

    A Purely Functional Grimoire

    The Nix package manager is different from other package managers in several ways, but one key aspect is that Nixpkgs contains only recipes to build packages, not the packages themselves. This is a fundamental idea to Nix: If we build a package reproducibly, we can separate the specification of a build from its built output. When we build a “recipe” for a package, we determine the output hash of that build–without actually building it. This means you don’t need a cache of build artifacts: Recipes are enough to rebuild your software world.

    Nixpkgs is just that: The big fat recipe book of open source. Anybody can view it, fork it, patch it and use Nix to inspect the largest set of free software recipes in the world. Anybody can create a cache that uses those build recipes (or your own!) to publish packages: The default cache is the official one run by the NixOS Foundation at cache.nixos.org.

    Derivation and Substitution

    So what happens, in practice, when I download a package from the Nix cache? When you query for a package, the CLI creates the specification (we call it a derivation) of the package to be built. This derivation includes a hash: The summary identifier of all source code, patches, build flags and hashes of the dependencies themselves.

    With that output hash in hand, Nix the package manager just asks whether that path exists in its cache. If it does, Nix fetches the contents of the package you asked for; if it doesn’t, Nix will build the package locally. We call this move substitution: Creating a derivation from a Nixpkgs expression gives you all the information you need about the output of the build, so the cache can substitute a binary with that output hash if we have it. Of course, if you end up having to build the package locally, it will result in the expected output hash anyway.

    Democratizing Package Curation

    The term ‘lingua franca’ refers to the dialect spoken by sailors across the Mediterranean in previous centuries, and it was the shared language with which one could communicate and do business from Morocco to Greece. Today we package software for independent islands, each with its own rules, languages and cultures and, until Nix, we had not found a way to “speak” across the many ports-of-call of our software archipelago. What this allowed for is packages being available on more platforms, which drew more developers in, which resulted in Nixpkgs becoming more complete, and so on in a positive feedback loop.

    Nix is a cross-platform, cross-architecture package manager. If we had to design the next-generation package manager, this would probably be at the top of the requirements list, yet few package managers enable this today. Each of the teams at RedHat or Debian maintains their own packaging format and package cache and has a fixed team of vetted contributors to keep packages secure and up-to-date. This results in few who effectively contribute to packaging open source projects and do so for one of many platforms.

    An open book of package recipes written in this lingua franca results in another surprising effect: Nixpkgs maintains more packages than anybody with an order of magnitude fewer maintainers [1]. It makes sense: Since it’s a common language, people can review the packaging code for packages they don’t necessarily know intimately, which in turn leads to contributors and reviewers cycling through updates faster and providing more up-to-date packages, which reinforces the previous feedback loop from earlier.

    Even if the rest of the Nix ecosystem stalled out, we’d still have this incredible resource in Nixpkgs to curate and share open source software; flox sources its packages exclusively from Nixpkgs for this (and many other) reasons. As of February 2023, there were around 6,000 contributors on the Nixpkgs repository and growing.

    (via (repology.org)[https://repology.org/repositories/graphs])

    Stay Unstable, my Friends

    Nixpkgs embodies a new model for package management and one that opens it up to more people to contribute. Providing cross-platform and cross-language packages is obviously attractive, but it’s the fact that it provides a lingua franca for software packaging that makes it a pillar of open source moving forward. Because Nixpkgs unifies a broader swath of software and makes it easy for anybody to contribute new packages and fix existing ones, it continues to grow and serve the open source community: A positive feedback loop propelling the existence of a more free and open world of software.

    References:

    Nixos.org

    https://reproducible-builds.org/

    https://wiki.debian.org/SecureApt

    https://rpm-packaging-guide.github.io/

  • Why We Need to Rethink Data Pipelines

    Why We Need to Rethink Data Pipelines

    Data is the fuel that powers modern business. But as demand for data surges, so does the pressure on data leaders and practitioners to deliver it. Businesses need resilient data pipelines that deliver critical insight for real-time decision-making to users on demand. However, against the backdrop of today’s chaotic modern data ecosystems, this is much easier said than done.

    DevOps teams are struggling with a complex combination of legacy and diverse technology systems across multiple environments. The resulting siloed systems and an ever-shifting data supply chain cause significant data integration friction. This friction keeps businesses from creating seamless and resilient pipelines to drive digital transformation and other business outcomes. Businesses need a better way to take the pressure off data leaders and DataOps practitioners and enable line-of-business teams to do more of the “last mile” data collection and analysis themselves. To achieve this, they need a unified, end-to-end platform designed to build resilient pipelines.

    Data Demand Sparks Friction

    The acceleration of digital transformation has created a demand for data that far exceeds supply. This has become especially true as competitive advantage gets harder to carve out and worrying macroeconomic headwinds gather. In today’s climate, all areas of the business are demanding data, but it’s hard to meet every request all the time.

    Research that polled data leaders and practitioners shows nearly half (48%) of admin and operations and customer service departments request data at least weekly, followed closely by accounting and finance (44%), IT and digital (43%) and sales and marketing teams (40%). And the strain is showing. Over half (59%) of data leaders said changing priorities have created significant data supply chain challenges.

    The problem is one not just of capacity but complexity. Building pipelines from source to destination requires rules to integrate, transform and process data. But when data is siloed across cloud, legacy and mainframe systems and stored in inconsistent formats, creating bespoke data pipelines to fulfill departmental requests is a huge challenge.

    Legacy systems also bring another challenge. Over half of data leaders say it’s so difficult to unlock data from legacy systems like mainframes that they simply don’t bother. That will ultimately undermine the value of the cloud analytics tools that many business leaders see as a single source of truth for decision-making–especially as legacy systems often hold decades of vital business insights.

    This patchwork approach creates extra toil for DevOps, DataOps and technical teams and means that over two-thirds (68%) of data leaders said they are being prevented from delivering data at the speed the business needs.

    The Scourge of Broken Pipelines

    For many organizations, building pipelines is also a labor-intensive job requiring a high degree of manual effort to produce hand-coded, one-off solutions. The resulting pipelines are brittle and vulnerable to being disrupted whenever there is a shift in the environment–such as adding new data sources. In fact, two-fifths (39%) of data leaders admit their pipelines crack at the first sign of trouble, and 87% have experienced a break at least once a year. More than one in 10 say it happens at least once a day.

    The most significant cause of breakages is due to bugs and errors being introduced during a change (44%) while infrastructure changes such as moving to a new cloud (33%) and credentials altering or expiring (31%) are also creating disruption. These broken pipelines immediately impact the corporate bottom line. They also force technical teams into a vicious cycle of firefighting. Data engineers spend, on average, almost a third (31%) of their time troubleshooting and recoding broken pipelines–time that could be better spent on value-adding tasks.

    But the negative impacts aren’t only felt operationally. The breakage of any pipeline can also lead to bad decision-making. For example, a supply chain director working with old data may over- or under-order goods or a financial trader making stock picks on out-of-date intel may lose money for a client.

    Unleash the Power of Data

    For organizations looking to overcome these challenges, the first goal is to reduce the workload on DevOps, DataOps and technical data teams by empowering business units and end users to do more. Research shows that 70% of technical data leaders are responsible for the last mile of data collection and analysis while 86% of them would prefer lines of business teams, such as marketing or finance, to be empowered to do this independently. However, this can only become a reality if organizations invest in the right data integration platform. It must be able to build, run, monitor and manage smart data pipelines at scale from a single console–across all cloud and on-premises environments. And it must be able to deliver resilient pipelines built to withstand continuous change.

    Only by eliminating data integration friction and enabling self-service analytics for lines of business can organizations accelerate digital transformation. Doing so will unleash the power of data across the enterprise and reduce the burden of overworked data leaders and practitioners.

  • A Supportive Engineering Culture is Key for Recruiting and Productivity

    A Supportive Engineering Culture is Key for Recruiting and Productivity

    Skilled engineers are in high demand, as their ability to build new products quickly is extremely valuable for businesses competing in a tight market. Strong engineering talent allows businesses to increase productivity and meet customer demand faster, which is essential for long-term success. However, in the current economic context, companies are being forced to reduce spending while trying to achieve the same results. This means engineers have fewer resources available to achieve their goals–increasing the pressure on their role. At the same time, engineering talent remains a valuable asset; in the UK, for example, just under half (49%) of engineering businesses struggle to find workers with a sufficient skillset for the job.

    So, businesses must find a way to attract and retain talented developers while ensuring they feel supported through challenging times. A vital step to achieving this is creating a workplace culture that promotes developer satisfaction. Below, I explore how businesses can create a positive working environment for engineers, make transparent decisions and ensure their teams feel supported – to create a culture where developers want to stay and supercharge business performance.

    The Key Pillars to Creating a Positive Culture

    Leaders of all stripes are tasked with ensuring that their employees are supported, engaged and motivated to do their best work. However, the route to achieving these outcomes can sometimes seem vague and elusive.

    Research shows that there are several key pillars for creating a strong organizational culture. The first is psychological safety. Developers must have the freedom to take risks without the fear of being criticized. Software engineering is an industry in which professionals must be creative enough to be able to create something out of nothing, so it’s important that companies ensure developers have the space they need to experiment and try new approaches.

    Secondly, dependability is essential. Developers are more likely to produce high-quality work in a timely manner if their employer believes in them and their abilities. Developers often face immense pressure to ship functional code on a daily basis; feeling trusted can give them the confidence they need to take on more responsibility and drive projects forward. Alongside this, it’s the leader’s role to ensure developers have a very clear and structured mission. They must know exactly the goals they’re aiming to achieve, the part they’re playing in reaching these goals and the plan for execution.

    Finally, ensuring developers understand the meaning of their work and are fully aware of the impact it has is an important pillar for cultural success. Sometimes, software engineers spend a huge amount of time on small fixes and updates that may seem insignificant. Business leaders must make efforts to communicate the importance of developers’ work and the value they bring to the business. If developers feel valued and rewarded for their difficult work, they will be more motivated to commit themselves to their mission and stay at the organization.

    When engineering teams are happy and supported, they are also more likely to speak highly of their organization and encourage other developers to join the team, helping the company build a strong pipeline of engineering talent.

    Making Transparent Decisions Using Concrete Data

    Transparency is another crucial element to ensure engineering teams are happy and motivated to do their best work. Developers can’t do their jobs effectively if they don’t have insight into the workflows on their teams or don’t have context into decisions that impact their ability to perform effectively.

    With tools such as software delivery intelligence platforms, developers can see the positive impact of their contributions and how they’re helping the business succeed. These tools also enable engineering managers to rely on concrete data to celebrate developer progress on previously invisible work and show how their work contributes to wider business success. This helps developers gain more autonomy to achieve the company’s goals, and eventually create better business outcomes.

    For example, Manulife adopted an engineering analytics tool to create an engineering culture that is based on trust and transparency. The concrete data generated is used to analyze engineering processes and highlight talent that may have remained hidden without a data-driven approach. Data also helps managers decide who within the team deserves to be promoted based on precise insights into everyone’s contribution.

    Use Data to Help Overcome Complex Challenges

    The mission of developers is constantly evolving, with new technologies, programming languages and methodologies changing very quickly and making the job more complex over time. It is up to engineering leaders to help their teams remove potential barriers that are blocking them from being successful in their roles.

    With data-driven insights, engineering leaders can better support and uplift their teams. Data about engineering workflows allows leaders to understand exactly where bottlenecks exist and remove them quickly before slowing their projects down. This means that employees can learn what worked–or didn’t work–from one project to the next, as well as develop the necessary skills to remove roadblocks to future success.

    Timelines and targets for projects set by business leaders can also then be tailored to each person’s abilities and based on concrete data that show where they can boost efficiency to ensure everyone produces their best quality work and remains engaged and motivated.

    In the current technology industry, employee culture is one big factor determining where developers wish to work. In a context where engineering talent is in high demand, it’s the job of every organization to create a stellar work culture to attract and retain talent.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    LinkedIn

    Northwestern Mutual
    Milwaukee, Wisconsin
    Assistant Director for Product Management for Cloud and DevOps Platforms
    $139,300 to $186,200

    CareerBuilder.com

    Brown Brothers Harriman
    New York, New York
    DevOps Operations Manager
    $130,000 to $190,000

    SimplyHired.com

    Spiff
    Salt Lake City, Utah
    DevOps Engineer
    $120,000 to $170,000

    Indeed.com

    Rose International
    Jefferson City, Missouri
    Azure DevOps Architect
    $114,000 to $145,000

    Dice.com

    ComTec Information Systems
    Remote
    DevOps Engineer
    $100,000 to $150,000

  • How LinkedIn is Investing in Developer Productivity and Satisfaction

    How LinkedIn is Investing in Developer Productivity and Satisfaction

    More and more companies are beginning to examine the role developer experience has in improving engineering productivity and morale. And one company often referenced as a model for developer productivity engineering is LinkedIn—its engineering leadership has invested heavily into reducing time-consuming developer tasks and constructing an internal platform to gauge developer satisfaction on the tools they use.

    Engineers at LinkedIn recently shared how the company has invested in measuring developer productivity and happiness as well as their strategies for automatically cherry-picking commits. I also met with LinkedIn engineering leadership for a closer look at how the organization is working to improve the developer experience across its internal platform.

    Below, we’ll take an inside look at LinkedIn Engineering’s efforts to enhance developer happiness. We’ll explore their approach to improving their internal software development trends and consider the benefits LinkedIn has gained throughout the process.

    A Look at LinkedIn Internal Developer Solutions

    Productivity is challenging to measure, said Jared Green, VP of engineering, head of developer productivity and happiness at LinkedIn. So, the company opted to take a different approach by including happiness as a variable in the equation. Thus, they measure not only how fast a workflow is but continually monitor how enjoyable the process is to ensure developers have a delightful experience.

    “We don’t just strive for productivity, but for productivity and happiness,” said Green. And to his point, on-the-job satisfaction can have real-world results—on average, happy workers are 13% more productive, according to a University of Oxford study.

    LinkedIn now oversees a custom-built internal platform for evaluating the sentiment of developer tools. The platform, known as iHub, is a holistic listening channel to gauge developer satisfaction across various tools. It collects real-time qualitative feedback from users and correlates it with objective quantitative telemetry. It then surfaces this data in a UI dashboard for team leads and individual contributors to view.

    But there’s an art to engendering the trust required to collect accurate feedback, says Max Kanat-Alexander, principal staff software engineer at LinkedIn. For example, individual contributors need to be sure the data won’t be used for performance reviews. Once that trust is established, folks are more willing to give genuine feedback, he said.

    Benefits of Investing in Developer Enablement

    The benefit of centralizing developer experience metrics with a shared platform means that tech leads can make data-backed decisions on how to improve their DevOps processes. “It’s really about thinking about giving people insights versus giving them data,” said Green. The idea is that frontline engineers can rapidly investigate any metric to expose problems and see trends. “The optics have given leaders ways to improve various parts of developer experience,” he said.

    Through their investment into developer experience, LinkedIn Engineering has discovered and streamlined hangups in the Git flow, exposed inefficiencies in their code review processes, and fixed specific developer build times. LinkedIn Engineering reported it has reduced time spent on certain time-consuming developer tasks by 99.6%.

    To construct iHub, LinkedIn Engineering wrote its own front end. The hub pre-computes data dimensions and stores them in an offline data store, increasing speed, said Kanat-Alexander. The system uses a sophisticated survey technology to collect the data and hides the complexity of manual collection in a sleek dashboard.

    LinkedIn managers have already leveraged iHub data to identify macro trends and resolve specific points of friction. For example, Kanat-Alexander describes how one director noticed that developer build time was worsening in their organization. They were able to isolate it to a single developer who wasn’t using remote development, which was spiking the metric. Leaders then assisted in unblocking them.

    Takeaways For Your Organization

    Most companies could benefit from a UI like iHub, explained Kanat-Alexander. But, you really must have a large engineering organization to see the benefits of the work. Plus, up to 80% of the legwork is simply data integrations—this involves hooking into systems, cleaning up data and deciding how to measure certain metrics. “Whatever you can measure, you can improve,” said Green.

    LinkedIn Engineering has flirted with the idea of open sourcing iHub, but it’s a bit too custom to LinkedIn to be valuable to other enterprises. “There’s a lot of variance in build technologies nowadays, and large-scale engineering systems are bespoke,” said Green. Implementing such a dashboard also requires a deep understanding of developer personas and how people work, which can be entirely unique for each organization.

    Although other organizations can’t fork LinkedIn’s efforts, there are some takeaways that could help others invest in their internal DX. First, Green recommended starting with the fundamentals. “Having a core of understanding about what you’re trying to move the needle on is important, as is coming up with ways to talk about it consistently.” This forethought should help establish what you want to measure.

    Turning DX Metrics Into Action

    Studies indicate that more companies are seeking to implement platform engineering or similar developer productivity engineering initiatives. The end result of these efforts should be to empower people and ensure they are satisfied with their work and not just a cog in the system.

    On that note, it’s critical that there are mechanisms for action to happen on the data that is collected, said Kanat-Alexander. Someone needs to monitor the insights and make changes to improve the metrics over time. “Then you can win hearts and minds.”

  • Black Box SLIs

    Black Box SLIs

    This article is a preview of a talk by Stephan Lips for SLOconf 2023, on May 15 – 18. To watch this talk and many more like it, register for free at sloconf.com.

    SLOs are fast becoming the industry standard to measure reliability and help teams decide when to prioritize it. The first step in adopting a service level objective (SLO) culture is to identify the metrics that matter without drowning in noise and alert fatigue. This article explores how to apply the black box concept to aggregate granular metrics into service level indicators (SLIs) that focus on the user experience as an indicator of system reliability.

    To SLI or Not to SLI

    In general terms, SLOs define targets for the proper level of reliability of a given product, such as a service or a website. SLOs are applied to or informed by SLIs. An SLI is a measurement determined over a metric, or a piece of data, representing some property of a service. And this is where we, as engineers, can get lost in the details, since the perpetual proximity to the systems we build and support often leads us to think of system reliability in technical terms or metrics (e.g., response time, error rate, throughput). While these are certainly valuable metrics, the user experience may be compromised even if the error rate is zero and the duration is well within SLOs. Consider, for example, the response data. Even if well-formed, it may not be current, or flat-out wrong. An error-free and quick response is of no value to a user that expects current and correct data. Error rate and response time remain valuable metrics and SLIs, but focusing exclusively on them would leave higher-level issues undetected.

    We could add freshness and correctness SLIs, but by doing so, we increase the number of signals we monitor. And with each signal—or SLI and associated SLO and error budget—we increase alert frequency and make reliability reports unnecessarily complex. In other words, adding SLIs may address a particular aspect of system reliability, but it also introduces additional complexities.

    Tales of Black and White Boxes

    So, let’s take a step back and borrow a concept from a related discipline: Quality engineering—in particular, software testing. Tests commonly fall into one of two categories: Black box tests or white box tests.

    In systems theory, the black box is an abstraction representing a class of concrete open systems that can be viewed solely in terms of its stimuli inputs and output reactions, without any knowledge of its internal workings. A given input is expected to result in a particular output, without any consideration for the processing steps. Common examples include end-to-end tests.

    White box tests, on the contrary, are designed with knowledge of, and to test, internal structures and workings of an application. Common examples include unit and integration tests.

    User Journey as Black Box

    Now that we understand the concept of black box versus white box tests, let’s apply it to our SLIs. As mentioned above, a good SLI considers the entire user journey. Conceptually, a user journey aligns with the black box paradigm: For a given input, a particular output is expected. For example, requests to our API (the “input”) result in responses that provide fresh data to clients within a given time frame (the “output,” including success criteria). There are several aspects worth mentioning with this SLI:

    ● The SLI is applied at a system level
    ● The SLI aggregates lower-level metrics implicitly and explicitly
    ● The SLI is binary; it is either true or false.

    These aspects combine to inform an SLI that represents the user experience (system level), via measuring many indicators by measuring only a few and supporting pass/fail attribution to an SLO target (by being binary). In other words, the user journey is measured as a black box SLI.

    White Box to Black Box: An Example

    Let’s consider a concrete example. A user requests a new account for a website. After the request is processed successfully, the user receives a confirmation email with an activation link. The user follows the link to activate the new account and log in. This workflow is visualized in the following sequence diagram.

    Of particular interest are the account creation and user notification via email steps. Both steps occur asynchronously. In particular, the event processing engine where the request for account creation is queued offers several opportunities for insightful SLIs: Queue length, average processing time, etc. Those SLIs, however, fall into the white box category: They contribute to the user experience, yet are opaque to the user (black box). The user journey begins with the initial request for account creation (input) and ends with the email containing the activation link (output). Rephrasing the example from earlier—a user-focused (black box) SLI could be a request for a new account (the ‘input’) that results in sending an email with a valid activation link within 1 minute (the ‘output’, incl. success criteria). This single high-level SLI aggregates several lower-level metrics; it measures many things by measuring only a few.

    Let’s switch to the engineering mindset mentioned in the introduction and assume the processing queue is stuck. The high-level black box SLI does not capture queue-specific metrics, suggesting a more granular SLI specific to queue size may be needed. However, white box metrics like this will affect the error budget burn of the aggregate SLO associated with the high-level black box SLI. Monitoring and observability tools will allow engineers to diagnose and troubleshoot particular issues, such as a stuck queue, while understanding the impact on system reliability (via the higher level, black box SLO’s error budget and burn rate). The solution to the stuck processing queue used in this example is not an SLI dedicated to the queue, but reliability-focused work to diagnose and correct the root cause of the queue getting stuck.

    Summary

    This article introduces an SLI thought model that uses a common paradigm from quality engineering. This thought model offers a different way to think about SLIs. It supports implementing the fundamental objective of SLIs and the associated reliability stack they inform: Ensure a positive and reliable user experience by measuring reliability and providing quantitative support for decisions on prioritizing development efforts. Only a happy user is a continuous user.

  • How Good Developers Become Good Engineering Managers

    How Good Developers Become Good Engineering Managers

    Being a good engineering manager means that, to some degree, you have a natural affinity for the work involved. And if you’re a good developer, you likely already have some of those innate skills and attributes.

    But simply having the skills doesn’t mean you’ll enjoy being an engineering manager. For example, the nature of the role is that you’re no longer the one writing the code. I’ve known great architects who became great managers, but they ended up going back to being individual contributors (ICs) because they missed working with nitty-gritty coding details and the joy that brought them.

    It helps to take a realistic inventory of your skillset and preferred working style before making the transition. There are certain competencies you should have completely dialed in, and other tips to know that will help your transition.

    Basic Skills of a Good Engineering Manager

    If you’re a good engineer, you likely already have the following foundational skills that also make a good engineering manager. With these in your toolbox, you’ll have a better chance at success.

    1. Attention to detail. The buck stops with the engineering manager. You need to know when to push and ask questions and how to set outcomes for your team’s accountability. If you work loosely as an IC and need reminders about your own work, you probably won’t do well as a manager.
    2. Amazing personal planning skills. As an engineering manager, you’ll be planning an entire team’s work. If you’re terrible at planning your own work, maybe the engineering manager role isn’t for you.
    3. Speak intelligently about broad technical topics. If you’re a good developer, this shouldn’t be a concern. You’ll need this knowledge to advocate for your team and justify their work to upper management.
    4. Team player attitude. Even as an IC, you should have a collaborative, team-oriented mindset. It goes without saying that you’ll need this when leading others.
    5. Interest in managing others. This might seem obvious, but if you don’t have a natural interest in managing others and in the people side of engineering, you’ll be unhappy as an engineering manager.

    Making the Transition From Developer to Engineering Manager

    If you can do all of the above and are ready to make the transition from developer to engineering manager, here’s what to be aware of as you navigate that process. These tips aren’t meant to deter or discourage you but rather provide a helpful dose of reality.

    Satisfaction in your work takes new forms. If you’re an excellent developer, you know the value you bring and likely get personal satisfaction out of the contributions you make. As a manager, it’s harder to see the direct value you provide. Shift your mindset and realize that now, you’re the one enabling developers to do that rewarding work, and that’s the value you provide. You’ll learn to take enjoyment and fulfillment from your team’s accomplishments.

    People take up a lot of your time. What most frequently catches new engineering managers off guard are the people-related aspects of the role. They’re not prepared for how much time managing people takes. I’ve seen it over and over again where a new manager thinks they can still set aside time to code, but you really have to prioritize learning the craft of management.

    Remember to communicate upward. In the absence of information, it’s human nature to assume the worst. When I was at BitBucket, I assumed that because I had always done a good job, the leaders would think my team was crushing it. The opposite was true; leadership thought everything was on fire.

    I started sending updates every two weeks to various levels of management, and that’s when I learned the power of upward communication. I realized I needed to sell my team’s work, contributions and challenges. Learn what higher-level leaders are looking for and then give them that information regularly.

    Context-switching is part of the job. As an IC, constant context switches kill your day. But being an engineering manager means dealing with endless context-switching. Going to multiple meetings is baked into the job, and you’ll still have to find ways to block off time to get other work done.

    Shield your team. You also need to be prepared to help protect your team from context-switching. You might learn information that’s important to your team during your many meetings, but not all of that needs to be communicated with them. Filter out the junk so it doesn’t rain down on them and they can maintain their flow.

    Being an engineering manager can be a rewarding role that has a considerable impact on an organization. The more prepared you are for the realities of the role and its differences from being a developer, the more you’ll enjoy it and the more successful you’ll be.

  • Loosely Couple Your Schema and Keep Data Separate

    Loosely Couple Your Schema and Keep Data Separate

    In software development, one of the primary goals is to write clean, maintainable and extensible code. To achieve this, developers often strive to design systems that are loosely coupled and separate concerns into smaller, more manageable components. Loose coupling refers to the practice of designing components that are not tightly dependent on one another, allowing them to be modified or replaced without affecting the rest of the system. Separating data into distinct layers, such as a database layer, further contributes to this modularity by allowing developers to focus on writing business logic without worrying about data management. Together, loose coupling and data separation provide a solid foundation for building scalable and adaptable systems that enable developers to concentrate on writing high-quality business logic. This approach also lends itself to platform engineering best practices, a popular methodology that is being implemented or considered by developer teams today.

    Loose coupling is an important concept in microservices architecture because it helps to achieve the goal of creating modular, scalable and easily maintainable software systems. In a microservices architecture, individual services are designed to be independent, self-contained and loosely coupled, meaning they can be changed, updated or replaced without affecting other services or the system as a whole.

    There are several benefits to loose coupling in microservices:

    • Scalability: Microservices architecture allows scaling of individual services independently based on their needs, and with loose coupling, the interdependencies between services are minimized, allowing for better scalability.

    • Maintainability: With loose coupling, it’s easier to make changes to individual services without disrupting the system as a whole, reducing the risk of introducing bugs and making maintenance tasks easier to perform.

    • Flexibility: With a loosely coupled architecture, services can be developed, tested and deployed independently, enabling teams to work in parallel and make changes to the system without impacting each other.

    Platform engineering plays a crucial role in ensuring that microservices are loosely coupled. A well-designed platform should provide a standardized set of APIs, protocols and interfaces for service communication, as well as tools and frameworks for building, testing, deploying and managing services. This enables service teams to focus on developing their services rather than worrying about the underlying infrastructure and communication protocols.

    Moreover, platform engineering can also ensure that there is a standardized and centralized approach to security, monitoring and logging, which are critical aspects of microservices architecture. By providing a consistent and standardized approach to these areas, platform engineering can help ensure that individual services remain loosely coupled and can evolve independently, without negatively affecting the system as a whole.

    It’s All About Keeping the Schema Separate

    In platform engineering, the concept of keeping the schema separate typically refers to separating the schema of a database or data store from the application code that interacts with it. This separation allows for greater flexibility and scalability in the platform, as changes to the schema can be made independently of the application logic. By decoupling the schema from the application code, developers can avoid introducing breaking changes to the platform, and can more easily evolve and scale the platform over time.

    Separating the schema also allows for different parts of the platform to interact with the same data store using their own specialized schemas, without interfering with each other. This is particularly important in large and complex systems, where different teams may be working on different parts of the platform concurrently. By keeping the schema separate, teams can work independently and avoid stepping on each other’s toes, while still interacting with the same underlying data store.

    In summary, it’s all about the schema – don’t share databases and tables. The separation of data is fundamentally driven by that desire for loose coupling. Microservices should be as independent as possible from each other from design to development to production. Why? Efficiency. Disparate teams can change data structures and will not affect others versus the more traditional approach where we do share and it’s more cumbersome and involves more coordination between teams.

    Keeping schema separate in platform engineering is important for several reasons:

    ● Scalability: Each microservice owns its private schema, which enables the platform as a whole to scale more effectively. As the platform grows and evolves, it may become necessary to change the data schema without necessarily changing the underlying code. Separating the schema makes it easier to make these changes without disrupting the rest of the platform.

    ● Maintenance: Separating schema from code makes maintenance easier. It allows developers to make changes to the schema without needing to change the code, which reduces the risk of introducing bugs or errors.

    ● Collaboration: Separating schema from code makes it easier for multiple developers to work on the platform at the same time. When developers work on the code and schema separately, they can work independently without interfering with each other’s work.

    ● Flexibility: Separating schema from code provides greater flexibility for integrating with other systems or applications. It enables the platform to adapt to changing requirements or to interface with different data sources more easily.

    Separating Schema From Code is an Essential Practice

    By hiding data from service components from other service components, It provides guardrails and “just works.” While it does provide a form of coupling, it’s through messaging (typically asynchronous). Platform engineering encourages good design patterns from the start versus having to be more aware of these things. It’s a lot easier for especially newer developers that are not familiar with some of these architectural patterns and can get themselves into trouble.

    Overall, separating schema from code is an essential practice in platform engineering that provides scalability, maintenance, collaboration and flexibility benefits. It allows developers to build more robust and flexible platforms that can adapt to changing business requirements and technical environments.

  • Linux 6.3: What’s New ¦ AWS Layoffs are a Worry

    Linux 6.3: What’s New ¦ AWS Layoffs are a Worry

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: A new Linux kernel drops and layoffs at Amazon Web Services point to trouble. (more…)