devopsy.kinsta.cloud

Tag: devops

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Dice.com

    ERP International
    Fort Meade, Maryland
    Senior DevOps Infrastructure Engineer
    $150,000 to $180,000

    LinkedIn

    Lark Health
    Remote
    Lead DevOps Engineer
    $127,700 to $183,900

    Simply Hired

    Shake Shack
    Remote
    Senior DevOps Engineer
    $123,000 to $155,000

    Indeed.com

    Peraton
    Arlington, Virginia
    DevSecOps Data Automation Engineer
    $112,000 to $179,000

    CareerBuilder.com

    Booz Allen Hamilton
    Fort Meade, Maryland
    DevOps Engineer and Release Manager
    $73,100 to $166,000

  • The Hidden Configuration Tax Affecting Uptime SLO

    The Hidden Configuration Tax Affecting Uptime SLO

    This article is a preview of a talk by Greg Arnette for SLOconf 2023 on May 15 – 18. To watch this talk and many more like it, register for free at sloconf.com.

    Tax season was just a few weeks ago, and that got me thinking about how frustrating it is to get hit with a surprise tax bill and how that can relate to uptime: Configuration sprawl. DevOps practitioners constantly strive to maintain uptime service level objectives (SLO) while working with complex and ever-evolving infrastructure and applications. Configuration sprawl is a significant challenge many teams face, negatively impacting system performance and reliability.

    My company’s recent survey of 900 DevOps and platform engineering leaders revealed a startling (but not too surprising) conclusion: Most teams struggle managing secrets and configs at scale for infrastructure and applications.

    Where Does Configuration Sprawl Come From?

    As organizations scale their operations, the number of configuration files and settings grows, making it increasingly difficult to manage and maintain consistency across the board. They may have multiple environments, such as development, staging and production environments, often requiring different configurations, leading to duplicated settings and increased complexity.

    Configuration sprawl may also lack standardization as teams adopt different naming conventions, file formats and storage locations without enforced configuration standards, resulting in fragmentation and confusion. Relying on manual processes for configuration management increases the likelihood of errors and inconsistencies.

    Grinding to a Halt

    So you’ve got a lot of configuration–why should you care? Sure, it means more files to manage, but does it really impact your environment? There are a few places where configuration sprawl will negatively impact you. You might see an increase in error rates because as complexity grows, the likelihood of errors and misconfigurations increases, leading to system instability and downtime.

    Increased configuration complexity also means slower troubleshooting. Identifying the root cause of issues becomes more challenging and time-consuming when configurations are disorganized and fragmented. Ultimately this means less agility as you are forced to slow the deployment of new features and updates, impeding the organization’s ability to respond to market demands.

    How to Overcome Configuration Sprawl

    Don’t worry–you can tackle configuration sprawl, keep your SLOs on track and reduce the efforts required to manage and maintain configurations. To mitigate the harmful effects of configuration sprawl, consider adopting these best practices:

    Don’t repeat yourself (DRY)
    DRY config means variable names are declared once. Then flexible variable values are injected across build, deploy and runtime phases as defaults, inheritances and overrides. DRY avoids errors by allowing single changes to propagate where they are needed.

    DRY solves the common problem of forgetting to update critical secrets or parameters and failing to synchronize related changes (e.g., client and server port numbers or rotating a database password.)

    Decouple and abstract
    Decoupling and abstracting config means externalizing config from source code, separating how code and configuration interact, and isolating the specific values (log level) from the configuration interface (how the log level value is fetched.)

    This approach allows the same code to run in different environments using parameterized config. Abstracting config supports a best practice of implementing independent “code and config change” life cycles. Teams that frequently need to change config independently from code should decouple and abstract.

    Centralized config
    Consolidate configuration complexity into a single location because it’s easier to manage complexity within a single location versus multiple locations. Syncing config to the edge (where it is consumed) mitigates downtime. For example, it’s easier to fetch the configuration for service X versus coding all the logic needed to generate the configuration for service X.

    Standardize your configuration management by enforcing consistent naming conventions, file formats and storage locations across teams to reduce confusion and streamline configuration management. Centralizing the configuration and generating that configuration with transformations allows all the places that consume the configuration to be simple. Simple equals easy to understand and troubleshoot.

    Version control & monitoring
    Use a version control system like Git to track config changes to configuration files, making identifying and resolving issues easier. And continue to monitor changes and identify potential problems before they impact system performance.

    Train your team
    Ensure all team members know your best practices and patterns for configuration management. The proliferation of microservices and teams operating in parallel, establishing their unique naming schemes and patterns, exacerbates this problem for DevOps teams.

    Cutting the Uptime Tax

    By understanding the causes of configuration sprawl and implementing best practices to manage and reduce its impact, teams can improve their uptime SLOs and maintain a more reliable, high-performing infrastructure. Remember, investing in the proper tools and processes for configuration management is not only a best practice but also a critical factor in ensuring the long-term success of your DevOps initiatives. Those are tax cuts I can get behind.

  • Nixpkgs: How to Build Better, More Open Software

    Nixpkgs: How to Build Better, More Open Software

    The world we know today depends in fundamental ways on open source software. As providers of this software, package managers serve their respective ecosystems from their centralized, opaque hubs in ways that are sometimes difficult to reconcile with the open source nature of the software they’re serving. I want to talk about the extent to which that’s true and discuss Nixpkgs—the largest open source package repo in existence today—in this context.

    People sometimes focus on Nix the language or Nix the package manager, but it’s the Nixpkgs package collection (pronounced ‘nix packages’) that is core to the success of Nix. But it’s more than that: We can also understand Nixpkgs as a powerful tool to ensure the continued growth, transparency and resilience of open source software, first and foremost because it democratizes the process of curating this software and its build instructions. It is the living compendium of the efforts of thousands of contributors who both stand on the shoulders of giants (the software they’re packaging) and provide the ladder for others to climb further.

    A Purely Functional Grimoire

    The Nix package manager is different from other package managers in several ways, but one key aspect is that Nixpkgs contains only recipes to build packages, not the packages themselves. This is a fundamental idea to Nix: If we build a package reproducibly, we can separate the specification of a build from its built output. When we build a “recipe” for a package, we determine the output hash of that build–without actually building it. This means you don’t need a cache of build artifacts: Recipes are enough to rebuild your software world.

    Nixpkgs is just that: The big fat recipe book of open source. Anybody can view it, fork it, patch it and use Nix to inspect the largest set of free software recipes in the world. Anybody can create a cache that uses those build recipes (or your own!) to publish packages: The default cache is the official one run by the NixOS Foundation at cache.nixos.org.

    Derivation and Substitution

    So what happens, in practice, when I download a package from the Nix cache? When you query for a package, the CLI creates the specification (we call it a derivation) of the package to be built. This derivation includes a hash: The summary identifier of all source code, patches, build flags and hashes of the dependencies themselves.

    With that output hash in hand, Nix the package manager just asks whether that path exists in its cache. If it does, Nix fetches the contents of the package you asked for; if it doesn’t, Nix will build the package locally. We call this move substitution: Creating a derivation from a Nixpkgs expression gives you all the information you need about the output of the build, so the cache can substitute a binary with that output hash if we have it. Of course, if you end up having to build the package locally, it will result in the expected output hash anyway.

    Democratizing Package Curation

    The term ‘lingua franca’ refers to the dialect spoken by sailors across the Mediterranean in previous centuries, and it was the shared language with which one could communicate and do business from Morocco to Greece. Today we package software for independent islands, each with its own rules, languages and cultures and, until Nix, we had not found a way to “speak” across the many ports-of-call of our software archipelago. What this allowed for is packages being available on more platforms, which drew more developers in, which resulted in Nixpkgs becoming more complete, and so on in a positive feedback loop.

    Nix is a cross-platform, cross-architecture package manager. If we had to design the next-generation package manager, this would probably be at the top of the requirements list, yet few package managers enable this today. Each of the teams at RedHat or Debian maintains their own packaging format and package cache and has a fixed team of vetted contributors to keep packages secure and up-to-date. This results in few who effectively contribute to packaging open source projects and do so for one of many platforms.

    An open book of package recipes written in this lingua franca results in another surprising effect: Nixpkgs maintains more packages than anybody with an order of magnitude fewer maintainers [1]. It makes sense: Since it’s a common language, people can review the packaging code for packages they don’t necessarily know intimately, which in turn leads to contributors and reviewers cycling through updates faster and providing more up-to-date packages, which reinforces the previous feedback loop from earlier.

    Even if the rest of the Nix ecosystem stalled out, we’d still have this incredible resource in Nixpkgs to curate and share open source software; flox sources its packages exclusively from Nixpkgs for this (and many other) reasons. As of February 2023, there were around 6,000 contributors on the Nixpkgs repository and growing.

    (via (repology.org)[https://repology.org/repositories/graphs])

    Stay Unstable, my Friends

    Nixpkgs embodies a new model for package management and one that opens it up to more people to contribute. Providing cross-platform and cross-language packages is obviously attractive, but it’s the fact that it provides a lingua franca for software packaging that makes it a pillar of open source moving forward. Because Nixpkgs unifies a broader swath of software and makes it easy for anybody to contribute new packages and fix existing ones, it continues to grow and serve the open source community: A positive feedback loop propelling the existence of a more free and open world of software.

    References:

    Nixos.org

    https://reproducible-builds.org/

    https://wiki.debian.org/SecureApt

    https://rpm-packaging-guide.github.io/

  • Why We Need to Rethink Data Pipelines

    Why We Need to Rethink Data Pipelines

    Data is the fuel that powers modern business. But as demand for data surges, so does the pressure on data leaders and practitioners to deliver it. Businesses need resilient data pipelines that deliver critical insight for real-time decision-making to users on demand. However, against the backdrop of today’s chaotic modern data ecosystems, this is much easier said than done.

    DevOps teams are struggling with a complex combination of legacy and diverse technology systems across multiple environments. The resulting siloed systems and an ever-shifting data supply chain cause significant data integration friction. This friction keeps businesses from creating seamless and resilient pipelines to drive digital transformation and other business outcomes. Businesses need a better way to take the pressure off data leaders and DataOps practitioners and enable line-of-business teams to do more of the “last mile” data collection and analysis themselves. To achieve this, they need a unified, end-to-end platform designed to build resilient pipelines.

    Data Demand Sparks Friction

    The acceleration of digital transformation has created a demand for data that far exceeds supply. This has become especially true as competitive advantage gets harder to carve out and worrying macroeconomic headwinds gather. In today’s climate, all areas of the business are demanding data, but it’s hard to meet every request all the time.

    Research that polled data leaders and practitioners shows nearly half (48%) of admin and operations and customer service departments request data at least weekly, followed closely by accounting and finance (44%), IT and digital (43%) and sales and marketing teams (40%). And the strain is showing. Over half (59%) of data leaders said changing priorities have created significant data supply chain challenges.

    The problem is one not just of capacity but complexity. Building pipelines from source to destination requires rules to integrate, transform and process data. But when data is siloed across cloud, legacy and mainframe systems and stored in inconsistent formats, creating bespoke data pipelines to fulfill departmental requests is a huge challenge.

    Legacy systems also bring another challenge. Over half of data leaders say it’s so difficult to unlock data from legacy systems like mainframes that they simply don’t bother. That will ultimately undermine the value of the cloud analytics tools that many business leaders see as a single source of truth for decision-making–especially as legacy systems often hold decades of vital business insights.

    This patchwork approach creates extra toil for DevOps, DataOps and technical teams and means that over two-thirds (68%) of data leaders said they are being prevented from delivering data at the speed the business needs.

    The Scourge of Broken Pipelines

    For many organizations, building pipelines is also a labor-intensive job requiring a high degree of manual effort to produce hand-coded, one-off solutions. The resulting pipelines are brittle and vulnerable to being disrupted whenever there is a shift in the environment–such as adding new data sources. In fact, two-fifths (39%) of data leaders admit their pipelines crack at the first sign of trouble, and 87% have experienced a break at least once a year. More than one in 10 say it happens at least once a day.

    The most significant cause of breakages is due to bugs and errors being introduced during a change (44%) while infrastructure changes such as moving to a new cloud (33%) and credentials altering or expiring (31%) are also creating disruption. These broken pipelines immediately impact the corporate bottom line. They also force technical teams into a vicious cycle of firefighting. Data engineers spend, on average, almost a third (31%) of their time troubleshooting and recoding broken pipelines–time that could be better spent on value-adding tasks.

    But the negative impacts aren’t only felt operationally. The breakage of any pipeline can also lead to bad decision-making. For example, a supply chain director working with old data may over- or under-order goods or a financial trader making stock picks on out-of-date intel may lose money for a client.

    Unleash the Power of Data

    For organizations looking to overcome these challenges, the first goal is to reduce the workload on DevOps, DataOps and technical data teams by empowering business units and end users to do more. Research shows that 70% of technical data leaders are responsible for the last mile of data collection and analysis while 86% of them would prefer lines of business teams, such as marketing or finance, to be empowered to do this independently. However, this can only become a reality if organizations invest in the right data integration platform. It must be able to build, run, monitor and manage smart data pipelines at scale from a single console–across all cloud and on-premises environments. And it must be able to deliver resilient pipelines built to withstand continuous change.

    Only by eliminating data integration friction and enabling self-service analytics for lines of business can organizations accelerate digital transformation. Doing so will unleash the power of data across the enterprise and reduce the burden of overworked data leaders and practitioners.

  • Black Box SLIs

    Black Box SLIs

    This article is a preview of a talk by Stephan Lips for SLOconf 2023, on May 15 – 18. To watch this talk and many more like it, register for free at sloconf.com.

    SLOs are fast becoming the industry standard to measure reliability and help teams decide when to prioritize it. The first step in adopting a service level objective (SLO) culture is to identify the metrics that matter without drowning in noise and alert fatigue. This article explores how to apply the black box concept to aggregate granular metrics into service level indicators (SLIs) that focus on the user experience as an indicator of system reliability.

    To SLI or Not to SLI

    In general terms, SLOs define targets for the proper level of reliability of a given product, such as a service or a website. SLOs are applied to or informed by SLIs. An SLI is a measurement determined over a metric, or a piece of data, representing some property of a service. And this is where we, as engineers, can get lost in the details, since the perpetual proximity to the systems we build and support often leads us to think of system reliability in technical terms or metrics (e.g., response time, error rate, throughput). While these are certainly valuable metrics, the user experience may be compromised even if the error rate is zero and the duration is well within SLOs. Consider, for example, the response data. Even if well-formed, it may not be current, or flat-out wrong. An error-free and quick response is of no value to a user that expects current and correct data. Error rate and response time remain valuable metrics and SLIs, but focusing exclusively on them would leave higher-level issues undetected.

    We could add freshness and correctness SLIs, but by doing so, we increase the number of signals we monitor. And with each signal—or SLI and associated SLO and error budget—we increase alert frequency and make reliability reports unnecessarily complex. In other words, adding SLIs may address a particular aspect of system reliability, but it also introduces additional complexities.

    Tales of Black and White Boxes

    So, let’s take a step back and borrow a concept from a related discipline: Quality engineering—in particular, software testing. Tests commonly fall into one of two categories: Black box tests or white box tests.

    In systems theory, the black box is an abstraction representing a class of concrete open systems that can be viewed solely in terms of its stimuli inputs and output reactions, without any knowledge of its internal workings. A given input is expected to result in a particular output, without any consideration for the processing steps. Common examples include end-to-end tests.

    White box tests, on the contrary, are designed with knowledge of, and to test, internal structures and workings of an application. Common examples include unit and integration tests.

    User Journey as Black Box

    Now that we understand the concept of black box versus white box tests, let’s apply it to our SLIs. As mentioned above, a good SLI considers the entire user journey. Conceptually, a user journey aligns with the black box paradigm: For a given input, a particular output is expected. For example, requests to our API (the “input”) result in responses that provide fresh data to clients within a given time frame (the “output,” including success criteria). There are several aspects worth mentioning with this SLI:

    ● The SLI is applied at a system level
    ● The SLI aggregates lower-level metrics implicitly and explicitly
    ● The SLI is binary; it is either true or false.

    These aspects combine to inform an SLI that represents the user experience (system level), via measuring many indicators by measuring only a few and supporting pass/fail attribution to an SLO target (by being binary). In other words, the user journey is measured as a black box SLI.

    White Box to Black Box: An Example

    Let’s consider a concrete example. A user requests a new account for a website. After the request is processed successfully, the user receives a confirmation email with an activation link. The user follows the link to activate the new account and log in. This workflow is visualized in the following sequence diagram.

    Of particular interest are the account creation and user notification via email steps. Both steps occur asynchronously. In particular, the event processing engine where the request for account creation is queued offers several opportunities for insightful SLIs: Queue length, average processing time, etc. Those SLIs, however, fall into the white box category: They contribute to the user experience, yet are opaque to the user (black box). The user journey begins with the initial request for account creation (input) and ends with the email containing the activation link (output). Rephrasing the example from earlier—a user-focused (black box) SLI could be a request for a new account (the ‘input’) that results in sending an email with a valid activation link within 1 minute (the ‘output’, incl. success criteria). This single high-level SLI aggregates several lower-level metrics; it measures many things by measuring only a few.

    Let’s switch to the engineering mindset mentioned in the introduction and assume the processing queue is stuck. The high-level black box SLI does not capture queue-specific metrics, suggesting a more granular SLI specific to queue size may be needed. However, white box metrics like this will affect the error budget burn of the aggregate SLO associated with the high-level black box SLI. Monitoring and observability tools will allow engineers to diagnose and troubleshoot particular issues, such as a stuck queue, while understanding the impact on system reliability (via the higher level, black box SLO’s error budget and burn rate). The solution to the stuck processing queue used in this example is not an SLI dedicated to the queue, but reliability-focused work to diagnose and correct the root cause of the queue getting stuck.

    Summary

    This article introduces an SLI thought model that uses a common paradigm from quality engineering. This thought model offers a different way to think about SLIs. It supports implementing the fundamental objective of SLIs and the associated reliability stack they inform: Ensure a positive and reliable user experience by measuring reliability and providing quantitative support for decisions on prioritizing development efforts. Only a happy user is a continuous user.

  • Open Source Communities Need More Than Funding From Big Tech

    Open Source Communities Need More Than Funding From Big Tech

    In recent years, major players like Microsoft, Amazon and Google have increased their involvement in open source projects, providing both financial and technical support. More consistent contributions from big tech—whether in the form of people or resources—is exactly what the open source community needs. However, underdeveloped collaboration guidelines can lead to friction between large corporations and open source developers, which ultimately harms both parties. Friction jeopardizes tech companies’ ability to rely on open source for operations while leaving the open source community without the support it needs. To make their relationship more sustainable, both parties need to cultivate a greater level of trust, develop clear guidelines for collaboration, and prioritize community.

    Big Tech Contributions: Moving From Cash to Collaboration

    For years, major tech companies have used open source solutions in their tech stacks. But many open source developers—who receive little to no money for their work—have voiced concerns about companies that take more from the open source community than they give.

    In many ways, big tech has answered the call. For example, Amazon recently passed IBM as the fifth-largest contributor to open source projects. Companies like IBM and Amazon have a vested interest in supporting these projects—if open source projects are understaffed and underfunded, the quality of the solutions suffers.

    But many tech companies are grappling with an unstable financial environment, which makes hefty donations more difficult to justify. As a result, support from big tech often involves donations of their developers’ time. However, this is also challenging as tech companies like Twitter and Google continue to lay off the developers who were keeping many open source projects afloat.

    Yet, even before recent layoffs, collaborations between big tech and open source teams were strained. There’s a natural power imbalance between a corporation like Amazon and the open source community, which includes members who contribute to these projects in their free time, sometimes for little or no pay. Big tech developers must juggle the interests of the open source team they’re collaborating with and their own employer—with little guidance on how to do so.

    A lack of transparency and communication from big tech also puts open source developers in a tricky situation. Many developers need financial and personnel support but may not want to relinquish control of their solutions to an outside corporation.

    To alleviate this tension, big tech needs to establish clear, mutually beneficial guidelines for open source collaboration that prioritizes sustainability.

    Reconciling the Fraught History Between Big Tech and Open Source

    Past tensions between big tech and the open source community can affect collaboration if left unaddressed. For example, Microsoft has become one of the biggest contributors to the Linux kernel, but many open source teams still remember former Microsoft CEO Steve Ballmer calling Linux a “cancer” in 2001.

    To overcome this tension, big tech companies need to clarify their intentions from the beginning of a project. They should disclose their reasons for selecting the project, the developers who will work on that project and how their contributions will look. Open source maintainers also have a role to play here. They must properly vet these companies and inquire about their prior open source experience and strategies before welcoming them into the fold.

    One way to reduce the risk of power imbalances or other tension is by collaborating through a project like the Linux kernel. Google, IBM and many other corporations contribute toward Linux kernel, and this distribution of contributors creates a much more balanced playing field. Rather than one company assuming operational control, multiple companies play a small part in keeping the Linux kernel up and running.

    The Benefits of an Open Source Program Office

    While these guardrails certainly help, big tech companies also need to formalize their open source contribution processes. At Aiven, we’ve done this through our open source program office (OSPO), which helps us manage our relationships with the open source communities we rely on. OSPOs ensure organizations remain compliant with open source licensing and provide a foundation to expand their involvement with the community over time.

    When developing an OSPO, tech companies should be intentional about how they measure success. The most obvious metrics relate to the work developers do—for example, the number of features delivered, the number of fixes and the number of projects they contributed to. But OSPOs should also measure how developers contribute to the community itself, not just the projects within it. For example, do developers review work from other developers in the community? Do they attend conferences, write blog posts and share knowledge with other developers?

    Big tech leaders must place as much emphasis on community-building as they do the number of contributions developers make. If they don’t evaluate developers based on their engagement with the community, the developer is less likely to focus their efforts there, injecting toxicity into the communities they’re trying to support.

    Open Source Thrives on Collaboration. Let’s Keep it That Way

    The challenge of reducing the stress and burnout of open source developers can’t be solved solely through financial means. No amount of money will reduce the amount of work it takes to maintain an open source project.

    To make collaborations between open source teams and big tech sustainable, both sides need to prioritize supporting the community above all else and develop guidelines that support this mission. More sustainable collaborations will ensure that the open source community gets the support it needs while big tech continues to reap the rewards of the community’s innovations.

  • How Good Developers Become Good Engineering Managers

    How Good Developers Become Good Engineering Managers

    Being a good engineering manager means that, to some degree, you have a natural affinity for the work involved. And if you’re a good developer, you likely already have some of those innate skills and attributes.

    But simply having the skills doesn’t mean you’ll enjoy being an engineering manager. For example, the nature of the role is that you’re no longer the one writing the code. I’ve known great architects who became great managers, but they ended up going back to being individual contributors (ICs) because they missed working with nitty-gritty coding details and the joy that brought them.

    It helps to take a realistic inventory of your skillset and preferred working style before making the transition. There are certain competencies you should have completely dialed in, and other tips to know that will help your transition.

    Basic Skills of a Good Engineering Manager

    If you’re a good engineer, you likely already have the following foundational skills that also make a good engineering manager. With these in your toolbox, you’ll have a better chance at success.

    1. Attention to detail. The buck stops with the engineering manager. You need to know when to push and ask questions and how to set outcomes for your team’s accountability. If you work loosely as an IC and need reminders about your own work, you probably won’t do well as a manager.
    2. Amazing personal planning skills. As an engineering manager, you’ll be planning an entire team’s work. If you’re terrible at planning your own work, maybe the engineering manager role isn’t for you.
    3. Speak intelligently about broad technical topics. If you’re a good developer, this shouldn’t be a concern. You’ll need this knowledge to advocate for your team and justify their work to upper management.
    4. Team player attitude. Even as an IC, you should have a collaborative, team-oriented mindset. It goes without saying that you’ll need this when leading others.
    5. Interest in managing others. This might seem obvious, but if you don’t have a natural interest in managing others and in the people side of engineering, you’ll be unhappy as an engineering manager.

    Making the Transition From Developer to Engineering Manager

    If you can do all of the above and are ready to make the transition from developer to engineering manager, here’s what to be aware of as you navigate that process. These tips aren’t meant to deter or discourage you but rather provide a helpful dose of reality.

    Satisfaction in your work takes new forms. If you’re an excellent developer, you know the value you bring and likely get personal satisfaction out of the contributions you make. As a manager, it’s harder to see the direct value you provide. Shift your mindset and realize that now, you’re the one enabling developers to do that rewarding work, and that’s the value you provide. You’ll learn to take enjoyment and fulfillment from your team’s accomplishments.

    People take up a lot of your time. What most frequently catches new engineering managers off guard are the people-related aspects of the role. They’re not prepared for how much time managing people takes. I’ve seen it over and over again where a new manager thinks they can still set aside time to code, but you really have to prioritize learning the craft of management.

    Remember to communicate upward. In the absence of information, it’s human nature to assume the worst. When I was at BitBucket, I assumed that because I had always done a good job, the leaders would think my team was crushing it. The opposite was true; leadership thought everything was on fire.

    I started sending updates every two weeks to various levels of management, and that’s when I learned the power of upward communication. I realized I needed to sell my team’s work, contributions and challenges. Learn what higher-level leaders are looking for and then give them that information regularly.

    Context-switching is part of the job. As an IC, constant context switches kill your day. But being an engineering manager means dealing with endless context-switching. Going to multiple meetings is baked into the job, and you’ll still have to find ways to block off time to get other work done.

    Shield your team. You also need to be prepared to help protect your team from context-switching. You might learn information that’s important to your team during your many meetings, but not all of that needs to be communicated with them. Filter out the junk so it doesn’t rain down on them and they can maintain their flow.

    Being an engineering manager can be a rewarding role that has a considerable impact on an organization. The more prepared you are for the realities of the role and its differences from being a developer, the more you’ll enjoy it and the more successful you’ll be.

  • Platform Reality Has a Surprising Amount of Detail

    Platform Reality Has a Surprising Amount of Detail

    Developer platforms and the platform teams that create them have been the center of attention for quite some time now. There have been many (wildly different) opinions about the subject, ranging from how bad the notion of a platform team is to how essential platforms are to make an organization’s DevOps journey to cloud native successful.

    Since early this year, the CNCF TAG App Delivery team has been working on a whitepaper about platforms. It explains what platforms are, why they are needed and how platform teams might work to create them. Furthermore, there’s an abundance of end-user talks at conferences that show how platform teams and their platforms were essential in enabling developers at scale. Even on the KubeCon + CloudNativeCon keynote stage, we’ve seen success stories of platform teams like the one at Mercedes Benz.

    Developer Platforms Should Enable Developers

    There is no silver bullet to successful platform teams and platforms. However, it is important to understand that the main goal of a developer platform is to enable developers. It is essential for platform teams to focus on delivering that value, which is why we are seeing a trend to introduce actual product ownership and product management roles to platform teams.

    The CNCF whitepaper stated that the three core jobs of a platform team as follows:

    1. Research platform user requirements and plan a feature roadmap.
    2. Market, evangelize and advocate for the platform’s proposed values.
    3. Manage and develop interfaces for using and observing capabilities and services including portals, APIs, documentation and templates and CLI tools.

    I couldn’t agree more. But the reality of day-to-day business for many platform teams means they rarely get to focus on those three jobs.

    Reality Has Detail

    The situation reminded me of a blog post I read a few years ago, titled “Reality has a surprising amount of detail.” The author explained how it is easy for people to get stuck even when faced with a task that might look simple from the outside. They explain how a task as simple as building wooden stairs can, in its many subtasks, become quite complex. Furthermore, when faced with reality, even the tasks that seemed plannable get more complex and need real-life adjustments.

    Most engineers have had analog experiences in their day-to-day work. We can translate this notion into platform engineering quite well. Building a developer platform sounds straightforward at the onset. However, once we get into it, it becomes a rabbit hole of figuring out the CNCF landscape, adding company-specific glue, compliance, security, fixing bugs, getting deep into the open source projects and sometimes, faced with the reality of open source, even becoming contributors and maintainers of such projects.

    Now, I’m not saying that it’s all bad. All these things can be great and rewarding in their own way. Especially with a community like the CNCF, participating and contributing to open source can be a very rewarding experience. You’ll make lots of friends, learn DevOps and development skills and might even further your career significantly.

    While it’s a lot of work, there is a risk that every platform team basically reinvents the wheel. This is because an organization’s requirements seem too large to rely on a single product (or even a suite of products). We end up with bespoke platforms that need a lot of attention. These platforms also have platform teams that don’t have enough time to focus on the higher-level jobs that they were tasked with.

    Platform Evolution

    Over the years talking to and working with platform teams we have built a model of prototypical platform evolution and its risks at each stage.

    Usually, a platform team starts with the task of building a developer platform to enable the product teams to build great software and deliver value to the end users.

    As they build the platform from the first Kubernetes clusters to observability tooling to security, the backlog for features constantly grows. Even with a growing platform team and sub-teams for certain technology areas, there’s a risk that some of these backlog items never get attention. Especially, delayable, non-critical, daunting tasks (like upgrading to the next Kubernetes release) often fall victim to this problem.

    Next, when it comes to operations, the platform team needs to limit the possibilities and breadth of what they offer. Operations responsibility for functionality that is outside of those limits, (for example, a specialized data store) goes back to the product teams themselves, which results in developer distraction.

    Even with a wide coverage of the platform, Day 2 and operations, the next big task for the platform team is advocating for adoption. This is not just marketing for their platform, but also means the platform team has to build user-friendly interfaces and documentation. The risks and alternatives we often see here are that product teams build their own mini-platforms and systems that result in shadow IT. Another common risk is ending up with several competing platforms within big enterprises that are hard to integrate and centralize, which in turn leads to inefficiencies.

    Once we get over the adoption hurdles, it is the platform team’s job to deliver value and high-level capabilities to the product teams’ developers. However, at this stage, often the platform teams have grown and might encompass several sub-teams for technology areas. Additionally, some areas like security might already be under central IT control. The risk we see here is that the capabilities offered are reduced to technology silos. They might cover all areas like observability, connectivity, security and release engineering, but to achieve real maturity, a platform team needs to offer capabilities that span several of those areas.

    Many of the advanced capabilities that we see with more mature end users cover several or all technology areas. Think about a simple high-level capability, like progressive rollouts. It starts with release engineering capabilities to build and deploy the software. From there, we need connectivity capabilities to dynamically route traffic to the new service. Then we need observability capabilities to observe our new version and from there have automated feedback loops to roll back or go forward with the rollout to production. And if that wasn’t enough, as we’re a security-aware organization, we want the whole software supply chain that facilitates this to be secure and trusted.

    Question for the curious cloud native enthusiasts: How many tools do you need to integrate to achieve the above “simple” high-level capability in an automated way? And this is only one of many capabilities.

    PaaS-Like Platforms Without the Downsides

    So, while platforms and platform teams are definitely the way to go, we need to be aware of the amount of work and effort required.

    We need to be aware of the risks and strive towards building highly integrated platforms that deliver high-level capabilities to the product teams.

    However, cloud-native developer platforms, while having similar goals to the PaaS systems of the past, should not follow the same approach of full abstraction. If we just abstract away into bespoke systems that hide everything from the user, we go back to the world of PaaS where at some point the end-user developers hit limits and build workarounds around the platforms.

    Additionally, if we build these bespoke internal platforms, we’ll be stuck with them and need to maintain and evolve everything by ourselves.

    Or, to go back to the blog post I referenced earlier, “At every step and every level there’s an abundance of detail with material consequences.” And we have to remember what we care about. The way to avoid both of these downsides is to bet on open source and stand on the shoulder of giants.

    By building our platforms on open source and using as many of the established or soon-to–be-established standards of our community we can avoid our platforms becoming bespoke. Recent developments like Cluster API, Gateway API and sigstore are some good examples.

    Furthermore, we need to make our abstractions poke-able. While our platforms should come out of the box with user-friendly interfaces and sensible production defaults that enable new product teams to onboard quickly and get their jobs done without having to bother with deep understandings of the underlying technology, we need to keep them open to mature users that know their way around and want to tweak it to their needs. We achieve this by giving mature users the option to access said open standards and APIs safely.

    This is the strength that we gain from a community like the CNCF, so we should use it to build great platform products that stay true to the core of open source and strive to make our product teams as happy and successful as possible. And remember to be happy with ourselves as successful and appreciated platform teams.

  • You’re Already a Distributed Developer

    You’re Already a Distributed Developer

    For many coders, the idea of distributed systems development might seem like a dark art. They think of high-performance computing wizards crafting mysterious spells with job schedulers to work magic with hundreds of connected nodes in massively-parallel systems. Or large, distributed content delivery networks that balance resources to deliver content across the world no matter how high the demand. The folks doing those jobs have often worked in specialist distributed computing environments for years. But the complex problems they face at a massive scale aren’t so different from those that you deal with every day. In short, you’re a distributed developer too. It’s just that no one bothered to tell you.

    Everything is Distributed

    The world is a lot more distributed today than it used to be. Back in the day, code would talk directly to one mainframe or microcomputer, generally on the same LAN. Your PDP-11 and your batch order processing software had an intimate, exclusive kind of relationship.

    Computing environments began to change with the advent of open computing in the late eighties. It ushered in the era of multiple computers working together other over a network. The concept continued to evolve beyond PCs. Now phones, servers and smart kettles all expect an easy way to talk to each other.

    Today, distributed operation is so ubiquitous that people barely see it happening. When you type anything into a browser, it fires off dozens of queries to different backend systems owned by different organizations. These systems generally run on different infrastructure.

    Even a small application that makes a database query is a distributed system. The application often talks to a system—or even multiple systems—halfway around the world.

    Many apps these days are compositions of multiple API calls to different online services. When a developer gathers weather data from a service provider and feeds it into a cloud-based online geographical information system that decides how much to irrigate crops in Iowa tomorrow, what is that if not a distributed query?

    Software is becoming more distributed over time. As software became more complex, developers have long wanted to carve it up into smaller pieces. This began in earnest with the software component movement around CORBA in the mid-to-late nineties, gravitating to XML web services in the early noughts and then to RESTful APIs. All of these involved forms of remote procedure call to increasingly granular services.

    In the last decade, this atomization of monolithic software has become the norm rather than the exception. Container-based architectures ushered in the use of microservices for better development agility and maintenance. Most developers that started their careers in the last five years or so will have been forced to think about how these services communicate from the outset. Distributed design patterns are a fact of everyday life.

    And it’s Still Difficult

    While your distributed applications might not be as complex as those facing HPC and hyperscale services, they still come with some challenges that increase as systems become more granular. What happens if a remote API takes longer than expected to deliver a result because it is overloaded, meaning that API queries to different systems arrive in a different sequence than expected? How should you react if a network or system outage or an expired access key prevents an API from responding at all?

    Problems like these arise unexpectedly, meaning that you have to get more creative with code from the outset. Good developers do more than master object-relational mapping and send queries. They program defensively to ensure that their programs retry steps when a failure somewhere else delays that job or stops it altogether.

    There are different design patterns to handle most common distributed programming scenarios today, even though you might not think of them as such. For example, developers have been building load-balancing code into their applications for decades, distributing requests between different servers to maintain performance.

    Anyone who has tried to maintain transaction integrity when writing to multiple databases will also have dabbled in two-phase commit design, which ensures that all of the databases get updated (or that none of them do).

    The more experienced you get, the more complex and esoteric these patterns become. At some point, many developers move beyond basic CRUD models in their applications to handle more complexity when reading and writing data. For example, they might read data from a website in one format but collapse multiple records into one when writing data to the website’s back-end store. That means writing code that creates different types of read and write queries. That pattern, called command and query responsibility segregation (CQRS), is also a form of distributed programming.

    Then, there’s the Saga pattern, which people use to maintain application state across long-running processes comprising multiple transactions. This pattern relies on an event bus along with messaging and queuing code to handle services with unpredictable delays in response times. It ensures that the overall process transaction finishes in an acceptable state or rolls back.

    As you venture into more sophisticated patterns that hold transactions together while maintaining performance, you’ll find your code becoming more complex and, therefore, application development becoming more difficult. Such is the life of a distributed developer, even if you didn’t realize that you were one until now.

    Gaining this knowledge and experience as a developer is important, but once learned the hard way, you don’t have to keep doing it manually. There are technology solutions that manage some of these complex distributed patterns for you. As the volume of plumbing code to support distributed design patterns increases, so does the need to offload it. In an industry increasingly built on abstraction, it’s time to turn distributed systems management into a separate service and unburden your code.

    And—if you’re looking for an example platform specifically designed to address this issue—check out Temporal, a completely open-source (MIT license) microservice orchestration platform which enables developers to build scalable applications without sacrificing productivity or reliability.

     

  • Preparing for Machine Learning Readiness to Unlock its Full Potential

    Preparing for Machine Learning Readiness to Unlock its Full Potential

    With the rapid advancement of artificial intelligence (AI) and machine learning (ML), companies need to understand and evaluate their readiness to adopt these technologies and drive material business outcomes.

    To assess a company’s machine learning readiness, we must consider several key factors, including:

    ● Vision/strategy
    ● Data availability
    ● Expertise, infrastructure and resources
    ● MLOps and governance

    In this article, I will review these three key factors in assessing ML readiness and how they can help your organization prepare.

    1. Vision/Strategy

    A company’s vision for its AI/ML strategy is essential to consider when assessing its machine learning readiness. This includes ensuring that the expected AI transformation aligns with the company’s strategic goals and that it will ultimately be meaningful for the business. To evaluate this, organizations should consider their overall strategy for AI adoption and determine how it fits into their broader business objectives. Additionally, companies should identify specific use cases and applications of AI that support their goals and how those can be implemented to drive business outcomes. A company that has already woven an AI/ML strategy into its business planning is in a far better position to assess its ML readiness.

    2. Data Availability

    I believe data availability is another crucial factor in assessing ML readiness. Machine learning algorithms require vast amounts of data for training and validation to offer real value. It would be challenging for a company to successfully implement machine learning without access to sufficient data, or they’d risk a model that cannot be trained effectively and that would, in turn, perform poorly. Companies must consider the quality and quantity of data at their disposal, as well as any potential future data sources when assessing their readiness for this important element of machine learning. I also think organizations should consider the feasibility of collecting and integrating data from various sources, whether internal or external, to have the most impactful machine learning initiative.

    3. Expertise, Infrastructure and Resources

    It’s no secret machine learning requires a diverse set of skills, including data science, programming and statistics, and every organization must be sure they have all the necessary talent and resources when building that function. These resources include data preprocessing, model development, model validation, deployment, monitoring, maintenance and the ability to train and onboard new employees to oversee AI/ML programs. Companies should also enable collaboration across these different units and roles. If a company chooses to outsource machine learning projects to third parties, the skills needed internally may need to shift toward project management and collaboration with external teams, as well as a nuanced and sophisticated understanding of the business problem, industry and technical capabilities of the vendors, to ensure proper execution.

    A company’s systems and technology stack are key to any operation. This includes selecting the right technologies and tools to enable the end-to-end creation and use of AI/ML-powered analytical applications. Before adopting ML, companies should evaluate their current stack to determine if it’s equipped to handle AI, including the availability of hardware, software and programming languages. Organizations should also consider the costs associated with selecting and implementing the right technology stack for AI/ML, including the costs of any necessary upgrades or new investments. Decision makers should also research and evaluate different technology vendors, their products and services, and find the one that best suits their needs in regards to cost, value and alignment with long-term goals.

    4. MLOps and Governance

    To be machine learning-ready, I find it important for companies to have machine learning ops (MLOps) in place. MLOps is a set of practices that allows companies to manage the end-to-end machine learning life cycle. It covers the entire machine learning process, from data preparation to model deployment and monitoring. Implementing MLOps includes continuous integration and delivery, model versioning and management and monitoring and alerting. Additionally, MLOps practices such as automated testing, model evaluation and deployment can help organizations improve the overall efficiency and effectiveness of their machine-learning efforts over time.

    A company’s governance structure for AI/ML adoption is another factor to consider when assessing machine learning readiness. Companies should evaluate their existing governance structures and determine if they are equipped to handle the implementation of AI, including data governance, compliance and risk management. We must also consider how we will manage the deployment and scaling of AI, including the development of policies and procedures to govern the use of AI and the identification of internal stakeholders and decision-makers who will be responsible for its implementation.

    Ultimately, I believe the key to machine learning readiness is to assess current capabilities, identify areas for improvement and develop strategies to address them. By taking the time to assess their machine learning readiness, companies can ensure they are in the best position to take full advantage of the benefits that machine learning can provide.