devopsy.kinsta.cloud

Category: Doin’ DevOps

  • Session Tokens Vs. JWTs: Choosing Your Session Management Solution

    Session Tokens Vs. JWTs: Choosing Your Session Management Solution

    In the world of authentication today, session tokens and JSON Web Tokens (JWTs) are the two most popular ways to manage user sessions and maintain a user’s authentication state between calls. Impassioned debates pit these solutions against each other, but each has pros and cons worth evaluating. Depending on the needs of your application, it’s even worth considering using them together to get the best of both worlds.

    The Main Differences Between Session Tokens and JWTs

    To understand the differences between session tokens and JWTs, it’s helpful to look at their setup and their impact.

    Setup

    The biggest difference in how session tokens and JWTs are set up is in where and how a user’s authentication information is stored.

    With session tokens, the user’s authentication state is stored in a server-side database as a record that includes a primary identifier for the session (typically a random string that is at least 128 bits long), an identifier for the user, the time the session started, the expiry of the session and, sometimes, additional contextual information like the IP address. Once stored in the database, the session identifier is sent back to the client to be stored as a cookie in the user’s browser.

    With JWTs, the user’s authentication data is stored as a JSON object, client-side, as soon as it’s issued by the server. The object contains a header, a payload (where sensitive user information is stored) and a signature that’s created by combining the header and payload, and then hashed with a secret key to protect the user’s information.

    Impact

    You can also look at session tokens and JWTs based on performance and control.

    Generally, JWTs win out based on performance: They enable faster authorization and more interoperability with external apps. But they demand more developer investment to address their security complexities and ensure that the right guardrails are in place to prevent vulnerabilities.

    Session tokens, on the other hand, enable more control but introduce some latency. While they provide stronger guarantees that each individual request is authorized and is simpler to implement securely, their bottleneck on the server-side database validation comes with a latency overhead that might ruin the user experience for highly responsive applications.

    While this framework can be helpful shorthand for summing up JWTs and session tokens, it falls into the trap of pitting security against performance, as if the two are mutually exclusive. But forward-thinking leaders in growth and security alike recognize that the best solutions leverage security to optimize user experience and accelerate product adoption and growth. One of the best ways to do that with session management is to combine JWTs and session tokens into a powerful hybrid.

    Combining JWTs and Session Tokens

    To date, there are a few different ways companies have combined session tokens and JWTs. One of the simplest ways is to return both a session_token and a JWT when a user starts a session. The session_token is a static value that is good for the lifetime of the session (stored server-side), while the JWT has its own, shorter-lived expiry.

    In this setup, expired JWTs can be passed to the session API in order to retrieve a fresh JWT, and the servers ensure that the underlying session is still active before passing back a new JWT. If the user logs out, this revocation of access will take place within whatever token age you set for your JWT. In other words, you only call the server when the JWT expires or before granting access to particularly sensitive actions.

    Configured this way, this approach to secession management greatly reduces the performance overhead while also protecting you and your end users from the risk of authorizing actions based on stale information. Instead of a tradeoff, JWTs and session tokens are leveraged together to optimize both security and performance.

    Picking the Solution That’s Right for You

    Despite new hybrid approaches like the one described above, there are still maximalists out there who will tell you that one approach is always superior. The truth, of course, is that every application is unique and the security and latency tradeoffs need to be evaluated in context.

    Whether you pursue session tokens, JWTs or a hybrid solution like the one described above, the choice of session management really boils down to how you answer four key questions:

    1. How sensitive is the information you’re storing?

    An extremely security-conscious organization, like a bank or government agency, might want to just use session cookies to ensure that every single call is authorized at that exact moment. Choose your solution based on the risk and cost of a data breach weighed against the cost greater latency might have on your customers.

    2. What are your ambitions for the scalability of your product?

    As mentioned earlier, scaling is much easier with JWTs because no call needs to be made to the server to re-authenticate the session. If handling high-volume traffic is a must for your product, you need to have a plan in place to address potential latency issues.

    3. Which modern features does your application rely on?

    For many modern features like serverless computing, cross-domain functionality, mobile-specific, or single-page applications, JWTs are either preferred or required. Understanding which of these features your product uses will help you understand the session management tools available to you.

    4. How important are performance and uptime to my end-users?

    How quickly do I want to get up and running? For companies whose value proposition relies on a fast, smooth user experience (say, live gaming), JWTs may offer the kind of speed and response time they need. For other kinds of products where latency has less of an impact on their user experience, or where they want a guarantee of security with less initial investment, session tokens may provide the stability and assurance they require.

    The Choice is Yours

    While some die-hard loyalists may always insist on choosing between session tokens and JWTs, modern session management solutions are much more nuanced, so companies can optimize the performance and security requirements for their unique products. With the ability to switch between JWTs and session tokens as needed, there’s more choice now than ever.

  • When Self-Service DevOps Forgets Who it Serves

    When Self-Service DevOps Forgets Who it Serves

    For years, self-service has been touted as the salvation of DevOps. Why? Well, every department in your organization has its own reasons for putting self-service DevOps at the top of its wish list:

    ● Ops teams need to ensure high availability, performance and security
    ● Dev teams need speed, self-sufficiency and transparency
    ● Security teams need guardrails, compliance and accountability
    ● And, of course, the C-suite needs ROI, strategy and insight on top of everything else.

    These goals often cause tension—especially between Dev and Ops. Verifying security takes time that developers usually don’t have. And once at scale, even the fastest manual processes break down. Like at the U.S. Department of Defense, for example, with a workforce of close to 200,000 internal and external developers.

    Self-service DevOps platforms promise virtual self-service kiosks for software delivery, uniting teams and creating a frictionless path to independent developer access and provisioning. According to the State of DevOps report from Puppet, self-service DevOps is a hallmark of “highly evolved” DevOps organizations.

    Many self-service solutions are available, but has anyone actually figured it out yet? After all, if everyone is happy with their self-service approach, why are they still searching for answers?

    Approaches to Self-Service DevOps

    True self-service DevOps promises a wide range of benefits: It lets Ops stop rushing to put out fires and shift to a big-picture strategy; it makes compliance simpler with automation, safety guardrails and clear accountability. And, there is the promise of tremendous capital efficiency.

    It’s no surprise that so many vendors are rushing into the space, offering so many flavors of approaches that Baskin-Robbins would be jealous:

    ● Internal developer portals
    ● Shared service platforms
    ● Workflow automation
    ● ChatOps/Chatbots
    ● Service catalogs
    ● API marketplaces
    ● Cloud competency centers

    But let’s face it: In reality, most of these approaches fail. And when self-service isn’t working as it should and companies fall behind on their service request queue, many DevOps teams default to risky practices that aim to expedite operations but actually expose the company to attack:

    ● Over-privileged and over-permissioned accounts that give broad access to anyone who gets their foot in the door, as happened with the SolarWinds breach of 2020
    ● Inactive accounts that leave a door open to cybercriminals, as we saw with the Colonial Pipeline breach of 2021.

    What you might not realize is that these two risks are so prevalent that IBM Security X-Force penetration testers managed to use them to hack into 99% of client cloud environments. So when we say self-service DevOps is failing, it’s failing big-time.

    So what’s the solution? Why is true self-service DevOps so hard to achieve?

    Why They Fail

    There are actually a number of reasons why today’s self-service DevOps platforms fail:

    ● They break end-user workflows, forcing users to change their behavior and learn new processes.
    ● They force devs to switch context which kills productivity.
    ● They need domain expertise to operate, reducing the target audience that can use them.
    ● They’re heavily reliant on human-in-the-loop involvement, often creating more work.
    ● They require significant effort to update and maintain workflows and meet or exceed SLAs.

    Most importantly, with current self-service DevOps solutions, users are often flying blind. They don’t know what they don’t know; for instance, how to locate, search, identify, create and use resources and workflows.

    Imagine, for example, choosing a self-service checkout to avoid a long line at the grocery store cashier. But then …

    a) You get stuck behind someone frantically trying to scan a wrinkled, expired coupon.
    b) You get stuck behind someone who doesn’t understand barcodes and scans all their items twice.
    c) You forget that alcohol requires age verification, causing an alarm bell and flashing lights to start going off above the register (spoken from experience; embarrassing).
    d) All of the above.

    In all these scenarios, a human attendant must come running to fix things, defeating the whole point of self-serve. Self-service is only self-service if it’s easy to use even when things go wrong.

    With most so-called self-service DevOps solutions, if something goes wrong, you’re back to waiting—drawing the operator back into the loop—just like with the old system.

    What True Self-Service Devops Looks Like

    To achieve true self-service DevOps, you need a few elements in place.

    Headless UI
    Most developers are (gasp!) human beings. So the best interface is no interface at all. Why not let developers express their needs freely in natural language using existing tools such as Slack, Microsoft Teams or a command line interface?
    Workflows
    Developers are frustrated when processes don’t make sense or take more steps than necessary. Let’s adopt workflows that actually handle all the nitty-gritty—so that all developers need to do is ask (in their own words) to be granted access to resources, information and workflows.
    Intelligence
    For developers, there’s nothing more frustrating than not knowing where to turn for the information you need. True self-service DevOps, powered by AI, will use your existing data to provide a single source of truth with all the answers and insight devs need.
    End-User Feedback Loop
    An ideal self-service DevOps platform will ask the developer for feedback that is then used, in turn, to improve the platform itself. First, because everybody enjoys having their voices heard and second because this reinforces the role of the platform, which is to serve DevOps by improving the end-user experience. Similar feedback loops have been used in apps like Waze, which uses social navigation to continuously improve the quality of its end-user experience. Self-service DevOps is next in line.

    Once you’ve accomplished all of this, you can work more efficiently, providing immediate on demand assistance so your Dev and Ops teams can work without interruption.

    How will you know when you’ve achieved true DevOps self-service? When you forget you ever did things any other way—and wouldn’t go back for a million dollars. When you wake up one day and realize that your platform not only lets you build trust and accountability but also gets all your teams working together…and—(gasp) dare I say it?—having fun.

    That will be the last DevOps platform you ever buy.

  • Fire at Data Center Causes Chaos | 20% Costlier Cloud

    Fire at Data Center Causes Chaos | 20% Costlier Cloud

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: A South Korean conflagration leads to a ridiculously long outage, and the price of public cloud is skyrocketing. (more…)

  • Transforming Observability

    Transforming Observability

    Walking into an unfamiliar operations center some time ago, I immediately noticed database error alerts racing down the primary monitor faster than a Matrix screensaver morphs letters. Strangely, no one seemed particularly excited about it. The situation was a head-turner for me, coming as I did from the “everything must balance” banking world.

    A senior ops tech quickly explained that airline reservation systems are highly optimized for speed and volume, sacrificing nearly everything else to meet peaks. “No worries; the agents just hit Enter again. Retry logic just slows us down,” they told me. The lesson here was removing constraints can profoundly change a system, much as Dr. Eliyahu Goldratt described in his game-changing Theory of Constraints.

    Today, cloud-native architecture, multi-cloud and hybrid cloud platforms, dynamic infrastructure-as-code (IaC), DevOps and our ability to store boundless amounts of data each change or remove constraints in their own way. Observability could be on this list, but I think observability’s transformation is still happening right in front of us.

    Observability and OpenTelemetry

    Through open source OpenTelemetry (OTel), highly competitive vendors collaborate to shift their offerings up the value chain by supplementing and replacing once highly-prized proprietary agents, interfaces and data specs. There’s no better example of removing a constraint than making these once proprietary agents with an open solution that will have profound effects across the industry.

    OTel is a thriving ecosystem of 11 tech companies participating on various boards and committees that also collaborates and integrates with open source projects such as Jaeger, Kubernetes, Prometheus and OpenMetrics. Over 20 observability companies natively support and provide OTel distributions. OTel continues to mature, reaching stability 1.0 release in 2021 and announcing its roadmap for metrics specifications in 2022.

    OTel-native support represents a significant commitment from vendors because it means rearchitecting products to distribute data via OpenTelemetry Protocol (OTLP) rather than using OTel SDKs and Collectors frontend internal interfaces. With SolarWinds’ October 2022 announcement, I suspect they will also join the ranks of companies supporting OTel natively in their commercial offerings. 

    Digital transformation, product and technology leaders see value in observability because of its potential to measure digital experiences and measure the performance of business and digital services. To do this requires observability to meet three significant challenges.

    First, observability must effectively cross the complex boundaries of microservices, containers, cloud and traditional applications, multiple cloud providers, database sources, SaaS services, infrastructure and internal and external APIs. Today’s challenge is far beyond the central aggregation of large volumes of log data and suppressing non-essential alerts. 

    Most enterprise architectures look eerily similar to a breadboard wiring project with applications, systems and data sources crisscrossing each other, representing the various pathways and interfaces across systems. Virtually any of these elements could contribute to the degradation of a digital experience, and observability must operate across these elements whether they live in our tightly controlled data centers or are distributed in microservices, cloud services or third-party interfaces.

    With this breadth and depth of visibility, we also need context to match and correlate what appears to be disconnected information and sources. Open source enthusiast Chris Engelbert describes this challenge well:

    “The data correlation, the knowledge of how the infrastructure and services are deployed, as well as the dependency tree of applications, services and (eventually) hardware, must be taken into account when providing hard evidence of what is going on in your system.”

    Following the Silver Thread

    With context and dependencies, observability can allow us to see what software developers call the “silver thread,” the ability to collect the components of systems involved in measuring an experience or triaging a performance issue. Then we can follow the pathway, or silver thread, across all the components, whatever or wherever they are, to find constraints or bottlenecks. For example, a particular API’s poor performance may be due to the location and traversal necessary to reach a needed data source rather than an issue in that API’s microservice or application.

    In summary, observability is invaluable to monitoring, operating and triaging modern applications and cloud infrastructure. The adoption and maturation of OpenTelemetry will deliver many benefits, including the removal (or, at least, greater transparency) of traditional boundary constraints. And with the context to understand and follow the silver thread, observability transforms beyond operations to measure digitally delivered business services and experiences.

  • More Choices, More Problems: The Double-Edged Sword of Multi-Cloud

    More Choices, More Problems: The Double-Edged Sword of Multi-Cloud

    Multi-cloud environments can be a double-edged sword. On the one hand, there are clear benefits to using whatever cloud or tool is best for your particular needs, regardless of vendor – in theory. In practice, however, the more choices you have, the more complex and expensive your environment becomes. In fact, Andreessen Horowitz postulates that multi-cloud architectures are oftentimes costing enterprises more than if they had stayed with on-premises systems.

    Enterprises need a multi-cloud environment to compete in such a rapidly expanding digital economy. For the majority of companies, completely repatriating to on-prem isn’t an option–the train has already left that station for most. But what do you do when the costs of staying competitive become more and more prohibitive, ironically making you less competitive?

    Hybrid Clouds Became Multi-Cloud

    When hybrid cloud–a mix of public cloud services and on-prem systems–became table stakes for enterprises, companies only needed one public cloud service provider. Most put all of their eggs in the AWS basket, splitting their workloads between AWS public cloud services and on-premises systems.

    This setup provided more agility and flexibility to enterprises and lowered costs by reducing dependence on expensive, cumbersome on-prem hardware. A lot of capex was replaced by opex; with cloud services, you only pay for what you use as you use it without the physical infrastructure and maintenance costs.

    Inevitably, the cost and efficiency benefits of the cloud drove companies to migrate more and larger workloads from their on-premises systems, increasing cloud costs over time. And the more workloads you had with a single vendor, the more beholden to that vendor you became. If AWS raised their prices, you had no alternative but to pay more.

    Soon, hybrid clouds grew into multi-clouds, with other public cloud providers such as Azure and GPC in the mix. Sometimes this was by choice–why be locked into one vendor and suite of tools when you could choose from multiple vendors that might offer better pricing or have some products that are better suited for your needs?

    More often, though, it was involuntary–developers spinning up their own clouds and resources for their own purposes without consulting IT, creating shadow IT. Or companies acquiring other organizations whose cloud choices varied. Either way, the enterprise is using multiple clouds whether they planned to or not.

    Regardless of how it happened, once more clouds were introduced, complexity multiplied.

    The Multi-Cloud Hydra

    Too many organizations underestimate the complexity they’re introducing with each additional cloud. Operating systems, networking, cost monitoring and tracking, workload methodologies and data aggregation are all different for each vendor. How many people in your organization know what all these differences are and how to navigate between them? How do you effectively manage, govern and monitor usage, costs and security and compliance issues when each public cloud does it differently? You have to manually collect data from each cloud and aggregate it in spreadsheets for analysis. How accurate can that be, how long does it take and at what cost in terms of people and time?

    Furthermore, how will you be able to accurately pinpoint waste and inefficiency? If you’re chasing down engineers to turn off resources or having to choose between one manual cost-efficiency action versus another, the waste will add up and your budget will be inaccurate.

    As if that’s not chaotic and complicated enough, development teams spread out across the enterprise adopt different tools and use them for their own purposes, without any alignment or coordination. The same tool is purchased multiple times. Best practices between teams aren’t shared. Everything needs to be custom-coded, creating layers of “spaghetti code” that blur visibility and are ripe for misconfigurations, which results in security vulnerabilities. And let’s not forget shadow IT, which IT departments don’t even know about, let alone monitor.

    You started your journey with a unified cloud strategy, but now you’re struggling with an ever-growing number of different clouds and strategies for different groups. Each group prefers different tools, clouds, methods and processes, and this has left you stuck with siloed teams, unending tool sprawl, infinite lines of custom code, worsening security risks and no clear visibility. And the meter keeps spinning, costing you millions.

    So, what do you do now?

    Taming the Chaos

    To survive and thrive in the new cloud order, enterprises can’t just write off exploding cloud costs as “the cost of innovation,” or otherwise justify their cloud chaos as par for the course. Too many are doing that now, and it won’t end well, as we’re likely headed into recession and growing uncertainty and turbulence in markets.

    First and foremost, taming cloud chaos and getting multi-cloud costs under control requires visibility and interoperability. You must be able to view, monitor, manage and analyze all clouds from one place; manually collecting that data with spreadsheets won’t do anymore.

    Development teams can no longer be siloed off from each other. There should be a single repository where all teams can access and utilize all tools, scripts, and previously coded integrations. When everything is viewable and available in a single place, no one has to buy the same tool twice or reinvent the wheel when using it. Once a tool has been integrated, any other team that wants to use that tool can simply re-use the integration, configuration, and policies that have already been created and vetted. If multiple groups are doing the same thing, an approved automation process can be created and shared.

    True cost optimization also requires organizational awareness and shared responsibility. If people don’t know they’re wasting resources, how will they be able to stop? Enterprises need a system that detects waste and cost overruns and notifies stakeholders accordingly.

    For example, a development environment may be up and running even though it’s no longer being used. An automatic alert could be sent to the appropriate team leader, reminding them to shut it down. A developer provisioning cloud resources might not choose the most cost-effective option, but the alert system would flag their activity and notify them of a cheaper alternative. Cloud resources that are left on overnight or on weekends would also be detected and shut down. Commitments would be conducted automatically every day and no longer require individuals to remember to do them manually.

    There’s just no going back to the traditional whack-a-mole style of cloud management and cost control. Enterprises that understand this and create a holistic framework for cloud management capabilities based on simplicity, visibility, unity and awareness will be in a much better position to navigate the choppy economic waters to come.

  • Of Max and Min: When Performance Engineering Plans Go Awry

    Of Max and Min: When Performance Engineering Plans Go Awry

    Modern software developers have access to powerful tools and services which allow them to quickly develop, demo and deploy fully functional applications. But what happens when the single-user prototype satisfies all required functionality but the user says the system seems slow? Or if the initial implementation turns out fine for one user, but multiple users start to experience delays? Or if users are happy, but the cost of auto-scaling grows prohibitive?

    The Problem with Performance Engineering

    Concern over non-functional aspects of computing systems—response times, resource utilization and costs—falls under the domain of computer systems performance engineering (CSPE). Unfortunately, the practice of CSPE has evolved into something more akin to art than engineering—with few standardized principles and practices. Because software is constantly evolving its languages and abstractions, CSPE practitioners appear obligated to evolve and change their abstractions and languages, too.1 As a result, part of our jobs as performance engineers is to understand what the heck other performance engineers really intend or mean.

    There are many reasons for this state of affairs, but ultimately this lack of a common language and engineering standards for CSPE is a deficiency in the education and training of software engineers. There is very little in the typical university software engineering curriculum that prepares software engineers for doing CSPE. This sometimes leads to practices that are not backed by scientific or engineering principles. As a result, practitioners are forced to “wing it”—creating their own art or hitching their wagons to someone else’s star and learning (or not) through trial and error.

    The PMWG – A POSIX for Performance?

    The Performance Management Working Group (PMWG) was started in the late 1980s during the “UNIX wars,” when computer vendors were feuding over claims to the UNIX leadership mantle and had formed two factions: UNIX International and the Open Software Foundation. The group was unique in that its members included computer systems vendors on both sides of the “war.” Its members put aside those differences to try to come up with performance management standards and practices that would put UNIX on par (at least) with the performance management tools found on mainframes.

    In retrospect, the PMWG was probably the best chance UNIX had for establishing, promoting and enshrining some basic performance engineering principles—a kind of POSIX for performance. Unfortunately, the group failed—in large part, in my opinion, because it put too much priority on the latter stages of the performance management data pipeline and did not spend enough effort toward establishing key primitives. Today, in the absence of established rules and practices for performance engineering, history is repeating itself. Current efforts in observability continue to focus mainly on data presentation and data transport and storage, without much focus on the right metrics and data quality based on basic performance engineering principles. Presentation, transport and storage are important, but the efforts around metrics, logging and tracing coverage and data quality are sporadic at best.

    Inertia, Expediency and Superstition

    Why are more formal, engineering-based practices around CSPE needed? Consider the use of CPU utilization as a key performance indicator (KPI) of system performance. I’m a member of Bloomberg’s Trading Solutions SRE team, which manages hundreds of machines on which the company’s Trading Solutions software runs. We often see or hear statements like “This host’s CPU is too busy,” or “That host is out of CPU resources.” I’ve found that these observations are not actually very helpful and sometimes divert our attention from the real problem(s). When misused as a system KPI, focusing solely on CPU utilization can lead us down the wrong path and draw incorrect conclusions. To understand why, let’s first look at how we assess the performance of real-world, non-computing systems.

    Take fast-food restaurants. One of the most important attributes of fast-food restaurants (besides their health inspection rating) is that they are fast. But we’ve all been to fast food restaurants that are anything but fast. Without actually standing in line and waiting, we can often tell that we will have a slow experience at a fast food joint if there is a long line. The line length gives us a lot of information. Our assessment of the situation is different whether there are 100 customers versus 10 customers versus a single customer waiting in line.

    In the real world, this line length (or queue length) assessment is almost universal. Supermarket checkouts, ATMs, airport security screening, toll booths, elevators and COVID-19 testing are all areas where we apply this semi-conscious assessment. In contrast, we almost never use “cashier utilization” or “self-serve soda fountain utilization” as a way of assessing how slow a fast-food restaurant is. We always say “the lines are long” or, if we actually choose to wait in line, we’ll make the more direct statement that “service is slow.”

    Which brings us back to our computing systems. Why do we reflexively look to CPU utilization when computing systems are slow while, in the physical world, we rely on queue lengths? The question is even more relevant when we consider that the concept of CPU is less well-defined today. Are we measuring the CPU, its cores or hardware threads?2 I believe the contributing factors to this misuse include inertia, expediency and superstition.

    In the early days of computing, CPUs were perhaps the most expensive component of the large monolithic computer systems of the time. Measuring how much these expensive components were being used was an important part of maximizing the financial investment in these large machines. Fortunately, measures of CPU utilization were relatively easy to estimate. Ease of implementation meant that CPU utilization measures were almost universally available.

    The universal availability of this simple metric, combined with some correlation (sometimes weak) between high CPU utilization and system slowness made it a default indicator of systems performance. When a belief becomes ingrained to the point of superstition, even weak correlations are enough to validate and perpetuate it. Better metrics—for example, run queue lengths at the CPU(s)3—can help us shed our superstitions to better understand our systems and lead us in the right direction in identifying and correcting problems.

    Dispelling Superstition and Other Irrational Practices

    Which brings us to the title and purpose of this series. As a title, “Of Max and Min” is meant to evoke John Steinbeck’s classic novella “Of Mice and Men,” which in turn got its title from the famous line in Robert Burns’ poem “To a Mouse”: The best-laid plans of mice and men oft go awry. The two main characters in “Of Mice and Men” fail in their efforts to better their lives during the Great Depression because they are tragically ill-equipped to do so. Similarly, we should not be surprised if plans go awry when software engineers are tasked with doing performance engineering without the proper training in performance engineering principles.

    “Max and Min” also highlights the importance of data analysis and statistics in understanding systems behavior. Effective CSPE requires us to be familiar with concepts in statistics, numerical analysis, and even operations research—in addition to the more standard computer science areas of algorithmic complexity and computer architecture.

    With “Of Max and Min,” in addition to calling out superstitions, I will be pointing out some common mistakes we make and blind spots that we have.4 Through case studies, I’ll illustrate how our blind spots can lead to mistakes that can persist for years. I will also try to cover some basic CSPE principles that I believe are skipped in most software engineering training. Like everyone else, I’ve been winging it for 40+ years. And I’m still learning. If you disagree with any of my points or have further insights into the topics that I am presenting, please let me know.

    1But, as in the application of the principles behind algorithmic complexity, there are some performance engineering principles that can and should be adhered to regardless of the system and language du jour.

    2There are in fact some bizarre implementations of “CPU utilization”. Here’s an example of how AIX “broke” the meaning of CPU utilization on their multiprocessor systems running in hyper-threading mode.

    3UNIX and Linux do in fact have metrics that could be used as CPU queue length indicators.  In a future installment, I will go into the quirks of some of the CPU queue length implementations and further argue for more consistent mathematical bases for queue length metric implementations.

    4For example, have you ever wondered why, after so many outages caused by logging, malloc, DNS and other low-level services, we still don’t have good, out-of-the-box visibility into these and other low-level components? We are accepting key blind spots as a fact of life when we really shouldn’t.

    About Me

    Like many software engineers, I started off studying a different field (Physics). Unlike most software engineers, I’ve always wanted to be a performance engineer. When I started taking computer classes as an undergraduate in Columbia College’s Physics program, I did not wish for a career writing programs in Fortran (yes, it was that long ago). But I did envision myself making systems run better and faster.

    To that end, when I added a Computer Science major to my undergraduate studies, I also added a minor in Industrial Engineering and Operations Research. For those unfamiliar with IEOR, it is a multidisciplinary engineering field that leverages mathematical and analytical methods to address complex system problems like resource allocation, supply chains, wait times, fairness, etc. Tools and methodologies from IEOR are invaluable to CSPE.

    At the start of my career, I joined the Bell Labs Computer Center as a computer systems performance engineer, where I made the somewhat contrarian decision (for 1980) to work in the group supporting their in-house, low-key UNIX systems (as opposed to their commercially popular MVS mainframes). I didn’t realize it at the time, but having access to the source code of the system you support/study can be a boon to understanding. At Bell Labs, I had the opportunity to contribute to the UNIX System V kernel, where I co-developed the first general-purpose kernel and user-land tracing system for UNIX and made significant performance enhancements to the virtual memory subsystem, and participated in the PMWG.

    Since then, I’ve spent the bulk of my professional career working on financial software systems—ranging from market data to messaging to transaction management. I’ve worked on monitoring systems on an enterprise level. I’ve made and witnessed many CSPE mistakes. In my current role as an SRE on the Trading Solutions team at Bloomberg, I’m seeing and helping remedy lots of operating system issues that stem from our runtime scale.

    Thanks

    Thanks to everyone who reviewed this document and provided constructive feedback. Special thanks to Nate McNamara and Peter Wainwright for suggesting substantive organizational and structural improvements. Peter also reined in my default stream of consciousness, run-on style. None of them had any hand in the writing of this paragraph.

     

     

  • Kill the Password: Google on Board | 4-Day Week Proves Worthy

    Kill the Password: Google on Board | 4-Day Week Proves Worthy

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Passkeys is getting another big-tech supporter, and the four-day workweek train picks up speed. (more…)

  • Distributed Systems: Turning Networking to Your Advantage

    Distributed Systems: Turning Networking to Your Advantage

    Whether it’s the office, the classroom, the store or the bank, a great many spheres of human life have moved online. Each day, we grow more and more used to consuming applications on demand that instantly connect us with the rest of the world. Our expanding appetite for the internet depends on an ongoing trend toward distributed systems, with most companies today spreading their information technology across hybrid and multi-cloud architectures that consist of small, independent and loosely coupled containers. Companies are scaling their applications by deploying more clusters of containers across an ever-broadening surface. There are four key lessons businesses can observe from the proliferation of distributed systems.

    Don’t Underestimate the Benefits of Distributed Systems

    There’s a reason why the trend toward distributed systems persists; they are chock full of benefits.

    Distributed systems are incredibly scalable, as they allow companies to scale applications by adding, replacing or removing nodes as necessary. New applications can be rolled out quickly and in multiple iterations, accelerating time-to-market. Codebases are smaller and easier to change. Distributed systems optimize developer efficiency and team collaboration because numerous teams can work on different parts of a system simultaneously.

    Distributed systems are highly resilient. When (and only when) built effectively, distributed systems will continue working even if a significant part shuts down, as traffic can be rerouted to functioning servers—just look at the resilience of the internet. Fewer single points of failure create more stability and fault tolerance.

    With so many benefits, it’s not difficult to see how distributed systems have enabled our connected, digital economy. Netflix is a prime example of how they have supported the growth of SaaS-based applications. Thanks to a complicated distributed system, anyone on Earth can access a massive library of movies, tv shows, and documentaries to stream whatever they want on demand, paying a monthly subscription.

    While Netflix is considered to be in the entertainment industry, we think of it as a technology company first and foremost because the success of its business model is based on the efficacy of its distributed system. Netflix’s platform is scalable, resilient, performant, efficient and therefore profitable because it is distributed.

    Know That Distributed Systems are Complex and Challenging

    Distributed systems are innately complex and therefore difficult to build and operate. They consist of many asynchronous components that must work in tandem with one another. As the system scales, it becomes even more challenging to understand, let alone operate.

    These challenges are hardly unique to Netflix. Any company that manages a distributed system at scale suffers because of the convoluted nature of distributed systems. Other internet giants like Spotify, YouTube and Twitter also grapple with compounding complexity. If you’re overwhelmed by the labyrinth that is your distributed system, take comfort in knowing that these challenges are (to an extent) unavoidable.

    Turn Networking to Your Advantage

    A distributed system is a set of computers, containers or any other nodes that connect to a network to communicate with other entities by passing messages. What truly allows distributed systems to be ‘distributed’ and to function is the network. This makes the network both a liability and an opportunity. Yes, even the best systems will fail at some point on various levels. There is no such thing as complete control in distributed systems, but that doesn’t mean you can’t turn networking to your advantage.

    Developers can efficiently manage their distributed system by prioritizing networking and employing the right tools across an architecture. Netflix has built exceptionally well-connected systems with servers deployed in thousands of locations worldwide that connect to the control plane in the cloud. If a server fails, traffic can easily be rerouted elsewhere. The distributed system is scalable, resilient, performant and efficient. Robust networking solutions give the team at Netflix visibility into and control over the application.

    Networking truly is the key to bolstering the advantages and overcoming the challenges of distributed systems. Invest resources in your network and it will take care of your distributed system.

    Aim for Simplicity

    Turn networking to your advantage by taking care when building a networking stack. The two most predictable pitfalls to avoid here are not choosing a stack that addresses your unique challenges and differentiators and selecting tools that are overly complicated for your use case.

    It’s all too common for companies building and managing distributed systems to adopt popular or default tools that come with their orchestration platform without giving their suitability much thought. It is never a good idea to copy-paste a networking stack seen elsewhere, assuming that what has worked for others will also work for you. It may, but do not blindly select the stacks everyone is talking about without giving them much thought.

    Many organizations followed the herd by adopting Istio, only to later discover that the service mesh is far too complicated and resource-intensive for their platform. While Istio’s numerous features are helpful for some, they quickly become burdensome. The trending tools are not necessarily the best for your use case.

    Some companies go the opposite route and build their own custom solutions instead of following the herd. Just as Netflix built Zookeeper for service discovery and Spotify built Backstage for API management cataloging, global companies with expansive distributed systems can benefit from the control offered by a custom tool. This option is expensive, time-consuming and not viable for non-tech or smaller companies.

    You’re (probably) not Netflix, so don’t try to be. You may think you have the same problems, but you don’t. Yes, distributed systems are always complex and full of challenges. But the complexity of distributed systems compounds at scale to such an extent that it becomes incomparable. Internet giants like Netflix and Spotify require the advanced features of complicated tools or custom solutions built in-house (and they have the means to invest in these resources). If your platform is a fraction of the size of Netflix’s, your distributed system relies on a far simpler network.

    It’s extremely easy to choose solutions that are overly engineered for your use case, but it’s best to opt for simpler alternatives. Start with simple solutions, gain control of your networking stack and then advance to another level of complexity later on. Gaining control of your network will allow you to tap into the benefits of distributed systems while avoiding some of the challenges. Turn networking to your advantage and make your infrastructure scalable by choosing simplicity over complexity.

  • Forget Automation: Why IT Should Take UX Cues From Fine Dining

    Forget Automation: Why IT Should Take UX Cues From Fine Dining

    As a college student, I worked at a five-star Italian restaurant where I learned the discipline and practice of customer service. The general manager knew exactly which corner to stand in to get a view of every single table during service. They would gauge diners’ experience according to how they were interacting, their microsignals, how much food was left on their plate and the way they glanced around the restaurant. What does this intense attention to detail and individual customer experience have to do with running a service-oriented technology business? Everything.

    The tech industry is notorious for automating processes at the expense of personalization, and that aversion can extend to building the relationships necessary to produce a successful end product. Understandably, some companies and their developers prefer to keep relationships with their clients strictly functional because these connections are messy, people are unpredictable and nothing ever goes completely according to plan. It’s a dance we have to do over and again with every new client and project. 

    For those of us who wish to function as service-oriented tech companies, keeping on top of client and customer experiences requires us to stay attentive and listen with active care. Just like the veteran restaurant manager, we have to constantly scan our environment for cues of satisfaction and dissatisfaction.

    Service is More Than an End Product

    Before opening every night, the general manager would make staff hold long pieces of string to measure the angles and distance between tables to ensure they were straight and the edges perfectly aligned. Then, after telling staff the evening specials, he would randomly quiz them about the menu. What color is a fava bean? 

    Seeing the value of such old-school standards left a deep impression and helped me embed attention to detail as a foundational practice across the board, from designing good UX to cultivating relationships. Creating a desirable environment is about successfully orchestrating a million tiny details and always looking for tweaks to make the customer experience better. 

    That seemingly omniscient Italian from my earliest experience in the restaurant industry also taught me that being empathic goes hand-in-hand with the hard-nosed realities of running a service-oriented business. They showed me how to take care of people, how to make them feel good and how to read body language. 

    It might be easy to stereotype tech companies as impersonal dispensers of products (and their developers as cogs in the machinery), but we don’t want to just take orders. Our job is to become strategic partners from the ground up for long-term success, and that requires constant care and continual adaptation. 

    Mapping the Client Experience

    We must be willing to redesign how we work with customers and end users over and over again—it’s always changing because expectations are always changing, and market norms are always changing. The moment that we stop trying to get better through small incremental improvements is the moment we stop being service-oriented.

    In the same way that a restaurant has a sequence of service that maps a diner’s journey from the moment they enter the door, we can architect a customer experience from every touchpoint. 

    In the sales process, it’s valuable to layer in different people to create more relationship touchpoints throughout an organization, so, in our case, a new client meets a member of our operations team and then somebody from our engineering team in order to ensure that they feel fully supported. Then, we set expectations in terms of communication as part of a scripted onboarding experience. In our review meetings, we seek feedback to determine whether we have under-delivered and course correct if necessary. 

    The other thing I learned from that Italian restaurant is that no mistake is irredeemable. It doesn’t really matter if you get it right or wrong the first time, it is how you respond when the unexpected happens that matters. Turning a situation around after making a horrible mistake—even sending out a cold steak—can breed even more loyal customers because it demonstrates that you really care.

    The Creative Heart of Service

    Despite the image of tech as cold and distant, it’s important to recognize that coding and development itself is creative and we should seek to nurture a healthy environment so that people can perform at their best. To that end, it makes sense to think of your responsibility as running the entire length of the chain, from employees to clients and, finally, to end users. Finding humanity in the work along each link is key to building successful partnerships. 

    In the context of building products, we must be creative to design seamless, curated experiences for end users. There are plenty of apps that sit on someone’s phone unused before ultimately being deleted. The only way to grab the attention of people that share the same interests and problems is by understanding their needs. 

    To that end, we must always be asking our clients: What is the easiest point of entry for the user? Creating a low barrier is a quick win that in and of itself is an onboarding experience and an opportunity to build trust. 

    Equally, what may make one customer happy may not actually have any resonance in the wider market. If customers were only hiring us to write code, we would only be interested in meeting their expectations. The goal is not just to build products, but successful products that establish or reinforce a client’s reputable presence in the marketplace. 

    Adding Value Through Relationships

    When we stand in our metaphorical corner of the room to observe the customer experience, we are consolidating data from products that have millions of users and harvesting insight. We are not interested in quick fixes. Customer service and a focus on the end-user experience means being oriented to what the market wants, not designing in a vacuum. We can then continue to build value over time.

    In the restaurant business, the little surprise at the end of dinner adds value because it is unexpected. A complimentary after-dinner liqueur with a pithy origin story is a point of detail at its finest. Creating a richer, fuller customer experience can set a tech business apart in an exploding marketplace. For my old mentors, paying attention to every nuance from the beginning to the end is what defines five-star service.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Indeed.com

    Harnham
    Remote
    Senior DevOps Engineer
    $160,000 to $180,000

    Monster.com

    Riley Consulting Services
    Folsom, CA.
    Senior Full-Stack .Net Developer
    $150,000 to $180,000

    LinkedIn

    Flowhub
    Remote
    Senior DevOps Engineer
    $140,000 to $160,000

    Dice.com

    Northrup Grumman
    Aurora, Col.
    Senior Principal DevOps Engineering Lead
    $120,000 to $180,000

    SimplyHired.com

    OBMedia
    Remote
    Senior DevOps Engineer
    $120,000 to $160,000