devopsy.kinsta.cloud

Tag: LinkedIn

  • How LinkedIn is Investing in Developer Productivity and Satisfaction

    How LinkedIn is Investing in Developer Productivity and Satisfaction

    More and more companies are beginning to examine the role developer experience has in improving engineering productivity and morale. And one company often referenced as a model for developer productivity engineering is LinkedIn—its engineering leadership has invested heavily into reducing time-consuming developer tasks and constructing an internal platform to gauge developer satisfaction on the tools they use.

    Engineers at LinkedIn recently shared how the company has invested in measuring developer productivity and happiness as well as their strategies for automatically cherry-picking commits. I also met with LinkedIn engineering leadership for a closer look at how the organization is working to improve the developer experience across its internal platform.

    Below, we’ll take an inside look at LinkedIn Engineering’s efforts to enhance developer happiness. We’ll explore their approach to improving their internal software development trends and consider the benefits LinkedIn has gained throughout the process.

    A Look at LinkedIn Internal Developer Solutions

    Productivity is challenging to measure, said Jared Green, VP of engineering, head of developer productivity and happiness at LinkedIn. So, the company opted to take a different approach by including happiness as a variable in the equation. Thus, they measure not only how fast a workflow is but continually monitor how enjoyable the process is to ensure developers have a delightful experience.

    “We don’t just strive for productivity, but for productivity and happiness,” said Green. And to his point, on-the-job satisfaction can have real-world results—on average, happy workers are 13% more productive, according to a University of Oxford study.

    LinkedIn now oversees a custom-built internal platform for evaluating the sentiment of developer tools. The platform, known as iHub, is a holistic listening channel to gauge developer satisfaction across various tools. It collects real-time qualitative feedback from users and correlates it with objective quantitative telemetry. It then surfaces this data in a UI dashboard for team leads and individual contributors to view.

    But there’s an art to engendering the trust required to collect accurate feedback, says Max Kanat-Alexander, principal staff software engineer at LinkedIn. For example, individual contributors need to be sure the data won’t be used for performance reviews. Once that trust is established, folks are more willing to give genuine feedback, he said.

    Benefits of Investing in Developer Enablement

    The benefit of centralizing developer experience metrics with a shared platform means that tech leads can make data-backed decisions on how to improve their DevOps processes. “It’s really about thinking about giving people insights versus giving them data,” said Green. The idea is that frontline engineers can rapidly investigate any metric to expose problems and see trends. “The optics have given leaders ways to improve various parts of developer experience,” he said.

    Through their investment into developer experience, LinkedIn Engineering has discovered and streamlined hangups in the Git flow, exposed inefficiencies in their code review processes, and fixed specific developer build times. LinkedIn Engineering reported it has reduced time spent on certain time-consuming developer tasks by 99.6%.

    To construct iHub, LinkedIn Engineering wrote its own front end. The hub pre-computes data dimensions and stores them in an offline data store, increasing speed, said Kanat-Alexander. The system uses a sophisticated survey technology to collect the data and hides the complexity of manual collection in a sleek dashboard.

    LinkedIn managers have already leveraged iHub data to identify macro trends and resolve specific points of friction. For example, Kanat-Alexander describes how one director noticed that developer build time was worsening in their organization. They were able to isolate it to a single developer who wasn’t using remote development, which was spiking the metric. Leaders then assisted in unblocking them.

    Takeaways For Your Organization

    Most companies could benefit from a UI like iHub, explained Kanat-Alexander. But, you really must have a large engineering organization to see the benefits of the work. Plus, up to 80% of the legwork is simply data integrations—this involves hooking into systems, cleaning up data and deciding how to measure certain metrics. “Whatever you can measure, you can improve,” said Green.

    LinkedIn Engineering has flirted with the idea of open sourcing iHub, but it’s a bit too custom to LinkedIn to be valuable to other enterprises. “There’s a lot of variance in build technologies nowadays, and large-scale engineering systems are bespoke,” said Green. Implementing such a dashboard also requires a deep understanding of developer personas and how people work, which can be entirely unique for each organization.

    Although other organizations can’t fork LinkedIn’s efforts, there are some takeaways that could help others invest in their internal DX. First, Green recommended starting with the fundamentals. “Having a core of understanding about what you’re trying to move the needle on is important, as is coming up with ways to talk about it consistently.” This forethought should help establish what you want to measure.

    Turning DX Metrics Into Action

    Studies indicate that more companies are seeking to implement platform engineering or similar developer productivity engineering initiatives. The end result of these efforts should be to empower people and ensure they are satisfied with their work and not just a cog in the system.

    On that note, it’s critical that there are mechanisms for action to happen on the data that is collected, said Kanat-Alexander. Someone needs to monitor the insights and make changes to improve the metrics over time. “Then you can win hearts and minds.”

  • The Value of Connections

    The Value of Connections

    value-of-connections

  • LinkedIn Job Scams: Out of Hand ¦ 4-Day Workweek: Let’s Get Serious

    LinkedIn Job Scams: Out of Hand ¦ 4-Day Workweek: Let’s Get Serious

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Don’t get scammed looking for a job, and momentum grows for the 32-hour working week. (more…)

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    LinkedIn

    CapitalOne

    Annapolis

    Director of Software Engineering – Risk Management and Compliance

    $190,000

    Dice.com

    Nigel Frank International

    Remote

    Senior Azure DevOps Consultant

    $165,000 per year.

    Indeed.com

    ROM Technologies

    Brookfield, Conn.

    DevOps Engineer

    $120,000 per year.

    SimplyHired

    Comcast

    Philadelphia, Penn.

    Senior Engineer Cloud Infrastructure

    $109,932 – $164,898.

    Monster.com

    Target

    Remote

    Lead Engineer – High Performance Distributed Computing

    $105,800 – $190,500

  • Dev Job Phisher Steals $540M | Patch OpenSSL NOW | Systemd Dev Joins Microsoft

    Dev Job Phisher Steals $540M | Patch OpenSSL NOW | Systemd Dev Joins Microsoft

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Spear-phishing causes $540 million loss, a high severity bug in OpenSSL might be “worse than Heartbleed,” and Lennart Poettering is now working for Microsoft. (more…)

  • Ego Panic

    Ego Panic

  • Analyzing SRE Job Postings

    Analyzing SRE Job Postings

    You can find plenty of high-level definitions out on the internet about what site reliability engineering means and what site reliability engineers do.

    But if you want to understand what it’s actually like to work as a site reliability engineer, there is perhaps no better source than job descriptions. SRE job ads explain what real companies are looking for and what they expect on a day-to-day basis.

    To provide insight on what SRE looks like in practice, we’ll look at some job ads from several leading companies. As we’ll see, each company has a slightly different take on what the site reliability engineer role entails, but it’s possible to identify some core themes that underline what businesses expect.

    To understand the content of SRE job descriptions, let’s examine examples from four leading companies.

    GitLab’s Job Ad

    GitLab’s SRE job description contains most of what you would expect from a typical site reliability engineer job ad. It highlights the importance of having a mix of technical skills and collaboration skills. It also emphasizes expertise in areas like Unix/Linux, infrastructure-as-code and Kubernetes.

    What may be slightly surprising, however, is the emphasis that GitLab’s job description places on programming skills. GitLab expects SREs to know how to code in specific languages, like Ruby and Go. The takeaway is that SREs need to know more than just how to use code to solve IT operations problems. They must also be able to develop applications or contribute code to application platforms (the job description mentions “contributing to code in GitLab” as a required SRE technical skill).

    Also interesting is the fact that GitLab expects site reliability engineers to be able to write blog posts. Blogging may not be one of the first skills that come to mind when you think of a site reliability engineer’s responsibilities, but at GitLab, at least, it’s part of the job.

    LinkedIn’s Job Ad Template

    LinkedIn provides an SRE job ad template that businesses can use when posting their own SRE jobs on the site.

    LinkedIn’s ad is pretty high-level (which is unsurprising, given that it’s a template rather than an ad for a particular job). For the most part, it doesn’t mention specific tools—although the main exception is programming languages, where it says SREs should know how to code in languages like C, Python, JavaScript and Ruby. Like GitLab, then, LinkedIn clearly sees development as a core SRE job responsibility.

    The LinkedIn job description also places a heavy emphasis on the ability to “gather and analyze metrics.” It doesn’t use the word “observability,” but that seems to be what LinkedIn is getting at.

    An SRE Job at Microsoft

    Continuing the theme of development-centric SRE work, a recent job ad for a specific SRE opening at Microsoft reads a lot like a DevOps engineer job description. The ability to “design, write and deliver software” is the top responsibility listed in the ad, which also mentions familiarity with CI/CD tools.

    The Microsoft job description also mentions the importance of written and verbal communication skills. Unlike GitLab’s ad, this one doesn’t say SREs will need to write blog posts; still, it’s clear that companies expect people in this role to be creative communicators in addition to technical experts.

    It’s worth noting, too, that Microsoft wants site reliability engineers who know both Windows and Unix. If you need further proof that Unix skills are absolutely central to SRE work, look no further than this job ad from the company that created Windows.

    Amazon’s Site Reliability Engineer Ad

    Even Amazon—a company that is more of a platform provider than a software vendor—places major emphasis on the ability of SREs to code in one of its recent SRE job ads.

    Interestingly, the Amazon job also highlights networking skills as a major qualification. That’s understandable given the complexity of networking in the AWS cloud. But it’s a bit surprising because most other site reliability engineer job ads don’t call out networking expertise as a key skill—despite the fact that network engineering is critical to SRE work.

    The Amazon job description mentions the importance of an applicant’s willingness to be on-call, too. We suspect that’s critical in virtually any SRE job, but at least Amazon is upfront about the fact that, at time, you’ll be woken up at 4 a.m. to manage incidents.

    Conclusion: Takeaways

    There are a few key takeaways from these various SRE job descriptions.

    • Development skills are absolutely critical in order to be an SRE. Scripting for administration or IT Ops purposes is not enough on its own.
    • The ability to communicate clearly—including both verbally and in writing—is more important to SRE work than you may realize.
    • Network engineering skills, too, are a crucial part of SRE jobs, at least at some companies.
    • In practice, SRE job descriptions sound more similar to DevOps job ads than you might expect.

    This is not to say that all SRE roles are identical or that all companies expect the same things from their SREs, of course. But the jobs ads or templates we’ve surveyed do highlight how the high-level understanding of SRE translates to on-the-ground reality at actual companies.

  • The Value of Attention

    The Value of Attention

  • Code Yellow: When Operations Isn’t Perfect

    Code Yellow: When Operations Isn’t Perfect

    Every now and then, engineering teams will get into trouble for any number of reasons. Sometimes, explosive growth catches the team by surprise. Or, on-call has gotten out of control, with alerts going off every five minutes. Or, development and operations teams simply have stopped seeing eye to eye. Regardless of why, the team is in a bad spot and something needs to be done to resolve it.

    If it’s a new problem, the fix might be easy: Add a few more servers, roll back to a known good application version or get everyone together over pizza and beers to clear the air. Often, however, the problem creeps up on you over time and suddenly the hole is so deep you can’t find the way out. At LinkedIn, a team that has gotten to this point will often declare a state known as “Code Yellow.”

    Yellow Alert

    Some people assume the name Code Yellow is based on traffic lights, but more accurately—and with a geekier twist—it’s actually from your favorite “Star Trek” series. More precisely, it’s how the crew of the Starship Enterprise indicates their current defensive condition. Either way, the definition is clear: Something is wrong, and we need to move forward with caution. True to both metaphors, we also have a Code Red. That’s better described as an immediate crisis, with everyone working 24 hours a day until it’s resolved. The Code Yellow is slightly more laid-back: This is everyone’s primary focus, but during business hours only. Code Yellows also tends to last on the order of months, while a Code Red should last on the order of days.

    Other companies may use a different term than Code Yellow, but the effect is the same: The team is communicating to the rest of the company that they’ve identified a serious problem that is a priority to be fixed to ensure the success of the team and, therefore, the company. The ability to do this is an important aspect of open and honest communication, a value that is critical to a healthy culture and can often get overlooked. Talking about our problems is just as important, if not more so, than celebrating our successes. Teams can learn more from fixing a problem than they can from a total success.

    This is Not a Failure

    The first step in starting a Code Yellow is to understand this is not a failure. There is no shame in admitting that the team has a problem that needs to be fixed. Bugs happen, despite our best efforts to avoid them. The only thing we can do is diagnose the problem and remediate it. The only time we fail is when we turn a blind eye to these problems. This applies to how our engineering teams interact with each other just as much as it does to the software and systems we produce and run.

    It is critical, however, that the right problems are addressed. Most of the time, we’ve gotten into the current situation because of a slow boil—increasing technical debt, many small issues or breakdowns in a process—that eventually built up into a crisis. The goal of the Code Yellow must be to not only remediate the current problems (a reactive component), but also make sure they are not repeated in the future (the proactive component).

    Planning for a Successful Code Yellow

    There are several components required for a successful Code Yellow, as well as necessary pieces to get buy-in from the rest of the organization:

    • Problem Statement: There must be a clear and agreed-upon statement of the problems facing the team that prompted the Code Yellow. This should include not only what the current problems are, but also what the current understanding of their root cause is.
    • Exit Criteria: Next, you need to have specific goals that the team will work toward to exit the Code Yellow. These should be traditional SMART goals: specific, measurable, achievable, relevant and time-bound. These goals are what make it possible for the team to enter a Code Yellow in the first place, as it covers a fixed scope and is not open-ended.
    • Communication: All information about the Code Yellow, including the announcement (which includes the problem statement and exit criteria), the successful conclusion and periodic status updates should be sent to the larger organization. This may be your department, or it may be the entire engineering organization. It may even be the entire company, depending on the nature of the problems.
    • Project Management: Like all large projects, there needs to be someone responsible for organizing the work and communicating information. As this represents an “all hands on deck” scenario for the impacted team, it is usually helpful to have a dedicated project manager (PM) to help with this. This is typically a PM who is knowledgeable about the team and the work, but not directly involved with the execution. This frees up the managers and individual contributors to focus on the work at hand.

    Once each of these aspects have been thought about, and the decision has been made to enter Code Yellow, the team’s first act is to reorganize their priorities around the exit criteria. This often means putting quarterly goals on the shelf. It may also be necessary to establish a dedicated meeting around discussing the status of the exit criteria.

    Space to Breathe

    It is all well and good for a team to enter Code Yellow and work with a single focus on the goals that have been set to make things right, but this is not enough for the team to succeed. For true success, everyone surrounding the team must understand the situation and give them the space to do their work. This is the place where a healthy engineering culture rises to meet the challenge.

    • Expect Delays: The most common way that a tangential team will be affected is through delays. They must expect that any requests that have been made of the impacted team may be delayed if they are not within the scope of the exit criteria. The Code Yellow involves, at its core, a reordering of priorities to address the stated problem. Outside teams need to factor this in and understand that their own project timelines may need to be adjusted.
    • Minimize New Requests: Other teams should also refrain from asking the impacted team for new things that are outside the scope of the defined exit criteria. Minimizing these requests, in addition to accepting delays on any existing requests, allows the impacted team to spend their limited engineering hours on getting to the other side of the Code Yellow.
    • Requests for Assistance from other teams: The team in Code Yellow may find they need outside help to reach their goals. For example, if there is sudden, explosive growth in traffic, they may need to accelerate the provisioning of new hardware. Finding yourself on the receiving end of a request such as this may require shifting your own priorities. Always remember that the team is all part of the same company, and as such, everyone succeeds or fails together.

    Engineering teams rarely stand alone, and it is important that everyone understand the value in having those teams working well, and working together well. A little temporary delay in goals to assure that this is the case is well worth it.

    Light at the End of the Tunnel

    Code Yellow represents a significant amount of high-priority work and working through it will often be stressful for the team. Saying “no” to coworkers that are making a reasonable request is hard, and the work that is in scope rarely involves spending time on interesting new features. In addition, if the problems being addressed include communication issues between groups, there are going to be some difficult conversations that need to happen. However, as the team approaches the end of the work, it will be much easier to see through to what lies on the other side of the exit criteria.

    The ultimate goal of the Code Yellow is to get the team out of a reactive mode where they are running from crisis to crisis and into a proactive state where they are able to work on the right big projects. Achieving the exit criteria will mean the engineers are more effective and able to work proactively. This is a stronger team—engineers are happier because they’re not under a heavy stress of operations work, the team is working well because they’re talking to each other effectively, and customers are pleased because requests are handled either through automation or in a reasonable amount of time.

    Does your company have an internal process that’s similar to how LinkedIn does a Code Yellow?

    — Todd Palino

  • LinkedIn Open-Sources Fossor Log Mining Tool

    LinkedIn Open-Sources Fossor Log Mining Tool

    Over the years there has been a steady flow of technologies developed by web-scale companies that have benefited the larger IT community by becoming open source projects. The latest two examples of this trend are Fossor, a tool for investigating broken hosts and services, and Ascii Etch, a Python library designed to complement Fossor that takes streams of numbers and turns them into visual graphs using ascii characters. Both were developed by LinkedIn.

    Steven Callister, senior site reliability engineer for LinkedIn, said Fossor, which is Latin for grave digger, was developed to provide a mechanism to mine logs to more easily discover unknown issues such a memory fragmentation. Typically there are 10 commands that DevOps teams employ to investigate whether certain issues might be occurring. Fossor eliminates the repetitive tasks associated with issuing the commands manually by enabling them to be automated then executed in parallel.  This approach  allows DevOps teams to continuously execute tests to discover any potential unknown issues that might arise at any time, he said.

    Callister noted LinkedIn decided to open-source Fossor to give back to the IT community in a way that is extensible. Fossor has two basic components: a core engine and a mechanism for creating plugins that DevOps teams can use to add tests specific to their site architecture. Linkedin is hoping that an entire library of plugins will emerge around a Fossor open source community.

    Of course, no organization is obligated to contribute the plugin they create to the open source community. There may be information in that plugin an organization considers too proprietary to share. But Callister said there are plenty of use cases where a plugin developed for one type of site might apply to another.

    One of the single biggest issues most IT organizations struggle with in managing IT at scale is a lack of engineers on staff. The average IT organization tends to rely on administrators that don’t have much in the way of coding skills. Organizations that manage IT environments that operate at web-scale typically have hired software engineers capable of developing their own tools. Open source projects such as Fossor provide a means through which software engineers working at web-scale companies can share their skills and expertise with the rest of the IT community.

    It’s not clear to what degree open source tools will replace commercial software tools. It’s not unusual for an IT vendor to leverage an open source project to create a commercial offering around which they provide extensions and professional support. But many DevOps teams are becoming more self-reliant out of necessity, either technical or budgetary. The challenge now becomes sorting through all the open source code available to determine which of the many tools available today solves a problem well enough to not only rely on every day, but also to make ongoing contributions.

    — Mike Vizard