devopsy.kinsta.cloud

Category: DevOps Culture

  • CircleCI Report Surfaces DevOps Productivity Gains

    An analysis of more than 14 million workflows from thousands of organizations conducted by CircleCI, a provider of a continuous integration/continuous delivery (CI/CD) platform, revealed the mean-time-to-recovery (MTTR) decreased by about 13% year-over-year.

    Half of CI workflows (50%) recovered from a failed run in 64 minutes or less. That compares to 73 minutes or more in previous years, the report noted.

    In general, the report found 50% of workflows completed in 3.3 minutes or less. However, the average duration was approximately 11 minutes. The median workflow ran 1.54 times per day.

    Success rates, as defined by the number of passing runs divided by the total number of runs, was, on average, 77% for default branches and 67%, on average, for non-default branches.

    Naturally, the level of productivity any given DevOps team can achieve is impacted by everything from the size of the team to the vertical industry they operate in. CircleCI CTO Rob Zuber also noted there are other crucial developer productivity metrics, such as the amount of time developers are focused on writing code, that need to be tracked as well.

    The goal for any DevOps team is to abstract away as much complexity as possible to enable developers to spend more time on writing business logic, said Zuber. Unfortunately, as more responsibility for infrastructure and security has shifted left toward developers, many of them are spending too much time managing application environments instead of writing code, noted Zuber. He noted that the cognitive load being placed on developers today is too high.

    Developing applications is, of course, as much an art as it is a science. Organizations can set aside time for sprints to write code, but it’s difficult to dictate when inspiration might strike a particular developer. There is always going to be a need to let developers, for example, write code at odd hours of the day and night. The more friction they encounter in terms of spinning up application environments, the less likely it becomes a developer will spend time squarely focused on writing code, said Zuber. The “zone” is when most of the best code is developed, he added.

    Naturally, there is no set of metrics that are equally applicable to all DevOps teams, but they should pay attention to metrics as an indicator of whether teams are improving, added Zuber. Most developers and software engineers are not going to want to work in a “software factory” where every minute of the day needs to be accounted for within a project management application. Rather than employing metrics as a stick, they should instead be used as an educational tool—a carrot—that drives steady improvement. In addition, organizations should take care to make sure the collecting of metrics doesn’t impede the pace at which applications are being developed.

    As the saying goes, things measured are usually things done; but there is a fine line to be observed when it comes to any goal that requires a certain level of creativity to achieve.

  • Declarative Compliance With Policy-as-Code and GitOps

    Declarative Compliance With Policy-as-Code and GitOps

    In recent times, the quest for greater agility, faster releases, enhanced scalability, security and performance brought forth the advent of several automation tools, technologies and frameworks. Software development has evolved considerably over the years to mitigate these challenges.

    Monoliths have been split into microservices for improved scalability, maintenance and faster releases. Since DevOps is adept at bridging the gap between development and operations teams, now these two teams can work together to improve productivity and speed up release cycles.

    The Need for Compliance

    One of the critical strategies for business success is to stay compliant since processes are always improving and evolving. Compliance ensures that your DevOps processes and practices are consistent and meet regulations or regulatory standards.

    An organization with a compliance-driven DevOps culture can cut down on operational costs, improve efficiency and reduce risks considerably.

    What is Declarative Compliance?

    Declarative compliance is defined as the process of managing and enforcing compliance in which policies are defined using code, and the system’s desired state is declared rather than explicitly programmed. To enforce compliance policies, you don’t write scripts or programs; instead, you use a high-level language or configuration file.

    Declarative Compliance provides several benefits:

    • Increased efficiency
    • Better managed compliance
    • Minimized risk of human errors
    • Enhanced security
    • Faster deployment
    • Better collaboration

    With declarative compliance tools, the system’s state is defined as a set of policies, and the tool makes sure it matches the desired state. The tool will automatically fix anything out of compliance, bringing the system back into compliance if there’s a deviation.

    Tools for Declarative Compliance

    Some of the popular declarative compliance tools are:

    • Open Policy Agent (OPA)
    • Terraform
    • Kubernetes Policy Controller
    • Chef Compliance
    • AWS Config

    Declarative Compliance for Infrastructure and Application Deployment

    Often, declarative compliance is used in infrastructure management and application deployment, where policies are written as code and enforced across multiple environments. In addition to managing compliance at scale, declarative compliance tools ensure that the system’s desired state stays consistent across multiple systems and applications over time.

    With declarative compliance, organizations can manage compliance more efficiently, effectively and securely since they can define policies as code and enforce them automatically. In addition to reducing errors, downtime and security breaches, this approach enables organizations to stay compliant and maintain their system consistently.

    Introduction to Policy-as-Code

    A policy-as-code approach allows you to write programs that govern security, compliance and rules throughout your application’s life cycle. It entails defining and applying code to manage and automate policies. It shares the same ideas as infrastructure-as-code (IaC) and other DevOps practices like continuous integration and continuous delivery (CI/CD).

    A policy enforces constraints and restrictions to prevent unauthorized access to resources such as databases, storage, services, etc. By codifying policies, you can enforce rules and constraints throughout the development life cycle to prevent non-conforming resources from being deployed by integrating these policies into the software development life cycle.

    compliance policy-as-code

    Figure 1: Policy-as-Code at work!

    Why Policy-as-Code?

    Typically, businesses have the required processes, tools and procedures to guarantee the deliverable’s quality. Performance, scalability, ease of use and accuracy are typical quantifiable quality metrics used to ascertain the quality.

    You can examine the quality of your application using these metrics, but a manual procedure will take time and is prone to mistakes. A better alternative is to leverage policy-as-code.

    The traditional policy enforcement processes have certain drawbacks. It is challenging to perceive that your policies would be entirely adhered to, i.e., they will not be broken. It is cumbersome to check against a list of policies manually.

    Moreover, the traditional policy enforcement method cannot scale in today’s agile world, where organizations have been growing continuously with more and more employees, services and teams.

    Here are the key benefits of policy-as-code at a glance:

    • Faster and more efficient deployments
    • Automated governance, approvals and policy enforcement
    • More reliable and secure applications
    • Centralized policy management
    • Leverages version control
    • Better visibility of policies
    • Enhanced collaboration

    How Does it Work?

    Policy-as-code involves three key elements:

    1. The policy itself, which contains the required code to model the decision-making process
    2. The data and information about the environment, service or application
    3. The query that is responsible for triggering the decision-making process

    Here’s how policy-as-code works:

    1. Policies are defined as code in a human-readable format
    2. These policies are then translated into machine-readable code using Rego or HCL
    3. The policies are integrated with the IaC pipeline
    4. A policy engine evaluates the infrastructure and the application resources against these policies
    5. The policy engine accepts the policies as input, processes them and then produces a query result
    6. The policy engine reports violations (if any) to the respective stakeholders, such as the development or security teams
    7. If violations are reported, the necessary remediation is performed automatically or manually

    Types of Policies

    • Security policies: Security is enforcing adequate technological controls to safeguard company assets.
    • Compliance policies: Compliance policies are designed to help organizations adhere to governance standards. Typical examples include PCI DSS, GDPR, HIPPA, etc.
    • Operational excellence: This encompasses policies pertaining to service degradation or interruption.

    What is GitOps? How can GitOps Help With Compliance?

    GitOps is a subset or extension of DevOps encompassing a collection of practices that can simplify deployments using Git. It is an operating model for cloud-native applications that leverages Git as a single source of truth for declarative applications and infrastructure. Using GitOps, you can implement a standardized policy-as-code approach across your CI/CD and GitOps pipelines.

    The key principles of GitOps are:

    • The state of the system is captured and stored in a Git repository
    • The entire system is configured declaratively
    • Automated deployment

    You can create policies in Git and store them in a particular repository, which is essential since Git offers sophisticated change management tools like version control. Only the compliance team would have admin access to the repository.

    It’s easy to enforce policies with Git, and you can apply them to any execution engine, like CI or CD. With GitOps, you can audit and document every action that affects data consumption.

    You can expose system changes, optimize deployment and use a version-controlled infrastructure. By managing compliance with GitOps, you can make sure that the changes are visible, verifiable and auditable.

    GitOps

    Figure 2: GitOps at work!

    Using GitOps as the Execution Engine in a GitOps Process

    There are several benefits to using GitOps as a process execution engine, including:

    • Increased automation: GitOps automates software delivery, reducing manual intervention and increasing deployment accuracy and speed.
    • Better transparency: GitOps records all changes to the system in a transparent and auditable way so that you can track compliance easily.
    • Maintaining consistency: GitOps keeps the system in the desired state defined in the Git repository, reducing non-compliance risk.
    • Fosters collaboration: All changes to the system are made through the Git repository, so GitOps encourages collaboration across teams.

    Using Policy-as-Code and GitOps: Policy-Based Compliance

    By using GitOps and policy-as-code, teams can manage compliance-as-code in an auditable, declarative way and enforce it via GitOps, ensuring that the desired state is always maintained. This can help your teams to codify compliance requirements as code, store them in version control and automatically enforce them through GitOps.

    Organizations can improve the consistency and reliability of their systems by defining and enforcing compliance requirements using policy-as-code and managing their deployment processes with GitOps. While this can minimize compliance violations, it can also maximize efficiency.

    By following policy-based compliance declaratively, businesses can define policies as code and enforce them automatically and repeatedly. By adopting such an approach to compliance management, you can better manage compliance in complex systems in a consistent, scalable, agile, transparent and collaborative manner.

    For example, you can leverage policy-as-code to define policies for infrastructure provisioning and configuration management to ensure that the infrastructure complies with organizational policies and industry standards. Typical examples are defining policies for network configuration, access control and resource allocation.

    Use Cases: Using Policy-as-Code and GitOps in a Production Environment

    Infrastructure Provisioning

    Open Policy Agent (OPA) can define policies for infrastructure resources like AWS Elastic Compute Cloud (EC2) servers. You can use GitOps to manage the deployment of infrastructure resources by storing IaC files in a Git directory. When the IaC files are changed, GitOps is adept at automatically deploying the changes while OPA checks for policy compliance.

    Deployment

    You can use policy-as-code tools such as OPA to create application deployment policies requiring specific security settings, while GitOps manages the application deployment by storing manifests.

    Manage Kubernetes Clusters

    You can leverage policy-as-code to better manage your Kubernetes clusters. So, you can write your code to define policies to manage Kubernetes resources, such as nodes, pods, clusters, etc., declaratively.

    Compliance Management

    You can use policy-as-code to establish and implement rules and regulations, such as those pertaining to security, to help businesses ensure that their systems comply with industry regulations and standards.

    OPA can be used to set compliance policies, like restricting access to sensitive data to authorized users only. You can use GitOps to manage the deployment of compliance policies seamlessly.

    Trusted Delivery: Policy-as-Code Integrated With GitOps Workflows

    Trusted application delivery is a software delivery approach emphasizing security and trust in the software supply chain. This ensures that any software delivered to the clients is devoid of security flaws, malicious code and other attacks.

    This approach prioritizes security and trust at the development, testing, packaging and deployment stages and integrates security and compliance checks into the GitOps pipelines. This quality check reassures clients about the product’s quality and ensures that any modifications adhere to predetermined rules and principles.

    Trusted application delivery allows for quick deployment while securing apps with automated guardrails. These guardrails are implemented using policy-as-code. By enforcing these guardrails, you encourage frequent deployments while ensuring that the reliability and security of the application are not compromised.

    Conclusion

    Declarative compliance with policy-as-code and GitOps is a best practice for achieving greater compliance, reliability and agility in software development and deployment processes. You can simplify defining and enforcing compliance policies using declarative configuration files and automation tools. Using Git as the single source of truth for all configuration changes, an organization can achieve greater efficiency, visibility and auditability in the compliance process.

  • Are you Being ‘Nice’ to Your Developers? Think Again

    Are you Being ‘Nice’ to Your Developers? Think Again

    As a developer, you’ve learned your craft. From your fingers comes the magic that builds the applications that run the world. You can literally create something from nothing! And not just software programs. You can create value that means revenue for your company. You’re a revenue machine — no, you’re a god, creating anything that’s needed! That is, you could be if you could spend all of your time writing that magical code. But this dream is not so simple. If you’re like most developers, you’re spending way too much time on other things, such as:
    ● Meetings
    ● Dealing with interruptions
    ● Security hassles
    ● Operations
    ● Debugging
    ● Fighting with infrastructure
    ● Burnout

    Every minute you’re dealing with any of those, you’re not making magic. Because you’re human, that is going to make you unhappy.

    If you really are a developer and not just imagining you are, you already know all of this. Now let’s turn our attention to those of you who are responsible for developers. Because it’s your job to make them happy.

    Yes, it is, if you want them to keep working productively for you and add value. Luckily, it’s within your reach to do so. It will benefit you, add value to the organization and, most importantly of all, the employees themselves.

    Simply put: If you want to get value from your developers, you have to be nice to them. If you expect them to pour themselves into you and into the company, you must first prioritize them and invest in them. Success can only be realized and achieved by a happy balance of developers who are motivated to create.

    Do you know who feels more motivated than anyone else? Who will always go above and beyond?

    Those who are appreciated.

    So let’s get started.

    Increased Productivity and Value

    The most obvious issue is productivity. The reality is that software development can be an arduous mental task, and when developers are unhappy, it will inevitably affect their productivity—especially when they spend mental cycles on things that are not directly related to the task at hand.

    Even if they’re not spending work time searching for another job (or quiet quitting), they’re probably thinking about what could be different and mentally grumbling about it every time something comes up that irritates them.

    By treating developers well and eliminating those annoyances as much as possible, you prevent that problem and get 100% of their attention.

    And there are other productivity benefits, as well. When developers trust their company and their co-workers, it fosters greater collaboration between those developers and business stakeholders. It also engenders greater creativity, which can lead to innovation and value above and beyond what you are expecting them to deliver.

    Retention

    The most obvious issue when it comes to developer happiness is, of course, retention. After all, if you’re not nice to your developers, you should expect them to leave for other jobs. If the Great Resignation taught us anything, it’s that it is human nature to move along if dissatisfied.

    That may not seem like a big deal to you, but think about the implications if your developers are leaving in droves. Never mind the turnover costs, what about the inevitable delays as you bring new developers up to speed? Hiring and training are arduous, time-consuming endeavors. And if turnover is bad enough, you may never be able to actually complete anything because nobody ever gets a chance to build up a good head of steam.

    Compare this to a company with a dedicated team of happy developers who are motivated to grow and innovate for themselves, their team and your organization.

    Which organization do you think is going to come out on top? Yes—the organization with the happier team of developers is probably going to have a competitive advantage—and a big one!

    When developers are happy, they are likely to be loyal to your company. That loyalty means they stay long enough to become experts in what you do, making them more likely to create the kinds of innovations that move the company forward for added value.

    But most of all, can you really afford to have your best employees leave and take all of the knowledge they’ve gained with them, potentially to a competitor? Probably not.

    Maintaining a Good Reputation

    All of this adds up to reputation; get a bad reputation in the developer world and it is going to be hard for you to attract top talent. In a world where technology changes so fast, it’s already difficult to find developers who have stayed with one technology long enough to become experts. You need to take advantage of every tool you can to make it easier. Your reputation can be one of them—if you treat your developers right.

    But there’s another reason to protect your reputation among developers: It could easily bleed over to your reputation among your customers. If you’re a big company, word of your treatment of developers may spread. If you’re not so big, and you have enough developer turnover, some of your former employees may work for prospective customers or know someone who does.

    Today, customers are trying to be socially responsible by buying from companies that ‘do the right thing.’

    Without much effort, you can be one of those companies by trying to foster developer happiness. Ultimately, it’s not that difficult.

    How to be Nice to Your Developers

    Fostering developer happiness doesn’t have to be a hugely expensive proposition. It’s not always about huge paychecks (though they do help smooth out some rough edges). Some things you can do to be nice to your developers include the following.

    1. Provide what they need: Developers should be able to spend as much of their time as possible actually coding; make sure they have the tools that they need so that they’re not having to improvise or struggle with creating infrastructure. For example, engaging a ZeroOps partner can take some of this off their plates.

    2. Eliminate distractions: Even in ideal situations, it can take as long as 20 minutes for a developer to get back into their train of thought every time they’re interrupted. Try to provide an environment with as few distractions as possible.

    3. Foster a training-centric culture: Developers need to feel as though you have their best interests at heart, and one way to do that is to help them learn new skills. And there are multiple benefits for you: For one, it’s much easier to train your existing people in an in-demand technology (such as Kubernetes) than it is to hire someone who already has those skills. For another, training your entire team in these new technologies can unlock creative ways of solving problems.

    4. Help developers understand how their work relates to the customer: While it can be tempting to expect developers to simply do what you ask them to, when they understand why they’re doing it, they’re much more likely to feel invested in the work.

    5. Listen to ideas: Your developers are at the heart of what you’re trying to do. It’s very possible that they have a new way of doing something or an idea for something that could help your company. Even if you don’t end up going forward with the idea, your developers need to know that you respect them enough to listen.

    6. Respect work-life balance: Developers have a reputation for working long hours and many even take pride in it. But it’s not always the most efficient way to get things done. A tired developer works more slowly and makes more mistakes. Be flexible and make sure that you’re not overloading your people with more than they can reasonably accomplish. If your developers have to work 70, 80 or 100 hours a week to meet deadlines, you either need to hire more people (if you can find them) or find a way to partner with other organizations to take things off their plate. (See number one).

    All of these things make for a positive work environment that enables your developers to work their magic for you, day in and day out, making something from nothing and moving you forward the way nobody else can.

  • How To Keep Engineers Happy and Reduce Burnout

    How To Keep Engineers Happy and Reduce Burnout

    Software development can be a mentally taxing profession. It requires intense mental focus, often under heavy workloads, with unclear job expectations and a lack of control. And with recent labor shortages due to layoffs and resignations, teams may also be understaffed, making matters more complicated. These issues can compound and lead to prolonged excessive stress—that’s when burnout hits.

    In fact, one report found that 82% of developers surveyed indicated that they had experienced burnout in the last six to eight months. Some top causes of burnout include increased workload, a poor work culture and personal reasons. But whatever the cause, for 73% of these developers, burnout is negatively impacting their productivity or personal life. So, what can leaders do about it?

    I recently talked with Randall Degges, head of developer relations at developer security company Snyk, to learn how to address ever-present burnout issues in software development teams at scale. Snyk, which provides popular automated vulnerability detection tools, recently closed a staggering $196.5 million series G funding at a $7.4 billion valuation. With that kind of momentum, they must be doing something right. According to Degges, there are a handful of tactics to consider to prevent burnout, like ensuring developers are heard, not assigning tasks without context and making improvements to the DevOps workflow.

    Understanding Burnout

    Burnout is an ever-present issue, explained Degges. It’s common within any high-stress job where you’re on the hook for constantly delivering new features to support the business. Specifically, in software development, burnout can quickly arise when developers don’t feel they have a voice. Or worse, when they’re tirelessly producing software they know won’t ever see the light of day.

    A paper entitled Burnout in software engineering: A systematic mapping study took an interesting systematic approach and analyzed decades of research into burnout in software engineering. It found the following qualitative work-related factors to be most correlated with burnout. (Interestingly, the paper suggested machine learning might soon help produce more quantitative indicators of burnout.)

    causes of burnout diagram
    Causal-relationship of burnout, taken from ‘Burnout in software engineering: A systematic mapping study.’ (Deciphering this diagram gives me burnout!)

    7 Ways to Fight Burnout

    Since we know burnout is a pervasive problem, how can engineering leaders minimize the issue? Below, we’ll outline some specific strategies to consider and how they might help.

    1. Give Developers a Voice

    First, Degges encouraged leaders to ensure developers have a say in what they’re building. He recalled working within a large enterprise, sitting in a cubicle and having absolutely no say in the features his team was working on. This often included code they all knew would never make it into production.

    Slaving away on irrelevant work is never motivating. Instead, a good internal culture provides a comfortable atmosphere where developers can raise issues early on, said Degges, hopefully before leadership puts targets destined to fail on the roadmap.

    2. Provide More Context

    On that note, when decisions are made, don’t throw orders over the wall in a vacuum. Instead, Degges recommended providing more context into what you’re all creating. Giving visibility into the reason behind decisions will align everyone on project roadmaps and create a shared purpose. This helps foster a culture of empathy and listening to others, said Degges. (David Shackelford affectionally calls this attitude HugOps.)

    3. Optimize Communication Patterns

    Optimizing communication patterns for your working environment is also vital to encourage a healthy culture, said Degges. For example, this could equate to optimizing for the physical reality of distributed teams—no one should have to clock into daily standup at 4:00 a.m. If you are working under timezone constraints, Degges recommended adopting more asynchronous communication.

    For instance, his team relies heavily on Slack and provides dedicated channels for sharing ideas. They’ve also incorporated automation to summarize verbose Slack communications with a recap of conversations. In general, removing unnecessary meetings and optimizing communication patterns could greatly reduce the baggage an engineer might carry in their day-to-day work.

    4. Practice Internal Developer Relations

    As they say, practice what you preach. Many tech-driven companies have adopted a developer relations (DevRel) division to build a community with external developer consumers. But are you practicing DevRel internally? According to Degges, DevRel teams should act as a bridge between external and internal teams.

    “Our goal is to not only to think and empathize with customers but also to encourage internal people to be authentic community members themselves,” said Degges. This can equate to empowering internal employees with resources like mentoring, blog writing or speaking opportunities, he said.

    Additionally, other groups are emerging to help increase internal developer satisfaction. For example, platform engineering and developer productivity engineering (DPE) are being applied by many organizations to improve developer experience around internal tools and processes. Naturally, this will require working closely with internal developers to source feedback and make quality improvements.

    Another method to upkeep engineering morale is through automating away toil. And security vulnerability discovery is one area full of painstaking manual tasks. Half of applications have high-risk vulnerabilities, and new exploits in open source libraries are being discovered almost daily.

    As more software supply chain risks emerge, it’s challenging to keep up with the pace of vulnerability discovery and necessary software patching. Therefore, implementing real-time automated vulnerability detection, said Degges, can help remove many manual processes and free up energy to devote to other areas.

    6. Hire Based on the Work, Not the Interview

    Before developers even get the job, they can be put through high-stress on-the-spot tests during the interview process. Experiences here can set the tone for the entire engagement with the company. But activities like white-boarding algorithms are pretty disconnected from day-to-day work, said Degges. In his opinion, engineers should instead be hired based on the quality of their past performances and the work they’ve produced. This could be artifacts like their personal blog or contributions to open source projects.

    7. Apply Artificial Intelligence (AI)

    Lastly, what’s a tech blog post in 2023 without a mention of AI? Generative AI like ChatGPT has rocked the programming world and is being integrated into more development environments and engineering workflows. The future of AI looks very bright and is poised to help developers produce more with less time and resources.

    Of course, automated code generation will require sanity checks, but these tools can certainly help many ancillary functions. For example, Degges’ team is using ChatGPT to spin up content outlines and automate keyword research, helping support their educational content production.

    Soothing the Burnout

    Even with new tools emerging and a newfound emphasis on developer experience, burnout is still prevalent. A full 58% of developers and security professionals said they are currently experiencing burnout, according to a 2022 Cobalt study. This is, perhaps, due to the rapid release cadences of today’s software delivery.

    “In the past, it took months or years for new software updates to roll out. Now, developers only have a matter of weeks,” wrote Bill Mann, CEO of Styra, on Forbes. “Suffice it to say the stakes are high for today’s software teams.”

    Burnout is psychologically taxing and can negatively affect employee happiness and retention. Therefore, it’s an excellent idea for engineering leaders to take it seriously and consider what positive cultural changes they can make.

  • Things We Should Acknowledge, Part Two: Skills Are Less Relevant

    Things We Should Acknowledge, Part Two: Skills Are Less Relevant

    This is perhaps the toughest issue for all of us to acknowledge: The longer you are in the space, the more skills you have filled out on LinkedIn (or wherever you are tracking your career for HR reps to look at) but the skills you possess today are, by definition, a subset of the skills you will possess even a couple of months from now if you change roles. Note that I said “change roles,” not “change employers.” The fact is that we use an insane number of tools in the development, DevOps and operations environments, and we’re addicted to adding new ones (that might be a hint about part three of this series). Changing roles within a company can involve minor changes, or it can take you to a whole new environment where no one cares that you are a Java genius; they want to know how fast you can pick up Python and be truly productive. Or they nod at your talk of configuring servers and ask if you can do that with Ansible on containers. It’s just not as important anymore for most employees to have a specific skillset. That’s a major contributor to the problems I talked about in part one.

    Of course, there are roles in IT that involve far less change. Email administration, for example, isn’t all that different from the way it has been since, well, forever. We have changed how/where the server is configured and many orgs have added SSO or multifactor authentication, creating new issues, but it’s all evolutionary. When NoSQL systems came out, it looked like a major change in DBA work, but it really wasn’t. Within a year, reality put SQL over top of NoSQL stores and DBAs kept chugging along.

    But DevOps has impacted both development and operations in profound ways. We pretty much standardized on Java, with all of its warts, because it was object oriented, general purpose and could be compiled if needed. Though embedded largely stayed with C, there was no serious competition in the enterprise for Java and Java-like (.NET) languages. Knowing them meant you could do everything but real-time/embedded development jobs—and even some embedded work shifted to Java as prices for miniaturized compute power dropped.

    But that changed. The inflection point was JavaScript, and from there, we went a little bit crazy. When JavaScript took off, full-stack developers needed to know one front-end and a couple of backend languages. Other languages still existed—the staying power of both Perl and PHP is somewhat surprising to me; they’re both good languages but have limitations. But still, no serious competition. But Node.js took JavaScript to the server, Ruby was popular for backends, Python came along for quick scripting … And now we have apps written in those and a dozen more languages floating around.

    I chose languages to fit the whole story into a blog because they see less change than the operations side, where we’ve gone from annual big releases to rolling deployments updated frequently and all the support tools changed in the process. Infrastructure has changed, data stores have changed, mobility has changed, UI targets have changed—basically, it’s a chaotic mess.

    And you’re in the heart of it, keeping the data flowing and the org running. It’s astounding, and you deserve more kudos than you’re likely getting. So approach hiring differently. Some orgs are already there, but skill-based hiring is still too prevalent. Hire broad technical knowledge and a desire to know more. As someone I know honestly said in a job interview once, “No, I never used it, but it is just another programming language. I’ll be productive in a week.” She got the job and delivered as promised. For all but those few corner-case roles, hire for potential and achievements and skills will follow.

  • Managing Engineering Teams During Economic Downturns

    Managing Engineering Teams During Economic Downturns

    The tech industry has faced strong economic headwinds over the past year or so. We’ve seen thousands of layoffs at public tech companies, along with weak stock performances and a decline in venture capital funding. Although some analysts see light at the end of the tunnel, the current slump doesn’t seem like it’s quite over yet. As such, some IT leaders are learning to cope with a slashed budget and a leaner engineering team.

    Simultaneously, engineers commonly desire more flexibility from their employers, both in terms of remote working opportunities and flexible hours. In fact, only 15% of developers are working 100% on-site, according to The State of Tech Hiring 2023 survey, which surveyed 14,000 professionals. In addition to being open to flexible working conditions, some organizations are examining how their internal tools could be improved to increase developer experience and productivity. Amid a downturn, leaders might need to consider these new strategies and many others to maintain steady progress.

    I recently met with Amanda Richardson, CEO of CoderPad, to discuss tips to lead your engineering team during this downturn. According to Richardson, finding the right blend of contractors, remote work opportunities, and helpful tooling is necessary amid the current climate. Leaders can also adopt many creative methods to encourage productivity and motivate developers. Below, we’ll overview four ways technical leaders can build and sustain their teams.

    1. Consider Hiring Contractors Vs. Full-Time Developers

    Recruiters often struggle to fill highly technical positions like full-stack developers, backend developers or DevOps professionals. If a company can’t fill full-time roles, contractors are another valid option. According to Richardson, contractors provide companies with more flexibility because they are easier to hire and cut. These mercenaries could swoop in to pick up the slack by solving common tedious problems, like Salesforce integrations or less-sexy administrative work, she said.

    Of course, contractors don’t solve every need, and they’re not bought into the company’s vision. That being said, more and more companies are leaning toward contractors to help fill technical gaps. In fact, 63% of recruiters now say their company hires contingent workers for tech needs, up from 42% last year, the report found.

    2. Target Remote Work

    Work-life balance has become a top requirement among today’s engineering workforce. For example, remote working options rank as the third-highest priority when looking for work, right behind salary and flexible working hours. Many engineers work better without the interruptions and noise of a traditional office space, explained Richardson.

    Yet, the desire to work remotely is at odds with the policies of some large tech companies, who are attempting to push developers back into the office. Interestingly, this may present an opportunity for small-to-medium enterprises to embrace flexibility and get a leg up on the competition. “Give the people what they want,” said Richardson. “If you want to retain talent, remote work is what you must do.”

    3. Use Tools to Fill in the Gaps

    Another method for filling technical gaps amid a downturn is improving efficiency with new tools. For example, platform engineering is a strategy that aims to develop reusable, self-service platforms to manage internal infrastructure. Centralizing common automation and builds could bring more consistency to deployments and reduce grunt work, freeing up engineers and making them more productive.

    Of course, we’ve also seen a lot of interest in new generative AI technologies, as well as development automation using low-code/no-code platforms. According to Richardson, leveraging AI and new automations presents a newfound opportunity to empower the workforce. For example, ChatGPT can solve interesting challenges in the prototyping phase, like generating a mock database or co-authoring support materials. Technical experimentation is a critical aspect of any team, and new AI capabilities could help bolster and support the operational sides.

    4. Retain a Strong Culture

    Lastly, maintaining a vibrant culture is another strategy to help retain talent. Richardson suggested internal hackathons and coding challenges to bring teammates together in an organic way. An internal hackathon could also be organized to rethink internal processes, providing an outlet for engineers to suggest innovative solutions. There’s also the basic stuff, like daily standups or brown bag lunches, that can create a space to share ideas. “It takes a good leader to figure out what’s going to engage their team,” said Richardson. (One business owner I interviewed actually condoned cooperative gaming during work hours to build community and has seen moderate success with this approach).

    Final Thoughts

    Of course, the above suggestions are easier said than done, and it will take a leader with a keen understanding of their unique engineering team’s attitudes and requirements to find their own version of success. On that note, talent comes in many shapes and sizes and traditional education isn’t everything when assembling a development team. In fact, a full 59% of developers do not have a university degree in computer science, the CoderPad report found, indicating that leaders may have to be more flexible when it comes to recruiting.

    In general, following the above tips could be helpful when managing technical talent. Enabling the company to hire contractors and remote workers increases your potential talent pool, and investing in new tools and company culture could help retain workers. Considering half of today’s developers are considering a job change in the next year, leaders should act now to avoid high turnover before it negatively impacts their team’s output and morale.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Dice.com

    NerdRabbit
    New York, New York
    DevOps jobs
    $120,000 to $150,000

    LinkedIn

    Tata Consulting Services
    Sunnyvale, California
    AWS DevOps Engineer
    $105,000 to $130,000

    Monster.com

    VANET, Inc.
    Irvine, California
    Software Engineer
    $100,000 to $300,000

    SimplyHired

    Renaissance
    Remote
    DevOps Engineer 2
    $100,000 to $125,000

    Indeed.com

    GEICO
    Remote
    Senior DevOps Engineer
    $82,000 to $185,000

  • npm is Scam-Spam Cesspool ¦ Google in Microsoft Antitrust Thrust

    npm is Scam-Spam Cesspool ¦ Google in Microsoft Antitrust Thrust

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: The npm registry suffers spam infestation, and Microsoft makes Google sad. (more…)

  • Debunking Myths About Reliability

    Debunking Myths About Reliability

    “Our service should always be up.” Some myths just won’t die.

    Engineering for reliability is well understood by engineering leaders, less so by bosses demanding unreasonable uptime with minimal resources and immense feature pressure. Business leaders tend to waffle between ignoring reliability with a hand wave and freaking out after an outage. How can they understand the reality of reliability and forget the myths?

    Myth One: Our Service Should Always be Up

    Reality: Our Service is Engineered to Consistently Exceed Expectations

    Some might imagine reliability like a light switch—it is either on or off. But reliability is about consistent repetitions and managing the risk of minor, occasional, recoverable failures. Can we consistently exceed customer expectations?

    Understanding customer expectations is challenging because you can’t ask people what they expect from you (unless you have a peculiar customer). So instead, we want to define and quantify the impact of our service not working and how that could negatively impact customer experience based on observed behavior. For example, “People abandon their shopping cart when our checkout experience takes more than 10 seconds to load, which means we lose money.”

    We reflect business impact in our reliability goals and then engineer our system and processes to meet this goal. We might change how we do releases, ensure tests pass or avoid changes during times of peak usage. These engineering decisions use reliability as a business metric based on customer expectations.

    Myth Two: Innovation is More Valuable Than Reliability

    Reality: You Need to Balance Innovation and Reliability Engineering Work

    The constant drive for new features over reliability is the most frustrating myth. “We need to release new feature X, or we won’t have customers; we can worry about reliability later.” In reality, most customers care about reliability; they don’t mention it until it’s a problem.

    You can have the most fantastic whizbang feature in the world, but if it doesn’t work–and Murphy’s Law will make sure that it doesn’t work at the worst possible time–no one can use it, no one will be impressed and it will turn your technology into a laughingstock.

    Innovation excites customers, but trust comes from reliability, which you must earn through hard work and clever engineering. Depending on your business context, you may require reliability more than ever. If you are facing headwinds, you may need to scale back your ambitions regarding innovative features. Still, you can’t scrimp on reliability or customers will justifiably leave.

    Myth Three: Five Nines is Normal and Incremental From Four Nines

    Reality: Five Nines is Expensive—10X the Cost of Four Nines

    No one–not even massive cloud providers or telcos–can consistently deliver at 99.999% across all their services by accident. Reliability at that level–less than six minutes downtime per year!–is an engineering marvel. A bridge or a dam might look simple after completion, but the engineering required to create a reliable physical infrastructure is immense, as everyone knows. Why is it so hard to understand the complexity, design, engineering and redundancy required to deliver a highly available and performant digital system? Further, it’s easy to think that 99.999% is just a bit more than 99.99%. After all, it’s “just one more nine.” Remind your less-technical counterparts that each nine requires ten times more effort!

    Why is it so expensive to deliver? Because the failure tolerance (also known as an error budget) is 1/10th the size but the risk of missing the goal increases exponentially. You’ll need more redundancy, careful testing and certification of releases, increased on-call rotations, extra hardware or cloud capacity and extensively tested backup plans to achieve this goal.

    Worst of all, higher reliability will slow you down. You can’t innovate or deliver updates as fast when you need to ensure absolute uptime.

    But what if there was a limit to how much reliability we need?

    Myth Four: More Reliability is Always Good

    Reality: Reliability Engineering Has Diminishing Returns

    There is a point at which being “too reliable” is terrible for business. It’s expensive to build all that redundancy, testing, responding to tiny glitches and all the rest. And most of your users won’t notice. We must avoid the large blowups that put us in the headlines and manage expectations everywhere else. The significant outages that can impact thousands, if not millions of customers come from this reductionist view of reliability as a by-product of conscientious work rather than an engineering problem with well-defined tolerances and thresholds. You earn the trust of your customers by properly engineering reliability into your delivery process.

    Consider tardiness at meetings. If you wanted to be 99% on time, you’d need to join a one-hour call within 36 seconds and at 99.9%, you would need to enter a Zoom within 3.6 seconds of its start time, a timescale so small you don’t even notice it. You would have to do this for every meeting you attended, no excuses–bio breaks, last meeting ran long, someone at the door, etc. None of these things matter when defining and measuring reliability. This metaphor also provides a common sense way to think about risk and error budgets. Your other meeting attendees can’t possibly notice (or care) if you’re 3.6 seconds late, no matter how prestigious or impatient the other party is.

    You could apply this same reasoning to catching a flight, picking up your kids from school, completing an exam, building a woodworking project or any human endeavor. The concept is so intuitive to daily life that even pointing it out seems absurd. But this is the fundamental concept from which reliability engineering stems. To build a reliable system, we must define acceptable failure boundaries. Otherwise, we will spend precious time and resources to eliminate the 3.6 seconds of delay that no one cares about and miss the more significant issues–like being present and engaged in the discussion.

    Busting Reliability Myths

    Understanding reliability is vital for engineers and business people alike. It all comes down to intentionally designing a customer experience, keeping up with expectations and, in some cases, even promises. Right-sizing reliability lets you find the perfect balance between delivering excellent service and efficiently running your organization.

    Image Source: Indira Tjokorda via Unsplash 

  • New Relic Bets on AI to Advance Observability

    New Relic Bets on AI to Advance Observability

    New Relic is preparing to extend the capabilities of its observability platform by making use of both additional machine learning algorithms and the ChatGPT generative artificial intelligence (AI) platform.

    Peter Pezaris, senior vice president for strategy and experience at New Relic, said during an interview on an episode of TechStrong TV that a prototype of an AI model created using machine learning algorithms will automatically notify developers when code in a production environment is not performing well—right within their integrated development environment (IDE). The notification will then take developers to the precise line of code identified as the root cause of the issue, he said.

    DevOps teams will need to add two lines of code to applications running in production environments to take advantage of this capability. New Relic also plans to use a similar approach to integrate with generative AI platforms such as ChatGPT to create suggestions for fixing code, he added. ChatGPT now makes it possible to create entire programs in ways that output HTML and JavaScript code, so DevOps teams should expect to be able to use the platform to create natural language queries that invoke the proprietary query language New Relic makes available for its observability platform, he noted. Conversely, ChatGPT will make it possible to explain how a piece of code functions in plain language, Pezaris added.

    Previously, New Relic employed a similar approach to integrate its observability platform with machine learning operations (MLOps) platforms used to build and deploy AI models. This was part of a larger effort to integrate MLOps and DevOps workflows, added Pezaris.

    These AI extensions to the New Relic platform are part of a larger effort to extend the reach of the New Relic One observability platform further left toward developers. Most recently, New Relic added a free CodeStream module to its observability platform to give developers access to metrics and telemetry data that will enable them to write higher-quality code faster.

    That approach eliminates the need to wait for IT operations teams to surface issues that usually don’t manifest until long after the code in question was originally developed. The overall goal is to provide development teams with frictionless access to observability data at every stage of the software development life cycle to reduce mean-time-to-detection (MTTD) and mean-time-to-resolution (MTTR) of issues.

    Organizations should also be able to reduce technical debt faster by making it easier to identify issues that, for example, adversely impact application performance.

    In general, most organizations are still in the early stages of achieving full-stack observability. A recent New Relic survey found only 27% of respondents have achieved full-stack observability, and only 5% claimed they have a mature observability practice in place. A third (33%) of respondents also said they still primarily detect outages manually or based on complaints, the survey found.

    On the plus side, the survey also found nearly three-quarters of respondents said C-suite executives in their organization are advocates of observability, and more than three-quarters of respondents (78%) saw observability as a key enabler for achieving core business goals. However, more than half (52%) of respondents said they experienced high-business-impact outages once per week or more and 29% said they take more than an hour to resolve those outages.

    The hope is, of course, that by augmenting DevOps teams with AI capabilities, the number of those outages will be dramatically reduced in the months and years ahead.