Tag: DevOps practices

  • Breaking Down Silos: Applying Open Source Practices in the Workplace

    Breaking Down Silos: Applying Open Source Practices in the Workplace

    DevOps plays a prominent role in mainstream technology and companies’ ability to deliver results. While the definition of DevOps encompasses the complete range of characteristics it covers, in practice, current tech culture tends to focus solely on the tooling aspect, neglecting the equally essential focus on the people and process critical in delivering value to end users. While cultures are difficult to change and redefine, a transformation is needed to fully capitalize on the potential DevOps has to accelerate software development.

    Conway’s law–the concept that organizations ship software based on their organizational structure–outlines the need for visibility. This law revolves around the idea that in order for a software product to successfully operate, its developers must have established open pathways of communication to ensure each component works as a whole rather than as individual pieces. Unfortunately, today, many organizations still run into problems with how their teams operate when individuals own limited pieces of development.

    Take a retail platform, for example. In the development of the platform, there will likely be several sub-teams that focus on individual components of the design. Perhaps there’s a team for the shopping cart service, another for the catalog service, a team for the underlying data platform and one for user account services. The list goes on. Typically, those teams do not have visibility into each other’s work and operate in their own silo. As soon as teams start thinking about ‘their’ component or ‘their’ microservice, a dangerous tunnel-vision takes hold. In some cases, there may be standup meetings across teams for coordination efforts, but this takes time, energy and steals focus from the important task: Delivering value to end users.

    Alternatively, think about your favorite open source project. The inner workings were likely  designed asynchronously, openly, remotely and at a global scale. Given the remote nature of open source innovation, developer teams don’t need to be co-located in the same workspace. While physically operating separately, open source developers have mastered alternative methods to operate with transparency in such a way that external contributions aren’t just welcome, but encouraged.

    This same idea applies to proprietary software development, or InnerSource, a modern engineering approach that applies open source practices inside your organization to proprietary code. While a technological shift in practice, InnerSourcing is also a cultural revolution toward collaboration.

    Companies of all types can benefit from implementing InnerSource practices. Some enterprise organizations that operate using InnerSource methodologies today include 3M, Ford, KPMG, Mercedes-Benz, Nubank, Spotify and more.    

    A critical component in making this change is prioritizing team visibility. If other teams in your organization are unable to see what you’re working on, you’re inherently (intentionally or not) creating silos and limiting efficient collaboration. In practice, there will be scenarios where company-wide visibility may not be feasible due to external regulations, highly secure environments or sensitive projects; however, these moments of limited project visibility should be the exception rather than the default.

    When approaching InnerSource through an open source mindset, you will find that many DevOps principles still hold true. Here are a few strategies for implementing a collaborative open source culture when developing proprietary code:

    1. Use existing resources: When problem solving, using the existing materials available within your organization is a great place to start—it is possible someone has already tackled the issue you are addressing. If you find the right fit, you can reuse prior work and, if it has not been started, this is an opportunity to add value to your organization by creating a project that could help others in the future. When building upon someone’s work or starting a project, prioritizing visibility by providing an explanation of any technical requirements of the change will allow future project maintainers to determine whether it’s in the project scope, already a proposed change, on the backlog, or not a prioritized piece of work. This use of existing resources will help develop a collaborative culture over time.
    2. Documentation of work: Documentation is critical. Keeping track of the expected standards of the project and which tools or dependencies are required to build the project for local testing will help others in your organization know what to contribute. When in the early stages of contributing code to a project, documenting proposed changes without directly impacting the production version will allow you to receive feedback early on from the project maintainer about the direction you should take as they will be able to see your steps clearly outlined. To contribute code to a project, you can propose changes as a pull request, which doesn’t impact the production version but means you can get valuable feedback on the direction the project should take. Handling contributions by a pull request also ensures that branch protection rules apply and continuous integration practices are required.

    While implementing these practices may not result in immediate changes, over time they will result in a significant cultural shift. And while cultural transformations require continued conscious efforts, embracing modern software engineering practices–learned from the open source community–will enable teams to break barriers for organizational collaboration.

  • Strategies To Consider When Adopting Platform Engineering

    Strategies To Consider When Adopting Platform Engineering

    Platform engineering helps centralize the responsibilities associated with maintaining internal infrastructure. And it continues to be an emerging discipline—according to the 2023 State of DevOps Report, 51% of organizations have already adopted platform engineering within the last three years, and 93% said it’s a step in the right direction.

    I recently met with Hope Lynch, senior director of platform at Cloudbees, to gain some insights on implementing platform engineering. According to Lynch, platform engineering has emerged at a critical juncture. Developers are overloaded, burnout is on the rise and tooling sprawl is starting to rear its ugly head. To respond to these trends, platform engineering has emerged to take the heavy load off internal software infrastructure management. But the practice requires keenly balancing responsibility with control and treating the platform as a product.

    Interestingly, Lynch believes that many IT departments—whether they know it or not—are already implementing some form of platform engineering. They might just need to work out the kinks. Below, we’ll define platform engineering as it relates to DevOps and share some best practices to consider when rolling out (or reiterating) a platform engineering plan within your organization.

    State of DevOps and Platform Engineering

    So, what is the state of platform engineering and DevOps? Contrary to the salacious ‘DevOps is dead,’ headlines, Lynch doesn’t believe DevOps is going anywhere. Instead, platform engineering is helping overburdened developers, having emerged at a time when developers and operations seem to be at a crossroads.

    Developers are now responsible for coding, security, deployment and infrastructure maintenance. “Developers are talented and creative, but everyone has only so much capacity,” said Lynch. She believes platform teams can relieve some of these pressures by creating standard builds and common CI/CD pipelines. “I see it as a partnership between the DevOps teams and platform operations teams.”

    Lynch views DevOps as more of a practice than a specific job title. And although developers should have control over their DevOps toolchain, she believes they were granted too much responsibility. For example, the amount of effort required to search the market, talk to vendors, and try out tools can all add up significantly.

    By creating proven patterns and managing tools, platform engineering teams can help narrow down the options, working behind the scenes to ensure the platform is self-serviceable and continually updated. Of course, this partnership requires regularly collecting feedback to ensure platform teams are enhancing development workflows, not hindering them.

    Best Practices for Platform Engineering Adoption

    So, what are some best practices that enterprises can apply to their internal DevOps platforms? Lynch shared some areas platform engineering teams should focus on to improve their process. By following these tips, a platform can enhance DevOps principles within a company.

    • Be open to feedback from developers. As previously mentioned, platform engineering and development should be partners. Platform engineers should seek to gather as much feedback as possible from developers to improve their experiences and ensure the right DevOps tools are acquired.
    • Don’t control everything. As such, a degree of flexibility should be granted—you don’t want to mandate standardized adoption of all tools. Yet, you do want to decrease rogue tooling and sprawl. The platform team should thus work to find a balance and build trust.
    • Maintain a DevOps culture. When people discuss DevOps, they often talk about the tools and technology, not why DevOps came to be. “DevOps is here because we were trying to find a way to allow teams to work better together and be high-performing,” said Lynch. As such, we shouldn’t forget to include the people element here. Platform engineering should reinforce a healthy culture with positive dialogues and ensure work is “a fantastic place to be.”
    • Treat the platform as a product. When delivering internal-facing tools, your developers are your consumers. Thus, it’s a good practice to treat the platform as a product in its own right. This includes publishing roadmaps and eliciting feedback. “The platform team really needs to win the business of their internal customers, or else they’ll buy or use something else, which could cause shadow IT.”
    • Focus on increasing developer experience. As part of viewing internal developers as customers, platform teams should invest in improving the developer experience associated with using the platform. This means reducing friction and integrating new capabilities into preexisting workflows whenever possible.

    Goalposts For Platform Engineering Success

    For any new initiative to be successful, it must have goalposts and be measured in some capacity. The same is true for platform engineering. So, what are some goalposts that IT leaders should have in mind when introducing platform engineering? According to Lynch, some positive end results include collaboration and efficiency, cost analysis, cybersecurity and tooling centralization.

    • Collaboration and efficiency: First, platform engineering should improve work in the best way for everyone, says Lynch. “The people component is always there and should always be a priority,” said Lynch. To reach these goals, she recommends having the right conversations to co-discover bottlenecks in the value stream management of software development.
    • Cost optimizations: Secondly, platform engineering is an opportunity for garnering insight into usage rates, which could inform cloud cost analysis. Platform teams can implement specific policies, such as spinning resources up or down to reduce expenses.
    • Improved security: Platform engineering teams can be the first to test new products, giving them a unique opportunity to synchronize requirements with security teams. They can also take the reins of continually patching dependencies to reduce vulnerabilities.
    • Increasing consistency. In general, another benefit is centralizing tooling control. Because when tens and thousands of developers are each using their own tool, it becomes harder for teams to work together. While some cutting-edge technology organizations may seek to give developers control over the full cycle to increase agility, most will benefit from centralizing procurement and oversight, said Lynch.

    The Human Side of Platform Engineering

    “Platform engineering, when done right, furthers the principles of DevOps,” said Lynch. And interestingly, it may be the case that teams don’t even realize they are already doing platform engineering. Once they realize they are gravitating toward these ideals, it will be an even greater reinforcer for DevOps and DevOps principles, she said.

    Currently, the tech space is in a tricky spot, with rising layoffs and tightening budgets. At many organizations, fewer people are now expected to perform the same high caliber of work. As a result, we need to place more emphasis on retaining the remaining talent and boosting morale. This means caring about what the developer’s experience is like throughout the day and supporting the human side of DevOps. According to Lynch, platform engineering is a natural choice in these circumstances.

  • Why SREs Are Critical to DevOps

    Why SREs Are Critical to DevOps

    Although a relatively new concept, site reliability engineers (SREs) have become crucial for DevOps teams, helping to solve an array of operational problems such as network availability and user experience. However, in previous years, some people have questioned the longevity of such a role. 

    This article will discuss several reasons why site reliability engineering roles are here to stay and why they have become critical to DevOps. 

    What Are Site Reliability Engineers (SREs)?

    A site reliability engineer helps connect development teams and IT operations by completing tasks previously assigned to operational roles. SREs use a range of automation tools for troubleshooting purposes, creating software systems that are both scalable and reliable.

    The role revolves around automation and standardization, as cloud systems become increasingly prevalent. SREs also play a vital role in ensuring the user experience of systems is of the highest standard.

    Why Site Reliability Engineering Roles Are Here to Stay

    Let’s take a look at two of the primary reasons SRE roles are here to stay. 

    User Expectations

    The key reason for the rise in popularity and the expected longevity of site reliability engineering roles is user experience. In the modern climate, user expectations are very high and more dedicated roles are required to help deliver a service that is not only acceptable but attempts to exceed user expectations.

    Even back in the 2010s, users were much more patient in terms of website load speeds and how quickly they could access information. Even crashes and load failures were tolerated without the user exiting a website.

    However, that has all changed due to the competitiveness of the market, as well as the growth of remote working, which has increased dependency on online resources. This means that if a service is unavailable for any period, users are unable to do their job. 

    An SRE’s role includes optimizing systems, ensuring they perform at the required level and that resources are always available to those that need them. This makes SREs critical to DevOps teams that may not have the capacity to focus on meeting user expectations sufficiently. 

    The Complexity of Software Architectures

    Another factor that has added a lot of value to the role of an SRE is their ability to provide much-needed support for software environments and architectures that continue to develop in terms of complexity. Kubernetes is one such architecture that requires dedicated attention. 

    Applications that are distributed across systems such as Kubernetes and other cloud-native platforms require specialized personnel that have the necessary knowledge and experience. Machine learning is also becoming more prevalent and requires special focus from a performance perspective. In 2020, just more than 22% of companies had machine learning models in production for one to two years.

    This level of complexity is only expected to grow further over the coming years as cloud-native systems become the norm. This is why new roles must be created that go beyond that of a normal IT engineer. 

    Challenges for SRE Roles

    Like any IT role, the role of a site reliability engineer always faces challenges that have led people to question whether SREs are a long-term solution or whether it is a niche specialization. 

    A Lack Of Opportunity Away From Hyperscale Companies

    The role of an SRE was created by hyperscale companies like Google that manage extremely large IT systems. For companies that do not have such a wide and varied IT infrastructure on a global scale, there are questions about whether such a role is necessary. 

    However, as previously mentioned, the growth of cloud-native technology shows no sign of slowing and, with this growth, opportunities for SREs at smaller organizations may arise. 

    The Role of an SRE is Considered Obscure

    The ambiguity of an SRE role can be considered a downfall in some cases, with many businesses unsure of what these professionals offer when compared to a traditional systems engineer. In many cases, SREs are deemed to be a hybrid of a software engineer and an IT operations role. 

    Fortunately, DevOps has realized the potential of SREs and their ability to provide an important bridge between developers and engineers, helping to achieve dedicated tasks to a high standard. 

    SREs Vs. DevOps

    Although both roles are connected and share some similarities, there are some clear differences.

    • Processes – DevOps has full visibility into the development environment which allows them to make changes from the initial development stages through to production. SREs, on the other hand, only have visibility into the production process, allowing them to make suggestions to ensure performance levels are maintained.
    • Implementation – The task of implementing new features on a system falls on the DevOps team. Meanwhile, SREs are tasked with ensuring the new features do not cause any system failures or impact performance during the production stage.
    • Key Focus – The primary focus of a site reliability engineer is to ensure the system’s reliability, availability and scalability. DevOps focuses on how quickly product development is completed, as well as its continuity.
    • Structure – The structure of an SRE team and a DevOps team have obvious differences. SRE teams are made up of individuals who have similar skill sets relating to development and operations. Whereas DevOps teams consist of professionals who have specific roles related to individual aspects of the project. These roles can include the team leader, product owner, cloud architect, software developer, QA engineer, system administrator, release manager and more.
    • Tools – DevOps teams often use tools such as integrated development environments (IDEs) when developing a product. These can include Jenkins (continuous integration and continuous development), JIRA (change management), Splunk (log monitoring) and GitHub (distributed version control). SREs commonly use tools such as Prometheus and Grafana (collection and visualization of metrics, such as CPU usage and available disk space), OP5 and PagerDuty (incident alerts), Ansible and Kubernetes (container orchestration) as well as a range of cloud platforms.
    • Bug Reporting – DevOps debug code whenever a bug is reported in the end product. SREs are not involved with debugging from a development perspective and are only required to perform such tasks if there is a production outage, there are infrastructure issues or perhaps if they are dealing with common AWS misconfigurations.
    • Measuring Performance – The typical metrics that DevOps uses to measure performance are deployment frequency and deployment failure rate. SREs measure service level objectives (SLOs), service level indicators (SLIs), service level agreements (SLAs) and error budgets.
    • Incident Handling – In the case of incident handling, DevOps teams work to mitigate an issue based on the feedback provided. SREs then analyze the issues post-incident, including the root cause of the issue. This is then documented and provided to the developers to fix. 

    Why SRE Roles Are Critical to DevOps – Conclusion

    As you can see, the two roles differ significantly but SREs can play a critical role in ensuring the DevOps team delivers a product that offers maximum performance. 

    The core benefits of employing professionals in a site reliability engineer capacity are high levels of product performance and reliability to meet user expectations while helping to manage complex IT architecture. 

    In the future, organizations that manage a large amount of cloud-based IT assets may struggle to deliver efficient products without such professionals. As such, demand for SRE roles is likely to grow. 

  • Do’s and Don’ts of Phased Rollouts Software Delivery

    Do’s and Don’ts of Phased Rollouts Software Delivery

    When it comes to app stability and user experience, the “move fast and break things” approach is a recipe for disaster. To protect brand reputation and to stay ahead of evolving needs and customer expectations, leading Silicon Valley companies’ software engineering teams have adopted phased rollouts, a software development method that involves the gradual implementation of a new feature to a smaller cohort of users at one time rather than all users at once.

    Phased rollouts help companies maintain the highest-quality customer experience (CX) for their users while making changes and software updates. When significant features and updates aren’t rolled out to production all at once, the organization’s software engineering team doesn’t have to deal with all the potential implementation issues at once. With phased rollouts, engineering teams catch issues early in the process so they can quickly fix them and resume the rollout of the experiment, which prevents issues from impacting millions of users down the road.

    While companies everywhere try to emulate the app development techniques of these Silicon Valley tech leaders, many are unable to achieve successful phased rollouts at scale. What they often don’t realize is that before even implementing a phased rollout approach—whether it be to 5%, 10% or 15% of their user base—there are a few steps that must be taken to ensure success. Below, I’ll walk you through the do’s and don’ts of a successful phased rollout plan.

    Do: Release Stability Management

    Companies must drive consistently stable app and web experiences in order to succeed in the market today. Therefore, it’s critical to continually monitor the stability of releases throughout development and testing to catch problems early and decide when to roll features out to users. By measuring and monitoring app stability in real-time, release managers inject accountability, predictability and complete peace of mind into the organization. Taking control of release stability management helps engineers deliver stable applications to drive world-class digital experiences.

    On top of that, software development teams must constantly improve mobile and web apps with new features and capabilities that better enable users to accomplish what they want to do. These kinds of improvements give companies the competitive edge they need to survive in today’s competitive business environment. In addition, monitoring release stability provides engineering teams with real-time visibility into every issue impacting application stability and CX. This level of visibility is critical for quickly diagnosing and understanding the root cause of errors so that engineers can roll out fixes before users are negatively impacted.

    Do: Testing and Early Access

    For a phased rollout, engineering teams must run A/B tests and quality assurance (QA) testing. A/B testing (also referred to as split testing or bucket testing) is a method of comparing two versions of a web page or app against each other to determine which one performs better. This includes trying out features like different screenshots, videos and even different app icons to measure which format users like the most.

    QA testing ensures that the organization delivers the best products or services possible by detecting usability issues that might not be technical. It tests for any problems that impact the end-user experience, from poor navigation to slow load times or unclear web copy.

    With phased rollouts, you must also provide early access to new features for select users to collect feedback and stamp out critical issues before launch. Many software products collect user feedback by including a small checkbox somewhere asking users to help make the software better by sending insights into how they use the application. In addition, social media has become an undeniable avenue for collecting candid user feedback. We’ve all seen it—when a platform changes its interface for certain users before others, users will post on social media sites about whether they are pleased with the new update or not. Historically, there has been backlash when significant changes are made and the company then has to decide whether to keep, change or cancel the new changes to the platform.

    As mentioned above, with an app stability monitoring tool, you can see if new features or experiments are introducing new stability issues and make decisions about pulling back or rolling out to a larger subset of users. By conducting testing and allowing early access to only a specific cohort of users, software engineering teams can balance release speed with application stability to create the best experience possible.

    Do: Allow for A Staging Period Before Going into Production

    As you launch new features, you’ll want to test them in a production-like environment after they go through QA testing. This is called staging, and it has become a popular concept with Silicon Valley companies. Staging is a near-exact replica of a production environment for software testing. Staging is meant to test code, builds and updates to ensure quality under a production-like environment before application deployment.

    A production-like environment is similar to when the app is released into production for users, but it’s not public yet. It provides an accurate view of how the update or new feature will operate and look in a production environment to avoid glitches that are only noticeable in production. Staging can include manual tests or, in some cases, engineering teams use automated software testing, which is more costly to implement and may only be attainable for larger enterprises.

    In this phase, software engineers should conduct additional tests and experiment with the new features to ensure that these capabilities—when released to production or to the larger user base—will not compromise the app’s stability. The last thing you want is for a new feature to break the user experience.

    Don’t: Try to Fix Every Bug or Error as it Occurs

    Since engineering teams often don’t have the bandwidth to fix every coding error that should arise, they need to prioritize and fix the errors that impact customers directly in order to protect revenue. Although organizations should have visibility into every error or bug, not all of them are worth fixing. This causes the signal-to-noise ratio that plagues developer efficiency.

    This is where app stability management plays an essential role. However, many of the app stability monitoring solutions on the market today can help with identifying errors, but leave it up to development teams to figure out which ones need to be prioritized. Engineering teams have to quickly determine which errors are the most urgent to fix, so they need to have reliable tools such as an app stability and error monitoring platform to provide them with visibility to make these important decisions in real-time.

    Phased Rollouts for Progressive Delivery

    Phased rollouts allow software engineers to continuously optimize the digital experience, without impacting the whole user base and negatively affecting the brand. Once mastered, phased rollouts can be used as part of a cutting-edge progressive delivery approach to software engineering and development.

    A progressive delivery approach includes shipping code (sometimes unfinished code) to a production environment multiple times a day, exceeding many companies’ risk tolerance. In order to do this successfully, engineers must have control over who can ship releases and when, and which users can access certain features and updates at what time. Phased rollouts provide a way to shield the majority of customers from the failures that occur at those rapid release speeds. To ship new software releases faster, engineers have to spend less time on QA and testing, but that doesn’t mean they should be overlooked completely. Then, by delivering the new release to a small subset of users first, you can manage risks more effectively and iterate accordingly.

    It’s important to call out that if the engineering team is making minute changes that no users will see or care about, you don’t need to use a phased rollout approach. That said, if it’s a significant cosmetic or operational change, such as updates to the UI, then phased rollouts are crucial. A phased rollout strategy is key for making software or app updates at scale that are necessary for a successful launch in today’s highly competitive market.

  • Cattle, Not Kittens – Revisited

    Cattle, Not Kittens – Revisited

    The early days of DevOps were filled with pronouncements that we needed to treat our servers as cattle, not pets (kittens). For a year or so, the refrain was everywhere. And we did. We detached servers from dedicated hardware, we made spin up and spin down so easy that we just replaced malfunctioning servers (which has caused its own problems), we have even made servers designed to spin up, do one thing well, then spin back down until needed again – something unthinkable even a decade ago.

    And we’ve reaped benefits. Servers suck up less attention from the DevOPS side, and standardization has made it easier to code for them, moving things along and allowing members of another team to pitch in when there is a crunch because of familiarity.

    But IT is filled with people, and something happened in the transition. In hindsight, we should have seen it as a possibility, but things changed so much, so fast that I don’t think anyone was considering this possibility.

    Our DevOps tools/toolchains/processes became our pets (kittens).

    Now the special snowflake that requires specific knowledge and has to be treated with kid gloves is the DevOps step, not the servers it uses. Like pet servers, a fragile step that is working is avoided like the plague, when it should be shored up. Like pet servers, a process that is locked into a specific tool, OS, or platform is left that way, even when other options are a better solution. Like pet servers, the person who “knows all about” that is left to manage the step that should, by all rights, be easily understandable by anyone on the DevOps team.

    It’s time to start treating DevOps like cattle, not kittens. Some steps (like some servers) will be more difficult to make less pet-like and more cattle-like, but if history is any gauge, the payoff is worth the investment.

    We don’t have a cross-toolset language, other than doing things in the various shells and calling the shell scripts from a given toolset (not the best solution, just an example of where we’re at), but that shouldn’t stop us from standardizing and genericizing as much as possible. Less fragility, more reusability and a broader support team is good for everyone.

    You all are knocking it out of the park. We – all of us – are spending more time at work and at home in the digital space than ever before, and, generally speaking, the tools “just work.” That’s all on you. Keep kicking it. But standardizing to keep things running smoothly just makes sense. You know your org; it is possible that the easiest route to cattle is to standardize on a single toolchain (perhaps one or two toolchains with reusability documented) possibly to shove a bunch of steps into scripts and share amongst whatever tools each team chooses (I, and history, strongly discourage this approach, but neither I, nor history, has to live in your environment. There might be a good reason for this, so I include it here).

    All I’m saying is, don’t re-implement every step over and over if you can avoid it, and don’t make steps so arcane that you need a specialist (not internal, and definitely not external). Keep rocking it, and get those cattle moving!