devopsy.kinsta.cloud

Category: DevOps Practice

  • Survey Surfaces Substantial Platform Engineering Gains

    Survey Surfaces Substantial Platform Engineering Gains

    A global survey of 500 application development and deployment professionals finds that while 43% work for organizations that have had a platform team in place for three to five years, there doesn’t appear to be much consistency regarding where that team reports within an organization.

    The survey, conducted by Puppet by Perforce, a provider of an automation framework, found that among organizations that have platform teams, well over half (58%) operate within the context of a larger DevOps or infrastructure management organization, compared to 40% that have their own dedicated leadership.

    David Sandilands, a principal solutions architect for Puppet by Perforce, said the survey makes it clear that platform engineering, in general, is not a new concept, but it’s clear more organizations are trying to reduce the level of toil currently associated with building and deploying applications.

    Regardless of where platform teams reside in an organization, nearly two-thirds of respondents (65%) said they are important to their organizations and would receive continued funding. The primary purpose for forming these teams is to increase productivity (58%) and automate standardized processes (51%), the survey finds. Top goals for platform engineering teams are to solve problems (30%), enforce security processes (27%) and accelerate transitions to cloud-native computing environments (26%). A full 70% said security was built into their platforms from the start.

    That suggests that, in addition to assuming responsibility for traditional DevOps workflows, many platform teams are now also being asked to ensure DevSecOps best practices are followed at a time when application security requirements are becoming more stringent, noted Sandilands.

    More than three quarters (76%) also reported that they had deployed two or more self-service portals, with 27% having deployed five or more.

    Only 22%, however, said they have deployed applications on Kubernetes in a production environment, with just under half (46%) reporting they have no current plans to deploy a cloud-native computing platform, which are widely seen to be challenging to deploy and maintain.

    Overall, the survey makes it clear the primary focus is to ensure platforms continue to evolve as developer requirements continue to evolve, said Sandilands. Platform teams are not necessarily trying to dictate what tools should be used; instead, they are finding ways to automate processes that are increasing the level of friction developers encounter, he added.

    In the longer term, advances in artificial intelligence (AI) will make it simpler to achieve that goal using natural language interfaces that DevOps teams can more easily invoke, noted Sandilands.

    Each organization will need to decide for itself whether platform engineering makes sense as a methodology for managing DevOps at scale, but as application development and deployment continues to evolve, more nuanced approaches will be required. In general, organizations are trying to strike a balance between empowering developers and the need to streamline backend processes to increase productivity while also simplifying compliance and improving security.

    The challenge, of course, is enticing developers to buy into that concept rather than using their expertise to resist platform teams that might, from their perspective, limit their prerogatives in ways that ultimately stifle innovation.

  • Survey Surfaces Lack of Significant Observability Progress

    Survey Surfaces Lack of Significant Observability Progress

    A global survey of 500 IT professionals suggests organizations are not making a lot of progress in their ability to truly observe application environments, especially as they become more complex with each passing day.

    The survey, conducted by Logz.io, a provider of an observability platform, found only one in 10 respondents reported they have full observability into their application environments.

    Logz.io CTO Asaf Yigal said while more DevOps teams than ever are collecting logs, metrics and traces, most of them have yet to determine how to operationalize all the data being collected. As data is ingested, it doesn’t just need to be stored; it also needs to be correlated to the various services that make up an application, he noted.

    The more organizations deploy cloud-native applications in production environments the more pressing this issue becomes. All the microservices that make up those applications are now generating a massive amount of telemetry data that results in more alerts than ever being generated.

    Not surprisingly, the biggest challenges organizations encounter when managing Kubernetes clusters in production environments are monitoring/troubleshooting (40%), followed closely by security (37%) and networking (33%).

    Many organizations simply don’t have the skills required to manage cloud-native applications. Nearly half of respondents (48%) specifically cited a lack of knowledge as the biggest challenge they encountered when trying to observe these types of applications. From an IT management perspective, most microservices all use the same basic template, so it’s difficult for DevOps teams to identify which microservices are likely to have the biggest impact on service level objectives (SLOs) and service level agreements (SLAs) in the event of a disruption, noted Yigal.

    In the absence of the ability to determine the actual root cause of an issue, alert storms conspire to increase fatigue—and that eventually results in higher levels of burnout across the DevOps team, noted Yigal. In fact, 82% of respondents said their mean-time-to-resolution (MTTR) during production incidents was more than an hour.

    The survey also noted that more than half of respondents (52%) worked for organizations that are simultaneously trying to rein in monitoring costs. More than three-quarters (76%) of respondents also noted that open source OpenTelemetry (OTEL) or OTEL-centric tooling was at least somewhat important to their overall observability strategy.

    In addition, 87% of respondents said their organization is already using some form of platform engineering to manage DevOps workflows at scale.

    It’s not clear how quickly organizations are adopting observability tools and platforms, but too many DevOps teams lack the visibility needed to identify the root cause of an issue. As a result, variations of the same problems often keep manifesting themselves because previous remediation efforts simply didn’t go deep enough to resolve the core issue.

    Of course, there may come a day when machine learning algorithms, along with other forms of artificial intelligence (AI), will make it simpler to surface these issues. The challenge, in the meantime, is laying down the observability foundation today to provide access to the data that will be required to train those AI models.

  • Is Backstage the Right Internal Developer Portal for You?

    Is Backstage the Right Internal Developer Portal for You?

    There has never been a better time to be a software developer. There is a language and framework to solve virtually any challenges we encounter. New tools, architectural patterns and methodologies like DevOps and Agile have changed how we work, architect our applications and organize ourselves. This has enabled teams to be nimbler than ever before. But this has all come with a hidden cost: Sprawling complexity.

    Sprawl is a complex web of disparate tools, languages and frameworks used across an organization’s projects. While this diversity allows for specialized solutions tailored to specific problems, it creates a labyrinth of interdependencies and varied skill requirements. This complexity often results in developers constantly navigating through a maze of tools, leading to inefficiencies and a steep learning curve for new team members. As organizations adopt more tools and technologies, the cohesive understanding of the entire software ecosystem diminishes. There are two areas in particular where this pain is felt acutely.

    The first area sprawl acutely impacts is the ability to find essential knowledge about systems, which becomes fragmented across tools and resources as sprawl increases. This fragmentation leads to longer incident response times and makes it more difficult for newly hired engineers to ramp up. Knowledge fragmentation also makes it considerably harder for enterprises to measure and manage initiatives to achieve high standards for security and reliability, as well as track migrations to new architectures like containers or clouds.

    Sprawl also impacts frequently occurring tasks for developers, such as managing “day two operations” like changing environment variables, adding secrets or modifying a replica count, or creating something new like a microservice or an environment. Given the complexity required to create and operate infrastructure, many organizations have invested in creating automations via infrastructure-as-code and CI/CD systems; however, these automations are often difficult for developers to discover and use. As a result, developers often file tickets with ops teams to complete these tasks, creating a process bottleneck that slows development velocity while burdening ops teams with mundane, repetitive tasks. This is commonly referred to as TicketOps.

    All of this has made the developer experience worse than ever. Developers report high levels of cognitive overload and burnout, which can cause unwanted developer attrition and reduced overall productivity. But it doesn’t have to be this way!

    Introducing Backstage

    In the early days of the cloud, only a handful of enterprises with lofty valuations could afford to solve these challenges by dedicating 50 – 100 engineers to build internal tooling designed to address the challenges of sprawl. Everyone else was left to suffer with subpar developer experiences or partial solutions like stale spreadsheets, manual processes and information that were often out of sync with the underlying system.

    Just as the pain was becoming unbearable for most enterprises, Spotify donated the tooling it created to address these challenges internally to the Cloud Native Computing Foundation. As a free, customizable, open-source project focused on these pain points, Backstage quickly captured the attention of enterprises worldwide.

    Over 2,000 enterprises have tried Backstage, making the project a tremendous success by promoting a vision for how internal developer portals can alleviate the pain of sprawl. However, a surprising number of enterprises struggle to adopt Backstage. In an interview with the head of engineering for Backstage, it was revealed many companies that have deployed Backstage have achieved only ~10% adoption rates, which is typically equivalent to a proof of concept. Understand why enterprises have struggled to succeed and decide if Backstage is right for you.

    Deciding if Backstage is Right for You

    Here are key questions to help guide your decision.

    Are You Ready to Build a New Product?
    Backstage is a framework to build an internal developer portal, but it is not an off-the-shelf tool or a solution. Many adopters have not understood what this means, at least at first.

    As a framework to build a tool, Backstage requires dedicated product and engineering resources to configure, customize and maintain. It is not uncommon for organizations to require four to six or more fully dedicated resources to succeed with Backstage. The engineers you dedicate to building this framework into a product will need to be skilled in TypeScript and React, and willing to work with an open source community and product.

    Getting up and running with Backstage involves more than 70 steps. It is not intuitive and can result in months of work before you can demonstrate value. Some of the steps and associated time commitments are obvious, such as deploying your instance of Backstage. Other steps may not be obvious to new adopters.

    For example, Backstage has historically relied on open-source plugins to enable organizations to connect their tools. The open-source nature of these plugins has been a key selling point of Backstage; however, the plugins have uneven quality and depth and can require additional customization to enable basic functionality like search. In another example, tuning the Backstage data model to your organization is so difficult that many adopters abandon hope and work with the default data model. Upgrading versions and plugins and the effort required to keep the data in Backstage synced with your system contribute to the engineering resource load required to succeed.

    The advantage of Backstage is the infinite customization you can accomplish thanks to its open-source code base. You can change anything you want if you write and maintain it. This includes adding custom data sources, customizing the data model, extending plugin functionality and having the UI reflect your corporate brand identity. Again, the trade-off is time and complexity relative to your alternatives. For instance, configure8 and Port offer the flexibility to tailor the portal’s data model to the contours of your organization and process any custom data source for use throughout the porta. Several vendors can support custom logos and color schemes in the portal’s UI (or you can build a more tailored UI on top of their APIs).

    Does it Support Your Scenarios of Interest?
    Internal developer portals as a product category have evolved well beyond their microservice catalog roots. If you are considering Backstage, it is important to ensure it offers functionality that addresses the needs of your enterprise’s stakeholders.

    For example, many organizations leverage functionality called Scorecards to define and measure teams’ compliance with reliability standards, security and architecture. Open source Backstage has a version of this functionality, but it is not intuitive and has a fairly high difficulty level (i.e., writing code, writing data collectors, etc.). The commercial vendor community typically all offer Scorecards. Adopters should consider the breadth and depth of information that can be checked, the ease of creating new scorecards, and the quality of the developer workflow and UI when evaluating Scorecards.

    In another example, Backstage enables users to scaffold new services via the Cookiecutter open-source package. Why take a principled dependency on a code generation solution instead of an open approach that enables you to use the best tool for the job? There is a robust landscape of available solutions to generate code and orchestrate workflows. Having a flexible way to directly integrate these reduces complexity.

    Backstage also offers functionality that enables developers to see the cloud costs of the code they operate. Upon closer inspection, adopters find this module only enables high-level aggregations of costs over a fixed timeframe. To truly provide developers with the knowledge and functionality they need to improve costs, consider closed-source vendors that offer more granular insights and actions, such as drilling in beyond aggregate costs at the microservice level to see the per resource cost drivers, an individual resource’s cost drivers, config settings and blast radius and weave in self-serve actions so developers can safely make changes where appropriate and permitted.

    The resource and environment catalog is another area to consider. Is it necessary to have the environments and resources that power them automatically mapped to each service? Do you want a centralized catalog of your resources across clouds and data centers that you can sort and filter using various keys? Backstage can support some of these scenarios, but not without significant work and at present, the connection to this data is brittle and manual versus automatic. This stands in contrast to certain closed-source vendors, such as configure8 which has extensive functionality.

    In a prior section, we discussed the importance of data model flexibility. This is an important evaluation criterion for organizations that want the flexibility to push in and present any data source, to customize the schema of the catalog, and finally to insert custom keys for results filtration and analysis (i.e., does this resource touch personally identifiable information; show me results related to a line of business, etc.). Ensure you understand whether you are poised to succeed in tailoring Backstage’s data model, which has stymied many adopters, and that any commercial vendor you evaluate can easily support customizations.

    Another consideration is Backstage’s TechDocs, and whether you want to adopt TechDocs for your entire enterprise or allow teams to use best-of-breed solutions. Whereas Backstage locks you into TechDocs (which can also make a future migration to a new portal solution more difficult once end-user behavior becomes entrenched), commercial vendors generally support open and flexible approaches that enable integration with any documentation solution.

    Also, ensure Backstage meets your information security needs. For example, most organizations desire a robust role-based access control model so developers only see permitted services, scorecards and actions in the UI, via search and the API. Backstage does not offer this functionality in the open-source edition. There is a framework to work with permissions included in Backstage but to get real value out of it, you need to budget to license a closed-source RBAC solution from Spotify or build it yourself.

    Finally, evaluate the pace of innovation you desire from your internal developer solution. Closed source vendors, two in particular, have demonstrated an extremely rapid pace of innovation, launching many industry firsts. Compare this to the pace of innovation in the open source community as internal developer portals are sticky once installed and tend to be long-term relationships.

    Are You Prepared for Success?
    Achieving broad internal adoption of an internal developer portal requires more than getting a portal configured, it requires solving the pain points that matter to your organization, demonstrating success, making the organization aware of that success, and creating a feedback loop to overcome challenges. To accomplish this, you must be aware of best practices, create content and forums to engage the organization, and ultimately adopt a product-oriented approach to ensure your platform engineering effort succeeds.

    Success in this area often combines budgeting and access to best practices.

    The Backstage community is vibrant and active, and you can pick up best practices by engaging with community members. But for enterprises and internal champions who need to succeed in driving the adoption of a new internal developer portal, you may wish to consider alternatives to community support.

    This can be accomplished via multiple methods. Consulting firms and individual practitioners specialize in this area and can coach you. Organizations above a certain size often invest in dedicated product management resources to own this function. Finally, certain vendors like configure8 focus on delivering success versus a tool (or framework to build a tool like Backstage) by providing consultative best practices, content and other value-adds packaged with their software to ensure you achieve robust adoption levels.

    Finally, to the extent your scenarios of interest involve self-serve actions to alleviate TicketOps, ensure that you have budgeted the appropriate engineering investment to develop the golden paths or blueprints that are vended inside an internal developer portal. Certain vendors like configure8 offer a solutions layer that creates custom golden paths for you. However you procure golden paths, you must balance your investment to ensure you get a great “razor” and all of the “razor blades” you will need to support the unique needs of your various internal customers.

    Do You Understand Your Total Cost of Ownership and Alternatives?
    One of the considerations of Backstage is understanding its total cost of ownership versus its best alternative.

    While Backstage is open source and free, we have noted that a meaningful engineering investment is required for installation, customization, maintenance, internal support and product management. Quantify the fully loaded cost (i.e., salary, taxes, benefits, facility charges, etc.) of each human resource required to succeed with Backstage.

    We have noted there may be licensing costs from Spotify for any essential features you do not wish to build yourself. There will be internal hosting costs and budgeting for any consulting costs.

    Several enterprises who have evaluated Backstage now realize the total cost of ownership of alternative vendors is much lower and more predictable. Leading vendors offer robust alternatives for catalog and scorecarding, self-serve actions, and collaborative cost management, and two vendors have evolved to provide compelling functionality and high degrees of flexibility. The key is finding a vendor who balances overall functionality and flexibility with ease of use, speed to value, and enterprise-grade support. This is where configure8 has struck a unique balance in addition to a lower cost of ownership relative to Backstage.

    Finally, any return on investment analysis must capture the opportunity costs. Here are some common examples:

    • Time to market trade-offs. Setting up Backstage for your proof of concept can take over a month, getting to a first releasable version can take 3-6 months, and full adoption can take 6-12 months. During this time, you may generate no or minimal business value. Specialist vendors can deliver success faster. The incremental business value you can generate with a specialist vendor versus Backstage related to ‘speed to value’ represents an opportunity cost you need to quantify.
    • Foregone features: Scenarios like Scorecards or RBAC that are important to your organization and you will have to pay for or forgo, you need to factor these incremental costs (or the cost of not having these features) into your analysis.
    • Opportunity cost of internal engineering resources: How many engineers are required to support Backstage over and above what’s needed to succeed with a specialist vendor? What business value could these incremental engineers have generated by instead supporting the unique value your enterprise brings to its customers versus an internal tool in a world of compelling alternatives? This becomes another cost of owning Backstage you need to compare v. the cost of buying a solution from a vendor.

    Understanding the direct and opportunity costs of your best alternatives will help you put your budget to work in the most efficient way and aid in your internal discussions with finance and other stakeholders.

    Carefully Consider Any Clones

    Given the initial popularity of Backstage, it is not surprising to see clones emerge built on Backstage. A clone is a vendor who has taken Backstage source code and added some feature crust on top, often a security wrapper, hosting and maintenance, light UI customizations, and potentially filling in a functionality gap like Scorecards. This approach can alleviate some of the burdens of open source Backstage in exchange for licensing costs you incur, it is important to understand if the clone is the best fit for your unique needs given the highly competitive vendor landscape that is now available. An easy way to think about it might be comparing Jenkins versus hosted Jenkins versus Jenkins with a vendor security wrapper relative to what enterprises get with Azure DevOps and GitHub Actions.

    Final Thoughts

    Building an internal developer portal from an open-source framework can be rewarding for organizations that can achieve value from extensive customization that is impossible elsewhere. For many organizations, however, it is important to properly scope the effort required to succeed with Backstage and compare it to your alternatives in the vendor community. While Backstage does offer certain advantages, DevEx and platform teams are increasingly finding greater success more quickly and at a lower total cost of ownership via specialist vendors like configure8.

  • Our Infrastructure is Still Expanding

    Our Infrastructure is Still Expanding

    You would think that sooner or later, marketers and early adopters would take a rest, but it seems that if one group slows down for a bit, another picks up. Our infrastructure is expanding in almost every possible way, and this creates more of a burden on every aspect of DevOps, specifically in IT.

    We have more security needs because we have a larger footprint, for example. We have long had APIs, but now we have massive API groups that are used and re-used, and while standard code scanning works with the source, the companies doing “API security” start with security of the interface, not the code. That is slowly changing, but API management is also impacted, and organizations that determined they didn’t need API management because adjacent markets were serving enough of their needs are slowly discovering that with proliferation comes the need to look at API management again.

    I stay up to date on most new technology because, as an industry analyst, what is coming can inform my discussions of what a given technology space looks like today. But even for me—someone who is not spending a huge chunk of my day coding or working on the toolchain or perfecting the new deployment platform—it can be hard to stay abreast of everything. While one of the foundational tenets of DevOps was no specialization and a shared workload, I think everyone now agrees with what a few of us were saying from day one: Some specialized skills are still needed, except in corner cases. But we certainly are sharing workloads more broadly and far more responsive, so DevOps is absolutely a massive success. Site reliability engineering (SRE) extends that success but focuses more on Ops. I’m really not a huge fan. I like that DevOps wanted to merge the processes and make the management of applications more holistic; I don’t think re-dividing it is the best idea out there. Like every title, “SRE” is interpreted pretty broadly. Some roles really involve DevOps (but SRE sounds cooler), and other organizations are just replacing ‘developer’ with DevOps and ‘operations’ with SRE. However, most are using SRE to manage the entire environment in which the app runs.

    Where are we going? Constant change. That’s the answer. I’ve waited for the breather for a couple of decades, but since VMs were popularized, it has been non-stop and will continue to be.

    This means a lot to you, to your viability as an employee, and to hiring. With the glut of people on the market, we’re seeing a resurgence of “must know every tool in our toolchain” type position postings. Of course, you must not, in a world where the combination of languages, frameworks, CI/CD tools, alerting, and monitoring tools would make a great college-level combinatorics equation. You add value by being adaptable. Both to your current employer and to future ones. The key thing you should seek to demonstrate is your ability to pick up and master new technologies when required. Do it on your current job to show worth, and keep a record of it for your next job. I’ve been a hiring manager off and on forever, and when the market is flooded with good candidates, that is the type of differentiator that gets the team’s attention. Do it; make it something worth doing and document that you did it. You won’t regret it.

    A group of us were helping a recent grad figure out what to do with their career a couple weeks ago, and the consensus was “avoid AI until it shakes out.” This is not the first time I’ve heard that advice—it’s not even the first group I was a part of giving that advice. And I agree in the general sense, but certainly in places where it is being implemented by all vendors in the space and it adds value, you do need to be aware of how it works and what you can do with it, and a project is one way to get that knowledge.

    It is still a rolling opportunity, and you are still keeping the lights on. I’ve seen people who worry they’re not doing enough. Any employer worth working for wants you to keep the systems running first, and everything else after—so if you’re doing that, you’re doing enough. Now go beyond. And keep rocking it. This new technology hasn’t implemented itself, and it isn’t self-maintaining (yet) … That’s all you.

  • Survey Sees Cloud Developer Environment Adoption Gaining Momentum

    Survey Sees Cloud Developer Environment Adoption Gaining Momentum

    A global survey found that while 95% of developers and business leaders are familiar with cloud development environments (CDEs), the reasons for adopting one vary widely. The survey polled 223 developers and business leaders working for organizations with more than 2,000 workers and 250 developers.

    Conducted by Coder, a provider of a CDE, the survey, for example, found that while 15% cited optimizing the developer environment setup process as a reason for adopting a CDE, another 14% cited making management tasks such as onboarding or improving remote access easier.

    Coder CEO Rob Whiteley said while CDEs clearly improve the productivity of application development teams and improve security by reducing the amount of source code running on notebook and desktop workstations, organizations looking to adopt CDEs need a significant amount of cultural resistance. Application developers have historically set up their own development environments on their own local machines. That approach, however, means that in addition to making it more challenging to collaborate also requires organizations to spend significant time and effort locking down each endpoint being employed by their developers, noted Whiteley.

    In fact, those issues are why many platform engineering teams formed to centralize the management of DevOps workflows are implementing CDEs, he added. The ultimate goal is to establish a set of guardrails while still enabling developers to self-service their tooling requirements, noted Whiteley.

    In fact, many DevOps teams have already built a homegrown version of a CDE that they currently maintain. Coder and other providers of CDEs are making a case for using a platform supported by a vendor rather than devoting resources to supporting a custom CDE, said Whiteley.

    As DevOps workflows continue to evolve in the age of artificial intelligence (AI), it’s only a matter of time before more organizations embrace CDEs, he added. The sheer volume of data that developers will be working with will make it impractical to continue to use local machines to build applications, said Whiteley.

    Regardless of the motivation for adopting CDEs, significant cultural changes are coming to DevOps workflows that need to evolve as application development continues to become more distributed. In the post-COVID-19 era, it’s now not uncommon for application development teams to consist of IT professionals working in different time zones. That creates a pressing need to leverage cloud services to provide the level of collaboration those teams need to achieve.

    It’s not clear how many CDEs have been deployed, but as the pace at which applications are being built and deployed continues to accelerate, the days when organizations could allow each developer to maintain their own development environment are coming to an end. Organizations need to determine as quickly as possible whether the code being created will actually run in a production environment. The more time spent trying to reconcile differences between development environments and the production environment where the application runs, the longer it will take to complete any given project.

    Unfortunately, time is the one thing that most DevOps teams no longer have the luxury of spending on application development initiatives that, in addition to multiplying, are also becoming much more complex to manage with each passing day.

  • Best Practices for Configuring MySQL Replication

    Best Practices for Configuring MySQL Replication

    MySQL provides several replication configuration options. However, ensuring it is done correctly may take time and effort, with considerable choices. Replication is a crucial initial step to enhance availability in MySQL databases. A properly designed replication architecture can significantly impact the accessibility of your data and prevent potential management complications. This article will delve into best practices for replication, covering local and long-distance scenarios.

    Active and Passive Setups

    When using active/passive mode for replication in MySQL, one server acts as the source while others serve as read-only replicas. The replicas can handle read-only queries, but all writes must be directed to the source. This configuration distributes the load across all replicas. Still, in the default asynchronous replication mode, there might be a delay between when data is written to the source and when it’s available on the replica. Keep this in mind while designing your application.

    Alternatively, the active/active configuration allows multiple servers to handle both read and write operations. While it may seem beneficial to have two servers for write requests, each server still needs to process the query workload of the other, making write distribution less effective. Additionally, MySQL lacks native conflict resolution logic, so conflicts can quickly arise, making it challenging to determine the true source of data without significant data loss when a rebuild is necessary.

    I recommend using the active/passive configuration for replication and sharding if you require higher throughput from your database.

    Enable GTIDs

    Global transaction identifiers (GTIDs) are unique IDs attached to transactions in a replicated environment. By default, replicas track processed records in the binary log file of a source database. However, this method is fragile if the source crashes and logs need to be restored.

    With GTIDs enabled, each transaction receives a UUID-based ID, allowing replicas to determine if a transaction has been processed. The GTID is a UUID representing the source server, followed by an auto-incrementing ID (e.g., 14a54b2f-2ad0-43b6-b803-72b5d7151d3b:1). As transactions occur on the source, the UUID remains the same while the ID increases. When a replica processes a transaction, the GTID or GTID range (e.g., 14a54b2f-2ad0-43b6-b803-72b5d7151d3b:1-10) is stored in the gtid_executed table.

    This process significantly reduces the risk of data becoming unsynchronized due to unprocessed or excessively processed transactions.

    Replication Modes for Optimal Configuration

    MySQL offers two replication modes that behave differently and should be understood to ensure optimal configuration for your environment. By default, MySQL employs asynchronous replication, where transactions are sent to the source, read by replicas and processed independently without source validation.

    Alternatively, MySQL supports semi-synchronous replication, enabled as a plugin. In this mode, transactions are received by the source and processed by replicas, but the source waits for at least one replica to accept the transaction before responding. Semi-synchronous replication provides greater data consistency, though it introduces some overhead in response time.

    With semi-sync mode, there are additional options that can be adjusted. By default, the primary server waits 10 seconds for a replica with semi-sync mode enabled to acknowledge the transaction. If data consistency relies on semi-sync, consider increasing this timeout value to ensure guaranteed consistency.

    It’s important to note that these modes can be mixed and matched. For instance, you can configure one replica with semi-sync and another without, ensuring that one specific server always has an up-to-date copy of your database while other replicas catch up at their own pace. This approach aids disaster recovery by easily identifying the best candidate for recovery.

    Understanding the distinctions between these modes will help you make informed decisions that align with your business needs.

    Optimize Replica Instances With Separate Log Storage

    Replica instances in your environment rely on the binary logs of the source for accurate information, as mentioned earlier. These logs are stored on the same disk as the database by default. However, this setup can overwhelm busy databases due to high throughput, including database manipulation and reading binary logs for replication. To address this, storing the binary logs on a separate disk from the database is recommended.

    This approach offers performance benefits and cost savings in cloud environments where free volumes have strict IOPS limits.

    Monitor Replication and Metrics in Managed Clusters

    Monitoring is crucial for all infrastructure, including replication. With monitoring, it is possible to know if data is being replicated after configuration. SolarWinds database performance (formerly VividCortex) and Prometheus are popular solutions for monitoring replication and other metrics in managed clusters. Regardless of the chosen solution, it is essential to promptly notify the appropriate individuals when issues arise and address them before they escalate.

    Create a Failover Plan for Minimizing Disruptions

    Inevitable software or hardware failures necessitate failover planning to minimize potential disruptions. Utilizing replication offers resilience by ensuring multiple servers house your online data simultaneously. Therefore, your team must possess a well-prepared strategy should the primary data source fail. The following example outlines an unplanned failover process:

    ● Ensure the downed source remains offline to avoid unexpected replication issues.
    ● Identify and remove the read_only option from the chosen replica that will serve as the new data source. If using semi-sync, select the replica previously configured with the plugin and source.
    ● Update your application to direct queries to the newly promoted source.
    ● Modify the remaining replicas to start replicating from the new source.

    Optimize Cross-Region Replication

    Cloud providers typically provide services across multiple geographic regions comprising several data centers. These data centers, known as availability zones (AZs), are strategically located within the same region but far enough to withstand disasters. Replicating databases to different physical locations is recommended but requires careful consideration. Replication introduces additional latency due to the distance the data needs to travel. Fortunately, latency between AZs is generally low, with AWS claiming single-digit millisecond latency within the same region.

    In contrast, there is significant latency between regions. For example, at the time of writing, the latency between us-east-1 and us-west-1 was reported to be over 60ms by Cloudping.co. Replication across regions results in replication lag, which is the time difference between data being written to the source and being written to a replica. This lag is even more pronounced when replicating across longer distances.

    Asynchronous mode should be used for replication across regions to avoid unnecessary delays for applications making requests.

    A comprehensive understanding of replication configuration options is crucial. However, it is equally important to determine the optimal configuration for these options. This can significantly impact your MySQL cluster’s operational efficiency and data-serving capabilities.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Indeed.com

    Western Climate Initiative
    Sacramento, California
    DevOps Engineering Manager
    $175,000 to $200,000

    CareerBuilder.com

    Leidos
    Herndon, Virginia
    Senior DevOps Engineer
    $122,200 to $220,900

    SimplyHired.com

    Hallmark
    Remote
    DevOps Engineering Manager
    $120,000 to $160,000

    Dice.com

    LTIMindtree
    Atlanta, Georgia
    Azure DevOps Lead
    $100,000 to $130,000

    LinkedIn

    Nintendo
    Redmond, Washington
    Associate Engineer – DevOps
    $84,800 to $127,200

  • OpenTelemetry Project Maintainers Add Code Profiling Capabilities

    OpenTelemetry Project Maintainers Add Code Profiling Capabilities

    This week at the KubeCon + CloudNativeCon Europe conference, the maintainers of the open source OpenTelemetry observability agent software project being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) revealed they have added profiling capabilities to enable DevOps teams to identify the root cause of issues down to a specific line of code.

    Austin Parker, director of open source for Honeycomb and a maintainer of OpenTelemetry, said while several providers of IT observability platforms have in the past year added code profiling capabilities, it makes more sense to provide this as a capability surfaced by OpenTelemetry in a way that any IT platform can easily consume.

    The overall goal is to reduce the amount of time it takes to troubleshoot IT issues by identifying not just the code at issue but also the development team responsible for it, noted Parker.

    OpenTelemetry defines a format that makes it simpler to centrally manage the collection metrics, logs and traces using open source software. In general, OpenTelemetry continues to gain traction as DevOps teams look to rationalize the amount of commercial agent software they need to deploy, update and manage.

    In effect, OpenTelemetry is aggregating agent software that, in terms of observability, provides no differentiated value, said Parker. That’s critical because as application environments become more complex, it’s all but impossible to manage them without being able to broadly apply the agent software required to instrument them.

    In the future, OpenTelemetry is expected to add additional capabilities such as data pre-processing, filtering or routing to redact sensitive data in a way that still fosters observability.

    Ultimately, as OpenTelemetry becomes more widely employed, it makes it significantly easier for developers to understand how their application is performing, noted Parker. Over time, armed with those insights, developers will build and deliver higher-quality applications, he added.

    It’s still early days as far as adoption of OpenTelemetry is concerned, but it’s clearly well on its way to becoming a de facto standard for enabling observability. The number of developers who have access to not just metrics and logs but also traces should steadily increase as the cost of instrumenting code declines.

    Of course, finding a way to collect that data inexpensively will also be crucial for applying artificial intelligence (AI) to DevOps workflows. After all, before an AI model can be trained, it needs access to verified data.

    In the meantime, DevOps teams might want to consider at what rate they can replace existing commercial agents deployed across the enterprise. Much of that agent software is more mature than OpenTelemetry, but over time, the economic arguments for using open source agent software become compelling. Many of the providers of observability platforms have already made support for OpenTelemerty available by default.

    Regardless of approach, the need for observability has always been apparent. The issue is that there is a world of difference between monitoring a pre-defined set of metrics and being able to launch queries to identify the root cause of an issue that metrics alone are not going to be able to surface.