devopsy.kinsta.cloud

Category: Platform Engineering

  • Is Backstage the Right Internal Developer Portal for You?

    Is Backstage the Right Internal Developer Portal for You?

    There has never been a better time to be a software developer. There is a language and framework to solve virtually any challenges we encounter. New tools, architectural patterns and methodologies like DevOps and Agile have changed how we work, architect our applications and organize ourselves. This has enabled teams to be nimbler than ever before. But this has all come with a hidden cost: Sprawling complexity.

    Sprawl is a complex web of disparate tools, languages and frameworks used across an organization’s projects. While this diversity allows for specialized solutions tailored to specific problems, it creates a labyrinth of interdependencies and varied skill requirements. This complexity often results in developers constantly navigating through a maze of tools, leading to inefficiencies and a steep learning curve for new team members. As organizations adopt more tools and technologies, the cohesive understanding of the entire software ecosystem diminishes. There are two areas in particular where this pain is felt acutely.

    The first area sprawl acutely impacts is the ability to find essential knowledge about systems, which becomes fragmented across tools and resources as sprawl increases. This fragmentation leads to longer incident response times and makes it more difficult for newly hired engineers to ramp up. Knowledge fragmentation also makes it considerably harder for enterprises to measure and manage initiatives to achieve high standards for security and reliability, as well as track migrations to new architectures like containers or clouds.

    Sprawl also impacts frequently occurring tasks for developers, such as managing “day two operations” like changing environment variables, adding secrets or modifying a replica count, or creating something new like a microservice or an environment. Given the complexity required to create and operate infrastructure, many organizations have invested in creating automations via infrastructure-as-code and CI/CD systems; however, these automations are often difficult for developers to discover and use. As a result, developers often file tickets with ops teams to complete these tasks, creating a process bottleneck that slows development velocity while burdening ops teams with mundane, repetitive tasks. This is commonly referred to as TicketOps.

    All of this has made the developer experience worse than ever. Developers report high levels of cognitive overload and burnout, which can cause unwanted developer attrition and reduced overall productivity. But it doesn’t have to be this way!

    Introducing Backstage

    In the early days of the cloud, only a handful of enterprises with lofty valuations could afford to solve these challenges by dedicating 50 – 100 engineers to build internal tooling designed to address the challenges of sprawl. Everyone else was left to suffer with subpar developer experiences or partial solutions like stale spreadsheets, manual processes and information that were often out of sync with the underlying system.

    Just as the pain was becoming unbearable for most enterprises, Spotify donated the tooling it created to address these challenges internally to the Cloud Native Computing Foundation. As a free, customizable, open-source project focused on these pain points, Backstage quickly captured the attention of enterprises worldwide.

    Over 2,000 enterprises have tried Backstage, making the project a tremendous success by promoting a vision for how internal developer portals can alleviate the pain of sprawl. However, a surprising number of enterprises struggle to adopt Backstage. In an interview with the head of engineering for Backstage, it was revealed many companies that have deployed Backstage have achieved only ~10% adoption rates, which is typically equivalent to a proof of concept. Understand why enterprises have struggled to succeed and decide if Backstage is right for you.

    Deciding if Backstage is Right for You

    Here are key questions to help guide your decision.

    Are You Ready to Build a New Product?
    Backstage is a framework to build an internal developer portal, but it is not an off-the-shelf tool or a solution. Many adopters have not understood what this means, at least at first.

    As a framework to build a tool, Backstage requires dedicated product and engineering resources to configure, customize and maintain. It is not uncommon for organizations to require four to six or more fully dedicated resources to succeed with Backstage. The engineers you dedicate to building this framework into a product will need to be skilled in TypeScript and React, and willing to work with an open source community and product.

    Getting up and running with Backstage involves more than 70 steps. It is not intuitive and can result in months of work before you can demonstrate value. Some of the steps and associated time commitments are obvious, such as deploying your instance of Backstage. Other steps may not be obvious to new adopters.

    For example, Backstage has historically relied on open-source plugins to enable organizations to connect their tools. The open-source nature of these plugins has been a key selling point of Backstage; however, the plugins have uneven quality and depth and can require additional customization to enable basic functionality like search. In another example, tuning the Backstage data model to your organization is so difficult that many adopters abandon hope and work with the default data model. Upgrading versions and plugins and the effort required to keep the data in Backstage synced with your system contribute to the engineering resource load required to succeed.

    The advantage of Backstage is the infinite customization you can accomplish thanks to its open-source code base. You can change anything you want if you write and maintain it. This includes adding custom data sources, customizing the data model, extending plugin functionality and having the UI reflect your corporate brand identity. Again, the trade-off is time and complexity relative to your alternatives. For instance, configure8 and Port offer the flexibility to tailor the portal’s data model to the contours of your organization and process any custom data source for use throughout the porta. Several vendors can support custom logos and color schemes in the portal’s UI (or you can build a more tailored UI on top of their APIs).

    Does it Support Your Scenarios of Interest?
    Internal developer portals as a product category have evolved well beyond their microservice catalog roots. If you are considering Backstage, it is important to ensure it offers functionality that addresses the needs of your enterprise’s stakeholders.

    For example, many organizations leverage functionality called Scorecards to define and measure teams’ compliance with reliability standards, security and architecture. Open source Backstage has a version of this functionality, but it is not intuitive and has a fairly high difficulty level (i.e., writing code, writing data collectors, etc.). The commercial vendor community typically all offer Scorecards. Adopters should consider the breadth and depth of information that can be checked, the ease of creating new scorecards, and the quality of the developer workflow and UI when evaluating Scorecards.

    In another example, Backstage enables users to scaffold new services via the Cookiecutter open-source package. Why take a principled dependency on a code generation solution instead of an open approach that enables you to use the best tool for the job? There is a robust landscape of available solutions to generate code and orchestrate workflows. Having a flexible way to directly integrate these reduces complexity.

    Backstage also offers functionality that enables developers to see the cloud costs of the code they operate. Upon closer inspection, adopters find this module only enables high-level aggregations of costs over a fixed timeframe. To truly provide developers with the knowledge and functionality they need to improve costs, consider closed-source vendors that offer more granular insights and actions, such as drilling in beyond aggregate costs at the microservice level to see the per resource cost drivers, an individual resource’s cost drivers, config settings and blast radius and weave in self-serve actions so developers can safely make changes where appropriate and permitted.

    The resource and environment catalog is another area to consider. Is it necessary to have the environments and resources that power them automatically mapped to each service? Do you want a centralized catalog of your resources across clouds and data centers that you can sort and filter using various keys? Backstage can support some of these scenarios, but not without significant work and at present, the connection to this data is brittle and manual versus automatic. This stands in contrast to certain closed-source vendors, such as configure8 which has extensive functionality.

    In a prior section, we discussed the importance of data model flexibility. This is an important evaluation criterion for organizations that want the flexibility to push in and present any data source, to customize the schema of the catalog, and finally to insert custom keys for results filtration and analysis (i.e., does this resource touch personally identifiable information; show me results related to a line of business, etc.). Ensure you understand whether you are poised to succeed in tailoring Backstage’s data model, which has stymied many adopters, and that any commercial vendor you evaluate can easily support customizations.

    Another consideration is Backstage’s TechDocs, and whether you want to adopt TechDocs for your entire enterprise or allow teams to use best-of-breed solutions. Whereas Backstage locks you into TechDocs (which can also make a future migration to a new portal solution more difficult once end-user behavior becomes entrenched), commercial vendors generally support open and flexible approaches that enable integration with any documentation solution.

    Also, ensure Backstage meets your information security needs. For example, most organizations desire a robust role-based access control model so developers only see permitted services, scorecards and actions in the UI, via search and the API. Backstage does not offer this functionality in the open-source edition. There is a framework to work with permissions included in Backstage but to get real value out of it, you need to budget to license a closed-source RBAC solution from Spotify or build it yourself.

    Finally, evaluate the pace of innovation you desire from your internal developer solution. Closed source vendors, two in particular, have demonstrated an extremely rapid pace of innovation, launching many industry firsts. Compare this to the pace of innovation in the open source community as internal developer portals are sticky once installed and tend to be long-term relationships.

    Are You Prepared for Success?
    Achieving broad internal adoption of an internal developer portal requires more than getting a portal configured, it requires solving the pain points that matter to your organization, demonstrating success, making the organization aware of that success, and creating a feedback loop to overcome challenges. To accomplish this, you must be aware of best practices, create content and forums to engage the organization, and ultimately adopt a product-oriented approach to ensure your platform engineering effort succeeds.

    Success in this area often combines budgeting and access to best practices.

    The Backstage community is vibrant and active, and you can pick up best practices by engaging with community members. But for enterprises and internal champions who need to succeed in driving the adoption of a new internal developer portal, you may wish to consider alternatives to community support.

    This can be accomplished via multiple methods. Consulting firms and individual practitioners specialize in this area and can coach you. Organizations above a certain size often invest in dedicated product management resources to own this function. Finally, certain vendors like configure8 focus on delivering success versus a tool (or framework to build a tool like Backstage) by providing consultative best practices, content and other value-adds packaged with their software to ensure you achieve robust adoption levels.

    Finally, to the extent your scenarios of interest involve self-serve actions to alleviate TicketOps, ensure that you have budgeted the appropriate engineering investment to develop the golden paths or blueprints that are vended inside an internal developer portal. Certain vendors like configure8 offer a solutions layer that creates custom golden paths for you. However you procure golden paths, you must balance your investment to ensure you get a great “razor” and all of the “razor blades” you will need to support the unique needs of your various internal customers.

    Do You Understand Your Total Cost of Ownership and Alternatives?
    One of the considerations of Backstage is understanding its total cost of ownership versus its best alternative.

    While Backstage is open source and free, we have noted that a meaningful engineering investment is required for installation, customization, maintenance, internal support and product management. Quantify the fully loaded cost (i.e., salary, taxes, benefits, facility charges, etc.) of each human resource required to succeed with Backstage.

    We have noted there may be licensing costs from Spotify for any essential features you do not wish to build yourself. There will be internal hosting costs and budgeting for any consulting costs.

    Several enterprises who have evaluated Backstage now realize the total cost of ownership of alternative vendors is much lower and more predictable. Leading vendors offer robust alternatives for catalog and scorecarding, self-serve actions, and collaborative cost management, and two vendors have evolved to provide compelling functionality and high degrees of flexibility. The key is finding a vendor who balances overall functionality and flexibility with ease of use, speed to value, and enterprise-grade support. This is where configure8 has struck a unique balance in addition to a lower cost of ownership relative to Backstage.

    Finally, any return on investment analysis must capture the opportunity costs. Here are some common examples:

    • Time to market trade-offs. Setting up Backstage for your proof of concept can take over a month, getting to a first releasable version can take 3-6 months, and full adoption can take 6-12 months. During this time, you may generate no or minimal business value. Specialist vendors can deliver success faster. The incremental business value you can generate with a specialist vendor versus Backstage related to ‘speed to value’ represents an opportunity cost you need to quantify.
    • Foregone features: Scenarios like Scorecards or RBAC that are important to your organization and you will have to pay for or forgo, you need to factor these incremental costs (or the cost of not having these features) into your analysis.
    • Opportunity cost of internal engineering resources: How many engineers are required to support Backstage over and above what’s needed to succeed with a specialist vendor? What business value could these incremental engineers have generated by instead supporting the unique value your enterprise brings to its customers versus an internal tool in a world of compelling alternatives? This becomes another cost of owning Backstage you need to compare v. the cost of buying a solution from a vendor.

    Understanding the direct and opportunity costs of your best alternatives will help you put your budget to work in the most efficient way and aid in your internal discussions with finance and other stakeholders.

    Carefully Consider Any Clones

    Given the initial popularity of Backstage, it is not surprising to see clones emerge built on Backstage. A clone is a vendor who has taken Backstage source code and added some feature crust on top, often a security wrapper, hosting and maintenance, light UI customizations, and potentially filling in a functionality gap like Scorecards. This approach can alleviate some of the burdens of open source Backstage in exchange for licensing costs you incur, it is important to understand if the clone is the best fit for your unique needs given the highly competitive vendor landscape that is now available. An easy way to think about it might be comparing Jenkins versus hosted Jenkins versus Jenkins with a vendor security wrapper relative to what enterprises get with Azure DevOps and GitHub Actions.

    Final Thoughts

    Building an internal developer portal from an open-source framework can be rewarding for organizations that can achieve value from extensive customization that is impossible elsewhere. For many organizations, however, it is important to properly scope the effort required to succeed with Backstage and compare it to your alternatives in the vendor community. While Backstage does offer certain advantages, DevEx and platform teams are increasingly finding greater success more quickly and at a lower total cost of ownership via specialist vendors like configure8.

  • Survey Sees Cloud Developer Environment Adoption Gaining Momentum

    Survey Sees Cloud Developer Environment Adoption Gaining Momentum

    A global survey found that while 95% of developers and business leaders are familiar with cloud development environments (CDEs), the reasons for adopting one vary widely. The survey polled 223 developers and business leaders working for organizations with more than 2,000 workers and 250 developers.

    Conducted by Coder, a provider of a CDE, the survey, for example, found that while 15% cited optimizing the developer environment setup process as a reason for adopting a CDE, another 14% cited making management tasks such as onboarding or improving remote access easier.

    Coder CEO Rob Whiteley said while CDEs clearly improve the productivity of application development teams and improve security by reducing the amount of source code running on notebook and desktop workstations, organizations looking to adopt CDEs need a significant amount of cultural resistance. Application developers have historically set up their own development environments on their own local machines. That approach, however, means that in addition to making it more challenging to collaborate also requires organizations to spend significant time and effort locking down each endpoint being employed by their developers, noted Whiteley.

    In fact, those issues are why many platform engineering teams formed to centralize the management of DevOps workflows are implementing CDEs, he added. The ultimate goal is to establish a set of guardrails while still enabling developers to self-service their tooling requirements, noted Whiteley.

    In fact, many DevOps teams have already built a homegrown version of a CDE that they currently maintain. Coder and other providers of CDEs are making a case for using a platform supported by a vendor rather than devoting resources to supporting a custom CDE, said Whiteley.

    As DevOps workflows continue to evolve in the age of artificial intelligence (AI), it’s only a matter of time before more organizations embrace CDEs, he added. The sheer volume of data that developers will be working with will make it impractical to continue to use local machines to build applications, said Whiteley.

    Regardless of the motivation for adopting CDEs, significant cultural changes are coming to DevOps workflows that need to evolve as application development continues to become more distributed. In the post-COVID-19 era, it’s now not uncommon for application development teams to consist of IT professionals working in different time zones. That creates a pressing need to leverage cloud services to provide the level of collaboration those teams need to achieve.

    It’s not clear how many CDEs have been deployed, but as the pace at which applications are being built and deployed continues to accelerate, the days when organizations could allow each developer to maintain their own development environment are coming to an end. Organizations need to determine as quickly as possible whether the code being created will actually run in a production environment. The more time spent trying to reconcile differences between development environments and the production environment where the application runs, the longer it will take to complete any given project.

    Unfortunately, time is the one thing that most DevOps teams no longer have the luxury of spending on application development initiatives that, in addition to multiplying, are also becoming much more complex to manage with each passing day.

  • How Generative AI Enables Unified Continuous Testing Platforms

    How Generative AI Enables Unified Continuous Testing Platforms

    It is time for a unified continuous testing platform.

    Existing software testing platforms and tools specialize in testing activities for a subset of the different stages of value streams. The landscape of software testing tools includes a wide range of solutions, each with its strengths, focusing on specific aspects of software development, delivery and operations.

    For instance:
    • Unit Testing Tools focus on the earliest stages of development, allowing developers to test individual units of code for correctness.
    • Integration Testing Tools aim to test the interactions between different modules or services within an application.
    • System Testing Tools are designed for end-to-end testing of the complete system before it goes live.
    • Performance Testing Tools assess the application’s behavior under load and stress conditions.
    • Security Testing Tools focus on identifying vulnerabilities within the application.
    • User Acceptance Testing (UAT) Tools facilitate the final testing phase, where end-users validate the solution against their requirements.

    The challenge with unifying these diverse testing needs into a single continuous testing platform is multifaceted. Such a platform needs to seamlessly integrate with a wide variety of development tools and environments, support different testing methodologies, and be flexible enough to adapt to different organizational processes and quality standards.

    While there are Continuous Integration/Continuous Deployment (CI/CD) tools that incorporate multiple testing stages, they often do so by integrating with specialized testing tools rather than offering a unified testing solution themselves. These CI/CD tools are closer to orchestrating the various testing activities rather than unifying them under a single platform’s capabilities.

    However, the concept of a fully unified continuous testing platform, covering every stage from requirements through to deployment and in-production testing, represents a significant opportunity for innovation in the software development tools space. Achieving this would likely require leveraging advancements in areas like generative AI, as discussed in my prior blog Applying AI/ML to Continuous Testing to create adaptable, intelligent testing processes that can cover the full spectrum of testing needs in a cohesive manner.

    Benefits of a Unified Continuous Testing Platform

    Improved Efficiency and Speed: A unified CT platform integrates testing across all stages of the software development lifecycle (SDLC), significantly streamlining the testing process. This integration facilitates automated, continuous feedback loops that promptly identify and address defects, allowing development teams to quickly iterate and improve. As a result, teams can release new features and fixes more rapidly, accelerating the product’s time to market and enhancing responsiveness to customer needs and market changes.

    Enhanced Quality and Reliability: By ensuring comprehensive and consistent testing across the entire application, a unified CT platform plays a crucial role in identifying and mitigating issues early in the development cycle. This early detection helps maintain a high level of software quality and reliability, leading to increased user satisfaction and trust in the product. The consistent application of quality standards across all testing phases contributes to a robust and dependable software product.

    Cost Savings and Reduced Maintenance: Automating the testing process through a unified CT platform not only optimizes resource utilization but also significantly reduces the costs associated with manual testing and late-stage defect remediation. Early defect detection translates to lower fix costs, and the streamlined process decreases the overall time-to-market. Additionally, high-quality, reliable software requires less maintenance, further reducing long-term costs and freeing up resources for innovation and development efforts.

    Improved Collaboration Across Teams: A unified CT platform fosters a culture of collaboration and transparency among development, testing, and operations teams. By providing a common framework and tools for all testing activities, it breaks down silos and enables seamless communication and collaboration across the SDLC. This improved collaboration ensures that teams are aligned with the project goals, can share insights more effectively, and work together to identify and solve problems more efficiently, leading to better outcomes.

    Together, these benefits highlight how a unified Continuous Testing platform can transform the software development process, making it more efficient, cost-effective, and collaborative while ensuring the delivery of high-quality and reliable software products.

    While many tools address specific parts of the testing lifecycle, the vision of a single platform that seamlessly unifies all stages of testing from requirements to in-production remains an ambitious goal. It’s a gap in the current technology landscape that presents both a challenge and an opportunity for future development.

    Challenges for Unified Continuous Testing Platforms

    Creating a continuous testing platform that unifies test activities for all stages of the end-to-end value stream is a complex challenge due to several factors:
    1. Diversity of Technologies and Tools: Modern software development environments are highly diverse, incorporating various programming languages, frameworks, and technologies. Creating a platform that seamlessly integrates with all these technologies is challenging.
    2. Complex Integration Points: Continuous testing needs to integrate with multiple stages of the development pipeline, including development, deployment, and operations. Each of these stages might use different tools and processes, making it hard to create a one-size-fits-all solution.
    3. Varying Quality Metrics: Different teams and projects may have different definitions of quality, success criteria, and performance metrics. A unified testing platform needs to be highly customizable to cater to these varying needs.
    4. Change Management: Adopting a new platform requires changes in the organization’s processes and workflows. Resistance to change is common in organizations, and transitioning to a new way of testing can be met with skepticism and inertia.
    5. Scalability and Performance: Ensuring the platform can scale to handle the testing needs of large organizations with thousands of tests running concurrently is a technical challenge. Performance issues can become a bottleneck, affecting the overall efficiency of the development process.
    6. Security and Compliance: Integrating testing across all stages of development also introduces security and compliance challenges. The platform must ensure that sensitive data is protected and that testing practices comply with regulatory requirements.
    7. Cost and Resource Constraints: Developing, maintaining, and supporting a unified continuous testing platform requires significant investment. Organizations might be hesitant to commit the necessary resources without clear evidence of return on investment.
    8. Evolution of Practices: Software development practices and tools are constantly evolving. Keeping the platform up-to-date with the latest practices and technologies requires ongoing effort and innovation.

    Despite these challenges, there is a growing recognition of the value of continuous testing throughout the software development lifecycle. Some companies and open-source communities are making strides in this direction, creating more integrated and flexible testing solutions. However, achieving a fully unified platform that addresses all these challenges is an ongoing effort and represents a significant opportunity for innovation in the software development and testing industry.

    Generative AI Can Facilitate a Unified Continuous Testing Platform

    Generative AI can play a significant role in overcoming the challenges associated with creating a continuous testing platform that unifies test activities for all stages of the end-to-end value stream. Here’s how generative AI might address each of the challenges:
    1. Diversity of Technologies and Tools: Generative AI can be trained on a wide range of programming languages, frameworks, and technologies to understand and generate code or testing scripts. This capability allows it to adapt to different environments and create testing materials that are compatible with various tools and technologies.
    2. Complex Integration Points: AI can analyze the workflow of development pipelines and suggest optimal integration points for testing. By learning from different CI/CD (Continuous Integration/Continuous Deployment) configurations, AI can recommend best practices for integrating testing seamlessly into existing workflows.
    3. Varying Quality Metrics: Generative AI models can be customized to understand and apply different quality metrics and success criteria based on project-specific requirements. By training on diverse datasets, these models can adapt to various definitions of quality and generate relevant tests or analysis.
    4. Change Management: AI can assist in the change management process by simulating the outcomes of adopting new testing platforms, thereby providing evidence-based benefits and mitigating resistance to change. Furthermore, AI-driven analytics can highlight the efficiency gains and quality improvements to support the transition.
    5. Scalability and Performance: Generative AI can optimize testing processes by identifying redundancies and suggesting improvements, thus enhancing performance. Additionally, AI can dynamically allocate resources based on testing needs, ensuring scalability without compromising efficiency.
    6. Security and Compliance: AI models can be trained to identify and flag potential security and compliance issues in the testing process. By continuously learning from the latest security standards and compliance regulations, AI can help ensure that testing practices meet the necessary requirements.
    7. Cost and Resource Constraints: By automating the generation and optimization of test cases, generative AI can significantly reduce the manual effort required, lowering costs and resource demands. AI can also help prioritize testing efforts based on risk assessment, ensuring that resources are focused where they are most needed.
    8. Evolution of Practices: Generative AI models are inherently adaptable and can continuously learn from new development practices, tools, and technologies. This ensures that the testing platform remains up-to-date with the latest advancements in software development.

    Generative AI has the potential to transform continuous testing by providing adaptive, efficient, and intelligent solutions to the complex challenges of unifying test activities across the end-to-end value stream. However, realizing this potential requires careful design, extensive training of AI models, and ongoing management to ensure that the AI systems remain effective and aligned with evolving testing needs.

    Summary: Call to Action

    The need for a unified Continuous Testing (CT) Platform has never been more urgent. Current testing platforms, each adept in their niche, cover only fragments of the software development lifecycle (SDLC), leading to a disjointed and inefficient testing process. This fragmentation not only slows down development and deliveries but also compromises the quality and security of the final product. The dream of a single platform that seamlessly integrates all stages of testing, from requirements through to deployment and in-production testing, represents a monumental leap toward efficiency, security, and quality in software development.

    The challenges in creating such a platform are manifold, ranging from the diversity of technologies and tools to the evolving nature of software development practices. Each challenge, from integrating diverse technologies and managing change within organizations to ensuring scalability and compliance, adds complexity to the development of a unified CT platform. Yet, the potential benefits of overcoming these hurdles are immense, promising a significant boost in the speed and quality of software delivery. The industry’s recognition of these benefits is growing, evidenced by the efforts of some companies and open-source communities moving towards more integrated and flexible testing solutions.

    Generative AI emerges as a beacon of hope in this quest, offering innovative solutions to the multifaceted challenges of unifying test activities. By harnessing the power of generative AI, the industry can address the diversity of tools, integrate complex testing stages, adapt to varying quality metrics, manage organizational change, scale efficiently, ensure security and compliance, and evolve with the practices of software development. The path forward requires a concerted effort, but investments in AI-driven testing innovations can realize the vision of a comprehensive, unified, continuous testing platform. This is not just an opportunity for enhancement but a call to action for the industry to redefine the future of engineering platforms.

  • Platform Engineering: The 2024 Game-Changer in Tech

    Platform Engineering: The 2024 Game-Changer in Tech

    Platform engineering, a key driver in the ever-evolving technological landscape, is poised for significant momentum in 2024. This article explores the reasons behind its rising popularity, the types of organizations that stand to benefit, potential drawbacks and the crucial role of test data management (TDM) in this context.

    What’s Driving Demand for Platform Engineering?

    Platform engineering is anticipated to gain significant momentum in 2024 as it offers crucial advantages in accelerating business value, reducing cognitive loads and enhancing the efficiency of application development and management processes. According to Gartner, by 2026, about 80% of software engineering organizations will establish platform teams as internal providers of reusable services, components and tools for application delivery. This trend underscores the increasing adoption and importance of platform engineering in the tech world.

    The growing popularity of platform engineering can be attributed to several factors:

    • Developer productivity: Enhancing developer productivity is a key driver, accounting for approximately 21% of the reasons for its popularity.
    • CI/CD pipeline implementation: About 20% of the popularity is driven by the need for efficient continuous integration and continuous delivery pipelines.
    • Standardization of tools and processes: Another 20% comes from the necessity to standardize tools and processes.
    • Security enhancements: Security improvements contribute 20% to its popularity.
    • Infrastructure-as-code (IaC): Finally, 19% is driven by the adoption of infrastructure-as-code (IaC) methodologies.

    In addition, platform engineering addresses the inefficiencies caused by the decentralization of tools and processes in DevOps, especially as organizations scale. This approach provides a more organized and centralized framework, which is essential in managing the complexities of modern software development environments.

    As highlighted by Gartner, platform engineering’s potential to offer a frictionless self-service experience to customers and business partners is a significant factor in its rising popularity. This adaptability is crucial in an era where technology and market demands are rapidly evolving, requiring businesses to remain agile and competitive.

    These factors collectively make platform engineering a strategic approach for software engineering organizations, particularly looking forward to the year 2024 and beyond.

    platform engineering, TDM

    Key Benefits of Platform Engineering

    While it isn’t a fit for every organization, there are some shared attributes of companies that are most likely to benefit from it.

    If your organization is seeking to:

    • Reduce tool sprawl by standardizing tools and services
    • Improve productivity through automation, leading to faster development cycles
    • Establish governance frameworks for software development
    • Increase scalability through technologies like containerization and orchestration
    • Enhance security by integrating automated security scanning tools into the CI/CD pipeline,

    then platform engineering may be right for you. These benefits are especially relevant for businesses facing complex software development cycles and those in need of robust, scalable, and secure solutions.

    Drawbacks of Platform Engineering

    However, platform engineering is not without its challenges. Increased complexity in infrastructure can be daunting for small-scale teams. High dependency on the platform may lead to difficulties in switching technologies or providers. Additionally, compatibility issues with existing development tools and technologies can slow down processes.

    The Crucial Role of Test Data Management (TDM)

    In a platform engineering environment, test data management (TDM) becomes a pivotal component. TDM’s importance lies in its role in agile development, ensuring test accuracy and reliability, thus enhancing the effectiveness of the testing process. Agile methodologies, which divide projects into smaller sprints, rely heavily on TDM for providing high-quality, compliant test data in a continuous integration and deployment model.

    TDM addresses the challenge of managing data fragmented across various applications, environments and compliance levels. An entity-based TDM approach enables testers to provision test data on demand, streamlining the testing process and reducing the risk of data breaches. Ultimately, TDM practices significantly reduce time, cost, and risk in software delivery, making it a vital component in modern platform engineering strategies.

    As technology continues to evolve rapidly, the role of platform engineering, bolstered by effective TDM, becomes increasingly crucial. Companies that do not adapt to these changes by integrating components like TDM into their platform engineering strategies will risk falling behind in the competitive tech market. Therefore, embracing these innovations is essential for survival and success in today’s fast-paced technological landscape.

  • The Age of Platform Engineering: What to Expect in 2024

    The Age of Platform Engineering: What to Expect in 2024

    Platform engineering has been gaining traction over the last few years, with some likening it to the next iteration of DevOps. Gartner has called it one of the top strategic trends of 2024 – and has predicted that 80% of large software engineering organizations will establish platform engineering teams by 2026. But what’s driving the momentum?

    I believe it’s a result of maturity in the adoption of the cloud by enterprises of all sizes and shapes. As more and more enterprises started using the cloud and embraced DevOps as a path towards quickly taking features to the market, there was a clear need to rethink how the underlying infrastructure and platforms were used for enterprise applications. Platform engineering emerged as the new path to building and managing the underlying application infrastructure.

    It’s been a growth year for platform engineering, but in today’s era of hybrid-cloud-meets-generative-AI, we see new trends emerging in this space. So, what’s next? Here is what we expect to see as we head into 2024.

    Product-Centric Funding Model

    Traditionally, IT was seen as a gatekeeper and cost center. With the cloud, developers gained more leverage in procuring the infrastructure they wanted for their applications. Enterprises realized the value of modernizing their IT operations to meet this modern need and wanted to transform their cloud operations to be part of the innovation team. They see platform engineering as the transformational approach that aligns cloud operations with developers and business users. However, platform engineering requires a product-centric funding model instead of project-centric models in the past, as a product-centric model doesn’t have a start and end date like projects and aligns closely with the product lifecycle with ongoing funding. Enterprises need to recognize this shift from the status quo widely, and we expect this realization to come for businesses in 2024.

    Platform Engineering as an Extension to DevOps

    As enterprises embraced DevOps to accelerate their application development, they still used the traditional gatekeeping models with cloud operations. The embrace of traditional guardrails in the cloud created friction and slowed innovation. With its “as code” model, platform engineering is well suited for reducing friction and accelerating innovation. We expect organizations to embrace platform engineering as the natural extension of DevOps to operate at the speed of business.

    Generative AI Capabilities

    Generative AI is at the peak of the hype cycle as we close out 2023, and in 2024, we will see some enterprise-ready use cases come to life. Generative AI will augment platform engineering with code assistants to create infrastructure-as-code, governance-as-code and security-as-code to ensure smooth Day 2 operations. We will also see chatbots leveraging generative AI to help SREs and other platform engineering pros with documentation, infrastructure playbooks and other relevant use cases.

    2024 is a critical year for organizations as they fend off the economic forces and embrace technological advancements to compete in the global market. Most CXOs will see platform engineering as the top strategic trend to focus on because it brings productivity and cost efficiencies to their organization. This is only the beginning, and the coming decade will be defined by platform engineering, laying a firm foundation for AI-driven innovation in enterprises.

  • AWS re:Invent 2023: Cloud Development Environments

    AWS re:Invent 2023: Cloud Development Environments

    Speaker 1: This is Techstrong tv.

    Shira Rubinoff: My name is Shira Rubinoff. I’m the chief cybersecurity officer with the Techstrong Group. I’m here with Rob Whiteley, CEO of Coder. Rob, it’s a pleasure to be with you here today.

    Rob Whiteley: Yes, thank you so much for hosting me. This is great.

    Shira Rubinoff: Rob, we’ve had some great conversations, and I’d love you to share with the audience who is Coder, what is your story? Please explain who you are, how you came about. Let’s go from there.

    Rob Whiteley: Yeah, I hope you have about two hours. No, I’m just kidding. I’ll give you the short version.

    Shira Rubinoff: Sure.

    Rob Whiteley: Coder is a cloud development environment solution or CDE is sort of the industry terminology. What that basically means is we’re trying to solve the problem of helping enterprises shift their development from local laptops to more of a cloud native where you’re asking developers to access the cloud and develop there.

    The idea is developing locally on laptops is expensive, it’s error-prone, and you often end up with the, “Well, it worked fine on my machine, but then when I ship the code, it doesn’t work.” So we try to solve that. We’ve been around for about seven years, and we actually started life as an open source project called Code Server, which is an IDE. We’ve since grown to have a more complete solution in this CDE space.

    Shira Rubinoff: Well, that’s wonderful and certainly very needed. Let’s dive into the security benefits of this because there are many and much are needed. I’d love to for your audience to understand that.

    Rob Whiteley: Yeah. Yeah. No, I’m super excited to be here because I think it’s an underappreciated element, right? So to me, if I step back, the reason why companies are investing in cloud development is for developer productivity, right? Only about a third of a developer’s time is actually spent coding. The remaining two-thirds is doing operational stuff, administrative. By going to the cloud, you get productivity gains. Great. That’s why I came.

    Why I stay is usually a security benefit. What I mean by that is so many of our customers have told us that, “Well, a developer left his laptop in a cab,” or, “She did not have access when she was on the road.” So by putting things in the cloud, it allows you to connect from any device, and it gets source code off of laptops. That’s now a major source of intellectual property for most companies. The applications run your business, and so to get source code off laptops is a huge benefit.

    That’s just the beginning. That’s usually what triggers. So for example, one of our large banks, it was actually the CISO that brought us in solely to solve that problem of getting source code off laptops.

    Shira Rubinoff: That’s a huge problem.

    Rob Whiteley: It is. I think that alone is worth it, but once you’re in the cloud, I have better access controls, I can do role-based access, I can be more thoughtful about encryption and how I secure it and whether I enforce certain forms of SSH. So there’s a lot of benefits, and I don’t think customers realize that until they’re underway. Part of what we’re hoping to do is front end that conversation a bit more. Security is actually a major driver on shifting to the cloud. It’s not just the productivity gains.

    Shira Rubinoff: Certainly. Let’s talk about the no extra steps that we were discussing earlier. The human factors piece of security is a massive piece that organizations and technology companies and all types of security, and certainly with coding there’s many different steps to do in order to get to what you want to do. Where there’s extra steps and people are in a rush, they don’t want to do that. They want to circumvent it, they want to break it, they want to leave it open so they don’t have to think about it, leave passwords around, all sorts of things. So please tell me what Coder does and why.

    Rob Whiteley: Yeah. No, it’s great. In fact, Coder’s co-founder Ammar, has a phrase that developers are inherently lazy. He doesn’t mean any disrespect by that. It’s just they’re-

    Shira Rubinoff: It’s everybody, right? It’s the multitasking. I wouldn’t just call it coders. Coders are fabulous.

    Rob Whiteley: But I think part of the coding ethos is do things as elegantly and simply as possible. That’s what makes good code. If you shift the security burden to the developer, like we in the industry love this concept of shift left, which basically means shift things earlier in the development lifecycle. What that really means is you’re asking the developer to do more, to take on more steps, to build security into his or her workflow. But that is against that ethos of being lazy and trying to do things efficiently.

    I think one of the things that we’ve noticed is, let’s flip that on its head a little bit. Let’s shift to the developer right. What if we could make the developer more productive and just make security seamless built into it? They’re not having to worry about it. How does that work? Well, if I’m logging into a cloud to do my development, I’m inheriting the entire security infrastructure I’ve already put in place for my cloud, my firewalling, my access controls, my encryption, my backups, everything that keeps that environment that I’ve probably already put in production. I now automatically shift those security controls to my development environment, which again, used to be laptops, and I was very dependent on making sure my antivirus or my mobile device management kept that device secure. My developer had to battle that. They had to toil, they had to update things, they had to make sure.

    We reduce that toil, but more importantly, we take the security off the plate, off the mind. If you access the cloud, it is more secure. Period. Full stop. So that’s what makes I think this work better as opposed to having to send developers through more security training. Yeah, you’ve got to write clean code and have fewer bugs, but that is what the developer should be focused on, not complying with your security policies, which is too much toil.

    Shira Rubinoff: Certainly. My background is heavy in security, and I will tell you the best technologies out there have the security built in and have the no extra steps. You want something that’s better, stronger, faster, without the onus being on the user, which is the developer or somebody who has to be responsible for it. You want to make sure they’re doing their job great and they’re doing their job well without having to burden them with thinking about those extra pieces.

    Just think about our daily lives. Moving a million miles an hour, and certainly the developers, that’s something they have to keep on top of mind, they have to be answering everything, running around and traveling, everything they’re doing. The last thing you want them to think about is, “Wait a second, did I do all these steps?” Because most of the time they haven’t. So that’s an excellent thing that you had certainly built in.

    In terms of the competitors out there that you see or you come face when you’re at the marketplace, what would you say your main differentiators are in the marketplace?

    Rob Whiteley: Yeah, sure. I think for us, there’s two primary ways these solutions get deployed. They’re self-hosted, which means you download the software, that’s Coder. So we start as open source, and you can upgrade to the commercial if you want, but you own the software, you deploy it. Or you can consume it as SaaS. The problem with this space in particular when deployed as SaaS, which is pretty much all of our competitors, is I now have a lot of very sensitive metadata about my developers. What environments are they using? What tools are they using? How often are they in those? What languages? What versions? That all could be used to exploit the application. So that metadata is now flowing out to some third party cloud, which by the way, I had to poke a hole in the firewall just to do, only for them to poke a hole right back into my environment in order to provision the developer workstation.

    For us, keeping it all hosted inside your cloud, whether it’s on-premise or public, we don’t care, is what has been the biggest inherent differentiation. I love SaaS. SaaS is the future, but in an early adopter market like this, there’s a clear leaning towards self-hosted. Most of these organizations can run software as well as a cloud provider anyway, and so they would rather just do it themselves. So that’s difference one.

    The second one I would just say is the majority of solutions in this market, and again I’ll call this the CDE or cloud development environment, is they’re designed with the operator in mind, the backend DevOps team that has to own the infrastructure. We’ve tried to approach it from the developer. What’s going to create the best developer experience? Because one of the things that you hinted at is if you make things difficult, it will not get adopted.

    Shira Rubinoff: Correct.

    Rob Whiteley: We don’t live in a world where we can force developers anymore. They pretty much have to opt in. So if it’s not a good experience, they just won’t adopt that structure.

    Shira Rubinoff: Well, that’s the mindset of developers, and it’s very good that you’re honed in on that. A lot of people are like, “Well, this is what we do for everybody else.” I like the fact that you’re really honed in to what do the developers do? How do they like to work? And work along the path of that. When they talk about security in general or how operating anything really works is think about your user base, work the way they work, and build the security around that.

    Rob Whiteley: Exactly.

    Shira Rubinoff: Don’t try to force them into something. So that’s key.

    Rob Whiteley: Yeah, exactly. So our mindset’s simple. If you make the developer experience clean and you take security friction out of it, they will adopt it. Then you finally benefit from all those advantages we talked about. The worst thing I think you could have is you deploy the solution and they just continue to work on their laptop as if nothing’s changed because they found the new thing too difficult.

    Shira Rubinoff: Right. True. I know you had some really interesting announcement that came out in September. I’d love you to reiterate that and tell our audience a little bit about that and also hint about some new things coming down the pike.

    Rob Whiteley: Yes. Yes. We’re very excited.

    Shira Rubinoff: If you could give us a little bit of a sneak peek here.

    Rob Whiteley: Yeah, back in September we announced something called the Coder Registry. Think of that as a marketplace for Coder plugins. The reason why that’s important is because we believe in developer choice. Developers should choose the tools that make them most productive. What we’ve done is we’ve created this registry so that anybody in the developer tooling ecosystem can write a plugin to Coder. If you want to be able to provision your tool to a developer … Because we sort of own that developer eyeball, if you will … We are the thing they log into to do their work so we can merchandise different tooling. If you want to use a code gen AI tool for assistance, we can make that. By doing these plugins, we’ll create a much more, I think, robust ecosystem where it is not on the developer to do that integration.

    We’re just a tile they can click on now to say, “I want that tool.” But we still give the platform team the chance to curate that. If they don’t want to introduce that tool, then they can just kind of exclude that from the developer’s pick list. That registry is what we announced. We’re building more and more third party support for that. That’s sort of the big thing. So that was big announcement.

    What I can say for next year that I think is going to be most exciting is we’re really trying to focus on how do we make these platform teams that are responsible for making developers productive, for helping secure source code, and really equip them with a much more advanced set of insights. One of the things that Coder does is we collect a lot of that metadata, and like I said, we don’t beacon it to the cloud. It stays local to your environment.

    Shira Rubinoff: That’s important.

    Rob Whiteley: Well, that’s a rich data set. There’s a lot we could do with that data set to help make operators’ lives easier. One of the things we’ll be looking at next year early is how do we expose that in a way where maybe you can figure out, “If I increased the amount of compute, what would that correlate to for an outcome?” By kind of front-ending that we can make it a lot easier for operators. Again, if we can remove friction from the developer experience, they will adopt it. So that’s part of what we’re looking at.

    Shira Rubinoff: Well, that’s very exciting. Everyone stay tuned for that announcement coming up. Anything else you want to add for our audience? Anything that they should take note of or any helpful hints around anything around the AWS Conference or your company itself that you’d like to share with us?

    Rob Whiteley: Yeah, I think the biggest thing would be … So the cloud development environment is new. It’s an emerging market. I think a lot of companies are not necessarily investigating it because they think it’s not right for them, or they don’t want to fit force developers to change behavior. I think what we’ve experienced is every customer that does it has quickly expanded. To me, my biggest call to action is just go look at it. I mean, it’s free. It’s open. You can get started. I’m not even asking you to purchase anything.

    But I think just shifting the behavior from local to cloud, getting those security benefits and demonstrating that win back to the business is the first step in I think what’ll be a sea change in arguably the last mile the cloud has not effectively touched yet, which is developers. Production environments have been in the cloud for a while now. Development environments are next. So I think now’s the time to get started.

    Shira Rubinoff: Well, that’s wonderful. Thank you so much for sharing your insight with our audience.