devopsy.kinsta.cloud

Tag: platform engineering

  • Red Hat Unveils Developer Hub to Boost Dev Productivity

    Red Hat Unveils Developer Hub to Boost Dev Productivity

    At its Red Hat Summit conference today, Red Hat unveiled its Red Hat Developer Hub, a developer portal that provides access to an opinionated set of development tools based on the open source Backstage project.

    The portal pulls together a set of curated tools and resources so that developers encounter less friction and avoid the complexity of numerous optional tools that span disparate vendors. The goal is to increase developer productivity while reducing the volume of optional tools and information that must be sifted through to actually deliver production code.

    Red Hat Developer Hub aims to streamline the development process and allow organizations to maximize existing resources and developer skills while reducing sprawl, said Chris Wright, CTO and SVP for global engineering at Red Hat. This will help organizations maintain application development velocity and drive a consistent user experience for developers based on tools and skills they’re already familiar with, Wright told attendees.

    Rather than building and maintaining their own internal developer portal, Red Hat is making a case for using Developer Hub to deliver a platform approach to addressing the complexity and sprawl DevOps teams face. Red Hat Developer Hub is, in essence, an opinionated and optimized version of Backstage that aims to deliver improved developer experiences across environments, including Kubernetes and container application platforms such as Red Hat OpenShift.

    Developer Hub will offer a single pane of glass to view all available developer tools and resources, developer self-service capabilities and security and compliance guardrails for cloud-native application development that comply with enterprise-wide best practices, according to the company.

    Additionally, Red Hat Developer Hub includes a set of ‘golden path’ templates—pre-defined, pre-architected and supported templates for further accelerating application development.

    The developer hub is based on the Cloud Native Computing Foundation (CNCF) incubating open source project Backstage, originally developed by Spotify and donated to the CNCF in 2020. Backstage is now being advanced by contributions from multiple vendors, including Red Hat.

    Backstage has been gaining traction in cloud-native application environments that are especially challenging to develop using multiple tools. There’s a lot more focus on developer productivity, which Backstage addresses by creating a catalog of blueprints that developers can readily consume rather than requiring them to build these capabilities themselves multiple times over. The goal is to create scaffolds that developers can consistently reuse across multiple application development projects.

    Red Hat has also developed Red Hat Plug-ins for Backstage, a bundle of six plug-ins that integrate various key systems into Backstage, helping to extend the project’s functionality, ensure a consistent user experience and improve workflow efficiency. The plugins can be used across all Backstage installations, providing even more flexibility for developers across platforms and Kubernetes distributions and include:

    • Application Topology for Kubernetes, which enables developers to visualize real-time status of application and infrastructure workloads deployed to any Kubernetes target, including Red Hat OpenShift, with greater consistency.
    • Multi-Cluster View with Open Cluster Manager (OCM), which provides a view into clusters from Open Cluster Manager’s MultiClusterHub and MultiCluster Engine in Backstage.
    • Container Image Registry for Quay, which improves the integration and speed of interactions with Quay registries by providing a view into container image details. This includes security vulnerabilities (CVEs) associated with deployed images.
    • Pipelines with Tekton, which gives users a view into the details of all Tekton pipeline runs and their respective statuses across all services.
    • Authentication and Authorization with Keycloak, which enables platform engineers to load users and groups from Keycloak into Backstage, allowing the use of multiple authentication providers to be applied to Backstage entities.
    • GitOps with Argo CD, which helps track the health and status of the Argo CD status for services within Backstage.

    Red Hat said additional plugins are in the works. Red Hat Developer Hub will be generally available later this year, but Red Hat is offering a ‘test drive’ through the upstream IDP project Janus.

     

  • Infrastructure-as-Code Pitfalls in Platform Engineering

    Infrastructure-as-Code Pitfalls in Platform Engineering

    In the context of cloud operations, the last decade was ruled by DevOps and infrastructure-as-code (IaC). But what is true DevOps? Is it developers running their own operations? Is it operators learning development skills to automate infrastructure provisioning using infrastructure-as-code (IaC) tools like Terraform?

    Though DevOps is widely understood to incorporate both of these, software developers are far from fully understanding or operating infrastructure. IaC is purely a DevOps tool. When an organization claims to have automated its infrastructure, they mean automation for the efficiency of the DevOps teams, not automation for developers. Developers are still left waiting on DevOps for days for even small infrastructure changes.

    While there were efficiencies gained by the current DevOps model, there is an increasing acknowledgment that we require a better approach to enable developer self-service. This has led to the platform engineering discipline.

    The high-level goals of platform engineering are:

    • Developer self-service for significant parts of infrastructure updates without DevOps subject matter expertise.
    • Built-in security and compliance controls.

    Infrastructure-as-Code (IaC) Limitations in Platform Engineering

    A DuploCloud survey of 40+ enterprises that have made substantial investments in platform engineering showed that the prevalent approach is for DevOps teams to build a DevOps platform with IaC as a core underlying technology. They create templates for the organization’s use cases and publish them in a CI/CD pipeline or a self-service catalog.

    Here are the top reasons why platforms that build on top of IaC are failing these platform engineering goals:

    1. IaC templates are rigid and not flexible enough to meet changing developer requirements: It is true that DevOps teams can anticipate cloud infrastructure topologies to some extent and have IaC templates for those with a few customizable parameters. But in a microservices world, there are thousands of other workflows and topologies possible based on changing application needs and security controls. Large data centers and edge infrastructure make this even more complex. A manual approach, relying solely on DevOps personnel to constantly build and update myriad combinations in static scripts simply can’t scale.
    2. Scripting tools can’t build life cycle management: In cloud operations, people-triggered changes are only a subset of possible use cases. Many asynchronous operations need to be continuously performed. These range from detection of configurations, from desired state to reverting, or complex configuration scenarios where individual components have to be set up asynchronously and brought together later. It could be as simple as a certain component going down and needing to be restored. IaC is a script that runs when triggered to completion. It has no active life cycle to operate continuously in the background.
    3. Inability to build a concept of an environment: In any orchestration system, users have a concept of the environment they want to build. When they log in to the platform and navigate to their environment, they expect to update as well as view the state and aspects of resources in that environment, be it the provisioning status, metrics, logs, faults, audit logs, compliance posture, etc. They may want to perform debugging functions, such as restarting services, SSHinginto a VM or accessing a resource’s cloud console (S3 Console, for example) using access control boundaries within that environment. For example, in Kubernetes, you choose the namespace to be the environment and management software like Rancher on top of K8S to provide these functions. If we must replicate this same concept across a broad infrastructure-level platform, we cannot do it using only IaC. We need something with a scope that spans all cloud operations, like Kubernetes. Terraform is a configuration updating and management software, not an orchestration system.

    Many platform teams have tried to work around these problems with point solutions by using a disparate set of jobs for certain aspects of life cycle management, building a thin UI shim on top of cloud accounts for visualization of resources while redirecting to other systems like Datadog for logging, metrics and alerts. But for most of the use cases, the DevOps team is still very much in the operations workflow. This completely defeats the concept of developer self-service as well as continuous compliance goals.

    Learning From Other Successful Platforms

    Two recent examples of successful cloud platforms are Amazon Web Services (AWS) in the context of infrastructure-as-a-service (IaaS) and Kubernetes for container orchestration. These are distributed system implementations using higher-level programming languages like Java and Go. You can’t build such complex systems using scripts and jobs.

    Building a true DevOps orchestration platform requires a systems design approach. It also takes expert systems engineers and many years to build and mature, like it did for K8s and IaaS in the Public Cloud.

    Conclusion

    Since platform engineering teams need systems design expertise that is likely outside of the scope of the typical DevOps role, we need the software development function to intervene with distributed systems programmers and many years of investment. Most organizations trying to build in-house platforms don’t have or don’t want to make this type of investment, as it may not be their core business. Megascale tech companies like Facebook, Uber, Netflix, etc. will likely build an in-house solution, as they have the talent and scale, but for the masses, it is likely that the solution will be an off-the-shelf product from an ISV.

    The industry has solved IaaS with public cloud, container orchestration with Kubernetes and observability with platforms like Datadog. The platform engineering category is heading in the same direction.

  • What’s the Difference Between DevOps and Platform Engineering?

    What’s the Difference Between DevOps and Platform Engineering?

    The DevOps space is buzzing with a new trend—platform engineering. Platform engineering has been gaining steam over the last few years but has really peaked in interest lately. In fact, according to a study conducted by Puppet Labs, 51% of organizations have adopted platform engineering or plan to do so in the next year. But what exactly is platform engineering, and how is it different than DevOps?

    Below, we’ll explore the nuances between these two concepts. We’ll review what DevOps and platform engineering are and see how they’re similar and dissimilar. We’ll also outline the benefits of platform engineering and suggest some best practices to make the most of this emerging discipline.

    What is DevOps?

    DevOps is a broad topic, but it generally involves efforts to increase software release efficiency and frequency. Some of the hallmarks of the DevOps discipline include reducing siloes between teams and increasing collaboration. DevOps is often directly associated with various tools that automate the process of moving code from a local environment into production, also known as continuous integration and continuous delivery (CI/CD). Monitoring, incident management and configuration management tools also support the DevOps process.

    “DevOps reflects more of a philosophy and a way of working,” said David Sandilands, senior solutions architect for Puppet. DevOps enables more of a shift left mentality. Instead of having operations teams “guarding the gates,” developers work in partnership with Ops. In doing so, groups can decrease build time significantly, he said. This style helps engineers incrementally and rapidly release new features, replacing old waterfall development cycles.

    The ultimate goal of DevOps is to automate the software life cycle and shorten the feedback loop. Thus, DevOps takes a life cycle approach to shipping applications more efficiently, streamlining phases such as build, release, configure, monitor, plan, create and verify. Microservices architecture and containerization also often play a crucial role in enabling DevOps.

    What is Platform Engineering?

    Platform engineering is all about developing a self-service internal platform that serves the needs of developer consumers. Although DevOps opened things up and empowered individual developers with advanced automation, it left organizations with scattered tooling and processes, especially at large enterprises. For example, various infrastructure-as-code (IaC) configurations might exist for similar functions within the same organization but be unknown to engineers in different departments. Not to mention, overseeing a growing DevOps pipeline means added responsibilities around continual updates and security.

    Platform engineering places more control in the hands of a central platform team rather than individual developers. These teams typically treat the platform as a product. They do so by evaluating the needs of developers across an organization, building out internal self-service tools that suit the needs of most consumers and maintaining the platform.

    The 2023 State of DevOps Report found that this self-service approach typically handles areas like deployment, operation of infrastructure, monitoring production applications and security and compliance. Other capabilities, such as abstracting Kubernetes or overseeing secrets management, may also play a role in these efforts.

    Benefits of Platform Engineering

    • Improve developer experience: Platform engineering can enhance the experience a developer has with their software build and release systems. A better developer experience can do loads to improve morale and create a more productive engineering atmosphere.
    • Standardize DevOps practices: By bringing together common DevOps approaches into a shared platform, organizations can begin to standardize with reusable build processes and automated infrastructure. This can unite an organization on common runtimes, observability methods and deployment processes. This can streamline the introduction of DevOps practices to an organization.
    • Secure the DevOps pipeline: Various vulnerabilities exist within tooling that comprises modern CI/CD chains and cloud-native open source software. By outsourcing DevOps responsibility to a shared platform team, the organization can coalesce around more secure practices and avoid overlooking important updates or holes.
    • Supply additional guardrails: Platform teams might also assist in the acquisition of new tools as well as outlining and enforcing policies for software infrastructure. While this may come at the cost of customization, additional guardrails can help avoid shadow IT and governance around internal development. Letting platform teams handle it could make maintenance easier across an organization.

    Platform Engineering: Under the DevOps Umbrella

    I don’t see platform engineering as replacing DevOps. Instead, I view platform engineering as falling under the umbrella of DevOps since platform engineering takes DevOps processes and makes them more reusable and self-service. Both use a lot of automation and share many of the same end goals.

    However, there are nuances between the two approaches that reflect the recent trends in the IT market. With DevOps, the market was more interested in shifting responsibility left and empowering individual developers to oversee the entire software life cycle. Now, as sprawl and shadow IT emerge, the pendulum seems to be shifting back toward a bit more centralization and governance with platform engineering.

    Platform engineering is typically more of an in-house development initiative. So, what are some best practices IT divisions should remember to get the most out of platform engineering? Hope Lynch, senior director of platform at Cloudbees, shared some great strategies to consider that revolve around treating the platform as a product. Doing this involves accepting feedback from developers, not controlling everything and always maintaining a focus on developer experience.

    Hopefully, by following these tips, organizations should have a better grasp of the emerging discipline of platform engineering and have the confidence to begin introducing platform-based concepts within their own environments.

  • Spotify Adds More Plugins for Backstage Developer Portal

    Spotify Adds More Plugins for Backstage Developer Portal

    Spotify this week added additional plugins for its open source Backstage platform that is used to build developer portals. The new plugins make it simpler to address role-based access and access Insights, a tool from Spotify that tracks Backstage usage trends.

    In addition, Spotify is also enhancing a Soundcheck plugin for Backstage that is used to visualize and track development of software components. Forthcoming capabilities include a no-code interface that makes it possible to programmatically create checks of code without writing any code.

    DevOps teams will also be able to view, export and understand how teams and components are doing compared to established best practices and monitor trends, graphs and historical views, and receive notifications when levels change.

    Finally, Spotify is committing to adding additional integrations with third-party tools and platforms such as Snyk, Sonarqube and the open source Argo continuous delivery (CD) platform.

    There are now five plugins available via a Spotify Plugins for Backstage subscription service, and Spotify said more are planned.

    Meg Watson, a group product manager at Spotify, said Backstage has been gaining traction in cloud-native application environments that are especially challenging to develop using multiple tools. Because of Backstage, developers are not only more productive but turnover has also been reduced because developers are provided with a better experience, she noted.

    There’s a lot more focus than ever on developer productivity, which Backstage addresses by creating a catalog of blueprints that developers can readily consume rather than requiring them to build these capabilities themselves multiple times over. The goal is to create scaffolds that developers can consistently reuse across multiple application development projects.

    Backstage was donated to the Cloud Native Computing Foundation (CNCF) and is now being advanced by contributions from multiple vendors. Much of that focus is on lowering the barrier of adoption for the platform, said Watson. Specifically, Spotify is working on a QuickStart for Backstage edition of the platform that is simpler to install, she said.

    In general, Spotify is trying to strike a balance between centralizing the management of DevOps workflows and the need to enable developers to define workflows that are natural to them versus ones that have been imposed on them, added Watson.

    It’s not clear to what degree Backstage will help fuel the adoption of a shift toward platform engineering to centralize the management of DevOps tools and platforms. The concept of a portal through which developers can self-service their own needs may not be new, but as an open source project, Backstage has made it simpler for many organizations to achieve that goal.

    Just about every DevOps team is now being asked to find ways to help improve developer productivity at a time when many organizations are trying to do more with fewer resources. Regardless of the motivation, however, it’s apparent that DevOps workflows are continuing to evolve and mature.

    In the meantime, DevOps teams would be well-advised to eliminate as many bottlenecks as possible before they negatively impact developer productivity.

  • Loosely Couple Your Schema and Keep Data Separate

    Loosely Couple Your Schema and Keep Data Separate

    In software development, one of the primary goals is to write clean, maintainable and extensible code. To achieve this, developers often strive to design systems that are loosely coupled and separate concerns into smaller, more manageable components. Loose coupling refers to the practice of designing components that are not tightly dependent on one another, allowing them to be modified or replaced without affecting the rest of the system. Separating data into distinct layers, such as a database layer, further contributes to this modularity by allowing developers to focus on writing business logic without worrying about data management. Together, loose coupling and data separation provide a solid foundation for building scalable and adaptable systems that enable developers to concentrate on writing high-quality business logic. This approach also lends itself to platform engineering best practices, a popular methodology that is being implemented or considered by developer teams today.

    Loose coupling is an important concept in microservices architecture because it helps to achieve the goal of creating modular, scalable and easily maintainable software systems. In a microservices architecture, individual services are designed to be independent, self-contained and loosely coupled, meaning they can be changed, updated or replaced without affecting other services or the system as a whole.

    There are several benefits to loose coupling in microservices:

    • Scalability: Microservices architecture allows scaling of individual services independently based on their needs, and with loose coupling, the interdependencies between services are minimized, allowing for better scalability.

    • Maintainability: With loose coupling, it’s easier to make changes to individual services without disrupting the system as a whole, reducing the risk of introducing bugs and making maintenance tasks easier to perform.

    • Flexibility: With a loosely coupled architecture, services can be developed, tested and deployed independently, enabling teams to work in parallel and make changes to the system without impacting each other.

    Platform engineering plays a crucial role in ensuring that microservices are loosely coupled. A well-designed platform should provide a standardized set of APIs, protocols and interfaces for service communication, as well as tools and frameworks for building, testing, deploying and managing services. This enables service teams to focus on developing their services rather than worrying about the underlying infrastructure and communication protocols.

    Moreover, platform engineering can also ensure that there is a standardized and centralized approach to security, monitoring and logging, which are critical aspects of microservices architecture. By providing a consistent and standardized approach to these areas, platform engineering can help ensure that individual services remain loosely coupled and can evolve independently, without negatively affecting the system as a whole.

    It’s All About Keeping the Schema Separate

    In platform engineering, the concept of keeping the schema separate typically refers to separating the schema of a database or data store from the application code that interacts with it. This separation allows for greater flexibility and scalability in the platform, as changes to the schema can be made independently of the application logic. By decoupling the schema from the application code, developers can avoid introducing breaking changes to the platform, and can more easily evolve and scale the platform over time.

    Separating the schema also allows for different parts of the platform to interact with the same data store using their own specialized schemas, without interfering with each other. This is particularly important in large and complex systems, where different teams may be working on different parts of the platform concurrently. By keeping the schema separate, teams can work independently and avoid stepping on each other’s toes, while still interacting with the same underlying data store.

    In summary, it’s all about the schema – don’t share databases and tables. The separation of data is fundamentally driven by that desire for loose coupling. Microservices should be as independent as possible from each other from design to development to production. Why? Efficiency. Disparate teams can change data structures and will not affect others versus the more traditional approach where we do share and it’s more cumbersome and involves more coordination between teams.

    Keeping schema separate in platform engineering is important for several reasons:

    ● Scalability: Each microservice owns its private schema, which enables the platform as a whole to scale more effectively. As the platform grows and evolves, it may become necessary to change the data schema without necessarily changing the underlying code. Separating the schema makes it easier to make these changes without disrupting the rest of the platform.

    ● Maintenance: Separating schema from code makes maintenance easier. It allows developers to make changes to the schema without needing to change the code, which reduces the risk of introducing bugs or errors.

    ● Collaboration: Separating schema from code makes it easier for multiple developers to work on the platform at the same time. When developers work on the code and schema separately, they can work independently without interfering with each other’s work.

    ● Flexibility: Separating schema from code provides greater flexibility for integrating with other systems or applications. It enables the platform to adapt to changing requirements or to interface with different data sources more easily.

    Separating Schema From Code is an Essential Practice

    By hiding data from service components from other service components, It provides guardrails and “just works.” While it does provide a form of coupling, it’s through messaging (typically asynchronous). Platform engineering encourages good design patterns from the start versus having to be more aware of these things. It’s a lot easier for especially newer developers that are not familiar with some of these architectural patterns and can get themselves into trouble.

    Overall, separating schema from code is an essential practice in platform engineering that provides scalability, maintenance, collaboration and flexibility benefits. It allows developers to build more robust and flexible platforms that can adapt to changing business requirements and technical environments.

  • Platform Reality Has a Surprising Amount of Detail

    Platform Reality Has a Surprising Amount of Detail

    Developer platforms and the platform teams that create them have been the center of attention for quite some time now. There have been many (wildly different) opinions about the subject, ranging from how bad the notion of a platform team is to how essential platforms are to make an organization’s DevOps journey to cloud native successful.

    Since early this year, the CNCF TAG App Delivery team has been working on a whitepaper about platforms. It explains what platforms are, why they are needed and how platform teams might work to create them. Furthermore, there’s an abundance of end-user talks at conferences that show how platform teams and their platforms were essential in enabling developers at scale. Even on the KubeCon + CloudNativeCon keynote stage, we’ve seen success stories of platform teams like the one at Mercedes Benz.

    Developer Platforms Should Enable Developers

    There is no silver bullet to successful platform teams and platforms. However, it is important to understand that the main goal of a developer platform is to enable developers. It is essential for platform teams to focus on delivering that value, which is why we are seeing a trend to introduce actual product ownership and product management roles to platform teams.

    The CNCF whitepaper stated that the three core jobs of a platform team as follows:

    1. Research platform user requirements and plan a feature roadmap.
    2. Market, evangelize and advocate for the platform’s proposed values.
    3. Manage and develop interfaces for using and observing capabilities and services including portals, APIs, documentation and templates and CLI tools.

    I couldn’t agree more. But the reality of day-to-day business for many platform teams means they rarely get to focus on those three jobs.

    Reality Has Detail

    The situation reminded me of a blog post I read a few years ago, titled “Reality has a surprising amount of detail.” The author explained how it is easy for people to get stuck even when faced with a task that might look simple from the outside. They explain how a task as simple as building wooden stairs can, in its many subtasks, become quite complex. Furthermore, when faced with reality, even the tasks that seemed plannable get more complex and need real-life adjustments.

    Most engineers have had analog experiences in their day-to-day work. We can translate this notion into platform engineering quite well. Building a developer platform sounds straightforward at the onset. However, once we get into it, it becomes a rabbit hole of figuring out the CNCF landscape, adding company-specific glue, compliance, security, fixing bugs, getting deep into the open source projects and sometimes, faced with the reality of open source, even becoming contributors and maintainers of such projects.

    Now, I’m not saying that it’s all bad. All these things can be great and rewarding in their own way. Especially with a community like the CNCF, participating and contributing to open source can be a very rewarding experience. You’ll make lots of friends, learn DevOps and development skills and might even further your career significantly.

    While it’s a lot of work, there is a risk that every platform team basically reinvents the wheel. This is because an organization’s requirements seem too large to rely on a single product (or even a suite of products). We end up with bespoke platforms that need a lot of attention. These platforms also have platform teams that don’t have enough time to focus on the higher-level jobs that they were tasked with.

    Platform Evolution

    Over the years talking to and working with platform teams we have built a model of prototypical platform evolution and its risks at each stage.

    Usually, a platform team starts with the task of building a developer platform to enable the product teams to build great software and deliver value to the end users.

    As they build the platform from the first Kubernetes clusters to observability tooling to security, the backlog for features constantly grows. Even with a growing platform team and sub-teams for certain technology areas, there’s a risk that some of these backlog items never get attention. Especially, delayable, non-critical, daunting tasks (like upgrading to the next Kubernetes release) often fall victim to this problem.

    Next, when it comes to operations, the platform team needs to limit the possibilities and breadth of what they offer. Operations responsibility for functionality that is outside of those limits, (for example, a specialized data store) goes back to the product teams themselves, which results in developer distraction.

    Even with a wide coverage of the platform, Day 2 and operations, the next big task for the platform team is advocating for adoption. This is not just marketing for their platform, but also means the platform team has to build user-friendly interfaces and documentation. The risks and alternatives we often see here are that product teams build their own mini-platforms and systems that result in shadow IT. Another common risk is ending up with several competing platforms within big enterprises that are hard to integrate and centralize, which in turn leads to inefficiencies.

    Once we get over the adoption hurdles, it is the platform team’s job to deliver value and high-level capabilities to the product teams’ developers. However, at this stage, often the platform teams have grown and might encompass several sub-teams for technology areas. Additionally, some areas like security might already be under central IT control. The risk we see here is that the capabilities offered are reduced to technology silos. They might cover all areas like observability, connectivity, security and release engineering, but to achieve real maturity, a platform team needs to offer capabilities that span several of those areas.

    Many of the advanced capabilities that we see with more mature end users cover several or all technology areas. Think about a simple high-level capability, like progressive rollouts. It starts with release engineering capabilities to build and deploy the software. From there, we need connectivity capabilities to dynamically route traffic to the new service. Then we need observability capabilities to observe our new version and from there have automated feedback loops to roll back or go forward with the rollout to production. And if that wasn’t enough, as we’re a security-aware organization, we want the whole software supply chain that facilitates this to be secure and trusted.

    Question for the curious cloud native enthusiasts: How many tools do you need to integrate to achieve the above “simple” high-level capability in an automated way? And this is only one of many capabilities.

    PaaS-Like Platforms Without the Downsides

    So, while platforms and platform teams are definitely the way to go, we need to be aware of the amount of work and effort required.

    We need to be aware of the risks and strive towards building highly integrated platforms that deliver high-level capabilities to the product teams.

    However, cloud-native developer platforms, while having similar goals to the PaaS systems of the past, should not follow the same approach of full abstraction. If we just abstract away into bespoke systems that hide everything from the user, we go back to the world of PaaS where at some point the end-user developers hit limits and build workarounds around the platforms.

    Additionally, if we build these bespoke internal platforms, we’ll be stuck with them and need to maintain and evolve everything by ourselves.

    Or, to go back to the blog post I referenced earlier, “At every step and every level there’s an abundance of detail with material consequences.” And we have to remember what we care about. The way to avoid both of these downsides is to bet on open source and stand on the shoulder of giants.

    By building our platforms on open source and using as many of the established or soon-to–be-established standards of our community we can avoid our platforms becoming bespoke. Recent developments like Cluster API, Gateway API and sigstore are some good examples.

    Furthermore, we need to make our abstractions poke-able. While our platforms should come out of the box with user-friendly interfaces and sensible production defaults that enable new product teams to onboard quickly and get their jobs done without having to bother with deep understandings of the underlying technology, we need to keep them open to mature users that know their way around and want to tweak it to their needs. We achieve this by giving mature users the option to access said open standards and APIs safely.

    This is the strength that we gain from a community like the CNCF, so we should use it to build great platform products that stay true to the core of open source and strive to make our product teams as happy and successful as possible. And remember to be happy with ourselves as successful and appreciated platform teams.

  • How Platform Engineering Makes Software Sustainable

    How Platform Engineering Makes Software Sustainable

    The world is constantly becoming more and more aware of the importance of sustainability in every aspect of our daily lives from buying groceries to traveling, and the IT world is no exception. Indeed, sustainability is already gaining importance in software development. For example, in the latest version of Thoughtworks’ Technology Radar, “carbon efficiency as an architectural characteristic” is one of the new techniques of interest. Moreover, in its research “Predicts 2023: Environmental Sustainability Is Now an IT Sourcing Imperative,” Gartner states that “by 2026, 75% of organizations will increase business with IT vendors that have demonstrable sustainability goals and timelines, and will seek to replace vendors that do not.”

    This shows that there is a growing need for sustainability. How is the software development industry responding to this need? Many great initiatives are arising, but platform engineering, in particular, helps make software more sustainable.

    What is Platform Engineering?

    According to Gartner’s definition, platform engineering is “the discipline of building and operating self-service internal developer platforms (IDPs) for software delivery and life cycle management.” This trend has gained momentum in the last few years, and an increasing number of organizations are adopting internal developer platforms (IDPs) to scale their development process. IDPs allow developers to self-service the technology they need, whether it is infrastructure, runtime resources or ready-to-use services. This helps speed up the development process and ensures higher-quality software.

    Platform engineering is becoming more and more popular and foundational. According to Gartner, “by 2026, 80% of software engineering organizations will establish platform teams as internal providers of reusable services, components and tools for application delivery.” Therefore, given the expected widespread adoption of IDPs in the next few years, sustainability emerges as a requirement. Ignoring or underestimating the importance of sustainability may cause organizations to waste all the benefits ensured by an internal developer platform.

    Platform Engineering for Economically Sustainable Software

    To survive and operate properly in the market, a company has to be economically sustainable and therefore stable. Usually, software is an asset that contributes to revenue generation for the company, but it still has maintenance costs. To make software economically sustainable, the total cost of ownership (TCO) needs to be lower than the generated income.

    An internal developer platform can lower software product costs, making them more sustainable. How so? In a nutshell, by sharing resources that can serve multiple products and can be reused across multiple projects. Such resources include, but are not limited to, capabilities, processes, infrastructure, configuration, etc. Since the platform at scale manages shared capabilities, there product teams won’t waste time and effort replicating it every time. While lowering the TCO, this characteristic of platforms also contributes to reducing the software’s time to market.

    Moreover, platform costs are shared across the whole organization and are not part of the TCO of each software product. This helps software to be more sustainable from an economic point of view.

    Platform Engineering for Environmentally Sustainable Software

    When talking about sustainability, most people usually think about the environment. While respecting the environment is one of the most important factors, it is important to note that sustainability is not limited to this aspect.

    An internal developer platform can help software be more sustainable on two levels.

    • Observability: The platform should collect and easily expose metrics on the environmental impact of the software. Such metrics can include hardware resource usage, energy consumption and CO2 footprint.
    • Optimization: The platform should foster the adoption of sustainable patterns, like the ones provided by the Green Software Foundation. It should also include solutions that actively help reduce energy consumption, such as kube-green.

    It is important that both of these capabilities are present and implemented in the platform. Observability without optimization is useless, and optimization without observability cannot be done.

    Platform Engineering for Socially Sustainable Software

    When speaking about sustainability, social responsibility cannot be forgotten. The first social responsibility of each organization is toward its members and employees. Does developer self-service also have social implications? Yes, it does; mainly by improving the developer experience.

    Indeed, an internal developer platform helps organizations’ members to:

    • Reduce friction between teams: Since everyone can have everything they need right when they need it, there will no longer be bottlenecks and misunderstandings between different teams. This contributes to a better working environment where everyone feels welcome and more productive.
    • Decrease repetitive tasks and cognitive load for developers: Since the platform will manage and automate interactions with infrastructure and runtime, developers will no longer need to waste time and energy on such out-of-scope secondary tasks. This way, developers are more focused on their actual job of developing new features and they will feel more satisfied.
    • Enable work from anywhere at anytime: Since the platform can provide all technology and capabilities whenever a developer needs them, it will no longer be required that people work together in the same location and at the same time. In other words, IDPs enable remote and asynchronous ways of working. This allows organizations to offer remote working and to create distributed teams, ensuring a better work-life balance for the team members.

    All of this brings happier employees that will be more productive and satisfied with their work, which will reduce turnover.

    Conclusion

    The need for a more sustainable world is gaining momentum and involves every industry, including software development. No wonder Gartner included sustainable technology in the “Top 10 Strategic Technology Trends for 2023,” along with platform engineering. As platform engineering is being adopted by more and more organizations to accelerate software development, sustainability has to be addressed.

    An internal developer platform can help developers build more sustainable software. Platform engineering can help in all of these aspects, improving not only the developer experience but also the organization and the environment. This is just the beginning of the sustainable path platforms should follow, but the outlook is very encouraging.

  • Strategies To Consider When Adopting Platform Engineering

    Strategies To Consider When Adopting Platform Engineering

    Platform engineering helps centralize the responsibilities associated with maintaining internal infrastructure. And it continues to be an emerging discipline—according to the 2023 State of DevOps Report, 51% of organizations have already adopted platform engineering within the last three years, and 93% said it’s a step in the right direction.

    I recently met with Hope Lynch, senior director of platform at Cloudbees, to gain some insights on implementing platform engineering. According to Lynch, platform engineering has emerged at a critical juncture. Developers are overloaded, burnout is on the rise and tooling sprawl is starting to rear its ugly head. To respond to these trends, platform engineering has emerged to take the heavy load off internal software infrastructure management. But the practice requires keenly balancing responsibility with control and treating the platform as a product.

    Interestingly, Lynch believes that many IT departments—whether they know it or not—are already implementing some form of platform engineering. They might just need to work out the kinks. Below, we’ll define platform engineering as it relates to DevOps and share some best practices to consider when rolling out (or reiterating) a platform engineering plan within your organization.

    State of DevOps and Platform Engineering

    So, what is the state of platform engineering and DevOps? Contrary to the salacious ‘DevOps is dead,’ headlines, Lynch doesn’t believe DevOps is going anywhere. Instead, platform engineering is helping overburdened developers, having emerged at a time when developers and operations seem to be at a crossroads.

    Developers are now responsible for coding, security, deployment and infrastructure maintenance. “Developers are talented and creative, but everyone has only so much capacity,” said Lynch. She believes platform teams can relieve some of these pressures by creating standard builds and common CI/CD pipelines. “I see it as a partnership between the DevOps teams and platform operations teams.”

    Lynch views DevOps as more of a practice than a specific job title. And although developers should have control over their DevOps toolchain, she believes they were granted too much responsibility. For example, the amount of effort required to search the market, talk to vendors, and try out tools can all add up significantly.

    By creating proven patterns and managing tools, platform engineering teams can help narrow down the options, working behind the scenes to ensure the platform is self-serviceable and continually updated. Of course, this partnership requires regularly collecting feedback to ensure platform teams are enhancing development workflows, not hindering them.

    Best Practices for Platform Engineering Adoption

    So, what are some best practices that enterprises can apply to their internal DevOps platforms? Lynch shared some areas platform engineering teams should focus on to improve their process. By following these tips, a platform can enhance DevOps principles within a company.

    • Be open to feedback from developers. As previously mentioned, platform engineering and development should be partners. Platform engineers should seek to gather as much feedback as possible from developers to improve their experiences and ensure the right DevOps tools are acquired.
    • Don’t control everything. As such, a degree of flexibility should be granted—you don’t want to mandate standardized adoption of all tools. Yet, you do want to decrease rogue tooling and sprawl. The platform team should thus work to find a balance and build trust.
    • Maintain a DevOps culture. When people discuss DevOps, they often talk about the tools and technology, not why DevOps came to be. “DevOps is here because we were trying to find a way to allow teams to work better together and be high-performing,” said Lynch. As such, we shouldn’t forget to include the people element here. Platform engineering should reinforce a healthy culture with positive dialogues and ensure work is “a fantastic place to be.”
    • Treat the platform as a product. When delivering internal-facing tools, your developers are your consumers. Thus, it’s a good practice to treat the platform as a product in its own right. This includes publishing roadmaps and eliciting feedback. “The platform team really needs to win the business of their internal customers, or else they’ll buy or use something else, which could cause shadow IT.”
    • Focus on increasing developer experience. As part of viewing internal developers as customers, platform teams should invest in improving the developer experience associated with using the platform. This means reducing friction and integrating new capabilities into preexisting workflows whenever possible.

    Goalposts For Platform Engineering Success

    For any new initiative to be successful, it must have goalposts and be measured in some capacity. The same is true for platform engineering. So, what are some goalposts that IT leaders should have in mind when introducing platform engineering? According to Lynch, some positive end results include collaboration and efficiency, cost analysis, cybersecurity and tooling centralization.

    • Collaboration and efficiency: First, platform engineering should improve work in the best way for everyone, says Lynch. “The people component is always there and should always be a priority,” said Lynch. To reach these goals, she recommends having the right conversations to co-discover bottlenecks in the value stream management of software development.
    • Cost optimizations: Secondly, platform engineering is an opportunity for garnering insight into usage rates, which could inform cloud cost analysis. Platform teams can implement specific policies, such as spinning resources up or down to reduce expenses.
    • Improved security: Platform engineering teams can be the first to test new products, giving them a unique opportunity to synchronize requirements with security teams. They can also take the reins of continually patching dependencies to reduce vulnerabilities.
    • Increasing consistency. In general, another benefit is centralizing tooling control. Because when tens and thousands of developers are each using their own tool, it becomes harder for teams to work together. While some cutting-edge technology organizations may seek to give developers control over the full cycle to increase agility, most will benefit from centralizing procurement and oversight, said Lynch.

    The Human Side of Platform Engineering

    “Platform engineering, when done right, furthers the principles of DevOps,” said Lynch. And interestingly, it may be the case that teams don’t even realize they are already doing platform engineering. Once they realize they are gravitating toward these ideals, it will be an even greater reinforcer for DevOps and DevOps principles, she said.

    Currently, the tech space is in a tricky spot, with rising layoffs and tightening budgets. At many organizations, fewer people are now expected to perform the same high caliber of work. As a result, we need to place more emphasis on retaining the remaining talent and boosting morale. This means caring about what the developer’s experience is like throughout the day and supporting the human side of DevOps. According to Lynch, platform engineering is a natural choice in these circumstances.

  • Platforming the Developer Experience

    Platforming the Developer Experience

    This article is part three of three on the boom in platform engineering and its role in supporting a streamlined, positive developer experience.

    In a perfect world, the developer experience would be on that is completely free of obstacles. In the previous articles in this series, I argued that supporting and improving the developer experience is a critical business requirement. To continuously improve and keep up with the constant change that typifies the cloud-native development landscape, platform engineering is–at least for now–the golden path to reducing developer toil and increasing productivity.

    I’ve also countered the controversial idea that DevOps is on life support because centralized developer platforms eliminate the need for DevOps. If anything, DevOps and developers share a symbiotic relationship, and the developer platform is where their functions converge. In this third and final article of the series, I will explore this relationship, examining the considerations for how the developer experience is “platformed” and by whom.

    Optimizing Developer Experience: The Circular Economy of the Developer Platform

    Let’s put it this way: The more relevant and useful the developer platform, the likelier the developer is to use it. If it reduces toil, cognitive load and makes shipping software easier, the developer has more time to focus on their core task: Developing features. With intuitive, self-service workflows and all the tools developers need, they rarely, if ever, have to think about ‘the how’ of getting their software into the hands of users. And this works if and when an organization does at least a couple of things right:

    • The organization prioritizes the developer experience and empowers other parts of the organization to answer the question, How can we create the optimal developer experience?
    • The organization puts resources behind understanding and building the best developer experience–and that’s where both the developer platform and DevOps teams as “fixers” ideas emerge.

    Does this mean the “optimal experience” can’t be optimized? Does that mean developers cannot have input into their own (or more general) developer experience(s)? No. In fact, part of what makes the developer platform idea compelling is that developers don’t have to weigh in or make decisions on the platform or tooling. Still, it’s possible to let them have that freedom if the team or organization wants to. Bottom line: There is no one-size-fits-all developer platform any more than there is a single developer experience. This is why DevOps, along with PlatformOps, is an ideal complement to facilitating the “circular economy” of the developer platform. The platform is unlikely to stay static. And to keep it dynamic and responsive to the changing needs of the developer teams using it, the DevOps teams of the future can proactively remove, recycle, repurpose and restructure the way the platform runs and what tools are available in it, automating and evaluating … almost like a feedback loop on the platform itself.

    Your Developer Platform is the Golden Path to Developer Productivity

    The developer experience is built around the tools and workflows developers use and, in the case of a developer platform, the tools and processes you define and provide, i.e., a “golden path” to productivity.

    Whatever that platform includes, it will demand that you answer questions about what tools, processes and best practices will be most effective for delivering the optimal developer experience. For example, what and how much should you build yourself? How should the platform be assembled to best serve the needs of your developer team? What is the most effective configuration? How much freedom can or should be given to your developers, depending on your business goals and the security, stability and scalability requirements of your organization and products?

    Boiling down the route to the golden path, there are three critical points to help guide the creation of an effective internal developer platform:

    • Treat your platform as a product. This is key to reducing cognitive load and paving the way to shipping software fast and safely. This also means that you are guaranteed to have an MVP and alpha, beta, etc. version of the product–and subsequent versions serve their purpose temporarily but are always works in progress.
    • Realize you can’t have good DevX without good UX and develop accordingly.
    • Focus on workflows and tooling interoperability; that is, a developer(-led) control plane. What is going to work for developers in practice to ensure that they can focus on key deliverables? How will you gather intelligence to discover when it’s time to make changes to the platform? Who will be responsible for this?

    Paving and Using the Golden Path Developer Platform Simultaneously

    Developer platforms can significantly improve developer productivity and the overall developer experience. But you can’t just build a platform and hope for the best without seeking developer input or understanding how developers work. Nor can you build a platform and decide it’s finished and forget about maintenance, enhancements and inevitable change. Developer platforms don’t exist as one-and-done items on a checklist and are themselves productivity tools that are in a constant state of flux.

    Misunderstanding why you would platform the developer experience in the first place defeats the purpose.

  • flox Unveils More Accessible Instance of Nix Framework

    flox Unveils More Accessible Instance of Nix Framework

    This week, flox launched a more opinionated implementation of the open source Nix framework that programmatically automates package management and system configuration.

    Fresh from raising an additional $16.5 million in funding, flox CEO Ron Efroni said the company will also make an enterprise edition of its namesake flox framework available later this year.

    The Nix framework has been available since 2003 and provides DevOps teams with a portable, fully isolated development, test and production environment along with a catalog and tools for managing software packages. There are more than 5,000 contributors to the platform, but Nix is based on a unique programming language that many DevOps teams have found challenging to master.

    The flox framework provides a higher layer of abstraction for invoking Nix, and provides a set of guardrails to make the framework more accessible to a wider range of DevOps teams, said Efroni.

    That approach will make it easier to share development environments and packages across teams, machines, individual developers and entire organizations. The framework also provides visibility into the software bill of materials (SBOM) for applications used in production environments to enable real-time vulnerability management and detection without scanning, noted Efroni. That capability makes it easier to discover dependencies spanning multiple architectures; many organizations today still try to track dependencies using spreadsheets and tools that don’t serve to make DevSecOps processes simpler to manage, he added.

    It’s not clear how many organizations are using Nix today, but Efroni said that as software development management continues to become more complex across multiple platforms and architectures, there is a clear need for a different approach to enabling portability at scale. The biggest obstacle is overcoming the psychological biases many DevOps teams have developed toward Nix over the last two decades, he noted. A more opinionated framework should reduce the overall fear factor associated with Nix, noted Efroni.

    Regardless of the frameworks employed, it’s become apparent that DevOps teams are struggling under the weight of all the code they must manage, much of it written in multiple programming languages. Many organizations that embraced DevOps early are now finding the platforms they deployed have become challenging to manage and maintain. As application environments become more complex, there is a need to modernize DevOps workflows. Less clear is whether those efforts might be focused on, for example, containers and serverless computing frameworks versus a framework such as flox that is designed to span multiple architectures and platforms.

    It might be a while before IT organizations develop a unified approach to application development and deployment, given all the preferences of individual development teams. However, with the rise of platform engineering, it’s apparent more organizations are trying to strike a balance between developer freedom and the need to reduce the total cost of DevOps. That cost continues to rise as additional software architectures and platforms are employed.

    In the meantime, flox plans to contribute to both Nix and its own more opinionated version of the framework in the hopes of one day making it easier to achieve that goal.