devopsy.kinsta.cloud

Tag: self-service

Empowering Self-Service Users in the Digital Age

We live in an increasingly self-serviced world where digital natives have become the dominant generation in the workplace. Most individuals today are not only more comfortable with self-service, but they also actively choose it. Therefore, creating and evolving an intuitive yet secure self-service portal for users consuming IT resources has become a key undertaking for many organizations.

Portals can take platform teams years to deliver, and maintenance will remain ongoing. Still, if done well, they can massively lighten the load on IT departments for end users by improving the overall experience and reducing the cognitive load, and for platform teams by optimizing operational efficiency and reducing the number of open tickets. Therefore, it speeds up business processes and places autonomy and control in the hands of a happier workforce.

Self-service IT is about centralizing and giving access to important items such as applications, automation, deployment, documentations, cloud, security, FinOps and GreenOps simply and comprehensively. To use a metaphor, it is not about building a hugely over-engineered Boeing 747 with every feature for a quick city break. Quite the opposite. It is about creating a streamlined glider equipped with the right amount of ability to manage the operationally intensive tasks it needs to.

But, of course, self-service has advantages and disadvantages, and achieving a balance is essential when empowering users in the digital era.

Managing Shadow IT

There has already been a creeping level of unauthorized self-service in many organizations. Therefore, shadow IT, the use of IT-related hardware or software by an individual or department without the knowledge or approval of the IT department has become a major consideration for CIOs, and while not all bad, it is far from ideal.

Shadow IT is a clear signifier of the demand for self-service, it also points to a challenge that IT in isolation cannot overcome — people. On traditional infrastructure, organizations are already starting to develop interfaces on top of virtualizations because they have identified the need to provide self-service guardrails to users. However, when it comes to handling multi-cloud deployments at scale, with security, governance, FinOps and GreenOps considerations, this becomes challenging.

People are unpredictable, they can be set in their ways and/or willing to experiment. They might use a new service for a while and then migrate to another without much thought for security or privacy. They can deploy a new service for proof of concept and forget to delete it. They can become attached to processes or systems with which they’re familiar. It is all well and good to declare an intention to build a self-service portal, but there is no guarantee that individuals will flock to use it.

Assuming that any new self-service portal has all the functionality and features to cover the existing demands of the people who will use it — there are (broadly speaking) two options in change management. Force people to do the new thing or make the new thing easier to do than the old thing. Spoiler alert — one of these options tends to be more effective than the other.

Providing comprehensive support to users as they navigate through a self-service portal is, therefore, essential. Organizations should supply clear and accessible guides, conduct informative training sessions and establish a reliable support system. When developers feel empowered and well-supported, they won’t simply use the portal, they will integrate it as a vital component of their toolkit.

In the long-term, this will help foster loyalty and retention among top talent. It is also critical when it comes to building the best possible developer experience and helping the organization and people focus on what matters — improving and maintaining automation and infrastructure on one side and developing their applications on the other.

Maximizing User Experience

Developer experience, or DevX (as I have written about before), is of paramount importance for the successful uptake of any self-service portal.

It might sound obvious but when creating a portal an organization needs to consider the individuals who will use it, rather than creating something that makes people work differently. A well-designed self-service portal is one characterized by its ability to be used without the need for outside assistance, with both the sophistication suited to a skilled developer community and intuitiveness to encourage engagement with a less technically savvy user group. A successful portal therefore is straightforward, intuitive and enriched with the functionality to enhance developer workflows.

On the topic of workflow, the portal must integrate seamlessly with existing tools and systems. Leveraging everything-as-code serves as an effective foundation to ensure smooth integration across all systems, extending beyond just the self-service portal, with minimal error or delay. Incorporating automation features and orchestration can streamline operations, freeing up time from mundane repetitive tasks to improve DevX, allowing for a greater focus on innovation and problem-solving.

Freedom vs Control

Ultimately, portals must strike the balance between freedom and control, which can be achieved by ensuring flexibility with role-based access control. Granting end users the freedom to deploy within a secure framework of predefined permissions creates an environment ripe for innovation within a robustly protected environment. This means users can explore, experiment and innovate without concerns about security boundaries or unnecessary hurdles.

But of course, as with any project, organizations can’t afford to build something and consider that job done. Measuring success is ongoing. Metrics such as how often the portal is accessed, who uses what, which service catalogs are used and how the portal usage should be tracked, along with other relevant data will help point to any areas that need improvement.

It is also important to remember that it is collaborative work between the platform team and end users. And in technology, there is always room for improvement. For instance, recent advances in AI/ML could soon be leveraged to analyze previously inaccessible datasets and generate smarter and faster decision-making.

A well-designed self-service portal should deliver an experience that demystifies and democratizes access to services and data and enhances the project lifecycle process between teams. To achieve this, they should simplify usability, strive to assist developers in leveraging its features effectively, ensure alignment with existing workflows, enhance its functionality and regularly assess its impact on productivity. This will ensure that self-service users survive and thrive in the digital age.

July 9, 2024
Survey Surfaces Significant Platform Engineering Gains

A survey of 100 software engineers working at organizations in the U.S. and Western Europe published today found nearly all (99%) have embraced platform engineering to some degree as a methodology to manage DevOps workflows at scale, with more than half (53%) having done so in the last year.

Conducted by Port, a provider of a proprietary internal developer portal (IDP) platform, the survey also found 83% of respondents have also incorporated GitOps workflows into their platform engineering practices.

The survey also found 85% are working for organizations that have either started deploying IDPs or plan to do so in the coming year. However, what constitutes an IDP is not exactly clear, with more than a third (35%) that have implemented one reporting they are relying on spreadsheets to manage their IDP. A little over half (53%) of respondents, in contrast, are using an IDP to provide developers with self-service capabilities for setting up environments to write code that either was custom built, is based on open source Backstage software originally developed by Spotify or a commercial IDP platform.

Among the 85% of respondents using or planning to use an IDP, the main measure of success is to improve developer productivity (56%), followed by reduced time to deployment (25%). Despite those initiatives, only slightly more than a third (34%) said IDPs are helping DevOps teams enforce software engineering standards.

Roni Floman, chief marketing officer for Port, said overall, the survey suggests developers are wasting a lot of time on tasks that have little to do with writing code. For example, the survey finds that prior to embracing platform engineering, developers spent three to four hours a day on non-core activities, resulting in weeks being required to eventually deploy software in a production environment.

The biggest challenge organizations face when embracing platform engineering is convincing developers that a centralized team has their best interests at heart. Many application development teams embraced DevOps as a best practice in the first place to escape the tyranny of a centralized IT team. The issue is that, as a result, many of them now find themselves managing tasks that have little to do with writing code. Platform engineering seeks to strike a balance between giving developers the ability to adopt new tools when they see fit and the benefits of having a dedicated platform engineering team that automates processes on their behalf.

Unless platform engineering teams find a way to get developers to buy into the effort, however, it’s unlikely they will succeed, noted Floman.

Each organization will need to determine for itself what level of platform engineering makes sense for them to implement, given their unique culture. In most cases, a platform engineering initiative will start small before increasing in size and scope as momentum is gained. In the meantime, the most important thing any platform engineering team clearly needs to do in the short term is engage developers. After all, the best way to get support for any change to a DevOps workflow is, as always, to convince developers it was their idea in the first place.

December 13, 2023
The Benefits of Self-Service Infrastructure

Self-service infrastructure is a modern approach to IT management that enables development teams to take greater control of their systems and services without relying on dedicated IT staff for assistance. This approach offers numerous advantages that help businesses become more efficient and agile.

What is Self-Service Infrastructure?

Self-service infrastructure is a type of IT infrastructure system that allows users to access, manage and maintain their own applications, data and resources without requiring intervention from an external provider. It combines software automation tools, cloud computing services, open source technologies, network management solutions and other digital tools to provide an environment where users can interact with technology on their own terms, although within defined boundaries (created, i.e., for security reasons).

Self-service infrastructure has helped transform traditional service delivery models by increasing the development team’s autonomy when they use technology for various purposes. With self-service infrastructure in place, it can be easier for businesses to reduce costs associated with maintaining IT systems while also improving the speed at which operations can be performed since most tasks do not require manual intervention.

Some self-service infrastructure platforms include AWS Service Catalog, Google Cloud Deployment Manager, Spacelift and Backstage.

Self-service infrastructure comes with a bit of a challenge. There is often a learning curve associated with setting up an effective self-service architecture. Since software platforms are constantly evolving due to updates or new versions being released, it can be difficult for administrators to stay up to date on changes to ensure that their systems are running smoothly. But self-service infrastructure also offers a number of benefits to compensate for the troubles.

What are the Benefits of Self-Service Infrastructure?

First, self-service infrastructure provides teams with the flexibility they need to quickly develop, deploy and scale applications as needed. It eliminates the burden of non-core work from development teams so that they can work faster and more efficiently with fewer resources. For example, they don’t need to care about Terraform but have a service where they select templates that cover their needs.

This allows for more streamlined delivery processes – reducing the amount of time it takes for projects or applications from concept through production launch. Self-service infrastructure helps speed up product development cycles and reduces risk by decreasing overall time spent on debugging and testing infrastructure throughout the development stages. It removes infrastructure management from development teams entirely, even when automated, eliminating potential human errors that could lead to expensive downtime or security issues in the future.

Additionally, when combined with automation tools and DevOps best practices such as continuous integration/continuous delivery (CI/CD), it helps automate even more tedious tasks, further streamlining overall product delivery times without sacrificing quality control measures already in place.

In terms of collaboration, self-service infrastructure helps promote transparency throughout organizations by providing visibility into usage analytics while also allowing teams greater autonomy over asset utilization, which will increase productivity levels between departments.

It also helps establish better communication channels between software engineering teams who are responsible for application releases since they can deploy timely updates through automation. There’s no need for long waiting periods due to delays in approvals from vendors or other stakeholders like during traditional manual process setups.

Finally, one of the most important benefits of self-service infrastructure is scalability capabilities. Whether your business needs capacity for a few thousand users or over one hundred thousand users, it can handle any scale you require at any given moment in time so that you’re never left behind due to a lack of computing resources when demand unexpectedly increases.

Wrapping Up

Self-service infrastructure provides many opportunities for businesses to modernize their IT operations easily. With features such as scalability, automation and agile development, companies can benefit from the ability to quickly and efficiently deploy applications.

Not only does self-service IT infrastructure reduce operational costs, but it also offers greater control over resources and enables organizations to leverage their limited resources more effectively.

November 7, 2023
Atlassian Makes Compass IDP Generally Available

Atlassian this week made generally available a Compass developer experience platform through which DevOps teams can employ a software-as-a-service (SaaS) platform to centralize the management of the environments used to create code.

Taylor Pechacek, head of product for Compass at Atlassian, said Compass provides an instance of an internal developer platform (IDP) that is simpler for DevOps teams to set up and maintain.

At its core, Compass provides access to a software component catalog to track, search and discover the component and approved infrastructure that can be used to build and deploy applications.

In addition, templates enable developers to self-service environment setup using a set of policies that provide guardrails to ensure developers follow best practices for provisioning cloud resources and setting up pipelines.

Finally, Atlassian is making available scorecards based on the DevOps Research and Assessment (DORA) metrics defined by Google.

Compass is designed to be integrated with a wide range of continuous integration/continuous delivery (CI/CD) platforms and third-party observability, testing, collaboration and source code management tools, said Pechacek. The goal is to minimize context switching as much as possible for developers accessing the Compass platform, he added.

That approach also provides a level of consistency that both fosters collaboration and makes it simpler to onboard new members to a development team, noted Pechacek.

While IDPs as a concept have been around for some time, they are now central to any effort to embrace platform engineering methodologies to implement best DevOps practices at scale. The issue platform engineering teams need to come to terms with is determining whether they want to build and maintain an IDP versus invoking one provided as a SaaS application.

Regardless of approach, platform engineering teams are trying to strike a delicate balance between centralizing the management of development environments and the ability of developers to innovate by adding new tools to their development environment. Organizations clearly want developers to become more productive by spending less time managing development environments, but many of those same developers tend to jealously guard their prerogatives when it comes to the tools they prefer.

The challenge, of course, is that spending less time managing a development environment doesn’t necessarily mean more code will be written. All it does is afford developers the opportunity to write more code. The issue is that the writing of quality code only really occurs when developers are inspired, but at the very least, configuring a development environment shouldn’t stand in the way of creativity.

It’s not clear how many organizations are embracing platform engineering methodologies, but it’s apparent that more supervision is being brought to how applications are built and deployed within the context of a set of DevOps best practices. The challenge and the opportunity now is to make sure developers buy into, rather than resist, those efforts.

October 18, 2023
The DevOps Pendulum: Agility Vs. Control

Managing changes to cloud assets is a universal pain felt by many engineering leaders today, despite all of the advancements in tooling and practices like GitOps. This is because, in reality, it is simply impossible to always have everything completely locked down—we don’t live in a utopia with zero incidents. If your engineering organization prevents any changes from being made through the cloud console or to your infrastructure-as-code (IaC) without complying with strict GitOps practices or change management processes via CI/CD, then it’s likely you have very frustrated developers—developers who can’t troubleshoot or debug in real-time and who have little control in a real-world incident.

Engineering, like everything else in life, is all about balance.

The Pendulum Swings

The pandemic created a whole new mindset and practice for managing distributed and high velocity engineering and operations. Overnight, companies that were not built for remote ways of working had to continue their operations in a global, distributed and asynchronous way that they weren’t completely familiar with. This required a new way to think about software delivery—and it accelerated the DevOps practices that support that delivery. Self-service infrastructure removed barriers for developers to ensure continued performance and velocity.

At the same time, your cloud cannot be the wild west with everyone creating bespoke infrastructure. It becomes impossible to manage, and misconfigurations can be risky. Guardrails and policy automation have become a hot topic. Today, with tech markets depressed and cloud costs escalating, there seems to be a growing trend toward locking things down again, even at the risk of frustrating developers.

This begs the question: How can you achieve unencumbered infrastructure for your developers while simultaneously following policies and best practices for compliance, risk and cost? There is a way to find balance.

Like many aspects of security, we’ve learned that when constraints and barriers are too high, users ultimately find ways to bypass them. This is true for operations, as well. While sometimes it might seem easier to lock everything down than design a better and more balanced way to enable developers to move fast, eventually this approach backfires. This is the exact same evolution the application and cloud-native security industry are undergoing right now. All the guardrails and controls applied have created too much friction in development processes and developers ultimately bypass them.

Real-Time Insight is Key

CloudOps can learn a lot from the disruption the security industry is going through today. In the same way that point-in-time security has been rendered completely useless, non-real-time alerting or eventual alerting to infrastructure drift just won’t cut it when managing an ephemeral cloud. What is really needed is the same kind of real-time and continuous scanning of cloud assets and IaC, similar to what we apply to our systems through monitoring and observability. These solutions became an essential backbone of our business to ensure continuous operations and availability of cloud services.

As we embrace IaC and the benefits it brings, everything-as-code unlocks greater agility and visibility, enabling you to automatically remediate without locking everything down. As DevOps says, “fail forward and fail fast.” Rather than focusing on never making a mistake, focus on how to fix it immediately.

The Solution

By providing a continuous comparison of actual cloud assets with their desired state through IaC and GitOps, it’s possible to surface configuration drift and policy violations immediately, much like any other kind of breach or major system failure. Failures and incidents are inevitable. It’s unrealistic, and even dangerous, to build systems with an inherent underlying design that prevents you from changing something at the cloud console at 2:00 A.M. during an outage.

With a locked-down approach, a developer would need to wait for the change management approval during a high-pressure production incident, oftentimes at 3:00 A.M. (because the incident gods of always make them happen in the middle of the night) or on a weekend. We don’t need to look much further than the catastrophic 40-minute and $440 million Knight Capital incident to understand that sometimes time is not on our side and that a delay can have severe consequences.

Cloud drift has become equally troubling as uptime or any other mission-critical aspect of continuous business operations, and that’s why we need to apply the same principles for monitoring drift in real-time as we do to our CPU and load. This will make it possible to be alerted when your cloud and your IaC are no longer aligned and even flag issues that can be prevented in real-time. You can automate ticket creation and escalation and also triage whether this is something that needs to be fixed in real-time with remediation suggestions provided or open a ticket to fix it later. The choice should be yours, not made by an administrator who has a lot less context about the systems and the ultimate business impact.

It is Not ‘One and Done’

Some vendors preach codifying your cloud into IaC and then locking it down so that you’ll never have to do it again. This feels like deja vu; we’ve definitely heard this before. It sounds a lot like the ‘write once, deploy anywhere’ JVM promise that never quite panned out.

By going down this path, you’re resigning yourself to never adding new cloud assets or changing your cloud configurations. If there’s one thing we’ve learned about the cloud, it’s that it’s a constantly moving target. (Read what cloud asset management can learn from the world of finance—another industry in constant flux). If this didn’t work decades ago with the JVM, it certainly won’t work in a dynamic and constantly changing environment like the cloud. That’s a fool’s journey.

Continuous Cloud Up to Code

There are major downsides of locking down agility and velocity. This is hard-earned advice, witnessed firsthand from tools like legacy CMDB and startups who have forgotten that cloud is ephemeral—and possibly the reason why we all use it in the first place! So if our cloud is constantly changing and is a continuously moving target, then shouldn’t our tools be built in just the same way to keep up? Solutions built for the rapidly evolving cloud-native era must continuously look for new, unmanaged and changed assets and ensure their cloud infrastructure always follows policies and regulatory standards.

Once we are able to find the balance between agility and control, we’ll be able to tap into the benefits of speed and safety (as DORA research shows time and again) that are the benchmark of high-performing teams. Automating cloud asset management can provide governance, policies and control while still enabling the velocity and agility engineering organizations today require.

Agility Versus Control

There’s a minority of people who claim they don’t worry about cloud asset configuration drift because they have everything completely locked down. Zero changes can be done at the cloud console or to IaC without going through GitOps for strict change management processes via CI/CD. I bet they have happy developers—not!

Life is All About Balance

Throughout the pandemic, organizations had to make developers happy for fear they would leave. So DevOps went full tilt aiming for self-serve infrastructure for developers to remove barriers to speed and velocity. Now, with tech markets depressed, maybe it’s easier to lock things down even if that risks losing engineers.

As we embrace IaC and its benefits, you need not live in the stone age of locking everything down. How can you achieve unencumbered infrastructure for your developers while following policies and best practices for compliance, risk and cost? There is a way.

The solution? Real-time continuous scanning of your cloud assets and your IaC with continuous comparison can surface configuration drift and policy violations immediately. Emergencies are inevitable. It’s bold/stupid/foolish to think you’ll never need to change something at the cloud console during an emergency or that a developer will wait for change management when production is down at 3:00 A.M. on a weekend. It happens, and you must deal with it.

August 2, 2023
How Can DevOps Evolve? Platform Engineering

Kaylee Van De Voorde co-authored this article.

Has your team adopted a DevOps mindset? Are your engineers using tools like Docker containers, Kubernetes and Terraform? Over the last decade, many engineering teams that adopted DevOps culture and pivoted to cloud-native applications that required rapid scaling may face a new set of problems.

DevOps

DevOps practices have brought operations and software engineering teams together throughout the entire development life cycle, from initial concept through the engineering process to production support. This collaboration paved the way for iterative development: Improving velocity, providing rapid feedback and consequently improving application quality by uncovering bugs and environment requirements earlier without user impact. DevOps automation created a more “blameless” culture that improved trust, collaboration and innovation, which boosted team morale and increased engagement.

That said, DevOps teams have encountered many obstacles, including fears of job obsolescence and steep learning curves for every member. Teams that learned how to implement DevOps practices on their projects may have moved more slowly at first while learning how to work together in new ways and adopting new tools.

What is Platform Engineering?

So, what is platform engineering? Platform engineering bridges the existing gaps between software, hardware and infrastructure that come with adopting DevOps practices: Automated horizontal scaling, service discovery and cost optimization. Platform engineering makes it even easier and simpler for engineers produce high quality applications quickly by standardizing CI/CD deployments, ensuring that infrastructure is auditable and monitored, automating deployment methods and processes, providing accurate documentation and incorporating FinOps.

Platform engineering teams can make these improvements available to their teams by creating tools such as an internal developer platform (IDP) that includes APIs that expose internal tools and features within the platform.

The Next Evolution

Evolution does not mean starting from nothing.

The core tenets of DevOps practices still hold their value; aligning software engineering and operations teams to better create applications that embody business and user functionality and run smoothly within an incorporated infrastructure, automation, security and dedicated site reliability engineering (SRE). Many DevOps teams may already be moving toward an overarching platform engineering culture and simply need to refine their standards. Distinct application teams may already be building similar features to provide platform functionality; a strong indication that a platform engineering team would be beneficial.

Choosing to create a dedicated platform engineering team has tradeoffs: Building a platform engineering team pulls resources from building applications and new features that the business needs, but application teams could experience an increase in productivity if a separate platform engineering team is created. A dedicated platform engineering team can also provide clear, unified direction and nuanced goals for application teams; this is incredibly important when creating standards and internal tools.

A platform engineering team is just one way that to advance and improve engineering culture. DevOps teams that may not require platform engineering can continue their growth by refining their existing practices and patterns in their everyday work.

June 29, 2023
What’s the Difference Between DevOps and Platform Engineering?
The DevOps space is buzzing with a new trend—platform engineering. Platform engineering has been gaining steam over the last few years but has really peaked in interest lately. In fact, according to a study conducted by Puppet Labs, 51% of organizations have adopted platform engineering or plan to do so in the next year. But what exactly is platform engineering, and how is it different than DevOps?

Below, we’ll explore the nuances between these two concepts. We’ll review what DevOps and platform engineering are and see how they’re similar and dissimilar. We’ll also outline the benefits of platform engineering and suggest some best practices to make the most of this emerging discipline.

What is DevOps?

DevOps is a broad topic, but it generally involves efforts to increase software release efficiency and frequency. Some of the hallmarks of the DevOps discipline include reducing siloes between teams and increasing collaboration. DevOps is often directly associated with various tools that automate the process of moving code from a local environment into production, also known as continuous integration and continuous delivery (CI/CD). Monitoring, incident management and configuration management tools also support the DevOps process.

“DevOps reflects more of a philosophy and a way of working,” said David Sandilands, senior solutions architect for Puppet. DevOps enables more of a shift left mentality. Instead of having operations teams “guarding the gates,” developers work in partnership with Ops. In doing so, groups can decrease build time significantly, he said. This style helps engineers incrementally and rapidly release new features, replacing old waterfall development cycles.

The ultimate goal of DevOps is to automate the software life cycle and shorten the feedback loop. Thus, DevOps takes a life cycle approach to shipping applications more efficiently, streamlining phases such as build, release, configure, monitor, plan, create and verify. Microservices architecture and containerization also often play a crucial role in enabling DevOps.

What is Platform Engineering?

Platform engineering is all about developing a self-service internal platform that serves the needs of developer consumers. Although DevOps opened things up and empowered individual developers with advanced automation, it left organizations with scattered tooling and processes, especially at large enterprises. For example, various infrastructure-as-code (IaC) configurations might exist for similar functions within the same organization but be unknown to engineers in different departments. Not to mention, overseeing a growing DevOps pipeline means added responsibilities around continual updates and security.

Platform engineering places more control in the hands of a central platform team rather than individual developers. These teams typically treat the platform as a product. They do so by evaluating the needs of developers across an organization, building out internal self-service tools that suit the needs of most consumers and maintaining the platform.

The 2023 State of DevOps Report found that this self-service approach typically handles areas like deployment, operation of infrastructure, monitoring production applications and security and compliance. Other capabilities, such as abstracting Kubernetes or overseeing secrets management, may also play a role in these efforts.

Benefits of Platform Engineering
- Improve developer experience: Platform engineering can enhance the experience a developer has with their software build and release systems. A better developer experience can do loads to improve morale and create a more productive engineering atmosphere.
- Standardize DevOps practices: By bringing together common DevOps approaches into a shared platform, organizations can begin to standardize with reusable build processes and automated infrastructure. This can unite an organization on common runtimes, observability methods and deployment processes. This can streamline the introduction of DevOps practices to an organization.
- Secure the DevOps pipeline: Various vulnerabilities exist within tooling that comprises modern CI/CD chains and cloud-native open source software. By outsourcing DevOps responsibility to a shared platform team, the organization can coalesce around more secure practices and avoid overlooking important updates or holes.
- Supply additional guardrails: Platform teams might also assist in the acquisition of new tools as well as outlining and enforcing policies for software infrastructure. While this may come at the cost of customization, additional guardrails can help avoid shadow IT and governance around internal development. Letting platform teams handle it could make maintenance easier across an organization.
Platform Engineering: Under the DevOps Umbrella

I don’t see platform engineering as replacing DevOps. Instead, I view platform engineering as falling under the umbrella of DevOps since platform engineering takes DevOps processes and makes them more reusable and self-service. Both use a lot of automation and share many of the same end goals.

However, there are nuances between the two approaches that reflect the recent trends in the IT market. With DevOps, the market was more interested in shifting responsibility left and empowering individual developers to oversee the entire software life cycle. Now, as sprawl and shadow IT emerge, the pendulum seems to be shifting back toward a bit more centralization and governance with platform engineering.

Platform engineering is typically more of an in-house development initiative. So, what are some best practices IT divisions should remember to get the most out of platform engineering? Hope Lynch, senior director of platform at Cloudbees, shared some great strategies to consider that revolve around treating the platform as a product. Doing this involves accepting feedback from developers, not controlling everything and always maintaining a focus on developer experience.

Hopefully, by following these tips, organizations should have a better grasp of the emerging discipline of platform engineering and have the confidence to begin introducing platform-based concepts within their own environments.
May 18, 2023
The Now and Next of Automation

Today’s never-ending flood of data has many of us drowning in noise. And while humans are still essential for thoughtful analysis and driving action, it’s no longer possible for a human to parse gigabytes of data alone.

How do you begin to filter out the noise and then understand what to do with it? How can you get to the root cause of an issue so you can act on and resolve it? Most of all, how can you do all of this while being more productive and driving more innovation instead of constantly putting out fires?

Automation is the obvious answer. For many organizations, automation isn’t just about efficiency – it’s about survival. Automation can make organizations more agile, resilient, and capable.

Where you sit within an organization or team, however, often dictates how automation can help with your specific goals or alleviate your immediate pain points. Mandi Walls, a DevOps advocate with PagerDuty, offers some advice on how even less experienced teams can use automation to perform simple tasks, solve problems and improve the work environment.

We hear a lot these days about A.I. and machine learning, or tools like Chat GPT which to some is a harbinger of robots finally replacing humans. Putting the hyperbole aside, where do you see automation today? Is it still a “nice to have” for engineering teams or a “must-have” to do their jobs effectively?

Automation is definitely a “must-have.” Manual processes are risky; they can inadvertently introduce too many errors, especially in complex environments. The scope and complexity of modern technical ecosystems is just too vast for teams to effectively cope with using only manual methods.

This is especially true for teams that develop and manage customer-facing services. Having a successful product leads to more customers, which requires more capacity and creates more complexity. To deliver the kind of performance customers expect, teams need to be fast and efficient. That’s where automation comes in.

What are some specific types of automation that engineers and IT pros should be aware of?

Some automation has become more or less ubiquitous, such as tools that enable configuring and building large numbers of systems with little human intervention. Other processes might feel less like “big-A” automation–for example, continuous integration/continuous delivery (CI/CD) pipelines are automation.

Humans also aren’t manually building many software packages anymore. The use of automation in the software testing process has greatly expanded in the last several years, changing the jobs of QA engineers significantly.

Is self-service automation – aka true autonomy that doesn’t require human intervention – the ideal end state for a digitally mature organization? When is the best time to jump in?

Organizations that wait for a perfect opportunity to automate their processes risk wasting a lot of time and resources in the interim. Teams can look at any of the tasks that they regularly perform and assess them as candidates for automation.

Tasks can be assessed in a number of different ways, but for teams just starting out with automation, seeking tasks that are low complexity and low impact is a strong starting point. Even if we look just within the realm of tasks that might be utilized during incident response, teams might have any number of tasks related to information gathering, log preservation, system diagnostics, and other telemetry that responders run for every incident. There’s no reason to require humans to run those well-understood tasks!

As teams get more comfortable with creating and using automation, they can seek out more complex tasks to automate. And it’s entirely possible that there will be tasks that teams aren’t able to automate fully! Those tasks might be too complex, require a lot of human input or interaction or have too many edge cases for the team to create automation for.

Are the principles of automation the same in simple versus more complex environments like Kubernetes?

The principles are essentially the same, but the constraints and tools may be different. In some environments, a simple shell script might be enough to accomplish our goals. In other environments, we might need to make use of APIs or vendor SDKs to do what we want.

On that topic, is self-service automation in cloud environments the preferred path?

Actually, self-service is sort of how the cloud really got going! Development teams armed with nothing more than a credit card got the environments they needed without going through lengthy procurement processes.

Now that cloud is a critical part of many workloads, organizations are putting more rigorous guidelines in place. The cloud platforms are also much more complex, with more service offerings than in the early days, which makes automation more important than ever. Automation helps organizations ensure that their cloud environments meet their security and compliance requirements, helps manage spend and still provides enough flexibility for all teams to get work done. And with self-service automation, developers can request an environment in the cloud to develop their service and have that service meet all the organization’s standards without having to keep up with all the details themselves.

What about toil? Can that be fully eliminated through automation, or is some toil still necessary?

I’m not sure I’d say toil is necessary, as much as it tends to be inevitable. There will likely always be some boring, repetitive tasks that are too unpredictable or unstable to automate well.

That said, teams certainly don’t need more toil; having too much repetitive work tends to lead to less job satisfaction and higher attrition rates, so teams should be looking to minimize as much toil as possible where they can.

Is there one myth or misconception, either about automation overall or self-service automation in particular, that you’d like to clear up? What’s the reality?

First of all, don’t worry. You won’t be able to replace your team with a set of shell scripts! Seriously though, automation isn’t about “doing more with less.” It’s about doing more with the same. Automation is an approach to help teams cope with the day-to-day running of complex environments and will need to be managed as part of those environments. Managers need to make sure they are allotting time for the creation and maintenance of automation in the same way they would for other tools and applications.

Interested in learning more about PagerDuty? Stop by Booth S49 at KubeCon to speak with an expert or see a demo!

April 12, 2023
How Platform Engineering Makes Software Sustainable
The world is constantly becoming more and more aware of the importance of sustainability in every aspect of our daily lives from buying groceries to traveling, and the IT world is no exception. Indeed, sustainability is already gaining importance in software development. For example, in the latest version of Thoughtworks’ Technology Radar, “carbon efficiency as an architectural characteristic” is one of the new techniques of interest. Moreover, in its research “Predicts 2023: Environmental Sustainability Is Now an IT Sourcing Imperative,” Gartner states that “by 2026, 75% of organizations will increase business with IT vendors that have demonstrable sustainability goals and timelines, and will seek to replace vendors that do not.”

This shows that there is a growing need for sustainability. How is the software development industry responding to this need? Many great initiatives are arising, but platform engineering, in particular, helps make software more sustainable.

What is Platform Engineering?

According to Gartner’s definition, platform engineering is “the discipline of building and operating self-service internal developer platforms (IDPs) for software delivery and life cycle management.” This trend has gained momentum in the last few years, and an increasing number of organizations are adopting internal developer platforms (IDPs) to scale their development process. IDPs allow developers to self-service the technology they need, whether it is infrastructure, runtime resources or ready-to-use services. This helps speed up the development process and ensures higher-quality software.

Platform engineering is becoming more and more popular and foundational. According to Gartner, “by 2026, 80% of software engineering organizations will establish platform teams as internal providers of reusable services, components and tools for application delivery.” Therefore, given the expected widespread adoption of IDPs in the next few years, sustainability emerges as a requirement. Ignoring or underestimating the importance of sustainability may cause organizations to waste all the benefits ensured by an internal developer platform.

Platform Engineering for Economically Sustainable Software

To survive and operate properly in the market, a company has to be economically sustainable and therefore stable. Usually, software is an asset that contributes to revenue generation for the company, but it still has maintenance costs. To make software economically sustainable, the total cost of ownership (TCO) needs to be lower than the generated income.

An internal developer platform can lower software product costs, making them more sustainable. How so? In a nutshell, by sharing resources that can serve multiple products and can be reused across multiple projects. Such resources include, but are not limited to, capabilities, processes, infrastructure, configuration, etc. Since the platform at scale manages shared capabilities, there product teams won’t waste time and effort replicating it every time. While lowering the TCO, this characteristic of platforms also contributes to reducing the software’s time to market.

Moreover, platform costs are shared across the whole organization and are not part of the TCO of each software product. This helps software to be more sustainable from an economic point of view.

Platform Engineering for Environmentally Sustainable Software

When talking about sustainability, most people usually think about the environment. While respecting the environment is one of the most important factors, it is important to note that sustainability is not limited to this aspect.

An internal developer platform can help software be more sustainable on two levels.
- Observability: The platform should collect and easily expose metrics on the environmental impact of the software. Such metrics can include hardware resource usage, energy consumption and CO2 footprint.
- Optimization: The platform should foster the adoption of sustainable patterns, like the ones provided by the Green Software Foundation. It should also include solutions that actively help reduce energy consumption, such as kube-green.
It is important that both of these capabilities are present and implemented in the platform. Observability without optimization is useless, and optimization without observability cannot be done.

Platform Engineering for Socially Sustainable Software

When speaking about sustainability, social responsibility cannot be forgotten. The first social responsibility of each organization is toward its members and employees. Does developer self-service also have social implications? Yes, it does; mainly by improving the developer experience.

Indeed, an internal developer platform helps organizations’ members to:
- Reduce friction between teams: Since everyone can have everything they need right when they need it, there will no longer be bottlenecks and misunderstandings between different teams. This contributes to a better working environment where everyone feels welcome and more productive.
- Decrease repetitive tasks and cognitive load for developers: Since the platform will manage and automate interactions with infrastructure and runtime, developers will no longer need to waste time and energy on such out-of-scope secondary tasks. This way, developers are more focused on their actual job of developing new features and they will feel more satisfied.
- Enable work from anywhere at anytime: Since the platform can provide all technology and capabilities whenever a developer needs them, it will no longer be required that people work together in the same location and at the same time. In other words, IDPs enable remote and asynchronous ways of working. This allows organizations to offer remote working and to create distributed teams, ensuring a better work-life balance for the team members.
All of this brings happier employees that will be more productive and satisfied with their work, which will reduce turnover.

Conclusion

The need for a more sustainable world is gaining momentum and involves every industry, including software development. No wonder Gartner included sustainable technology in the “Top 10 Strategic Technology Trends for 2023,” along with platform engineering. As platform engineering is being adopted by more and more organizations to accelerate software development, sustainability has to be addressed.

An internal developer platform can help developers build more sustainable software. Platform engineering can help in all of these aspects, improving not only the developer experience but also the organization and the environment. This is just the beginning of the sustainable path platforms should follow, but the outlook is very encouraging.
April 11, 2023
Puppet Makes IT Automation More Accessible

Puppet today launched two efforts to make its IT automation platforms accessible to a broader range of IT teams.

The Puppet Enterprise Platform integrates the company’s flagship Puppet Enterprise offering with Relay by Puppet, the company’s existing event-driven automation platform.

At the same time, Puppet announced Puppet Connect, a self-service platform for executing tasks on-premises or in the cloud without requiring any agent software to be installed, managed or updated. Based on the open source Bolt orchestration engine, Puppet Connect is a commercial platform scheduled to be available in early 2021.

Puppet CTO Deepak Giridharagopal said Puppet Enterprise Platform is unique in that it allows IT teams with varying levels of skill and expertise to employ the same platform to automate IT operations. It is the first to support task-driven orchestration, model-driven infrastructure as code and application programming interface (API) event-driven workflows in a single platform, he said, noting as such, Puppet Enterprise Platform is designed to enable organizations to better approach IT management as a team sport.

The company is also providing access to thousands of pre-built automation modules and hundreds of API integrations for Puppet Enterprise Platform that can be downloaded from the Puppet Forge repository.

Puppet Connect, meanwhile, is designed to make it easier for IT teams to build self-service portals more consistently. IT teams regularly build self-service portals to automate tasks to help reduce the number of help desk requests generated by end users. According to the “2020 State of DevOps Report” published by Puppet, 63% of respondents have at least one self-service internal platform. Puppet Connect streamlines the process of creating a self-service framework using the orchestration capabilities of Bolt, said Giridharagopal.

Interest in automating IT tasks has risen sharply as IT teams cope with more end users working from home to limit the spread of COVID-19. In some cases, the size of the IT staff has been reduced because of the economic downturn brought on by the pandemic. Regardless of motivation, IT staffs that have varying levels of skills all need to find an easier way to automate tasks. The Puppet platforms provide a more accessible layer of abstraction to enable IT teams to automate tasks spanning multiple platforms, noted Giridharagopal.

It may be a while before IT teams of all sizes automate every IT task; however, the more those tasks become automated the further down the path they move toward embracing best DevOps practices. Those organizations may not deliberately start out to embrace DevOps, but once an IT team automates its first set of tasks it’s not long before other manual tasks become automated.

Of course, there’s no shortage of IT automation frameworks to standardize on. At this point, the more important thing for many IT teams is to simply get started at a time when no one knows for sure how many years the impact of the COVID-19 pandemic will be felt.

November 17, 2020