When a production system fails, one of the first questions we usually ask is: What broke? Was it a storage service, a bad deployment, a memory leak, a dependency or a network problem?
That is a useful place to start, but it can also narrow the investigation too early. Some production failures happen even when every component behaves as designed; the problem appears only in their interaction. Systems-safety methods give us a useful way to reason about this class of failure, and I ran into a good example in a large-scale provisioning system.
When Everything Works and the System Still Fails
The system created storage partitions on demand and, as part of that process, applied an access-control policy to every new partition. The policy logic lived in a shared library that already had another user: An operator CLI that engineers used to apply policies manually, one storage instance at a time.
For the CLI, the design worked well. Each invocation read the current policy state of one storage instance into memory, calculated the required changes and applied them. As the CLI operated on one instance at a time, the amount of work and the memory footprint were reasonably bounded.
The automated provisioning service later reused the same library, but with a very different usage pattern. It could create many partitions at once and apply policies concurrently so that a single request could fan out into many simultaneous calls to the policy library. This worked for ordinary requests and had the obvious benefit of improving throughput.
The problem only became visible when a much larger request arrived. The provisioning service accepted it and started processing many policy operations in parallel. Each operation independently loaded the state it needed from the storage service, and the combined memory footprint kept growing until the provisioning service exhausted its memory limit.
At first, this looked like a normal resource problem. Maybe the policy library was inefficient, concurrency was too high or the service simply needed more memory.
But the interesting part was that every component was doing something reasonable. The policy library worked as designed, the provisioning service used concurrency to improve throughput and the requester submitted a valid request. The failure came from how those behaviors combined.
The Hidden Assumption
The policy library had an implicit assumption built into its original design: Its resource cost was safe because callers would use it in a bounded way. That assumption was true for the CLI, but it was no longer true once the same operation was used concurrently by the provisioning service.
The library had also been optimized shortly before the incident, but those improvements had been tested mainly against the original serial usage pattern. That told us little about the aggregate memory cost of many concurrent calls. A locally bounded operation had become an unbounded workload at the system level.
The above figure illustrates how a bounded policy operation becomes an unbounded system workload: Serial use keeps the resource cost predictable, while concurrent fan-out multiplies the same per-call cost across the provisioning service.
The original assumption had never been turned into an explicit system constraint. Nothing limited the size of a provisioning request based on its expected resource cost, and nothing tied the amount of concurrent work to the memory pressure created by those policy operations.
Make the Constraint Explicit
What the design needed was a system-level rule: The provisioning service must not accept or execute more work than it can process within its resource budget.
A fixed concurrency limit can help, but the safe level may change with request size, state size, traffic or available memory. The important part is that the provisioning service can account for resource pressure when deciding how much new work to accept.
Those signals might come from internal counters, runtime metrics or monitoring. What matters is not where the metrics live, but whether the provisioning logic actually uses them. In the original design, engineers could see the pressure after it appeared, but the service did not use that feedback to limit new work.
This is broader than one provisioning service. We often review components independently: Can this function handle errors? Can this service recover from a failed dependency? Can this API retry safely? Those questions are useful, but they miss a system-level one: What happens when several components behave correctly at the same time?
A function that is safe once may not be safe when it runs hundreds of times concurrently. A retry that helps one request may become harmful at scale. Reliability depends on the behavior of the whole system, not only on whether each component works as designed.
Ask About Interactions, not Only Failures
Traditional reliability analysis often starts from component failures: What if this service crashes, this call times out or this storage service becomes unavailable? Systems-safety methods such as STPA take a broader view. STPA describes a system using three main ideas:
Control actions — what one component tells another to do
Feedback — how the controller learns what happened
Constraints — rules that must hold for the system to remain safe
That way of thinking fits this incident well. The provisioning service was the controller accepting and creating work, while the policy library and storage service performed the requested operations.
Memory usage, concurrency, outstanding work and queue depth were the signals the provisioning service needed to consider before accepting more work. The problem was that the provisioning logic did not use resource pressure as feedback.
The above figure explains that the provisioning service ultimately runs out of memory because accepted work is not constrained by its resource budget and resource pressure is missing from the control loop.
You do not need to run a formal STPA exercise every time to benefit from this perspective. During a design review, architecture review or incident follow-up, a few questions are often enough to expose assumptions that would otherwise remain hidden:
What assumptions does this component make about its callers?
What happens if this operation runs hundreds of times concurrently?
Can several individually safe actions combine into an unsafe system state?
What limits the amount of work the system can accept before it starts doing that work?
What feedback does the controller need to enforce that limit?
If we had stopped at ‘the service ran out of memory’, we would have described the symptom correctly and still missed the more useful lesson. The system allowed a valid request to create an aggregate workload that none of the individual components understood or controlled.
Sometimes a component really is broken. But some of the more dangerous reliability failures happen when every component is doing exactly what it was designed to do. In those cases, the problem is not in the individual parts, but in the system they form together.
The New York Times recently ran a long feature with a dramatic headline declaring the end of computer programming as we know it. The article describes developers at Google, Amazon, startups and elsewhere who now spend their days talking to AI agents instead of writing code by hand. Some barely type at all. They describe what they want, review a plan, approve it, and the machines do the rest.
If you learned to code in the old days, this sounds somewhere between science fiction and heresy.
The piece follows engineers who once measured progress in painstaking increments. A good day might produce a few dozen lines of production code after hours of thinking, debugging, testing and rewriting. That was not laziness. That was reality. Every line had consequences. Every change could break something critical. Careful work was the job.
Now those same engineers can generate thousands of lines in minutes. Entire features appear before lunch. Systems that once took weeks can be assembled in hours. One founder quoted in the article says productivity gains feel like ten to one, twenty to one, even a hundred to one.
That part is not hype. Anyone actually using these tools knows it is real.
But the article frames this as the end of programming. That is not quite right. Programming is not disappearing. It is being industrialized.
We have seen this movie before.
Assembly gave way to higher-level languages. Manual memory management gave way to garbage collection. Libraries replaced handwritten routines. Open source replaced writing everything from scratch. Each step removed drudgery and raised abstraction. Each time, old hands warned that real programming was dying. Each time, software kept getting bigger, more complex and more important.
AI is not another step on that ladder. It is an elevator.
Code production is no longer constrained by human typing speed or cognitive bandwidth. Machines can produce far more code than any team of humans ever could. The marginal cost of trying an idea approaches zero. If something does not work, you throw it away and generate another version.
Scarcity is gone.
And when scarcity disappears, behavior changes everywhere.
Startups can build products with tiny teams. Enterprises can modernize ancient systems faster. Individuals with no formal training can create software that once required a full department. The Times article even describes a print shop manager who built a working application without understanding the code it produced.
That is not the end of programming. That is the democratization of it.
But there is a catch. Software is not valuable because code exists. It is valuable because systems behave correctly, reliably and securely over time. AI can generate code far faster than humans can understand, validate, integrate or maintain it.
So the bottleneck moves.
For decades, the limiting factor in software development was producing code. Now the limiting factor is controlling what that code does once it exists.
Welcome to the governance era.
Engineers are becoming supervisors of automated production. The job is shifting from writing instructions to defining constraints. Instead of crafting every line, developers decide what should exist, what must never exist, what risks are acceptable, how systems fit together and how failures are contained.
In other words, less construction worker, more building inspector.
Large companies already live in this world. In mature codebases, engineers spend much of their time reviewing changes, managing dependencies, planning releases and responding to incidents. The Times article notes that even at Google, where AI is deeply integrated, productivity gains are meaningful but not explosive because safety and integration still dominate. You cannot just spray new code into systems that billions of people rely on.
Startups, on the other hand, are in greenfield territory. Fewer constraints, fewer legacy systems, lower blast radius. That is why they are seeing the wild gains.
But even there, cheap code can create expensive problems. Technical debt accumulates faster. Security vulnerabilities multiply. Systems become harder to reason about. Velocity can outrun understanding.
Cheap code is like cheap plastic. Useful everywhere, dangerous in bulk.
There is also a human dimension the article touches on but does not fully explore. Junior engineers traditionally learned by doing the tedious work. Writing tests. Fixing small bugs. Implementing well-defined features. That apprenticeship path built intuition. If AI handles that work first, where does the next generation gain experience?
We may be trading short-term productivity for long-term expertise.
None of this means programmers are obsolete. If anything, experienced engineers become more valuable. When machines produce endless output, judgment becomes the scarce resource. Knowing what to build, what to ignore, what will fail in production and what will age well matters more than ever.
The new elite skill is not typing code. It is steering complexity.
For years, we told young people to learn to code. That was good advice in a world where code production was the bottleneck. In the emerging world, the better advice might be to learn systems thinking, architecture, security, operations and critical evaluation. The machines can generate syntax. They cannot take responsibility.
The Times article describes developers who feel both exhilarated and uneasy. That mix is telling. They sense they have been handed enormous power without clear rules for using it. Productivity gains feel intoxicating, but the long-term implications are murky.
This is not the end of programming as we know it. It is the end of programming as manual labor.
Software development is becoming less about making code and more about managing the consequences of code. The organizations that thrive will not be the ones that generate the most lines. They will be the ones that keep systems stable, secure and comprehensible despite the flood.
When code becomes cheap, engineering becomes governance.
And governance, unlike typing, cannot be automated away.
Serverless architecture is projected to grow more than 15% annually for the next five years. The rise of serverless APIs could translate into higher quality code, greater productivity and even happier developers; but the transition can be rocky if it is not managed well.
A high-quality serverless application requires investment in two areas:
● Application design: Follow serverless architecture best practices for each application layer, including infrastructure, data and client API layers.
● Tools & practices: Equip your developers with tools and practices built for modern cloud and serverless architecture, especially ephemeral architecture.
When done right, serverless applications are lightweight and flexible. Developers will have the most success when their tools deliver the same.
Learn Your Way Around the Serverless Ecosystem
Realizing the benefits of serverless architecture and avoiding its pitfalls starts with understanding what it is and how it fits into the cloud computing ecosystem.
Below is a glossary of some important (and interconnected) terms related to serverless architecture. If you need more depth on cloud computing, check out the Cloud Native Computing Foundation (CNCF)’s glossary and courses.
Cloud Architecture
● An umbrella term for technology that allows users to run full-stack applications on remote server networks, replacing on-premise infrastructure.
● Most cloud applications use immutable infrastructure.
Ephemeral Environments
● A developer tool that provides temporary, simulated prod-like environments available on demand for feature development, debugging and testing.
● A SaaS solution for IaaS and FaaS developers allows rapid creation of cloud-like environments without allocating cloud resources.
Function as a Service (FaaS)
● The central technology in serverless architectures is a refinement of IaaS.
● Containers run single functions and deploy in response to triggers from application code, sometimes called event-based compute.
● Highly scalable, true ‘pay-as-you-go’ cloud capacity; configuration is declarative.
Infrastructure as a Service (IaaS)
● A common model of cloud architecture, sometimes called backend as a service (BaaS).
● Replaces on-premise infrastructure like servers and firewalls with managed services.
● Often used with conventional development tools like IDEs and data scrapers.
Immutable Infrastructure
● Infrastructure that can’t be changed once deployed. Often used in containers or virtual machines.
● More secure and reliable than conventional infrastructure.
● Closely connected to infrastructure as code (IaC). Configuration is declarative and checked into Git version control.
Managed Services
● Components of architecture or infrastructure (e.g., data warehousing, CRM, network monitoring, security services) provided by a third party, usually via API.
● Can be used with on-premise or cloud servers.
Platform as a Service (PaaS)
● A comprehensive model of cloud infrastructure that hosts your development environment and testing tools.
● Combines features of SaaS and IaaS.
Serverless Architecture
● Application design that uses FaaS for multiple components of architecture, including content delivery, data storage, message queues and API gateways.
● Segments of cloud resources are allocated at the function level rather than allocating a whole cloud server to an application.
Software as a Service (SaaS)
● A subset of managed services, usually cloud-hosted.
● Full-scale applications and tools available over an API.
Why Go Serverless?
While serverless architecture isn’t appropriate for every API or application, it has a broader range of use cases than you might think. Done right, serverless can offer multiple benefits such as enhanced scalability, simpler operations, better developer productivity, faster iteration and reduced costs. Serverless applications can also be quick to build. This is a good match for APIs since many are spun up quickly for short-term use, especially in microservices.
Reasons to like serverless architecture go deeper than use cases and basic efficiency. In two meaningful ways, serverless design gives developers greater control over their work and puts the focus back on high-quality, functional code.
Declarative Code is Good, Clean Fun for Developers
In the glossary above, you may have noticed ‘declarative’ popping up a couple of times — the alternative is ‘imperative’, a term you might hear developers use with some disdain.
These terms often describe two approaches to writing code. In imperative code, developers spell out, step by step, what the computer should do. In declarative programming, developers tell the system their desired end result and let the compiler determine the best way to get there.
You can view infrastructure similarly. Conventional on-premise infrastructure is imperative — you manually set parameters on each piece of equipment and connect machines and services. It takes a lot of expert knowledge and time.
Cloud infrastructure is declarative. You tell your cloud provider details like what services you need and how much traffic you expect, and the provider finds the best way to deliver that infrastructure capacity.
PaaS and FaaS are even more declarative than standard cloud architecture, which starts with a commitment to a specific server capacity and configuring your tenant yourself. PaaS and FaaS don’t require that. Instead, they tell the cloud to run a specific function and return a result, and the cloud service figures out how much capacity you need and how to access it — everything after the function call is up to the cloud provider.
Developers prefer their code as well as their infrastructure to be declarative for similar reasons:
● Less room for bugs: Devs rely on well-tested compiler or cloud server code to execute instructions.
● More energy to write good application code: Infrastructure details don’t take up bandwidth.
● Easier to envision and action new solutions: This is helped by what-you-see-is-what-you-get logic.
● More reliable development and production environments: Infrastructure is outsourced to third-party experts.
In short, declarative infrastructure, especially serverless architecture, makes it easier for everyone to focus on what they do best. DevOps teams can design more efficient systems without fiddling with hardware or worrying about implementation details. Developers get to dive deeper into application logic and write clean code that others can understand. The end result is a better API and a happier team.
Separation of Concerns, Cloud-Style
A second advantage of serverless architecture connects to another basic clean code principle. Think back to your early days in tech. You probably learned about separation of concerns somewhere around week 1 of your technology education — it is that fundamental.
Clean code makes the entire software development lifecycle (SDLC) smoother. You can easily separate and recombine application layers and functions with distinct purposes to serve different use cases. It is a particularly important principle for APIs designed to be flexible and front-end agnostic.
IaC and serverless architecture help achieve the separation of concerns in your infrastructure layer. They reduce connections between your application and infrastructure to discrete programmatic methods. It is hard to change cloud configurations accidentally — any infrastructure changes must be intentional and declarative.
Serverless architecture also helps deliver the benefits of functional programming for your API, making functions more independent. Serverless functions have no side effects, are typically declarative and are easy to combine in new ways to serve different purposes.
It all boils down to control: Immutable architecture means server configurations are a controlled variable, and serverless architecture means greater control over your cloud usage. With fewer variables, developers can translate ideas directly into code and clearly see their impact.
The net impact of a move to serverless architecture can be transformative for your organization. By reducing the need for infrastructure expertise and investment in physical assets, you free up resources to invest in application development. You also allow API and app developers to find more satisfaction and focus on their work; but all of that is only true if you also update your tools and practices to match.
Ephemeral Environments are a Best Practice
To make the most of immutable infra and serverless architecture, you need to modernize your development practices and tools to keep pace. You will find useful guidelines and reference architectures from cloud providers to help design your production API. Choosing tools designed for IaaS like API gateways and rapid API mocks can make your development teams more nimble and efficient.
One of the most useful things to support your teams is to adopt ephemeral environments during development.
Ephemeral environments make earlier phases of the SDLC more compatible with modern cloud practices. An ephemeral environment is a simulation of cloud infrastructure that runs locally so developers can mock, run and debug code under the same conditions as the production application without using cloud resources. Exactly how it does that takes some wizardry — but Blackbird, from Ambassador, actually uses some serverless code to make it happen.
Other names for this practice are production-like test environments and on-demand feature environments. These names suggest the benefits dev teams can gain from using ephemeral environments.
Why Devs Love Ephemeral Environments
At one time or another, every dev probably wished they could just push changes to production — but we have all heard enough horror stories to know better. But even code that has gone through unit testing and code reviews can sometimes cause unexpected chaos in production.
Standard testing and QA processes can feel pointless when you know deployment conditions will differ from your local environment.
Ephemeral environments create mirror images of cloud infrastructure and production data, so pushing to production is less of a leap of faith. If you can run and debug in production-like conditions from the earliest stages of development, you can shorten your entire CI/CD pipeline and get code updates released faster. That is a huge advantage for serverless architecture, where devs can push changes at the function level.
Why DevOps Might Love Them More
Ephemeral environments also offer significant advantages for your DevOps team. For one, they offer a better way to test potential infrastructure changes. Use them to test changes locally, rather than deploying an expensive cloud instance. Ephemeral is particularly useful for performance testing and understanding how your compute load will change, helping make the cost of cloud computing more predictable.
On-demand ephemeral environments also mean DevOps isn’t receiving constant requests from devs. Devs need to spin up prod-like environments quickly, which can add significant unbudgeted work to DevOps teams and disrupt focus on larger projects.
When developers have tools that give them more independence, DevOps gets more done, too.
Create Time and Space for Innovation
When done right, IaC and containerization make web services stable, scalable and iterable; they also reduce the overhead of maintaining on-premise systems. But realizing the ideal isn’t guaranteed.
You can’t get the full benefits if you only modernize production infrastructure. Developers are your most valuable resource, and you will only get meaningful bottom-line improvements if you can enable them to be more productive. Using ephemeral environments like Blackbird makes following best practices easier across your technical organization — a strong choice for any team looking to modernize their practices and maximize their resources.
Choose serverless and IaC if they are a good fit for your API — they will be for many teams. Regardless of architecture, teams should prioritize tools and practices that let people focus on their area of expertise. The right tools let developers write and debug code with fewer distractions, free up DevOps teams to take on meaningful projects and give everyone the space to do better work.
As we close out 2023, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2023.
Given how demand for reliable application performance is always growing, it’s no surprise that many firms build their systems with future expansion in mind. Many industry pundits have pointed to microservices as the best way to future-proof a system—but it might be better to use the term “right-sized services” instead, as there’s no single strategy that works for every firm at every point in time. Rather than thinking about which of the latest trends to adopt as a strategy, teams should instead recognize and focus on the underlying factor of coupling. Systems are more flexible and dynamic if there is loose coupling between the components that are mostly likely to change.
Coupling, in the systems context, refers to how components of a system are connected. Interfaces are necessary for loose coupling. For electrical systems, those interfaces are implemented as physical connectors with pins and sockets and protocols described in terms of voltage levels. The software systems analogy to an electrical connector is an application programming interface (API), which could be implemented as a set of functions or HTTP-based resources, among others. But an interface alone is not enough for loose coupling. That interface must also be somewhat stable, which is to say that a user can count on that interface to be there, with its inputs remaining the same for a long time or, at least, changing predictably.
Why Loose Coupling Matters
When components of a system are loosely coupled, those components can be changed internally without significantly disrupting the rest of the system. Consider a quotidian example: The light bulb. Light bulb sockets have a standard size with standard threads and a standard voltage. The light bulb itself has evolved through various types of materials for the filament to LEDs, all without requiring a change to lamps and light fixtures. Now consider a simple example in software systems: An HTTP server and a datastore, such as a cache, queue or database. In most cases, it would be beneficial to place an interface between the server and datastore that would make changing the implementation of the datastore easy. Once these two components are only loosely coupled, things that are likely to change can do so without requiring significant changes to the other things.
If the components of your system are too closely intertwined, then even the smallest change could wreak havoc somewhere else in the system. You might compare this structure to a group of plants that were grown in proximity. Because all of the stems are intertwined, trying to swap out one plant for a new one will be challenging, as one false move could easily uproot many other plants.
In contrast, in a loosely coupled system, changes can be made with the confidence that when one component experiences an issue, the rest of the system will remain resilient. This confidence results in greater flexibility to make changes, enables shorter time-to-market for new features and products, and potentially affords a competitive advantage.
A corollary benefit of loose coupling is evolvability: When interfaces are located in the places most likely to change, it will be easier for the system to evolve. Interfaces are meant to be stable and opaque so that anything that goes on behind them is generally unknown and irrelevant to a user of that interface. So, as long as interfaces remain stable, the systems behind them are free to change and, therefore, evolve.
For Small Projects, Microservices May Negate Loose Coupling Benefits
While it is true that microservices strategies do support loose coupling, they’re not the only way. Simpler architectural strategies can afford smaller or newer projects the benefits of loose coupling in a more sustainable way, generating less overhead than building up microservices-focused infrastructure.
Architectural choices are as much about the human component of building and operating software systems as they are about technical concerns like scalability and performance. And the human component is where microservices can fall short.
When designing a system, one should distinguish between intentional complexity (where a complex problem rightfully demands a complex solution) and unintentional complexity (where an overly complex solution creates unnecessary challenges). It’s true that firms like Netflix have greatly benefited from microservices-based architectures with intentional complexity. But an up-and-coming startup is not Netflix, and trying to follow in the streaming titan’s footsteps can introduce a great degree of unintentional complexity.
The microservices approach is a way to manage the complexity of a system once it has gotten too large to handle safely. Still, there is a trade-off between complexity and overhead. There is substantial overhead associated with managing a number of services. If adopted prematurely, a microservices approach can result in staff members spending all of their time supporting the tools that support the services and neglecting the actual product.
Two Ways to Move Fast and Remain Nimble
There are at least two options that may be worth considering for any team that wishes to stay nimble by fostering loose coupling and to make progress by avoiding the overhead of maintaining a microservices-based infrastructure.
The first option is to embrace monolithic architecture. While monoliths are often maligned as inflexible and outdated, a properly planned one can support loose coupling and require much less support overhead. The key is to build a code-based interface that can take one component’s inputs and relay them to another. Even if one component of the monolith is altered, such an interface minimizes the impact on any other component, and the system can evolve more easily where interfaces exist.
The second option is to employ a serverless platform provided by a major cloud provider. This approach is much closer to microservices in spirit, but the overhead cost is assumed by the cloud provider. As a result, serverless architecture can afford the benefits of loose coupling with minimal unintentional complexity. It may turn out that serverless is not for you because of other constraints, but as long as you construct your interfaces carefully by focusing on the interactions that are most likely to change in the future, it doesn’t hurt to try serverless first.
Choosing the Best Loose Coupling Strategy
Even if you decide to pursue one of these two strategies for loose coupling, it may not be clear which one to choose. Here are a few factors that may help you determine whether to opt for monolithic or serverless cloud-based architectures:
Cost. If you do not have experience in optimizing utilization with providers like AWS or if you underestimate your resource utilization, serverless architecture can become very expensive.
Prior Knowledge. A monolith will require a little more maintenance and, therefore, is a better fit for companies that have worked with one before. Otherwise, serverless architecture may have more appeal because the provider handles much of the upkeep.
Traffic Patterns. If traffic for your system tends to come in bursts, you may benefit from the use of a serverless platform, which will only activate when it is needed. An “always-on” monolith may be a better fit for use cases with continual traffic.
Stage of the Product. If you’re testing out hypotheses about technical or market viability, then serverless solutions could help you test them more quickly and with less overhead. Serverless solutions are typically quick to build and configure, quick to tear down and they obviate the need to think about infrastructure, even more so than managed compute solutions.
Focus on ‘Right-Sized Services,’ Not ‘Microservices’
The objective is not to decouple everywhere, nor is it for everything to be loosely coupled. Rather, it is to make the system evolvable by strategically choosing where and by how much components are loosely coupled. This is a game of educated guesses and trade-offs. There is no best way, just a least-worst one.
When analysts and industry leaders talk about “microservices” like it is the universal answer for software systems, this only applies to teams building the most mature systems that have the capacity to support the associated overhead. The term “right-sized services” better conveys that, even within a company, each team has different constraints and, therefore, different ways of achieving loose coupling. This may entail the use of microservices, monoliths, serverless platforms or a mixture of these things; but, in any case, teams that embrace loose coupling are better equipped to respond to challenges, gain a competitive advantage and adapt to a constantly changing tech landscape.
Organizations of all sizes are feeling the pressure to introduce more aggressive sustainability measures to combat climate change. For example, most cloud service providers have announced significant climate pledges to either reduce or offset their carbon emissions. But these initiatives can only go far to move the needle—it’s up to individual companies and departments to participate in curbing their climate impact, and DevOps certainly has a role to play in enacting positive change.
One way to design for sustainability is to optimize information technology systems to reduce waste. Enterprise IT typically supports a lot of unnecessary technical debt and perpetuates toilsome tasks that contribute to energy inefficiencies. Removing unused instances and features or idle workloads that aren’t contributing value could be an easy way to reduce waste. Yet other optimizations will require more effort.
I recently chatted with André Christ, co-founder and CEO of LeanIX, to discover what techniques enterprise-level software architects can use to get the most out of their technology modernization efforts while leaving the least carbon footprint. According to Christ, optimizing existing processes, using cloud-native technologies and leveraging value stream management are some tactics architects can take to “do more with less.”
1. Match Technology to Sustainability Targets
First, technology leaders must understand the capabilities of their business that could help contribute to meeting sustainability targets. “This requires understanding and systematic mapping,” said Christ. “It’s really about understanding what you can do as a company to contribute to sustainability in a more holistic way.”
For example, a routing and logistics provider might optimize operations by redesigning its floor plan to maximize efficiency and reduce energy use. Similarly, Christ recommended software professionals take stock of their existing capabilities to determine what should change. This could equate to discovering internal microservices that aid one department and extending them across the entire organization.
Optimizing data transfers is another way to create a leaner IT system. For example, teams could reduce constant polling by eliminating overfetching and underfetching or embracing a new API standard, like GraphQL, said Christ. Cutting out these superfluous calls reduces unneeded energy consumption.
2. Ensure Cloud-Architected, Not Lift-and-Shifted
Cloud migrations can reap significant benefits in terms of scalability and flexibility. But not every cloud migration delivers tangible sustainability benefits. AWS identified six unique strategies for migrating applications into the cloud: Rehosting (known as lift-and-shift), replatforming, repurchasing, refactoring, retiring and retaining. While early gains may be met simply by rehosting applications, replatforming or refactoring them completely with cloud-native components will likely deliver the best optimizations.
“If you only lift-and-shift applications into the cloud, they will keep running just as inefficiently as a virtual machine,” said Christ. Therefore, architects should design for cloud-native whenever possible rather than simply porting legacy applications to the cloud. On the other hand, it may be prudent to replace specific legacy components with SaaS or retire them altogether. As such, it’s crucial to analyze your existing landscape before making a cloud migration to ensure changes are both cost-effective and advancing sustainability goals.
3. Practice Value Stream Management
Practicing value stream management is another potential strategy to cut bloat by ensuring product teams always create more value for the end customer. Planview defined value stream management as such:
“Value Stream Management is a management technique or practice that focuses on increasing the flow of business value from customer request to customer delivery.”
This frame of mind could help engineers make more streamlined decisions about what they’re creating. But to achieve value stream management, organizations will require enhanced visibility into how individual services perform, their dependencies under the hood and their approximate emissions.
The core question you should be asking product teams, said Christ, is how carbon-efficiently are you operating in relation to the number of users you are serving? The answers will highlight which aspects of the business require the most optimization.
4. Tie Sustainability to the Business Bottom Line
Many sustainability pledges are founded on carbon offsetting, in which the company purchases “offsets” of its emissions, usually by planting trees or preserving forests. But, the practice of carbon offsetting has received criticism as being ineffective at stemming rising global temperature levels. So what does an honest climate pledge look like that’s more than greenwash? Well, perhaps one more closely associated with the bottom line.
Taking the idea of value stream management one step further, Christ argued that a real climate pledge would involve reducing carbon footprint by a certain percentage for each new dollar generated. As a business scales and its revenue increases, it should constantly reduce emissions in parallel with this growth. This would equate to assigning a dollar value for every CO2 ton and putting it into the balance sheet, said Christ.
Doing More With Less
Nations are striving to reduce carbon emissions through sweeping legislation. Yet, national goals will be insufficient to meet the target decline of 43% percent by 2030 and net zero by 2050, the United Nations reported. Thus, it will take effort from the private sector, as well.
We’ve identified some steps above to do more with less, yet many current, unsustainable practices are culturally ingrained, making them challenging to change. It will be the work of enterprise software architects to break it down and apply smarter designs to cloud-native applications. This will likely require centralizing views of internal services to enhance visibility and reporting. “You need good visibility into what you’re building and what it’s composed of to reduce the footprint,” said Christ.
My company keeps a sample project running pretty much all of the time. Our current project (multi-target web/Android app using Java and Spring built with Gradle and Ant that we’ll call “Finder” here) is getting a bit long in the tooth, so we have been casting about for a new one over the last several months. From non-work sources, we came upon the idea of multi-targeted Godot with C# as a viable side project. Moving to a completely different set of tools is kind of the point; it makes us stretch, and that is the idea of our ongoing projects. The stated goals for these projects are:
Stretch knowledge about different languages, DevOps tools and platforms.
Keep doing what you all are doing. Too many pundits have “grown out of” hands-on work, and it shows.
Try out new ideas/architectures.
Avoid “kitchen sink.” We can’t have a current test app that includes all available technologies, so keep it streamlined. Approach this as a production app that we use for testing, not as a test app.
That’s why we’re looking to replace our Java/Spring project. We’ve got that pipeline down, we’ve tweaked the app until we are happy with the core and it hasn’t changed in a good while. But change is a part of DevOps, so we’re off.
The Obvious
Let’s take care of the obvious elements in our toolchain right away:
Git. We’ll use GitHub, but it was a given that some form of Git would be holding our code.
Visual Studio Code. We’re doing C#, so this is going to be the most useful option for code editing. The built-in Godot editor is good and will be used, but VS-Code offers C# functionality add-ons.
Antsle is the deployment tool of choice, though this has changed over the years. We’ll use it to set up install sets for clients.
Jenkins or Hudson. We’re going to use this only for test management.
The Less Obvious
There are several constraints we want to (or have to) meet. First, we want to avoid using Jenkins or Ant to control the entire build process, mostly because we’ve done both in previous projects, and most DevOps teams are no longer using these methods for new projects. Second, this is a sample we keep, not an investment, so we want to use open source technology if at all possible, to keep expenses down. Third, we’re using a game engine to do app development. This has some impacts–like most game engines, it is in control at runtime and we need to bundle the engine in the final application.
The Godot Engine. Godot is written as a game platform, but its graphics/cross-platform abilities make it useful for graphics-heavy apps like ours. But we have to include the engine and its config files in the pipeline.
Code Scanning. Static analysis was a late addition to Finder, and it’s in Java. This project is in C# and Godot Script. We have not yet found a tool that supports Godot Script, but C# we can get; it just adds restrictions. So we will use a static analysis tool, but we’re still deciding which.
Build Automation. We’ve used Jenkins, Gradle, Maven and Ant to handle this step at different times in the past, but something newer would be a good idea. We are still evaluating this step. We will likely end up back on one of those tools because they are still used today. Godot is a new element to us, so we are also evaluating what is required to hook into it and make functioning apps in a repeatable process.
Fueled Innovation
We’re having fun—first couple sprints have us running a working prototype, so we’re happy with that. There’s still a lot to implement in the app and in the pipeline, so we’ll no doubt have more to say here soon. We’re talking about putting a server component in—not because the original concept needs it, but because our outgoing app—Finder—had server and client sides, and our need for things like Postman go away without this architecture. And yet, API security, testing, etc. is an important part of the DevOps architecture. As you can see, we’re still iterating.
As to Finder, we normally take down the pipeline for a previous playground project when the new one goes up, but Finder costs us little. There are five containers (three permanent for the app hosting; two for build/test) and some processing power, so we’ll probably keep it since it is using a different set of pipeline tools (and languages—it is mostly Java/JavaScript).
Meanwhile, you all keep rocking it. We’re building an(other) app that no one but us will ever see—but you are keeping the enterprise humming. Kudos to you!
Organizations are embracing the cloud to modernize legacy applications, create more resilient business infrastructures and to support remote work. In fact, Deloitte’s “TMT Predictions 2021: Cloud Migration Trends and Forecast” report suggests that, spurred by this growth, cloud revenues will likely continue to remain above 30% through 2025. While cloud migration is predisposed to embrace an agile development methodology enabled with DevOps, many organizations still leave themselves open to risk when modernizing and migrating applications to the cloud by neglecting to embed security into their development operating model, architectural design and processes.
When migrating to the cloud to modernize applications, DevOps and DevSecOps, as an architecture process, can enable organizations to rethink and rearchitect the security model with a “security by design” approach. If done right, organizations stand to create more secure and agile applications that balance the need for continuous releases in an evolving threat landscape with building customer trust.
DevSecOps Requires a New Operating Model
DevOps is a demonstrated approach to achieving better value, sooner, from IT programs and is seeing new developments in an increasingly distributed work environment. Previously, organizations may have relied on a shift-and-adopt strategy for incremental cloud replatforming. But, today’s rapidly shifting business strategies demand fast reaction time and resilient solutions, as well as flexible and agile solutions supported by DevSecOps to help development and security move at the same pace as the business.
Importantly, DevSecOps requires an integrated team of cross-skilled cloud and cybersecurity specialists working under a shared operating model. A modernization and migration center of excellence (CoE), often led by the digital transformation leader, can help bring together cloud and cyber specialists from across the business with external cloud service providers via a shared responsibility model. Through collaboration, cross-teaming, cross-skilling and a shared operating model across cloud developer and security functions, organizations can achieve better outcomes.
Embracing “Security by Design”
DevSecOps, then, is about more than moving existing security processes earlier into the development process. It is about elevating, embedding and evolving your organization’s risk response, as well as rethinking and rearchitecting the way applications are designed with security as a guiding factor in the architectural decisions. Secure by design means setting up a whole DevSecOps capability to make sure security is embedded early on in application architecture design and then further safeguarded through strategies like segmentation, zero trust and attack surface management.
Before the migration begins, DevSecOps would have developers and security specialists considering data flows, functional requirements and work streams related to workload protection, secure landing zones, operating model, network segmentation, access/controls to be implemented in a zero trust environment, attack surface management and more. An organization, for example, might use microservices to segment application access for internal versus external users to achieve enhanced security through system design.
DevSecOps Requires Process Innovation
DevOps and DevSecOps bring the security and application teams together with shared processes and communication to quickly, securely and efficiently roll out products from concept to production at pace. During the pandemic, teams have pushed their use of communication and collaboration tools to better support distributed teams, including the use of ChatOps, to enable real-time knowledge sharing and knowledge management, increased DevOps automation through incorporation of cloud artificial intelligence (AI)/machine learning (ML) services and reimagining traditional roles to embrace more of an IT-as-a-service operating model.
As DevOps continues to shift left beyond DevSecOps to embrace operations, governance, and customer support, developers will need to work on increasingly integrated teams. These foundational communication leading practices can be valuable as a model for agile working across functions.
Rethinking Cloud Security and Development
DevSecOps can help support cloud migration and agile development programs that require speed and resilience by rethinking the development operating model, architecture approach and collaborative processes to improve security and compliance and enhance customer trust. Security specialists must understand the demands placed on developers for fast migration and continuous releases and developers need to work with cyber professionals collaboratively to make applications designed to be secure and resilient. A security-by-design approach to cloud cyber collaboration can help organizations to rethink and re-engineer their DevSecOps approach. Security specialists should also work toward making their services easily consumable in the DevOps process to enable frictionless security.
Vikram Kunchala, Deloitte’s Cyber Cloud leader and Principal, Deloitte & Touche LLP, and Amod Bavare, Deloitte’s Global Cloud Migration and Modernization leader and Principal, Deloitte Consulting LLP contributed to this article.
The story you’re about to read and its recommendations are real; they contrast with what you have heard so far about digital transformation and DevOps implementation. Its objective is to convince you about that the time to go digital is now; it’s about transforming your business, not only IT; and to do it you’ll have to implement DevOps entirely.
In September, Toys R Us, one of the world’s largest toys store chains, filed for bankruptcy protection. A better understanding of how disruption in the retail industry impacted the company would have helped to prevent its downfall and to reinvent its business model. For example, providing Toys 2.0 (a generic term referring to a new generation of toys built on cloud, mobility, IoT and AI technologies) gaming services could have helped the company survive.
Shifting to Toys 2.0 demands a revolution in thinking, vision, practice and value that business leaders, CIOs and IT vendors haven’t done yet. They must forget the obsolete notion that technology and speed determine business success. Adopting paradigms that stress innovation and solve competitive constraints is the safest path toward fast and durable revenue.
The biggest problem Toys R Us faced was the fact that consulting firms and IT vendors either didn’t understand the challenge as an industry disruption issue—or if they did, they improperly addressed it.
Let’s clarify: To tackle your industry’s disruption, you must adapt your business model including the set of interactions, staff and skills, processes and practices, beliefs and values and infrastructure and tools used across your company to generate revenue. Industry disruption is the challenge, and DevOps through its many assets is the unique business capability companies need to quickly transform and win their digital battles.
The problem, as Alan Shimel, founder, CEO and editor in chief of devopsy.kinsta.cloud, puts it:
“While so many companies have already started to use DevOps or planning to use DevOps and related methodologies soon, there is only a small percentage of companies that are end-to-end DevOps. This means there is still tremendous opportunity in bringing DevOps to market.”
How should have Toys R Us leveraged DevOps to tackle its industry’s disruption? Why did investing in that global digital e-commerce platform prove insufficient? How well did they transform their business model? How would have implementing DevOps 100 percent helped to overcome Amazon and Walmart competition?
The principles, ideas, practices and solutions in this article are part of a case study we developed at ITaaSNow to see how to help companies facing similar challenges.
What is DevOps?
Alan, who is a major contributor to DevOps, is right: Only a small percentage of companies have fully adopted it. DevOps is much more than the automation toolchains consultants and vendors talk about. If they want to help lines of business survive and keep their job, CIOs must get a grasp of DevOps’ business facets.
So, what is DevOps? Why do I think, it would have saved Toys R Us from digital downfall?
DevOps Beyond the Operations Automation
From my experience implementing it, DevOps is a business capability that seeks to optimize business operations (versus the sole IT operations). It provides companies with four competitive advantages needed to meet the digital business competitive challenges:
Enterprise collaboration
Business agility
Short time-to-value
Innovation culture
Figure 1 – DevOps Overview
As the picture shows, DevOps is a set of concepts aggregated around the notion of value stream to form a digital business management capability. It’s structured around four stages that format the company’s interactions and workflows into a powerful digital business management capability. These stages include:
Plan & Measure – addresses strategic watch, service prioritization and revenue generation through innovation portfolio management practices and infrastructure.
Develop & Test – takes advantage of agile and lean practices and infrastructure to ensure innovations design, development and testing.
Release & Deploy – leverages continuous delivery practices and infrastructure to timely deliver the right innovations to the right markets.
Monitor & Improve – stresses continuous improvement to guarantee not only premium customer experience but also customer value.
Four business value catalysts are used to act as revenue generators:
Culture & Values
People & Skills
Methods & Practices
Tools & Infrastructure
As you can see, DevOps transcends application development issues; it covers the entire business life cycle.
DevOps Framework as a Facilitator of DevOps Implementation
So, what is a DevOps Implementation framework?
Concretely speaking, it’s a bundled DevOps platform combining managed PaaS infrastructures and optimized value streams to help companies boost their digital business:
Figure 2 – DevOps Implementation Framework
The platform seeks to help businesses implement DevOps end to end and builds on three principles to guarantee digital profits:
Innovation as the revenue driver – It’s the belief that business can resist the competition imposed by technology companies such as Apple, Google, Amazon and Facebook when they use innovation to keep customers loyal, attract new ones and generate revenue.
Value stream agility as the foundation of market responsiveness – It’s the demonstrated notion that companies are market-responsive when they rely on agile and lean value streams.
Managed cloud as the catalyst of business performance – It’s the proven idea that when companies are focused on their core business, they innovate and offer superior customer experience.
The 5-Step Digital Transformation Blueprint
The 5-Step Digital Transformation Blueprint makes DevOps implementation fast and easy. As illustrated, it’s a comprehensive business transformation framework I developed to help clients with their digital transformation initiative:
Figure 3 – The 5-Step Digital Transformation Blueprint
It’s an integrated and inclusive approach to business transformation that involves the company’s relevant contributors. It structures the transformation effort into a five-stage transformation journey:
Digital strategy: In this step, the transformation team uses tools like the SWOT analysis matrix to help business and IT executives understand the changes in the disrupting industry along with the impacts on the company’s business model.
Value stream: In this step, the transformation team uses tools such as the value stream mapping to identify the required changes and get business and IT executives agree on the company’s future value stream. Interactions, staff, skills, processes, practices and tools are at the heart of the discussions.
Digital platform: In this step, enterprise and solution architects work together to identify and implement the technology and tools selected to support activities in the value stream.
Digital organization: In this step, a panel of business and IT users at the operational and executive levels experiment the new digital business model under actual business conditions. The tested digital business model is then deployed as the new digital organization.
Digital business management: In this operation phase, the value stream and the digital infrastructure are extensively used to run the business.
How Chief Digital Officers Set Up Top Digital Businesses
It would be remiss on my part not to recommend the online digital transformation course that gives DevOps practitioners—executive and operational from both the business and IT—the insights about implementing DevOps as the foundation of digital business.
Analyzing Amazon and Walmart Disruptive Impacts to Spot Guidelines for Reinventing the Company
Many bankruptcies happen because business leaders don’t take the time to understand their industry’s disruption and the impacts on their overall business.
The reality on the ground is, industry analyses and competitive strategies are either overlooked or totally ignored. Understanding the toy industry’s disruptive effects would have helped to identify Toys R Us’s transformation as the safest solutions to avoid bankruptcy.
DevOps implementation framework makes the difference, it sees DevOps as the foundation of digital business success. It considers industry and competitive analyses essential steps of the DevOps transformation initiative. Figure 4 outlines the outcomes that resulted from our industry analysis:
Figure 4 – Toy Sector’s SWOT Analysis
As illustrated, it was shown that Toys R Us should have exploited Amazon’s vulnerabilities and take advantage of the technological opportunities to reinvent itself as a Toys 2.0 gaming service provider. These vulnerabilities include little product differentiation while technological opportunities refer to the new toy generation built on cloud, mobile, IoT and AI technologies.
The company’s strengths—reputation, longtime industry presence, brand equity—combined with the cost savings resulting from managed cloud adoption would have not only simplified its transformation but also challenged Amazon on costs.
Changing the Value Stream to Establish Innovation and Invent the Company’s Digital Business Model
Business leaders don’t take the right transformation decisions because they lack insights into the changing competitive environments. Aggressive marketing and sales campaigns supporting short-term financial gains. Toys R Us failed because it bet on its debt reimbursement at the expense of its business model reinvention.
A DevOps implementation offers a value stream mapping framework that not only facilitates innovation culture implementation but also speeds up digital business models development.
Figure 5 – Value Stream Mapping, Agile, and Lean
The objective of the value stream mapping I performed was to outline the features of work environments that take advantage of DevOps Three Ways to equip the company with the competitive advantages needed to resist Amazon and Walmart. The following lines summarize the outcomes.
Implementing DevOps Flow Principle to Defeat Amazon and Walmart on Innovation: The value stream mapping recommended to leverage Agile and Lean principles to properly implement the flow principle and defeat Amazon on its little product differentiation. The Agile’s features to stress included collaboration and prioritization while focus on people engagement and customer value were the Lean’s benefits to emphasize. Their combination were likely to not only foster innovation culture but also accelerate time-to-market and time-to-value.
Implementing DevOps Feedback Principle to Guarantee Superior Customer Experience: As to the feedback principles, the value stream mapping recommended to use Lean’s value stream assessment and Agile’s review mechanisms. The objective was to institutionalize continuous improvement practices as a means to ensure superior customer experience.
Implementing DevOps Experimentation and Learning as the Foundation of the Innovation Capability: As to the experimentation and learning principle, the recommendation was to take advantage Agile’s prototyping and learning curve-lessening features to institutionalize innovation and risk-taking cultures. They were likely to not only help the company challenge its competitors on innovation but also ensure durable revenue and customer loyalty.
Enabling the Value Streams with Managed PaaS Services
Using technology ingeniously to boost business activities is clearly one of the keys to succeed in the digital business. Today’s DevOps-timid approach focused on speeding up IT operations alone not only is ineffective, but it also doesn’t make sense. It’s the entire value stream that needs optimization, not just IT.
The findings of the digital platform assessment I performed, showed as illustrated, that the global e-commerce platform wasn’t enough to compete against Amazon and Walmart. The company was at the embryonic stage of digital transformation maturity practices:
Figure 6 – Toys R Us Possibly at the Embryonic Stage of Digital Transformation Maturity
The findings suggested that using technology to innovate or creating the operational conditions to cut cost, accelerate time-to-market and be responsive to market opportunities were either overlooked or improperly addressed.
Using a digital platform engineering framework allowed me to design in a record time the following architecture:
Figure 7 – Possible PaaS Platform for Toys R Us
The digital platform would have built on two cloud services supporting the value stream: managed software as a service (SaaS) and platform as a service (PaaS). Let’s discuss them.
How Managed SaaS Services Would Have Helped to Fix the Cost Leadership and Business Agility Issues
SaaS services would have supported the Plan & Measure stage. In addition to supporting varied management activities, it would have provided two fundamental tools that possibly could have enabled the Toys 2.0 strategy: customer relationship management (CRM) and innovation management portfolio. CRM tracks, measures and improves customers’ experience, keeps them loyal and guarantees revenue. Innovation portfolio management identifies innovation ideas, supports their implementation into digital services and monitors generated value. It gives the company opportunities to disrupt and control its industry.
The on-demand and metered billing features of SaaS services would have allowed Toys R Us to get the needed software in record time while enabling time-to-market, time-to-value and responsiveness competitive advantages.
How a Managed PaaS Platform Could Have Helped to Address the Express Delivery Issue
PaaS services would have supported activities covering the Develop & Test, Release & Deploy and Monitor & Improve stages. In addition to providing the technologies needed to implement Toys 2.0 gaming services, it would have built on microservices to set up the automation tool chain and accelerate innovations development and delivery.
Wrapping Up
The recent IT innovations are disrupting several industries and expanding the digital economy. To avoid troubles, businesses must urgently start their digital transformation. Toys R Us, Sears and many others learned it the hard way.
It’s urgent to understand that the digital transformation of your business isn’t only about migrating your IT to the cloud, implementing automation tool chains or deploying artificial intelligence tools across the company. It’s primarily about transforming with a strategic mindset your company’s business model from the ground up.
Digital transformation is a complex process that engages the entire company, and requires specific methodologies, practices and tools. I renew my recommendation to take the online course, “How Chief Digital Officers Course Set Top Digital Businesses“—you’ll get from it the practical vision, approach, processes, practices and tools that’ll help you create the conditions for successful digital business with DevOps.
Snowflakes are beautiful, unique creations. But, let’s keep them in nature. They don’t belong in our server infrastructure. Snowflake servers, where every configuration is just a little different, can introduce unnecessary security vulnerabilities and complications. While common in IT infrastructure, in the DevOps realm, they are gradually becoming ancient history.
At the recent All Day DevOps conference, Erlend Oftedal (@webtonull), with Blank and head of the OWASP Norway chapter, discussed the benefits of immutable infrastructure practices within serverless architectures. Erland walked us through the positive effects on security as well as offered insight on potential problems.
Erlend outlined the progress from physical servers to immutable servers, discussing the benefits of each step and the concerns that lead to the next phase, ending with immutable servers—that is, they can’t be changed. Once set up, they are set. Automated processes coupled with immutability principles can eliminate snowflake servers.
Immutability isn’t a new concept. Some classic examples include: live Linux CDs, internet cafes, schools, libraries, hotel business centers, etc. To set up immutable servers, Erlend outlines these steps:
Find a suitable base image
Build a specific image for the application (dependencies, application, secrets)
Deploy image to server(s)
Repeat steps 2-3 for new versions.
Of course, this immediately raises questions.
What about data? He notes that, because data inherently changes, data has to be stored externally.
How do you deploy changes? Well, you don’t change the server. You take one out and put a new one in. If that one fails, roll back to the previous one.
How do you manage secrets? You can build secrets in, leverage cloud key management or use third-party services.
How do we apply security patches? If truly immutable, you deploy a new server with the patch. However, you can have a semi-immutable server that allows for automatically installing patches but still doesn’t allow for logins or manual changes.
Speaking of security, what are the security advantages? For starters, if an attacker is in your server, it will be thrown out when you change servers, although it may come back if the vulnerability isn’t patched. It also allows for very specialized images by removing all unneeded packages. With these specialized images, you can more easily audit and monitor the system for unexpected file changes, logins and connections.
Erlend discussed containers because they are a step closer to a serverless world. Containers isolate processes and all run on the same OS. Of course, you can make them immutable, but, whether they are or not, there are some key security best practices you’ll want to consider:
Don’t run as root inside container
Implement user namespaces so the root account within the container is separated from the root account outside the container
Utilize the least privileges necessary for capabilities and resources
Keep them up to date
Automate scanning using tools such as Sonatype Nexus, Twistlock, Clair or Aqua.
Separate untrusted and trusted containers
Erlend also dove into serverless architecture. While there is debate over what exactly qualifies as serverless, there are five principles that Erlend highlighted:
Use a compute service to execute code on demand
Write single-purpose, stateless functions
Design push-based, event-driven pipelines
Create thicker, more powerful front ends
Embrace third-party services
Migrating to immutable, serverless infrastructure is a topic to be discussed well beyond this blog post and even Erlend’s session. Whatever you do in the future, though, take his parting words to heart: “The cloud can solve many of your problems, but in the end you cannot transfer your responsibility for security.”
Erlend’s session was packed with information. If this piqued your interest, you should watch his complete session in his full All Day DevOps conference session (just 30 minutes). The other 56 presentations from the All Day DevOps Conference are also available online, free of charge here.
The word “legacy” evokes mixed reactions from the IT and business crowds alike. For those who depend on legacy systems such as the zSeries and iSeries platforms, it is a way to deliver stable and reliable applications into production. Business stakeholders who may not know the intricacies of the technology often take a view of, “Why fix it when it ain’t broken?”
On the other hand, for those who haven’t experienced the joys and pains of legacy systems , legacy world will continue to defy their logic and assumptions because their view of the world is shaped by the open system world which has completely different characteristics: being more malleable to change and easy to maintain etc. Hence there is this great divide between the legacy and new worlds. Add Agile and DevOps to the equation with the lofty goals of increased release frequency and reduced lead times, we end up with a divide that is too far to bridge.
The likes of Gartner have come up with constructs such as two-speed delivery to help deal with this legacy challenge. In my view, there is a problem with the idea of dual-speed because it is based on a wrong analogy to start with — that of allowing vehicles to move at different speeds depending on whether you are on the motorway or a city road. That analogy works well for traffic management scenarios but not for complex IT architectures we see in the enterprise. The dependencies across the systems in the enterprise landscape are just too intricate and too tightly coupled to be defined using such a binary logic. In my experience, I find that many Agile and DevOps transformations hit the roadblock and never achieve the desired levels of productivity and effectiveness because of an oversimplified view of these complex dependencies across legacy and open systems.
The three biggest challenges that prevent a legacy DevOps transformation are mindset problems, architectural constraints and tooling limitations.
The Mindset Problem
“There are no legacy systems, just legacy thinking.” That’s a great quote I heard recently from an engineer in a large organization trying to adopt DevOps at enterprise scale. The initial resistance from teams to move into DevOps ways is due to reluctance to change, emerging from an inertia of doing things a certain way for years if not decades. The mindset problem is as much a problem for those outside the legacy world as those inside. The “outsiders”—typically leadership, management team and consultants—tend to oversimplify the problem and, too often, rely on examples from the open systems world. This just alienates the incumbent legacy teams and makes them even more skeptical of DevOps driven change. The outsiders need to have a genuine appreciation of the challenges with legacy architecture and need to work in collaboration with the legacy teams if they really want to transform their organization.
Architectural Constraints
Legacy systems have certain characteristics such as being static, tightly coupled and monolithic blocks of procedural code. These systems were not built for the modern Agile workflows that focus on incremental and iterative deliveries. A significant majority of these systems are written to operate in batch mode.Testing the systems requires cycling through multiple business days, which simply limits the level of agility you want to introduce in the organization. I am currently working with a legacy engineering team in the financial services space where you have to cycle through a minimum of 20 business days to simulate even 70 percent of the test scenarios. However, there are a number of solution options to overcome the architectural challenges; they range from killing applications to migrating to modern platforms or re-engineering legacy code. Not surprisingly, the right approach depends very much on your context.
Tooling
Tooling in the legacy world to support a DevOps ways of delivery has always been a challenge. But it is fast changing now, thanks to a number of startups and even old-school IT product companies investing in this space. If you consider zSeries as an example, there are better IDEs (integrated development environments) such as RDz that can replace traditional code editors typically based on “green screens.” These IDEs provide great features such as better navigation, debugging, split views and searches on the legacy code. One of the CIOs I worked with made it clear to his team that he wanted to get rid of “green” screens from the IT floor. There are tools such as SmartBear Collaborator that support a great peer review experience and help bring one of the original XP practice of pair programming to the legacy world. There are tools such as endeavor from CA that promise automated build and deployments in the z-series world. Similarly, there are tools that support test automation of mainframe applications and tools for code coverage and code quality checks on legacy programming languages like COBOL. XATester looks like a promising tool for unit test automation from a Danish startup called Xact.
While this write-up is not intended to be a compendium of all DevOps and engineering tools, the central message is that there are a number of DevOps tools emerging on the legacy landscape which can help your legacy delivery teams get on to the DevOps and Agile journey.
What is more important is for organizations to realize that many of these legacy systems will remain part of their IT landscape for the next foreseeable future and it is time to invest in these core systems. They may not look as appealing as the digital platforms, but they are fundamental for businesses and you cannot ignore them on your DevOps journey. DevOps in the legacy world is not going to be smooth, but it is certainly not a mission impossible!
