devopsy.kinsta.cloud

Tag: vendor lock-in

Cloudflare Brings the Deno Team In to Make Self-Hosted Workers Practical
Cloudflare is bringing the entire Deno team on board. The goal is narrow and practical: Make it easy to run the Workers programming model on your own infrastructure, not just on Cloudflare’s network.

Ryan Dahl, who created Node.js and later Deno, will lead the effort with Deno co-founder Bert Belder. Their job is to make self-hosted workerd a first-class, supported way to build and run Workers apps.

The plan pulls together two open-source projects. Workerd is the runtime behind Cloudflare Workers. Celld is Deno’s self-hosted implementation of Workers and Durable Objects, released in August. Cloudflare plans to merge code and ideas from celld back into workerd.

What Changes for Deno Users

For DevOps teams, the bigger story may be what’s winding down. Deno is putting its future work into the shared platform instead of maintaining its own runtime and hosting service. Here’s the timeline Dahl laid out:
- The Deno runtime gets one more year of monthly releases with bug fixes and security patches. After that, Deno ends development. The code stays open source, and others can pick it up.
- Deno Deploy runs for six more months, then shuts down. Paying customers get migration support to Cloudflare Workers.
- JSR, the TypeScript-first package registry, keeps running. Its infrastructure moves to Cloudflare.
- Work on rusty_v8 continues, intending to integrate it into workerd.
Dahl called it a consequential change for people who built on Deno. That’s fair. Six months isn’t long to move production workloads off a hosting platform. Teams on Deploy should start their inventory now, not in month five.

The Gap Celld Fills

Cloudflare open-sourced workerd in 2022. It’s the same code Cloudflare runs in production, and some former customers have used it to leave. But it has a big hole. Durable Objects only run as a single instance. That works for local testing. It doesn’t scale.

Cloudflare’s Kenton Varda was candid about why. The company’s production routing for Durable Objects is built for hundreds of locations and leans on outside services run by a team of SREs. No self-hoster wants that. Varda also admitted Cloudflare never built the tooling and adjacent services that would make workerd practical outside its network.

Celld takes the opposite approach. It’s a single Rust binary with object storage as its only external dependency. You run many celld instances against one bucket. Each Durable Object acts like a small, addressable server with its own SQLite database, single-threaded JavaScript, and WebSocket support.

Celld is still alpha, though. Its developers warn it isn’t safe for multi-tenant setups running untrusted code. So the merge into workerd is what will matter for most production teams.

The Lock-in Question

Varda used the announcement to push back on a common complaint: That Workers is built differently to trap customers. He argued the reverse. Cloudflare open-sourced workerd because large customers such as Shopify wouldn’t commit to Workers for Platforms without an exit path. Some customers have used that exit. Varda’s view is that an escape hatch is good business.

Not everyone will read it that way. Celld was, in part, an independent check on Cloudflare. Deno published cost estimates claiming big savings over Cloudflare for always-on workloads, which Cloudflare disputed. Now the most credible independent implementation sits inside Cloudflare.

“What to watch is parity,” said Mitch Ashley, vice president and practice lead for CIO & Technology Buyers and Software Lifecycle Engineering at The Futurum Group. “If self-hosted workerd handles state and routing differently from Cloudflare’s network, teams end up running and governing two platforms.”

Platform teams should also watch licensing, governance, and how quickly scalable Durable Objects show up in workerd.

Why It Matters

This continues a consolidation trend among JavaScript runtimes. Anthropic acquired Bun in December 2025. Now Deno has a corporate home, too. Picking a runtime is increasingly tied to a vendor’s broader platform strategy. That deserves a line in your architecture reviews.

AI agents are the other driver. Dahl said agents are what he’s most excited to explore, and why celld focuses on Durable Objects.

“Runtimes are the execution layer for AI agents, which is why platform vendors want to own them,” Ashley said. “Durable Objects give an agent harness state, a persistent connection, and cheap compute in one primitive. A supported self-hosted path lets regulated enterprises keep agent data on their own infrastructure without rewriting the app.”

What DevOps Teams Should Do Now
- Deno Deploy customers: Start migration planning now. Map dependencies, test on Workers, and schedule cutover well before the deadline.
- Deno runtime users: You have a year. Deno’s Node.js compatibility should make a move to Node or Bun easier for many apps. Or watch for a community fork.
- Workers teams: Track workerd releases for distributed Durable Objects. Celld works today, but treat it as experimental.
Cloudflare says more announcements are coming in the next few months. The idea is simple: Write once against Workers primitives and run on Cloudflare or your own servers. Whether it delivers depends on execution. But the effort now has a team with a strong track record of building runtimes.
October 13, 2026
What I Learned Building Cloud-Portable Services Across Multiple Providers

Most teams I’ve worked with start their multi-cloud journey with a clear business case and a tidy architecture diagram. Within weeks, the diagram stops matching reality. The development experience across cloud providers is fragmented in ways that no amount of upfront planning fully prepares you for.

After building services across AWS, GCP, and Alibaba Cloud at several companies, I’ve seen a consistent pattern: teams start these journeys aiming for portability and get derailed by semantics. A 2024 Gartner report found that over 92% of large enterprises now operate in multi-cloud environments, which means the scale of this problem is enormous. Yet the real engineering challenge, reconciling the behavioral differences across cloud services, is far more subtle than most architecture discussions acknowledge.

Where Semantic Differences Actually Bite

Each cloud provider builds services with different assumptions about how operations should behave. Consider something as basic as deleting an object that doesn’t exist. One major provider returns a success response; another returns a 404 error. This might seem trivial until you realize every service team writing cloud-portable code must handle this semantic difference, along with dozens of others, independently.

Pagination tells a similar story. Some platforms provide explicit continuation tokens. Others rely on building cursors from the last document returned. These aren’t bugs to fix, they’re fundamental design decisions baked into each platform. Without abstraction, teams independently implement provider-specific code paths, duplicating effort and creating technical debt that compounds over time.

Why Raw REST APIs Fall Short

Previous attempts at multi-cloud abstraction, including projects like Apache jclouds, took a REST-based approach. The logic was straightforward: REST is the shared layer. But in practice, t cloud providers evolve their APIs constantly, adding features, changing authentication flows, and optimizing performance. REST-based abstractions inevitably lag behind, leaving gaps that grow wider with each provider update.

Building on official provider SDKs changes this dynamic. Cloud vendors invest heavily in their SDKs: they handle request signing, header management, retry logic, timeout handling, and endpoint discovery. Leveraging that work means an abstraction layer can focus on normalization rather than infrastructure plumbing. When providers update their SDKs, those improvements flow through automatically rather than requiring manual catch-up work.

Layered Architecture for Provider Abstraction

The architecture pattern that I’ve found works best separates concerns into three layers. A portable client layer gives developers stable, cloud-neutral APIs. A driver layer handles validation and coordination. Provider-specific implementations sit underneath, normalizing behavior by translating status codes, unifying pagination patterns, and standardizing error handling. Application teams interact only with the top layer; provider complexity stays hidden below.

The approach aligns with the broader shift in how organizations manage infrastructure complexity. Internal developer platform adoption rose from 23% in late 2024 to 27% in 2025, according to industry surveys, reflecting growing recognition that developers need insulation from platform-level details. Multi-cloud SDKs serve a parallel function at the provider level: creating consistent interfaces that let engineers focus on business logic rather than cloud-specific integration work.

The tradeoff between portability and provider-specific capabilities ultimately comes down to cost leverage versus business-critical functionality. A useful framework: roughly 90% of cloud services are commodities (compute, object storage, pub/sub messaging) standardized enough across providers that abstraction makes sense. The remaining 10% represent genuine differentiators. When a provider-specific feature offers a significant cost advantage or competitive edge, teams should embrace native integration rather than forcing it through an abstraction layer. The goal isn’t portability for its own sake; it’s making deliberate choices about where portability matters and where specialized capabilities justify the tradeoff

Testing Across Providers

Ensuring uniform behavior across providers requires rigorous conformance testing. The most effective approach involves writing tests against abstract driver classes, then running the same suite against each provider’s implementation. Any behavioral deviation surfaces immediately. The challenge is running these tests in CI environments where managing live credentials for multiple cloud providers introduces security risks.

One solution involves using tools like WireMock as a forward proxy, recording real HTTP transactions on developer machines with credentials, then replaying them in CI. This enables reliable integration testing without exposing production secrets while still validating actual request and response flows.

What the Industry Still Gets Wrong

Surveys consistently show that vendor lock-in ranks among the top concerns driving multi-cloud adoption, with 86% of enterprises citing flexibility as a primary motivation. But that strategic intent rarely translates into developer tooling. The gap between what cloud architects plan and what development teams can actually execute remains a persistent friction point, and most multi-cloud strategies are written at the infrastructure level, where the conversation is about regions, availability zones, and failover policies. The developer experience rarely enters that discussion until teams are already deep into implementation.

Provider-level abstraction layers help close that gap, but they require deliberate investment. The ability to switch clouds by changing configuration rather than rewriting code sounds simple in a design review. Getting there means confronting semantic differences one operation at a time, building conformance tests that run across every provider, and accepting that some provider-specific features are worth keeping outside the abstraction. The work is unglamorous and detail-heavy. It’s also what separates a multi-cloud strategy that exists on paper from one that works in production.

September 24, 2026
What the Microservices Era Can Teach Us About AI

The most expensive mistake teams are making with AI agents right now is treating them as microservices with a language model bolted on. The approach is understandable; microservices are the architecture most platform engineers know best, agents feel like services that happen to think, and the migration path looks familiar. But the analogy only holds long enough to lead you somewhere painful.

Microservices changed how software gets built. The idea was to help teams work independently, ship faster and scale efficiently. In those days, shipping software meant one giant codebase compiled together, tested together and released on 12- or 18-month cycles. Then the cloud came along, and teams needed to ship in weeks or days. The only way to do it was to split systems apart and let each piece move on its own schedule.

It worked, but the systems underneath stayed simple in one crucial way: they were deterministic. Distributed systems are hard, but the behavior was predictable. A service received a call and returned a result, and when it failed, it failed in milliseconds and threw an error you could see.

That is the assumption agents break. An agent workflow can run for hours or days, touch a dozen systems, and make a non-deterministic decision at step three that you don’t discover was wrong until step forty. Nothing threw an error, nothing lit up red on a dashboard; the process just quietly went the wrong way and kept going. Where a broken microservice pages you, a misbehaving agent sends a confident, wrong result and moves on.

This is why the microservices playbook both applies and doesn’t, and why “agents are just distributed services” is a trap. Agents do certainly build on microservice foundations, and many of the era’s lessons carry straight over. But the practices that were optional when systems were deterministic and short-lived are now mandatory, and even the ones that carry over don’t carry over cleanly. Treat agentic AI as just another microservice, and you’re making an architectural bet that will come due in the next 12 to 18 months.

Here are the lessons the agent era rewrites, and the ones it simply reinforces.

The Lessons That Get Rewritten

1. Durable execution stops being a pattern and becomes the runtime. This is the primary one, and it’s where the microservices instinct fails quietly. In a deterministic world, you could add resilience later, layering retries and queues around services that mostly completed on their own; the process finishing was the default, and durability was just another pattern. That default is gone. When an agent workflow fails five steps into a multi-day run, detecting that something broke is often harder than recovering once you know, and the resilience you meant to add later never gets added. It has to be part of the runtime from the start.

But here’s the part teams miss: it is not the same durability. The durable execution engines built for the microservices era assumed high concurrency and millisecond steps, payment flows, order processing, thousands of shorter workflows multiplexed per worker and agents violate every one of those assumptions. A single step might be a 45-second model call. Another might pause for hours while it waits on a human, and during that wait the process should be able to die and resume cleanly later, because nobody cares whether the resume takes seventeen milliseconds or two seconds. You cannot even replay a step and expect the same result, because the step is non-deterministic by nature. So the answer isn’t to adopt the durability you skipped; it’s that the durability layer itself has to be rebuilt for a workload it was never designed to carry.

2. Security and governance need a new model, not a stronger boundary. Microservices pushed authentication to the edge, a fence around the building that worked when the actors inside were fixed services you deployed. Agents can start other agents at runtime, which raises questions the boundary model simply can’t answer. Does the sub-agent inherit the same permissions? How do you trace what it did on your behalf three steps later? The model that fits is less a fence and more a series of locked doors, each requiring its own credential, with identity and cryptographic attestation following every step of every workflow, so you can prove not just who called, but what acted and what it was allowed to do. Most teams are still guarding the front door of a building that now rearranges its own rooms.

3. Observability has to reach every step, not just the endpoint. Operators need to retain control, and non-determinism makes that harder than it ever was for microservices. When something breaks in a 100-step workflow, an operator needs to see exactly where it broke and why, not just that it failed. Typically, a status indicator on a dashboard was enough when a service either returned or errored, but it is useless when the failure is a plausible-looking wrong turn ten steps back. Distributed tracing was essential for microservices. With agents, you’re no longer debugging whether a call succeeded; you are reconstructing why an autonomous system made the choice it did. That demands full visibility into every step an agent takes, as a first-class property of the platform rather than a log you grep after the fact. It’s the difference between the application performance monitoring (APM) tools of the microservices era and LLM observability tools in the agentic era.

These Three Demands Have a Name

Treated separately, recovery, per-step identity, and per-step visibility look like three infrastructure projects competing; they aren’t. They’re three faces of one capability the agent era requires, and it’s worth naming, because you can’t build toward something you can’t name.

Call it agentic durable execution: a runtime that automatically recovers from failures, carries autonomous AI agents and deterministic workflows to completion, and provides cryptographic, tamper-proof evidence of what happened. The first two answer the durability problem: the process finishes, while the third answers governance and observability. In a non-deterministic system, proving what an agent did and being able to see and trust it are the same requirement. It is not a feature you add to an agent framework; it is the layer the framework should be standing on.

The Lessons That Carry Straight Over

These three lessons didn’t change. They just matter more, because the cost of ignoring them is higher.

4. Complexity still demands a platform. Around 2010, most teams were stitching together virtual machines with Chef and Puppet, trying to build something that resembled a distributed application. Kubernetes won because it gave teams a platform layer that absorbed that complexity instead of asking every team to rebuild it. The same pattern is repeating now, as every organization races to stitch together agent frameworks and rebuild infrastructure a platform should already provide. The stakes are just higher this time, because the failures are harder to see coming.

5. Vendor lock-in is still costly. Committing to a single cloud provider’s agentic stack seems efficient until that provider has a regional outage and stalls every dependent workflow, and data sovereignty rules, especially across the EU, only add more pressure to a single-cloud strategy. The microservices lesson holds: architecture decisions made for short-term convenience almost always turn into long-term constraints.

6. Open standards still win the long game. The cloud-native projects that lasted, Kubernetes chief among them, were built in the open and governed by communities rather than single vendors, which is why they worked across environments instead of locking teams in. The agent world is already showing the same trait, with standards for how agents communicate and authenticate emerging through open governance rather than proprietary roadmaps. Betting against openness didn’t pay off for infrastructure a decade ago, and there’s little reason to expect it will now.

Why the Timeline is Shorter This Time

Organizations that approached microservices thoughtfully ended up with systems that scaled cleanly, while those that rushed are often still paying down that architectural debt years later. The same is playing out with agentic AI, except the debt comes due faster. With microservices, a bad decision surfaced as a broken deploy or a paged engineer, but with agents, non-determinism compounds silently and autonomously across long-running workflows, so the debt accrues in decisions you can’t see until the blast radius is large.

The lessons are already on the table. The teams that will win the agent era will be those who recognized which lessons carried over, which ones didn’t, and built for agentic durable execution before the debt came due.

August 19, 2026
Survey Surfaces Increased Reliance on Open Source Software to Build Apps

A Perforce survey of 712 IT professionals finds that programming languages and frameworks (49%), followed closely by databases and data technologies (46%), DevOps/GitOps/DevSecOps tooling (39%) and cloud and container technologies (38%) are the areas where open source software is most widely adopted.

Conducted in collaboration with the Open Source Initiative (OSI) consortium and the Eclipse Foundation, the survey also finds nearly half (49%) of respondents reporting they have increased use of open source software in the last year, with 21% describing that increase as significant. Nearly half (49%), however, said usage of open source software remained the same in the last year.

Not surprisingly, the primary reason cited for adopting open source software was reduced costs derived from no licensing fee (62%), followed by avoiding vendor lock-in (55%).

Despite that level of adoption, roughly a third of respondents also noted they still struggle with security updates and patches (39%), installations, upgrades, and configurations (30%) and technical support (29%). Nearly half (47%) are also spending more than three quarters of their time on maintaining the open source software that has been deployed, the survey finds.

Matthew Weier O’Phinney, principal product manager for the OpenLogic arm of Perforce, said that suggests many organizations that adopt open source software are still challenged by how best to operationalize it once it’s been installed, especially when it comes to staging current and applying software patches.

On the plus side, regulations such as the Cyber Resilience Act (CRA) and the Digital Operational Resilience Act (DORA) enacted by the European Union will push more organizations toward embracing best DevSecOps practices when installing and maintaining open source software, noted Weier O’Phinney.

Overall, the survey finds well over half of respondents (58%) have deployed open source software in an on-premises IT environment, followed by 32% that have deployed open source software on the Amazon Web Services (AWS) cloud. In comparison, 18% are running open source software on the Microsoft Azure cloud, followed by 11% using Google Cloud Platform (GCP).

The most widely used open source programming languages are JavaScript and Python at 72% each, followed by PHP at 49% and Java at 46%. The most widely used open source application development frameworks are jQuery 31%, Symfony (30%), VueJS (28%), ReactJS (28%) and Laravel (26%), the survey finds.

Finally, the most widely deployed open source operating systems are Ubuntu (55%), Debian (35%) and Alpine Linux (29%).

Despite a wave of cyberattacks against software supply chains targeting open source software, it’s clear that organizations are either as or more dependent on the maintainers of multiple projects. The challenge is that many of these maintainers lack the resources needed to address new vulnerabilities as they are discovered, many of which are now being surfaced by artificial intelligence (AI) tools.

Unfortunately, cybercriminals are becoming more adept at using AI tools to create malicious code that exploits vulnerabilities. The resulting paradox is that organizations find themselves heavily dependent on open source software that may be more fundamentally insecure than ever.

Hopefully, there will come a day when more organizations contribute the resources required to truly secure open source software. In the meantime, DevOps teams remain well-advised to proceed with much care when downloading any type of open source software from a public repository.

April 2, 2026
Case Study: DevOps Key to AWS Cloud Migration

DevOps practices halved the time it took this SaaS company to migrate to AWS

Australia-based Whispir provides communications applications as a service. One of its goals is to get products into users’ hands as soon as possible, and that requires development, operations and product staff working together, said Jonathan Swift, head of product and operations.

Whispir formerly operated from a co-location data center, a model that continued as it expanded internationally. But the time and effort needed to run the systems was not cost-effective for an organization of its size, so the company switched largely to AWS. Governance issues faced by some clients force the company to still run its own systems in certain geographical locations.

In the process, “[We] learned a lot about our app and how to make it cloud-ready,” said Swift. “Cloud isn’t free, cloud isn’t cheap” and is potentially more expensive than placing servers in co-lo data centers if the software isn’t built correctly.

For example, it makes sense to make the most use of a physical server as it is largely a fixed cost. But in the cloud, the fewer resources you use, the less it costs.

Crises Lead to Spikes

On the other hand, a lot of Whispir’s business is crisis-focused (eg, Caltex uses Whispir at its Kurnell oil refinery in Sydney to send emergency messages), and it takes a certain amount of time to spin up additional instances to deal with spikes in message volumes. “That was a real challenge,” said Swift.

Those crisis-communications clients expect Whispir to have sufficient resources to meet sudden demand. That’s an expensive proposition with co-located hardware, whereas Ccloud resources can be scaled up quickly, though not instantaneously.

Seasonality has to be taken into account. One relatively predictable driver of crisis messaging is Australia’s bushfire season. Unfortunately, this overlaps Christmas, which is a peak time for marketing messages. So Whispir plans to run at around 200 percent of its normal baseline capacity during the fire season. This is easily done with cloud infrastructure, which Swift describes as “a beautiful thing” in this context.

Some organizations are scared by multi-tenant environments. The ability to stand up dedicated instances on AWS is a great feature that Whispir can offer, especially to its clients in the financial sector.

The company’s DevOps practices had been focused on the co-location environment, so even though kanban still “works very well for us” after moving to AWS, some previous choices had to be revisited. In particular, new monitoring tools were needed to cope with the elasticity inherent in the cloud and because the focus expanded from the hardware to the whole stack.

Selected tools included the Zabbix open-source monitoring platform and Sumo Logic for trend analysis.

Earlier this year, Whispir moved all of its Australian customers into a “pure AWS” environment. There are some great tools for migration, Swift said, “but it’s not as straightforward or seamless as you might think.” He recommends doubling any initial estimate of the time required for such projects.

There were some nasty surprises: The first month’s bill from AWS was “huge” and it took a couple of months to determine the cause, even with the aid of AWS’s “really beneficial” premium support service.

Security requirements in various jurisdictions had to be taken into consideration, with updates to various procedures to ensure the new cloud environment was compliant. For instance, the procedures required when a physical system was decommissioned were well-understood, but how could they be translated into a cloud environment? One specific example is that an in-house storage device can be destroyed when it reaches the end of its life, but in the cloud it’s largely a matter of relying on the provider’s controls.

Avoid Lock-In

But Swift warned that “the challenge is to not lock yourself in” to a particular cloud provider. For Whispir, which operates a mixed cloud/on-premises environment (no cloud provider offers the right onshore facilities in New Zealand and the hardware that Whispir has installed in Singapore is too new to be replaced by IaaS), that is a requirement rather than a strategic choice.

Whispir would have avoided AWS-specific features that would make the platform easier to use and more difficult to get out of, despite the fact that using provider-specific toolsets can yield significant cost savings.

Whispir is keeping an eye on Azure—”great tools, great partnerships, great ecosystems,” observed Swift—and may migrate to Microsoft’s cloud in around 18 months.

A particular advantage is that Azure Stack can be run on private hardware, which could be useful if Whispir decided to enter a market where Microsoft Azure wasn’t present.

Either way, the intention is to go 100 percent cloud to the extent that is possible.

‘Infrastructure as Code’ Means More Power for Ops

The move to the cloud has made Whispir’s operations people more powerful, he observed. The infrastructure-as-code concept allows for rapid deployment, enables operations to contribute more to product development, reduces risk and enables the company to be more responsive to the market.

Furthermore, dealing with on-premises hardware still very much a waterfall situation, thanks to the delays and dependencies involved. But moving to IaaS means infrastructure issues sit comfortably within the kanban model.

IaaS also means that various network-related tasks simply disappear, but other issues arise in their place. For example, some external services used by Whispir (eg, SMS gateways) require the use of VPN connections from a known IP address, but with cloud infrastructure the best you may be able to do is specify a range of addresses. Whispir’s short-term fix was to route such connections through a specific server in the co-lo center, and in the longer term it found providers with less-restrictive arrangements.

Moving from co-lo to cloud would have been “incredibly hard” without DevOps, Swift said. It would have taken at least twice as long, he noted, and the company wouldn’t be able to run the mixed environment as well as it does.

“There’s no way we would have been so successful” without DevOps, he said.

— Stephen Withers

November 3, 2017
Multi-Cloud: The Danger of Data Lock-In
Enterprises rarely think about cloud portability issues before transferring large amounts of data onto a cloud service provider such as Amazon AWS, Microsoft Azure or Google Cloud Platform. They were likely to ask questions on usage by other players in the industry (“the herd effect”) as well as the reliability of the cloud service and durability of data, among other things. Enterprises are starting to realize that as they store increasing amounts of data onto the cloud, it is easy to import data but difficult to get data out. While it is true that sophisticated users may be able to solve the problem, most run the risk of locking in their databases—the “crown jewels”—with the cloud provider.

A common solution to the cloud portability issue is to develop an abstraction layer. Having an abstraction layer in place between an application and the cloud data service helps by minimizing the amount of code restructuring that would be required should an enterprise wish to migrate to a new cloud service. One example is the Amazon S3 API: The Google Cloud Framework has developed S3-compliant API to make it easy for enterprises to migrate their applications. The open-source community is also working to make it seamless; for example, the Python Boto library supports both the Amazon S3 service as well as the Google Cloud Storage service under the same abstraction layer.

However, while an abstraction layer solves the application migration issue, what about the “data” itself? How easy would it be to migrate swathes of data from one cloud service provider to another? Let us do some simple mathematics on this subject. For example, let us assume you would like to migrate 10TB of bucket data from Amazon S3 to Google Cloud Storage. How long do you think it will take? The answer would surprise you—almost a month, by reasonable estimates! The sheer operational cost of managing the data transfer over a month would dwarf the storage and network bandwidth charges accrued in both cloud providers. It might be cheaper to transfer the contents of an Amazon S3 bucket onto a local hard drive, ship the drive to Google and ask the company to upload the drive contents onto a Google Cloud Storage bucket.

How would someone solve the multi-cloud data lock-in problem? Would you:
- Run copies of applications on multiple clouds and keep data in sync?
- Maintain one cloud as the primary and other clouds as passive secondaries?
- Trust one cloud for Tier 0 applications and other cloud for Tier 2 or Tier 3 applications?
One of the key issues is that of the data transfer time. We use intelligent change capture techniques to minimize the amount of data sent over the WAN links that connect multiple cloud providers. For example, take a version of an Apache Cassandra database running in AWS S3, store the version in Google Cloud Storage and restore the version to an Apache Cassandra database running in Microsoft Azure. This is the future of data protection: Not only does it allow you to protect against user errors and logical corruptions, but it also gives you protection across clouds to provide insurance against lock-in onto a single cloud service provider.

Application Recovery Management: The Rise of Polyglot Persistence

Enterprises are using a variety of different data stores to capture the needs of a varying set of applications and their access patterns. Relational data stores might be the right choice for multi-faceted normalized data, but data stores today have to cater to a varying degree of access patterns—for example, a relational database cannot keep up with the high volume of a social media stream, and, likewise, a NoSQL data store cannot perform joins efficiently on a normalized data set. This term is referred to as “polyglot” persistence, wherein composite applications are sprayed over multiple data stores for varying types of data and methods of data manipulation. In the earlier days, data architects chose a relational data store such as Oracle and then mapped the application to this data store. Today, the landscape has changed significantly: Data architects first classify the data types and expected manipulation methods and then choose the appropriate data store to fit the needs.

Let us use the example in which an enterprise needs to identify all the customers in the past year, their purchase characteristics, the customer acquisition method, the social media comments on the post-purchase experience as well as any support interactions. Remember, that in today’s information age, the processing has to be in real time: it would be very imprudent for a business to react to a negative social media reference months after it happened. As one can guess, multiple sources of data with varying degrees of structure need to be collected and analyzed to meet these basic requirements of the enterprise. This type of problem cannot be solved easily or cost-effectively with one type of database technology. Even though some of the basic information is transactional and probably in a relational data store, the other information is non-relational and will require a few different types of persistence engines: document stores for static images and text collateral, spatial for geo-locating mobile customer interactions and graph for deriving relations between the multi-faceted data described above.

But has data protection kept up? Remember that the data is highly correlated with complex interdependencies between the data stores. A logical error in one data store is likely to propagate to others, so recovery has to span the entire spectrum of polyglot persistence. Today’s data protection is highly siloed with recovery limited to a single type of data store. This is clearly not enough!! What one has to do is to synchronize versioning across multiple data stores according to the needs of the application—this is an application consistent polyglot version. The key advantage of this version is to allow us to recover an entire application rather than sub parts of it.

Conclusion

It is time to rethink and reinvent data protection. The world of enterprise IT is experiencing a massive change as cloud is becoming the de-facto infrastructure of choice and multi-faceted big data is becoming the hallmark of the next generation of applications. Data protection must adapt to these changing environments and deliver the next generation of services to the enterprise.

— Shalabh Goyal
October 4, 2016