devopsy.kinsta.cloud

Author: Bill Doerrfeld

  • Managing GraphQL at Scale

    Managing GraphQL at Scale

    GraphQL, the query language invented by Facebook, has attracted a lot of interest as a means to conveniently access data programmatically. GraphQL structures internal services with a unified schema for data, empowering product engineers to easily grab what they want from many microservices to populate the user interfaces (UIs) they’re creating. By unifying company data into a single graph, GraphQL combines the benefits of SQL with the ubiquity of API access.

    However, GraphQL does introduce some challenges when you’re trying to scale it across a large enterprise and partner ecosystem. For example, since GraphQL channels all requests through a single endpoint, security must carefully segment access to all the underlying infrastructure to avoid data overexposure. Matt DeBergalis, Apollo’s co-founder and CTO, also identified potential issues around changeability and performance when adopting a graph at scale.

    As the use of GraphQL within large organizations grows, development teams will have to overcome these obstacles—especially if companies intend to productize public APIs and reuse the same single graph company-wide. I recently met with DeBergalis for an update on managing GraphQL at scale. According to DeBergalis, we’re still in the early days of the graph revolution. And, future GraphQL maturity will require more rules-based declarative architecture, efficiency improvements and the adaptability to enact agile iteration of graph schemas.

    State of GraphQL Adoption

    So, what is driving the rise of GraphQL? Well, DeBergalis noted three key contributing factors:

    1. Many many microservices. In recent years, there has been an explosion of microservices. These modular services can be challenging to wrangle and are often exposed inconsistently throughout a software ecosystem.
    2. The rise of omnichannel platforms. Product engineers must now port the same data across disparate devices, whether streaming in a browser or on a Peloton screen.
    3. Users expect a hefty, data-rich experience. Expectations are at an all-time high. It’s now expected that an e-commerce shop, for example, will include product recommendations, auto-fill keyword search, pricing by geography and many more advanced features.

    Juggling all these factors creates a “complexity bottleneck,” DeBergalis explained.

    GraphQL responds to such complexity, helping unify the backend to enable product developers to fetch precisely what their client applications require. And, GraphQL adoption took off pretty quickly, particularly among product engineering teams, DeBergalis added. GaphQL plays nicely with familiar languages and frameworks like JavaScript, React and Relay. It’s also a bit different than previous trends, with demand being driven more by the database folks; “query people are really leading the charge,” he said.

    With the right technology stack, developers can add a graph layer between the underlying core APIs and the UI they are building, creating a universal API with a self-documenting schema. “For the developer, the amazing thing about GraphQL is throwing away all this boiler code,” DeBargalis said. GraphQL acts as “a foundation for tomorrow” to help deliver future business requirements, he said.

    With regard to architectural styles for APIs, 94% still use REST and 31% use GraphQL, according to the 2021 State of API Report from Postman. Though this percentage may seem small now, GraphQL adoption is steadily growing among large companies, such as PayPal, Netflix, Shopify, GitHub and Airbnb.

    Three Issues When Scaling GraphQL

    Using GraphQL is like using a mobile phone. You wouldn’t want to use a different phone every time you called a different number, would you? You would rather have all your contacts accessible from the same device. Similarly, GraphQL excels the more functionality is added.

    “Due to the network effect, a graph becomes more interesting as we publish more into it,” said DeBergalis. “What we have found is that GraphQL is most interesting and valuable at scale.” So, what are some inevitable consequences of GraphQL at scale? DeBergalis outlines three main architectural issues.

    Performance

    Performance downsides are one disadvantage that developers often cite when considering GraphQL. GraphQL has no built-in caching features, and the payload query could introduce additional latencies. Mitigating lower-level issues around throughput and reliability will likely require additional tooling from the community, DeBargalis said. This is partially why the Apollo team has begun to embrace Rust in new attempts to make infrastructure for GraphQL routing hyperefficient.

    Changeability

    A second problem enterprises may encounter when scaling GraphQL is embracing agility for such an all-encompassing fabric. “The companies that do GraphQL really well change it all the time—they treat it like a product,” said DeBergalis. This agility is at odds with the traditional REST API approach of building for longevity, which makes that approach extremely averse to breaking change.

    The reality is that when a company embraces a single graph, it may be pushing hundreds of schema changes every day. Different internal teams may maintain their own slice of the graph, defining it differently. Plus, maintenance roles may shift from team to team over time, noted DeBergalis. “If you try to waterfall what your inventory model should look like, you won’t get anywhere,” he said.

    Balancing change and stability will likely require a two-pronged approach to GraphQL. You want to continually iterate the graph with experimental features but also depend on certain functionality to preserve backward compatibility for client integrations. Satisfying both sides will likely require a change management strategy for iterating quickly, as well as enterprise-grade service level agreements (SLAs) to retain a firm bedrock on which to build more permanent data models.

    DeBergalis also described how observability is important for obtaining a clear picture of graph environments. This is one area where GraphQL shines—since clients request precisely what they need, you are never left wondering which methods are fetched but not implemented in production, as could happen with REST API management. According to DeBargalis, tracking fine-grained GraphQL usage with KPIs can help direct how the data model should evolve over time.

    Authorization

    Finally, perhaps the most significant concern surrounding GraphQL at scale is apportioning the appropriate access to users and devices. As a single unified endpoint can connect to a vast array of data behind the scenes, large GraphQL implementations will require a strategy for how that one unified graph can be implemented in parts.

    Without the proper access control, GraphQL could improperly expose personally identifiable information (PII). For example, a FinTech company may have a rule that requires a customer service representative to only have visibility into a customer’s transaction history if the customer has filed a support ticket in the last few days. Or consider data custody—a business working in the EU must comply with regulations on where data can and can’t live.

    Safeguarding GraphQL will require a declarative setting that integrates authorization control and context into GraphQL queries. To implement this, DeBergalis advocated for a command-and-control style. First, you build the schema around the data and problem statement. Next, you integrate rules field-by-field, marking sensitive touchpoints as PII.

    With rules baked into the query, you can have a more solid understanding of the surrounding context. “Authorization is actually a lot better at graph level,” said DeBergalis. “It’s the layer that has semantic understanding.”

    Future of GraphQL Scalability

    “Once you have a graph, you want to use it for everything,” said DeBergalis. The heritage of GraphQL is JavaScript, the language of the product world. And, the graph approach delivers an excellent experience for product engineers. The sheer usability and productivity benefits of GraphQL indicate incredible potential for projected use in the years to come.

    As we’ve discussed, scaling GraphQL across an enterprise will likely require performance improvements, a path for schema adaptability, policies to handle authorization for queries and greater accountability for business metrics. While GraphQL certainly has existing momentum among a subgroup of engineers, there are plenty of opportunities to introduce more training and educational curriculum to spread greater awareness.

    Furthermore, for those coming from a traditional REST API mindset, GraphQL is a bit of a new paradigm that will require some massaging to expose graph endpoints as a productized API. Doing so will likely involve marking select parts of the larger graph to expose to partners. “Starting with an internal GraphQL use case works really well, but productization around GraphQL is coming,” said DeBergalis.

    The value of a unified graph is so strong, and the bottom-up hunger for a graph environment is so intense that increased GraphQL adoption feels like a given. “At this stage, the focus is on expansion,” said DeBergalis. “How do we roll this out to more teams?”

    “We know there’s going to be a graph in every company—not a lot of little ones,” said DeBergalis. “Every company’s going to need a graph.” When that day comes, the architectural challenge will be breaking apart the graph into pieces. “That’s the next big frontier.”

  • DevSecOps Tech Radar Highlights Diverse Tooling Adoption

    DevSecOps Tech Radar Highlights Diverse Tooling Adoption

    The DevSecOps Technology Radar showcases the opinions cloud-native groups have about DevSecOps tools. To review, The Technology Radar is a periodic report from the Cloud Native Computing Foundation (CNCF), a burgeoning host for new open source projects that are becoming pivotal underpinnings to cloud-native architecture at large.

    Just like previous radars, the DevSecOps Technology Radar was assembled using feedback from the CNCF end-user community, which is a group of over 155 organizations that use cloud-native technologies to build and distribute their services, spanning startups and enterprises. The Radar report organizes popular tools into adopt, trial and/or assess status, based on a technology’s active use and its maturity.

    ‘Adopt’ means these tools are stable and come highly recommended for use in production systems.

    ‘Trial’ means users have had some success with these tools, and it is recommended that others consider them.

    ‘Assess’ means these are promising tools that are good for solving specific needs.

    So, why was DevSecOps selected as this Radar’s theme? According to the Radar team, DevSecOps was chosen because it is highly relevant to current development focuses. “We realized that more than half of the tools we were using in CI/CD were actually security-oriented,” said Sergiu Petean, head of DevOps at Allianz Direct. This goes beyond DevOps—”this is definitely an uppercase ‘Sec’,” he said.

    “More than half of our time is spent discussing designs around security,” said Keith Nielsen, director, cloud architecture at Discover Financial Services. Financial services especially must find innovative methods to balance speed-to-market with security. Nielsen also cited the rapid rate of change in this space, which requires companies to continually reevaluate their security posture.

    DevSecOps Radar Results

    So, where did the CNCF end-user community place popular DevSecOps tools? The Radar ended up rating 16 tools across the three different levels:

    • Adopt: Istio, Sonarqube, Artifactory, Hashicorp Vault, Calico/Tigera, Terraform, ArgoCD and OPA.
    • Trial: XRay.
    • Assess: Cilium, Harness, SonaType Nexus, Hashicorp Sentinel, GitHub Actions, Linkerd and Trivy.

    The above results were based on 252 votes from end users across 35 tools, with most respondents coming from large companies across multiple sectors.

    Out of these tools, Terraform collected the highest number of ‘Adopt’ votes. This technology aids the move toward infrastructure-as-code (IaC) with declarative configuration files, which is important for DevSecOps automation. Tied for second place were Hashicorp Vault and Artifactory, indicating that secrets management, access control and continuous updates for security are pressing requirements.

    Other open source tools on the Radar are providing impressive ways to unify security controls. For example, OPA is a universal layer for applying policies that could help unite fragmented authorization across applications. Or, service mesh options like Linkerd and Istio place a sidecar proxy around applications to centrally manage access control, observability and networking across a large microservices ecosystem. Such projects are programmable to fit into the DevSecOps pipeline.

    Analysis

    In general, the radar team was somewhat surprised to see such a high number of security tools in use today. Some tools compete in the same arena, making hunting for the right tool challenging. “The sheer number of products is a bit staggering … what we’re seeing is a highly fractured strategy from different companies,” explained Nielsen, who said he expected more consolidation on capabilities.

    After evaluating the DevSecOps tooling landscape, The Radar team organized their core findings into three main takeaways:

    Usability is compromised at the expense of security. Security features are necessary, but they often come at the cost of usability. The DevSecOps Radar team found that the developer experience for most tools is poor. Many disparate tools must be stitched together to have a truly automated pipeline which is cumbersome to implement. “The tools are getting better, but still there’s not a cohesive, prescriptive way to develop for the cloud,” said Nielsen.

    The pace of change in the security space is rapid. Major cloud providers continue to rapidly release new services, which has led to an explosion in new tools. Simultaneously, the speed of innovation is accelerating within companies, opening up unique security needs. Combined, this means the security landscape is constantly shifting. “Security tools are trying to keep up with the services and how you host them and run them,” Nielsen said. “We definitely weren’t expecting to see so many newcomers in the security space,” Petean agreed. Though DevSecOps is spoiled for choice these days, “it’s always better to have options,” he added.

    Microsegmentation is important but presents a significant challenge. Many great tools enable companies to improve how they segment traffic, such as service mesh tools like Linkerd and Istio. However, implementing cloud-native microsegmentation is bumping against legacy implementations for established companies. Between API gateways, service mesh, edge firewalls and Kubernetes federated firewall functions, DevSecOps has much complexity to deal with, Nielsen said.

    The DevSecOps Technology Radar: In Review

    In recent years, cloud-native technology has placed increasing emphasis on solving the security puzzle. Companies continue to reevaluate their security approach in light of cloud-native strategies, novel vulnerabilities and new compliance requirements. DevSecOps is proving to be a useful, if not necessary, practice to maintain both security and rapid software delivery.

    With new security tools and unique approaches coming out so frequently, it’s easy to have “analysis paralysis.” However, DevSecOps must start somewhere. And, for those who trust the opinions of the CNCF Technology Radar, you now have some helpful insights on where to begin.

  • GraphQL’s Greatest Strength is Also its Greatest Weakness

    GraphQL’s Greatest Strength is Also its Greatest Weakness

    Michael Scott: Why don’t I tell you what my greatest weaknesses are? I work too hard, I care too much and sometimes I can be too invested in my job.
    David Wallace: Okay. And your strengths?
    Michael Scott: Well, my weaknesses are actually strengths.
    David Wallace: Oh. Yes. Very good.

    GraphQL is a new query language that offers a simple way to interact with structured data. The best thing about GraphQL’s design is its convenient data fetching process which enables the requesting party to collect limitless data and perform multiple actions, all with a single API call. This interaction method makes it super useful and very easy to use. However, from a security perspective, its greatest strength could also be its greatest weakness.

    I recently met with Elad Koren, chief product officer, Salt Security, to understand the nuances of GraphQL security. According to Koren, GraphQL affords better usability than REST APIs, yet its bundled request model may compromise systems that don’t employ proper object-level authorization checks. Also, since developers interact with GraphQL services differently than typical web APIs, hackers could avoid detection by typical firewalls, requiring a new approach to security.

    These concerns aren’t dealbreakers for adoption; rather they are a reality check for those using GraphQL in production. Below, we’ll explore some potential weaknesses and look at some general mitigations that providers can take to improve the confidentiality of their GraphQL endpoints.

    What is GraphQL?

    GraphQL was initially developed by Facebook and open sourced in 2015. It’s a query language that makes data retrieval and manipulation with external sources very easy. Instead of programming and linking together complex calls with traditional HTTP API requests, GraphQL enables a client to bundle queries into a single request. This design pattern reduces the under- and over-fetching pattern that REST APIs typically suffer from.

    “You get exactly what you need, not more or less,” described Koren. This makes GraphQL a popular choice for modern database mechanics over REST, as it’s similar to SQL querying.

    Adoption of GraphQL is steadily growing; a 2020 RapidAPI survey found GraphQL usage had quadrupled. GraphQL is now used by 22.5% of API developers. But when it comes to overall API design patterns, REST is still the dominant architectural style used by 93.4% of APIs, found Postman’s 2020 State of API Report.

    GraphQL is promising and has a number of benefits, but remains in an early adoption phase. “It’s still a technology that early adopters are starting to look into—it’s not so widespread,” said Koren. Due to its age, GraphQL security practices are immature, exposing a few potential vulnerabilities for platforms that use GraphQL.

    Unique Security Concerns

    So, what are some unique security concerns in the GraphQL paradigm? Well, according to Koren, it begins with a lack of understanding. “Security is not necessarily top-of-mind most of the time,” he said. “When developers choose to implement GraphQL, they assume that they’re not using APIs anymore and that traditional REST API vulnerabilities do not apply.”

    In fact, the opposite may be true. GraphQL APIs can easily suffer from broken object-level authorization, OWASP’s number-one API vulnerability, just as REST APIs do. But, with greater ease of access to many underlying systems, a single GraphQL query could exfiltrate much more information than a typical API request. If the GraphQL provider does not have granular authorization checks for each particular method and resource, a GraphQL endpoint could suffer from data overexposure.

    Another issue concerns spotting malicious behavior. Typically, when hackers perform reconnaissance on an API, they brute-force it with many requests; trying different namespaces and changing user agents in the process. With REST APIs, this sort of malicious behavior is easier to catch with standard web application firewalls (WAFs) since it’s easier to notice hundreds of peculiar malformed requests. However, with GraphQL, a hacker conducting query batching could completely fly under the radar of typical thresholds.

    This could easily lead to an account takeover, said Koren, who has seen this in production. He described a production payment service that used GraphQL; callers could perform mutations and capabilities they shouldn’t have been allowed to perform, such as pulling up other users’ account information and even initiating transactions.

    It’s easier to limit this type of nefarious traffic with REST APIs by assigning access parameters to each resource and handling authorization checks once at a time. But, this playbook is thrown out the door in GraphQL. “When you’re using a single point to access the data, you may have some false sense of security,” said Koren. In practice, having a single access point requires hard work to retain its integrity. A usable GraphQL frontend may also often mask a convoluted backend made of disparate API styles, Koren explained, making it much harder to correlate user authorizations to specific resources and mutations across various microservices.

    Tips to Mitigate Weaknesses

    So, with these weaknesses in mind, how should GraphQL-based platforms respond? Koren provided the following advice to protect GraphQL APIs:

    • Adopt a security mindset in development: It’s vital to start with a security culture from the beginning. Usability shouldn’t compromise security; ideally, they should complement each other.
    • Ensure proper authorization checks are in place: Whenever an asset is provided to the user, you need to consider whether the user is authorized to access it—especially if they are hitting multiple resources at once.
    • Set a new baseline specific to GraphQL: GraphQL owners need to adapt their perimeter security systems to look for new queues. These validations should consider typical REST API security checks and nuanced thresholds that consider nested queries and batch queries.
    • Know your API inventory: Maintain a clear picture of the underlying architecture of what is beneath the GraphQL endpoint. Verify the data sources and ensure they correctly correspond to user privileges.
    • Increase visibility into API calls: If possible, increase your visibility into queries, mutations, attributes, variables and underlying specifications and how they interact with each other.

    Strengthening GraphQL Endpoints

    API attacks are rapidly increasing. Gartner estimates APIs will become a top threat vector in 2022, and Salt Labs data showcased a shocking 300% growth in malicious API traffic in mid-2021.

    “Like any other innovative tech that comes into the mainstream, it will rise in popularity, but it will not surpass REST in the foreseeable future,” says Koren. Perhaps in three to four years, GraphQL’s current 20% adoption rate will rise to 30% to 50%. While GraphQL is still a relatively niche tool, its use is growing, especially among API-first cloud-native tech companies. And with this adoption will come novel security concerns.

    According to Koren, query batching makes catching malicious attacks using ordinary firewalls tricky. Thus, training a new baseline to discover anomalies unique to GraphQL will be necessary to find and limit hackers while they are still in the reconnaissance phase. Thus, we’ll likely see future security investment go into spotting these atypical GraphQL interactions.

    In general, perhaps the best advice is to treat GraphQL with the same level of intensity you would with any other type of API—if not more. As a single doorway to many underlying systems, GraphQL requires added security forethought and careful application of user authorization.

    With that squared away, hopefully, GraphQL’s greatest strength will remain just that—a great strength.

  • Sustainability No Longer Just a Buzzword for Cloud IT

    Sustainability No Longer Just a Buzzword for Cloud IT

    A recent CloudBolt survey on IT sustainability finds that sustainability is no longer just a buzzword — the market is demanding real change, and it appears that environmental concerns finally are influencing large IT investments. For example, 67% say that a cloud provider’s sustainability initiatives are important in deciding whether to do business with them. A significant portion of organizations are even willing to pay a premium to support green initiatives!

    Below, I’ll review the report to find its major takeaways. We’ll see how sustainability is directing real action in cloud IT and briefly compare how the market perceives current green initiatives among three top cloud providers: AWS, GCP and Azure.

    Sustainability Gains Real-World Traction

    In June and July 2021, CloudBolt polled 256 global CxOs and tech leaders on its Pulse platform, and found that 76% of respondents now consider climate change an existential threat. Not only are IT purchasers factoring compute and cost, but a sustainability conscience is now influencing executives to direct sweeping change. In fact, 79% of respondents indicate their IT departments are expected to help their companies achieve specific sustainability initiatives driven by the board of directors of the C-suite.

    Interestingly, other studies found that a sustainability commitment results in tangible business outcomes. For instance, 88% of the research conducted by Oxford University and Arabesque showed that solid environmental, social and governance (ESG) practices result in better operational performance of firms and higher profits. This study also noted that 80% of the time, a company’s stock price is positively influenced by sustainability practices.

    Climate Conscience Guides Cloud Choices

    Most organizations now consider a cloud vendor’s sustainability initiatives when choosing whether to do business with that provider. A full 78% of respondents agree that the public/private cloud choices they make can materially affect the environment for better or worse. New environmental cloud commitments not only attempt to minimize carbon emissions from their data centers but factor in ecological ramifications and aim to erase companies’ historical carbon footprints, as well.

    Cloud consumers also want to contribute to addressing these green initiatives. An impressive 79% of respondents indicate that they will pay a premium to work with vendors who incorporate sustainability into their business. And 41% of companies are willing to pay a premium of 11 % to 15% of their current bills. This shows that sustainability is now a vital factor within the marketplace, and companies are willing to pay the price to increase their own environmental standing.

    Though this commitment is undoubtedly growing, the report did find that operational requirements such as security, cost and agility, still rank significantly higher than sustainability when comparing cloud providers. Though IT professionals recognize climate change as an existential threat, they still have day jobs and must keep operations moving.

    AWS: King of Green Clouds

    Slightly more than half (51%) of respondents agree that the industry-wide green energy commitment is real and growing among all cloud providers. However, when comparing the top three—Amazon Web Services (AWS), Azure and Google, AWS appears to be emerging as an IT sustainability leader; 41% of respondents perceived AWS as doing the most to address green initiatives.

    AWS is on track to power its entire operations with 100% renewable energy by 2025. With their solar plant investment in April 2021, Amazon became the largest corporate buyer of renewable energy in Europe. Around the globe, AWS has invested in a combination of on-site solar, solar farms and wind farms to power their data centers and Whole Foods divisions.

    To their credit, both Microsoft and Google also committed to climate pledges that affect their cloud footprint. Azure supports a similar initiative to reach 100% renewable energy by 2025, in addition to further investments to improve water, waste disposal and surrounding local ecosystems. Google Cloud’s commitment is to operate on carbon-free energy 24 hours a day by 2030, and the company has already achieved 100% renewable every year since 2017.

    Not only does AWS appear to be doing the most to address its environmental impact, it also retains the most market traction as a cloud leader. The majority of respondents (62%) said that AWS is winning the cloud wars while only 31% said Azure and just 7% said GCP.

    Final Thoughts

    It is heartening to see that ESG standards are having a measurable impact on a company’s cloud allegiances. The market now demands real change and is willing to pay the price for more ethical and sustainable cloud hosting.

    But choosing a cloud provider is only one aspect of reducing IT’s impact. As I’ve covered before, there are many ways IT can curb its climate impact. Such actions include optimizing overall infrastructure, shuttering zombie VMs, avoiding over-provisioning resources, increasing performance visibility, refactoring code and other measures. What’s great is that most of these initiatives have the added benefit of reducing cost to the business.

    The CloudBolt Industry Insights Report, The Truth About IT Sustainability: Paying More Green to Get More Green, surveyed 256 IT leaders on their perception of cloud sustainability. For additional insights, you can pick up a copy here.

  • How do we Bring APIs Into Decentralized Web3?

    How do we Bring APIs Into Decentralized Web3?

    Blockchain-based applications are growing in popularity and underscore the decentralized Web3 movement. Simultaneously, web APIs are helping port data and reusable functionality across application types and are driving microservices development practices. So, how do we bring these two paradigms together?

    There are many cases where a blockchain-based smart contract may require data updates that only a web API request can fulfill. For example, perhaps the smart contract needs to access the latest price of a given stock ticker. Or, perhaps on-chain activities are contingent on the weather at a particular time. The ubiquity of APIs means there are thousands of helpful features available for the task at hand. Yet, integrating them into Web3 applications is tricky.

    Interestingly, the API3 Alliance is making strides to introduce an industry-standard solution to bring APIs to Web3 applications. API3 provides a vendor-neutral standard gateway to apply on top of existing APIs that helps them appeal to decentralized apps. The vendor-neutral API Alliance already consists of many APIs across various verticals and appears to be gaining more traction.

    I recently met with API3’s co-founder, Heikki Vänttinen, to learn more about the challenges of introducing traditional web APIs into blockchain development. According to Vänttinen, whereas previous solutions relied on a third-party middleware or ‘oracle,’ the best way to incorporate an API request and response system into smart contracts is through a node operated by the API provider themselves. Such a system could help blockchain-based applications leverage myriad software-as-a-service solutions, essentially uniting the two worlds.

    The State of APIs in Web3

    The number of APIs is growing exponentially. As companies reinvent themselves with digital transformation, APIs have become the glue for interconnecting the applications we use on a daily basis. When it comes to APIs, there are a growing number of potential use cases for blockchain—the technology is not just relegated to cryptocurrency; rather, it can power a number of scenarios that require smart ledgers.

    Since APIs drive much of the functionality in modern software development, it makes sense that they may be useful in decentralized app development. For example, a blockchain-based decentralized finance (DeFi) app that handles lending and borrowing may want to call an API to collect raw data for a liquidation price, says Vänttinen. Or, a smart contract may wish to acquire the cost of an asset as a conditionality for making a purchase. Hooking into public APIs to check local weather or flight arrival information could affect how smart contracts respond.

    There are many use cases for incorporating APIs into a distributed ledger system. However, “blockchain applications are not able to call APIs natively,” Vänttinen explained. The incompatibility is due to the nature of blockchain technology, said Vänttinen, as it requires consensus from multiple parties to maintain the smart contracts. For this reason, blockchain applications require a middleware piece to connect a blockchain node to an API, relaying data responses from outside of the chain.

    Incorporating APIs Into Blockchain

    As Vänttinen described, previous attempts to connect APIs to Web3 architectures have revolved around a third-party middleware node. Yet, externalizing integration responsibility in this way creates a centralized point of failure, which defeats the purpose of a decentralized environment altogether.

    Instead of relying on such a dependency, Vänttinen advocates for a “first-party oracle approach,” in which API providers operate a layer in front of their APIs. This enables a request-response mechanism for smart contracts, allowing blockchain applications to initiate API requests from on-chain and incorporate the responses into the contracts. This method retains immutability and trustlessness, which is inherent to the distributed ledger.

    One solution enabling such an architecture is Airnode, a Web3 middleware that connects any web API directly to any blockchain application. Companies that have adopted Airnode make up the API3 Alliance, which is “a collective of API providers who have signed up to enable this first-party use of their API for Web3 applications,” said Vänttinen. “Adopting Airnode is a pledge from API companies to make their API blockchain-compatible.”

    The API3 Alliance is one significant initiative aiming to extend the capabilities of smart contracts and blockchain applications. This could enable such environments to call an API to send a text message or email, or grab sports scores, collect maps data and so on. Thus far, participating APIs in the API3 Alliance come from sectors such as agriculture, biotechnology, communications, open banking, insurance, supply chain management and many others. The initiative also accepts applications from API providers that want to join the marketplace.

    Benefits of a Common Web3 Integration Layer

    So, what are the benefits of embracing an industry-standard middleware for web-API-to-blockchain connectivity? Well, according to Vänttinen, it is the best way to enable complex off-chain computation. Enacting intricate computing on the chain is quite resource-intensive, resulting in high transaction fees. By extending particularly complicated calculations externally, blockchain applications could reduce operational overhead. “The industry is long overdue for connection between the real world and blockchain,” said Vänttinen. (By real-world, I assumed he meant other architectural styles that pre-date blockchain.)

    Having a standard implementation could also reduce the development burden for the API provider to support nuanced technologies. Whereas other solutions would require someone in-house to operate blockchain nodes, Airnode is more of a plug-and-play API gateway that could be deployed on AWS as a Lambda function, Vänttinen said. This usability makes it easy for API-based services to appeal to Web3 applications, thus quickly opening a new potential growth segment.

    Of course, Vänttinen admitted, the concept of syncing with off-chain processes is still at a very early stage. There are competing philosophies about how to address it on the market, and some foundational issues have not yet been resolved. For example, most efforts have focused on Read operations to acquire raw data. The next step will be having the middleware ‘oracle’ incorporate off-chain Write capabilities, too. Furthermore, more PubSub-like formats will be important to enable event listeners capable of reactive qualities.

    Final Thoughts

    The Web3 trend is bringing more decentralized, blockchain-based applications to fruition. But, such applications can’t operate in a closed-off world. To experience the full breadth of digital possibilities, they’ll likely need to figure out how to incorporate off-chain computations and mesh with alternative communication software libraries. Naturally, this will require integrations with web APIs, which have become an important method of cloud-based communication.

    The API3 Alliance is “here to propose a solution to the API connectivity problem for Web3,” said Vänttinen. “When a data provider is blockchain-compatible, you can apply the reputation of the data provider in this decentralized data feed … The importance of this becomes more clear when you talk about connecting off-chain services and microservices with smart contracts.”

    Airnode and the surrounding API3 Alliance is one initiative spearheading the connection of APIs and smart contracts. But, this is just one method in a maturing field. What do you think about bridging the two paradigms? Comment on your experience below!

  • The Role of APIs in Mainframe Modernization

    The Role of APIs in Mainframe Modernization

    Mainframes are historically clunky systems. They are incredibly secure and can process impressive calculations, yet they are complex to manage and challenging to extract data from. This reality is at odds with modern cloud-native infrastructure, which emphasizes distributed computing and portable microservices. So, how can existing mainframes adapt to this connected, data-driven economy?

    I recently met with Dr. Alex Heublein, president at Adaptigent, to talk about the state of legacy modernization. According to Heublein, many mainframes are here to stay for the foreseeable future. However, enterprises are continuing to get value from these systems by adding an integration layer on top of them. By using a REST API layer on top of a mainframe, enterprises could finally open up these systems. This could facilitate bidirectional communication with newer cloud-native services, enable phase migration and edge closer to real-time processing.

    State of Mainframe Modernization

    The software industry changes every day. “Nothing in the tech field is permanent,” said Heublein, yet some legacy technologies will persist into the foreseeable future. In terms of mainframe adoption, Heublein sees a couple of categories of users. First are the large financial services organizations, such as large banks and insurance companies, which are considerably invested in their mainframe architectures and which will likely continue to rely on mainframe environments for their policy management and core infrastructure. “I think there’s a lot of good reasons for that,” explained Heublein. “If you want to process billions of transactions a day, [mainframe] is still a good option.”

    That said, regulations and market pressures are influencing a global movement toward open banking. For example, PSD2 mandates open banking in Europe and industry consortiums like Financial Data Exchange (FDX) are directing it in the U.S. market. These initiatives are encouraging financial services to transform their legacy environments and spurring cloud migration, especially for applications that aren’t mission-critical.

    Heublein pointed out a few different patterns in enterprise mainframe modernization. One approach is that enterprises are rewriting or re-architecting old COBOL applications and keeping them on-premises. Or they are porting and re-platforming the application; moving a COBOL app to a Windows or Linux system, for example. Finally, some are shifting straight to the cloud. For solutions that don’t need to be on the mainframe, enterprises may shift toward cloud-native to free up capacity.

    APIs Enable Legacy Modernization

    Distributed cloud technology is helping businesses scale to the right level. Yet, it’s difficult to jump off the mainframe entirely, noted Heublein. “Very rarely can you take all your stuff and move it all at once,” he said. “It’s a tough pill to swallow.” It’s especially difficult to move core banking infrastructure to a distributed environment—an instant shift could introduce security risks. For this reason, he recommended a ‘phase migration’ approach in which individual components are slowly migrated. For example, an insurance provider may shift their claims engine, policy engine and ratings engine to the cloud one at a time.

    As a result of phase migration, however, engineers must simultaneously support both on-premises and cloud-based applications. They must also ensure bidirectional communication so that these systems can exchange data. This is where a REST API transactional layer is necessary, Heublein explained. Such a layer could enable bidirectional communication between the mainframe and cloud-native applications, abstracting complexity when calling out to the modern world, or intercepting cloud data and filtering it into a format mainframe technologies can process.

    To increase developer usability, abstracting complexity from the mainframe integration process will be the key to building out connectivity. This could involve a codeless development approach with a COBOL sub-routine to call external systems, Heublein said. Automatic restructuring of data will be necessary to enable fluent bidirectional communication, too.

    Four Business Drivers of Mainframe Integration

    So what is the end result of putting an integration layer on top of mainframe systems? According to Heublein, there are four key business benefits of using such an abstraction layer:

    1. Orchestration: Software ecosystems often need to make multiple calls to different applications to collect and transform data. Yet, mainframe environments are very complex and challenging to integrate with. More usable integration fabric for mainframe systems could enable better orchestration of workflows in a developer-friendly manner. By using a REST interface, all apps could be calling the same interface, thus greatly simplifying orchestration between disparate services.
    2. Sustained Adaptability: The world is rapidly accelerating, and enterprises need to maintain and modify these APIs very quickly. Organizations require “sustained adaptability” to adapt and survive in the digital world, said Heublein. Mainframe systems, however, were not designed for the 21st century. Thus, low-code integration layers are critical to empowering rapid development.
    3. Data Transformation: Data is powerful for the business. Data is not only a commodity in and of itself but also is required for AI and machine learning. However, mainframes have a completely different way of encoding data. Thus, there is a need to transform this data into a modern format that businesses can consume. An integration layer between these systems can deliver an improved data consumption model.
    4. Real-Time Processing: You may think online payments happen in real-time; however, payment apps like Venmo are really just an abstraction layer over bank transactions, which may clear as little as twice a day. In short, mainframes are not great at real-time processing. However, more usable integrations to this infrastructure could enable user-facing applications to interact more closely to real-time.

    Another concern is security. Opening up mainframe environments that house core financial infrastructure must be done with extreme care—especially as API attacks remain the number-one rising threat in the software landscape. All organizations building out new integrations will require a secure layer from an outbound standpoint. Heublein said this often involves placing an API management layer or gateway in front of the integration layer. This system enables authorization and functions like a firewall between the internal and external world. It can also enable logging and prevent denial of service attacks, he explained.

    Improving Agility for Legacy Systems

    Legacy systems are entrenched in older technology and carry lots of technical debt, forcing operations to move at a slower pace. Maintainers are also culturally a bit different than the newer breed of cloud-native engineers. “Mainframe people are not … fast,” said Heublein.

    DevOps folks want to build, test and promote projects to production very quickly and, arguably, this agility is necessary to satisfy rising digital innovation demands. Improving integration capabilities for on-premises systems could enable businesses to better utilize preexisting investments while also improving agility.

    Low-code/no-code capabilities could also increase collaboration potential, Heublein added, which could bridge the gap between on-premises engineers and fast-moving business folks. When adopting low-code/no-code tools for mainframe integration, he cautioned against general-purpose citizen developer frameworks. Whereas universal integration layers can work well for many situations, he stressed that a very specific boundary box is best for building APIs for mission-critical applications.

    How is your organization approaching modernization for legacy systems? Let us know in the comments below!

  • 4 Tips To Plug The Cloud IT Skills Gap

    4 Tips To Plug The Cloud IT Skills Gap

    Technical talent is few and far between. The programming world faces a developer shortage, and with this shortage comes a lack of cloud know-how. By now, most organizations have become multi-cloud, yet 86% of IT professionals believe a lack of these skills will slow down cloud projects.

    As more teams transition from older architectures to cloud environments, the IT skills gap could pose a severe bottleneck to software development and deployment. Cloud servers require ongoing maintenance, and managing security and compliance is challenging, even for seasoned developers. Furthermore, it’s easy to overspend in the cloud—without keen insight into vendor nuances, it becomes challenging to select the most cost-effective package for the application at hand.

    I recently talked with Joe Spurrier, chief technology officer and co-founder, cloudtamer.io, to see how teams can improve their cloud strategies while navigating a short supply of IT cloud know-how. According to Spurrier, increasing automation, sharing reusable templates, centralizing best practices and maintaining agnosticism are strategies to amplify cloud results. Below, we’ll expand on these areas to see how they maximize cloud benefits while retaining a lean engineering footprint.

    Automation, Automation, Automation

    “Automation is key to furthering your adoption and use of the cloud,” said Spurrier. The drive to automate comes as no surprise to many DevOps and SRE professionals whose goal is to automate their job away. When automating the cloud, infrastructure-as-code (IaC) is a fundamental necessity. “If you’re getting into cloud, you should probably have some scripting skills,” said Spurrier.

    Automation can significantly aid scalability when setting up additional accounts since engineers must replicate similar operations, such as server provisioning, deployment patterns, monitoring and so on. Automation is also essential for shift-left testing and to increase observability into production environments. “It’s one thing to deploy an application via automation, but another to have tests in automation—that’s the next level,” said Spurrier.

    Now that so many tools are API-driven, orchestration becomes programmable. A cloud service provider’s CLIs or SDKs could also be leveraged to hook into these integrations to construct such automation.

    On the flip side, too much automation for high-level privileges could backfire. “Someone could break into a data center, steal a server, disconnect it, then run off,” Spurrier explained. Thus, automation will require a higher sense of security to ensure that every change is appropriately authorized and logged.

    Reusable Templating

    Along with automation, documenting operational processes to be shared amongst engineering teams is vital for maximizing efficiency in a lean talent landscape. The key here is to make the process self-documenting and reusable, stresses Spurrier. This will most likely involve someone on the team creating shared libraries and repositories for other team members to leverage.

    One such example is Amazon’s golden AMI pipeline. Golden AMIs are virtual machines that teams can “standardize through configuration, consistent security patching and hardening.” Golden AMIs can also enable a repeatable process for performing monitoring, logging and even distribution to business units.

    Reusable patterns are important for building CI/CD systems so that when you commit code, it is automatically deployed to preferred cloud providers. In general, Spurrier recommends building shared repositories that are easy to clone, with step-by-step instructions for usage. Sample applications that demonstrate components in different languages can also aid developer usability and save time.

    Adopt Centers of Cloud Excellence

    Another way to encourage better adoption is by creating a center of cloud excellence (CoCE). For an enterprise, having a cloud-centric project management group can be very helpful to organize knowledge and offer support. A CoCE could centralize operations, supply engineers with pre-configured containers and help distribute access to cloud accounts.

    A CoCE is a good way to start to unite disparate teams around shared best practices, said Spurrier. Depending on your organization’s needs and workloads, a CoCE will focus on varying skills, he added. In the beginning, a company may have just one center for cloud knowledge. However, as adoption grows, dedicated centers may likely emerge around specific cloud providers, too.

    Retain Agnosticism

    Contrary to popular belief, multi-cloud steups aren’t always about lift and shift. Business units often adopt a particular cloud to suit specific workload requirements. For example, researchers may choose GCP for its data science functions, or Microsoft shops may prefer Azure.

    In a multi-cloud world, management layers could help retain a modicum of vendor neutrality. For example, take Cloud Custodian, a tool developed at Capitol One and recently contributed to the CNCF. Using Cloud Custodian, engineers can run security checks across AWS, Azure and GCP and generate metrics for reporting. Such control and insights could empower teams to automate some of the multi-cloud management burdens.

    When working with multi-cloud, it’s also good to consider agnostic tooling that can work across different providers, like Terraform or Kubernetes, Spurrier said.

    Templating

    Cloud transformation is ongoing at most organizations. With this rise, more cloud-native processes and tools are emerging. Yet, with talent in short supply, DevOps isn’t always fully prepared. To review, we covered four ways businesses can circumnavigate this issue to get an edge:

    • Increased cloud automation is a necessary response to a slimmer labor pool. Though building automation requires upfront technical skill, it pays dividends in increased efficiencies.
    • Reusable libraries and templates can rapidly configure new projects and do so consistently across business units.
    • Consider maintaining a center of cloud excellence to centralize policies and offer support.
    • Opt for agnostic tools. This helps to retain vendor neutrality and apply consistent controls across clouds.

    Of these above strategies, Spurrier emphasized the importance of creating shared templates. When used right, a templating approach can be very effective, he says. “If you have a templating model, it should be easy for DevOps teams to migrate to new features. If you’re leveraging tools that set up templates, it’s easier to figure out how much infrastructure is going to cost.”

  • Can Web 3.0 Solve the Linkrot Crisis?

    Can Web 3.0 Solve the Linkrot Crisis?

    Data on the web is more ephemeral than you realize. A recent examination of linkrot and content drift by Harvard Law School found that as time progresses, web-based content is becoming more and more lost. By analyzing half a million links within New York Times articles, 6% of deep links from 2018 were found to be inaccessible, and 72% of links from 1998 were completely dead.

    The rotting web could have severe consequences that affect society’s collective understanding. Consider links used to substantiate pivotal arguments in the course of law. Studies now show that 49% of hyperlinks in Supreme Court decisions no longer function. The linkrot situation undermines humanity’s shared intelligence, which the Internet once promised.

    So much of the content on the web is outdated, inaccurate or has changed its URL destination, a problem known as content drift. Therefore, band-aids like the Wayback Machine internet archive have risen as a steward of the historical web archives. However, some proponents in the industry argue we need a more modern, decentralized, immutable reference for this historical data. You may have guessed where this article is heading—blockchain.

    I recently met with Jonathan Victor, Protocol Labs, to see how Web 3.0 and blockchain might be used to solve the linkrot endemic. According to Victor, smart contracts that reference canonical URLs could help create a more resilient web. This would, in effect, require a peer-to-peer model with shared ownership and incentive models to sustain content longevity.

    The State of Linkrot

    The longer data sits, the higher the likelihood of losing track of it. Forgotten databases and rising data storage costs can easily lead to whole networks of websites and user data being accidentally (or intentionally) destroyed. Putting so much faith in private companies to preserve data indefinitely is not viable, explained Victor.

    Public data isn’t really as public as you think it is. And older, user-generated content is often the first domino to fall. For example, a massive amount of media on MySpace was deleted in a botched server migration in 2015. In 2021, Yahoo Answers similarly erased 16 years of user-generated content in its shutdown. Surprisingly, users have little ownership or control of their data on forums and social media. A ban on Twitter, for example, could result in inactive links within posts.

    When an article points to a website, this imbues an implicit dependency on an external organization to keep the domain live, said Victor. The second dependency is on the domain owner to keep content unchanged. “If we can talk about content and how to reference it in a different way, maybe we can create references that are much more resilient,” suggested Victor.

    How Web 3.0 Can Help

    One way to solve the linkrot issue is to create a ‘fingerprint’ for web resources. This could be a canonical reference to specific pieces of content. Organizations would store and retrieve content based on a hash of that contract itself, Victor explained. Such a system would likely require an incentive-based, non-exploitative, decentralized network where web users own their data and control who profits from it.

    Some have coined this concept Web 3.0. Victor describes Web 3.0 as a broad classification of distributed technologies and tools that enable a peer-to-peer internet underpinned by blockchain.

    For example, one open source project spearheading progress on this front is IPFS, the InterPlanetary File System. IPFS is a peer-to-peer hypermedia protocol that uses blockchain to store cryptographic hashes. The project is “designed to preserve and grow humanity’s knowledge by making the web upgradeable, resilient and more open.” With IPFS, a client requests content using the hash and retrieves content. IPFS is based on Filecoin, a permissionless storage network.

    Benefits

    A decentralized nature fits the original goal of the open web, allowing creators to own and share content horizontally. If you can decouple data storage and retrieval from a specific location, it gives you an impressive ability to rearchitect modern communication, said Victor. This could have a number of benefits:

    • Redundancy: First off, a major benefit is the redundancy of data. Distributed nodes assure data is globally available and temporally accessible. Resources are in multiple locations, ensuring better global availability and temporal persistence.
    • Incentives: An incentive-based decentralized network could use block rewards to subsidize costs, helping fund storage initiatives. Building on the blockchain can also ease payment acceptance aspects — great for donation-based data hosting groups.
    • Efficiency: By using smart contracts, you can programmatically handle storage requests. “Blockchain gives you higher-level primitives to run this stuff automatically,” said Victor. This helps tie on-chain activities to real-world storage, Victor added.
    • Low-cost: It can be cheaper to store content on distributed networks than a typical cloud provider. Impressively, Victor estimated it would cost 1/1000 of the cost of Amazon. A distributed environment could also bring speed gains; similar to torrenting, seeding from distributed sources increases speed in orders of magnitude, said Victor.
    • Communal ownership: Such a model could spur more communal ownership for some content. This could benefit public-good assets that don’t have a clear owner. According to Victor, this aspect especially benefits universities and open source data initiatives that rely on public support to function.

    A Compelling Problem

    Constructing a new peer-to-peer Internet obviously has significant barriers. Namely, a decentralized structure requires a large group effort, and this community action may be difficult to encourage, let alone coordinate. It will also require consolidation around open standards and protocols, as many different decentralized projects crowd the scene. Shifting pre-existing architectures to this new style could also introduce roadblocks.

    “A lot of content on the internet becomes a public good,” says Victor. Yet, perpetual access to this content is fickle, even for well-established online communities. Some research has found the average lifespan of a website to only be 100 days. Hosting fees or accidental failures are often root causes for the loss of data.

    The fragility of the web is an issue that will take not only innovative development effort, but a radically new mindset to tackle. “It’s a really compelling problem,” added Victor. Using blockchain-based smart contracts to store canonical references for content is one proposed solution. But Web 3.0 is in a primitive stage, and to progress, it will take significant community action.

    Doing so in a way that is sustainable will be the trick. Open standards and blockchain-based protocols could allow the technical means for such a network to flourish. And a cryptocurrency backing could help incentivize participation and provide a means of funding. This decentralized non-ephemeral storage could help enable perpetual storage for the web’s most precious resources.

  • Enterprise Blockchain Adoption Hinges on DevOps

    Enterprise Blockchain Adoption Hinges on DevOps

    There is a lot of excitement around blockchain right now, for a good reason. Production use cases continue to emerge across sectors such as manufacturing, energy, air travel, insurance and finance. Gartner predicts that by 2023, organizations using blockchain smart contracts will see a 50% increase in their data quality.

    Yet, Gartner also predicts that blockchain will reduce data availability by 30%. Before its use can reach ubiquity, blockchain-based application development has some hurdles to overcome. And if enterprises are to get on board, executives will need to see tangible proof of its value in the form of reduced expenses and increased profits directly correlated with blockchain adoption.

    “Enterprises aren’t going to change their business model to adopt a new technology,” said CasperLabs CEO Mrinal Manohar. Just as it’s difficult to dramatically alter the UX for established products, it’s similarly difficult to change the developer experience for established software production routines.

    I recently met with Manohar to discuss the limitations and the changes required to accelerate enterprise blockchain adoption. According to Manohar, much of it boils down to taking a DevOps approach—reducing energy use, enabling CI/CD, ensuring stable development environments and providing room for change and ongoing testing—will be critical to spur adoption. This could also mean abstracting complexity away from the developer and end users. However, once more production proofs are out there, the shift toward blockchain will be swift and it will be massive.

    What’s Hindering Enterprise Adoption?

    Enterprise adoption comes down to UI and UX, said Manohar. He predicts blockchain will follow a similar path as the internet. In its command-line days, the internet was only adopted by die-hard techies. But once visual-based actions were possible through the browser, consumer interest exploded.

    Similarly, blockchain might see more production use once the complexity is pushed to the background. “A technology is awesome when it’s 99% invisible,” said Manohar. However, at the moment, blockchain is 99% visible, he adds. For example, it isn’t easy to spin up a node and test a blockchain app. Lowering the requirements to 50% with more abstraction, simulation and integration with cloud providers could help tremendously.

    In general, Manohar believes that four key factors are hindering enterprise adoption:

    1. Enterprises need a lower energy footprint. Attaining high throughput on a public network without sacrificing decentralization and security is tricky. If blockchain drains operations when scaled, organizations may simply opt for a centralized data store. Therefore, enterprises will require a blockchain with minimal energy requirements.
    2. Large companies want developer environments that are similar to what’s already in place. In many ways, blockchain throws out the old playbook. But such a significant shift is a hard sell across multiple IT divisions. Large companies will require blockchain abilities that plug into the existing developer stack. Using a blockchain with familiar programming languages and plugins to popular IDEs like Visual Studio and IntelliJ will decrease onboarding effort.
    3. They need more control. Blockchains typically use immutable contracts. This is great for sustaining a shared distributed history, but Manohar believes that contracts need to be more malleable to enable continual upgrades. Furthermore, there is the issue of privacy when adopting blockchain. Some implementers don’t want all 100% of data to be on the public chain. Having the ability to maintain both private and public nodes for smart contracts could help enable more flexibility.
    4. Enterprises want ongoing support. Some companies will be autonomous using open source, but most enterprises will not adopt new or emerging technology without a support email attached to it, said Manohar. Thus, there may be a need for professional services and outsourced blockchain experts.

    Applying a DevOps Perspective

    Another big boon to accelerating blockchain adoption could be considering a DevOps perspective. To Manohar, introducing DevOps into blockchain requires more continuous updates, mitigations to avoid breaking change and simulations to enable better testing environments.

    An immutable blockchain infrastructure seems to be at odds with the DevOps creed of rapid deployment. Thus, Manohar said that choosing a blockchain that enables more mutability will help upgrade contracts and allow functionality to change over time. For core infrastructure changes that affect the entire public chain, the majority of validators must, of course, accept upgrades before they are distributed.

    Simple functionality, like a push notification, could ease this process to allow a quick Yes or No response from operators.

    Another potential concern is backward compatibility. What if the aggregate introduces significant breaking change? Manohar said that while this is technically possible, it’s improbable. “That would be a controversial upgrade,” he said, and validators have an incentive to keep a sturdy system intact. “They only do it if they think it’s better for the long-term health of the network.” But guardrails on this process will help avoid edge cases. Thankfully, if a change did disturb a minority of users, a blockchain can be forked. We’ve already seen this occur with Bitcoin Cash and Bitcoin, or Ethereum Classic and Ethereum, for example.

    Another shift that will help enterprises embrace DevOps in their use of blockchain will be better testing and deployment processes. It’s notoriously challenging to test blockchain applications while they are tapped into the broader network. Thus, engineers could benefit from a simulated runtime wholly divorced from the main network for testing purposes.

    The Future of Blockchain Development

    So, what does the future of blockchain enterprise adoption look like? Well, it appears we will continue to see new novel use cases emerge that use blockchain in innovative ways. For example, Manohar pointed to the use of blockchain to power an NFT digital twin, a patent marketplace, a tagging system for supply chains and the mortgage processes for a core banking system. Within these scenarios, he noted how blockchain is exorcising cost in many ways; smart contracts are being used to avoid legal fees, eliminate manual processing and improve fault attribution.

    Before blockchain can go mainstream, however, work must be done to bring other enterprises into the fold. Namely, it will necessitate more abstraction of the infrastructure layer. In this effort, low-code integration capabilities could help make API connections to ERP systems or cloud service providers more accessible.

    To inspire adoption, the economy must also change its perspective on what constitutes blockchain success. As of now, common KPIs are surface-level, like: How many wallets? What is the trading volume? Instead, Manohar believes we need to focus on proving actual cost savings from blockchain adoption. Once accountants are publicly reporting revenue gains, shareholders will demand that every company do this, he said.

    “Once that catalyst happens, adoption will happen very, very fast, and demand for blockchain in enterprises will be massive.” At that point, some years down the line, blockchain may truly power a new economy.

  • Majority of Organizations Now Adopt Multi-Cloud

    Majority of Organizations Now Adopt Multi-Cloud

    As IT’s digital footprint grows, its cloud footprint is rising both in size and complexity. As a result, most organizations have matured into multi-cloud operators. A multi-cloud approach empowers different divisions with the autonomy to select the right computing environment for their needs. It also enables a lift-and-shift ability to port workloads across cloud service providers (CSPs) when needed. Adopting a multi-cloud model could bring performance benefits, cost savings or better scalability, all while encouraging a more vendor-neutral stance.

    We’re in a multi-cloud world. A recent HashiCorp study found three out of every four organizations are already multi-cloud. This is a staggering increase from just a few years ago when mono-cloud was more accepted, and operators had zombie-like allegiance to AWS. The uptick in multi-cloud signals the maturity of offerings at other CSPs, like Azure and GCP. It also signals differentiation among clouds as each hones in their unique value propositions.

    HashiCorp’s inaugural State of Cloud Strategy Survey offers some very clear results about the state of multi-cloud adoption. Below, I’ll review major takeaways from the report to better understand the state of multi-cloud adoption. To sum it up, large organizations tend to benefit most from multi-cloud adoption, as there are still significant roadblocks to truly reaping a sizable return from a multi-cloud status. Yet, more infrastructure automation may help companies of all sizes respond to multi-cloud complexities.

    The Multi-Cloud Era

    DevOps truly operates in a multi-cloud era. The report revealed 76% of organizations are already multi-cloud; that leaves only one in four companies leveraging a single cloud. And, that trend won’t stop—within two years, 86% predict they will be multi-cloud operators. The study also confirms that the larger a company is, the more likely it will adopt multi-cloud—94% of large enterprises with 5,000 or more employees are multi-cloud.

    So, what’s driving this cloud fragmentation? Well, 34% of respondents cite digital transformation as a top driver for multi-cloud adoption. In the race to meet new digital innovation demands, multi-cloud has emerged as a strategic force. To enact this rapid change, many organizations are investing in a cloud center of excellence (CCoE) to guide cloud transformation efforts. A CCoE is most common within large companies with a cloud budget of $5 million or more.

    Speaking of budgets, cloud spending is going through the roof. In 2020, only half of companies met their projected cloud spend, and 39% of companies overspent their allotted cloud budgets. As I’ve covered before, cloud-native costs are notoriously difficult to identify and trim. Without the proper guardrails to manage spending, surprise data egress fees or hidden server costs can strain the cloud checkbook. As it stands now, 40% of companies lie in the $100,000 to $2 million per year range, and 6% of enterprises are investing $50 million a year or more into multi-cloud. This spending will likely increase in the coming years as cloud reliance expands.

    Enterprises Benefit Most

    So, which clouds are companies using? If we compare cloud adoption, AWS is a clear market leader; 88% of companies use AWS, 74% use Azure and 61% use GCP. AWS’s hegemony is not expected to fade in the next two years. Yet, while AWS usage will remain the same, competing CSPs will experience modest increases. This is likely due to Azure and GCP making gains in geographical zones outside North America, such as Asia-Pacific and Africa.

    In terms of technology used in cloud environments, open source software (OSS) is a popular choice. Cloud-native open source projects, such as Kubernetes, OpenStack and Terraform, have taken on a life of their own, with vibrant community support. However, not all shops build upon OSS and run it themselves. Somewhat ironically, we’re seeing more managed OSS emerge. One stand-out area is provisioning, where 42% of organizations build with open source and run it themselves.

    On the whole, multi-cloud is working at most enterprises, with 53% of survey respondents saying multi-cloud has helped achieve their business goals. The top benefit of multi-cloud is digital transformation, at 34%. This is followed by avoiding lock-in (30%), reducing cost (28%) and scalability (25%). But multi-cloud does not treat all companies as equals; the likelihood of multi-cloud success increases as the company size increases. Less than half (42%) of small companies say they have multi-cloud success, which jumps to 64% for large enterprises. With fewer resources for support, multi-cloud may be less beneficial for startups and small to medium businesses. In fact, 41% of professionals say a lack of in-house skills inhibits their cloud success.

    Challenges in a Hybrid Multi-Cloud World

    An ongoing talent gap is just one potential challenge to cloud success. Just over half (51%) say cost concerns are a top inhibitor to their cloud program success. Hybrid cloud complexity is another prominent challenge, with 35% of organizations citing it as a major cloud obstacle. Many enterprises are still grappling with the remnants of legacy on-premises estates. But technical debt is not the only reason for hybrid modalities. Surprisingly, large platforms even have constructed their own data centers or used colocation as an intentional strategy.

    Security is a top cloud inhibitor, according to 47% of IT professionals. The leading cloud security concerns include data protection (40%), data theft (33%) and regulatory compliance (31%). The myriad cybersecurity threats evolve each day, and cloud technologies are not immune; for example, web APIs are now a significant threat vector. A separate report found a 300% increase in malicious API traffic in the past six months. Other issues, such as exposed keys, insecure defaults and misconfigurations are known to hamper cloud-native infrastructure.

    Contrary to popular belief, nearly half (46%) of companies reported that COVID-19 has not quickened their cloud adoption. This reality could indicate that cloud migration has been planned for years, and recent adoption is only part of an ongoing trend. The study found that the pandemic did, however, accelerate things like infrastructure-as-code (IaC), container orchestration, compliance and governance and other operational factors.

    Infrastructure Automation Supports Multi-Cloud

    Multi-cloud feels like an inevitable result of accelerating cloud adoption in a more diversified market. Rising multi-cloud use is also arguably in line with Conway’s Law. As organizations develop more and more digital experiences across new verticals, the platforms and backend processes to support these initiatives will inevitably grow more and more complex.

    To mitigate some of the issues posed by multi-cloud, many organizations are turning toward increased infrastructure automation. In fact, 70% of those surveyed said infrastructure automation tools are extremely important to their organization. Of these tools, provisioning automation (75%) is the most popular kind in use. Automation for other areas, like networking (58%), security (50%) and app development (69%) follow.

    Hashicorp’s State of Cloud Strategy Survey boldly announces, “Welcome to the Multi-Cloud Era,” and its findings back up that sentiment. For further insights, you can view the results freely on this website.