devopsy.kinsta.cloud

Category: DevOps in the Cloud

  • env0 Extends Workflow Platform for Provisioning Infrastructure

    env0 Extends Workflow Platform for Provisioning Infrastructure

    env0 today announced it has added support for additional infrastructure-as-code (IaC) tools and the Microsoft Azure DevOps platform to its workflow automation and management platform.

    Fresh from raising an additional $35 million in funding, env0 CEO Ohad Maislish said while Terraform remains the most widely used IaC tool, there are now more organizations using alternatives such as CloudFormation from Amazon Web Services (AWS) along with Pulumi and Terragrunt.

    Other capabilities added to the env0 platform include automated drift detection, multi-tier workflows, auditing, external logging platform integration, support for plug-ins and expanded role-based access controls and OpenID Connect (OIDC) authentication.

    Finally, env0 is extending its existing Terraform support to include support for remote backend, module registry, remote plan, and the ability to plan and apply pull requests. The capabilities are especially critical for organizations that are now building microservices-based applications that require more sophisticated workflows to provision infrastructure resources, noted Maislish.

    Most IaC tools are designed to enable developers to address technical implementation and provisioning issues without regard for user experience, security and governance, or business goals. The env0 platform makes it possible to create infrastructure environments with guardrails in place that enable DevOps teams to centrally address governance issues, noted Maislish.

    In general, there’s now a lot more focus on continuous delivery (CD) as organizations look to continuously update applications, said Maislish. That requires a more structured approach to provisioning infrastructure across teams of developers that is managed by a platform engineering team, he added. The challenge is that CD is dangerous—in the sense that mistakes can easily be made, noted Maislish.

    It’s not clear whether organizations are embracing platform engineering to centralize the management of DevOps workflows, but the number of organizations that have mastered CD best practices remains low. Most organizations are adept at continuous integration (CI), but given the number of unique platforms on which applications are deployed, CD has proven more challenging to adopt. In theory, a platform engineering team should make it easier for DevOps teams to define a consistent approach to CD that could be adopted across an organization.

    Regardless of the DevOps approach, the overall goal is to improve developer productivity. The problem is that in the absence of any guardrails, developers have been provisioning cloud infrastructure as best they can. Not surprisingly, cloud computing environments are rife with misconfigurations that create vulnerabilities that cybercriminals have become more adept at exploiting. The fundamental issue is not such much the cloud infrastructure that is available as much as it is the processes employed to provision the resources. In their rush to build and deploy applications, developers will sometimes ignore best practices unless guardrails force the issue.

    One way or another, IT organizations are exercising more control over cloud computing environments as the security of software supply chains becomes a bigger issue. The only thing that remains to be resolved now is how to put much-needed guardrails in place in a way that adds the least amount of friction possible.

  • Cloud Management Issues Are Coming to a Head

    Cloud Management Issues Are Coming to a Head

    A survey of 1,300 technology and data executives found that more than a decade after the initial rise of cloud computing, organizations are still struggling to manage these environments.

    Conducted by Wakefield Research on behalf of NetApp, the survey found 98% of respondents reported their business has been impacted by everything from the increasing complexity of data and heightened cybersecurity risk to lack of visibility into business operations and staff burnout.

    Top challenges include cybersecurity risk (45%), staff not taking full advantage of business applications (44%), increased skepticism about the cloud from leadership (44%), lack of visibility into business operations (42%), going over budget (37%) and burning out staff (31%).

    A full 88% continue to encounter technical challenges, including data mobility between clouds (33%), security risks (33%) and interoperability between cloud services (30%).

    The top drivers of cloud investments are scaling artificial intelligence (AI) and automation (39%), followed by driving business innovation (35%) and creating digital experiences (34%).

    Organizational challenges encountered include having a clear vision for cloud strategy (32%), establishing the right level of governance (31%) and managing costs (30%). A full 84% are under pressure to show short-term return on investments (ROI) in cloud computing platforms, with 76% noting CFOs and other business leaders are skeptical of savings. Nearly half of the tech executive respondents (49%) reported that when cloud strategy discussions occur, cost concerns come up often or all the time. As a result, nearly three-fourths of executives (74%) said tiered workloads are either very or extremely important, the survey found.

    Other factors impacting multi-cloud strategy include environmental, social and governance (ESG) concerns (79%) and data sovereignty (74%), the survey found.

    Ronen Schwartz, senior vice president and general manager for cloud volumes at NetApp, said as more organizations employ multiple clouds, total costs tend to rise. Each cloud platform requires mastering another control plane, which means someone in the IT organization must have the right skills to manage, he noted. IT leaders are attempting to exercise more control over cloud deployments, but developers, for a variety of reasons, have already deployed workloads across multiple cloud computing environments, added Schwartz. One approach to achieving that goal is to make sure data is stored on the least expensive tiered service possible for any given specific use case, he noted.

    Less clear is whether an uncertain economy might push organizations to consolidate workloads on fewer cloud computing platforms. Even then, the cost of moving a workload is not trivial given the proprietary application programming interfaces (APIs) that many developers have likely invoked as DevOps built and deployed cloud applications.

    Of course, there is no going back; the issue now becomes how to define a set of best practices for managing cloud computing environments in a way that minimizes costs. It turns out that cloud computing can be more costly than an on-premises IT environment simply because the infrastructure resources made available to developers are almost infinite. In the final analysis, however, that can turn out to be too much of a good thing—especially if organizations wind up spending more on IT to support these environments. Many of those environments are not rigorously managed because, for example, developers may not be inclined to spin down a development environment on a Friday because it takes too long to spin it back up the following Monday.

    One way or another, however, a level of long overdue scrutiny of cloud computing costs may be on the horizon.

  • Enfabrica Makes Cloud Networking Performance Advances

    Enfabrica Makes Cloud Networking Performance Advances

    Enfabrica Corp. today emerged from stealth at the MemCon conference to launch a class of processors for networking in cloud computing environments that are optimized for moving data between CPUs, graphical processor units (GPUs), accelerators and memory.

    Enfabrica CEO Rochan Sankar said the Accelerated Compute Fabric (ACF) enables scalable, streaming, multi-terabit-per-second movement of data in a way that also reduces the total cost of cloud networking. That’s critical as the amount of data being generated by applications infused with artificial intelligence (AI) capabilities that require access to larger amounts of data continues to increase, he added.

    At the same time, DevOps workflows will need to evolve as it becomes possible to compose AI fabrics of compute, memory and network resources capable of scaling to tens of thousands of nodes versus relying on top-of-rack network switches, network interface cards (NICs), PCIe switches and CPU-controlled DRAM that creates too many bottlenecks when moving large amounts of data.

    Enfabrica’s first chip, the Accelerated Compute Fabric Switch (ACF-S), supports standards-based interfaces, including multi-port 800 Gigabit Ethernet networking and high-radix PCIe Gen5 and CXL 2.0+ interfaces. The company claims that Enfabrica’s ACF-S is the first data center silicon product in the industry to deliver headless memory scaling to any accelerator by incorporating CXK memory bridging to enable a single GPU rack to have direct, low-latency, uncontended access to local DDR5 DRAM at more than 50 times greater capacity than GPU-native High-Bandwidth Memory (HBM). That approach should enable customers to cut their cost of GPU compute by an estimated 50% for large language model (LLM) inferencing and 75% for deep learning recommendation model (DLRM) inferencing, the company claimed.

    Simulation tests suggested ACF-enabled systems could achieve the same target inference performance using only half the number of GPUs and CPU hosts compared to the latest GPU servers.

    The cost of building AI applications today is prohibitive for many organizations, but as processor advances are made, there may come a time when the cost of building and training AI models substantially decreases. Advances in networking processor technologies will, as a result, accelerate the pace at which network operations are integrated into programmable DevOps workflows versus relying on dedicated networking teams to manage top-of-rack switches.

    It’s not clear yet whether cloud service providers will embrace ACF, but the chip makes it clear there is a significant opportunity to reduce the amount of networking infrastructure needed to build and deploy AI applications.

    Less clear, of course, is to what degree machine learning operations (MLOps) might ultimately be folded into DevOps workflows. While developers of AI models create a unique type of software artifact, they still need to be integrated with application code running in a production environment via application programming interfaces (APIs). One way or another, the total cost of building and deploying applications infused with AI models will need to decline.

    In the meantime, the networking bottlenecks that conspire to make deploying distributed computing applications difficult today may soon become overcome as latency issues are increasingly overcome.

  • Tabnine Extends Generative AI Alliance With Google

    Tabnine Extends Generative AI Alliance With Google

    Tabnine has extended its alliance with Google Cloud to advance the adoption of generative artificial intelligence (AI) to automate the writing and testing of code.

    The generative AI platform provider has already developed its own large language model that is hosted on Google Cloud. Tabnine is now also committing to leveraging the large language model that Google is developing to automate software development life cycle processes.

    Brandon Jung, vice president of ecosystems for Tabnine, said it’s apparent that application development and deployment teams will be making use of multiple large language models across SDLC workflows. Not all those large language models will be developed by one single platform provider, he noted.

    Rather than being dogmatic about large language models, Tabnine plans to make it possible to invoke multiple models via application programming interfaces (APIs) that are integrated with development environments, he added.

    The Tabnine platform supports multiple programming languages, including Python, Java and JavaScript and is designed to be integrated with integrated development environments (IDEs) such as Visual Studio Code and Jetbrains. The company also previously integrated its code completion tool with the GitLab continuous integration/continuous delivery (CI/CD) platform. The overall goal is to make it easier for developers to automatically write code based on custom models using approved source code hosted in a secure private repository.

    Generative AI creates a large language model that assesses the probability of what the next line of code will be based on what has preceded it. It’s not likely DevOps teams will be replaced as generative AI is extended across workflows, but the overall size of those teams might shrink as it becomes possible to do more with fewer people. At the same time, the barrier to DevOps adoption will also fall as AI platforms make it simpler for more organizations to embrace DevOps best practices.

    The challenge will be making sure the data collected to train AI models is of a high enough quality to ensure the desired outcome. Organizations will need to find a way to apply generative AI to data models that have been validated by DevOps teams.

    One way or another, it’s only a matter of time before generative AI capabilities are applied more broadly. Platforms such as OpenAI’s ChatGPT are only the tip of an iceberg that impacts almost every manual process, including software development and deployment. The issue will be determining how quickly those innovations will become practical enough to employ.

    In the meantime, it is already apparent generative AI platforms are having a significant impact on the rate at which code can be developed. Inevitably, that means the amount of code moving through DevOps pipelines at any one time should increase significantly. DevOps teams should expect generative AI technologies to be applied both before and after application code is built and deployed, noted Jung.

    In the meantime, the biggest issue is, arguably, keeping up with a rate of generative AI innovation that is occurring faster than many organizations can absorb.

  • Dylibso Releases Tool for Tracking and Validating Wasm Modules

    Dylibso Releases Tool for Tracking and Validating Wasm Modules

    Dylibso today made generally available a system of record for tracking, investigating, searching and validating modules built using the WebAssembly (Wasm) binary instruction format.

    Fresh from raising $6.6 million in additional funding, Dylibso CEO Steve Manuel said Modsurfer exposes the internal contents to enable developers to debug Wasm modules and ensure compatibility with runtime environments.

    Modsurfer provides development teams with a console for accessing a fully auditable database of modules indexed for search and debugging. In addition, development teams have the ability to extract critical information from binary formats in a way that can be integrated with DevOps pipelines. In effect, Modsurfer provides application development teams with X-ray vision into previously opaque Wasm modules, said Manuel.

    Dylibso previously developed Extism, an open source universal plug-in system that supports multiple programming languages used to build Wasm modules. That reduces the level of effort required to integrate Wasm modules and provided the foundation upon which Modsurfer was created, noted Manuel.

    Wasm, at its core, is a portable binary instruction format for building software that describes a memory-safe, sandboxed execution environment. The World Wide Web Consortium (W3C) drove the development of Wasm as part of an effort to create a common format for browsers executing JavaScript code. Wasm is now being extended beyond browsers and JavaScript to enable developers to create a set of universal binaries that could work on any platform without modification.

    That approach replaces the current predominant method for building software that relies on the aggregation of software components that tend to lack distinct boundaries between them. One of the issues of that approach is that it becomes relatively simple for malware to infect all the components of an application.

    It may be some time before WASM becomes the preferred artifact for building applications, but it’s clear the walls that have separated one IT platform from another are finally starting to crumble. Once applications become much more portable, the longstanding goal of writing an application once and then deploying it anywhere may finally be achieved. That’s not only critical in terms of overall application developer productivity; it also has major implications in terms of streamlining the management of DevOps workflows as it becomes easier to both initially deploy applications and continuously update them.

    As organizations continue to review their application development practices to better secure software supply chains, it’s also become apparent that legacy approaches to building applications are fundamentally flawed. Rather than attempting to reengineer processes to encourage developers to better secure applications, it may be more effective to change the way applications are built. After all, considering all the processes required to secure legacy development approaches, it may be cheaper and easier to adopt a new method for building applications that lowers overall risk, said Manuel.

    Of course, the amount of Wasm expertise that exists in the developer community is still limited. However, DevOps teams should expect to see more Wasm modules moving through DevOps pipelines as Wasm tooling continues to improve.

  • Data APIs: Realizing the Future of Data Warehousing

    Data APIs: Realizing the Future of Data Warehousing

    Data is the currency of business in today’s digital economy, with organizations collecting mountains of data from their customers, products and services. Enterprises are increasingly turning to data warehouses to store this valuable enterprise data and to make it useful and actionable. Data APIs–both GraphQL and REST–are emerging as a tool to power frictionless and high-quality integrations between applications/services and data in enterprise data warehouses. But what exactly is driving this shift toward using APIs on top of data warehouses? In this article, I will talk about the drivers behind the rise of data APIs on data warehouses, challenges with current approaches to building, operating and scaling data APIs and how organizations can address those challenges.

    The Rise of Data APIs

    Data APIs enable organizations to create powerful integrations between applications without having to go through lengthy development cycles or write complex code. As a result, businesses can deliver improved user experiences faster than ever before by quickly connecting disparate applications and services together using intuitive REST or GraphQL endpoints. Because these endpoints are standardized across multiple systems (e.g., REST or GraphQL), developers don’t have to learn new languages or coding techniques every time they want to connect two different applications together. This makes it much easier for developers to work with multiple systems simultaneously while still ensuring that all applications remain connected via one interface–the API endpoint itself.

    Challenges With Current Approaches

    Despite all these advantages provided by Data APIs, there still remain a few challenges that organizations need to consider when building out their own solutions. For example, many existing approaches require manual setup processes, which can be extremely time-consuming and tedious for developers working with multiple systems at once. In addition, many existing solutions only allow for limited query functionality, which means that more complex queries may not be possible without additional coding or manual setup processes being performed. Finally, some existing solutions require an extensive amount of organizational resources in order to build out a full-fledged solution—time that could be better spent on more important tasks like product development or customer service initiatives.

    How Organizations Can Address These Challenges

    Fortunately, there are now tools available that make it easy for organizations to quickly build out powerful data API solutions with minimal effort required from their developers or IT staff members. These tools provide pre-built templates that can be used as a starting point for any integration project while also allowing users to customize their own solution based on specific needs and requirements without having to write any code themselves. Additionally, these tools allow users access to advanced query functions such as joins across multiple tables so they can get the most value out of their databases without compromising either speed or security either. Finally, these tools also include powerful monitoring capabilities so users can ensure that their data APIs remain running smoothly at all times, no matter how much traffic they receive.

    Benefits of Using an API for Integrating Cloud Data Warehouses

    When it comes to integrating cloud data warehouses into your application or service, there are several advantages that come with using an API as opposed to traditional methods such as manual coding or ETL tools:

    • Increased Flexibility: By leveraging an API, you are able to access data from multiple sources quickly and easily without having to manually code every integration yourself. This also allows your application or service more freedom when it comes to accessing additional data sets in the future; all you need is an existing API connection in place.
    • Improved Security: Since APIs are designed specifically for secure communication between different systems, they offer enhanced levels of protection against malicious attacks such as DDoS attacks or unauthorized access attempts. You can also add additional layers of security by implementing authentication protocols like OAuth or JWT tokens on top of your existing API connection.
    • Enhanced Performance: APIs can be designed with specific performance optimizations in mind; this means that you can create custom solutions tailored towards the exact needs of your application or service which will ultimately result in improved end-user experiences and higher engagement rates from customers who interact with your product or service regularly.

    Data warehouses have become an essential part of modern digital businesses due to the sheer amount of valuable enterprise information they store. However, traditional approaches can often lack the flexibility or scalability needed in order for businesses to stay competitive in today’s ever-changing market environment. This is why smart organizations are using data APIs on top of them for quick integration among disparate applications/services along with advanced query capabilities.

    By leveraging tools designed specifically for this purpose, companies can easily overcome current challenges associated with building out their own solutions while still getting maximum bang for their buck when it comes down to successful implementation and operation over the long term.

  • How to Maximize Telemetry Data Value With Observability Pipelines

    How to Maximize Telemetry Data Value With Observability Pipelines

    The software landscape has shifted significantly in recent years as companies digitize their operations and adopt cloud and microservices technologies. The complexity of modern software systems has led to an increasing need for real-time visibility into their performance—resulting in an explosion of telemetry data that is stretching IT budgets and outpacing engineering teams’ ability to find value in that data. In fact, data volume is growing at 23% yearly, while IT budgets are only growing at 5%.

    Managing all this data at increasing cost is further compounded by unpredictable software failures and ever-evolving security threats. As a result, companies are forced to pick which parts of their software to instrument and monitor. To overcome this challenge, a different approach to managing data is required.

    Observability pipelines are emerging as a way to address this problem. They centralize telemetry data (i.e., logs, metrics and traces) from many sources, transform that data to fit downstream needs, and route it to various destinations for analysis. With complete control of the data, companies can sort through large amounts and prioritize what is essential, allowing them to act swiftly to avoid disruptions while reducing costs by only storing the data they need.

    Research indicates 62% of organizations currently have an observability practice. This means 38% operate with limited awareness of what’s happening in their software. Even those with an observability practice can only afford to store some data and have to choose which applications to instrument. This leads to blind spots and, ultimately, issues that are time-consuming to fix due to a lack of data.

    Managing Limited Resources With Observability Pipelines

    Engineers face more demands in today’s fast-paced business environment than ever before. Despite an increase in their responsibilities, the size of engineering teams has yet to grow accordingly. Given this, there has been a rise in full-stack engineers responsible for technologies from the front end to the back end. Short on time and resources, it’s difficult for these engineers to instrument and monitor software.

    Observability pipelines help control the amount of telemetry data using various processors such as sampling, throttling, filtering and parsing, and only forward valuable data to the downstream systems. The rest of the data can either be discarded or stored in a low-cost system, such as Amazon S3. This reduces costs and lets engineers focus on analyzing relevant data to identify issues.

    Engineers can also use observability pipelines to consolidate data from various tools. Many teams use different platforms or open source tools, like Prometheus or Jaeger. The fragmented data makes it hard to solve issues and understand application health. Observability pipelines combine all this data to make it easier to act on.

    It’s also important to remember that real people work on these technical challenges. Engineers face burnout working long hours to keep up with software development demands. Constantly switching between tools and manually moving data adds to the pressure. Moreover, the number of data sources, observability tools, and platforms that need that data are constantly increasing. Embracing open standards such as OpenTelemetry can ease this integration burden. Engineers can swap out tools without extensive testing and integration. The increased control and easy integration enable engineers to focus on delivering business value.

    Making Data More Usable

    Teams across organizations want access to data but struggle to use it effectively. The difficulty stems from the lack of data usability: Telemetry data is unstructured; varying formats make it hard to use; data preparation is time-consuming and sensitive data in logs may lead to compliance violations.

    By most estimates, 80%-90% of data is unstructured, which makes it hard to store and analyze because it doesn’t adhere to traditional data models. Observability pipelines make sense of unstructured data before it reaches its final destination. This is achieved using data processors that shape and transform data to make it more actionable. These processors include advanced parsers that identify and extract relevant information, transforming data into consumable formats that make it easier to work with and analyze.

    The advantage of performing these operations within the pipeline is that the same data can be prepared to fit different use cases downstream. For instance, while one team may require data optimized for visualization and trend analysis, another may require complete data for threat hunting. These transformations are managed from a single control point, eliminating the need to maintain separate data streams.

    Observability pipelines can also help companies meet data compliance requirements. For example, teams can scrub, mask or redact PII data based on a defined key structure before the data reaches a SIEM or audit platform.

    Taking Immediate Steps to Make the Most of Telemetry Data

    Organizations can take immediate action to make the most of their telemetry data. First, they need to adopt a visibility-first rather than a cost-first approach. They should assess what software they’ve instrumented. This will identify gaps in telemetry, which can be used to develop a plan to instrument all software. Next, they need to create a strategy for managing their telemetry data, being particularly thoughtful about how and why the data is stored and analyzed.

    Once a company has taken these two steps, it can decide which tools to use. One of the great things about OpenTelemetry is that it provides flexibility for future decisions about what tools to use.

    As companies invest more resources into building observability practices, they’ll encounter problems processing an extraordinary amount of telemetry data for any meaningful action. Observability pipelines provide a competitive advantage of prioritizing the essential data that will enable companies to make better decisions faster.

  • Four Technologies Transforming Data and Driving Change

    Four Technologies Transforming Data and Driving Change

    Data is getting a facelift in 2023 courtesy of event-driven architecture (EDA). These four technologies are set to transform DevOps and IT operations over the next year–whether it be how companies make use of data or the customer experience they offer.

    1. Applied Observability: A Powerful Source of Decision-Making

    Applied observability has been recognized by Gartner as a key 2023 strategic tech trend at its most recent Orlando Symposium. It ensures that the right data is delivered at the right time for rapid action based on confirmed stakeholder actions rather than intentions. “Observable” data include key digitized artifacts that appear when any stakeholder takes any kind of action. Applied observability feeds these observable artifacts back in a highly orchestrated and integrated approach to accelerate organizational decision-making and allows the business owner to track how long an action took to fully process at every point in the workflow.

    But unearthing these insights requires an event-driven approach to software architecture. Distributed tracing, for example, is a method of tracking application requests as they flow from front-end devices to back-end services and databases. By definition, and in contrast to traditional tracing, distributed tracing can be visualized to show a searchable, graphical picture of when, where and how a single event flowed through an enterprise, regardless of the number of hops the workflow took to fully process.

    Consider the potential traceability and end-to-end observability benefits across a payment ecosystem underpinned by EDA. Embedding distributed tracing into an event mesh emits trace events in OpenTelemetry format so banks can collect, visualize and analyze them in any compatible tool. This empowers them to not only confirm that a given message was published but easily understand exactly when and by whom, where it went, down to individual hops, who received it and when … or why not.

    2. Hyper and Intelligent Automation: Taking Automation to the Next Level

    Gartner and IDC identified both hyper and intelligent automation in their yearly technology trends for 2023. Hyperautomation moves automation up a level, adding more intelligence to automation and using a broader set of tools so that previously un-automatable tasks can be automated. Hyperautomation initiatives come in many different shapes and sizes and are being seen across a wide range of industries, from banking and insurance to manufacturing and health care.

    But hyperautomation requires an organization to be underpinned by an EDA and–more specifically, an event mesh–to maximize success. An event mesh is an interconnected network of event brokers that allows data to be pushed in real-time to parts of the organization where the data is needed. It does this dynamically, meaning that new event types can be added at any time and interest in events can be registered to allow a seamless interchange of data for the applications that are interested in using it.

    Take the aviation industry as an example. An event mesh streams information such as flight routes, delays, cancellations and mileage accruals between applications, connected devices and people anywhere in the world, instantly. With an event mesh, information about events can be continuously streamed to multiple systems; filtered so each system only receives the data it needs. This ultimately enhances the customer experience as passengers, pilots and crews are notified in real-time when something relevant occurs on each and every flight.

    3. The Acceleration of the Event-Driven API Economy

    The API economy has exploded. Research from MIT Initiative on the Digital Economy showed that over a four-year period, businesses using APIs saw 12.7% more growth in market capitalization compared to those that did not adopt APIs.

    But the API economy is changing as event-driven architecture and asynchronous event-driven APIs become increasingly important to companies striving to operate in real-time. Using both synchronous and asynchronous methods can result in an application environment in which system resources are most effectively used.

    AsyncAPI and RESTful APIs will be the next evolution of the API economy. API management platforms will need to adapt quickly to this new reality. As a prime example, Forrester recently spotlighted a global biotech company that is blazing a trail in the API-driven area. As the case study demonstrates, the organization understood that EDA is a complement to RESTful APIs–another tool in their digital business toolbox, filling in the gaps of a REST-only approach.

    As a result, they planned a unified event + REST approach from the outset. Both APIs and events are managed as enabling digital products, with the governance and life cycle of both unified as one process. A platform team builds a unified API + event platform for app dev teams rather than building each as a separate siloed platform. The results include faster speed to market and new business opportunities. The company also embraces FAIR data principles: Data that is findable, accessible, interoperable and reusable. A digital marketplace consisting of both events and APIs makes the data findable and machine-readable via Open API Spec (OAS) and AsyncAPI metadata. Data can then be accessed on demand via APIs and events push data in real-time.

    4. The Metaverse Continues at Pace

    Driven by technologies including digital twins, augmented reality and virtual reality, McKinsey estimates the metaverse will be worth $5 trillion by 2030. Data in real-time will be the vital common denominator to link the digital and physical divide and optimize the Metaverse. McKinsey lists real-time data as a vital element to facilitate adoption of the metaverse in several key industries including retail, banking/financial services and transportation.

    The metaverse cannot be static. It needs to be in real-time, in motion. It needs an EDA and a real-time event mesh to support it.

    For example, if an avatar visits a retailer in the metaverse, the retailer will need to keep track of countless user actions such as how long someone spent looking at a particular product, which products got the most interest, etc. The metaverse will produce a significant amount of data that companies will be interested in analyzing in real-time. This data can be used to make decisions from real-time adjustments to inventory to incentivizing commerce through real-time offers/discounts/coupons and even dynamically pricing products.

    Designing a Data-Driven Future With new Tech

    Together, these four predictions have the potential to revolutionize company operations, communications and decision-making–but to take advantage of their full potential and reap success, these four trends will need EDA by their side.

  • How Database DevOps Fuels Digital Transformation

    How Database DevOps Fuels Digital Transformation

    In today’s real-time, hyperconnected reality, data is a core driver behind most digital strategies. User data is constantly gathered and shared across various technology backends and platforms—as is system performance data to inform engineers. Data is pivotal for everything, from generating machine learning models, refining user experiences and even supporting new data-centric services. But although data is a clear backbone to many enterprises, there is no clear winner regarding which database to use.

    Large organizations often find themselves supporting different database styles, from relational databases to NoSQL and distributed SQL. And as organizations continue to adopt multiple databases, safely managing the processing and storage of data becomes more challenging to enact at scale.

    I recently met with Jakub Lamik, CEO, Redgate Software, to understand the role DevOps, specifically database DevOps, is playing in advancing data-driven digital transformations. According to Lamik, applying DevOps practices for database management can improve performance, increase productivity and help organizations stay compliant. Below, we’ll consider the state of today’s data-driven digital transformations and highlight ways to improve how DevOps is applied to database management.

    State of Data-Driven Digital Transformations

    The 2022 StackOverflow Developer Survey found the most popular databases to be MySQL, PostgreSQL, SQLite, MongoDB and Microsoft SQL Server. But most often than not, businesses deploy more than one of these in production. Redgate’s 2021 State of Database DevOps report found that 70% of businesses had more than one DBMS in use, and 48% had teams working on three or more different databases.

    Engineers operate in increasingly hybrid and distributed technology environments. This isn’t for no reason—in fact, 90% of organizations said multi-cloud is helping them achieve their business goals. Lamik also notes a diverse array of software technologies in practice today, from multiple clouds to decoupled microservices architectures and database types. But the desire to work with diverse toolsets causes fragmentation, which could hinder software delivery efforts. For example, organizations want to deploy the same framework and similar methodologies across different databases, said Lamik, and have it be database- and cloud-agnostic.

    Tips to Improve Database DevOps Operations

    To increase the pace of software delivery, infrastructure automation is often needed. And like other roles, database professionals are increasingly trying to automate operations and free themselves from mundane tasks. So, what are some ways to streamline database DevOps? First off, Lamik encourages adopting a common approach across multi-database environments, using standard toolsets that are flexible and portable. “The organizations that standardize deployments across different database environments can increase the frequency and reliability of deployments,” he said.

    Catching and resolving errors quickly is also becoming increasingly essential to meet objectives. And although DevOps brings many advantages, the CI/CD pipeline has gaps. “Observability supported by intelligent automation is key to closing those security gaps,” said Lamik.

    Thus, he encouraged additional oversight and tracking of changes to ensure stability throughout the database deployment pipeline. Also helpful is encouraging collaboration with security professionals (also known as DevSecOps) and integrating security principles into the DevOps workflow to ensure the process is secure by design.

    Another element of database security involves the proper observability and classification of sensitive data. As Lamik describes, organizations must be careful to distinguish between data used for production purposes and data used for development purposes. Ensuring sensitive data is cataloged correctly is imperative to avoid production data from being breached or accidentally publicly exposed.

    Final Thoughts

    In summary, applying DevOps ideals to database management could be a boon for development efforts. For those who haven’t yet considered it, a DevOps transformation will pay dividends, says Lamik. Furthermore, a common management layer could help wrangle multiple database types and help adopt new emerging database types, such as Snowflake and Postgres. Yet, when acquiring new database toolsets, Lamik admitted it may not be a quick transition. He recommends a land-and-expand strategy rather than a top-down approach (which typically isn’t as effective in empowering individual developers).

    Commenting on the industry’s recent move toward platform engineering, Lamik sees it as the next evolution of DevOps. Whereas DevOps brought helpful toolsets to streamline deployment and digitization, platform engineering is taking these practices and productizing them to be reusable at scale across teams and departments, he explained.

    Also, in the new cloud-native stack, containers are designed to be ephemeral, causing engineers to craft workaround solutions to enable stateful storage on platforms like Kubernetes. Regardless of deployment, data must have proper backup and recovery controls as well. In this new cloud-native paradigm, the role of database professionals is changing, explained Lamik, and “APIs are becoming the focal point of communication.”

  • Addressing Software Supply Chain Security

    Addressing Software Supply Chain Security

    It’s essential for organizations to learn more about the software supply chains they rely on and the steps needed to secure them. In just the past few years, we have seen a major uptick in malicious actors exploiting vulnerabilities in software supply chains to facilitate attacks on organizations. However, it’s important to remember that these attacks are not a new phenomenon. In fact, they’ve been around for decades. Therefore, the need to secure the software we rely on is not new, though the industry turning its attention to this problem is.

    Software supply chain attacks are also far from simple. Given the fact that software supply chains are highly complex and made up of a number of constantly changing components, the avenues that malicious actors can take to pull off an attack are plentiful. For example, software artifacts are partially made up of components downloaded from open source software repositories. These platforms have become a favored attack avenue, given how frequently they are used by developers and the trust extended to them by development organizations. According to GitHub, anywhere from 85-97% of enterprise codebases come from open source repositories. Two of these repositories, npm and PyPI, have seen a nearly 300% spike in the number of attacks on their platforms in the past four years, impacting thousands of software supply chains globally.

    One example of the kinds of software supply chain attacks that are now common is IconBurst, a major attack from last year where more than two dozen typosquatted packages were discovered on npm. As with other recent attacks, the IconBurst attackers used typosquatting – uploading malicious packages to repositories with names that closely resemble popular and legitimate packages on the same platform. Developers, in a rush to complete their work, are often tricked into downloading these typosquatted packages. In the case of the IconBurst campaign, for example, one of the malicious packages discovered was downloaded over 17,000 times!

    Given the significant increase in the number of software supply chain attacks, organizations must make more of an effort to understand how these attacks work and then take steps to secure their software supply chains from compromise.

    Say Goodbye to “Shift Left”

    To begin with, development organizations need to reevaluate the mantra to “shift left.”

    For years, industry leaders promoted the notion of “shifting left” as a call to marry application security with application development: Embedding secure development tools and practices into the continuous integration/continuous delivery (CI/CD) process. But times are changing and the industry is beginning to understand that while shifting left is necessary, it is not sufficient to secure software supply chains.

    “Don’t shift left; shift everywhere!” said Tanya Janca, founder and CEO of WeHackPurple. Janca was speaking as a member of a recent Techstrong panel, “Software Supply Chain: It’s All About The Code” at the Predict 2023 virtual event. Janca pointed out the industry’s past fixation on “shift left” and the gaps in security that this caused since threats to the software supply chain not only happen during development but also throughout the entire software release process.

    In fact, the notion of shifting left may do development organizations and developers a disservice: Leaving the work of securing applications entirely to them. “It’s unfair to expect developers to write perfect code,” said Jeff Williams, co-founder and CTO of Contrast Security, another member of the panel.

    Software engineers are already under pressure to develop software quickly to meet development and project milestones. That makes it difficult for them to implement perfect security measures, like vetting every single line of third-party code they use while also watching out for social engineering attacks.

    A New Approach to Secure the Software Supply Chain

    For organizations looking to secure the software supply chains they rely on, they must choose the right tools. But much of the secure development and application security testing (AST) technology out there won’t cut it. For instance, tools such as dynamic or static application security testing (DAST/SAST) are critical components of the secure development life cycle (SDLC). However, they don’t provide holistic coverage of all of the supply chain threats out there, including software tampering and the risk posed by compromised open source and third-party libraries. On the other hand, tools such as software composition analysis (SCA) can help vet open source components but often fail to detect malicious modules or provide total security coverage of software supply chains. Knowing that your application depends on a malicious open source package after the fact is of little consolation when you release code at the speed of DevOps.

    This is why modern tools that go beyond these capabilities should be considered. “It’s about giving developers the tools they need to be successful,” said GitLab’s CPO David DeSanto at the event. Experts in the space agree that change is needed but that developers and development teams aren’t set up for success as it stands.

    According to Mat Matthews, an experienced practitioner in the DevOps space: “While developers are responsible for ensuring clean code, it takes a cross-functional team with a specific focus on security to reduce the attack surface against supply chain attacks.”

    To accomplish that, organizations need to look to their security operations centers (SOCs) to take on a greater role in helping to secure software supply chains. Close collaboration between development and SOC teams is the surest way to provide end-to-end security for an organization’s software supply chains. And, supplied with the right tooling, SOCs and development teams working together will be better positioned to scan software packages; achieve greater visibility into development processes; detect deviations from a set baseline; and pursue possible supply chain compromises before they can cause real harm to software publishers and their customers.

    As threats and attacks on software supply chains evolve, so must protections. A modern software supply chain security program should aim to protect the entire software development process, from the developer IDEs and their plugins to the open source components, CI/CD workflows and the release pipelines, while also being equipped to spot and pursue compromises at any point in the development life cycle. Only then will development teams be able to prevent harmful attacks like SUNBURST, IconBurst and countless others that have happened since.