devopsy.kinsta.cloud

Category: DataOps

  • Database Virtualization – Driving the Next Big Transformation

    Database Virtualization – Driving the Next Big Transformation

    Data is both the most important asset and the biggest challenge for most corporations. It impacts every aspect of the business from financial considerations to AI/ML training. Data is used by development and DevOps teams, engineering teams, quality assurance, business intelligence teams and database administrators. Despite its central role in operations, traditional forms of database delivery remain complex, costly and time-consuming.

    In many cases, enterprise data is growing exponentially, increasing management challenges and driving up the cost of data storage. Duplicating data to each new environment as use cases arise is overwhelming DBA teams and incurring a significant storage footprint. This has resulted in data becoming a significant bottleneck for DevOps teams, highlighting the importance of finding a quick and automated solution for the replication process across non-production environments.

    What’s Changed?

    In the past, development teams would work with the same dataset for upwards of six months. This left DBA teams with plenty of time to meet all database delivery needs. However, as technology became more sophisticated and consumers grew more demanding, the data delivery life cycle was cut down dramatically. Today, with agile development, new features may be added every few days, if not hours. Coupled with QA teams’ requirement for testing data continuously, AI/ML teams requiring millions of data points for accurate training, and data analysts constantly needing fresh data for organizational growth, this means DBA teams are flooded with demands they are incapable of managing.

    What Now?

    Database virtualization is an automated and easy solution to the dataset bottleneck. By simplifying the creation and distribution of database replicas, DevOps teams will be able to create fully functioning virtual environments via a self-service process–allowing DBA teams to focus on more critical duties relating to production databases.

    Speed Up

    The first—and most obvious—advantage of this approach is the increased speed at which data is delivered. To keep DevOps teams productive and avoid the use of old, stale data, new environments need to be produced on demand. By integrating the production of new environments into the CI/CD pipeline, enterprises will be able to automate the process and provide DevOps teams with full autonomy. By cutting out the dependency on the DBA, teams will be able to access the data they need, when they need it.

    Scale

    Agility is fundamental to the success of database management. Each team requires an independent environment, unaffected by other use cases. The creation and supply of virtual databases allow for quick provisioning. Once created, these databases look identical to the original physical database and function completely independently of each other. Any changes made to a vDB remain unique to the environment it’s in.

    Save

    As the need for data grows, so does the cost of storage. Creating a physical copy for each team and use case results in a significant storage footprint. This means that these datasets come with a hefty cost. However, the creation of virtual databases led to a dramatic decrease in storage needs. Where traditionally, physical datasets can weigh upward of several TB, vDBs only weigh a few hundred megabytes. Further, virtualization solutions also compress the weight of physical copies – such as the golden copy – actively decreasing the weight and cost of storage across all fields.

    Secure the Database

    Even after the database delivery problem has been solved, enterprises are faced with another major issue – security. With so many different teams accessing data in virtual and low-level testing environments, the risk of ransomware and other cyberattacks increases dramatically. Generally, the investment in securing non-production environments is significantly weaker than in production.

    A common approach to address this issue is masking sensitive data so it is rendered useless to outside sources. However, traditional masking requires a data expert to manually sift through all existing data and implement data masking algorithms on any sensitive information found. This time-consuming solution creates a new data bottleneck. By automating the process and applying it to the golden copy, corporations can be confident that all vDBs created are completely secure and privacy compliant.

    Looking Ahead

    Database virtualization is the future of database replica management. Virtual databases are delivered in a matter of minutes, significantly accelerating the speed of delivery. These environments are easy to use, independent, and can be maintained via a self-service portal. The ongoing sync between the original database and the vDB ensures continuous access to the latest data without the need for timely or expensive processes.

    Effective database virtualization will allow well-established enterprises to more easily compete against those agile startups that face significantly fewer barriers to entry.

    Image Source: joshua-sortino-LqKhnDzSF-8-unsplash

  • The Benefits of a Streaming Database

    The Benefits of a Streaming Database

    Streaming databases offer an innovative approach to data processing that combines the familiarity of traditional models with a powerful data transformation engine. This hybrid system provides DevOps and DataOps teams with extremely fast results for complex SQL queries, aggregations and transformations that are impossible or take hours to process in the traditional batch computation model.

    By pre-specifying query results upfront (in materialized views) and then incrementally updating them as new data comes in, streaming databases offer a valuable alternative to traditional databases that require complex ETL pipelines or overnight batch processing.

    As the demand for faster results continues to rise, streaming databases are emerging as a fundamental new building block for data-driven organizations.

    streaming database

    History

    Streaming database concepts originated in capital markets, where fast computation over continuous data is highly valued. The first products were basic event processing frameworks that addressed specific needs within hedge funds and trading desks. However, their creators quickly recognized that SQL works as the declarative language for streaming data just as well as it does for traditional static databases.

    Today, modern streaming databases are most often used downstream of primary transactional databases and/or message brokers, similar to how a Redis cache or a data warehouse might be used.

    Trade-Offs

    Experienced engineers understand that no software stack or tooling is perfect and comes with a series of trade-offs for each specific use case. With that in mind, let’s examine the particular trade-offs inherent to streaming databases to understand better the use cases they align best with.

    Unique Features

    • Incrementally updated materialized views – Streaming databases build on different dataflow paradigms that shift limitations elsewhere and efficiently handle incremental view maintenance on a broader SQL vocabulary. Other databases like Oracle, SQLServer and Redshift have varying levels of support for incrementally updating a materialized view. They could expand support, but will hit walls on fundamental issues of consistency and throughput.
    • True streaming inputs – Because they are built on stream processors, streaming databases are optimized to individually process continuous streams of input data (e.g., messages from Kafka). Scaling streaming inputs involves batching them into larger transactions, slowing down data and losing granularity. In traditional databases (especially OLAP data warehouses), larger, less frequent batch updates are more performant.
    • Streaming outputs on queries – Many databases have some form of streaming output (e.g., the Postgres WAL), but what’s missing is output streams involving any kind of data transformation. Streaming databases allow for streaming output of complex joins, aggregations and computations expressed in SQL.
    • Subscribe to changes in a query – As a side effect of streaming outputs, streaming databases can efficiently support subscriptions to complex queries: Updates can be pushed to connected clients instead of forcing inefficient polling. This is a key building block for pure event-driven architectures.

    Missing Features

    • Columnar optimization – OLAP databases have advanced optimization techniques to speed up batch computation across millions of rows of data. Streaming databases have no equivalent because the focus is on fast incremental updates to results prompted by a change to a single row.
    • Non-deterministic SQL functions – Non-deterministic functions like “RANDOM()” are common and straightforward in traditional databases. But imagine running a non-deterministic function continuously, resulting in chaotic noise. Therefore, streaming databases don’t support non-deterministic SQL functions like “RANDOM().”

    Performance Gained

    • Minimized time from input update to output update – The time between when data first arrives in the streaming database (input) and when changes to results reflect the change (output) is sub-second. Additionally, it keeps up as the dataset scales because results are incrementally updated.
    • Repeated read query response times – When a query or query pattern is known and pre-computed as a persistent transformation, reads are fast because they require no computation: You’re just doing key-value lookups in memory, similar to a cache like Redis.
    • Aggregations – The resources needed to handle persistent transformations are often proportionate to the number of rows in the output, not the scale of the input. This can lead to dramatic improvements in performance in aggregations in a streaming DB versus a traditional DB.

    Performance Sacrifices

    • Ad-hoc query response times – While running ad-hoc queries in a streaming database is possible, response times will be much slower because the computation plan is optimized for continuously maintaining results, not answering point-in-time results.
    • Window functions – A window function performs calculations across table rows that are related to the current row. They are less performant in streaming databases because updating a single input row requires updating every output row. Consider a “RANK()” window function that ranks output by a computation. A single update can force an update to every row in the output.

    Factors Impacting Scalability

    • Throughput of changes – The changes or updates to input data are what triggers work in the system, so changing data will often require more CPU than data that changes rarely.
    • Cardinality of the dataset – The total number of unique keys will slow down read queries in traditional databases. In streaming databases, high cardinality increases the initial “cold-start” time when a persistent SQL transformation is first created and requires more memory on an ongoing basis.
    • Complexity of transformations – Unlike the on-request model in a traditional DB, SQL transformations are always running in a streaming database in scale in two ways:
      • Memory required to maintain intermediate state – Imagine how you would incrementally maintain a join between two datasets: You never know what new keys will appear on each side, so you must keep the entirety of each dataset in memory. This means that joins over large datasets can take a significant amount of memory.
      • Quantity and complexity of transformations – When a single change in inputs needs to trigger a change in outputs in many views, or when many layers of views depend on each other, more CPU is required for each update.

    Use Cases

    Like any software primitive, there are many use cases. Here are some categories of products and services particularly well-suited to streaming databases:

    • Real-time analytics – Use the same ANSI SQL from data warehouses to build real-time views that serve internal and customer-facing dashboards, APIs and apps.
    • Automation and alerting – Build user-facing notifications, fraud and risk models, and automated services using event-driven SQL primitives in a streaming database.
    • Segmentation and personalization – Build engaging experiences with customer data aggregations that are always up-to-date: personalization, recommendations, dynamic pricing and more.
    • Machine learning in production – Power online feature stores with continually updated data, monitor and react to changes in ML effectiveness – all in standard SQL.

    Conclusion

    Streaming databases offer a powerful yet accessible way for data and software teams to leverage stream processing capabilities. By using familiar SQL DB concepts and a stream processor to compute SQL transformations, data and software teams can focus on shipping complex data-driven products quickly, with improved performance and scalability. With a streaming database, organizations have the power to transform data in real-time and build the applications of tomorrow.

  • Data APIs: Realizing the Future of Data Warehousing

    Data APIs: Realizing the Future of Data Warehousing

    Data is the currency of business in today’s digital economy, with organizations collecting mountains of data from their customers, products and services. Enterprises are increasingly turning to data warehouses to store this valuable enterprise data and to make it useful and actionable. Data APIs–both GraphQL and REST–are emerging as a tool to power frictionless and high-quality integrations between applications/services and data in enterprise data warehouses. But what exactly is driving this shift toward using APIs on top of data warehouses? In this article, I will talk about the drivers behind the rise of data APIs on data warehouses, challenges with current approaches to building, operating and scaling data APIs and how organizations can address those challenges.

    The Rise of Data APIs

    Data APIs enable organizations to create powerful integrations between applications without having to go through lengthy development cycles or write complex code. As a result, businesses can deliver improved user experiences faster than ever before by quickly connecting disparate applications and services together using intuitive REST or GraphQL endpoints. Because these endpoints are standardized across multiple systems (e.g., REST or GraphQL), developers don’t have to learn new languages or coding techniques every time they want to connect two different applications together. This makes it much easier for developers to work with multiple systems simultaneously while still ensuring that all applications remain connected via one interface–the API endpoint itself.

    Challenges With Current Approaches

    Despite all these advantages provided by Data APIs, there still remain a few challenges that organizations need to consider when building out their own solutions. For example, many existing approaches require manual setup processes, which can be extremely time-consuming and tedious for developers working with multiple systems at once. In addition, many existing solutions only allow for limited query functionality, which means that more complex queries may not be possible without additional coding or manual setup processes being performed. Finally, some existing solutions require an extensive amount of organizational resources in order to build out a full-fledged solution—time that could be better spent on more important tasks like product development or customer service initiatives.

    How Organizations Can Address These Challenges

    Fortunately, there are now tools available that make it easy for organizations to quickly build out powerful data API solutions with minimal effort required from their developers or IT staff members. These tools provide pre-built templates that can be used as a starting point for any integration project while also allowing users to customize their own solution based on specific needs and requirements without having to write any code themselves. Additionally, these tools allow users access to advanced query functions such as joins across multiple tables so they can get the most value out of their databases without compromising either speed or security either. Finally, these tools also include powerful monitoring capabilities so users can ensure that their data APIs remain running smoothly at all times, no matter how much traffic they receive.

    Benefits of Using an API for Integrating Cloud Data Warehouses

    When it comes to integrating cloud data warehouses into your application or service, there are several advantages that come with using an API as opposed to traditional methods such as manual coding or ETL tools:

    • Increased Flexibility: By leveraging an API, you are able to access data from multiple sources quickly and easily without having to manually code every integration yourself. This also allows your application or service more freedom when it comes to accessing additional data sets in the future; all you need is an existing API connection in place.
    • Improved Security: Since APIs are designed specifically for secure communication between different systems, they offer enhanced levels of protection against malicious attacks such as DDoS attacks or unauthorized access attempts. You can also add additional layers of security by implementing authentication protocols like OAuth or JWT tokens on top of your existing API connection.
    • Enhanced Performance: APIs can be designed with specific performance optimizations in mind; this means that you can create custom solutions tailored towards the exact needs of your application or service which will ultimately result in improved end-user experiences and higher engagement rates from customers who interact with your product or service regularly.

    Data warehouses have become an essential part of modern digital businesses due to the sheer amount of valuable enterprise information they store. However, traditional approaches can often lack the flexibility or scalability needed in order for businesses to stay competitive in today’s ever-changing market environment. This is why smart organizations are using data APIs on top of them for quick integration among disparate applications/services along with advanced query capabilities.

    By leveraging tools designed specifically for this purpose, companies can easily overcome current challenges associated with building out their own solutions while still getting maximum bang for their buck when it comes down to successful implementation and operation over the long term.

  • GraphQL Documentation Generators: How They Work and Why They Matter

    GraphQL Documentation Generators: How They Work and Why They Matter

    GraphQL is an open source query language developed by Facebook that provides a more efficient, powerful and flexible alternative to traditional REST APIs for fetching and manipulating data over the web. It allows clients to specify the shape and structure of the data they need and receive exactly that, no more and no less.

    Unlike REST APIs, which expose fixed endpoints for specific resources, GraphQL enables clients to construct complex queries that describe the exact data they need, with the server returning only that data in response. This allows for more efficient network usage and improved integration with CI/CD pipelines, as changes to the data requirements can be easily made without breaking existing clients.

    GraphQL also offers real-time capabilities through subscriptions, allowing clients to receive live updates as data changes on the server and supports a range of programming languages and platforms through a variety of libraries and tools.

    What are Documentation Generators?

    Documentation generators are tools that automatically create documentation from source code, making it easier for developers and other stakeholders to understand and work with a codebase. These tools parse the source code and extract information about the structure, behavior and dependencies of the code and use this information to generate documentation in various formats, such as HTML, PDF or markdown.

    Documentation generators can help improve the quality of code documentation by automating the process and reducing the risk of errors or omissions. They can also save time and effort for developers, as they can quickly generate documentation without having to write it manually.

    There are different types of document generation techniques, each with its own advantages and use cases. Let’s discuss some of them:

    • Batch generation: A generic technique that involves generating documents in bulk using a set of rules and templates. It’s commonly used for generating reports, invoices and other similar documents that follow a specific format. Batch generation can save time and effort by automating the document creation process and reducing errors.
    • Text block correspondence: Involves creating documents based on pre-defined text blocks. These text blocks can be customized and combined to create different types of documents, such as contracts, letters, and proposals. Text block correspondence is commonly used in document assembly tools, which allow users to create documents by selecting the appropriate text blocks and filling in the necessary information.
    • Forms: A type of document generation that involves creating web-based forms that users can fill out to generate documents. Forms can be used for a variety of purposes, such as generating applications, registrations and surveys. Forms can be customized to match the specific needs of the organization or business.
    • Documentation synthesis: Involves combining data from multiple sources to create comprehensive documentation. This technique is commonly used in software development where documentation needs to be created for different parts of the system. Documentation synthesis can save time and effort by automating the documentation process and ensuring consistency across different parts of the system.

    Documentation Generators for GraphQL

    Documentation generators for GraphQL work by analyzing the GraphQL schema and generating documentation based on the information in the schema. The GraphQL schema defines the structure of the data and the available operations, and it serves as the basis for generating documentation.

    Documentation generators typically follow a similar process:

    • Analyze the schema: The documentation generator analyzes the GraphQL schema to identify the available types, fields and operations.
    • Generate documentation: Based on the information in the schema, the documentation generator generates documentation in a specific format, such as HTML, markdown or PDF. The documentation typically includes information about the available types, fields and operations, as well as any associated documentation, such as descriptions or examples.
    • Include additional information: Some documentation generators may also include additional information, such as query examples, error handling information or performance considerations.
    • Update the documentation: The documentation generator can be run periodically to update the documentation based on changes to the GraphQL schema.

    Documentation generators for GraphQL are typically designed to be flexible and customizable so that developers can customize the generated documentation to meet their specific needs. For example, developers may be able to customize the format of the documentation, include additional information or exclude certain information.

    Documentation Generators for GraphQL: Why They Matter

    Documentation generators for GraphQL are important tools for DevOps teams for a number of reasons:

    • Efficiency: Documentation generators automate the process of creating and updating documentation for GraphQL APIs. This can save DevOps teams a significant amount of time and effort, particularly when working with large or complex APIs.
    • Consistency: Documentation generators can help ensure that documentation is consistent with the GraphQL API and up-to-date. This can help reduce confusion among developers and other stakeholders and improve the overall quality of the documentation.
    • Collaboration: Documentation generators can help improve collaboration among DevOps teams by making it easier to share and review documentation. This can help ensure that everyone has access to the most up-to-date information and that documentation is reviewed and approved by all stakeholders.
    • Accessibility: Good documentation is essential for developers who are working with GraphQL APIs, and documentation generators can help ensure that documentation is clear, concise, and accessible. This can help reduce the learning curve for new developers and make it easier for developers to understand and work with the API.
    • Scalability: Documentation generators can help DevOps teams manage the documentation of large or complex GraphQL APIs, making it easier to keep up with changes and ensure that the documentation is accurate and up-to-date.

    Conclusion

    In conclusion, by automating the documentation process, documentation generators can save time and effort for developers, reduce the risk of errors and inconsistencies in the documentation and provide a common source of truth for different teams and stakeholders. In addition, documentation generators can help DevOps teams better understand the API, improve collaboration and communication and speed up development cycles. 

  • How to Maximize Telemetry Data Value With Observability Pipelines

    How to Maximize Telemetry Data Value With Observability Pipelines

    The software landscape has shifted significantly in recent years as companies digitize their operations and adopt cloud and microservices technologies. The complexity of modern software systems has led to an increasing need for real-time visibility into their performance—resulting in an explosion of telemetry data that is stretching IT budgets and outpacing engineering teams’ ability to find value in that data. In fact, data volume is growing at 23% yearly, while IT budgets are only growing at 5%.

    Managing all this data at increasing cost is further compounded by unpredictable software failures and ever-evolving security threats. As a result, companies are forced to pick which parts of their software to instrument and monitor. To overcome this challenge, a different approach to managing data is required.

    Observability pipelines are emerging as a way to address this problem. They centralize telemetry data (i.e., logs, metrics and traces) from many sources, transform that data to fit downstream needs, and route it to various destinations for analysis. With complete control of the data, companies can sort through large amounts and prioritize what is essential, allowing them to act swiftly to avoid disruptions while reducing costs by only storing the data they need.

    Research indicates 62% of organizations currently have an observability practice. This means 38% operate with limited awareness of what’s happening in their software. Even those with an observability practice can only afford to store some data and have to choose which applications to instrument. This leads to blind spots and, ultimately, issues that are time-consuming to fix due to a lack of data.

    Managing Limited Resources With Observability Pipelines

    Engineers face more demands in today’s fast-paced business environment than ever before. Despite an increase in their responsibilities, the size of engineering teams has yet to grow accordingly. Given this, there has been a rise in full-stack engineers responsible for technologies from the front end to the back end. Short on time and resources, it’s difficult for these engineers to instrument and monitor software.

    Observability pipelines help control the amount of telemetry data using various processors such as sampling, throttling, filtering and parsing, and only forward valuable data to the downstream systems. The rest of the data can either be discarded or stored in a low-cost system, such as Amazon S3. This reduces costs and lets engineers focus on analyzing relevant data to identify issues.

    Engineers can also use observability pipelines to consolidate data from various tools. Many teams use different platforms or open source tools, like Prometheus or Jaeger. The fragmented data makes it hard to solve issues and understand application health. Observability pipelines combine all this data to make it easier to act on.

    It’s also important to remember that real people work on these technical challenges. Engineers face burnout working long hours to keep up with software development demands. Constantly switching between tools and manually moving data adds to the pressure. Moreover, the number of data sources, observability tools, and platforms that need that data are constantly increasing. Embracing open standards such as OpenTelemetry can ease this integration burden. Engineers can swap out tools without extensive testing and integration. The increased control and easy integration enable engineers to focus on delivering business value.

    Making Data More Usable

    Teams across organizations want access to data but struggle to use it effectively. The difficulty stems from the lack of data usability: Telemetry data is unstructured; varying formats make it hard to use; data preparation is time-consuming and sensitive data in logs may lead to compliance violations.

    By most estimates, 80%-90% of data is unstructured, which makes it hard to store and analyze because it doesn’t adhere to traditional data models. Observability pipelines make sense of unstructured data before it reaches its final destination. This is achieved using data processors that shape and transform data to make it more actionable. These processors include advanced parsers that identify and extract relevant information, transforming data into consumable formats that make it easier to work with and analyze.

    The advantage of performing these operations within the pipeline is that the same data can be prepared to fit different use cases downstream. For instance, while one team may require data optimized for visualization and trend analysis, another may require complete data for threat hunting. These transformations are managed from a single control point, eliminating the need to maintain separate data streams.

    Observability pipelines can also help companies meet data compliance requirements. For example, teams can scrub, mask or redact PII data based on a defined key structure before the data reaches a SIEM or audit platform.

    Taking Immediate Steps to Make the Most of Telemetry Data

    Organizations can take immediate action to make the most of their telemetry data. First, they need to adopt a visibility-first rather than a cost-first approach. They should assess what software they’ve instrumented. This will identify gaps in telemetry, which can be used to develop a plan to instrument all software. Next, they need to create a strategy for managing their telemetry data, being particularly thoughtful about how and why the data is stored and analyzed.

    Once a company has taken these two steps, it can decide which tools to use. One of the great things about OpenTelemetry is that it provides flexibility for future decisions about what tools to use.

    As companies invest more resources into building observability practices, they’ll encounter problems processing an extraordinary amount of telemetry data for any meaningful action. Observability pipelines provide a competitive advantage of prioritizing the essential data that will enable companies to make better decisions faster.

  • Four Technologies Transforming Data and Driving Change

    Four Technologies Transforming Data and Driving Change

    Data is getting a facelift in 2023 courtesy of event-driven architecture (EDA). These four technologies are set to transform DevOps and IT operations over the next year–whether it be how companies make use of data or the customer experience they offer.

    1. Applied Observability: A Powerful Source of Decision-Making

    Applied observability has been recognized by Gartner as a key 2023 strategic tech trend at its most recent Orlando Symposium. It ensures that the right data is delivered at the right time for rapid action based on confirmed stakeholder actions rather than intentions. “Observable” data include key digitized artifacts that appear when any stakeholder takes any kind of action. Applied observability feeds these observable artifacts back in a highly orchestrated and integrated approach to accelerate organizational decision-making and allows the business owner to track how long an action took to fully process at every point in the workflow.

    But unearthing these insights requires an event-driven approach to software architecture. Distributed tracing, for example, is a method of tracking application requests as they flow from front-end devices to back-end services and databases. By definition, and in contrast to traditional tracing, distributed tracing can be visualized to show a searchable, graphical picture of when, where and how a single event flowed through an enterprise, regardless of the number of hops the workflow took to fully process.

    Consider the potential traceability and end-to-end observability benefits across a payment ecosystem underpinned by EDA. Embedding distributed tracing into an event mesh emits trace events in OpenTelemetry format so banks can collect, visualize and analyze them in any compatible tool. This empowers them to not only confirm that a given message was published but easily understand exactly when and by whom, where it went, down to individual hops, who received it and when … or why not.

    2. Hyper and Intelligent Automation: Taking Automation to the Next Level

    Gartner and IDC identified both hyper and intelligent automation in their yearly technology trends for 2023. Hyperautomation moves automation up a level, adding more intelligence to automation and using a broader set of tools so that previously un-automatable tasks can be automated. Hyperautomation initiatives come in many different shapes and sizes and are being seen across a wide range of industries, from banking and insurance to manufacturing and health care.

    But hyperautomation requires an organization to be underpinned by an EDA and–more specifically, an event mesh–to maximize success. An event mesh is an interconnected network of event brokers that allows data to be pushed in real-time to parts of the organization where the data is needed. It does this dynamically, meaning that new event types can be added at any time and interest in events can be registered to allow a seamless interchange of data for the applications that are interested in using it.

    Take the aviation industry as an example. An event mesh streams information such as flight routes, delays, cancellations and mileage accruals between applications, connected devices and people anywhere in the world, instantly. With an event mesh, information about events can be continuously streamed to multiple systems; filtered so each system only receives the data it needs. This ultimately enhances the customer experience as passengers, pilots and crews are notified in real-time when something relevant occurs on each and every flight.

    3. The Acceleration of the Event-Driven API Economy

    The API economy has exploded. Research from MIT Initiative on the Digital Economy showed that over a four-year period, businesses using APIs saw 12.7% more growth in market capitalization compared to those that did not adopt APIs.

    But the API economy is changing as event-driven architecture and asynchronous event-driven APIs become increasingly important to companies striving to operate in real-time. Using both synchronous and asynchronous methods can result in an application environment in which system resources are most effectively used.

    AsyncAPI and RESTful APIs will be the next evolution of the API economy. API management platforms will need to adapt quickly to this new reality. As a prime example, Forrester recently spotlighted a global biotech company that is blazing a trail in the API-driven area. As the case study demonstrates, the organization understood that EDA is a complement to RESTful APIs–another tool in their digital business toolbox, filling in the gaps of a REST-only approach.

    As a result, they planned a unified event + REST approach from the outset. Both APIs and events are managed as enabling digital products, with the governance and life cycle of both unified as one process. A platform team builds a unified API + event platform for app dev teams rather than building each as a separate siloed platform. The results include faster speed to market and new business opportunities. The company also embraces FAIR data principles: Data that is findable, accessible, interoperable and reusable. A digital marketplace consisting of both events and APIs makes the data findable and machine-readable via Open API Spec (OAS) and AsyncAPI metadata. Data can then be accessed on demand via APIs and events push data in real-time.

    4. The Metaverse Continues at Pace

    Driven by technologies including digital twins, augmented reality and virtual reality, McKinsey estimates the metaverse will be worth $5 trillion by 2030. Data in real-time will be the vital common denominator to link the digital and physical divide and optimize the Metaverse. McKinsey lists real-time data as a vital element to facilitate adoption of the metaverse in several key industries including retail, banking/financial services and transportation.

    The metaverse cannot be static. It needs to be in real-time, in motion. It needs an EDA and a real-time event mesh to support it.

    For example, if an avatar visits a retailer in the metaverse, the retailer will need to keep track of countless user actions such as how long someone spent looking at a particular product, which products got the most interest, etc. The metaverse will produce a significant amount of data that companies will be interested in analyzing in real-time. This data can be used to make decisions from real-time adjustments to inventory to incentivizing commerce through real-time offers/discounts/coupons and even dynamically pricing products.

    Designing a Data-Driven Future With new Tech

    Together, these four predictions have the potential to revolutionize company operations, communications and decision-making–but to take advantage of their full potential and reap success, these four trends will need EDA by their side.

  • How Database DevOps Fuels Digital Transformation

    How Database DevOps Fuels Digital Transformation

    In today’s real-time, hyperconnected reality, data is a core driver behind most digital strategies. User data is constantly gathered and shared across various technology backends and platforms—as is system performance data to inform engineers. Data is pivotal for everything, from generating machine learning models, refining user experiences and even supporting new data-centric services. But although data is a clear backbone to many enterprises, there is no clear winner regarding which database to use.

    Large organizations often find themselves supporting different database styles, from relational databases to NoSQL and distributed SQL. And as organizations continue to adopt multiple databases, safely managing the processing and storage of data becomes more challenging to enact at scale.

    I recently met with Jakub Lamik, CEO, Redgate Software, to understand the role DevOps, specifically database DevOps, is playing in advancing data-driven digital transformations. According to Lamik, applying DevOps practices for database management can improve performance, increase productivity and help organizations stay compliant. Below, we’ll consider the state of today’s data-driven digital transformations and highlight ways to improve how DevOps is applied to database management.

    State of Data-Driven Digital Transformations

    The 2022 StackOverflow Developer Survey found the most popular databases to be MySQL, PostgreSQL, SQLite, MongoDB and Microsoft SQL Server. But most often than not, businesses deploy more than one of these in production. Redgate’s 2021 State of Database DevOps report found that 70% of businesses had more than one DBMS in use, and 48% had teams working on three or more different databases.

    Engineers operate in increasingly hybrid and distributed technology environments. This isn’t for no reason—in fact, 90% of organizations said multi-cloud is helping them achieve their business goals. Lamik also notes a diverse array of software technologies in practice today, from multiple clouds to decoupled microservices architectures and database types. But the desire to work with diverse toolsets causes fragmentation, which could hinder software delivery efforts. For example, organizations want to deploy the same framework and similar methodologies across different databases, said Lamik, and have it be database- and cloud-agnostic.

    Tips to Improve Database DevOps Operations

    To increase the pace of software delivery, infrastructure automation is often needed. And like other roles, database professionals are increasingly trying to automate operations and free themselves from mundane tasks. So, what are some ways to streamline database DevOps? First off, Lamik encourages adopting a common approach across multi-database environments, using standard toolsets that are flexible and portable. “The organizations that standardize deployments across different database environments can increase the frequency and reliability of deployments,” he said.

    Catching and resolving errors quickly is also becoming increasingly essential to meet objectives. And although DevOps brings many advantages, the CI/CD pipeline has gaps. “Observability supported by intelligent automation is key to closing those security gaps,” said Lamik.

    Thus, he encouraged additional oversight and tracking of changes to ensure stability throughout the database deployment pipeline. Also helpful is encouraging collaboration with security professionals (also known as DevSecOps) and integrating security principles into the DevOps workflow to ensure the process is secure by design.

    Another element of database security involves the proper observability and classification of sensitive data. As Lamik describes, organizations must be careful to distinguish between data used for production purposes and data used for development purposes. Ensuring sensitive data is cataloged correctly is imperative to avoid production data from being breached or accidentally publicly exposed.

    Final Thoughts

    In summary, applying DevOps ideals to database management could be a boon for development efforts. For those who haven’t yet considered it, a DevOps transformation will pay dividends, says Lamik. Furthermore, a common management layer could help wrangle multiple database types and help adopt new emerging database types, such as Snowflake and Postgres. Yet, when acquiring new database toolsets, Lamik admitted it may not be a quick transition. He recommends a land-and-expand strategy rather than a top-down approach (which typically isn’t as effective in empowering individual developers).

    Commenting on the industry’s recent move toward platform engineering, Lamik sees it as the next evolution of DevOps. Whereas DevOps brought helpful toolsets to streamline deployment and digitization, platform engineering is taking these practices and productizing them to be reusable at scale across teams and departments, he explained.

    Also, in the new cloud-native stack, containers are designed to be ephemeral, causing engineers to craft workaround solutions to enable stateful storage on platforms like Kubernetes. Regardless of deployment, data must have proper backup and recovery controls as well. In this new cloud-native paradigm, the role of database professionals is changing, explained Lamik, and “APIs are becoming the focal point of communication.”

  • Improving the DevOps Process for Mobile App Developers

    Improving the DevOps Process for Mobile App Developers

    Development teams—perhaps especially mobile development teams—have heavily invested in systems to automate their processes and accelerate the delivery of mobile apps. From build, test and release to tracking and monitoring, the mobile DevOps team depends on systems like Fastlane, Bitrise, Jenkins, Azure Pipelines and GitLab—and really, that list just scratches the surface.

    The mobile app market moves so fast that automation is the only way mobile DevOps teams can keep up with their competitors and rapidly changing customer expectations. They have to iterate and release rapidly—the more frequently a publisher puts out new features, the more highly customers rate their app. In the GitLab 2022 Global DevSecOps Survey, 70% of respondents said their teams release code every day or every few days.

    Unfortunately, there’s one major component of mobile app development that sits outside of these automated processes in most mobile DevOps teams: Security. For the most part, developers still implement security manually, and the process for ensuring apps are secure mostly comes down to code scanning and penetration tests. Again, from the GitLab survey, 53% of developers are running static application security testing (SAST), but unfortunately, the data from those scans is often not feeding back into developer workflows. Less than three out of 10 teams (29%) pull scan results into a report for developers.

    Additionally, there’s still a large disconnect between the security and development teams. Nearly half (47%) of security pros said that developers miss more than three-quarters of the bugs in the code, leaving them for the security teams to find, and more than half (56%) said it was hard to get developers to prioritize fixing code vulnerabilities. In fact, prioritizing vulnerability remediation was security professionals’ biggest challenge. Additionally, it’s interesting that when it comes to shift left security, the emphasis appears to be on early code scanning, not actually building security into apps earlier in the process.

    Data-Driven Decisions and Security Automation

    First, DevOps teams need to integrate data about the security of their mobile apps early in the process so they can make informed decisions about what protections to incorporate into the next build. Scanning information and results from penetration tests clearly need to feed back to the development team as quickly as possible—there’s no point in doing these tests if the information remains unavailable and not acted on.

    But mobile devices have the capability to collect and send much richer data about the security threats apps are actually facing in the field. By collecting this data, DevOps teams can make data-driven decisions about which threats are the highest priority to combat.

    However, as noted earlier, it does little good to collect data if it’s never used. And the slow pace of manual security implementation does not allow DevOps teams to implement protections quickly or nimbly enough to keep pace with the rapidly changing threat environment. DevOps teams need to automate the build, testing, release, tracking and monitoring of security to the same degree as every other aspect of an app’s development. Specifically, they need:

    • A system that can store, provide version control and audit security in every release
    • An automated system that can build the desired protections into the app within the organization’s existing CI/CD processes
    • Automated verification that the protections slated for implementation are actually included in the release
    • A feedback system from data collected in the field, including data about the effectiveness of protections already implemented. This proves security measures are working and reinforces the value of the DevSecOps process.

    With this combination of data and automation, mobile app developers can transition from DevOps attempting to shift left security towards a DevSecOps process that is completely data-driven. Instead of reacting to the latest threat that’s making headlines in the news or working off gut feelings about the direction the threat environment is moving, DevOps teams can look at trending threat data from their very own apps, slicing it according to growth, geography, device, OS version and a multitude of other filters. As a result, the organization can pinpoint exactly which threats are emerging as the next big thing so they can defend against them early.

    And with the assistance of automation, the DevOps team can keep up with the trending data, building security protections into the app within days or even hours of making a decision on what to include.

    To improve the DevOps process, security has to shift left, but moving code scanning and penetration testing earlier in the process won’t do much good to harden protection without real-time data from the field and automation for security implementation. What DevOps needs is a transition to data-driven DevSecOps.

  • Linux Tweak Brings Big Speedup ¦ DCs in SPAAACE (Redux) ¦ Atlassian Fires 500

    Linux Tweak Brings Big Speedup ¦ DCs in SPAAACE (Redux) ¦ Atlassian Fires 500

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Intel optimizes Linux multithreaded networking, data centers in space (again), and more DevOps toolmaker layoffs. (more…)

  • Five Tips for Moving IT Ops to DevOps

    Five Tips for Moving IT Ops to DevOps

    The era of siloed development and operations functions is rapidly coming to an end. The concept of DevOps has already closed the gap between IT and developers, but as I speak with customers around the globe, one theme continues to arise. Many systems administrators are ill-equipped to handle the shift from IT Ops to DevOps and, in some cases, don’t even know the right questions to ask to ensure they are planning accordingly.

    Having the appropriate equipment and software in place means nothing without people trained and ready to implement the shift from IT Ops to DevOps. As IT strives to support and protect developers continuing to innovate with wild-west-style creativity, how can we empower them with the tools and knowledge to ease the transition? This starts with asking the right questions, but requires IT to think about new things: Automating processes, revamping config management, building efficiencies in data orchestration and standardizing a set of APIs with enterprise-level protections.

    Asking the Right Questions About DevOps and Infrastructure-as-Code

    Enterprises need to look holistically at how infrastructure services are created, delivered and managed. Continuing current practices, such as manual provisioning complicated by different interpretations of imprecise instructions, will continue to lead to snowflake architectures and brittle systems. Organizations must modernize or replace this existing infrastructure to exploit cloud architectures and implement true platforms, actualizing the promise of the cloud.

    Infrastructure-as-code is an underlying framework that removes the manual process of managing and provisioning resources and opens doors to enabling DevOps. Using code to define the infrastructure configuration means that there is greater consistency between platforms. That consistency allows IT to seed the agile processes, methods and technical approaches needed to evolve a DevOps culture.

    To get to a point where infrastructure-as-code is facilitating a DevOps culture, it’s important for IT Ops to understand how to develop an early-stage continuous delivery and deployment pipeline. They also need to begin to seed practices and values native to DevOps for software engineering principles and practices with IT Ops. It’s also important to define success criteria–referred to as the “definition of done” in agile and begin to think about how to implement the infrastructure-as-code using DevOps best practices.

    Don’t Expect Developers to Become Expert Sysadmins

    Developers don’t want or need an extra layer of oversight or management, nor do they want to ask for permission before doing what they do best: Innovating. DevOps can more rapidly provision compute and storage resources and provide infrastructure-as-code while automating DevOps processes, empowering developers to become an innovation engine for their businesses.

    So much of what we’ve seen puts pressure on IT to be proactive when protecting developer cycles in containers and beyond. The NSA, as well as other industry data protection companies, offer tips for IT to stay ahead of these threats, but manual patching and scanning is not only handcuffing IT but stifling innovation in DevOps. The challenge is that IT can’t move as fast as developer cycles and often lose sight of what developers are putting into containers or other IT environments.

    While global data protection restrictions might have an impact on things like machine learning and AI from a consumer experience standpoint, it does not affect how IT might use these technologies to automate protection. It’s also not about extending protection to developer workloads, it’s about having an environment that recognizes and automates protection against threats in all environments. This way, developers and IT don’t have to become system admins or data protection experts. They can focus on the innovation at hand.

    Config Management is Critical to Empowering Developers

    As we continue to prioritize the automation of processes in the developer environment, it’s also important to consider how to move beyond just automating protection and infrastructure while allowing IT Ops to still have an eye on identifying, documenting and tracking changes in the DevOps stack. To do this, DevOps should look to configuration management tools to help manage and automate the architecture through strong community support, customization, scalability and the simplification of complex configurations.

    These tools, such as Ansible, Terraform, etc., can be deployed to meet specific needs and preferences. The point is, once you create infrastructure-as-code, you need a way to configure and manage it that fits the model that’s appropriate for your business. While you can use IaC to configure and manage, you still need to determine how to best implement automation and configuration in the most appropriate environment.

    Data Orchestration Eliminates Repetitive Cycles

    Data orchestration tools bring all data together regardless of where it originates or its format. An ecosystem that can collect data efficiently and convert it to a format that is easily understood by any and all endpoints is critical to the app development process. There’s no need to iterate at multiple levels, on multiple platforms or in different environments. IT can enable developers to innovate and learn from any environment, data warehouse or platform and build applications that can do the same.

    In the same way data orchestration can drive efficiency, there are additional benefits. Automating the collection, analysis and formatting of data saves IT countless hours of doing these processes manually. While there are obvious cost benefits, there are also a few other important benefits.

    To allow developers to continually innovate at a wild west pace, IT/DevOps must also ensure they can meet data privacy compliance and governance policies. Data orchestration allows businesses to delete or store data based on policies preset in the environment. It also allows businesses to ensure that internal policy is upheld throughout the development process. Orchestration then takes that data and makes it workable so that any iterative process can be applied without fear of bottlenecks or additional cycles.

    Standardized APIs Trump Those Created by Unaffiliated Coders

    Today’s developer profile expands far beyond the enterprise. In fact, according to Developer Nation, we can expect a whopping 45 million developers in the world by 2030. These numbers don’t include the unaffiliated coders working out of their basements or garages. That’s not to say that creativity isn’t appreciated or eventually adopted by industry leaders. But the code and software created don’t necessarily abide by enterprise governance and privacy standards. Yet we continue to see wild west developers pulling in anything and any kind of API to ensure it meets the needs of the next business application they are developing. How does IT police this?

    They don’t have to. For IT/DevOps to identify a set of APIs that meet business imperatives and certify them for use within the organization is imperative. Think about a scenario in which a developer has not only a comfortable environment to sit in without learning new skills but also an environment that automates protection, orchestration and gives them a preset library of helpful and open APIs. Businesses can begin to focus on delivery rather than policing and upskilling developers to work within the confines of the business ecosystem.

    As data continues to flow in and out of organizations and more data is being created and stored at the edge, a DevOps culture built for tomorrow’s developers is critical. These tips are a great starting point to ensure you’re installing a future-proof culture of innovation. What are some of the obstacles you’ve faced and overcome during the transition to DevOps?