devopsy.kinsta.cloud

Category: DataOps

  • Snowflake Makes Case for Building and Deploying Apps on Platform

    Snowflake Makes Case for Building and Deploying Apps on Platform

    During a Snowday 2022 event, Snowflake this week moved to make it simpler to build applications on top of its cloud platform for managing data in addition to enhancing the overall security of the platform.

    The company has made generally available Snowpark, a framework for building Java, Scala, SQL or Python applications. Snowflake also committed to making generally available in early 2023 an open source Streamlit framework for building Python applications that Snowflake acquired earlier this year.

    Snowflake also revealed it is adding observability tools to the platform to make it easier to build, test, debug, deploy and monitor data pipelines, including alerting tools (private preview), logging capabilities (private preview), event tracing (private preview), task graphs and history (public preview).

    Torsten Grabs, director of product management for Snowflake, said those capabilities will make it easier to alert DevOps teams to issues that might arise with applications deployed on the Snowflake platform.

    At the same time, Snowflake is also making it simpler to build data pipelines by making a Serverless Tasks capability generally available. In addition, the company is previewing a Schema Inference engine that makes it simpler to ingest data and a Dynamic Tables capability to declaratively manage data pipelines.

    Other capabilities being added to the platform include a public preview of a Query Acceleration Service to make additional compute resources available on demand. Snowflake has also improved query efficiency with join elimination in addition to providing search optimizations.

    There are also now cost optimization tools that enable IT teams to run a cost-benefit analysis of Snowflake services in addition to tools that make it easier to replicate data across multiple cloud computing environments running the Snowflake platform.

    IT teams can now also replicate Streams and Tasks via a capability in public preview that enables data pipelines to seamlessly work on secondary Snowflake accounts and configure failovers using a graphical tool that is available in private beta.

    Finally, Snowflake has also enhanced its cross-cloud Snowgrid Capability to make it easier to share both public and private listings of data in addition to applying tags to and searching masked data.

    The latest editions to the Snowflake platform come at a time when more organizations are starting to adopt DataOps best practices to manage data. Snowflake is now making a case for building and deploying applications on top of a platform that automates the management of the data pipelines used by applications to access data.

    It’s not clear to what degree the Snowflake platform will be employed to host applications, but the company is clearly betting the volume of data residing in its cloud platform will attract a wide range of application developers that, in turn, will need to be supported by DevOps teams.

    Of course, Snowflake is not the only provider of a data warehouse or data lake that has similar ambitions. Regardless of platform, however, the one thing that is clear is that a convergence of DevOps and DataOps workflows is now underway as IT organizations look to streamline the management of the massive amounts of data that now reside in the cloud.

  • Momento Launches Elastic Cloud Caching Service

    Momento Launches Elastic Cloud Caching Service

    Momento this week emerged from stealth to launch an elastic Serverless Cache platform that promises to improve the performance of applications running on databases deployed in the cloud while simultaneously reducing costs.

    Fresh from raising $15 million in seed funding, Momento CEO and co-founder Khawaja Shams said Serverless Cache gives IT teams access to a serverless computing framework for accessing cache memory that can be used to improve the interactivity of applications.

    Cache is, of course, already widely employed to improve the performance of applications running on databases in the cloud. However, organizations today are required to provision excess cache memory to address performance workload peaks. That approach, over time, conspires to increase the total cost of cloud computing, he noted.

    A serverless cache makes it possible to more easily share an elastic pool of memory that can be invoked in seconds via an application programming interface (API) that is itself invoked by adding five lines of code to an application. The Momento cache service can then handle millions of transactions per second that can be spread across multiple applications, said Shams.

    The Momento Serverless Cache service also automatically manages hot-keys, maintains high cache hit rates and keeps tail latencies low.

    Organizations that are already using Momento on either the Amazon Web Services (AWS) or Google Cloud Platform (GCP) include CBS, NTT Docomo and Wyze Labs, Shams said.

    In general, there’s a lot more scrutiny of cloud costs in the wake of the economic downturn, noted Shams. More IT teams are starting to take note of the fact that various cloud-native platforms enable them to more efficiently consume cloud resources, he added.

    The simple truth is that far too many organizations are managing cloud resources in much the same way they managed on-premises IT environments. Capacity is being dedicated to virtual machines to ensure application availability with even less regard for cost because the perception is that the cloud makes it easy to throw hardware resources at any issue. The challenge is the bill for those cloud resources tends to add up at the end of every month. Cloud database services, in fact, are among the most expensive to consume, noted Shams.

    It’s not clear how much organizations will optimize the consumption of cloud resources in the weeks and months ahead, but it’s clear more supervision is going to be required. Organizations have historically let developers provision resources in the name of agility with little to no regard for cost. Now that finance teams are looking to, at the very least, make sure cloud resources are optimally consumed, there will undoubtedly be more reviews of cloud spending patterns.

    In the meantime, many organizations may also decide to accelerate the rate at which they are embracing cloud-native technologies to optimize that spending. One of the more compelling attributes of Kubernetes clusters and serverless computing is they make it easier to scale compute resources up and down as required to eliminate the need to overprovision cloud infrastructure. The challenge, of course, is that the cost of acquiring and mastering those technologies remains significant.

  • InfluxData Delivers Revamped Database for Real-Time Processing

    InfluxData Delivers Revamped Database for Real-Time Processing

    InfluxData has made good on a promise to deliver a next-generation time-series database that enables observability and monitoring tools used by DevOps teams to ingest high volumes of data more cost-efficiently.

    In addition, for the first time, the latest iteration of the InfluxDB database supports the de facto standard SQL query language. Developers can run unlimited time-series workloads and contextualize data by any dimension without restrictions using SQL or existing tools such as the InfluxDB application programming interface (API), Flux scripting language or the InfluxQL query language

    Brian Gilmore, director of internet-of-things (IoT) and emerging technologies for InfluxData, said InfluxDB is now based on a columnar engine, dubbed IOx, that leverages the open source Apache Arrow memory format and the Rust programming language accessed via a multi-tenant InfluxDB Cloud service that company manages on behalf of application developers. Use cases that require processing large volumes of data such as metrics, events, traces and other high cardinality data can now be queried in milliseconds, he noted.

    The goal is to make this latest incarnation of InfluxDB available via a managed cloud service before providing IT teams with a version they can install themselves.

    The arrival of a revamped InfluxDB architecture comes as the amount of data being generated across application environments continues to rapidly expand. Many DevOps teams are especially struggling with ways to process and store all the data being generated by cloud-native applications made up of microservices generating a steady stream of logs, metrics and traces.

    InfluxData, of course, has ambitions for its database that go beyond application development and deployment environments. Edge computing platforms, for example, will increasingly process and analyze data in real-time at the point where it is generated and consumed. Similarly, applications infused with machine and deep learning algorithms will require databases that can process data at scale. Regardless of the use case, however, it’s apparent more IT teams will be embracing DataOps best practices alongside DevOps to manage data as applications are deployed and continuously updated.

    It remains to be seen how many observability and monitoring tools might embrace this latest iteration of the InfluxDB database, but it’s clear DevOps teams should expect to see major advances as the underlying databases upon which these platforms depend continue to evolve. The goal is to ultimately eliminate the tradeoff that now often occurs between how much data is collected and analyzed versus the cost of processing and storing it.

    In the meantime, DevOps teams will need to decide whether they want to manage databases themselves as more managed services become easier to invoke in the age of the cloud. Most database selections these days are made by development teams that aren’t especially eager to take on the tasks normally performed by a database administrator. A managed service essentially outsources responsibility for managing a database to a services provider instead of requiring IT teams to hire a DBA themselves.

    One way or another, however, the speed at which data can now be processed in real-time should sharply reduce the need to rely on existing and cumbersome batch-oriented approaches to processing data.

  • Mezmo Adds Observability Pipeline to Analyze DevOps Data

    Mezmo Adds Observability Pipeline to Analyze DevOps Data

    At the KubeCon + CloudNativeCon North America conference this week, Mezmo launched an Observability Pipeline platform that promises to make it simpler to manage, enrich and correlate machine data.

    Previously known as LogDNA, Mezmo is expanding the scope of its reach to add the ability to augment and analyze data on top of its core log management platform. Over time, that capability will be extended to add support for metrics and traces using the open source OpenTelemetry agent software.

    Mezmo CEO Tucker Callaway said the goal is to provide a framework for centralizing the flow of observability data that is now being generated by a wide range of sources across a DevOps workflow. It’s clear that DevOps teams require a means to enrich and correlate high volumes of data in motion that goes beyond simply pre-processing so it can be stored more cost-effectively, he added.

    The Mezmo Observability Pipeline achieves that goal by ensuring the right data is flowing into the right systems in the right format for analysis, noted Callaway.

    As the volume of data being collected by DevOps teams continues to dramatically increase the need to embrace a set of DataOps best practices is becoming more apparent. DataOps has emerged recently to describe a set of data engineering practices that applies many DevOps principles to the way data should be managed. Mezmo’s Observability Pipeline aims to make it simpler to apply DataOps best practices specifically to the data collected by DevOps teams.

    Of course, each DevOps team will need to decide for themselves how much data they want or need to store long term. However, increasingly DevOps teams are holding onto data much longer as IT environments become more complex, noted Callaway. That creates a need to more efficiently analyze as much data as possible and store it more cost-effectively, he said. Those twin goals, however, often create a physics problem in the sense that they can be diametrically opposed.

    In the longer term, it’s not quite clear how much DevOps teams are embracing observability. Most DevOps teams make use of monitoring tools that enable them to track a set of pre-defined metrics. Observability platforms promise to make it easier for DevOps teams to query a wide range of data types to discover potential issues before they disrupt an IT environment. The challenge is that many DevOps teams may not know what queries to form to discover those issues. It’s likely that, over time, machine learning algorithms will be increasingly applied to observability data to make it simpler to surface potential issues.

    In the meantime, however, the need for DevOps teams to find ways to manage all the data being generated by multiple types of applications is becoming a higher priority. In fact, in the absence of any meaningful way to manage all that data, the benefits of an investment in an observability platform is likely to be limited, at best.

  • Where Does Your Data Go?

    Where Does Your Data Go?

    One of the most interesting developments in security and compliance in recent years is the ability to follow a piece of data through an application from input to consumption and see each bit that touches it. For me, the reason this is so interesting is that it allows postmortems to actually determine exactly what was lost in a data breach. “Oh, that library was compromised? Okay, let’s see what inputs that library actually touches …” It’s so very cool, given the free-for-all that data has traditionally been in applications.

    I’m not a huge compliance fan for a variety of reasons, but this is one great result of mandating compliance. I don’t think that we would have this level of data tracking without compliance requirements driving it.

    So what can we do with this information? That’s the interesting part that I think we’ll see developing over time. If we know everywhere that a given piece of data is used, we have a powerful bit of development information. It is still early days for implementation–this is not easy stuff–but think about it. I could see a future where all redundant data access is eliminated. If we’re getting the same set of data from multiple sources, and it is being fed to the same internal consumers, then we can standardize the you-know-what out of it. A singular library to standardize formatting, a singular library to standardize look-ups, etc.

    For the most part, we currently validate and format on input. Like straight away as the data is brought into the system. That is a great idea for a system where we are uncertain where the data is going, and are sure that some attacks are coming through those inputs. But what if we discovered 10,000 more lines of code to maintain than is necessary? What if data tracing allowed us to eliminate that code by centralizing those functions? We would have to harden the routine doing the validation and formatting–because it will be getting dirty data–but no more so than we are doing literally everywhere today.

    If we can reduce source code by using a tool that is designed to improve security, is that not the ultimate shift left? The volume of source code that handles data entering the system is staggering, also. For enterprise systems, most of what we develop is actually data processing, no matter how convoluted it gets. The data store is the central repository and apps are simply accessing and modifying it. So shift left with regard to data is huge.

    Consider it. I know that you are all super-busy with other projects, but less source code lightens the load on everyone; take a look at data tracing functionality in products you already own (mostly available in vulnerability management and source code scanning toolsets), and see if it suits your needs. It might save everyone time and will help with some data-centric compliance requirements, too. That leaves you more time to make systems better.

  • SaaS Data Backup and the API Bottleneck

    SaaS Data Backup and the API Bottleneck

    The need to protect SaaS data has never been greater. A recent global survey from Odaseva found that 51% of ransomware attacks are targeting SaaS data, and they are more likely to succeed (52%) than were attacks on cloud, endpoint and on-premises data. 

    But there are plenty of reasons beyond the threat of ransomware to protect SaaS data. Employees accidentally—and maliciously–delete data. And while outages are not common, they do happen, and when they do, it can cause a lot of pain. Atlassian, for instance, suffered a two-week outage in April. If the data isn’t backed up, the organization is completely at the mercy of the provider to get the service back online before they can access it.

    And though all respondents said their organizations say they have some form of SaaS data protection in place, 57% said they still have SaaS data that remains unprotected. Likely, that’s why only half of respondents whose SaaS data had been successfully encrypted by an attack said they were able to recover all of it, compared to more than eight in 10 (81%) who were able to do so for on-premises data.

    That’s not surprising, because backing up SaaS data is very different from on-premises data protection. For starters, of course, IT has limited control, at best, over the application and data. IT will need to rely on APIs to back up and restore SaaS data, and those API calls will have hard caps. SaaS providers almost always run their services on a multi-tenant architecture, so they need to make sure API resources are available for everyone. To accomplish this goal, they set hard caps on API calls, typically over a 24-hour period, which means that IT must carefully manage API usage for backup and restore operations. And with many different APIs with different capabilities and limitations to choose from, IT will need to make some hard, complex choices. 

    Navigating APIs With Hard Caps

    Managing API usage is important, and not just for backup. These same APIs are also vital for connecting other applications integrated into the SaaS application. And for core SaaS apps, such as CRM or ERP, there will likely be many such integrated applications that require the use of those APIs. 

    To get the best performance, IT needs to be very careful with the APIs it uses for backup and recovery. Let’s take Salesforce as an example. The Salesforce REST API can transfer up to one million records per hour, while the BULK API can do up to ten times more in the same amount of time. And if you use parallel calls to multiplex data, the BULK API can do up to 300 million. The upshot is that the API you use can make a huge difference in your recovery point objectives (RPOs) because transferring more data faster means more backups throughout the day.

    Unfortunately, Salesforce customers won’t be able to depend on BULK API for all their backup and recovery needs. Some objects can only be accessed via the REST or other APIs, for instance, and that BULK API will also be useful to other apps—you don’t want to hit that hard cap and leave other applications unable to connect. And you can’t always use the same API to write data as you do to read it.

    The SaaS backup infrastructure has to achieve a balance of API usage, which must be closely managed. And don’t forget about restore, which can consume a ton of API resources. Make sure you have enough to account for a big restore—you really don’t want to hit the API cap while you’re trying to get your users back in business after a data loss event.

    Lastly, APIs change over time and the backup system has to be able to adapt. You’ll need to make sure you keep up with API documentation so the system can be updated regularly; otherwise, data may go unprotected.

    It’s now becoming widely accepted that SaaS data needs to be protected, but building a SaaS backup solution that meets an organization’s RPOs and RTOs is more complex than many in IT may realize. A strategic plan for API management is a must-have for any SaaS backup management solution. 

  • When the Cloud and the Edge Are the Same Thing

    When the Cloud and the Edge Are the Same Thing

    Edge computing remains a growing and exciting new architecture for enterprises across multiple industries. Distributing computing power across a decentralized network gives new opportunities for AI- and IoT-based applications. Demand is such that Gartner predicts by 2025, 75% of enterprise-generated data will be created and processed on the edge. 

    However, the cost and latency of sending all this data back to the cloud for processing are significant. To retain profitability, enterprises will need to keep as much data on the edge as possible while also moving processing to the edge. The benefits of these requirements are many, including:

    • Real-time processing, lower latency, faster answers
    • Resilience and availability, not relying on a stable connection to the cloud
    • More resource- and cost-efficient, less data shipping

    I believe organizations achieve these requirements and realize these benefits via a concept I call the cloud-to-edge data plane.

    A cloud-to-edge data plane unifies cloud and edge into a single solution that allows you to define your data and business logic once, package it up as a service and simply deploy it without having to worry if it should run in the cloud or out at the edge (which might change over time). This frees up the data plane automatically and adaptively to move services around, optimizing data access, coordination and replication, ensuring physical co-location of data, compute and user for the lowest possible latency with the highest throughput and availability.

    Choosing between cloud and edge is not a binary, black-and-white decision but has a lot of shades of gray in between. From an architectural perspective, the edge consists of many hierarchical layers between the cloud and the devices; each layer is further away from the cloud but closer to the end-users—a continuum. 

    Where you need your services to run in this continuum is very much use-case-dependent and might change over the lifetime of an application, depending on usage patterns or changes in features and business requirements. Critical parameters like latency, throughput, consistency, availability, scalability, compute resources, etc. change drastically as you move through this continuum. This results in both new opportunities and challenges:

    edge cloud

    One of the new services enabled by this architecture is mobile location transparent services. This new class of services is a thing of beauty. It means services are free to run anywhere and collaborate with other services without restrictions. It also allows the underlying platform to optimize overall system behavior by relocating services as needed, based on changes in usage, resources available, communication patterns, failure scenarios, and more. 

    If services are not location-transparent, then the system’s runtime topology needs to be part of the design and encoded into the services themselves. In practice, this means hardcoding decisions such as: 

    • If the communication between services is local or distributed—allowing for concurrency or distribution
    • If services are to be considered stationary or mobile—pinned to a specific physical address or free to be relocated
    • How the services can respond to failure—self-healing bulkheads that can be migrated to other nodes if needed or tied at the hip with the failed node

    Locking in decisions like this upfront will make the system harder to run efficiently, evolve, adapt and maintain over time. These should not be design or development decisions, not deployment decisions, but runtime decisions, not affecting the code. 

    It’s hard to predict the future. This includes knowing how an application will be used over time, how changes in usage and load patterns will evolve, and how future features or changes in business requirements will impact the initial design, assumptions, and runtime/operational characteristics. 

    Not having to take these decisions or worry about how they will be maintained efficiently but only focusing on API, data model, and business logic reduces time to market and overall risk substantially. 

    Here’s my vision for data…

    Data always exists wherever and whenever needed, but only for the time it is needed.

    • Having the latest and correct data always available to you right when you need it and wherever you happen to be located—in the central cloud or at the far edge—sounds like a developer’s dream.
    • On the other hand, it’s been said that “laziness is a virtue,” which is very much true of data management in a distributed system. Replicating data to locations where it will never be needed is a loss (unless needed for resilience/failover). The same is true for holding on to data longer than needed (holding resources hostage and forcing needless consistency mechanisms). 
    • Balancing this is very hard and should ideally be done by the platform adaptively at runtime at a global system level, rather than by the developer.

    Data is always co-located with processing and end-user—ensuring ultra-low latency, high throughput and resilience. 

    • If not, as in traditional three-tier stateless architecture, data is often somewhere other than where you need it to be and therefore has to be fetched before being used and pushed out to storage after modification. This increases latency, reduces throughput and puts the service at the mercy of the availability of the storage service.

    Data and compute move adaptively together with the end-user. 

    • Many use cases at the edge serve users on the move (e.g., cell phones, cars). To serve these users as efficiently as possible, their data and compute need to move geographically with them. 

    Data is injected into the services on an as-needed basis—automatically, timely, efficiently and intelligently. 

    • Data storage management is often very intrusive in application design and business logic: How to map domain objects to DB tables (OR-mapping), transaction management, storage API calls for fetching and updating the state, etc. 
    • Externalizing all of this to the platform and instead having the data injected into the service when new data is available (just as we are used to with events in a pub/sub system) makes the code easier to design, write, understand and maintain. 
    • Additionally, if the service is doing all data management itself, then it’s a black box to the platform. The platform has no idea what access patterns have been used, on what data, when data can be safely cached, replicated, sharded, prefetched, shared, etc. On the contrary, if the platform is doing all the data management, then it can look broadly across all services and safely optimize data management for the overall system holistically. 

    I believe RedMonk analyst Steve O’Grady’s concept of vertical integration fits well into this narrative. It all leads to what I feel is a distillation of a cloud-to-edge programming model (a continuum, as described previously). This model is about abstracting away everything that is not core business value, leaving the three things that can never be delegated: Data model, API and business logic. 

    As Timothy Keller said, “Freedom is not so much the absence of restrictions as finding the right ones, the liberating restrictions.”

    By using a serverless developer experience (DX), I believe the answer to extending real-time data and computing to the edge can be realized in a fully managed developer PaaS model. That model abstracts away all infrastructure into a single declarative programming model and effectively unifies cloud-native and edge-native development. This is “vertical integration” and the cloud-to-edge data plane vision.

    Conclusion

    Today, cloud computing and edge computing are typically treated as separate domains. However, rapidly evolving demands for intelligent low-latency processing of edge-generated data are creating requirements for a new approach combining cloud and edge into a single continuum. With cloud-native applications, architects and developers have already embraced the concepts required to build services that run in an essentially infinitely scalable (but logically centralized) cloud infrastructure. Extending this thinking to a fully decentralized infrastructure, including the edge, is a natural next step in enterprise application architecture. Programming models that enable a data-centric, location-transparent approach combined with intelligent runtimes that automatically enable data and processing to be co-located where and when needed will allow developers to take full advantage of such a wholly distributed edge-to-cloud environment.

  • Cisco Unveils 800G Networking Platform to Advance DataOps

    Cisco Unveils 800G Networking Platform to Advance DataOps

    At the Open Compute Project (OCP) Global Summit conference, Cisco announced it developed an 800-gigabit switch that consumes significantly less power than the previous generation of its networking equipment.

    Thomas Scheibe, vice president of product management for cloud networking for Cisco’s Nexus and ACI product line, said the throughput provided by the latest 7-nanometer iteration of Cisco One ASIC processors is primarily needed by organizations processing massive amounts of data to train artificial intelligence (AI) models either in a local data center or in the cloud. The challenge is that many of the organizations are looking to build AI models that process massive amounts of data while simultaneously reducing their IT infrastructure’s carbon footprint, he noted.

    Cisco has been able to achieve that goal by continuing to invest in proprietary ASIC processors; they are at the core of its networking portfolio to improve data operations (DataOps), said Scheibe. Many IT teams are also looking to replace legacy network routers and switches that consume more power and reduce energy costs by as much as 77%, added Scheibe. In terms of climate impact, Cisco claimed its 8111-32EH switch can now provide 25.6Tbps at 15% of the power requirement. Cisco projected that switch will save about 10,000 kg CO2e per year compared to a 12.8Tbps switch. That equates to the greenhouse gas emissions generated by an average gasoline-powered passenger vehicle driving 26,155 miles, according to Cisco.

    IT teams also have the option of deploying either Cisco’s network operating system or SONiC—Software for Open Networking in the Cloud—network operating system (NOS).

    Cisco is also providing IT organizations with the option to configure its latest routers and switches to run at either 800, 400 or 100 Gigabits with an eye toward upgrading throughput sometime in the future.

    In general, DataOps as an IT discipline is maturing as organizations realize they need to implement best practices to optimize data flows across their organization. Cisco is giving IT teams the option of employing a standard Ethernet fabric or an alternative fabric that increases throughput by predicting which packets need to be delivered to a specific location based on their attributes and the behavior of previous network traffic. As the volume of data that needs to be accessed by low-latency applications continues to expand, DataOps will become a more critical IT discipline. Historically, storage administrators tended to be responsible for data management, but as these applications continue to proliferate, a new class of DataOps engineers is emerging to optimize the flow of data across distributed computing environments.

    In the longer term, it’s not clear how much carbon dioxide emissions are factoring into IT decisions these days, but there are organizations that have begun to track it as part of an effort to lower their carbon emissions. Most cloud service providers have also committed to being carbon neutral. Cisco is betting achieving that goal will require IT infrastructure upgrades at a time when the amount of data that needs to be processed will only continue to exponentially increase.

  • 5 Ways to Transform DataOps With Human-in-the-Loop Automation

    5 Ways to Transform DataOps With Human-in-the-Loop Automation

    We are in the middle of a data renaissance. Today, it’s not just about data instrumentation but also learning how to make DataOps a real business advantage for the entire organization. Data has become an inextricable part of products, used to enhance the quality of user experiences. Reliable access to data and the integrity of that data is imperative to drive innovation and business success.

    But just like any process, problems will arise. There will be pipelines that break or data that isn’t instrumented properly. The question then becomes, how can teams more quickly identify problems and continually improve? How can the journey toward data integrity become one of iteration and improvement?

    DataOps teams are by no means strangers to automation—automated CI/CD pipelines and infrastructure-as-code (IaC) are a huge part of their skill set. But there are some components of automation that are less pervasive throughout DataOps, ones that can help promote data integrity and encourage a culture of improvement. 

    It all starts with a different approach to automation—a human-in-the-loop (HITL) approach. Human-in-the-loop automation enables parts of a process to be fully automated while enabling humans to step in at critical points to take action, make decisions and decide the path forward.

    Here are five ways to use human-in-the-loop automation to drive continuous improvement and achieve optimal data integrity.

    Create Better, Earlier Interruption of Problems

    As is life, inevitably things break, issues arise and teams need to step in to remediate. The goal should be to enable better interruption of these problems when they do occur—identifying issues earlier and bringing the right stakeholders together more easily and swiftly.

    HITL automation can help teams not just identify issues earlier but kickstart the incident response and coordinate across teams and stakeholders. Ideally, automated workflows can be triggered from an incoming signal; for instance, from an observability tool like DataDog or BigPanda. An incident response could be started automatically, triggering further automation—like creating tickets, starting a Zoom meeting, creating a Slack channel and bringing together the right team members. 

    By instantly bringing the right people together, the team can investigate collaboratively and more quickly understand the state of the issue. HITL automation should then enable teams to run further scripts to take action and remediate as needed.

    Bake Improvement Into the Process Through Self-Documenting Workflow

    Many companies today are asking, “How do we make sure that we have that full integrated loop, down to the collection point?” Communication between teams is critical to this endeavor. But even before teams can collaborate on improvements, they need to understand the current state of their data so that if, for instance, a team finds that a report isn’t providing what’s needed, there’s a really clean way to get that communication back to the DataOps team to improve the instrumentation in the first place.

    HITL automation can bake improvement into DataOps workflows by automatically documenting every human and machine action throughout a process. Because it’s connected through APIs to all a team’s tools and services, a self-documenting workflow slurps up every action and creates an audit trail. This audit trail is the basis by which teams can more accurately understand not just the integrity of their data, but also how teams are implementing and running processes.

    Bring Human Data to the Forefront of Improvement

    DataOps teams play a big role in making sure the right data is being collected. However, a lot of people are still using systems that are very manual. They may have a good understanding of their log and monitoring data but still don’t have a good sense of the human data around their organization—what processes and workflows people are running.

    Understanding how humans run processes is just as important as understanding technology problems. The self-documenting workflow described above should include documentation of human data, from manually-run scripts to Slack or Teams conversations and, ideally, even auto-update tickets so even maintaining a system of record becomes less manual. Only by looking at the full picture—of human and machine data together—can teams find areas of improvement in their automation, communication and workflows.

    Take an Incremental Approach

    Don’t let “perfect” be the enemy of “done”. Every great transformation I’ve seen started with a small team, a great mandate, and strong support from the executive team. Adding tons of new technology and hiring more people may not be the silver bullet teams need.

    Instead, teams should look at the expertise they have in-house first—the people, technology and processes. Then, begin by taking an incremental approach. HITL promotes incremental, approachable automation by enabling teams to first codify institutional knowledge, analyze, learn and then begin to automate pieces in small batches.

    Diving head-first into a fully automated approach can leave teams in a worse state than nothing at all. Automation is a journey—we learn by doing, by recording human and machine data and then, using insights as our guide, begin to find real value in small doses of automation. These small steps can make a huge impact and, over time, add up to substantial business value.

    Buy for Industry Standards and Build for the Gaps

    As I mentioned above, teams should first start by evaluating their current resources. What skill sets and technology do they already have and how can they use those resources to achieve their data goals? At the same time, many teams starting their automation journey face challenges—lack of development resources, scripts that live on one person’s machine, difficulty bringing all the human and machine data together into a recorded timeline.

    To help data teams effectively and reliably implement HITL automation, investment into an off-the-shelf platform can help bridge the gaps teams and organizations have. Platforms should enable automation through low-code interfaces while providing code-level customization. This combination enables teams to buy for industry standards while building for the gaps—not managing the platform means development resources can be focused on expanding and improving automation.

    Platforms should enable teams to easily connect to APIs and adapt to custom APIs. The continuous change of DevOps processes and tooling coexist with rapidly changing APIs. Therefore, the most promising automation platforms must manage the complexity of APIs, enabling users to focus on the intent, not the mechanics of custom APIs. Automation technologies should seamlessly handle identity, authentication, pagination and caching. 

    Find Leverage With HITL Automation and Drive Business Impact

    As data becomes increasingly ingrained in product life cycles and user experience, automation holds the key to faster innovation, product quality and enhanced reliability. Creating a culture of continuous improvement and implementing technology like HITL automation that promotes data integrity throughout the product life cycle is where I’ve seen the greatest leverage for organizations. 

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2022 – October 24-28, 2022.

  • Where DataOps and Opportunities Converge

    Where DataOps and Opportunities Converge

    In today’s data age, getting data analytics right is more essential than ever. A robust data analytics implementation enables businesses to hit key performance metrics, build data and AI-driven customer experiences (think ‘personalize my feed’) and capture operational issues before they spiral out of control. The list of competitive advantages goes on, but the bottom line is that many organizations successfully compete based on how effectively their data-driven insights inform their decision-making.

    Unfortunately, implementing an effective data analytics platform is challenging due to orchestration (DAG alert!), modeling (more DAGs!), cost control (Who left this instance running all weekend?!) and fast-moving data landscapes (data mesh, data fabric, data lakehouses …). Enter DataOps. Recognizing modern data challenges, organizations are adopting DataOps to help them handle enterprise-level datasets, improve data quality, build more trust in their data and exercise greater control over their data storage processes.

    What is DataOps?

    DataOps is an integrated and agile process-oriented methodology that helps businesses develop and deliver effective analytics deployments. It aims to improve the management of data throughout the organization. 

    While there are multiple definitions of DataOps, below are common attributes that encompass the concept while going beyond data engineering. Here’s how we define it:

    We broadly define DataOps as a culmination of processes (e.g., data ingestion), practices (e.g., automation of data processes), frameworks (e.g., enabling technologies like AI) and technologies (e.g., a data pipeline tool) that help organizations to plan, build and manage distributed and complex data architectures. DataOps includes management, communication, integration and development of data analytics solutions, such as dashboards, reports, machine learning models and self-service analytics.

    Why DataOps?

    DataOps is attractive because it eliminates the silos between data, software development and DevOps teams. The very promise of DataOps encourages line-of-business stakeholders to coordinate with data analysts, data scientists and data engineers. Via traditional agile and DevOps methodologies, DataOps ensures that data management aligns with business goals. Consider an organization endeavoring to increase the conversion rate of their sales leads. In this example, DataOps can make a difference by creating an infrastructure that provides real-time insights to the marketing team, which can help the team to convert more leads. Additionally, an Agile methodology can be employed for data governance, where you can use iterative development to develop a data warehouse. Lastly, it can help data science teams use continuous integration and continuous delivery (CI/CD) to build environments for the analysis and deployment of models.

    DataOps can Handle High Data Volume and Flexibility

    The amount of data created today is mind-boggling and will only increase. It is reported that 79 zettabytes of data were generated in 2021 and that number is estimated to reach 180 zettabytes by 2025. In addition to the increasing volume of data, organizations today need to be able to process it in a wide range of formats (e.g., graphs, tables, images) and with varying frequencies. For example, some reports might be required daily, while others are needed weekly, monthly or on demand. DataOps can handle these different types of data and tackle varying big data challenges. Add in the internet of things (IoT), such as wearable health monitors, connected appliances and smart home security systems, and that introduces another variable for organizations that also have to tackle the complexities of heterogeneous data as well. 

    OK, so, how can we make this a reality? First, to manage the incoming data from different sources, DataOps can use data analytics pipelines to consolidate data into a data warehouse or any other storage medium and perform complex data transformations to provide analytics via graphs and charts.

    Second, DataOps can use statistical process control (SPC)—a lean manufacturing method—to improve data quality. This includes testing data coming from data pipelines, verifying its status as valid and complete, and meeting the defined statistical limits. This enforces the continuous testing of data from sources to users by running tests to monitor inputs and outputs and ensure business logic remains consistent. In case something goes wrong, SPC notifies data teams with automated alerts. This saves them time as they don’t have to manually check data throughout the data life cycle.

    DataOps can Automate Repetitive and Menial Tasks

    Around 18% of a data engineer’s time is spent on troubleshooting. DataOps enables automation to help data professionals save time and focus on more valuable high-priority tasks.

    Consider one of the most common tasks in the data management life cycle: Data cleaning. Some data professionals have to manually modify and remove data that is incomplete, duplicate, incorrect or flawed in any number of ways. This process is repetitive and doesn’t require any critical thinking. You can automate it by either setting customized scripts or installing a built-in data cleaning software tool.

    Additional processes that can be automated via DataOps include:

    • Simplifying data maintenance tasks like tuning a data warehouse
    • Streamlining data preparation tasks with a tool like KNIME
    • Improving data validation to identify flags and typos, such as types and range

    Building Your Own DataOps Architecture 

    To develop your own DataOps architecture, you need a reliable set of tools that can help you improve your data flows, especially when it comes to crucial aspects of DataOps, like data ingestion, data pipelines, data integration and the use of AI in analytics. There are a number of companies that provide a DataOps platform for real-time data integration and streaming that ensures the continuous flow of data with intelligent data pipelines that span public and private clouds. Looking to increase the likelihood of success of data and analytics initiatives? Take a closer look at DataOps and harness the power of your data.