devopsy.kinsta.cloud

Tag: data warehouse

  • Dynatrace Adds Grail Data Lakehouse to Observability Platform

    Dynatrace Adds Grail Data Lakehouse to Observability Platform

    Dynatrace today announced it has added a data lakehouse, dubbed Grail, to the Dynatrace Software Intelligence Platform to make it simpler to aggregate all the observability data an IT team collects.

    Bob Wambach, vice president of product marketing for Dynatrace, said Grail will initially drive analytics of log data but in time will also be applied to additional types of application development, security and business intelligence data.

    The goal is to streamline processes such as data indexing and rehydration across a unified observability platform that eliminates the need to maintain multiple disparate repositories, he added.

    Grail, at its core, is a causational data lakehouse based on a massively parallel processing (MPP) analytics engine that will initially be made available on the Amazon Web Services (AWS) cloud. IT teams can use the Dynatrace Query Language (DQL) to query data stored in the data lakehouse. Additionally, the Davis artificial intelligence (AI) engine that is already embedded within the Dynatrace Software Intelligence Platform will be able to apply machine learning algorithms to the data stored in Grail.

    Data lakehouses have recently emerged as an approach to enable organizations to apply the structure of a data warehouse platform to massive amounts of data that previously would have been stored in a data lake. In effect, a data lakehouse combines the attributes of both types of platforms for centralizing the management of data. One of the biggest issues with any approach to observability is the massive amount of data that winds up being stored to allow DevOps teams to track issues over time.

    Of course, it’s up to each individual IT team to determine how much data they want to save to drive analytics, but the more data they save the more cloud storage costs will naturally increase. Grail provides a purpose-built means of consolidating data silos in a way that reduces the total cost of observability, noted Wambach.

    In general, because of the need to embed a data lakehouse capability, most observability platforms will be consumed as a software-as-a-service (SaaS) application. That means the number of IT teams that will likely build and maintain their own observability platform will be limited. IT teams are also now struggling to manage increasingly complex IT environments made up of both cloud-native applications built using microservices and legacy monolithic applications. In fact, a global survey of 1,303 CIOs and senior IT practitioners commissioned by Dynatrace found 71% of respondents believed the explosion of data produced by cloud-native technology stacks is beyond the ability of IT staffs to manage without help from an AI platform.

    It’s not clear how quickly IT teams will be migrating from legacy monitoring tools that are designed to track specific metrics to observability platforms that promise to make it easier to identify the root cause of an issue by simplifying the launch of queries against a massive pool of data. However, as organizations continue to become more dependent on software, it’s apparent that the level of tolerance for any kind of disruption is steadily declining; it’s now more a question of when rather than if a different approach to managing IT will be required.

  • DataOps: How to Turn Data into Actionable Insights

    DataOps: How to Turn Data into Actionable Insights

    Enterprises‌ ‌have‌ ‌struggled‌ ‌to‌ ‌collaborate‌ ‌well ‌around‌ ‌their‌ ‌data, which can impact everything from digital transformation to advanced concepts like AI and ML.‌ ‌DataOps is not without challenges; building, managing and scaling data pipelines requires careful thought around reusability, portability across infrastructure and applications and long-term maintenance and governance. And these are just a few of the issues facing enterprises. For this reason, DataOps technology stacks need to focus on providing key capabilities include data extraction, integration, transformation and analysis.

    The Importance of DataOps

    The enterprise is undergoing a seismic shift from siloed application development and data repositories to more composable and reusable architectures. Additionally, there is a growing demand for speed and agility as well as the influx of disruptive technologies such as the cloud, IoT and AI. Organizations that want to win big must act and adapt quickly to deploy and scale new software and solutions that provide customers with a superior experience and satisfy their rapidly evolving needs. To do that, they also must be able to rapidly aggregate, integrate and analyze data sources.

    Even with this need for speed, the tools and processes for the creation and processing of data are not standardized‌ in‌ a way that promotes rapid innovation‌ and ultimately helps organizations transform. DataOps is critical to address the challenges involved in acquiring, storing and governing data. In addition, companies are dealing with increasing complexity in their IT environments. In a recent survey, more than 80% of enterprises have a hybrid cloud or multi-cloud strategy. In order to have cost-effective and secure management of increasingly large amounts of data, enterprises must adopt DataOps.

    Managing Data Growth

    DataOps has been around for about a decade, but has recently gained momentum because of the overwhelming challenges companies face today in dealing with large, complex sets of data that are being generated at increasingly faster rates. With new technologies like the internet of things (IoT), cloud computing and the power of big data now integrated into everyday use, companies are generating at least 50 times more data than they were just five years ago. And with more data comes a need for greater efficiency and higher demand for data experts.

    DataOps potential impact on businesses might lead you to think of it as a radical new methodology. But many companies have already been using similar practices to deal with some aspects of data management, particularly around data warehousing and analytics. And just like DevOps, DataOps isn’t a product, but rather a cultural shift supported by many products—many existing products. So it’s important for businesses looking to adopt DataOps to consider what they already have, such as enterprise data warehouses and ETL tools, and what they may need to acquire, replace or modernize. In the end, companies will end up with several systems to support the data pipeline.

    DataOps and Big Data Challenges

    Most enterprises have invested in significant data infrastructure to extract, load and store their data to take advantage of Big Data analytics and technologies. However, these infrastructures are often layered, hard to manage and full of legacy tools that hinder the transfer and integration of data. Additionally, organizations have invested significant resources in tools such as data warehousing and data marts. These data warehouses have been used to model data but have been typically implemented with a predefined data model. A business intelligence (BI) system, an enterprise data warehouse or any other set of tools might assist in the execution of a data transformation job.

    The technological revolution is creating unprecedented opportunities to make things happen with data. By analyzing the data that powers modern business, we are able to provide value in every dimension of our organizations. As is the case with every transformation, the benefits of building an integrated, self-service analytics environment will be realized by users rather than IT. Leveraging the power of data science, business leaders can achieve operational excellence and compete effectively against the growing waves of competitors. For the data science and analytics staff to be successful, they must be exposed to all of the necessary technology and processes to support effective use of data.

  • Understanding Data Storage: Lakes vs. Warehouses

    Understanding Data Storage: Lakes vs. Warehouses

    Now more than ever, companies are looking for new ways to incorporate data analytics into their daily operations and leverage data-driven insights to improve business functions. They are frequently turning to complex data for tasks like machine learning and artificial intelligence, which are becoming necessary to understand and reach customer segments across industries. However, understanding data storage is a key factor in developing a successful strategy. The two most common storage formats are data lakes and data warehouses – but there are benefits and pitfalls to each that organizations must understand in order to properly capitalize on them.

    Data Lakes vs. Data Warehouses Use Cases

    One effective way to better understand the different functions of data lakes and data warehouses is to focus on the end use case. Typically, there are three types of data consumers: farmers, explorers and executives.

    • Farmers need data and information to execute their day-to-day activities. They must report on the key performance indicators (KPIs) required to execute their job by providing data in a structured format.
    • Explorers are users who want to experiment with data, look at new types of data pools and gather specific insights. If data is presented in a fixed format, it hinders their progress, as they seek data that doesn’t have a pre-defined structure.
    • Executives are responsible for making business decisions, and often require information presented at different aggregates, with the ability to narrow in on data subsets as needed. However, they require information that is somewhat structured, and speed is critical for them.

    If we look at these perspectives more broadly, they essentially boil down to two key types of data usage – the exact needs that data warehouses and data lakes are designed for. Farmers and executives are largely served through data warehouses, which require information to be timely and structured. Some of the main use cases for data warehouses include operational analytics, business intelligence and predictive analytics for data science and machine learning applications. Explorers are best served through data lakes, since the data formats are not predetermined. Data lakes are often most effective for use cases like a centralized data catalogue that enables organizations to view all sources of data in a single place, or recent advances in AI application development that require the processing of unstructured data, such as text, images, video and audio.

    Challenges and Convergence of Storage Formats

    Despite the differences, most companies can use both data lakes and data warehouses for their various analytics needs. However, to successfully capitalize on the benefits, it is also important to understand the challenges. In particular, data lakes present difficulties when it comes to parsing data. They were initially designed to capture data across the enterprise in its natural format, without enforcing schema, so that users could garner more insights; a fundamental aspect in creating a data-driven culture across the organization. However, without a predetermined use case, data lakes can quickly become data swamps. End users were historically unable to figure out whether the data was stale, and ownership was not well established. This has been remedied by taking a use case-driven approach and only including data that has defined use cases or ownership.

    Now that many of the issues surrounding data lakes have been resolved, the two storage formats must complement each other to meet the varying needs of the customer. This has contributed to blurred lines between data lakes and data warehouses. Data lakes are now capable of schema enforcement and answering rapid business intelligence queries, which were traditionally qualities of data warehouses. Data warehouses have separated compute and storage and can read directly from big data file systems, enabling users to read semi-structured data. As such, we are rapidly moving toward integrated data environments and the convergence of data lakes and data warehouses.

    Supporting Data-Driven Initiatives

    For initiatives like artificial intelligence and machine learning to succeed, data must be presented as an immutable entity that can be used for experimentation. With data lakes, it is crucial to separate the data into different zones and maintain a refined zone after transformation. Then, in the refined zone, companies can enforce schema and allow the schema to evolve to ensure the data is ready for machine learning and data science needs. Most importantly, they must catalog the data and enforce metadata management, data quality and governance. Data warehouses, on the other hand, excel in providing data sets ready for discovery and consumption. Companies should integrate these data sets with an interactive data catalog so they are discoverable – this is the most important step in making artificial intelligence and machine learning possible.

    If organizations can learn how to best capitalize on data warehouses and data lakes for their intended purposes, they will be well equipped to uncover key data-driven insights to help guide their business strategy. This will help enable use of other advanced technologies and help transforming them into data-driven companies.

  • How Cloud Data Platforms Help Enterprises Drive Business Value

    How Cloud Data Platforms Help Enterprises Drive Business Value

    Cloud data platforms make businesses more scalable and flexible and more cost-efficient

    The need for actionable data is nothing new. Enterprises spend time, money and resources working to get their data into a format that is usable for analytics. But most are still struggling to do just that. More than 90% of enterprises say that making data available for insights is a key challenge in their analytics programs. The journey to modernizing your data infrastructure is challenging, especially if performing data transformation quickly is holding back your business.

    To compete effectively with data, you need “analytics-ready” data. This requires scalable data transformation—the ability to join together siloed data, denormalize, enrich and apply business logic. This process is imperative for getting to the insights that will help you solve business challenges at pace. So how much can data transformation benefit your business? It can rapidly improve your data preparation process, consolidate your data sources, save money, make accurate forecasts and predictions and most importantly, provide you with the insights you need for business growth.

    Cloud-native Solutions and the Cloud Data Warehouse

    A modern data infrastructure works best with cloud technology. Legacy data warehouses hold businesses back from scaling alongside data volumes and bringing data together into a single view. Cloud data warehouses help companies move quickly, and cloud-native solutions use the economics and power of the cloud to handle big and varied data volumes. This is especially helpful with a cloud ELT (extract, load, transform) solution. Traditionally ETL (extract, transform, load) is used to perform data transformations. But the modern ELT approach allows you to move data directly into your cloud data warehouse and then perform the necessary transformation. When enterprises employ a modern data environment and cloud-native tools, it is easier to proactively use data to glean insights—and save on provisioning costs.

    Take, for instance, a large cloud security organization that needed a centralized data repository that was modeled to handle diverse data sources and increasing data storage and reporting needs. This company moved off of its legacy system onto a cloud data platform with a cloud data transformation tool, allowing them to reduce application spends by 84%. Now with a single view of data, they can easily run unified reports to analyze data quickly, opening up greater possibilities to extract insights.

    Preparing and Transforming Data Quickly for Faster Time to Insights

    The act of wrangling data is time-consuming. In a recent survey by IDG, nearly half of data professionals reported that it takes more than a week to prepare data for analytics. ETL, ELT and data preparation workloads are bogging down precious data engineering resources and wasting valuable time that could be better spent innovating with data. Data preparation is a challenge for organizations because the work is code-heavy and manual.  Cloud-native ELT solutions can reduce the burden of data wrangling and provide a much faster, low-code process to prep data.

    Data transformation tools that take advantage of the power of the cloud, have intuitive and automated data preparation processes that eliminate the need for coding.

    This enables businesses to transform data for advanced use cases such as machine learning. For example, a global logistics company that made 100 million deliveries to 37 million unique customers across 72 countries needs to use massive amounts of data, from a variety of sources, to fuel its daily operations. Using data transformation inside their cloud data warehouse helped them to join together disparate data and improve business processes. The company deployed 200 machine learning models in a single year, used to make 500,000 predictions every day. This significantly increased the efficiency of its delivery operations and helped the customer service team reduce inbound call center inquiries by 40%.

    High-quality, Accurate, Timely Data Makes the Difference

    New data sources are proving beneficial—and challenging—for businesses to integrate into their organization. As the pace of data increases, the need for recent and accurate data rises along with it. An analytics platform built in the cloud can provide a much-needed pipeline of high-quality data that can be used in a timely manner, with insights provided quickly.

    As another example, a software company that wants to improve its data-driven culture by relying on fast data integrations using a cloud analytics platform to ingest information. Instead of analyzing data that was days old, the software company used a cloud data transformation tool to load data in real-time from all external sources in the business. By taking advantage of reusable components built in the cloud, the team reduced their data ingestion process from as many as 10 workflows down to just one. This allowed them to reduce their data warehouse and ETL administration, freeing up developer resources to innovate within the business.

    Scaling Resources and Saving Costs With a Cloud Data Platform

    The results of cloud data transformation speak for themselves, as enterprises from different industries see accelerated time to value, scalability and cost savings that validate their investment in modern solutions.

    Data transformation, performed in the cloud, can help you rapidly improve data management and existing processes. Analytics-ready data can help you solve critical business problems and ultimately generate revenue. As external factors contribute to pressure on budgets, resources and road maps, transforming data with an ELT solution will help reduce development time and accelerate innovation with new insights. Cloud ELT and data platforms make businesses more scalable and flexible, cost-efficient and better-prepared to leverage data in advanced use cases involving IoT data, machine learning and artificial intelligence that can predict what comes next and what to do about it.