devopsy.kinsta.cloud

Tag: database

Database Virtualization – Driving the Next Big Transformation

Data is both the most important asset and the biggest challenge for most corporations. It impacts every aspect of the business from financial considerations to AI/ML training. Data is used by development and DevOps teams, engineering teams, quality assurance, business intelligence teams and database administrators. Despite its central role in operations, traditional forms of database delivery remain complex, costly and time-consuming.

In many cases, enterprise data is growing exponentially, increasing management challenges and driving up the cost of data storage. Duplicating data to each new environment as use cases arise is overwhelming DBA teams and incurring a significant storage footprint. This has resulted in data becoming a significant bottleneck for DevOps teams, highlighting the importance of finding a quick and automated solution for the replication process across non-production environments.

What’s Changed?

In the past, development teams would work with the same dataset for upwards of six months. This left DBA teams with plenty of time to meet all database delivery needs. However, as technology became more sophisticated and consumers grew more demanding, the data delivery life cycle was cut down dramatically. Today, with agile development, new features may be added every few days, if not hours. Coupled with QA teams’ requirement for testing data continuously, AI/ML teams requiring millions of data points for accurate training, and data analysts constantly needing fresh data for organizational growth, this means DBA teams are flooded with demands they are incapable of managing.

What Now?

Database virtualization is an automated and easy solution to the dataset bottleneck. By simplifying the creation and distribution of database replicas, DevOps teams will be able to create fully functioning virtual environments via a self-service process–allowing DBA teams to focus on more critical duties relating to production databases.

Speed Up

The first—and most obvious—advantage of this approach is the increased speed at which data is delivered. To keep DevOps teams productive and avoid the use of old, stale data, new environments need to be produced on demand. By integrating the production of new environments into the CI/CD pipeline, enterprises will be able to automate the process and provide DevOps teams with full autonomy. By cutting out the dependency on the DBA, teams will be able to access the data they need, when they need it.

Scale

Agility is fundamental to the success of database management. Each team requires an independent environment, unaffected by other use cases. The creation and supply of virtual databases allow for quick provisioning. Once created, these databases look identical to the original physical database and function completely independently of each other. Any changes made to a vDB remain unique to the environment it’s in.

Save

As the need for data grows, so does the cost of storage. Creating a physical copy for each team and use case results in a significant storage footprint. This means that these datasets come with a hefty cost. However, the creation of virtual databases led to a dramatic decrease in storage needs. Where traditionally, physical datasets can weigh upward of several TB, vDBs only weigh a few hundred megabytes. Further, virtualization solutions also compress the weight of physical copies – such as the golden copy – actively decreasing the weight and cost of storage across all fields.

Secure the Database

Even after the database delivery problem has been solved, enterprises are faced with another major issue – security. With so many different teams accessing data in virtual and low-level testing environments, the risk of ransomware and other cyberattacks increases dramatically. Generally, the investment in securing non-production environments is significantly weaker than in production.

A common approach to address this issue is masking sensitive data so it is rendered useless to outside sources. However, traditional masking requires a data expert to manually sift through all existing data and implement data masking algorithms on any sensitive information found. This time-consuming solution creates a new data bottleneck. By automating the process and applying it to the golden copy, corporations can be confident that all vDBs created are completely secure and privacy compliant.

Looking Ahead

Database virtualization is the future of database replica management. Virtual databases are delivered in a matter of minutes, significantly accelerating the speed of delivery. These environments are easy to use, independent, and can be maintained via a self-service portal. The ongoing sync between the original database and the vDB ensures continuous access to the latest data without the need for timely or expensive processes.

Effective database virtualization will allow well-established enterprises to more easily compete against those agile startups that face significantly fewer barriers to entry.

Image Source: joshua-sortino-LqKhnDzSF-8-unsplash

April 4, 2023
How Database DevOps Fuels Digital Transformation

In today’s real-time, hyperconnected reality, data is a core driver behind most digital strategies. User data is constantly gathered and shared across various technology backends and platforms—as is system performance data to inform engineers. Data is pivotal for everything, from generating machine learning models, refining user experiences and even supporting new data-centric services. But although data is a clear backbone to many enterprises, there is no clear winner regarding which database to use.

Large organizations often find themselves supporting different database styles, from relational databases to NoSQL and distributed SQL. And as organizations continue to adopt multiple databases, safely managing the processing and storage of data becomes more challenging to enact at scale.

I recently met with Jakub Lamik, CEO, Redgate Software, to understand the role DevOps, specifically database DevOps, is playing in advancing data-driven digital transformations. According to Lamik, applying DevOps practices for database management can improve performance, increase productivity and help organizations stay compliant. Below, we’ll consider the state of today’s data-driven digital transformations and highlight ways to improve how DevOps is applied to database management.

State of Data-Driven Digital Transformations

The 2022 StackOverflow Developer Survey found the most popular databases to be MySQL, PostgreSQL, SQLite, MongoDB and Microsoft SQL Server. But most often than not, businesses deploy more than one of these in production. Redgate’s 2021 State of Database DevOps report found that 70% of businesses had more than one DBMS in use, and 48% had teams working on three or more different databases.

Engineers operate in increasingly hybrid and distributed technology environments. This isn’t for no reason—in fact, 90% of organizations said multi-cloud is helping them achieve their business goals. Lamik also notes a diverse array of software technologies in practice today, from multiple clouds to decoupled microservices architectures and database types. But the desire to work with diverse toolsets causes fragmentation, which could hinder software delivery efforts. For example, organizations want to deploy the same framework and similar methodologies across different databases, said Lamik, and have it be database- and cloud-agnostic.

Tips to Improve Database DevOps Operations

To increase the pace of software delivery, infrastructure automation is often needed. And like other roles, database professionals are increasingly trying to automate operations and free themselves from mundane tasks. So, what are some ways to streamline database DevOps? First off, Lamik encourages adopting a common approach across multi-database environments, using standard toolsets that are flexible and portable. “The organizations that standardize deployments across different database environments can increase the frequency and reliability of deployments,” he said.

Catching and resolving errors quickly is also becoming increasingly essential to meet objectives. And although DevOps brings many advantages, the CI/CD pipeline has gaps. “Observability supported by intelligent automation is key to closing those security gaps,” said Lamik.

Thus, he encouraged additional oversight and tracking of changes to ensure stability throughout the database deployment pipeline. Also helpful is encouraging collaboration with security professionals (also known as DevSecOps) and integrating security principles into the DevOps workflow to ensure the process is secure by design.

Another element of database security involves the proper observability and classification of sensitive data. As Lamik describes, organizations must be careful to distinguish between data used for production purposes and data used for development purposes. Ensuring sensitive data is cataloged correctly is imperative to avoid production data from being breached or accidentally publicly exposed.

Final Thoughts

In summary, applying DevOps ideals to database management could be a boon for development efforts. For those who haven’t yet considered it, a DevOps transformation will pay dividends, says Lamik. Furthermore, a common management layer could help wrangle multiple database types and help adopt new emerging database types, such as Snowflake and Postgres. Yet, when acquiring new database toolsets, Lamik admitted it may not be a quick transition. He recommends a land-and-expand strategy rather than a top-down approach (which typically isn’t as effective in empowering individual developers).

Commenting on the industry’s recent move toward platform engineering, Lamik sees it as the next evolution of DevOps. Whereas DevOps brought helpful toolsets to streamline deployment and digitization, platform engineering is taking these practices and productizing them to be reusable at scale across teams and departments, he explained.

Also, in the new cloud-native stack, containers are designed to be ephemeral, causing engineers to craft workaround solutions to enable stateful storage on platforms like Kubernetes. Regardless of deployment, data must have proper backup and recovery controls as well. In this new cloud-native paradigm, the role of database professionals is changing, explained Lamik, and “APIs are becoming the focal point of communication.”

March 17, 2023
Stream Big, Think Bigger: Analyze Streaming Data at Scale

Data streams are all the rage. Once a niche element of data engineering, streaming data is the new normal—more than 80% of Fortune 100 companies have adopted Apache Kafka, the most common streaming platform, and every major cloud provider (AWS, Google Cloud Platform and Microsoft Azure) has launched its own streaming service.

This shift is fueled by the growing demand to deliver events quickly, reliably and at scale to support a wide range of internal and external applications. From always-on, customer-facing applications to mission-critical operations, a growing number of use cases depend on subsecond updates—further upping the ante for real-time data processing and dissemination.

There’s no doubt 2023 will be a big year for streaming technology.

Meanwhile, as streaming moves toward ubiquity, we’re seeing another shift in the way businesses use the data: Analyzing events as they’re created for real-time insights. With the right tools in place, people can instantly compare what’s happening now to what happened in the past, triage problems as they occur and make time-sensitive decisions in the moment.

With the proliferation of data streams, a new set of use cases and requirements for real-time analytics has emerged. To unlock the power of streaming in 2023, data teams will need to look beyond the traditional batch-oriented stack and adopt a streaming-native approach to the analytics architecture.

The Streaming Paradigm Shift

Data has always been at the center of how businesses operate. Until recently, we lived in a very batch-dominant world of business and data. The goal of data infrastructure was to identify data at a fixed point in time and store it for eventual use. The evolution from daily batch operations on mainframes to the internet-driven always-on world replaces fixed, static “data at rest” with fast-moving data in motion, as information flows freely between applications and data systems within and between organizations.

Data at rest still exists, and batch systems continue to serve a variety of reporting purposes. But reality isn’t fixed or static, and to meet the demands for seamless, authentic data experiences, the systems we build must be designed for data in motion.

Hence, the rise of streaming technology—and with it, a new mindset for how we think about data. As streaming increases, streaming data platforms increasingly act as an organization’s central nervous system, connecting the different functions and driving mission-critical operations. Evolving technologies are purpose-built to support data in motion systems: Stream processors and event databases.

Apache Druid fits into the purpose-built category; as a real-time analytics database, it is designed to enable users to query every event as it joins the data stream and do it at a tremendous scale while enabling subsecond queries on a mix of stream and batch data.

Thousands of businesses are already using streaming processors like Kafka and Amazon Kinesis with Druid to build cutting-edge systems that make terabytes to petabytes of streaming data accessible to applications and humans in milliseconds. Some of those businesses—including Reddit, Citrix and Expedia—were highlighted at Current 2022, the annual streaming event organized by Confluent.

The next evolution of data intelligence is here, and it’s based on the ability to react to events as they are happening. But on the growth chart of streaming adoption, we are only at the beginning of an upward curve—a critical inflection point where streaming (and the technology that is built for it) starts to become the bedrock of everyone’s data architecture.

So, when it comes to enabling scalable, subsecond analytics on streaming data, many developers and data innovators are wondering, ‘How?’

We’re Streaming! Now What?

At Current, we spoke with hundreds of Kafka users who were turning their attention to building out the next phase of their data streaming pipelines and embedding them more deeply into their organizations.

While streaming adoption is becoming more widespread, most companies still have just one or two use cases they’re solving with a streaming platform. At Current, many people spoke about how Kafka effectively set their data in motion. But when it came time to analyze or put those streams to work in a user-facing application, their “data in motion” turned into “data in waiting” because their analytics system was designed for batch data, not streaming.

To solve this problem, a new type of database is needed—and that’s why Druid was created. Turning billions or trillions of events into streams that can then be immediately queried by thousands of users simultaneously is where stream processors like Kafka, in combination with Druid, have unlocked a new set of capabilities that are enabling new developer-built analytics applications.

For example, Reddit generates tens of gigabytes of event data per hour from advertisements on its platform. To let advertisers understand their impact and decide how to target their spending, Reddit needs to enable interactive queries across the last six months of data—hundreds of billions of raw events to slice and dice. But they also needed to empower advertisers to see user groups and sizes in real-time, adjusting based on interests (sports, art, movies, technology…) and locations to find how many Reddit users fit their target. To achieve this, they built a Druid-powered application that ingests data from Kafka and enables Reddit’s advertising partners to make real-time decisions and get the best possible ROI on their campaigns.

Stream-to-Batch Vs. Stream-to-Stream

Reddit chose Druid as the database layer of their application in large part because of its close integration with Kafka, as Druid was designed to ingest and analyze streaming data. This sets Druid apart from virtually every other analytics database, the rest of which were built for batch ingestion.

Batch ingestion works exactly the way it sounds—it takes chunks of stream data, dumps them into a file, processes the file, then loads it into the database. Using a batch-based system to analyze streams introduces a bottleneck to your streaming pipeline. By contrast, Druid provides a connector-free integration with the top streaming platforms and handles the latency, consistency and scale requirements for high-performance stream analytics cost-effectively.

In addition to out-of-the-box integrations, Druid has built-in indexing services that provide event-by-event ingestion. This means streaming data is ingested into memory and made immediately available for use. No waiting for events to be processed through batch processes before they can be queried. This capability, coupled with exactly-once semantics, guarantees data is always fresh and consistent. If there’s a failure during stream ingestion, Druid will automatically continue to ingest every event exactly once (including events that enter the stream during the outage) to prevent any duplicates or data loss.

However, perhaps the most powerful reason to use Druid for streaming data analytics is its near-infinite scalability. For even the largest, most complex ingestion jobs, Druid can easily scale into the tens of millions of events per second. With variable ingestion patterns, Druid avoids resource lag (as well as overprovisioning) by enabling dynamic scaling. It combines the query and ingestion performance of shared-nothing architecture with a cloud data warehouse’s flexibility and non-stop reliability. This means you can add computing power and scale out without downtime or rebalancing, which is handled automatically.

In essence, while using a cloud data warehouse to serve both batch-oriented and real-time use cases might sound efficient, doing so defeats the purpose of a streaming pipeline.

“The weakest link in the chain sort of dominates everything,” Databricks CEO Ali Ghodsi told the audience at Current. “If you have one step of your pipeline that is batch-processed and really slow, then it doesn’t matter how fast you are in the rest of the pipeline.”

That’s why when Confluent’s data engineering team needed to build a real-time observability platform, they chose to use Kafka and Druid together. Today, the platform ingests over five million events per second from Kafka and handles hundreds of concurrent queries from a large team of analysts.

“We do a lot of things today that we didn’t imagine doing three or four years ago,” Matt Armstrong, who leads the observability and data platform team at Confluent, said in a Druid Summit 2022 interview. “From a [product] perspective, you could think of Druid as being able to power a great product [Confluent Cloud] with a great support experience—but perhaps more importantly—an unconstrained future.”

It’s Time to Think Big

We’re already moving to this entirely new way of thinking about moving, analyzing, and sharing data. Streaming technology has opened the door for a plethora of new use cases and products to come to fruition. We believe organizations that view streaming data as a force multiplier and implement a streaming-native approach to analytics will gain a competitive advantage in 2023 and beyond.

Druid is purpose-built for these new streaming use cases, but it’s also built for a new type of application—one that draws characteristics from both the analytics and transactional database worlds. Industry-leading teams have built mission-critical applications that enable operational visibility, customer-facing analytics, drill-down exploration and real-time decision-making using streaming data and Druid. This trend is now starting to take hold and the next wave of analytics applications is already being built.

January 27, 2023
The Database of the Future: Seven Key Principles

The database of the future will be more than just a database. Instead, it will be a platform that focuses on the user experience and abstracts the technology. Looking at the evolution of application development, an enormous amount of effort has made it easier to build and deploy applications. Similarly, databases need to evolve for ease of use with a delightful developer experience and an exemplary user interface that surfaces features and functionalities that engineers will actually use.

Here are seven principal features needed to chart the development of the future database.

1. Serverless by Default

Serverless is unquestionably the most significant paradigm shift in software engineering in the last five years and the one that’s been realized the least. There aren’t that many people at this point that are all in on serverless. It’s the shift away from buying a server to a service that does what you need and scales with your growth.

Serverless by default is the only way to ensure the database of the future will be easy to operate while being consistently reliable. Serverless is evolving from running functions in the cloud to becoming the standard for fast-paced development teams that don’t want to worry about vCPUs or node pools but focus on outcomes.

The companies that are doing it the best still deploy servers, but they are leveraging products that scale independently, regardless of how many servers they have. Their component vendors can increase the number of resources allotted to them, and they only pay for what they use, and the architecture of the solution scales in that way. They’re not buying a single server and pushing it as hard as they can. They’re telling one platform, ‘here’s my code,’ they’re telling another, ‘here’s my data,’ and then that scales up based on usage much, much farther than buying a single server.

When people think serverless, they think JavaScript and Lambda, but there’s serverless PHP out there, too. It’s not just the JavaScript community. Other software paradigms are going for this. We will see more and more people pulling this out when they need to do some quick code and they want to ship to production as quickly as possible and scale as quickly as possible.

2. Easy to Operate

The most significant advancements in DevOps have been freeing developers from complex operations that overwhelm their day-to-day operations. Engineering teams today are organized around product and business goals, not maintenance and servicing. But databases have yet to catch up. Product teams still rely on experts to perform overly complicated database operations tasks.

The database of the future must fit into everyone’s DevOps workflows. It must be usable for any developer and leveraged by every team within a technical organization. The database needs to be operable by everyone and no longer require database admins to manage backups and capacity planning or worry about disaster recovery.

Developer-centric features like database branching, database developer environments, autoscaling and usage-based billing will be commonplace.

3. Reliable

Reliable simply means it’s harder to hit the scaling limits. Not only does it mean that the database is there, but that you get the right information when something is going wrong, and you can collaboratively help scale.

Nothing scales infinitely. There are not enough computers in the world to scale something to infinity. You’ll always be able to query a database and pull too much information out of it too fast. No matter how many cars you get or how much horsepower you have, you can’t go a million miles an hour. A good database must let you know when you’re hitting those limitations and help you move past them, whether at a small scale, medium scale or large scale.

4. Interoperable

Future databases will be accessible to every imaginable client–from smart refrigerators to fleets of autonomous delivery vehicles. But this client-agnostic approach requires planning for many more transactions and connections than most current databases could ever manage simultaneously.

In the future, databases can handle unlimited connections through smart connection pooling, query replication and intelligent query routing. Databases will also need to serve a wide set of connections and clients. Although we are already seeing many databases offering web APIs and new connection types, as these get tested at scale, we will begin to see them adopted and used to power the next generation of applications.

5. Globally Distributed, Locally Available

Today, most businesses and applications don’t need a global database, which often introduces unnecessary complexity and operating costs. However, there are many reasons the future database will need to be globally distributed, as they must be located where you need them the most.

The key to this is knowing what your requirements are. Many people look at multi-region and think it’s faster reads, failover, scaling and breaking the speed of light. Ultimately, it’s all a series of tradeoffs. For example, when you read from a local replica, you trade off consistency because you have asynchronous replication. But this is really leveraging the global internet to deliver a better outcome where it’s necessary.

Being compliance-ready with regionally stored data enables you to scale rapidly to a new region and provides a great user experience by quickly serving locally available data. High-transaction, low-latency applications such as mobile games already need this capability.

But there’s one caveat: Global data distribution cannot break principle number two—being easy to operate. Tools will need to be elegant and have straightforward interfaces as their main goal is to make operations easier, not harder.

6. Intelligent

The future database is intended to be used by people and will be able to tell them how it needs to be used and how to make it better. All the tools you buy today can only give you the data you need to make some educated guesses. The future database will replace that with concrete recommendations, then self-healing actions.

This starts with basic data collection and observability and moves to, “let’s record every query that someone makes and use some decision engines and some AI and recommend how they can shard automatically.” It will be able to say, “I’ve recorded your database use for the last 30 days; this is how you should shard for optimal performance. Click the button, and we’ll do it.”

The future database will say, “You’re scaling too far for the single machine at a per-user level. You must break it down into two or break it down into four, and we figured out how to do it for you, so you don’t have to figure that out. We’ve sharded your data on your behalf.”

7. Linearly Horizontally Scalable

Future databases will also need to be designed for future scale. Only a handful of companies on the planet have hit a scale that tests the limits of current technologies, but as information grows, more companies will get there. The next databases will do that by automatically scaling both vertically and horizontally within seconds. Many existing databases require a maintenance window, replication and a failover before they can be scaled up or out. But downtime is unacceptable, and scale should not come at the cost of performance. And it doesn’t have to. The database of the future will scale intelligently and without a blip on the status page.

In short, the consumer should be able to get what they need when they need it, with no risks. It should be easy to say, “I’ve doubled the number of servers, I’ve doubled the amount of throughput. I’ve quadrupled the number of servers and the amount of throughput.” This is what sharding provides. Good sharding happens automatically at a super high scale, and you’ll only pay for what you use as you go along. The future database will do all of this without users having to turn a knob.

In conclusion

It will take the entire database industry to agree on these principal features, adopt a user-focused approach and build future database platforms designed to make engineers more successful by delivering superior products and services to their customers. I envision a day where every company no longer worries about its infrastructure and can simply focus on its business. I want development teams to take back control over their development workflow. Using a database should be as easy as writing code.

January 25, 2023
FAA Ground Stop due to Technical Debt? | Don’t Do DIY Crypto!

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: The FAA’s NOTAM database gets corrupted, and Threema shows why DIY cryptography is bad. (more…)

January 12, 2023
Key Findings From the 2022 State of GraphQL Report

GraphQL, the open source query language, has been steadily growing in popularity over the past few years. The language is excellent for frontend development, as it creates a usable interface to fetch precisely the data required for the specific client at hand. You can also use it to aggregate multiple backend services and data sources.

Studies show that the ecosystem around GraphQL is maturing—it has a very global user base and a growing number of tools exist to aid in its adoption. The 2022 State of GraphQL Report found that 61.8% of developers are happy with the overall state of the ecosystem. Yet many native GraphQL features go unused by developers, which suggests there is a wide range of experience levels, from novice to advanced users, working with the technology. This indicates there is plenty of room for growth and knowledge sharing.

Below, I’ll review the key takeaways from the 2022 State of GraphQL Report, considering overall usage data and focusing on the popularity of specific features. We’ll also highlight the state of standard tools used in conjunction with GraphQL to get the job done.

Analyzing Overall Usage Statistics

First off, there is a wide variance in the types of GraphQL APIs in production today, ranging from private to public endpoints. The report found 47.9% of developers use GraphQL for exposing an API intended for a personal website or app, 40.4% use it for private unexposed APIs for internal usage and 19.5% use GraphQL for publicly available APIs intended for third-party developer consumption. This demonstrates its flexibility to cater to multiple scenarios. And while the majority of clients that connect to GraphQL APIs include browsers (62.3%), it’s commonly consumed by other client types such as native mobile apps, other servers and desktop apps.

Previously, we discussed using GraphQL as a composition layer to combine underlying APIs and services. And the report backs up this idea that developers commonly use it to aggregate various data sources: 55.4% of GraphQL APIs hit databases, while 35% consume REST APIs and 14.6% eat other GraphQL endpoints.

In terms of benefits, one strong point is the ability to enforce and validate types for every object in the API. Users find other top benefits to be avoiding overfetching and aggregating requests. Yet, some hurdles still present themselves. These top drawbacks include error handling, performance issues, client-side caching and security woes.

Regarding user base statistics, most users have been working with GraphQL for less than five years. If we look at the type of sector utilizing GraphQL, we see it primarily leveraged in the programming and technology sector. This is followed by e-commerce and retail (16.4%), finance (8.9%) and news, media and blogging (7.2%).

Many Advanced Features Go Unused

Many of the more advanced GraphQL features still go unnoticed by the majority of users. For example, live queries is a feature that helps obtain live data from a GraphQL server, yet only 25% who know about it have used it. Over half of respondents had never heard of @skip, a directive that skips a field based on a value passed to it. Similarly, other directives like @specifiedBy, @defer and @stream also share little-to-no awareness or use.

Analyzing feature usage becomes a bit more serious if we consider the unadopted security and performance features. For example, only 43.2% of respondents have disabled query introspection. Disabling query introspection is a suggested security best practice to limit a hacker’s ability to perform surveillance. But as we’ve previously covered, security by obscurity isn’t enough to protect GraphQL, and an additional security emphasis is often needed.

As most GraphQL APIs are intended for personal or private scenarios, you would think that more security features would be leveraged to limit the threat of introspection and denial-of-service attacks. Yet, the report discovered low usage and awareness of security and performance features like query timeouts, query rate limiting and query cost analysis, all of which could be turned on to curb malicious behaviors.

Tools Used In Conjunction With GraphQL

Developers often seek to combine, aggregate or merge GraphQL schemas from different APIs. To do so, Apollo Federation is the most common solution, used by 22.5% of respondents. This is followed by Schema Stitching (16.9%), GraphQL Modules (4.4%), and GraphQL Mesh (2.9%).

Looking at other technologies used in connection, Next.js is the most popular framework and PostgreSQL is the most common database used. TypeScript and JavaScript comprise the most common languages used to write a GraphQL backend. Developers also use a balanced mix of IDEs to query and test endpoints, including GraphiQL, GraphQL Playground, Postman, Apollo Studio, and Insomnia.

The report also found relatively high usage and retention for tools like the GraphQL Code Generator schema builder, Apollo Client and Apollo Server. The data shows a crowded solution market with Apollo as an industry-leading outlier—many other clients, servers, API generators and schema builders have medium-to-high retention rates yet experience far less use.

Final Thoughts

The 2022 State of GraphQL Report sheds some interesting light on the state of GraphQL, validating a lot of the interest and trends we’ve covered lately on the blog. Of course, there is more to be said about many areas, such as smoothing the transition to GraphQL.

One key finding is the low number of security features being leveraged by developers. A greater emphasis on API security is required to avoid incidents and retain a zero-trust approach. This emphasis is part and parcel of responding to ongoing software supply chain threats and will require more awareness and thought leadership around GraphQL security best practices.

The 2022 State of GraphQL survey ran from June 15, 2022, to July 15, 2022, and collected 3,094 responses. The above analysis only scratched the surface and there are many other insights. For more findings, you can read the full results here.

December 1, 2022
Zilliz Unfurls Managed Vector Database Service

Zilliz this week made generally available a managed cloud service that can now be spun up in less than five minutes. The service is based on the company’s open source Milvus vector database.

Frank Liu, director of operations for Zilliz, said the managed Zilliz Cloud service running on the Amazon Web Services (AWS) cloud will make vector databases much more accessible to a wider range of application developers.

Vector databases are gaining traction because they provide artificial intelligence (AI) application developers with a more efficient method for managing the unstructured data need to train AI models. Currently, many of those teams are relying on labels and tags to classify and manage data. The Milvus vector database employs deep learning algorithms and its own AI model to create and store, index and manage vectors to more efficiently classify data, said Liu.

Based on a cloud-native architecture that, for example, employs Kubernetes to scale consumption of cloud resources more efficiently, Zilliz Cloud can also scale to process tens of billions of high-dimensional vector data for data-intensive applications, he added. The company claims Zilliz Cloud delivers up to 10 times greater throughput and five times lower latency for read and write operations for data-intensive workloads.

Other attributes of the managed Zilliz Cloud include support for multi-language software development kits (SDKs), a web-based graphical user interface to search and explore data, SOC 2 compliance, network isolation using IP whitelists and full data encryption in transit (TLS).

In the future, Zilliz also plans to add support for Microsoft Azure Cloud and Google Cloud Platform in addition to graphical processor units (GPUs) to optimize the performance of its deep learning algorithms.

Potential use cases for a vector database span everything from image retrieval and video analysis to fraud detection to drug discovery. Organizations that already use Milvus, which is being advanced under the auspices of the LF AI & Data Foundation, include eBay, Shopee and SmartNews.

In general, the rise of data-intensive workloads is driving more organizations to embrace DataOps best practices to optimize the management of massive amounts of data. Organizations that build AI models are typically at the forefront of those efforts given the massive amounts of data typically required to train AI models. The challenge organizations have historically faced is not many have the internal expertise required to manage a vector database. The Zilliz Cloud eliminates the requirement by relying on the expertise of a team of database administrators trained by Zilliz to operate a managed service.

It’s not clear at what rate organizations are embracing vector databases to build AI models. As the next era of applications are constructed, it’s only a matter of time before the machine learning operations (MLOps) practices used to build AI models converge with DevOps practices used to build the applications in which those models will be embedded. The challenge is the data science teams that build AI models tend to also have a distinct DataOps culture all their own; DevOps teams will need to find a way to embrace and extend that.

In the meantime, the number of types of databases being used to build applications will only continue to expand so it will be up to each DevOps team to know when to manage each one themselves versus relying on a managed service.

November 29, 2022
Momento Launches Elastic Cloud Caching Service

Momento this week emerged from stealth to launch an elastic Serverless Cache platform that promises to improve the performance of applications running on databases deployed in the cloud while simultaneously reducing costs.

Fresh from raising $15 million in seed funding, Momento CEO and co-founder Khawaja Shams said Serverless Cache gives IT teams access to a serverless computing framework for accessing cache memory that can be used to improve the interactivity of applications.

Cache is, of course, already widely employed to improve the performance of applications running on databases in the cloud. However, organizations today are required to provision excess cache memory to address performance workload peaks. That approach, over time, conspires to increase the total cost of cloud computing, he noted.

A serverless cache makes it possible to more easily share an elastic pool of memory that can be invoked in seconds via an application programming interface (API) that is itself invoked by adding five lines of code to an application. The Momento cache service can then handle millions of transactions per second that can be spread across multiple applications, said Shams.

The Momento Serverless Cache service also automatically manages hot-keys, maintains high cache hit rates and keeps tail latencies low.

Organizations that are already using Momento on either the Amazon Web Services (AWS) or Google Cloud Platform (GCP) include CBS, NTT Docomo and Wyze Labs, Shams said.

In general, there’s a lot more scrutiny of cloud costs in the wake of the economic downturn, noted Shams. More IT teams are starting to take note of the fact that various cloud-native platforms enable them to more efficiently consume cloud resources, he added.

The simple truth is that far too many organizations are managing cloud resources in much the same way they managed on-premises IT environments. Capacity is being dedicated to virtual machines to ensure application availability with even less regard for cost because the perception is that the cloud makes it easy to throw hardware resources at any issue. The challenge is the bill for those cloud resources tends to add up at the end of every month. Cloud database services, in fact, are among the most expensive to consume, noted Shams.

It’s not clear how much organizations will optimize the consumption of cloud resources in the weeks and months ahead, but it’s clear more supervision is going to be required. Organizations have historically let developers provision resources in the name of agility with little to no regard for cost. Now that finance teams are looking to, at the very least, make sure cloud resources are optimally consumed, there will undoubtedly be more reviews of cloud spending patterns.

In the meantime, many organizations may also decide to accelerate the rate at which they are embracing cloud-native technologies to optimize that spending. One of the more compelling attributes of Kubernetes clusters and serverless computing is they make it easier to scale compute resources up and down as required to eliminate the need to overprovision cloud infrastructure. The challenge, of course, is that the cost of acquiring and mastering those technologies remains significant.

November 3, 2022
Database Observability: How to Circumvent your Weakest Link

Today, IT pros field an increasing number of assets they need to monitor to secure systems and understand what’s happening in their database and applications. This is further complicated as organizations adopt hybrid IT strategies and IT pros are tasked with managing tech stacks that span across on-premises and public clouds.

To cut through complexity and create end-to-end visibility into their IT systems, organizations must enact an observability approach.

Monitoring Vs. Observability

When it comes to observability, many think it means the same as monitoring. Whereas monitoring is a verb; observability is a measure of how well the internal states and conditions of a system can be inferred from its external outputs.

When monitoring systems are set up, dashboards are built around certain assumptions that an IT pro may expect will happen. When monitoring databases, teams are building multi-level component IP systems. The algorithms must process that data, and the user interface must interact with it.

However, to make that system highly observable, performance information and other kinds of metrics across all kinds of different endpoints must be incorporated. Especially since IT pros must deal with microservices, mesh, IoT or containers orchestrated through Kubernetes.

How to Monitor Effectively

In the past, IT pros would typically monitor a SQL Server or even other database platforms like PostgreSQL or MySQL. In this new era of observability, on the other hand, IT pros need to do something different with the front-end code with regard to the different kinds of algorithms processed against that data set. Typically, developer teams would write the code, but there wasn’t anything written in that code to send out debug signals.

Today that has changed. To make sure our systems are fully observable (because we can’t anticipate all possible problems!), we must log different steps to keep track of the many moving pieces. Organizations must collect as much of everything as possible. The more data, the more information organizations have about what’s happening in their application, from the database through the virtualization or containerization layer to the virtualization host or the orchestration to the bare metal. If organizations can get visibility into those elements, we can enable our systems to be fully observable and more easily diagnose any problem or anomaly that happens.

An Observability Approach

When using the observability approach, high-level dashboards are handy. They can help IT pros roll up everything that’s happening and get to a root cause analysis if there is a problem. However, sometimes there’s not a specific problem; there’s just a general performance issue. With observability platforms, teams can quickly discover, map and understand dependencies in ways that make it easier to achieve and maintain service level objectives (SLOs).

As IT teams increasingly distinguish between merely monitoring IT environments and achieving observability, they can proactively surface anomalies using machine learning algorithms and other forms of data science. And the depth and insight that can be applied to managing IT is increasing dramatically. Observability is quickly becoming table stakes for any modern IT organization.

September 12, 2022
Time-Series Database Basics
There’s been a lot of buzz about time-series data for the past few years. No matter what you’re monitoring—financial data, server performance, social media activity or something else entirely—time-series data can give you valuable insights into trends and patterns.

But what exactly is a time-series database, and why do you need one? This article will look closely at time-series databases and how they can help you make sense of your data.

What is a Time-Series Database?

A time-series database (TSDB) is a database management system optimized for storing and querying data that changes over time. Time-series data is often used in monitoring applications where it’s important to quickly retrieve information about a system’s current state and trends and patterns over time.

In most definitions, time-series data, often referred to as time-stamped data, is a sequence of values indexed in time order. Time stamping refers to data collected at various times, where each value is time stamped. These data points are usually gathered from the same source and are used to measure progress over time.

Important Notes:

You can use relational or NoSQL databases to crunch time-series data, but purpose-built time-series databases are tailored to exploit the unique features of time-series data. This implies that time-series databases ingest at a faster rate, query more quickly and compress data more efficiently. Furthermore, time-series databases include special analytical capabilities and management features that are not found in most relational or NoSQL databases.

Some Advantages

This list is not exhaustive, but here are some of the advantages that you might get from using a time-series database:

1.) Time-series databases can handle high-velocity data very well.

2.) Time-series databases are purpose-built for storing time-series data, making them more efficient in storage and querying.

3.) Time-series databases often have built-in analytics and management features designed specifically for time-series data.

4.) Many time-series databases are open source, which means they’re free to use.

Some Disadvantages

There are also some disadvantages that you should be aware of before using a time-series database:

1.) Time-series databases are often more complex to set up and manage than relational or NoSQL databases.

2.) There are relatively few time-series databases to choose from, so you might not have as much flexibility in choosing a platform that meets your specific needs.

3.) Time-series data can be very large, so you’ll need enough storage capacity to accommodate your data.

What Are the Characteristics of Time-Series Data?

There are several characteristics of time-series data that make it unique and require special handling. From a database standpoint, the most essential ones are as follows:
1. Timestamp: Every data point has its timestamp. The timestamp is crucial for calculating or analyzing the information.
2. Structure: Metrics from devices or monitoring are almost always structured, unlike those produced by Internet applications. They have predetermined data types and lengths, and the structure will not alter until the device’s firmware is updated.
3. Stream-like: Data sources, such as audio or video programs generate data at a set or constant rate. These data streams are completely unrelated to one another.
4. Stable flow: Time-series data traffic is constant over time, and it may be calculated and predicted if enough data sources are used within a certain sampling period.
5. Immutability: The source of this data is a time-series data store. Each data point is created only once and never updated or corrected. Time-series data, like log data, is typically append-only.
How Is Time-Series Data Used?

Many of you may be wondering why time-series data is so important. The answer is that it allows us to track changes over time, which can be incredibly valuable for monitoring and troubleshooting purposes.

For example, let’s say you’re a system administrator tasked with keeping an eye on server performance. By monitoring various metrics—such as CPU usage, memory usage and network traffic—you can quickly identify when a server is starting to experience issues.

Time-series data can also be used for predictive purposes. For example, if you notice that CPU usage spikes at certain times of day, you can use that information to plan for future capacity needs.

Of course, time-series data isn’t just limited to server performance. It can be used for any application where it’s important to track changes over time. This includes everything from weather and financial data to social media and website analytics.

Let’s Get Specific

Here’s what you should know about time-series data: It’s typically used to seek insights into operations, create alerts based on real-time analysis and forecast future trends.

The following characteristics are found in time-series data applications:
1. High write-read ratio: Twitter and LinkedIn are internet applications with single articles that millions of people read, but raw time-series data is primarily scanned and evaluated by apps and algorithms.
2. Retention policy: In general, time-series data is not kept permanently. Organizations have a retention policy that specifies when and how their data is destroyed.
3. Real-time analytics and computing: Time-series data must be calculated in real-time to identify out-of-the-ordinary activity and sound alarms based on the acquired information or aggregate findings.
4. Query scope: Time-series data is generally requested over a period or a set of data sources, and filters are used to prevent all historical information from being requested. Furthermore, all or a portion of the data sources with a filter condition are always aggregated.
5. Trends: Single data points are typically not significant in time-series data. The emphasis is on how data evolves, such as fluctuations in the previous hour or day.
Popular time-series solutions like TDengine or InfluxDB, for example, create more efficient processing of time-series data and greater performance than general databases by taking advantage of these qualities.

Do Time-Series Databases Need Specialized Databases?

We live in a modern world where data is constantly being generated at an unprecedented rate. To keep up with the demand, businesses need to be able to store and process large amounts of data quickly and efficiently. This is where specialized time-series databases come in.

Everything is online, including meters, automobiles, lifts, assembly lines and even bicycles. And these items are sending out a never-ending stream of numbers and events that IoT and the cloud have unleashed an explosion in time-series data generation.

With this in mind, it’s not surprising that several specialized time-series databases have been created to deal with this type of data influx.

Time-series data sets are enormous and pose a significant problem for general database management systems, such as relational and NoSQL databases. Non-specialized databases have trouble with the following elements of time-series data:
1. Data ingestion rate: In many time-series data scenarios, millions of data points are generated every second and must be ingested in real-time. Relational databases are not built to handle this quantity of data, and while NoSQL databases may be scaled to do so, the amount of resources required soon becomes prohibitive.
2. Query latency: It is often the case that a time-series system must scan a massive number of data points to get an aggregate result, which might lead to sluggish performance. For example, it would take days for a basic database to compute the average response time of all Amazon.com clicks, by which time the overall conclusion may be incorrect.
3. Storage cost: Data generated by internet-connected devices and apps is nonstop 24/7, with a single day producing as much as a terabyte of data. Because relational and NoSQL databases cannot compress this data effectively, storage costs can quickly skyrocket.
These problems are generally linked to efficiency in processing big data sets, although there are some places where general databases frequently do not meet even the fundamental demands of time-series applications:
1. Data life cycle management: Time-series data is generally removed in bulk, not one data point at a time, as it ages out.
2. Roll-up: In most cases, time-series data is collected and rolled up over a set period before being stored in the new table. Raw data and rolled-up data can have distinct life cycles and retention policies.
3. Special analytic functions: Time-series applications need more specialized features than general databases, such as time-weighted average, moving average, cumulative sum, rate of change, elapsed time for a specific state and the delta between two consecutive data points.
4. Interpolation: The database management system must be able to interpolate data based on the adjacent data points and rules to regularize data sets when applications or algorithms require.
5. Continuous query: Therefore, if you hear a user or customer say they want to know when their data is updated, it’s reasonable to assume they want this functionality.
6. Session and state windows: Aggregations and analytical procedures may be performed on a session or state window, not just time—for example, consider one that only calculates average power consumption when a machine is switched on.
With general databases, developers must write custom code to implement features specific to their data set. Different data workloads require different database solutions; one size does not fit all. For time-series data, no matter the size of your data set, a purpose-built time-series database is the best tool for the job.

Why Are Time-Series Databases Becoming Popular?

Two years ago, time-series databases were the most popular database type in the business, owing to its expanding number of use cases. It is especially useful if you’re executing sophisticated transactions like advertising, e-commerce, supply chain management and so on.

However, because of the rapid expansion of the IoT, they are becoming even more popular. As more devices become internet-connected and send data—time-series, of course—to the cloud, many industries are interested in purpose-built time-series databases.

Time-series data is proving to be an important tool, not just for decision-making and optimization but also for industrial applications. Finally, IT infrastructure has been growing rapidly, with everything from servers, containers, network devices, apps, and microservices being monitored to create massive quantities of time-series data.

Older time-series databases, on the other hand, are often closed systems employing antiquated structures that do not scale to handle the increasing amount of data. A million time-series data points used to seem like a lot, but now millions and even billions of data points are commonplace.

Finally, integrating old (previous) time-series solutions with popular data analysis tools like artificial intelligence and machine learning platforms is difficult, if not impossible. These legacy systems cannot be transferred to the cloud without a lot of work and their licensing terms are no longer adequate for current apps.

Final Thoughts

The expanding market and the constraints of previous time-series databases are making room for a new generation. These new time-series databases are built from the ground up to take advantage of the cloud, big data and AI/ML technologies.

If your business or application deals with time-series data, you need a purpose-built time-series database. The advantages of using such a database are too great to ignore, and the disadvantages of using an older, less capable database are becoming more and more apparent.
August 18, 2022