devopsy.kinsta.cloud

Tag: database

  • Observability: The Central Watchtower Your All-Seeing IT Needs

    Observability: The Central Watchtower Your All-Seeing IT Needs

    If there’s something you need to keep secure, it’s natural to want to keep it sealed away under lock and key. However, even keeping something in a vault, far from prying eyes and ill intent, may not be enough to offer peace of mind. After all, how can you have the peace of mind of knowing it’s safe if you don’t regularly check to ensure it hasn’t bolted, been burgled or broken down?

    Now, multiply the volume and value of things being monitored tenfold — and you get a sense of the daily inefficiencies and anxieties IT professionals battle.

    Instead of jewels, these teams safeguard, manage and maintain something far more valuable: The data that underpins everything from business transactions to system performance to threat detection to service delivery and beyond. To make matters more complicated, today’s digital ecosystems are more complicated, distributed and disparate than ever before.

    Databases are critical to managing a successful IT department — and it’s time to discard the widespread industry’s perception of them as an unknowable black box. To be sure, databases represent the most difficult component of the IT ecosystems to observe, tune, manage and scale. But the database isn’t a Magic 8 ball in terms of its internal processes; We no longer have to shake the mystery toy and accept one of the six answers it’s designed to spit out.

    The fact of the matter is that database specialists and IT teams need a clear view of database performance telemetry if they want any hope of maintaining the health, stability and scalability of their services.

    Observability Tools

    This is where observability tools can play a transformative role. Taking its cues from the all-seeing panopticon watchtower, observability eliminates the dark corners of the database dungeon by providing a comprehensive view across the full cloud-native, on-premises and hybrid technology stacks.

    The word panopticon derives from the Greek word panoptes, or “all-seeing.” The panopticon model features a central, lighted guard tower at the center of a circular room — allowing 360-degree lines of sight that allow the guards to observe every surrounding cell from a single, centralized vantage point.

    English philosopher Jeremy Bentham had no way of foreseeing the challenges of today’s database professionals when he first conceived of the panopticon in the 1700s. The concept would lay the blueprint for a prison system designed around one core concept: Enabling the minimum number of guards to effectively monitor the maximum number of inmates. Bentham’s system also works as a strategy for ensuring the health and performance of your organization’s critical databases.

    The panopticon was designed so that every cell could be monitored from one central point — ensuring not only better and more straightforward security but preserving resources in saved time, manpower and effort expended. Think of your ITOps, database, DataOps and DevOps teams as the wardens of your IT performance. Traditional monitoring can be likened to patrolling a traditional prison: Rows of cells facing out into a hallway that is patrolled on a schedule or used to spot and address misbehavior as it’s noticed.

    How can they be sure they’re focusing their efforts in the right place? Or trust that Cell Block A won’t stage a breakout while they’re breaking up an argument in Cell Block B?

    In short: They can’t. Odds are, the team is either running from outage to outage in an everlasting firefighting mode or relying on random spot checks for something as vital as the health of our IT systems.

    Observability, as opposed to monitoring, puts teams into that central panopticon watchtower, allowing them to not just see everything, but to see it within the context of the bigger picture. This comprehensive visibility and always-on observation enable IT teams to identify critical issues as they occur, even those caused by complex dependencies between the database, operating system, storage subsystem and network.

    Database Performance Issues

    Furthermore, database performance issues are a starting gun for costly bottlenecks or serious outages that can handcuff your company’s ability to compete or grow. Without complete and precise database monitoring and observability, IT operations struggle to accurately determine an application’s root cause of performance issues. This increases the risk of downtime, data loss and poor customer experiences.

    Discovering the root cause of performance issues is quite literally a system-critical undertaking, whether you’re simply running your enterprise, scaling your operations or deploying new code. Observability also offers teams the rare opportunity to get ahead of performance issues since you can see performance anomalies in advance of a service disruption rather than relying on real-time remediation.

    With traditional methods, DevOps and IT teams must manually analyze the data presented to them, correlate it to the problem, and locate the error before they can finally begin addressing it. In contrast, observability collects data to provide information on what’s not performing as expected and why. This allows teams to take a proactive approach to solving downtime and bottlenecks and proactively preventing them.

    In the software world, that translates directly to more time spent where it matters: Developing new business value within your applications and infrastructures, fueling innovation and exceeding customer expectations.

    Given how complicated the modern database has — and continues to — become, arming your enterprise with the right observability solutions can provide a boost to efficiency and performance and pardon your teams from a sentence to time-consuming, mundane and error-prone work.

  • Couchbase Adds Columnar Database to DBaaS Environment

    Couchbase Adds Columnar Database to DBaaS Environment

    At the AWS re:Invent 2023 conference today, Couchbase, Inc. announced it added a columnar database based on the JSON file format to its database-as-a-service (DBaaS) portfolio to enable organizations to build real-time analytics applications.

    Scott Anderson, senior vice president for product management and business operations at Couchbase, said the Capella columnar database is designed to be integrated with the company’s JSON-based document database via a database change protocol (DCP) that Couchbase has embedded within its DBaaS environment.

    Deployed on cloud services from Amazon Web Services (AWS), the overall goal is to reduce the level of friction that IT teams would otherwise encounter when integrating a document database with a columnar database optimized to process analytics using columns rather than the rows typically associated with a relational database.

    That approach gives IT teams the ability to use a schemaless database architecture that provides an alternative to employing disparate databases that would need to be deployed and then integrated using extract, transform and load (ETL) processes to move data from one database to the other, noted Anderson.

    Capella columnar also uses the same SQL++ language as the company’s document database to streamline queries in addition to supporting Capella iQ, a copilot based on generative artificial intelligence (AI) that enables queries to be crafted in natural language.

    In addition, Capella columnar provides connectors to move data from Amazon DynamoDB, Amazon DocumentDB, Amazon Relational Database Service (Amazon RDS) and other data sources deployed on the AWS cloud.

    In the longer term, Couchbase is also planning to add vector capabilities to its DBaaS environment to make it simpler to securely extend generative AI models, said Anderson.

    In general, document databases have been widely adopted because they provide developers with an alternative to legacy database platforms that they can download and deploy themselves. The Capella DBaaS platform further simplifies that process via a managed service through which Couchbase assumes operational responsibility for managing databases. The columnar database extends that capability using a columnar database that can drive analytics capabilities to deliver a more personalized application experience, noted Anderson.

    It’s not clear to what degree organizations will ultimately wind up relying on as-a-service platforms, but since the start of the COVID-19 pandemic, the shift to this approach to consuming IT resources has increased significantly. As a result, the way internal IT teams are organized is shifting as lower-level tasks are either automated or handled by an external IT team that works for a vendor.

    Regardless of how databases are managed, however, there remains a need to integrate the data operations (DataOps) processes used to manage them with the DevOps workflows used to build modern applications. Each organization will need to determine how much economic sense it makes for them to manage that data versus relying on an external services provider. The challenge when relying on an external service provider is, as always, integrating that service into an existing IT environment.

  • Multi-Tenant Database Strategies for Mixed Workloads and Real-Time SLAs

    Multi-Tenant Database Strategies for Mixed Workloads and Real-Time SLAs

    Multi-tenancy—where a single instance or multiple instances of one or more applications operate in a single shared infrastructure environment—is not a new concept. First used in mainframe hardware, timesharing enabled a single massive piece of computing iron to be sliced up and used by multiple applications on a single hardware instance. This simplified both the hardware landscape as well as the skills and resources needed to manage environments. With the rise of software-as-a-service (SaaS) in today’s modern private and public cloud environments, multi-tenancy is now a common and critical function. It allows cloud companies to scale and customers to save money by essentially pooling resources. As more applications demand more and more data, multi-tenancy is now a core function at the database layer.

    But when you have a lot of different applications running on the same infrastructure and accessing the same database, how do you prevent one application from impacting the data performance of another? And how do you determine from the underlying database infrastructure which tenants get what infrastructure and at what performance level?

    Here’s a look at the driving forces behind today’s real-time data needs and how they impact multi-tenant data architecture management.

    SLAs Are at the Heart of Multi-Tenant Data Management

    Everyone wants faster performance and more resilient systems. We’re impatient, so reliable, real-time performance delivers large, competitive business benefits. However, every tenant does not require the same level of performance. Actual performance needs may be plenty fast for a user or machine without requiring the fastest execution path possible with the available resources.

    Service-level agreements (SLAs) ultimately govern the resources delivered to a tenant. Real-time may need to be measured in sub-milliseconds in one instance (i.e., for a stock trade) but in sub-seconds for others (i.e., in a Zelle funds transfer). This is where developers, architects and operators partner with a business to understand its actual user or system needs and to build a system that can deliver the SLA using the minimum amount of resources possible.

    Mixed Workloads and the ‘Noisy Neighbor’ Problem

    As soon as workloads are shared, there’s the potential for one application to quickly grab and use resources in a way that impacts other applications. Often called a “noisy neighbor,” the impacting applications can dramatically interfere with real-time response time SLAs when multi-tenant apps share an infrastructure base. For example, one node in a database cluster becomes heavily loaded, and the performance of other nodes begins to be impacted, causing SLAs to be missed.

    Typically, multiple virtual machines (VMs) are hosted on a single physical machine or node but share system resources (i.e., processor, network, storage, etc.) with the other VMs. It’s possible for applications to overuse those shared resources, impacting other applications and sometimes even the database itself. Such overusage of the infrastructure layer (in this case, sharing compute, storage, network queues, etc.) leads to poor database and end-to-end system performance. It’s like plugging a 100-amp appliance into a 50-amp circuit. All the circuit breakers will trip, and the entire system comes to a screeching halt!

    Strategies to Maximize Performance for Database Multi-Tenancy

    Configuring and managing the database itself (not just infrastructure) with multi-tenancy in mind can maximize performance and avoid problems like noisy neighbors. Isolating databases in your multi-tenant architecture can help minimize non-database problems that can be the root cause of poor performance and missed SLAs.

    Start by picking databases that allow applications to isolate their data access based on sets (or tables in relational parlance) that can have storage limits attached to them. When combined with rate limits for transactions on a per-user basis, applications can be guaranteed their quota of runtime resources. Using the per-app quota configurations for items like storage and TPS (transactions per second), the database can evaluate the total capacity available in the system and ensure that enough resources are reserved and available to ensure that the SLAs of all applications are met all the time.

    Note that unexpected events that cause rebalancing of data within the distributed database, such as node and network failures, require additional resources. The database system will subtract that resource capacity from what is available for apps so that the system is not overcommitted during failure situations.

    Enterprises also need very good automation and observability to fix immediate problems quickly and have the requisite visibility into long-term trends that impact performance and SLA compliance. Otherwise, by the time a human notices a problem, the SLA may already have been missed and the business impacted.

    Good observability can help you understand how data is used and create those automations. As more applications access and use the same data set in different ways, you need modern observability tools integrated into (or interoperable with) the underlying database.

    With a good understanding of how your data-rich applications interoperate with other tenants in mixed-load environments, you can begin to deploy these strategies to make sure you conform to all required SLAs and keep all that data working hard for the business.

  • Building Scalable Cloud Applications With MySQL

    Building Scalable Cloud Applications With MySQL

    Relational database systems such as MySQL have long been the go-to choice for handling large and complex data handling tasks. But, with the introduction of new technologies like NoSQL, many developers started to question whether traditional relational databases still have the upper hand. However, when building high-scalability cloud applications, relational databases like MySQL still have the edge over NoSQL. Here are five reasons MySQL remains superior to NoSQL in supporting developers of high-scale cloud applications.

    1. Enablement of Complex Queries

    Structured query language (SQL) is the most widely adopted language for relational databases, and MySQL is no different. SQL gives developers a powerful platform for creating complex queries and ad hoc reports, efficiently managing large datasets and creating tables and indexes for data optimization. By contrast, NoSQL usually offers only rudimentary query and data manipulation capabilities, often proving inefficient when handling complex data sets. Most data analysts are also familiar with SQL, so it’s a drop-in tool they can use to help gain insights.

    2. Scalability and Reliability

    MySQL is noted for its scalability when handling massive data loads and is particularly well suited to cloud-based applications that require consistently high performance. MySQL ensures that the consistency of data is maintained. MySQL is the go-to option for scalability and reliability for larger cloud applications. Companies such as Facebook, GitHub and WordPress have built billion-dollar businesses on MySQL, and it’s been tested and validated in some of the harshest and most demanding workloads of any database.

    3. High-Performance Replication at Scale

    MySQL replication offers a reliable solution for cloud applications that need to scale. Unlike other databases, MySQL’s replication capabilities provide effective data safety, making it a robust choice for applications that require high availability and durability. Additionally, MySQL enables seamless version upgrades without disrupting operations, ensuring the system always has the latest features and security patches. The read scaling feature of MySQL improves performance by distributing the read load across multiple instances, making it ideal for large-scale cloud applications. With a proven track record at scale, MySQL replication is the optimal choice for applications that want to grow and scale efficiently.

    4. High Availability and Durability

    MySQL provides enhanced high availability and durability, ensuring that critical data is always available even during maintenance and disaster recovery. This is because MySQL is fully ACID-compliant (atomicity, consistency, isolation, durability) and provides full support for transactions. Even when the system fails, the transaction can be rolled back and the data is not lost. With NoSQL databases, data is primarily written in memory and doesn’t provide the same degree of durability or consistency.

    5. Ease of Implementation and Debugging

    MySQL is easy to install, implement and debug compared to NoSQL. MySQL is supported by a large community of developers and database administrators who use it for a diverse range of applications. MySQL is also open-source and free to use, making it a great choice for businesses of all sizes. NoSQL systems require the installation of additional tools, libraries, and other components for implementation, making it more complex and time-consuming.

    MySQL provides many benefits for developers of high-scale cloud applications, ranging from its powerful SQL language for complex querying to its high availability and flexibility for managing multi-table joins. NoSQL remains a viable option in some cases, but developers who are building high-scale cloud applications are mainly sticking with MySQL due to its scalable and reliable nature, easy implementation and flexible query-handling capabilities. Choosing the right database engine is critical for any organization that relies heavily on large and data-heavy applications. MySQL continues to stand the test of time when it comes to ensuring performance and getting consistent results.

  • Oracle Bill is 5x Client’s Budget ¦ Toyota Out of Space

    Oracle Bill is 5x Client’s Budget ¦ Toyota Out of Space

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Birmingham looks like the Detroit of the UK—is it Oracle’s fault?
    Plus: Was Toyota’s factory failure caused by running out of disk space? (more…)

  • The Tragedy of the Data Commons: Balancing Innovation With Risk

    The Tragedy of the Data Commons: Balancing Innovation With Risk

    Clean drinking water and fisheries are classic examples of the “tragedy of the commons,” the problem in economics and ecology where an individual is incentivized to consume as much of a shared resource as possible at the expense of the larger society. With fisheries, for example, a logical fisherman would like to make as many catches as possible. However, this can ruin the environment for everyone, as overfishing can endanger an entire species and even threatens whole ecosystems. The modern IT environment has given us a new case to consider related to the tragedy of the commons: Data.

    Data, much like drinking water, is a critical resource. Developers need to use this data and, logically, they want to use it as much as possible to build innovative applications and products. However, too much data flowing around a modern distributed enterprise environment carries risks to the larger company.

    The data developers use often includes customers’ personal data like names and birth dates. The risks of this information being damaged or stolen can be catastrophic. This is where database administrators (DBAs), who are tasked with managing and securing this data, come in. There is an inherent tension between developers, who want as much access as possible, and DBAs, who want to limit that access for security and privacy reasons. How data is used within an enterprise is critical, because it must be used to create new solutions and be kept secure. An overly open data policy could lead to breaches and leaks, while an overly restrictive policy limits innovation.

    Thankfully, there is a solution that is helping ease this tension. Observability can automatically analyze massive amounts of information across a business’s entire IT environment. Users can use it to pinpoint the causes of outages or performance issues and receive actionable insights to resolve these problems quickly. Modern observability solutions even use AI to predict and proactively prevent problems before they occur. DBAs and developers can use observability to strike a balance while boosting innovation and improving productivity. Here are the three biggest ways that observability can solve the tragedy of the data commons and ease the tension between DBAs and developers.

    Ensuring Data is Secure

    One of the biggest concerns that DBAs have regarding data access is security. Sensitive production data are no longer stored exclusively on a company’s on-premise servers, where DBAs have full visibility into how it is being used and who has access. Instead, it is now stored across various private and public clouds as well as on-premises, which has reduced visibility for DBAs.

    This sensitive data is the same information that developers often need to build and launch applications in a continuous delivery model, where developers build software in short cycles so it can be released at any time. This involves working across different clouds, including AWS and Azure, while leveraging complicated cloud-enabled technologies like microservices and Docker containers. When data is deployed across multiple clouds, the attack surface increases while visibility decreases, leading to new security concerns.

    Observability provides DBAs with a better understanding of how data is being used by developer teams so they can more easily ensure that the data remains secure and protected. Additionally, many modern IT environments deploy threat intelligence solutions that observe actions and detect threat signals to help predict security incidents and respond before they happen. Some threat intelligence applications are powered by AI, which can automatically gather and organize complex log data from the entire network. This data is then compared with an out-of-the-box threat database, helping to detect suspicious activity early. Observability and threat intelligence working in tandem provide a comprehensive overview to help relieve some of the security concerns for DBAs and create a more secure environment for data to be shared between teams.

    Remaining Data Compliant

    Ensuring that data is compliant is top of mind for every organization. Compliance regulations such as GDPR, CCPA and HIPAA carry severe penalties for data leaks, meaning that companies must maintain the integrity and availability of sensitive and regulated data.

    Ensuring compliance while managing requests for information from developers working across geographies and clouds can be a daunting task for DBAs. Not only must they remain compliant, but many businesses must demonstrate continuous compliance, which is only possible using complex, dynamic and automated systems.

    Observability allows teams to clearly see and follow the complex flow of data as it moves between DBAs, developers and other teams. Additionally, observability can be deployed to help ensure that any sensitive data that is required to be anonymized is done so before it is shared. DBAs can further use observability to track the lineage of data as it moves between teams to monitor that it is not changed or transformed in any way that risks a compliance violation. Making compliance a regular part of the IT environment should not be viewed as a burden. In fact, it provides many benefits, including improving data quality, which can be used to further innovate and build better products.

    Making Teams More Productive

    Observability has shown it can create harmony between DBAs and developers by helping mitigate security and compliance concerns. Not only is it addressing these issues, but it is being used to make each team more productive on their own.

    For DBAs, this means gaining additional insights that help ensure applications and services are running properly. Observability collects information from across hybrid and multi-cloud environments and presents it in a single view to allow DBAs to respond quickly and resolve issues. When DBAs gain the insights that observability provides, they can better manage critical data and anticipate future problems before they arise.

    Observability is key to helping developers build applications as productively as possible. The ongoing shift left has necessitated the use of observability as a way to gain single-pane-of-glass visibility into the build process. Developers are also provided insights into the performance of web applications and can move quickly to address any problems. Finally, observability is improving the speed at which teams can innovate, as it provides live code profiling that can be used to predict and prevent future problems.

    Observability provides a real solution to ensure that data is not a resource that falls victim to the tragedy of commons, where its use can harm an entire organization. Instead, it is powering teams to use data securely to work more productively and build innovative applications.

  • InfluxData Makes Processing Observability Data at Scale More Efficient

    InfluxData Makes Processing Observability Data at Scale More Efficient

    InfluxData today made available an update to its open source time series database that can now analyze metric, event and trace data in a single datastore with unlimited cardinality in terms of how they are aggregated.

    The company is now making available a single-tenant instance of InfluxDB as a managed service alongside its already-existing multi-tenant cloud service that is based on a serverless architecture.

    Finally, InfluxData also announced that later this year it will make available InfluxDB 3.0 Clustered and InfluxDB 3.0 Edge to provide a curated version of the database that organizations can deploy themselves where they best see fit.

    InfluxData CEO Evan Kaplan said InfluxDB has been developed to support a wide range of emerging applications that require access to time series data, including the observability platforms core to DevOps workflows that require visibility into metrics, events and trace data.

    Over the past three years, InfluxDB has been revamped to run on a columnar engine, dubbed IOx, that leverages the open source Apache Arrow memory format and written in the Rust programming language. Kaplan said that approach makes it possible to continuously ingest, transform and analyze hundreds of millions of time series data points per second without limitations.

    At the same time, InfluxDB takes advantage of high compression object storage to reduce the total cost of storing all that data. It also provides interoperability with Open Data Architecture (ODA) to integrate with data lakes based on open source platforms such as DataFusion, Flight SQL and Parquet that are being advanced by the Apache Software Foundation.

    The arrival of version 3.0 of InfluxDB comes as many DevOps teams are starting to struggle with the amount of observability data being generated. DevOps teams want to be able to move beyond monitoring a set of pre-defined metrics and query data in a way that enables them to surface anomalies indicative of a potential issue before there is a major disruption to an application service.

    One of the major challenges today is the tradeoff between how much data is collected and analyzed versus the cost of processing and storing it. InfluxDB provides a mechanism for analyzing high cardinality data involving metrics, events and traces cost-effectively.

    Observability is, of course, only one of several use cases for a time series database capable of processing data in near-real-time. As organizations embrace digital business transformation initiatives, they need to be able to process data in near-real-time at the point where it is being created and consumed. Those applications won’t necessarily replace existing applications based on batch-oriented processing but, over time, they create two distinct classes of applications that process data in fundamentally different ways. The challenge, as always, will be defining the DevOps workflows required to manage applications running on multiple distinct types of architectures.