devopsy.kinsta.cloud

Tag: data storage

  • Tiering Cold Data to the Cloud Without Tears

    Tiering Cold Data to the Cloud Without Tears

    The cloud provides inexpensive storage. But like everything else in the cloud, there are hidden costs. Cloud storage providers (CSPs) charge to put data into and retrieve data from the cloud. They charge for the API calls and they generally charge an egress cost when the data is extracted from the CSP. So, to keep enterprise storage costs low, infrequently accessed data such as snapshots, logs, backups and cold data are best suited for tiering to the cloud.

    By tiering data, on-premises storage arrays need to only keep hot data and the most recent logs and snapshots. Typically, 60% to 80% of enterprise data has not been accessed in over a year. By tiering the cold data as well as older log files and snapshots, the capacity of the storage array, mirrored storage array (if mirroring/replication is being used) and backup storage is reduced dramatically. This is why tiering cold data can reduce overall storage costs by as much as 70%.

    The many advantages of this approach include:

    • Lower acquisition cost. Flash storage, used for fast access to hot data, is expensive. By tiering off infrequently used data, you can purchase a much smaller amount of flash storage, thereby reducing acquisition costs.
    • Lower backup and mirroring costs. By continuously tiering cold data, you can reduce your footprint, license costs and storage costs for backups and replication if the cold data is placed in robust storage (such as that provided by the major CSPs).
    • Improve performance and lower capacity of your storage array. By running storage at a lower capacity and by moving access to cold data to another storage device or service, you can increase the performance of your storage array and get by with a smaller storage array.
    • Enable processing of cold data without burdening the storage array. Processing and feeding your cold data into your AI/ML/BI engines is critical to staying competitive. With cold data in the cloud, you’re also reducing the load on your storage array, thereby extending its life.

    Know First, Then Tier

    A key challenge of cloud tiering is determining what data to tier. End users should not have to decide which data to archive or handle the management of shared files. This issue may result in IT organizations keeping too much data in expensive, hot storage. One way around this is to automate the process through business policies dictating when and how the process is applied. It’s also helpful to provide transparency of the data by keeping it in the existing namespace. Users can still find the data and access it as if it had never been tiered. Transparency enables IT to tier cold data continuously and systemically across all storage devices in the entire organization without requiring end user assistance.

    To help IT make the right decisions, analytics is needed to see just how much data you have, and how much has not been touched in three, six or 12 months. It will be even more compelling if IT can run what-if scenarios to determine how much they will save with cloud tiering. The last thing you want to do is tier data in the dark. Take time to understand your data assets before you invest in the effort.

    Here are other considerations, beyond analytics, when embarking on a cloud tiering initiative:

    • Transparent, continuous tiering. Tiering should be transparent to the user. If someone can still search for and access data that’s tiered without any change to the experience, IT can realistically tier data systemically across the enterprise. Without transparent tiering, it’s an onerous process to tier data because you will need user permissions.
    • Flexible tiering policies. You need the flexibility to exclude certain data sets, include others and select from a large range of data ages. Be aware that many tiering solutions provide an extremely limited set of policies.
    • Tiering across multi-vendor storage arrays. Most enterprises have storage arrays from multiple vendors, and each appliance will have different tiering solutions if any. This makes it difficult to roll out a consistent, global tiering policy across the organization. Look for a solution which can encompass all the storage devices you have today and might acquire tomorrow.
    • Fast access to tiered data. Accessing data from the cloud will incur a much higher latency than accessing data on the local storage array. But once the cold data has been retrieved you can cache it locally to eliminate future latency and reduce egress costs. If the data set is large, the tiering solution should stream it so that users can access it even while the rest of it is being recalled.
    • Native cloud access to tiered data. The tiering solution should allow access to the cold data directly using the cloud storages native access tools. For instance, if you tier cold data to AWS S3, you should be able to access the data directly from AWS using a standard S3 browser like CloudBerry. Unfortunately, most storage array tiering devices store the data in a proprietary format, meaning that you can only access it from the source storage array.

    Cloud tiering is a practical and easy step on the path to the cloud. When done right, it gets you into the cloud while reducing storage costs. There are new solutions available that make this approach seamless with no disruption to users and your existing data protection workflows. By running an analysis of data assets across your storage ecosystem, and setting up policies for migration of cold data, you can ensure that data is always living in the best places from both a cost and business perspective.

  • Understanding Data Storage: Lakes vs. Warehouses

    Understanding Data Storage: Lakes vs. Warehouses

    Now more than ever, companies are looking for new ways to incorporate data analytics into their daily operations and leverage data-driven insights to improve business functions. They are frequently turning to complex data for tasks like machine learning and artificial intelligence, which are becoming necessary to understand and reach customer segments across industries. However, understanding data storage is a key factor in developing a successful strategy. The two most common storage formats are data lakes and data warehouses – but there are benefits and pitfalls to each that organizations must understand in order to properly capitalize on them.

    Data Lakes vs. Data Warehouses Use Cases

    One effective way to better understand the different functions of data lakes and data warehouses is to focus on the end use case. Typically, there are three types of data consumers: farmers, explorers and executives.

    • Farmers need data and information to execute their day-to-day activities. They must report on the key performance indicators (KPIs) required to execute their job by providing data in a structured format.
    • Explorers are users who want to experiment with data, look at new types of data pools and gather specific insights. If data is presented in a fixed format, it hinders their progress, as they seek data that doesn’t have a pre-defined structure.
    • Executives are responsible for making business decisions, and often require information presented at different aggregates, with the ability to narrow in on data subsets as needed. However, they require information that is somewhat structured, and speed is critical for them.

    If we look at these perspectives more broadly, they essentially boil down to two key types of data usage – the exact needs that data warehouses and data lakes are designed for. Farmers and executives are largely served through data warehouses, which require information to be timely and structured. Some of the main use cases for data warehouses include operational analytics, business intelligence and predictive analytics for data science and machine learning applications. Explorers are best served through data lakes, since the data formats are not predetermined. Data lakes are often most effective for use cases like a centralized data catalogue that enables organizations to view all sources of data in a single place, or recent advances in AI application development that require the processing of unstructured data, such as text, images, video and audio.

    Challenges and Convergence of Storage Formats

    Despite the differences, most companies can use both data lakes and data warehouses for their various analytics needs. However, to successfully capitalize on the benefits, it is also important to understand the challenges. In particular, data lakes present difficulties when it comes to parsing data. They were initially designed to capture data across the enterprise in its natural format, without enforcing schema, so that users could garner more insights; a fundamental aspect in creating a data-driven culture across the organization. However, without a predetermined use case, data lakes can quickly become data swamps. End users were historically unable to figure out whether the data was stale, and ownership was not well established. This has been remedied by taking a use case-driven approach and only including data that has defined use cases or ownership.

    Now that many of the issues surrounding data lakes have been resolved, the two storage formats must complement each other to meet the varying needs of the customer. This has contributed to blurred lines between data lakes and data warehouses. Data lakes are now capable of schema enforcement and answering rapid business intelligence queries, which were traditionally qualities of data warehouses. Data warehouses have separated compute and storage and can read directly from big data file systems, enabling users to read semi-structured data. As such, we are rapidly moving toward integrated data environments and the convergence of data lakes and data warehouses.

    Supporting Data-Driven Initiatives

    For initiatives like artificial intelligence and machine learning to succeed, data must be presented as an immutable entity that can be used for experimentation. With data lakes, it is crucial to separate the data into different zones and maintain a refined zone after transformation. Then, in the refined zone, companies can enforce schema and allow the schema to evolve to ensure the data is ready for machine learning and data science needs. Most importantly, they must catalog the data and enforce metadata management, data quality and governance. Data warehouses, on the other hand, excel in providing data sets ready for discovery and consumption. Companies should integrate these data sets with an interactive data catalog so they are discoverable – this is the most important step in making artificial intelligence and machine learning possible.

    If organizations can learn how to best capitalize on data warehouses and data lakes for their intended purposes, they will be well equipped to uncover key data-driven insights to help guide their business strategy. This will help enable use of other advanced technologies and help transforming them into data-driven companies.

  • Nebulon Unveils Hybrid Cloud Computing Platform

    Nebulon Unveils Hybrid Cloud Computing Platform

    Nebulon has emerged from stealth to launch a hybrid cloud computing platform that enables workloads to be processed in the cloud while data continues to be stored on a local application server.

    Company COO Craig Nunes said the Cloud-Defined Storage platform employs PCIe cards embedded with a Nebulon Services Processing Unit (SPU) based on an ASIC to connect local storage systems to a cloud-based control plane dubbed Nebulon ON that can span Amazon Web Services (AWS) and Google Cloud Platform (GCP) services.

    The PCIe card is installed in an application server residing in a local data center, where it connects to solid-state drives (SSDs) installed on the server. The Nebulon SPU offloads compression, encryption, deduplication, erasure coding, snapshots and data mirroring.

    The Nebulon SPU also transmits storage, server and application metrics to Nebulon ON, where machine learning algorithms are applied to surface recommendations to improve performance.

    Nunes said both Hewlett Packard Enterprise (HPE) and Supermicro have already agreed to resell the Cloud-Defined Storage platform.

    At its core, the Cloud-Defined Storage platform takes hyperconverged infrastructure (HCI) to another level. Instead of combining compute and storage in a local data center, however, compute processing is shifted to the cloud, said Nunes. That approach, he noted, makes it possible to take advantage of inexpensive compute resources in the cloud without losing control over data.

    In addition, Nunes noted, the Cloud-Defined Storage platform ensures application service levels are maintained while reducing costs by eliminating the need to transfer massive amounts of data into and out of the public cloud.

    Organizations that work within highly regulated industries can now take advantage of cloud resources while continuing to meet compliance requirements, he added.

    Finally, the Nebulon Cloud-Defined Storage platform can support both monolithic and microservices-based applications regardless of the type of virtual machine or container employed.

    At a time when many organizations are under increased pressure to reduce the total cost of IT, there’s a lot more interest in ways to leverage cloud computing to reduce costs. However, for many organizations, moving data into the cloud is problematic for a host of performance, security and compliance issues.

    A hybrid cloud computing architecture that doesn’t require data to be stored or moved into and out of a cloud could provide a way for organizations to re-evaluate their overall cloud computing strategies. Organizations beyond those in highly regulated industries could conclude they would prefer to retain control over their data.

    Cloud service providers are, of course, banking on massive amounts of data being stored in the cloud. They have long argued that as the amount of data stored in the cloud exponentially increases, the more sense it makes to deploy applications in the cloud as close as possible to that data. Each new application deployed in the cloud then increases the amount of data gravity that cloud exerts on other application deployment decisions.

    It’s not likely hybrid cloud platforms such as the Nebulon Cloud-Defined Storage system will completely reverse that trend. However, it does signal that data gravity in the cloud may not be as inexorable a force as many cloud service providers have long assumed.

  • Commvault Deepens Ties With NetApp to Automate Data Protection

    Commvault Deepens Ties With NetApp to Automate Data Protection

    Commvault has extended its alliance with NetApp to make it possible to recover files nearly instantaneously from an all-flash memory array.

    Ranga Rajagopalan, vice president of product management for Commvault, said NetApp Scale-out Data Protection (SDP) from Commvault builds on its backup and recovery software to make it possible to use to NetApp hyperconverged infrastructure in combination with all-flash arrays running StorageGRID object storage software to now recover data stored on-premises or in the cloud.

    The on-premises capability is critical because it eliminates potential downtime in the event of a ransomware attack, Rajagopalan said, noting data can be recovered from an all-flash array residing on-premises using a scale-out architecture in a matter of seconds.

    At the same time, IT organizations can take advantage of an object storage system from NetApp that is compatible with the S3 application programming interface (API) to store data in the cloud, he added.

    Most IT organizations are by now aware the only way to combat ransomware is to be able to access a pristine copy of their data. The challenge they face, however, is finding a way to make that data accessible as fast as possible. It not only takes a significant amount of time to extract data from a cloud service, but it is also expensive. By keeping an organization’s most recent data available in an all-flash array locally, the time to restore data is sharply reduced, said Rajagopalan.

    Of course, organizations have been keeping copies of data on magnetic storage systems for years. However, as the cost of all-flash arrays continues to decline it’s now more affordable for organizations to consider employing much faster all-flash arrays to back up and recover their most mission-critical data, he added.

    Commvault also makes available application programming interfaces (APIs) through which the whole data protection process can be automated, he said.

    In the wake of the COVID-19 pandemic, there are two issues that many organizations are now struggling to manage. The first is the need to automate more processes at a time when many organizations will have smaller IT staffs. The number of IT personnel that can be dedicated to manually managing data protection tasks will be fewer during the economic downturn.

    At the same time, however, the number of ransomware attacks being launched continues to increase. All it takes is one mistake for an organization to find all its data encrypted. As a result, the relationship between data management and cybersecurity has become much tighter as organizations strive to ensure they always have a pristine copy of their data available.

    In the months ahead there is no doubt there will be a lot more focus on the total cost of IT operations. The challenge is, finding a way to reduce those costs must be balanced against the existing commitment to service level agreements (SLAs). The only way to meet those often-conflicting goals is to rely more on IT automation frameworks to make data always available, in a way that requires much less human intervention.

  • Scalyr Alters Log Analytics Economics

    Scalyr Alters Log Analytics Economics

    Scalyr today announced that its log analytics cloud service is now capable of ingesting over 200TBs of data per customer per day in real-time.

    Company CEO Christine Heckart said the goal is to make as much log data available at a price point that will continue to be less than $5 per GB at scale.

    Heckart said Scalyr expects to be able to continue to expand the amount of data it makes available to customers because its namesake platform is built on top of a columnar data store rather than an index engine. The goal of a project dubbed Sonic Boom is to eventually make a petabyte of data available to customers of the cloud service, she said.

    One of the paradoxes of log analytics is that while IT organizations are encouraged to use this data to proactively identify issues before they become a major problem, the cost of storing that data often results in IT organizations limiting the amount of log data they analyze. As a result, they often miss outlier events that could have major implications for the IT environment.

    As a cloud service that is compatible with the S3 storage interface defined by Amazon Web Service (AWS), the columnar store engine employed by Scalyr makes it easier to pass the economics of cloud storage back to the end customer, she said. IT costs in the wake of the COVID-19 pandemic are, of course, going to be a much bigger issue going forward. Almost every aspect of IT, including log analytics, is likely to be subject to a cost review in the week and months ahead as organizations adjust to an economy that has changed for the worse overnight.

    Heckart said the issue many organizations will face when it comes to logs is that data is the lifeblood of any digital process. Organizations may not need to store that data, forever, but analyzing log data to identify anomalies indicative of, for example, a cybersecurity breach is now almost a daily routine. As such, how much log data to make available where at what cost becomes a subject of debate.

    Of course, it may be less costly to store log data locally. However, someone on the IT staff will need to manage those logs. Scalyr and other providers of SaaS-based platforms for log analytics contend it’s more cost-effective to rely on a cloud service to manage log data that also is more accessible via shared analytics tools.

    Regardless of the amount of IT resources made available, the one thing that is certain is neither the amount of data nor the rate at which it needs to be analyzed will be slowing down anytime soon. In fact, many organizations are likely to accelerate their transition to digital processes that are ultimately both more flexible and cost-efficient than existing processes. It will be up to DevOps leaders to figure out how best to keep pace with that rate of change with whatever resources are at hand.

    — Mike Vizard

  • Humio Adds Unlimited Ingest Plan to Log Management Service

    Humio Adds Unlimited Ingest Plan to Log Management Service

    Fresh off raising an additional $20 million in funding, Humio, a provider of a platform for analyzing log data, has added an unlimited ingest plan to a cloud-based service to make it easier for organizations to retain massive amounts of log data.

    The unlimited ingest plan for the company’s cloud service extends a similar option Humio already makes available to customers that deploy its log management platform in an on-premises environment.

    Morten Gram, executive vice president for Humio, said one of the significant limitations of rival log management platforms is they charge customers based on the amount of data ingested. That approach only serves to dissuade organizations from ingesting data to limit their costs. The trouble is that organizations won’t be able to discover outlier issues in data they never see in the first place, he said, noting issue becomes even more problematic as organizations embrace best DevOps practices to deploy more applications both in the cloud and in on-premises IT environments. The Humio Bucket Storage can retain more than a petabyte of log data.

    Given the scale and scope of modern applications, Morten said organizations are now using log data to drive a wide variety of analytics tools to automate processes in real-time. Log data is also playing a critical role when it comes to applying artificial intelligence (AI) to IT management. The more log data made available, the more accurate the AI models being employed become, he noted.

    In addition, Gram said that because Humio doesn’t rely on an index engine to organize data for a search engine, IT teams can launch any type of query they like. In contrast, rival platforms are optimized for specific types of queries, he said, which also substantially reduces the amount of compute and storage required to analyze log data. That capability is why Humio can offer unlimited ingest of data and still remain profitable, as long as there are enough customers to enable Humio to operate at scale, he noted.

    Given the current need for many IT employees to work from home to help combat the COVID-19 pandemic, it’s clear more organizations will be relying on tools hosted in the cloud to manage IT environments. Many of those organizations will also be a lot more sensitive to costs given the impact the COVID-19 pandemic is having on the economy. Of course, it’s probable many organizations will postpone new application development projects until sometime after the COVID-19 pandemic subsides. However, as more organizations start to appreciate the extent to which people may adopt social distancing, many of them will soon be re-engineering business processes to adjust to a new reality. As such, it may now only be a matter of time before there are even more digital business transformation projects being launched than ever before.

    — Mike Vizard

  • Storj Labs Unfurls Decentralized Storage Service

    Storj Labs Unfurls Decentralized Storage Service

    Storj Labs has made generally available a distributed storage platform based on the excess capacity available on devices ranging from servers to individual desktops.

    Company CEO Ben Golub said the goal is to provide IT organizations with an alternative to storing data inexpensively using storage resources that have been vetted by Storj Labs.

    At the core of the Tardigrade Decentralized Cloud Storage Service provided by Storj Labs is an open source object storage system that is compatible with the S3 application programming interface (API) defined by Amazon Web Services (AWS). That object storage system distributes 80 slices of a file across a subset of thousands of nodes that each contain only a slice of the data required to reconstruct that file when needed. Only 29 of those slices are needed to reconstruct a file, so Golub said it’s relatively simple to ensure data is secure without having to manage encryption keys.

    Following an extensive beta test of the Tardigrade, the company claims to have achieved 99.95% availability, with a service level agreement of 99.9999999% achieved in terms of durability. Golub said Tardigrade makes use of erasure coding to make sure data is distributed in a highly resilient fashion.

    Storj Labs compensates owners and data centers along with individuals with access to a lot of excess storage capacity for making capacity available, Golub said. Most storage drives are not anywhere near full capacity, which Golub noted has made it relatively easy for Storj Labs to amass 19PB of storage for its Tardigrade service. Each provider of storage capacity is required to pass a series of data stewardship tests for a period of time before being allowed to become a member of the Tardigrade service.

    Storj Labs has also set up an open source partner program through which any open source project with a Tardigrade connector will receive a portion of the storage revenue generated by users of that software. That approach allows participants in projects to be partially compensated for their efforts as alternatives to cloud service providers, which routinely commercialize open source software without helping to fund its development, said Golub. Based on his previous experience as CEO of Docker Inc., he noted many open source projects die on the vine simply because the team that created it simply can’t afford to sustain it.

    Interest in Tardigrade Decentralized Cloud Storage Service for certain use cases as an alternative to pubic clouds could be high, he said, given the fact that there has been no meaningful decline in cloud storage service pricing in the last three years. Use cases that should prove to be especially appealing include data protection and image retention. Storj Labs is also making available a library of bindings for Go, Android, C, .NET/Xamarin, Node.js, Swift and Python to make it easier for developers to invoke Tardigrade storage services as code.

    It may be a while before IT organizations are completely comfortable with decentralized storage services as an alternative to public clouds. However, as the amount of data that needs to be stored continues to grow, every option available to store data may be needed soon.

    — Mike Vizard

  • You Still Have to Make Time for Maintenance

    You Still Have to Make Time for Maintenance

    I’m going to take a side trek today into home computing with applicability to IT. Bear with me.

    We maintain a pretty complex home network, as home networks go. With three NAS servers, a small one-box SAN, some web servers and connections to several hosted environments. We both work from home, and my business is contained in one of the NAS servers and some of those hosted environments, so it’s all necessary. Scheduling downtime with several people using the network darned near 24/7 is difficult. Not medium enterprise level of difficult, but still an issue.

    Two of our NAS servers each have a disk out. It happens, that’s why they’re NAS servers. In general, the only day that replacing the disks makes sense is Sunday, when things are a little slower (both in schedules and network usage) than normal. One of our web servers is just plain dead–but there’s info on there we have to get off. The code is mostly being replaced, but the data is collected over years and we need to see how much we can salvage.

    None of these things are shiny DevOps things. Yet, they still have to get done. Importantly, they have to get done when automated tools aren’t running across the network or counting on the NAS to be responsive. The rush isn’t huge, because there are off-site backups, but restoring a remote backup is a long-term proposition–weeks, not days in our current environment–so it must happen as soon as practicable.

    Don’t forget that with the drive to automate everything. If you own the hardware, it’ll need maintenance. If you own the software, same thing. If you need data to move platforms, it will take time. All things we kind of hand-wave in the DevOps world. They’re seen as bottlenecks, if seen at all. Generally, we don’t even consider maintenance issues until they impact DevOps, but we know they are going to happen so we should have a plan for how to deal with them.

    Those few of you whose entire environment is hosted in a cloud provider have an easier job of it, but eventually you will be asked to move cloud providers. As sure as the sun will rise tomorrow, something will drive a move away from your chosen platform. Normally TCO, but other factors play in, too. When that happens, the work will be more than “Find time for the NAS to rebuild without overly impacting everything else.”

    So, spend some time thinking about it. Do you have plans to make room for maintenance? Are you scheduling maintenance into release cycles? Are you coordinating releases with updates that are not required by this specific release?

    This is where dev (typically Agile/Scrum) meets ops (typically system/network maintenance), and it is every bit as important as what is normally seen as DevOps. Get it right and have fun, keep cranking stuff out. Get it wrong and struggle with the problems infrastructure issues can inflict upon your cool new update. One of our new websites outright stopped development when the web server went down–for a variety of reasons, but the catalyst was the loss of the original. We’re working out what to do about that.

    Meanwhile, we didn’t make time yesterday, so I’ll be replacing disks and tearing apart a web server in a much smaller window that is available today. See you when I poke my head back up. Hopefully, you’ll have better planning than we did.

    — Don Macvittie

  • The Acceleration of Data and Data Storage Trends

    The Acceleration of Data and Data Storage Trends

    There’s never been a more exciting time for data. Increasing demands for size, speed, performance and reliance have propelled the data market forward at a faster rate than could have ever been anticipated, and new technologies are making some dreams for data a reality.

    From open source adoption in place of legacy databases to cloud-native app stacks and more, here are seven trends driving the acceleration of data and data storage. 

    Oracle’s Slow Fade Drives Cloud-Native Adoption

    Organizations of all sizes continue to eschew Oracle for cloud-native databases, shrinking the user base of the monolithic legacy provider. Two major trends driving this activity are the ongoing move to microservices-based redesign of applications and the rise of cloud-native deployments running in public clouds or Kubernetes. Overall, Oracle has lost market share every year since 2013, and legacy relational database players have dropped about five percentage points per year. Major organizations, such as Amazon and Salesforce, have already figured out that it’s to their benefit to use less, not more, Oracle.

    Open Source DB Usage Increases

    Open source databases that claimed 0% of the market 10 years ago, now make up more than 7%. It’s clear that the market is shifting and an increase in open source database adoption is on the rise. This goes against the recent trend of database and data infrastructure companies abandoning open source licenses for some, or all, of their core projects. However, as technology rapidly advances it will be in the best interest of database providers to switch to a 100% open source model, since freemium models take a significantly longer period of time for the software to mature to the same level as a true open source offering.

    Furthermore, Gartner predicts that, by 2022, more than 70% of new in-house applications will be developed on an OSDBMS or OSDBMS-based dbPaaS. It’s clear that open source has proven to be the most successful approach to developing and distributing business-critical infrastructure software, driven by the benefits of reduced total cost of ownership, access to community support and innovation and compatibility with commercial databases, making it easy to migrate towards an open source solution. 

    Kubernetes For Stateful DBs

    There has been continued growth in the adoption of both microservices and containerization. With the combination of massive adoption and the support of a large and thriving open-source community, it is clear that the dynamic nature of containerization and microservices deployments will have huge implications on fundamental application architecture. To put it simply, this is the future of how applications will be built. This is also why the adoption of Kubernetes has continued to grow.

    Multi-Cloud Goes Mainstream

    Multi-cloud deployments are becoming the norm in today’s enterprise. A multi-cloud approach is critically important for organizations that run on-premise, since they need to stay in a hybrid mode when moving microservices to the cloud. As a result, enterprises are widely embracing distributed SQL databases to ensure agility without the availability constraints of traditional monolithic databases, such as Oracle. 

    Rise of Edge Data Centers

    While the rise of edge data centers is assuredly a trend, Gartner predicts that three quarters of all databases will either be deployed or migrated to a cloud platform by 2022. There will also be an increase in multi-cloud adoption as a result of the geo-distribution of data. Edge data centers are already starting to offer data services with geo-distribution, a clear sign that demand is creating supply.

    Cloud-Native App Stacks Gaining in Popularity

    Modern microservices-oriented architectures are becoming increasingly popular. Over time, this leads to environments with lots of microservices that should be able to notify one another of changes and store these changes in a system of record. The application stack that is getting very popular for such use-cases is a combination of:

    • Spring framework to build the applications.
    • Kafka as the message bus to send notification between the microservices.
    • A Distributed SQL database to store the data in a scalable manner and query it efficiently.

    Other application stacks that are rising in popularity are the JAMstack ( Javascript, APIs and Markup) where there is no dependency on any particular web server. Javascript frameworks such as ReactJS and VueJS have become very popular for building the frontend portion of the app. GraphQL frameworks such as Apollo and Hasura are getting increasingly adopted as efficient ways to build the API portion of the app.

    Advances in innovation for data and data storage will undoubtedly continue, driven by major trends across all areas of business, as long as organizations are prepared to embrace change and hit the accelerator on their data strategies.

    — Karthik Ranganathan

  • Building a Real-Time App: 13 Open Source Projects You Should Be Tracking Right Now

    Building a Real-Time App: 13 Open Source Projects You Should Be Tracking Right Now

    Open source is the dominant model of software consumption in the modern era. Cutting-edge startups and entrenched incumbents alike find open source software development and community building a significant part of their overall business strategies.

    For many open source projects, independent steering committees ensure that projects remain stable and are not beholden to a single company or profit motive. Alternatively, for open core projects where a single company financially supports and contributes the vast majority of open source code, open sourcing makes a project publicly available and accessible for potential new users. Opening a project to public contribution widens the pool of potential users and contributors, therefore ensuring more rapid innovation than commercial software offerings.

    Choosing the Right Open Source Projects for Your Real-Time App

    There are seemingly limitless open source projects to evaluate. Further complicating matters, each project has its own domain competence and cultural nuances. Considering a single distributed application will have tens or (many) more open source dependencies, development teams must continuously test new open source offerings and hone new skills to use these solutions, while simultaneously architecting their applications.

    In order to help sort through the noise, I’ve looked at six categories of open source projects for this article: data storage, message systems, service meshes, REST frameworks and streaming frameworks. For each category, I’ll identify any dominant players as well as some other projects of note.

    Here are the top 13 open source projects by category.

    Data Storage: Cassandra, Hazelcast

    Apache Cassandra, originally developed at Facebook, is one of the most widely used open source projects for real-time applications. Functionally, it’s a distributed NoSQL database management system. But practically, it’s become the linchpin of most at-scale, real-time applications. This is because Cassandra was designed to handle large amounts of real-time data across many servers, and has robust clustering support.

    For applications that require extremely low latency or have exceptionally high data rates, Hazelcast may provide a useful alternative. Unlike Cassandra, Hazelcast is an in-memory data grid database, which enables development teams to benefit from the latency advantages of in-memory computing. 

    Message Systems: Kafka, Pulsar, NATS

    Apache Kafka is the dominant open source messaging system used by application development teams today. Originating at LinkedIn, Kafka has grown into the de facto standard for high performance open source messaging systems. For streaming use cases, Kafka users can also turn to Kafka Streams.

    Apache Pulsar, developed at Yahoo!, is another alternative messaging system which has been optimized for streaming data sources. Pulsar further differentiates itself with TTL message support and queuing via shared subscription.

    For applications where latency is a critical consideration, NATS.io (a CNCF project) is one of the fastest pub-sub messaging systems available.

    Service Meshes: Istio, Linkerd

    Trends such as microservices, serverless and hybrid cloud deployment have contributed to the increasing architectural complexity of distributed applications. With hundreds or thousands of distributed application services, service meshes provide a mechanism for pulling all the disparate services together into a manageable and cohesive whole.

    The most notable open source service mesh offering is Istio. One of the most popular open source service mesh projects, Istio is primarily used for applications built on top of Kubernetes. Another alternative is Linkerd, a flexible service mesh solution for Kubernetes or other frameworks.

    REST Frameworks: Node.js, Spring

    Node.js is a popular JavaScript framework for building scalable web applications. When comparing web development frameworks, Node.js typically ranks among the top three most (along with frontend frameworks React.js and Angular.JS). While Node.js can support large scale applications, it does not support real-time use cases well at scale.

    Spring is another application framework, optimized for distributed microservices-style architectures. Like Node.js, Spring “focuses on the ‘plumbing’ of applications so that teams can focus on application-level business logic, without unnecessary ties to specific deployment environments.”

    Streaming Frameworks: swimOS

    SwimOS is a software framework for distributed streaming applications. Unlike REST-based frameworks, swimOS utilizes a unique and lightweight Web Agent architecture, optimized for composing and transforming many data streams in real-time. Whereas DevOps oriented projects, such as Spinnaker, seek to provide an abstraction layer to converge distributed cloud services, swimOS provides an abstraction across distributed data streams, so teams can build applications by composing many distributed data streams in a single mesh environment.

    In order to ensure real-time consistency of distributed services, swimOS automatically ensures eventual consistency between all Web Agents. Complete with a UI toolkit and it’s own stateful messaging protocol called WARP, swimOS is the fastest way to build stateful streaming applications.

    DevOps: Kubernetes, Spinnaker, RancherOS

    Kubernetes is one of the most talked about open source projects in the world. Kubernetes describes itself as “an open-source system for automating deployment, scaling and management of containerized applications.”

    While it may simply be a platform for container orchestration, rightly or wrongly, Kubernetes has become a stand-in for DevOps on the whole. In spite of its dominance, there are alternatives to Kubernetes, for example there’s RancherOS to manage Docker containers.

    Furthermore, as application architectures continue to become more distributed and complex there is a need for platforms which can converge multiple distributed edge and cloud environments. Spinnaker provides an abstraction layer to connect the various environments and provides a simple mechanism for continuous integration/continuous delivery (CI/CD). 

    Final Thoughts

    Your own particular use case should determine the appropriate open source technologies for your team’s real-time application. Whether you’re building an ultra-high performance low latency application or simply trying to tame large volumes of real-time data, knowing your options and choosing the right framework or platform can save you significant time and effort.

    The open source software landscape is changing every day. If you have something to add about one of the technologies above or think I skipped over the next OSS breakthrough, I’d love to hear about it. Let me know your thoughts in the comments section below.

    — Chris Sachs