Tag: apache

  • Stream Big, Think Bigger: Analyze Streaming Data at Scale

    Stream Big, Think Bigger: Analyze Streaming Data at Scale

    Data streams are all the rage. Once a niche element of data engineering, streaming data is the new normal—more than 80% of Fortune 100 companies have adopted Apache Kafka, the most common streaming platform, and every major cloud provider (AWS, Google Cloud Platform and Microsoft Azure) has launched its own streaming service.

    This shift is fueled by the growing demand to deliver events quickly, reliably and at scale to support a wide range of internal and external applications. From always-on, customer-facing applications to mission-critical operations, a growing number of use cases depend on subsecond updates—further upping the ante for real-time data processing and dissemination.

    There’s no doubt 2023 will be a big year for streaming technology.

    Meanwhile, as streaming moves toward ubiquity, we’re seeing another shift in the way businesses use the data: Analyzing events as they’re created for real-time insights. With the right tools in place, people can instantly compare what’s happening now to what happened in the past, triage problems as they occur and make time-sensitive decisions in the moment.

    With the proliferation of data streams, a new set of use cases and requirements for real-time analytics has emerged. To unlock the power of streaming in 2023, data teams will need to look beyond the traditional batch-oriented stack and adopt a streaming-native approach to the analytics architecture.

    The Streaming Paradigm Shift

    Data has always been at the center of how businesses operate. Until recently, we lived in a very batch-dominant world of business and data. The goal of data infrastructure was to identify data at a fixed point in time and store it for eventual use. The evolution from daily batch operations on mainframes to the internet-driven always-on world replaces fixed, static “data at rest” with fast-moving data in motion, as information flows freely between applications and data systems within and between organizations.

    Data at rest still exists, and batch systems continue to serve a variety of reporting purposes. But reality isn’t fixed or static, and to meet the demands for seamless, authentic data experiences, the systems we build must be designed for data in motion.

    Hence, the rise of streaming technology—and with it, a new mindset for how we think about data. As streaming increases, streaming data platforms increasingly act as an organization’s central nervous system, connecting the different functions and driving mission-critical operations. Evolving technologies are purpose-built to support data in motion systems: Stream processors and event databases.

    Apache Druid fits into the purpose-built category; as a real-time analytics database, it is designed to enable users to query every event as it joins the data stream and do it at a tremendous scale while enabling subsecond queries on a mix of stream and batch data.

    Thousands of businesses are already using streaming processors like Kafka and Amazon Kinesis with Druid to build cutting-edge systems that make terabytes to petabytes of streaming data accessible to applications and humans in milliseconds. Some of those businesses—including Reddit, Citrix and Expedia—were highlighted at Current 2022, the annual streaming event organized by Confluent.

    The next evolution of data intelligence is here, and it’s based on the ability to react to events as they are happening. But on the growth chart of streaming adoption, we are only at the beginning of an upward curve—a critical inflection point where streaming (and the technology that is built for it) starts to become the bedrock of everyone’s data architecture.

    So, when it comes to enabling scalable, subsecond analytics on streaming data, many developers and data innovators are wondering, ‘How?’

    We’re Streaming! Now What?

    At Current, we spoke with hundreds of Kafka users who were turning their attention to building out the next phase of their data streaming pipelines and embedding them more deeply into their organizations.

    While streaming adoption is becoming more widespread, most companies still have just one or two use cases they’re solving with a streaming platform. At Current, many people spoke about how Kafka effectively set their data in motion. But when it came time to analyze or put those streams to work in a user-facing application, their “data in motion” turned into “data in waiting” because their analytics system was designed for batch data, not streaming.

    To solve this problem, a new type of database is needed—and that’s why Druid was created. Turning billions or trillions of events into streams that can then be immediately queried by thousands of users simultaneously is where stream processors like Kafka, in combination with Druid, have unlocked a new set of capabilities that are enabling new developer-built analytics applications.

    For example, Reddit generates tens of gigabytes of event data per hour from advertisements on its platform. To let advertisers understand their impact and decide how to target their spending, Reddit needs to enable interactive queries across the last six months of data—hundreds of billions of raw events to slice and dice. But they also needed to empower advertisers to see user groups and sizes in real-time, adjusting based on interests (sports, art, movies, technology…) and locations to find how many Reddit users fit their target. To achieve this, they built a Druid-powered application that ingests data from Kafka and enables Reddit’s advertising partners to make real-time decisions and get the best possible ROI on their campaigns.

    Stream-to-Batch Vs. Stream-to-Stream

    Reddit chose Druid as the database layer of their application in large part because of its close integration with Kafka, as Druid was designed to ingest and analyze streaming data. This sets Druid apart from virtually every other analytics database, the rest of which were built for batch ingestion.

    Batch ingestion works exactly the way it sounds—it takes chunks of stream data, dumps them into a file, processes the file, then loads it into the database. Using a batch-based system to analyze streams introduces a bottleneck to your streaming pipeline. By contrast, Druid provides a connector-free integration with the top streaming platforms and handles the latency, consistency and scale requirements for high-performance stream analytics cost-effectively.

    In addition to out-of-the-box integrations, Druid has built-in indexing services that provide event-by-event ingestion. This means streaming data is ingested into memory and made immediately available for use. No waiting for events to be processed through batch processes before they can be queried. This capability, coupled with exactly-once semantics, guarantees data is always fresh and consistent. If there’s a failure during stream ingestion, Druid will automatically continue to ingest every event exactly once (including events that enter the stream during the outage) to prevent any duplicates or data loss.

    However, perhaps the most powerful reason to use Druid for streaming data analytics is its near-infinite scalability. For even the largest, most complex ingestion jobs, Druid can easily scale into the tens of millions of events per second. With variable ingestion patterns, Druid avoids resource lag (as well as overprovisioning) by enabling dynamic scaling. It combines the query and ingestion performance of shared-nothing architecture with a cloud data warehouse’s flexibility and non-stop reliability. This means you can add computing power and scale out without downtime or rebalancing, which is handled automatically.

    In essence, while using a cloud data warehouse to serve both batch-oriented and real-time use cases might sound efficient, doing so defeats the purpose of a streaming pipeline.

    “The weakest link in the chain sort of dominates everything,” Databricks CEO Ali Ghodsi told the audience at Current. “If you have one step of your pipeline that is batch-processed and really slow, then it doesn’t matter how fast you are in the rest of the pipeline.”

    That’s why when Confluent’s data engineering team needed to build a real-time observability platform, they chose to use Kafka and Druid together. Today, the platform ingests over five million events per second from Kafka and handles hundreds of concurrent queries from a large team of analysts.

    “We do a lot of things today that we didn’t imagine doing three or four years ago,” Matt Armstrong, who leads the observability and data platform team at Confluent, said in a Druid Summit 2022 interview. “From a [product] perspective, you could think of Druid as being able to power a great product [Confluent Cloud] with a great support experience—but perhaps more importantly—an unconstrained future.”

    It’s Time to Think Big

    We’re already moving to this entirely new way of thinking about moving, analyzing, and sharing data. Streaming technology has opened the door for a plethora of new use cases and products to come to fruition. We believe organizations that view streaming data as a force multiplier and implement a streaming-native approach to analytics will gain a competitive advantage in 2023 and beyond.

    Druid is purpose-built for these new streaming use cases, but it’s also built for a new type of application—one that draws characteristics from both the analytics and transactional database worlds. Industry-leading teams have built mission-critical applications that enable operational visibility, customer-facing analytics, drill-down exploration and real-time decision-making using streaming data and Druid. This trend is now starting to take hold and the next wave of analytics applications is already being built.

  • How Log4j Becomes a Serious DevOps Problem

    How Log4j Becomes a Serious DevOps Problem

    The recent discovery of the Apache Log4j vulnerability has wide-ranging implications for anyone who develops software, especially for those in the DevOps realm. What’s most troubling about the vulnerability (CVE-2021-44228) is how prevalent the use of Log4j is. The vulnerability is reported in a vast array of applications and directly impacts numerous Apache projects, including Druid, Dubbo, Flink, Flume, Hadoop, Kafka, Solr, Spark and Struts. There are also numerous Cisco programs impacted by the vulnerability. The list of impacted applications and libraries seems to be almost infinite, extending to Elastic LogStash, GrayLog2, Neo4J, Steam, Twitter and even the VMware Tanzu family of programs. 

    Simply put, any application that uses Log4J for logging and tracing is subject to the identified family of attacks known as Log4Shell. Those attacks prove easy to exploit and can be used to take control of vulnerable servers.

    Why DevOps Practitioners Should be Concerned

    The world of DevOps is all about reiteration, where features and improvements to applications are slipstreamed into production environments. To accelerate development, the QA process is usually limited to the most recent changes.

    In other words, developers, via their pipelines, can often establish a baseline of quality and then speed up their CI/CD pipelines by testing only what has changed since the last iteration of the application. This process ensures an acceptable level of quality without introducing what are perceived to be unnecessary steps.

    However, those processes do not take into account newly discovered flaws in older APIs and libraries, meaning that those flaws could telegraph into deployed applications and leave DevOps teams completely unaware of the impact and possible disruption. Detecting flaws after an application is deployed is like putting the cart in front of the horse.

    Further complicating the situation is that many developers lack a well-defined and well-maintained software bill of materials (SBOM), which means developers may be unaware of the components and/or libraries being used in a deployed application.

    Without an SBOM, it becomes increasingly difficult to remediate an issue with an application—especially if developers are unaware of the issue to begin with.

    Where Should Responsibility Lie?

    More often than not, vulnerabilities introduced into applications become a game of finger-pointing, where developers blame library suppliers or infosec staffers, and infosec blames developers for not performing due diligence on the components they are selecting to incorporate into their applications.

    Of course, finger-pointing only adds to the angst and delays the process of remediation and does nothing to determine where the responsibility for detecting and remediating vulnerabilities lies. DevOps teams need to put these squabbles aside and establish a process that deals with unintentionally introduced vulnerabilities. What’s more, DevOps teams need to invest in DevSecOps to make sure that vulnerabilities are detected and remediated as quickly as possible.

    To avoid the potentially massive disruption that a vulnerability like Log4Shell can cause, those running DevOps should consider:

    • Creating and maintaining an SBOM 
    • Deploying automated tools that scan for discovered vulnerabilities
    • Automatically comparing the SBOM against vulnerabilities
    • Including DevSecOps processes within their CI/CD pipelines
    • Frequently testing applications for flaws
    • Coordinating with cybersecurity personnel when a flaw is discovered
    • Patching frequently

    With more and more vulnerabilities discovered on an almost daily basis, DevOps teams must be ever vigilant about what components are incorporated into their applications. What’s more, with the increased use of low-code/no-code tools and integrated development environments (IDEs), DevOps will also need to consider if those tools use other libraries that can become compromised.

     

     

  • Log4j Puts Effective IT Operations at Center Stage

    Log4j Puts Effective IT Operations at Center Stage

    News of the Apache Log4j vulnerability exploit is striking fear into the hearts of both software makers and users. Log4j is the most popular Java logging service used today, with over 400,000 GitHub downloads and has been embedded in most internet services and products from companies all over the world, including Apple, Amazon, Cloudflare, Steam, Tesla, Twitter and Baidu, just to name a few.

    Exploiting this vulnerability is simple and allows threat actors to control Java-based web servers and launch remote code execution (RCE) attacks. New variations of the original exploit are being introduced rapidly—over 60 in less than 24 hours. The attack surface grows by the minute. Good credential management and advanced authentication will not help as this exploit is pre-authentication, which means an attacker doesn’t need to authenticate to your web applications to attack. Your web server is vulnerable.

    The good news is your organization likely has a team of seasoned IT operations pros that can help protect against the worst of Log4j’s devastation.

    How IT Operations Can Protect Against Log4j Vulnerabilities

    Step One: Find Log4j Wherever it Exists

    You need to investigate every internet-facing application, website and system that you own or use. This includes self-hosted installations of vendor products and cloud-based services. Focus on systems that are internet-facing that contain sensitive data. Once you’ve completed assessing your hosted apps and vendor systems, move on to endpoint applications. Java-based apps like WebEx, Minecraft, Citrix and many more.

    Step Two: Patch, Patch, Patch

    Patch in the same order as above: Internet-facing applications, websites and systems. If no patch is yet available for your apps and systems, uninstall.

    The pandemic has forced many endpoints into workers’ home environments, which creates extra challenges for patching those endpoints and keeping them compliant. Operations teams can’t rely on work-from-home folks to patch their own systems, even if they’re given clear instructions. Many of them will just ignore those directions, so you need to adopt systems that enforce continuous compliance.

    Step Three: IT Operations is Essential

    Communicate with your senior leaders. They should be in the know about this one! Senior leadership needs to position their IT operations teams among their most vital employees. Don’t think this is going to go away any time soon—we’re just starting to get a glimpse of what is being tried out there in the wild. Buckle up. 

     

  • DevSecOps: Old Security Bugs Still Performing New Tricks

    DevSecOps: Old Security Bugs Still Performing New Tricks

    New Security Flaws Creep into Reliable Old Systems

    In cybersecurity, we are often like hunters. Our eyes are firmly glued to the horizon, scanning for the next breakout vulnerability (along with the right designing tools, techniques and tactics to stop it). However, this forward-looking focus can have the surprising effect of dampening our overall security awareness, blinding us to deep-seated dangers that exist all around, and which attackers are all too happy to exploit.

    I often compare modern cybersecurity to a suit of Kevlar armor. The seemingly ethereal properties of Kevlar can block high-velocity rounds and all manner of modern, powerful weaponry. It may even make a wearer feel somewhat invincible. However, a comparatively ancient bow and arrow weapon system, first crafted sometime around 1000 B.C., can often penetrate that protection. A sharp knife, probably the second oldest weapon in the world behind rocks, can slice right though Kevlar as easily as if it were shredding a cotton sweatshirt. And then there’s the little matter of Kevlar not being able to protect every single millimeter of the human body. If an attacker can find any gap to deliver a damaging blow, they will—much like small, exploitable areas in software.

    In cybersecurity, many organizations are similarly vulnerable to flaws on systems that are eight or 10 years old, which in modern computing terms just about qualifies them for a gold watch and a pension. But if you think flaws on these elderly systems are harmless, then you’ve probably got a blue screen of death or two in your future.

    A Vulnerability for a Veteran

    One of the oldest and most used JavaScript libraries is jQuery, an open source resource that helps with everything from event handling, to DOM tree traversal and manipulation, to generating animations. It’s quite a workhorse, and has been used for many years. People assume that because the library is so established at this point, that it must have been completely vetted, with any vulnerabilities removed.

    Sadly, this is not the case. By default, most applications that rely on jQuery use the internal library’s instructions for authentication. For example, with Apache servers, this means checking the .htaccess files. Few developers designing programs that use Apache probably thought to check to make sure that Apache server updates included .htaccess. After all, why would Apache remove that critical component, which has been a bedrock of security for years?

    As strange as it may seem, that is exactly what Apache did in version 2.3.9. Apparently, having to check the .htaccess configuration files every time a program needed to run was slowing things down too much. Removing it improved overall Apache performance, but also created a vulnerability that most people didn’t know about. If developers didn’t bother to check to see if their apps could still reach the .htaccess files, most requests would simply be accepted without scrutiny.

    Recently, experts discovered that flaw and noted that using it would allow unauthorized users to upload and run shells or almost any type of code on supposedly secure systems. This led to the creation of a vulnerability alert designated CVE-2018-9206 in October. But, the ease at which the flaw was discovered by a security researcher implies that professional hackers, whose sole aim is to search for vulnerabilities such as this, probably already have discovered it. After all, despite the publicity, patches and fixes brought about in the aftermath, a similar high-impact attack happened just a few weeks later, in which Bitcoin-thieving malware was unleashed on a popular NPM lib downloaded by millions every week.

    The Butler Did It

    Like jQuery, Jenkins is an open source offering, and one of the most popular of its kind. With its helpful servant-like name, it makes sense that Jenkins is used as an automation server by development teams in many industries. When Jenkins is functioning correctly, it’s an extremely helpful tool. However, newly discovered flaws, and one recently uncovered crypto mining operation that is truly massive in scale, suggests that Jenkins was also doing a lot of work for the bad guys.

    One of the most dangerous Jenkins vulnerabilities is called Java deserialization, which is designated as CVE-2017-1000353. It’s a complex attack, but one that has been around for a while. An attacker must submit two requests. The first one starts a bidirectional channel for downloading which is initially rejected by the server. However, the second request adds an upload channel which contains a payload with whatever commands the attacker wants, and utilizes the payload.jar script. Once the second request is sent, the communication is allowed on unpatched Jenkins servers.

    Even on patched servers, exploits exist. For example, when running Jenkins in a Windows environment, by default it uses the NT AUTHORITYSYSTEM account to authorize users. This is dangerous because SYSTEM is granted full permissions on Windows servers. Developers can change the authority account, but often don’t. Their logic not to do so is based on the fact that Jenkins has been around forever, so people figure that any vulnerabilities have been patched a long time ago.

    Most recently, a hacker used these aging Jenkins vulnerabilities to compromise several servers. The goal was adding a crypto miner program on every vulnerable Jenkins instance they could find. The miners took up valuable computing resources in their constant search for cryptocurrency. So far, they have found approximately 10,800 Monero crypto coins, with a value of almost $3.5 million. 

    What is Old is New Again

    In both of these examples, vulnerabilities are being exploited by opportunistic attackers on platforms that many people consider to be safe. On the defensive side, the lack of security-aware development is allowing these hackers to breathe new life into old tricks. And despite a new round of success using aging vulnerabilities, many organizations have no plan in place to stop this vicious cycle.

    Just because something is old does not mean that it’s harmless. And just because common libraries and resources have been around for years doesn’t mean that they are completely secure (for example, the No. 9 entry in the current OWASP top 10 is dedicated to dealing with Using Components With Known Vulnerabilities). Only through diligence and constant security training can we protect ourselves not only from dangerous threats creeping over the horizon, but also those which have already settled insidiously into our own backyards.

    — Pieter Danhieux

  • DataStax Announces DataStax Distribution of Apache Cassandra

    Continues Commitment and Engagement with Open Source Community with Industry-Leading Code Contributions, Coaching and Conference

     SANTA CLARA, Calif.–(BUSINESS WIRE)–DataStax, the leading provider of the always-on, Active Everywhere distributed hybrid cloud database built on Apache Cassandra™, today announced DataStax Distribution of Apache Cassandra (DDAC). DDAC is a new subscription offering designed for running a certified version of open source Apache Cassandra for development and production use cases that do not have the higher enterprise demands better served by DataStax Enterprise.

    The DDAC release reinforces DataStax’s engagement with the open source community. DataStax enables and supports data professionals who want to build applications on Apache Cassandra. DataStax does this by hosting frequent community events nationwide and by also hosting the premier Apache conference—DataStax Accelerate—where Cassandra practitioners, experts, and developers come to learn and share ideas.

    DataStax is now making it easier for professionals and enterprises to obtain and use the production-certified version of Apache Cassandra from DataStax.

    DDAC consists of a production-certified release of Apache Cassandra with drivers, bulk load utilities, and professional support. It combines the benefits of open source software with professional software maintenance and the expert support needed by customers who wish to deploy open source software in less critical production or development environments.

    Deploying DDAC enables enterprises to:

    • Apply premium features and performance with no lock-in while standardizing on open source software (OSS).
    • Scale environments and manage costs effectively to compete in a dynamic marketplace.
    • Streamline operations and support by removing a patchwork quilt of in-house, regional integrators or partners and third party vendors providing Apache Cassandra support and services.
    • Access the best-in-class Apache Cassandra software, support and services from the experts.

    “Many organizations are interested in using open source software when there are concerns about cost, vendor lock-in and availability of developer skills, but most organizations also want to use products that are backed by commercial support,” said Doug Henschen, Vice President and Principal Analyst, Constellation Research. “DataStax Distribution of Apache Cassandra addresses both concerns, building on open source software while leveraging the company’s experience and expertise supporting high-scale Apache Cassandra-based database deployments.”

    “We’ve put DDAC through rigorous testing to offer enterprises a production-ready release of Apache Cassandra with maintenance management and support from DataStax and our team of Apache Cassandra experts,” said Amit Chaudhry, Vice President of Product Marketing at DataStax. “We believe DataStax is the only company that has deployed the world’s best distribution of Apache Cassandra at hundreds of real-world enterprises. Our expertise, services, and support are unmatched in the industry when it comes to Cassandra.”

    DataStax Distribution of Apache Cassandra includes:

    • A certified version of Apache Cassandra that goes through the same rigorous quality assurance (QA) and certification testing process as DataStax Enterprise.
    • Business day, 8×5 support or 24×7 worldwide support.
    • DataStax Bulk Loader Utility for making data loads and migrations easy.

    “When enterprises deploy just open source Cassandra, we find that over time they run into security, support and compliance issues. DDAC solves this. Enterprises now have an option to deploy a production-certified release of Apache Cassandra with DDAC,” concluded Chaudhry.

    DDAC is available for download now at:
    https://academy.datastax.com/quick-downloads

    To learn more about DDAC, please join DataStax at DataStax Accelerate, The World’s Premier Apache Cassandra Conference, on May 21-23, 2019 in the Washington D.C. Area. To register, please visit https://www.datastax.com/events/datastax-accelerate.

     

  • Google Looks to Simplify Building of Workflows

    Google Looks to Simplify Building of Workflows

    Google this week launched the beta of Cloud Composer, a managed Apache Airflow service for building workflows that promises to simplify management of integrated DevOps processes. Apache Airflow is an incubation project being developed under the auspices of the Apache Software Foundation that enables IT teams to programmatically author, schedule and monitor workflows using directed acyclic graphs (DAGs) that visualize pipelines running in production, monitor their progress and can be used to troubleshoot issues.

    James Malone, a product manager for Google Cloud, said Cloud Composer leverages Apache Airflow to not only make it simpler to manage workflows, but also port them between various public and private clouds, including on-premises IT environments.

    Cloud Composer also includes the Google Developer Console and Cloud software development kit (SDK), a cloud identity-aware proxy, Stackdriver logging and monitoring, identity access management (IAM) and Python support, along with simplified DAG management and streamlined Airflow runtime.

    Google plans to add support for autoscaling and availability in additional Google Cloud regions over time, Malone said, as well as integrate Google Composer with Kubernetes to extend the reach of Apache Airflow workflows to containers.

    Pricing for Cloud Composer is consumption-based as measured by vCPU/hour, GB/month and GB transferred/month.

    Malone said Cloud Composer will play a critical role in enabling DevOps teams to construct pipelines spanning hybrid cloud computing environments. Today IT administrators regularly make use of text editors or scripts they write to automate a process. But it’s difficult for those scripts to be shared between multiple IT administrators. Cloud Composer and Apache Airflow make it easier to share workflows that automate processes across DevOps teams. Just as importantly, those workflows will be accessible to IT administrators who don’t have the programming skills required to develop a script.

    Those workflows will also play a critical role in unifying DevOps and DataOps processes around a common set of continuous integration/continuous deployment (CI/CD) processes, Malone added.

    Most organizations today are managing multiple clouds in isolation from one another. But as organizations look to build applications faster, they want to be able to deploy those applications on the most optimal platform available without having to develop a unique set of pipelines for each cloud environment. Having an application locked into a specific cloud platform not only is economically unacceptable, it’s also fundamentally inefficient from an application development and deployment perspective.

    Apache Airflow is not the only tool that can be employed to construct pipelines. But Malone noted that because it is an open source Apache project, organizations can be assured that workflows they develop can run anywhere now and into the future. Google chose Apache Airflow to build Cloud Composer largely because an active development community is emerging around this open source project, he said.

    The way IT platforms are managed is transforming rapidly. The days when servers and applications where managed in isolation are over. IT administrators are being asked to manage fleets of server and storage systems running more applications than ever. The tools required to achieve that goal are thankfully now becoming a lot more accessible to everyone in IT.

    — Mike Vizard

  • The Top 10 Reasons to Use Cassandra

    The Top 10 Reasons to Use Cassandra

    The Apache Cassandra database is the right choice of database if you are looking for scalability and high availability without compromising performance for your mission-critical applications. Additionally, Cassandra’s support for replicating across multiple data centers is best-in-class, providing lower latency for users and the peace of mind of knowing that you can survive regional outages. To that end, we’ve compiled a list of the Top 10 reasons why enterprises onboarding and deploying mission-critical applications should use Cassandra (Apache Cassandra, DataStax Enterprise).

    1) Requirement for fast writes: Easily deals with data velocity, data variety and data complexity issues

    Many of the challenges associated with next-generation cloud applications center around data volume and data velocity. Is Cassandra able to handle the speed of data coming into the system? The answer is “yes” based on the amount of data and cluster size. Not only does Cassandra come with this ability out of the box, but there are systems of data pipeline architectures being built around ingestion speed. And to top it off, it scales linearly, making it easy to determine the right amount of capacity based on data flow.

    But, there are also two often overlooked components it provides: 1) data variety and 2) data complexity. Data variety is an alternate way of saying that data coming into one database can come in different forms. An example of this would be sensor input from a heart monitor and sensor input from an IV both for the same patient. The second component, data complexity, extends the previous example. The heart monitor might report 100 metrics twice per second under normal operating circumstances and up to 125 metrics once per second while the wearer is sleeping. This means the write patterns, locations, and frequencies can vary. Cassandra handles these situations gracefully.

    2) Can handle massive data sets

    If there were any questions about whether Cassandra is capable of handling large data sets, there is no need to look any further than the companies using it. They operate at massive scale—Netflix, Hulu, Instagram, eBay, Apple, and Spotify all have Cassandra working in interesting ways as part of their offerings.

    The other way you know Cassandra is up to the challenge is in use case examples. Many organizations use it for applications where data grows in an unbounded way very quickly. These include Twitter clones, a web log analytics data warehouse and telemetry or sensor data.

    3) Homogeneous environment

    Unlike some of the legacy distributed systems, Cassandra does not require outside support for synchronization. All of the required components for basic operation are built in. Since Cassandra also operates in a peer-to-peer fashion, this means that there is no master-slave or sharding setup and that all nodes in the ring are equal. Additionally, there is only one machine type that an administrator needs to worry about.

    4) Highly fault-tolerant

    Cassandra employs many mechanisms for fault tolerance. Since it is masterless, there is no single point of failure. There is also the potential for zero downtime rolling upgrades. This is because Cassandra can support the temporary loss of multiple nodes (depending on cluster size) with negligible impact to the overall performance of the cluster.

    The safety net Cassandra offers extends outside of your data center as well. Cassandra allows you to replicate your data to other data centers and keep multiple copies in multiple locations. This helps satisfy many regulatory requirements in addition to being a part of an strong disaster recovery and business continuity plan.

    5) Proven success across enterprise applications and in many use cases already

    There are already many examples where Cassandra is being used effectively. Banks and other financial institutions are storing large quantities of financial data in Cassandra. Analytics companies are using Cassandra to store web analytics data. Medical companies are using Cassandra to store sensor data and other time series inputs. There are also many companies making use of Cassandra for storing internet of things (IoT) data.

    6) Ease of administration

    Cassandra is a straightforward system to administer. With Cassandra being a masterless system, all nodes in the ring are the same—a homogenous system. It’s fault-tolerant and can support the temporary loss of nodes with minimal impact to production performance. This means that nodes are easy to replace and the requirement to replace downed nodes immediately isn’t as strict.

    7) Custom tuning

    There are a lot of knobs and levers that can be turned to get Cassandra to perform optimally for your workload and environment. You can set it up to operate in a way that is consistent with your workload. For example, if you write lots of log data and read infrequently, then there are configuration tweaks to be made for write-heavy systems. If you write heavily to one data center and then do all your reading from another data center, then you can adjust the settings on a data center-by-data center basis.  This idea of tuning isn’t just available at the application level; you can also tune the JVM and Java settings, including GC and logging levels. Changes can even be made by the drivers at connection time to aid in the performance of your system.

    8) Easy to integrate core applications

    A lot of work has been done on data manipulation and parsing systems to integrate with Cassandra. For instance, the full text search engine Apache Solr has been packaged to work with Cassandra to provide full featured search capabilities to an existing Cassandra database. Apache Spark, a big data analytics engine, also has been plugged in to work on an existing database. There are entire suites of tools that can be integrated or bolted on to increase its capabilities. These include Apache Mahout, Kafka and Zipkin, to name a few. This is important because the more tools you have available to you, the more powerful your data becomes. You also have the ability to gain more insight about your data without having to build and maintain the application systems that were previously required.

    9) Excellent Monitoring Options

    Included in the system of tools referenced in No. 8 are monitoring packages. If you are a user of automated monitoring platforms such as Datadog or Netuitive, you’ll find examples of prepackaged agents to monitor the important parts of Cassandra. You can then tack on your own additions of other metrics that are important to you. This is made possible by Cassandra taking advantage of Java MBeans and exposing them to the client. You can use these to get at much of the internal information Cassandra uses to make its own decisions and decide on its own health. Datastax also offers its own monitoring and control application called Opscenter.

    10) Amazing community

    One of the best things about any piece of software is having a great community of developers and experts available to you for help or guidance. There is a huge yearly database summit put on by Datastax, the primary backer and largest contributor of code to Cassandra. It also sponsors community events all over the world so you can meet and interact with other developers around you.

    For open-source software to be successful, there needs to be an ecosystem that develops around it. In the case of Cassandra, there are consultancies, monitoring and troubleshooting systems, plugins, instrumentation systems and backup systems. That is all a set a competencies that your organization no longer needs to own and can use what the greater community has already created. There are even PaaS companies that will completely take over the management for you, leaving you just the development of your application to focus on.

    Given the sizable number of organizations and people that are a part of the ecosystem and the Cassandra community as a whole, there is no shortage of articles, documentation and people willing to help. A welcoming and helpful community isn’t always a given, but in the case of Cassandra, it’s alive and thriving. This is important because software is always about people.  The more of them that you can interact with that have shared your experiences, the better. It also will be easier to find solutions to your problems having a network of people who might have faced them before.

    There are many reasons why Cassandra could be the right tool for your application. Knowing your systems requirements, workloads and future will help you make the right choice. As you can see, if you do choose Cassandra, you are bound to be in good company.

    — Michael Colby

  • Databricks Offers APIs to Enable Agile Application Development With Apache Spark for the Enterprise

    Fusion of API and Visualization Meets Application Deployment and Data Science Needs

    SAN FRANCISCO, CA–(Marketwired – Mar 30, 2016) – Databricks, the company behind Apache Spark, today launched a new set of APIs that will enable enterprises to automate their Spark infrastructure to accelerate the deployment of production data-driven applications.

    Today’s enterprise data platform must satisfy the disparate needs of multiple constituents as they work together to deploy production data-driven applications. While developers and DevOps teams operate in command line and APIs to automate infrastructure, data scientists demand a visual environment to create algorithms. Without a unified platform, enterprises suffer from a sprawl of tools, leading to long application development cycles and high IT infrastructure complexity. With the addition of secure APIs to an integrated workspace, Databricks natively supports this full spectrum of enterprise needs in one platform — a capability not yet available anywhere else.

    Databricks users will now have the ability to implement Agile Development best practices, such as continuous integration, in tandem with performing real-time data analysis and visualization. Added benefits to enterprises include:

    • Faster release cycles of Spark applications;
    • Higher reliability in Spark infrastructure deployments;
    • Improved oversight and management of Apache Spark infrastructure expense in the cloud.

    “Agile data-driven application development requires data scientists to work in perfect sync with DevOps, but these teams lack a unified platform to connect their separate workflows,” said Ali Ghodsi, CEO of Databricks. “The addition of Databricks APIs will enable these constituents to work seamlessly in one platform, allowing them to act on hidden insights from their data and create unique competitive advantages for the business.”

    “The APIs provided by Databricks combined with its integrated workspace and security framework has allowed us to release our applications twice as often as before,” said Doug Stein, chief technology officer at Metacog. “Instead of waiting for multiple weeks to incorporate improvements into our product, we are now able to do so every week.”

    The new APIs are available today to all Databricks customers. To sign up for Databricks, visit www.databricks.com or contactsales@databricks.com.

    About Databricks:
    Databricks’ vision is to make big data simple for the enterprise. The company was founded by the team that created and continues to drive Apache Spark, a powerful open source data processing engine built for sophisticated analytics, ease of use, and speed. Databricks is the largest contributor to the open source Spark project providing 10x more code than any other company. The company has also trained over 20,000 users on Apache Spark, and has the largest number of customers deploying Spark to date. Databricks is venture-backed by Andreessen Horowitz and NEA. For more information, contact info@databricks.com.

    CONTACT INFORMATION