devopsy.kinsta.cloud

Tag: big data

  • Avoid the DevOps Tool Trap

    Avoid the DevOps Tool Trap

    Software development, at its most complex and dynamic is a full-court press of effort, knowledge, patience, and hard work. Anything that can reliably and cost-effectively reduce the huge number of tasks, redundancies, reports, and delays associated with the application release process is something to be embraced.

    In particular, many organizations today are turning to the agile development methodology, DevOps, in order to rapidly produce software and improve operations performance. According to research by Gartner, by 2016 DevOps will become a mainstream strategy that will be employed by 25% of Global 2000 organizations. Moreover, by 2019, Gartner predicts 50% of enterprises will implement automated configuration and release management as a result of DevOps initiatives.

    What many in the development community don’t realize is that solutions exist, using current technologies, to automate many of the routine—and even not so routine—elements of the application release process needed for DevOps. Instead of adding multiple application-specific automation tools, resulting in a confusing and expensive array of various silos of automation, organizations can take advantage of their existing automation suite to automate their application release processes. What we’re talking about is not simply scheduling workflows, but the actual execution of tasks, from administrative duties to build deployments, release management, and more.

    Using modern IT workload automation solutions, it’s possible to offload the manual execution of a number of processes to improve overall DevOps agility. Many actions necessary to go from code to production, not to mention maintenance and cleanup tasks, can be taken off the backs of developers and operations, thus freeing them for higher level projects.

    With a modern workload automation solution, organizations can automate routine and time-consuming build kickoffs and installations. For example, using pre-built job steps for Microsoft Team Foundation Server (TFS), organizations can create an automated workflow that deploys builds onto machines, installs the kits onto production environments, verifies installation files are correct, and then sends an alert to the proper users that the build is ready. After QA performs testing of the individual stories, the automation solution can then close the stories and merge them in time for a daily or weekly review meeting. Teams can also integrate TFS with a QA/test system, in order to automatically create a new work item with TFS every time a bug is logged.

    On the Operations side, one of the biggest challenges in application release management is making sure machines are correctly architected. Excessive instances and regression tests can cause machine slowdowns that create havoc with development schedules. Again, workload automation comes to the rescue via workflows that purge databases, increase CPU, or anything else necessary to ensure a properly-provisioned testing environment.

    Adding to this, many popular tools used in software development lack even basic scheduling capabilities. If you wish to create a work item or trigger a build, for example, you probably have to do so using a Web service or the TFS Task Scheduler Service. Even then, all you likely have at your disposal is date/time scheduling. This is where advanced workload automation really shines, since it can trigger tasks according to a pre-defined event, or even a complex combination of events. As a result, DevOps teams can avoid unnecessary project delays or lag time due that would otherwise occur with manual handoffs between teams or individuals.

    While workload automation has been around for many years, today’s IT departments are increasingly using workload automation as a foundation for their wider IT systems. As new methodologies and strategies like DevOps and Big Data emerge, automation provides a foundational piece upon which to automate essential processes needed to support these new initiatives. In the area of DevOps, workload automation is helping both developers and operations build and automate release processes more effectively, with capabilities that can make the difference between headaches and headway. Don’t fall into the DevOps tool trap.

    About the Author/ Jim Manias

    Jim ManiasJim Manias is Vice President at Advanced Systems Concepts, Inc. and is responsible for the overall market strategy and planning for a range of products including ActiveBatch® Workload Automation and Job Scheduling.  Jim has been with Advanced Systems Concepts since 1991 and has held multiple senior management positions in the enterprise software and hardware market.  Jim can be reached at JManias@advsyscon.com

  • Jut Operations Data Hub for DevOps

    Jut Operations Data Hub for DevOps

    “You have to be able to ask questions of your data in real-time and then respond to the answers,” says Apurva Dave, VP of Marketing, Jut. Jut leverages live streaming, batch analytics, and visualization to enable enterprises to ask important queries of their operational data using the Jut operations data hub, which Jut created with DevOps goals (such as Continuous Deployment with Quality) in mind.

    Jut supports real-time and historical data queries across all log, event, and metrics data. “Our platform is built on dataflow, dealing with large scale streaming data,” says Dave. Jut is an option for organizations that don’t want to go out and build their own operations data hub software.

    Core Data Types

    Jut intends to join data such as user activity (on websites and software), event data (triggered operationally such as support ticket filings, customer transactions, and software upgrades or migrations), and unstructured data (application and system logs) together for analysis, says Dave. Jut works with structured data such as metrics data (such as application level data like the user numbers and key action numbers or system level data like CPU utilization, memory utilization, and network I/O data), explains Dave.

    Jut attempts to analyze the whole of your data to answer questions you may have like: “What resulted when I deployed a new version of my application into production?” Jut might answer based on an analysis correlating system resource utilization (“Was the app too resource intensive this time? More so than we expected?”).

    Or you may ask Jut this question: “How did my code respond to user activity / user demand? How did it perform?” Jut could ask this question based on its assimilation of both unstructured and structured data. “It’s common for enterprises to A/B test two separate versions of a site or software feature. Using Jut, you can ingest those test results and analyze them in real-time to determine what version produces the desired effect,” says Dave.

    Big Data Analytics & a Huddle about Juttle

    “We have users ingesting tens of billions of data points per month. That number is growing fast,” says Dave. Jut targets big data management for easier analysis using data infrastructure techniques that simplify / streamline, integrate, and manage big data backend infrastructure. “We also use our own dataflow language to quickly iterate and visualize lots of data,” says Dave. All this is designed to ease DevOps teams’ work in accessing and assessing operational data to discern how the software and the business that it serves are performing.

    Jut designed its Juttle dataflow programming language for data analysis and data visualization. And Juttle is dedicated to those tasks alone. “While you could try to cobble together the right packages and libraries in a general language like Python or Java to do these analytics, it would be relatively messy, time-consuming, and error prone,” says Dave.

    Juttle is an extensible language enabling analytics and visualization. “Jut’s data scientist built an anomaly detection algorithm entirely in Juttle and then published it on GitHub for anyone to use inside Jut,” says Dave.

    Juttle uses a structured approach to querying your data. “The approach looks like a series of piped UNIX commands, so anyone familiar with scripting languages such as JavaScript or query languages such as SQL should feel at home with Juttle,” says Dave. Jut responds to questions in seconds to tens of seconds (for larger data sets) based on iterative, interactive analysis.

    Investigate, Question, Consider (Try?)

    Outsourcing to Jut puts the enterprise in a position to focus on existing coding duties while proving whether Jut is the better operational analytics move for analytics performance and cost-savings (Jut offers a free trial). But as with the analytics themselves, ask a lot of questions of Jut. Do your homework before committing even to trial software (Jut has a demo and an online “playground” as well).

    Tip: Jut made sparkling comments from its Beta-development-phase customers such as NPM and Mylio available. Rather than include those for your consideration, I suggest you reach out through your connections to these and other enterprises that are experimenting with this data hub software.

  • Where Oh Where has my Hadoop gone?

    Where Oh Where has my Hadoop gone?

    Where is Hadoop? Just a few years ago, it was everywhere. Internet heavy hitters such as Facebook and Yahoo boasted about the size of their Hadoop clusters and applications. It was easy to imagine the day when Hadoop would be the one indispensable tool for anyone who dealt in large volumes of data.

    (more…)

  • Managing many hosts is more than a Big Data problem

    Managing many hosts is more than a Big Data problem

    Back in my days at StillSecure a major problem we had with our vulnerability management platform was managing vulnerabilities over a large number of hosts. How can you effectively manage so many hosts, visualize them across the entire deployment? It gave our GUI designers and developers fits.

    In today’s world of virtual instances, containers and apps, managing a large number of hosts is even a bigger challenge. It is a daunting task that even the largest companies struggle with. In fact in some ways the bigger the company, the greater the amount of hosts and the harder it is to manage. I wanted to highlight today how one company, Datadog has tackled this problem with their Host Maps feature.

    The trick to managing large numbers of hosts is to balance a global view with an individual host view. It is harder than it sounds though. It is almost like joining Einstein’s Theory of Relativity with Quantum Mechanics. How can we use one view to see things on both the galactic scale and the molecular level at the same time. Datadog has developed one of the best ways to do this with their instant grouping and drill down capability.

    Instant grouping allows you to virtually (no pun intended) define groups on the fly by a large number of tags and filters. So for instance you can use standard AWS tags. You can also group hosts by tags and they will be visualized this way.

    This was where we ran into the wall on our vulnerability manager. We were never able to create groups on the fly. You could define groups by certain filters but drilling down from there was out of reach. With Host Map feature you can create a group of hosts that have a certain feature or attribute in common and then drill down into that group with the click of a mouse. The YouTube video below shows you much better than I have described what I am talking about:

    Datadog’s designers have also made great use of color selection and shaping to help make visualization much easier for the end user. Pretty nifty use of simple technology.

    Not to turn this into a love fest for Datadog Host Map, but I am not aware of others who have instituted this kind of on the fly grouping and drill down capability to address this Big Data type of problem. If you know of one, I would love to hear about it.

    From the point of view of looking at how the state-of-the-art has evolved from when we were trying to tackle this issue 10 years ago, it is impressive to see how far GUI has progressed. I think it is a clear case of “mother is the necessity of invention”. It would be near impossible to manage the kind of host deployments we see now without this type of functionality.

    I did have a chance to speak to a few Datadog customers when I reached out to Datadog about Host Maps. It was unanimous that they feature was as valuable to them as it looked to me. Now of course one could say that Datadog wouldn’t refer me to customers who didn’t like the product. But my impression was that this feature was indeed a big help to them.

    As we continue learn to better manage data flows and manage ever growing infrastructure, I think we will continue to see the evolution of management GUIs like what Datadog has done here.

    What about you? What GUI functionality have you run across that you think is worthy of praise? I am sure others would like to know.

  • New Relic Announces Groundbreaking Real-Time Analytics Platform

    Cloud-based New Relic Insights collects, stores and processes trillions of events from web and mobile software to allow customers to easily and quickly answer the most important business questions

    SAN FRANCISCO AND NEW YORK – March 19, 2014 – Software analytics company New Relic today announced the arrival of Insights, a real-time analytics platform that collects, stores and presents valuable data directly from modern software, and transforms the data into insights about customers, applications and the business. Delivered as a cloud-based software-as-a-service (SaaS) offering and using New Relic’s lightning-fast and custom-built Big Data database platform, New Relic Insights empowers application developers and business users alike to make ad hoc and iterative queries across trillions of events and metrics and get answers in seconds.

    The Big Data Challenge

    Traditionally, there hasn’t been a fast, easy or cost-effective way to collect, store and interact with real-time data to gain business or IT insights. With today’s fast-paced economy, businesses can no longer afford to wait hours or days for a static report that may not provide the data required for data-driven, software-powered businesses. Frequently, the data is out of date and ad-hoc queries take hours to complete, limiting what and how many questions decision makers can ask – making data-driven, iterative decision making impossible.

    New Relic Insights to the Rescue

    • Collect data about your software, customers and business directly from your software using New Relic’s intelligent agents
    • Store volumes of data in New Relic’s super-cluster and take advantage of a custom-built database to query billions of metrics in seconds
    • Present visualizations and dashboards immediately with the New Relic Query Language (NRQL), a simple SQL-like language that automatically generates data visualizations, so even novice users get answers fast

    New Momentum for the Category of Software Analytics

    New Relic Insights is only one offering from New Relic within the much broader Software Analytics category. Every modern business is software-driven and interacts with its customers digitally whether via mobile or the web. As a result, software holds answers about your customers’ behaviors and actions, which reflect on the state of your business and your relationship with customers. There are five components that make up Software Analytics:

    • Real-time Analytics Platforms – Collect trillions of events from applications and allow users to query the data easily to answer tough business questions
    • Application Performance Management – Developers and IT operations groups monitor and troubleshoot the performance of their web and mobile software
    • Operational Intelligence – IT operations groups analyze and identify trends in operational systems
    • Infrastructure Management –IT Operations groups monitor and troubleshoot the performance of IT infrastructure
    • Business Optimization and Analytics Apps – Business user groups optimize their job functions

    Ways to Gain From New Relic Insights

    • Product Management – Make a query about the adoption of new product features that were recently launched, determine which customers are actually using the features to enable actions and next-steps.
    • Marketing – Find out if the new marketing campaign is a homerun or a strikeout in real-time. If it’s a homerun – spend more. If it’s a strikeout, kill the campaign before the budget is blown.
    • Sales – Track customers’ experiences and product usage during a free trial. Immediate insight into key stakeholders and issues that they might have with the product.
    • Customer Loyalty and Support – Manage customer engagement and happiness through each click of their mouse. Understand why a customer makes a call to support before they make the call.
    • Application Developers and IT Operations – Track customer interactions with a site over the previous 10 minutes to find the root cause of a support issue that cannot be reproduced.

    The New Relic Insights Beta Program Highlights

    • This new Real-Time Analytics Platform (previously code-named Project Rubicon) is now in public beta. More than 275 customers have participated in the private beta to-date. All paying New Relic customers get immediate access to the new product for free for a limited time.
    • Each account gets 1 billion metrics stored for up to seven days for free during the beta period. More than 1 trillion events have already been stored in the new product over the last month.

    Supporting Quotes

    “New Relic Insights is like a data Swiss army knife, a tool you can use for a multitude of analytics needs and opportunities. For any modern publisher, analytics is omnipresent with millions of data points – think March Madness and World Cup 2014 just for starters. Bleacher Report sees New Relic Insights adding significant engagement opportunities with our sports fans around the globe.” – Sam Parnell, CTO, Bleacher Report

    “The New Relic API and the software analytics now available through New Relic Insights are allowing us to analyze and iterate feature changes to our website and donor experience in sophisticated A/B testing that wouldn’t be available any other way with any other tool. Software Analytics is helping us save the lives of children in countries like Zambia and Uganda, all made possible by New Relic.” – Joel Worrall, CTO, CURE

    “New Relic Insights is a spotlight that rapidly uncovers major insights about our properties — and is an invaluable tool for both our tech and marketing teams. The custom dashboards and ability to add custom attributes allow us rapidly to ask better questions and quickly learn from those, so we can dig further and make better business decisions — both for the platforms as well as the user experiences they support. The monitor-test-get results cycle has been reduced from two weeks to daily (or even intra-day) — the speed with which you can build a query and get a response has created a near real-time intelligence capability. Finally, the instrument-launch-monitor cycle now fits in our agile development and marketing processes.” – Paul Ryan, CTO, Yellow Pages Group, Canada

    “I am so excited that New Relic Insights is now a reality, a completely new type of product that signals New Relic’s second act. It builds upon our ability to collect, store and make sense of billions of metrics and offers the ability to instantly query live production software, iterate on the fly and act upon the results. I truly think New Insights will transform the way businesses use data from their software to make better decisions.” – Lew Cirne, founder and CEO, New Relic

    Online Resources

    • Learn more about New Relic Insights by clicking here
    • Discover more about the Software Analytics category with the paper, “Learn Why Your Software Should Rely on Software Analytics,” by The Big Data Group, an independent market research firm
    • Enterprise customers can access the independent market research firm Enterprise Management Associates paper, “New Relic Unveils New Relic Insights: Software Analytics for Business Insights” here
    • See how New Relic Insights helps developers in this infographic here
    • Read CEO and Founder Lew Cirne’s blog post here and get lots more details about how New Relic Insights works as well as a technical perspective on New Relic Insights here

    About New Relic

    New Relic is a software analytics company that makes sense of billions of metrics about millions of applications in real time. Our comprehensive SaaS-based solution provides one powerful interface for web and native mobile applications and consolidates the performance monitoring data for any chosen technology in your environment. Our 90,000 customers use our cloud solution every day to optimize more than 200 billion metrics for 3 million applications. When your brand and customer experience depend on the performance of modern software, New Relic provides insight into your overall environment. Learn more at newrelic.com.

  • Tokutek Announces Reselling Partnership With SkySQL

    Leading MariaDB Distributor Adds TokuDB as Complementary Tool to Extend Database Performance and Scalability

    With TokuDB and the SkySQL MariaDB, database administrators and DevOps engineers can easily operate on continuously incoming data and incorporate analysis on large volumes of data.

     

    LEXINGTON, MA and LONDON, UNITED KINGDOM and HELSINKI, FINLAND–(Marketwired – Feb 12, 2014) – Tokutek, delivering database performance at scale, today announced that it has entered into a partnership with SkySQL™, a trusted provider of open source database technology, to resell its TokuDB® database storage engine. SkySQL will offer the TokuDB storage engine as a complement to its MariaDB® distribution to increase scalability and performance and to enable real-time management of transactional and analytical data.

    TokuDB enhances the SkySQL MariaDB distribution with its unique ability to handle high rates of incoming data, including updates, while maintaining query performance on large volumes of data. This occurs without requiring expensive hardware build-out and with no operational downtime. With TokuDB and the SkySQL MariaDB, database administrators and DevOps engineers can easily operate on continuously incoming data and incorporate analysis on large volumes of data. Businesses benefit from richer analysis, more up-to-date information, and access to detailed data.

    “Tokutek’s TokuDB storage engine gives the database administrators and DevOps engineers who support MariaDB applications a ready-made tool to improve their database performance and scalability,” said Tom Callway, Director Marketing, SkySQL. “The combination of high data ingestion, compression and Fractal Tree indexing in Tokutek’s TokuDB makes it a great complement to the SkySQL MariaDB for high-performance, real-time analytics.”

    Paybox, the leading provider of electronic payment solutions in Europe, is a current SkySQL MariaDB customer also using TokuDB to manage their database. According to Marc Thomas, chief technical officer at Paybox, “As we continue to grow, so do the demands on our databases. We’ve benefited from SkySQL’s deep knowledge of MariaDB and best-in-class support, and Tokutek’s unparalleled scalability and performance. The increased collaboration between SkySQL and Tokutek ensures that these two industry leading providers will become more tightly coupled on both the technology and the service level, which is a big plus for Paybox.”

    “SkySQL offers both a MariaDB distribution as well as the tools and services necessary to support it,” said John Partridge, CEO of Tokutek. “With our TokuDB storage engine available as a SkySQL download, SkySQL is giving its customers the ability to make MariaDB into a much more performant database.”

    TokuDB is the ideal choice for complex high-volume applications that require fast response time and that must simultaneously store and query large volumes of rapidly arriving data. These include social networking applications, real-time clickstream analysis, logfile analysis, ecommerce personalization, and high-speed webcrawling applications. TokuDB is fully compatible with MySQL and MariaDB.

    To learn more about the SkySQL MariaDB distribution with TokuDB v7.0:

    Download a free trial of the suite,
    Attend a Webinar overview of TokuDB v7.0,
    Read the Tokutek blog for more information on TokuDB.

    About Tokutek Inc.
    Tokutek is a performance database company that delivers Big Data capabilities to leading open source data management platforms. Its breakthrough technology lets customers build a new class of applications that handle unprecedented amounts of incoming data and scale with the data processing needs of tomorrow. Tokutek applies patented Fractal Tree™ indexing to increase MySQL performance and MongoDB performance, decrease database size, and minimize downtime. The company is headquartered in Lexington, MA, and has offices in New York, NY. For more information, visit Tokutek.com or follow us on Twitter@Tokutek.

    About SkySQL
    SkySQL is a trusted provider of open source database solutions for MySQL and MariaDB users on enterprise sites and in the cloud. SkySQL provides deployment and database management solutions to more than 400 client companies, including Canal+, ClubMed, Constant Contact, Deutsche Telekom, La Poste, Virgin Mobile, Western Digital, and XING. With more than 300 accumulated years of experience in the original MySQL solution, SkySQL has a wealth of core competencies in MySQL and counts on the support of the first MySQL creators, as well as those of MariaDB. The company has also developed database management solutions in the cloud, which are easy to use and offer improved productivity. For more information, go to www.skysql.com, and follow our twitter feed:http://www.twitter.com/skysql

    MySQL is a registered trademark of Oracle, and/or its affiliates.
    MariaDB and SkySQL are registered trademarks of SkySQL Corporation Ab. SkySQL and MariaDB are not affiliated with MySQL, or Oracle.
    Other names may be trademarks property of their respective owners