devopsy.kinsta.cloud

Tag: data analysis

  • To Accelerate Digital Transformation, Accelerate Data Transformation

    To Accelerate Digital Transformation, Accelerate Data Transformation

    Successful digital transformations are not possible without successful data transformation. After all, it’s data—and the ability to analyze that data—about an organization’s customers, employees, infrastructure, market dynamics, supply chain, business-technology systems and more that make digital transformation possible.

    However, last year Gartner said that “information as an asset” is still in its infancy. “Data and analytics are the key accelerant of an organization’s digitization and transformation efforts. Yet today, fewer than 50% of documented corporate strategies mention data and analytics as fundamental components for delivering enterprise value,” the analyst firm wrote in its report, “Why Data and Analytics Are Key to Digital Transformation.”

    Further, Gartner predicts change is coming and quickly: Gartner says that 90% of corporate strategies will unequivocally list information as a critical enterprise asset and analytics as an essential competency within two years.

    Getting there and being successful with data and data analytics won’t come without significant challenges.

    According to a survey released by IDG (sponsored by Matillion, a cloud data transformation provider for cloud data warehouses), the challenges are steep. The survey polled more than 200 IT, data science and data engineering professionals at North American organizations with at least 1,000 employees.

    What are the challenges enterprises are facing? According to the survey results, 47% said data control issues are the biggest challenge to data analytics projects. In contrast, other top challenges indicated are lack of a scalable, reliable technology platform to process large data sets (45%), having too many manual processes (38%) and challenges cleansing and preparing data (36%). Additionally, 40% cited a lack of visibility and control of data silos as a challenge when collaborating with business users.

    With that, it’s not surprising to learn that it takes on average one week to pull together and prepare data for useful analysis, and 45% of all time spent on data analytics initiatives is consumed by data preparation. About 60% said they lack the scalability and flexibility they need when preparing data for analysis.

    It was almost unanimous when it came to the number of companies that sought ways to accelerate their approaches to data analysis improvement.

    So how are enterprises accelerating their efforts to improve data analysis to better power their digital transformations? Respondents said data portability (57%), ease of onboarding (57%) and cost-effectiveness (52%) are the top features of an analytics platform that can help them move past current obstacles. The IT professionals surveyed favored user-friendly (50%) and easier connections to data sources (50%) as top features; however, data engineers, data scientists, data architects and others cited time-to-value (57%) and ability to provide self-service (51%) as essential capabilities.

    A sizeable portion of those surveyed indicated they are already on their way, adopting the technologies and processes they need to improve their ability to analyze their data for better decision-making rapidly. According to the survey results, 38% are already using cloud data warehouses. And in the future, 43% said they expect to have all of their data in the cloud, with 57% planning to use both cloud and on-premises data strategies.

    Those that master their data analytics more rapidly may enjoy an early mover advantage. “A company’s ability to compete in the emerging digital economy will require faster-paced, forward-looking decisions,” said Douglas Laney, distinguished VP analyst at Gartner. “Data and analytics leaders need to assert themselves into corporate strategic planning to ensure that data and analytics competencies are incorporated within the highest-level public-facing enterprise plans.”

  • DataOps and DevOps: A Match Made in Heaven

    DataOps and DevOps: A Match Made in Heaven

    The adoption of IT technologies has increased exponentially over the last few decades. Nowadays, the need to create quality web applications while simultaneously managing their extensive data effectively is a major concern for any organization. As such, there have been continuous efforts to come up with better approaches to software development, data analysis and data management.

    In order to meet the needs of today’s internet users, technology companies have intensified their efforts and are increasingly making use of all their available resources to find progressively efficient ways to create web applications within a shorter time-space and with superior quality, and to find more effective ways to integrate data-driven decision-making.

    In order to accomplish these feats when it comes to application development as well as data pipelining management, most of the larger organizations are now following a set of practices, DevOps and DataOps, respectively. Within this environment, developers must create applications that can function across various hosted providers, each with its own specific administrators and requirements. 

    This can only be accomplished by looking past the traditional storage capabilities. Instead, developers and organizations they work for are creating an underlying data fabric that facilitates what they’re trying to do, and any cloud provider they want to do it with.

    An Introduction to DevOps and DataOps

    DevOps and DataOps are a set of practices that relates to software development and data management, respectively. 

    These practices can be applied to a single project or can be followed at an organizational level. DevOps emphasizes the automation of the processes of the different teams (software development team, quality assurance team and IT operational team) so that they can easily complete their relative tasks (i.e., design, code, create, check, deliver, deploy, operate and monitor).

    DataOps, on the other hand, can be described as DevOps that includes data analysis, which empowers businesses to create processes to meet the needs of users throughout any data usage life cycle. 

    DevOps emphasizes the cooperation of the software development team (including testers) with system administrators (operational teams). In a similar way, DataOps relies on the cooperation of teams of data engineers, data scientists and other data practitioners who manage data-related processes (i.e., data storage, data processing, data decision-making, data archiving or data delivery).

    The concept underscores both communication and collaboration between the different teams within the same organization, ultimately removing all obstacles for seamless integration between the practices and the end product, including cross-device compatibility. 

    The combined team can make the project more visible and deliver faster results. It also increases the morale of the company by showing that, when it comes to project management, people will still be above processes and resources.

    Benefits of DevOps

    The benefits of DevOps include:

    • Transparency — By regular communication and coordination with the development team, the QA team and the IT operations team will be updated with the latest project details and will be more aware of the processes of other departments.
    • Quicker and high-quality deployment cycles — On any great DevOps platform, the pace of deployment will be improved by applying DevOps practices. The scope of the teams will be limited (i.e., for a small task/module) for a deployment period, as such their efficiencies will increase.
    • Faster bug fixes — Because tasks are smaller and continually distributed on the appropriate servers, the development team can receive input very easily from the QA team, allowing the development team to provide much faster resolution of any non-working components. Application stability would be a by-product of this method, because the cause behind any bug can be established very easily.
    • Flexibility — Because agile methodology insists on managing the raised change request, teams aligned with DevOps will be able to manage any unplanned work more effectively.

    Benefits of DataOps

    Now, let’s take a look at the benefits of DataOps:

    • Faster data processing — When different teams cooperate and interact on a daily basis and work on micro databases rather than on all at once, they not only avoid data overwhelm, they also accomplish much faster processing of data at all the different stages. 
    • Quicker problem solution — Similarly to DevOps, businesses can benefit from DataOps in multiple ways. DataOps data is processed in batches as this helps teams to recognize issues related to a batch of data faster as the emphasis is limited.
    • Better data security — We can protect our data in the cloud by using a range of safety features such as two-factor authentication and virtual private networks, the best of which provide a greater level of security than is available merely through your ISP. As data security professionals are in constant communication with data integrators and data analysts in a DataOps scenario, data security at each process level is made easier.
    • Improved resource output — When the team conforms to the basic principles of DataOps, seamless communication and transparency between the different teams will help the performance of each team member and lead to better resource output.

    The Lessons We Can Learn from eharmony

    The eharmony membership pool consists of singles in the U.S. as well as more than 200 countries around the world, making up about 5 million visitors per month.

    The company is continuously bringing out new features such as their most recent one-on-one video communication feature aimed at helping their members to maintain meaningful connections in the current era of social distancing. But the company has come a long way since its debut in 2000, learning very valuable lessons along the way.

    Overcoming Several Challenges

    eharmony is leading the online dating industry through innovation by continuously improving its two key applications — Singles and Matching. But they’ve had to make significant changes on their journey to remain at the top. Their challenges included:

    • The IT team needed a considerable amount of time to refresh the database, requiring the involvement of DBAs, storage specialists and network administrators.
    • The labor-intensive process of creating, scanning and storing database snapshots made it difficult to create test datasets for new releases. As a result, QA often carried out tests using stale data. 
    • Database improvements triggered development freezes, which hindered the team’s ability to deliver new features and functionality.
    • Their data volumes were increasing constantly, with several Oracle databases requiring between 20 to 70 TB of storage.

    Combining DevOps and DataOps

    eharmony has also implemented a self-service data management solution called Delphix, which enables the individual teams to develop and update environments as needed.

    Today, their refreshes can be completed within an hour, providing accurate and relevant information to development teams on demand. 

    Other solutions included:

    • They have saved over five thousand person-hours in just three years by automating database provisioning, refresh and rollback tasks. In the process, this has also saved the company over $3 million in operational costs.
    • Through compression and virtualization, eharmony has reduced their database storage footprint to a fraction of the original size, reducing their storage requirements by almost 80%.
    • As development teams adopt agile methods and more frequent releases, demand for non-production environments is expected to increase, further increasing storage cost savings. eharmony has also eliminated data related defects by providing accurate, up-to-date test data. Unrestricted access to test environments significantly improves the ability of QA teams to identify issues before they reach production.

    Why DataOps and DevOps Are a Match Made in Heaven

    It is easy to see why DataOps and DevOps go together so well. DevOps calls for continuous software monitoring, testing and deployment. DataOps are inherently modular because they are designed to execute a single function. Software that is scalable and easily fits into the DevOps framework.

    Using DataOps in the DevOps process will help to fully recognize the benefits of continuous integration by validating the implementation promised by DevOps. These tools can be combined to optimize computing in a way that should be used during development whenever possible.

  • The Taxonomy of DataOps

    The Taxonomy of DataOps

    Data Operations, or DataOps for short, is one of those IT buzzwords that lots of people use, yet few can define precisely. Like the cloud or DevOps, DataOps doesn’t make sense until you sit down and really think through what it entails.

    To that end, let’s take a look in this article at the taxonomy of DataOps—or, in other words, which IT disciplines, areas of expertise and types of work make DataOps happen. I won’t explain everything about how DataOps works (that would require more than one blog post), but I will offer a primer on how DataOps workflows break down.

    What Is DataOps?

    Before delving into the taxonomy of DataOps, let me first define at a high level what I’m referring to when I use the term “DataOps.”

    I use DataOps as a shorthand for data operations of all types—from data collection and storage to data management, monitoring, analytics and performance optimization.

    I mention this because when some people hear the term DataOps, they think only of the practices and ideas described in the “DataOps Manifesto” and similar writings, which focus on extending the core principles of the DevOps software delivery philosophy to data operations. I’m a big believer in the “DataOps Manifesto,” but because it is targeted at data scientists rather than the IT community at large, I think it presents a narrow image of what data operations entails.

    To me, data operations encompasses the work performed by anyone who works with data in any way. While data scientists and data engineers may perform the bulk of the heavy lifting related to data transformation and data analytics, developers and IT engineers also have important roles to play in data operations. When I use the term DataOps, I’m thinking of them, too.

    The Taxonomy of DataOps

    With that clarification out of the way, let’s explore what goes into DataOps.

    Data Collection

    You can’t have data operations without data—and so, data collection is a core part of DataOps.

    Data collection refers, of course, to all of the processes by which you accumulate data and store it somewhere, for some length of time.

    It’s important to note that data collection can be both passive and active. Much of the data that organizations store and analyze these days is not data that someone goes out and collects by hand. It’s the information produced through the everyday operations of these organizations—their data exhaust, so to speak.

    Data Storage

    Data storage may seem simple and straightforward. After all, how much do you really need to think about the storage of data? Don’t you just pick a storage location and put your data there?

    Well, no. Data storage is much more complex than that, which is why data storage is a distinct discipline within DataOps.

    Effective data storage requires finding the right type of storage infrastructure—the one that will deliver the best balance between data performance, availability and cost. It also involves defining data retention policies to determine how long data will be stored. It encompasses data backups, too, which are important for ensuring that the data you intend to keep in storage actually will be available to you if the unexpected happens.

    Data Management

    Data management is a broad discipline that involves all of the tasks required to keep data available and accessible once it has been collected and stored. It entails work such as transforming data from one format to another to make it compatible with a given set of tools. It also involves tasks such as moving data between storage locations and identifying and resolving bottlenecks within the overall DataOps pipeline that are slowing down performance.

    Data Monitoring

    In the IT world, the word “monitoring” usually refers to the monitoring of applications or infrastructure for problems. But you have to monitor your data also if you want to do DataOps effectively.

    Data monitoring includes tasks such as identifying data quality problems (such as missing data or data formatting issues) that will make your data less reliable or more difficult to work with. It also means checking for signs of failures, such as a failed database or infrastructure crash, that could cause a rupture in your overall DataOps pipeline.

    Data Analysis

    Everyone (or everyone’s boss, at least) wants to “turn data into dollars,” as Gartner put it. Data analysis, or the process of seeking insights within sets of data, is key to making that happen.

    Sophisticated data analytics is the only discipline on this list that is usually performed by data scientists alone, not the rest of the IT team—although that is changing somewhat as more and more simplified data analysis and visualization tools enter the market that makes it easy to make sense of large data sets without having a Ph.D. in statistics or knowing how to program in R.

    Data Security

    Data security is a big topic, especially in the present age of high-profile data security breaches. Keeping data secure requires a range of processes and tools, from setting the right access control policies for data to monitoring for signs of unauthorized access and scanning data to root out malware or other threats that could exist within data sets.

    DataOps is Everyone’s Ops

    You might have noticed that many of the tasks described in the DataOps taxonomy above typically are performed by employees of the IT department (or, in some cases, programmers), not data scientists or engineers who specialize in data operations.

    This is an important point to drive home because when many people talk about DataOps, they assume that every organization has a team of data experts on staff who handle all work related to data, from collecting and storing it to analyzing and securing it.

    In reality, many companies don’t have dedicated data operations teams. And even if they do, the general IT department still plays a huge role in tasks such as configuring tools and platforms for data collection, provisioning data infrastructure and keeping databases running healthfully. Security engineers, too, are key for things such as enforcing data access control and monitoring for signs of data security problems.

    Thus, whether your company has a dedicated data operations team or not, DataOps is the responsibility (at least in part) of everyone who does anything related to IT. It’s everyone’s Ops.

    This sponsored article was written on behalf of Unravel.

    — Chris Tozzi

  • The Value of Visual Data in Decision-Making

    The Value of Visual Data in Decision-Making

    Gone are the days when a gut-feeling was a major decision-making driver in business. Instinct has ruled business leaders’ mind for a very long time, but as companies expand their business globally, instinct should give way to data as the best measure to remain competitive.

    We live in a data-driven world. Today, number-crunching algorithms decide who we should hire, where we should invest and even who we should date. In fact, data-driven organizations that harness insights to create a competitive advantage are growing rapidly—at an average of 30% annually. What’s more, data-driven organizations are on a track to earn $1.8 trillion by 2021.

    Data in Businesses

    Technology is tucked inside every nook and corner of modern business. Smart leaders use data generated to understand what’s happening in the organization, and this analysis helps them bring agility to their business.

    Top companies around the world use data to add value to their business. They gain a strategic advantage over their competitors putting data—not instincts—to good use.

    Being data-driven offers multiple advantages:

    • Data is the new gold. A long-term relationship with data helps organizations set trends.
    • Data-driven decisions offer a better customer experience. Data reveals customer patterns and their journey with products.
    • Data-driven organizations respond better to market demands with innovative solutions.
    • Data-driven decisions hone key business areas such as finance and operations.

    Data offers insights and executives turn those insights into key decisions. But you cannot ignore that fact that data is complex, let alone its analysis. We’re all familiar with excel sheets full of thousands of rows and hundreds of columns of numbers. So, how can this turn into insights? That’s where data visualizations come into action.

    Data Visualizations Make Hidden Insights Obvious

    Interpreting a voluminous dataset for actionable insights is tough. Data visualizations make voluminous data accessible and comprehensive. How?

    Data viz, as it is often called, is the transformation of any unstructured data into visuals. Visuals add a component to human understanding.

    For example, the World Bank releases its doing business project data every year, providing insights about countries with conditions for establishing businesses. It was launched in 2002 to measure the regulations applied to various small and medium-size organizations.

    A total of 190 economies were surveyed on 12 topics including employing workers, tax benefits and resolving insolvency. A visual summary of data on doing business in 2018 was produced to make these crucial insights digestible for businesses, which then could put them to use.

    The data science company that created the visualization took the data and plotted it on a world map. It added shades from red to green to indicate the least and the most favored economies for doing business. The data is made easy, appealing and comprehensive. Any business manager or executive can easily identify emerging economies for investment.

    Using visual data, insights such as the best and worst country for doing business and economy rankings are now obvious. Furthermore, companies can turn these insights into key decisions regarding expanding business across boundaries.

    Take a look at the example above: The United States is an economically powerful country; however, data tells that New Zealand tops the list of economies with ease of doing business in 2018 with 86.55 DTF*. Many countries, including Singapore, Demark, Korean Republic and Hong Kong, also rank favorably in the index.

    Three major reasons that data visualization can be crucial for any data-driven organizations are:

    1. Data visualizations amplify a story with pictures and visuals. Its impact on the audience is more powerful than bland numbers and text. Moreover, human minds are hardwired to remember visuals. Data visualizations tap into pictorial strengths to drive executive decision-making.
    2. Data visualizations make difficult data comprehensible. Business executives and key decision-makers are not masters in every field. They need comparative measures to drive their decisions. Data visualizations provide just that.
    3. Data visualizations help in decision analysis. It offers a platform to interact with the facts. Jeff Bezos said that not all the decisions are made by factual analysis. But whichever decisions can be made on facts, should be.

    Conclusion

    Data should be an influence to any major business decision. And so, data visualization is a powerful tool to bring dramatic changes to organizations. It can provide answers to many key questions, and make the data visually appealing, easy to play with and interesting to explore. CEOs and managers should emphasize the importance of the democratization of their data to add value to their processes.

    *DTF: Distance to Frontier. It is a score between zero to one hundred, which illustrates the worst and best performance of an economy observed on each doing business parameter.

    — Naveen Gattu

  • How AIOps Helps You Get More Out of DevOps

    How AIOps Helps You Get More Out of DevOps

    DevOps may seem like an overused term. It involves taking an application’s source code and running it in an environment. It can cover the processes, the technology or even the people that maintain the technology that is running those very processes.

    At its heart, though, DevOps is about helping developers to be self-sufficient when it comes to the basic operations around getting their application to a real environment. Once things are running in a test or production environment, there are other IT functions that are required to keep it running and performing well. While there are teams that specialize in these functions, such as site reliability engineers (SREs) and system administration, they are usually overwhelmed by the sheer amount of data their areas of responsibility generate without considering the data that applications going through their DevOps processes are also creating.

    This is what makes AIOps the perfect complement to the DevOps space.

    What AIOps Does

    What AIOps does best is break down silos within IT. Having data from all the functional areas processed and handled by modern machine learning (ML) in a single place allows correlation, which leads to faster problem identification and resolution.

    Before machine learning was introduced, for humans to process data, more or less they operated in three ways:

    • They knew exactly what data to look for, such as when a specific business function isn’t working. There is a typical message in the monolithic application that handles all the core business.
    • They routinely sifted through data only from their area of expertise.
    • When they went outside their area, any logs and other data they had access to often was summarized data, which has time gaps and can miss a lot of detail that proper correlation needs.

    Now with ML, the AIOps data platform can perform trend analysis, look for patterns and exceptions across all the data that is being generated in the environment and highlight what it considers important through web interfaces or using an event management solution for after-hours support.

    Over time, as the AIOps platform continues to learn the environment, it can be given the ability to automatically resolve certain error conditions, as it will be able to identify them faster than human operators and often resolve them before they become service-impacting.

    How This Helps DevOps

    Machine learning aids DevOps processes in three ways when working with logs and data:

    • Knowing exactly what data to look for: In a true polyglot DevOps environment that is leveraging microservices, it is next to impossible for any one person to know the most common error messages that might occur. In this model where a new business function can be conceived, written in one of many languages, and deployed into production without Operations even being notified, ML allows the AIOps platform to quickly identify what “normal” looks like for a new microservice and start looking for anomalies.
    • Only routinely sifting through data from their area of expertise: When working in DevOps, developers need to be able to quickly find data from multiple areas in IT that is related to whatever they are trying to diagnose. Using AIOps, they will be able to quickly pull data from systems directly related to their incident. Whether it is as simple as using timecodes or a correlation GUID (globally unique identifier) that is put in the logs purposefully to aide tracking, AIOps will guide the developer through the other systems’ data.On the flip side, if there is an infrastructure problem, the same AIOps platform can help correlate what downstream services may have been impacted, from failed builds to Kubernetes pods that were restarted.
    • Summarized and incomplete data: The true value of machine learning is that it has the ability to process a higher volume and variety of data at a much higher speed and accuracy than a human, and sort and index it for later use. Not having to summarize the data to one data point every five minutes could result in a problem being discovered that has occurred often but not consistently, as it may have been dropped in the summarization.

    Conclusion

    AIOps is very young in the technology industry, but has great promise as it builds on previous generations of IT operational analytics products, combined with the latest in machine learning capabilities. DevOps is a key field in IT that has more logs than it knows what to do with. Combining the technologies and expertise in these two growth areas will have a huge positive impact on productivity across enterprises with improved service management.

    — Vince Power

  • Top 5 Best Practices for DevOps Monitoring

    Top 5 Best Practices for DevOps Monitoring

    Today organizations are required to deliver on higher levels of customer satisfaction for their online services. Many, however, are forced to support these initiatives with an interrupt-driven approach as they react to fix things after they break. However, for a more proactive approach and to manage expected high levels of SLAs, organizations can opt to reduce their amount of unscheduled downtime by implementing a continuous delivery (CD) model to their development efforts.

    What is critical in the CD model is the ability to monitor and manage systems in a structured way, getting early detection of problems so organizations can make changes before the system is impacted. While unexpected failures make interrupt-driven work unavoidable at times, organizations can become more proactive by examining their current approach and toolset against business needs to help create a path to continuous service delivery optimization.

    Effective CD Models Need Effective Monitoring

    As a starting point, organizations need control and visibility into their DevOps environment by collecting and instrumenting everything. Considering the amounts of data, this can be an insurmountable challenge for most organizations. To get started, follow these five best practices to perform DevOps monitoring efficiently and quickly in a measurable and scalable way.

    Step 1: Collect the Data. You can’t manage what you don’t measure! It is important to take inventory of what is being collected today and align with business and executive teams to collaborate on the aim of the services delivered. Analyze the inventory collection of metrics with questions such as: “Why aren’t we collecting this? How does this fit into our goals? Have you ever seen a failure in X? How often should we be measuring this? How long should we keep it for? Is that important?” Teams also should evaluate how they are collecting the information and consider the best architectural approach for collecting their metrics, including whether pull, push or pull collection methods are better. Once goals are understood and the inventory of data is collected, look at what else your organization should consider collecting.

    Step 2: Correlate and Triage. Correlation of data is needed to understand it, but data comes in at different frequencies, in different time frames and from different sources. Work to normalize the data to understand it, perform comparisons against the different incoming metrics and establish baselines for basic service availability. Since organizations aim to go beyond basic service level agreements (SLAs) and offer a high-performing solution, constantly ask what the organization is missing from a data perspective and how it relates to business initiatives. Asking that question from the collection, correlation and triage perspective is critical.

    Step 3: Identify Trends. Organizations need to examine historical data to identify trends and take action before issues arise and customers complain. Establish alerting thresholds by outlining what a normal day looks like from a monitoring performance and customer perspective, and then identifying examples of what makes an abnormal day. This ties in with managing infrastructure inventory and understanding safety thresholds for each of the components that potentially could impact the service offering. It’s critical to communicate these findings with teams and business-line managers to prevent service delivery problems from happening and optimize offerings based on identified trends.

    Step 4: Notify and Act (Automation). In manual mode, a notification is delivered and then the team reacts. But teams are continually pushed to do things faster, and automation can help. To get to that point, organizations must understand where best to add more automation. How do you gather the right telemetry that delivers consistent answers that the machine can operate rules against, and do you leverage an automated process or notify a person? The desire to improve to a faster process requires a shift from manual to automated practices.

    Step 5: Predict (What-if Analysis). If you don’t take the time to go through the first four steps in a methodical way, it’s very difficult to reach this final step without always remaining in a manual mode. To balance costs and availability, it’s critical to discuss with the executive team how to predict customer service consumption (revenue) against the amount your services are going to cost the business.

    For example, developing a service that can alert customers of a potential service disruption due to disk space can be accomplished only by clearly setting business goals, then using the right metrics and events platform leveraging a time series database. In this case, the business goal is to ensure no service disruption due to failures such as low disk space; the metric monitoring is for disk space for each customer instance with the appropriate threshold alert set off by an automated trigger; and the action is an email to the customer letting them know of the situation and what action they can take (reduce load or upgrade for more disk space). In this example, it is important to marry business logic with the monitoring practice to make service a successful experience for customers. Furthermore, this approach will help teams to not only predict user experience but also help to predict OpEx and CapEx better into the future.

    That’s the benefit of the ability to predict with the right monitoring approach—to make informed business decisions based on results. Customers appreciate that, and proactive support is much better than being reactive.

    Emerging trends such as microservices, containerization, elastic storage, software defined networking and hybrid clouds are pushing the boundaries of DevOps monitoring. The right monitoring plan can identify and resolve problems before they affect critical business processes and enables customers to plan for upgrades before outdated systems begin to cause failures and outages for users.

    About the Author / Mark Herring

    Mark Herring is CMO at InfluxData. He is a well-rounded silicon-valley executive with proven experience in taking complex technology and making it understandable to the broader audience. His deep developer roots are never far from his mind as he looks at trends and asks the tough questions on whether the technology is here to stay or just another fad. Currently Mark is Chief Marketing Officer at InfluxData. Follow him on Twitter and connect with him on LinkedIn.

  • Useful Big Data Terminologies, Part 1

    Useful Big Data Terminologies, Part 1

    As data continues to increase at an evermore rapid pace, organizations struggle to deal with this data torrent, let alone use it to analyze and capture value. The ways used to understand this big data also is increasingly rapidly, which introduces myriad terms used to define these methods.

    The follow is an attempt to provide natural explanations to some of the significant terms and technologies you will come across when you’re getting into big data.

    Algorithms: Mathematical and analytical formulas that also include statistical processes used to analyze data. Algorithms are implemented in software to analyze, process the input data and produce output or results.

    Analytics: The course of depicting conclusions based on the raw data. With the help of analysis, otherwise-meaningless numbers and data can be converted into something useful. The emphasis here is on interpretation and not on big software systems. That may be why data analysts are very experienced in the art of storytelling.

    Biometrics: Using analytics and technology to identify people by one or many of physical characteristics, such as fingerprint recognition, face recognition or iris recognition.

    Cassandra: A very well-known open-source database management system managed by the Apache Software Foundation, which has been constructed to handle high volumes of data throughout distributed servers.

    Cloud: A term used to describe data or software running on remote servers rather than locally. Data stored in the cloud is usually reachable over the internet, wherever the owner of that data in the world might be.

    Database: A systematized collection of data, such as schemas, charts or tables. A database management system (DBMS) is software that helps in data analysis and exploration.

    Data Mining: This term can mean different things for different context. To the layman, it means the automatic examining of large databases. To an analyst, it refers to the pool of statistical and machine learning methods used in the databases.

    Dark Data: The information collected and managed by a business that is never put to use, yet sits waiting to be studied. Most companies don’t realize they have a lot of this kind of data lying around.

    Data Scientist: A skilled expert in extracting value and insights from data. A data scientist typically is someone with skills in computer science, analytics, mathematics, creativity, statistics, communication and data visualization, as well as strategy and business.

    Gamification: The process of creating a game-like environment in areas that typically would not have games, such as websites, to attract users and increase engagement. In the terms of big data, a gamification is a powerful tool for incentivizing the data collection.

    Hadoop: An open-source software structure that works mainly by processing and storing files and data. Hadoop is known for its big processing power, which makes it easy to run a host of tasks alongside. It helps companies access, save and analyze enormous amounts of data.

    IoT: An acronym that stands for internet of things. Principally, it defines an ecosystem of things, from diapers to self-driving cars, that can communicate with each other via the internet. Their sensors generate a large amount of data that can be analyzed.

    Machine Learning: A highly casual way performing data analysis. Machine learning mechanizes logical model building and trusts on the ability of the device to adapt. With the use of algorithms, models dynamically learn and improve themselves every time they process any new data. Machine learning is not new; however, it is receiving massive attraction as a modern tool for data analysis. It allows devices to grow and acclimatize without demanding numerous hours of extra work by the scientists.

    MapReduce: A model for programming, generating and processing massive data sets. It does two different things: the Map, which includes rotating one dataset to the other, more valuable and fragmented dataset made of bits known as tuples; and the Reduce, which takes all of these fragmented tuples and breaks them even further. It results in a useful breakdown of information.

    NoSQL: Database management systems that do not use relational tables used in most old-style database systems. The data retrieval and storage system is designed for managing massive volumes of data without tabular categorization.

    SaaS: An acronym for software as a service, a method of application delivery in which vendors host applications and make them available through the internet. SaaS providers deliver their services via the cloud.

    Spark: An open-source computing structure developed at the University of California, Berkeley, and donated to the Apache Foundation. It is used for interactive analytics and machine learning.

  • Why You Need to Embrace Automation

    Just in case you hadn’t noticed: The robots are coming.

    Well, you probably had noticed. It’s hard to miss. Newspaper articles and blogs everywhere herald their arrival. The stories usually follow the same general formula: A catalogue of amazing things that robots (really, AI systems) can do. Debate about forced unemployment vs. people freed up to do more interesting work. And then some comments about whether it’s for the good or bad. You can generally infer the general tone from the choice of image accompanying the story: a cute Japanese robot with a plastic smile or a more sinister face with the Terminator’s jawline.

    Of course, workplace automation has been around for almost as long as we have had workplaces. Each new wave of technology disrupts the employment landscape in its own particular way. However, this time it’s serious: It’s not just manual labor that’s under threat, but increasingly, we’re told, white-collar workers are being supplanted by machines. A recent McKinsey report has generated a wave of headlines: “Even the CEO’s job is susceptible to automation,” wrote Patrick Nelson at Network World; and Ana Swanson at the Washington Post reported, “How robots will even affect the jobs of people we thought were immune.”

    Not surprisingly, a closer read of the McKinsey material gives a more nuanced appraisal than a headline can offer. Few of us will actually be replaced by a robot, but many workers will lose some part of their current jobs to automation. The McKinsey report states that up to 45 percent of the activities that employees perform today could be automated, but very few occupations will be automated entirely in the near or medium term. The things that require creativity and emotion remain uniquely in the domain of humans, and they are likely to remain there for quite some time.

    It’s against this background that we see a steady increase in what is often referred to as robot journalism: machines writing news stories. So far, these applications rely on scenarios where the general ‘shape’ of a story remains pretty consistent because of the regularity of the data underlying those stories. For example, a report on stock performance is always going to mention the current stock price, what the stock price was at some particular point in the past, the price highs and lows, and perhaps something on volume or analyst sentiment. These reports often are found in the back pages of a newspaper, and are the stories that few journalists actually want to write, since they allow almost no space for creativity and emotion.

    The reality is that today’s news-writing software is still a long way from replacing human journalists. Which is not to say that what a robot journalist produces isn’t useful; it surely is. Last month I spoke about robot journalism at the news:rewired conference. It was fascinating to see how most of the major news media are exploring how to get robots on staff. So while we’re going to see more robot journalists, it doesn’t necessarily mean fewer human journalists. A recent study by researchers at Oxford University and Deloitte explored how susceptible various jobs were to automation—journalists ranked 285th out of 366 jobs, with an ‘automation risk’ of just 8 percent.

    But if you really want to see where the action is regarding automated writing, just look at it from a different angle. Rather than Natural Language Generation displacing people from their jobs, it’s offering a solution to the problem of shortage: A shortage of workers or expertise. And that story isn’t about news; it’s the Big Data story. You can take issue with the astounding numbers that are thrown around, but even if they’re out by a factor of 10, they’re still significant: by 2020, about 1.7MB of new information will be created every second for each person on earth. But less than 0.5 percent of all data is ever analyzed.

    In 2011, McKinsey noted, “The United States alone faced a shortage of 140,000 to 190,000 people with analytical expertise, and 1.5 million managers and analysts with the skills to understand and make decisions based on the analysis of big data.” And subsequent reporting points to massive year-on-year leaps in demand.

    Regardless of how few or how many people are displaced by robots, they are not all going to become data scientists. The only way this volume of data will be analyzed and explained is if we have machines doing it for us. And that’s the real calling and promise of Natural Language Generation (NLG). Interactive visualizations and dashboards don’t cut it; the meaning in all that data will be lost unless we call in the robots. But you need more than robot journalists, tirelessly churning out one formulaic story after another. You need articulate robots that use sophisticated NLG to construct fluent narratives from idiosyncratic data sets. So, accept the data reality, embrace automation, and find an NLG robot to explain the insights buried in your data.

    About the Author:

    2,41 CWDr. Robert Dale, Chief Technology Officer and Chief Strategy Scientist at Arria NLG, is recognized as one of the world’s foremost experts in Natural Language Generation (NLG) research and Development, having authored or edited seven books and 160 papers on computational linguistics. He was a Professor in the Department of Computing, Director of the Centre for Language Technology at Macquarie University.  He co-authored the seminal textbook “Building Natural Language Generation Systems” with Arria NLG Chief Scientist and co-founder Professor Ehud Reiter.