devopsy.kinsta.cloud

Tag: data monitoring

  • Getting Started With Continuous Monitoring

    Getting Started With Continuous Monitoring

    DevOps continues to gain traction among organizations as demand grows for digital product and platform development. According to Gartner, 87% of business leaders believe digitalization is a priority. Every DevOps transformation requires a dedicated, continuous learning process and effective implementation to reach maturity. If a practice or pattern is passed over or ignored, it can put a damper on DevOps success. Continuous monitoring, for example, is a very important part of every DevOps life cycle that is often overlooked.

    Through effective continuous monitoring efforts, organizations can scale and deliver digital products and platforms faster and more securely. In fact, these continuous monitoring advantages tie directly to modern customer needs. If they haven’t already, business leaders must embrace continuous monitoring along every point of the DevOps life cycle. These best practices can help you get started.

    Understanding Continuous Monitoring

    The practice of continuous monitoring helps to collect and analyze outcomes, statuses, exceptions and key metrics within each step of the DevOps process – from development to deployment and production. Additionally, it offers deep insight for DevSecOps teams by enabling compliance triggers and security alerts configuration. It allows the organization to detect issues or security concerns throughout every phase of the DevOps lifecycle.

    Understanding what continuous monitoring does is a good start. However, to effectively adopt continuous monitoring, the organization must know what to monitor.

    What Should Be Monitored

    An effective continuous monitoring process collects enough data to gain sufficient understanding of both successes and failures of the organization’s systems and processes. Some of the events collected by the continuous monitoring systems will require immediate attention, while others may lead to longer-term strategy discussions.

    Collecting the right information is always tricky considering the number of endpoints that generate logs and events. One should rank the significance of each source in relation to the type of data it produces, the functional area it supports and the impact it has on the system’s performance, as well as on organization’s goals and commitments (requirements posed by SLA’s, regulations, contracts, etc.).

    A combination of technology and strategy helps ensure that the right data is collected at the right time. The next step, and perhaps one of the most significant challenges, is finding the balance between monitoring and analysis.

    How to Achieve Balance

    Balance is, by far, one of the most difficult decisions in log monitoring and analysis. Gather too little data and risk missing early trouble indicators or important alerts. Collect too much and get buried in analysis paralysis – essentially, leading to the same situation of overlooking items of significance. Striking the right balance requires understanding the system’s continuous monitoring processing capacity and using its full potential. This may also involve data preparation steps during intake to ensure optimal processing and storage of the collected material.

    Offloading mundane tasks and parts of the analysis to machine learning and artificial intelligence (ML and AI) and fuzzy logic systems will enable big data processing gains. With this approach, the continuous monitoring capacity can be significantly increased, along with the semantic quality of the alerts and notifications produced by the system. As a result, security operation center (SOC) staff can devote their attention to the analysis of events which are more meaningful to the system and operations, leading to a more efficient continuous monitoring process.

    A DevOps transformation will likely not reach full maturity unless the organization applies continuous monitoring to the DevOps life cycle. Although often overlooked, it remains central to DevOps success. With this in mind, a basic understanding of continuous monitoring, what to monitor and finding the balance between log monitoring and analysis, serve as a foundation for getting started with continuous monitoring.

  • The Top 5 Challenges to Overcome with Heroku Logging

    The Top 5 Challenges to Overcome with Heroku Logging

    Heroku can be a very powerful tool for monitoring the massive amount of data generated by your applications, systems, APIs and add-ons.

    One thing you may have discovered with Heroku logging is that making sense of data within your log files can be time-consuming. When you’re running large numbers of applications which process tons of data each day, the sheer volume of data that appears in the logs can seem overwhelming.

    Managing Massive Log Output 

    Due to the sheer amount of data that is gathered, it’s important to find ways of making the process a little bit easier, limit information to what is useful and organize what you have into digestible pieces.

    Parsing with Drains

    Using Logplex and log drains can help you filter out large chunks of your logs. Syslog drains will help you move any system messages to another server for archiving purposes.  HTTPS drains can help you better take control of your system messaging for sending it to third-party apps for analysis. 

    By assigning clear indicators that identify the application, system or API that creates a log, these can be streamed into channels that can be better understood in relation to their given purposes. 

    Log Retention Period

    While we may think that more information is always good, there is a limit to the amount that we can process. Some log data becomes meaningless or redundant after a certain length of time. It is wise to set realistic log retention periods on data based on business needs.

    Limit Verbosity

    Don’t log everything. Turning on debugging is appropriate during development and if there is a specific problem that needs investigating, but in most cases, these shouldn’t be stored. 

    Complexity in Stream Patterns

    The challenge with large amounts of data is that there may be subtle differences between successful activity and erratic behavior. The details may be small, and essentially invisible to the naked eye unless examined carefully. 

    Use Regex for Identifying Patterns

    Through the use of log analytics tools, you can set these to pick up specified regex patterns to subtle changes or variations in your logs that can provide meaningful insights. 

    Different Log Data Needs

    You may have people with different needs who wish to make use of the information you gain in your Heroku logs. While you have data that is of interest to developers and QA engineers, there are other tech-savvy users who may wish to gain business intelligence (BI) from the massive output of data, to be able to identify key performance indicators (KPI).

    Analytics Tools

    Using intelligent add-on analytics applications can help by filtering and sorting information into usable chunks that can be understood by users in different roles. By accurately filtering, you can provide intelligent real-time alerts for a specific application or system behavior to those who may wish to make use of it. 

    Application Issue Forecasting

    One of the best potential uses of your detailed Heroku logs involves the opportunities to spot patterns of problems that have occurred in the past and use them to predict problems that may occur in the future.

    As an expert, you may be able to spot patterns in your logs, however, even the smartest and most attentive engineers will not be able to manually define every scenario worth detecting.

    Occasionally patterns will occur that are only detectable through the rote process of comparing and contrasting working behavior with non-working behavior. Often, the subtleties are too small for a human to be able to notice the difference.

    Machine Learning Assistance

    Naturally, utilizing ML-powered techniques can help catch the unknowns. By investing in ML, using some of the better libraries or choosing a Heroku logging add-on with these capabilities, you can monitor your applications and infrastructure with less effort. 

    Securing Sensitive Data

    With the rapid pace of CI/CD, a large amount of activity regularly flows through your pipelines from development into production. Throughout this process, it’s possible for sensitive data to start appearing in logs. While your teams may be diligent about keeping control over OAuth credentials, security keys, authentication tokens and more, nobody is perfect. Sensitive security credentials can accidentally make their way onto production servers. Sensitive customer data can also accidentally find its way into public logs if mistakes are made.

    Credential Detection and Security Alerts

    Logs can serve as a saving point for the identification of sensitive information. Using a parsing engine, these can be immediately redacted for compliant log management. By monitoring your log files for any patterns which could include secret keys or other potential security breaches, you’ll be able to continue ingesting critical log data without increasing business risk.

    Heroku Private Spaces

    With Heroku Enterprise, you can make use of Heroku Private Spaces which are an isolated network of private apps and data services, which will help you keep sensitive data separate from public interference. If you are handling a large amount of sensitive customer data, ensuring that any information gets filtered into these places will help you remain compliant with GDPR and other privacy-related regulations.   

    Conclusion

    Heroku can be a fantastic solution for monitoring the activity which goes on in your cloud clusters.  However, unless you have solid analytical tools in place, you may find yourself spending a lot of time wondering how to make use of this sea of data that has been collected. Consider using an add-on as an analytic layer to help spot patterns, organize data better and create smart alerts to make sure your operations continue to run smoothly.

  • The Taxonomy of DataOps

    The Taxonomy of DataOps

    Data Operations, or DataOps for short, is one of those IT buzzwords that lots of people use, yet few can define precisely. Like the cloud or DevOps, DataOps doesn’t make sense until you sit down and really think through what it entails.

    To that end, let’s take a look in this article at the taxonomy of DataOps—or, in other words, which IT disciplines, areas of expertise and types of work make DataOps happen. I won’t explain everything about how DataOps works (that would require more than one blog post), but I will offer a primer on how DataOps workflows break down.

    What Is DataOps?

    Before delving into the taxonomy of DataOps, let me first define at a high level what I’m referring to when I use the term “DataOps.”

    I use DataOps as a shorthand for data operations of all types—from data collection and storage to data management, monitoring, analytics and performance optimization.

    I mention this because when some people hear the term DataOps, they think only of the practices and ideas described in the “DataOps Manifesto” and similar writings, which focus on extending the core principles of the DevOps software delivery philosophy to data operations. I’m a big believer in the “DataOps Manifesto,” but because it is targeted at data scientists rather than the IT community at large, I think it presents a narrow image of what data operations entails.

    To me, data operations encompasses the work performed by anyone who works with data in any way. While data scientists and data engineers may perform the bulk of the heavy lifting related to data transformation and data analytics, developers and IT engineers also have important roles to play in data operations. When I use the term DataOps, I’m thinking of them, too.

    The Taxonomy of DataOps

    With that clarification out of the way, let’s explore what goes into DataOps.

    Data Collection

    You can’t have data operations without data—and so, data collection is a core part of DataOps.

    Data collection refers, of course, to all of the processes by which you accumulate data and store it somewhere, for some length of time.

    It’s important to note that data collection can be both passive and active. Much of the data that organizations store and analyze these days is not data that someone goes out and collects by hand. It’s the information produced through the everyday operations of these organizations—their data exhaust, so to speak.

    Data Storage

    Data storage may seem simple and straightforward. After all, how much do you really need to think about the storage of data? Don’t you just pick a storage location and put your data there?

    Well, no. Data storage is much more complex than that, which is why data storage is a distinct discipline within DataOps.

    Effective data storage requires finding the right type of storage infrastructure—the one that will deliver the best balance between data performance, availability and cost. It also involves defining data retention policies to determine how long data will be stored. It encompasses data backups, too, which are important for ensuring that the data you intend to keep in storage actually will be available to you if the unexpected happens.

    Data Management

    Data management is a broad discipline that involves all of the tasks required to keep data available and accessible once it has been collected and stored. It entails work such as transforming data from one format to another to make it compatible with a given set of tools. It also involves tasks such as moving data between storage locations and identifying and resolving bottlenecks within the overall DataOps pipeline that are slowing down performance.

    Data Monitoring

    In the IT world, the word “monitoring” usually refers to the monitoring of applications or infrastructure for problems. But you have to monitor your data also if you want to do DataOps effectively.

    Data monitoring includes tasks such as identifying data quality problems (such as missing data or data formatting issues) that will make your data less reliable or more difficult to work with. It also means checking for signs of failures, such as a failed database or infrastructure crash, that could cause a rupture in your overall DataOps pipeline.

    Data Analysis

    Everyone (or everyone’s boss, at least) wants to “turn data into dollars,” as Gartner put it. Data analysis, or the process of seeking insights within sets of data, is key to making that happen.

    Sophisticated data analytics is the only discipline on this list that is usually performed by data scientists alone, not the rest of the IT team—although that is changing somewhat as more and more simplified data analysis and visualization tools enter the market that makes it easy to make sense of large data sets without having a Ph.D. in statistics or knowing how to program in R.

    Data Security

    Data security is a big topic, especially in the present age of high-profile data security breaches. Keeping data secure requires a range of processes and tools, from setting the right access control policies for data to monitoring for signs of unauthorized access and scanning data to root out malware or other threats that could exist within data sets.

    DataOps is Everyone’s Ops

    You might have noticed that many of the tasks described in the DataOps taxonomy above typically are performed by employees of the IT department (or, in some cases, programmers), not data scientists or engineers who specialize in data operations.

    This is an important point to drive home because when many people talk about DataOps, they assume that every organization has a team of data experts on staff who handle all work related to data, from collecting and storing it to analyzing and securing it.

    In reality, many companies don’t have dedicated data operations teams. And even if they do, the general IT department still plays a huge role in tasks such as configuring tools and platforms for data collection, provisioning data infrastructure and keeping databases running healthfully. Security engineers, too, are key for things such as enforcing data access control and monitoring for signs of data security problems.

    Thus, whether your company has a dedicated data operations team or not, DataOps is the responsibility (at least in part) of everyone who does anything related to IT. It’s everyone’s Ops.

    This sponsored article was written on behalf of Unravel.

    — Chris Tozzi