devopsy.kinsta.cloud

Tag: data collection

  • What the Convergence of Observability and Security Means for Devs

    What the Convergence of Observability and Security Means for Devs

    There is a scene in the movie Apollo 13 when the mission control flight director asks why the carbon dioxide scrubber in the command module was a different shape than the one used in the lunar module (and therefore incompatible). The engineer simply replied, “This just isn’t a contingency we’ve even remotely looked at.”

    A similar scenario plays out at many organizations when they examine security and development practices. When it’s time for code to go into production, every developer hopes their code will run as planned–smoothly and error-free. Unfortunately, something will be missed, performance could degrade or new, unanticipated problems can emerge after an application is deployed to production.

    Developers fear the “unknown unknowns.” And it keeps them up at night.

    When an “unknown unknown” problem occurs, it takes time for teams to do the forensics to identify the problem. It can take days or even weeks for teams to complete that initial investigation. If a mission-critical application is offline for a significant period of time due to a forensics investigation, then it can negatively affect your customer’s experience and, ultimately, your organization’s bottom line.

    The “unknown unknowns” problem is another challenge facing developers in today’s unpredictable macro environment. That includes the deluge of data, increasing cyberthreats, an increasingly distributed infrastructure and a remote and hybrid workforce as well as a mix of modern cloud-native and monolithic apps.

    To help developers build for the future and address security concerns, it is imperative they rethink the way they are collecting, operationalizing and storing different types of data from a variety of sources.

    Security and Observability: Better Together

    Today, security and observability have recently begun to overlap, driven by the growing need for organizations to better understand the activity inside their environments. Many forward-thinking organizations are operationalizing the massive amounts of log and event data currently being generated to understand and assess issues in their infrastructure and apps and using these actionable insights to optimize workload, app resource availability, security and uptime.

    So how can developers take advantage of the security and observability convergence trend happening right now?

    Currently, it’s common for developers to have two agents as part of their tech stack. One agent for observability, which is collecting data such as logs and events to get relevant insights into the health and performance of their application in real-time. The other agent is for security to collect data in their devtest, integration and production environments.

    Since these two agents often collect the same data, there is an opportunity to consolidate it into a single location, such as a unified security platform with a single agent, where this data can be overlaid with time series analytics to deliver relevant insights to users. Using this approach of having both security and non-security data in one place will significantly speed up the process of getting these valuable insights into the developer’s hands.

    In addition to getting insights faster, another benefit of having this data in a single location is the ability to access historical data. Giving developers the ability to store and manage data for long periods of time allows them to get insights from this historical data, so they can better understand alerts, incidents and adversaries. This can help them correlate in terms of understanding the risk to an application, such as potential “unknown unknown” threats, which they can proactively address before it causes any damage.

    Adopting a converged security and observability approach is the most mature step that organizations can take to help developers deliver the best outcomes and best experiences for their customers and users. As more organizations rely on the cloud and the speed of business rapidly increases, developers must transform their view of the health, performance and security of applications and infrastructure. Giving developers this perspective will dramatically help them take rapid action against an adverse event before it impacts their company’s bottom line.

  • Guide to Privacy Management for Developers

    Guide to Privacy Management for Developers

    Privacy has taken on an unprecedented level of importance in 2022, primarily online privacy. There are multiple reasons behind this increase. The primary one is that users online are now more educated, informed and concerned than ever before about the collection of their personal data and, perhaps more importantly, how the website collecting this data manages and uses it.

    Privacy management refers to the practices, tools and strategies that an organization develops behind the scenes to ensure that any and all data collected is managed in a way that is  fully compliant with data regulations.

    Like every other division within an organization, the dev and DevOps teams play a vital role in successfully implementing and executing privacy management solutions. But how can the dev and DevOps teams contribute in this regard? Where should they begin? What practices to adopt?

    Be Clear About Your Privacy Policy

    This has less to do with the organization’s practices and more to do with the overall philosophy about privacy. There’s no point in beating around the bush regarding data collection and users’ privacy since the better educated and informed your users, the higher their chances of giving you the appropriate data collection permissions.

    You should be straightforward and let your users know why you need to collect their data. While this topic will undoubtedly be covered in the website’s privacy policy, it should also be reflected in the user’s browsing experience while on the site.

    This can be done by ensuring there’s an easy-to-find and quickly accessible link to the site’s privacy policy across all web pages on the website.

    Collect Only the Most Essential Data

    This may seem like a given in 2022, considering how unyielding and austere most data protection regulations are. However, there are still organizations that are in the middle of transforming their data collection practices. That means that even though the legislation they must comply with requires minimal data collection, in actuality, their practices paint another picture.

    This is not necessarily a deliberate or malicious practice. Most legislation, even those that are quite clear in stating that only the most essential data required for a website to function properly may be collected, gives organizations leeway regarding when they must adhere to these laws. The CPRA is perhaps the most vivid example of this.

    While the CPRA was passed in 2020, it will not go into effect until January 2023. Most organizations are expected to ensure their data collection practices are in line with the law when it goes into effect. There’s no point in delaying the inevitable. If your organization hasn’t begun transforming and altering its data collection practices, it is high time to start.

    Have a Data Transfer Mechanism and Strategy in Place

    This is arguably the most sensitive part of any organization’s privacy management infrastructure. This is down to the fact that while an organization may have to comply with a data protection regulation in one country to process and collect data on its residents, it may find itself having to balance out its practices to be able to transfer this data to another jurisdiction.

    Moreover, nearly every data protection regulation has a stringent set of requirements that an organization must fulfill before it can transfer the data out of the jurisdiction in the first place.

    Naturally, this can all become incredibly messy and escalate into a crisis unless you have a proper data transfer mechanism and strategy in place that takes into account every possible step that may hinder your compliance efforts.

    Ultimately, the buck stops with the dev team and DevOps team, in this case, to ensure that whatever mechanism the organization ends up adopting is fully capable of transferring data securely across jurisdictions without leading to a breach of any regulatory statutes an organization is required to follow.

    Ensure Accountability

    Some would argue that the entire philosophy of privacy management is built upon accountability. While various data protection regulations globally are meant to ensure all businesses follow a certain set of practices that reduce any chances of data breaches, true accountability comes from within.

    The most practical method of ensuring such accountability is by maintaining a regular and up-to-date record of processing activities (RoPA). Again, this is something that most data protection regulations will require businesses to maintain anyway, with processing activities, data flows and categories of data subjects the most common items that need to be covered.

    Similarly, make sure any third parties or vendors you work with have the relevant practices in place before going forward with any sort of data sharing, even if you have user consent to do so.

    Remember, accountability means holding yourself and those you work with to the highest standards.

    The most effective way to do this is to automate the process of record keeping. Owing to both the sheer volume of data that may become involved and the risks associated with human error in record keeping at scale, the best way forward is to opt for a data-centric automated approach.

    Conduct Regular Assessments

    Depending on which laws your organization must adhere to, regular assessments, known as privacy impact assessments (PIA), could be legally required. Most organizations design and implement their own PIA depending on their current practices and data collection methodologies.

    However, there are some fundamentals that each PIA is supposed to follow, such as the following:

    • The what, why, when and how of all data being collected
    • Will the data be shared or sold to any third parties?
    • What measures are in place to ensure the data is stored correctly once collected?

    The purpose of any PIA is to help you evaluate just how at-risk your organization is based on its current data collection practices. These evaluations can help you identify gaps and flaws in your current practices while also highlighting areas for improvement.

    On the off chance that you do discover discrepancies, the best option going forward is to contact the organization’s internal data protection officer (DPO) and develop a roadmap for eliminating those discrepancies before they cause any real damage.

  • Incidence Avoidance in Action

    Incidence Avoidance in Action

    Today, CIOs are faced with the challenge of making enterprise technologies perform on par with offerings from consumer-facing technology leaders, such as Facebook or Netflix, meaning technology or services being seamlessly available with no performance lag. Of course, these expectations are not always aligned with the realities of managing complex hybrid enterprise systems that were not necessarily designed to deliver this type of performance.

    On the other hand, IT operations supporting new-age “digital” business services are dealing with on-demand resources that are supported by software-defined data centers extending into the public cloud by using serverless computing platforms. This is resulting in a “new generation hybrid enterprise” that must deliver the functionality of, and operate as, real-time infrastructure.

    Lexmark, a U.S.-headquartered global company that manufactures laser printers and imaging products, wanted to achieve zero latency operational performance and enhance its business processes to achieve incidence or error avoidance.

    This article details Lexmark’s IT objective and the solution implemented by Wipro Ltd., a global information technology, consulting and business process services company.

    Problem: Lexmark required a technology solution that would provide visibility into its core business process—managed print services. The solution would also need to factor in other supporting systems and functional processes. The objective was to achieve operational excellence by minimizing errors and risks.

    Solution: To address this critical IT operations problem, Wipro was able to incorporate unique design principles to create an operationally simple, scalable and focused solution that brings IT and business process owners to a common understanding regarding achieving incident avoidance.

    Wipro built a new way of visualizing operations management, based on the fundamentals laid out by IT operational analytics (or AI operations, as it is called now).

    Working only with the tools and data that Lexmark already owned, Wipro and Lexmark created a system that measured the heartbeat of Lexmark’s managed print services business. That heartbeat was synchronized to the printer data collection systems, with the thresholds determined entirely by machine learning techniques. Every underlying system was arranged in a “server room.”  Only out-of-variance conditions are displayed in the main visualization panel, with the root cause just a single click away. This paper provides insights into the challenges faced by IT enterprises to deliver an on-demand solution to clients, similar to the ones offered by digitally evolved B2C companies.

    Digital enterprises that use hybrid environments must respond in real-time and need to achieve incident avoidance, rather than incident management.

    If we look behind the scenes at operation command centers today, we see operations that struggle to keep up the pace. Very few operations have been built from the ground up. In the enterprise world, Severity 1 (incidents that are categorized as high impact by business users) incident resolution times, still run for a couple of hours if IT support is not in a position to find the root cause of the problem. The culprit is often the lack of a proper configuration management database (CMDB), insufficient logging or lack of application discovery and dependency mapping. Every IT professional can recognize these failings, yet projects to correct these flaws never seem to get the necessary capital appropriation. Clearly, a new approach is needed to fix this.

    Identify the Business Process

    The process should start by identifying the key driving force or the heartbeat of the core business operation. Depending on the industry, this heartbeat might be customer orders, service requests or vendor purchase orders. All other systems are connected to this heartbeat. To select the right attribute to monitor as the heartbeat, start with business analysts who have a good feel for the day-to-day struggles of the operation. These analysts often have a series of scripted reports they run to track system health. More advanced analysts might even have automated these with alerts. By leveraging this information and using data discovery techniques, we found that the values can be configured as business KPIs.

    In the case of Lexmark, Wipro streamed the KPI values in real time and used machine and wire data to broadcast the underlying monitoring components by using an existing streaming analytics platform that is supported by AI and machine learning techniques. We found that the amount of data needed to assess process health is time sensitive, and normally less than 24 hours of data is displayed during that time.

    The platform visualization shown above, tracks the creation of sales orders for a business unit. The green line represents the flow of orders through the system. Any anomaly in an underlying system is reflected directly on this flow. This is achieved by streaming analytics and AI techniques that use the business KPIs.

    Apply the Architecture Overlay

    The next step is to subordinate the supporting infrastructure to the business process heartbeat. This feat is accomplished without complex business rules and sophisticated orchestration engines. We call this the “server room” and it is a view of the entire infrastructure on one web page, leveraging a lattice chart format that allows for easy comparison. Each of these underlying components will have a direct impact on the business process.

    Applying the same streaming analytics and AI/ML techniques that identified the anomaly in business process, we identify what is causing that anomaly and can quickly see the problem areas.

    Color coding helps the technician quickly understand the severity of the event. Blue indicates the system performance has not yet violated the set threshold, but an underlying infrastructure issue exists and needs to be proactively addressed. Amber indicates the system has violated the warning threshold. Red indicates the system has violated the critical threshold, a condition that by design we will never allow to happen.

    The user can scroll over any given point and a callout box will display the actual level, as well as a brief note on what has gone wrong and the timestamp.

    Analysts can now instantly focus their efforts on the specific systems that are in an error state, even if an event has not yet created an incident.

    Machine Learning to Define the Thresholds for an Anomaly

    What is normal behavior? This is often the most difficult question to answer because common monitoring and management processes rely on a single data value that was set during implementation. But thresholds must be extremely flexible to capture the ever-changing nature of the business process. Batch jobs that run at midnight, end-of-month closing activities, shift changes and a host of other potential variables must be considered when determining the right thresholds.

    For any component supporting the business process, we need to understand what is normal for right now. To determine that at Lexmark, we applied machine learning to dynamically set the thresholds for each sub-system at the month, day and hour level based on the healthy business KPIs we established.

    Incident Avoidance, Not Incident Management

    To fully understand the potential of this approach, it is necessary to change the way system analysts interact with the system. IT professionals who have been doing system support for several years, may be conditioned to react to the work in their queue. A well-honed incident management process is the enemy of proactive action that can prevent emergencies.

    By aligning the underlying system activity and thresholds with the business process, the IT professional can truly understand the whole environment, not just their individual technical component. The most effective analysts will extend their knowledge into other business areas. At Lexmark, we discovered that we were able to solve “intractable” performance issues by simply adding hardware or capacity.

    Integrated IT operations is not a goal, it is the means by which digital enterprises can be managed in real time. The path to real time management is through service capacity guarantees, performance and availability management and using anomaly detection in the context of the business process to identify issues well before an incident occurs. We advise starting small and building an anomaly detection system for the heartbeat of the process that enables you to see both the forests and the trees.

     

    This article was co-authored by Murthy Malapaka, general manager, Transformation Services Cloud and Infrastructure Services at Wipro Ltd. He helps enterprises digitally rearchitect their operations to run on hybrid IT environments to achieve incident avoidance instead of incident management. He has assumed various technology leadership roles across application and infrastructure architecture domains, specializing in availability and reliability. Murthy has been providing consulting services to CIOs and CTOs in their journey from client server to on demand infrastructure services.

    — Brad Clay

  • Top 5 Best Practices for DevOps Monitoring

    Top 5 Best Practices for DevOps Monitoring

    Today organizations are required to deliver on higher levels of customer satisfaction for their online services. Many, however, are forced to support these initiatives with an interrupt-driven approach as they react to fix things after they break. However, for a more proactive approach and to manage expected high levels of SLAs, organizations can opt to reduce their amount of unscheduled downtime by implementing a continuous delivery (CD) model to their development efforts.

    What is critical in the CD model is the ability to monitor and manage systems in a structured way, getting early detection of problems so organizations can make changes before the system is impacted. While unexpected failures make interrupt-driven work unavoidable at times, organizations can become more proactive by examining their current approach and toolset against business needs to help create a path to continuous service delivery optimization.

    Effective CD Models Need Effective Monitoring

    As a starting point, organizations need control and visibility into their DevOps environment by collecting and instrumenting everything. Considering the amounts of data, this can be an insurmountable challenge for most organizations. To get started, follow these five best practices to perform DevOps monitoring efficiently and quickly in a measurable and scalable way.

    Step 1: Collect the Data. You can’t manage what you don’t measure! It is important to take inventory of what is being collected today and align with business and executive teams to collaborate on the aim of the services delivered. Analyze the inventory collection of metrics with questions such as: “Why aren’t we collecting this? How does this fit into our goals? Have you ever seen a failure in X? How often should we be measuring this? How long should we keep it for? Is that important?” Teams also should evaluate how they are collecting the information and consider the best architectural approach for collecting their metrics, including whether pull, push or pull collection methods are better. Once goals are understood and the inventory of data is collected, look at what else your organization should consider collecting.

    Step 2: Correlate and Triage. Correlation of data is needed to understand it, but data comes in at different frequencies, in different time frames and from different sources. Work to normalize the data to understand it, perform comparisons against the different incoming metrics and establish baselines for basic service availability. Since organizations aim to go beyond basic service level agreements (SLAs) and offer a high-performing solution, constantly ask what the organization is missing from a data perspective and how it relates to business initiatives. Asking that question from the collection, correlation and triage perspective is critical.

    Step 3: Identify Trends. Organizations need to examine historical data to identify trends and take action before issues arise and customers complain. Establish alerting thresholds by outlining what a normal day looks like from a monitoring performance and customer perspective, and then identifying examples of what makes an abnormal day. This ties in with managing infrastructure inventory and understanding safety thresholds for each of the components that potentially could impact the service offering. It’s critical to communicate these findings with teams and business-line managers to prevent service delivery problems from happening and optimize offerings based on identified trends.

    Step 4: Notify and Act (Automation). In manual mode, a notification is delivered and then the team reacts. But teams are continually pushed to do things faster, and automation can help. To get to that point, organizations must understand where best to add more automation. How do you gather the right telemetry that delivers consistent answers that the machine can operate rules against, and do you leverage an automated process or notify a person? The desire to improve to a faster process requires a shift from manual to automated practices.

    Step 5: Predict (What-if Analysis). If you don’t take the time to go through the first four steps in a methodical way, it’s very difficult to reach this final step without always remaining in a manual mode. To balance costs and availability, it’s critical to discuss with the executive team how to predict customer service consumption (revenue) against the amount your services are going to cost the business.

    For example, developing a service that can alert customers of a potential service disruption due to disk space can be accomplished only by clearly setting business goals, then using the right metrics and events platform leveraging a time series database. In this case, the business goal is to ensure no service disruption due to failures such as low disk space; the metric monitoring is for disk space for each customer instance with the appropriate threshold alert set off by an automated trigger; and the action is an email to the customer letting them know of the situation and what action they can take (reduce load or upgrade for more disk space). In this example, it is important to marry business logic with the monitoring practice to make service a successful experience for customers. Furthermore, this approach will help teams to not only predict user experience but also help to predict OpEx and CapEx better into the future.

    That’s the benefit of the ability to predict with the right monitoring approach—to make informed business decisions based on results. Customers appreciate that, and proactive support is much better than being reactive.

    Emerging trends such as microservices, containerization, elastic storage, software defined networking and hybrid clouds are pushing the boundaries of DevOps monitoring. The right monitoring plan can identify and resolve problems before they affect critical business processes and enables customers to plan for upgrades before outdated systems begin to cause failures and outages for users.

    About the Author / Mark Herring

    Mark Herring is CMO at InfluxData. He is a well-rounded silicon-valley executive with proven experience in taking complex technology and making it understandable to the broader audience. His deep developer roots are never far from his mind as he looks at trends and asks the tough questions on whether the technology is here to stay or just another fad. Currently Mark is Chief Marketing Officer at InfluxData. Follow him on Twitter and connect with him on LinkedIn.