devopsy.kinsta.cloud

Author: Ariel Assaraf

  • Using Observability to Create a Smart Data-Driven Culture

    Using Observability to Create a Smart Data-Driven Culture

    The first words that come to mind when talking about smart technologies are often artificial intelligence, machine learning and automation. While these smart technologies play a crucial role in smart-driven cultures, they are not the sole factor in making workplaces “smart.” Organizations that have a smart-driven culture, or data-driven culture, also promote innovation. An innovative organization actively encourages creativity, exploration and the pursuit of new ideas, continuously developing and implementing groundbreaking solutions, products and processes. Innovative companies with a culture of learning and growth are often seen as desirable employers and have high employee retention rates, attracting top talent.

    Let’s go over the benefits of a smart-driven culture and how an observability platform can help your company become data-driven in five steps.

    Why is Smart-Driven Culture Important

    You can increase efficiency and productivity among your employees by fostering a smart-driven culture. Smart technologies are able to automate repetitive tasks and streamline processes, which helps free up workers’ time and focus on other issues instead.

    A data-driven culture can also drastically improve decision-making on a company level. Your organization will begin to strategize and act on objective insights rather than guesswork or assumptions. Through collecting, analyzing and interpreting data, you will not only be able to make informed decisions but also know how to optimize processes and mitigate risks.

    Use data to better understand your customers’ preferences and improve your products to meet their demands more effectively. For example, leverage smart technologies and create high-tech touchpoints to tailor each of your customer’s experiences. A culture that encourages continuous learning, experimentation and adaptation is ready to navigate market shifts, changing customer needs and emerging technologies.

    How to Create a Data-Driven Culture in Five Steps

    According to the 2023 Data Analytics Survey conducted by New Vantage Partners, 79.8% of data leaders identified people, business processes and organizational alignment as the principal challenge to becoming data-driven. These folks cited cultural issues, including organizational receptivity to change, communication and organizational alignment, as the greatest obstacles to realizing business value.

    One way to overcome these challenges is by harnessing the power of observability, building data literacy and organizational transformation. Observability refers to the practice of gaining real-time insights into systems, applications, and processes through the collection and analysis of relevant data. Real-time insights will enable your organization to make data-driven decisions, understand your customers better and quickly identify areas of optimization to improve your products.

    These characteristics are fundamental to smart-driven companies, enabling them to stay competitive and drive innovation in the market. Take the following five steps to ensure you’re maximizing the potential of observability and paving the way toward becoming a smart-driven organization:

    1. Define Clear Goals

    Start by defining what a smart-driven culture means for your company. Write down clear outcomes that you want to achieve with a new culture, and identify the specific areas where observability can make a significant impact on your team.

    Focus on areas where real-time insights derived from observability can generate substantial business value. Set SMART (specific, measurable, achievable, relevant and time-bound) goals that can be tracked over time. For example, you could aim to reduce website loading time by 20% in the next two months in order to increase customer conversion.

    2. Implement Robust Observability Tools

    Evaluate and choose observability tools that align with your organization’s specific needs and goals. Look for tools that offer features such as real-time monitoring, log management, distributed tracing, advanced analytics and visualization capabilities.

    Consider factors like scalability, ease of use, integration with existing systems, and the ability to handle diverse data sources. Remember that the availability of accurate and timely insights promotes a culture that values data and encourages performance for achieving business objectives.

    Ask your IT department to help set up your observability tools to gather the relevant data. For our previous example goal of improving website loading times, this will mean gathering real-time data on website performance, such as server response times, page load speeds, and user interactions.

    3. Promote Data Literacy

    Your observability tools are of little help if no one understands how to use and derive helpful insights from the data you gather. Educate all of your employees on good data practices and the role of analytics in decision-making.

    Provide training and support on the data tools you have chosen and encourage your employees to understand, interpret and utilize data effectively in their day-to-day work. For instance, in meetings, you can ask for data points to back up business decisions.

    4. Encourage Collaboration and Knowledge Sharing

    Creating cross-functional teams and encouraging collaboration among different departments, such as IT, operations and analytics, is crucial for the successful implementation of observability practices.

    Taking an interdisciplinary approach allows for a better understanding of the systems and data within the organization. This can lead to the identification of optimization opportunities, help break data silos and practice collective problem-solving as a result.

    5. Iterate and Improve

    Embrace a culture of experimentation and continuous improvement by using observability data to identify areas for optimization and innovation. Encourage teams to test hypotheses, implement changes and measure the impact, fostering a cycle of learning and refinement. Track progress against the goals you set at the beginning and adjust your trajectory if needed.

    Coming back to our goal to improve website loading time. Now that your interdisciplinary team implemented some changes and reduced the loading times, you can assess the impact on user experience and business metrics, such as bounce rates, conversion rates and customer satisfaction.

    Becoming a smart-driven organization will grant you a great competitive advantage. You will make better and faster decisions, employ the best talent, maximize their productivity, and provide an enhanced experience for your customers.

  • Is ChatGPT Helping or Hurting Your Developers’ Productivity?

    Is ChatGPT Helping or Hurting Your Developers’ Productivity?

    ChatGPT’s ability to translate natural language into working code has sparked tremendous interest in the programming community. Developers are exploring ways to use ChatGPT to reduce redundant tasks, from creating code snippets to analyzing and debugging programs. This also allows them to focus on more complex tasks like data analysis and monitoring with the help of a strong full-stack observability platform. Despite ChatGPT’s impressive capabilities, developers and their employers are split on whether the AI chatbot significantly boosts developer productivity.

    We will explore how to use ChatGPT to benefit your developers’ productivity and go over the advantages and disadvantages of ChatGPT to help you make the right decision for your business.

    How to use ChatGPT for Better Productivity

    ChatGPT has quickly become the most popular intelligence chatbot, with over a million user signups in its first five days and over 100 million in the first two months. Here are a few ways ChatGPT, developed by OpenAI, can help boost productivity in your organization.

    Improve Their Code
    Sometimes manually written code snippets consume additional CPU resources. Using ChatGPT, developers can optimize their functions for better performance. Additionally, ChatGPT can also write test cases and spot security vulnerabilities in their code, helping developers work faster.

    Simplify Documentation
    ChatGPT can give a simple explanation for the layman, avoiding technical jargon to a large extent. For instance, a developer at Twitter fed ChatGPT their code and it responded with a pretty detailed explanation. Using ChatGPT to generate the first draft of a company’s internal documentation would make knowledge transfers easier. New hires could quickly get up to speed on the company’s codebase. Further, ChatGPT could also be used for external-facing documents where other developers can learn how to use the company’s APIs and frameworks.

    Test and Understand Code
    ChatGPT can write automated tests for code that is fed into it. Although many developers consider testing boring and stressful—and sometimes a waste of time—it’s a critical step in ensuring that published applications are bug-free.

    With ChatGPT, developers can output reliable, well-tested code much more quickly and easily. ChatGPT can also give high-level and line-by-line explanations of code and even teach developers how to use external modules and APIs in their own code, helping them build much faster.

    Debug Code
    ChatGPT makes a great debugging tool for developers. It can take in error messages and output information on the likely cause of the error and how to fix it. This is typically a lot more time efficient than browsing forums and blogs for a solution.

    The tool can also suggest helpful code alternatives. Many developers claimed that ChatGPT can be much faster and less mentally challenging than the conventional approach of searching the web for a solution.

    As the CEO of Replit, Amjad Masad, put it, “ChatGPT could be a good debugging companion; it not only explains the bug but fixes it and explains the fix.”

    Code Comprehension
    ChatGPT’s terabytes of training data come from various sources, including common crawl and WebText2. This training data makes it easy for ChatGPT to simplify explanations of complex pieces of code for developers. Therefore, developers can save time deciphering any code’s underlying logic and structure.

    For instance, Stanford instructor and well-known developer Matt Harrison used ChatGPT to build a simple graphic design tool. He explained, “I’m using ChatGPT to create a GUI program for drawing lines. I haven’t used QT [a development framework] in years, yet I can throw this together really quickly.”

    Generate Documentation
    Writing code documentation is often considered a mundane task by developers. With ChatGPT, developers can significantly cut down on the amount of time needed for documentation. For instance, they can expedite the creation of troubleshooting guides, tool user manuals and test cases by asking ChatGPT to perform code analysis and generate detailed technical reports. Accelerating this documentation process allows for easier onboarding of new developers and faster product release times.

    Writing Tests
    ChatGPT can analyze code snippets and automate testing for them, saving developers lots of time and helping them output reliable code. Test coverage is an essential metric for most enterprises, and ChatGPT could help streamline the process.

    ChatGPT Disadvantages

    While many developers have praised ChatGPT for increasing developer productivity, others have criticized it as unreliable. Let’s understand some key areas where the tool falls behind.

    Outputting Buggy or Unusable Code
    Developers have complained that the chatbot would sometimes output plausible-looking code that doesn’t work. This might result from misunderstanding the user’s requirements or simply being unable to come up with a correct answer—then faking it.

    Programming Q&A forum Stack Overflow even issued a temporary ban on ChatGPT-generated answers. The ban was because the chatbot generated an influx of answers whose average accuracy was pretty low. ChatGPT sometimes uses nonexistent commands and outdated modules in its code, so while they theoretically might make sense, they don’t work in practice.

    Writing Outdated Code (and Forgetting Previously Written Code)
    Since ChatGPT’s training data cut off in September 2021, the chatbot has no knowledge of the recent developments in programming languages and modules after that. As a result, ChatGPT often needs to use outdated commands, modules and paradigms in its code.

    ChatGPT also occasionally forgets the context within which it is working, affecting code output. Some users have reported that ChatGPT sometimes forgets about the task at hand and begins generating incompatible code.

    Security Concerns
    Security concerns are the biggest drawback to having developers work with ChatGPT. Open AI uses the data users input into ChatGPT to train the underlying AI model.

    Research by cybersecurity firm Cyberhaven has shown that 11% of the data employees input into ChatGPT is confidential. This presents a significant risk, as such data can be used in the chatbot’s next responses. Corporations like JPMorgan and Verizon have even gone as far as to block employees’ access to ChatGPT because of these concerns.

    Lacks Problem-Solving Skills
    A developer’s job is to understand the business problem and develop customized solutions to address it. Although AI/ML tools like ChatGPT help developers code faster, they can’t help in problem-solving or implementing a software application.

    Furthermore, when developers become overly reliant on AI-generated code, they’re discouraged from upskilling on new programming frameworks and languages. This can hurt the organization in ways that might not be evident immediately.

    To ChatGPT or not to ChatGPT

    As Google’s head of developer experience Bret McGowen said, “While AI will transform software development in many ways, human judgment, intuition and creativity will remain essential to building great products.”

    While ChatGPT may not be a perfect solution for every aspect of software development, it can be a useful addition when combined with human problem-solving skills. However, companies should be careful while allowing the usage of AI in day-to-day processes. Guidelines educating in-house developers on the fair usage of AI are a must to protect sensitive information and maintain a high-quality control process.

  • Using Chaos Engineering to Build Resilient Systems

    Using Chaos Engineering to Build Resilient Systems

    Over the past decade, chaos engineering has become one of the most popular approaches in DevOps. It’s uniquely adapted to complex cloud-based systems and has the potential to succeed where more conventional approaches may not.

    Chaos Engineering Explained

    Traditionally, DevOps teams worked with systems hosted on single servers. They would analyze logs with a fine-toothed comb, looking for potential anomalies. They implemented automated tests to ensure the system worked to spec.  They employed root cause analysis to debug applications.

    Modern systems use cloud-based distributed architectures and are much too complex for this approach to be successful; they contain hundreds, and sometimes thousands of instances.  In many cases, their behavior can be chaotic, making it nearly impossible to predict.

    Because of this, modern systems need to be designed for resiliency from the outset. Chaos engineering was developed to foster resilient system design by deliberately anticipating failure modes.

    Two Steps to Chaos

    Here’s how it works. After characterizing a state of “health” for their system, DevOps teams perform a chaos experiment. Just like physics or chemistry experiments, chaos experiments involve setting up two versions of a system: the experimental version and the control.

    The experimental system is then subjected to a barrage of virtual kicks and punches which could range from injecting faults to disrupting a data center. DevOps teams then pore through the resulting logging data, attempting to understand how the chaos experiment affected the state of the system.

    Chaos engineering has two major differences from conventional DevOps practice. First, while conventional forms of testing supply a controlled input and test for a predefined output, chaos experiments are far more open-ended. They’re designed from the outset to expect the unexpected, requiring analysis of the whole system to give novel insights.

    Second, they are performed on live systems. This makes chaos engineering controversial, as its experiments have the potential to affect real users. Still, it makes it much more powerful than alternative approaches. Chaos engineering encourages resilient design by taking genuine risks with systems in the wild.

    Why Companies Embrace Chaos Engineering

    Chaos engineering has been around for over a decade. In that time, it’s been adopted by several of the biggest names in tech. Netflix, the company that pioneered the approach, is still the most well-known for their use of fault injection.

    In 2008, Netflix was moving from an in-house data center to the cloud. They knew that, as they expanded, their application would undergo massive horizontal scaling.  Because any one of its thousands of instances could fail at any moment, it needed to be resilient.

    They created the first chaos experiment, Chaos Monkey. This worked by randomly turning off instances in Netflix’s live application to check it was fault-tolerant. This practice sounds dangerous, but it encouraged Netflix engineers to design resilient system architectures.

    Chaos Monkey sounds bananas, but it worked. In its first five years of operation, there was only one failure which was promptly fixed by an on-call engineer. That failure was caused by Chaos Monkey itself.

    Facebook is another big company to adopt chaos engineering. They created Project Storm in the wake of 2012’s Hurricane Sandy.  Although the hurricane itself didn’t affect Facebook, it served as a wake-up call. A storm hitting one of Facebook’s data centers could wreak havoc.

    In 2014, Facebook took the step of turning off traffic to one of their data centers. Although no end users were affected, traffic loading went haywire on all sorts of different subsystems.

    Once Facebook had successfully identified this weakness in their system’s resilience, they could preemptively design traffic management software that would be resilient in the face of a live data center shutdown.

    As more and more companies use cloud computing and hyperscale data centers, chaos engineering is likely to become an essential feature of DevOps strategy.

    Applying Chaos Engineering In Your Company

    Chaos engineering is the ideal approach for complex, distributed systems. Chances are, your DevOps team either runs their application in the cloud or is planning to migrate to it.

    Serverless computing, in the guise of systems like Kubernetes and Heroku, is popular with an increasing number of organizations. Now it’s common for systems to contain hundreds or even thousands of instances, hosted in massive data centers.

    So far, we’ve only seen chaos engineering from 10,000 feet. It’s time to look at how you can make it work on the ground.

    First, we need to understand the roles observability and monitoring play in chaos engineering.  To successfully implement chaos experiments, there needs to be a high level of system visibility.

    Dealing with Data

    Marius Moscovici outlines four strategies for extracting valuable insights from data. He discusses the importance of detecting anomalies in your data stream, something a chaos engineer needs to be good at.

    Whether it’s the haywire traffic in Facebook’s Project Storm or the deactivation of Netflix instances, chaos experiments rely on the ability to diagnose unusual patterns in data.

    1. Profiling Normal System Behavior

    In order to detect anomalies, you need to have a good idea of what your system normally does. In their book on chaos engineering, Netflix used the analogy of human vital signs. Just as doctors know that 98.6°F is a normal human body temperature, their IT counterparts need to characterize a “normal” state of “health” for their DevOps system.

    Many log analysis and observability tools in the market offer machine learning capabilities to help achieve this.

    2. Detecting Anomalies Across The Board

    Anomaly detection should be integrated into every step of your data analysis. You need to be able to detect anomalies across the board, not just in places you happen to be looking at. Because chaos engineering, unlike more conventional forms of testing, is designed to reveal new and unexpected information it’s vitally important to have a completely open data strategy.

    3. Data Scientists On the Lookout

    Always be on the lookout for potentially anomalous behavior. Chaos engineering will require your company to take its DevOps strategy to a new level, both in terms of your team’s mindset and the sophistication of their logging tools.

    Because chaos engineering requires you to analyze all available data without making assumptions, it’s necessary to leverage machine learning to extract patterns from your logs.  Chaos engineering requires a scientific mindset and an appreciation of the power of data.

    4. Create Alerts

    Finally, make sure that anomalous behavior is being flagged and that the team is aware when it occurs. If possible, dynamic alerts should be used to avoid false positives and alert fatigue which is common with static thresholds.

    Engineering with Chaos

    More and more organizations are migrating to the cloud and building highly distributed application infrastructures. DevOps teams are being forced to confront chaos and complexity on a daily basis.

    While the complexity of modern IT systems has necessitated chaos engineering, it has also enabled it. In a chaotic world, chaos engineering brings the order of an engineering approach.

  • How Logs Can Empower Your CI/CD Delivery Pipeline

    How Logs Can Empower Your CI/CD Delivery Pipeline

    Change is the currency of software success. This need to change and grow is the basis for continuous integration and continuous delivery (CI/CD). Each year, new industries embrace this philosophy of constant change, but without the necessary data underpinning their decisions, every deployment is a risk. When you enrich your software delivery with log data coupled with best practice log management, you mitigate this risk and enable rapid, safe change in your organization.

    Logs are often the most common source of information about an application’s behavior. They provide context as well as result, opening up far more information than a simple count metric will. Despite the immense insight that they can offer, organizations regularly overlook them. Many leave their logs in files on servers, far away from the CI/CD pipeline that needs them. It doesn’t have to be this way; with some best practices, every part of your business can collect, analyze and utilize the hidden knowledge within your logs.

    Add Life to Log Data With JSON

    Logging in an unstructured format dramatically increases the complexity of detecting patterns in your logs. By logging your application output in JSON, you open up the possibility of analysis of all of your logs. This gives you a top-down view of your entire system while still maintaining readability. This empowers your CI/CD pipeline by allowing you to query and filter your logs, which then enables you to focus in on the problem and precisely diagnose any unwanted side effects of your latest change.

    Create Actionable Alerts

    If alerts are defined and channeled correctly, have context and can be interpreted easily, they will be actionable, add context and offer greater value. It is difficult to provide context when a single metric exceeds a threshold, but log lines are more sophisticated. They can include additional information to provide context and allow us to pinpoint the source of the issue fast. This is an essential capability for a company embracing CI/CD.

    Prioritize Your Logs

    Structured logs often come attached with a severity, which indicates how seriously we should investigate an event. An INFO level log signals business as usual, but an ERROR demands immediate attention. Once your logs have these tags, you can make intelligent, automated decisions that will automatically respond to unwanted changes in your system.

    Benchmark Each Version to Understand ‘Normal’

    “Normal” is a complicated idea when working in a microservices architecture. It may be safe to ignore a minor slowdown of a single service, but an unfortunate combination of events can lead to disaster. Benchmarking enables us to see these disasters well before our users. Benchmarking is the act of recording what the normal behavior of an application is. After each deployment from your CI/CD pipeline, compare your new behavior with your previous benchmark. If something is out of the ordinary, logs provide the information you need to act decisively. They provide an outstanding baseline signal for your benchmark.

    Analyzed Logs Can Level up Your CI/CD Pipeline

    Building a CI/CD pipeline is not the most difficult part. Neither is the deployment of new features. In fact, the greatest challenge facing any organization wanting to deliver their changes via a CI/CD pipeline is observability. With a best-practice approach to the preparation, curation and analysis of application and system logs, we can overcome this challenge and confidently change at a pace that propels us to the forefront of our market.

  • How to Overcome Challenges With AWS Lambda Logging

    How to Overcome Challenges With AWS Lambda Logging

    Monitoring AWS Lambda can be a complex and potentially costly endeavor. Here’s what you need to know to stay on track and on budget

    Organizations are already experiencing a shift toward serverless cloud computing. According to the O’Reilly “Serverless Survey 2019,” around 40% of respondents said they had implemented serverless architecture. Forrester “Global Business Technographics” Developer 2019 found that 49% of companies are using it or planning to use it in the next 12 months.

    AWS leads the way in this space. According to Serverless.com, 96% of respondents choose AWS Lambda over Azure Functions and Google Cloud Functions. AWS Lambda helps us to design a lightweight and robust application architecture that can be built and deployed without any hassle. However, nothing comes for free. There are several, potentially costly, complexities when monitoring AWS Lambda. Fortunately, there is an array of solutions.

    Logging

    By default, AWS Lambda’s stdout and stderr messages are sent to CloudWatch Logs, where every function has its log group. A log group is a collection of log streams (sequence of log events) that can be stored as long as you want. Storage of logs costs USD 0.03/GB and data ingestion costs around USD 0.50/GB.

    It is common practice to break up your Lambdas into single responsibility functions. This means, for example, if you need one Lambda to read a certain object and another to write it, you’re setting up two separate functions. However, this also means two log groups.

    This creates a challenge when you want to analyze your application’s output log. There is no direct way to log queries across multiple log groups. In response to this, AWS enabled CloudWatch Log Insights to conduct cross-log querying. While this brought many of the improvements users wanted and needed, analyzing multiple log groups together meant a higher cost for queries. This prohibitive cost model encouraged many to look to external services.

    Tracing

    AWS X-Ray enables you to trace the underlying components of your application and helps you to dig out the fundamental reason behind distributed issues. It has quickly become an essential troubleshooting and optimization tool for engineers.

    For example, cold start latency can be problematic for AWS Lambda functions with slow startup times. Although you can counter this by enabling Provisioned Concurrency, you will need to pay extra.

    If you have budget constraints and still want to resolve the issue, X-Ray might help you eliminate or at least optimize your application by removing the bottleneck. It lets you know, among other things, if your application is consuming more time during initialization or invocation. You can find X-Ray SDK on AWS Lambda for Go, Java, Node.js, Python and .NET.

    Debugging

    Traditional architecture and serverless architectures are as different as day and night. Serverless requires changes across the software engineering spectrum, including debugging.

    A serverless application is a combination of various services or resources within infrastructure, which can include tools such as AWS API Gateway, CloudWatch and SNS. Such distribution of web services helps avoid a single point of failure (SPOF), but this design creates a problem. Debugging or replicating an issue on a local machine is not always feasible, and debugging a production application is always a dangerous proposition.

    Moreover, the abstraction of AWS Lambda makes debugging complicated, as you don’t have direct access to the underlying OS. Hunting down dispersed logs adds to this overhead. As your serverless infrastructure scales, it becomes difficult to identify the root cause.

    To combat this, developers can use AWS native tools such as CloudWatch, X-Ray or Serverless Application Model (also known as SAM). Any of these tools on their own can be difficult to use; together, however, they form a powerful debugging suite.

    Outside of the AWS-native toolset, some tools provide further assistance with debugging. For example, the Serverless framework, which is an open source tool to develop and deploy serverless applications over AWS, lets you see your live application’s function log, for example:

    Monitoring

    Robust monitoring ensures the smooth operation of an application and helps to detect issues quickly. In traditional infrastructure, metrics only appear from a handful of sources, so collecting them together is less of a challenge. In a serverless architecture, this problem is much more complex.

    AWS gives you the freedom of converting CloudWatch Logs to metrics. These metrics can be filtered, measured and, when a threshold is crossed, alerted on. You can define a metric filter by choosing a log group created by your serverless function and target the log stream. This enables you to create a visual reference for your log events. Every custom metric you create will cost $0.30/10000 metrics per month.

    Metric filters can be a powerful tool, but it can also be a bit overwhelming to set up everything, and there are certain bottlenecks as well. For instance, you can’t parameterize metric names, which forces you to create a filter for every custom metric you want to track. There is also a hard limit of 100 metric filters per log group, which restricts you as your application evolves.

    Other than this, AWS resources are tightly coupled with each other, which makes AWS-native monitoring tools a fine choice when you need to monitor individual AWS resources or Lambda functions. To get an infrastructure-wide view and to monitor other third-party tools, a centralized logging approach is necessary to take full advantage of the log data.

    Conclusion

    There is no denying that serverless tooling quickly unlocks high availability, scalability and cost optimization. To make the most of this new paradigm, engineers need to build a robust and reliable monitoring system. The ability to manage log data streaming from many components, detect and trace issues on the fly, and deliver accurate insights to keep on improving the application is crucial to remain online and competitive.

    While serverless solutions give access to many of the technical features we need, the dangers of vendor lock-in and the power of third-party providers must be considered if you want to scale your organization and its infrastructure smoothly.

  • The Top 5 Challenges to Overcome with Heroku Logging

    The Top 5 Challenges to Overcome with Heroku Logging

    Heroku can be a very powerful tool for monitoring the massive amount of data generated by your applications, systems, APIs and add-ons.

    One thing you may have discovered with Heroku logging is that making sense of data within your log files can be time-consuming. When you’re running large numbers of applications which process tons of data each day, the sheer volume of data that appears in the logs can seem overwhelming.

    Managing Massive Log Output 

    Due to the sheer amount of data that is gathered, it’s important to find ways of making the process a little bit easier, limit information to what is useful and organize what you have into digestible pieces.

    Parsing with Drains

    Using Logplex and log drains can help you filter out large chunks of your logs. Syslog drains will help you move any system messages to another server for archiving purposes.  HTTPS drains can help you better take control of your system messaging for sending it to third-party apps for analysis. 

    By assigning clear indicators that identify the application, system or API that creates a log, these can be streamed into channels that can be better understood in relation to their given purposes. 

    Log Retention Period

    While we may think that more information is always good, there is a limit to the amount that we can process. Some log data becomes meaningless or redundant after a certain length of time. It is wise to set realistic log retention periods on data based on business needs.

    Limit Verbosity

    Don’t log everything. Turning on debugging is appropriate during development and if there is a specific problem that needs investigating, but in most cases, these shouldn’t be stored. 

    Complexity in Stream Patterns

    The challenge with large amounts of data is that there may be subtle differences between successful activity and erratic behavior. The details may be small, and essentially invisible to the naked eye unless examined carefully. 

    Use Regex for Identifying Patterns

    Through the use of log analytics tools, you can set these to pick up specified regex patterns to subtle changes or variations in your logs that can provide meaningful insights. 

    Different Log Data Needs

    You may have people with different needs who wish to make use of the information you gain in your Heroku logs. While you have data that is of interest to developers and QA engineers, there are other tech-savvy users who may wish to gain business intelligence (BI) from the massive output of data, to be able to identify key performance indicators (KPI).

    Analytics Tools

    Using intelligent add-on analytics applications can help by filtering and sorting information into usable chunks that can be understood by users in different roles. By accurately filtering, you can provide intelligent real-time alerts for a specific application or system behavior to those who may wish to make use of it. 

    Application Issue Forecasting

    One of the best potential uses of your detailed Heroku logs involves the opportunities to spot patterns of problems that have occurred in the past and use them to predict problems that may occur in the future.

    As an expert, you may be able to spot patterns in your logs, however, even the smartest and most attentive engineers will not be able to manually define every scenario worth detecting.

    Occasionally patterns will occur that are only detectable through the rote process of comparing and contrasting working behavior with non-working behavior. Often, the subtleties are too small for a human to be able to notice the difference.

    Machine Learning Assistance

    Naturally, utilizing ML-powered techniques can help catch the unknowns. By investing in ML, using some of the better libraries or choosing a Heroku logging add-on with these capabilities, you can monitor your applications and infrastructure with less effort. 

    Securing Sensitive Data

    With the rapid pace of CI/CD, a large amount of activity regularly flows through your pipelines from development into production. Throughout this process, it’s possible for sensitive data to start appearing in logs. While your teams may be diligent about keeping control over OAuth credentials, security keys, authentication tokens and more, nobody is perfect. Sensitive security credentials can accidentally make their way onto production servers. Sensitive customer data can also accidentally find its way into public logs if mistakes are made.

    Credential Detection and Security Alerts

    Logs can serve as a saving point for the identification of sensitive information. Using a parsing engine, these can be immediately redacted for compliant log management. By monitoring your log files for any patterns which could include secret keys or other potential security breaches, you’ll be able to continue ingesting critical log data without increasing business risk.

    Heroku Private Spaces

    With Heroku Enterprise, you can make use of Heroku Private Spaces which are an isolated network of private apps and data services, which will help you keep sensitive data separate from public interference. If you are handling a large amount of sensitive customer data, ensuring that any information gets filtered into these places will help you remain compliant with GDPR and other privacy-related regulations.   

    Conclusion

    Heroku can be a fantastic solution for monitoring the activity which goes on in your cloud clusters.  However, unless you have solid analytical tools in place, you may find yourself spending a lot of time wondering how to make use of this sea of data that has been collected. Consider using an add-on as an analytic layer to help spot patterns, organize data better and create smart alerts to make sure your operations continue to run smoothly.

  • Why DevOps are Keen on Open Source Logging

    Why DevOps are Keen on Open Source Logging

    With the advent of DevOps, we are now moving toward more collaborative workflow models, where the integration of development and operations teams make it easier to quickly move development projects into production, and with fewer roadblocks. However, within DevOps, many of our activities are not as clear-cut as they were in previous siloed methods. While previous methods made it easier to keep track of activities and processes, these systems were not particularly efficient.

    In the DevOps model, moving software from development to production is much faster. But, we need to be able to identify actions that occur in our increasingly complicated applications. We now have many chefs working at the same time, and visibility suffers.

    Logging helps us better understand applications throughout each stage in the process so that we can identify problems or larger issues that might arise already during the development process. This is particularly true within CI/CD environments where items are regularly pushed into production. We need methods for identifying not only if there are critical errors, but also the ability to attribute the errors to specific versions throughout the cycle.

    To deal with the increasing complexity and decreasing visibility, DevOps identifies complex issues using logs to keep track of services, APIs, containers, infrastructure issues, network activity and also security related logs.

    Log files provide a tremendous amount of useful information about what goes on beyond the CI/CD pipeline and also identify issues that directly impact customers. Good log analysis tools make it possible for DevOps teams to get much closer to the end user. And open source is the preferred starting point for many organizations.

    Why Open Source?

    Lack of Lock-in 

    Many DevOps teams make use of a wide-range of tools and find that they can work much better without vendor lock-in, which can be caused when using proprietary stacks exclusively. This makes it far easier to make any needed changes as the need arises, and not be reliant on individual vendors. This also helps avoid many risks, such as vendors disappearing, ending support, changing pricing models, etc.

    Widely Used

    Within CI/CD, there are many commonly used open source tools, which makes it easier for new members to be on-boarded. There is already a widely accepted suite of tools that almost all DevOps teams use. It’s hard to imagine working without Jenkins, Docker, Kubernetes, Git, etc.

    Regular Skill Development

    Open source work ensures that DevOps teams are actively learning and directly interacting with software and systems. It becomes a way of ensuring that skills and abilities of teams are continually staying updated. This can translate into better troubleshooting and problem solving skills. When working with closed systems, there’s considerably less opportunity for knowledge advancement, as closed systems tend to encourage rote work with existing software rather than skill development.

    Quality

    Fears about subpar open source software are largely unfounded, particularly if the projects are widely used and have a large community. There are far more developers working on a project in the open source community than in proprietary systems. Closed systems can lag behind, and there may actually be bad code but that is hidden due to its closed-source nature. The ability to make modifications to create functionality beyond that which already exists in closed source systems, makes open source extremely appealing for many DevOps personnel.

    Some Drawbacks

    It’s important to note there are a few drawbacks when attempting to work entirely within open source environments. Many resources used are not open by nature–for example leading clouds are decidedly not open source in nature. Individual users (outside these companies) have much less control over how they actually operate. However, if we look at the toolkits that we described above, we can think of open source tools as needed resources for working in and around these environments.

    Another drawback to consider is the actual costs of managing open source software versus a fully-managed service such as Coralogix, for example. Many times, teams will start with the self-managed approach and move to a managed service when faced with scaling and complexity issues as the organization grows. 

    Types of Features to Look for in Open Source Logging Models

    When setting up to work with open source logging tools, it’s important to understand there is a range of different types of models available, which can work better or worse depending upon needs and capacities.

    • OpenAPI: provides a useful interface into many different systems. It is language agnostic and can generate clear documentation of methods, parameters and models. In most cases it will work within RESTful interfaces, but will also work with other protocols such as SOAP.
    • Open Standards: Even if a tool is not open source, typically these should follow a set of standards to make sure that different pieces can talk to each other. OpenAPI is one example, but not the only one.
    • Federated Model: This is a broader type of open source model which is like many default open tools, but allows a much wider set of input and may allow individual users to provide input and maintain some control over local development areas. This means that aggregation, processing and control remain local. However there’s still a central organization which will be able to collect summarized or complete code. The advantage of a federated model is it increases flexibility for individual teams, while still contributing largely to the project as a whole. 

    Elasticsearch, Logstash, Kibana (ELK)

    One of the most popular open source logging stacks is known as ELK. It is particularly powerful and effective for the purposes of those wishing to create a clear aggregation and understanding of their log files, with the ability to find both problems and solutions quickly. 

    ELK is a combination of three tools.

    • Elasticsearch: a popular NoSQL Lucene search engine.
    • Logstash: a pipeline system which will take data from logs, ingest it and transform it into a data store (such as Elasticsearch), so that it can be searched and analyzed.
    • Kibana: a tool which makes it possible to create clear visualizations for Elasticsearch to appear in a human-readable format. 

    What ELK provides is a single place for all of your data, where it can be stored, searched and analyzed in a method. It is useful for DevOps teams wishing to make use of the vast amounts of data stored in log files. 

    Upfront expenses for open source tools like ELK are minimal; the software itself is free. However, this does come with a caveat. Professional management of ELK tools is required, otherwise costs can grow quickly out of control. 

    Generally, for low volumes of data, one can manage ELK for as little as a few hundred dollars a month on a platform such as AWS. However, without careful management, the cost can grow exponentially for large amounts of data.   

    Work required for ELK involves:

    • Setup, which may take some time but that’s typically not a problem for many ops teams—and is a project many will enjoy.
    • Maintenance of elastic search clusters can be quite a bit of work—several hours per week simply solving issues or resolving downtime. This can vary based on the size and growth of clusters. The more activity you have occurring, the more time you will need to spend maintaining your logs.

    FluentD

    FluentD is an open source alternative to Logstash which can collect, parse and transform data for further analysis. It has features which may be particularly helpful if you are running your logs in a Kubernetes environment.

    One of the drawbacks of Logstash is that it is written in JRuby, which means it requires a Java runtime for each implementation. If you are running many different microservices in Kubernetes, this can result in a large amount of your memory being taken up.

    Because FluentD is developed in CRuby, it doesn’t require as much memory to use with each new pod creation. This can help solve some memory allocation issues which could cause your logs to slow down your applications.

    FluentD also has a wide range of plugins, which can cover pretty much any use case that you come up with. Another advantage is that its tag-based routing makes it slightly easier for developers to work with than Logstash. 

    What to Log?

    With continuous integration, there are many things that merit monitoring. These can include everything from checking the frequency of merge conflicts in Git, or whether pulls are preceded by pushes prior to any other pulls.

    Are you receiving a high number of compiler warnings? You can identify code that has a strong likelihood of breaking builds, change your unit testing procedures and implement better automated checking in Jenkins.

    Much of this data is worth visualizing in Kibana for analysis. Deciding what specific activities to log can be a bit involved. Areas that are a good idea to log include:

    • Diagnostics: For example, understanding the causes of repeated errors.
    • Auditing: Track how well errors in code have been fixed, and whether they succeed or fail a second time.
    • Profiling: For example, get an idea of how long it takes for certain parts of code to execute, for the purpose of identifying whether this has any impact on customer experience, and identify areas for improvement.
    • Statistics: Code performance, such as execution time or memory consumption.

    Conclusion

    Gaining complete visibility or your infrastructure will go a long way to make sure that your operation runs smoothly and that your CI/CD pipelines function at optimum efficiency. Open source tools remain an attractive option for DevOps because of its advantages and knowing how to get around drawbacks.

    Finding the right tools is, of course, only the first step, but if you’ve picked good architectures and have implemented them properly, you are already moving in the right direction to a more streamlined operation.