devopsy.kinsta.cloud

Category: DevOps Practice

  • Python’s Popularity Continues: Examining the Top Use Cases

    Python’s Popularity Continues: Examining the Top Use Cases

    Python continues to be one of the most popular coding languages in use. In fact, the TIOBE index of programming language popularity named it the 2018 Language of the Year. Its usage and adoption continue to grow and proliferate into additional use cases.

    Python’s popularity and growth are in part because it’s flexible and easy to use. Python has grown from a scripting solution for sysadmins to web development for programmers to the driving force behind machine learning. We’ve observed this growth since 1999, when we began to build engineering Python distributions. As we’ve seen the use cases grow, we’ve examined the top use cases by customers ActiveState has observed.

    Let’s look at the three most popular Python use cases within the traditional tech space: infotech, software and hardware.

    Infotech: IT modernization

    The IT team of a state legislature is challenged to be able to remotely manage thousands of user desktops. Its 147 representatives, as well as its hundreds of supporting personnel, work on-premises and/or remotely. One of the key tools that empowered the team was Windows Powershell. Unfortunately, Powershell is unsupported and no longer works under newer Microsoft operating systems.

    Rewriting any application that’s a cost center is always difficult to justify. As a result, rewrites are rarely budgeted, planned and resourced appropriately. But the alternative was much costlier in terms of capital and resources: dispatch IT personnel to physically install, update and manage each user’s desktop, on an on-call basis.

    It’s famously easy to get started and develop with Python. While there was no way around a rewrite of the desktop management scripts, the IT team could at least minimize the time and resources required. Python allowed Powershell script rewrites to be done incrementally in IT’s spare time. This translated into a significant lowering of costs, and reduction in the risk of a “big bang” rip and replace project.

    Software: Adding Extensibility to Legacy Applications

    A U.S.-based public company has a 30-plus-year history of providing a suite of electronic design automation (EDA) tools to their more than 5,000 customers worldwide. Its suite of tools historically was built in C++ and is packaged for deployment on-premises.

    Because each customer has their own instance, it’s natural for the customer to want to customize the suite of tools to their needs. The U.S. public company used Python to wrap some of its C++ APIs to expose key functionality, making their software solutions more extensible by customers. Ensuring Python was deployed properly at customer sites required including Python within their existing installer, rather than relying on their customers to install Python correctly.

    The company used an OEM distribution Python that added a standard version of Python within their application and deployed it, so all teams can customize and leverage the EDA tools as required. Plus, by deploying a standardized Python version across all customer sites and developer desktops, the company can decrease support costs and speed time to market by eliminating the need for legal department reviews. Further, using a third-party-standardized Python version eliminates the opportunity cost of managing one’s own Python build and can facilitate the discovery of issues, bugs and performance issues.

    Hardware: Automating Network Administration

    A Silicon Valley-based American technology leader develops, manufactures and sells networking hardware. A key enterprise demographic for the company is network administrators, who often are confronted with managing the vendor’s multiple devices across extended corporate networks.

    The challenge: Network admins aren’t typically developers. Yet, they want to improve their efficiency by being able to programmatically make configuration changes to multiple network devices at once. While sample scripts are easy for any network admin to use and modify, the vendor did not want to be held responsible for creating all the scripts its user base might require.

    Python was the solution because it is simple enough for any technical person to learn and start using effectively. The company chose a version of Python that allows their customers to quickly create sample Python scripts to cover the majority of typical network admin use cases.

    This choice also allows network admins to create Python libraries they can leverage to create their own custom scripts to manipulate the configuration, operational state or event data from network devices. Most importantly, the cost of administering enterprise networks is significantly lowered and network admins are dramatically more efficient with programmatic access to physical networking devices and virtual network resources.

    Looking Outside the Tech Space

    Other examples that round out the top 10 use cases for Python include:

    • Consulting services: Bespoke web development.
    • Insurance: Creating business insights with machine learning.
    • Finance: Data mining to identify cross-sell opportunities.
    • Retail banking: Flexible data transformation and manipulation.
    • Aerospace: Meeting software system deadlines.
    • Business services: API access to financial information.
    • Health Care: Predicting disease prognosis.

    Ongoing Innovation

    Python’s usage continues to grow due to its simplicity and flexibility. Developers clearly want to use it for a variety of reasons that surpass its creator’s original intent. To support this usage, organizations need to ensure their developers can do so securely, by overcoming open source vulnerabilities while still iterating quickly. It seems that use cases for Python are limited only to the imaginations of its growing, enthusiastic user base.

    — Bart Copeland

  • HybridOps: Driving Better Collaboration and Productivity in IT

    HybridOps: Driving Better Collaboration and Productivity in IT

    Over the past 10 years, enterprise technology has experienced an enormous amount of upheaval. Developers started self-servicing, leading to “shadow IT.” The DevOps movement and alignment with Agile methods started gaining popularity, and thought leaders started making statements about how DevOps is the future and ITIL/ITSM and other traditional practices are the past.

    With all these changes, it’s no wonder that the words “digital transformation” have become highly charged and central IT teams feel—not entirely without justification—that their budgets and influence are on the wane. As a result (and partly due to misplaced self-preservation and wanting to stay relevant), these IT teams push back against any kind of cooperation with more developer-centric teams, leading to low levels of digital sophistication and decreased productivity.

    Introducing HybridOps

    This is where HybridOps comes in. We at PagerDuty coined the term HybridOps following feedback from customers to describe the reality of digital transformation in large enterprises—which is one in which technology teams will have a variety of operating models, velocities, tools and practices.

    Rather than a dogmatic focus on velocity and stack-ranking teams according to how “mature” they are, HybridOps focuses on three principles:

    1. Learning culture: It doesn’t matter what operating model (DevOps, ITIL or something in between) teams are using, as long as they can meet the needs of the business and have a culture of continuous learning and improvement.
    2. Communication and empathy: Open communication, constant sharing of operational data between teams and empathy for colleagues are much bigger drivers of digital operations productivity than rigorous adherence to a methodology handed down from executive management.
    3. Autonomy: Teams should have the autonomy to choose and use the tools they prefer—as long as those tools allow for open data exchange with other teams. Instead of forcing teams to use tools they don’t need or want, a better approach is to implement technologies that can bridge data and written communication gaps between teams that need to collaborate.

    HybridOps is ultimately a pragmatic, incremental, big-tent approach to digital transformation. Our most successful enterprise customers use this approach to roll out their digital transformation efforts, as opposed to the top-down, “Send everyone to Agile training,” enforced-compliance approach. In our experience, using the forced approach typically means teams end up using the language and processes of Agile development practices but actually fail to collaborate with others or make a meaningful impact on becoming more customer-centric.

    Measuring HybridOps Success

    All this talk about HybridOps is well and good, but how can organizations measure success in a HybridOps world? It’s actually very similar to other areas of a mature business: Leaders need to establish behavioral norms and measure outcomes (instead of outputs) relative to business needs.

    For example, it’s unnecessary to benchmark the mainframe team against the web team with metrics such as the number of deploys per day, tickets resolved per month or even system throughput. After all, a mainframe batch process that runs twice a week isn’t necessarily less operationally mature than a serverless function that can return an instant response.

    In short, when it comes to measuring performance, it all depends on the operational context and needs of the business—and it’s best to establish some common metrics that prevent overwork and burnout and use them across all teams.

    Culture, Culture, Culture

    Finally, one last word on culture. HybridOps can only succeed when senior leadership creates a high-trust culture. Blamelessness and fostering a learning culture are key here. Open communication and empathy for colleagues are important cultural values underpinning HybridOps that leaders need to establish and reinforce continuously. If leaders use ownership and control of data as a way to name and shame others when something goes wrong, a HybridOps approach won’t work.

    Ultimately, HybridOps is fundamentally a kinder, gentler route to digital transformation—one that’s focused on meeting people where they are and how they’re used to work. It encourages data-driven process improvements free of blame, avoids shaming people who aren’t “DevOps enough,” and creates high levels of empathy between technical professionals within an organization. It puts the focus on outcomes instead of outputs and favors collaboration and productivity over arguments about tools.

    — Julian Dunn

  • Building Amazing Apps, Part 3: Optimizing the Network

    Building Amazing Apps, Part 3: Optimizing the Network

    This is part three of a series of three articles on building and delivering amazing apps at the edge. Read part one here, and part two here.

    Introduction

    There are millions of apps available today, and it’s incredibly easy for any user to download and install any of them with just a few taps on a screen or click of a mouse. If you want to build and deliver an app that will amaze users, you need to be able to differentiate from the rest of the pack.

    Over the past 14 years, I’ve been working closely with the top companies delivering the most popular apps, helping them optimize their apps for speed and scale so users get a great experience. In this three-part series, we’ll explore the fundamental steps you must take to build and deliver your own amazing app at scale that will keep users coming back for more.

    We’ll break these steps down into three key areas:

    • Front end: Covers the parts of the app that end users interact with.
    • Back end: Looks at the required origin infrastructure to support the app, including databases, application, web servers and more.
    • Network: The glue that connects the front end and the back end.

    In the final article of this series, we’re going to explore four things you can do on the network to optimize your app and make it more amazing:

    • Distributed DNS
    • Protocol optimization
    • Latency optimization
    • Edge computing

    Distributed DNS

    The Domain Name System (DNS) is a critical part of making sure your clients can access your applications (in fact, it’s a very popular attack vector by hackers today because if your DNS is down, nobody can access your PPP). Distributing your DNS across many servers can help ensure that your DNS (and thus, your app) is fast and highly available, so you should use a DNS vendor that has a large DNS server network in a wide range of locations.

    Validating whether you have a properly distributed DNS infrastructure is rather easy. Just follow these steps:

    • Identify how many name servers your domain uses and their IP addresses. Ideally, you want a number between three and 12—the more the better. You can use command-line tools such as dig and nslookup to identify the name servers of a given domain.
    • Use an IP geolocation system to find out the physical location of the name servers. At this point, you can get a rough idea if those servers are close enough to your end users. Hint: Some CDNs offer diagnostic tools that can be scripted via APIs to automate IP geolocation.
    • If you want to be more thorough, you may want to run connectivity tests between the main locations of the users and the name server IP addresses to make sure the network latency—i.e., how long it takes for information to go back and forth between the client and the server—stays under 100 ms (the lower the better). Some internet providers and CDNs provide looking glass technology, which enables you to run connectivity tests that originate from servers around the world.

    Protocol Optimization

    Once your clients know where to connect, how they connect and the protocol used is something that shouldn’t be neglected. HTTP/2 is the modern, superior network protocol, as it uses a more efficient binary format, header compression, single network connection, stream multiplexing and server-push. Actually, I was very involved in the early years of HTTP/2 (the standard was approved in February 2015) and have since helped some of the largest websites on the internet implement HTTP/2 and still spend a considerable amount of my time testing the new protocol in all sorts of environments. I’ve shared my findings in talks at conferences including Velocity and in the “Learning HTTP/2” O’Reilly book, which I co-authored. In a nutshell, most sites can improve page load performance between 5% and 15% by just switching the protocol from HTTP/1.1 to HTTP/2, which is as simple as flipping a switch if your website uses HTTPS.

    To illustrate that improvement, I created a demo site that tests the download speed of many small images over HTTP/1.1 and over HTTP/2 and displays the differences.

    Figure A

    As you can see in Figure A, the content loaded eight times faster over HTTP/2 than over HTTP/1.1 on my mobile phone over an LTE connection. That’s a huge improvement considering how simple it is to implement HTTP/2.

    According to the HTTP Archive, as of June 2018, 20% of hosts support HTTP/2, but those hosts are responsible for 40% of all requests—meaning HTTP/2 powers some of the web’s most popular sites.

    However, as explained in the HTTP/2 Performance chapter of the Learning HTTP/2 book by O’Reilly, there are a lot of factors that can impact the benefits of HTTP/2. Some of them you can control, while others—such as network packet loss—are much more difficult to tackle and can impact HTTP/2 performance because HTTP/2 runs over a single TCP connection.

    Note: TCP stands for Transport Control Protocol and some parts of its implementation (e.g., slow to start and cumbersome network congestion algorithm) can impact HTTP/2 performance because all the communication with the server goes through a single TCP connection.

    The future of the web is HTTP/3, which uses HTTP/2 over QUIC (a newer and more efficient transport layer protocol). However, as of April 2019, HTTP/3 is still in draft state and hasn’t been approved as a standard.

    Ideally, your servers should monitor the network connection and use different network protocols depending on the current conditions. Certain content delivery networks (CDNs) offer this functionality. For example, to maximize throughput, a top CDN provider uses custom software on its edge servers to dynamically use different network protocols, TCP settings and congestion control algorithms, depending on the specific conditions of each individual connection the server makes with an end user. In short, using the optimal protocol makes your app faster and more amazing.

    Latency Optimization

    Latency is a physical limitation of nature that impacts the throughput on computer networks. You probably know that whenever you are outdoors and there is a storm with lightning, you can count the number of seconds until you hear the thunder to identify how close the storm is (and if you count less than a second, you better get inside!). In computer terms, latency is measured as the time it takes to send a network packet back and forth between two systems. This concept is known as round-trip time (RTT) and is typically measured in milliseconds. You can use network tools such as ping or mtr to easily measure the RTT between two systems.

    Latency has a huge impact on performance because the further away you are from the server you want to connect, the slower the communication will be. As a rule of thumb, you want to ensure your latency is well under 100 ms. The easiest way to achieve that is by leveraging the local device cache to avoid extra network connections to the server and using a CDN to put your server closer to the end user.

    Here is a graph (see Figure B) showing that latency plays a larger role on page load time than bandwidth. As you can see, doubling the bandwidth from 5Mbps to 10Mbps results in just 5% improvement in page load times, and any bandwidth increase over 10Mbps doesn’t make a big difference. Reducing the latency, however, drastically improves the page load times. That makes your app more amazing for your end users.

    Figure B

    Edge Computing

    If you are a developer or a DevOps engineer (or just love tech) chances are you are within the TOP 1% of world population in terms of mobile computing power. But things you take for granted, such as resizing images on a mobile phone, are going to take much longer for many millions of internet users who have less powerful mobile devices.

    So, what can we do if running logic at the origin causes delays because of the network latency and running logic on the client side may be too slow on less powerful devices? The solution is edge computing, which consists of moving the origin logic to a server close enough to the end users that latency does not impact performance. CDNs are in the best position to offer this proximity because of their natural distributed architecture; a CDN can set cookies, rewrite HTML and resize and deliver optimized images for the screen size of your users so those images don’t take a long time to display on low-end mobile devices with limited CPUs.

    Edge computing is the next step beyond cloud computing. You should try to use a CDN company that offers you the ability of running business logic as close as possible to where your users are, to ensure that your app is fast and can scale without impacting your infrastructure.

    Summary

    In this series of articles, we reviewed some basic things you should consider if you want to build and deliver amazing apps at scale. This is not a comprehensive list of all the things you can do but provides you with some of the most impactful areas for app improvement.

    — Javier Garza

  • Focusing on the True DevOps North Star: Continuous Improvement

    Focusing on the True DevOps North Star: Continuous Improvement

    When we look beyond the fundamentals of tearing down silos and establishing greater collaboration, communication and transparency, DevOps is a label assigned to a framework of best practices and technologies that organizations leverage to deliver systems faster, more efficiently and without sacrificing quality. The best practice within that framework is continuous improvement, which provides stakeholders with data and feedback to constantly improve their ability to deliver products that advance strategic business objectives. This is done through key performance indicators (KPIs) that align with the goals of the business.

    Continuous improvement thrives on a robust program of KPIs that measure key attributes of a successful development team in the context of what is important to your organization. These KPIs provide you with the opportunity to continuously evaluate your system delivery processes and supporting technologies, and identify opportunities to make changes that produce better outcomes. In general, all organizations should be continuously improving their system development and delivery against three primary KPIs:

    • Quality: The value and performance of software you deliver to customers.
    • Velocity: The speed at which you develop and deliver software to customers.
    • Efficiency: The ease (low resource consumption, minimal latency) with which you develop and deliver software to customers.

    For now, let’s focus on what’s required to continuously measure and improve against them.

    Continuous Improvement KPIs vs. Metrics

    KPIs provide baselines for minimally acceptable performance levels across critical areas that impact the success of your business. Metrics measured against these KPIs help development teams improve what they’re doing to support the performance of the business in light of its goals.

    There are hundreds of metrics available (escaped versus trapped defects, features deployed and mean time to X, to name a few). The problem is most organizations focus too much on non-business-related metrics and data points and end up measuring the wrong things, such as how many lines of code have been written. This data may be interesting, but it tells you nothing of how well or within what time frame the code was written, or how that affects the delivery of value to customers.

    Your metrics should align with the goal-oriented KPIs of your business. For example, if a goal is to increase the quality of software after an influx of customer support issues related to defects, you should measure how many defects are escaping and how many are being trapped in testing. However, the remaining challenge for most organizations is actually capturing this metrics data to accurately measure progress and continuous improvement against, for example, a quality baseline.

    Collecting Data for Continuous Improvement

    A large reason organizations struggle to capture the right metrics data is they rely on outdated project management tools and manual effort. Developers and other team members painstakingly capture and contribute metrics data through status reports and meetings. Project managers then spend the bulk of their time analyzing and producing even more reports to provide the necessary feedback to support continuous improvement.

    In a digital economy where fast beats slow, this slow approach of manual data collection constrains and negates the business objectives of increasing quality, velocity and efficiency simultaneously. Core to Agile is the production of working software and minimizing the amount of work not directly related to that endeavor. If you want to move to an agile, iterative world, you want to minimize activity or any work that’s not directly related to the software delivery effort.

    That’s why, for true continuous improvement, you want your tools to automatically produce the right metrics data to measure and continuously improve upon KPIs, rather than relying on developers to attend status meetings and feed project managers information used to manually build reports.

    Once you’re able to visualize this data with a modern dashboard-driven analytics tool, you can use that information to constantly assess and evaluate system development and delivery productivity, and help developers make behavioral changes. You will have the necessary information to tweak processes and tooling and better educate team members.

    For example, if your software quality drops below its KPI-defined baseline and your metrics show increased efforts utilized to resolve break/fix issues, you can make a valuable correlation leading to smarter actions to improve quality, such as tweaking or changing testing methods and possibly using more effective testing tools.

    In this way, continuous improvement becomes embedded in your software delivery architecture, as you’re no longer making gut-based decisions that require you to wait for an outcome and guess again. Instead, you’re using data to drive strategic, fact-based shifts in behavior as you develop software, with the knowledge that these changes will have a real impact on the business.

    Making Continuous Improvement Across Platforms

    Ultimately, continuous improvement must reach across platforms in your organization—from the cloud to the mainframe. Development teams should work to continuously improve system delivery across all development platforms.

    The KPIs mentioned must apply to the entire development team and business, not just developers on one particular platform or skill set. You want to identify KPIs that are critical to the successful delivery of systems that support your customers, employees and business partners.

    Now that we have covered the importance of continuous improvement through the use of system delivery KPIs, stay tuned as we explore the individual topics of quality, velocity and efficiency at a deeper level in subsequent articles.

    — Rick Slade

  • Monitoring the DevOps Toolchain

    Monitoring the DevOps Toolchain

    DevOps has grown in popularity among many companies seeking greater agility and higher-quality code. When using different tools for automation, these organizations need monitoring and tracking of these tools and metrics from different portals on a common dashboard. Toolchain monitoring helps to bring all reports into a common portal.

    Enabling monitoring tools from a centralized platform can save a lot of effort and time. These tools have many in-build capabilities that bring value to the business, such as predictive alarming, graphical representation of data and integration with other tools including email, ticketing system and auto remediation. These added features become game-changers in today’s competitive SLA-driven and standard environments.

    Monitoring can be categorized in three ways:

    • Infrastructure monitoring: Monitoring the environment, machine and resources used to create a platform for an application.
    • Application monitoring: Monitoring the application’s performance, resource consumption, processing time and responsiveness.
    • DevOps monitoring: Monitoring the pipeline and toolchain from continuous exploration through release management. DevOps monitoring tools have the capability to monitor job health, job status, performance and quality.

    Why DevOps Monitoring is Essential

    In DevOps, we use many tools to create a toolchain and pipelines. In some big teams with a long release train, monitoring a project’s status from requirements-gathering to deployment can be difficult, as you have to track a number of different tools. And even then, you might not get the status in a desired format. DevOps monitoring tools bring everything to one single dashboard, where you can see the status and other metrics, including useful information about the project quality and progress.

    DevOps Monitoring: Not the Same as Tools Monitoring

    Tools monitoring is specific to the health and performance of tools; DevOps monitoring focuses on things such as the status of sprint, code quality, code branch status, test results and deployment status. All of which are required by project management for continuous improvement in productivity and quality.

    What to Monitor?

    There are a number of things we want to monitor in DevOps orchestration, and all these metrics provide a perspective of the current status of the project or product development and productivity:

    • Performance: Performance of the application.
    • Development (sprint or Kanban progress) and defects status from the application lifecycle management (ALM) tool.
    • Code repository monitoring for a project specific branch.
    • Build job or pipeline monitoring from the continuous integration (CI) tool.
    • Code analysis and security test results and reports.
    • Deployment and upgrade status in different environments.
    • DevOps maturity using the tools’ usage and quality data.
    • Artifacts (versioned artifacts).

    These are the basic and mandatory areas any project would monitor on a single dashboard.

    Monitoring Use Case

    In agile projects, CI/CD tools are implemented following the best practices of DevOps. An ALM tool is implemented to manage the backlog and development sprints. On this project a CI tool is used to build code and all the code is getting committed in a code repo, using a code review and analysis tool. A binary repository has been implemented to store all the binary of every version.  An automated deployment tool is being used to deploy the application in QA, staging and production environment.

    Now the struggle comes when monitoring each tool, the status of the build and the progress on different portals (CI tool, code repo, analysis tool, deployment tool, etc.).

    Using centralized monitoring tools can help project management to monitor the metrics mentioned below. These metrics give the overall picture of build, quality and deployment status on one dashboard.

    Available Tools for DevOps Monitoring

    Below are few available products that provide similar solutions:

    • Hygieia: Freeware tool that supports all DevOps tools, but metrics are very limited.
    • KlipFolio: Has lots of tool support, but it’s not free.
    • DevOps Insights: An IBM tool that supports IBM cloud platform only.
    • DataDog: Has a wide range of tool support, but it’s not free.
    • Prometheus: Freeware tool that monitors lots of tool-using exporters.

    Build vs. Buy

    Deciding which tool to use is always a tough task—you need to compare all the available tools and their offerings from different perspectives. Below are a few factors that play an important role when deciding on available options or to build your own solution:

    • Range of supported tools (plugins or data collectors).
    • Freeware or paid.
    • Support to customize data collection.
    • Rich UI for data representation and customization of fields and views.
    • Range of metrics supported.
    • Ease of installation and configuration.
    • Licensing and distribution policy.

    Solution Approach for Building Your DevOps Monitoring

    If you have decided to build or extend any existing tool, below are some considerations:

    • Identify the tools you have to monitor: List the DevOps tools you have to monitor; the list should cover all the areas of DevOps (build, repo, code quality, deployment, monitoring, etc.) to get the complete picture.
    • Identify the metrics: Once you have the list of tools, identify the important metrics from each tool. Your metrics should be capable of providing KPIs of DevOps areas.
    • Identify API or command-line interface to fetch the metric data: You need a good understanding of tools and their implementation to identify the best way to fetch the data. It could be predefined APIs, REST APIs or any command line interface. Make sure you are getting all the desired data.
    • Correlate the metrics with the business value or define the DevOps KPI: Your metrics should represent the KPIs and business value of all the tools involved in achieving DevOps. The dashboard should cover all the aspects and represent the current status and gaps.
    • Determine how to store and process those metrics: You need to define the schema to store the data in defined database; your schema should be generic enough to cater any data from different tools. To process the data, you need a middleware component between database and UI to process in a presentable format.
    • Build a scoreboard based on metrics: Use a rich GUI to display the scoreboard in different formats such as table, graphs and charts.
    • Support high availability: The tool should be designed to support high availability to avoid downtime and loss of data points.

    How Monitoring Tools Collect Data

    Most of the tools we use for implementing DevOps and CI/CD are web-based tools such as Jenkins, Sonar, Git, SVN, UDeploy, Gerrit, Jira, Confluence or others. These tools provide REST APIs, from which we get the data in a described format—either in json or xml. Some tools support predefined APIs or CLIs to connect and fetch the data. Once you identify the communication mechanism with the end system, the next step will be writing robust code that can fetch the desired metrics data and handle all the cases to parse the data and put it in a relational or non-relational database.

    Once you have the data in the database, you need a GUI to display those in tabular and graphical formats and a middleware component that can fetch the data from the database and provide it to GUI in the desired format.

    Getting the data from the system you want to monitor is the real challenge in this whole exercise, as everything else depends on it. Below are a few challenges you might face while developing a toolchain monitoring system:

    • Identify the API or any other mechanism to fetch the data.
    • Identify the metrics to monitor.
    • Identify how you want the data to be presented on the GUI—whether a trend chart, pie chart or just tabular presentation.
    • Determine the authentication mechanism for the monitoring system, as each supports a different kind. Sometimes it needs a predefined token while other times a token needs to be generated at runtime.

    Simple Architecture of DevOps Pipeline Monitoring

    Below is the simple generic architecture, on which most of the monitoring tools work. In the picture below, the different DevOps tools are on the left side; these tools are the end system from where we want to bring the data. Only a few tools have been shown as an example.

    A data collector brings the data from each end system and communicates with the end system using a REST API or any API exposed by that tool or a CLI, depending on the case. Using the defined communication mechanism, it will bring the data and pass it to the parser to fetch the required data and put it in the database in the desired format.

    While developing data collection, you need to have good a understanding of the tool you are going to monitor as well as available APIs including REST, CLI, SMIs or any pre-defined API built within the tool.

    The data parsing layer will fetch the desired metric value from the raw data collected by the data collector. Raw data could be in any format, including key value pair, json, xml or csv. The parsing methodology will differ in each case depending on the type of data. You can use any generic API for the parsing available.

    You must identify a good API to parse the data available in a different format and complexity.

    The database will hold the metrics data of each interval for each tool and user credentials.

    The GUI will request the data from the parsing component. It will represent it in tabular or graphical user-friendly format.

    Other features: Once we have data, we can have the trend chart to make some critical decisions, we can generate alarm on breaching any defined thresholds and integrate it with email or SMS to take immediate action.

    Monitoring Tool High Availability

    If we are relying on our monitoring tool to get reports of all the tools used in DevOps, we cannot afford downtime. We may lose data points and trends if the monitoring tool goes down, and management may not get the actual status of an entire project without having to gather the data manually.

    To avoid such situation, high availability (HA) should be implemented on the monitoring tool. In case of a HA setup, we will keep gathering the data in case the monitoring node goes down.

    To implement the HA, there are two aspects we need to consider: the HA of GUI and middleware (business logic component) and the HA of database.

    HA in the GUI and middleware could be achieved using an Apache or Nginix cluster. In the case of containerized environments, we can use any orchestration platform’s default feature to achieve high availability, such as Kubernetes. Similarly, in the database we can implement a clustered database feature provided by the most of available databases.

    Conclusion

    DevOps toolchain monitoring is very useful. It provides a good snapshot of the health of a project to management and is able to identify any blocker issues that might impact the release so they can be corrected. Monitoring can help identify pain points to focus on in defining the road map. Organizations following DevOps practices and using DevOps tools should have a tool to monitor the DevOps pipeline.

    — Prateek Kumar Asthana

  • Tax Day IT Lessons, or How to Avoid Wiping Out on Your Biggest Day of the Year

    Tax Day IT Lessons, or How to Avoid Wiping Out on Your Biggest Day of the Year

    This time last year, as the Internal Revenue Service (IRS) geared up for its biggest day of the year, a firmware bug deep in its storage network was waiting to take down the system. When Tax Day arrived, the onslaught of last-minute filers triggered a systemwide outage that lasted 11 hours and forced the IRS to extend its filing deadline by a day.

    The IRS has another chance to get it right next week, but its IT team is far from alone in having dropped the ball on its most important day of the year. In 2015, the government’s Healthcare.gov website was plagued by crashes on the day Obamacare launched. Last year, Amazon’s website crashed just as Prime Day kicked off, and a few months later, Walmart’s systems fumbled at the start of Black Friday.

    These glitches can cost organizations millions, not to mention the hit to their reputations and inconvenience to customers. As Tax Day approaches, it’s worth looking at some steps any organization can take to avoid embarrassing or costly glitches when a big day arrives. These occasions are fraught with risk due to sudden traffic spikes or untested code.

    Here are steps organizations can take to minimize the chance of disruption.

    Have a Deployment Process You Trust

    People often talk about a code freeze as a solution, but this shouldn’t be your first instinct. A code freeze ties your hands from making last minute changes, and let’s face it, few teams have their code buttoned down months ahead of time. Far better is to have a rock-solid deployment process that gives you confidence that what you’re shipping won’t break. Here are some elements it should include:

    • Trusted testing gate before production. You and your team need confidence that you’re shipping code that works on the devices and browsers your customers use. There are many ways to do this but focus on these elements: speed of throughput needed, the level of risk you are willing to tolerate and the cost to your business of bugs making it to production.
    • A totally automated process with no human bottlenecks. There’s nothing like trying to ship a fix to production, only to realize the engineer with permission to approve the release is on PTO.
    • Proper environment escalation. Contrary to the above, never ship direct to production! Have a proper staging server that is a good enough replica of production, so you can test your application in conditions that match production.
    • Good test data management. This is one of the areas where software companies fall down. Stale data, data that doesn’t have edges or nuances like in production and data that contains personally identifiable information (PII) are all very common culprits—this is one of the most difficult things to get right with testing.

    Stress Test for Robustness

    In high-traffic periods, new stuff breaks. Stress testing helps to ensure services will continue running under a heavy load, but you need a realistic approach grounded in actual user behavior. Can you handle 20,000 users, all trying to register at once when your new app goes live? What pathways will customers take through your website or application? What features have caused problems in the past? Identify common pathways and potential problem areas and stress-test the hell out of them.

    Build a Culture of Quality

    One reason big events are scary for engineers is the sudden emphasis on accountability and quality that was lacking before. Solving for quality at the eleventh hour is guaranteed to be less effective and efficient than building a culture of quality from the start. Much like most management, the only way to do this well is to create accountability for your teams and give ownership of quality directly to them. A few quick ideas: Add a functionality review to your code review step, provide tooling to your team so they can drive testing themselves on pre-production features and make them feel the pain of bugs getting into production by having an engineering support rotation.

    Measurement and Metrics

    Achieving this culture of quality depends partly on good measurement and metrics. Sadly, this is one of the weakest and least-defined areas of development today. Since it’s impossible to directly observe quality in production—since you can only know about the bugs that are caught—it’s more productive to create some measure for QA process coverage and tie that off to the business. One way to do this is to look at the differences between how your customers are using your application in production, versus how you are testing it pre-production. This can give you a lot of insight into the areas that are under-covered or over-covered and can be a useful way to measure whether incremental investments in QA will actually be ROI-positive.

    If You Can’t Do Any of That … Fine, Resort to a Code Freeze

    If you don’t have a bulletproof deployment process, a code freeze is a brute-force option. It ties your hands from making last-minute changes, but at least ensures you won’t add new risks without adequate time to discover them. The duration depends on how long it takes your team to surface production issues, based on past experience. If you don’t have that data, six to eight weeks may be a safe period. But really, if this is your main strategy for avoiding problems on game day, you’re probably in the wrong job.

    Review Patch Management Policies

    This one might have helped the IRS. Its storage vendor, IBM, had issued a patch for the bug that brought down the system months earlier, but the IRS didn’t apply it because it was part of a code bundle that didn’t meet its production testing requirements. A government report concluded that the IRS needs to vet those decisions more effectively and document the process. Bottom line: Have a structure and process in place that ensures you’re making informed, sensible decisions about whether to apply new patches.

    Other issues contributed to the IRS meltdown: Its storage system had no automatic fail-over or built-in redundancies, creating a single point of failure. It was unfortunate that several factors conspired against it, but luck isn’t something you should depend on. Executing successfully for a big event isn’t achieved in a few short months beforehand. It involves building the right culture and practices to guide your development and operations year-round. This will greatly minimize risk and the work you need to do to be prepared.

    — Fred Stevens-Smith

  • Developing Simple and Stable Machine Learning Models

    Developing Simple and Stable Machine Learning Models

    A current challenge and debate in artificial intelligence is building simple and stable machine learning models capable of identifying patterns and even objects. Is it possible to know which models will work best or to simply see the data? It is possible, and the question is figuring out how to get there.

    Simple models are popular because they are easy to understand, trustworthy and tend to be more resilient to change than complex models. Simple models also have the benefit of overcoming transfer learning pains. While everyone wants simple models, they also want accurate models. In practice, there are always transfer learning issues when applying learning from the past, into future practice.

    An issue arises when trying to fit a model from a known family to the available data and hoping it will generalize effectively. It could be any proposed family: linear models, trees or trained deep neural networks.

    There remains a fundamental problem with any overly complicated model—linear, tree or neural network—in the risk of “overfitting” the data resulting in poor generalization and results. At the same time, an overly simple model risks missing the nuances of the data and sacrificing accuracy. There are techniques to handle this problem.

    First is to reduce the size of the set of models. This works well with the PAC model. For example, reduce the number of parameters, reduce the depth of the tree and reduce the nodes in the network. Reducing the size of the set means it’s easier to understand, and with fewer possibilities built into the model, it’s less likely to overfit the data.

    Next, add L1 or L2 regularization. Models with small weight, or many zero weights, are preferable and will help the model generalize. The question is, why? Why do we use trees or neural networks rather than another hypothesis family? Why do we want models with small weights? Why do some network topologies work well?

    These questions bring in the actual world we live in and the types of problems that likely need solving. There is no learning when any function is allowed, and the model family isn’t limited. It is well-known that when the family of allowed hypotheses is too large, the model overfits the data ad-absurdum and becomes useless. Thus, machine learning almost always optimizes along a well-understood family of hypotheses while human feature engineering and representation construction captures our prior knowledge about the world. The question is how to exploit such knowledge when automating machine learning. Arbitrary limiting is better than not limiting at all, but we can do better.

    The world has patterns repeating themselves across many domains. Different applications will use similar building blocks to analyze rocket movement or medical sensors, or to solve a predictive maintenance task. But we are all experts in the world we live in. Even studying a domain where we don’t have specific expertise, we still recognize some patterns and rules.

    For example, take a look at this ring-tailed lemur. Even if you’ve never seen one before, you are now likely to identify this type of primate if you come across one. My 5-year-old son was shown this picture and he could visit the zoo and identify a ring-tailed lemur. It only took one photo.

    Compare that result to modern neural network research which typically requires many examples—often into the millions—when training to identify objects, and still they suffer from embarrassing mistakes when new data differs from the data the network was trained on.

    How did my 5-year-old son learn to identify ring-tailed lemurs from a single picture? He’s an expert. Not a lemur or primate expert, but an expert in the world we live in. He understands identifying an object from its background. What a tail, eyes, ears, legs and fur look like. He knows how to imagine a 3D object from different directions. At 5, he already brought a ton of knowledge with him when learning to identify lemurs.

    This is the essence of learning. We understand the notion of simplicity is not in number of nodes in the network or number of lines of code, but that it relates to simplicity when explaining human language.

    This process can be formalized by codifying human knowledge to solve tough problems. We can formalize the many patterns seen before in various domains and in the various tools data scientists use when constructing features. Then we can look for patterns that make sense and are similar to what we’ve seen before.

    We can also look for more novel patterns represented within the complex combinations of things we’ve seen before. As an industry, we need to do exactly this: Codify human knowledge in curated code libraries, and in structured knowledge and facts, use them to search for patterns in data and build highly accurate resilient models. This will bring simplicity and stability to machine learning models without sacrificing accuracy by leveraging humanity’s existing knowledge.

    — Meir Maor

  • 5 DevOps Trends Worth Knowing in 2019

    5 DevOps Trends Worth Knowing in 2019

    DevOps today is more than the buzzword it was 10 years ago. Back then, the idea of combining development with production in your IT infrastructure was not common; the logistics were nonexistent and most of the time businesses decided it was far too much effort.

    However, if the last nine years has taught us anything, it’s that DevOps is worth every penny businesses invest. It helps companies streamline production, keep customers happy and provide agile services to attract new audiences.

    But as with anything in the IT field, there is a paradigm of change. This article discusses some of the top DevOps trends happening now and how to implement them in your own system.

    Automation Will Be a Bigger Focus

    Automation has always been the way for businesses. Whether it’s speeding up the pipeline with machines, providing better response times for angry customers with chatbots or better inventory management with tracking and logging system, automation is behind it all.

    DevOps is no exception.

    But as with everything, there can be too much automation. The best thing to do is break down processes and find simple, redundant tasks that can save time if automated and apply the changes.

    The most important aspect of automating DevOps is to understand the six pillars in the DevOps cycle, which are:

    • Continuous business planning
    • Collaborative development
    • Continuous testing
    • Continuous release and deployment
    • Continuous monitoring
    • Collaborative customer feedback and optimization

    DevOps Assembly Lines: The Next Big Thing

     

    To track the development of your application, we have pipelines. These visually represent the product from its initial stage right through until final development and release.

    However, continuous integration (CI) is no longer the priority focus when it comes to effective monitoring and implementation. Continuous delivery and automation are a major part that businesses have already started delving into in 2019.

    This is where DevOps assembly lines enter the picture.

    Assembly lines are focused on bridging the gap between manual and automated tasks, ensuring there is no redundancy while providing several tools for teams to implement, to ensure apps are hitting the production lines as reasonably quickly as possible.

    Microservices Architecture Adoption Taking Off

     

    Microservices architecture may as well be that new kid in school trying to fit in. These smaller services compound into agile, implementable tools that DevOps teams can use to scale and test software over the course of weeks—not years.

    As you can imagine, this can help some businesses streamline their development and production lines, setting them way ahead of their original production estimates—which is never a bad thing.

    Implementing microservices into your DevOps team can also help companies focus on the runtime and efficient delivery of new development sources, without the hassle of creating dependency errors when something decides to bug out for no apparent reason.

    Security Needs to Step Up

    Cybercrime is on the rise—we all know that’s old news. But as a business, cybercrime can either make or break your sales and weigh down your DevOps team.

    That’s why more companies are hopping onto enterprise security options. Although there are third-party programmers for almost every situation you need, none of them have the same security levels as the tools from the enterprise version provides. Safeguarding both your company and your clientele.

    Riding the Artificial Intelligence Wave

    Finally, we have artificial intelligence (AI) and machine learning (ML). When you think about it, these two elements are literally the sum of everything we want in the DevOps community—the ability to crunch out hours upon hours of menial labor and saving your teams hours to focus on higher-priority tasks.

    The combination of AI, used to predict algorithmic influencers, and ML’s ability to adapt to new situations will be key for DevOps teams to get to the next level in 2019.

    — Gil Artmoore

  • 6 Considerations for Secure Cloud Migration

    6 Considerations for Secure Cloud Migration

    Planning a secure cloud migration? Here are some tips to help make the process smooth.

    Three out of every four organizations on the planet have some sort of cloud presence. According to a 2018 IDG survey, 77% of enterprises now have at least one application or some portion of their enterprise computing infrastructure in the cloud. In addition, enterprises plan to invest an average of $3.5 million on cloud apps, platforms and services, the report found.

    Looking forward, executive management at technology-dependent industries—including manufacturing, high-tech and telecom—are increasingly driving toward becoming 100% cloud-enabled. Which also means that, except for those new startups starting with a full cloud-based infrastructure, most organizations are currently either actively migrating their infrastructure and/or applications to the cloud; trying to bridge their business processes, applications and workflows between a local physical network and one or more networks residing in the public cloud; or are planning to do so.

    One of the challenges these organizations face is creating a consistent security posture across their local and cloud-based resources. The reality is, not all security policies can be successfully and seamlessly implemented across a multi-cloud environment, especially when using a variety of tools. That’s because most vendor solutions either do not support all of the major cloud platforms. And for those that do, many do not support native cloud integration leveraging cloud resource description. This can create challenges in enforcement consistency and policy definition as workflows and applications move between different cloud environments, resulting in security gaps and blind spots that can be exploited.

    Successfully Migrating Security to the Cloud

    Migrating infrastructures, applications or services to the cloud without expanding the attack surface or increasing security overhead requires careful preparation. This starts by understanding that any cloud-based deployment, whether building out a new infrastructure or building a new application, requires clear communication between lines of business, IT and security teams. Without clear communications about business needs and objectives and a candid discussion of related threats, organizations open themselves to a whole array of new risks, including denial-of-service attacks targeting cloud resources, cloud malware injection, web application exploitation, cloud-API attacks and account or service hijacking.

    Successful cloud migration also requires successfully migrating security to the cloud, enabling organizations to deploy and manage a single, consistent security framework that spans the entire multi-cloud infrastructure.

    Here are six steps every organization should consider when planning a cloud adoption or cloud migration strategy:

    Baseline Your Security Before Cloud Migration

    Far too many organizations own security architecture built around isolated security devices, decentralized management and an inconsistent application of security policies.

    If this is the case for your organization, you will need to start by getting control of your security sprawl and imposing a central security strategy. Once that is in place, you then need to ask the following questions:

    • Do you know your current security posture and its implications for your future business goals?
    • Do you have the proper policies and procedures in place for your current and future environments?
    • Have you performed a gap analysis for how cloud will change your security paradigm?
    • What are the impacts of a distributed, cloud-based network on risk management?

    Plan for Bandwidth Requirements

    You will need to model and understand data flows and bandwidth requirements to ensure that your security solutions can meet performance requirements, especially for latency-sensitive services that will need to go over VPN tunnels.

    Understand Compliance Issues

    What requirements do you have to meet for data processed and stored on the cloud, as well as for data that moves between different cloud and physical network environments? It is crucial that your legal team be consulted before you begin to build or adopt any sort of cloud program.

    Provide for High Availability and Disaster Recovery

    The biggest fear for most organizations looking at a cloud solution, after addressing security concerns, is the availability of cloud-based resources. You also need to recognize whether dynamic scaling is required and whether your security solution can meet new performance requirements. Finally, you need to consider things such as flow symmetry and load balancing, especially for legacy applications, to maintain availability, performance and protection even when utilizing dynamic cloud-based services.

    Apply the Right Security at the Right Place

    Cloud security requires much more than simply placing a firewall at the perimeter of the cloud infrastructure. A wide range of security solutions will need to be applied depending on the applications running and services being used. A next-generation firewall (NGFW) solution is the most common security tool to be applied, but other solutions are often also required, including web application firewall (WAF), intrusion protection or detection service (IPS/IDS) and a cloud access security broker (CASB).

    Establish a Lifecycle Management Framework

    Ensuring consistency between security solutions and policy enforcement, especially when they span multiple environments, is crucial. Security tools need to be chosen not only for their ability to operate natively in a specific cloud platform, but also for their ability to interoperate seamlessly through the entire security policy life cycle with sister solutions deployed in other environments. This includes consistent security change policy, dynamic provisioning and scaling, single point of management—including integration with a central ITSM solution and central log collection—and correlation.

    Don’t Let the Ease of the Cloud Trick You into Shortcutting Security

    Adopting a cloud service can be as simple as clicking a link. And adding a new cloud-based infrastructure, while much more complex, is far easier than building its physical counterpart. But that can be deceptively simplistic. Far too many organizations have had to pay the price for rushing into a new cloud solution without carefully considering challenges related to security. These have ranged from opening new attack vectors into their network, to being unprepared for new cloud-based threats or being blindsided by fines and penalties for failing to adequately prepare for new compliance considerations.

    Careful preparation before you begin to build out your new cloud infrastructures, platforms or services can save you time, money, reputation and enable you to compete effectively in today’s new digital marketplace.

    — Lior Cohen

  • Using Jira Software Search to Accelerate DevOps Workflows

    Using Jira Software Search to Accelerate DevOps Workflows

    Imagine driving down the highway and seeming to get a traffic jam at a distance. You realize your car is equipped with GPS to have clear visibility into what’s next and can show exact directions for alternative routes and estimated travel time. It is very easy to pick the fastest and most favorable route to reach your destination by looking ahead proactively and having the set of right tools for collecting information.

    The combination of Dev and Ops teams that are working together to build a faster and more reliable released pipeline is defined as DevOps. Similar to GPS, Jira Software acts as a singular source of truth for information development across your DevOps workflow.

    Complete traceability and visibility of work across different teams can get you surface information and data from an integrated toolchain to the right people anytime. No matter if you’re a team lead, release manager or a tester, the workflow visibility can ensure everyone by letting them know what’s in development and what’s being supported.

    In the latest release of Jira Software version 7.9, there is support added for the dev cycle to show development data from Bamboo, Bitbucket and GitHub in Jira Software search from your DevOps practice. It makes it a lot easier to identify development issues by making informed decisions and faster shipping with software development services.

    Get Answers to Development Questions via Basic Searches

    A new column called Development is added to the default view of Jira’s Issue Navigator in the latest release of Jira Software Server (v7.9). It allows viewing specified information pulled out from Bitbucket Server without clicking into the individual issue. Instead of limiting yourself to the single-issue information, you can see a high-level snapshot of issues coupled together by filtration, to help surface larger trends or problems.

    If this gets stopped at their source, does it really need to move the needle for improving your dev process within DevOps? For instance, if you wish to know how many issues under development, are they in the review?

    If you ever find that one or two developers are getting loaded down with the bulk of pull requests, you can immediately access if redistribution of work is required or not.

    Filter Out Your Development Information Using JQL

    What happens if you need to answer too many complex or open-ended questions? This can become monotonous after a time. You can use Jira Query Language (JQL) to create search queries that are based on development information to uncover key information for avoiding all pending problems.

    The latest version of Jira Software Server comes updated with the syntax to align the JQL query formatting. The users of Jira Software power use JQL for a long time to look at a subset of issues for driving reports and focus on Scrum and Kanban boards. With the help of cross-reference data from developer tools, Jira Query Language seems to be more powerful now.

    Examples of Jira Query Language in Action

    The teams who practice DevOps need critical internal and customer feedback, which helps them to inform features by making them higher quality and more valuable to the end user. Hence, focusing on feedback often creates tension when you decide if the first version is ready to deploy.

    You may ask: What number of issues are done but have not been deployed?

    From this question, you can determine whether the feature is ready to be released and starts collecting feedback or is blocking any issues that need to get resolved prior to release.

    Often, the team’s test builds locally before jumping off to deployment. If the same is the case for your team, then it is a good practice to regularly check the status of the builds.

    You may ask: What number of issues are failing builds?

    If you figure out a lot of failing builds, it may be a signal that the developer environments do not match the deployment. Generally, this is a problem you want to catch and fix right away, so conversely, the set of issues can be the areas where the testers can check acceptance criteria and begin exploratory testing.

    Boost Up Your Search

    You need to surface the information when and where it is needed as most of the teams get the most out of DevOps by bringing development, QA, high-quality releases, more-informed decisions and happy customers. Keep learning!

    — Kibo Hutchinson