devopsy.kinsta.cloud

Author: Thomas Theakanath

  • Break-Even Analysis: Understanding the Impact of Automation

    Break-Even Analysis: Understanding the Impact of Automation

    Determining whether automation is helpful beyond the CI/CD pipeline can be easier through a break-even analysis

    The pursuit of automating away all the operational tasks in your organization doesn’t stop after you finish building a solid CI/CD pipeline infrastructure using the myriad tools available and making it stable.

    There are many more one-off tasks not part of the CI/CD pipeline, but which are very important for managing the infrastructure and operating the applications running in production, that are not yet automated. These tasks could be very specific to your organization, but some common categories of these tasks are:

    • Provisioning and maintenance of infrastructure, including patching, updating certificates and licenses.
    • Maintenance of monitoring systems.
    • Generation of reports and dashboards.

    To Automate or Not to Automate?

    A CI/CD pipeline is an end-to-end automated system that handles all aspects, from builds triggered by code change through deploying the built artifacts into production, with very little or no human intervention in between. The non-CI/CD tasks described above could be automated or be carried out manually.

    An inexperienced DevOps team can easily get carried away by the notion of  automating everything and end up spending time on low-priority tasks—sometimes, simply because using a new tool or the coding work involved in automating a task can be very attractive to the team members. While such endeavors would look great in a sprint demo and on a resumé, they won’t help the business achieve operational velocity. When there are multiple automation projects competing for the team’s time, it would make sense to do a break-even analysis to obtain better insight on the returns of automating an operational task.

    The Agile methods to select and quantify a task such as backlog prioritization and estimating time for a story are not adequate to assess how the organization would benefit from that project overall. The break-even analysis proposed here would help to determine whether an automation project is worth doing by looking at the overall impact of rolling out that automation in the organization. Agile is limited by its narrow focus on the team that rolls out the automation—the DevOps team—and it cannot provide any metrics on the overall impact of an automated process on the organization.

    The Break-Even Analysis

    Let’s define various metrics involved first and then do the simple math of overheads and breaking even with the benefits.

    Development time (d) – This is the time needed to develop and roll out an automation task. This would be same as the time estimate used for a related story in an Agile sprint. In hours.

    Maintenance time (m) – As any experienced DevOps engineer can attest, developing a script or tool suite is not a one-time deal. The changes will be needed due to many factors:

    • Infrastructure change.
    • Fixing bugs in the tool.
    • Enhancement requests.
    • Training users of the tools, such as SREs and fellow team members.

    Budget the hours needed annually for this task, depending on the complexity and possibility of changes down the road.

    Manual working time ™ – The time needed to complete the task without automation. Doing things manually can cause human errors which adds to the total time. Factor that in the estimate  if this task has been causing issues. In hours.

    Automated working time (ta) – The time needed to complete the task using automation. In hours.

    Jobs count (c) – The number of times this task is run. Make an annual estimation so even infrequent operational tasks can have a score.

    Though arbitrary, my suggestion is to try out an automation for a year for its benefits. Check if the project would bring in impressive returns during that time frame:

    Project overhead = d + m

    Savings from automation = (tm – ta) x c

    Net benefit to organization = Savings from automation – Project overhead

    To embark on an automation project, the organization should benefit greatly from it to do things faster and better. While an Agile way of prioritizing an automation task would narrow the focus only on the DevOps team, this method also gauges and factors in the users’ time.

    DevOps Benefiting Customers

    The tools developed might not be used internally always, which is a major assumption in the above break-even analysis. It is quite possible that an automation process could benefit end users as well. I think it is very important to factor in the time saved by our customers also.

    Happy customers always bring in goodwill to a company.

    — Thomas Theakanath

  • DevOps Stack on a Shoestring Budget

    DevOps Stack on a Shoestring Budget

    There are competing products, both open source and licensed, in any DevOps area that you can think of. The Operations groups in companies tend to accumulate such applications over time as determined by ongoing projects. Team members’ familiarity with a certain application could be a factor in adding it to the tools stack.

    Companies, therefore, end up with a mixed bag of freeware, open source or community versions of applications and software licensed from the get-go. In a startup environment where the Operations group might be on a shoestring budget, it might make sense to build the DevOps infrastructure on the cheap; using open source software (OSS) products would be one way to achieve that goal.

    The Shoestring Budget

    Below is a list of some of the applications and software for an effective DevOps environment. The list here includes only core applications that are needed to build the DevOps infrastructure. The basic communication and productivity tools that would be used by everyone in the company—and usually managed by IT—are not in the scope. Neither are the compilers and debugging and testing tools used by product development teams. However, in a small company setting, managing all kinds of tools might drop into the laps of the same people driving the DevOps initiatives.

    The objective here is to make a list of software that could be used license-free. That means some of the obvious applications in certain areas might not make it to the list. That doesn’t mean that the open source alternative is an inferior choice; availability of reliable product support is a major factor in most of the software purchase decision-making processes, and licensed software scores over OSS on that front.

    Git for Source Code Management (SCM)

    A code repository might be the first DevOps infrastructure component that would be set up even before a product idea is formalized into a company. Also, both developers and DevOps engineers need it for collaboration, within their own teams and across various engineering teams. Typically in a startup, developers get started with the DevOps work such as build and deployment automation before a dedicated DevOps team would even be part of the engineering staff.   

    In this category, It is hard toimagine choosing anything other than Git. Even though GitHub is not free for maintaining private code repositories, Git server can be setup on-premise for free. AWS has CodeCommit, another SaaS version of Git, as one of the services, and there is no extra cost for using it if you are already on AWS.

    Bugzilla for Bug Tracking

    The popular Atlassian suite of products, JIRA in this category and Confluence for collaborative documentation, seems to be default choices in respective categories. But, both on-premise and SaaS offerings of  JIRA and Confluence are licensed. Bugzilla is a powerful OSS alternative to JIRA that can support a large user base and it has been around for a while.

    If you are open to customizing Bugzilla for specific requirements, it could work better than some of its licensed competitions. Though its primary use is bug tracking, Bugzilla is feature-rich enough to be used for managing all phases of software development and even as an IT ticketing system.

    MediaWiki for Collaborative Documentation

    Though there are multiple, open source wiki applications available, MediaWiki might be the best supported when it comes to the availability of third-party extensions. Out-of-the-box features of MediaWiki are limited, but it can be extended using the free extensions available. Such customizations are very straightforward.

    Jenkins as Build & Continuous Integration Platform

    Though there is an enterprise offering of Jenkins from CloudBees, its open source version combined with freely downloadable plugins would make up a powerful integration platform for build, release and deployment automation—and any other areas in between. Describing Jenkins any more here would be simply preaching to the choir.

    Artifactory for Artifacts Management

    This is one of the applications a lot of companies might not have in their DevOps stack in the initial stage. However, the need to store built artifacts and to cache third-party tools is real, and the packages and binaries would normally end up in S3 buckets or shared filer volumes. There is no reason for not using the OSS version of Artifactory right from the beginning, as it covers the storage requirements already mentioned and much more:

    • Automated uploading of artifacts as part of the build process, typically done from Jenkins
    • Caching third-party tools locally for builds and deployments
    • Storing software artifacts for meeting build dependencies
    • Distributing packages and third-party tools for deployments

    Ansible for Configuration Management and Orchestration

    There are two great products in this space that could be used for free: the OSS versions of Puppet and Ansible. If you are on AWS, Chef is also available, at no extra cost, in the form of OpsWorks. But I would recommend Ansible for those getting started for these reasons:

    • It is an agentless tool and beginners would feel safer to roll out automation using it. A configuration management agent can play havoc with production systems if something gets misconfigured inadvertently.
    • Using the playbooks, deployments can be orchestrated very well using Ansible. In many companies, Puppet or Chef primarily covers system management aspects and there would be additional tools for deployments. Ansible simplifies that scenario.
    • Ansible’s support of Python turned out to be more useful, in my experience, than the support of Ruby in Puppet and Chef. The chances of a DevOps engineer knowing Python is much more than Ruby.
    • Ansible’s handling of host inventory is advanced, especially the dynamic generation of inventory lists. The latter feature is compatible with the provisioning of cloud infrastructure where its elasticity demands the use of dynamic inventory lists in configuration management systems.

    Nagios for Monitoring

    A lot of us in the DevOps community despise Nagios for its blandness and lack of features out of the box. But if you are willing to customize it by developing application-specific plugins, it could be converted into a very effective monitoring platform that can be used to consolidate outputs from the rest of the monitoring and various admin applications. I recommend it for these reasons:

    • Writing plugins for Nagios is very simple. Also, there are tons of free plugins available that would meet most of the system-level and third-party component level monitoring.
    • Developing plugins to add application-level monitoring and to integrate with third-party admin applications is easy.
    • When you want to move on to a next generation cloud-based monitoring application such as Datadog, existing Nagios plugins are normally migrated natively or as an integration.

    In a sysadmin environment where coding skills would be in short supply, Nagios could be of limited use as a systems monitoring application. But DevOps teams should treat it as an easily extendable tool that can be used to implement specific and fine-grained monitoring goals of the company.

    ELK for Log Management

    Splunk was instrumental in changing the way we dealt with log files, and, now there are several good products including Loggly, Logentries and Sumo Logic in that space. Though some of these SaaS products have a free tier and that might be enough for a small operation, it is better to start with ELK (Elasticsearch, Logstash and Kibana stack), mainly because the size of the logs can grow very fast and so would the usage of the log management application in the company. ELK requires substantial DevOps resources to set it up and maintain, as there are three applications involved.

    Closing Remarks

    It is important to have these essential applications, or similar ones with comparable features, implemented or made available to empower the engineering  staff. They facilitate the application development, lay the foundation for automation initiatives, and provide the toolkit to maintain production environments 24×7.

    Though it is normal, the organic growth of the DevOps organization and infrastructure can bring in OSS products with overlapping features, mainly because they could be used license-free. Carefully selecting the tools would help avoid the otherwise-inevitable clutter and the confusion later.

    One aspect not discussed is the total cost of use of a software. If SaaS offerings such as GitHub, Datadog and Loggly, and hosted versions of JIRA, Confluence and Artifactory are used, the cost would only be the subscription charges. OSS applications will need computing resources and some amount of customization and administrative overheads, which could add up to the overall cost.  

    If the objective is to minimize total cost, the upfront decision-making can become much more complicated than it looks. However, for a startup or a small company, getting started with the OSS stack for setting up the DevOps infrastructure can be advantageous if the applications are selected carefully.

  • Proactive Monitoring

    Proactive Monitoring

    I think, most of the DevOps discussions are centered around testing the code change, packaging and deploying it in production in an automated fashion. However, stinging site-down issues would eventually force any nascent production engineering team to make monitoring a high priority down the road.

    If functional, integration and acceptance tests are run to verify a code change before that was deployed in production, in my opinion, similar checks should be part of monitoring to make sure that the application works fine in production also, as designed. In production, code change is not only the factor that could impact the application. Updates of OS, network configuration changes, automatic upgrades of third-party tools, availability of system resources etc. can also impact application in production. But, I am yet to come across any framework that integrates testing and monitoring efforts.

    In a traditional sysadmin centric environment, the scope of monitoring doesn’t go beyond infrastructure layer. The operations team that owns monitoring might not have visibility into what needs to be monitored beyond infrastructure. For the latter, out-of-the-box solutions are readily available on popular monitoring platforms like Nagios.

    Here is an attempt to classify the monitoring requirements in production. The requirements are generic and are applicable for any system hosted in-house, in a data center or in the cloud. A systematic and proactive plan for testing code changes and monitoring production would help to prevent incidents in production. Classification of tests and tools to implement them have been around for a while and that hardly require any additional attention. This article is an attempt to classify the monitoring types and to document their implementation methods.

    Monitoring Infrastructure

    The infrastructure that hosts an application environment would be made up of multiple components: servers, storage devices, load balancer etc. Checking the health of these devices is the most basic requirement of monitoring. The popular monitoring platforms support this feature out-of-the-box. Very little customization is required except for setting up right thresholds on those metrics for alerting.

    Monitoring Platform

    An application would typically be built using multiple third-party tools such as databases, both RDBMS (MySQL, Postgres) and NoSQL (MongoDB, Couchbase, Cassandra) data repositories; full-text search engines (ElasticSearch) ; BigData platforms (Hadoop, Spark); messaging systems (RabbitMQ); memory object caching systems (Memcache,Redis); and BI and reporting tools (Microstrategy, Tableau). Checking the health of these application components is important too. Most of these tools provide some interface, mainly via REST API, that can be leveraged to implement plugins on the main monitoring platform.

    Monitoring Application

    Having a healthy infrastructure and platform is not good enough for an application to function correctly. Buggy code from a recent deployment or third-party component issues or incompatible changes with external systems can cause application failures. Application level checks can be implemented to detect such issues. As mentioned before, a functional or integration test would unearth such issues in a testing/staging environment, and, an equivalent of that should be implemented in the production environment also.

    The implementation of application level monitoring could be simplified by building hooks or API endpoints in the application. Monitoring is usually an after-thought and the requirement of such instrumentation is overlooked during the design phase of an application. The participation of DevOps team in the design reviews improves operability of a system. Planning for application level monitoring in production is one area where DevOps can provide inputs.

    Monitoring Business

    The applications run in production to meet certain business goals. You can have an application that runs flawlessly on a healthy infrastructure but still the business might not be meeting its goals. It is important to provide that feedback to the business at the earliest to take corrective actions which might trigger enhancements of the application features and/or require the way business is run using the application. These efforts should only complement the more complex BI based data analysis methods that could provide deeper insights into the state of the business. The business level monitoring can be based on transactional data readily available in the data repositories and the data aggregates generated by the BI systems.

    Both application and business level monitoring are company specific, and, plugins have to be developed for such monitoring requirements. Implementing some framework to access standard sources of information such as databases and REST APIs from the monitoring platform could minimize the requirement of building plugins from scratch every time.

    Last-Mile Monitoring

    A monitoring platform deployed in the same cloud or data center environment where the applications also run cannot check on the end-user experience. To address that gap, there are several SaaS products are in the market, such as Catchpoint and Apica. These services are backed up by actual infrastructure to monitor the applications in specific geographical locations. For example, if you are keen on knowing how your mobile app performs on iPhones in Chicago, that could be tracked using the service provider’s testing infrastructure in Chicago.

    Alerts are setup on these tools to notify the site reliability team if the application is not accessible externally or if there are performance issues with the application.

    Log Aggregation

    In a production environment, huge amount of information is logged in various log files, by operating system, platform components and application. They will get some attention when issues happen and normally are ignored otherwise. The traditional monitoring tools like Nagios couldn’t handle the constantly changing log files except for alerting on some patterns.

    The advent of log aggregation tools like Logstash, Loggly and Splunk changed that scenario. Using the aggregated and indexed logs, it is possible to detect issues that would have gone unnoticed earlier. Alerts can be setup based the info available in the indexed log data. For example, Splunk provides a custom query language to search index for operational insights. Using APIs provided by these tools, the alerting could actually be integrated with the main monitoring platform.

    To leverage the aggregation and indexing capabilities of these tools, structured data outputs can be generated by the application or scripts that will be indexed by the log aggregation tool later. Such data aggregation can be used for generating data for reporting applications. For example, if storage usage on a set of computing nodes has to be tracked, a daily storage usage report can be generated on related nodes which the aggregation tool can track. Weekly and monthly aggregates could also be computed once the daily aggregates would be available.

    Monitoring the Monitoring

    It is important to make sure that the monitoring infrastructure itself is up and running. Disabling alerting during a deployment and forgetting about enabling it later is one of the common oversights I have seen in operations. Such missteps are hard to monitor and only improvement in deployment process could address such issues.

    Pinging hosts

    If there are multiple instances of the monitoring application running, or if there is a stand-by node, then cross checks can be implemented to verify the availability of hosts used for monitoring.

    In AWS, CloudWatch can be used to monitor the availability of an EC2 node.

    Health-check for monitoring

    Checking on the availability of monitoring UI and activity in monitoring application’s log files would ensure that monitoring system itself is fully functional and it continues to watch for issues in production environment. If a log aggregation tool is used, tracking monitoring application’s log files would be the most effective method to check if there is activity in the log file. The same index can be queried for any potential issues also by using standard keywords like “Error”, and “Exception”.

    Conclusion

    Monitoring efforts are normally in response to issues happen in production. A systematic approach to rolling out monitoring can minimize the reactive ways it is normally associated with. Proactive monitoring adds to better user experience, and, it avoids costly data reprocessing and rollback in production.