Category: DevOps Toolbox

  • How Python is Transforming the DevOps Landscape

    How Python is Transforming the DevOps Landscape

    DevOps is the combination of operations and software development which reduces or almost eliminates the disconnect between systems administrators who run the infrastructure and software developers who develop applications.

    During software development, when the Agile methodology is utilized properly, a bottleneck often occurs during the frequent operations and deployment phases. New fixes and updates are produced so rapidly in every sprint that the infrastructure teams can be swamped with deployments. To eliminate few of these issues, the operations personnel and application developers are asked to work closely to automate the code delivery to production from deployment.

    As DevOps is an approach for handling change and agility, engineers must master multiple languages. The Python programming language is one of the most crucial components of the DevOps toolchain. Many DevOps teams utilize it for building web applications for visualizing custom data, building custom utilities and more.

    Ansible and other tools are written in Python, which means you can build custom scripts or modules to automate your tasks, for example, or do other things.

    Reasons for Using Python for DevOps

    The accessibility and flexibility of Python are reasons why it is a preferred language for the DevOps toolset. If you are working in the role of DevOps, you must require an adaptable skillset; you must know how to work with multiple languages. For anyone looking to explore new tools and languages, and are curious about technology, the language can form a solid foundation, as it doesn’t require the commitment levels that a specialist language requires.

    In addition, it is a great scripting language, and scripting refers to automation. A few major configuration management tools, including SaltStack and Ansible, are written in Python underscores. This shows how useful the language is when it comes to orchestration and infrastructure automation.

    Python and Ruby are compared often, and with good reason: Both are highly accessible and utilized in applications developed by many organizations. They also feature in the DevOps toolset. We can do almost all the things with Ruby that we do with Python. But, when it comes to syntax, Python gains the upper hand, as it is much more direct than the syntax of Ruby.

    In the realm of learning to debug problems and code, Python has the advantage over Ruby, as it takes a much more direct approach to programming. Its directness and simplicity are invaluable if you are working in DevOps environment and agility is highly important.

    Conclusion

    As a growing number of engineers and developers take on decision-making responsibilities, Python can be the stabilizing language. And not only for DevOps—the language also can be used for developing blockchain, which is the underlying technology for various cryptocurrencies such as bitcoin wallet. As it can be utilized in various ways, it enables you to be open to broad range of technical possibilities. Python is the go-to programming language for the DevOps landscape.

    — savaramravindra

  • 4 Amazing Quantum Computing Applications

    4 Amazing Quantum Computing Applications

     Researchers across the globe, from the quantum researchers at University of Maryland synthesizing time crystals, to the research team at the University of New South Wales who are embedding atoms and their electrons into silicon, are pushing quantum computers forward, attempting to arrive at the most powerful quantum computing technology.

    IBM, Google and Microsoft are racing for quantum supremacy. IBM expects to create a 50-qubit general purpose quantum computer “within the next few years.” Indeed, the next iteration in quantum computing may be near.

    Why? Quantum computing could be the best path to ultra-powerful computers, and our best bet to solve unimaginably difficult problems. As our need for powerful processors continues to increase, and problems become larger in scope and complexity, we’ll need more complex computational systems to power solutions.

    Quantum Computers in Use

    Here are four amazing applications of quantum computers:

    Machine Learning

    Artificial intelligence needs to be able to pull from large datasets of image, video and text. Thankfully, there doesn’t seem to be a shortage of content. In fact, there may be an overabundance. Big data is out there to be analyzed, but we need more powerful computers to process the petabytes of unanalyzed data.

    Quantum computers could empower machine learning by enabling AI programs to search through these gigantic datasets concerning medical research, consumer behavior and financial markets—and make sense of them.

    Optimization

    Imagine, say, you are a traveling salesman. You wish to visit a handful of cities and want to know what the most optimal routes would be. This would be an example of an optimization problem. It sounds simple enough, but, in reality, the process can get quite involved as you up the number of variables. With only 270 destinations, for example, there are more combinations of travel than atoms in the universe.

    With quantum computers, however, we could expect a machine to be able to handle almost innumerable permutations and combinations, which could advance system design and analysis in massive ways.

    Biomedical Simulations

    With quantum computers, we can create, simulate and model molecular structures. Researchers at Harvard University used a D-Wave One quantum computer to solve the puzzle of how some proteins fold in 2012.

    “The model consisted of mathematical representations of amino acids in a lattice, connected by different interaction strengths,” writes Geoffrey Brumfiel in a news article for Nature about the Harvard researchers’ protein folding models. “The D-Wave computer found the lowest configurations of amino acids and interactions, which corresponds to the most economical folding of the proteins.”

    While the technology was nowhere near perfect—as the researchers cited that “10,000 measurements using an 81-qubit version of the experiment gave the correct answer just 13 times”—it was able to accomplish an amazing feat by modeling the behavior of protein folding with some degree of accuracy.

    Financial Services

    Quantum computing is already on its way. D-wave, a company backed by Goldman Sachs and Bezos Expeditions, among others, deployed its first commercial quantum computer: the D-Wave 2000Q, a quantum annealing system with 2000 qubits and advanced feature controls.

    Despite their prohibitive price, these computers are being utilized by a small niche, as illustrated by Harvard’s use of D-wave’s first model back in 2012. The systems could be used for complex financial modeling and risk management within the financial industry as well. Quantum computing could be used to find “new ways to model financial data” and isolate “key global risk factors,” according to IBM.

    “It would be great to build systems to help Wall Street better manage risk using this type of technology,” D-Wave Systems President and CEO Vern Brownell told Bloomberg. “They spend a lot on computing power [to manage risk].”

    Conclusion

    From managing money to massive datasets, quantum computing’s applications are seemingly endless. Advances in machine learning and optimization could boost efficiency dramatically. Biomedical and chemical simulations could help us land more drug discoveries and uncover new medical treatments in record time. Advanced computing power could lead to more than just innovation; it also could lead to lessened risk. Indeed, enhanced financial services could fundamentally change how we invest.

    As tech companies, computer scientists, and theoretical physicists help to perfect this technology, these truly amazing applications of quantum computing could become ubiquitous—and drive us further into a quantum age.

    — Ellie Martin

  • Akamai Extends Scope of DevSecOps Reach

    Akamai Extends Scope of DevSecOps Reach

    Akamai pressed its case for becoming a strategic DevSecOps platform by adding a range of security capabilities to its network service platform that help isolate applications from cybersecurity threats.

    The company has added support for scripting to automate application security using its Kona Site Defender web application firewall as well as improved management and versioning support for application programming interfaces (APIs). Those capabilities complement an Akamai API Gateway for securing and managing APIs.

    Akamai is also updating Akamai Bot Manager, adding a client SDK for mobile applications to help mitigate distributed denial of service (DDoS) attacks. The company also is adding capacity to its scrubbing centers to further defend against large-scale DDoS attacks.

    Ari Weil, vice president of product marketing for Akamai, said the company is evolving into a platform that securely provides web applications with access to caching software and network infrastructure that dynamically scale up and down as required.

    The goal is to make the Akamai network a seamless extension of any DevOps processes that developers can programmatically invoke on demand regardless of whether their applications reside in a public cloud, on-premises or in a third-party hosting center, said Weil. Because Akamai sits between users and production applications, Akamai now plays a critical role in terms of bother DevOps and DevSecOps, he said.

    In term of accomplishing that goal, Weil conceded Akamai doesn’t have as much visibility in the DevOps community as it would like. The company has integrated its platform into continuous integration/continuous development (CI/CD) platforms such as Jenkins to reduce as many steps as possible for DevOps teams. But developers are not always fully cognizant of the security services that Akamai makes programmatically available, he said.

    Weil noted that as DevSecOps continues to gain ground, Akaimai expects many developers will increasingly appreciate those security services, especially as cyberattacks become both more frequent and lethal. A 2017 security report published by Akamai finds DDoS and web application attacks rose 14 percent and 10 percent, respectively, year over year.

    In effect, Akamai turns web application security into a large-scale zero-trust managed service that is available on thousands of points-of-presence (PoPs) around the world. In time, developers will only have to express their intent concerning the level of security to be applied. The execution of that intent will all be automated, Weil said.

    In an ideal world, all cybersecurity battles would be fought as far away from applications as possible. Rather than focusing solely on the edge of the corporate network, Akamai is encouraging DevOps teams to think in terms of layers of defense that starts where applications are cached at the edge of the network. In theory, platforms such as Akamai provide a lot more visibility into what types of cyberattacks are circulating in the wild hopefully long before they ever get a chance to impact an application. The challenge, however, lies in distinguishing between what security technology needs to be embedded in the application itself versus an on-demand service that’s only a simple API call away.

    — Mike Vizard

  • Google Taps Netflix for Open Source Canary Analysis

    Google Taps Netflix for Open Source Canary Analysis

    Google and Netflix today jointly announced an open-source automated canary analysis service, dubbed Kayenta, that applies statistical analytics to optimize DevOps processes.

    Andrew Phillips, a product manager for Google Cloud Platform (GCP), said goal is to make it easier to consume advanced analytics by integrating Kayenta with the open source Spinnaker continuous delivery pipeline platform developed by Google. However, because Kayenta is based on software developed by Netflix, an IT organization can feed data from any cloud or on-premises IT environment into the canary analysis service, said Phillips. Kayenta also can ingest data from a broad range of monitong tools, including Stackdriver, Prometheus and Datadog.

    Phillips said the goal is to continually iterate on Kayenta to make it simple enough for almost any organization to take advantage of canary analysis, which enables DevOps teams to track metrics such as memory usage of a critical component. Kayenta fetches user-configured metrics from their sources, runs statistical tests and provides an aggregate score for the canary. Kayenta can automatically promote or fail “the canary” or trigger a request for human approval based on those test results.

    Specifically, Kayenta creates a pair of control/experiment time series datasets and calls a canary judge, which performs statistical tests, evaluating each metric individually and returns an aggregate score from 0 to 100 using preconfigured metric weights. Depending on the configuration, the score is then classified as “success,” “marginal” or “failure.” Success promotes the canary and continues the deployment, a marginal score triggers a human approval path, and a failure triggers a roll back.

    The basic canary concept involves routing a small subset of production traffic through a deployment that reflects the changes (the canary) and a newly deployed instance that has the same code and configuration as production environment, which serves as the baseline for testing. The analytics engine then compares the metrics between the canary and the baseline to identify any significant impacts the changes might have on the production environment. Kayenta in effect provides a mechanism for modeling risks based on proposed changes to the production environment, said Phillips.

    Google and Netflix expect third-party vendors will soon be incorporating Kayenta into a variety of DevOps tools and cloud platforms. In the meantime. Google views Kayenta as one more way of enticing DevOps teams to take advantage of Spinnaker to manage their DevOps pipelines in the cloud.

    Historically, setting up a canary analysis system has been a complex endeavor requiring significant engineering skills. Google is moving to democratize canary analysis by making it available as a service that can be invoked on demand, said Phillips.

    Phillips said Kayenta provides access to data that can be used to drive a wide range of predictive analytics applications that can be used to automate DevOps processes. It will be a while before Kayenta data is used to drive artificial intelligence (AI) models to automate those DevOps processes.

    Given the rate at which applications are being added and updated in production environment relying on manual testing is quickly becoming unfeasible. The days when DevOps teams needed to manually create their own statistical analytics models are clearly coming to a merciful end.

    — Mike Vizard

  • Early Automation: A Key Requirement for DevSecOps Success

    Early Automation: A Key Requirement for DevSecOps Success

    According to the “2017 DevSecOps Community Survey,” by Sonatype, almost 60 percent of the respondents consider security to be an inhibitor to DevOps agility, while more than 50 percent of developers say they do not have sufficient time to allocate to security.

    Hence arises the need for enhanced security automation in DevOps. Security automation helps in refining and improving application security by reducing the time taken for detecting an attack and responding to any identified issue.

    Tools for Security Automation

    Organizations looking to implement DevSecOps by integrating security into their DevOps pipelines have an increasing number of tools such as Tanium, InSpec, Splunk, Checkmarx, Metasploit, FireEye and Contrast Security for security analysis and testing throughout the software development life cycle (SDLC), right from source-code analysis to post-deployment monitoring.

    Early Automation, A Key Requirement for Success Through DevSecOpsFor security to become an integral part of the DevOps workflow, which includes continuous integration and continuous deployment (CI/CD), automation is essential. As most organizations are pushing tens of new versions of code into production for each application every day, it becomes necessary to embed security controls early in the development life cycle, which is possible only through automation.

    Around 40 percent of the 2,300 IT professionals surveyed have said that they have deviated from the waterfall development model, where automated security tests are run just ahead of production, by running them throughout the development life cycle.

    Using SAST Tools

    Static application security testing (SAST) tools help in scanning code and providing instant feedback to developers on security-related issues who, in turn, can remediate the potential vulnerabilities as part of the standard workflow.

    However, static analysis may not be alone sufficient for detecting all problems in the code.

    Automating DAST

    Automated dynamic application security testing (DAST) searches for vulnerabilities in real time while the application is running and is really a major improvement over static analysis that only looks for potential security issues in the code.

    Inclusion of automated security analysis helps in limiting the introduction of vulnerable code earlier in the development life cycle. The runtime analysis of the issues detected through automation enables developers to prioritize the code problems that need to be fixed.

    Red Teaming and Threat Modeling

    Organizations have begun to employ the concept of red teaming, a military terminology, by having separate teams to simultaneously take up attacking the code and performing security testing respectively. This ensures that any potential flaws are uncovered in the production stage itself and code fixes are pushed as soon as possible.

    Threat modeling is critical for the success of DevSecOps as it leads developers to look at software from an attacker’s perspective. While the threat modeling process cannot be automated, it helps in identifying flaws in the application design and architecture better than other security approaches.

    Are You Ready for Automation?

    Organizations can automate the provisioning and commissioning of DevSecOps environments to achieve a predictable, consistent and secure service delivery.

    While the goal of DevOps Services automation is to help development teams in faster deployment and monitoring of applications, DevSecOps adds security to the automation and contributes to enhancing the quality and efficiency of the software.

    DevSecOps adoption provides organizations with a strong foundation for mitigating risk in a proactive, efficient and streamlined manner.

    — Veritis

  • NGINX Extends Microservices Ambitions

    NGINX Extends Microservices Ambitions

    NGINX has extended the reach and scope of a namesake application platform that the company is positioning as a bridge between legacy applications and emerging microservices architectures.

    The NGINX Application Platform now includes NGINX Unit 1.0, an open source application server designed to simplify microservices across multiple application types simultaneously using a single instance of a server. It supports Go, Perl, Python, PHP and Ruby, with additional support planned for Java and JavaScript support soon. Sidney Rabsatt, vice president of product management at NGINX, said NGINIX Unit 1.0 is different from other application servers in that it is configured via an application programming interface (API) and is 30 percent faster than existing app servers.

    NGINX in the second quarter plans to add new monitoring dashboards, alerting and management via NGINX Controller R1 offering that provides the capabilities of a service mesh for microservices without having to set up a separate dedicated framework. Service meshes are a critical capability because they provide a means to manage interdependent microservices at scale.

    Finally, NGINX Plus R15 for enterprise customers builds on new HTTP/2, gRPC and JavaScript features that have been added to the core open source NGINX platform to provide support for high-availability clustering and OpenID Connect capabilities. These capabilities extend the ability to deploy NGINX as a web server, load balancer, API gateway, Kubernetes Ingress controller and sidecar proxy, said Rabsatt.

    Rabsatt noted most IT organizations do not want to set up separate infrastructure for managing microservices. Managing legacy and microservices-based applications in DevOps environments will be a fact of life for years to come, he said, adding that based on the number of downloads of NGINX software being used on Kubernetes platforms running Docker containers, that message is already resonating.

    NGINX claims users have pulled a billion NGINX instances from the Docker Store, including more than a million pulls of the NGINX Kubernetes Ingress controller. According to the company, more than 3 million NGINX instances are deployed in production microservices environments across more than 250 customers. That’s critical for the company because some rivals have positioned service meshes as an existential threat to the NGINIX platform. Overall, NGINIX says there are more than 11 million instances of NGINX are downloaded each month, an average of four instances every second.

    Rabsatt said most developers have made it clear they don’t have the patience to wait for IT operations teams to stand up dedicated microservices infrastructures. Most IT operations teams still don’t have the skills required. NGINX provides the ability to deploy microservices at scale using a platform most developers already know how to programmatically invoke on their own, he said.

    At the same time, NGINX is also committed to making available best practices that DevOps teams can employ when trying to manage what amounts to a new type of hybrid computing environment spanning legacy applications and microservices, he added.

    It remains to be seen exactly where the divide between microservices and legacy applications will be laid down inside most organizations. Some will draw a proverbial line in the sand, while others will look to deconstruct existing applications into series of microservices. Regardless of the approach taken, all those existing legacy applications will not be disappearing anytime soon.

    — Mike Vizard

  • Can Videoconferencing Solve the DevOps Collaboration Conundrum?

    Can Videoconferencing Solve the DevOps Collaboration Conundrum?

    Every DevOps expert has their own definition of what DevOps really means, but a common thread running through each definition is that it is based on collaboration and evolution of a culture of sharing and communication between IT and operations. How that collaboration is achieved has evolved rapidly, and in today’s most successful digital transformation initiatives, video integration is playing a significant role in improving that teamwork.

    Success tactics in a DevOps initiative depends less on the specific tools and more on that theme of creative collaboration, yet many tools claiming a DevOps heritage actually do very little to facilitate that collaboration. Development platforms, for example, do achieve the goal of faster development, which is a goal of DevOps, but DevOps is less about the deliverable and more about the journey. Tools that facilitate better collaboration between teams, such as videoconferencing, lie at the heart of that DevOps journey.

    How businesses handle issues such as videoconferencing decides, to a large degree, how effective and successful they are in a globally connected marketplace and how well they achieve the twin goals of faster software delivery and a final product that meets the needs of the business. 

    Solving the Essential Development Struggle

    The fundamental development struggle is based on a constant battle between developers and line-of-business managers, who often have different goals and different perspectives on how to achieve those goals. Developers may not have a full grasp of the business goals of their projects, while the managers may fail to recognize the technical challenges being faced in delivering the final product.

    Agile, lean and Scrum development strategies do help to hold the development team accountable to the desired timeline of the business, but beyond these tactics, simply understanding the “big picture” of the project is what makes it work and avoids developers getting bogged down in less important details. How, though, does the development team facilitate this understanding?

    The obvious answer is collaboration, but how is that collaboration achieved? Emails? Nobody pays attention to them. Conference room meetings with whiteboards? People will pay attention half the time and then get bored. Team messaging tools? They do facilitate workflow and communication, but still often lack the real-time interaction necessary for getting a true understanding of the project’s requirements.

    Videoconferencing Integration Meets the DevOps Challenge

    New forms of collaboration have been a major catalyst for new development initiatives over the past year, with cloud-based systems have shown the greatest potential. Simple tools such as Office 365 or Google Docs allow users to share and co-edit documents, while Slack, Zoom and other communication tools allow for easy group communication. However, videoconferencing has been shown to have the greatest potential because of its real-time and highly visual nature, and its use has become much more practical with the ubiquity of 4G and high-quality cameras on smartphones and laptops.

    DevOps is as much about creating a culture of collaboration and sharing as it is about specific development tools and tactics. Videoconferencing facilitates the cultural aspect of DevOps, an especially important factor given that teams are often in different physical locations. Even when those functional teams are in the same building, it is common for those teams to operate in a silo. Creating an inclusive and interdepartmental DevOps team is always the starting point, but getting those teams to communicate requires a cultural change—and a little boost from videoconferencing and other collaboration tools.

    Integrating and Executing Videoconferencing in DevOps Environments

    When it comes to video conferencing in the workplace, businesses have to stop taking a laissez-faire approach to integration and execution. While they may assume it’s as simple as plugging in a solution and getting to work, it often requires much more technical competence and involvement from the appropriate parties. 

    For starters, the IT department needs to be involved in its deployment. There’s a lot to consider from a technical perspective before you can even begin to think about details related to specific software solutions and platforms.

    Cloud-based systems have made deployment of videoconferencing and other tools simple, and there is a growing tendency for departmental managers to source and deploy their own tools in a “shadow IT” approach. But since the goal is inter-departmental communication, in this case IT should play the central role in its deployment and execution.

    One of the issues organizations need to think about is how the integration of video technology will affect current processes. The goal of video is to make things more efficient, not to drag processes down.

    It’s also wise to consider how much videoconferencing will be used and whether it’s better to have multiple small-video equipped rooms or one larger, more expensive boardroom setup.

    While IT involvement is vital for successful implementation and execution, it’s not the only factor involved. Failing to involve outside sources can actually result in an incomplete setup.

    “IT departments are incredibly adept at managing network, servers, processes, and internal applications. They know their business and the needs associated with it,” noted videoconferencing provider IVCi in its blog. “However, in certain circumstances deploying videoconferencing can require an organization to act more like a service provider, rather than an IT department. IT departments are rarely experienced in video conferencing implementations and are forced to learn while doing.”

    This lack of experience has become increasingly evident over the past few years—something that The New York Times recently exposed in a story on the lack of security in many videoconferencing systems in major organizations.

    The article exposed a story in which a security chief assumed the role of “hacker” to find holes in his client’s network security. What he found was alarming. While companies spend millions on securing their networks, he was able to hack into multiple conference rooms relatively and control the cameras with relative ease.

    “Many systems are sold, installed and maintained without any level of access security, with auto-answer enabled by default,” the “hacker” HD Moore said. “It boils down to whether organizations are aware of the risk, and our research indicates that many, even well-heeled venture capital firms, were not aware and do not implement even the most basic of security measures.”

    It’s not just security, though. Many businesses simply don’t know how to use their video conferencing solutions. The IT department might understand how the system works, but individual users don’t even know how to start a video session. This results in cumbersome processes that reflect poorly on the business, especially when video is used in client-facing situations.

    It’s Time to Take Video Seriously

    While business leaders spend a lot of time talking about how to be successful with videoconferencing, in terms of knowing what to wear and how to interact with participants, very few take implementation, security and execution seriously. Until it’s given the priority it deserves, videoconferencing will lag in organizations.

    — Dan Blacharski

  • VMware Adds Analytics to Automate IT Operations Based on Intent

    VMware Adds Analytics to Automate IT Operations Based on Intent

    VMware is moving to make it easier for organizations to automate data centers and public cloud based on its core hypervisor technology by injecting advanced analytics into its VMware vRealize Suite, enabling them to manage IT environments based on intent rather than having to manually manage every process themselves.

    At the core of those update is the Capacity Analytics Engine, which is being added to version 6.7 of vRealize Operations. Mahesh Kumar, senior director of product marketing for cloud management at VMware, said the capacity analytics engine makes use of Autoregressive Integrated Moving Average (ARIMA) techniques to forecast future system events based on historical data points that identify a long-term trend of sudden spikes in usage. Armed with that data, vRealize Operations can now provide cost analytics that include advice on ways to adjust application size and reclaim idle resources on either a public cloud or inside a local data center. IT organizations can also run “what if” scenarios, Kumar said.

    On top of that capability, VMware is also now adding predictive analytics that enables IT organizations to define a set of intents such as workload balancing to reduce software license cost, prioritize one set of application to maintain a service level agreement (SLA) or make sure certain workloads only run on servers that have been shown to comply with a specific regulation or mandate. Tighter integration between vRealize Operations 6.7 and vRealize Automation 7.4 then can be continuously employed to automate DevOps processes in a closed-loop manner at higher levels of scale without having to add additional headcount, Kumar said.

    In addition, the latest version of VRealize Automation includes more than 120 free, curated blueprints and OVF (Open Virtualization Format) files out of the box. VMware has teamed with Bitnami to add 20 new blueprints and 100-plus new OVFs of applications and databases including GitLab, Hadoop, Jenkins and MongoDB to speed application development and deployment. VMware also updated the Custom Form Designer and added multitenancy capabilities.

    Finally, VMware vRealize Operations is sporting a new user interface and integrates with VMware Wavefront monitoring software.

    Longer term, Kumar said vRealize Operations will be where VMware applies machine learning algorithms to enable artificial intelligence (AI) models to automate the management of VMware environments.

    Collectively, Kumar said the VMware vRealize Suite moves IT organizations that have standardized on VMware in both their local data centers and in the public cloud much farther down the DevOps maturity curve. In effect, VMware vRealize Suite creates an “easy button” for managing complex VMware environments spanning multiple clouds, Kumar said.

    VMware is also betting that AI models will make it a lot easier for organizations to make a cultural adjustment as the management of compute, storage and networking becomes more unified. It’s not clear how long it will take for those machine learning algorithms to manifest themselves in VMware management software. But it’s clear most IT organizations running VMware soon will need to rethink who is responsible for running what inside an IT operations team.

    — Mike Vizard

  • Using Inspec to Validate Infrastructure

    Using Inspec to Validate Infrastructure

    So, here we were, at one of the bigger financial corporations in New Zealand, working on an elaborate infrastructure built on Docker containers. It had configurations, SSL certificates, DNS entries and all the other condiments required to spice up a tasty infrastructure curry. Though we were able to stand up our complete infrastructure environments from scratch using automated configuration management tools, we weren’t really sure if it was indeed working correctly, or rather the way it was supposed to. The question staring at us was, Is there a way we can declare the desired state of our infrastructure and then validate our infrastructure against it?

    Inspec to the Rescue

    The open source tool Inspec has been used with considerable success over the last few years to achieve security compliance for infrastructure. The syntax/constructs happens to be unbelievably simple. For instance, let’s consider the code snippet below. It doesn’t really require a lot of programming knowledge to make sense of it. Isn’t it?

    This made us wonder, what if we use Inspec to validate our infrastructure setup? What if we map our infrastructure configurations in an easy-to-read declarative syntax? What if this actually solves our problem?

    In line with the best practices of DevOps, we started with a simple experiment; an experiment that could solve a tangible problem. We had a Docker host which was supposed to have a particular docker container running inside it, but it used to fail quite often, for some reason. We decided to write a test in Inspec to check whether this container was running on this host:

    … and tried executing the test.

    This was just what we wanted!

    Excited with this success, we started writing a battery of tests that validated each and every component of our infrastructure. Within a single day, we churned out 192 tests. The number of tests wasn’t as important as the scope and breadth of the test coverage. Once the tests were there, we plugged them on to our infrastructure pipeline (which was built using Jenkins) to ensure the components align with the desired state. We called them ‘component-readiness’ tests.

    So far so good.

    Now that our components match the desired state, are we still confident that our platform would work the way we intended? Perhaps we need some “platform readiness tests” as well. Can we use Inspec for that, too? The answer: Yes!

    Now that the platform readiness tests were ready, we extended our infrastructure pipeline to incorporate them as well.

    Setting our component and platform readiness tests using Inspec had one more fantastic offering: live documentation for infrastructure. Inspec allows the audit results to be displayed in a variety of report formats, including JUnit, JSON, HTML and others. We used the JUnit report format and displayed them using the Jenkins JUnit Plugin. With this, we were able to have human-readable JUnit reports that doubled as live documentation for our infrastructure.

    Needless to say, this infrastructure testing with Inspec provided that (much needed) confidence on our setup and readiness. Our pipeline is designed in such a way that whenever any infrastructure component changes in our ecosystem, the pipeline is triggered which reconfigures and (if required) rebuilds the ecosystem. These Inspec tests are triggered with this infrastructure pipeline and validates the newly configured infrastructure setup. Thanks to this, we are never in a situation in which the client application running on this infrastructure complains or aborts because of incorrect configurations. Moreover, as described above, the living documentation helps document the expected configuration of the environment.

    So far, the Inspec implementation I have referred to is using the freely available Inspec tool and the Jenkins platform. Inspec can also be combined with Chef Compliance to produce more descriptive reports and charts. Chef Compliance also comes with out-of-the-box CIS profiles that can be used to jump start an Inspec implementation for security compliance and audits.

    Inspec is an easy tool to master with simple and intuitive constructs and one could get up and working quite easily. However, this simplicity has a downside. It’s very easy to speed ahead and gloss over the documentation while missing out on the best practices and tips/techniques to implement the solution in a better way. One way to master this is to have a crack at the newly released Auditing with Inspec certification, which provides the impetus to go through the documentation in detail while preparing for the certification. And, you also get to show off a shiny badge for your efforts!

    In Summary

    Infrastructure validation is quite an important step in any DevOps implementation. Apart from being very useful for security compliance testing, Inspec can be used to validate infrastructure using a simple and easy-to-understand declarative syntax. Moreover, it has a small footprint, a vibrant community looking after it and up-to-date and clear documentation. It definitely is a tool with the potential to provide quick and effective results.

    — Mrinal Mukherjee

  • Google Unveils Suite of APM Tools

    Google Unveils Suite of APM Tools

    Google today announced that it is making available application performance management (APM) tools based on the same technology it employs to manage its internal IT environment.

    Morgan McLean, a product manager at Google, said Stackdriver APM is based the open source StackDriver Trace and Debugger tools Google originally developed for its DevOps teams.

    The APM offering, available in beta, also includes Stackdriver Profiler, a tool Google developed to enable DevOps teams to explore how their code is running in a production environment.

    Google also said it has integrated Stackdriver Debugger with GitHub Enterprise and GitLab to extend the number of repositories developers can use to create mirrors of their code.

    DevOps teams will have the option of deploying Google APM tools either on-premises or on a public cloud.

    McLean noted that rather than throwing additional compute resources at a performance issue, Stackdriver Profile enables DevOps teams to discover and remediate issues and ultimately reduce costs. To minimize the impact of Stackdriver Profile on the performance of a production application, Google opted to employ statistical methods to create a profile of a piece of code. Specifically, Stackdriver Profiler collects data via lightweight sampling-based instrumentation that runs across every instance an application. Data showing CPU time, wall time, RAM used and contention, for example, is then displayed on a flame chart.

    At the core of the Google APM toolkit is a the Stackdriver Trace software the company developed to collect latency data about applications deployed in a distributed computing environment. McLean said that in the age of microservices it’s not feasible to managed modern distributed applications without any ability to instrument them. For many IT organizations, however, the cost of acquiring APM software or employing a cloud service has forced them to limit their use of APM technologies to only their most critical applications.

    Google is trying to remove cost as significant APM factor, which ultimately will result in the applications deployed on the Google Cloud Platform (GCP) becoming more robust and reliable. The Google APM tools can be applied to applications running on virtual machines and containers, as well as the Google AppEngine, a framework Google exposes on its cloud service to enable the development and hosting of applications. In effect, Google is making it easier for organizations to consume open source tools that IT operations previously would have had to stitch together on their own.

    Reliance on APM tools is at the heart of any DevOps process. But DevOps processes are not universally employed, largely because in addition to acquiring new tools, organizations must evolve their processes and culture. The debate many of them have comes down to an argument over whether they need to buy tools before they change those processes or change their processes to justify buying the tools. Most often, it’s the initiative of handful of developers that force the issue. It’s only once that team is successful that senior managers start to think about replicating the DevOps process employed by that team across the rest of the organization. Via Stackdriver APM, Google is clearly hoping to accelerate the rate at which those transition to DevOps occur.

    — Mike Vizard