devopsy.kinsta.cloud

Author: contributor

  • Differentiating Between CI Pipelines and DevOps Assembly Lines

    Differentiating Between CI Pipelines and DevOps Assembly Lines

    “What’s in a name?” wrote the greatest Bard who ever lived.

    All I can say is, he lived in simpler times, when DevOps and the confusing world of continuous integration (CI), continuous deployment (CD), CI/CD, the other CD, application release automation (ARA), pipelines, assembly lines, etc., didn’t exist.

    The biggest challenge today is that vendors use overloaded terms to describe completely different use cases. CI vendors claim to support CI/CD, even though the “D” here is just a deployment to a development environment and not staging or production. ARA vendors claim to enable continuous delivery, even though they do not include critical steps such as infrastructure provisioning, security patches or configuration management.

    The terms “pipelines” and “assembly lines” are relatively new. Multiple vendors have recently announced pipelines and assembly lines, and the differences aren’t obvious at first glance. So let’s do a deep dive.

    What are CI Pipelines?

    Continuous integration has evolved tremendously over the last few years. What started as a simple process of automating build and unit tests for each code change has evolved into a pretty complex workflow.

    As an example, look at the different CI workflows shown below:

    In classic CI, all build and test instructions were included in a single blob and instructions were executed serially. If all instructions executed successfully, CI was successful, and if anything failed, then it was marked failed. Almost all CI providers, until recently, offered just classic CI.

    However, as CI became mainstream, developers wanted to define more complex workflows with decision trees and parallelism. CI pipelines support this complexity, such as:

    • Defining the CI workflow in stages, so you can get results from a stage faster without having to wait for the entire workflow to finish. This also enables scenarios such as running a basic test group and only running the extended test suite if the basic tests pass.
    • Running stages in parallel, so you can split tests or test your code against different environments or language versions, and get faster feedback.
    • Forking your CI logic based on the results of a stage.

    While CI pipelines are a huge improvement over classic CI, they are still limited to developer-focused workflows. Jenkins, Teamcity, Shippable, Gitlab and CircleCI all support some form of CI pipelines.

    What are DevOps Assembly Lines?

    DevOps assembly lines are focused on automating and connecting activities performed by several teams, such as CI for devs, infrastructure provisioning and configuration management for ops, test automation for test, security patching for SecOps, semantic versioning and approval gates for release managers, deployments for multiple environments, and so on.

    Today, most organizations use various tools for automating specific DevOps activities. However, the DevOps toolchain is fragmented and gluing it together to achieve continuous delivery is a challenging task.

    Most teams adopt one of the following approaches:

    • Using cultural collaboration as the glue between silos. As an example, when a new application version is available, the test team creates a JIRA ticket and assigns to ops. Ops then copies the application version information into their deployment scripts and run them.
    • Writing ad-hoc scripts to trigger one activity from another, including passing state information through intermediate storage such as S3. In this case, the DevOps engineer would likely write a Jenkins job that runs automated tests, and then pushes application version information to S3. Another Jenkins job polls the S3 bucket, and when a change is detected, runs the deployment scripts.

    The first approach introduces inefficiency and unnecessary human-dependent steps. The second approach is better, but only works well for small teams working on a single application. It creates a Frankenstein of spaghetti automation when you have many applications or even a single application with microservices.

    DevOps assembly lines address this problem by focusing on gluing together these various DevOps activities into streamlined, event-driven workflows with the ability to easily share state and other information across activities.

    The following image shows a typical assembly line for a single application or service:

    See the difference? The CI pipeline is just ONE activity in the assembly line. Each of the boxes in the picture above is a pipeline, just for different activities! Each has specific needs with respect to tools integration, runtime, configuration, notifications, etc. In addition, each pipeline is owned by a different team, yet needs to interact with other pipelines and exchange information. For example, if the ops team changes provisioning scripts for staging environment, the test team needs to redeploy the application and run performance tests.

    A DevOps assembly line is, therefore, a “pipeline of pipelines.”

    DevOps assembly lines need to support the following:

    • Ability to easily define workflows across multiple pipelines.
    • Versioned and reusable workflows, enabling rapid changes and scaling for multiple applications and/or microservices.
    • Integrations with all popular source control systems, clouds, artifact repositories, DevOps tools, languages, services, etc.
    • Runtime to execute every pipeline, including pre-installed tools and CLIs, and the ability to use the right runtime, depending on pipeline type.
    • Accelerators/playbooks for common pipelines and tools.
    • Ability to pass state and other information while triggering dependent pipelines.
    • Automatic triggers or manual approval gates between each pipeline.
    • Configurable notifications for each stage of every pipeline.
    • Release automation features such as semantic versioning of packages.
    • Audit trails for each pipeline, with the ability to go back or forward to a specific state.
    • Abstraction of all sensitive information like passwords, tokens, keys, etc. for security reasons.
    • Roles and permissions restricting assembly line and pipeline actions.
    • Manage DevOps infrastructure, including spinning VMs and containers up and down as required.
    • Visibility into each pipeline and stage, including logs, status and versioned data.
    • Metrics and Analytics across pipelines to help identify bottlenecks.

    In short, DevOps assembly lines help automate and scale end-to-end workflows across all teams and tools, enabling continuous delivery.

    About the Author / Manisha Sahasrabudh

    Manisha Sahasrabudhe is co-founder and vice president of product management at Shippable. Prior to Shippable, Sahasrabudhe was senior program manager at Microsoft. She can be found online at Linkedin.

     

  • Digital Transformation: Thrive, Not Just Survive

    Digital Transformation: Thrive, Not Just Survive

    My conversations about innovation and the enterprise have afforded me an intimate understanding of the technological challenges companies are facing. Chief among these concerns, regardless of industry or sector, is digital transformation.

    Just the other day, a C-level financial executive came to me seeking guidance on this topic. He said his company was on a mission to become a lean, mean, digital machine, and was currently leading four or five initiatives to modernize operations and convert systems from legacy to new. “We’re moving everything to the cloud, we’re building new apps, and our teams are working toward agile methodologies,” he said, “I’m proud of my teams for all the hard work they’ve put in—in fact, they’re worn out from all their efforts! But honestly, I feel like all this work hasn’t paid off. We’re not seeing the dramatic progress that was promised. What are we doing wrong?”

    He’s not alone in his frustration. I’ve seen many organizations get excited about these kinds of changes, high on the notion that once they’ve knocked out a few initiatives, they will have conquered digital transformation and secured the future of their business through technological fortification. But I reminded him not to get too distracted by all of the technology jibber-jabber, and instead make sure to keep the big picture in mind: Digital transformation is, first and foremost, a fundamental shift in your business model.

    It isn’t simply about switching from outdated technologies to newer, slicker solutions. It’s not about transitioning to the cloud or performing a complete upheaval of legacy systems or processes. And it definitely can’t be accomplished in a few steps or one fell swoop. In reality, digital transformation is a journey that requires a road map and strategic planning to ensure all of your initiatives support the greater movement toward agile, streamlined and more efficient, and directly address and impact your business needs and objectives.

    To this end, I posed a critical scenario to this financial executive: Consider an industry competitor who is smaller, newer, more agile and less bound by the market. If this player were to become super successful, how would your company respond? And which parts of your business specifically would be responsive? His reply was that his company is adopting ideas from new-entry players, primarily online-only banks. He’s building a new mobile platform and revamping his web properties to make banking more efficient for his customers.

    While these efforts certainly have value and no doubt will improve how he does business, it’s not enough to constitute a digital transformation. These changes won’t be enough to drive the fundamental shift that needs to happen. It’s more of a blended approach, which won’t be enough. Hybrid models can allow companies to survive—but not thrive.

    We can draw similarities to the evolution of automotive technology. While internal combustion engine (ICE) cars have long dominated the market, about a decade ago electric vehicles gained real popularity as a superior alternative. However, there have been real obstacles in the way of pure electric vehicles going mainstream. For one thing, we have so much invested in the infrastructure to support ICE cars and very little in the way of making electric cars practical. While charging stations are expanding and becoming more common, we don’t have nearly enough to make pure electric travel the practical choice—for now.

    What has been our interim solution? Hybrid vehicles. In response to the electric contender, we’ve developed a number of vehicles that bridge the divide between infrastructure and efficiency needs. We created ICE-electric hybrids to harness the beauty of electric without having to sacrifice on practicality as the infrastructure works to catch up.

    So this is what our financial executive is actually doing: he’s going the hybrid route by borrowing some of the nice, fancy features from the new entry companies while keeping an entire foot of his company in the traditional ICE world. And though the hybrid movement has been serving us reasonably well with cars, it’s not a good approach when it comes to a technology strategy. In the world of enterprise tech, bi-modal IT is like a hybrid vehicle. Sure, it allows companies to say, “We’re getting more electric!” But if they make the mistake of designating a hybrid operating model as their end goal, they’ll be stuck with split IT personalities and bifurcated cultures.

    Organizations can get by on a hybrid approach, but only for so long. They may be able to modernize in part and reap some of the benefits of better technology, but it just gets too expensive to be sustainable. Hybrid necessitates investing in both types of infrastructure—legacy and new, gas stations and charging stations—to keep services running. In the long run, this holds companies back from truly taking advantage of all the benefits of electric, and they need to build transition technologies between ICE and electric components to make them run as smoothly and efficiently as possible.

    “Okay, so hybrid’s not the way to go. What do I do instead?” the financial executive asked. My answer is to create a space expressly for innovation. You should compile whatever resources you can to create an entirely independent division of your company (or even a separate company) that makes room for innovation. This arm would be free from the constraints of legacy systems, free to ignore the norms of its parent, and free to create its own rules and business model. Using our example from the automotive industry, you can think of it as akin to General Motors creating an entire, standalone electric division, rather than simply converting Chevy models into electric versions.

    While your organization may be in survival mode in the near term, you can’t lose sight of the pure electric play. Think about the inevitability of the electric car age: Battery prices will continue to plummet, driving ranges will increase, charging stations will multiply and the auto industry itself is investing in both EV and air pollution standards. We’re not putting all of our money on oil anymore.

    So as you move forward on your digital transformation journey, be mindful of your true end goal: to have your business thrive through this technology cycle and the next (and hopefully the one after that, too). Going hybrid is a necessarily survival mechanism, but only as long as it builds momentum and helps drive you toward success.

    About the Author / Abbas Haider Ali

    Abbas Haider Ali brings over 16 years of experience in networking, cloud services, software, and cloud communications. As CTO of xMatters, he is responsible for evangelizing the adoption of communications-enabled business processes, and has worked closely with more than 400 global enterprises and IT organizations to create a vision for adopting intelligent communication strategies across business scenarios. Abbas holds a BASc in Computer Engineering from the University of Toronto.

  • Incorporating Performance Engineering into DevOps

    Incorporating Performance Engineering into DevOps

    Performance testing historically has been a challenge for most IT organizations. It is highly misunderstood, from executives down to the few resources allocated to perform an extremely difficult task. Most companies consider performance testing expensive and unnecessary and simply refuse to acknowledge it is a critical component of the software development life cycle.  Have we forgotten about major performance failures such as Healthcare.gov? It is time companies start to take performance engineering seriously.

    IT organizations using the Waterfall methodology relied on a few performance testers to develop performance scripts and run them after most of the testing had been completed and the application was a few weeks from a production deadline. If the performance team was lucky, they would get in one or two successful tests, typically working off-hours and executing performance tests on small, patched-together ancient hardware. The problem with this approach is when issues are found, there is not enough time to fix them adequately and performance degrades over time.

    Developing a Performance Engineering Strategy

    Here are some items to consider when incorporating performance engineering into DevOps:

    • Learn—It is important to learn what performance engineering is and how it can be incorporated into your organization. If you don’t have the right skills within your organization, go find someone who can help. There are many organizations that can help with an initial assessment of where things stand.
    • Develop Your Strategy—This is perhaps were most teams fail. Upfront planning is critical and it will save you in the end. It is important to develop a performance engineering road map. This is not something that will be built in a few weeks; a solid performance road map could take 18-24 months, so be patient and build it right.
    • Build Your Baseline—It is important that you build a baseline of your performance tests so that you can determine if you have successful results. Performance testing is different from functional or automation because you have to consider the performance run as a whole. A few high transactions usually are acceptable. It is a good idea to build a history of runs to see how the application performance is trending.
    • Identify Key Business Workflows—It is too time-consuming to build hundreds of performance tests, so focus on those that are the most critical to your business. Typically those workflows will account for the majority of your performance load.
    • Test Upfront—With DevOps you need to start early and you will need to run performance tests frequently. The more you can run upfront, the more performance issues will be avoided in production.
    • Tools and Hardware—Performance engineering requires tools and hardware. It is important that your assessment evaluates what you currently have and what you will need to do it properly. There are many open-source tools on the market today that can help and other tools that are making a big difference in the performance space.
    • Budget—Developing a world-class performance engineering practice is going to take money. When you develop your performance road map, you might need two to three budget cycles to get the tools and infrastructure you need.
    • Load Performance Environment—Yes, you will need to have a performance environment that is scaled to match your production environment. If there isn’t enough budget, then you will have to know how much smaller the environment is so that you can be confident in your performance results.
    • Data—It is important to have data that is similar size to your production environment. This will help you understand how your system will handle performance within the production environment.

    Incorporating performance engineering within DevOps will require work, but the rewards will certainly put the organizations that embrace it at the top of the list. DevOps enables teams between Dev and Ops to produce a stable application that performs exceptionally well under load. Here are some benefits of DevOps performance engineering:

    • Predictable production performance results
    • Production deployment confidence
    • Eliminate catastrophic performance failures
    • Increased performance efficiency

    DevOps gives organizations a real opportunity to get performance engineering right; unfortunately, most organizations will carry over the same mistakes into DevOps. I hope you will take the opportunity to embrace the performance engineering challenge!

    About the Author / Ron Wilson

    Ron Wilson is a passionate champion of Agile, DevOps and Quality Engineering. He has managed large global Quality Engineering teams across multiple industries. He is currently the director of Quality Engineering at Arch MI. He regularly shares insights through social media where you can find him at www.qarevolution.com and on LinkedIn.

  • Understanding the Data Scientist’s Role in Cross-Functional Teams

    Understanding the Data Scientist’s Role in Cross-Functional Teams

    In our previous article, “Data Science Industry Perspectives in the Cloud,” we discussed that evolution is key if you plan to grow your business. You can start with ready-made solutions, then, in time, you can switch to in-house solutions created with the help of a group of data scientists.

    This time around, we’ll talk about the cases and solutions when machine learning as a service (MLasaS) doesn’t work. Your company’s first move is not to go out and hire a data scientist to solve your urgent business needs; rather, your company should invest in a custom-made solution. Only when you have a workable solution can you dive in deeper and create a proper team that can create an in-house data science solution.

    Custom Made End-to-End Solution as a Start

    Data Science in the classical sense involves data, goals and models to solve a pressing issue. The best way to jump start a data science process is by putting together various ready-made services into a single workable product to show your customers clear-cut results in short order. This can be done without any complex or global research, and you can comfortably formulate specifications taking into account all the feedback from your customers to create a more high-level data science offering later.

    One of the biggest issues for any data science project is in formulating the specifications for it. The usual request is, “Create something for my company using data science. Analyze data for me.” This type of a job has lots of trial and error. Having a custom end-to-end solution from the start lets you add on extra services as needed, and enables greater insights and predictions into a specific business workflow.

    I believe companies should build their data science system not from the data science research but from the point of view of an end-to-end solution. And then, bit by bit, they can adjust their services in the process.

    Moving on to Data Science Research

    Companies start data science research by employing a data scientist who matches their needs. You can use the standard data scientist qualifications: This employee should have a firm footing in the application environment, mathematics and programming. Also, their business skills are key to success—I believe an understanding of the topical area is key here because we are solving business issues. And a data scientist who solves problems that are more academic will be more focused on winning a Kaggle competition than on addressing the business needs.

    It is important this person understands the product development cycle. This way, the models and analyses data will be built in such a way that they could be deployed in production. For example, if a person is using R (the language and framework for data scientists), we should note that it is more research-oriented and it is not production-ready. Correspondingly, the results of such research cannot be deployed in any end-to-end solution. Therefore, the data scientist should work with a programmer.

    Although in the above diagram we see that the Data Scientist should have hacking skills, in reality, this is not the case. In their vast majority, Data Scientists are not able to write quality code. And if you need not just the research but a solution then process-wise you need to have a Data Scientist working together with a Data Engineer.

    CAP Theory as an Analogy

    It is impossible to have three database properties at once: consistency, availability and partition-tolerance. This basic rule is known to all developers. This is true for a data scientist as well: People envision a data scientist overlapping all three  properties, but in real life, you cannot find such an ideal candidate. Usually, people tend to lean one or another way in their work and keeping a balance is not always a priority.

    In principle, data scientist should have business insight, understand the math behind it and work in tandem with a skilled developer. Of course, a proper data scientist should be able to write any semblance of code, but a data scientist should work with a data engineer to write and realize code, being responsible for both the quality and sustainability of this solution.

    The classic tragedy of a company that decides to initiate any data science research is when a data scientist says, “I’ve got 40,000 lines of Python code on my PC. Can you make it work in production?” And, of course, this is virtually impossible to do. When that happens, all the research is simply wasted.

    Cross-Functional Teams

    Any team that deals with data science has to be cross-functional. In other words, it has to cover a whole stack of the solutions it writes. In a normal infrastructure there should be a DevOps engineer, data scientist, data engineer and a product developer writing the web app and/or mobile app. This single team is responsible for the result. The team members should work together and solve tasks that are interconnected in their interactions.

    All of this means that the whole team is responsible for the business result. This is also true for the transitionary research done by a data scientist that is impossible to use in production on its own.

    Old-School vs. Vertical Teams

    To dig in deeper, let’s take a classic old-school layered company organization structure when you have a department of data scientists, operations, UI developers, a big data department, QA engineers and so on. In this case, we have every project penetrating most of these teams. And the classic problem is that tickets and tasks are being thrown around by one team to another, and the real business goals are being watered down along the way and not solved in the end. So, instead of this horizontal division, we have divided the teams vertically. This allows us to create teams that see a clear-cut goal they need to achieve. And at the same time, they can improve their cross-skills and boost their responsibility levels.

    As a result, such teams began to deliver, Scrum and Agile properly. It is not directly related to data science, but companies need to realize the difference between a data scientist, which works in academia, and a production data scientist. Companies should aim at employing the latter one within their teams, and not let a data scientist work alone remotely.

    About the Author / Stepan Pushkarev

    Stepan Pushkarev is the Head of DevOps Practice at Squadex.com and CTO of Hydrosphere.io. Co-founded and managed engineering teams for eCommerce, IoT and Ad-tech companies. He has been responsible for the full products stack: math models, infrastructure & operations, enterprise applications as well as hiring, establishing engineering culture and delivery process. Stepan combines strong technology, management skills and entrepreneurial spirit. Connect with him on LinkedIn.

     

  • Redefining NoOps to Better Inform IT Decision-Making

    Redefining NoOps to Better Inform IT Decision-Making

    It is safe to say that since the term NoOps escaped from its confines as an assembly language instruction (NOP or NOOP mnemonic for no operation) and in microelectronics (NoOps or null instructions inserted into pipeline, creating bubbles*) into enterprise IT, it has caused more than its fair share of consternation and controversy in the DevOps community. This is hardly surprising, when some of the more prominent claims for NoOps placed emphasis on ending operations entirely, threatening the jobs and livelihoods of thousands of knowledge workers. As with previous movements for automation, NoOps came under immediate attack—and, somewhat ironically, the criticism came from IT industry workers. For better or worse, IT has more than its fair share of hyperbole, religiosity and grandstanding, with the NoOps debate being no exception.

    As someone who started his technology career as a developer, moved into infrastructure and is now accountable for resources being allocated to IT, I find the debate does little to inform my decision-making. In this article I want to share my definition of NoOps with the broader community to clear path for greater adoption of NoOps principles among the DevOps community. I see both of these communities as being distinct and yet highly complementary, as long as our ultimate goal is to generate value for the organizations we choose to work for and, by extension, the customers which they serve.

    NoOps as an Ideological Principle

    Various definitions of DevOps have been refined and become more stable over time. I don’t believe it would be particularly controversial to recognize DevOps as a culture, a methodology for creating high-quality applications and delivering them into production in the most efficient manner. In my book, “NoDev NoOps NoIT,” I choose to define NoOps as a guiding principle that makes assertions to help IT decision-makers inform their decisions. It is more useful to separate our notions of DevOps and NoOps into methodological and ideological categories, because doing so strengthens our understanding of both terms and equips us to make better IT decisions.

    Before we examine the assertions made by the NoOps principle, I want clarify why I have rejected other definitions of NoOps and critiques of NoOps itself.

    NoOps Spells the End of Operations

    One of the most common and enduring uses of the word “NoOps” is to refer to eliminating IT operations entirely. The problem with such definitions is that they conveniently scope IT operations in a manner that allows them to be eliminated. IT operations roles are not static in the face of changing technology; arguably, they evolve more rapidly than application roles. The original activities of racking, stacking, cabling and administering IT infrastructures in a typical enterprise has been hugely diminished by outsourcing, commoditization and the advent of cloud infrastructure. However, new activities in software development life cycle, automated provisioning, monitoring, metering and security have created new roles for infrastructure/operations developers. NoOps is not about eliminating IT operations; rather, it is concerned with maximizing automation in IT operations.

    NoOps is the Evolution of DevOps

    While superficially it may well seem that NoOps takes DevOps to its logical conclusion, this line of thinking chooses to ignore the huge swath of DevOps methodology that does not overlap with the automation of operations or the fact that DevOps is itself an evolving methodology. I simply don’t find this definition to be particularly useful in day-to-day decision-making. At best, it’s a dimly lit observation.

    DevOps is NoOps—NoOps is Just a Confusing Term

    The idea of NoOps is an expression of the benefits of automation. How we choose to define the term will dictate its utility. If we choose to define it in a fashion that leads to confusion, then fault lies with the definition and not with the idea itself, which I believe is a powerful one that should not be ignored. Attempting to simply define away NoOps is as wrong-headed as stating that NoOps is the end of operations. Overloading DevOps methodology to encompass NoOps ideology stretches the definition of DevOps beyond its useful boundaries.

    NoOps Assertions: Eliminate All Manual Intervention in IT Operations

    Eliminating manual intervention in IT operations is a fundamental assertion of NoOps. This assertion is made from an understanding of the benefits of the automation and the observation that the very tools we use in IT—computers—were invented to automate conative and algorithmic processes.

    While we may never be able to fully automate every aspect of our operations—for example, some regulators demand four-eyes processes for approval and execution—we must rigorously challenge the unthinking orthodoxy that currently governs many IT processes.

    Automate, simplify and standardize: I subscribe to this mantra but one has to proceed in the right order; that is, simplify, standardize and then automate. Otherwise, you literally risk making an ASS out of your IT automation efforts.

    I can say with a high degree of certainty that any failure to automate a process could probably be traced back having made no attempt to simplify the process before automating it. No attempt was made to challenge the process, to question who is doing what and why before automating it. Indeed, no effort, was made to question the existence of the process in the first instance.

    Once a process has been simplified, standardizing it follows almost naturally. Stakeholders adopt your process because it is simple, it saves them time and generates value for them.

    The opposite is also true: A poorly thought-out, complex process will never win buy-in from users. Any attempt to put lipstick on such a pig will be fraught difficulty, often resulting in the vast majority of users and stakeholders finding exceptions to your attempts to increase adoption of the offending process. 

    NoOps Assertions: Eliminate All Dependencies Between Application Developers and Infrastructure/Operations Developers

    As the trend toward software-defined everything matures, we need to change our organizational structure to adapt with the new landscape. Within enterprise IT, the war between application and operation teams is over, even if there are still combatants that don’t realize that the application teams won. In the software-defined paradigm we are all developers. Application developers write business/end user logic and infrastructure/operations developers write run-book instructions using the tools and APIs made available to them by infrastructure-as-a-service (IaaS) and platform-as-a-service (PaaS) vendors. Both of these types of developers write code and we find more and more frequently it is the same individuals writing both application and infrastructure code; hence, the rise of the full stack developer and the full spectrum technologist.

    As a CTO I organize my technology division into three teams: Application Development, Infrastructure Development and User Experience Development. These teams are physically situated next to each other and work toward a single technology strategy defined by a NoOps ideology and DevOps methodology.

    In Summary

    When we are presented with ideas that are threatening and challenge fundamental elements of our existence, such as the way we work, the best response is to seek out opportunities presented by those ideas and adapt to them. For the DevOps community, the first step in engaging with NoOps is to accept the ideology is based on a very real and powerful foundation of automation. As a community we can choose to dismiss NoOps or we can choose to embrace the possibilities it offers and allow it to enrich DevOps methodology. I would posit that those among in the DevOps community who can adapt to a NoOps reality will not only survive but thrive.

    About the Author / Hussein Badakhchani

    Hussein Badakhchani is a Distinguished Technologist and author with over 20 years of professional experience in financial services technology. Having worked for institutions such as the Bankers Automated Clearing System (BACS), Deutsche Bank and VocaLink – MasterCard, Hussein has a proven track record of delivery in some of the most competitive global capital markets. As a trusted academic, practitioner and adviser, Hussein also provides critical analysis of technology to executive and investment decision makers in government, banking, financial, commercial and defense sectors. He is the author of “NoDev NoOps NoIT,” principles governing the ideology, methodology and praxeology of informed IT decision-making. Connect with him on LinkedIn.

    *”Microelectronics,” by Jerry C. Whitaker.

  • Today’s DNS: DevOps’ New Best Friend

    Today’s DNS: DevOps’ New Best Friend

    Continuous development and deployment is a primary focus of DevOps teams. As part of this process, DevOps teams also require the automatic creation or removal of networks, and automated deployment to production. Automation is a closer guarantee that the process will be done right the first time because the fewer the number of manual steps involved, the less likelihood for human error. Intelligent DNS improves DevOps project life cycles by integrating DNS tasks into your software development and automation tools.

    DNS Needs an Upgrade

    The internet has come a long way since its humble beginnings, from the Advanced Research Projects Agency Network (ARPANET) of the 1960s to an entity that is mission-critical for the world’s commerce. This system is now tested and taxed in ways that were not even conceived of when it was originally designed. One of these early elements, DNS, is now a potential bottleneck for network performance in general and DevOps teams in particular.

    One of the benefits of cloud computing is that it enables DevOps teams to automatically create and remove servers as needed for development and testing. Infrastructure as code (IaC) enables cloud computing to build infrastructure automatically using robust APIs that build servers and add applications as needed. Traditional DNS, with its mostly manual management, becomes a bottleneck for this automated rapid configuration of infrastructure. This is why organizations need an intelligent DNS solution. Intelligent DNS with a robust API can seamlessly interface with current DevOps software, removing the bottleneck that traditional DNS can become.

    Extending Agile with DevOps

    The underlying ideas of agile programming are extended with DevOps. Rather than the traditional way of producing and testing the code, then tossing it over the wall to the network operations team to deploy, DevOps merges the development, testing and operations responsibilities. In short, DevOps should have the ability to break the silos within your IT environment and among the development, QA and operations teams.

    DevOps’ blend of cultural philosophies, practices and tools increases an organization’s ability to deliver applications and services at high velocity. DevOps allows organizations to evolve and improve products at a faster pace than organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market.

    Rapid Change Propagation is Essential

    In addition to rapidly delivering applications and services, a goal of DevOps continuous release is to remove bottlenecks to the development, testing, and deployment cycles. This is aimed at creating an environment where the work isn’t hampered with computer delays and frustrations.

    When you’re automating the creation or deletion of 100 virtual servers with a single mouse click using IaC, waiting for 30 minutes to see if DNS changes appear at the edge servers delays your project. Your DNS needs to move as fast as your IaC software and service discovery processes. At this point, intelligent DNS with rapid change propagation becomes essential. When choosing DNS vendors to support your DevOps environment, make sure to include speed testing for the network’s propagation for changes.

    Where IaC Comes In

    The aim of DevOps professionals is to create and sustain server environments that closely resembles the way software developers build and maintain application source code. IaC is the way DevOps teams make that happen.

    IaC is based on three primary constituents:

    • Unit/Integration Testing – Lets you validate your infrastructure code within various phases of the DevOps pipeline and lets you feel confident about what you are pushing to production.
    • Infrastructure Blueprints – Supports the reusable, consistent and rapid deployment part of IaC.
    • Version Control – Facilitates the method track and rollback changes to your infrastructure as needed.

    Infrastructure blueprints can be created using configuration as code (CaC). Using tools such as Terraform, Ansible and Chef allows DevOps teams to build these blueprints by enabling a declarative way of handling infrastructure configuration. Infrastructure blueprints should always go hand in hand with unit/integration testing. Having a robust, API-driven DNS network to couple the full automation of DNS changes alongside the infrastructure and configuration changes ensures a smoother process.

    IaC is next-generation technology for DevOps environments. When combined with CaC products, it changes the way software is built and delivered. Continuous integration and continuous deployment are a large part of the DevOps world that all still depends on DNS technology to connect everything on the network.

    The Role of Service Discovery

    Service discovery is needed to enable your apps or servers to identify which services are available on the network and which IP addresses and ports are associated with them. Previous methods of tracking this information no longer work in the DevOps age.

    Since modern infrastructure is so scalable and changes so quickly, keeping track of service information inside a periodically updated database doesn’t work anymore. Instead, you have to be able to discover services in real time and update your service information continually. Three common service discovery tools are: Consul, Apache Zookeeper and Etcd (with Registrator and confd).

    A tight integration with your managed DNS provider as part of your overall strategy means that service discovery can work hand in hand with the DNS configuration so both systems remain in sync.

    Moving Toward Intelligent DNS

    DevOps needs the same types of robust features you would want for your external internet traffic: end user-facing DNS. You should align with a vendor that can provide the flexibility, agility and reliability you require to deliver your applications.

    Look for features including:

    • Zone Transfers allowed and easily facilitated
    • An anycast DNS network
    • A robust API to support automation, IaC, service discovery and other architecture and tool needs
    • Native integration with common DevOps tools such as Terraform, Ansible and Chef
    • An SLA that meets your needs for reliability and performance

    These elements are necessary, but a more important item on your checklist needs to be redundancy in all layers of the stack, including DNS. Because no vendor is completely bulletproof, use at least two authoritative DNS providers to set up redundancy. Following these guidelines will improve your DevOps’ life cycle by using intelligent DNS.

    About the Author / Kimberly Lacerte

    Kimberly Lacerte has 14 years of experience in computer and technical writing. She is currently the Content Marketing Manager at NS1 in Bedford, N.H. Previously, she has written for Veritas, Symantec, Health Dialog and Dyn. Kimberly holds an English BA from UNH and an MBA from Southern NH University.

  • Simulating DevOps to Discover the ‘Culture Thingy’

    Simulating DevOps to Discover the ‘Culture Thingy’

    DevOps … is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably. — Wikipedia

    We are all, I believe, familiar with the fact that culture is an important factor when adopting DevOps.

    Groan!

    Most engineers and technologists are probably tired of having the “C” word thrown at them: culture.

    Because let’s be brutally honest: DevOps value comes from the automation stuff. The latest “State of DevOps” report confirmed “that automation is a key differentiator for organizations.” All the shiny technology that supports the “continuous-es”: continuous integration, continuous deployment, continuous delivery, with infrastructure as code and spinning up servers in the cloud and then collapsing them when no longer needed, with other cool technology concepts such as containers, microservices and a whole array of sexy build, test and monitoring tools. Just when we start getting excited about it and drooling about the new tools we get to play with, somebody else pipes up with that culture. stuff again …

    Groan. Just let us get on and do IT!

    But equally, probably the most important part of the “continuous-es” is continual learning and improvement, because again, let’s be honest: The real value of DevOps takes years to realize. It is not a one-time, overnight migration with a simple press on the ‘Install DevOps” button. It is an iterative exercise involving people from the end-to-end value stream; therefore, it is all about continual experimentation, learning and improvement—and doing it with people from different siloes who don’t usually like talking to each other. So, it’s a “cultural thingy.”

    If it’s not somebody talking about culture and collaboration and communication, it’s one of those “frameworky-geeks” talking about agile, scrum, lean thinking and IT service management (ITSM) and how we need “flow,” “process engineering,” “voice of the customer,” “kaizan,”kanban,” and a whole host of other Japanese-zen-like mysterious terms which only consultants are qualified to talk about and do, as they are the enlightened-ones (probably with their own secret handshake or Doctor Spock type split fingered salute!).

    “DevOps is a philosophical movement,” we hear. Now, if there are two things that I wouldn’t normally associate with each other it is technical engineers (I was one!) and philosophy. Although I do remember trying to stare down a server once, hoping by the power of positive thought and sheer will power for it to start working again; the alternative was a sledgehammer …

    Where was I? Oh, yes. So what are some of these “culture things” we should care about, apart from the “Wall of Confusion” that everybody keeps talking about? “We’re not confused! It’s the other lot on the other side of the wall that don’t get it!”

    Below are some of the key discoveries relating to the culture, or soft skills, issues that teams experience and recognize during the DevOps business simulation, “The Phoenix Project.” Before you start getting all excited again, this is not a computer game in which you can win points by zapping Ops people who get in the way or setting phasers to stun when you see a developer about to release some more untested code or beam up an uppity product owner who tells you that what you coded may have been what they asked for but isn’t what they were thinking. This business simulation is a dynamic, interactive, classroom-based workshop in which people have to get together—face to face—and, well, communicate and collaborate! The simulation workshop has been performed with literally hundreds of DevOps teams across the globe and across different industries, and is used to support DevOps training and to help empower teams. (see a blog posted on this site).

    What are Some of These Key Learning Points and Takeaways?

    I have mapped these discoveries onto the DASA (DevOps and Agile Skills Association) competence model, particularly the “Skills & Behaviors” area.

    DASA has developed the competence model to help organizations assess and develop the right knowledge and skills within their DevOps teams. The competence model is based upon industry feedback from practicing organizations, and helps organizations assess their level in terms of Novice, Competent, Proficient, Expert and Master. It looks at four key Skills & Behavior areas and eight Knowledge areas.

    Skills & Behavior Areas: Courage, Team-building, DevOps Leadership, Continuous Improvement.

    Simulation exercise discoveries for Courage:

    • “Feedback” is often seen as negative. People don’t like or dare to give feedback. It is perceived as personal, as criticism and often as blame. A blame culture makes it even more difficult to create open, honest feedback which is a barrier to learning and improvement.
    • “Fail fast” often means “blame fast,” or “pass the blame.” A blame culture creates a lack of trust and willingness to experiment for “fear of failure.” Often in the simulations if things go wrong, people revert to asking,”Whose fault was that?” rather than asking, “What happened? How can we stop that happening again?”
    • Sharing knowledge. Still in many organizations, knowledge is power; people don’t like to share knowledge or don’t think others are capable of using knowledge. A critical aspect of DevOps is multi-functional teams with broad shared skills and knowledge.

    Simulation exercise discoveries for Team-building:

    • Teams are unaware of the need for active listening, confirmation of agreements and decisions and verifying “assumptions”—many mistakes are the results of “assumicide” and poor understanding, which causes disruptions and excessive hand-off moments to gain the right information resulting in waste, delays and rework.
    • Ownership, responsibility and being proactive. Many IT people wait to be told what to do. Many people don’t like to admit they don’t know or that they need help. others don’t actively seek out and ask, “How can I help,” asking upstream and downstream, “What do you need from me to be able to do your job?”
    • Many teams have an aversion to getting together as they have poor experiences of pointless meetings with lots of talking and then not a lot of decisions being made or action being taken.
    • Many teams don’t set for themselves goals for a meeting or learning objectives from a training session. They do not ask themselves, “What do I need to take away from this session?” They often don’t ask, “What do you expect from me?” “What should my contribution be?” They often cannot answer the question, “What problem do you want to be able to solve after attending this?” Many times in a session people say, “I was told to come.”
    • Silo thinking: Many teams still think from siloes, e.g the technology team they are in or the development team, and don’t think end-to-end. They are building and supporting a technology solution and not managing end-to-end business value.

    Simulation exercise discoveries for DevOps Leadership:

    • When there are managers in the simulation, people often sit back and wait to be told or managers feel they must come up with the solution and give instructions.
    • Managers have difficulty letting go, and sometimes don’t have the patience to let teams learn and improve themselves.
    • Managers fear experimentation and failure, as they may be blamed.
    • Managers often blame teams for not becoming self-steering, becoming self-empowered or for not taking ownership, yet they do not help coach and empower teams.
    • Often managers are unable to answer the “why” question: “Why are we doing DevOps?” If managers do not know then how can they expect team members to know?

    Simulation exercise discoveries for Continuous Improvement:

    • Teams are poor at reflecting, analyzing, thinking and coming up with solutions. They lack skills in “problem thinking,” also in terms of Kolb learning cycle. Often they recognize one symptom and then leap to conclusions and solutions.
    • Lack of skills in facilitating a stand-up meeting. See the five key team skills mentioned by Jan Schilt on devopsy.kinsta.cloud. These skills are: The stand-up; Facilitating a retrospective; Managing the Kanban (Visual Management System); Managing the Andon: (and swarming problems); and Collaboration.
    • Many teams struggle to record issues in their work, such as a CSI (continual service improvement) register. They agree it is a good idea, then nobody does it, or if they do then nobody looks at the list again. If I ask, “Who owns this?“ they usually look at each other the point to the business or IT manager. There is no sense of ownership for improvements. This confirmed our own mini-survey of 650 organizations: Less than 20 percent had formal CSI.
    • With improvements, teams wait for somebody to say, “You will do this,” rather than suggest, “I can pick up that.” Improvements mean more work and teams are already busy enough, allocating time for improvements and rewarding improvements are not valued.
    • Teams often say they do NOT reserve time to reflect and discuss improvements. They are often too busy and managers do not stimulate, or allow them to reserve time for thinking and acting on improvements to the way of working. Priority is given to completing the ever growing backlog of work.

    Knowledge Areas

    Business Value Optimization:

    • Many technology people do not understand the business they are supporting, the critical business activities and business times, how a system delivers value to the business, what happens at a given time if a critical IT feature, function or technology is not available. There is a technology mindset as opposed to a business mindset.
    • Many technology teams have never seen a user or seen how users or customers use and experience the technology. Many think they don’t need to know this. This is a top-scoring ABC (Attitude, Behavior, Culture) card worldwide in global workshops and has been for 15 years. “IT has too little understanding of business impact and priority,” and is also a top concern for business managers.
    • In every single simulation and workshop I conduct worldwide, the issue of prioritization is raised, again in relation to business impact and business value, yet all teams recognize a growing pressure on scarce resources, a dilemma between “innovation” and “maintenance work” and a need for clear prioritization.
    • Product managers in teams are often unaware of technical debt and its impact. ITSM teams have difficulty quantifying the impact of technical debt and how this impacts current and future business plans.

    And finally, the latest “State of DevOps” report also confirmed the importance of skills and behaviors, particularly what it calls “Transformational Leadership”: “By 2020, half of the CIOs who have not transformed their teams’ capabilities will be displaced from their organizations’ digital leadership teams.” The survey continues: “Leaders are the ones who set the tone of the organization, and reinforce the desired cultural norms.” Oh! There is that “C” word again. Are we ever to escape it?

    What are the characteristics of a transformational leader, according to the State of DevOps findings?
    Vision, inspirational communication, intellectual stimulation (challenges people to think about problems in new ways), supportive leadership, personal recognition. These characteristics are one of the reasons why leadership is one of the four key Skills & Behaviors in the DASA competence model.

    Conclusion

    A simulation experience helps make the “C” word become a reality. Delegates can translate it from a buzzword into meaningful behavior. The simulation allows delegates to see, feel and experience the benefits of communication and collaboration, while helping them to translate DevOps theory into practice.

    As a leader, you can use a simulation exercise as an instrument to communicate your vision in an inspirational way. End-to-end stakeholders can be brought together and stimulated to explore and develop DevOps skills, knowledge and competences together. At the end of the simulation experience teams have been empowered to capture takeaways they want to apply.  As a leader, you can support them in translating the captured takeaways into real behavior and helping shape together a DevOps culture for your organization.

    About the Author / Paul Wilkinson

    Paul has been actively involved in the IT industry for more than 35 years. Fulfilling a wide range of roles ranging from computer operator to IT infrastructure operations manager. He was co-author of the ITIL publication, “Planning to Implement IT Service Management,” and was a member of the ITIL advisory group for ITIL Version 3, and the architects team for ITIL Practitioner. Paul is also co-director and owner of GamingWorks, the company that developed the internationally renowned “Apollo 13 – an ITSM case experience” ITSM simulation game and most recently the “Phoenix Project – DevOps business simulation,” which is based upon the Phoenix Project book . He was also co-author and developer of the “ABC of ICT (The Attitude, Behavior and Culture of ICT)” publications, having conducted ABC workshops and simulation workshops with delegates representing more than 4,000 organizations worldwide. Follow him on Twitter or connect with him on LinkedIn.

  • 10 Ways to Make Outage Mitigation Fun

    10 Ways to Make Outage Mitigation Fun

    It’s 3:47 a.m. You and the rest of the Ops team have been summoned from your peaceful slumber to mitigate an application delivery outage. As you catch up on the frantic emails, Slack chats and text messages from everyone on your international sales team (Every. Single. One.), your mind races as you switch to problem-solving mode. It’s time to start thinking about how to make this mitigation FUN!

    1. Gently remind everyone that the SaaS-delivered automated application delivery strategy you proposed would have prevented this. No need to rub it in, but … if you had turned on a software-defined application delivery platform for your hybrid infrastructure, you’d all be sound asleep right now. Automated real-time delivery decisions and failover would be nice, right? Just sayin’.
    2. Drop a generationally divisive Giphy in the emergency Slack channel. Your coworkers’ opinions on gaming consoles, Spiderman movies and music are different than yours! The perfect animated GIF will remind them of your erudite tastes, while you have their attention. Extra credit if you can work in some low-key shade about not listening to your equally sophisticated opinions on optimized outage mitigation.
    3. Hold an impromptu cloud health dashboard page customization contest. Oh hey, look at that! Your cloud provider’s health dashboard page says everything is fine … because it’s powered by the services that went down. Help your team vent their creative energy (and frustration) with some fun customized MS Paint updates to the offending page. Bonus points for art that reminds everyone of the value of a multi-cloud strategy powered by a programmable application delivery platform.
    4. Use health metrics to cure ills instead of just conducting another really good post-mortem. Tired of “learning lessons” from these emergency room drills? You depend on NGINX for your LLB, but don’t have a way to use those LLB health metrics and data to automate global delivery. Disjointed delivery intelligence means you don’t know how your apps will land with users. You need an end-user-centric approach to app delivery that automates the best delivery path and ensures failover is in place at all times. Micro-outages often fly under your passive monitoring radar, but that doesn’t mean your users don’t notice them. An active, integrated app delivery approach re-routes automatically before you lose business. Post-mortems are fun … but so is making sure your apps survive the last mile. Arguably more so?
    5. Open (and then close) some tickets for the sales team. “Sales needs to sell more stuff.” You’ll feel better.
    6. Build a countdown clock for when you can finally replace your old ADC hardware. Sure, your Mode1 ADC hardware was a sunk cost so you’re stuck with them for a while. But you’re one unnecessary emergency closer to having a fully software-defined application delivery platform for your hybrid cloud. And now you’re even closer. And closer … Tick. Tock.
    7. Leverage real user measurements (RUM) to foresee and fix problems before they happen. Probably best to do this during normal work hours. User experience data from around the world can detect degrading, sluggish resources in real time and user-centric app delivery logic powered by RUM can make quick rerouting decisions automatically. No more getting woken up after the application crashes. While you’re wishing you had RUM on your side, you can look up some fun facts from the countries experiencing app outages. Did you know Luxembourgish is an official language?
    8. Send an email. You, too, can be the weird colleague who sends emails with crazy, middle-of-the-night time stamps.
    9. Do some virtual window shopping. Browse around to see what you could have purchased with the money you were just forced to spend on unplanned cloud instance provisioning to keep your app running. That desktop Nerf missile launcher (or 700 of them) would have been pretty nice.
    10. Swear off the game of chance. You’ve just proven it’s not that much fun after all. Don’t just dump everything onto one cloud and call it done. Clouds go down for so many reasons. Use an application delivery platform control layer to build in the capability to auto-switch to an available resource, while you sleep soundly. Running on multi-cloud without an abstracted control layer removes most of the value of the cloud. Swear off the game of chance. Out loud. Right now.

    About the Author / Andrew Marshall

    Andrew Marshall is the Director of Product Marketing for Cedexis, and has driven product marketing efforts at New Relic, Amazon Web Services and Microsoft. Connect with him on LinkedIn.

     

     

     

  • Codeless Automation: Shattering the Testing Bottleneck

    Codeless Automation: Shattering the Testing Bottleneck

    In his article, “Moving from ‘Tester’ to ‘Quality Enabler’,” Scott Edwards clearly lays out the predicament in which QA testing is currently at: with one foot in the quicksand of manual testing, while the other tries to jump over the hurdle of automation. Ever tried hurdle racing with one foot? You probably won’t reach the finish line.

    Edwards mentions the infamous label that is attached to testing—bottleneck—and cites the “DevOps Review 2017” research paper of Computing.co.uk that found two-third of DevOps practitioners report QA testing as the bottleneck of the development process.

    Why is that? Why is testing bottlenecking development, and how can it be untangled?

    What Creates the Development Bottleneck?

    The testing bottleneck is created primarily due to two reasons:

    1. The reliance on manual testing, and
    2. “All the code that is getting thrown” QA teams’ way.

    Manual testing is extremely time-consuming, sometimes to the point that it’s close to impossible for test teams to keep up with the fast pace of DevOps. Manual testing is also error-prone, simply because the testing is performed by humans.

    The code issue is related to automation. Due to the pressure imposed upon test teams, automation is being hailed as the great savior. While the word “automation” has the flair of hands-free simplicity, the reality of automated testing is quite the opposite.

    Automation relies on code and testers aren’t coders. So instead of simplifying and speeding up testers’ work, the opposite is being achieved. Testers are thrown out of their comfort zone, being bogged down by tasks that aren’t in their “job description.” The frustration of all involved is easy to understand. The development team feels that testing is slowing them down, while the testing team feels they are required to jump through hoops.

    Add to that the complex setup and heavy maintenance required of automation, and you start to doubt whether it is indeed the right path for testing.

    Well, it is. With the ever-faster pace of DevOps, the pressure to shorten release cycles and the cutthroat competition software and web application companies are facing, test automation is the right path, despite all the challenges.

    What must be done to clear that path is remove coding from automation.

    Codeless Test Automation to the Rescue

    Codeless is a growing force in mass technology. It has liberated the web. Codeless platforms such as Wix and Shopify allowed millions of businesses to create an online presence quickly and effortlessly and led to the incredible expansion of the web. On the enterprise side, codeless platforms are making strides with the likes of Salesforce, FileMaker and Nintex, which offer drag-and-drop solutions to speed up and optimize efficiency of large-scale administrative tasks.

    The idea behind codeless is the modern-day “power to the people.” Just a decade ago, a small family business from Minnesota would have never been able to afford and maintain an online presence, but now their 22-year-old can set up an online shop, promote it on Facebook and in no time start selling their goods to a consumer across the ocean.

    Applying the same logic to testing instead of coding from scratch test after test—which is time-consuming and costly, and requires acquired skill—codeless test automation allows for quick and simple creation of tests, no coding skills required. It allows QA testers to focus on what they do best: testing.

    Automation shouldn’t be a hurdle that testers need to jump above; it should be a tool that pushes them forward. As things stand now, automation in testing is at a counterproductive standstill. Don’t get me wrong: Automation means well, but at the moment it’s like a misguided teenager that brings havoc even when he aims for harmony.

    How Does Codeless Test Automation Actually Work?

    Codeless test automation functions very similarly to enterprise codeless platforms. Both take complex tasks traditionally achieved by coding and simplify them through automated code generation.

    With test automation a tester creates a flow by visually binding elements that represent “clicks” in the software or web application. The tester simply goes through motions of the scenario he wishes to test, and the code for the test is generated under the hood.

    We can get more technical than this, but there’s no need. That’s exactly the idea with codeless, isn’t it? Testers could focus their efforts on testing functionality and evaluating the user experience rather than struggling with test creation.

    This ties back to Edwards’ main point: Testers get so entangled with test creation that the whole point of testing—quality assurance—gets lost.

    Side Note: The proposed job title change suggested by Edwards—Tester to Quality Enabler—might sound merely cosmetic, but it can deliver real value. As illustrated in this collection of QA testers’ venting from a reddit forum, QA testers often harness feelings of underappreciation and of being looked down on by their developing colleagues. Recognizing testers’ valuable contribution to the overall quality of the product will surely elevate their feelings of self-worth and enable them a stronger sense of pride in their position.

    Using Codeless for Deeper Integration of Testing Into DevOps

    Codeless automation is best utilized in sanity and regression testing, the two most basic and frequently applied tests. Sanity testing ensures that all incorrect inputs a user could enter are accounted for. Regression testing is like inventory counting, in a way: you just need to go over your stockpiles and see that everything is in order.

    Thinking on inventory: Let’s say I have a huge warehouse full of thousands of items. Checking inventory by hand is a very time-consuming and error-prone process. How many boxes on a piece of paper (representing real boxes on shelves) can you tick before you’ll make a mistake? Now think of Amazon’s warehouse. Inventory counting is done using automation with the press of a button. It is instantaneous and error-free. It’s an action that’s been done so many times before, there’s no need to plan it—or to code it—all over again. Simply run the same command—“inventory”—and you’re done.

    If you’re using codeless automation, all you need for regression testing is to click a button and the scripts run. You are done and able to divert your attention to sanity testing, making sure the new iteration is solid.

    For the next iteration, you simply pile the sanity testing of the previous round on top of the regression scripts and, again, you’re done.

    Continuous testing is getting a whole new meaning with codeless automation as part of the equation. If regression testing is done with the click of a button, and sanity taken care of simultaneously, the reporting back to development is swift and focused and the bottleneck is shattered.

    Codeless enables deeper integration of testing into the development process simply by arming testers with automation tools that simplify tasks, not complicate them.

    About the Author / Assaf Dudai

    Assaf Dudai is the content marketing manager at TestCraft, a pure SaaS continuous test automation solution for QA professionals. Assaf is mostly looking into the future of testing, how automation, codeless and machine learning can and will help testing evolve. You can connect with Assaf on LinkedIn.

  • 4 Predictive Analytics Challenges Facing SRE Teams

    4 Predictive Analytics Challenges Facing SRE Teams

    “Predictive analytics” refers to the many techniques used to analyze data to make predictions about the future, such as data mining, statistics, modeling, machine learning and artificial intelligence. Being able to predict technical and business outcomes is one of the main promises of big data. Indeed, many businesses have invested heavily in infrastructure to anticipate key performance indicators such as demand, pricing or maintenance. The investment required is substantial but, when it works, it yields a very positive ROI.

    In this post, we’ll examine why implementing predictive analytics in a site reliability engineering (SRE) context requires a significant investment in both setting up the right data sets and applying the right logic to that data.

    SRE should in theory be a prime target area for predictive analytics. Why? Because companies have already widely adopted monitoring tools over the past decade. In doing so, they’ve amassed extensive system health and behavior data that is ripe for analysis. SRE teams have also increased their ability to make sense of this data as data science tools have become more accessible. Basic data science aptitude is increasingly become a desired skill when hiring engineers.

    Additionally, the SRE team is responsible for a mission-critical KPI: system availability. This single metric is directly tied to customer SLAs in digitally dependent businesses. SRE teams have a strong incentive to implement predictive analytics. Why? Having the ability to prevent downtime by addressing potential issues before they hit the production environment is hugely valuable. More system availability translates into more revenue, higher customer satisfaction and lower costs of problem mitigation.

    However, the reality is that an SREs job is still mostly focused on the diagnostic, mitigation and “fixing” parts, as opposed to more proactive tasks. This is in big part because predictive analytics applied to SRE/DevOps has proven difficult to implement and perform in a disciplined manner. Therefore, it is still an aspirational goal at most companies.

    SRE Challenges in Predictive Analytics

    Up next, let’s explore the four challenges that need to be overcome to implement high-quality predictive analytics practice in an SRE context, including:

    • Data collection
    • Data quality
    • Data volume
    • Model usability

    Challenge 1: Data Collection

    First, the working data sets need to fit what you are trying to predict. In this case, it is challenging to get the right data set because the SRE team’s mission is very broad; specifically, ensuring the reliability of the entire system. The difficulty lies in the fact that there is no such thing as “full system” data readily available. All the relevant data is distributed across siloed, or overlapping repositories and monitoring tools. Centralizing the data requires either a full system agent (which takes a lot of work to set up) or through APIs, which requires mastering a multitude of API integrations. An additional obstacle in getting the right data lies in the inconsistent nature of events, time series data and logs. These disparate data types can’t be analyzed together unless they can be transformed into a uniform data set.

    Challenge 2: Data Quality

    SRE environments emit lots of data and some won’t make sense for the purpose of the analysis. Whichever data collection method is used, the information tracked often reflects some level of bias on the part of whoever set up the tracking mechanism in the first place. This leads to capturing both irrelevant data or inadvertently leaving out important information. In other words, the data is often both noisy and lacking at the same time.

    The key to successful modeling lies in the selection of prediction variables with data that is obtained before the event predicted happens (in this case, a failure). Defining the variables that lead to the determination of a subsystem failure is critical and requires specific domain knowledge. It is difficult to define and extract a pure data set sanitized of noise. The more data you leverage, the higher the chances that you are actually “feeding the beast” with irrelevant information. Too much irrelevant data leads to inaccurate models and false positives.

    Challenge 3: Data Volume

    Here you can have two different types of challenges: too little data and too much.

    Not enough data

    You typically need at least 100 instances of what you are trying to predict and probably at least 100 where it didn’t work to train a model. This assumes the data set is clean and directly relevant to a very narrow, specifically defined problem. In the case of SRE, the scope of issues that can affect a system is such that the data set required is very substantial. Likely, it’s several months worth of a wide variety of system behavior measurements. For example, if you collect data only through passive collection, think about the lag time between t-zero (when you start collecting data) and the time when you would have results. During that time, you have a bootstrapping problem—your model can’t be trained to predict accurately what you want.

    Too much data

    There are two practical points to make around the notion of “too much data.”

    • Lack of standardization: Processing large volumes of data for analysis remains a difficult engineering challenge. There is no standardization for data pipeline; every method has a limit in terms of how many inputs it can ingest on a per-second basis, and ingest rates often end up becoming bottlenecks in large enterprise environments.
    • Volume does not equal accuracy: There is a point beyond which you get diminishing returns: statistical theorems show that after a certain point, feeding more data into a predictive analytics model will not provide more accurate results.

    Overall, getting the quality and quantity of the training data right is the best investment you can make in building your predictive capabilities.

    Challenge 4: Model Usability

    From a pure “mechanics” standpoint, the analytical concepts and methods are there to reflect the complex thinking required to predict the behavior of an enterprise’s combination of technology, information and infrastructure. Predictive software relies heavily on advanced algorithms and methodologies such as logic regressions, time series analysis and decision trees. The problem is that the workings of the kinds of systems that SREs deal with are very hard to understand in the first place, let alone predict. It is not uncommon even for SREs to not entirely understand how a system works if they were not the ones who initially set it up. In addition, every company’s system is different.

    When data scientists design predictive models without years of experience in an SRE capacity in multiple environments—in other words, without leveraging subject matter expertise—it rarely works perfectly. Often the models they create lack usability, even after the validation process to confirm its quality. A modeler will split the available data into a training set where the model is run to generate predictions and compare performance against the actual outcomes. That will require multiple iterations to generate a model that has high overall predictive accuracy, and 9 out of 10 times the result is overfitting, or perfect for the training data. A subject matter expert in the domain of technical operations can cut the cycles of iteration and get to a predictive model that works with an acceptable level of risk.

    In short, implementing predictive analytics in an SRE context takes a significant investment both in setting up the right data set and applying the right logic. Stay tuned for a future blog post that will provide some pointers on how to achieve the best prediction results.

    About the Author / JP Emelie Marcos

    JP Emelie Marcos is the co-founder and CEO of SignifAI. Previously, JP served as the Chief Operating Officer of Tango, a leading mobile consumer application that reached 400 million regular users in five years. JP’s career spans 20+ years leading teams in technology startups and large companies as an investor, GM and CEO.