devopsy.kinsta.cloud

Category: DevOps Practice

  • NodeSource, AppDynamics Ally for Node.js DevOps

    NodeSource, AppDynamics Ally for Node.js DevOps

    The prevalence of Node.js as the most widely deployed instance of JavaScript is putting a lot of pressure on DevOps teams to find ways to automate the management of Node.js applications. To provide the visibility needed to achieve that goal, NodeSource has announced it has integrated its N|Solid development environment with the application monitoring tools developed by AppDynamics.

    Pravin Halady, product manager for NodeSource, says one of the things that differentiates N|Solid, which is based on Node.js, most is the amount of telemetry data NodeSource is able to collect from Node.js applications. The alliance with AppDynamics provides a mechanism for DevOps teams to now tap into that data to better manage those applications alongside the panoply of existing applications they need to support and manage, says Halady.

    AppDynamics already supports Node.js. But Halady says the alliance with NodeSource provides deeper visibility into, for example, asynchronous activity. Advanced metrics can be used to augment the basic Node.js metrics that AppDynamics currently provides to better ensure service level agreements (SLAs) are both achieved and maintained, Halady says.

    Adoption of Node,js exploded across the enterprise once it became feasible several years ago to deploy JavaScript code on both the client and the server. That capability in many ways fulfilled the early promise of Java in the eyes of many developers. But DevOps teams have been slow to include support for JavaScript applications within their DevOps processes, mainly because of a lack of visibility. Now that the base of Node.js applications in the enterprise has reached a level of critical mass, tools vendors such as AppDynamics are looking to provide that visibility by expanding the scope of support they provide for Node.js.

    Halady says NodeSource chose to partner with AppDynamics in part because in the wake of AppDynamics being acquired by Cisco Systems, it’s clear that NodeSource needs to participate in the end-to-end visibility platforms that larger companies are starting to build. Cisco has already made significant investments in a Cisco Tetration analytics application in addition to committing to imbuing the IT infrastructure environment it provides with machine-learning algorithms. That data will then be combined with data about applications collected by AppDynamics to identify the source of any potential performance issue faster. NodeSource, adds Halady, will look to foster similar agreement with other providers of application monitoring tools.

    In the meantime, DevOps team can take heart in the fact the Node.js Foundation is making providing visibility into DevOps environments a higher priority. The number of Node.js applications in the enterprise is only expected to increase. There’s a high correlation between usage of Node.js and containers that collectively are being used to facilitate the development of microservices. As those microservices bgin to replace monolithic applications developed originally developed in languages such as Java, the size and scope of the DevOps challenge exponentially increases.

    Application development in the enterprise is going to be polyglot well into the next decade. Today there are more programming languages being actively used in the enterprise than any time in history. In fact, the days when IT operations could mandate what technologies developers can employ are long over.

    — Mike Vizard

  • DevOps and Security is Like Smoking Meat

    DevOps and Security is Like Smoking Meat

    It isn’t everyone who thinks, “Doesn’t Ubuntu remind you of wild boar?” Or labors over his pit of slow-roasted pork shoulder while contemplating containers. Or dwells on e2e testing while mesquite smoke permeates spareribs.

    But, Apollo Clark (@apolloclark) does. Apollo is a foodie of smoked meats (is that a meatie or smokie?), working to master the craft: understanding different cuts, what each type of wood adds, and the subtleties of sauces. But, alas, it is a hobby; security and DevOps are his career.

    Being passionate about both, he naturally sees parallels between the two, so Apollo presented at the 2016 All Day DevOps conference with a session titled, “What Smoking Meat Taught Me About DevOps and Security.”

    Understandably, you are asking yourself, “What does smoking meat have to do with DevOps and security?” Apollo notes both have tremendous complexity and nuances, there are multiple ways of getting the job done, lots of ways to mess it up, a couple of ways of doing it right, and you are always learning.

    apollo 1.png

    For both, there are many tools and processes to get the job done. For smoking, your wood is a critical component, along with time. Let’s look at some parallels:

    • Oak = unit testing
    • Maple = coverage
    • Apple = dynamic analysis
    • Peach = static analysis
    • Mesquite = e2e testing. People love it, but it is difficult to handle
    • Wine barrel = browser support. This is when you are doing it really well
    • Bourbon barrel = device support. Pretty complicated
    • Smoking time = testing. You can test or smoke for five minutes or five hours, but there is a sweet spot. You can oversmoke meats and you can overtest.

    Getting hungry?

    apollo2.png

    Meat is critical to a meal of smoked meat, but so are many other components. Likewise, software applications take a suite of other tools and components to deliver the full package. Get your taste buds ready, because Apollo likens all of the goodness that goes with smoked meats to tools you use to keep applications running:

    • Sauces = auto-scaling. They both can get pretty complicated. Always a little different, but you have to make it work for you.
    • Bread = monitoring, which is the bread and butter of infrastructure. Make sure you have it up and down your stack.
    • Salad = system logs. Not the sexiest things, but you can rely on them.
    • Fruit = custom application logs. Takes a lot of time to pair them, but takes a really good thing and makes them better.
    • Beer = firewall. You should always have both.
    • Wine = Intrusion detection systems. These are your fine wines.
    • Whiskey = IR training. It takes time and there are so many ways of doing it. When things break, we have procedures on how to deal with them.

    When smoking meats and in DevOps and security, Apollo asks and answers, “Do we have to do everything? No, but the more we do, the better it will taste.

    “Does it cost money and take time? Yes, but you can do great things even without money and time. You can use cheap meats and you can use open source. Start simple, build up complexity, and always be learning.” Constantly ask yourself, “Am I better today than I was yesterday.” If not, be better.

    You can watch Apollo’s entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online Oct. 24.

    — Derek E. Weeks

  • Incorporating Performance Engineering into DevOps

    Incorporating Performance Engineering into DevOps

    Performance testing historically has been a challenge for most IT organizations. It is highly misunderstood, from executives down to the few resources allocated to perform an extremely difficult task. Most companies consider performance testing expensive and unnecessary and simply refuse to acknowledge it is a critical component of the software development life cycle.  Have we forgotten about major performance failures such as Healthcare.gov? It is time companies start to take performance engineering seriously.

    IT organizations using the Waterfall methodology relied on a few performance testers to develop performance scripts and run them after most of the testing had been completed and the application was a few weeks from a production deadline. If the performance team was lucky, they would get in one or two successful tests, typically working off-hours and executing performance tests on small, patched-together ancient hardware. The problem with this approach is when issues are found, there is not enough time to fix them adequately and performance degrades over time.

    Developing a Performance Engineering Strategy

    Here are some items to consider when incorporating performance engineering into DevOps:

    • Learn—It is important to learn what performance engineering is and how it can be incorporated into your organization. If you don’t have the right skills within your organization, go find someone who can help. There are many organizations that can help with an initial assessment of where things stand.
    • Develop Your Strategy—This is perhaps were most teams fail. Upfront planning is critical and it will save you in the end. It is important to develop a performance engineering road map. This is not something that will be built in a few weeks; a solid performance road map could take 18-24 months, so be patient and build it right.
    • Build Your Baseline—It is important that you build a baseline of your performance tests so that you can determine if you have successful results. Performance testing is different from functional or automation because you have to consider the performance run as a whole. A few high transactions usually are acceptable. It is a good idea to build a history of runs to see how the application performance is trending.
    • Identify Key Business Workflows—It is too time-consuming to build hundreds of performance tests, so focus on those that are the most critical to your business. Typically those workflows will account for the majority of your performance load.
    • Test Upfront—With DevOps you need to start early and you will need to run performance tests frequently. The more you can run upfront, the more performance issues will be avoided in production.
    • Tools and Hardware—Performance engineering requires tools and hardware. It is important that your assessment evaluates what you currently have and what you will need to do it properly. There are many open-source tools on the market today that can help and other tools that are making a big difference in the performance space.
    • Budget—Developing a world-class performance engineering practice is going to take money. When you develop your performance road map, you might need two to three budget cycles to get the tools and infrastructure you need.
    • Load Performance Environment—Yes, you will need to have a performance environment that is scaled to match your production environment. If there isn’t enough budget, then you will have to know how much smaller the environment is so that you can be confident in your performance results.
    • Data—It is important to have data that is similar size to your production environment. This will help you understand how your system will handle performance within the production environment.

    Incorporating performance engineering within DevOps will require work, but the rewards will certainly put the organizations that embrace it at the top of the list. DevOps enables teams between Dev and Ops to produce a stable application that performs exceptionally well under load. Here are some benefits of DevOps performance engineering:

    • Predictable production performance results
    • Production deployment confidence
    • Eliminate catastrophic performance failures
    • Increased performance efficiency

    DevOps gives organizations a real opportunity to get performance engineering right; unfortunately, most organizations will carry over the same mistakes into DevOps. I hope you will take the opportunity to embrace the performance engineering challenge!

    About the Author / Ron Wilson

    Ron Wilson is a passionate champion of Agile, DevOps and Quality Engineering. He has managed large global Quality Engineering teams across multiple industries. He is currently the director of Quality Engineering at Arch MI. He regularly shares insights through social media where you can find him at www.qarevolution.com and on LinkedIn.

  • Understanding the Data Scientist’s Role in Cross-Functional Teams

    Understanding the Data Scientist’s Role in Cross-Functional Teams

    In our previous article, “Data Science Industry Perspectives in the Cloud,” we discussed that evolution is key if you plan to grow your business. You can start with ready-made solutions, then, in time, you can switch to in-house solutions created with the help of a group of data scientists.

    This time around, we’ll talk about the cases and solutions when machine learning as a service (MLasaS) doesn’t work. Your company’s first move is not to go out and hire a data scientist to solve your urgent business needs; rather, your company should invest in a custom-made solution. Only when you have a workable solution can you dive in deeper and create a proper team that can create an in-house data science solution.

    Custom Made End-to-End Solution as a Start

    Data Science in the classical sense involves data, goals and models to solve a pressing issue. The best way to jump start a data science process is by putting together various ready-made services into a single workable product to show your customers clear-cut results in short order. This can be done without any complex or global research, and you can comfortably formulate specifications taking into account all the feedback from your customers to create a more high-level data science offering later.

    One of the biggest issues for any data science project is in formulating the specifications for it. The usual request is, “Create something for my company using data science. Analyze data for me.” This type of a job has lots of trial and error. Having a custom end-to-end solution from the start lets you add on extra services as needed, and enables greater insights and predictions into a specific business workflow.

    I believe companies should build their data science system not from the data science research but from the point of view of an end-to-end solution. And then, bit by bit, they can adjust their services in the process.

    Moving on to Data Science Research

    Companies start data science research by employing a data scientist who matches their needs. You can use the standard data scientist qualifications: This employee should have a firm footing in the application environment, mathematics and programming. Also, their business skills are key to success—I believe an understanding of the topical area is key here because we are solving business issues. And a data scientist who solves problems that are more academic will be more focused on winning a Kaggle competition than on addressing the business needs.

    It is important this person understands the product development cycle. This way, the models and analyses data will be built in such a way that they could be deployed in production. For example, if a person is using R (the language and framework for data scientists), we should note that it is more research-oriented and it is not production-ready. Correspondingly, the results of such research cannot be deployed in any end-to-end solution. Therefore, the data scientist should work with a programmer.

    Although in the above diagram we see that the Data Scientist should have hacking skills, in reality, this is not the case. In their vast majority, Data Scientists are not able to write quality code. And if you need not just the research but a solution then process-wise you need to have a Data Scientist working together with a Data Engineer.

    CAP Theory as an Analogy

    It is impossible to have three database properties at once: consistency, availability and partition-tolerance. This basic rule is known to all developers. This is true for a data scientist as well: People envision a data scientist overlapping all three  properties, but in real life, you cannot find such an ideal candidate. Usually, people tend to lean one or another way in their work and keeping a balance is not always a priority.

    In principle, data scientist should have business insight, understand the math behind it and work in tandem with a skilled developer. Of course, a proper data scientist should be able to write any semblance of code, but a data scientist should work with a data engineer to write and realize code, being responsible for both the quality and sustainability of this solution.

    The classic tragedy of a company that decides to initiate any data science research is when a data scientist says, “I’ve got 40,000 lines of Python code on my PC. Can you make it work in production?” And, of course, this is virtually impossible to do. When that happens, all the research is simply wasted.

    Cross-Functional Teams

    Any team that deals with data science has to be cross-functional. In other words, it has to cover a whole stack of the solutions it writes. In a normal infrastructure there should be a DevOps engineer, data scientist, data engineer and a product developer writing the web app and/or mobile app. This single team is responsible for the result. The team members should work together and solve tasks that are interconnected in their interactions.

    All of this means that the whole team is responsible for the business result. This is also true for the transitionary research done by a data scientist that is impossible to use in production on its own.

    Old-School vs. Vertical Teams

    To dig in deeper, let’s take a classic old-school layered company organization structure when you have a department of data scientists, operations, UI developers, a big data department, QA engineers and so on. In this case, we have every project penetrating most of these teams. And the classic problem is that tickets and tasks are being thrown around by one team to another, and the real business goals are being watered down along the way and not solved in the end. So, instead of this horizontal division, we have divided the teams vertically. This allows us to create teams that see a clear-cut goal they need to achieve. And at the same time, they can improve their cross-skills and boost their responsibility levels.

    As a result, such teams began to deliver, Scrum and Agile properly. It is not directly related to data science, but companies need to realize the difference between a data scientist, which works in academia, and a production data scientist. Companies should aim at employing the latter one within their teams, and not let a data scientist work alone remotely.

    About the Author / Stepan Pushkarev

    Stepan Pushkarev is the Head of DevOps Practice at Squadex.com and CTO of Hydrosphere.io. Co-founded and managed engineering teams for eCommerce, IoT and Ad-tech companies. He has been responsible for the full products stack: math models, infrastructure & operations, enterprise applications as well as hiring, establishing engineering culture and delivery process. Stepan combines strong technology, management skills and entrepreneurial spirit. Connect with him on LinkedIn.

     

  • Redefining NoOps to Better Inform IT Decision-Making

    Redefining NoOps to Better Inform IT Decision-Making

    It is safe to say that since the term NoOps escaped from its confines as an assembly language instruction (NOP or NOOP mnemonic for no operation) and in microelectronics (NoOps or null instructions inserted into pipeline, creating bubbles*) into enterprise IT, it has caused more than its fair share of consternation and controversy in the DevOps community. This is hardly surprising, when some of the more prominent claims for NoOps placed emphasis on ending operations entirely, threatening the jobs and livelihoods of thousands of knowledge workers. As with previous movements for automation, NoOps came under immediate attack—and, somewhat ironically, the criticism came from IT industry workers. For better or worse, IT has more than its fair share of hyperbole, religiosity and grandstanding, with the NoOps debate being no exception.

    As someone who started his technology career as a developer, moved into infrastructure and is now accountable for resources being allocated to IT, I find the debate does little to inform my decision-making. In this article I want to share my definition of NoOps with the broader community to clear path for greater adoption of NoOps principles among the DevOps community. I see both of these communities as being distinct and yet highly complementary, as long as our ultimate goal is to generate value for the organizations we choose to work for and, by extension, the customers which they serve.

    NoOps as an Ideological Principle

    Various definitions of DevOps have been refined and become more stable over time. I don’t believe it would be particularly controversial to recognize DevOps as a culture, a methodology for creating high-quality applications and delivering them into production in the most efficient manner. In my book, “NoDev NoOps NoIT,” I choose to define NoOps as a guiding principle that makes assertions to help IT decision-makers inform their decisions. It is more useful to separate our notions of DevOps and NoOps into methodological and ideological categories, because doing so strengthens our understanding of both terms and equips us to make better IT decisions.

    Before we examine the assertions made by the NoOps principle, I want clarify why I have rejected other definitions of NoOps and critiques of NoOps itself.

    NoOps Spells the End of Operations

    One of the most common and enduring uses of the word “NoOps” is to refer to eliminating IT operations entirely. The problem with such definitions is that they conveniently scope IT operations in a manner that allows them to be eliminated. IT operations roles are not static in the face of changing technology; arguably, they evolve more rapidly than application roles. The original activities of racking, stacking, cabling and administering IT infrastructures in a typical enterprise has been hugely diminished by outsourcing, commoditization and the advent of cloud infrastructure. However, new activities in software development life cycle, automated provisioning, monitoring, metering and security have created new roles for infrastructure/operations developers. NoOps is not about eliminating IT operations; rather, it is concerned with maximizing automation in IT operations.

    NoOps is the Evolution of DevOps

    While superficially it may well seem that NoOps takes DevOps to its logical conclusion, this line of thinking chooses to ignore the huge swath of DevOps methodology that does not overlap with the automation of operations or the fact that DevOps is itself an evolving methodology. I simply don’t find this definition to be particularly useful in day-to-day decision-making. At best, it’s a dimly lit observation.

    DevOps is NoOps—NoOps is Just a Confusing Term

    The idea of NoOps is an expression of the benefits of automation. How we choose to define the term will dictate its utility. If we choose to define it in a fashion that leads to confusion, then fault lies with the definition and not with the idea itself, which I believe is a powerful one that should not be ignored. Attempting to simply define away NoOps is as wrong-headed as stating that NoOps is the end of operations. Overloading DevOps methodology to encompass NoOps ideology stretches the definition of DevOps beyond its useful boundaries.

    NoOps Assertions: Eliminate All Manual Intervention in IT Operations

    Eliminating manual intervention in IT operations is a fundamental assertion of NoOps. This assertion is made from an understanding of the benefits of the automation and the observation that the very tools we use in IT—computers—were invented to automate conative and algorithmic processes.

    While we may never be able to fully automate every aspect of our operations—for example, some regulators demand four-eyes processes for approval and execution—we must rigorously challenge the unthinking orthodoxy that currently governs many IT processes.

    Automate, simplify and standardize: I subscribe to this mantra but one has to proceed in the right order; that is, simplify, standardize and then automate. Otherwise, you literally risk making an ASS out of your IT automation efforts.

    I can say with a high degree of certainty that any failure to automate a process could probably be traced back having made no attempt to simplify the process before automating it. No attempt was made to challenge the process, to question who is doing what and why before automating it. Indeed, no effort, was made to question the existence of the process in the first instance.

    Once a process has been simplified, standardizing it follows almost naturally. Stakeholders adopt your process because it is simple, it saves them time and generates value for them.

    The opposite is also true: A poorly thought-out, complex process will never win buy-in from users. Any attempt to put lipstick on such a pig will be fraught difficulty, often resulting in the vast majority of users and stakeholders finding exceptions to your attempts to increase adoption of the offending process. 

    NoOps Assertions: Eliminate All Dependencies Between Application Developers and Infrastructure/Operations Developers

    As the trend toward software-defined everything matures, we need to change our organizational structure to adapt with the new landscape. Within enterprise IT, the war between application and operation teams is over, even if there are still combatants that don’t realize that the application teams won. In the software-defined paradigm we are all developers. Application developers write business/end user logic and infrastructure/operations developers write run-book instructions using the tools and APIs made available to them by infrastructure-as-a-service (IaaS) and platform-as-a-service (PaaS) vendors. Both of these types of developers write code and we find more and more frequently it is the same individuals writing both application and infrastructure code; hence, the rise of the full stack developer and the full spectrum technologist.

    As a CTO I organize my technology division into three teams: Application Development, Infrastructure Development and User Experience Development. These teams are physically situated next to each other and work toward a single technology strategy defined by a NoOps ideology and DevOps methodology.

    In Summary

    When we are presented with ideas that are threatening and challenge fundamental elements of our existence, such as the way we work, the best response is to seek out opportunities presented by those ideas and adapt to them. For the DevOps community, the first step in engaging with NoOps is to accept the ideology is based on a very real and powerful foundation of automation. As a community we can choose to dismiss NoOps or we can choose to embrace the possibilities it offers and allow it to enrich DevOps methodology. I would posit that those among in the DevOps community who can adapt to a NoOps reality will not only survive but thrive.

    About the Author / Hussein Badakhchani

    Hussein Badakhchani is a Distinguished Technologist and author with over 20 years of professional experience in financial services technology. Having worked for institutions such as the Bankers Automated Clearing System (BACS), Deutsche Bank and VocaLink – MasterCard, Hussein has a proven track record of delivery in some of the most competitive global capital markets. As a trusted academic, practitioner and adviser, Hussein also provides critical analysis of technology to executive and investment decision makers in government, banking, financial, commercial and defense sectors. He is the author of “NoDev NoOps NoIT,” principles governing the ideology, methodology and praxeology of informed IT decision-making. Connect with him on LinkedIn.

    *”Microelectronics,” by Jerry C. Whitaker.

  • Securing Immutable Servers in a Serverless World

    Securing Immutable Servers in a Serverless World

    Snowflakes are beautiful, unique creations. But, let’s keep them in nature. They don’t belong in our server infrastructure. Snowflake servers, where every configuration is just a little different, can introduce unnecessary security vulnerabilities and complications. While common in IT infrastructure, in the DevOps realm they are gradually becoming ancient history.

    At All Day DevOps 2016, Erlend Oftedal (@webtonull), with Blank and head of the OWASP Norway chapter, discussed the benefits of immutable infrastructure practices within serverless architectures. Erlend walked us through the positive effects on security as well as offered insight on potential problems.

    Erlend outlined the progress from physical servers to immutable servers, discussing the benefits of each step and the concerns that lead to the next phase, ending with immutable servers—that is, they can’t be changed. Once set up, they are set. Automated processes coupled with immutability principles can eliminate snowflake servers.

    blog pic 1.png

    Immutability isn’t a new concept. Some classic examples include live Linux CDs, internet cafes, schools, libraries and hotel business centers. To set up immutable servers, Erlend outlines these steps:

    1. Find a suitable base image
    2. Build a specific image for the application
      1. Dependencies
      2. Application
      3. Secrets
    3. Deploy image to the server(s)
    4. Repeat steps 2-3 for new versions

    blog pic 2.png

    Of course, this immediately raises questions:

    • What about data?  He notes that because data inherently changes, data has to be stored externally.
    • How do you deploy changes?  Well, you don’t change the server. You take one out and put a new one in. What if that one fails? Roll back to the previous one.
    • How do you manage secrets?  You can build secrets in, leverage cloud key management, or use third-party services.
    • How do we apply security patches?  If truly immutable, you deploy a new server with the patch. However, you can have a semi-immutable server that allows for automatically installing patches but still doesn’t allow for logins or manual changes.
    • Speaking of security, what are the security advantages? For starters, if an attacker is in your server, it will be thrown out when you change servers, although it may come back if the vulnerability isn’t patched. It also allows for very specialized images by removing all unnecessary packages. With these specialized images, you can more easily audit and monitor the system for unexpected file changes, logins and connections.

    blog pic 3.png

    Erlend discussed containers because they are a step closer to a serverless world. Containers isolate processes and all run on the same OS. Of course, you can make them immutable, but regardless of whether they are, there are some key security best practices you’ll want to consider:

    • Don’t run as root inside container
    • Implement user namespaces so the root account within the container is separated from the root account outside the container
    • Utilize the least privileges necessary for capabilities and resources
    • Keep them up-to-date
    • Automate scanning using tools such as Sonatype Nexus, Twistlock, Clair, or Aqua
    • Separate untrusted and trusted containers

    Erlend also dove into serverless architecture. While there is debate over what exactly qualifies as serverless, there are five principles that Erlend highlighted:

    • Use a compute service to execute code on demand
    • Write single-purpose, stateless functions
    • Design push-based, event-driven pipelines
    • Create thicker, more powerful front ends
    • Embrace third-party services

    Migrating to immutable, serverless infrastructure is a topic to be discussed well beyond this blog post and even Erlend’s session. Whatever you do in the future, though, take his parting words to heart: “The cloud can solve many of your problems, but in the end you cannot transfer your responsibility for security.”

    Erlend’s session was packed with information. If this piqued your interest, you should watch his complete session from All Day DevOps 2016 (just 30 minutes). The other 56 presentations from the conference are also available online, free-of-charge here.

    Also, this year’s All Day DevOps is Oct. 24. It’s online and free, so be sure to check out www.alldaydevopsy.kinsta.cloud to learn more and register.

    — Derek E. Weeks

  • Codeless Automation: Shattering the Testing Bottleneck

    Codeless Automation: Shattering the Testing Bottleneck

    In his article, “Moving from ‘Tester’ to ‘Quality Enabler’,” Scott Edwards clearly lays out the predicament in which QA testing is currently at: with one foot in the quicksand of manual testing, while the other tries to jump over the hurdle of automation. Ever tried hurdle racing with one foot? You probably won’t reach the finish line.

    Edwards mentions the infamous label that is attached to testing—bottleneck—and cites the “DevOps Review 2017” research paper of Computing.co.uk that found two-third of DevOps practitioners report QA testing as the bottleneck of the development process.

    Why is that? Why is testing bottlenecking development, and how can it be untangled?

    What Creates the Development Bottleneck?

    The testing bottleneck is created primarily due to two reasons:

    1. The reliance on manual testing, and
    2. “All the code that is getting thrown” QA teams’ way.

    Manual testing is extremely time-consuming, sometimes to the point that it’s close to impossible for test teams to keep up with the fast pace of DevOps. Manual testing is also error-prone, simply because the testing is performed by humans.

    The code issue is related to automation. Due to the pressure imposed upon test teams, automation is being hailed as the great savior. While the word “automation” has the flair of hands-free simplicity, the reality of automated testing is quite the opposite.

    Automation relies on code and testers aren’t coders. So instead of simplifying and speeding up testers’ work, the opposite is being achieved. Testers are thrown out of their comfort zone, being bogged down by tasks that aren’t in their “job description.” The frustration of all involved is easy to understand. The development team feels that testing is slowing them down, while the testing team feels they are required to jump through hoops.

    Add to that the complex setup and heavy maintenance required of automation, and you start to doubt whether it is indeed the right path for testing.

    Well, it is. With the ever-faster pace of DevOps, the pressure to shorten release cycles and the cutthroat competition software and web application companies are facing, test automation is the right path, despite all the challenges.

    What must be done to clear that path is remove coding from automation.

    Codeless Test Automation to the Rescue

    Codeless is a growing force in mass technology. It has liberated the web. Codeless platforms such as Wix and Shopify allowed millions of businesses to create an online presence quickly and effortlessly and led to the incredible expansion of the web. On the enterprise side, codeless platforms are making strides with the likes of Salesforce, FileMaker and Nintex, which offer drag-and-drop solutions to speed up and optimize efficiency of large-scale administrative tasks.

    The idea behind codeless is the modern-day “power to the people.” Just a decade ago, a small family business from Minnesota would have never been able to afford and maintain an online presence, but now their 22-year-old can set up an online shop, promote it on Facebook and in no time start selling their goods to a consumer across the ocean.

    Applying the same logic to testing instead of coding from scratch test after test—which is time-consuming and costly, and requires acquired skill—codeless test automation allows for quick and simple creation of tests, no coding skills required. It allows QA testers to focus on what they do best: testing.

    Automation shouldn’t be a hurdle that testers need to jump above; it should be a tool that pushes them forward. As things stand now, automation in testing is at a counterproductive standstill. Don’t get me wrong: Automation means well, but at the moment it’s like a misguided teenager that brings havoc even when he aims for harmony.

    How Does Codeless Test Automation Actually Work?

    Codeless test automation functions very similarly to enterprise codeless platforms. Both take complex tasks traditionally achieved by coding and simplify them through automated code generation.

    With test automation a tester creates a flow by visually binding elements that represent “clicks” in the software or web application. The tester simply goes through motions of the scenario he wishes to test, and the code for the test is generated under the hood.

    We can get more technical than this, but there’s no need. That’s exactly the idea with codeless, isn’t it? Testers could focus their efforts on testing functionality and evaluating the user experience rather than struggling with test creation.

    This ties back to Edwards’ main point: Testers get so entangled with test creation that the whole point of testing—quality assurance—gets lost.

    Side Note: The proposed job title change suggested by Edwards—Tester to Quality Enabler—might sound merely cosmetic, but it can deliver real value. As illustrated in this collection of QA testers’ venting from a reddit forum, QA testers often harness feelings of underappreciation and of being looked down on by their developing colleagues. Recognizing testers’ valuable contribution to the overall quality of the product will surely elevate their feelings of self-worth and enable them a stronger sense of pride in their position.

    Using Codeless for Deeper Integration of Testing Into DevOps

    Codeless automation is best utilized in sanity and regression testing, the two most basic and frequently applied tests. Sanity testing ensures that all incorrect inputs a user could enter are accounted for. Regression testing is like inventory counting, in a way: you just need to go over your stockpiles and see that everything is in order.

    Thinking on inventory: Let’s say I have a huge warehouse full of thousands of items. Checking inventory by hand is a very time-consuming and error-prone process. How many boxes on a piece of paper (representing real boxes on shelves) can you tick before you’ll make a mistake? Now think of Amazon’s warehouse. Inventory counting is done using automation with the press of a button. It is instantaneous and error-free. It’s an action that’s been done so many times before, there’s no need to plan it—or to code it—all over again. Simply run the same command—“inventory”—and you’re done.

    If you’re using codeless automation, all you need for regression testing is to click a button and the scripts run. You are done and able to divert your attention to sanity testing, making sure the new iteration is solid.

    For the next iteration, you simply pile the sanity testing of the previous round on top of the regression scripts and, again, you’re done.

    Continuous testing is getting a whole new meaning with codeless automation as part of the equation. If regression testing is done with the click of a button, and sanity taken care of simultaneously, the reporting back to development is swift and focused and the bottleneck is shattered.

    Codeless enables deeper integration of testing into the development process simply by arming testers with automation tools that simplify tasks, not complicate them.

    About the Author / Assaf Dudai

    Assaf Dudai is the content marketing manager at TestCraft, a pure SaaS continuous test automation solution for QA professionals. Assaf is mostly looking into the future of testing, how automation, codeless and machine learning can and will help testing evolve. You can connect with Assaf on LinkedIn.

  • Using Application Deltas in Deployments

    Using Application Deltas in Deployments

    Developers have been versioning their source files since the early days of Unix. Anyone else remember SCCS? Back in those days, storage was at a premium: Disks were small and very expensive and it was too inefficient to store a complete copy of each version of the source file. So version control tools would simply store deltas—the changes made from one version of the file to another. This was more efficient; if you only changed one line in a 3,000-line file, the system would simply store the new line and some metadata about where it fit in the file and who made the change.

    Some tools would save the latest version of the file and use reverse deltas to record the changes if you wanted to revert to an earlier version. This was usually more efficient, since generally developers want the latest version of a file. As such, it was more efficient to apply deltas to get earlier versions than it was to start with a 3-year-old file and roll forward through 472 changes to get the file you wanted.

    These days, disk storage is literally 3 million percent cheaper than it was in 1981. (The storage in my phone would have cost $38 million in 1981!) Because of this, storing deltas is not the panacea it used to be. In addition, CPUs now can crunch numbers faster than they used to, so it’s possible now to use math to compress and store a complete copy of the new version of the file. That’s what modern revision control tools such as Git do.

    Now that’s all very interesting, but the title of this post is, “Using Application Deltas in Deployments.” So why am I rattling on about source code versioning?

    Well, here’s the rub: While deltas may no longer have much of a part to play in tracking changes to source files, they most certainly do when it comes to deploying applications, and for similar reasons that plagued us (or our parents!) 30 years ago.

    Applications are Made of Components

    Let’s be frank: An application that consists of a single binary on a single server is not going to need deltas. That would be akin to a change to a single-line source file. All you need to do is to replace the old binary with the new and you’ve deployed. But applications are getting more complex:

    • The application may be split across multiple servers. An n-Tier application may have different tiers running on different servers (and even different operating systems).
    • The application could be wholly or partially containerized. You can have multiple containers each running different microservices.
    • There could be database changes associated with an application change. Indeed, the application change could consist solely of a database change. Database updates are pretty much always deployed as deltas; alter scripts are used to add and remove columns, tables and indexes.

    In the same way a single binary can be built from lots of different source files (each of which is under its own version control), an application can be made up of lots of different binary components. Clearly, if the application consists of lots of different components and those components are resident on different servers or containers, then deploying every component every time a new application version is deployed would be wasteful, time-consuming and error-prone.

    An application version “delta,” therefore, would track which components have changed between the version of the application being deployed and the versions of the components on the target environment. This allows the deployment tool to only deploy the components that have changed. So what do we need to track to make this work?

    • Component Versions. Obviously, we need to version each individual component so we know which version of which component is included in any particular application version.
    • Which component version is present on any particular endpoint within a target environment. That way, if the application we’re about to deploy has version 5 of a component (a DLL or a WAR file, say) and version 5 is already present in the target environment, then we don’t deploy that particular component.

    This approach has a number of advantages. You can deploy to an existing environment and only deploy the components that have changed, minimizing the downtime and reducing the risk of failure. You can deploy to a newly provisioned environment and deploy all the components within the application. Then, if you need to fire up a new virtual machine (or a container), the deployment pushes everything. But the next time you deploy to that environment, only the components that have changed are deployed.

    Database Deltas

    We may need to consider special cases when it comes to database components. Remember we mentioned that database changes are nearly always delivered as deltas, SQL “alter” scripts that make changes to the previous version of the database schema? Well, if all you’re doing is deploying the next version of the application, then that works fine. However, what if you’re deploying a version that’s several releases ahead of what’s in the environment? This can easily happen if the test and production environments are “pulling” deployments into their environment (as opposed to having them “pushed” via a continuous delivery process). Now, just applying the alter script that is associated with this application version won’t work in any environment that doesn’t contain the previous version of the application.

    Picture an application (“My Application”) that is up to version 6. It has two components: a WAR file and a database component that contains alter scripts that roll the database schema forward. Here is what the last four versions look like:

    So version 3 has a new version of the WAR file; version 4 has a new version of the WAR file and a database alter script that adds a column to a table. Version 5 only changes the database (it amends a stored procedure which uses this new column). Version 6 applies a new change to the WAR file.

    Now, in a continuous delivery process, the test rig being targeted will receive each new version. So when version 4 is deployed, the rig will receive version 4 of the WAR file and the alter script will be executed to add the column to the table. Then, when version 5 is deployed, only the DB alter script will run (alter.sql;2) to amend the stored procedure. The WAR file is not deployed, since the server already contains version 4 of the WAR file. When version 6 is deployed, only version 5 of the WAR file is deployed. This is exactly what we want.

    But now what happens when we move to user acceptance testing (UAT)? By this stage, we’ve done as much automated testing as we can get away with; in UAT, real users need to log on to our test system to make sure they’re happy with everything. So we’re not going to do a “push.” In UAT, the test lead does a “pull” when they’re ready to accept the new version of the application for testing. So they pull “My Application Version 6” into their test environment and get their testers lined up with coffee and pizza.

    But what happens if the test environment is currently on version 3? Well, for the WAR file, there’s no problem—there’s a difference in the component version for the WAR file so the new version is deployed. But what happens to the database? There are no alter scripts associated with version 6 of the application, but without applying alter.sql;1 and alter.sql;2, the database schema is not going to be valid for use with myapp.war;5. The testing will fail, not so much “falling at the first hurdle” as “falling in the paddock.”

    Deltas allow us to cure this issue. What we need to do is identify the components that represent the database changes and get them to be applied in sequence for each interim version. In that way, successive database “deltas” are applied to roll the database forward to the correct schema version before the required version of the application is deployed.

    So, in our case, going from version 3 to version 6 would deploy the following components:

    So we will deploy (and run) alter.sql;1 to add the new column, then alter.sql;2 to amend the stored procedure and, finally, deploy myapp.war;5. Deltas mean we do the minimum required to get the application version to the desired state.

    Conclusion

    Versioning the components that make up an application means it is easy to determine the deltas (changes) between one application version and another. Recording what version(s) of the components are on the endpoints in a target environment makes it easy to only deploy the components that have changed, minimizing downtime and making the deployment quicker and less error-prone. Identifying the deltas for a database gives us the ability to roll databases forward and jump between releases, which is invaluable when transitioning from an agile “push” to a waterfall-centric “pull” deployment model.

    — Phil Gibbs

  • 4 Predictive Analytics Challenges Facing SRE Teams

    4 Predictive Analytics Challenges Facing SRE Teams

    “Predictive analytics” refers to the many techniques used to analyze data to make predictions about the future, such as data mining, statistics, modeling, machine learning and artificial intelligence. Being able to predict technical and business outcomes is one of the main promises of big data. Indeed, many businesses have invested heavily in infrastructure to anticipate key performance indicators such as demand, pricing or maintenance. The investment required is substantial but, when it works, it yields a very positive ROI.

    In this post, we’ll examine why implementing predictive analytics in a site reliability engineering (SRE) context requires a significant investment in both setting up the right data sets and applying the right logic to that data.

    SRE should in theory be a prime target area for predictive analytics. Why? Because companies have already widely adopted monitoring tools over the past decade. In doing so, they’ve amassed extensive system health and behavior data that is ripe for analysis. SRE teams have also increased their ability to make sense of this data as data science tools have become more accessible. Basic data science aptitude is increasingly become a desired skill when hiring engineers.

    Additionally, the SRE team is responsible for a mission-critical KPI: system availability. This single metric is directly tied to customer SLAs in digitally dependent businesses. SRE teams have a strong incentive to implement predictive analytics. Why? Having the ability to prevent downtime by addressing potential issues before they hit the production environment is hugely valuable. More system availability translates into more revenue, higher customer satisfaction and lower costs of problem mitigation.

    However, the reality is that an SREs job is still mostly focused on the diagnostic, mitigation and “fixing” parts, as opposed to more proactive tasks. This is in big part because predictive analytics applied to SRE/DevOps has proven difficult to implement and perform in a disciplined manner. Therefore, it is still an aspirational goal at most companies.

    SRE Challenges in Predictive Analytics

    Up next, let’s explore the four challenges that need to be overcome to implement high-quality predictive analytics practice in an SRE context, including:

    • Data collection
    • Data quality
    • Data volume
    • Model usability

    Challenge 1: Data Collection

    First, the working data sets need to fit what you are trying to predict. In this case, it is challenging to get the right data set because the SRE team’s mission is very broad; specifically, ensuring the reliability of the entire system. The difficulty lies in the fact that there is no such thing as “full system” data readily available. All the relevant data is distributed across siloed, or overlapping repositories and monitoring tools. Centralizing the data requires either a full system agent (which takes a lot of work to set up) or through APIs, which requires mastering a multitude of API integrations. An additional obstacle in getting the right data lies in the inconsistent nature of events, time series data and logs. These disparate data types can’t be analyzed together unless they can be transformed into a uniform data set.

    Challenge 2: Data Quality

    SRE environments emit lots of data and some won’t make sense for the purpose of the analysis. Whichever data collection method is used, the information tracked often reflects some level of bias on the part of whoever set up the tracking mechanism in the first place. This leads to capturing both irrelevant data or inadvertently leaving out important information. In other words, the data is often both noisy and lacking at the same time.

    The key to successful modeling lies in the selection of prediction variables with data that is obtained before the event predicted happens (in this case, a failure). Defining the variables that lead to the determination of a subsystem failure is critical and requires specific domain knowledge. It is difficult to define and extract a pure data set sanitized of noise. The more data you leverage, the higher the chances that you are actually “feeding the beast” with irrelevant information. Too much irrelevant data leads to inaccurate models and false positives.

    Challenge 3: Data Volume

    Here you can have two different types of challenges: too little data and too much.

    Not enough data

    You typically need at least 100 instances of what you are trying to predict and probably at least 100 where it didn’t work to train a model. This assumes the data set is clean and directly relevant to a very narrow, specifically defined problem. In the case of SRE, the scope of issues that can affect a system is such that the data set required is very substantial. Likely, it’s several months worth of a wide variety of system behavior measurements. For example, if you collect data only through passive collection, think about the lag time between t-zero (when you start collecting data) and the time when you would have results. During that time, you have a bootstrapping problem—your model can’t be trained to predict accurately what you want.

    Too much data

    There are two practical points to make around the notion of “too much data.”

    • Lack of standardization: Processing large volumes of data for analysis remains a difficult engineering challenge. There is no standardization for data pipeline; every method has a limit in terms of how many inputs it can ingest on a per-second basis, and ingest rates often end up becoming bottlenecks in large enterprise environments.
    • Volume does not equal accuracy: There is a point beyond which you get diminishing returns: statistical theorems show that after a certain point, feeding more data into a predictive analytics model will not provide more accurate results.

    Overall, getting the quality and quantity of the training data right is the best investment you can make in building your predictive capabilities.

    Challenge 4: Model Usability

    From a pure “mechanics” standpoint, the analytical concepts and methods are there to reflect the complex thinking required to predict the behavior of an enterprise’s combination of technology, information and infrastructure. Predictive software relies heavily on advanced algorithms and methodologies such as logic regressions, time series analysis and decision trees. The problem is that the workings of the kinds of systems that SREs deal with are very hard to understand in the first place, let alone predict. It is not uncommon even for SREs to not entirely understand how a system works if they were not the ones who initially set it up. In addition, every company’s system is different.

    When data scientists design predictive models without years of experience in an SRE capacity in multiple environments—in other words, without leveraging subject matter expertise—it rarely works perfectly. Often the models they create lack usability, even after the validation process to confirm its quality. A modeler will split the available data into a training set where the model is run to generate predictions and compare performance against the actual outcomes. That will require multiple iterations to generate a model that has high overall predictive accuracy, and 9 out of 10 times the result is overfitting, or perfect for the training data. A subject matter expert in the domain of technical operations can cut the cycles of iteration and get to a predictive model that works with an acceptable level of risk.

    In short, implementing predictive analytics in an SRE context takes a significant investment both in setting up the right data set and applying the right logic. Stay tuned for a future blog post that will provide some pointers on how to achieve the best prediction results.

    About the Author / JP Emelie Marcos

    JP Emelie Marcos is the co-founder and CEO of SignifAI. Previously, JP served as the Chief Operating Officer of Tango, a leading mobile consumer application that reached 400 million regular users in five years. JP’s career spans 20+ years leading teams in technology startups and large companies as an investor, GM and CEO.

  • Fast Pass: Your Ultimate Guide to the Best of Jenkins World

    Fast Pass: Your Ultimate Guide to the Best of Jenkins World

    At the end of the month, Jenkins users and DevOps professionals from around the world will head to San Francisco for the annual Jenkins World conference, taking place Aug. 28-31. The conference lasts four days, but time flies when you’re attending trainings, keynotes, sessions and networking events. To make sure you get the best of everything at Jenkins World, we’ve put together a guide that will ensure you have the most productive and enjoyable experience.

    NOTE: devopsy.kinsta.cloud readers can get 20 percent off registration with the code JWDEVOPSCUST.

    Take the Time to Develop Your Skills

    The first two days of the conference will be full of workshops and training sessions. Regardless of skill level, there are trainings for everyone. Those just getting familiar with Jenkins can attend an “Introduction to Jenkins” training for free and learn how to easily create pipelines while putting in place best practices for continuous integration (CI) and continuous delivery (CD).

    Those a bit farther along their Jenkins journey can attend workshops such as “The Final Mile: Jenkins Pipeline and Patterns for Operations,” where attendees will review patterns of common problems and solutions specific to operations teams in a CD environment , as well as how to create Jenkins pipelines that address the needs of operations.

    Don’t Miss These Essential Sessions

    All attendees should make sure they make it to each of the keynotes, of course! Keynote speakers include Sacha Labourey, CEO of CloudBees; Kohsuke Kawaguchi, founder of the Jenkins project; and Jez Humble, CTO at DevOps Research and Assessment LLC.

    In addition to top-notch keynote speakers, there are a number of DevOps professionals you’ll want to hear from while at the show. Try to attend at least one of these sessions:

    Keep an Eye Out For …

    Make sure you listen in to find out who the winners are of the first annual Jenkins World Awards Program. There are two awards tracks: Jenkins Community Awards and CloudBees Innovation Awards. Each track has several categories including Jenkins Contributor Award, DevOps Automation Excellence Award and more. Winners will be announced Thursday, Aug. 31, at the show.

    You’ll also want to stop by the sponsor expo hall to check out dozens of great vendors. As with any good conference, there will be plenty of swag passed out. Exhibitors will include GitHub, BlazeMeter, Docker, JFrog, Fugue, Sonatype and others. Don’t miss out on taking a photo with the famous Jenkins butler at the CloudBees booth!

    Can’t Forget to Network

    With so many great sessions to attend, don’t forget to make time for networking events. It would be a waste to not take advantage of dedicated networking time to speak with experts and peers who have likely dealt with some of the challenges your organization may be facing. There will be several networking opportunities, including a reception Aug. 30 sponsored by CA BlazeMeter. Or if you’re looking for ways to stay active, attend a scheduled walk or yoga class.

    If you can check off all of the above activities, you’re bound to leave Jenkins World a better professional! Make a checklist at the start of the conference to ensure you hit all the events and sessions you’re most excited for.

    If you haven’t registered yet, there’s still time! Use the promotional code JWDEVOPSCUST for 20 percent off.

    About the Author / André Pino

    André brings more than 20 years of experience in high technology marketing and communications to his role as vice president of marketing at CloudBees. He has experience in several enterprise software markets including application development tools, middleware, manufacturing and supply chain, enterprise search and software quality and testing tools. André has particular expertise in working with the executive teams of enterprise software companies to create unique competitive positioning, build pipeline and establish sustainable market leadership. Prior to joining CloudBees, he served as principal analyst serving technology marketers at Forrester Research. He has also held a number of senior management positions in marketing and communications with companies such as Segue Software, Hewlett-Packard, Unisys Corp. and Bluestone Software. André holds a BS from Eastern University.