devopsy.kinsta.cloud

Tag: testing

  • Give Test Automation the Respect It Deserves

    Give Test Automation the Respect It Deserves

    One of the key mantras of DevOps is “continuous.” Effective DevOps relies on automating routine tasks. Any step in the process that requires human intervention or manual input becomes a roadblock that can slow or halt productivity. To the extent that it’s possible, anything and everything should be automated—and that includes testing.

    Test is like the Rodney Dangerfield of DevOps—it gets no respect. It should, though. Test—and test automation—is a crucial element of a successful DevOps program. A recent Gartner survey found that 50 percent of respondents listed test automation as one of the top enablers of DevOps success.

    Joan Wrabetz, CTO of Quali, is focused on automating test capabilities and ensuring that testing gets the respect it deserves in DevOps. DevOps isn’t really a process with a beginning and end. It’s a continuum—a self-perpetuating cycle. When test teams take the lead, they need to find ways to push automation back into development and forward into staging and production. “In fact, the problem of test automation is particularly difficult, and this means that the team that implements test automation is in a great position to lead DevOps initiatives beyond test,” Wrabetz says.

    That self-perpetuating cycle “starts” with planning and development and “ends” with deployment into operations, but that flows right into monitoring and management, which leads to discovering issues and identifying new feature requests—and that takes you right back to planning and development. If done properly, test falls somewhere in the middle of that cycle, so it’s imperative to streamline and automate that functionality as well.

    “Sometimes test drives DevOps. Test organizations have been focused on automation already for 10 years, so often they are the driver for DevOps,” explains Wrabetz. “When test is the focus of DevOps, it is called continuous integration. However, it is also important to work to continuously expand automation outside of test.”

    Key Drivers

    According to Quali, there are five key components that are essential to successful automation in a DevOps environment:

    1. The infrastructure. It starts with one very important characteristic that is critical to testing and often is not automated—the ability to automatically mimic the production infrastructure in which the software (or hardware) being tested is expected to run. Having the ability to give each tester (or set of automated tests) their own personal replica of the target production infrastructure—a.k.a. a “sandbox”—is critical to ensuring that the software coming out of test can be deployed automatically and continuously into production. Automating tests is not nearly as difficult as automating the underlying infrastructure in which those tests run.

    2. The production environment workload. Often not considered in automating testing is the requirement to mimic the workloads that would be common in production. This includes the network traffic, other application workloads and network security profiles. These activities need to be automated within the sandbox to create a more realistic environment for testing.

    3. The Tests themselves. Of course, the tests themselves also need to be automated and there are a variety of tools to assist in this process. As software gets more complex, this continues to be a challenge, with mobile testing replacing GUI testing as the most challenging type of testing to automate.

    4. Reporting and result analysis. To provide continuous automation from development through testing to delivery, it is important to see and analyze test results automatically.

    5. Tool Integration. Automation into and out of testing needs to be enabled. This means sandboxes and test automation suites need to be API-driven so that they can be started by a previous tool in the DevOps tool chain and can also initiate a tool that follows testing in the tool chain.

    Show Some Respect

    Testing is a crucial part of the DevOps continuum. It needs to be done properly and effectively, but it also needs to be a streamlined process that drives DevOps forward rather than a hurdle that slows everything down.

  • It’s Time to End the Automated Tests Arms Race

    It’s Time to End the Automated Tests Arms Race

    I grew up during the end of the Cold War. During that time, I remember hearing that there were enough nuclear weapons to destroy the world 20 times over. That always seemed silly to me. Wasn’t once or twice plenty? Wasn’t it more important to have the fastest and most effective weapons rather than to simply have the most weapons? Before I could devise a satisfactory explanation, “Rocky IV” was released, successfully ending the Cold War.

    Number of Automated Tests:  The New Iron Curtain

    Now, 30 years later, I’m beginning to notice another arms race taking place. At a recent DevOps conference, I saw speaker after speaker boasting about the number of automated tests their respective shops were running daily. Again, I was confused. Those numbers provide no real information. What’s more important than the quantity of tests you are executing is the efficacy of those tests and the speed of execution. Why did no presenter have those numbers? It felt like Cold War braggadocio all over again.

    issues_arms-race_nuclear Why do I care? Shouldn’t I view the embracing of test automation as a good thing, regardless of the metric used to measure it? When creating a culture of automation, test counts do serve a purpose. First of all, It’s a simple metric. Just count the number of tests and dance in the self-exulting joy of your automation prowess. It also shows the power of automation compared to manual execution. So what’s the problem? The reason I care is this: Test-case count, while simple to gather and easy to understand, puts the focus on irrelevant data and ignores more critical information, similar to judging a military on number of weapons alone (yes, I’m forcing the analogy). Number of test cases executed daily isn’t just the wrong metric; it’s the wrong unit of measurement! Trying to determine a development shop’s test automation maturity by counting number of tests executed daily is like trying to determine how many gallons a good developer weighs. It’s just nonsensical data that doesn’t answer the important questions:

    • What percentage of your code base is covered by your automated tests? If you knew that your 5,000 tests only covered 15 percent of your code, would you still stand on a stage at a conference and boast?
    • How long do your tests take to run? If it takes all night to execute the suite, then you can kiss multiple releases a day goodbye.

    Using Proper Metrics, or the Strategic Defense Initiative

    So what are the automated metrics we need to worry about? I have a few favorites, in no particular order:

    • Test execution time: Speed matters. A decrease in testing time allows for an increase in number of test iterations, an increase in features per release, an increase in release frequency, etc.
    • Percentage of code covered: The tool you use to get this number doesn’t matter. The key thing is that you have a standard way to track this metric and whether you are trending upward. This trend gives you the next important metric …
    • Automation progress: How fast is the discipline growing? When are you projected to hit the point where all tests that can be automated have been automated?
    • Defect density: While this isn’t specific to automation, the trend in this metric helps show the benefit of automated testing. Bonus points for having a pre-automation baseline.
    • Number of automated test executions: WHAT? Hey, sometimes you have to parade the military through Red Square to put on a show of strength. This isn’t a bad metric when used in conjunction with the others. But it is not enough by itself!

    Keep on Rockin’ in the Free World

    By focusing on the above metrics and not solely on test execution counts, you will ensure you are moving in the right direction in terms of test automation. Similar to how Gorbachev moved the Soviet Union in the right direction with his policies of perestroika and glasnost. OK, OK. The analogy is officially dead.

  • DevOps Matchmaking: APM + CI = A Perfect Marriage

    DevOps Matchmaking: APM + CI = A Perfect Marriage

    As anyone who has ever been set up on blind dates by friends can attest, finding the right match in today’s world is a tricky feat.

    There’s a lot of evidence suggesting that the frantic pace of life makes such matchmaking more difficult than ever. That said, when the right set of complimentary traits exist and there’s a willingness to find common ground, things can work out.

    Spike Jonzian-notions of artificial intelligence aside, certain technological partnerships are increasingly being perceived as “a perfect marriage” within the context of DevOps automation.

    Register today for the upcoming webcast: “Using APM to Turbocharge Continuous Integration”

    These technological matrimonies closely support the underlying tenets of DevOps methodology—collaboration, experimentation and increased velocity. In fact, it’s easy to wonder why no one ever conjoined them before. But we were using traditional waterfall development and operations and so couldn’t conceive of such a thing. On the other hand, emerging availability of tooling supporting new integrated workflows represents a significant catalyst.

    Finding the Perfect Fit

    One such DevOps confederation getting a lot of attention is the matchmaking of continuous integration (CI) and application performance management (APM) technologies. Tying together these once-disparate systems makes a lot of sense, despite the fact that few tools would seemingly appeal to more (traditionally) polarized constituencies.

    Continuous integration has been a staple of the development community for many years. It emerged as teams evolved the practice of long build times into more frequent check-ins to deliver code and look for potential errors more quickly. Today, with the rise of CI tools such as the Cloudbees Jenkins platform, some development teams are touting tens of thousands of builds per day.

    On the other end of the spectrum, squarely on the turf of IT operations, APM has been evolving rapidly over the past few years as organizations have sought to improve use experience and applications reliability via dedicated monitoring. Enabling this ability to identify and address emerging issues has resulted in APM becoming the fastest-growing segment in the IT ops management market, according to Gartner.

    DevOps matchmaking factors in when you bring these tools together to create an automated workflow that spans their respective use cases. A cross-functional team can leverage APM-level monitoring and KPIs within the domain of CI and testing to identify potential performance obstacles introduced in new code. Doing so then rather than waiting to respond later in the applications life cycle benefits everyone.

    Reaping the Benefits

    For example, say an e-commerce applications developer has implemented a new inventory search functionality—something considered a key differentiator by business management. The developer knows this feature could rely heavily on some back-end database system and wants to ensure it’s tested properly, even as the code is being introduced.

    Integrating APM and CI allows the entire team to monitor the process and test various conditions related to the update, tracking multiple builds and looking out for related performance issues. Such integration also allows for centralized tracking of other key attributes such as build number, status and build date, to further ease management.

    Most importantly, by tying these two critical aspects of the applications delivery life cycle together, the collaboration and feedback loops required to truly embrace DevOps methodology become ever more valuable as these core Dev and Ops functions become intrinsically linked.

    If this type of DevOps advancement gets your heart pumping, you’re not alone. Later this month, we’ll host a webcast here on devopsy.kinsta.cloud featuring experts from CA Technologies and Cloudbees. They’ll be discussing the benefits of matchmaking with APM + CI integration. They’ll also detail real-world benefits organizations leveraging integration of our respective technologies (CA APM and Cloudbees Jenkins) enjoy.

    We obviously feel that this is a DevOps tooling marriage that will mature, change and stand the test of time. We’d love to garner your questions and feedback on the topic.

    Which is all good, because thankfully there’s no such thing as Tinder for DevOps … yet.

  • Infrastructure-Testing Dominoes

    Infrastructure-Testing Dominoes

    Understanding the artifacts of a given technology component is critical to an enterprise-class DevOps service offering. Second-most critical is understanding the dependencies between given components. Testing helps illuminate those dependencies.

    For example, let’s assume I have a server running the previous version of the Windows operating system (OS) that I now want to upgrade to the latest version. The previous deployment package of this version of the OS would be an artifact. It should also include all of the test automation I run to validate that version of the OS is working after I install it. The artifact also might include configuration files I deploy when I install that version of the OS on this type of server in this class of environment. Having these bits of automation bundled together allows me to do a successful install of my OS (assuming no hardware errors) on perhaps many servers.

    As I create a new artifact for the latest Windows OS, I keep in mind all the things I had in my last version. I might include a newer config file and perhaps a set of tests (albeit new ones) to make sure it is working when I am done. Bundling all these bits gives me a new Windows OS artifact or package that I can use over and over on this kind of server in this class of environment. The idea is that I am creating a single kind of artifact or package to cover this kind of technology—in this case the latest version of the Windows OS operating system. All my artifacts should be version-controlled. So when I’m done I have an artifact for the previous Windows OS and one for the latest version. Each artifact version has embedded or included testing (pointers to, or discreet automation) to verify the validity of the install, and perhaps a little more to prove the OS works.

    Within my DevOps tooling, I likely have some sort of artifact management system. It may not be as formal as a full-on configuration management database (CMDB) system, but whichever tool I use will have some sort of database to manage my artifacts or packages by version. How long I retain these artifacts will be defined in my retention policies, but likely I will retain every version in use at my company today. This is step one, and it forms the foundation for my next effort in cataloging the dependencies that exist between artifacts.

    Enter the Spider’s Web

    Under the scenario I have described, when I deploy either the previous OS or the latest one, I would run a series of tests specific to that version to ensure the OS is working properly. As an infrastructure engineer, I tend to think my job is done at this point. But it isn’t. To truly validate that my OS is working, I really need to know what is running on the server I just deployed to. If my server was bare or new, then I am really done. But if my server has middleware running on it, then the job is not finished until I re-deploy the right version of the middleware on top of my new OS and verify it is still working properly.

    To pull this off, I will need a catalog of the technologies that depend on each other. The OS example is kind of simple, because it lives at the bottom of the stack where a server is concerned. But on top of the OS might reside a middleware platform (perhaps messaging or a database). On top of the middleware might reside an application or perhaps several applications, depending on the horsepower of the server in question. Upgrading the OS component implies I am redeploying the middleware layer and then the application layer on top of that.

    Enter full-stack engineering. It takes an application-centric view of the world, but this implies a single application-centric view. It looks at one app from top to bottom and treats the entire stack as a single version-controlled artifact. But if you have legacy environments that run multiple applications on the same physical box, perhaps on the same virtual instance, you have the makings of a spider web of dependencies. There are no full-stack solutions (to my knowledge) that have figured out how to incorporate multiple applications at the top of the same stack, varying middleware in the middle and “n” number of virtual instances that ride on one OS on one physical box. Instead, each piece of the technology tends to be treated separately. Thus the need for some sort of catalog that tracks the interdependencies between them all.

    Starting the Domino Testing

    So once I have discreet testing that is version-controlled (or associated with) each type of technology component, I will have to re-execute those tests if anything “under” me undergoes a change. For example, if the version of my WebSphere is upgraded, beyond just the core WebSphere testing that occurs, I really should execute the tests associated with each application to ensure “that app” is still working after the upgrade. You could apply this premise to a database layer (say, upgrading the current version of Oracle, as an example). To really know if the database upgrade was successful, you should run the testing of each dependent app and ensure “that app” still works after the upgrade. The results are likely to be surprising.

    To execute this kind of domino testing, even if you have the web of dependencies defined and the artifacts or packages managed, you can take one of two approaches: redeploy everything, although this method may not actually kick off tests if it detects the versions did not change from the last deployment (smart overwrite), or set up discreet testing jobs within your release orchestration software to execute the entire web of dependencies and track the results. In either case, you are going to need human oversight on whether it is better to keep the new version or roll back to the old one. Only a human can prioritize the impact of what might seem small or catastrophic (based on a human perspective) of a given app or series of apps.

    Impact to the Bottom Line

    True quality can be assessed only through testing. Part of breaking down silo-thinking is taking a holistic view of the technology your organization supports. Just because the OS support engineer’s testing passes does not mean the upgrade was a success. For an upgrade to be a success, the tests of everyone else who relies on that technology underpinning must pass to prove it worked. The entire team needs validation when the underlying infrastructure undergoes change. The dominoes of tests must pass from end to end. Only then can true quality or true success be measured.

    Of course, most engineers are reluctant to have “app software” guys influence whether they did their own jobs right. After all, most software quality problems are in the code, not due to infrastructure errors. However, if the applications upon which your business relies cannot work, no matter what the reason, after an infrastructure upgrade, it is a problem for the entire business, not just the software guys. To tear down the silos, we need to look at impacting events for change across boundaries and drive toward overall results, not just departmental ones. When this occurs, the cost of quality is significantly reduced.

    To continue the conversation, feel free to contact me.

  • Why a Dev/Test Cloud Is Not Optional

    Why a Dev/Test Cloud Is Not Optional

    How do you create a private Dev/Test cloud to standardize processes, tooling and environments across teams? Recently, I participated in an online panel on the subject of, “Creating an Internal Dev/Test Cloud,” as part of Continuous Discussions (#c9d9), a series of community panels about agile, continuous delivery and DevOps.

    Continuous Discussions is a community initiative by Electric Cloud, a provider of continuous delivery to businesses including SpaceX, Cisco Systems, GE and E*TRADE. I am not an employee or vendor of Electric Cloud.

    Below are a few insights from my contribution to the panel:

    What are your thoughts on Dev/Test cloud?

    It’s not a question of why—You don’t really have a choice. I don’t think that this is an optional thing. If you’re moving to this modern software delivery, you’re going to have to build this environment. If by chance you’re already doing some of the continuous integration continuous delivery (CICD) processes without it, it’s not going to last long. If I were to take a slightly different angle, I think the interesting question is where? So where is your Dev/Test cloud?

    I think of Dev/Test as a conceptual thing; it’s your factory. If we talk about it from a singleton point of view—as one environment, for example, or one integration environment that it is not going to support microservices and some of the future architectures—that’s when you might add multiple Dev/Test clouds. For example, if you think you’re going to have your browser grid on-prem or intermingling with your other integration environments, that’s a lot of work, and that’s a lot of maintenance. It’s just not a good idea. But at the same time, it’s actually amazing if you’re doing anything with artifact repos. Even though some of these artifacts are really small—and Docker images can be very small, if those repositories are not on the same network as the rest of your Dev/Test—it could be really slow pulling assets down. So that’s what can be brought on-prem. I think the bottom line is that you’re going to have to get there no matter what. It’s just going to happen. I think how you set it up is the interesting question, since there are a lot of options as far as how to do that.

    What are the challenges of creating a Dev/Test cloud?

    I think the first challenge is parity between your Dev/Test environment and your production environment. Replication would be insane because you’re talking about data centers, a lot of overhead, and it reduces flexibility. And everything we talked about before—the ability of developers to spin out Dev/Test environment and integration environment on a whim. However, there are things that you can absolutely consider, which is consistency between all your configurations, making sure that all your frameworks are consistent because chasing down bugs that cross applications throughout the system are very hard to deal with. But there is always the need to address that.

    And we’re even talking about agents sitting on your nodes. I mean, if you’re running some tooling on your servers, it’s good to know what the potential impact of having those, or not having those on your Dev/Test environment could actually be. I’d say that parity is probably one of the biggest challenges. And then the other challenge is just knowing who owns it, because I think that the best scenarios I’ve seen are when you have a steward of the Dev/Test environment. It could be DevOps, it could be somebody else who’s responsible for maintaining the strategy of what you do with those environments versus it being a wild west. So they don’t own it, but they facilitate making sure that parity exists, and facilitate making sure the automation and the provisioning and everything is as efficient as possible. Developers shouldn’t think about that. Dev/Test should be more of an offering. At the bigger organizations, that’s exactly what they do. They’ve created a shared services or cloud services division that is responsible for setting those up.

    What are the challenges you see people running into when they’re trying to create a Dev/Test cloud?

    It is a huge habit of developers to build locally, and I think that most developers I tell to stop doing that complain. It is a big problem, just from a pure pollution standpoint and the fact that you don’t really know the condition of each developer box, there is no consistency. Hopefully a Dev/Test Cloud is easy enough to utilize that they don’t see the need, and there is some training—I think it goes back to the idea that if you treat it as this singleton thing, you know that some Dev/Test environments live and die within an hour time period. They’re very short. That should be the idea, that you spin it up, put your code and test it quickly, get the IP address. You test it and kill it, and that’s it. You move on and continue coding, and as it moves downstream, you can have a more robust integration environment.

    I do think that there are best-of-breed Dev/Test clouds. I think you have the Dev/Test clouds that are great for the spinning up and down a pole instances. And then I think you have past types environments, and I think you have load testing clouds, and you have functional testing clouds. There may be a reason to break them out into different types. The most important thing is that everything is tightly integrated and that there is a strategy. It’s not just off the cuff, because that’s equivalent to shadow IT. Even if everybody knows it’s happening, it’s the same difference.

    What Are the Prerequisites for a Dev/Test Cloud?

    I think you have to have a steward, somebody who owns it, standardization. You need to have full master scripts, snapshots, images, that represent the de facto Dev/Test environment, and somebody needs to make sure that those are up to date and current so you have consistency across all the locations where these have been spun up. The contradiction is if you are using a performance testing cloud for load testing, a functional testing cloud, where you don’t have control over that infrastructure. In those scenarios, if you’re running Selenium scripts, you’re going outside in, so it doesn’t really matter as much.

    How to do it

    If you’re starting bottom up, you could think about it all the way through before you develop it. But most organizations aren’t in that position; they’re slip-streaming that into something else, and chances are you’ve already had this concept and you’re already using virtualization, so part of it you probably already have, but maybe not in the most efficient way. Many of the organizations I’ve seen start with something small. For example, they start with a robust integration environment. Druva is an example of a company that’s made this transition. But in all the scenarios I’ve seen, there’s been another catalyst, it’s never just been Dev/Test cloud. There’s always been something else. An architecture change to microservices, changing your product from an on-premises solution to a cloud-based solution, which was the scenario with Druva. There always seems to be something else. I don’t think it has to be that way, but it seems like there are other driving factors that necessitate change and the opportunity to change your pipeline, and they’re driven by the application. I don’t think there’s an easy answer. I think you have to rely on automation, you have to rely on tooling. You can’t be afraid of tooling. Don’t let the tool dictate what you do. You should be telling the tool what to do. If eventually you are a Chef shop and Jenkins, and it dictates what you’re doing, then that’s all you’re going to do. And you’ll never be able to adopt a future tool, so think about that. Inertia is a powerful force.

    I have very strong opinions about how Development environments should be set up. Stop building locally, and put it in the cloud.

  • What Happens to Test in a DevOps World?

    What Happens to Test in a DevOps World?

    I am always surprised by the urban legends around DevOps. One of my favorites is, If you implement a DevOps methodology, you can stop doing test. People famously quote that Netflix only tests in production as evidence of this. This is perhaps more like of game of Telephone, where much of the story gets lost in translation. It is true that Netflix created a tool called “Chaos Monkey” which generates failures in production as a way to test the fault tolerance of its software in large-scale production use.  However, this is not Netflix’s only test; it is an additional type of testing at scale that allows the company to test in new and different ways. Netflix still tests in software development.

    Okay, now that we are past the notion that DevOps is about eliminating testing, let’s look at what really happens with test in the DevOps world. In fact, the goal is for testing to happen everywhere and all the time. Let’s look at the two directions test can shift: left or right.

    Shift Left

    DevOps methodology encourages automation of activities across the entire development process. This can be realized by automating the link between development and test. Often implemented with an automated build tool such as Jenkins, a set of automated tests are triggered whenever a developer completes a build of new code. Of course, this means that both development and testing processes must be automated. Once you reach this goal, then testing can happen all the time. Every time developers want to check in new code, they can trigger a small set of “smoke tests.” If those succeed, they can check in code and when the build happens, they can trigger a larger set of regression tests. The result is a left shift of testing into development.

    Benefits of left-shifting test include:

    • Move testers into scrum teams to work on stories together with developers.
    • Design tests in parallel with development.
    • Give developers and testers the flexibility to help each other to reduce bottlenecks.
    • Early detection of bugs, shortening the critical metric for time between creating a bug and fixing it.
    • Increased transparency of code dependency issues because they are raised during the nightly (or more often) regression testing.
    • Virtual elimination of the test bottleneck that occurs after development is completed.

    Shift Right

    The urban legend about Netflix may be wrong, but that doesn’t mean that there isn’t merit to right-shifting test into production. Automated testing in production is effective for tests that would be impossible to perform outside of production, including scale testing and resiliency testing. Production is also a great place to introduce A/B testing by offering different interfaces to some customers and measuring the differences in business results.

    Benefits of testing in production include:

    • Better understanding of the real production environment and workloads.
    • Early feedback from customers on new features and user experience.
    • Better resiliency.
    • Overall better user experience for customers.

    No Shift

    Wait, if we shift left and shift right, is there any testing left to be performed in test? There are still a range of tests that need to be performed by the test team. With full automation, these tests can be performed at a variety of times and don’t need to be a bottleneck to release.  These tests include:

    • Performance testing
    • End-to-end use-case testing
    • Security testing (don’t think about doing this in production!)
    • Manual testing – often for user interface or mobile testing that is most difficult to automate

    DevOps methodology is not promoting the end of testing, but in fact a widening of testing to encompass all of the steps in the DevOps cycle. You know what they say: Test early and often! And next time you are sitting around the campfire discussing urban legends, let’s leave Netflix out.

    About the Author/Joan Wrabetz

    Joan's image croppedJoan Wrabetz is the CTO for QualiSystems. Most recently, she was the VP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.

  • Q&A: CircleCI’s Kevin Bell on Continuous Delivery

    Q&A: CircleCI’s Kevin Bell on Continuous Delivery

    There’s been a lot of talk and momentum lately when it comes to continuous delivery. And it seems every enterprise has moved part, or all, of its development efforts to continuous delivery—or is at least talking about doing so. But there are many challenges such as security, dealing with legacy systems and internal politics associated with the move. With that in mind, we wanted to get a sense of where continuous delivery stands in the market today. And for that we turned to Kevin Bell of continuous integration and delivery platform maker CircleCI.

    Kevin Bell has been a both developer and product manager of cloud-based software products, and is currently developer evangelist at CircleCI, where he gets to play with almost every deployment-related technology out there, hack a little Clojure and absorb the wisdom of very smart colleagues and customers. We thought it’d be a good idea to get his perspective on continuous delivery in the marketplace.

    Kevin Bell Headshot

    devopsy.kinsta.cloud: I wanted to get a sense from you where your customers are in the range of continuous integration and continuous delivery? What are you observing in the market?

    Bell: I would say continuous integration is a hard prerequisite to continuous delivery. You need to have continuous integration first, and once you have that down, you can expand from that into continuous delivery. Maybe there are people that automate deployments with a minimum of testing in the middle, but you really want to have continuous integration set up.

    Generally, I would say of our customers, on the order of half of the active customers, are deep into continuous delivery. There are certainly people that just use us to test and then do something else to deploy, and then there are those who aggressively, many times a day, deploy to production directly from CircleCI in a totally automated fashion. So we definitely have both extremes and then people that are experimenting in the middle on trying to figure out how to make that jump.

    devopsy.kinsta.cloud: Have you seen many of these changes in the past year? And are many of the changes to continuous integration and deployment happening outside of Internet and SaaS (software-as-a-service) companies?

    Bell: Lots of companies want to do it. So one change I’ve seen in the past year is that, not necessarily a change in the market, but a change in CircleCI’s world is that we have rolled out an enterprise version, which basically means a version of CircleCI that you can deploy behind your firewall. We are seeing a lot of interest in this among bigger companies. And we’re talking to a lot of different types of companies that are not SaaS companies, but are just any kind of company from apparel to finance institutions—just all manner of companies that take their internal software development practice very seriously, and very much want to do continuous delivery and have a modern software delivery methodology.

    It’s kind of just becoming table stakes that you’re going to follow these kinds of practices today, even if you’re not a SaaS startup. It’s not just about software deployment, either. It’s also necessary to attract talent.

    devopsy.kinsta.cloud: I would imagine, because we’re starting to see pilots and success stories, enterprises are seeing that it’s not just speed when it comes to feature release, but once the continuous delivery pipeline is up and running, it’s a more effective way to create and maintain software.

    Bell: That’s definitely a big part of the promise of continuous delivery, and I think it delivers. Pardon the pun. It does let you get software features out faster. If you do end up releasing a bug, it lets you fix it faster. But I think some of the most emotional stories of success that I hear from people talking about the dark old days when they deployed at once every two weeks, or once every month, is just how risky that felt. And when they clicked the big deployment button in those days, how much was at stake and how many things could go wrong because it was such a big blob of software being deployed all at once. A big part of the idea of continuous delivery is that you don’t do those big bang software deployments. You’re always just putting out a little bit at a time, and keep everything in an equilibrium throughout the process.

    devopsy.kinsta.cloud: These enterprises that you were talking to, what are some of the challenges or hurdles they have moving from legacy development to continuous deployment? What are the barriers they need to overcome to get to where they want to be?

    Bell: There’s certainly various barriers, and it depends on the organization. Some organizations have a momentum built around a certain process. Maybe there’s political issues in an organization, maybe they have a new CIO that wants to change how they do things, but they have a lot of legacy software and processes that still operate in the old-mold paradigm that they need to update. So there are a lot of systematic challenges organizations run into. But there are definitely technical challenges, too. And I think just broadly to compare, for example, continuous integration and continuous delivery, I would say that continuous integration has been around longer now and has kind of matured a bit.

    And a lot of the challenges have been uncovered within continuous integration when it relates to complicated and offbeat projects, whether that’s using things like mobile emulators like you can use on CircleCI to do iOS testing, or mobile device clouds, or browser testing, or all these things to kind of fill in the gaps, and test suites for big complicated software has sort of been explored and addressed to a large extent. But in continuous delivery there remain complicated, nontrivial systems that they’re deploying, not just little apps that go to Heroku or something. There’s this whole world of different policies and structures of software and systems that need to be continuously delivered. I think the industry and toolset makers are still exploring how to provide that.

    devopsy.kinsta.cloud: Earlier you mentioned that if an organization wants to attract talent, they need to have these modern development processes in place. So I would imagine developers don’t want to get bogged down, in their view, in old methodologies and want to get aligned with companies that have these more modern practices as part of their career development?

    Bell: That’s come up often at big companies. I would say that category of Fortune 500 companies, and not just software companies, they need to assure potential hires that they have modern software practices, and that even though they’re not a software company per se, that they value developers, and that they’re going to be a good place to work.

  • 5 Development Trends Reshaping Testing

    5 Development Trends Reshaping Testing

    There’s no shortage of new development trends, but virtual reality, containers, big data, hyperconnected apps and IoT are affecting testing the most today.

    If you like change and challenge, it’s a fun time to work in software. Some technology developments, such as cloud computing, have been gaining speed slowly over many years. That steady progression has meant that developers can stay on top of trends easily and be informed about market changes. Other areas, such as containers, have come to fruition much faster without as much notice. Development teams have had to scramble to stay ahead of these trends, hoping to capitalize on a their efficiency gains while staying ahead of the curve. Here’s a look at five important software trends which are having a profound effect on testing.

    Containers

    Docker made containerization cool and exciting in 2013, even though containers have arguably been around for many years, if not decades. Developers love containers because the technology makes it much easier to set up and install software in different environments, such as in the public cloud or in a private data center. Containers wrap up all of the infrastructure components that otherwise would be installed separately into one easily portable package with a common standard. And since multiple containers can run on the same hardware without creating problems, you can do more with less infrastructure. Typically, containers can move between different environments with relative ease—a nice benefit for multicloud infrastructures. Containers have evolved quickly from an experimental platform to a mainstream infrastructure technology, with many major businesses taking advantage of containers to deliver their product at scale.

    Testing considerations: It’s hard to find much at fault for containers when it comes to facilitating software testing. Because of the ease in setting up test environments and moving code from one environment to another during the production cycle, testers can work much faster. Before containers, there was a lot more waiting around by both developers and testers. When it takes two weeks to set up an environment for a testing cycle that took two hours, teams would tend toward less frequent releases or would do less testing. Now, it is a matter of minutes to set up a testing environment. This is great for companies and development teams that value testing and want to test more frequently with greater access to a variety of environments.

    Another boon with containers is stability. In the past, bugs sometimes appeared due to configuration errors from setting up dev/test environments. Given the plug-and-play nature of containers, there’s little if any concern about the proper configurations being completed before a testing environment can be used. Security has been an ongoing concern with containers, especially in highly regulated sectors such as health care, government and financial services. In those cases, companies may have to support two versions of the code—one in a containerized platform and one in a more traditional virtual or bare-metal environment. We believe this will be a short-term situation as containerization technology matures and becomes adopted across all industries.

    Big Data

    Whether you want to use this vague and overhyped term or not, big data is simply a new way to refer to larger than normal data sets. Big data became a big trend as the hardware and software to store and manage it got a lot more powerful and cheaper, and the cloud shift helped to centralize data sets into one location. Some large companies and research organizations are seeing interesting results from big data projects already, whether in more targeted drug development (precision medicine) or just-in-time environmental data for farmers. Now, every Tom, Dick and Harry is trying to get into the big data market, which means there are too many solutions to count. A natural weeding-out process will occur over the next few years, as the prominent big data players and NoSQL standards converge into two or three leading providers.

    Testing considerations: Big data applications bring value through the way they manage and/or analyze data, not so much through the user interface. Therefore, testing for big data applications is going to require more technical, database-oriented skills. Those skills are also different from the typical relational database skills of the past, given the fact that big data platforms typically are designed to manage structured and unstructured data. These are open source tools and systems such as NoSQL and Hadoop, along with large vendor systems such as SAP HANA. Testers in this field will need to be constantly adapting learning these new technologies.

    Virtual Reality

    Investors have funneled $8.8 billion into the sector since 2012, according to SuperData Research. Google, Facebook, Microsoft, Samsung and HTC are all in the game. The market is segmented between VR, augmented reality (AR) and mixed reality (MR) and consists of both traditional (PC/console) and mobile device applications. VR is a fully immersive, fully virtual experience that mimics reality, while AR and MR are different variations of blending real and virtual environments together. VR applications aren’t just for gaming anymore: We now are seeing the technology being used for product design and collaboration, tourism/travel and entertainment, including film.

    Testing considerations: VR is an emerging technology still, with many different types of platforms, no clear leaders and no real standards in place. That makes it more difficult for developers and testers to make strategic decisions around technology. As well, there aren’t any great tools for automated testing in these environments yet. Therefore, the need for manual and real-world testing will add time and cost to the project. Yet, investing in doing this right is critical for ensuring optimal user experience. In virtual reality environments, it’s difficult to simulate experiences, such as testing for motion sickness, without actually playing with the real devices. This is especially challenging with the newer mobile platforms, such as Google Cardboard, which incorporate a viewing device that transforms the mobile phone into a VR headset. That experience simply can’t be simulated accurately on a PC, a common testing strategy for other mobile apps. Companies testing mobile VR apps may need to invest in the purchase of dozens or hundreds of different mobile devices to gain an accurate picture of how the application performs for most users.

    Hyperconnected Apps

    Web-based and consumer apps comprise the bulk of hyperconnected apps, characterized by having deep, often real-time integration with other applications. Instagram and Facebook, Gmail and Google Calendar, Salesforce and their wealth of apps—these are just a few examples of hyperconnectivity. The advent of REST APIs has facilitated far easier and cheaper integration so any company can build a platform of third-party software in ways they couldn’t have imagined five or 10 years ago. Similarly, with the explosion of the app economy, including open source software, it doesn’t make good economic sense to write 100 percent of code in your application any longer. Product value is increasing the result of how developers can patch together different technologies in innovative ways.

    Testing considerations: Given the importance of APIs in hyperconnected applications, a primary testing consideration will be ensuring the performance validity of those connection points. It’s imperative for both developers and testers to be aware of API update schedules and to test those updates immediately for any negative impact on the application. Beyond API testing, load and performance testing are also important considerations for hyperconnected applications, which typically involve heavy and/or unpredictable transaction processing. Overall, these types of testing are much more technical in nature than manual UI testing and will require testers to expand their skill set to optimize value.

    Internet of Things (IoT)

    The interest in (and hype) for the IoT sector rivals the largest IT trends, such as big data. Initial applications have focused on connecting devices, such as home lighting and thermostat systems, which use sensors to stream data to smartphones. Over time, we will see more device-to-device connections such as a thermostat collecting data from a smoke detector to ensure that the house isn’t on fire if the temperature rises dramatically. Products such as Amazon Echo and Google Nest might be at the heart of many new consumer-based IoT applications, but there is also ample potential in sectors such as utilities, health care, industrial equipment and government (a.k.a. smart city initiatives).

    Testing considerations: As IoT applications mature, connectivity complexity will grow. IoT applications can involve many components: the hardware of the smart device or product and the software that controls it, along with the user applications that interact with the product. When troubleshooting an issue or vulnerability, a tester must investigate the applications that collect, analyze and display the IoT data, the various network connections across which that data travels, the IoT appliance’s hardware and its software. Hardware testing must incorporate typical scenarios such as what happens to the user experience when a device starts to run low on battery or goes offline. Testers will be most valuable when they have experience in both hardware and software testing; alternatively, an organization can have two teams of testers but they must be in very close contact. IoT apps should also be configured to collect and deliver more alerts and logging data than mobile apps, due to the higher number of components involved. This will make it easier to prevent, troubleshoot and fix problems when they arise.

    About the author/Kevin Dunne

    Kevin Dunne picKevin Dunne is VP of Strategy and Business Development at QASymphony.

     

  • SmartBear Presents on DevOps Testing at Better Software West

    SOMERVILLE, Mass.–(Business Wire)–Nikhil Kaul, Product Marketing Manager, Testing Products at SmartBear Software, the leader in software quality tools for the connected world, is presenting, “Where’s Test in DevOps?” at Better Software West, taking place on Sunday, June 5 – Friday, June 10, 2016 in Las Vegas. SmartBear is also exhibiting its industry leading test automation solutions and dev collaboration tools including, TestComplete, TestLeft, LoadComplete and Collaborator in booth #13.

    WHAT

    &WHEN:

    		 SmartBear at Better Software West:

    • “Where’s Test in DevOps?” Presented by Nikhil Kaul on Thursday, June 9, 2016, 3:00 p.m. PDT

    • SmartBear’s Automated Test Solutions and Dev Collaboration Tools Exhibited in Booth #13 on Wednesday, June 8 – Thursday, June 9, 2016

    WHERE:

    		 Caesars Palace, 3570 Las Vegas Blvd., Las Vegas, Nevada

    As organizations try to meet faster delivery schedules and improve collaboration between development and operations, DevOps has become a hot topic. And with Apple, Google and Microsoft making it easier for Dev and Ops to build and deploy cross-platform apps, there is more to test in less time. The question then arises, where does testing fit in such a DevOps strategy? Nikhil Kaul gives an overview of recent trends in DevOps, specifically changes that have taken place in the testing industry over the past year. He explores why testing is becoming more important than ever and how testing can facilitate the transformation to DevOps. Finally, Nikhil touches upon ways to build a scalable testing framework and important reusability practices that can be applied—from development and test to operations—to reduce post-deployment debugging time.

    Prior to SmartBear, Nikhil served Digité, a leading provider of Application Lifecycle Management solutions, in various roles including Senior Software Executive. There, he gained insights into software development and testing market. Nikhil received a master’s degree in business administration from Georgetown University. He is very involved in discussions taking place in development and testing communities. His most recent thoughts can be found in Forbes and CIO.com. Follow him on Twitter @kaulnikhil or read his blog.

    SmartBear has a suite of testing automation tools, including automated testing platform,TestComplete, LoadComplete, the industry leading performance and load testing tool for Web and mobile assets, its newest developer focused test automation tool, TestLeft and test management tool, QAComplete. For more information, visit:https://smartbear.com/products/testing/.

    Collaborator is the only tool available that allows teams to peer review code, user stories and test plans in a transparent, collaborative framework – instantly keeping the entire team up to speed on changes made to the code. For more information, including a free trial of Collaborator, visit: http://smartbear.com/product/collaborator/overview/. On Twitter, follow @CollaboratorSB.

    The Better Software West Conference keeps industry professionals current on the entire development lifecycle. SmartBear is a Silver Sponsor of the event. For more information, visit: https://bscwest.techwell.com/.

    About SmartBear Software

    As the leader in software quality tools for the connected world, SmartBear supports more than four million software professionals and over 25,000 organizations in 194 countries that use its products to build and deliver the world’s greatest applications. With today’s applications deploying on mobile, Web, desktop, Internet of Things (IoT) or even embedded computing platforms, the connected nature of these applications through public and private APIs presents a unique set of challenges for developers, testers and operations teams. SmartBear’s software quality tools assist with code review, functional and load testing, API readiness as well as performance monitoring of these modern applications. For more information, visit: http://smartbear.com, or for the SmartBear community, go to: Facebook, Twitter, LinkedIn or Google+.

    All trademarks recognized.

    BroadPR
    Tracy Wemett, +1-617-868-5031
    tracy@broadpr.com

  • Embedding Predictive Analytics

    Embedding Predictive Analytics

    For an enterprise with a deep DevOps maturity model, the initial “wow” factor of automation has worn off. At this point, the sexier concepts involve proving continuous improvement. As an example, our first baseline of metrics might have taken the pre-DevOps timings of a typical build and deployment of a given application. We can measure this time against our post-implementation of DevOps automation and show a significant savings in time. This is opportunity time, meaning an employee must be ready to turn around the decision cycle faster to take advantage of the time saved (not be playing solitaire, for example). But once DevOps automation is old hat, the difference between our initial post-DevOps typical build and deployment time and our current one still using DevOps may not be all that significant. But is there more hidden there than meets the eye?

    Establishing Context & Baselines

    At this point context may be a more meaningful measurement. As an example, if my first version of DevOps automation for a deployment contained 35 steps and my current one contains 160 steps it would stand to reason I am “doing more” now in my deployments than I used to do. These additional steps may involve executing more tests than I originally thought to do. They may be interacting with monitoring systems (turning them on and off as appropriate), which reduces “noise” in the alerts my monitoring teams have to respond to. In short, over time, my deployment processes may become “longer” because I am increasing the capabilities or the sophistication of what I do during these deployments. While from a pure timing perspective this may make them longer, it also makes them more powerful and, so, is worth measuring. To know the delta, however, I must have retained older versions of automation to compare against, not just throw away the old every time I update with new steps.

    So let us assume the context of my automation is now fairly stable. I do the same things, more or less, the same way every time I deploy my application. At this point, measuring the functionality of what I do becomes more meaningful. As an example, my deployment automation may complete four main functional tasks to complete, I deploy the code (the app), I deploy the dataset, I deploy the config files and, finally, I run a series of tests on the above. While I may not ever incorporate too many new kinds of things to do in this automation, understanding how long each function takes to complete is relevant to proving continuous improvement. For instance, if my code takes roughly the same amount of time each deployment, as does my config files, but all of the sudden my dataset begins to grow in each deployment (from normal usage), my overall deployments may be getting slower. This is not due to hardware inefficiency or network inefficiency; it is simply due to a larger dataset that keeps growing.

    If on the other hand, it is my test set that is getting progressively longer to execute, that is an indicator of test script inefficiency (given same code, same config and same dataset). If the number of tests being executed has not increased, but the length of time it takes to run them has, there likely is inefficiency in “how” the testing is being constructed or executed. Understanding the context for what is going on in a deployment or similar kinds of automation must occur before introducing predictive analytics can occur.

    Historical Trends Extended

    Predictively evaluating automation is nothing more than taking the data from a historical perspective and extending the trend lines into the immediate future. If for example, my deployment automation is getting four seconds longer each month for the last 12 months, I can reasonably infer the trend will continue, and I can predict the length of time it takes to execute a deployment in the coming months. Understanding “why” this occurs requires a contextual investigation, but predicting its occurrence does not. This may seem inconsequential as we talk about the length of deployment processes of a given app, particularly if my portfolio of apps is small. But if the trend holds true for a portfolio of thousands of applications, the impacts become more consequential. Or if the phenomenon is directly connected to a particular cloud provider and not seen elsewhere, again, the consequences become meaningful.

    Where having “ground-up” predictive analytics (build and/or deployment automation) becomes much more useful is in the construction of release events. Often, release managers are asked to supply an estimated time duration for a given event to factor into the construction of a new release. If a vendor were to integrate more predictive analytical tooling into the release orchestration (RO) toolsets, “random” or “best guess” timing estimates could be replaced by extending historical data into the timing estimation fields. The RO tooling could provide a default timing estimation based on examining historical trends for a period defined by the user. Estimation defaults could always be replaced by a user (assuming discreet knowledge of other impacting events), but at least the new overall lengths of releases then could become significantly more accurate.

    This has special significance for manual tasks. Let’s say our database admin team traditionally takes 40 minutes to perform a given release event update, but now is running significantly longer than that. Having the RO software aware of the historical trends could alert the release manager, as well as the database team manager, during the event to check out why this is occurring. For manual tasks in particular it may be hard for anyone besides a predictive analytics module to determine what timings “should” be, as compared with what they are in a given release.

    Impacts to the Bottom Line 

    The real questions are, Why should a customer care about this kind of feature set? Why should we be looking at the vendors of our tooling to incorporate forward-thinking architectures such as predictive analytics into the tooling they sell us? Because the efficiency it can bring to our DevOps set of services lowers the costs of administration and increases the speed at which we can deliver innovation in general. Over time, trending combined with context can statistically prove continuous improvement of our DevOps services, which is the promise we made to the executive teams that sponsored and made the investment in DevOps to begin with.

    Metrics analysis can occur (usually with a dedicated investment on our part) without a predictive view of the impacts. But this changes the perspective of our analysis to hindsight thinking instead of forward thinking. In essence, we then are simply “reacting” to what has occurred instead of tailoring our environment to alter the picture of future predicted events to be better than the software predicts it to be. We start tweaking the variables to instantly impact future predictions, instead of only examining them in a post mortem.

    This perspective is yet another cultural catalyst for change your business needs. When I talk about changing how an organization “thinks” about innovation, this is one of the key objectives. We take an enterprise from knowing little about the costs of innovation, to having an extensive knowledge, to having a forward view of what is—or what should be, possible. All of this is possible despite the loss of waterfall controls and in the midst of an agile construction model. All it takes is the integration of forward-thinking architectures into the products we select. If our vendor does not offer them, we begin to enlighten our vendors on the value they would bring to us in cost reduction and to them in market capture. If you need help in these areas, feel free to contact me, I am looking at the moment 🙂 .

    To continue the conversation, feel free to contact me.