devopsy.kinsta.cloud

Tag: testing

  • As Business Speeds Up, Testers Must Adapt

    As Business Speeds Up, Testers Must Adapt

    It should be no surprise to anyone working in software today that delivery speed is the name of the game. Irrespective of business sectors or B2B/B2C focus, global competition in application development and delivery has flooded the market. Due to the increased number of options in the marketplace, users expect an excellent experience from your application or site—and if you cannot meet expectations, they’ll find it elsewhere.

    A company’s ability to be highly responsive to (if not proactive about) customer’s needs is also valuable for adoption and branding. Established companies understand that delivering high quality in their products and services is core to brand value.

    This shift isn’t always easy for testers. Moving to a faster pace of work can be unsettling when your job is all about assessing the perceived quality of the software. Testers like to be cautious and check everything twice. The current work environment has made testers more germane than ever in software and IT organizations.

    Testers help build confidence in the quality of an application, and this perception of quality is what typically drives user adoption. If two applications have more or less the same functionality but one is faster, more reliable and has a better UI and design, you can guess which one will be more popular.

    Setting the Foundation

    To thrive in this new world, testers must learn to prioritize and focus on the most important needs of the team’s delivery goals versus testing every last mouse click. Testers will need to quickly incorporate feedback from multiple places, and work well with stakeholders across the development organization and the business. Efficiency is imperative, and that means increasing the use of tools whenever possible.  The following framework to develop a world-class testing organization when your company is moving faster all the time can make the transition a little easier.

    1. Understand business goals. While this may sound obvious, testers haven’t always had to think about the bigger picture. With more competition from everywhere, everyone on the development team should have a solid understanding of customer needs, business goals such as to drive new revenue or grow customer loyalty, and how the competition stacks up for the solution in development.
    2. Define a testing strategy. This exercise begins with understanding the business goals as they relate to the product or project and, at a more granular level, the goals for each sprint. The testing strategy should include a description of the overall project, priorities for testing mapped out into phases, timelines, testing approach, tools and methods to use and how progress will be measured.
    3. Develop a risk management plan. What will happen if those business goals are not achieved? For instance, the business stakeholder might say the project must go live in four months with new functionality. But is this an arbitrary goal or one that is embedded in an actual customer commitment? What will be the financial impact if the deadline slips? Which enhancements are critical and which are nice to have from the user perspective? Once those particulars are documented (or discussed), the testing team can create the appropriate plan, prioritize and determine what resources are needed to execute it.
    4. Resources. Testing teams with mandates to move faster will need support in the form of new tools, training and even new hires. Software that testers (as well as others) use can include DevOps and CI/CD tools, ALM tools, reporting and analytics, UI and API testing tools, systems for recording and logging manual testing, tools for usability feedback and more. This can get overwhelming for startups and other companies that don’t have the budget to purchase new software. Fortunately, there are many credible open source tools that can fit the bill. This reality is also why development teams must understand well enough what customers really want so they can prioritize on which features must be perfect and which areas of the application could meet a lower bar of quality.

    A Word of Caution

    Testers are well-equipped to make all the changes required for the faster pace of modern software development. This shift has been underway for a few years now, and many testers are embracing the change and the elevation of their role on engineering and IT teams. However, as testing relies on more automated tools, there are a few things to consider:

    1. Avoid tools mania. Testers, like many technical people, love to try out the newest toys, but it’s important to justify a tool’s adoption before acquiring it. With too many tools, there’s just too much noise and complexity. That will counteract any speed and efficiency gains that automation supplies.
    2. Automation tools do not replace testing. It’s not necessary nor helpful to automate every testing task. Testers should view automation tools as enablers, not a replacement for the function. Automation is wonderful for reducing effort and giving testers time to work on value-added activities that can boost the overall user experience and usefulness of an application. Be sure to baseline test objectives before recommending an automation solution.
    3. Institute process around automation. Ensure you have defined a proper process in sharing artifacts that are produced by your automation tools. Artifacts such as scripts must be versioned with proper comments and stored in a central repository instead of on individuals’ machines. Defining and enforcing good development practices such as check-in/check-out process, scripting style guides and object organization with automation will ensure a higher return on effort in the long run.

    In a few years, this point may be moot. But there are still many testers coming into new roles from the waterfall mindset who may struggle in the transition.

    Testers once lived simply in the silo of sluggish procedures. They knew exactly when they would be called upon for action and how they would complete a task. Now, though, testers must put on their business assurance hats and work side by side with developers, operations managers and others early in the process. They will have to learn new ways to test, such as with behavior-driven development and test-driven development. They need to get more technical, given the push toward test automation.

    What hasn’t changed though is the tester’s quest for quality in software. Great testers will add the most value when they can be proactive about alerting others to brewing problems before they present themselves to the user.

    — Mush Honda

  • What Andreessen’s ‘Software’ Essay Means to Development Today

    What Andreessen’s ‘Software’ Essay Means to Development Today

    It’s been nearly seven years since Marc Andreessen’s essay, “Why Software is Eating the World,” brought to light software’s role in determining the next wave of winners and losers.

    Today, his essay is still relevant. The past decade saw the consumerization of IT in business. It also witnessed the shift in software consumption from desktops to the web and to mobile apps today, reflecting a mobile revolution. Software continues to eat the world, having moved to a “mobile only” development model.

    This revolution is defined by disruptive companies including Airbnb, Uber and others that run with more leanness and agility compared to their 20th century counterparts. These companies have few physical assets, relying instead on software.

    Since Andreessen’s essay, the software development life cycle has expanded and evolved—from waterfall to agile processes, from testing against just a handful to thousands of mobile devices and from serving a regional to a global one user base, to name a few examples. Demands for speed and quality control have been key drivers, and as a result, the life cycle moves faster than ever. A record number of developers are writing a record number of programs. This expands each and every day as businesses small and large, in markets old and new join, the mad dash to create more software, better, faster and cheaper.

    Creating Software, Fixing Mistakes

    Indeed, the software development curve is evolving at breakneck pace. But while the cost of creation decreases, the cost to correct a mistake increases. How can developers and their organizations stay ahead of the curve? Here are three principles we’ve observed from studying more than 25,000 customers in the last seven years:

    Implement Modern Development Techniques Such As CI/CD

    Continuous integration (CI) and continuous delivery (CD) are two modern development practices designed to help teams release quality code faster and continuously. CI/CD has gained popularity over the past few years, and for good reason. When implemented effectively, CI/CD shortens delivery times, improves overall quality and enables teams to adapt quickly.

    Key to a successful CI/CD practice and ensuring quality software development is the practice of testing throughout the development process. The cost of fixing already deployed broken software is high, and it’s crucial to ensure your software works properly on all platforms and devices to minimize customer abandonment. This is especially true when you consider that the cost of regaining a customer is four times higher than winning them in the first place. Getting software right the first time through a culture of continuous testing is critical to success.

    What does ideal testing look like? It is automated, runs fast and includes good coverage. As the already broad ecosystem of operating systems, devices and platforms expands, this last point is critical. What good is your app if it doesn’t run (and, by extension, monetize) properly when it gains popularity in some part of the world with users on a different platform from the one you tested on? Developers need to ensure their software works on as many different combinations of browsers, devices and operating systems as possible, to help their business reach the maximum number of users.

    Introduce ‘Self-Driving Software’ Practices, Technology into Your Development Cycle

    When everyone is running so fast, it’s critical to ensure the software you’re writing actually works, won’t break your infrastructure and will actually be used by consumers. The “garbage in, garbage out” model is not an option, especially given the expense of creating software today. While the barriers to create software have lowered over time, the bar has risen in terms of creating something really remarkable.

    The good news is that we are at the cusp of attaining the holy grail of software development. I have interviewed multiple engineering and quality leaders at many of our customers who are pushing the limits on ultra-efficient DevOps. Through a streamlined process (which, frankly, is nothing short of beautiful), they are able to create a single, seamless flow between development to testing to operations to deployment and monitoring—without compromising on speed or quality. The key is to prioritize speed over cost and quality and agility over all else. If an organization isn’t following this principle, they risk being left behind. In the “Stack Overflow developer survey 2017,” more than 65 percent of developers indicated that they follow some kind of agile development methodology at work.

    The end result is deploying software almost immediately as soon as it’s written, for release as quickly as possible. Companies strive to achieve this scenario, and the highest performing organizations have already reached this development dream in some manner or another. Attaining CI/CD and automating all the tasks involved in creating a test system in real time is one of the arenas in which the next big software battle will be fought. The quest for the holy grail of software development is on.

    Support (and Attract) Your Engineering Talent by Providing Flexibility

    In  the mad dash to develop amazing software, engineering has become among the hardest roles for companies to recruit for.

    Organizations need to be nimble enough to attract top engineering talent. Part of being attractive to potential hires is understanding that software can no longer be monolithic. More than ever, the talent pool expects choice in the languages, platforms and tools they can use to develop and work most efficiently. In the Stack Overflow developer survey, participants valued the languages, frameworks and other technologies they’d be working with as a critical (4/5) decision factor while assessing potential jobs. This doesn’t mean providing a “free for all” for developers to use everything and anything they want—rather, it’s all about adopting flexibility. The companies that are most attractive to the best engineers follow this principle.

    Conclusion

    Today, we’ve moved from software eating the world to the world eating software. Developers simply have no control or knowledge over where their website or app is going to be downloaded or accessed from. Unless it’s an internal app behind a firewall, any software developed today by definition is consumed by the global community.

    Even if you test for all scenarios and devices, the future will always bring new devices, browser versions and operating systems. Your customers may use these instead of what you initially tested for. One option is to test for emerging devices in an in-house lab; however, that means committing to resources, time and qualified engineers to maintain it. In most cases, this approach is simply unsustainable.

    Companies must focus on ensuring their software works both with what’s out there today and with whatever browsers, devices and platforms will be out tomorrow. This is a major challenge, requiring considerable investment. Companies need to decide whether they are in the business of testing software themselves, or leverage solutions that are.

    Well before Andreessen’s seminal article, Geoffrey Moore wrote “Dealing with Darwin.” Here he he expounded upon the concept of core vs. context. Organizations will be well-served in deciding (quickly) whether building an in-house testing infrastructure is core or context to their business. The alternative is falling behind—or worse, being eaten up by the competition.

    — Ritesh Arora

  • Using Inspec to Validate Infrastructure

    Using Inspec to Validate Infrastructure

    So, here we were, at one of the bigger financial corporations in New Zealand, working on an elaborate infrastructure built on Docker containers. It had configurations, SSL certificates, DNS entries and all the other condiments required to spice up a tasty infrastructure curry. Though we were able to stand up our complete infrastructure environments from scratch using automated configuration management tools, we weren’t really sure if it was indeed working correctly, or rather the way it was supposed to. The question staring at us was, Is there a way we can declare the desired state of our infrastructure and then validate our infrastructure against it?

    Inspec to the Rescue

    The open source tool Inspec has been used with considerable success over the last few years to achieve security compliance for infrastructure. The syntax/constructs happens to be unbelievably simple. For instance, let’s consider the code snippet below. It doesn’t really require a lot of programming knowledge to make sense of it. Isn’t it?

    This made us wonder, what if we use Inspec to validate our infrastructure setup? What if we map our infrastructure configurations in an easy-to-read declarative syntax? What if this actually solves our problem?

    In line with the best practices of DevOps, we started with a simple experiment; an experiment that could solve a tangible problem. We had a Docker host which was supposed to have a particular docker container running inside it, but it used to fail quite often, for some reason. We decided to write a test in Inspec to check whether this container was running on this host:

    … and tried executing the test.

    This was just what we wanted!

    Excited with this success, we started writing a battery of tests that validated each and every component of our infrastructure. Within a single day, we churned out 192 tests. The number of tests wasn’t as important as the scope and breadth of the test coverage. Once the tests were there, we plugged them on to our infrastructure pipeline (which was built using Jenkins) to ensure the components align with the desired state. We called them ‘component-readiness’ tests.

    So far so good.

    Now that our components match the desired state, are we still confident that our platform would work the way we intended? Perhaps we need some “platform readiness tests” as well. Can we use Inspec for that, too? The answer: Yes!

    Now that the platform readiness tests were ready, we extended our infrastructure pipeline to incorporate them as well.

    Setting our component and platform readiness tests using Inspec had one more fantastic offering: live documentation for infrastructure. Inspec allows the audit results to be displayed in a variety of report formats, including JUnit, JSON, HTML and others. We used the JUnit report format and displayed them using the Jenkins JUnit Plugin. With this, we were able to have human-readable JUnit reports that doubled as live documentation for our infrastructure.

    Needless to say, this infrastructure testing with Inspec provided that (much needed) confidence on our setup and readiness. Our pipeline is designed in such a way that whenever any infrastructure component changes in our ecosystem, the pipeline is triggered which reconfigures and (if required) rebuilds the ecosystem. These Inspec tests are triggered with this infrastructure pipeline and validates the newly configured infrastructure setup. Thanks to this, we are never in a situation in which the client application running on this infrastructure complains or aborts because of incorrect configurations. Moreover, as described above, the living documentation helps document the expected configuration of the environment.

    So far, the Inspec implementation I have referred to is using the freely available Inspec tool and the Jenkins platform. Inspec can also be combined with Chef Compliance to produce more descriptive reports and charts. Chef Compliance also comes with out-of-the-box CIS profiles that can be used to jump start an Inspec implementation for security compliance and audits.

    Inspec is an easy tool to master with simple and intuitive constructs and one could get up and working quite easily. However, this simplicity has a downside. It’s very easy to speed ahead and gloss over the documentation while missing out on the best practices and tips/techniques to implement the solution in a better way. One way to master this is to have a crack at the newly released Auditing with Inspec certification, which provides the impetus to go through the documentation in detail while preparing for the certification. And, you also get to show off a shiny badge for your efforts!

    In Summary

    Infrastructure validation is quite an important step in any DevOps implementation. Apart from being very useful for security compliance testing, Inspec can be used to validate infrastructure using a simple and easy-to-understand declarative syntax. Moreover, it has a small footprint, a vibrant community looking after it and up-to-date and clear documentation. It definitely is a tool with the potential to provide quick and effective results.

    — Mrinal Mukherjee

  • Beware the Blind Spot in Performance Testing for DevOps

    Beware the Blind Spot in Performance Testing for DevOps

    According to Forrester, 64 percent of businesses are dissatisfied with the time it takes to release new features for customers. Brand loyalty has given way to customer loyalty; to drive it, organizations must focus on customers and their experiences using their products and services.  As a result, organizations of all sizes need to make customer experience their highest priority.

    To many business leaders, the speed of rolling out new updates is the most important part of satisfying customer expectations, maintaining customer loyalty and staying ahead of the competition. At the same time, user expectations for fast and reliable digital experiences across platforms continue to increase. As a result, development teams are pushing out service updates faster than ever before while trying to minimize any degradation of the experience as they move at a quickened pace.

    But with heightened and persistent focus on rollout speed, it is easy to forget that the overall user digital experience is critical to maintaining customer loyalty. Businesses must integrate their focus on performance testing with their DevOps workflows to make sure speedy product rollouts don’t come at the expense of customer experience. So how should businesses ensure strong delivery processes to both improve customer experiences and drive efficiency in the application development and delivery pipeline?

    Performance Starts with Constant Measurement in Unit Testing

    DevOps, as does Agile, breaks down testing into smaller chunks. Each new rollout has an iterative impact on the application or website. Whether good or bad, the results stack up. Customers may demand certain new features, but user experience will suffer if each update that meets that demand slows down the application more and more over time.

    Testing these units is very important and setting performance SLAs per unit per build is the first step to a performance culture. Organizations with a mature performance culture will turn back code if it doesn’t meet their performance SLA, keeping it from ever reaching QA.

    This is Not an Either/Or — It is an And 

    DevOps teams are very good at rolling out rapid iterations of products and features, but without careful consideration, the pace of those updates can build momentum for digital experience failure. There are false positives in DevOps testing. With unit tests constantly passing, it’s easy to lose sight of the bigger picture. It’s not the fault of the DevOps engineers or QA; they are working within an established process that is designed to sniff out errors on a per-build basis. But what happens when you put it all together? Organizations start to forget about end-to-end testing and can see failures when the application is stressed by heavy demand, when it really matters during critical business events like Black Friday, live sporting/news events, product launches, or changes in financial markets. Teams implementing DevOps-style testing must remember to test all the “units” working together end to end. This needs to be part of the DevOps workflow but is often being forgotten.

    What You Can’t See Can Hurt You

    Testing for performance in the development lab can help mitigate some of these performance issues before they get out in the wild. But it’s overly optimistic to think that you’ll catch everything. Delivery teams need to constantly maintain visibility in the real-world use of their applications with real-time real user monitoring to alert them of any degradation in the experience during or just following a release. A quick response can help mitigate a larger detrimental impact to the business, and the fidelity of real-user data gives assurance there aren’t any blind spots so you can fail fast, and shorten time to resolution. The web is a wild place, and users have sneaky ways of finding their way into and through your app that you may not foresee when building your test plan. You can use this real user data that you’ve collected while monitoring to create more realistic test plans that will truly replicate real-world browsing behavior.

    This data can also inform the development team about the impact of design decisions on performance in each new update. They can prioritize which parts of the site or app to fix based on what would be most impactful to the business and weigh that against plans to build out new features aimed to enhance functionality. The key factor is an understanding of the business impact of performance and using that to guide and prioritize development resources to achieve the best possible results.

    Adopting a methodology to monitor, optimize and validate ongoing enhancements to your application can help with the continuous improvement of user experiences and give your development efforts a customer focus. DevOps teams have a responsibility to the user experience and should consider how it correlates to business impact. Faster time to market is important, but as important is making sure the customer experience continuously improves as well. Uniting cloud delivery with DevOps workflows and keeping performance in mind will put your business on the path to ensuring both faster rollouts and superior user experiences.

    — Anthony Larkin

  • DevOps Chat: Continuous Testing with Sandeep Johri, CEO, Tricentis

    DevOps Chat: Continuous Testing with Sandeep Johri, CEO, Tricentis

    In this DevOps Chat we take a dive into the continuous testing market, which is one of the most dynamic in the DevOps space. Our guest is Sandeep Johri, CEO of Tricentis, a leader in the space.

    Sandeep gives us his own personal background, as well as the history of Tricentis. The company is not an overnight sensation in the continuous testing space. In fact, it has been perfecting its offerings for many years. Tricentis literally helped usher in the continuous testing age, making continuous testing part of the CI/CD pipeline.

    This chat is a great way to learn about what some of the important points for continuous testing are. Have a listen. As usual, the streaming audio is immediately below, followed by the transcript of our conversation to follow along. Enjoy.

    Audio

    Transcript

    [su_pullquote align=”right”][su_custom_gallery source=”media: 120253″ link=”post” target=”blank” width=”230″ height=”230″ title=”never”]Sandeep Johri
    As the CEO of Tricentis, Sandeep brings over 25 years of enterprise software experience. He manages all business-related functions including business strategy, products, services, sales, marketing, and operations to achieve business profit and sales objectives. Sandeep most recently served as the COO of Appcelerator, the leading open source mobile application development platform. He was Vice President of Strategy and Industry Solutions at HP, where he also served as Vice President of Corporate Development. There, he was instrumental in developing and implementing the growth strategy that resulted in the software business growing 2X market over the last four years. Sandeep was instrumental in the formation of two startups, Determina (VMware) and Bluelane (VMware). He was also the CEO and founder of Oblix (Oracle), and co-founded eBoodle (EW Scripps), one of the first ecommerce comparison shopping services. Sandeep has an MBA from Stanford University, a Master’s degree in Industrial Engineering from Wayne State University, Detroit, and a Bachelor’s degree in Mechanical Engineering from Pune University, India. Sandeep is involved as an advisor and investor in various startups in Silicon Valley and India.[/su_pullquote]    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Got a really great chat lined up for you all today. I’d like to introduce you to Sandeep Johri. He is CEO of Tricentis. Sandeep, welcome.

    Sandeep Johri: Thank you, Alan.

    Shimel: First of all, Sandeep, I just wanted to make sure I’ve got your – we pronounced your name correctly.

    Johri: Yes, you did, actually. You got it absolutely right.

    Shimel: Good. All right. It’s always a good start. So, Sandeep, let’s start foundational here. Tricentis is a company that, really, had a very, very strong presence over in EMEA, I think, and has most recently really begun to exert themselves in the North American market. But some of our listeners may not be familiar with Tricentis at all. So why don’t we start with that, if you can give us a sort of a – not an elevator pitch, but a little background on Tricentis?

    Johri: Sure. So Tricentis was founded about eight, nine years ago. It was founded by Wolfgang Platz, who was actually a tester and working on complex enterprise applications for testing and was frustrated with all the tools out there. So, like any good entrepreneur, he decided to write his own tool. And the tool initially was focused primarily on automation. And it sold – they were – Wolfgang was based out of Vienna, Austria, and they sold it locally into the German-speaking market, so Austria, Germany, and Switzerland.

    Shimel: Mm-hmm.

    Johri: And they had a nice, little business going and they had a great product. I ran into them about 4 years ago, in 2013. I found them and felt that this could be a really interesting offering because what the market needed was automation, especially as people were going to Agile. Given my background with Mercury at HP, I knew that there was not much automation, but, if you go Agile, you have to have automation. It becomes a critical component – automation on the testing side becomes critical – and that’s why I joined the company. You’re right; they were initially focused primarily on Europe. And, remember, this is pre-coining of the term “DevOps,” but the need for automation was always there, especially as you kind of shift towards Agile on the dev side.

    So now, you know, cycle ahead four years and we are now a global company. We did as much business in US as in Europe, so we are now really very well balanced. And we also have now created a presence, over the last few years, in Asia-Pac, so Australia, New Zealand, and Singapore, so we’re truly kinda global, with a typical balance of about equal between US and EMEA, and then another additional APAC component to it. And, obviously, once DevOps became the kinda mantra in the industry, we had a really nice fit and people started talking about continuous testing. Well, with automation, what you really get is continuous testing, so that’s why we are all about continuous testing. But the core engine is an automation – is a test-automation engine, so for the enterprise.

    Shimel: Absolutely. So I wanna get into the whole automation and testing market, but you mentioned a few things that I, if you don’t mind, I’d like to dive into. I don’t mean to embarrass you, Sandeep, but, when you get to my age – and you’re a contemporary – you know, we like to reminisce. So you mentioned Mercury and your experience there. I mean, for those who aren’t looking at your CV or the management team details up on the Tricentis site, why don’t you give a little background on yourself to our audience so they get a flavor for this?

    Johri: Sure. Sure, Alan. I’ll try to keep it short. I’ve been in Silicon Valley for almost 30 years and I’ve been entrepreneur. I started a few companies. One was Oblix, which was a leader in the identity space. And then a couple of other companies, Blue Lane and Determina, which were then acquired by VMware. I joined HP after that because they were looking to build out their IT-management portfolio, and, over 3 or 4 years, from 2005 to 2008, we acquired about 13, 14 companies, including Mercury Interactive, which was the largest, but we also acquired Opsware for data-center automation; we acquired Peregrine for asset and service management. And that whole portfolio became HP ALM, so it was – that business went from about $600 million to about $3.5 billion.

    Shimel: Wow.

    Johri: And then everyone knows HP was having some challenges and were increasingly focused less on the software business, which they eventually sold last year. But I felt the need – I felt the opportunity was there for a next-generation tool or next-generation testing platform that better suits the Agile automation environment, which we now call more “DevOps” because it takes the whole cycles and kinda creates a continuous-delivery chain, if you may.

    Shimel: Absolutely. So, Sandeep, that’s sort of the quintessential Silicon Valley entrepreneur story, where you’ve done everything from start-ups that had some very successful exits. As I mentioned offline, I remember Oblix and I remember Blue Lane almost like it was yesterday. And then, of course, you had this big company, HP kind of experience on the resume, and I would imagine, knowing what I know about the Insight Venture team, who I believe is an investor in Tricentis, I would imagine you would have been a very, very attractive candidate, when they were looking for a – the team was looking for a CEO to take Tricentis to the next level, so right person at the right time there. Fantastic story.

    Johri: Yeah, I joined before Insight came in. Yeah, I joined four years ago – Insight came in last year – because I thought that they were really promising, and we spent the first couple of years kind of building out the product to be more – you know, filling out the product. We added service virtualization. We added test data in it. You know, we’ve added a whole bunch of capabilities and we were beginning to get recognized by Gartner and the like as a leader and we also started having some really large enterprise customers in the US. That’s when Insight also had a similar thesis, that, “Hey, HP seems to be walking away from this market, but there’s a need – there’s an opportunity there,” and they found us. And that’s why they invested $165 million late last year, so.

    Shimel: Wow. That’s quite an investment. And so sort of I had the horse before the cart, but congratulations. Right? This is not an easy thing with that kind of investment, but also comes when there’s that kind of opportunity. Let’s turn now, if we can, Sandeep, and talk a little bit about the testing market. You know, testing – before DevOps and Agile, the whole QA testing thing was really, really super geeky and I don’t know how interesting to people. Back in my day, at StillSecure, the hardest thing for us in testing was just building up a test lab that could handle the scale we wanted to test at. Right?

    Johri: Right. Right. Right.

    Shimel: And testing software and so forth was relatively a bunch of open-source tools that the development team did as part of the release cycle in the old Waterfall. But, you know, with Agile and automation and, look, there’s Agile with automation and then there’s automation in and of itself, throughout the software life cycle.

    Johri: Uh-huh.

    Shimel: It’s become imperative, right? And, all of a sudden, now –

    Johri: Right.

    Shimel: – yeah – testing has taken on a super role, right? I mean, it’s really become kind of the – you know, when we look at the continuous integration, continuous delivery pipeline, right, that testing, man, it sticks out. And it’s like swamping everything else, if you will, right? It’s swallowing it.

    Johri: Right.

    Shimel: And so tremendous opportunity. We’ve seen a number of companies try to enter this fray, many of them based on open source, whether they’re doing Selenium or something like that, but there’s different aspects of testing. Can you share with our audience, Sandeep, what type of testing is it that Tricentis is doing?

    Johri: Right. And so, Alan, I think you cover a lot of the history and perhaps I can comment on that because that’s really interesting. So, you know, testing, for an enterprise, has always been critical, but you’re right. It was kind of a “Oh, God, I have to do testing and it’s a lot of manual process,” so people mostly outsourced testing. And, when you were in Waterfall mode, that was okay because you did a bunch of development and then you threw it over the wall. In most cases, at an enterprise, it went offshore and people did it manually. But it has always been a huge spend. Even today, there’s a $30-billion spend on testing services.

    Now the challenge is that the tools that were out there, both open-source and commercial tools, were primarily script-based tools, so it was really difficult to automate. And, because you had outsourced, you kinda got the labor arbitrage and people continued to do it manually. And, even today, 80 percent of testing, enterprise testing, is done manually, which is kinda crazy, right? ‘Cause here we’re getting ready with autonomous cars on the one hand, but, on the other hand, we’re doing software testing which is – software’s so fundamental to every enterprise – we’re doing all the testing manually. Well, now – and that was okay in the Waterfall environment.

    As you go to Agile on the dev side, you start doing frequent builds. And, as you start doing frequent builds, if your testing cycle is still six to 12 weeks, you’ll fall right back into a Waterfall mode, even though your dev might be Agile. So testing is becoming – the need for automated testing is becoming critical because, otherwise, it becomes the bottleneck in your delivery chain. And what we do or our view of testing is that testing should be at the same cadence as development, so, if you’re doing daily builds, you should be testing on a daily basis.

    Now you mentioned developers. Developers do unit testing – they do subsystem-level testing as they build – but, for a large enterprise, which has a complex environment that has a mix of technologies, just doing subsystem-level testing doesn’t work. It is necessary but not sufficient. And, therefore, what you need to be able to do to really deliver faster or to really be getting – to get into a continuous-delivery mode, you need to be able to build continuously but also test end-to-end on a continuous basis. And that is fundamentally what we enable through automation. We can get to 90-percent automation; we leave manual testing only for usability and exploratory. And then, you know, you truly have your DevOps tool chain in a continuous mode, where you’re doing continuous builds and then –

    Shimel: Mm-hmm.

    Johri: You’re doing continuous builds on the one hand, you’re doing continuous testing on the same cadence, and then you can push it out into ops, so you’re right. It becomes absolutely critical, and the manual testing is rapidly decreasing, and people are shifting more to automation now.

    Shimel: Absolutely. And not only shifting. You know, it’s a global, massive shift to automation and especially with the testing. And, frankly, look, that’s why you get the kind of valuations we’re seeing. That’s why you’re seeing the kind of acquisitions by large companies that we’re seeing. You know, I don’t see how you can do testing, going forward, without automation, so this is a fundamental change in the way this is being done. You know, you look at some of the companies from the old way of doing it, who are maybe selling testing equipment and so forth. Right? And they’re kind of dead men walking, to a certain extent. Right? And I –

    Johri: Yeah.

    Shimel: Geez, I don’t know what to tell you with it.

    Johri: But, you know, Alan, one more point on that. I think when you think about a large enterprise – a bank, an insurance company, or the like – their business processes span technologies, meaning they have modern front ends, but they also have a lot of legacy systems. And most of the early DevOps work has been done only on the front-end systems, where you can go dev and, as long as you’ve done some unit testing, you can move to production. When you’re thinking about enterprises with large end-to-end systems that span every technology that’s been invented, you have to get into a continuous-testing mode; otherwise, you cannot do DevOps for a large enterprise. So the enterprise DevOps is a slightly different animal than, let’s say, a smaller startup in Silicon Valley doing DevOps, where they have a relatively modern tech stack and a relatively monolithic application, if you may.

    Shimel: And, frankly, I always like to say, Sandeep, those companies – first of all, it’s DevOps, right? I don’t care whether you’re big or little – you’re all like do DevOps – but they do DevOps because they just don’t have the resources to do it any other way, frankly, right? Everyone wearing multiple hats and the way they – they have to – it is just no other way. It’s in their DNA, if you will. Where, with large enterprises, it’s DevOps or die, to a certain extent, in today’s competitive marketplace. And when I say “DevOps,” I’m talking about continuous – continuous testing, continuous delivery, continuous integration. You know, that automation is wherever – I mean, you know, all of the things we’ve come now to recognize as DevOps. So –

    Johri: Mm-hmm.

    Shimel: Sandeep, I guess we see these mega-trends. We see the way the market’s heading. We’ve heard about your background and the company’s history. I mean, it’s an eight-year-old or nine-year-old startup, right?

    Johri: Mm-hmm. That’s right.

    Shimel: What does the future hold? Specifically for Tricentis.

    Johri: Well, you know, we see ourselves as – and also we have about 600 large enterprises across those regions that I talked about – US, most of EMEA, and then Australia, New Zealand, and Singapore – and we focus primarily on large enterprises, the Global 2000 or the Global 5000. Competitively, we’ll end up replacing the old HP, which is now called “Micro Focus.” Most enterprises are looking for a modern tool set, just like they use AppDynamics or New Relic to replace the old app-monitoring solutions and they’re replacing old defect-management systems with newer things like Jira and the like. So we see a lot of replacement, but the really exciting part is that the $30-billion spend that I was talking about, on testing services, that is, like I said, mostly manual, and, as you start automating that, what we are really doing is disrupting that $30-billion manual-testing market. So the global opportunity for us is incredible. There’s a multibillion-dollar opportunity, in terms of what a platform like us can be selling out there.

    Shimel: Mm-hmm.

    Johri: Mercury was a billion-dollar company – almost a billion-dollar company 10 years ago, 2007, and so we see that not only as a replacement as a replacement opportunity, but also, because we’re disrupting the services play, as you disrupt manual labor, you know, a big chunk of it goes towards the automation tools, and that’s what we see. So, in terms of the future, we see a very, very bright future. And we wanna be the marketshare leader in testing tools, so we have a very comprehensive solution already for end-to-end testing, both performance and load testing, and we will continue to enhance it so that we’re the innovative leader, as we are right now. We will maintain that and continue the leadership, so.

    Shimel: Sandeep, I have little doubt of that, as long as you’re at the helm, driving this. It sounds like, based on your track record – and I have to agree with almost everything you said about the market. That’s kind of where we are right now and it should be an interesting next couple of years. The opportunity is certainly there, right? Someone’s gonna dominate this.

    Johri: Wonderful. Yes.

    Shimel: Yep.

    Johri: Mm-hmm. Mm-hmm.

    Shimel: Sandeep, as I told you when we started, the 15 minutes goes really quickly, so we’re at 20 minutes already. So we’re gonna need to kinda wrap up on this, but, before we do, for people who may be – we’re doing a bunch of activities with Tricentis, where I believe we’ll have some webinars and some content up on devopsy.kinsta.cloud, but, beyond that, where can people maybe either see you guys at conferences or speaking? Or, obviously, they can go to tricentis.com – T-R-I-C-E-N-T-I-S.com – for information, but just if you happen to know off top of your head where people might get more info even or a little hands-on –

    Johri: Yeah, well, we have – yeah, absolutely. Well, we, like you said, Alan, we do participate in most of the major conferences, either on DevOps or testing. You know, there’s testing-focused conferences we participate in. We also participate in larger vendor conferences, like the Appalachian Summit and SAP’s SAPPHIRE. But what is most exciting for me is that, this year, we’re organizing our first US user conference, which will really be a conference focused on continuous testing. And we have scheduled that for April, yeah, here in – actually, no, no, wait a minute. It is scheduled for –

    Shimel: I don’t know.

    Johri: Oh, God.

    Shimel: You know what?

    Johri: Yeah, I should –

    Shimel: I’m looking it up myself for you. Hold on here. Tricentis Accelerate? Would that be it?

    Johri: Yeah, it’s – yeah, it’s on the 8th or 9th.

    Shimel: San Francisco, May 7th to May 8th.

    Johri: Yes. Yes. Yes. May 7th and 8th. Sorry about that.

    Shimel: It’s okay.

    Johri: But that conference is gonna be completely focused on continuous testing. We’re gonna have – obviously, show all our product capabilities and the like, but what’s more interesting and exciting is that we’ll have a whole bunch of large enterprise customers that are gonna talk about how they’re tackling continuous testing, as they are on their journey of DevOps, so it should be really exciting. I would really welcome everybody to come attend that conference in San Francisco on May 7th and 8th.

    Shimel: Yep. And also you’re doing, it looks like, one in Vienna, Austria, October 9th and 10th. And ’cause – you know what? – our audience, like DevOps itself, about 36, 37 percent are US-based and the rest of the world makes up the majority. For anyone interested, though, you can go to accelerate.tricentis.com – that’s accelerate, A-C-C-E-L-E-R-A-T-E, at tricentis – or accelerate.tricentis.com, to get information on both of those. And we’ll probably be covering them from devopsy.kinsta.cloud as well. Well, this has been a great introduction to Tricentis and a great discussion of the testing, Agile, DevOps market. Sandeep, thank you so much for joining us.

    Johri: It was wonderful, Alan. Thanks a lot for the time. And I have a special offer for you. For your listeners, we will make some tickets available to the conference and you can distribute it to your favorite listeners, so we will follow up and get you a whole bunch of tickets to the conference that you can distribute to your readership. Your readership is exactly who we target and who we – that’s in our domain. That’s exactly our domain, so we’d love to have as many of your readers – or listeners and readers –

    Shimel: As both, right.

    Johri: attend the conference as possible. Yeah. Wonderful.

    Shimel: Really kind for you. I will put it in the show notes as well as on the article on devopsy.kinsta.cloud, for people who are interested. We’ll raffle ’em off, depending how many we have here. Sandeep Johri, CEO, Tricentis, powerhouse in automated testing. Thanks for being this episode’s guest on DevOps Chat and continued success.

    Johri: Thank you.

    Shimel: Thank you. This is Alan Shimel for devopsy.kinsta.cloud, DevOps Chat. Thanks for listening, everyone, and we’ll see you soon on another DevOps Chat.

    — Alan Shimel

  • Testing and Certification in the Age of Automation

    Testing and Certification in the Age of Automation

    If we do not change the way we teach, 30 years from now we will be in trouble. Because the way we teach, the thing we teach our kids are the things from the past 200 years. It is knowledge based. And, we cannot teach out kids to compete with machines, they [machines] are smarter.

    We have to teach something unique so that a machine can never catch up with us.

    —Jack Ma, CEO Alibaba

    tl;dr synopsis

    Given the complexity of modern DevOps, technical certification is becoming more granular. Also, its use is exploding as a requirement for employment. Most technical certification is awarded by passing a test composed of multiple choice questions. Passing a test with multiple choice questions requires the test-taker to absorb a significant volume of knowledge for a short period of time. This acquired knowledge does not need to be retained. Hence, the certification can only attest to the bearer’s knowledge of a technology at the time the test was taken. And, essentially the test measures only the bearer’s ability to take the certification examine. Finally, testing by multiple choice does not ensure the certification bearer’s qualification in terms of required experience or higher-level thinking capabilities.

    Technical certification is valuable in the current commercial landscape. However, at some point, as machine intelligence is able to perform more of the activities presently done by humans, the format and methods by which certification testing is administered will need to change. Future certification testing methods will need to put greater emphasis on analysis and creative problem-solving rather than the current focus on knowledge acquisition and rote application. Also, what and how we teach humans will need to be unique, beyond the capabilities of artificial intelligence.

    Over the last 10 years, DevOps has become a much more complicated landscape. Regardless of the service provider of choice—Amazon Web Service, Microsoft Azure, Google Cloud, OpenStack, IBM, Joe’s Homegrown Cloud Service—you gotta know a lot to play.

    At a high level, service providers offer products that are surprisingly similar. For example, they all provide some sort of machine virtualization, storage services, database services, queuing technology, authorization service, provisioning technology and API management. But the way you manipulate the conceptual knobs and switches to get any of these products to work can vary dramatically between providers. There’s a lot of minutiae involved. Absorbing the amount of detail required to be productive in a provider’s product line takes time. And, ensuring that someone can actually work competently with a service provider’s products is a crapshoot. For example, a deployment engineer may understand the concepts behind dynamic provisioning, but getting a Kubernetes orchestration to work in Google Cloud is where the rubber meets the road. Doing is always much more difficult. As I said, there’s a lot of detail involved.

    So, how do companies ensure that the people they hire to fiddle with their multimillion dollar, mission-critical technology stack will deliver the goods without blowing things up? Most have an extensive technical interview process accompanied by a stringent reference check. Some just hope. Others require certification.

    The Rise of Micro Certification

    The notion of certifying that a person has the skills and experience required to be competent in a profession has been around since 1885, if not before. That was the year Massachusetts implemented the first written bar exam. Before that time, all a person needed to do lawyering was to get a note from a court saying the person could be a lawyer. That’s how it went for Abe Lincoln. Eventually the profession matured. Professional qualification was determined through a test-based approach. Early bar exams used essay questions. In 1972, with the addition of the Multistate Bar Exam, multiple choice questions were included.

    Going to multiple choice questions may seem like a trivial change, but it’s not. Whereas a knowledgeable professional is required to grade an essay question, multiple choice answers can be evaluated by anybody with the answer key. Hence, the popularity of multiple choice questions on the SAT. It’s the most efficient way to grade millions of high school test-takers each year. Evaluation is fast, cheap and a machine can do it.

    As I mentioned at the beginning of this piece, IT in general and DevOps in particular has become a place of increasingly granular complexity. There are a lot of technologies on the landscape. Each of those technologies has a significant learning curve to be overcome to achieve mastery. The conventional wisdom is that the best way to determine mastery is to test a person’s knowledge of the given technology and then measure the results of the test—high score, good; low score, bad. As with early testing practices, the fastest, cheapest way to make an evaluation is to administer a test made up of questions with multiple choice answers. This is sort of like the SATs, except most technical certification is geared to a particular vendor’s product line, not a general body of knowledge. (You can view a sample of test for AWS Certification as a DevOps Engineer Professional here).

    This type of professional testing paradigm is exploding. It’s infiltrating every corner of the technology landscape. And, more employers are requiring certification as conditions of employment, both for prospective employees and those already on staff. This would be OK, except for one thing: After you scrape away all the preconceptions and assumptions about the value of test-based certification, when it comes down to the bare-bones validity of test measurement, the only thing a test really measures is the ability to take the test. For example, a deployment engineer might be sitting for a certification test after having been up all night working to get the company back online after a system failure. The taker, who is exhausted, ends up failing miserably. Yet, some slacker just out of high school who has never seen in inside of a data center can take the test using stolen answers and pass with flying colors. Tell me, please: Who is really the most qualified professional?

    Testing by Regurgitation

    Certification is not the problem. After all, I really do feel more comfortable knowing that the surgeon about to open up my wrist to alleviate my carpal tunnel syndrome is certified by the American Board of Surgery. Rather, the problem is the way that certification testing is conducted. In my opinion, a certification testing process that emphasizes the multiple choice question-and-answer format is flawed. Even when questions are answered correctly, the test format does not really verify the competence of the taker, only that some facts are known. It’s just testing by regurgitation.

    Such testing requires the taker to absorb a lot of knowledge and then dump it out according to the test requirement at hand. There is no guarantee of long-term retention. It’s the age-old process of cramming for the test. Once the “time’s up!” bell rings, the chances are that the test taker will forget most of what she absorbed. Those with photographic memories might remember it all, but for most of us mere mortals, it’s a one-shot deal.

    Few people in tech work using the cram-it-in method. Most of us just use this thing called the internet to get the answers we need, when we need them. It’s what Bill Gates calls “information at your fingertips.” Yet, we still test for the knowledge in your head at the moment.

    There is a better way.

    A Better Way

    Certification is good provided it ensures that the bearer knows what he or she is doing. When certification is about nothing more than checking the right checkbox on an online test, there’s a problem. Put a monkey in front of a multiple choice test for eternity and eventually the animal will score 100 percent.

    So what do we do?  We do the following:

    Support Open Book Testing

    If the purpose of the test is to ensure that a person has a certain body of knowledge at the time he or she takes the test, make the body of knowledge required available at test time. It’s called open book testing. That’s how it works in real life. For example, you need a NodeJS package to provide mock objects. Do you rely upon your existing knowledge about the packages you know? Great, but what if you don’t know? Does work come to a grinding halt? No. You go to NPM to find one that meets your need. That’s how it is in real life. A competent professional is expected to have a baseline, high-level knowledge of his or her professional domain. For the day-to-day details, a pro has the ability to find accurate answers to questions quickly.

    Instead of having a certification testing process based on cramming, maybe we should eat the dog food we make in real life and go open book. There is little downside to researching what you need, when you need it. Besides, the test question created a week ago might be best answered by a new technology created today.

    Put the Emphasis on Essay Questions

    The actual mechanics of testing vary according to the level of cognition being examined. I wrote an article about this a few years back. The best way to test higher-order thinking—analysis, synthesis, evaluation—is by using essay questions. After all, what is a Ph.D. dissertation if not a very big essay question? Essays require a level of thinking that is hard to capture using lower-order test methods such as multiple choice. Essays reveal not only what the test taker thinks, but also how the test taker thinks. There’s a reason why your high school math teacher wanted you to show your work when taking an algebra test. Sometimes how you get the answer is just as valuable as the answer itself.

    A good certification test will provide a question format that allows test takers to illustrate their thinking along with the way they arrived at an answer. Thus, the need for a bias toward essay questions. Again, how you think is just as important as what you think.

    Put a Human Back in the Examination Process

    I’ll share an intimacy with you: I’m not very good at answering multiple choice questions. When confronted with one, I spend a lot of time trying to figure out what the question is really asking. You’d be surprised at the number of test questions out there that are vague and confusing.

    This was a real problem for me back when it was time to go to college and do the SAT dance. Fortunately, my alma mater focused more on my essays than my SAT score. Also, a lot the exams during my collegiate years were administered as essays or conducted as interviews with a professor. Later, when it was my time to be the Guy Awarding the Grade, my process was to evaluate students by interviewing them about one or many of the projects they did. I based my grade on the quality of the project itself and the thinking that went into the work. I tested my students by talking to them and reviewing their work. I like to think that I got a pretty accurate idea of a person’s competence, more than a multiple choice test could ever provide.

    Providing the opportunity for human interaction in the examination process can go a long way to providing the information required for making an accurate evaluation of a person’s competence.

    Require Testimony of a Candidate’s Experience, Character and Competence

    Think about this: It’s possible to be an AWS Certified Solutions Architect Professional without ever having done a day’s worth of paid, real-world work. All you need to have done is pass the certification exam. Yes, of course passing the exam is meaningful. It’s a grueling test. At the least, the test might prove that should be able to work the conceptual dials and switches that go with a given AWS service. But, as far as having the real-world experience, competence and character to play a critical path role in a mission-critical enterprise, some other instrument of certification is required. This instrument is sometimes called a reference—or, as the Wizard said to the Tin Man in “The Wizard of Oz,” a testimonial.

    As you read above, back in Abe Lincoln’s day, a letter from a court attesting the good moral character of the bearer was all that was required to practice law. You can think of good moral character as the ability to show up on time, do reliable work, demonstrate a commitment to excellence and get along with others. Such a description might sound melodramatic, but it counts.

    While it’s true that some certifications require that bearer to have a modern version of “good moral character,”  in addition to good test scores, many do not, particularly in tech. An employer wants to hire people who know what they’re doing and have the wherewithal to work effectively in an organization. Potential candidates want to provide evidence that they have what it takes to be a good employee. Building a testimonial of the bearer’s experience, character and competence into a technical certification will add a great deal of value for all involved. Also, the testimonial needs to affirm the bearer’s ability to adapt to change. In the not too distant future, your commercial value will not be about what you know today, but rather how fast you can work with technologies that will appear tomorrow.

    How to Win the Race

    For better or worse, we’re in a race with machines to keep our jobs. They don’t really want our job. They’re incapable of desire. But, that’s not to say they can’t have our job should the machine prove to be able to do it. Sadly, the current practice of technical certification is focused on passing tests concerned with the acquisition of knowledge.

    Right now, we’re winning the race. Machines still have problems with knowledge-based tasks involving advanced image recognition and reading comprehension. We still know how to differentiate between a dog, cat and an apple tree better than a machine can.

    But, it’s not always going to be this way. Keep in mind that talking machines were very primitive 20 years ago. Their voice had an artificial, staccato rhythm that was distinctly robotic. Today, it’s hard to tell the difference between a human or machine when a robocall disturbs your evening meal. It’s only a matter of time before the machine can do mid- to high-level knowledge-based work better than any human. Jack Ma’s quote at the beginning of this article supports this assertion.

    We need to change. The commercial value we’ll bring to the workplace in the future will be about analyzing, synthesizing and evaluating. Machines will be able to to the lower-level thinking faster and more accurately than we ever can. The realm of human excellence will be about better understanding the world around us and bringing new things into that world. The way we start is by teaching something so unique that the machines can never catch up. This thing is creativity. The paradox is that creativity can always be learned, but rarely be taught. Still, we must try. We have little choice not to.

    — Bob Reselman

  • Applitools Advances AI Testing of User Interfaces

    Applitools Advances AI Testing of User Interfaces

    Given the number of platforms applications now need to be able to run on, it’s little wonder that mistakes and errors that compromise the graphical experience of the end user appear all too regularly in production. Keeping up with all the various nuances of each platform is just too much to ask of the average developer. To even those odds, Applitools has developed Applitools Eyes, the latest release of which extends the reach and scope of user interface testing in addition to providing expanded third-party tool integrations, enhanced analytics and team collaboration tools.

    Version 10 of Applitools Eyes extends the company’s mission to employ artificial intelligence (AI) to enable organizations to consistently deliver a better application experience, said James Lamberti, chief marketing officer for Applitools.

    Testing of graphical user interfaces (GUIs) is becoming more complex because the front-end of each application is increasingly made multiple microservices. Any change to the application requires developers to test each individual microservice and associated content that gets rendered on any number of platform. Applitools Eyes automates that testing process in a way that enables organizations to deliver applications faster without compromising the user (UX) experience, said Lamberti.

    “AI makes it easier to identify where the UX is broken,” he said.

    Much of the AI focus on DevOps these days tends to be on automating back-end testing. But Lamberti said it’s already been proven AI has a significant role to play in automating UI testing as well. To spur awareness of that requirement, Applitools also announced an Application Visual Management initiative, an attempt to create a new category of tools for visually testing graphical user interfaces.

    Lamberti noted that functional tests today are automated, and the visual elements of an application are still conducted manually. That invariably leads to errors that conspire to impede application release cycles no matter how much time and effort may have been invested in continuous integration and continuous delivery (CI/CD) processes.

    Applitools claims to have already executed almost 100 million visual tests using its visual AI engine to analyze billions of individual UX elements. Version 10 of Applitools Eyes adds improvements to the Layout Algorithm for dynamic content use cases as well as dashboards to make it easier to discover the root cause of a UI issue. Development teams, for example, can review visual test execution status and baseline status updates, bugs squashed and coverage levels specific to individual tasks and test assignments.

    Bugs discovered by Applitool Eyes can now also be programmed to automatically generate a ticket with a bug tracking tool such as Jira.

    In addition, integration with collaboration tools from Micro Focus as well as Slack is planned for March.

    Developers are often especially challenged when it comes to application usability. Application Visual Management may provide a much-needed framework for capturing UI best practices that would enable developers to avoid the same issue over and over by, for example, enabling a testing tool to make specific recommendations about where to place specific functions on a screen. There may even come a day when AI builds the application. In the meantime, when it comes to building applications end users will actually embrace regularly, most developers need all the help they can get.

    — Mike Vizard

  • We’re All Load Testers Now (Maybe)­

    We’re All Load Testers Now (Maybe)­

    More than 94 percent of organizations already have Agile or DevOps initiatives or plan to adopt them within the immediate future, according to a survey by VersionOne. As part of the process transformation required to adopt these more iterative and rapid methods of delivering software, organizations are reassessing all practices associated with software development and test, including load testing.

    Traditionally, load testing has been deferred until the late phases of waterfall release cycles. Specialized performance testers applied legacy load testing tools to validate application performance prior to release. Why is this no longer sufficient?

    1. With teams moving to more componentized applications (including cloud-native apps, microservices, etc.), an application involves many highly distributed components—and a performance issue in any of them could have a ripple effect across the entire application.
    2. Now that new functionality is being released weekly, daily or hourly, each team needs instant insight into whether their incremental changes negatively impact performance.
    3. The later you start load testing, the more difficult, time-consuming and costly it is to expose, debug and resolve performance problems.

    In-depth load testing by performance-testing specialists remains critical, but it doesn’t provide the level of fast, on-demand load testing that’s critical for Agile and DevOps. Developers and testers need a way to expose critical performance issues before new functionality progresses through the delivery pipeline. To achieve this, DevTest team members must be able to easily:

    1. Create load tests that provide fast feedback on the functionality they’re evolving; and
    2. Execute and scale those load tests as needed—without the exorbitant costs and efforts traditionally required to establish, configure and maintain a performance test lab.

    Traditional Load Testing Isn’t for Everyone

    However, load testing has long been the domain of performance testing specialists for a reason: it’s difficult. Today’s developers and testers don’t have the time (or desire) to wrestle with all the technical details required to get load tests working correctly and to keep brittle load tests in sync with the rapidly evolving application.

    The traditional way of approaching load test scripting is at the protocol level (e.g., HTTP). This includes load testing with open source tools such as JMeter and Gatling, as well as legacy tools including LoadRunner. Although simulating load at the protocol level has the advantage of being able to generate large concurrent load from a single resource, that power comes at a cost. The learning curve is steep and the complexity is easily underestimated.

    The main culprit for this complexity is JavaScript. In 2011, there was usually less than 100 KB of JavaScript per page, which spurred around 50 or fewer HTTP requests. Now, that’s doubled: We see on average  200 KB of JavaScript per page, and this gives us more than 100 requests per page.

    Just running a search on a simple search page involves things such as XML HTTP requests processed asynchronously after page load. You also find things such as dynamic parsing and execution of JavaScript, the browser cache being seeded with static assets and calls to content delivery networks.

    For a more business-focused example, consider the SAP Fiori demo app. Assume we want to load test two simple actions: navigating to a page and then clicking on the “My Inbox” icon. This actually generates more than 120 HTTP requests at the protocol level.

    When you start building your load test simulation model, this will quickly translate into thousands of protocol-level requests that you need to faithfully record and then manipulate into a working script. You must review the request and response data, perform some cleanup and extract relevant information to realistically simulate user interactions at a business level. You can’t just think like a user; you also must think like the browser.

    You need to consider all the other functions that the browser is automatically handling for you, and figure out how you’re going to compensate for that in your load test script. Session handling, cookie header management, authentication, caching, dynamic script parsing and execution, taking information from a response and using it in future requests … all of this needs to be handled by your workload model and script if you want to successfully generate realistic load. Basically, you become responsible for doing whatever is needed to fill the gap between the technical and business level. This requires both time and technical specialization.

    You might be thinking, “Okay, we’ll use ‘record and playback’ tools, then.” Theoretically, you could just place a proxy between your browser and the server, record all the traffic going through and be set. Unfortunately, it’s not quite that simple. Even though you’re interacting at the UI level, the testing is still based on the protocol level. Assume we were looking at the traffic associated with one user performing the simple “click the inbox” action described above. When we record the same action for the same user two different times, there are tens if not hundreds of differences  in the request payload that we’d need to account for.

    Of course, you can resolve those differences with some effort. Unfortunately, when the application changes again, you’re back to square one. The more frequently your application changes, the more painful and frustrating this becomes.

    Taking a Browser-Based Approach

    To sum up the challenge here: modern web applications are increasingly difficult to simulate at the protocol level. This raises the question: Why not shift from the protocol level to the browser level—especially if the user’s experience via the browser is what you ultimately want to measure and improve?

    When you’re working at the browser level, one business action translates to maybe two automation commands in a browser as compared to tens, if not hundreds, of requests at the protocol level. Browser-level functions such as cache, cookie and authentication/session management work without intervention. There are a number of ways to simulate traffic at the browser-level: Selenium is clearly the most popular, but there are a number of cross-browser tools available—some of which let you test without getting into scripting.

    However, historically, it just wasn’t feasible to run these tools at the scale needed for load testing. In 2011, if you wanted to launch 50,000 browsers with Selenium, you would have needed something on the order of 25,000 servers to provide the infrastructure. Moreover, it would have been prohibitively expensive and time-consuming to provision the necessary infrastructure.

    Today, with the prominent availability of cloud-based technology, the concept of browser-based load testing is feasible. At the same time, projects Google Chrome and other projects are offering fast automation and better memory profiles with headless or UI-less variants of the browser. In fact, tests with headless Chrome show that it’s possible to go far beyond the industry standard benchmark of two to five browsers per machine and achieve around 50 browsers per machine.

    Suddenly, generating a load of 50,0000 browsers is a lot more achievable—especially when the cloud can now give you access to thousands of load generators that can be launched from any web browser and can be up and running in minutes. Instead of having to wait for expensive performance test labs to get approved and set up, you can get going instantly at an infrastructure cost of just cents per hour. Fast feedback on performance is no longer just a pipe dream.

    At Flood IO, we’ve captured our browser-level user (BLU) research and development in Flood Chrome: a browser-based load generation approach that builds upon APIs from Google for headless Chrome automation. This approach reduces the complexity of writing and maintaining tests while maintaining a viable level of concurrency and performance per machine. It creates scripts at a higher level of abstraction—the user level—then distributes those tests via the cloud for better economies of scale.

    Why consider this new approach to load testing?

    1. Simple scripting—or no scripting at all
    2. Reduced test complexity
    3. Test entire stack in one shot from the user perspective
    4. Capable of testing any user behavior
    5. Record network and user interaction times for front-end optimization
    6. Easier to test earlier and often
    7. Easier to maintain
    8. 10X faster than other Selenium load testing

    Or, in just one line: because it’s purpose-built for DevOps.

    There’s Still a Time and a Place for Protocol-Level Load Testing

    Of course, no single load testing approach is not going to solve everyone’s load testing needs all the time. For example, if you’re trying to test an application that’s not accessible from a browser, the BLU approach isn’t going to work for you.

    Moreover, there are still some situations where you’d be remiss to overlook protocol-level testing with tools like JMeter, Gatling, or API-based test cases. For example, if you want to simulate load against APIs, I’d still recommend running tests that exercise them directly at the protocol level.

    Ultimately, though, it’s important to remember that protocol-level tests can be higher maintenance and use them accordingly. If you have a single click that makes 20 background requests, would you rather wrestle with all the technicalities of scripting that at the protocol level, or have one line of a BLU script that achieves the same business functionality?

    Final Thoughts

    By reducing the complexity traditionally associated with load testing, BLU load testing gives developers and testers a fast, feasible way to get immediate feedback on how code changes impact performance. It’s geared to help people who are not professional performance testers quickly get started with load testing and create load tests that can be run continuously within a CI/CD process—with minimal maintenance. With this new “lean” approach to load testing, any developer or tester can get started with load testing.

    — Tim Koopmans

  • A Crash Course in Continuous Testing

    A Crash Course in Continuous Testing

    What better way to start a blog on continuous testing than with this “I Love Lucy” clip, and then an analogy:

    https://youtu.be/8NPzLBSBzPI

    Like Lucy and Ethel, many software testers have been scrambling to keep pace with accelerated processes—then along comes the supervisor saying, “You’re doing splendidly … Speed it up!”

    This ever-increasing pace is just one of four aspects of the “perfect storm” driving the need for continuous testing.

    What is Continuous Testing?

    Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain feedback on the business risks associated with a software release candidate as rapidly as possible.

    Why do we need to rethink our approach to testing?

    The expectations associated with test are changing–but legacy testing platforms aren’t keeping up. Legacy testing platforms take a “heavy” approach to testing. They rely on brittle scripts, deliver slow end-to-end regression test execution and produce an overwhelming level of false positives. As a result, they’ve achieved limited success with test automation. The overall test automation rate is 18 percent, on average–8 percent for enterprises. In a polling question, the audience confirmed that the results of test automation have been so-so, at best.

    Now, software testers are facing increasingly complex applications, delivered at dramatically accelerated paces—and they’re expected to deliver trustworthy go/no-go decisions at the new speed of modern business. More of the same won’t get us there. We need to transform the testing process as deliberately and markedly as we’ve transformed the development process.

    The Differences Between Test Automation and Continuous Testing

    There are three main differences between test automation and continuous testing:

    • Risk
    • Breadth
    • Time

    Risk: Exposing more and more innovative functionality to the user is now a competitive differentiator—but it also increases the number, variety and complexity of potential failure points. Most tests are designed to provide low-level details on whether user stories are correctly implementing the requirements—not high-level assessments of whether a release candidate is too risky to release. Would you instantly stop a release based on test results? If not, your tests aren’t properly aligned with business risks.

    Breadth: Just knowing that a unit test failed or a UI test passed doesn’t tell you whether the overall user experience is impacted by recent application changes. To protect the end user experience, you need tests that are broad enough to detect when an application change inadvertently impacts functionality which users have come to rely on.

    Time: If your organization has adopted DevOps and is performing continuous delivery, software may be released hourly—or even more frequently. In this case, feedback at each stage of the process can’t just be “fast”; it must be nearly instantaneous. If the business wants to minimize the risk of faulty software reaching an end user, you need some way to achieve the necessary level of risk coverage and testing breadth—fast.

    What Are the Key Points of Continuous Testing?

    • Continuous testing’s primary goal is assessing business risk coverage.
    • Continuous testing establishes a safety net that helps the team protect the user.
    • Continuous testing requires a stable test environment to be available on demand.
    • Continuous testing is seamlessly integrated into the software delivery pipeline and DevOps toolchain.
    • Continuous testing delivers actionable feedback appropriate for each stage of the delivery pipeline.

    You can find a more extensive list (14 points) at the What is Continuous Testing page.

    Where Can I Learn More About Continuous Testing?

    Watch the complete continuous testing webinar to get more details on the above topics and also learn:

    • How continuous testing impacts the organization
    • What practices testers need to know to be successful with continuous testing
    • How non-automated practices such as exploratory testing fit into continuous testing

    If you want to learn more, download the white paper, “Continuous Testing vs Test Automation: What Testers Need to Know” or visit the What is Continuous Testing page.

    — Cynthia Dunlop

  • CA World 2017: DevOps As Far As the Eye Can See

    CA World 2017: DevOps As Far As the Eye Can See

    The show floor for CA World 2017 had one inescapable, very obvious feature: There was DevOps as far as the eye can see. No kidding. It was a stunning reminder that this multibillion-dollar-revenue company has gone all-in on DevOps and related technologies. While we know CA has spent almost a billion and a half dollars acquiring DevOps-related companies in the last year, the effect on this year’s CA world was dramatic.

    CA World 2017

    What Was All That DevOps?

    CA World is organized so that most of the sessions except the keynotes are all conducted right on the expo floor. The floor is broken into distinct areas. Each area has its own theater, where sessions on the subject of that area are conducted. It is actually a great way to organize a conference like this and saves you from going to room to room, floor to floor.

    DevOps everywhereCA World 2017’s main floor segments were:

    • Automation – Basically CA Automic and all of its related partners, integrations and offerings. Primarily a DevOps area.
    • DevOps: APIs and microservices – This has been a staple of CA World now for several years. APIs again are related to DevOps, automation and microservices.
    • Mainframe -This area focused on enabling customers to extend DevOps practices to legacy applications, which most critically included a modern mainframe experience for enterprise developers
    • DevSecOps – There was a whole area dedicated to DevSecOps. Primarily the Veracode team—they had their own theater and lots of space and attention in the middle of the floor. The DevSecOps section did bleed into a smaller security section with the legacy CA security tools.
    • Agile Management – All things agile, based primarily on the Rally business CA bought a few years ago.
    • DevOps Agile Operations – APM and things of that nature.
    • DevOps CD – This area also included continuous testing and the new CD Director.
    • Advanced analytics
    • MSP Zone
    • CA Agility Zone
    • CA Acceleration Zone

    As you can see, a lot of DevOps right there.

    All Your DevOps Are Belong To Us

    But more than the real estate of the show floor, even the messaging coming from CA was “all DevOps, all the time.” The keynotes from the executive team were heavy on DevOps and related topics. The sessions at each theater featured household-name customers mixed with partners and CA people discussing DevOps-related topics.

    One presentation I found interesting was from the GM of the DevOps continuous delivery business, Jeff Schaeffer. It was a demo of a new product CA rolled called DevOps Continuous Delivery Director, a SaaS product that is a UI wrapped around a lot of the CD-related solutions from CA and leading partners. All kinds of testing, including testing from Sauce Labs and security testing from Veracode was possible from within the interface.

    I moderated a good panel on Continuous Testing: Shifting Left, which was well-attended. I was joined by customer reps from Cigna, a large financial institution, Groupon and others. This was also in the DevOps CD area.

    The mainframe theater stage featured a DevOps panel session to showcase a cross-BU perspective. Panelists included Aline Gerew from Automation, Steve Feloney from Continuous Delivery, Scott Morrison from APIs and Microservices, and Jean Louis Vignaud from Mainframe. The video recording is not yet available, but I was able to catch one snippet in a promotional tweet – APIs enabling IAM on Mainframe

    What’s Still Needed?

    As impressive as the DevOps show by CA was, what is missing is a tighter integration between the DevOps solutions. While some rudimentary integration is in the works, really making a 1 + 1 = 3 equation will provide the enhanced value to customers that such a broad set of DevOps solutions promises. I spoke to executives from just about every DevOps-related business unit and to a person they mentioned that this was in the works. But what was pleasantly surprising was the mandate not to force tighter integration that would stunt the innovation that each of the DevOps companies brought with them when they were acquired by CA.

    Of course, we will see if this remains the case. However, I did speak with employees at each of the units and they were all happy with how CA has handled the acquisitions to this point.

    Conclusion

    It is hard to argue when a software company the size of CA Technologies makes such a bold statement regarding DevOps. The company is clearly betting its future on DevOps being the backbone of the modern software factory.

    — Alan Shimel