devopsy.kinsta.cloud

Tag: testing

  • Evolving to Continuous Testing

    Evolving to Continuous Testing

    As agile development practices mature and DevOps principles infiltrate our corporate cultures, organizations are realizing the distinct opportunity to accelerate software delivery. However, when you speed up any process, immature practice areas such as testing and roadblocks become much more pronounced. It’s the difference between driving over a speed bump at 5 mph vs. 50 mph … at 50 mph, that speed bump is going to be quite jarring.

    Accelerating any business process will expose systemic constraints that shackle the entire organization to its slowest moving component. In the case of the accelerated software development life cycle (SDLC), testing has become the most significant barrier to taking full advantage of more iterative approaches to software development. For organizations to leverage these transformative development strategies, they must shift from test automation to continuous testing.

    Drawing a distinction between test automation and continuous testing may seem like an exercise in semantics, but the gap between automating functional tests and executing a continuous testing process is substantial. This gap will be bridged over time as the process of delivering software matures. Both internal and external influences will drive the evolution of continuous testing. Internally, agile, DevOps and lean process initiatives will be the main drivers that generate the demand for change. Externally, the expense and overhead of auditing government and industry-based compliance programs will be the primary impetus for change.

    A Roadmap for Testing

    Any true change initiative requires the alignment of people, process and technology—with technology being an enabler and not the silver bullet. Yet there are some basic technology themes we must explore as we migrate to a true quality assurance process. In general, we must shift from a sole focus on test automation to automating the process of measuring risk. To begin this journey, we must consider the following:

    Driven by business objectives, organizations must shift to more automated methods of quality assurance and away from the tactical task of testing software from the bottom up.
    graphic

    From Causal Observations to Probabilistic Risk Assessment

    With quality assurance (QA) traditionally executing manual or automated tests, the feedback from the testing effort is focused on the event of a test passing or failing—this is not enough. Tests are causal, meaning that tests are constructed to validate a very specific scope of functionality and are evaluated as isolated data points. Although these standalone data points are critical, we must also use them as inputs to an expanded equation for statistically identifying application hot spots.

    The SDLC produces a significant amount of data that is rather simple to correlate. Monitoring process patterns can produce very actionable results. For example, a code review should be triggered if an application component experiences all of the following issues in a given continuous integration build:

    • Regression failures greater than the average
    • Static analysis defect density greater than the average
    • Cyclomatic complexity greater than a prescribed threshold

    From Defect Documentation to Simulated Replay

    The ping-pong between testers and developers over the reproducibility of a reported defect has become legendary. It’s harder to return a defect to development than it is to send back an entrée from a world-renowned chef. Given the aggressive goal to accelerate software release cycles, most organizations will save a significant amount of time by just eliminating this back and forth.

    By leveraging Service Virtualization for simulating a test environment and/or virtual machine record and playback technologies for observing how a program executed, testers should be able to ship development a very specific test and environment instance in a simple containerized package. This package should isolate a defect by encapsulating it with a test, as well as give developers the framework required to verify the fix.

    From Structured Data to Structured and Unstructured

    The current tools and infrastructure systems used to manage the SDLC have made significant improvements in the generation and integration of structured data (e.g., how CI engines import and present test results). This data is valuable and must be leveraged much more effectively (as we stated above in the “From Causal Observations to Probabilistic” section.

    The wealth of unstructured quality data scattered across both internal and publicly-accessible applications often holds the secrets that make the difference between happy end users and unhappy prospects using a competitor’s product. For example, developers of a mobile application would want constant feedback on trends from end user comments on:

    • iTunes app store
    • Android app store
    • Stackoverflow
    • Twitter
    • Facebook
    • The company’s release announcements
    • Competitors’ release announcements

    This data is considered unstructured since the critical findings are not presented in a canonical format: parsing and secondary analysis are required to extract the valuable information. Although these inputs might be monitored by product marketers or managers, providing these data points directly to development and testing teams—in terms that practitioners can take action on—is imperative.

    From Dashboards to Business Policies

    In a Continuous Everything world, quality gates will enable a release candidate to be promoted through the delivery pipeline. Anything that requires human validation clogs the pipeline. Dashboards require human interpretation—delaying the process.

    Dashboards are very convenient for aggregating data, providing historical perspectives on repetitive data, and visualizing information. However, they are too cumbersome for real-time decision making because they do not offer actionable intelligence.

    Business policies help organizations evolve from dashboards to automated decision making. By defining and automatically monitoring policies that determine whether the release candidate is satisfying business expectations, quality gates will stop high-risk candidates from reaching the end user. This is key for mitigating the risks inherent in rapid and fully-automated delivery processes such as Continuous Delivery.

    From Tool Dependent to SDLC Sensors

    Let’s face it—it’s cheap to run tools. And with the availability of process intelligence engines, the more data observations we can collect across the SDLC, the more opportunities will emerge to discover defect prevention patterns.

    Given the benefit of a large and diverse tool set, we need to shift focus from depending on a single “suite” of tools from a specific vendor (with a specific set of strengths and weaknesses) to having a broad array of SDLC sensors scattered across the software development life cycle. And to optimize both the accuracy and value of these sensors, it’s critical to stop allowing tools to be applied in the ad hoc manner that is still extremely common today. Rather, we need to ensure they are applied consistently and that their observations are funneled into a process intelligence engine, where they can be correlated with other observations across tools, across test runs and over time. This will not only increase the likelihood of identifying application hot spots, but will also decrease the risk of false negatives.

    About the Author/Wayne Ariola

    Wayne Ariola (1)Wayne Ariola, Chief Strategy Officer, leads the development and execution of Parasoft’s long-term strategy. He leverages customer input and fosters partnerships with industry leaders to ensure that Parasoft solutions continuously evolve to support the ever-changing complexities of real-world business processes and systems. Ariola has contributed to the design of core Parasoft technologies and has been awarded several patents for his inventions. A recognized leader on topics such as service virtualization, SOA and API quality, quality policy governance, and application security, Ariola is a frequent contributor to publications including software test & performance, SOA World, and ebizQ—as well as a sought-after speaker at key industry events. Ariola brings more than 20 years strategic consulting experience within the technology and software development industries.

  • BlazeMeter Uses the Force of Performance Testing to Prepare MovieTickets.com for the Release of “Star Wars: The Force Awakens”

    SAN FRANCISCO–(BUSINESS WIRE)–Movie releases are major traffic events for online ticket providers. So when the most storied movie franchise in history comes out with a new chapter, it creates a unique performance challenge. That’s whyMovieTickets.com, a worldwide leader in advance movie ticketing, turned to BlazeMeter to ensure that its new Amazon Web Services infrastructure could manage the volume of online ticket orders around opening weekend for “Star Wars: The Force Awakens.”

    The majority of online advance ticket buying occurs 1-2 weeks prior to a release date, but the release of “Star Wars: The Force Awakens” marked a historically unique buying trend, as MovieTickets.com was inundated with pre-orders months prior to the film’s December 2015 release date.

    “The demand for advance ticketing during the first buying opportunity was unprecedented,” said Craig Lefkowitz, Director of Infrastructure Systems at MovieTickets.com. “We had not seen that type of volume two months before a movie release, so we needed to make sure we were well prepared for opening weekend.”

    To prepare its site to meet demand, MovieTickets.com wanted to upgrade its infrastructure to manage spikes in volume, migrating to an Amazon Web Services (AWS) cloud environment that could scale alongside demand. However, in order to make sure the new cloud-based infrastructure could perform optimally, the company enlisted the help of BlazeMeter’s continuous performance testing technology.

    “We chose BlazeMeter because it’s the leader in performance testing,” said Lefkowitz. “BlazeMeter is a standout and very well-respected in the industry. We needed to utilize a proven solution to get us ready for such an important event.”

    MovieTickets.com used BlazeMeter’s performance testing solution to examine various load volumes and times across the AWS infrastructure. A member of the AWS CodePipeline partner network, inaugural DevOps Competency Program member and trusted partner, BlazeMeter’s testing solution can integrate directly into an AWS DevOps or continuous delivery framework, identifying trouble spots in real-time and readying the site for major traffic events.

    With BlazeMeter’s help, MovieTickets.com tested their infrastructure to manage more than 30,000 simultaneous users per second. As they continued to extend their infrastructure, that number expanded to more than 300,000 concurrent visitors.

    “The higher the performance expectation from an event, the greater the need to ensure an infrastructure can support the required service level,” said Alon Girmonsky, CEO and Founder of BlazeMeter. “The trust that large companies have in our technology around critical events speaks to the importance of performance overall. The inability to manage the volume directly equates to lost sales, making performance testing a critical investment for any company that needs to maintain peak performance at all times.”

    “Performance is our top priority,” said Lefkowitz. “Our customers have high expectations and we need to meet those expectations.”

    For more information on how MovieTickets.com works with BlazeMeter, visit https://www.blazemeter.com/case-study/movietickets?utm_source=BM&utm_medium=newswire&utm_campaign=MovieTickets. And for more information on how to improve site performance through continuous performance testing, visit https://blazemeter.com/.

    About BlazeMeter
    BlazeMeter ensures delivery of high-performance software by enabling DevOps teams to quickly and easily run open-source based performance tests against any mobile app, website or API at massive scale to validate performance at every stage of software delivery. The rapidly growing BlazeMeter community has over 100,000 developers and includes prominent global brands such as Adobe, Atlassian, Gap, NBC Universal, Pfizer and Walmart as customers. Founded in 2011, the company is headquartered in Mountain View, California with Research & Development in Tel Aviv, Israel.

    For more information, sign up for a free 14-day trial at www.blazemeter.com, visit our blog or find us on Twitter @BlazeMeter and on LinkedIn.

    About MovieTickets.com
    MovieTickets.com is the leader in advance movie ticketing with nearly 250 theater chains, representing over 28,000 screens in its group, offering moviegoers a quick and convenient way to purchase tickets online, via mobile devices and at 877-789-MOVIE. The company currently enables remote ticketing for consumers across 21 countries/territories including the United States, Canada, United Kingdom, Ireland, Argentina, Puerto Rico, the Dominican Republic and the Caribbean. Fans can stay connected with MovieTickets.com through Facebook (facebook.com/MovieTicketscom), Twitter (twitter.com/movietickets) and YouTube (youtube.com/c/movietickets).

    Contacts

    Press Contact:
    PAN Communications
    Adam Novak, +1-617-502-4300
    blazemeter@pancomm.com

  • Case Study: Sauce Labs Covers Animoto’s Testing Needs

    Case Study: Sauce Labs Covers Animoto’s Testing Needs

    Animoto’s web and mobile platforms enable businesses and consumers to drag and drop stock and user-defined visual and audio elements to create videos via a tiered monthly subscription service. Animoto updates service features with increasing frequency—some of them several times a day.

    “We have a broad range of applications and features including an online video maker application, where we enable users to import videos and video elements from all the known, established sources such as Facebook and through Dropbox,” says Siddharth (“Sid”) Jain, VP of Engineering at Animoto. “They can then export videos out to a variety of venues such as YouTube.”

    Animoto must iterate new innovations continuously into its application/platform for high-quality yet simple video creation to compete with similar solutions in the marketplace such as WeVideo and Stupeflix. This fast-paced development process is challenging, since Animoto must update separate versions of its software for the web, Android OS and Apple iOS (both iPhone and iPad). “The mobile apps are largely consumer-focused for very easy, on-the-go, frictionless video creation and sharing, while our web apps feature server audiences of professional photographers, educators and small businesses,” says Jain.

    In addition to testing service capabilities of different versions of the video maker application, Animoto must test the on-boarding/registration flow and the free product experience, where users get their first taste of Animoto. “We do lots of interesting things such as testing support for multiple currencies around the globe and running continuous multiple AB tests to ensure that the free product experience is working,” Jain notes.

    Notable Animoto testing includes user funnel testing in a simulated user flow environment starting at sign-up and continuing through video creation, feature exploration and the upgrade experience. “We test this several times a day on the different platforms,” Jain says, noting Animoto tests for the seamless success of import and export of videos and video elements to and from Facebook, Dropbox and YouTube. “The emphasis is on ensuring the quality and consistency of the user experience, i.e., making sure that nothing goes out that breaks revenue in terms of service.”

    Porting to Sauce Labs Promotes Progress

    Sauce Labs‘ cloud service helps Animoto scale up and out instantly to test Apple/iOS versions of its application so it can meet the high standards set by Apple and ensure fast inclusion of new app versions in the Apple App Store. This helps Animoto to compete by getting new innovations and features to market more quickly. “The biggest advantage to using Sauce Labs is that we don’t have to create or maintain a test lab internally,” says Jain.

    This accelerated testing speed is especially important since the app must adapt to every new iOS version while maintaining backward compatibility. “It would be a very nightmarish thing to do this all manually,” he says.

    In addition to faster testing, Sauce Labs is helping Animoto ensure that testing was delivering on its promises of software improvement through accurate feedback. “We used to be religious about doing extensive unit tests. Now we emphasize integration tests over unit tests because we got bit by unit tests that were all green even though the software functionality was broken,” says Jain. “Now we have a balance between unit and integration tests while frontloading the latter with tests of customer-facing functionality and collecting fine-grain feature use data.”

    The comprehensive testing helps to ensure that users aren’t shying away from certain features and that features are not broken. Frontloading uncovers regressions very early so the Animoto developers dedicated to fixing them can catch them before they lead to breakage.

    Sauce Labs helps Animoto shrink technical debt. “Debt rears its head in awkward ways. We do get surprised, but having good test coverage allows us to know where the app hot spots are and to have feedback to fix these and verify changes swiftly,” says Jain.

    Results: More Stability, More Capacity

    “We have been able to reduce testing time from three hours to under 30 minutes for some tests,” says Jain. “Thanks to Sauce Labs’ cloud, we added VM capacity, moving from 20 VMs to 45 VMs for the testing environment.”

  • Performance Testing: The Unexpected Missing Link in DevOps

    Performance Testing: The Unexpected Missing Link in DevOps

    It is no secret that DevOps and continuous delivery goes hand in hand in the enterprise, but how do processes such as performance testing fit in to the equation?

    Many enterprises still harbor the traditional way of thinking—that the QA and performance testing group is completely separate from the development group. A widely held myth is that each has different roles, responsibilities, backgrounds and areas of expertise. However, with DevOps gaining popularity in the enterprise, the distinct separation between testing, development and operations is no longer. In fact, to successfully implement performance testing, the once-siloed departments must work together.

    Creating a Culture for DevOps

    The most common purpose of DevOps is to have an automated pipeline of software and application deployment updates. An enterprise cannot implement DevOps unless all are on board, from the top down. To do this, there must be buy-in from not only the software engineers and technology buyers, but also the C-suite. This ensures DevOps processes and strategies are being implemented appropriately.

    However, it’s not just about getting the tools needed for DevOps; there must be a culture shift toward collaboration for it to be effective. That is when an enterprise gets the most from the shift to DevOps and becomes the most agile.

    Once DevOps is embraced, experts and power users can help facilitate how it is being used to drive maximum benefit and help the organization get more from it. The key way to determine and evaluate this maximum benefit is through continuously testing throughout the development process. While the first thought may be for “functional” testing (to ensure code operates without errors and to provide fast feedback to developers when things go wrong), no organization can afford to release applications that deliver slow user experiences, and so “performance” testing with fast feedback is equally critical. Continuous testing is only scalable and sustainable if it is automated.

    DevOps Makes Load Testing Easier

    A key benefit of DevOps is that automation makes everything—from implementation to functional and performance testing—faster and more predictable. Tool chains ensure that the “right things” are done with every code push. A concern commonly cited by those who have not yet implemented performance testing is that it is “too time consuming” and they will be unable to facilitate different tests at the rate they need to be done to be effective. But surprise! Organizations that already practice DevOps are in a better position to add performance testing to the mix, in a fully automated fashion. A solution like BlazeMeter takes just minutes to get API tests up and running, by just adding the HTTP transaction for specific pages or endpoints to test. Once set up, these tests will continuously run automatically and will notify the team when there are any problems or potential problem areas. When there are known surges in website traffic, more Uniform Resource Identifiers (URIs) can be added to the test, but this will not impact the autonomy of it.

    Another key function of DevOps is scalability through the enterprise, and scalability of user traffic is extremely important. While performance testing would still scale without DevOps, operating on the same principles makes each concept more relevant to the folks who are analyzing the outcome of both. The key is to integrate performance testing directly with an already existing continuous delivery or integration pipeline, such as Jenkins or CircleCI. Here, fast, small-scoped performance tests are able to run in parallel to ensure performance is as expected before promoting a build to the next stage. Tests run in staging or QA environments, such as IBM BlueMix or Chef, can also be automated and integrated.

    Performance testing is also important not just because of its easy integration into DevOps practices and how it fits into the CD pipeline, but the overall customer experience as well. A properly functioning webpage or mobile application is the connection between your brand and your target customer, and if it doesn’t load properly or in a timely manner, you may lose a customer to a competitor. Performance testing ensures the user doesn’t experience slowdowns or the dreaded downtime, which is often the result of a poorly functioning back-end system. Incorporating performance testing into workflows creates a centralized environment where all team members have what they need to design, run or analyze tests, while communicating quickly and clearly with each other and everybody in the enterprise.

    Performance testing and DevOps are not opposing forces; and performance tests run automatically within the tool chain save time, rather than consuming more. The reality is that during this age of rapid software delivery, testing is the glue that keeps the development process on track and high quality. Performance testing should be part of every software delivery workflow, and DevOps is allowing that to happen.

    About the Author/Michael Sage, Chief Evangelist

    unnamedMichael Sage brings a 15-year career of enterprise software pedigree and customer success to BlazeMeter. Prior to BlazeMeter, Michael (better known as “Sage”) was a solutions architect and consultant in software delivery and performance management. He has worked with industry-leading companies including Mercury Interactive, Hewlett-Packard and New Relic, helping teams implement integrated solutions that enhanced the performance and experience of their customers. A self-taught hacker and open source evangelist, Sage always has the pulse on the cultural zeitgeist inflected with a heathy dose of humor.

  • Making the Case for Secure, Defect-Tested Software Development

    Making the Case for Secure, Defect-Tested Software Development

    Creating a software security initiative in any organization is no easy feat. Often times, organizational culture or politics can provide development managers with a strong counterargument for implementing software security concepts. Unfortunately, building software without a consideration for security has become a less viable option given the increase in compliance pressures and widely publicized data breaches leading software consumers to expect more security from developers.

    There is no way around the fact that attacks have become increasingly more sophisticated and applications have become the central avenue of these attacks. As a result, organizations are increasingly confronted with the task of assuring secure, defect free, software development. Leaders within these organizations are finally coming to realize that vulnerable and defect rampant software undermines the productivity, privacy and security of their businesses, employees and consumers alike.

    As organizations are beginning to accept the fact that there is a need to test for best practices and defects throughout the development lifecycle, it is important that they are provided with the resources to make this happen. To change the way large organizations build software, an enterprise-wide initiative is required. At its core, an initiative to change the way an organization tests and validates its development processes and practices are a fundamental business process improvement effort.

    Typically, large organizations with unique operational requirements find themselves building custom software systems to address their specific needs. But building custom software on time, on budget, and without bugs is difficult. Building software that also complies with an organization’s software security policies presents an even more difficult challenge. Although the tools and practices to build secure software are maturing, most organizations find internal hurdles to be more daunting. Organizational impediments including culture, differing software development approaches, and short-term business drivers make it more difficult to effect meaningful change. In order for a software security initiative to be successful, organizations must take a phased approach that considers organizational culture at each step.

    Sure, there is the option of treating the symptoms by deploying web application firewalls or running an automated scanner against applications, but this does not get to the root cause and solve the problem for most organizations. It highlights that there must be deeper process improvements involving the systems development life cycle, particularly for higher level business logic or authorization vulnerabilities.

    Despite the hurdles that exist when approaching the issue of secure development, most organizations do realize that there is a problem that needs to be addressed. Organizations focus on technical means to write more secure code and strategies for putting controls around the software. The next step is to then educate executives on the process of leading a software security initiative as these initiatives are most likely to fail due to organizational issues, not technical issues. This means taking a disciplined approach by characterizing the landscape, securing champions, defining standards and strategy, executing, and then sustain the effort. These steps, tailored to the way an organization operates, will help ensure that corporate-wide efforts to secure applications are as productive as possible.

    So in conclusion, it is not an impossible feat to address security at the root cause, software development, as part of standard operating procedure. It just takes a little forward thinking to demonstrate its undeniable value and get the corporate buy-in needed to change the development culture. That being said, once the culture is changed don’t rest on your laurels. Always observe what’s occurring within your organization to determine whether there are new technology risk areas emerging. If you simply focus only on well trodden areas like web applications, you may find yourself behind the 8-ball once again when it comes to new advances in software development, like mobile and the Internet of Things.

  • An obituary to a Managed Test Service

    An obituary to a Managed Test Service

    1st October 2020, Enterpirseborough , F500 Inc street

    Managed Test Service, 66, died Friday, 3rd October, 2020 at the Delivery governance war room in the IT head quarters of F500 inc. following a terminal illness caused by a lifetime burden of 34 SLA’s, debt burden of 44 KPIs , 29 painful claw backs by procurement due to failed deliveries and overdose of turnaround specialists and consultants

    MTS leaves 4 programs , 45 projects, 12 application centres and a number of small changes behind.Born and raised in the HQ , MTS was a lifelong resident of the borough.

    In the early years, MTS would help do functional testing for the core system portfolio and eventually taking ownership of non-functional testing and user testing. After graduating from the delivery transformation high school , MTS attended the SLA University, graduating with a degree in cost optimisation and skill super-speciaisation. MTS eventually joined the IT Board, where he was the voice of quality(or so he believed)

    His attention to detail and obsession with centralisation of anything that had Test in it’s name was legendary. He was proud and preached the virtues of , albeit bloated and inflexible, test service

    However,he struggled with newer delivery approaches of Agile and DevOps which spread like cancer throughout the delivery organisation. Devops challenged the fundamental idea of skill super-specialisation and independent testing that MTS championed and built his empire on. DevOps challenged the way of living of MTS as unsustainable 

    DevOps valued people and interactions over process unlike MTS which obsessed about process

    DevOps valued working software while MTS was worried about comprehensive documentation

    MTS got caught up in time-wasting contract negotiations instead of allowing his team to focus on customer collaboration

    MTS preferred a predictable plan while his delivery friends had already moved on with adaptable plans

    A funeral service was held on Monday 6th October from the funeral Home of Procurement , with a private burial.

  • How do you identify gaps in your testing?

    How do you identify gaps in your testing?

    Every piece of software ships with some defects in it, but major gaps in your testing coverage can be disastrous, and they can occur for a variety of different reasons. Part of the problem with a typical root cause analysis is that it comes after the fact. Perhaps the customer has complained, there’s been a big outage, or the software quality simply isn’t where it should be. But reactive firefighting like this isn’t the most effective use of your resources.

    What if you could identify those gaps earlier in the testing process and dynamically shift strategy to close them before they cause major issues?

    The best possible start

    Lack of domain knowledge and an incomplete understanding of the purpose and intent of a piece of software is enemy number one. There are two ways you might consider mitigating this risk in order to get a good start:

    1. Automated Testing – Identify critical core functions and automate mundane repetitive tests in order to free up your testers to leverage their experience and domain knowledge to break the system.
    2. Exploratory Testing – Context driven testing encourages critical thinking and collaboration. Testers have to understand what the application is trying to do, they have to learn about the intent and direct their efforts at the most important areas.

    Making sure you build a solid foundation, understand the software, and define your scope at the outset can take time, but it will be time well spent.

    Communication and assessment

    The traditional approach with gap analysis is to start with a problem and trace it back. In theory, you should learn from your mistakes and avoid repeating them in the future, but it doesn’t always work that way. It’s also a great deal more time-consuming and expensive to identify and fix problems when they’re already released. There’s no reason you can’t apply the same logic from root cause analysis earlier in the process.

    The earlier testers are involved, the better. There should be wide-ranging discussions during the planning phase that include the development team, the business team, and the test team. This is an excellent opportunity to ensure that everyone understands the workflow of the new software, what it’s supposed to deliver for the end user, and where the priorities lie for validation and testing.

    By including testers from day one, you reduce the chance of misinterpretation and give yourself an opportunity to spot potential gaps or misunderstandings. But the communication and assessment has to be continuous throughout the project, the second it slips, you’re running the risk of a gap opening up. You need to make sure that communication, collaboration, and mitigation are an integral part of your process for each new sprint or cycle.

    Developing process oversight

    In order to ensure that your coverage is tight, and to immediately flag possible gaps, you need to have a good test data management strategy in place from the beginning. You need to be using analytical tools, like Jira, QMap, or VersionOne. Regardless of the tool in play, what you want is a complete record of all your manual testing activities. The ability to generate reports or visualizations of your coverage, and to analyze it from different business perspectives, is absolutely vital.

    Where are most of the defects coming from? How did you find them? How critical are they? Where are your efforts being concentrated? What topics are trending in tester’s comments? Are you seeing the same questions or recommendations popping up over and over again?

    There’s a good chance that you’re collecting this data anyway, but keeping an eye on all of it in real-time should enable you to strategize effectively about where to focus your resources. The best approach is never carved in stone, it should continually evolve throughout the development lifecycle. Rigorous and regular analysis presents an opportunity to identify gaps as soon as they begin to form, instead of waiting until they cause a problem and then tracing it back. It’s about taking a proactive approach instead of a reactive one.

     

    About the Author/Mush Honda

    fixed_Mush_Headshot-cropMush Honda is QA Director at KMS Technology, a provider of IT services across the software development lifecycle with offices in Atlanta and Ho Chi Minh City. He was previously a tester at Ernst & Young, Nexidia, Colibrium Partners and Connecture. KMS services include application management, testing, support, professional services and staff augmentation.

  • Testing Nightmares: Real-Life Frightening QA Experiences

    Testing Nightmares: Real-Life Frightening QA Experiences

    canstockphoto1396657In the spirit of Halloween, this past month testing experts from around the globe answered the call to join in a “crowd-sourced” article highlighting their most frightening testing QA experiences. These are real-life examples of testing gone wrong—leading to all sorts of mayhem in some cases. We allowed these testers to hide their identity, and to keep their organizations anonymous to solicit the most behind-the-scenes responses.

    In no particular order here are the top five Testing Nightmares, as told by the testers themselves:

    The Importance Of Making Sure Your Hosting Company Gets Paid

    By Ivar (from a medium sized company/ now acquired)

    The company I was working for at the time had bet its existence on signing an enterprise contract that it had already been in negotiations for more than year with. We were building a digital asset management system that controlled the licensing and delivery of assets to corporate devices. (Think ‘private app store’.) I had written backup tooling that wasn’t ready for use but was included, unused, in the branch that was being deployed. Ten minutes after the final deployment (in preparation for client review!) all of our assets disappeared. Nothing new in the deployment was related to asset storage except my code. There was an eight-figure contract on the line and we had no backups. Fueled by the mother of adrenaline rushes, I reviewed my work and remained convinced that my code was not the cause of the loss. I was already into investigating other possible causes when another developer informed our team that after calling our asset host, he discovered that we had neglected to update our billing credit card. The host had turned off access to all of our assets ‘to get our attention’. They succeeded, but at oh what a price…

    canstockphoto1397192

    Not The Best Way To Start As The New Test Automation Engineer

    By James Eisenhauer

    This happened my first week as the new and only Quality Engineer at a health and wellness startup. I was on a mission to get test automation created and show them what I could do for them! My first test was to invite all the members of a test company to a new company wellness challenge. It just so happened the company President’s test account was a member of that test company. I scheduled my tests to run at 5 a.m. At 5:38 a.m. we all received an email from the President that read, “Emails every few minutes telling me that QA wants me to walk by the damn river. Make it stop!” “Walk by the River” was a copy of a real wellness challenge we had in our test system. Something in my auto-retry logic had gone horribly wrong. First impressions are a bitch!

    Gaming Fail

    By Abe Chiu

    While working for a gaming app company, we pushed out a build with an engine update that reverted all of our users back to levels that were saved on their device. This as opposed to the one that was saved on our server side. (Example: If logged in to create an ID and was at level 50 on your device, that was the save that popped up even though on our server you had previously reached level 125). All of our users were affected. That’s not the worst part. Some of our users reverted back to level 1 when they were above level 100 and up. This all happened on a Friday and the build was pushed on Thursday night. On Friday we were only able to fix some issues with a small percentage of our user base but the ones that were reverted back to level 1, those were not able to be helped until the following Monday.

    The White Screen of Death

    By Anonymous

    I work as a Software Engineer in test at a media company. This company has its own video player that is embedded either as a single standalone video or a playlist. The player was re-done completely, and I was responsible for testing it before it went out to production. I tested the player as a single video and found a number of issues so the release to production was pushed out. After the issues had been fixed, I tested it again as a video and then gave my “go-ahead” to release it to production. The problem was I had COMPLETELY FORGOTTEN to test the playlist! We had figured that everything looked good and since the release had already been pushed off once, we made a decision to release it to production at the end of the day, around 5:30 p.m. I double-checked it on production as a single video and it looked fine. An hour later, when I got home, we started receiving urgent messages and emails that the playlist wasn’t working!!! You could only watch the first video and then it became a blank white screen! Since the playlists are embedded so deeply, this was a major problem. At 6:30 pm that night only one engineer was around and online but he wasn’t sure how to revert it. Finally after a very stressful hour and a half, he figured out how to do it and I verified that everything was back and working again with the old player. I still can’t believe that I had forgotten to test the playlists, which is such an important part of the feature.

    Global Cocanstockphoto1397050mmunication Breakdown

    By Joe Nolan

    We were trying to create a mobile on-line conferencing feature to include in our education app, much like Google Hangouts. Testing had gone well in controlled environments. Then we pulled all of the teams together to have a large group call using and testing the feature. We immediately had reports of crashes and all sorts of issues. It was impossible to talk over the people trying to diagnose the problems, while at the same time others were randomly pushing buttons to try the feature. Even worse, participants were located around the world with different networks and connectivity issues. It was worse than herding cats!

    About the Author/Sauce Labs

    unnamedSauce Labs is the leading cloud-based web and mobile application automated testing platform. Its secure and reliable testing infrastructure enables users to run JavaScript unit and functional tests written with Selenium and Appium, eliminating the time and expense of maintaining a test grid. With Sauce Labs, organizations can achieve success with continuous integration and delivery, increase developer productivity, and reduce infrastructure costs for software teams of all sizes.

  • Does Agile need continuous integration?

    Does Agile need continuous integration?

    Michael Oubre – Agile evangelist, practitioner and trainer, and Marc Hornbeek – DevOps continuous testing evangelist and thought leader, share their views on popular questions regarding Agile and DevOps from their respective viewpoints.

    If you are doing Agile do you need to do Continuous Integration?

    Michael: From an Agile perspective, let’s consider your company decided to take the plunge and switch its software development process to Agile, formed Agile teams, got them some training and set them up with a coach. The team have been scrumming for a couple of iterations but there’s not a lot of work getting finished, and many of the stories are having to be carried over from sprint to sprint to get the defects resolved.

    You might be tempted to start worrying about the quality of the code being written, or whether you have the right ratio of test engineers, but it might be more useful to look at your build/test cycle. One of the big benefits of Agile is the reduction of the time between the code being written and the identification of defects – the shorter that time is, the lower the cost is to get it fixed. Agile removes the process barriers, but if you only rely on unit testing and infrequent integration testing, you won’t achieve the results that you hope for.

    Agile teams will have a very difficult time of ever reaching high performance without the benefits of Continuous Integration providing the ability to deliver new code in an integrated, testable build to the Agile teams as close to “instantly” as you can manage.

    Marc: From a DevOps perspective development teams that are “doing Agile” are the primary customers of Continuous Integration together with Continuous Testing.

    Both Agile and DevOps emphasize the importance of incremental development processes rather than large, serial waterfall processes because incremental processes facilitate fast feedback and reduce process waste compared to long waiting periods between software creation and validation. In fact the DevOps “movement” was born out of a system test consultant’s frustration with the disconnect between some Agile development teams and separate operations teams. (Reference: http://itrevolution.com/the-history-of-devops/) . Today the words Continuous Integration and DevOps, together with Continuous Testing and Continuous Delivery are practically synonymous.
    The mere fact that there are plenty of successful Agile examples that existed prior to DevOps or Continuous Integration became a word tells you that Agile can indeed be done without CI, but clearly CI is a great complement to Agile and supports the goals of Agile while helping to tie it to operations. So the strict answer the question “If you are doing Agile do you need to do Continuous Integration?” is, well, “no” but to leave it at that would be incomplete, so a better answer is “ no, but CI is definitely recommended to complement Agile”.

    Bottom line is, from both Agile and DevOps perspectives continuous integration is really important to achieving performance goals. At Spirent we think Agile and DevOps work together to help teams develop and deliver quality software efficiently. More information about Spirent’s views can be found at the following link: Spirent.com/solutions/devops

    What do you think of these views and do you have other views?

    About Michael Oubre

    Michael Oubre is an expert in Agile practice, transformation and management and has spent more than 10 years as a practitioner, coach, trainer and evangelist. His long and eventful tech career has given him practical experience and insight into every aspect of building, deploying, supporting and managing software, systems and networks.

  • Digital Disrupters – Building a DevOps Leadership Profile

    Digital Disrupters – Building a DevOps Leadership Profile

    While the advancement of DevOps methodology remains fairly nascent for many organizations, an ever-expanding class of leading practitioners are evolving a wide range of related best practices.

    Beyond the well-worn concept of “unicorns” – the scant few organizations that stand as mythical examples of DevOps nirvana – this broader set of early adopters have begun to pull back the curtain on their efforts, lending further detail to the strategies that have bred this current model for success.

    Last week, industry analysts Freeform Dynamics published valuable research that highlights the ongoing work of such DevOps innovators, dubbed as “Digital Disrupters” for their accomplishments in pursuing larger digital transformation. The report “Exploiting the Software Advantage” – sponsored by CA Technologies, outlines a critical set of initiatives crucial to navigating the DevOps landscape.

    What are the specific hallmarks of those organizations leading the charge? Where are others struggling to catch-up? Many of the Freeform Dynamics survey results, based on 1442 respondents, reaffirm widespread DevOps observations, while other findings serve to shed new light, including:

    • Everyone builds software: Perception is, in fact, reality and to deeply engrain digital transformation, Digital Disruptors are increasingly identifying with this mantra. While 95% of leaders have committed to becoming more app-centric, only 51% of mainstream organizations have adopted this persona. Similarly, 84% of Disruptors directly identify as “software companies”, regardless of industry, compared to 40% among the masses.
    • Transformation is required: This point may seem obvious, but it certainly bears repeating. A whopping 92% of leaders have actively moved to break down traditional silos between Dev and Ops, with 51% of those organizations having already completed this work. By comparison, only 18% of mainstream organizations claim to have done the same, supported by 59% who at least share this more collaborative DevOps vision.
    • Commitment is critical: This may also seem like a no-brainer, but the numbers bear out this requirement. Some 77% of Disruptors have agile development methodologies in place, with 70% having “broadly” implemented DevOps activities. For mainstream organizations, these metrics rate at only 31% and 21%, respectively; leaders are going all-in and appreciating the benefits, and those that follow will need to do the same.
    • Experimentation is omnipresent: To garner the benefits of continuous delivery, embracing trial and error is a must. Among Disruptors, 93% recognized the need to “experiment and fail quickly on the road to success”, compared to only 58% of mainstream organizations. Some 95% of leaders have already shifted from fixed product cycles to continuous innovation, while only 56% of the mainstream have made this leap.
    • APIs are everywhere: Ubiquitous integration, both in-house and in connecting to third parties, has only increased emphasis on the API. Disruptors are resoundingly focused on API management, with 68% using APIs to integrate third party services into their applications, and 66% to offer external access to systems and data. Mainstream peers compare at only 26% and 24%, respectively, denoting a sizable API management gap.

    These are only a handful of the most dramatic and obvious areas where DevOps advocates have clearly re-architected their approach, separating themselves from those organizations that have yet to go all-in; it doesn’t stop there.

    As highlighted in the research, other areas of differentiation and dichotomy include cross-functional alignment, emphasis on digital channels, use of digital to drive core business efficiency, and even funding of digital investments.

    Beyond these key philosophical milestones that litter the pathway to DevOps success, the survey also highlights the vast array of strategic goals that these organizations have sought to accomplish. The results spotlight the massive range of significant business benefits that practitioners believe DevOps can deliver.

    Leading use cases include:

    • Improved customer satisfaction (49%)
    • Better workforce productivity (44%)
    • Improved customer experience (41%)
    • Improved competitive differentiation (38%)
    • Better agility / speed-to-market (37%)
    • Opening up of new markets (35%)
    • Creation of new business models (30%)
    • Creation of new routes to market (26%)

    If one were to take those outcomes out of context, unrelated to the impact of DevOps, it’s fair to say that they read like a generic list of leading business initiatives – further underlining the movement’s overall relevance.

    That’s not to say this is all accomplished easily. Report respondents detail hard won victories that required significant time and investment to achieve all this DevOps-driven advancement. After all, the notion of “disruption” doesn’t exactly evoke images of ease or comfort.

    Click here to read the report for yourself – there’s a lot more to be considered. Namely, how well your organization measures up in affecting digital transformation in its own right, and where it can take steps to move forward.