devopsy.kinsta.cloud

Tag: continous testing

  • Forrester Study Reveals Software Quality Metrics that Differentiate DevOps Leaders from Laggards

    MOUNTAIN VIEW, Calif., July 23, 2018 (GLOBE NEWSWIRE) — Results from new Forrester Consulting research reveal the 20 most important Agile and DevOps quality metrics that separate DevOps/Agile experts from their less advanced peers. The study, which publishes today, was commissioned by Continuous Testing leader Tricentis and was led by Forrester Vice President and Principal Analyst Diego Lo Giudice.

    The research examined 94 quality metrics that were classified into four categories. As a high-level summary, the research uncovered the following:

    • Understanding of business risk is the most important determining factor of DevOps and Agile maturity.
    • Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).
    • Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

    Detailed results and analysis are available in the complete report: The Definitive Software Quality Metrics for Agile+DevOps.

    Agile/DevOps Leaders vs Laggards
    The study also found that companies with the most successful Agile and DevOps initiatives do a number of things differently:

    • They are transforming software testing into Continuous Testing by adopting five core practices, such as automating end-to-end functional testing and integrating testers into cross-functional teams.
    • They are almost twice as likely to consider automating the software quality process to be a “critical business differentiator” (53% vs. 27%).
    • They are significantly more likely (by 23 percentage points) to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).

    “Organizations adopt DevOps to accelerate the delivery of innovative software that creates positive user experiences,” explained Clark Golestani, Managing Director of C Sensei Group and former President, Emerging Businesses and Global CIO of Merck. “If you start accelerating delivery without also rethinking your approach to quality, you will end up rapidly delivering updates that drive your customers to competitors. Automated, continuous testing is the gate that allows you to protect your organization from significant risk as you innovate at the speed modern business demands.”

    The Risk “Blind Spot”
    Risk is a prevalent thread throughout the various parts of the report. Most firms (80%) believe they deliver within acceptable business risk, but fewer than a quarter state that their QA and testing processes completely cover business risk. Moreover, just 15% of respondents say that their test suites reliably provide a good indication of business risk.

    Interestingly, although risk-related metrics do not rank high in overall popularity, Agile and DevOps experts measure them significantly more frequently than the non-experts do. In fact, Agile and DevOps experts consistently rank risk-related metrics among their top 3 most valuable metrics in different phases and categories.

    Additional Findings
    Other notable findings include:

    • Only 26% of firms with active Agile/DevOps adoptions are currently practicing the five core Continuous Testing practices shown to separate Agile/DevOps leaders from laggards.
    • Many CXOs and high-level decision makers are vastly overestimating the maturity of their firms’ Agile and DevOps practices.
    • Firms that rely on manual testing are more than twice as likely to report that testing is a bottleneck (59%), compared to those with more automated quality processes (26%).

    The complete Definitive Software Quality Metrics for Agile+DevOps report is available for complimentary download on the Tricentis web site.

    About Tricentis
    With the industry’s No. 1 Continuous Testing platform, Tricentis is recognized for reinventing software testing for DevOps. Through risk-based testing, scriptless end-to-end test automation, and the industry’s most extensive technology support, Tricentis breaks through the barriers experienced with conventional software testing methods. Our innovative technologies simplify testing for even the most complex enterprise applications—transforming testing from a roadblock to a catalyst for innovation. The result is accelerated software delivery speed, improved cost efficiency, and reduced business risk.

    Tricentis is the only vendor to achieve “leader” status in all three top analyst reports (i.e., the “Triple Crown”). This honor is based on our technical leadership, innovation, and a Global 2000 customer base of 800+ companies, including global enterprises such as Allianz, ANZ Bank, Experian, First Data, HSBC, Lexmark, Merck, Sentry Insurance, Starbucks, Telstra, UBS, Vantiv (now WorldPay), Vodafone, and Whole Foods. These customers rely on Tricentis to achieve and sustain test automation rates of over 90 percent—increasing risk coverage while accelerating testing to keep pace with Agile and DevOps.

    Tricentis has a global presence in Austria, Australia, Belgium, Denmark, Germany, India, Netherlands, Singapore, Switzerland, Poland, United States and the UK. To learn more, visit https://www.tricentis.com or follow us on LinkedIn, Twitter, and Facebook.

  • Clean Up Those Ever-Fail Tests

    Clean Up Those Ever-Fail Tests

    One of the points of DevOps is to streamline development and focus on what is important. Things that don’t raise to a high enough level of importance end up in the category of backlog, and their tests are normally disabled or marked as “ignore results” in CI/CD tools.

    With CI/CD, this is a good step in moving forward, because you have a list of the things that you haven’t gotten to yet, even if that list is not currently part of validation. The problem is that it can create the very kind of cruft that we adopted CI/CD to get rid of. Many organizations keep these tests in the queue for an indefinite amount of time on the assumption that they will come in handy when priority allows for time to address the issues at hand.

    It Isn’t Simple

    The problem is, of course, that some of those tests will never—and I do mean never—be resolved. Those are cruft, sitting in the way of productivity and cluttering up the build system. It can be a corporate decision that a given bug fix/feature request will never be implemented, but more often than not it is simply that the priority will never be addressed because there are always new features, more important bugs, etc.

    Purists will already have started sweating while reading this, because the goal is supposed to be to address every single test in the system, resolve every single bug in the system, and implement every change request that made it far enough to have tests written for it. But we don’t live in a purist world. We never have. I’ve argued with purists of every possible stripe in my time in high-tech, because reality isn’t pure. It’s interruptive and ugly and prioritized.

    So we have backlogs. And we have items in the backlog that won’t ever be addressed.

    Why Old Tests Matter

    Image of binThe real problem with tests that are backlogged for more than a few sprints is, of course, the changing code base. As an abstract example, you have a dialog that asks the user for some input. The tests assume the input values that are designed in at the time the test was written. Six months later, a higher-priority test comes through that changes what fields are available on the dialog, and what data is collected. The original test is marked as ignored, so when it fails that doesn’t impact things. But now the test is invalid. And still in your system. Multiply these two requests by hundreds or even thousands, and you have the same type of flotsam that DevOps was designed to resolve, except now it is part and parcel of the CI/CD system.

    Clean Up, Acknowledge and Move On

    So what is a productive team to do? Have a frank discussion about your backlog and what will/won’t ever be implemented. Business leaders can agree that some items just never will raise to the correct level of priority, and you can identify tests that have aged out of usefulness while sitting in the backlog.

    Set up a process to review them occasionally, to run parallel to the process that updates tests when assumptions change—just not as frequently.

    Then document the tests that won’t ever succeed, and delete them. They don’t belong in version control, even as documentation, because eventually and inevitably, they will be inaccurate representations of the system. And that can create confusion.

    Of Course It’s DevOps

    This is an extension of all that DevOps did to begin with: eliminate wasteful activity and non-productive items on the list of to-dos. You prioritized once and cut with a knife, this is the same process, operationalized for CI/CD. And it’s necessary.

    A slight change, another set of choices that are no harder than the others you’ve faced, and the code base (in particular the test base) is cleaner.

    Leaving you more time to keep rocking it.

    — Don Macvittie

  • Testing What’s Important: Keep Growing Your Test Base

    Testing What’s Important: Keep Growing Your Test Base

    This project has gone on soooo long…

    In most human endeavors that take a stretch of time to complete, the idea is latched onto and people become excited about being involved. It doesn’t matter if it is building a skyscraper, inventorying a massive warehouse or even going to war. People are intrigued by the possibilities and want to get started doing the thing. After a time, though, this enthusiasm wanes, and most often is replaced by a weary determination to get the thing done. The nature of the endeavor doesn’t have to change; the reality of its implementation just becomes the dominant factor.

    IT projects in general—and continuous integration (CI) projects in particular—are no different. Whether a new project or implementing comprehensive CI on an existing project, the beginning is the most exciting part. Soon the excitement wanes and achieving the desired level of test coverage and a consistent build success rate becomes the goal that is trudged toward. It is a rare team that reaches for 100 percent code coverage with testing on initial implementation. The key becomes “testing what’s important.” And that varies by team and project.

    But testing at all levels—unit, integration, smoke, system, regression—could be better on almost every project. That improvement can be done incrementally while running through sprints. While some of the testing is managed by QA, some is managed by the developers—and in a CI/CD environment, arguably more is managed by developers. They are more involved in the testing process in CI/CD, so are more likely to be more involved in test development/management.

    The point is, good enough coverage is just that. And there is a hidden cost in getting too carried away with testing every line of code for every possible input, but improving tests is more than simply cranking out more permutations. Improving existing tests, or writing tests for functionality currently poorly tested, is increasing testing.

    So, as you strive to make your code get better, strive to make your testing get better, too. Testing after the baseline is established is one of the first things to get minimum attention, even in automated test environments. But improving it reduces the risk of releases, and that’s a benefit to the team and to the company. So it’s worth staying up on.

    Unless you have my friend, who we will only call Pete, on your staff. Then, no matter the quality of your testing, no matter the number of tools and layers of testing, he’ll find a way to break your product within 5 minutes. That made him a good tester, made him a better developer and, now, a better architect. But most of us don’t have Pete, so work on those tests.

    — Don Macvittie

  • First-Time Quality for Software on a Blockchain Network

    First-Time Quality for Software on a Blockchain Network

    There is probably more trust in the world today than at any other point in history—not necessarily by choice, but because of our use of software to carry out almost every type of transaction. The use of blockchain networks brings visibility to those networks of transactions: Information about an asset that was once held by separate owners is changing to the lifetime history of an asset or transaction and is shared between all stakeholders.

    For software developers, this means ensuring the integrity of a blockchain network is paramount. To that end, three key interesting aspects regarding continuous testing are emerging:

    • New approaches to business networks require the ability to test supply-chain business processes and transactions.
    • Reliance on shared technology ecosystems requires the ability to test APIs from multiple perspectives and to simulate the behavior of consensus.
    • Increased dependency of APIs requires the ability to test integrations at scale without affecting distributed ledgers in which any interaction is immutable (unchangeable) and final.

    Project teams face an unprecedented demand for first-time quality while minimizing the impact of testing on the team’s delivery. To respond to this challenge, test teams are enhancing their capabilities so they can test:

    • Apps that integrate with a blockchain network;
    • Chaincode that manages the execution of transactions on a blockchain network; and
    • Provision and scalability of a blockchain network.

    In addition to these new capabilities, proven continuous testing techniques and tools can be adopted to help the team deliver quality software into their blockchain ecosystem.

    Test Entire Ecosystems

    Using a blockchain network means an organization is redefining the structure of its business networks and processes, connecting distributed supply chains with a single transaction backbone. Projects need testers to minimize the risk of business disruption by validating every business process works across all stakeholders on the blockchain network.

    Test State Early—Shift Left

    In a blockchain ecosystem, the new paradigm is state-based communication, not message-based communication. The behavior of the state of an asset is influenced constantly by market, internal and regulatory changes, so projects increasingly are adopting Agile and DevOps practices to respond as quickly as possible. Shifting testing to the left on an Agile project means the team can test early, and those tests can then be reused continuously. Providing early, iterative feedback on quality at the time of development means fewer problems are found late in the life cycle, where their business impact can be severe.

    Establish Technical Trust

    One of the great things about blockchain technology is the potential to create greater transparency throughout an ecosystem.  Trust is embodied in the transaction itself, which gives an important new role for digital systems of trust. Regulators and owners of a blockchain network must ask testers to validate the codification of contracts, compliance and certifications, which are then deployed into the blockchain network and embodied within the business transactions.

    Use of blockchain technology will fundamentally change business processes and enable new opportunities. Success depends on the level of trust stakeholders have in the blockchain network and the blockchain ecosystem.

    However, a blockchain ecosystem is an environment shared among different organizations, each of which has a stake in—and dependence on—the overall system. This means each organization must trust the other stakeholders to fully test any software that interacts with the blockchain ecosystem in a manner that will not impact the reliability of the overall system. As such, a blockchain ecosystem that has a culture and reputation of technical trust between the dependent stakeholders is more likely to be successful. Continuous testing brings trust throughout the blockchain ecosystem.

    To establish technical trust in a blockchain ecosystem, ensure these four key things:

    1. Your team can test when needed, not just when possible.
    2. Required technical environment works as required to support the business flows.
    3. Business processes and transactions run as expected.
    4. Solution scales to the expected usage.

    Benefits of Continuous Testing and ‘Shift Left’

    Enabling teams to shift testing to the left means projects can test when needed, not just when possible. Continuous testing means an organization can deliver capabilities faster, respond quickly to ongoing changes and realize the value of transforming applications to utilize the power of a blockchain network.

    Watch the webinar replay on how to implement quality checks as early as possible, run tests as often as possible and integrate automated testing with release management to minimize the complexity of deployment.

    About the Author / James Hunter

    James Hunter jpg

    James has spent a large part of his career architecting and implementing large-scale, distributed systems in the finance, public and defense sectors. He now leads a global business segment responsible for defining technology strategy then developing and implementing solutions for both IBM and clients. Mentoring and encouraging future technical leaders is important to James. He works with students of all ages and speaks in primary schools to universities to evangelize on the benefits of technology and the positive impact software can have on our society and the environment. He has a degree in Management Systems and he is a Chartered Fellow of the British Computer Society. Connect with James on LinkedIn/Twitter

     

  • DevOps Chat: John Harris, LeftShift on DevOps Consulting and Continuous Testing

    DevOps Chat: John Harris, LeftShift on DevOps Consulting and Continuous Testing

    This episode of DevOps Chat has us speaking with John Harris of LeftShift IT. LeftShift is based in London, but has customers the world over. He speaks with us about DevOps consulting and continuous testing. John has some great insights to share that I think you will find interesting.

     

    Alan Shimel:                Hi, everyone. Alan Shimel – devopsy.kinsta.cloud here for another DevOps chats. This episode’s guest is John Harris of LeftShift IT. John, welcome.

    John Harris:               Thanks, Alan. Good to be here.

    Alan Shimel:                Thank you. So John, what exactly is your role with LeftShift 

    John Harris:               My role is the Head of Continuous Delivery in DevOps with LeftShift.

    Alan Shimel:                Okay. And John, you know what? Some people in our audience are probably not familiar or may have not heard of LeftShift IT before. Why don’t you give us quickly a background of LeftShift?

    John Harris:               Yeah. We’re primarily a services consultancy in the continuous delivery – DevOps continuous testing space. We work quite a lot with IBM, a lot of other different vendors. Based in London and the UK, but we have a lot of our business offices based in the U.S. as well.

    Alan Shimel:                Sure. And John, before we jump into the subject of today’s chat, which is around continuous testing, I want you to take a moment and just talk about this burgeoning field of DevOps consulting. It seems to be exploding. I was in London for the DevOps Enterprise Summit last month and I was absolutely blown away with the diversity of firms from big to small that are offering DevOps consulting services. How long has LeftShift been in the DevOps consulting space?

    John Harris:               We’ve been in the area kind of on and off I guess you could say ten years. I guess most people would say DevOps has only been around since that original kind of DevOps coining days again in Belgium, but I guess a lot of people say the discipline itself had been around before the name, so I guess the processes around DevOps you could say we’ve been involved in for quite some time I guess. 

    Alan Shimel:                Sure. And John you know I don’t think we would be seeing this explosion of DevOps consulting if there wasn’t a wanting audience for DevOps consulting continuous delivery consulting services. Is that pretty much what you’re seeing with LeftShift? I assume business is good.

    John Harris:               Yeah. It’s one of those interesting areas where one of the things I guess the DevOps consultants could say is you don’t just want to create a new DevOps practice in your company. You don’t just want to bring in these pieces, but it’s a difficult loop because you’ve got to then support those processes and you’ve got to get that knowledge into the organization somehow, so I think our model is firmly to go and help people get started and gussy up their own practice and their own good processes and then kind of lead them to it, so yeah absolutely.

    Alan Shimel:                Got it. So John, let’s pivot a little bit here and talk about continuous testing. So, I will tell you that on devopsy.kinsta.cloud the whole notion of testing, continuous testing, automated testing is one of the most hotly debated and in some ways controversial topics within DevOps. Wondering if you see similar?

    John Harris:               Yeah. I think so. I mean I think traditionally in so far development, testing has traditionally kind of been seen as just the necessary part that you have to do at the end, but I think now that continuous delivery is becoming a real discipline and people are being forced to go faster and faster for getting ahead of the market and things like – can’t afford to see testing as a poor cousin of development anymore and I think testing is really becoming – people realize I can throw stuff out into production as quick as I want really, but actually it’s throwing it out with the quality that it needs to – that product to still be viable in the market. So I think absolutely. The drive towards testing is long overdue, but it’s definitely getting started. 

    Alan Shimel:                Got it. And John, can you have continuous without the automation though?

     John Harris:               Personally I don’t think so. I think that DevOps really – one of the biggest buzz words for me – DevOps and continuous delivery is really the organization. I think you have to have the process behind it, but I think when you go towards anything delivering more rapidly through all of these stages in our development life cycle in production, I think you just cannot scale at the size that we need to in order to get that stuff out without automating the process.

    I mean when you talk about the union it forms like Netflix and Etsy and all of those guys, now three, four years ago, there’s no way you can deliver that kind of scale without putting in the automated tool in and also the processes and the culture around that to enable that to where it will have the effect that you need.

    Alan Shimel:                Sure. So John, I think one of the controversy – not controversy, but one of the big discussion points around this though is what does this mean for the traditional Q/A role and for the people in the Q/A space?

    John Harris:               Yeah. I think teams have traditionally kind of been split. I mean obviously we have these silos and DevOps as kind of trying to address the silos of development and testing and ops and those kinds of things. I think again, testing’s one of the things that gets left out of that, but certainly in the customers that we work with, we’re seeing a lot of Q/A teams being retrained in certain ways, brought into the development side of the organization, having their practices changed to more of a development software engineering kind of way of working.

    We’re certainly seeing some of the geographic splits that have been traditional in this area kind of being broken down, perhaps testing, quite a lot of off shoring work in testing. Some of that’s coming on shore. Testers are becoming more technical but yeah, retraining and the different skills required now for the Q/A role, to bring that into more of a technical automation. That’s certainly a question a lot of our customers are grappling with right now and I think the decisions they aren’t just technical, but they go into the culture side of the company and how the company has organized itself. So yeah, it is a contentious area. I agree.

    Alan Shimel:                Yeah. It certainly is and it’s really one of those – for people in the Q/A space, it’s a pocketbook issue, right? They perceive a threat to a livelihood, even though I don’t think that’s the case. I think it allows them actually to soar and higher and do bigger and better things, but there’s certainly that element to it.

    John, I want to talk for a moment about service virtualization and we actually have a webinar coming up on I think it’s August 23rd where we’re going to looking at a real unicorn company – Mertz Shipping Lines, which is I think is 80 years old or something in traditional shipping, not what you would think of as your Google, Facebook kind of company. But they’re going to be talking about service virtualization, shifting left. Thoughts on that at all?

    John Harris:               Yeah. Service virtualization is an area that we work with quite a lot, especially around the IBM tool set. And yeah, Mertz is doing great work. Almost all of the organizations that we work that are trying to do some kind of parallel development, I mean one of the patents that we see is building the y and Rex API layers on top of some of their legacy – so business services and things like that and they want to do some parallel development with the ____, with the Rex, with the user in space layers. So SV’s really able to give those teams a really solid platform because everything’s based on their service interface definitions and it can really do some proper parallel development there and we’re seeing some great shifts left with those development teams and actually the ability to identify a whole bunch of defects before they would even go and integrate with each other.

    I think we had one team who identified over 100 defects in the early testing stages using service virtualization before they even would have integrated with the other services. They were able to get those fixed even before they went into their next integration environment, so that was a really great success story – service virtualization. 

    Alan Shimel:                Yeah, absolutely. I think I interviewed the Mertz guys at Interconnect last year and it was an interesting case study. I guess we should mention – there is a Service Virtualization for Dummies book that people can download and I don’t have the URL in front of me. I’ll try to include it in our notes. I think it comes out from IBM for people who are interested in this. But John, let’s return – what do you see – what in your role with LeftShift, what do you see as sort of the next frontier where the – not the battle, but where there are points to be made, where there are battles to be won for lack of a better word in the near term future?

    John Harris:               I think the biggest one really and we’re starting to see the shift now, but it’s still early stages and the custom is that – for the unicorns and the startups, this is probably not going to be a big bull, but certainly for enterprise – user interface testing is still very prevalent and we talk about the testing pyramid how most of our tests are one of the unit tests and then we’re gonna have a good base of integration tests and then a small percentage of UI test if you want.

    I think currently a lot of our organizations have one of them that’s flipped on its head because it’s easy to do UI testing and reason about it going to the business, but now when we go faster, I think the traditional focus on UI testing is going to have to go away, but when we talk about APIs on top of everything and things like the API economy and a lot of systems now just don’t have UIs. Coming out with a lot of systems that are all API driven and smart devices.

    I think everything’s going to have to be integration tested. I think the APIs and the logic behind them is really where the core business functionality is, so I think that’s where the importance is gonna be and it’s going to be moving organizations and showing them that integration testing and API testing is not the way to go and obviously automating that as part of their delivery pipeline, which is sometimes a little more difficult to do on the UI side.

    Alan Shimel:                Yeah. Quick question on that because I thought about how to deal firsthand with this issue. Shouldn’t the API testing – at least the tests themselves come from the same people who are writing the API? In other words, where’s the burden? Should the people who are providing the APIs also provide the testing for it or is that sort of letting the wolf in the chicken coop, right? Where you’re using someone else’s API. The burden is on you to make sure that that is in fact working with your system.

    John Harris:               Yeah. That’s a really interesting point I think and there are different kind of philosophies around it. Myself, we work a lot with the virtualization and the testing side of it, so we would say that if you’re developing the interface then you’re also in charge of developing the basic virtual service for that to integrate with your consumers, but I’m very much on the side of consumers developing the API test or the APIs that you consume, because if you think about the case where I’m a consumer, I may not need every field in an API. I may just use a few if I’m developing loosely coupled micro services let’s say.

    So I’m really in favor of each team who consumes an API developing their own tests and then contributing those tests to the API team so they can be run as call, every build or every deployment. So then if I change functionality and those tests fail, I know which team gave those tests to me and I can go and alert them, “Well, look I’ve made a change and I’m just going to break your implementation or your call of this API, but I may not have broken anyone else’s.”

    I mean I think putting the burden to test every piece of functionality, especially as we go to more loosely coupled services is kind of an unreasonable burden on the team developing that API.

    Alan Shimel:                Got it. You know what, John? Believe it or not as I told you I think before we went live, time goes really quick on these and we’re coming up on the end. If someone wants to find out more about LeftShift, where can they go?

    John Harris:               I think it’s LeftShiftIT.com and we’ve just revamped out website, so they can find out all about our services and products and everything we offer on there.

    Alan Shimel:                Okay. And then finally John, I wanted to end with the kind of question I guess I ask just about every guest, which is for our audience out there, if they’re gonna read, what should – if they –hopefully if they do read everyone should read books. What do you recommend is the next book our audience reads?

    John Harris:               I’m a big Jez Humble fan and I’m not going to recommend Continuous Delivery, because I’m sure everyone’s read that who’s listened to the podcast, but is Lean Enterprise book – How High Performance Organizations Innovate at Scale and it’s Jez Humble, Joanne Molesky and Barry O’Reilly is an awesome book for any enterprise looking to get into this kind of area.

    Alan Shimel:                Absolutely. I actually have it on my book shelf in the office. Good one, John. Anyway, I think that’s gonna use up our time here on this edition of DevOps Chat. John Harris, LeftShift IT, thank you very much for being our guest and we look forward to hearing more about LeftShift and continuous testing service virtualization going forward and continued success.

    John Harris:               Great. Thanks for having me, Alan.

    Alan Shimel:                Thank you. This is Alan Shimel for devopsy.kinsta.cloud and DevOps Chats. Until next time, have a great day.