Tag: forrester

  • The Software Testing Metrics That Make Or Break DevOps Success

    The Software Testing Metrics That Make Or Break DevOps Success

    Organizations today are racing to develop software at a breakneck pace through efficient DevOps processes. But it’s important for them to avoid losing sight of the forest for the trees. If too much of that speed comes with quality sacrifices that degrade the user experience, the rapid software iterations hurt the business more than they help it.

    “Organizations adopt DevOps to accelerate the delivery of innovative software that creates positive user experiences,” said Clark Golestani, former CIO at pharmaceuticals giant Merck and current managing director of C Sensei Group. “If you start accelerating delivery without also rethinking your approach to quality, you will end up rapidly delivering updates that drive your customers to competitors.”

    This is why continuous and automated testing is so essential to DevOps success. But instituting effective and streamlined testing is easier said than done. Organizations must continually tweak what, how and when they test their software to achieve the optimal test coverage while introducing as little friction as possible into the software delivery pipeline. It’s a delicate balance, and the only way to achieve it is to keep meticulous track of metrics that evaluate how well the testing process is improving quality throughout the development life cycle.

    “As the saying goes, ‘You can’t manage what you don’t measure,’” said Andrew Magarie, senior market impact consultant at Forrester Consulting. “So firms need to be able to define and track the right metrics that can help enable quality at speed while minimizing risk.”

    Magarie and his team at Forrester recently conducted research into 94 different quality metrics frequently measured by DevOps organizations across four development stages: build, functional validation, integration and end-to-end regression testing. The study showed that the most effective test metrics that bubbled up across all of stages tended to clump into at least one of three categories.

    Risk-Focused Metrics

    According to Magarie’s report, the most mature and effective organizations fundamentally chose testing metrics that best helped them understand business risk posed by the software throughout the development process.

    “For this study, we use the following definition of business risk: Any application shortcoming that impairs the end user’s (or customer’s) expected experience and ultimately erodes confidence in the business,” he explained.

    For example, during the build stage mature organizations consider measuring the number of automated tests prioritized by risk as one of the most important assessments.

    “Tracking ‘unit’ tests prioritized by risk is the key. As the code base evolves, developers and testers need immediate feedback about change impact. This feedback is significantly more actionable if prioritized by level of risk,” wrote Magarie.

    As he explained, business risk will be different for every application, and the complexity of an organization’s architecture and transaction dependencies adds more variables to the equation. However, organizations need to seek ways to measure ‘relative risk’ across the software development life cycle (SDLC) if they truly want to take a disciplined approach to incrementally improving quality in their software.

    Contextual Metrics

    Forrester found that simple “counting” metrics don’t offer QA nearly as much value as contextual metrics. So simply tracking the number of tests will not provide teams with as much actionable data as the measurement of the ratio of requirements covered by tests. Approximately 69 percent of mature organizations say running functional tests and checking against covered requirements is a top measurement and 75 percent say the same about API tests run against functional requirements.

    Holistic UX Metrics

    The most mature organizations recognize that while application-specific or team-specific QA metrics are important, the ultimate goal is ensure the quality of the user experience (UX) across their entire interaction with an application. This means measuring the thoroughness of end-to-end regression testing is crucial. Approximately 70 percent of leading DevOps organizations say they believe the percent of automated end-to-end test cases as a top metric.

    — Ericka Chickowski

  • 5 Can’t-Miss DevOps Sessions at Jenkins World

    5 Can’t-Miss DevOps Sessions at Jenkins World

    In less than a month many Jenkins users and DevOps professionals will gather in San Francisco for Jenkins World, taking place Aug. 28-31. With more than 60 sessions to choose from, covering a wide range of topics, it can be difficult to narrow down your schedule. That’s why it’s important to go into the show with a goal in mind.

    For those of you specifically looking to hone your DevOps skills, here are five can’t-miss sessions that fall under the DevOps tracks.

    NOTE: devopsy.kinsta.cloud readers get 20 percent off their registration with the promotional code JWDEVOPSCUST.

    DevOps: Delivering on the promise of innovation, agility and velocity

    Forrester Analyst Rob Stroud will discuss recent Forrester research that confirmed organizations are leveraging agile development techniques, automated CI/CD pipelines, testing and release automation to deliver differentiated products. The interactive discussion will allow attendees to review their DevOps maturity versus that of the market. Stroud also will lead discussions of how DevOps is adding value within enterprises and how the adoption of containers and serverless computing will further drive DevOps adoption.

    DevOps Leadership Panel

    Attend this panel to hear from a variety of experts on topics such as software delivery challenges, business value realized from DevOps, building internal support and more. Panelists will include:

    • Rob Stroud, Principal Analyst, Forrester – moderator
    • Kolby Allen, AWS Architect, WatchGuard Technologies
    • Andrey Falko, Principal Engineer, Salesforce
    • George Swan, Director of Engineering Solutions, Autodesk
    • Stas Zvinyatskovsky, Engineering Leader and Architect, Accenture

    Mapping Your DevOps Journey with the DevOps Quadrant Maturity Model

    In this session, Brian Dawson, DevOps evangelist at CloudBees, will map the DevOps journey to the DevOps Quadrant Maturity Model. He’ll go through the characteristics of organizations within each quadrant and present strategies for moving to the next stage of DevOps maturity. The DevOps Maturity Model is a great way to assess your current state of DevOps.

    Banking on DevOps: How Leading Financial Services Organizations Are Driving the DevOps Transformation

    Hear from financial services executives about how their organizations are embracing DevOps practices to speed innovation and deliver more value to customers. The financial services industry faces specific security, governance and regulatory hurdles that can prove challenging when changing processes and technology. This session will be moderated by Andre Pino, VP of marketing at CloudBees. Speakers will include:

    • Brock Beatty, Director, Software Engineering, Capital One
    • Karthrik Loganathan, Director, Business Technology, ESQM, Discover
    • Stefan Simenon, Head of IT Tooling & Software Development, ABN AMRO Bank N.V.
    • Moied Wahid, Director, Software Development, PayPal

    Leading a DevOps Transformation

    In this session, Jez Humble, CTO at DevOps Research & Assessment LLC (and also a Jenkins World keynote speaker), will discuss how implementing DevOps is more than just implementing tools. He’ll walk through how a DevOps transformation requires a different way of thinking about everything from budgeting to management to compliance. He’ll also break down exactly what DevOps is and isn’t.

    Don’t miss out on the hundreds of DevOps leaders and trailblazers to learn from at Jenkins World. You can view the full Jenkins World agenda to see what other sessions can help boost your DevOps skills!

    If you haven’t registered yet, make sure you to do so today with the promotional code JWDEVOPSCUST.

    — Alan Shimel

  • Compuware Survey Connects Mainframes, DevOps

    Compuware Survey Connects Mainframes, DevOps

    Mainframes are still necessary; integrating them with modern hardware and software is tricky; and DevOps is the answer. Those are the takeaways from a survey out this week that was commissioned by Compuware.

    The survey, which was released Dec. 14, was conducted by Forrester. It asked professionals how often their organizations rely on mainframe systems and what they do to overcome the challenges associated with using legacy mainframes within modern computing environments.

    The survey suggests that mainframes are not going anywhere soon: 57 percent of respondents said new business initiatives “always” involve the mainframe, according to Compuware. Even though mainframe systems are now decades old, they still have a place in modern IT.

    But bridging the gap between mainframes and more modern infrastructure and software delivery practices is tough, according to the survey. This is due not only to what Compuware calls “organizational bureaucracy” and “insufficient funding,” but also the difficulty of integrating mainframes with other types of technology that were designed in a different era.

    The survey shows that most professionals take one of three approaches to solving that challenge:

    • Move applications off of mainframes. The downside to this strategy is that mainframe applications may not run as fast or as securely on other hardware.
    • Avoid using the mainframe at all. This, too, is not always ideal, according to Compuware, because the mainframe is better at handling some types of workloads.
    • “Modernize” mainframes, which entails a lot of investment and effort.

    According to Forrester’s conclusion in the study, a better approach is to apply DevOps techniques to mainframe development. That means that the mainframe technology itself does not have to change or be hybridized; only the approach to mainframe app delivery changes. “Those companies embracing DevOps deliver faster and at a higher quality, all while fostering collaboration,” Forrester concluded.

    These are all things that a Compuware-sponsored study is likely to say, of course. Compuware specializes in DevOps solutions for the mainframe.

    Still, the survey is a good reminder that mainframe is alive and well—and DevOps teams should not leave it out of the picture.

    — Chris Tozzi

  • Murphy’s DevOps: 7 Habits of Rugged DevOps

    Murphy’s DevOps: 7 Habits of Rugged DevOps

    Attendees of Rugged DevOps Connect at RSA packed Moscone West’s second floor Feb. 29 to immerse themselves in security. Among the presenters of the daylong event was Forrester analyst Amy DeMartine, who provided a tutorial on how security, developers and operations folks can work together better in the new DevOps world.

    Through a slide show, DeMartine laid out her Seven Habits of Rugged DevOps, three of which are discussed in this column:

    1: Increase Trust and Transparency Between Dev, Sec and Ops

    We live in a world where technology’s goal increasingly is to automate everything, with less reliance on humans—at least, that’s what many marketers would have us believe. However, DeMartine emphasized that Security and Risk groups, if they are going to play a key role in DevOps, need to speak to the practice’s true owners—App Dev and Infrastructure and Ops—in a language they can understand. Here, it’s not automation that matters most. It’s communication—among actual human beings—that’s paramount.

    She emphasized a real disconnect between these groups, with AppDev being perceived as the department of “Anything Goes,” Infrastructure and Ops thought of as the division of “No,” and Security and Risk seen as the “Persistent Nagging” sector.

    7 Habits 4
    Imagine for a minute, a family unit comprised of Mom, Dad and a highly motivated teenager. The teenager is fast, full of energy and always pulling the trigger; Mom is pragmatic, a disciplinarian always having to tug at the teenager’s reins; and Dad is constantly frustrated that no one listens to his determined warnings. The family’s living room is the scene of many “heated” debates, where each side scores its share of wins but there are really no common agreements that satisfy everyone’s needs. When the teenager wins, it’s because Mom simply doesn’t have the energy to resist any further. When Mom wins, the teenager turns on the iTunes, plugs in the headphones, withdraws and possibly even starts searching for a new family. Dad’s victories are often followed by a series of door slams and the silent treatment.

    Now imagine if the three sides found a common language? One that not only got each to see the others’ viewpoints but also developed a system where the teenager was set free to operate within defined boundaries, and where he or she even wore enough safety gear to keep risk to a minimum. We’d have a much more harmonious place, one where the teenager continued to grow without risking the family ship.

    Interestingly enough, there really isn’t any technology available that addresses this very real human communication challenge. No mater how counter intuitive it may seem, an effective Rugged DevOps environment is highly dependent on human-to-human relations.

    3: Discard Detailed Security Road Maps in Favor of Incremental Improvements

    The U.S. Office of Personnel Management (OPM), Sony, Hollywood Presbyterian, Anthem—each of these organizations experienced a security breach of massive scale. If these breaches all shared a word in common it would be PANIC—or at least headlines would have us believe this. The OPM breach was the ultimate example, wherein almost as soon as the alarm bell rang the government announced an elaborate plan followed by the big “Sprint.” Although, it might have been wiser just to beef up phishing defenses (smiley face).*

    It turns out that a detailed long-term plan may not be the answer to data breaches, ransoms and system shut-downs.

    DeMartine said outright that a comprehensive road map for addressing threats should be discarded in favor of a vision for real-time measurement and incremental improvements. Below is the “circle of life in security and DevOps,” as envisioned by DeMartine:

    7: Test Preparedness with Security Games

    Who doesn’t like competition? Not many. I think security folks and hackers are probably some of the most competitive people around; visit a Black Hat conference and watch as attendees intently capture flags, break out of handcuffs and hack signs.

    With that in mind, it makes sense that this is one of the seven habits. There’s little doubt that simulations will do more to prepare people for real-life situations—after all, practice makes perfect.

    Communication is a Rugged DevOps Essential Tool

    Increasing trust between Dev, Sec and Ops mandates effective and, likely, improved communication. Perhaps one of the biggest takeaways is that, for all the technological bells and whistles the industry throws at combating cybercrime, in the end an age-old skill—communication—is at the foundation of the most-effective security.

    *Phishing is assumed the most likely cause of the OPM breach; it is important to disclose that one of my clients, PhishMe, is a phishing defense solutions provider.

  • DevOps Connect at RSA: In Security, Choose Increments

    DevOps Connect at RSA: In Security, Choose Increments

    Earlier this week the worlds of DevOps and information security collided at the DevOps Connect Rugged DevOps event, held in conjunction with RSA Conference 2016. The security-centric audience got to hear from a number of thought leaders across the DevOps and infosec worlds about ways security pros and DevOps teams can work together and establish incremental improvements in application security, as IT picks up the pace of software delivery.

    Among the talks, Forrester Analyst Amy DeMartine discussed her team’s findings after studying the “best of the best” in implementing Rugged DevOps patterns. There are seven habits universal across organizations that are succeeding with this new way of developing and deploying secure applications, she notes.

    “Think of it as a fitness program, and the muscles that you are building up are those that strengthen application security,” she says.

    These habits included a number of obvious practices, along with a few surprises, including:

    • Increase trust and transparency between Dev, Sec and Ops.
    • Understand the probability and impact of specific risks.
    • Use the continuous delivery pipeline to incrementally improve security practices.
    • Standardize third-party software and then keep current.
    • Govern with automated audit trails.
    • Test preparedness with security games.

    All are good tips for getting security, dev and ops to work better at delivering code both faster and more securely. But the seventh habit probably raised the most CISO eyebrows and could have the most potential to effect meaningful change and collaboration:

    • Discard detailed security road maps in favor of incremental improvements.

    According to her, the “best of the best” organizations in Rugged DevOps have security leaders that make it a habit to ditch detailed security road maps in favor of a establishing broad vision and achieving incremental security improvements over time.

    As she related in her talk, she’s seen too many organizations develop extensive and extremely detailed multiyear road maps only to see them grow obsolete not only because of rapid changes in the IT environment, but also the constant pivots attackers make in their attack techniques.

    Instead, she said, security leaders have to take the same continuous improvement approach that their peers in development and operations have made in their shift to DevOps.

    “This is a mind shift change that has to happen,” she warns.

    To achieve good results, security leaders should be focused on “real-time, actionable measurements” to focus on improvements that matter, such as decreasing remediation time for vulnerabilities.

    “In DevOps, we call this Plan, Do, Check, Act, where you’re taking very small changes. Say you want to insert static analysis right after a developer checks in new code. So you’re going to go off and plan. The plan is to insert static analysis,” she says. “Then you’ll do—we’re going to look at tools, we’re going to create our rules. We’re going to check to make sure it works, if it improves things. And then we’re going to move on to the next thing—maybe it’s tightening our rules or maybe it’s adding another tool.”

  • Webinar: Application Release Automation at Scale

    Webinar: Application Release Automation at Scale

    Logo_EC_corp_Large_Web_RGB

    Organizations of all shapes and sizes depend on feature-packed application releases to keep end users productive and happy. As application complexity grows, managing deployments manually becomes more difficult and slows down the pace of business.

    Join Amy DeMartine, Sr. Analyst at Forrester research and Mike Hamrick, Software Configuration Management Lead at Lockheed Martin, servicing CMS Centers for Medicare & Medicaid Services, as they discuss:

    1) How Application Release Automation (ARA) solutions eliminate error-prone manual work, decrease risk and increase uptime

    2) How Infrastructure and Operations teams in large organizations overcome modern service delivery challenges

    3) Best-practices and tools for overcoming the challenges of ARA at scale


    Event recorded Thursday April 2nd, 2015

    Click here to register to see it On Demand


     

    Video

    Slides

    [seoslides embed_id=”a3d2f6e4e229″ script_src=”https://devopsy.kinsta.cloud/embed-script/electric-cloud-webinar/6046/” overview_src=”https://devopsy.kinsta.cloud/slides/electric-cloud-webinar/” site_src=”https://devopsy.kinsta.cloud” site_title=”devopsy.kinsta.cloud” title=”Electric Cloud Webinar” /]


  • Eight critical DevOps practices:  innovate, deliver, repeat

    Eight critical DevOps practices: innovate, deliver, repeat

    DevOps has been garnering a lot of interest for some time. Discussions and debates on forums like devopsy.kinsta.cloud are improving our collective understanding of the right approach and now we are seeing concrete proof points of fast-movers applying DevOps practices to disrupt entire industries with rapid-fire, digital-innovation based on Cloud, Big Data and Mobile technologies. I have personally been part of this transformation at IBM for several years and would like to reflect on what we have learned, and also share with you results of a recent study from Forrester Research.

    DevOps helps organizations address a critical business challenge – that of continuously capturing and responding to feedback (both internal and external) so we can rapidly and repeatedly turn innovative ideas into highly relevant, and desirable, products and services.  This involves not only development and operations, but all aspects of the organization, as it entails the collaboration across the entire organization including business and supply chain. Compared to single-direction,  traditional models, DevOps is bi-directional and is based on not just pushing something out to the customer but also on getting their feedback to make improvements.

    In a recent webcast, I discussed eight DevOps practices that speed up software delivery with Kurt Bittner, Principal Analyst at Forrester and a leading expert on agile and iterative software development. This webcast highlighted a recent Forrester survey on DevOps practices, cycle times and where enterprises are in terms of the maturity of DevOps practices adoption. 
    You can watch the webcast replay here to know more –
    ibm.biz/Forrester-talk

    Their survey of 600 IT professionals with application development responsibilities found that about a third of teams consistently deliver at cadences of one to three weeks and that the fastest teams generated higher business satisfaction than slower teams, an indication they don’t sacrifice quality to deliver fast. The study reveals how these teams did it: by adopting the following eight DevOps practices:

    8_DevOps_practices

     

    1. Minimum Viable Product – Deliver the smallest viable product or application and then improve it in subsequent rapid releases, based on feedback, and delivering value incrementally.
    2. Dedicated Teams – Form dedicated, cross-functional teams, avoid over-specialization and ensure they are not interrupted by side projects.
    3. Loosely Coupled Architecture – Use loose architectural coupling within and between applications to reduce complexity and enable delivery in small increments.
    4. Minimize Hand-offs, Maximize Flow – Eliminate unnecessary steps, delays and friction between steps to increase the flow of work.
    5. Deliver in Small Batches – Deliver in smaller batches helps to expose the most uncertain issues first and enables feedback on the most valuable usage models earlier.
    6. Transparency – Transparency into progress helps everybody to see where you are at any point in time and also visibility into risk without decreasing productivity.
    7. Eliminate Overhead –Eliminate the overhead of periodic and manual status reports and status meetings which absorbs valuable time and distracts focus from delivering value.
    8. Automate Testing using APIs – Shift away from heavy manual testing to improve delivery speed, quality, and testing accuracy, and dramatically reduce cost.

     

    The study refutes the myth that releasing fast requires you to sacrifice quality and accept higher risk. Adopting these eight DevOps practices, especially automated testing and delivering in small batches, does exactly the opposite:  small incremental releases are more thoroughly tested, more stable and lower risk.

    The best part about DevOps adoption is the flexibility to choose which practices to adopt first, based on each organization’s priorities, and then to incrementally adopt the rest of the practices as needed to optimize the software delivery lifecycle and accelerate innovation.

    To learn more about the survey and the eight DevOps practices, download the Forrester study along with the approach that you can follow to adopt these as per your organization’s current state of delivery cycles.

    About the Author

    ashokAshok Reddy is responsible for Product Management & Strategy, product Design/user experience and Business Development for Rational Software Division of IBM Software Group and for the IBM DevOps portfolio.  He is based in Raleigh, NC, USA.  Ashok’s role involves all aspects of global product delivery including setting product strategy and for product line management, establishing product roadmaps and partnerships for IBM DevOps portfolio.

  • DevOps makes coding more social, social coding makes better code

    DevOps makes coding more social, social coding makes better code

    One of the main principles of DevOps is to break down silos. What’s more like a silo than the image of a lone developer, shutting himself off from the world in order to code more and interact with people less? “I’m just here to write code and that’s all I do.” There’s nothing collaborative or DevOps about that scenario and unfortunately, it’s one that is all too frequent within organizations.

    Enter social coding. This is a concept that didn’t exist twenty, or even ten, years ago and one that continues to grow in popularity. At its essence, social coding is an approach to software development that puts the focus on formal and informal collaboration. There are many benefits to social coding but the most important is probably the opportunity it presents for mutual growth and learning. Developers are allowed to collaborate in an environment that fosters discussion, creation and sharing.

    When talking social coding, you can’t avoid a mention of GitHub. Take everything you love about your favorite social network and then apply those same ideas to a code repository. That’s basically what GitHub did, and continues to do, with much success. You can create a profile, follow or message someone, see what others are doing with their code (or yours) and even go so far as to subscribe to RSS feeds of activity streams. GitHub started to take off a mere five years ago and is now held up as one of the more successful examples of how social coding can be commercialized.

    Companies like GitHub and SourceForge are leading the way in revolutionizing how developers commit and share their code but the foundation of social coding still rests with code review in a public forum. As Mark Chu-Carroll puts it…

    “If you’re programming and you know that your coworkers are going to look at your code, you program differently. You’ll write code that’s neater, better documented and better organized — because you’ll know that people whose opinions you care about will be looking at your code. Without review, you know that people will look at code eventually. But because it’s not immediate, it doesn’t have that same sense of urgency, and it doesn’t have the same feeling of personal judgement.”

    What’s next for social coding? After it’s start in the open source community, this is one idea (like many others) that is heading towards the enterprise market. Companies are beginning to recognize that even internal projects can be managed and treated as open source efforts. From wikis to crowdsourcing, there are many ways that the enterprise market can take advantage of social coding. The end user is the one who stands to benefit the most from such a change.

    Dave West, a Principal Analyst at Forrester Research, has this to say about social coding:

    “We’re social animals, and the development of stuff tends to be social in nature. We think a better way of thinking about it is as social development. It’s not just about coding; it’s about enabling business change and delivering business value in a social way. It will be interesting to see how companies broaden the idea from social coding to social development. Social coding is a fundamental change.”