devopsy.kinsta.cloud

Author: Ericka Chickowski

  • The Software Testing Metrics That Make Or Break DevOps Success

    The Software Testing Metrics That Make Or Break DevOps Success

    Organizations today are racing to develop software at a breakneck pace through efficient DevOps processes. But it’s important for them to avoid losing sight of the forest for the trees. If too much of that speed comes with quality sacrifices that degrade the user experience, the rapid software iterations hurt the business more than they help it.

    “Organizations adopt DevOps to accelerate the delivery of innovative software that creates positive user experiences,” said Clark Golestani, former CIO at pharmaceuticals giant Merck and current managing director of C Sensei Group. “If you start accelerating delivery without also rethinking your approach to quality, you will end up rapidly delivering updates that drive your customers to competitors.”

    This is why continuous and automated testing is so essential to DevOps success. But instituting effective and streamlined testing is easier said than done. Organizations must continually tweak what, how and when they test their software to achieve the optimal test coverage while introducing as little friction as possible into the software delivery pipeline. It’s a delicate balance, and the only way to achieve it is to keep meticulous track of metrics that evaluate how well the testing process is improving quality throughout the development life cycle.

    “As the saying goes, ‘You can’t manage what you don’t measure,’” said Andrew Magarie, senior market impact consultant at Forrester Consulting. “So firms need to be able to define and track the right metrics that can help enable quality at speed while minimizing risk.”

    Magarie and his team at Forrester recently conducted research into 94 different quality metrics frequently measured by DevOps organizations across four development stages: build, functional validation, integration and end-to-end regression testing. The study showed that the most effective test metrics that bubbled up across all of stages tended to clump into at least one of three categories.

    Risk-Focused Metrics

    According to Magarie’s report, the most mature and effective organizations fundamentally chose testing metrics that best helped them understand business risk posed by the software throughout the development process.

    “For this study, we use the following definition of business risk: Any application shortcoming that impairs the end user’s (or customer’s) expected experience and ultimately erodes confidence in the business,” he explained.

    For example, during the build stage mature organizations consider measuring the number of automated tests prioritized by risk as one of the most important assessments.

    “Tracking ‘unit’ tests prioritized by risk is the key. As the code base evolves, developers and testers need immediate feedback about change impact. This feedback is significantly more actionable if prioritized by level of risk,” wrote Magarie.

    As he explained, business risk will be different for every application, and the complexity of an organization’s architecture and transaction dependencies adds more variables to the equation. However, organizations need to seek ways to measure ‘relative risk’ across the software development life cycle (SDLC) if they truly want to take a disciplined approach to incrementally improving quality in their software.

    Contextual Metrics

    Forrester found that simple “counting” metrics don’t offer QA nearly as much value as contextual metrics. So simply tracking the number of tests will not provide teams with as much actionable data as the measurement of the ratio of requirements covered by tests. Approximately 69 percent of mature organizations say running functional tests and checking against covered requirements is a top measurement and 75 percent say the same about API tests run against functional requirements.

    Holistic UX Metrics

    The most mature organizations recognize that while application-specific or team-specific QA metrics are important, the ultimate goal is ensure the quality of the user experience (UX) across their entire interaction with an application. This means measuring the thoroughness of end-to-end regression testing is crucial. Approximately 70 percent of leading DevOps organizations say they believe the percent of automated end-to-end test cases as a top metric.

    — Ericka Chickowski

  • DevOps Security: 3 Privileged Access Management Best Practices

    DevOps Security: 3 Privileged Access Management Best Practices

    The tremendous upside of DevOps practices and tools are enough to keep organizations pressing forward at all costs. But when sloppy use of DevOps toolchains cause breaches, more than half the time it comes down to poor protection of privileged accounts. According to a recent study by security vendor Beyond Trust, 52 percent of IT practitioners say that overprivileged users are at the root of DevOps and other next-generation technology-caused breaches.

    If organizations are going to reap the biggest benefits from DevOps without putting their IT infrastructure and data at risk, they’re going to need to think more strategically about how they handle privileged access management (PAM), said Morey Haber, CTO at Beyond Trust. devopsy.kinsta.cloud recently caught up with Haber to break down three of the most important PAM best practices for DevOps teams to build into the continuous delivery pipeline.

    Discover and Inventory All Privileged Accounts and Assets

    You can’t manage accounts and assets you don’t know about, so this step is foundational to the whole process of PAM. Unfortunately, with so many scripts and so much automation layered all over the DevOps toolchain, it can be tremendously difficult.

    “If it’s embedded in other runtimes or it’s hard-coded into compiled executables, that discovery is the hardest,” Haber said. But it must be done. Organizations need to get clear visibility into exactly what tools are executing the automation and what the privileges are assigned to them. “Is it a power-shelf script? Is it embedded in Jenkins or somewhere else? What is actually running? Who’s running it and when?”

    Additionally, organizations need to understand where the automation is stored and, consequently where that embedded credential information is being stored, so it can be examined for tamperability and safety of the credentials.

    “Now, this isn’t something that’s easy to discover in any of the three contexts, but it has to be,” Haber said. “So, one you have to understand the entire process and where you’re actually embedding privileged credentials for the edit, storage, operation and scripting itself.”

    Manage Shared Secrets and Hard-Coded Passwords

    Hard-coded passwords are one of the biggest no-nos in credential management. Unfortunately, even when application security teams are meticulous about rooting hard-coded passwords out of their finished applications, they often leave them within the IT infrastructure that helps support the development of that software for the sake of expedience. Same goes for account sharing, which is a frequent mistake organizations make to just get the automation working and keep it working with stability.

    The problem is, this trashes any semblance of traceability or auditability of activity within the affected environment.

    “Whether that is using some type of proxy technology, a VDI environment—anything, you don’t want people logging in directly with shared accounts because you can’t trace that,” Haber said, noting there needs to be a control plane in the mix that can keep track of both individual users and script or automation accounts that are impacting the environments to track each process or activity against unique credentials. This is essential not just for compliance, but for forensics and the overall integrity of a DevOps team’s software “factory.”

    Enforce the Rule of Least Privilege

    Ultimately, strong PAM depends upon only giving individual users or specific automation accounts the exact amount of privileges they need to get their job done.

    “We see credentials being given to DevOps teams all the time that are way, way overprivileged,” Haber explains “It should come down to the least privileged model—that you’re giving enough privileges for the DevOps process to work, but not overprovisioning so that if any process or account is compromised, it doesn’t jeopardize the rest of the environment.”

    — Ericka Chickowski

  • 10 Views on What Microsoft’s GitHub Deal Does for DevOps Users

    10 Views on What Microsoft’s GitHub Deal Does for DevOps Users

    The DevOps and open source communities have been in a tizzy all week over the massive $7.5 billion Microsoft acquisition of GitHub. Even as many developers rage on Twitter about the deal and still others are rushing to migrate their repositories to alternative solutions—#movetogitlab has trended heavy this week—for the most part the DevOps community is taking it in stride. As devopsy.kinsta.cloud’s Don MacVittie explained, complaining or planning a move away from GitHub solely due to long-held grudges against Microsoft seems pretty counterproductive at the moment.

    He’s not alone. A lot of developers, evangelists, CTOs and other GitHub users out there doing the work that drives applications into production are looking at this as a potentially great thing. Others are more cautious, but at least believe this won’t set the movement backward. We’ve gathered a few thoughts from these professionals to help balance out the dialogue.

    Ensures GitHub Viability

    “For those developers that may be anxious and thinking about jumping ship, I am sure they also must realize the hard truth that for GitHub to succeed moving forward it needed an influx of cash. Inevitably, there would come a point where GitHub would need to start to actually turn a profit, something it has yet to achieve, before it would have to shut its doors. ”

    —Adam Mansfield, Microsoft practice leader at UpperEdge

    Provides Much-Needed Leadership

    “Satya Nadella has so far done a great job dropping the Windows religion to embrace the reality of the iOS, Android, Linux and multi-cloud world, which he will hopefully continue with the GitHub community. By putting Nat Friedman—former CEO & co-founder of Xamarin—in place as a technical CEO, Microsoft is sending a clear message that they’re committed to GitHub and the larger developer ecosystem.”

    —Jyoti Bansal, former CEO of AppDynamics and current co-founder of DevOps platform Harness

    GitHub Champions in Enterprise Have Easier Sell

    “For one, I wouldn’t be surprised if we start to see corporate firewalls allowing access to GitHub now that it’s a Microsoft owned property, and perhaps considered ‘enterprise-ready.’ Microsoft knows how to make products that make enterprise architects feel confident, so their blessing and ownership of GitHub could really encourage social coding within enterprise organizations.”

    —Matt Stratton, DevOps evangelist at PagerDuty

    Net Neutral Immediate Impact

    “Developers should not be anxious about this acquisition nor should they be excited. I honestly think this is an ‘oh well’ moment. Unless Microsoft has some crazy plan to sunset features or support for one vendor or another, everything should behave as business as usual for the foreseeable future. The only thing developers should watch is license fees and terms and conditions. Expect Microsoft to monetize the solution better in the coming months and potentially tie usage into MSDN.”

    —Morey Haber, CTO of BeyondTrust

    GitHub Already Had Its Own Problems

    “GitHub is not strong in this area (DevOps) and never has been; maybe if Microsoft made it easy to deploy to Azure it could be seen as an improvement. As a tangent, GitLab which is now the attractive alternative, has its own CI/CD tooling and can also be self-hosted for free. This is a major advantage over GitHub, and I’m wondering if corporate America will start looking more closely to the FOSS alternative to GitHub Enterprise.”

    —Mitch Pirtle, longtime developer and consultant at SpaceMonkeyLabs, former Tech Fellow at Capital One

    Opens Eyes for About Dangers of Centralization

    ” My hope is that – one way or another – this leads to greater competition, more innovation, and less reliance on any one single platform. I don’t feel that centralization around any one platform is good for the DevOps and FOSS communities. I also don’t feel consolidation to larger companies is a good thing, as it creates conflicts of interest and generally a lack of innovation. This consolidation could result in a more decentralized approach to both software development and DevOps integration, which will hopefully drive more innovation. I think that could ultimately be a positive outcome of this acquisition.”

    —Jeremy Steinert, CTO of WSM International

    Expect Better Integration for More Seamless CI/CD

    “Longer term, I expect GitHub’s integration with Azure to expand substantially, enabling some truly exciting developments for one-click code deployment to a Cloud infrastructure. DevOps for smaller companies will gain easier access to Azure resources. For DevOps as a whole, we will see acceleration of seamless code management within the context of actual deployment to containers. DevOps for enterprise has the greatest potential for improvement with this deal. There will be massive investment in improving developer interactions between infrastructure and code.”

    —Clint Wilson, VP of product management at DigiCert

    Here Come Better Enterprise DevOps Features

    “While I think the development community will have mixed emotions about the news, ultimately this is a massively positive moment for GitHub. GitHub is at the center of the developer community, but as a business, they have faced challenges in reaching the enterprise buyer—something Microsoft is extremely well-versed in. With Microsoft’s resources and expertise, GitHub no longer has to wrestle with the internal struggle of being open to the community and being open for businesses, ie. selling to the enterprise. Assuming it’s kept a separate entity and is not beholden exclusively to Azure, this acquisition gives GitHub the opportunity to deliver the features enterprise DevOps and development teams need.”

    —Tal Weiss, co-founder and CTO of OverOps

    Accelerates GitHub’s Servicing of DevOps Market

    “The new Microsoft is not the bogey of the past — VS Code is great; TypeScript is terrific; Microsoft has not merely accepted open source, but become an open source leader through its GitHub contributions. From a business perspective, I think the deal will pay off. Unlike competitors such as IBM and Oracle, Microsoft not only understands developers, but has a proven ability to sell to the SMB as well as to the enterprise market. Microsoft can probably grow the GitHub business substantially by executing better on GitHub’s existing strategic objectives, such as GitHub Enterprise, without alienating users. It may also be able to create significant value by applying its AI technology and VS Code innovation to GitHub’s enormous corpus of code.

    —Rod Johnson, co-founder and CEO of Atomist and creator of Spring

    Validates Value of What Open Source and Devs Do

    “I think overall this is good for the DevOps movement, and open source in general. It’s a very strong validation of how important open source is not just to open source developers, but also the business world. Developers have a right to be anxious, but there are many viable options for open source hosting in this day and age, so if Microsoft does not prove to be a good steward of GitHub, they have other options. However, Microsoft could very well keep GitHub going.”

    —Logan Abbott, president of SourceForge.net

    — Ericka Chickowski

  • The DevOps Management Perception Gap

    The DevOps Management Perception Gap

    The DevOps momentum is fierce in the enterprise today, as executives and middle management recognize the transformative power that continuous delivery of software has for a business. The message that DevOps can lead directly to increased profitability for organizations has seeped into technology leadership consciousness and tech leaders are gung-ho to reap the rewards.

    The problem is that their enthusiasm isn’t necessarily translating into a meaningful enough change in leadership patterns to really optimize the benefits of DevOps. The “2018 Global Developer Report” released by GitLabs recently offers some meaningful evidence of this. Study results hint at some Pollyanna-ish attitudes by managers that potentially could be leading to them resting on their leadership laurels.

    The GitLab study showed that managers are overestimating their DevOps teams’ performance as compared to the developers who are on the ground getting the work done. While 81 percent of managers say DevOps saves time in the development process, only 65 percent of developers agreed with that statement. There was a similar disparity when it came to awareness of speed of delivery capabilities: Approximately 47 percent of managers said their organization deploys code on demand or multiple times a day while only 39 percent of developers said the same.

    This gap could point to a potential disconnect growing within leadership that’s unable to adjust their management style to DevOps’ democratization of everyday development responsibilities. The study shows that almost a third of managers admit they don’t have visibility into what others are working on.

    While individual teams do need to be given the autonomy to get work done with less red tape in the way, leaders still need to provide companywide direction and strategic supervision. In the end, teams need to come together to meet global business goals, and that’s where managers need to step in.

    “What you need your leadership to do is apply systems thinking, and that’s a whole another subject we could get into, but I don’t think executives and leaders know how to really address the, ‘How do I optimize this as a whole?’” Carmen DeArdo, technology director for Nationwide Insurance, recently told devopsy.kinsta.cloud. “You need to really focus on the strategy and make sure you’re removing barriers and allowing teams to innovate and improve your delivery capability.”

    Unfortunately, the GitLabs study proves out DeArdo’s point, showing that leaders are missing their mark in this regard. The survey showed that “unclear direction” is the top challenge named by developers in getting their work done. It was cited by 67 percent of developers as their biggest impediment.

    Meanwhile, managers may not be aware that they’re not providing proper direction. Managers are 35 percent more likely to say they set project expectations up front than developers are to say they receive such requirements right out of the gate. In fact, only half of developers agree that they receive ample expectations and requirements from the start.

    — Ericka Chickowski

  • Open Source: Is Your DevOps Org Vulnerable to an Equifax-Style Hack?

    Open Source: Is Your DevOps Org Vulnerable to an Equifax-Style Hack?

    More than half of the Fortune 100 could be at risk of falling prey to the same kind of hack that caused devastation at Equifax last year, and it all comes down to poor open source component governance.

    A new report out from Fortune reports that in the year after Apache Software Foundation issued an update to its Apache Struts framework to fix a well-known and critical deserialization vulnerability, more than 10,000 organizations downloaded the vulnerable version instead. Among those were 57 percent of the Fortune 100, according to numbers provided to Fortune by Sonatype.

    The vulnerability is best known to have triggered the catastrophic breach at Equifax in 2017 that exposed the records of 148 million people. It’s a doozy of a flaw—a command injection vulnerability that allows for unauthenticated remote code execution—the kind of opening that gives attackers to gain a powerful foothold in affected systems.

    “Seven months should be enough time for organizations to install the necessary patches and it’s unfortunate that so many still choose to download the older vulnerable versions,” said Nick Bilogorskiy. “There is really no excuse for this.”

    This news should be sobering to DevOps organizations. Increasingly, they’re turning to an assembly-focused mode of software delivery that relies heavily on open source components rather than developers reinventing the wheel with new code. It offers a wake-up call for DevOps organizations to get serious about tracking and securing the third-party code they use.

    In a recent DevSecOps survey conducted by 451 Research on behalf of Synopsys, 40 percent of organizations don’t perform any kind of software composition analysis on their software. That’s problematic, because it means that organizations often don’t know when vulnerable open source components such as Apache Struts are woven into their code base.

    More disconcertingly, even fewer organizations have systems in place that can actually control when, where and what open source components are used within their software development. According to a different DevSecOps survey conducted by Sonatype, 62 percent of organizations don’t have any meaningful controls over which components are in their applications.

    It’s a disconcerting statistic, considering the sheer volume of open source component use in development today—whether in DevOps shops or not. In a talk titled, “We are ALL Equifax” at the DevOps Connect event held at RSA conference last month, Derek Weeks highlighted the fact that open source component download requests have increased 87 times in the last 10 years.

    “The growth rate and consumption volume is massive. And it’s coming into your organization,” Weeks told the audience. “And as a result of this behavior the applications your developers are building are now 80 percent to 90 percent open source components—built from code you did not write from scratch.”

    Sonatype’s survey shows that almost 1 in 3 organizations this year suspect or have verified that they’ve suffered from a breach related to open source components in the last 12 years.

    — Ericka Chickowski

  • Bridge the DevSecOps Experience Divide with Cross-Functional Teams

    Bridge the DevSecOps Experience Divide with Cross-Functional Teams

    Last week’s DevOps Connect event at RSA Conference offered up a ton of wisdom and real-world examples of the power of DevSecOps. With its best-ever attendance after several years and impressive participation levels, this year’s event stands as a good harbinger for the growing interest that the security community has in vesting itself in the DevOps phenomenon.

    But there’s still a lot of work to do in helping security professionals shift their mindsets and fully understand what DevSecOps is all about. Questions from the audience offered evidence of some of the trepidation and misconceptions that still exist in the security community.

    For example, after one of the first talks of the day, an audience member asked about a typical point of confusion for those new to DevOps—namely, the misunderstanding that the melding of disciplines and the requirement that DevOps team members flex their skills across traditional IT boundaries also means that everyone in IT must be renaissance men and women with a range of in-depth knowledge about all IT specialties.

    “Not everyone can know .NET, not everyone can know Python, not everyone can know firewalls,” the audience member said. “I’ve been doing this a long time and it takes a lot to become an expert in any one of these categories. And yet I’m hearing, ‘Oh, everyone needs to know it all.’ They can’t, so how do you deal with that?”

    This was a valuable question to revisit here, because the response from one particular speaker offers up an extremely important tenet that everyone in the DevSecOps world needs to keep in mind. The answer came from Paula Thrasher, a longtime DevOps evangelist in one of the hardest spaces to implement cutting edge IT processes: federal IT. As she explained, one of the biggest success criteria for a successful DevSecOps transformation is the ability to build fully engaged cross-functional teams. Individuals working within the DevOps are not meant to know everything. Rather, different specialists need to be paired up so the complete team has that full body of knowledge at its fingertips.

    “It’s the fidelity of your collaboration within the cross-functional team that makes this successful,” explained Thrasher, who is director of digital services for General Dynamics, a large federal IT integrator.

    For example, she pointed to one particular team she has seen in her work that is the epitome of a solid cross-functional team.

    “My teams all have mascots, and so one of our teams is the Tigers. The best quote I’ve heard from a team about this topic came from a new employee who said, ‘If one Tiger knows it, they all know it,’” she said. “This is a team that has a crazy diverse background—there’s developers, there’s ops people, there’s a former networker on there—and it doesn’t require everyone on that team to know everything. They collaborate so well that if there’s a network vulnerability, the network guy knows it and tells everybody.”

    This is the why Thrasher sees cross-functional teams that include security people as a big success criteria for DevSecOps.

    “I usually say the first thing should be if your teams aren’t cross-functional, start with that. You don’t have to change anyone’s boss, you don’t have to change anything else about who you report to, but make sure your teams represent all of the skills you need to deliver,” she said.

    Not only will the teams be able to deliver on all of the necessary skills when they’re working as a unit, there’s also an ancillary benefit: In the process, they’ll be training each other, and each team member will end up gaining a more diverse set of skills—and perhaps become renaissance men and women in the process.

    “They might have been a graduate of a cyberprogram and they’re next to a graduate of a CIS-type program and they’re next to a math major and they’re next to a hardware engineer, and they’re all kind of new because they’re recent graduates, but they’re learning with each other and we’re going to train them with some more senior engineers that we self-select to be good mentors,” she said. “This is how you are giving people the skills to work with this next-generation of automation and technology, both in the security realm and the Ops realm and the Dev realm.”

    — Ericka Chickowski

  • DevOps Connect at RSA: In Security, Choose Increments

    DevOps Connect at RSA: In Security, Choose Increments

    Earlier this week the worlds of DevOps and information security collided at the DevOps Connect Rugged DevOps event, held in conjunction with RSA Conference 2016. The security-centric audience got to hear from a number of thought leaders across the DevOps and infosec worlds about ways security pros and DevOps teams can work together and establish incremental improvements in application security, as IT picks up the pace of software delivery.

    Among the talks, Forrester Analyst Amy DeMartine discussed her team’s findings after studying the “best of the best” in implementing Rugged DevOps patterns. There are seven habits universal across organizations that are succeeding with this new way of developing and deploying secure applications, she notes.

    “Think of it as a fitness program, and the muscles that you are building up are those that strengthen application security,” she says.

    These habits included a number of obvious practices, along with a few surprises, including:

    • Increase trust and transparency between Dev, Sec and Ops.
    • Understand the probability and impact of specific risks.
    • Use the continuous delivery pipeline to incrementally improve security practices.
    • Standardize third-party software and then keep current.
    • Govern with automated audit trails.
    • Test preparedness with security games.

    All are good tips for getting security, dev and ops to work better at delivering code both faster and more securely. But the seventh habit probably raised the most CISO eyebrows and could have the most potential to effect meaningful change and collaboration:

    • Discard detailed security road maps in favor of incremental improvements.

    According to her, the “best of the best” organizations in Rugged DevOps have security leaders that make it a habit to ditch detailed security road maps in favor of a establishing broad vision and achieving incremental security improvements over time.

    As she related in her talk, she’s seen too many organizations develop extensive and extremely detailed multiyear road maps only to see them grow obsolete not only because of rapid changes in the IT environment, but also the constant pivots attackers make in their attack techniques.

    Instead, she said, security leaders have to take the same continuous improvement approach that their peers in development and operations have made in their shift to DevOps.

    “This is a mind shift change that has to happen,” she warns.

    To achieve good results, security leaders should be focused on “real-time, actionable measurements” to focus on improvements that matter, such as decreasing remediation time for vulnerabilities.

    “In DevOps, we call this Plan, Do, Check, Act, where you’re taking very small changes. Say you want to insert static analysis right after a developer checks in new code. So you’re going to go off and plan. The plan is to insert static analysis,” she says. “Then you’ll do—we’re going to look at tools, we’re going to create our rules. We’re going to check to make sure it works, if it improves things. And then we’re going to move on to the next thing—maybe it’s tightening our rules or maybe it’s adding another tool.”

  • 7 Sets Of DevOps Success Metrics From DOES ‘15

    7 Sets Of DevOps Success Metrics From DOES ‘15

    Beyond the moral support, the executive management coaching and the tips on how to build DevOps momentum in the enterprise environment, last week’s DevOps Enterprise Summit 2015 offered a boatload of excellent fodder for those seeking to prove the value of DevOps patterns once they got back home.

    DOES wasn’t just about cheerleading—alongside lessons on the soft, squishy elements of transforming IT, nearly every speaker offered up stone cold numbers to back up why those transformations make business sense. The performance metrics shared by speakers were impressive and could go a long way toward swaying skeptical stakeholders. Here are some of the highlights.

    Ticketmaster

    Ticketmaster CTO Jody Mulkey offered a ton of excellent examples of how an organization-wide commitment to DevOps has fueled innovation and stability at his company. For example, he described how what Ticketmaster calls the ‘Boom Team’ has used a completely integrated team of devs, operations, QA and other staff of about 20 to shift its Ticketmaster.com property—which accounts for 40 percent of the company’s revenue—from a duct-tape and bubblegum affair with 2000 ModPerl at its base into a modernized system continuously delivered by that team. Similarly, its technical operations team has embraced the DevOps ethos in order to deliver what it calls ‘support at the edge’ for WebOps incidents. The success of this approach has resulted in:

    • 90 percent reduction in mean time to repair (MTTR)
    • MTTR from 47 minutes down to 3.8 minutes

    “This was a phenomenal change,” Mulkey says. “The business went crazy over this. These are measurable outcomes that have really helped demonstrate that we are making progress.”

    Target

    Target brought forward speakers to discuss the retailer’s innovation in a number of key areas. Heather Mickman, director at Target, talked about how her API team worked hard to not only easily expose product and store information via its APIs, but to also challenge itself in the way it was delivering its work.

    “We wanted to deliver new capabilities in days and not months,” she explains, saying the team needed to be sure that it could scale its APIs because it new that they would be highly reused—both internally and by partners. Target has:

    • more than 90 APIs
    • monthly volume of calls to these APIs have grown in one year from 1.5 billion to 17 billion
    • deploy 80x per week across stage and production enviornments
    • all that, with fewer than 10 incidents per month
    • this volume helped support in 2014 a 42 percent increase in guest traffic and 280,000 orders fulfilled on Black Friday alone

    HP

    HP CIO Ralph Loura says HP didn’t go for the big bang approach, but through careful pilots and the use of communication afforded by ChatOps infrastructure, his team has been able to drag HP IT slowly into the DevOps fray. That’s a good thing, considering the massive scale of the organization:

    • 7,000 developer and operations staff
    • handling over 20,000 change requests per year
    • supporting 1,400 apps
    • and 1000s of builds per day

    CapitalOne

    After four years of hard work to move toward DevOps and continuous delivery, CapitalOne Product Manager Tapabrata Pal had a number of impressive statistics to share on its its progress:

    • Code commits have moved from random occurrence to hundreds per day for any given product team
    • Integration has moved from 1 per day to every 15 minutes
    • QA performance has shifted from monthly to four times per day
    • Production releases have moved from quarterly to one per sprint
    • Defects have been reduced by approximately 60 percent
    • Feature delivery rate has increased by approximately 50 percent

    CSC

    Paula Thrasher, application delivery lead with CSC, stepped up in front of the DOES audience to show them how it is possible to battle bureaucracy at any organization—coming from the government sector, she has first hand knowledge of what it takes. In one example she shared, her organization helped a client move its cyclew time from automating deployments to staging:

    • from 12 hours down to just 12 minutes and 30 seconds.

    IBM

    IBM’s Angel Diaz, VP of cloud architecture and technology, talked about how enterprises can still innovate at the pace of startups while delivering at scale. The thing with IBM is that it puts its money where its mouth is when it comes to DevOps—the company has doubled down to not only offer a number of products viable for the continuous delivery pipeline but to also completely transform its own business through DevOps practices. Some of the results it has seen by shifting its 2,000-plus developers and 100-plus squads in the Cloud Services division to DevOps:

    • 300 percent increase in “pure coding time” for its Cloud Services developers
    • reduction of deployment time from 32 hours down at first to 6 hours and now to minutes
    • an increase of release frequency from months to every three weeks.

    Disney

    Disney’s Jason Cox, director of systems engineering, got the crowd pumped up with a slick mash-up video of Star devWars and DevOps themes (not to mention a private showing of the most recent Star Wars trailer at the end of his talk!). But he didn’t favor style over substance. In his talk he described how much success Disney has grabbed up through DevOps, including:

    • 75 percent reduction manual steps
    • reduction in deployment time for large online/digital experience systems from 3 hours to 20 minutes
    • ability to provision 300-700 srvers within 15 to 30 minutes
  • 5 DevOps Success Factors Surfacing At DOES 15

    5 DevOps Success Factors Surfacing At DOES 15

    Two days into a breakneck schedule of energizing presentations at DevOps Enterprise Summit 2015 (DOES 2015) and  there are already some common themes bubbling up from the speakers and attendees. As IT practitioners from big name enterprises like Capital One, HP, Target, Ticketmaster and more step up to the podium to share their personal stories about the journey to continuous delivery, they’ve proven that their transformations rely on similar success factors.

    Success Factor: DevOps Champions

    DevOps change agents and champions are the engine that can help turn an enterprise IT ship around. And while it will eventually take a lot of people to power a large organization’s shift to DevOps, the passion and vision of a single DevOps change agent is contagious—particularly when they can demonstrate and document the results of new process patterns.

    “One person can make a difference,” says Sherry Chang, chief architect of DevOps and continuous delivery with Intel, who says when she first started helping her firm in its transformation, she had no formal DevOps title or role, but she believed strongly that changes needed to be made and was determined to drive them. She says she connected with others to start small and spread the DevOps gospel.

    Success Factor: The Yin/Yang of Grassroots/Executive Support

    Building DevOps momentum can’t depend solely on support from the bottom up or the top down. Those experienced in walking the DevOps path say that buy-in must come from both sides of the coin to truly transform the business.

    Most transformations tend to start at the grassroots level—that’s what those champions are especially good for kick starting—because risk-averse executives tend to like to grab onto new initiatives that already look like potential winners.

    “Those change agents are important in the early days because they can demonstrate success, which then helps to start growing a grassroots movement, where more and more passionate folks start to get bought into the different changes that are being talked about, and then that got us to gain our tops-down buy-in,” says Heather Mickman, director at Target.

    Success Factor: Internal DevOps Events

    Many of the biggest enterprises that shared their stories explained that a key component of the culture shift at their organization has been some sort of large-scale internal event—essentially in-house DevOps conferences. These events give DevOps champions the chance to bring in outside thought leaders and practitioners at other companies to demonstrate to people within the host enterprise that there really is something to DevOps, automation, continuous delivery and the like. For example, CapitalOne recently hosted more than 1,200 software engineers for an in-house event it calls SECon and brought in many of the same speakers we’re seeing this week at DOES.

    Target has also hosted numerous internal DevOps days, as well as an executive summit that Ross Clanton, senior group manager at Target, called a “tipping point” for gaining top down support from executives.

    Success Factor: Empowerment

    Giving staff the tools the need to do their job and then getting out of the way to actually let them do it has been another common theme weaving its way through DOES talks.

    “Teams crave responsibility,” says Jody Mulkey, CTO at Ticketmaster. “Folks on our development teams, they all want responsibility to run their products. They didn’t all want it in the beginning, to be quite honest. But once they saw how quickly teams were able to deliver and how they were able to minimize these business disruptions based on the ability to quickly make changes to their software, it was pretty incredible.”

    He says of the 73 development teams at Ticketmaster in North America, 100 percent push their own code.

    Success Factor: Test Automation

    Time and time again, presenters have called out the importance of test automation in eliminating waste and idle time in the development lifecycle and elevating the process. On Monday, there was a whole panel dedicated to the topic with representatives from Intel, Quicken Loans and Bluware talking about how to scale this across the enterprise. And as Paul Schmid explained in his description of the DevOps transformation at Sherwin-Williams, like many organizations, his has been learning the hard way that test automation ideally should be one of the first steps to shifting gears to continuous delivery.

    “Tests for us are really important, and we are not mature in this at all. So a lot of what was talked about this morning hit right at home,” he says. “Testing is really important and it’s definitely the next step for us, and we’re putting a lot of time, effort, and money into it.”

  • Cloud Visibility: The Security Achilles Heel

    Cloud Visibility: The Security Achilles Heel

    As organizations accelerate their DevOps practices and move toward continuous delivery, cloud adoption plays a pivotal role. But visibility into cloud infrastructure and provider practices will remain a security Achilles heel until businesses can get better monitoring and insight into public cloud assets. According to a new survey out by the SANS Institute, even as the amount of sensitive data housed in public cloud infrastructure explodes, many organizations haven’t found a way to lick the visibility problem.

    Sponsored by CloudPassage and conducted among nearly 500 IT professionals, the survey showed that both PaaS and IaaS are on a huge upswing. In the next 12 months organizations report that they will approximately double usage of both models. IaaS in particular shows the biggest growth among cloud usage models, as 29 percent of organizations report they’ll deploy IaaS for the first time in the next year. Among the major drivers for this growth, 61 percent of organizations said time to deployment was key, while 54 percent said they were moving to the cloud because they can’t scale their own solutions.

    In those cloud deployments, well over half of organizations are processing or storing sensitive data in their public cloud environments or aren’t sure if they have sensitive data in the cloud. At least one in five organizations said they use the cloud to store or process intellectual property, customer financial information or health records. Meanwhile, 52 percent said they used the cloud to store and process business intelligence data, while 48 percent store or process employee records.

    Visibility concerns trump all others when organizations related their problems with cloud adoption—nearly 60 percent reported having issues with a lack of visibility into service provider operations. Nearly half of organizations related they had issues getting proper incident response support from providers due to visibility issues. And 46 percent reported having issues with VM and workload visibility. While not quite as widespread as general visibility woes, a quarter of organizations reported that provider-introduced vulnerabilities resulted in a breach or incident. Unfortunately, organizations are more likely than not unable to test for these kind of issues proactively. Only about 24 percent of organizations work with providers that allow them to do penetration testing on their public cloud assets.

    Diving deeper into this issue, one of the big problems is the inability for organizations to fold their cloud operations into overall security monitoring and management—37 percent of organizations reported there’s just a lack of consistent security controls that integrate with on-premises tools and security management.

    “The biggest issue, which goes back to the problem of visibility into internal cloud provider operations, is the lack of access to log files and other forensic artifacts, experienced by 53% of respondents,” says the report’s author, Dave Shackleford. He recommends that as organizations evaluate their public cloud relationships, they consider controls like logging, multifactor authentication, encryption and compatibility with security products that work natively within that cloud infrastructure.