devopsy.kinsta.cloud

Category: Features

  • 7 Sets Of DevOps Success Metrics From DOES ‘15

    7 Sets Of DevOps Success Metrics From DOES ‘15

    Beyond the moral support, the executive management coaching and the tips on how to build DevOps momentum in the enterprise environment, last week’s DevOps Enterprise Summit 2015 offered a boatload of excellent fodder for those seeking to prove the value of DevOps patterns once they got back home.

    DOES wasn’t just about cheerleading—alongside lessons on the soft, squishy elements of transforming IT, nearly every speaker offered up stone cold numbers to back up why those transformations make business sense. The performance metrics shared by speakers were impressive and could go a long way toward swaying skeptical stakeholders. Here are some of the highlights.

    Ticketmaster

    Ticketmaster CTO Jody Mulkey offered a ton of excellent examples of how an organization-wide commitment to DevOps has fueled innovation and stability at his company. For example, he described how what Ticketmaster calls the ‘Boom Team’ has used a completely integrated team of devs, operations, QA and other staff of about 20 to shift its Ticketmaster.com property—which accounts for 40 percent of the company’s revenue—from a duct-tape and bubblegum affair with 2000 ModPerl at its base into a modernized system continuously delivered by that team. Similarly, its technical operations team has embraced the DevOps ethos in order to deliver what it calls ‘support at the edge’ for WebOps incidents. The success of this approach has resulted in:

    • 90 percent reduction in mean time to repair (MTTR)
    • MTTR from 47 minutes down to 3.8 minutes

    “This was a phenomenal change,” Mulkey says. “The business went crazy over this. These are measurable outcomes that have really helped demonstrate that we are making progress.”

    Target

    Target brought forward speakers to discuss the retailer’s innovation in a number of key areas. Heather Mickman, director at Target, talked about how her API team worked hard to not only easily expose product and store information via its APIs, but to also challenge itself in the way it was delivering its work.

    “We wanted to deliver new capabilities in days and not months,” she explains, saying the team needed to be sure that it could scale its APIs because it new that they would be highly reused—both internally and by partners. Target has:

    • more than 90 APIs
    • monthly volume of calls to these APIs have grown in one year from 1.5 billion to 17 billion
    • deploy 80x per week across stage and production enviornments
    • all that, with fewer than 10 incidents per month
    • this volume helped support in 2014 a 42 percent increase in guest traffic and 280,000 orders fulfilled on Black Friday alone

    HP

    HP CIO Ralph Loura says HP didn’t go for the big bang approach, but through careful pilots and the use of communication afforded by ChatOps infrastructure, his team has been able to drag HP IT slowly into the DevOps fray. That’s a good thing, considering the massive scale of the organization:

    • 7,000 developer and operations staff
    • handling over 20,000 change requests per year
    • supporting 1,400 apps
    • and 1000s of builds per day

    CapitalOne

    After four years of hard work to move toward DevOps and continuous delivery, CapitalOne Product Manager Tapabrata Pal had a number of impressive statistics to share on its its progress:

    • Code commits have moved from random occurrence to hundreds per day for any given product team
    • Integration has moved from 1 per day to every 15 minutes
    • QA performance has shifted from monthly to four times per day
    • Production releases have moved from quarterly to one per sprint
    • Defects have been reduced by approximately 60 percent
    • Feature delivery rate has increased by approximately 50 percent

    CSC

    Paula Thrasher, application delivery lead with CSC, stepped up in front of the DOES audience to show them how it is possible to battle bureaucracy at any organization—coming from the government sector, she has first hand knowledge of what it takes. In one example she shared, her organization helped a client move its cyclew time from automating deployments to staging:

    • from 12 hours down to just 12 minutes and 30 seconds.

    IBM

    IBM’s Angel Diaz, VP of cloud architecture and technology, talked about how enterprises can still innovate at the pace of startups while delivering at scale. The thing with IBM is that it puts its money where its mouth is when it comes to DevOps—the company has doubled down to not only offer a number of products viable for the continuous delivery pipeline but to also completely transform its own business through DevOps practices. Some of the results it has seen by shifting its 2,000-plus developers and 100-plus squads in the Cloud Services division to DevOps:

    • 300 percent increase in “pure coding time” for its Cloud Services developers
    • reduction of deployment time from 32 hours down at first to 6 hours and now to minutes
    • an increase of release frequency from months to every three weeks.

    Disney

    Disney’s Jason Cox, director of systems engineering, got the crowd pumped up with a slick mash-up video of Star devWars and DevOps themes (not to mention a private showing of the most recent Star Wars trailer at the end of his talk!). But he didn’t favor style over substance. In his talk he described how much success Disney has grabbed up through DevOps, including:

    • 75 percent reduction manual steps
    • reduction in deployment time for large online/digital experience systems from 3 hours to 20 minutes
    • ability to provision 300-700 srvers within 15 to 30 minutes
  • Devops Days Silicon Valley, you don’t want to miss it

    Devops Days Silicon Valley, you don’t want to miss it

    DevOps Days have become a global phenomenon. These are unique, community-organized events that bring together development and operations practitioners for two days of scheduled presentations, open spaces (more about that later), and networking.

    How instrumental to the DevOps movement are DevOps Days? “DevOps” actually got its name from the discussions that happened at the first DevOps Days organized by Patrick Debois in 2009 in Ghent, Belgium.

    The 6th annual DevOps Days Silicon Valley is happening on November 6th and 7th at the world-famous Computer History Museum. Silicon Valley is the oldest of the US events and was originally the third DevOps Days ever. Starting as a low-key event hosted on the campus of Linked-in in 2010, DevOps Days Silicon Valley has grown into a major event.

    True to past reputation, this year’s speaker lineup blends industry favorites, like John Willis, with a unique mix of soon to be well-known speakers who have been selected by the community. The 2015 line-up includes:

    • State of the DevOps Union by John Willis
    • Anomaly Detection 101 by Elizabeth A. Nichols
    • Not Everybody is Kelsey Hightower by Gregory Poirier
    • Securing the Heart of Automated Infrastructure by Jamesha Fisher
    • Caching Versioned MetaData with Git by Kirsten Hunter
    • Flaming Poo and the Human Response by Jason Hand
    • DevOps Army of One – How to Survive When You’re the SPoF by Jonathan Disher
    • Single Point of Failure… Expert by Sasha Rosenbaum

    But that’s not all! Unlike most conferences where the scheduled speakers are the main event, at DevOps Days, facilitating the interaction between attendees is the main event. Both afternoons are dedicated to Open Space, where the agenda is dedicated to small-group participant-lead discussion and the discussion topics are chosen on the fly each day. This is the chance to dive into specific topics, ask for help, discuss what you’ve done, find like-minds, or just be a fly on the wall and take it all in.

    The DevOps Days experience is one that is unique among technology events — from the local community driven organizing (each one is totally run by local volunteer organizers) to the 50% of time dedicated to Open Space discussions. The whole point behind DevOps Days is to push those with fresh knowledge and insights, no matter how small, into direct contact with those who can benefit from or add to it.

    Of course there is a good bit of fun involved as well. Evening events, ignite talks, and the seemingly cult-like status of the locally designed t-shirts all add to the unique flavor of a DevOps Days event. Austin 2015 was famous for their dueling social themes of Star Wars (May 4th / “May the Force” day) and Cinco de Mayo (complete with large Mariachi band).

    If you are in the San Francisco Bay / Silicon Valley area, get your tickets for the November 6th and 7th DevOps Days Silicon Valley ($150) before they sellout. If you live elsewhere, be sure to keep an eye on the main DevOps Days site for upcoming events. You can also watch presentations from past events on the DevOps Days video site.

    About the Author/Damon Edwards

    41368124db43a27257c31df174a61357_400x400Damon Edwards is the co-founder and managing partner of the DTO Solutions, a DevOps and automation engineering consultancy whose clients include leading SaaS, e-commerce, gaming, and financial services companies. Damon Edwards is also the co-founder and vice president of SimplifyOps, a provider of support and services for the users of the Rundeck open source project. Damon has spent over 14 years working on both the technology and business ends of IT operations. Damon is also a frequent conference speaker and writer who focuses on DevOps and operations improvement topics. Damon is active in the international DevOps community, including co-organizing the DevOps Days conference series, co-hosting the DevOps Cafe podcast, and a co-author of the upcoming DevOps Cookbook from IT Revolution Press.

  • PicsArt’s APM Transition Brings Clarity for Customers

    PicsArt’s APM Transition Brings Clarity for Customers

    PicsArt’s Technology Landscape

    PicsArt enables its creative user base to freely upload or draw, edit, and exchange digital images in a worldwide, web-based community. PicsArt works on the web and on Android, iOS, and Windows Phone platforms.

    PicsArt applies state-of-the-art backend infrastructure to provide its editing applications and social reach for its community. PicsArt uses popular DevOps technologies such as the MongoDB database, the Node.JS JavaScript runtime, and the Redis in-memory data tool. With a foundation of these tools and infrastructure, PicsArt builds and supports apps that avail its users of many free options for creating and appreciating original digital art.

    According to Mikayel Vardanyan, Chief Technology Officer, PicsArt, PicsArt apps and technologies process millions of user requests each day. “We will also soon provide a public API to all of our developers, giving them an opportunity to use our backend services to create their own apps,” says Vardanyan. Though this will satisfy customer requests for additional functionality, it will also further multiply the intricacies of the PicsArt systems, making it ever more critical that they perform properly from foundation to pinnacle.

    PicsArt Experiences the Same Pitfalls as Other Tech Firms, and Responds

    As with any technological system, PicsArt’s infrastructure is at risk of downtime, which looms ready to strike the business and to challenge customer satisfaction at any time. Unforeseeable events as well as planned maintenance outages create downtime and are an expected part of the systems lifecycle, even though they are not always welcome.

    But malfunctions and downtime can exacerbate DevOps processes and corrupt and diminish the user experience. Downtime can temporarily take out entire services, ones that customers will definitely notice and miss when they become unavailable. “If, for example, the users cannot upload their pictures, that would certainly be bad for us,” says Vardanyan.

    Sufficient downtime on the PicsArt backend systems could also cause user registrations functions to fail. The backend also supports PicsArt’s community activities such as sharing artwork, liking others’ artwork, and commenting on people’s creations, which could suffer. “We try to prevent downtime by providing redundancy and scalable systems. We properly test our systems on a regular basis,” says Vardanyan.

    “When we start to sense issues such as these, our operations team goes to work immediately to fix what we need to fix. We investigate all the issues we come across and respond by adding more layers of redundancy and scalability,” says Vardanyan.

    PicsArt Sought Picture Perfect Improvements from APM

    PicsArt was looking for increasing visibility into its infrastructure in order to monitor for the aforementioned problems at their very outset, to create alerts on trouble issues and status reports, and to derive metrics about what is happening across the entire infrastructure from nation to nation, and region to region.

    PicsArt knew that it needed extensive monitoring and that the best way to go would be to have that monitoring centralized in a single product. “We needed to monitor the infrastructure 24/7 in order to provide the best possible service for our community,” says Vardanyan; “we needed external as well as internal monitoring and we needed to monitor the user experience”.

    APM Frames Performance Challenges Nicely

    PicsArt currently uses Monitis. At PicsArt, systems are continually wrestling with pressing loads and with load balancing. If systems performance should lag for any reason, the network can take center stage as the weakest link and the strongest bottleneck. “We utilize the different alerting capabilities of Monitis, such as voice calls and SMS, using all the escalation procedures that its notification systems offer, too. We also use Monitis to create custom monitors for certain scripts and for Redis jobs,” says Vardanyan.

    The cloud-based monitoring solution enables the digital art editing and sharing platform provider to leverage views from a number of global vantage points, clarifying transparency into PicsArt infrastructure. “With Monitis, we can check our global CDN locations and find out what location may have an issue. The benefit is that a potentially issue-stricken location does not escape the monitoring routine,” says Vardanyan. The sooner the enterprise sees it, the faster it can fix it.

    The conveniences and efficiencies of dissecting it all inside a single dashboard enable PicsArt to repair the real problems in its DevOps and platform delivery processes without touching anything that is not out of tune.

  • DevOps success is everyone’s responsibility

    DevOps success is everyone’s responsibility

    For organizations with existing hierarchies and processes the shift to a DevOps mindset can be a challenge. The traditional departmental roles and separation of duties is anathema to an efficient DevOps culture. When it comes to DevOps there’s no room for finger-pointing—every team and individual is equally responsible for making sure everything is firing on all cylinders.

    Gene Kim, coauthor of The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win and the upcoming DevOps Cookbook, took part in a podcast earlier this year and shared that two-thirds of the DevOps success stories he has seen are actually from established businesses with existing infrastructures and processes as opposed to “greenfield” projects that embrace DevOps from the start. In those situations, the drive to embrace DevOps has to come from somewhere, but in order for DevOps to work the organization has to overcome the urge to place the entire burden for the success or failure of DevOps on that one team or individual.

    There is a reason that the word “culture” plays such a central role in any discussion of DevOps. Without a shift in the mindset of how teams and individuals communicate and collaborate a DevOps initiative isn’t much different than any other IT project in the organization. One team or person can use DevOps tools, but that’s not really a functional implementation of DevOps.

    That doesn’t mean that DevOps is an all-or-nothing game. It’s possible for different departments or elements within a company to embrace DevOps without the whole company jumping on board. There has to be a quorum of sorts, though—a broad enough consensus between teams and individuals to cooperate and collaborate in a different way than how it’s laid out on the company org chart. An important part of that shift is for everyone to accept responsibility for the outcome and work together toward a common goal.

    There’s a phrase that goes something like “When all you have is a hammer, everything looks like a nail.” Developers understandably view the world through the filter of developing. IT operations people look at projects from the perspective of operations. QA looks at things from the point of view of testing and quality assurance. Accounting considers projects from the financial angle. Security personnel tend to focus on the security aspects. You get the idea.

    Kim shared an excerpt of the upcoming DevOps Cookbook with me. In the book Kim talks about the need to move from functional-oriented teams optimized for cost to market-oriented teams optimized for speed. He talks about the challenges and bottlenecks created when some teams or elements are moving at an accelerated pace while others struggle to keep up.

    Part of optimizing for speed is shifting the perception of responsibility. Smaller, market-oriented teams are more agile and can deliver value to customers—both internal and external—while reducing or even eliminating reliance on other teams. Kim explains, “Taken to its extreme, market-oriented teams are responsible for not just feature development, but also testing, securing and deploying their code into production themselves, as well as being responsible production operations and availability.”

    In order for DevOps to work effectively, everyone needs to forget that they’re a “hammer” and view the project as a more comprehensive toolbox instead. DevOps teams need to strive to break out of the traditional roles and silos and make a conscious effort to see things in terms of the common goals or greater good.

  • DevOps Enterprise Summit 2015 kicks off

    DevOps Enterprise Summit 2015 kicks off

    Are you going to San Francisco? If you are you don’t have to wear a flower in your hair. But if you are into DevOps or want to be and are in San Francisco this week you should be at the DevOps Enterprise Summit 2015 (DOES 2015). The conference produced and hosted by Gene Kim’s IT Revolution and Electric Cloud, features a 3 day schedule chocked full of success stories of enterprises who have adopted DevOps.

    Additionally many leading DevOps companies are sponsoring, exhibiting and otherwise participating.  Several are making or touting recent announcements. Electric Cloud themselves have just made a big announcement, releasing ElectricFlow Release which according to the company “helps organizations speed application releases by streamlining and automating software deliveries into highly organized and well-choreographed release pipelines. For software teams adopting DevOps and Continuous Delivery, it replaces checklists and spreadsheets with a single, unified view across both traditional and Continuous Delivery style application releases to increase reliability, transparency and predictability while lowering risk.”

    Screen Shot 2015-10-18 at 10.48.59 PM

    More on this in another post coming this week.

    IBM is making a major announcement as well with the BlueMix Garage Method. I had a chance to speak to Randy Newell about this new release and you can listen and read the transcript of that conversation here. IBM is a major sponsor of DOES and have several sessions and events planned including a DevOps Garage event at the Galvanize building Monday night.

    devopsy.kinsta.cloud will be providing extensive coverage of DOES 2015.  Ericka Chickowski and I will be live blogging from the conference all three days in addition to filing stories to keep our readers up to date.  Additionally we are setting up a video booth in the foyer by the Grand Hall. We are video interviewing every speaker at the conference, as well as select other guests.  Look for those videos to be available soon.

    So if you are at DOES stop by our video set and say hello. If you can’t make it, devopsy.kinsta.cloud will do all we can to keep you up to date on all of the goings on at DOES 2015.

  • Continuous Deployment is king of the DevOps hill

    Continuous Deployment is king of the DevOps hill

    One of the promises of DevOps is continuous deployment. DevOps is, first and foremost, a matter of shifting the traditional business culture more than the tools and technologies that enable it. Once that culture exists, though, one of the defining aspects of DevOps is automation and continuity.

    Earlier this year I wrote about the seeming ubiquity of “continuous” solutions throughout DevOps:

    If DevOps is the buzzword du jour for tech, then continuous is the buzzword du jour of DevOps. Everything is continuous. Continuous development and continuous testing lead to continuous deployment and continuous delivery, which requires continuous support. Continuous monitoring produces continuous integration and continuous change. Continuous security results in continuous incident response…or vice versa. To top it off, all of the continuous activities continuously feed each other to drive more continuousness in some sort of DevOps Mobius strip of continuity.

    Taking a closer look at all of that continuous continuousness, however, continuous deployment stands out as one of the prevailing goals of effective DevOps. Continuous integration, continuous delivery and the other continuous tools and processes are all in some way related to facilitating the flow of continuous deployment or monitoring and managing it after the fact—which ultimately feeds back to the beginning of the loop and back to continuous deployment.

    Chris Riley tried to muddle through the confusion of continuous integration, continuous delivery and continuous deployment. In his explanation of continuous deployment, he stressed the impact of this symbiotic loop that surrounds it: “Continuous deployment has additional costs, as it relies on instrumentation, to ensure that new functionality does not result in bugs and also on infrastructure that allows effortlessly backing out new features when a defect has not been caught by automated tests.”

    I defer to Riley on most things, but I don’t agree with his summation that everyone should do continuous integration, some should do continuous delivery, but only the rare few should strive for continuous deployment.

    I guess it depends on what perspective you’re considering it from. From the standpoint of ease or simplicity and the likelihood that an organization can execute it effectively I can see where Riley is coming from. From the point of view of the goal of embracing DevOps, though, I would reverse his order and say that everyone should do continuous deployment, most should implement continuous delivery to facilitate it, and some (or maybe most as well) should adopt continuous integration to drive continuous delivery.

    The primary objective of DevOps, though, is to deploy apps. Everything else plays a supporting role.

    That said, there is a difference between embracing DevOps with the goal of developing and deploying apps more efficiently and effectively, and adopting continuous deployment tools that automatically pass all changes that pass automated testing and are verified as deployable by continuous delivery tools into production.

    A blog post from Puppet Labs explains, “There are business cases in which IT must wait for a feature to go live, making continuous deployment impractical. While application feature toggles solve many of those cases, they don’t work in every case. The point is to decide whether continuous deployment is right for your company based on business needs — not on IT limitations.”

    There are many acronyms and various interpretations out there for DevOps, so my opinion certainly isn’t gospel on the subject. What do you think? Should continuous deployment be the primary goal and centerpiece of DevOps automation? If not, which of the many “continuous” solutions would you suggest as the central focus?

  • Survey on Modern IT – We Need Your Opinion!

    Survey on Modern IT – We Need Your Opinion!

    Change is the only constant in the world of IT. Developers, ITOps and the DevOps community are undergoing significant changes in needs, processes and services. This survey will provide better understanding of how the latest trends are impacting innovation, security and your team.

    This survey is sponsored by Apcera, a highly secure, policy-driven platform that lets enterprises innovate at speed with full confidence and trust. You can learn more by reading an interview with Apcera CEO, Derek Collison, on Container Journal.

    The survey is under 20 questions and takes about 5 minutes to complete. Share your thoughts on what is leading innovation, what stands in your way and how new technologies are are influencing the IT landscape.

    By taking the survey, you’re helping the entire DevOps community better understand where we stand and where we’re headed. As always, the results will be released when the survey period is complete.

    Take the Survey

  • Digital Disrupters – Building a DevOps Leadership Profile

    Digital Disrupters – Building a DevOps Leadership Profile

    While the advancement of DevOps methodology remains fairly nascent for many organizations, an ever-expanding class of leading practitioners are evolving a wide range of related best practices.

    Beyond the well-worn concept of “unicorns” – the scant few organizations that stand as mythical examples of DevOps nirvana – this broader set of early adopters have begun to pull back the curtain on their efforts, lending further detail to the strategies that have bred this current model for success.

    Last week, industry analysts Freeform Dynamics published valuable research that highlights the ongoing work of such DevOps innovators, dubbed as “Digital Disrupters” for their accomplishments in pursuing larger digital transformation. The report “Exploiting the Software Advantage” – sponsored by CA Technologies, outlines a critical set of initiatives crucial to navigating the DevOps landscape.

    What are the specific hallmarks of those organizations leading the charge? Where are others struggling to catch-up? Many of the Freeform Dynamics survey results, based on 1442 respondents, reaffirm widespread DevOps observations, while other findings serve to shed new light, including:

    • Everyone builds software: Perception is, in fact, reality and to deeply engrain digital transformation, Digital Disruptors are increasingly identifying with this mantra. While 95% of leaders have committed to becoming more app-centric, only 51% of mainstream organizations have adopted this persona. Similarly, 84% of Disruptors directly identify as “software companies”, regardless of industry, compared to 40% among the masses.
    • Transformation is required: This point may seem obvious, but it certainly bears repeating. A whopping 92% of leaders have actively moved to break down traditional silos between Dev and Ops, with 51% of those organizations having already completed this work. By comparison, only 18% of mainstream organizations claim to have done the same, supported by 59% who at least share this more collaborative DevOps vision.
    • Commitment is critical: This may also seem like a no-brainer, but the numbers bear out this requirement. Some 77% of Disruptors have agile development methodologies in place, with 70% having “broadly” implemented DevOps activities. For mainstream organizations, these metrics rate at only 31% and 21%, respectively; leaders are going all-in and appreciating the benefits, and those that follow will need to do the same.
    • Experimentation is omnipresent: To garner the benefits of continuous delivery, embracing trial and error is a must. Among Disruptors, 93% recognized the need to “experiment and fail quickly on the road to success”, compared to only 58% of mainstream organizations. Some 95% of leaders have already shifted from fixed product cycles to continuous innovation, while only 56% of the mainstream have made this leap.
    • APIs are everywhere: Ubiquitous integration, both in-house and in connecting to third parties, has only increased emphasis on the API. Disruptors are resoundingly focused on API management, with 68% using APIs to integrate third party services into their applications, and 66% to offer external access to systems and data. Mainstream peers compare at only 26% and 24%, respectively, denoting a sizable API management gap.

    These are only a handful of the most dramatic and obvious areas where DevOps advocates have clearly re-architected their approach, separating themselves from those organizations that have yet to go all-in; it doesn’t stop there.

    As highlighted in the research, other areas of differentiation and dichotomy include cross-functional alignment, emphasis on digital channels, use of digital to drive core business efficiency, and even funding of digital investments.

    Beyond these key philosophical milestones that litter the pathway to DevOps success, the survey also highlights the vast array of strategic goals that these organizations have sought to accomplish. The results spotlight the massive range of significant business benefits that practitioners believe DevOps can deliver.

    Leading use cases include:

    • Improved customer satisfaction (49%)
    • Better workforce productivity (44%)
    • Improved customer experience (41%)
    • Improved competitive differentiation (38%)
    • Better agility / speed-to-market (37%)
    • Opening up of new markets (35%)
    • Creation of new business models (30%)
    • Creation of new routes to market (26%)

    If one were to take those outcomes out of context, unrelated to the impact of DevOps, it’s fair to say that they read like a generic list of leading business initiatives – further underlining the movement’s overall relevance.

    That’s not to say this is all accomplished easily. Report respondents detail hard won victories that required significant time and investment to achieve all this DevOps-driven advancement. After all, the notion of “disruption” doesn’t exactly evoke images of ease or comfort.

    Click here to read the report for yourself – there’s a lot more to be considered. Namely, how well your organization measures up in affecting digital transformation in its own right, and where it can take steps to move forward.

  • DevOps for Systems of Record – A new hope (Preview of DOES talk)

    DevOps for Systems of Record – A new hope (Preview of DOES talk)

    I am speaking at the DevOps Enterprise Summit later in the month and thought I give people a teaser of what I am going to talk about, so that they know what to expect. I am looking forward to share my story – a story of adversity but also of hope – with many of you. (See me on Wed, 21st October at 11:15 at the Bayview B)

    So we have all heard about the DevOps and Continuous Delivery stories from our favorite Web companies (Google, NetFlix,…). It is much harder to come across successful case studies with Enterprise-grade Systems of Record. While I cannot provide the 50 times a day deployments for SAP (yet ;-)), I wanted to share my story so far and what I have learned in the journey (both in regards to solved and unsolved issues).

    Hopefully this contribution to the discussion helps some of those out there who are looking for a first step. But why should we even deal with COTS and other systems of record? Ultimately for me this comes back to the two gear analogy that I keep using (Two Gear Blog Post) . If you can just ditch your “legacy” applications completely, then congratulations you don’t have to deal with this and can probably stop reading this blog post. For all of you out there that cannot do that, you will be confronted with the reality that even though your digital and custom applications can deliver at amazing speed you are still somehow constraint by your “legacy” applications. Hence speeding up those will help you achieve the ultimate terminal velocity of your delivery organisation. So how do I approach Systems of Record and the DevOps adoption of it?

    DOES1

    3 Key steps:

    1. Look under the hood to find the ingredients
    2. Recreate the IKEA manual
    3. Understand the path to production/operations

    Step 1.  Let’s open the Hood

    DOES2Ultimately the two things that I am always looking for are the moving parts of the application and the processes to build and deploy the application. Sounds easy, right? Well it often isn’t as easy as you think, especially COTS application try to abstract those processes away. Look at SAP where you make configuration changes in one environment that then somehow get transported to other environments, but you have little visibility into the underlying process (Ok – my SAP experience is 10 years old, so it might have changed, but you get the gist).

    Why do we look for moving parts? Well we want to use the common software engineering practices as much as we can and don’t necessarily want to trust the inbuild features for version control for example. So we look for ways to find the source code files or ways to extract the relevant information in text format (well SQL, XML, etc count as text in my world view), this is sometimes relatively easy (Siebel .sif files are just a type of XML for example) and sometimes it is pretty difficult or impossible. These underlying files is what we will store in source control to make sure we understand at each point in time the configuration of the application.

    Once you have found the source code, you will have 3 things to do with it:
    a) Get it out of the proprietary solution
    b) Integrate it with your IDE
    c) Solve for Merges
    (I will explore this further in another blog post later)

    Step 2. Recreate the IKEA Manual
    DOES3Once you have all the moving parts, you need to understand how it all fits together. Custom applications usually require explicit instructions which makes it much easier to define and automate the instruction set. For systems of record packaging and deployment is often done through UIs and there is usually more than one way of doing it. Experienced operators have found their own short cuts and the reliability of the system depends on having the right guy in the operating centre (I have some stories to tell about not having the right guy on go-live weekend).

    Your goal needs to be an “Ikea Manual” that creates the system from the moving parts we identified earlier without requiring complex interactions or decisions from human operators. Ideally all instructions work from the commandline or some programming API, but when all else fails a UI driver like Selenium can solve the really tricky parts of creating and automating the manual user entries in the application. Every black-box step should be drilled open so that you understand what is happening within the system and so that you understand the variations that could cause you trouble down the path.

    Step 3. Understand the Path to Production
    does4Speaking of paths…Of course for systems of record the same rules apply as for any other system. You should practice the exact things that you will do to establish the system in production in all previous environments and minimise the difference between configurations. The challenge for systems of records is often the cost associated with production like environments So you might have to look for creative ways to minimise the difference and to parameterise configurations so that you can at least automate for the differences.

     

    So all is good in Mirco’s land of the Systems of Record?
    Unfortunately not, there are still many challenges that I am working on. The progress we have made so far allowed for a step change in velocity and reliability, but there are still areas that I want to find better solutions for:

    • Unit test automation
    • Unsupported activities or no API
    • Performance
    • Configuration Management skills – the curse of the configurators
    • Common Objects
    • Operations – the last mile…

    I will speak at much more detail about my experiences at the DevOps Enterprise Summit, if you are attending, come say Hi. I am always happy to show you the scars of working with Systems of Record and share war stories.

    And for two “legacy” technology stacks that I have done extensive work for (Siebel and Mainframe ) I promise to write more in-depth blog posts in the future.

  • Building a Winning Software “A-Team” with DevOps

    Building a Winning Software “A-Team” with DevOps

    The 1980’s TV show The A-team was brilliant. Four ex-commandos with special skills who help the oppressed. Throw in some over-the-top violence where no-one gets hurt, together with some tech wizardry where the team magically create weapons out of old spare parts. Add a catchy theme tune – and voila – classic viewing.

    In many ways agile software development teams want the autonomy portrayed by the A-team. And why not? Let’s bring together small teams of bad-ass code junkies. Provide them with enough open source software to build, integrate and deploy virtually anything. While we’re at it, give them the power to smash all the evil villains’ conspiring to prevent the rapid release of software for the oppressed business – like out-of-touch operations, process-crazy service management, or security and compliance police.

    So on the face of it our A-Team looks like an IT winner. They save the day and no-one gets injured, right? Well not exactly.

    When Agile Autonomy meets an Immovable Object

    Autonomy is both a blessing and a curse. On the one hand, letting teams self-organize and self-manage without outside interference seems like a productivity no-brainer. However, when team autonomy descends into selfish technology-centric rants over rational program level thinking, all hell can break loose. Suddenly every team has access to a plethora of super open source tools — need a document style database for your new Python coded microservice? No problem, click here to download, quickly brush up by reading some appdeveloper articles, and hey presto you’re off and running. Rinse and repeat for different programming languages, build tools, code versioning, configuration management and your teams are all swimming in a fast river of technical goodness where considerations about resilience, performance and security can be easily lost.

    The only problem is that the river flows down to where it eventually meets guess who – all those out-of-touch operations folks, process crazies, and security guards. Not surprisingly, they view this last minute technical deluge with a more cautionary eye. And since they’re compensated and rewarded on preventing problems, security issues and compliance exposures, they’re not going let any software pass freely into production – well, not without a fight anyway.

    And herein lies the problem with the autonomous A-Team. The “A” suddenly becomes more associated with adversarial engagement than anything remotely agile. So cue delays and the business being ticked off yet again with IT.

    Lessons from the Art World – Rembrandt and Stradivarius

    DevOps with its focus on team collaboration aims to channel the healthy tension existing between Dev and Ops into business goodness. This starts by understanding how and where to engage all that natural team angst.

    There’s a classic old sketch from the late British comedian, Tommy Cooper — who walks on stage with a painting and violin, proclaiming – “I bought a Rembrandt and a Stradivarius – the only problem is Rembrandt made lousy violins and Stradivarius couldn’t paint”.  He then smashes the violin into the painting – hilarious.

    In development we have our own Stradivarius’ who write great code, but have little appreciation on what makes a “complete” system. They can add all functional bells and whistles, but when it comes to compliance, resilience and supportability, forget it – they’re like the maestro violin maker with a paintbrush – pretty crap. Worse still, the only time they get to learn is when they’re forced to slug it out during a gated security review or regulatory compliance session – this usually doesn’t end well.

    These issues may be resolved when development, operations and other stakeholders engage much earlier in the software development lifecycle. Here as requirements are established and code begins to take shape, stakeholders can work with development; guiding them on what’s needed for the software to become more secure, perform optimally and be a breeze to support.  For starters, this has to include IT Operations sharing years of expertise – like how to exploit monitoring tools in pre-production to equip developers with an early warning system on performance issues as they code, or explaining what’s needed to support new distributed architectures at scale.

    Engage Early and Broadly for Success at Scale

    Beyond operations, there are many other IT and non-IT stakeholders who should be drafted into the A-Team.

    When working on a new web to back-end system integration for example, security could advise development on end-to-end transaction tracing and static-code analysis techniques to help detect code or SQL vulnerabilities. In an API project, threat teams should engage early to model and mitigate against many new attack vectors that hackers can exploit, providing advice on strong, API-specific infrastructure and management. Naturally, if your team is doing anything with mobile payments you better hunker down early with the security and PCI compliance team – or if the agile team has devised some neat sensor, software and cloud technologies to capture and analyze customer-centric data ala the Internet of Things, then early involvement with the legal department on privacy gotchas is a must.

    Large scale success from agile teams can be realized when those teams become less feral about their own individual needs and more focused on supporting business objectives at a program level. To this end, and whatever the makeup of individual teams, it’s vital to establish DevOps style collaboration early in the lifecycle; engaging all business and IT stakeholders in order to establish resilience, quality, security and even supportability into the fabric of software as its developed.

    Do this and you’ll deliver great software without ever having to compromise speed. Fail, and your agile efforts may just stutter and fail – of course you’ll still have an A-Team, but the “A” now stands for something far less complementary!