devopsy.kinsta.cloud

Category: Enterprise DevOps

  • How do large enterprises do DevOps? Recap of my third video chat with the 2015 DevOps Enterprise Summit Speakers

    How do large enterprises do DevOps? Recap of my third video chat with the 2015 DevOps Enterprise Summit Speakers

    Earlier this morning I had the pleasure of hosting the third video chat with and some of the speakers of DevOps Enterprise Summit 2015.

    This live event featured:

    Paul Reed | Director of Site Reliability Services/Principal Consultant, Salesforce
    Attend Paul’s talk at DOES15: The Blameless Cloud: Bringing Actionable Retrospectives to Salesforce

    Kris Buytaert | CTO @ Inuits.eu
    Attend Kris’ talk at DOES15: Turning Around the Containership

    Michael W. Smith | Technical Director, Hardware & Infrastructure, Raytheon Software
    Attend Michael’s talk at DOES15: Melting the Glacier: Introducing DevOps to a Federal Systems Integrator

    Tapabrata Pal | Product Manager, Capital One
    Attend Tapabrata’s talk at DOES15: Shortening the Feedback Loop: DevOps Dashboard

    Anders Wallgren | Chief Technology Officer, Electric Cloud
    Attend Anders’ talk at DOES15: Continuous Delivery of Microservices: Patterns and Processes, and a panel discussion with Gary Grver, Jez Humble and others on DevOps and Continuous Delivery for Embedded and IoT.

    We discussed how some of the largest enterprises in the world are addressing the biggest challenges facing the DevOps community: from fostering a DevOps culture and leading the transformation, automated testing of legacy applications, security and auditing requirements, managing burnout, fostering an internal consulting capability, and more. Take a look at some highlight tweets and then watch the full recording below.

    The importance of avoiding burnout during a DevOps transformation:

    On security:

    It was such a pleasure to discuss these topics with the best minds guiding DevOps transformations in large, complex organizations. Watch the full recording and hope you join me at DevOps Enterprise Summit 2015 to discuss more!

    The Agile, Continuous Delivery and DevOps Transformation Summit
    October 19–21, 2015
    Hyatt Regency, San Francisco, CA

    Use the code DOES10 to get 10% off your ticket.

  • The perfect fit: what to look for when hiring a consultancy group and how to build a lasting relationship

    The perfect fit: what to look for when hiring a consultancy group and how to build a lasting relationship

    Whether your company is just getting started or has been in business for years, selecting and building the right team makes a significant impact on your output quality, team morale and retention. If you are looking for engineers that are skilled, experienced, and a good cultural fit to step into a new or ongoing technology project, consider hiring a consulting group.

    Once you’ve determined that a consultancy group is the right solution for your business, you will need to select a company to work with. But what should you look for when hiring a consultancy group? Make sure you’re choosing the team that is perfect for your company, and how should you manage them once you’ve started working together? A selective, thorough approach to hiring and managing a team will result in exponential benefits for both your product development and overall team morale.

    Hiring the Right Team

    Look for a dynamic, adaptable and nimble team. While the specs for your project may focus on one aspect today, the next day you may discover the need to pivot in an entirely new direction. Engineers need to understand programming principles and paradigms before being proficient in any language, so make sure that the consultancy group you select has engineers that are skilled enough to easily switch languages and architectures, learn new languages and migrate to different environments, among other things.

    Assess candidates holistically.When looking for the best-fit consultancy firm, be sure to evaluate both their technical and soft skills. Carefully assess their technical abilities and make sure that the skills and experience are there. Be sure to also evaluate their soft skills such as attitude, energy, cultural fit, personality, motivation and ethics. If you cannot meet the team you will be working with in-person, try video-conferencing to get to know them face-to-face, which can help with making sure it will be a fit. Lastly, take the time to look into the consultancy group’s track record of success and what current and past clients have said about their work.

    Setting Your New Team Up for Success

    Prior to beginning the engagement with the consultancy group, be sure to communicate expectations clearly and eliminate as many questions as possible to avoid mistakes and multiple iterations. It can also be very helpful to provide your consultant with a thorough “Requirements” document, wireframes and as much context as possible to help them get up to speed as quickly as possible.

    Motivation is also key with any successful software development team and can make or break a project. Make sure that everyone understands and believes in your vision before embarking on a new project. For a longer-term project, consider switching team members between projects and technologies to allow them the opportunity work on something new and keep them engaged in their work.

    For teams working remotely in various locations, managers should provide honest, regular and respectful communication with their remote teams to build camaraderie, trust and overall passion for the project. Get the most from your team by leveraging any differences that exist, they may turn out to be your greatest advantage in the end!

    Managing your Team 

    As a manager, it is important to create an environment that allows your engineers to focus on what they do best: building software. Be sure to communicate frequently with the remote teams to answer any questions they may have during the project lifecycle and keep it on track. Use a cooperation platform or Project Management platform to organize and track the technical details of your project. Establish any parameters to work around early on and as your project progresses always communicate any changes in the desired outcomes.

    Make sure to embrace your consultants as an extension of your team. If you don’t, there will be disconnect between the teams and the relationship will not work out. Trust is also incredibly important. Consultants are highly skilled and should be able to handle projects you give to them. Also, contrary to popular belief, all consultants aren’t “yes men” and can have great suggestions for how to streamline projects or tweak certain aspects. Keep an open mind and consider what they have to say!

    The process of hiring a consulting firm may seem daunting at first, but taking the time to know what to look for will streamline the process and help you recognize what is most important when building your team. Once you select a firm, trust and open communication between your company and your new team is key. Closely integrating the two teams and fostering mutual respect between them both will help you on your way to building strong, lasting relationships and successful projects.

    About the Author/Theo Nissim

    Theo headshotTheo Nissim, CEO at Gemini Solutions, Theo is a seasoned Silicon Valley technology executive with more than 20 years of experience. He has led pre-IPO start-ups, built dozens of technical teams, developed technology strategies and driven innovative products and solutions to market. One of Theo’s many strengths and passions is managing large scale, cross-functional development efforts to high profile product releases. Theo has held executive and management positions with large public companies, Sun Microsystems and Wind River Systems, and served as the Engineering Vice President for VC funded start-ups, Digital Fountain and AdKnowledge.

    Theo has a Masters Degree in Computer Engineering from the Technion, Israeli Institute of Technology in Haifa Israel. He lives in Palo Alto, California, with his wife and twin boys

  • How do large enterprises address the challenges of DevOps? Recap of my Video Chat with the DevOps Enterprise Summit Speakers

    How do large enterprises address the challenges of DevOps? Recap of my Video Chat with the DevOps Enterprise Summit Speakers

    Last Friday, I was delighted to spend an entire hour with seven leaders of DevOps transformations, discussing the practices and patterns they see for leading a DevOps transformation in large, complex organizations. It was an honor to have them share their lessons learned and tips for how large organizations address some of the challenges of DevOps: from automated testing for legacy environments, getting leadership buy-in and fostering a DevOps culture, DevOps roles and responsibilities, and security and compliance.

    Each of these experts also shared what they will be presenting on at this year’s DevOps Enterprise Summit, taking place October 19–21 in San Francisco.

    Chivas Nambiar  |  Director – Systems Engineering, Verizon

    Heather Mickman  |  Director, Target

    Ralph Loura  |  CIO, HP Enterprise Group

    Neil Manvar  |  Solutions Architect, Sauce Labs, Inc

    Tom Limoncelli  |  SRE, StackOverFlow.com

    Anders Wallgren | Chief Technology Officer, Electric Cloud

    Cornelia Davis  |  Director of platform engineering, Cloud Foundry team at Pivotal

    After quick introductions, I summarized the top five problems facing the DevOps community today – which also form the key focus areas for this year’s DevOps Enterprise Summit. At last year’s Summit, I asked that every speaker end their presentation with a slide addressing one of the two following titles: “Here’s what I don’t know how to do” or “Here’s what I’m looking for help with.” We wanted a glimpse into what the top challenges were that the entire community was facing, which we could then use as a research roadmap.

    Each speaker shared how their organizations are addressing these challenges, and gave a taste of what to expect from their talk at DevOps Enterprise Summit.

    1. How to address culture and leadership issues, particularly around transitioning and transformation

    Cornelia Davis suggested starting small, perhaps with sending some select members to work on pair programming, then have them return to bring the skills back in-house. “It’s one of the best ways of learning new programming. It’s more of a grassroots process,” elaborated Davis.

    Heather Mickman agreed, adding, “We started doing our own internal DevOps Days where we could start building momentum and showing results quickly. To accelerate adoption, we’ve created an internal DevOps dojo to help coach teams on tools and processes.”

    Chivas Nambiar suggested, “Find the leader who shares a common belief, who is close to the work and is aware of these types of technology trends. The second piece of it is you can’t do this [transformation] overnight. We’re entering our fourth year of transformation. Find your key partners, give them what they need to be successful, then share their successes.”

    When I asked, “How do I get my boss’s boss to care about DevOps?” Ralph Loura listed ways to counteract “corporate antibodies” who doubt and resist change: “The top things you need are trust, trust and trust,” he said. “To create this trust, we provide transparency through collaborative tools and pairing people together; we create and embed evangelists to help the next wave of teams succeed and spread knowledge, and help when something goes wrong; and we need data and evidence of success.

    The group discussed the balance of grassroots efforts with getting leadership buy-in, and everyone discussed the value of teams needing to create small successes in order win doubters over. Loura brought up the metaphor of the campfire: “If you start a campfire and you put too much fuel too soon, it can smother the fire. It’s important to keep it shielded at first, until the fire has enough heat to it.”

    1. How to deal with automated testing issues, specifically for legacy environments

    The conversation moved on to experiences we’ve all had with software products or services where we gave development, testing, operations, etc. to a different sourcing partner, thinking we would get better outcomes that way. Heather Mickman shared what the transition was like to go from that model to their current one, “The way we used to do software development in Target was handing off each function to different teams and centers of excellence. There were handoffs to 10–15 different teams. There was no end-to-end accountability for these once-per-quarter releases. So, as we started building out the team that creates APIs for Target, we decided we wanted end-to-end accountability, so we broke down those silos so everyone cares about the end product.” This was just one example of emergent patterns that are coming out of these complex organizations.

    1. How to address IT security and compliance issues – to get your Security team on board with your DevOps plans

    All these presenters are from large organizations who are publicly traded, and have numerous regulatory measures that they must comply with. I wanted to know more about how they work with security and compliance.

    Neil Manvar suggested that DevOps practitioners be completely transparent. He says, “Show the security teams the value, so they can get on board and that you’re not just trying to give them a hard time. Give them access to the change log and very nicely format it or link to the commit. Giving more information about what’s going on really helps with getting everyone on board. Any concerns that are raised, kind of create some automation around it and integrate that into the pipeline.”

    1. What metrics should organizations be using to track their DevOps implementation and performance.
    2. Organizational design, roles and responsibilities

    Holy cow, we didn’t even get around to discussing this last two problem areas. We will be revisiting this and the other problem areas with another group of DevOps experts in another live chat.

    When I asked for any concluding remarks with our last two minutes, Tom Limoncelli responded, “Everyone I know who’s been through this kind of transformation has said the same thing. That when you were done, when you looked back, they said ‘I never want to go back to the other way and in fact in future job hunting, the first questions will be do you have this environment?’ Having a DevOps environment can be one of your biggest recruiting tools.”

    Join the next live chat:

    How do Large Enterprises do DevOps?: Video chat with Gene Kim & the #DOES15 speakers

    Friday, September 11

    10:00 a.m. PDT

     

    Watch the replay of the video chat:

     

    To hear more from these great speakers and other DevOps luminaries, I invite you to join us at DevOps Enterprise Summit next month.

    Learn more and get your pass now at: http://devopsenterprise.io

  • Five things to understand about Enterprise DevOps Solutions

    Five things to understand about Enterprise DevOps Solutions

    There is a lot of buzz in the industry around selling and consuming DevOps. Unfortunately there isn’t the same level of understanding or even a common definition about DevOps between consumers and service providers. In my experience the past couple of years of evangelizing DevOps, the following 5 aspects come to my mind as critical factors while considering and implementing enterprise DevOps solutions:

    1. Evolving Architectural models

    If you are a geek like me, and if you haven’t heard any of the following terms, you’ll probably google them immediately – Container technology, Microservices architecture, single page web apps (SPA), Node.js, mobile web apps, The Internet of things (Mobile awareness, sensors & wearables etc.), web APIs – these are the current trends in the application development space. Almost all of them disrupt traditional architectural models. And the reason I point this out is because they’ll also disrupt your traditional software development lifecycle (SDLC). The number of code & deployable units is only going in one direction, and that’s up. Very rapidly actually! That means source code management, build management, deployment & release management, dependency management, all become so much more relevant and cumbersome. The ONLY meaningful way to make these kinds of disruption manageable is through a mature DevOps approach, and a robust platform to enable DevOps. Developers always want to do the new and coolest thing. Enterprises will be well served to adopt a DevOps strategy NOW, before their own developers will cause so much disruption, that their IT strategy may fall flat.

    1. Integration with existing security and governance models

    More and more enterprises are headed towards a hybrid model of IT which involves traditional on-premise data centers, public cloud, private cloud and consumption of domain specific Software as a Service & Platform as a Service capabilities. Hybrid model also poses the same kinds of challenges discussed with some of the new architectural trends. How do you manage releases across so many disparate entities and providers? A centralized DevOps strategy can help mitigate issues posed in a hybrid model to a large extent. But the question is – what about the corporate policies around security and governance? Do we throw away existing tools & capabilities around change management, ticketing etc.? Disruption is a good thing for innovation, but too much disruption can also lead to disastrous outcomes. One of the key challenges that I’ve encountered while implementing an enterprise DevOps solution is to integrate the platform with existing security models and tools. Not all of them work together seamlessly all the time. But the key to success is to evaluate where the gaps are, and only bringing in new tools or capabilities where necessary, and not just for the sake of it (sell a solution, not a bunch of products).

    1. DevOps enablement

    One of my mentors in IBM used to say, that “a fool with a tool, is still a fool”. How true that is in the world of DevOps! Tooling is a key ingredient towards automation. In fact the concept of DevOps is not new at all, but the kinds of tools available today make automation possible in so many levels that was unthinkable a decade ago. But just providing a platform of seamlessly integrated tools for SDLC isn’t going to work. A level of “design thinking” is absolutely necessary towards DevOps enablement. A DevOps engineer should always be slotted in as a key member of this enablement team. Someone that can coach teams while starting new projects on – What elements of the tooling would apply for them? How to on-board the DevOps platform? What elements of automation are already available vs ones that’ll need to be newly created? What will be the roles and responsibilities of current team members in a DevOps world? – These are the elements that enterprises struggle with. Without a credible enablement program, a DevOps strategy is bound to be very slow in adoption, or just simply fail.

    1. Transformation Investment

    An enterprise DevOps platform and enablement program doesn’t come for free. I’ve constantly been challenged with questions like – what Line of business (LOB) is going to pay for it? Can’t we just use “open source” and make this all free? Can we only pay for what & when we use? All these are fair questions. And the debate about centralized IT expenditure vs LOB expenditure isn’t a new one, and will not go away either. But DevOps is disruptive enough and WILL challenge some of these old ideas and habits. “Automation as a Service” is certainly a viable option that can be availed on a consumption model. Open source tools are available by the dozen. And what comes for free with open source tools is just the usage license, and nothing else. You’ll still have to integrate them with your platform, still write automation scripts and manage them, and enterprise grade tools are usually not totally free and there is some cost involved in acquiring and managing them. All that being said, DevOps IS very much an investment. It is an investment for speed of delivery, higher quality of deliverables through automation and perhaps some cost take out in the form of reduced or re-assigned resources that were performing traditional activities in SDLC. There are several models to ascertain return on investment (ROI) on such investments, but I’ll refrain from getting into that discussion in this article. I’ll just leave this with the infamous cliché – there is no free lunch!

    1. Top Down vs Bottoms up buy-in

    Typically I’ve been involved in DevOps strategy conversations for enterprises at the executive levels, and they seem to be all over it. They love some of the cost take out numbers when they see it, deployment cycle time reduction and aggressive time to market fascinates them. And legitimately so. Getting executive level buy-in and support is absolutely key for any transformation to be successful, DevOps included. But just a top down approach isn’t going to make DevOps successful. It is imperative to reach out to developers and architects under each tower or LOB and get their buy-in as well. Developers and testers love their tools and methods, and are very unwilling to give them up if the strategy calls for standardization of tools (which, by the way has to happen to some degree to gain efficiencies). DevOps also redefines roles in development and testing teams through the level of automation it brings in. This causes a lot of grief amongst the staff, and as a service provider and enabler, I’m sometimes perceived as the ‘enemy’, which is never a good sign for a successful transformation. Developers and testers also have the domain knowledge around how builds are done today, what the application component dependencies are, what test scenarios are more relevant and so on and so forth, information that is so critical to understand for automation. Make friends with the developers and testers to be successful.

    About the Author/ Sunil Joshi

    Sunil JoshiSunil is an Executive Architect with IBM. He is currently the Application Development & Innovation (AD&I) DevOps offering owner for Global Business Services. He is IBM Senior Certified and Open group certified Distinguished Architect. His specialization are in the areas of hybrid cloud solutions, platform as a service and DevOps strategy. He’s created complex hybrid solutions on cloud and DevOps for large enterprises. Sunil has authored several IBM Redbooks, blogs and also a co-author of the IBM Cloud Computing Reference Architecture (CCRA).

    Sunil is passionate about international music, multi-cultural cuisine, active sports and mentoring school & college students on career path and technology.

    Contact Sunil on  linkedin twitter

  • New DevOps Book: Leading the Transformation

    New DevOps Book: Leading the Transformation

    When I first started researching DevOps and was looking for early enterprises that were successful but not “Unicorns” I was introduced to Gary Gruver. At the time Gary had led the transformation at Macy’s and he was talking about it at several DevOps events.  For me it was strong proof that DevOps could work beyond the native-cloud companies and start ups. Gary had made it work at Macy’s and at HP before that. In fact Gary had co-authored a book on his experience at HP, “A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware (Agile Software Development Series)”

    I was lucky enough to meet Gary around then and have stayed in touch since then. He left Macy’s and was consulting helping companies transform from waterfall to agile and DevOps.  I knew that he was working on another book.  Well finally I am happy to report that Gary’s new book is now available:

    leading the transformationLeading the Transformation: Applying Agile and DevOps Principles at Scale. Gary is actually co-author of this along with Tommy Mouser. The forward is by Gene Kim and the book is published by Gene’s IT Revolution.  I have just started reading it, but knowing Gary and from what I have read all ready, it should be a must read on your DevOps book list.

    As a special treat for readers of devopsy.kinsta.cloud, we are very happy and proud to offer our readers a sneak peek at this exciting new book. At the bottom of this article is a link to a PDF with the first two chapters of the book free for you to read! Of course we are sure that once you read the first two chapters you will head over to IT Revolution and buy the full copy.

    BTW, don’t take our word about how good Gary’s book is.  Here is what Jez Humble, VP Chef, author and DevOps leader said:

    “If you want to understand how to lead a Continuous Delivery or DevOps transformation in your company, there’s no better book than this. Concise, practical, and based on hard-won executive experience, this book is essential reading for every IT executive.”

    So click the link and enjoy!

    Leading the Transformation

  • The Art of DevOps

    The Art of DevOps

    This is the third in a series of posts on DevOps. The first written by my colleague Lee Reid was titled The Simple Math of DevOps. The second The Calculus of DevOps was written by IBM client, and my friend Carmen DeArdo of Nationwide. Lee introduced mathematically improving delivery throughput by leveraging DevOps to improve Trust. And Carmen used Calculus to imply that the business value of IT can be improved by adopting DevOps to increase the frequency of releases. In my post I would like to visit the art of DevOps , or the art of adopting DevOps by picking up right where Carmen left off – by looking at the culture side of adopting DevOps.

    Lee and Carmen have already alluded to some basic premises behind DevOps as an approach – reducing batch sizeand improving collaboration. These result in reducing complexity, increasing throughput, and improving trust. Let’s talk about how to get there.

    As I have worked with customers adopting DevOps, across company sizes, across industries and countries, there are four common threads that make DevOps adoption successful, leveraging all three aspects of DevOps adoption – culture, process and automation. These are:

    1. Reduce batch size: The most effective way to manage risk and quality, while increasing speed is to reduce the batch size in each iteration or ‘sprint’. This is a mind shift to release smaller, more frequent new versions. Where it is not possible to release new versions that frequently to customers, release small batches to a pre-production area. The deliverable in pre-production is tested and made customer-ready, and then released to the customer at formal release dates. As Lee and Carmen have both discussed, reducing batch size is a pre-requisite to improving both trust and business value.
    2. Shift-left Operational engagement: Get operations engaged right from the beginning of a project/initiative. This has been a core principle of DevOps. Do not leave Ops in a silo that one throws code over the wall to, but have them engaged in the development process, right from requirement inception stage. This allows Ops to have visibility into what is coming from Dev; allowing them to deliver the right production-like environments, as and when needed to Dev and Test teams. It also allows Ops to shift operational concerns left, left into the minds of developers, making them more Ops conscious and engaged in what happens after they hand-off to Ops.
    3. Continuous funding: This is a significant change in an organizations funding model to enable continuous availability and engagement of SMEs and business functions, previously not engaged on a continuous basis – like security, legal, marketing, etc. Carmen in his post mentioned several examples of ‘waiting for…’ work, someone else to do something and environments. Another ‘waiting for’ in enterprises is for funding. Funding is typically provided in a waterfall manner, with specific hard dates (Months, quarters, fiscal year) and gates, not suitable for a Continuous Delivery model. Funding too should be continuous.
    4. Create a ‘Product Management’ team: This team includes a Designer, Development lead, operations lead and an Architect at the minimum (4-in-box). They own the ‘product’ through its entire lifecycle, and beyond transient projects. They are responsible for long-term thinking, design thinking, evolutionary architectural thinking, and overall ownership of the product, from concept to end-of-life.
    5. Setting up a DevOps Center of Excellence (CoE): This center of excellence is not an administrative organization or a ‘tools/enablement group’, but a place where DevOps adoptees come to learn from each other and share expertise and lessons learnt. As organizations adopt and scale DevOps adoption, this CoE also becomes a source of DevOps coaches, helping teams and programs adopt DevOps, and own the organizations DevOps framework – their own flavor of DevOps.

    Adopting DevOps is an art. It is not as simple as hiring a consultant who can show how to improve processes; it is not just buying and adopting tools to automate manual tasks; and it is not going through ‘fall-back-into-the-arms-of-the-person-behind-you’ exercises to build trust. It is a combination of all three – process improvement, organizational and cultural change, and automation with tools to replace manual processes. Adopting these in any enterprise, outside of startups, requires overcoming organizational inertia. By changing culture not by edict, but by showing the need and value of change. By building trust, not through meetings, but with improved communication and transparency. By improving business value, not by measuring mandated KPIs, but by improving organizational agility and throughput. Overcoming organizational inertia, by delivering towards a common goal for the business, irrespective of where in the organization one is, and what role one has. By developing a culture where everyone takes responsibility of delivering value to the business.

    Watch the webcast series hosted by devopsy.kinsta.cloud in partnership with IBM or download a copy of DevOps for Dummies book for more information.

    About the Author/ Sanjeev Sharma

    SanjeevSanjeev Sharma, CTO and Distinguished Engineer – DevOps Technical Sales and Adoption, IBM Cloud Unit, is a 20-year veteran of the software industry with expertise in DevOps, Mobile Development and UX, Lean and Agile Transformation, Application Lifecycle Management and Software Supply Chains. He is a DevOps Thought Leader at IBM and speaks regularly at conferences. He has written several papers and is the author of the DevOps for Dummies book.

     

    Contact Sanjeev on linkedintwitter

  • Jut Operations Data Hub for DevOps

    Jut Operations Data Hub for DevOps

    “You have to be able to ask questions of your data in real-time and then respond to the answers,” says Apurva Dave, VP of Marketing, Jut. Jut leverages live streaming, batch analytics, and visualization to enable enterprises to ask important queries of their operational data using the Jut operations data hub, which Jut created with DevOps goals (such as Continuous Deployment with Quality) in mind.

    Jut supports real-time and historical data queries across all log, event, and metrics data. “Our platform is built on dataflow, dealing with large scale streaming data,” says Dave. Jut is an option for organizations that don’t want to go out and build their own operations data hub software.

    Core Data Types

    Jut intends to join data such as user activity (on websites and software), event data (triggered operationally such as support ticket filings, customer transactions, and software upgrades or migrations), and unstructured data (application and system logs) together for analysis, says Dave. Jut works with structured data such as metrics data (such as application level data like the user numbers and key action numbers or system level data like CPU utilization, memory utilization, and network I/O data), explains Dave.

    Jut attempts to analyze the whole of your data to answer questions you may have like: “What resulted when I deployed a new version of my application into production?” Jut might answer based on an analysis correlating system resource utilization (“Was the app too resource intensive this time? More so than we expected?”).

    Or you may ask Jut this question: “How did my code respond to user activity / user demand? How did it perform?” Jut could ask this question based on its assimilation of both unstructured and structured data. “It’s common for enterprises to A/B test two separate versions of a site or software feature. Using Jut, you can ingest those test results and analyze them in real-time to determine what version produces the desired effect,” says Dave.

    Big Data Analytics & a Huddle about Juttle

    “We have users ingesting tens of billions of data points per month. That number is growing fast,” says Dave. Jut targets big data management for easier analysis using data infrastructure techniques that simplify / streamline, integrate, and manage big data backend infrastructure. “We also use our own dataflow language to quickly iterate and visualize lots of data,” says Dave. All this is designed to ease DevOps teams’ work in accessing and assessing operational data to discern how the software and the business that it serves are performing.

    Jut designed its Juttle dataflow programming language for data analysis and data visualization. And Juttle is dedicated to those tasks alone. “While you could try to cobble together the right packages and libraries in a general language like Python or Java to do these analytics, it would be relatively messy, time-consuming, and error prone,” says Dave.

    Juttle is an extensible language enabling analytics and visualization. “Jut’s data scientist built an anomaly detection algorithm entirely in Juttle and then published it on GitHub for anyone to use inside Jut,” says Dave.

    Juttle uses a structured approach to querying your data. “The approach looks like a series of piped UNIX commands, so anyone familiar with scripting languages such as JavaScript or query languages such as SQL should feel at home with Juttle,” says Dave. Jut responds to questions in seconds to tens of seconds (for larger data sets) based on iterative, interactive analysis.

    Investigate, Question, Consider (Try?)

    Outsourcing to Jut puts the enterprise in a position to focus on existing coding duties while proving whether Jut is the better operational analytics move for analytics performance and cost-savings (Jut offers a free trial). But as with the analytics themselves, ask a lot of questions of Jut. Do your homework before committing even to trial software (Jut has a demo and an online “playground” as well).

    Tip: Jut made sparkling comments from its Beta-development-phase customers such as NPM and Mylio available. Rather than include those for your consideration, I suggest you reach out through your connections to these and other enterprises that are experimenting with this data hub software.

  • The Calculus of DevOps

    The Calculus of DevOps

    This is the second in a series of blogs, partnering with Lee Reid, who wrote the first installment “The Simple Math of DevOps” and Sanjeev Sharma both of who are outstanding DevOps Thought Leaders from IBM.

    I am picking up on Lee’s mathematical DevOps model to apply a calculus perspective, a “calculus of DevOps” if you will, (OK, not quite core Big Bang Theory Calculus but rather approaching it from the aspects of engineering queuing theory).

    One of the cultural challenges that DevOps poses is, that some aspects of it run counter to engineering principles that many of us learned with respect to capacity and response time.   When I worked at Bell Labs, we would apply queuing theory to determine how to best package information to be sent across the network. A basic premise revolved around a tradeoff between throughput and response time. On one hand, the more messages that were packed into the packet, the more the payload would improve capacity because the overhead of processing the packet of information would be spread across the messages.   However, waiting to accumulate enough call processing information meant that the packets would be transmitted less frequently, increasing the response time.

    A simple example of this would be making trips to the grocery store. If one wanted to spend the least amount of resources (time, gas, car wear, etc.) on shopping, then one would make less frequent trips and then buy a larger amount of food. So if the Smith family made one trip a month and spent 2 hours, then their average time per week spent on shopping would be 30 minutes.   If the Jones family went every week and spent an hour, then they would be spending twice the time and four times the gas on grocery shopping. However if the Smith family run out of an item (say cereal), they would have to wait, on average 2 weeks before this was replenished. The Jones family average on any item that they ran out of would be 3.5 days in comparison.

    So when the 2014 DevOps report was published[1] which states that areas that deploy more frequently get customer feedback more quickly, that is to be expected. But what’s not expected is that these organizations also have higher productivity.   So exactly what is happening here that seems to be in a basic conflict to the principles of throughput and response time?

    Keeping up with the JONESES

    If we think of a pre- DevOps state for a medium to large enterprise, it may be typical for applications to only be released (deployed into production) a few times a year to provide business functionality. (I assume there are more frequent releases for “unplanned work” like defects and other changes or operational work). For the sake of this discussion, let’s call this the Smith Application. So from a business perspective, the business wants to pack as much work into the Smith releases as possible given they only have a few cracks a year at getting new business functionality into production.

    SMITH APPLICATION CHARACTERISTICS:

    • Deliver Business Value 3 times a year (every 4 months)
    • Cost is C1
    • Business Value (BV) delivered is V1
    • Average time from concept (e.g. Story) being ready to be pulled from the backlog to the time it is delivered to the customer is 2 months plus any additional wait times (e.g. prioritization)
    • BV delivered per year is 3*V1
    • Cost per year is 3*C1

    Now let’s transport over to the Jones Planet where the application is released into production with new business capabilities at the end of every 2 week iteration.   In this case the business is more concerned with providing a constant demand flow into IT than they are with debating over whether a given feature/story is going to be included in this particular release.   Instead there is a constant prioritization (or continuous planning) process that ensures the highest value work is being fed into the IT pipeline.

    JONES APPLICATION CHARACTERISTICS:

    • Deliver Business Value 26 times a year (every 2 weeks)
    • Cost is C2
    • BV delivered is V2
    • Average time from concept (e.g. Story) being ready to be pulled from the backlog to the time it is delivered to the customer is 1 week plus any additional wait times
    • BV delivered per year is 26*V2
    • Cost per year is 26*C2

    Based on the previous engineering analysis, we would expect the following:

    1. SMITH annual cost (3C1) < JONES Annual Cost (26C2)
    2. SMITH annual BV (3V1) > JONES annual BV (26V2)

    So the tradeoff (we would believe) JONES is making by delivering more frequently is that they are spending more money and are delivering less business value on an annual basis.

    However the data from the previously referenced DevOps report is just the opposite. In fact the findings are that enterprises that deploy smaller units of work more frequently have much higher quality and much higher productivity (Capacity). So what’s going on here?

    A NEW PARADIGM

    If you go up to any IT team (even an agile one) and ask them why they don’t deploy business value more frequently, they will generally refer to the following types of wait states of waste that they encounter:

    • Waiting for work (lack of consistent flow)
    • Waiting for somebody else to do something (dependencies from other teams)
    • Waiting for environments (contention)

    One major root cause of this is the dependencies that arise from large enterprises packing a lot of work into infrequent releases. From a planning perspective, the process is elongated due to prioritization discussions because being on the wrong side of the line can mean delay amounting to months, in delivering a product to a customer.   Ask a business portfolio leader what their highest priority is for IT and I am guessing they can tell you in a matter of seconds. However, most large organizations can spend several months a year planning their annual portfolio of work for IT. Some organizations spend more than 50% of their new project build time prior to a single story entering an agile development team’s backlog!

    From a development perspective, packing a lot of work into releases means many more dependencies that have to be managed across initiatives and impacted applications. From a testing perspective, downstream environments (ST, UAT) have to manage much more information in terms of application versions and especially test data that complicate and slow down the process. The complexity of test data increase is proportional to the number of pairwise relationships that exist within the datasets required for testing.   As such, the testing complexity will increase non-linearly (perhaps proportional to the square of the dependent systems) and from a release and change perspective; there is a lot of coordination necessary given how much is being changed at one time.

    All of this work to manage all of the dependencies found in medium to large enterprises completely eliminates any potential savings one might get from batching work into a single delivery. In addition, the possibility of having to pull some work out of a release (due to quality or business readiness or some other reason) sometimes leads to complicated source code management (feature branches in some cases) that further complicate the process of merging frequently into the trunk. These types of practices also serve to drive up technical debt. So looking at the problem from this angle shows why results we are seeing make perfect sense.

    Productivity is inversely proportional to the number of dependencies in a release

    As Lee related to in the initial blog, a lot of an organization’s success depends on trust. And one factor of trust is clarity which speaks to understanding the changes that each release is bringing to the IT and customer environments. So a factor that is working against trust is the notion of delivering more things less frequently and better controlling change somehow, is a good notion.

    Trust is inversely proportional to the number of dependencies in a release

    There is another hidden component of waste associated with “big bang” initiatives. Go back to our grocery example for a moment. Chances are that the Smith monthly excursion results in buying some stuff they end up not using. Perishable items are not spread out very well when they aren’t purchased more frequently. Likewise in IT planning, we tend to put a bunch of stuff in our IT Delivery basket that we don’t know for sure that we will need and we end up with a lot of wasted effort for features that are hardly used or not needed at all. Some enterprise estimates are that 50-65% of what they put into large releases does not actually result in true value to the business. Doing more frequent releases allows a constant discovery process to better guide the usage of IT resources on what is truly of value.

    Waste is proportional to the size of a release.  

    In other words:

    Yield (Value per IT investment) is proportional to the frequency of releases.

    REVISTING THE MATH OF DELIVERY TIME

    So if we revisit Lee’s model

    Lee Reid Blog equationand apply the “Calculus of Release” as we stated above, there are increased costs associated with having larger and less frequent releases in medium/large enterprises.

    • Planning cost TPLAN
    • Development complexity TDESIGN + TDEVELOP + TBUILD
    • Testing Complexity TTEST
    • Release and Change Management Complexity TRELEASE + TDEPLOY + TEVALUATE

    TRUST % – Even if everything was proportional between the JONES and SMITH models, the component of trust that impacts clarity, consistency and collaboration, is much lower for large (SMITH) releases.  Given that TRUST has a multiplicative impact, this alone can account for the results found in the studies.

    CULTURAL CHALLENGE

    While clearly one needs to be sensitive to the changes being introduced into the business, having the ability for the business stakeholders to pull capability when they want it and having it available more frequently is not only good for customers but also has extreme benefits from an IT perspective in terms of increased productivity and reduction of technical debt. The challenge is to convince leaders in both the business and IT areas on the merits of doing more continuous planning and frequent deliveries.   Unicorns may have led the way in this type of thinking but it is actually the horses that will benefit most from this type of culture shift because it is the horses[2] that have the most to gain in mitigating the dependency factor which can undermine quality, productivity and cycle times.

    Read more about the topic in the next blog of this series The Art of DevOps by Sanjeev Sharma.

    [1] “2014 State of DevOps Reports”, Puppet Labs, IT Revolution, https://puppetlabs.com/sites/default/files/2014-state-of-devops-report.pdf

    [2] “The Phoenix Project”, Gene Kim, Kevin Behr, George Spafford, http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592

     

    About the Author

    CarmenCarmen DeArdo is the Director of Application Development Tools and Technologies at Nationwide Insurance. Carmen is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across Mobile, Distributed and Mainframe and other technologies. This includes recommendations and implementation of technologies used across the development life cycle (e.g. IBM Rational tool suite, open source technologies).

  • Proactive Monitoring

    Proactive Monitoring

    I think, most of the DevOps discussions are centered around testing the code change, packaging and deploying it in production in an automated fashion. However, stinging site-down issues would eventually force any nascent production engineering team to make monitoring a high priority down the road.

    If functional, integration and acceptance tests are run to verify a code change before that was deployed in production, in my opinion, similar checks should be part of monitoring to make sure that the application works fine in production also, as designed. In production, code change is not only the factor that could impact the application. Updates of OS, network configuration changes, automatic upgrades of third-party tools, availability of system resources etc. can also impact application in production. But, I am yet to come across any framework that integrates testing and monitoring efforts.

    In a traditional sysadmin centric environment, the scope of monitoring doesn’t go beyond infrastructure layer. The operations team that owns monitoring might not have visibility into what needs to be monitored beyond infrastructure. For the latter, out-of-the-box solutions are readily available on popular monitoring platforms like Nagios.

    Here is an attempt to classify the monitoring requirements in production. The requirements are generic and are applicable for any system hosted in-house, in a data center or in the cloud. A systematic and proactive plan for testing code changes and monitoring production would help to prevent incidents in production. Classification of tests and tools to implement them have been around for a while and that hardly require any additional attention. This article is an attempt to classify the monitoring types and to document their implementation methods.

    Monitoring Infrastructure

    The infrastructure that hosts an application environment would be made up of multiple components: servers, storage devices, load balancer etc. Checking the health of these devices is the most basic requirement of monitoring. The popular monitoring platforms support this feature out-of-the-box. Very little customization is required except for setting up right thresholds on those metrics for alerting.

    Monitoring Platform

    An application would typically be built using multiple third-party tools such as databases, both RDBMS (MySQL, Postgres) and NoSQL (MongoDB, Couchbase, Cassandra) data repositories; full-text search engines (ElasticSearch) ; BigData platforms (Hadoop, Spark); messaging systems (RabbitMQ); memory object caching systems (Memcache,Redis); and BI and reporting tools (Microstrategy, Tableau). Checking the health of these application components is important too. Most of these tools provide some interface, mainly via REST API, that can be leveraged to implement plugins on the main monitoring platform.

    Monitoring Application

    Having a healthy infrastructure and platform is not good enough for an application to function correctly. Buggy code from a recent deployment or third-party component issues or incompatible changes with external systems can cause application failures. Application level checks can be implemented to detect such issues. As mentioned before, a functional or integration test would unearth such issues in a testing/staging environment, and, an equivalent of that should be implemented in the production environment also.

    The implementation of application level monitoring could be simplified by building hooks or API endpoints in the application. Monitoring is usually an after-thought and the requirement of such instrumentation is overlooked during the design phase of an application. The participation of DevOps team in the design reviews improves operability of a system. Planning for application level monitoring in production is one area where DevOps can provide inputs.

    Monitoring Business

    The applications run in production to meet certain business goals. You can have an application that runs flawlessly on a healthy infrastructure but still the business might not be meeting its goals. It is important to provide that feedback to the business at the earliest to take corrective actions which might trigger enhancements of the application features and/or require the way business is run using the application. These efforts should only complement the more complex BI based data analysis methods that could provide deeper insights into the state of the business. The business level monitoring can be based on transactional data readily available in the data repositories and the data aggregates generated by the BI systems.

    Both application and business level monitoring are company specific, and, plugins have to be developed for such monitoring requirements. Implementing some framework to access standard sources of information such as databases and REST APIs from the monitoring platform could minimize the requirement of building plugins from scratch every time.

    Last-Mile Monitoring

    A monitoring platform deployed in the same cloud or data center environment where the applications also run cannot check on the end-user experience. To address that gap, there are several SaaS products are in the market, such as Catchpoint and Apica. These services are backed up by actual infrastructure to monitor the applications in specific geographical locations. For example, if you are keen on knowing how your mobile app performs on iPhones in Chicago, that could be tracked using the service provider’s testing infrastructure in Chicago.

    Alerts are setup on these tools to notify the site reliability team if the application is not accessible externally or if there are performance issues with the application.

    Log Aggregation

    In a production environment, huge amount of information is logged in various log files, by operating system, platform components and application. They will get some attention when issues happen and normally are ignored otherwise. The traditional monitoring tools like Nagios couldn’t handle the constantly changing log files except for alerting on some patterns.

    The advent of log aggregation tools like Logstash, Loggly and Splunk changed that scenario. Using the aggregated and indexed logs, it is possible to detect issues that would have gone unnoticed earlier. Alerts can be setup based the info available in the indexed log data. For example, Splunk provides a custom query language to search index for operational insights. Using APIs provided by these tools, the alerting could actually be integrated with the main monitoring platform.

    To leverage the aggregation and indexing capabilities of these tools, structured data outputs can be generated by the application or scripts that will be indexed by the log aggregation tool later. Such data aggregation can be used for generating data for reporting applications. For example, if storage usage on a set of computing nodes has to be tracked, a daily storage usage report can be generated on related nodes which the aggregation tool can track. Weekly and monthly aggregates could also be computed once the daily aggregates would be available.

    Monitoring the Monitoring

    It is important to make sure that the monitoring infrastructure itself is up and running. Disabling alerting during a deployment and forgetting about enabling it later is one of the common oversights I have seen in operations. Such missteps are hard to monitor and only improvement in deployment process could address such issues.

    Pinging hosts

    If there are multiple instances of the monitoring application running, or if there is a stand-by node, then cross checks can be implemented to verify the availability of hosts used for monitoring.

    In AWS, CloudWatch can be used to monitor the availability of an EC2 node.

    Health-check for monitoring

    Checking on the availability of monitoring UI and activity in monitoring application’s log files would ensure that monitoring system itself is fully functional and it continues to watch for issues in production environment. If a log aggregation tool is used, tracking monitoring application’s log files would be the most effective method to check if there is activity in the log file. The same index can be queried for any potential issues also by using standard keywords like “Error”, and “Exception”.

    Conclusion

    Monitoring efforts are normally in response to issues happen in production. A systematic approach to rolling out monitoring can minimize the reactive ways it is normally associated with. Proactive monitoring adds to better user experience, and, it avoids costly data reprocessing and rollback in production.

  • The Whole Nine Yards with DevOps and Agile Operations

    The Whole Nine Yards with DevOps and Agile Operations

    Growing up as a boy in Manchester, UK, football (ok, soccer to US readers) was everything to me. I vividly remember being herded into the stadium stands on cold rainy Saturday afternoons to watch my team (#MCFC) win, draw or lose. Back then, football was for the die-hards – no frills or fancy tech, just hot tea and a suspect meat pie at half-time.

    How things have changed. Now sporting events have undergone a complete transformation – all digitally-driven.

    High Tech Sports

    Take Levi Stadium, home of the San Francisco 49ers for example; considered by many to be the most high-tech stadium in the world. There’s too much tech to describe in detail, but you can be sure that all the Wi-Fi beacons, 400 miles of cable, photo voltaic energy and a funky mobile app have been designed with one thing in mind – enhancing the fan experience. So no ‘cattle-herding’ at a 49ers game – nope, a personalized digital experience that’ll find you your seat, food concessions, the closest bathroom — and of course the shortest beer line (#1 feature in my opinion).

    Levi Stadium isn’t just another example of technical wizardry. It’s more an exercise in an organization deeply connecting with its customers through the delivery of highly experiential services. Understanding the needs, motivations and desires of their loyal fan base and servicing them accordingly.

    It’s a pity this same thinking isn’t always applied in IT Operations – that is, deeply understanding what Ops ‘fans’ need if they’re to ensure a good business user and customer experience across new highly dynamic agile environments.

    High Stakes IT Operations

    Similar to the Manchester game day experience years ago, IT Operations has often done it tough. They’ve worked in isolated NOC’s and service centers, been placed on 24/7 call; working weekends and missing their kid’s birthdays. They have been served specialized ‘team merchandise’ in the shape of complex tools and processes – always counting on their loyalty to save the day when the next mega release comes over the wall. No great surprise then that Operations in many organizations is seen as uncommunicative and uncompromising – almost, well – tribal.

    But IT tribalism doesn’t win matches in the software-driven economy game. Now, digital business realities means IT Operations must supplement its traditional fixation on reliability and resilience with a willingness to drive change – Operations must itself become more Agile. But how do you start? Thinking back to the innovation at Levi Stadium, here are three good ways.

    1. Gaining Loyal Supporters – Development and Operations aren’t enemies, but like rival UK football team supporters the teams are often segregated. Dev and their tools to the left, Ops with theirs to the right. From a business perspective this is plain dumb – especially when it comes to getting the most value from technologies like application performance management. Rather than frantically monitor applications as problems manifest in production and then call Dev for help, wouldn’t it be better to incorporate operational intelligence much earlier in the software lifecycle? So when code is released, monitoring could immediately provide development with an early warning on operational issues. Conversely, Ops practitioners gain accurate and timely insights into production performance requirements and KPI’s – that’s not only a DevOps style win-win, it’s an agile like best practice to increase quality and avoid technical debt.
    1. Filling the Business Stadium – The goal of Operations has always been keeping the business lights on, but Agile Operations now means making them shine brighter. Cloud, Microservices architecture and containers all give businesses the technical clout needed to support speed at scale, but it comes at a cost. Today’s sophisticated applications rely on specialized technologies, so it’s perhaps understandable that Operations chase the technology down the proverbial rabbit hole with more specialized tools. But the real truth is that any monitoring system which doesn’t deliver true business value by providing unified insight into how customers are actually experiencing IT services is of no use to anyone. Remember too that technology scale is about future proofing your business, so look for scalable monitoring to match it.
    1. Delivering a Complete Experience – The digital experience for a 49ers fan doesn’t start and end on game day. It is delivered by a linked series of services – a digital journey comprising seat selection, ticketing, parking, entry, action replays, and player stats reviews – every one of which can positively or negatively impact the fans experience. Whatever your industry, IT Operations will be called upon to support similar services, meaning deep and uninterrupted visibility across the myriad of front and back technologies underpinning every customer interaction, engagement and transaction will become essential.

    Successful digital transformation needs more DevOps and agile operations, agile development, faster release cycles and higher quality software. It’s time for IT Operations to stop fixing production problems and become an active participant in supporting these goals. Without Agile Operations, your role in the software-driven business game ferociously being played out will be like that meat pie I ate many years ago – suspect.