Category: DevOps Practice

  • So Many DevOps Tools, So Little Time

    So Many DevOps Tools, So Little Time

    I’ve received thousands of emails, ads and sales pitches for tools for every stage of the app life cycle, from automating infrastructure buildout to release planning to building and testing, to delivering applications. If you’re building or rearchitecting an application, you have a surfeit of choice of DevOps tools.

    Ironically, when an organization invests in various DevOps tools, it can lead to disconnected teams that are siloed by tooling. Gartner calls this “disconnected islands of automation.” In addition to a culture problem, it can lead to major incidents and downtime.

    You can’t tool yourself into DevOps. But here’s how to make sure you’re not hurting your team with the wrong tools.

    Step 1: Process Before Tools

    Find out what problems you need to solve, then find tools to fit that need. It sounds simple, but many companies do the opposite: they’ve already picked the tool, and then use it to attack every problem.

    “If all you have is a hammer, everything looks like a nail.” If all you have is one or two main tools, every problem gets fixed by those tools. Our team has been guilty of this, too. When you get really good at one tool, such as Puppet or Chef, the answer to every question is inevitably Puppet or Chef.

    For our team, the fix has been to create process and architecture diagrams without tool names. Instead, we describe what needs to happen at each stage. If I know I’m building a long-running virtual machine, I know I need a configuration management component. Once we agree on the process, then we discuss tools.

    Engineers are practical people who are trained to be constrained by the tools we use. To break out of this mindset, write out your desired code pipeline without tool names, even if you “know” the DevOps tools you’re going to use.

    Step 2: Give Equal Priority to Infrastructure and Application Tools

    A software company recently told me an unhappy story: The company spent years perfecting its continuous delivery pipeline and everything worked together like clockwork. Developers could test and run code in minutes.

    The development team knew how to push code, but nothing was reachable in their region. Some of its delivery tooling was running in the same cloud region, so even though other regions were up, the tools the team wanted to use to perform failover weren’t available. When the team went to fail to DR, the configuration for its DR infrastructure had not been kept up to date with production, so instance sizes were not sufficient to run the application.

    The moral of the story here is that your infrastructure and application need the same level of automation and agility. Even if you have a very mature CI/CD pipeline with gating, approvals, linting, etc., if you don’t have that same discipline and tooling around your infrastructure stack, then you’re equally down. You should have the ability to move your application from one region to another (and if you’re really ambitious, from one cloud to another). Also, if you’re not incorporating application development into infrastructure development, over time your infrastructure will grow out of date.

    You should be asking questions such as:

    • What steps can we take to automate infrastructure provisioning?
    • How can we continually test failure?
    • What steps can we take to survive a platform failure?

    Step 3. Integrate Infrastructure and Application Tools

    The best way to survive a platform failure is to continually “practice” building out entirely new infrastructure and application stacks—as one continuous process, not as two different activities.

    Here’s one extreme example. The team at Logicworks recently worked with a large enterprise software company that was launching an internal product with a custom deployment pipeline. The pipeline included dozens of parallel automated and manual tests, each in a separate development environment. When a new deployment to the development environment occurs, a new set of 45+ dev instances must be created that has the new version of code; when the test is finished, the instances must be terminated. This means that instances in their environment rarely last for longer than 24 hours, and over the course of a single week, hundreds of instances are terminated and rebuilt.

    As you can imagine, this company has never experienced an infrastructure outage it couldn’t recover from, since it already recover from “failure” hundreds of times a week. This is the true meaning of immutable infrastructure: virtual instances are disposable and once you instantiate the infrastructure and code, you never change the instance. This way the infrastructure never strays from its initial, known-good state, operations are simplified and the system is so good at replacing itself that failure is a non-event.

    Ideally, infrastructure automation and application automation are not different stacks with different teams, but the same stack with the same level of destructive testing, so that an advance in your application needs to be tested against the current state of your infrastructure.

    Summary

    The constant release of new DevOps tools and features can be overwhelming. But the biggest challenges for your DevOps team probably have nothing to do with CI/CD tools. I have found that it’s easy to overemphasize tools and deprioritize crucial (albeit less glamorous) automation components such as infrastructure templates and testing.

    Your developers are constantly looking for new libraries and experimenting with new languages and tools for your applications. You need the same level of attention and testing in your full infrastructure and application stack. This will make you choose better application-level tools and prepare you for failure.

    — Jason McKay

  • Common Hidden Dependencies and How to Uncover Them

    Common Hidden Dependencies and How to Uncover Them

    It’s no secret that a truly Agile approach minimizes the risk organizations assume with waterfall or other traditional project management methods, by letting teams iterate and inspect and adapt as they go. But many companies lack clear visibility into real-time dependencies at the team level—even with daily standups, iteration and release planning, and a regular cadence of cross-team check-ins. This lack of visibility impacts organizational agility and can erase the value of an investment in agile.

    As Dwight D. Eisenhower said, “In preparing for battle I have always found that plans are useless, but planning is indispensable.” When it comes to planning within an enterprise, being aware of the biggest risks of hidden dependencies will help you keep a proactive approach and drive value from strategic objectives to tactical execution. Here are some of the risks of the most common dependencies and how to uncover them:

    1. Teams that are distributed (or use multiple Agile platforms) do not have the flexibility to work across teams as easily as their co-located counterparts. This slows down response time, which in turn slows down releases. Teams must make dependencies transparent at the highest level to stress their importance. Make sure your team has an advocate, such as a Scrum Master, to raise red flags when dependencies or risks emerge.
    2. Medium- and large-scale organizations may run multiple release planning sessions for one product. Running them at the same time, or in a sequence but in different locations, increases the risk that teams will overlap work and not plan for dependencies.
    3. Many organizations and teams try to plan for dependencies by adding more time to their estimates as a buffer. While this seems like a good idea, it hides a problem by burying wait time and masking the real blocker. Organizations and teams should ensure that estimates are only based on complexity and effort and do not reflect “waste” such as wait time. Doing so helps reveal impediments for what they are.
    4. In larger organizational structures, infrastructures are complex and teams tend to be highly dependent on one another delivering work, with a chain effect of one team’s work beginning only after another finishes. As they look to adopt and scale Agile, organizations should closely examine their structure to ensure it is optimized. This will reduce risks and dependencies, rather than exaggerate them.
    5. Team members often allow “blockers” and “dependencies” to get in the way of completing work while still committing to a sprint or release goal. If team members foresee a dependency or blocker with key features, user stories, etc., they should not commit to the work. It is the responsibility of each individual to refrain from committing to work that they know they cannot complete.

    Visualization is Key

    Organizations can adequately manage dependencies and risks if they can plan for them. One helpful technique is to create a road map for releases. When consistently updated throughout at the end of each sprint, a road map helps teams identify risks and evaluate dependencies continuously. Selecting a platform that allows real-time visualization into dependencies is another way to prevent time-wasting while giving insight into the right details at the right time.

    — Andrey Mihailenko

  • Using Netflix’s HubCommander to Automate GitHub Organizations

    Using Netflix’s HubCommander to Automate GitHub Organizations

    Netflix owes a great deal of its exponential growth to its phenomenal tech stack. Throttling its content through a single internal API, the company was able to deliver content agnostic of device type and quickly disrupt its contemporary competitors in the consumer entertainment industry.

    We’re intrigued, then, when Netflix exposes some of its internal architecture for others to use. HubCommander is just one recent addition to Netflix’s growing collection of open source tools. It’s a ChatOps tool for GitHub management.

    We’ve previously covered an open source approach to ChatOps, tracking how to bring free operational tools into the conversation for increased transparency and efficiency. In this article, we’ll resume this research by learning how Netflix has utilized HubCommander and delve into its open source iteration to see how others might leverage it to automate its GitHub organization.

    Problems

    Managing many GitHub repositories while working in a large organization can be frustrating; especially difficult is the task of managing permission levels between various user subsets. Not only are various user permission levels difficult to trace, but there is an intrinsic security problem to address as well, especially when working with external teams.

    “One of the biggest challenges with using GitHub organizations is user management.” — Netflix

    Project managers need to administer different organizations, and GitHub requires admin privileges to manage such repository settings; if onboarding thousands of employees, editing this manually could be an extremely tedious job.

    Benefits!

    At Netflix, HubCommander standardizes its user management approach, enabling automation for GitHub repo permissions across development teams.

    “Management of many users on GitHub can be a challenge without tooling. We needed to provide enhanced security capabilities while maintaining developer agility. As such, we created HubCommander to provide these capabilities in a method optimized for Netflix.”

    HubCommander thereby provides this missing link, granting admins the ability to quickly grant privileges. Easily to embeddable into chat, the HubCommander is made with ChatOps in mind. Therefore, privileged GitHub organization management tasks can be performed from the Slack channel, “without granting administrative or owner privileges to your GitHub organization members.”

    Using HubCommander

    To use HubCommander, first, you must have Python 3.5+, Slack, a GitHub organization, and a GitHub bot user with ownership level privileges.

    Next, Slack channels with HubCommander installed can make the call !help to return a list of bot commands:

    Source: Netflix

    To create a repo, for example, you can simply use the command !CreateRepo:

    Source: Netflix

    Utilizing HubCommander also has security benefits as it reduces the number of GitHub API permissions granted per user—reducing attack vectors inherently increases security. Another point on security is HubCommander’s integration with Duo for additional authentication:

    Source: Netflix

    The total list of HubCommander features include:

    • Repository creation
    • Repository deletion
    • Repository description and website modification
    • Granting outside collaborators specific permissions to repositories
    • Repository default branch modification
    • Repository PR listing
    • Repository deploy Key listing/creation/deletion
    • Repository topics creation/deletion
    • Repository branch protection enabling/disabling
    • Enable Travis CI on a GitHub repo
    • Safeguard commands with 2FA via Duo

    Automating GitHub With ChatOps

    If your code resides in multiple Git repositories, like Netflix, having a method to configure permissions across all entities in a convenient way becomes a necessity. With a bot to perform operational tasks on GitHub repositories, Netflix is benefiting from automatic manipulation and a wider team access, adding to its agile goals:

    “The reduction in administrative overhead has significantly simplified our open source efforts.”

    Netflix operates with three development channels (a core Netflix OSS, Spinnmaker and a skunkworks project for new ideas). Thus, having a consistent permissions model across organizations can decrease friction. Also, since Netflix collaborates with third parties, such as contractors or other open source contributors, HubCommander also acts as an effective way to maintain security.

    Final Thoughts

    Netflix recognizes a growing popularity in the ChatOps approach and cites other benefits such as team transparency, a timestamp for each event and a self-service nature—all reasons for adopting a ChatOps strategy into its developer tooling.

    HubCommander appears like a good ChatOps tool for GitHub repository management. However, GitHub Actions (still in beta at the time of writing) is a new feature that will aid development workflows, offering automation for the GitHub operational duties themselves. This may spark alternative bots to be developed, especially in terms of workflows that respond to events.

    For now, with open source utilities such as HubCommander, certain GitHub duties can be quickened, stimulating a culture of more efficient management to decrease redundant tasks. While shorthand for GitHub repository management might not seem at first glance as a priority, it could scale down operational headaches tremendously.

    — Bill Doerrfeld

  • Image Management 101: The Web Developer’s Guide

    Image Management 101: The Web Developer’s Guide

    How understanding the user experience can help web developers in managing images effectively

    Online experiences have in many ways supplanted in-person experiences. Today, no one would hesitate to buy a luxury watch online instead of from a jewelry brick-and-mortar store. But as these online experiences become the norm, user expectations for a seamless and “in-person like” customer journey have increased as well. When customers buy that watch, they expect to see every gleam of precious metal and stone that proves its worth before they add it to their shopping cart. This can only happen with the highest quality visual imagery that kindles an emotional connection.

    But getting these high-quality images from the photographer’s camera to the shopper is a bigger challenge than most understand. Web development teams often receive raw photographs in obscure formats, or high-resolution files designed to preserve image quality to nearly microscopic detail; but these files will undoubtedly impact site performance due to sheer size.

    Speed and quality are often at odds in the web development process as companies embrace the need to provide richer and richer online experiences. Web developers need to master the best ways to deliver site performance while balancing marketing needs.

    Optimizations Based on User Experience

    Developers need to understand how image optimization can impact the user experience. No one disputes the goal of delivering images with the best quality possible to end users, but the part that is challenging is how subjective and wide the range of “best quality possible” can be.

    First, it’s important to note that quality can vary depending on whether a content delivery network (CDN) is used to deliver images, the connection speed and device used. For example, an overclocked gaming desktop with a hardwired, high-speed internet connection with images served from local servers is going to provide a drastically different experience than a smartphone overdue for an upgrade on a roaming signal. Also, delivering desktop images to mobile devices can result in site latency and user frustration due to over downloading. On the flip side, delivering images intended for mobile devices to desktops will result in pixelated images that will displease viewers.

    As for devices, different screen sizes mean different viewing experiences. A wide shot on a website viewed on a computer screen will show more detail than a 2-by-5-inch phone. For example, a scenic shot of a hotel with a beach in the background may clearly show the hotel name on a computer screen but be too small to make out on a mobile device. But since more and more people are using mobile devices to access digital content, it will benefit web development teams to ensure images serve their intended purpose with focus points and zoom capabilities.

    Delivering Based on Insight

    Once developers know their objectives in terms of user experience, they need a plan to deliver this experience at the highest possible quality. As mentioned above, the highest quality image does not necessarily translate into the highest quality experience; therefore, it’s important for developers to configure their delivery systems according to varying user needs. Developers can start by collecting accurate information about their users and how they access content. In addition to network conditions and screen size, the user experience can be impacted by what browser, operating system and device visitors are using. Collecting real user data can provide the insight needed to deliver the right quality and size image to the right user.

    To make matters more complex, many browsers and device environments have native image formats that differ from standard formats such as .png or .jpg. For example, Apple computers and devices support JPEG 2000, while Microsoft has a proprietary format called JPEG XR. These formats offer better image compression for delivery, meaning that web teams can use higher quality images without sacrificing faster load time or bandwidth. However, the fact that they are limited to specific environments requires web teams to create all the variant derivatives, store multiple copies and develop the logic to deliver the correct format to the appropriate browser and operating system.

    Automation All Day, Every Day

    Web teams and developers can benefit from automating their image optimization decisions given the challenges they face when delivering images, such as various combination of internet speed, screen size, device performance, operating systems and browsers. Automation to CDN cloud providers can also offload the decision-making logic (which can be CPU heavy) away from the main server, speeding up overall response times.

    Developers also need visibility into how images load and display on user devices. This means their job is not done after delivery. Having this level of insight will help identify bottlenecks and prioritize images for optimization. Automation services that have this option will be best positioned to help web teams.

    Web developers are faced with increasing pressure to meet aggressive marketing priorities. Quality and performance may seem at odds, but by taking stock of all factors that go into image requirements, web teams and developers can build websites that deliver stunning image-driven user experiences without sacrificing quality.

    — Charles Choe

  • DevOps Chat: Instrumentation with Influx Data’s Evan Kaplan

    DevOps Chat: Instrumentation with Influx Data’s Evan Kaplan

    Evan Kaplan, CEO of Influx Data, is someone who knows his way around the technology world. In a lifetime of experience that has seen him found and manage several successful companies from startups to public entities, Evan has had a great track record of success.

    Influx Data is yet another success for Kaplan. Building on the open source InfluxDB, the company boasts more than 200,000 users of Influx today.

    In this DevOps Chat, Evan and I discuss systems instrumentation today and what big data and velocity means for developers and everyone up the stack. It is a very different world from just 15 or 20 years ago.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Happy to be joined in this episode of DevOps Chat by, actually, someone who grew up in my neighborhood. We knew each other – well, we didn’t know each other way back when, but we’ve come from a very small area of concentrated – lot of people, you know, we know from that area, but, anyway, Evan Kaplan, CEO of InfluxData and of North Woodmere. Evan, welcome.

    Evan Kaplan: [Laughs] Thanks, Alan. Thanks for having me. Look forward to _____ –

    Shimel: My pleasure. So, Evan, it’s been a long, strange trip from North Woodmere. Why don’t you give our audience a little bit of your background and your journey? And  then we’ll jump into –

    Kaplan: Yeah. Sure. Sure. Very long for me. If those of you don’t know, that’s a suburb of New York City, right near Kennedy Airport, that we both grew up. My strange trip took me to Seattle, where I finished up college and worked as a climbing guide for years. And then worked in aerospace and started a company, the first company to develop SSL-based VPNs, called “Aventail,” back in the mid-90s. _____ through the bubble. Ran a public company called “iPass,” and, about two-and-a-half years ago, met Paul Dix, the founder of Influx. And Paul and I really connected about his vision and what he was doing, and we partnered together to start building Influx from 25 employees back then to north of 100 now. So growing fast.

    Shimel: Good stuff. Good stuff. And, Evan, I’m gonna assume most of our audience has at least heard of InfluxData, but, for those who have or even those who haven’t, just to set the level-set, tell us about what InfluxData does.

    Kaplan: Yeah, so InfluxData is an open-source time-series-based data platform. It’s the leading time-series platform out there. Time-series is the fastest growing market in the database, data category. And so Paul, our founder, spun up Influx back in 20 – I think it was late 2013. It received a lot of accolades on GitHub. It was a very popular project and it has grown dramatically since then, from roughly – I don’t know – 1,000 sites running it to, today, we’re approaching 200,000 sites running it.

    Shimel: Wow.

    Kaplan: And, you know, most of the large Global 2000 is running it somewhere, so.

    Shimel: That’s really impressive, man. That’s something to be proud of. So, Evan, we could probably spend our whole 20 minutes talking generally about Influx DB and what’s happening, but I’m wanted to kinda dive in a little bit and talk specifically today around instrumentation in modern software installs, in the modern software factory. This whole term “modern” has become sort of a catchall buzzword, dog whistle, if you will, for when we talk about  things like Agile and DevOps and serverless and microservices and all of these ways that we’re seeing software being written, managed, and deployed today. So talk to us a little bit about it, if you will, Evan. Where’s your take on this?

    Kaplan: Yeah, so I think we’re – we grew up in the industry, going back a couple of decades ago, the common myth for monitoring or instrumentation of systems was you would buy a package from HP OpenView or CA or a mainframe management package, and you’d buy a monitoring-for-a-service package. They were expensive and they were largely horizontal across the stack, and you’d implement those. And what’s happened over – certainly over the last ten years, during the DevOps revolution, is that stuff has fractured and you’ve seen companies, like Splunk and Elastic and a variety of other folks, arise to handle the modern workloads in an open-source sort of fashion. And now, if you take a look a look at DevOps world, most of solutions are dominated by open-source kinds of models.

    If you look at state of the art now, that people are really paying attention to, the revolution that brought you to logs and log views of stuff and the consolidation of logs has moved much closer to metrics and events. And what I mean by “metrics and events,” the instrumentation of real time of these complex systems. And they’re not just software systems that are DevOps things, so – those are obvious, but they’re also hardware systems; they’re IoT things. And so what’s happening is people wanna see these complex systems and they wanna interact with them in real time because the only way to characterize these systems is to watch them behave in real time. And “observability,” as you know, Alan, has become sort of a keyword here. And observability generally means real time and it means metrics and events; it doesn’t mean verbose logs.

    And so what we’re trying to do is help developers, specifically and, up the chain, architects and CTOs, build instrumentation or telemetry natively into these complex systems, so they can view these things in real time. As you can guess, the data loads with this are absurdly high. You’re talking about hundreds of millions of points a second in some of these systems and you’re talking about to visualize those in near real time, you know, control loops in near real time. And you’re talking about this sort of next generation of, lack of a better way to say it, system instrumentation and management.

    Shimel: Sure. Evan, if I could stop you there for a sec, you know what I find interesting? We’re talking about building this instrumentation and to give people insight and who are we giving insight? Well, you mentioned developers and managers and architects and CTOs, et cetera, but you didn’t mention ops. And is that –

    Kaplan: Yeah. Yeah.

    Shimel: Isn’t that a little bit of, you know – I don’t know if it’s Freudian or if it’s just insight into what today’s world looks like for these things.

    Kaplan: You know, that discrete ops, right? That discrete ops role is changing, right, into what you obviously cover, which is this DevOps. And so our view of how our – uptake of our platform into large corporations, companies like IBM and Cisco and SAP and Tesla and PayPal, has been from developer or DevOps people up. So we tend to think of “Make it super easy for developers, right? Get their time to value down to really, really minimal and then it will rise up and be taken as a complete operational system.” So we tend to think of it from developers up. Not to say the operations guys aren’t relevant – they’re super relevant – it’s just, when we enter into a large opportunity or business or that sort of stuff, it’s usually coming from a developer who figures out he can instrument his code, his service, his offering relatively quickly in Influx.

    Shimel: Got it. And I’ll tell you something I – and it goes back to what you mentioned earlier – the sheer volume and the acceleration of data, right, the – so, no, it’s how much data and the speed that it comes at us. With automation and everything that that brings, it easily outstrips the ability of a human, a lot of times, to stay on top of it, so you almost need to build automation, machine learning, AI into these types of instrumentations, that you bring what the human really needs to see and touch or make a decision on to his attention ’cause so much of it is just whizzing by right now.

    In the last week or two, Evan, I’ve probably had a couple of discussions – trying to think of a friend of mine who – CEO at ScienceLogic and another company called “Ravel Systems”; some friends of mine in security just launched something called “DisruptOps.” And they’re all talking about this term, “AIOps.”

    Kaplan: Yeah. Yeah.

    Shimel: And I’m wondering is this something relevant to what you’re talking about, with instrumentation? Where does that fit in? How do we account for just the big data nature of this?

    Kaplan: Yeah, I think, first of all, absolutely. So, if you think about – so the journey. And let’s call it the end state of all of this is true autonomy, which is very rare and, in fact, almost never the case, right? But, if end goal state is true autonomy, what’s that journey look like? It looks like, first and always and most importantly – and, by the way, the stuff that we skipped a lot, when we were in the early stages of our career – it is, in fact, instrumentation, is telemetry. Telemetry always was an afterthought before whether shipping a product or a service or that sort of stuff. Now the ability to drive that cost of instrumentation down so low that the ability to see what’s going on and create the data – not just see as in “human” see but create the data – has changed dramatically.

    So, if you follow that journey, you go from instrumentation; you go to observability; you go to automation in the form of “I’ve observed these sorts of things, whether it’s through AI capability or not. I built control loops because I’ve observed them, so I’ve built control loops to keep it in a relative stasis. Right? To autonomy.” All of that – the last two steps, right, even the last three steps – involves some form of machine processing of learning, whether it’s machine learning or AI or call it, but all of that gets increasingly sophisticated. But it means taking in, as you sort of articulated, you know, millions, hundreds of millions – in some case, billions – of points per second, right?

    Shimel: Yep.

    Kaplan: Think about it. Now that’s just completely – you know, what a different world, than doing server management with your CA Unicenter product 20 years ago.

    Shimel: [Laughs] To say the least.

    Kaplan: Right?

    Shimel: Yeah, I hear ya. I mean, it really is. If we transported – you know, if we could do that – if we could transport someone from not that long ago, Evan, an hour or _____ _____ –

    Kaplan: No. No.

    Shimel: From 1997, ’98, right? I was helping an early ASP, Interliant. We were building – opening data centers, managing Lotus Notes and PeopleSoft and stuff like that.

    Kaplan: Right.

    Shimel: If I could take a NOC guy from there and show him today’s velocity and the fact that we actually do have telemetry and we actually are building tools that allow us to manage at that level, it would be like taking someone from colonial times and showing them highways with cars. I mean, the supports –

    Kaplan: But, Alan, these solutions are still sitting out there. They’re a huge _____ _____ _____ –

    Shimel: No – well, that’s the scary thing. Yeah, I mentioned my friend, ScienceLogic, my friend Dave Link. He was saying that’s his – you know, and they’ve done very, very well, and his thing was just replacing the Tivolis and the Unicenters and that generation, if you will, of network management, of systems management.

    Kaplan: If I could tell you what, back in the late ’90s, is I was building an ASP also, based on the Aventail, the SSL VPNs, what we were standing on Netcool, back in the day, and things like that, _____ _____ –

    Shimel: And Netcool was a new program, right?

    Kaplan: Yeah.

    Shimel: Before IBM bought it, right? It was one of the cool, new kids at the time.

    Kaplan: Exactly. If I could tell you what somebody could take with Influx today and do, relative to what Netcool did back in the day, and the difference, right – first of all, they could do it for nothing. Forget paid licenses. And the amount of scale, the amount of data collection, and the ease of sort of visualization has dramatically – and all those _____ _____ –

    Shimel: It’s phenomenal. Phenomenal, man. It’s just crazy. So, Evan, let’s bring this back to Influx. How does all of this now manifest itself with your business today?

    Kaplan: Yeah. So that’s right. So we look at our businesses in two segments, right? We look at it as the instrumentation of and the collection of data around software, being the platform of record for this next generation of cloud software. And I don’t have to describe to you is is that, if, in the old days, it was an X Windows terminal talking to a Unix server app, today, there is no “there” there. There’s a virtual network; there’s VMs, containers, and microservices that are growing at exponential rate. The whole infrastructure is ephemeral, so the monitoring and the instrumentation workload has dramatically changed. So one side of our business is really focused on that, that opportunity, and we’d call that the “DevOps chain” of the business, right? Because it comes up into organizations through DevOps folks who bring in Influx and instrument.

    The other side is IoT, which is completely different. So, while DevOps is largely metrics-driven, IoT is largely event _____ driven. And that’s the instrumentation of the physical world, right? So our watches, our clothing, our health care, our cars, our homes, are all being instrumented with sensors. The amount of data pouring in through that is driving whole other systems into business. And, while most of our time was spent on the management of software systems back, now it’s this deep, deep integration of software and hardware. Deep. And so our business breaks out into those two categories, largely.

    Shimel: Got it. Got it. Evan, anything – any new news coming out of Influx you wanna share with our audience or –

    Kaplan: You know, where – we have – you know, we’re almost constantly releasing stuff into the community on a regular basis, so no news that I’d share at this moment. We can touch base in three or four weeks. We have our big InfluxDays event. We’ll make some announcements there, but you can look for us to build increasingly sophisticated solutions that allow people to do their machine-learning workloads, their application tracing, their log and metrics, their asset management, on top of Influx. What we want is people to use us as the platform of record for all of these metrics and events pouring in.

    Shimel: You know, we live in interesting times and I can’t wait to see where this all goes as well, so we’re about out of time, though, here. As I mentioned to you, the 15, 20 minutes goes so quickly.

    Kaplan: Yeah, it really does.

    Shimel: And you barely ever – you know, it is what it is. But, Evan, thanks for joining with us. Evan Kaplan, CEO, InfluxData. You know, so, in three, four weeks, you have your event coming up; maybe we’ll have you back. We can talk a little more about that. We’re doing a big – I didn’t even announce it yet, but I’m gonna announce it right here on the show with you – we’re gonna be doing a big virtual event, December 18th, called “Predict 2019,” and we’re gonna –

    Kaplan: [Laughs]

    Shimel: Yeah. We’re gonna be opening up the crystal ball and asking a lot of pundits and experts and so-called experts what their predictions are for what 2019 has in store. Maybe we can have you back then and we can do some crystal-ball gazing.

    Kaplan: That’d be great, Alan. Always a pleasure. Always a pleasure.

    Shimel: All right, my friend.

    Kaplan: Have a great day. All right.

    Shimel: Well, Evan Kaplan of North Woodmere and InfluxData, thanks for being our guest on this episode of DevOps Chat. This is Alan Shimel for devopsy.kinsta.cloud, and you’ve just listened to another DevOps Chat. Have a great day.

    Kaplan: Take care.

    — Alan Shimel

  • Steps for Migrating Code Between Version Control Tools

    Steps for Migrating Code Between Version Control Tools

    Code migration can be tricky if you’re not prepared. Follow these steps for a worry-free migration.

    Almost all source code contains some format of delicate information, whether it be configuration-related information such as database credentials or information on how the web application functions, such as a web template.

    If such information is disclosed, this information potentially could be used by an attacker to discover logical flaws and escalate into a successive chain of attacks. These include attacks such as SQL injection, database takeovers, cross-site scripting, remote code execution and, sometimes distributed denial of service.

    To achieve solution for conducting a risk-free and systemic migration, we need to plan three stages for the code migration process: pre-migration, migration and post-migration.

    Key Considerations During Pre-Migration

    • License cost
    • Administration cost
    • Developer productivity
    • Security of data
    • Training of team
    • Total cost of ownership
    • Knowledge on all connected tools with application lifecycle management
    • Agile adoption policies and CI practices
    • Team structure – distributed or co-located?
    • File types – Do you have large binary files?
    • Need file locks?
    • Users have the appropriate permissions to fetch and push
    • Builds are successful, and all your tests are passing

    Key Considerations or Challenges During Migration

    • Test environment
    • Capacity planning
    • Compute capacity
    • Benchmark the current version control system
    • Network connectivity
    • Access between source and target servers
    • Load testing
    • Access control
    • Aim to migrate over a weekend when nobody needs to check-in

    Key Considerations During Post-Migration

    • Data integrity verification
    • Data size check
    • Automated acceptance testing and reporting
    • Dependencies are intact and build doesn’t break
    • Link between files and folders is not broken
    • Load testing
    • Backup and sync jobs

    Moving to Distributed Version Control from Centralized Version Control

    When migrating code from a centralized version control system such as Apache Subversion (SVN) to a distributed version control such as GIT, here are the steps to take:

    • Evaluate the tools and methods you’re planning to utilize
      • Build and test
      • Code reviews
      • Release management
    • Select a branching strategy for GIT
      • long-lived feature branch
      • Short-lived feature branch
      • Bug fix branch
      • Dev branch

    Reference: https://guides.github.com/introduction/flow/index.html

    https://nvie.com/posts/a-successful-git-branching-model/

    • Decide whether you want to migrate history; if yes, then determine how you will achieve this—you may need another third-party tool to resolve your task.
      • There are many tools available in market that can migrate a whole history of all branches from a centralized tool to GIT.
      • Almost all version control systems collect how files change between versions, including rename, merge and rollback. GIT version control tool stores versions as a snapshot of the entire repository.
    • Retain your old version control system for a set time after migration.
      • During and after a migration, developers sometimes need access to the history in the old version control system, so you can’t just shut the old system down once you’ve migrated.
      • There should be some text file committed in GIT which should explain how/when the GIT branches were migrated from the old system and what all branches are migrated.
    • Remove binaries and executables from source control.
      • You should avoid commit large binary files to a repository. Once they’ve been committed to a repository, they will remain in the repository history and in every upcoming clone.
      • Migrating to Git provides you the opportunity to remove these binaries from your codebase and instead use a package management system.
    • Give knowledge transfer to your team in the concepts and practice used in GIT.
      • It is mandatory for developers to understand how Git stores change and how commits form a history of development and how all things work in GIT.
      • Identify training materials.
    • Perform the actual migration to GIT.

    Suggestions:

    • Keep the following types of files: code files, versioned assets such as graphics and large configuration files.
    • Remove the following types of files: database dumps, log files.
    • Use a vulnerability checking tool.

    Conclusion

    By following the above key steps, you can have a systemic and risk-free migration. Planning, then acting, is critical in a successful code migration process.

    — Amit Anan

  • How DevOps Can Nurture the IoT Brownfield For Results

    How DevOps Can Nurture the IoT Brownfield For Results

    Your latest IoT project sailed through proof-of-concept but started running into problems during rollout. Fortunately, your DevOps superhumans got things running. Now, perhaps a year or more later, you’re seeing more and more issues crop up as the project grows. If this scenario sounds vaguely familiar, or you’re worried you may be headed there, you’re not alone. Welcome to the IoT brownfield.

    “Nah, can’t be. We selected and evaluated all-new hardware and software during the PoC. This has been a greenfield project from the start.” Be careful of this trap. Many IoT projects indeed start off as greenfield efforts, free of the constraints of integrating legacy hardware and software. Your team developed requirements with collaboration between IT and line-of-business folks. You introduced these new tools, and got results on a small scale. According to Cisco’s 2017 IoT survey of 1,845 IT and line-of-business decision makers, 60 percent of IoT projects stall at PoC, so you’re ahead of the game.

    Or so it would seem. Now comes the escalating commitment. Management assumes that if it worked on a small scale, it should just work when rolled out across the enterprise. Data and insight will just come pouring in, and everyone will have visibility to what’s happening everywhere in real-time. They write a big check for the rollout, and away you go.

    First Signs of Trouble

    At the first rollout project review, folks start splitting hairs over the definition of “complete success.” You rolled out X locations, Y of them are working, X-Y of them are in various states of triage. How can that be? You took the “same” configuration and scaled it. As the team investigates the non-working population, you’re finding little things that aren’t the same, calling for tweaks here and there.

    Tossing out the trivial, over scale and time there is no such thing as an IoT greenfield project. Every IoT project with an extended life cycle becomes a brownfield, with a mix of new and old hardware and software. Rigorously controlling a configuration, à la defense or medical applications, is prohibitively expensive and unnecessary for most IoT applications. Your vendors won’t sell you the exact same hardware or software two years from now, five years from now, 10 years from now. Unless, that is, you pay them for things such as lifetime buys of electronic components or freezing open source code releases.

    Ideally, we should send DevOps teams into IoT projects. With a big tool stack acquiring data, the line-of-business types figure out more things to do as the picture develops. The whole point is gaining insight and adjusting for optimized results. Cycles of learning and security enhancements are part of the game.

    Coping with an IoT Brownfield

    Here’s the paradox: the longer your IoT application survives, the more complex this situation gets. How can you improve your chance of success by applying IoT brownfield thinking? Technical requirements vary, but some core philosophical tenets can help.

    Roll with the changes. Your team is already skilled at quickly rolling application software releases. Get more aggressive with ongoing changes at the firmware and network level. Incorporate over-the-air (OTA) technology for updating code on any wireless devices. Deploy multi-protocol gateways that can deal with revised protocol stacks (or entirely new ones) as IoT specifications continue evolving. Update security vulnerabilities relentlessly.

    Abstract everything. You’ll read tons about IoT interoperability, and what people typically mean is wire-level interoperability of protocols. Those multi-protocol gateways have one job—getting real-time data onto an IP network and into the cloud. What keeps me up at night is data-level interoperability. Use edge or fog computing for pre-processing streams so the cloud isn’t stuck sorting out various formats of the same class of data. Abstract your data using JSON or other lightweight techniques so that if a next-generation sensor outputs data in a slightly different format, pre-processing can be revised close to the source. Consider a publish-subscribe protocol for easier routing between nodes.

    Optimize the cloud. Scale can bite hard in the cloud tier. Heterogenous, distributed architectures usually win. Deploy a hybrid cloud solution where there is real-time data. Let the public cloud do what it’s good at: presentation, business intelligence and archiving. In a private cloud instance for heavier analytics lifting, use a framework such as Apache Spark to minimize data movement, with faster nodes sprinkled in for critical algorithms. Try dedicating a small processing core to each data stream for predictable deadlines, or using FPGAs in workflow optimized nodes for boosting compute performance.

    After All, It’s Just Business

    Ultimately, the evaluation of complete success of an IoT application will be up to the line-of-business decision-makers who wrote that big rollout check. A DevOps model is ideal for nurturing the IoT brownfield over scale and time, featuring ongoing strong collaboration between technologists and line-of-business types. Early in the project, reach out for some external IoT expertise until you have a few cycles of learning under your belt.

    As that Cisco 2017 IoT survey alluded to, 61 percent of practitioners believe we’re barely scratching the surface of IoT transformation for businesses. Avoiding the greenfield trap and incorporating these IoT brownfield coping strategies sets you up for the best possible chance of success in the long run.

    — Don Dingee

  • DOES18: Challenges and Enablers for DevOps Success

    DOES18: Challenges and Enablers for DevOps Success

    “What happens in Vegas, doesn”t stay in Vegas” … a great quote to end the recent DevOps Enterprise Summit 2018 (DOES18) in Las Vegas, urging attendees to take and share the new insights and knowledge gained. This article is an attempt at sharing my discoveries.

    This article contains a summary of the challenges that delegates told me about when visiting the GamingWorks stand at the summit. These closely match our global findings, with more than 400 teams having participated in the Phoenix Project business simulation workshop.

    This article also describes highlights from the sessions I had time to attend—highlights mostly in relation to the challenges mentioned above, and biased toward toward people, culture and behavior, as this to me is the No. 1 success or fail factor for the successful adoption of DevOps ways of working.

    Challenges

    I asked more than 100 people, “What is your top challenge or barrier for your adoption of DevOps ways of working?” I have ordered them based upon the number of times mentioned, highest first. Many of these I captured before the opening of the summit:

    Leadership:

    • Leaders applying the “pointy-fingered” approach. “You are now self-empowered. Make IT happen.‘
    • Lack of “empowerment” and not “walking-the-talk” to foster the right culture.
    • Not wanting to let go of a “command and control” management approach and fear of becoming less relevant.

    Product Management:

    • The need to shift from a “feature” focus to more “outcome” and “business value” focus.
    • Product owners “demand” but don”t share why.
    • Low focus on “technical debt” and “issues” versus “benefits.”
    • Need higher-level product management (business decision makers) to balance “innovation” with “risks” (e.g technical debt).

    Value Stream:

    • Poor end-to-end visibility (including business “before” and “after” activities).
    • A narrow focus on Dev and Ops, mostly a Dev “pipeline.”
    • No impediment backlog visible for the stream.
    • Too much local optimization.

    Value and Metrics:

    • Too much focus on internal, proxy metrics.
    • Need more “value” and “outcome” metrics to gain buy-in to scale up.

    End-to-End Communication and Team Collaboration:

    • “Them” and “Us” – lack of trust.
    • Poor listening and communication skills to explore and understand.
    • Lack of end-to-end understanding of the business impact and value we are all working toward.

    Poor Management of WIP:

    • Poor balance between types of work, feature vs technical debt, improvements, training and knowledge-sharing.
    • Engineer disconnect between “what you are working on and the business impact.”

    Continual Learning and Improvement:

    • Not enough WIP reserved.
    • Poor team ownership and ability to structure this.
    • No visible backlog of impediments/improvement.
    • No prioritization of improvements.

    Project to Product:

    • Projects claiming resources.
    • Product owner resource conflicts and challenges.
    • Flip-flopping, context switching work.

    Buy-in and Resistance:

    • Why should we change?
    • Fear, uncertainty, doubt.
    • Suspicion of intentions.
    • Managers not wanting to change and lose “power.”

    Certainly not a lot of CI/CD or technology stuff mentioned here!

    What were some of my highlights as they relate to these challenges?

    DOES18: Introduction and Opening

    In his introduction, Gene Kim revealed one of the goals of the summit was a need to “Elevate focus on spanning business/technology divide,” which clearly relates to a high-scoring challenge. Another topic in the intro was the need to “upskill,” announcing a new skills research project to be launched. The challenges above and the highlights from the sessions give a clear indication of some of the skills needed. Considering the challenges above, I sincerely hope the survey also focuses on business skills as well. In the “Lean Coffee” roundtable sessions, one of the announced topics was “Transformational Leadership,” clearly linking to the top scoring challenges mentioned. More about this topic in my closing.

    DOES18 Sessions

    In a session titled, “Product Management Meets DevOps” a key recommendation was to “build and run teams combined with service ownership,” stressing a need “to connect people to your strategy” and declaring “IT cannot be separate from the business,” and finishing with a need to “manage product value streams, not projects.”  (This was the only session in which I heard the word “service”). To me, these highlights clearly make a link back to the challenge teams have in understanding business impact and value. These recommendations will require a changing role for product management to ensure strategic priorities are embedded in the value stream, and a more engaged role of product owners in teams to ensure a shared understanding and focus on “outcomes” and “value” in relation to strategy, rather than on features.

    Benefit: Enabling the flow of strategic intent

    Mark Schwartz explained that almost all large enterprises are faced with “impediments” that are a barrier to DevOps success, and these are usually people and organizational issues, not technology. He introduced his new “War and Peace and IT” book, and explained an impediment faced by Napoleon in his Russian campaign—Napoleon had “long decision cycles.” By the time he got the right information to make a strategic decision, the fast-changing short decision cycles at the battlefield meant his decision-making was ineffective. The same applies to the way we currently manage IT: long decision-making cycles from idea to business case to program and project resourcing while DevOps teams operate at a fast, agile pace. Mark suggested a need to speed up the “entire lead time from idea to impact” (a topic also explored in Mark Smalley’s latest DASA whitepaper, “The Business Value of DevOps“).

    Benefit: Enabling the faster flow of strategic intent

    Schwartz went on to suggest three ways to build more agility into the end-to-end value stream and delegate decision-making to a lower level: the “product model,” decentralizing decision to the product team; the “budget model,” decentralizing budget to the team to manage both “innovation” and “keeping the lights on”; and the “objective model,” just give the team the key objectives to be achieved and let them decide how to make it happen anyway they can.

    This to me represents a considerable mindset shift from the hierarchical “command and control” management paradigm and requires transformational leadership. In my mind, whatever model is chosen needs to fit with the highlight from the previous session, “Connect people to your strategy.”

    Benefit: Enabling faster decision-making around strategic intent

    Anders Wallgren‘s session, “Measuring DevOps – The Key Metrics That Matter” was standing room only, with a packed audience taking pictures of all of the slides. I must admit to being surprised, as this is the fifth DOES conference and organizations have had five years to put metrics in place! I can only assume the vast majority were just starting their journey. I was pleased to see not just the traditional metrics often quoted such as “deployment frequency,” “lead time to change,” “mean time to recover (MTTR),” and “change failure rate,” but also business value metrics such as revenue growth; customer value metrics such as customer experience satisfaction; team culture metrics such as education and growth; and the pipeline metrics such as release velocity. This was one of the top-scoring challenges facing delegates who visited our table: “Fhifting from internal metrics to business value metrics.” This focus on business value metrics ties in again to the highlight from the first session above, “Connect people to the strategy,” and the important role product owners have in sharing the strategic intent.

    Benefit: Enabling measurement of strategic intent.

    Dr. Mik Kersten presented “Project to Product: How Value Steam Networks will transform IT and the Business,” suggesting that project management and traditional cost models are the wrong models, an idea supporting Mark Schwartz’s observations. He went on to add, “Fragmented value streams kill personal productivity,” as well as “impact employee satisfaction.” Silos and proxy metrics not related to business value will also destroy a DevOps transformation. He made some powerful arguments for a complete end-to-end value stream (Not just a DevOps value stream).

    These messages were further endorsed by Carmen De Ardo in his session, “Connecting IT to the Business at BMW Group,” during which he also stressed, “Flow time is not just code-to-commit but also, and more importantly, idea-to-value.” Carmen also stressed that the “flow to value” is not just the flow of “features” but also “defects,” “risks” and “debts,” which to me represents a need to manage “value optimization,” a balance of “value creation” and a reduction of “value leakage.”

    Benefit: Enabling value optimization of strategic intent.

    Damon Edwards, in his session “Operations: The Last Mile,” expanded on the theme of end-to-end value streams, with a powerful case about disruption, context switching, waiting, escalations, lack of trust and employee dissatisfaction—all this because of “four forces that undermine operations” that need to be addressed to empower operations:

    • Reduction of Toil—non-valued repetitive work that can be automated.
    • Queues, which disintegrate and obfuscate value streams.
    • Silos, causing disconnects and mismatches.
    • Low trust—the need for “psychological safety, the No. 1 predicate for success.”

    Benefit: Enabling the removal of behavioral flow barriers and waste.  

    Dominca Degrandis in her session, “Making Connections Visible: How to Defrag Your Value Stream,” stressed, “The hardest thing we do is communicate across teams,” making a call to make connections visible as well as “how the different types of work enters the system, where they enter and which tools are used.” This touched on a topic I discussed with DeArdo at itSMF Fusion: the information value stream.

    Benefit: Enabling the removal of end-to-end information and behavioral flow barriers and waste.

    One of the most powerful sessions for me was the presentation from Dr. Steve Spear: “Discovering Your Way to Greatness: How Finding and Fixing Faults is the Path to Perfection.” He suggested, “Design all work to SEE problems.” Make this part of a philosophy: “See, Solve, Spread (Share knowledge).” He suggested, don’t ask “How is it going?” Ask, “What is not working, and what can we do about it?” This ties in nicely with a finding from the latest State Of DevOps report: “Relentless improvement work leads to excellence and high performance.”

    Benefit: Enabling the relentless removal of barriers and waste and the optimization of strategic intent.

    DOES18: A Customer Case

    Mascha Boender  and Stefan Groot presented an ABN AMRO case, “Crossing the DevOps Chasm: From Taboo to Enterprise DevOps,” in which they discussed their bottom-up journey of experimentation. They designed their approaches to see problems. They admitted failure, demonstrating their learning and how they made iterative improvements. They stepped away from a maturity model to a capability model, recognizing that one size doesn’t fit all teams. They provided a “Moonshot workshop,” focusing on specific team goals and ambitions and how these aligned to business strategic goals. They provided a DevOps playbook (based on captured team best practices) to help new teams develop practices at their own pace. To develop team “soft skills” such as communication and collaboration, they used the Phoenix Project business simulation workshop. This was done also with more than 30 leaders to create a mindset shift and demonstrate the change of leadership behaviors required to empower the teams and foster an enterprisewide culture change.

    Benefit: Enabling bottom-up and top-down “skills” and “culture” and aligning team goals to strategic intent.

    The State Of DevOps

    Nicole Forsgren and Jez Humble presented, “The Data Behind DevOps: Becoming a High Performer,” in which they shared findings from the latest “State Of DevOps” report. I won’t go into detail. I suggest you read the report, then read it again. Suffice to say their key messaging echoed many of the other sessions above: “Focus on outcomes, not output,” the need to “identify constraints and continually improve,” and for me an important topic: “Influencing culture through leadership and autonomy.”

    The ABN case above was an example of how they gave teams autonomy to change and  how they helped change leadership behaviors.

    DOES18: Final Thoughts

    As I stated at the start, culture to me is the No. 1 success or fail factor. The “State of DevOps” report stressed the importance of leadership on culture. Leadership was the top challenge mentioned by delegates visiting our stand. I”d like to share with you an article on devopsy.kinsta.cloud on how to influence leadership behaviors: The article is written through the lens of “communication,” tying back to a comment from Dominica Degrandis: “The hardest thing we do is communicate across teams,” and tying in with the ABN AMRO case about developing communication and collaboration skills.

    And finally, getting back to the start of the summit and the need to “upskill,” with the announcement of a skills survey, I’d like to provide some input. Here is a DevOps Agile Skills Association (DASA) article, “DASA Provides Core Skills Key for DevOps Success,” which summarizes the “people and culture” elements from the “State of DevOps” report (this year and previous years) matching these with findings from more than 400 teams having participated in the Phoenix Project DevOps simulation. This I believe can provide input into the survey.

    — Paul Wilkinson

  • Simplifying Through DevOps in Banking

    Simplifying Through DevOps in Banking

    There are a great many assets available on the internet that talk about what DevOps is and how far DevOps adoption has come. So we will not repeat the “what” and the “how far.” Instead, we will focus on the road ahead and future of simplification and transformation at banking organizations and the role DevOps is likely to play in the same—in all three areas of “Run the Bank (RTB),” “Change the Bank (CTB)” and “Transform the Bank (TTB).”

    Figure 1: Change-Run-Transform Bank – POA with DevOps Maturity

    In this article, we take a look at what shape we think DevOps-driven simplification in financial organizations can take in the near future and also some opportunities for such simplification. As depicted in Figure 1, with DevOps maturity, the siloed processes of CTB, RTB and TTB will come closer together, allowing common activities and processes to be carried out in one place. The others will get simplified and automated thereby reducing elapsed time of cycle from ideation to production and application management drastically. This will yield benefits not only in terms of cost and time, but also in terms of feature availability to business.

    Banking Focus

    In banking, the early adopters are leading the march with a lot of good practices such as standardized tooling for specific technology areas, orchestration, feedback loop setup, process and technology rationalization, etc. The focus needs to continue on the same.

    We think that the challenge that DevOps architects need to now focus on is to be able to use DevOps to help financial organizations bring down the non-discretionary spend by simplifying the previously ignored maintenance, operations and infrastructure realms. The core of this DevOps driven simplification is needs to be move cost of maintenance to its lowest possible and hence releasing funds to discretionary areas in “Change the Bank” and “Transform the Bank.”

    While the point of arrival for DevOps-led transformation should be “full automation” and “zero maintenance” but they are a bit far in the future because the return on investment (ROI) on the monies and effort required to make that happen needs to justify the same. We see the focus shifting to value delivered to business by transformation of an application or suite of applications. Figure 2 depicts the core repeatable activities of a DevOps-led simplification exercise while keeping in mind these improvement cycles need the setup of feedback loops at each stage.

    Figure 2: Repeatable activities for DevOps-led simplification

    This brings us to our next idea, that the road map to DevOps-enabled simplification needs to be paved with metrics and measures and these metrics cannot be randomly chosen. These metrics have to be actually measurable and be divided into two areas of business metrics and IT metrics as we explain below and need to be diligently tracked to aid course correction.

    For example, release orchestration processes for applications are mostly complicated, need a lot of manual effort, have strict compliance requirements as well as security mandates, need to be carried out more frequently nowadays because of shorter deployment cycles and have to manage ever-increasing complexity of component interaction (upstream and downstream). Now, release orchestration is also a very important step in the entire CI/CD pipeline. So standardization and simplification of release orchestration process makes immense sense. But how does one measure success and quantify improvement?

    As already stated, intelligent insights (metrics and measures) are important in this process for both business and technical team members so that everyone has single source of truth for current status and distance from end goal. This kind of automation also helps focus on the continuous improvement pillar of DevOps adoption by logging everything that matters, have frame-by-frame data for everything that helps. This collection of metrics aids analysis of success as well as failure thereby furthering the cause of continuous improvement.

    Continuing with the example of release orchestration, there are many important best practices that help ensure that wheel is not reinvented. Some of the important learnings that only practitioners will be able to let you in on are as follows:

    1. Tools that need scripting will not be able to help you scale and create low-maintenance solutions.
    2. You need to invest in tools/orchestration solutions that help you visualize, get intelligent insights in a jiffy and can report in a variety of forms and shapes.
    3. Tools need to be able to capture and track business metrics, too.
    4. You need to spend effort on organization level compliance and security.
    5. Strive to automate your most effort-intensive and/or lengthy processes.

    Each step on DevOps maturity scale will have many such learnings and best practices that must get shared across the length and breadth of the organization.

    DevOps and Banking: Future Trends Prediction

    Based on our understanding & knowledge of financial organizations, we believe that the following areas will see focus in the next few years:

    Figure 3: Trends in short-medium term in DevOps adoption

     

    Continuous Improvement within Federation

    • Technology teams will need to enable and encourage localized innovation within a federated boundary of toolsets that will, in time, bring Dev and Ops toolsets to near mirror images of each other.
    • By federation, we mean a detailed homogeneous blueprint that gives flexibility but within an overarching boundary of tools and procedures while encouraging continuous improvement.
    • This must also inform relevant stakeholders about areas for continuous improvement that make the most evolutionary impact to business and use metrics suitably to track and reward improvement.

    Outcome Management

    • We also think that DevOps adoption needs to be driven by outcome management with evidence of impact taking the driving seat for road map management.
    • Focus will be on point of arrival (POA) and continuous measurement of business and IT metrics for outcome management.
    • We also believe that Culture-Automation-Lean-Measurement-Sharing (CALMS) model of DevOps will find more believers.

    Evidence of Impact

    • Metrics and metrics-based course correction will become the Holy Grail in the simplification exercise. The metrics collection will need to focus on areas of both business and IT. The metrics and measures will need to be redefined as:
      • Business Metrics for business value delivered or improved such as new products / services launched, conversion rate, net promoter score (NPS), customer retention rate, CSAT, ESAT, etc.
      • IT Metrics for value improvement and value management such as cost performance index, reduction in defect debt, reduction in lead time to change, etc.

    Design Thinking

    • We believe that the next big disruptor in this space will be bringing design thinking into the mix to ensure that DevOps-led automation is more customer-centric and iterative.
    • We see focusing on test data management (TDM) across the organization as an important step to ensure continuous delivery.
    • In our opinion, the testing process will undergo complete rethink with support of processes such as risk based priority and risk based test case design, test bed splits, TDM and so on.

    KM, Training and Governance

    • Tools and processes for knowledge management, training and governance need to ably support the DevOps-led simplification.
    • These processes will need systemic focus to create and nurture self-learning technology teams.
    • Transformation of the way of working to rationalize processes such as collaboration for lean and improved handover of work packages.

    DevSecOps

    • Security will be extremely crucial in the process of DevOps-led simplification. All the automation, cloud adoption, pipeline creation, IoT adoption and more is throwing up many new security concerns, thereby leading to a focus on DevSecOps.
    • Security processes will potentially need to move their execution points up in the CI/CD pipeline and be divided into stages based on iterations sizes and deployment cycles.
    • Focus on shifting left application security and role-based security.

    Conclusion

    The next wave of simplification of IT spaces in banking organizations will be aided by DevOps. To look for opportunities, a systematic approach toward DevOps will help—it will be perilous to ignore detailed assessment of existing spaces that will detail the current state, point of arrival simplified state, road map for simplification and the key guiding principles for the transformation.

    Carrying out pilots and proof of concepts in DevOps is not a problem; the challenge is with scaling—in all areas: tooling, training and processes. The key is to manage the horizontal and vertical spread of same/similar toolsets that also help drive efficiency associated with scale and standardization. DevOps-led simplification will need changes to processes, technology and people management (culture). The current role structures might need to be demolished to ensure there are no fiefdoms and cross-functional teams handle all requirements of Dev-Test-Ops. Finally, all the changes need to be able to define the new way of working  so that Change-Run-Transform organizations can be brought together.

    This article was co-authored by Babu Unnikrishnan, Head, Banking & Insurance Technology Group at Tata Consultancy Services.

    — Anshu Premchand

  • The Importance of Instant Feedback in Analytics

    The Importance of Instant Feedback in Analytics

     A feedback loop is essential for every part of the modern DevOps environment. From continuous integration and deployment to application development to infrastructure monitoring to team dynamics, feedback is the most important piece of the DevOps puzzle.

    But feedback is of little to zero value if it is not timely. And in the modern development world, that means instant or as near to instant as possible.

    In the past, prior to the current world of web and mobile development, feedback was a closed loop between the developer and the application. Often aided by tests or a compiler, desktop developers could spot issues and follow trails to solve problems without worrying about end users seeing the flaws or getting caught in a bad deploy. There were no deploys, unless you consider pushing a new version out on disc or CD a deploy.

    As the world moved to web-based software and mobile software, new components came into play including cloud, containers, test-driven development and constant iterative deployments. All these led to a new need to monitor systems far beyond the single machine with operating system ecosystems prior to the web revolution.

    Before fully understanding the value of instant feedback, we first need to define what feedback is. When we are running multiple, divergent systems, such as a CI/CD server, containers for our applications or separate containers for our databases, information will be pushed to us in the form of feedback. Without some sort of analytical engine, this is simply raw data.

    Raw data often has little value. To those working directly with it, there may be something to be gleaned without manipulating it further, but for those abstracted from direct interaction with raw data, it is important to have some sort of buffer between the information we are looking for and the data in the raw.

    Taking raw data and putting it into a usable form is the role of the analytical engine. It’s important to note here that not all data is necessary to all people. For example, the sales department has no need of a dashboard showing CPU usage over time. At the same time, the operations team likely doesn’t need to see the burn rate of the sales team. Getting the information is the first step. The next is to get the correct information to the right people in a timely way.

    The importance of instant feedback is compounded by the development style adopted by the modern application development team. We want our web-based and mobile applications to be deployed consistently, if not constantly, as well as securely and with stability. When an application enters the world, we want to know if the deploy occurred correctly, if there were errors, and how many resources were necessary.

    If the team needs to wait for even a few minutes, it can be detrimental to business. The concepts of high availability and constant uptime for users has made it so every deployment needs to be tested and retested, but all this needs to be done in a short time frame.

    Modern, instant feedback allows everyone to see what is successful (and what is not) with enough time to adjust before a slight issue becomes a major meltdown. The analytical dashboards are feeding this information, converting the raw deployment and application data and presenting it as valuable feedback. This helps the modern DevOps adherent to streamline processes and continue to deliver a better product to customers and end users.

    It’s easy to see the past was a time when analytics and feedback was of little consequence, but as we continue forward, with faster deployment and development cycles, the role of instant feedback will continue to increase and better, faster solutions will need to grow to meet that need.

    — Geeta Schmidt