devopsy.kinsta.cloud

Category: DevOps Practice

  • Achieving Reliable Observability Part 1 – Making Cloud-Native Observability More Robust

    Achieving Reliable Observability Part 1 – Making Cloud-Native Observability More Robust

    I was having a conversation with a CxO level customer as part of an AIOps/Observability workshop, and from what I could tell, most are confused about how to properly operationalize cloud-native production environments – especially the monitoring/observability portion. Here is how the conversation went.

    “Andy, we are thinking about getting [vendor] to use for our observability solution based on your recent research. What do you think?”

    “Well, I don’t want to endorse any specific vendor, as they are all good at what they do. But let’s talk about what you want to do, and what they can do for you, so you can figure out whether or not they are the right fit for you.” The conversation continued for a while, but the last piece is worthy of being called out specifically.

    “So, we will be running our production microservices in AWS in the ____ region. And we are planning to use this particular observability provider to monitor our Kubernetes clusters.”

    “Couple of items to discuss. First, you realize that this particular provider you are speaking of also runs in the same region of the same cloud provider as yours, right?”

    “We didn’t know that. Is that going to be a problem?”

    “Not particularly. However, you may get into a ‘circular dependency’ situation.”

    “What is that?”

    “Well, as an enterprise architect, I always call for separation of duties as a best practice. For example, having your developer testing the code is a bad idea, having your developer figuring out how to deploy is a bad idea. In much the same way as when your production services run in the same region as your monitoring software – how would you know about a production outage if the cloud region takes a hit, and your observability solution goes down at the same time your production services do?”

    “Should we dump them and go get this other solution instead?”

    “No, I am not saying that. Figure out what you are trying to achieve and have a plan for it. Selection of an observability tool should fit your overall strategy.”

    For those who don’t understand the above conversation, here is the reason why this scenario could be a problem.

    Coming from an enterprise architecture background, we were taught, as a best practice, to operationalize production systems to avoid circular dependencies. This includes not having two services depend on each other, or not to colocate monitoring, governance and compliance systems as part of the production systems themselves. If you were to monitor your production system, you would do it from a separate and isolated sub-system (server, data center rack, sub-net, etc.) to make sure that if your production system goes down, the monitoring system doesn’t go down, too.  The same goes for public cloud regions – although it’s unlikely, individual regions and services do experience outages. If your production infrastructure is running on the same services in the same region as your SaaS monitoring provider, not only won’t you be aware that your production systems are down, but you also won’t have the data to analyze what went wrong. The whole idea behind having a good observability system is to quickly know when things went bad, what went wrong, where the problem is and why it happened so you can quickly fix it. You can check out this blog where I explain this in detail.

    The best practice would be to either:

    1. Consider an observability solution that runs in a different region than your production workloads. Better yet, consider something that runs on a different cloud provider altogether. Although it is exceptionally rare, there have been instances of cloud service outages that cross regions. The chances of both cloud providers going down at the same time would be slim.For example, if the cloud region goes down (a region-wide outage in the cloud is quite possible, and seems to be more frequent of late), then your observability systems will also be down. You wouldn’t even know your production servers are down to switch to your backup systems unless you have a “hot” production backup. Not only will your customers find out about your outage before you do, but you won’t even be able to initiate your playbook, as you are not even aware that your production servers are down.
    2. Consider having your observability solution in a different location/region, yet still close enough to your production services so latency is very low. Most cloud providers operate in close proximity, so it is easy to find one.
    3. Another option is to get a solution that gives you deployment flexibility. For example, there are a couple of observability solutions that allow you to deploy it in any cloud and observe your production systems from anywhere – both in the cloud and on-premises.
    4. You can also consider sending the monitoring data from your instrumentation to two observability solution locations, but that will cost you slightly more. Or, ask what the vendor’s business continuity/disaster recovery plans are. While some think the costs might be much higher, I disagree, for a couple of reasons. First, because monitoring is mainly time-series metric data, so the volume and the cost to transport is not as high as logs or traces. Second, unless your observability provider is VPC peered, the chances are your data will be routed through the internet even though they are hosted in the same cloud provider. Hence, there will not be much more additional cost. Having observability data, all the time, about your production system is very critical during outages.
    5. A very commonly overlooked consideration is monitoring your full-stack observability system. While it is preferable to have the monitoring instance in every region where your production services run, it may not be feasible either because of cost or manageability. On such occasions, monitor the monitoring system. You could do synthetic monitoring by checking the monitoring API endpoints (or do random data inputs and check to ensure it worked) to make sure that your monitoring system is properly watching your production system. Better yet, find a monitoring vendor that will do this on your behalf.

    When/if you get the dreaded 2 a.m. call, what is your plan of action? Just think it through thoroughly before it happens and have a playbook ready, so you won’t have to panic in a crisis.

  • Python Remains Popular in Programming Language Study 

    Python Remains Popular in Programming Language Study 

    A ranking by analyst firm RedMonk reflects on Python’s continued popularity as a leading enterprise programming language for a number of reasons. According to a recent study by RedMonk, JavaScript also remains widely popular as the web development programming language of choice for a significant number organizations. 

    In many ways, Python’s strong showing in the ranking — drawn from GitHub pull requests and Stack Overflow data — indicates its long-term viability among DevOps developers. 

    “One lack of change that is worth surfacing is Python’s number-two spot,” Stephen O’Grady, a RedMonk co-founder and analyst writes in The RedMonk Programming Language Rankings: January 2021. “Java was extremely hot on Python’s heels – and was, in fact, closer to the number-one ranking than to PHP behind it – but Python’s ability to defend its new high ranking is notable.” 

    RedMonk

    Python’s prolonged high ranking is notable because of its versatility, Rachel Stephens, a RedMonk analyst who is responsible for the RedMonk Language Rankings’ design, told DevOps.com. “The breadth of use cases — and associated libraries that support those use cases — means there are a number of possible entry points for developers. The number of toeholds for the language within the enterprise helps its ubiquity,” said Stephens.

    There also remains much discussion about what accounts for Python’s continued popularity beyond the scope of the study. A popular subject in forums, including Reddit, Python is typically described as highly versatile, relatively easy-to-learn and a simple, yet elegant language. While not cited in the RedMonk study, some software engineers and coders — such as this writer — appreciate how Python can make more intuitive sense, and is also less rigid compared to, say, a classic language such as C++. While JavaScript is largely used for web apps and scripts, of course, Python could arguably be the most popular language outside of the sphere of web development, thus accounting for its second-highest ranking in the study.

    As such, “due to the accretive nature of our methodology, movement within the top-ranked languages is an infrequent occurrence,” Stephens said.  

    “[However], Python overtaking and sustaining its ranking over Java was a big deal. That said, we remain strong believers in the future of Java and the Java ecosystem, and we know there is a vast amount of enterprise Java that isn’t captured in the metrics we use,” Stephens said. “We don’t view Python’s ascendence as a zero-sum gain over the use of Java.”

    For JavaScript, “in spite of all of the competition from up-and-coming languages, all the discussion of fragmentation – and even criticisms of JavaScript the language itself – it remains remarkably popular,” O’Grady said.  

    “Simply put, JavaScript remains – its detractors notwithstanding – a force of nature like no other within the industry, and there are no indications in the data that this is likely to change any time soon,” O’Grady added.

    Meanwhile, in the book Hardcore Programming for Mechanical Engineers, published by No Starch Press, JavaScript trails Python in other rankings, while Python is also more popular than JavaScript for other reasons than those described in the RedMonk study. Python, for example, is cited as the ‘The Most Wanted Language,’ based on a Stack Overflow survey. 

    While Python is not the fastest language, as measured  in benchmarks listed in Hardcore Programming for Mechanical Engineers  — Go consistently outperforms it, for example, in execution times — Python’s popularity remains very high overall, according to the book’s author.

    “These results are not surprising; Python is an extremely versatile and productive language,” author Ángel Sola Orbaiceta wrote. “Writing code in Python is a delight, and its standard library is well-equipped: for just about anything you want to do, Python has something ready to help.” 

     

  • How to Make Self-Service IT More Intelligent

    How to Make Self-Service IT More Intelligent

    It’s an old story: DevOps’ need for speed and agility clashes with IT’s need for security and control. Striking the right balance between control and agility is key to fostering a smooth relationship between DevOps and IT.

    Many enterprises have achieved this balance by implementing a self-service IT model. In a self-service environment, IT creates a pre-approved catalog of resources that developers and engineers can access themselves, regardless of where they are, through a single portal. These resources might include storage, compute, networking or multi-tier app stacks. Thus, DevOps can get the things they need on their own, when they need them, avoiding the IT bottleneck while IT still maintains control.

    This gives DevOps the speed and agility they want while allowing IT to maintain control. IT can place guardrails on resources, such as role-based access controls and permissions, usage quotas, cost controls, security and compliance measures. IT can also set up approval workflows to ensure visibility and management. DevOps increases efficiency and expedites time-to-market. They will also be much less likely to go around IT and procure the resources they want on their own, such as spinning up public cloud resources with a credit card. This kind of activity leads to unauthorized apps and resources running in the background without IT’s knowledge – otherwise known as shadow IT – which can create significant security vulnerabilities and cost overruns.

    So, the balance between DevOps and IT, the yin and yang of speed and control, is attained. All is well in the kingdom, right? Not exactly. While self-service IT is essential, it needs to grow more intelligent to remain effective.

    Hybrid cloud environments are changing rapidly. As helpful as most current self-service IT systems may be, a multi-cloud strategy will continue adding more layers of complexity to IT infrastructure management. While a prudent enterprise will automate as many cloud management processes as possible and have stringent guardrails in place to enforce cost and security policies, there is still plenty of room for things to go wrong when you have multiple engineering and development teams consuming cloud resources in a rapidly-changing environment.

    When various combinations of resources are provisioned in ways that IT didn’t foresee, or new tools are added that didn’t exist when existing policies were drafted, you have an environment that’s susceptible to misconfigurations and inadvertent security risks. Furthermore, tools and resources available to DevOps via the self-service portal may become outdated or not be as effective or useful as they once were. If IT hasn’t replaced or upgraded them yet, DevOps might still be tempted to go outside the system to get what they need.

    Staying on top of the evolving challenges in hybrid cloud management will require a new, more intelligent approach to self-service IT. This approach will further empower DevOps and assist IT by embedding greater awareness into decision-making processes across the organization.

    Say, for example, a developer writes a piece of code. Unbeknownst to the developer, something they just wrote in the code could potentially create a security vulnerability. The old way of preventing cybersecurity breaches resulting from flawed code entails top-down, more centralized security scans or reactive audits.

    The new, more proactive approach would leverage an advanced automated alert and visualization system that informs people about the impact of the decisions they’re making, as they’re making them. In the example above, the developer would automatically receive an alert about the security issue in the code so it can be rectified right then and there, before resource deployment.

    Similarly, this shift-left-and-rectify approach can be applied to cost control and compliance issues, as well. If resources are being provisioned sub-optimally and would incur unnecessary expenses, DevOps would be automatically alerted to the oversight and presented with a cheaper option. IT can make the solution even more cost-effective by allowing developers to take direct actions from the platform, instead of going to various portals. (According to the FinOps Foundation’s State of FinOps 2021 report, 39% of engineers do not currently have the necessary tools to take these actions). This type of intelligent automation can also ensure regulations and standards such as PCI-DSS, CIS and the AWS Well-Architected Framework are always adhered to.

    While traditional self-service IT systems still serve a vital purpose, these types of proactive cloud management and optimization initiatives will keep self-service IT agile and able to adapt to rapidly changing conditions. By embedding greater awareness into decision-making across the enterprise, DevOps functions even faster and more efficiently, and IT maintains even greater control. Hybrid cloud environments can be managed more intuitively and with greater confidence that misconfigurations and oversights that lead to security issues, cost overruns, and compliance risks will be intelligently and continuously remediated. The yin and yang endures.

  • Why a Unified Enterprise DevOps Platform is Necessary

    Why a Unified Enterprise DevOps Platform is Necessary

    As a senior DevOps consultant working with large organizations over many years, I have been involved in more than 70 digital transformation projects, with hundreds of applications. While many things have changed over time, one thing has been a consistent source of problems that presents a formidable impedance to these transformations. The diverse number of DevOps platforms and tool variants used in those platforms results in a chaotic software delivery environment. From the point of view of any one application, almost any choice of DevOps platform and toolchain is fine, to an extent. But when you stand back and look at the larger picture, an enterprise with many variations of DevOps platforms and tools is at best chaotic; at worst, it is a gross impediment to overall organizational agility – which undermines the very purpose that DevOps is meant to serve.

    I relate to the quote “You don’t need another CI/CD automation tool. You have too many already.” The aggregate set of platforms and tools is like a Frankenstein’s monster with many incompatible parts resulting in an awkward looking product that is hard to control, replicate and evolve.

    DevOps Platform Blockages

    Success with DevOps requires frictionless interplay between the people, processes and technologies that need to cooperate for continuous deliveries to happen effectively. All three of these factors are impacted when an organization has too many variants of DevOps platforms.

    People factors – It starts with leadership. Business leaders want to leverage the values of continuous flow of software improvement made possible by DevOps. However, the flow is bottlenecked when the services of the organization must be delivered through a disparate combination of unconnected pipelines, each with separate processes to monitor, manage, troubleshoot and maintain.

    Process factors – Continuous delivery works because the DevOps platforms enable automation of processes to remove bottlenecks in flow. Process management and improvements are seriously slowed down when the details of automating the orchestration, execution and monitoring for software version management, integration, testing, security assurance and deployment processes are not standard between different DevOps platform variants.

    Technology factors – Often application development teams have their own favorite DevOps platform and a chosen set of tools. However, when application development teams in the same organization choose different DevOps platforms and tools, sharing knowledge, process data and practices becomes difficult. These incompatibilities slow down the overall evolution of continuous delivery for the organization.

    Requirements for a Unified DevOps Enterprise Platform

    Culture barriers, process problems, and DevOps platform “wars” are factors impacting the progress of digital transformations. A new unified enterprise DevOps platform would accelerate time-to-value and realize efficient, safe operation for digitally transforming enterprises and organizations. The following are general requirements for a unified enterprise DevOps platform.

    ● A new unified approach that supports end-to-end capabilities for continuous delivery of software, as a managed service, for multiple streams of pipelines is needed.
    ● Unified capabilities should include visibility, orchestration, integration, security, compliance, governance, traceability and management of flow for application value streams across multiple pipelines.
    ● Businesses need unified metrics that show business value, costs of the value stream and timing for end-to-end continuous delivery flow across multiple pipelines.
    ● A new unified approach is necessary to simplify and accelerate toolchains that support new cloud-native and cloud-adapted applications, infrastructures, machine learning, commercial off-the-shelf and open-source software.

    What This Means

    Digital transformations are an imperative for modern enterprises and organizations to remain competitive. However, the transformations of many enterprises and organizations are being seriously impeded by having too many DevOps platforms and tools. A new unified enterprise DevOps platform is needed to resolve the technical problems of broken, leaky, “Frankenstein” toolchains and processes.

  • Ambassador Insights: Value Stream Management

    Ambassador Insights: Value Stream Management

    Value stream management is the next critical evolution for enterprises and organizations that want to further mature their efforts toward smart, efficient and effective software development and delivery. The need for ongoing software innovation and delivery is becoming a top priority for all businesses today, thanks to the new digital demands organizations are facing from their employees and customers.

    Value streams help ensure that software and services under development deliver on the promise of value for employees or customers. But it’s not as easy as it seems; plenty of enterprises still struggle to know exactly what value is being derived from their software development efforts.

    In our third episode of Ambassador Insights we talk with DevOps Institute ambassadors Siddharth Pareek, the Global DevOps Center of Excellence Leader at NatWest Group; Garima Bajpai, the founder of Canada DevOps Community of Practice; and Steve Pereira, founder of Visible, about how value stream management can help enterprises on the next step of their software development journey.

    The video is below, followed by a transcript of our conversation. Enjoy!

    Transcript

    Charlene O’Hanlon: Hey, everybody, welcome back to TechStrong TV. I’m Charlene O’Hanlon, and I am here now with a group of very, very smart people in the value stream management space. They are part of the DevOps Institute Ambassadors, and they are here for our monthly installment of the DevOps Institute Ambassadors Insights Panel Discussion.

    This month, the topic is on value stream management, so like I said before, these guys are experts in the field, and I’m sure we’re gonna get some really insightful information from them. Thank you, all three of you, for joining me today.

    We have Siddharth Pareek, who is the Global DevOps Center of Excellence Leader at NatWest Group. Siddhartha, thank you so much for joining me today.

    Siddharth Pareek: The pleasure’s all mine.

    O’Hanlon: We also have Garima Bajpai, who is the Canada DevOps Community of Practice—is that correct, is that your correct title, Garima?

    Garima Bajpai: Yes, I’m the Founder for the Canada DevOps Community of Practice, and thank you for the introduction.

    O’Hanlon: Alright. Well, thank you for joining me today. And last but certainly not least, we have Steve Pereira, who’s the Founder of Visible. Steve, thank you so much, it’s great to see you again.

    Steve Pereira: Thanks. Great to see you, it’s great to be here.

    O’Hanlon: Alright, great. So, let’s talk value stream management. You know, it’s something that we’ve been talking a little bit about here and there on DevOps.com over the last couple years, but it seems like in the last year or so, especially, it has taken on more steam, if you will. It’s definitely more significance, a lot more people having the conversation about value stream management and its role in organizations, but I think it’s probably important that we at least start at the beginning, level set a little bit, and maybe, Steve, we can start out with you. Can you just kinda maybe loosely define what value stream management is and what it means to organizations?

    Pereira: Yeah, absolutely. That’s a great starting point. I think folks in the DevOps space are familiar with the origins to a degree where we had agile kind of be wildly successful in boosting development performance, right, and then all of a sudden, we hit this block where it’s like, okay, well, we can’t really go any faster unless we look at the bigger picture, right, and we start to involve Operations.

    Well, that’s where I see value stream management now is that DevOps has reached this point where we’re seeing massive performance improvements, we’re seeing a lot of positive outcomes, and we’re hitting the boundaries of that where all of a sudden, the bottlenecks and the opportunities are elsewhere at a further zoom point. So, if we now zoom out again and look at a larger sphere of influence, there’s a ton of opportunity outside of DevOps where value stream management is really the recipe for success. It’s a holistic operational capability that allows you to tie together this big picture view of your organization—and not just software development, but value streams are everything in your business, right? Your entire organization is just a collection of value streams delivering value to customers.

    So, the ability to organize, visualize, measure, and manage all of that in a very, let’s say, low touch, high automation way is going to be a game changer for organizations of all kinds, whether they are primarily software driven or building and delivering software, or if they are supply chain health care logistics, it really doesn’t matter. It’s gonna touch everything, because it has to, to drive the level of performance that we want and that end to end level of productivity.

    O’Hanlon: Okay, alright. Garima, have you, in your position, you’re obviously very deeply embedded in the DevOps space, have you seen kind of an increased interest in value stream management over the last year, especially in the DevOps space?

    Bajpai: Yes, I have, and I actually come from a large enterprise background and I work with complex systems, largely monolithic environments, which are transitioning into cloud native applications.

    Now, when we talk about these large enterprises, value stream mapping, to me, has been a tool in the toolbox of value measurement. So, I would actually reflect from the standpoint where Steve had left. So, value stream mapping, to me, is as much an art as it is science. So, what Steve was mentioning, the scientific aspect of value stream management, which is basically leading to more productive, cost-efficient organizations where we can bring speed and scale.

    But it has also an element of creativity where we bring the discovery part of the value, right? So, how do we ensure your value stream management is able to discover new value for the organization? That’s the first point. The second point is, visualization of that value is also important. So, how your value stream management is able to map all the complex value streams which you have in an enterprise and bring it together to visualize that value in one way. So, that’s the second thing.

    I also wanted to bring another point, which I have experienced when working with large organizations. When you think about a data-driven approach towards value, there is no unified process to combine the value in one way, to build consensus in the organization. There are various functional silos and complex structures, and there is a maze of organizations which we have to bring together for consensus building. And that is where value stream management is pretty useful.

    And the fourth aspect I would like to bring from a creative aspect is, we also combine value stream management with applied data science to produce the future of the organization, the future state, defining the future state.

    So, these are the four elements I would like to bring into the conversation, probably changing the conversation a bit as well, so, let’s see how it goes. [Laughter] 

    O’Hanlon: [Laughter] Alright, great. So, Siddharth, kind of expanding on that, is your organization currently using value stream mapping or value stream management, and if so, have you guys noticed any changes in when you’re using it versus before you started using it?

    Pareek: I’ll put it this way. We’re at the starting phase of the journey of adoption of these value stream mapping concepts. And I could easily answer and say that what Steve and Garima said, I’m done with it. But I’ll add something to the flavor so that the audience understands value stream mapping more.

    What you are doing when you’re talking about DevOps is, you’re talking about reducing failures. We are talking about releasing early, we’re talking about less failures, less bugs, all those things. What is really the customer’s, the end user who is really worried about it, really concerned about it, maybe not. They are probably concerned about the customer satisfaction. They’re talking about how much reviews they’re getting on their product on different platforms, or maybe more.

    And what is happening is, there is a gap between these two things—what developers are delivering and what the customer is expecting. And that is where the role of value stream mapping comes from, where it bridges the gap between the IT side and the business side, which is clearly visible to everyone across the stream. And what it does is, it tells you a similar story for everyone. When you ask a Dev guy what are the values you’re delivering, he will not talk about incident reductions or chain failure occurrence or DORA metrics, all those things. They will talk about, we are delivering what businesses expect of us.

    Coming back to the story of why, how much it is done within the organizations of ours, I think this needs to be picked up and probably come up in the later conversations. Why it has not been done so much, because of the problems that exist within the organizations.

    So, in short, we are starting the journey, but miles to go.

    O’Hanlon: Okay, alright. So, it sounds like value stream management is a great idea and organizations really should be considering it, but it might not be as simple as just flipping a switch or saying, “Hey, let’s do value stream management.”

    So, what are some of the things, then, that may be tripping up organizations as they start on their value stream management journey? And Siddharth, since your organization is kind of in that place right now, what are you seeing as kind of the roadblocks, if you will, to value stream management in your organization?

    Pareek: I mean, we talk about value stream mapping, we talk about a holistic view of the place on the left-hand side where the idea generates to the place on the right-hand side somewhere, where the value is delivered to the customer. And over there, software development teams or a testing team has a small role to play. 

    The bigger roles are your governance, your security, your compliance. How can you visualize them? How can you make, you know, process re-engineer them? For example, the NatWest Group which I worked for, they have huge compliance laws that need to be followed. Can just a simple value stream exercise streamline them? Maybe yes, maybe no. And why I say either of them is, because it depends upon the deep-rooted problems that could be solved or could not be solved going forward.

    O’Hanlon: Alright, great. Steve, I know your organization deals a lot with companies that are going through the value stream management kind of adoption journey. So, what are you seeing as some of the things that organizations are having trouble with as they start out using value stream management or as, even in their consideration phase?

    Pereira: Yeah, there’s a lot going on. There’s tons of complexity. There’s a huge spectrum of circumstances. Certainly, Siddharth really touched on a major one, which is the existing requirements in an organization that enforce the status quo, right? The things that keep us where we are, are these powerful forces of, like, things need to be done a certain way and we expect them to follow process. So, thinking of how to improve process or things that will affect the flow of value through the organization always get pushed to the wayside, right? Because it would be nice to have that, but we gotta keep everything rowing as is, right? We can’t risk that and we can’t ever forget about the fact that nothing can fall through the cracks in the status quo.

    But that’s where we see that drift, you know, the slow drift into legacy and calcification and friction is that. That doesn’t mean that you’re moving forward, right? It doesn’t mean that you’re going to make progress. And the promise, here, of value streams is really in re-orienting the entire organization to be driving vastly improved value flow, but also, as Garima mentioned, innovation, right? It’s not only just making things better, it’s making better things and it’s making different things, right? It’s this idea of thinking in terms of value and orienting towards value allows you to re-imagine your entire organization, and that is kind of the promised land. 

    But to get there, you have to get over the realities of the org chart, right? The realities that we structure our organizations based on legacy hierarchies where everyone has to fit in and report to somebody and then they have reports and you have to go up and over through the org chart. If you look at an org chart and you map the flow of value to the org chart, it’s just this wandering path that goes all over the organization and no wonder it’s inefficient, no wonder we struggle to deliver value and really make meaningful improvements.

    So, I think I would say demoting the org chart in terms of importance and focusing more on the streams and the interactions between the streams, starting to see those streams, and then aligning our teams and aligning our activities and our operations to those streams is no easy feat, but we can start really easily by just seeing it, right? Just seeing what’s going on, just observing the current state, that lays the groundwork.

    So, I would say that there can be a lot of defensive reactions to the idea that the org chart is not that important to customers, to value delivery. That could threaten people who are really invested in that hierarchy and their position in it. But ultimately, it does bring everybody together in a much more meaningful way to think in terms of streams.

    The other challenge, I think, is that, when we think about surfacing gaps and surfacing issues, a lot of cultures means that people get pointed at, right, and it’s, they’re scapegoating and there’s people, individuals who are responsible or those challenges, and we know that that’s not true. But the cultures that are in legacy trouble, are in these calcified states and not succeeding with transformation efforts, it’s often because that’s the mindset. The mindset is, we need to find the person accountable and force them to tackle the problem. It’s not our problem, it’s their problem, because I have my problems and someone’s trying to scapegoat me for something, so it’s only fair it’s going around, right? 

    And the level of visibility and the metrics and the measurements that you can get from value stream management can be used or abused, right? You can use those measurements to draw all kinds of conclusions, and in the wrong culture, it can be weaponized. That’s not to say that there’s not a ton of value and a ton of promise there, it’s just, if it’s misapplied or it’s done in a way that’s more kind of tailoristic or legacy founded, you’re potentially doing harm by surfacing all of this reality that people tend to ignore and just go about their business.

    O’Hanlon: You bring up some good points, and obviously, it doesn’t sound like value stream management, adopting it, is a walk in the park, but obviously, there are clear benefits, otherwise organizations really wouldn’t be considering it.

    I’m wondering if you could maybe kinda choose one or two things that maybe organizations need to do to derive the value from value stream management. Garima, I’d like to start with you. What are some of the things that, as I said, organizations really need to do to kinda derive that value from value stream management? Considering all the things that Steve just said, it sounds like there’s really an uphill battle for a lot of organizations.

    Bajpai: Right, and I think we will start from what Siddharth as well as Steve has mentioned that, coming from a large enterprise background, I see that there’s a lot of valueless complexity in large organizations, which quickly leads us to job security. So, there is a lot of organizations which are struggling with this problem.

    Now, to disable this problem, I think what we will necessarily have to do is, first of all, nobody wants to do a useless job. We need to be sure that we have inculcated behavioral aspects into value stream mapping. And, you know, looking from a bird’s eye view perspective, there are three things which we need to necessarily do.

    The first and foremost thing is empowering people and building accountability where you can actually visualize that in your value stream map, everybody is accountable for a value, you know? Nobody wants to do a useless job, right?

    The second part of this whole problem is making the value stream mapping as efficient as there is a loop that is basically, you’re incrementally building the value, you’re not trying to kind of big bang this approach of doing your value stream mapping and engaging everybody to kind of ensure that we decompose this complexity in the organization.

    And the third aspect, which is really very important, is that, first set the strategy. Whenever you are trying to do a value stream management, what is the objective? There can be four objectives—cost, you might need to ensure speed, you might need scalability in the organization, or you want to introduce new value into the value stream. So, there are four possibilities. So, what are your key objectives? Just define that strategy before you prioritize or you do any kind of decomposition of the value stream, from a value stream management perspective.

    O’Hanlon: Alright. Siddharth or Steve, do you have anything that you’d like to add onto that?

    Pareek: Yep. One thing is, and the organizations started very wrongly is, to confuse value stream mapping with the process mapping, to think both are somehow a similar exercise. However, people need to understand, process mapping talks about a very micro view of what things are happening, whereas value stream mapping talks about the macro view.

    And why that is important to know is, if you go into the process mapping stage, you may optimize one component of your whole floor, which is good for that component, but if you look holistically, it may create chaos for other components as well. 

    So, let’s suppose, you know, you will have streamlined a manual process of delivering certain products at a faster rate. You’ve improved that process, but what you’ve done is, maybe for other components, you have created a long stack of inventory to clear it out. And that is where this value stream mapping would come across. Because you’re going one level above the thought process, or from micro to macro. So, that is one proof that people make it happen.

    The second thing that makes it happen is, people go with jumping on the value stream mapping exercise or sessions they need to have. The first and primary thing is, they need to get the buy-ins of those stakeholders, because after the exercise, this scenario, the output would be very scary for people, accept and to action upon. So, they should have a good acceptance that yes, we agree to what happens after these sessions. Two things. I think that’s it from my side.

    O’Hanlon: Okay. Steve, any thoughts?

    Pereira: Yeah, I think that one thing that I want to clarify for the audience is that we’re talking a lot, I mean, this session is about value stream management and we keep talking about value stream mapping. And it speaks to the starting point, you know? It speaks to how you actually start to make traction with this and get going is understanding the big picture and it starts with mapping. And for me, it starts with mapping outcomes to obstacles and key objectives in the organization.

    So, starting with clarity, and we’re talking about the difficulty involved here and how complex this is. Really, we’re getting so much better at adopting these techniques, because we’re starting to realize that you can’t get there with agile, you can’t even get there with DevOps, and we struggled for decades trying to make this work. The value stream thinking approach, the holistic approach, the fact that this is the entire organization and it’s entire ecosystems that are involved in this. The flow of value is larger than we imagined initially, right? It’s not software development, it’s not even Development and Operations, right? It affects everything in the organization, but that perspective puts you decades ahead of where we were stuck in the mud.

    And so, just adopting that way of thinking and stepping back and saying, “Okay, well, what’s really going on here? What are we dealing with in the current state?” That’s gonna put you in this really strong position to adopt things like value stream management. But value stream management to me is the act of managing the value stream. It’s not software. There are tools that are available to do that job and we’re in early days of that, but management is the act of managing value, right? It’s visualizing, it’s tracking the flow, it’s understanding the flow of value through the organization.

    So, that really, to me, starts with mapping and this baseline view. And I’ll echo everything that the other folks said—bringing in the right perspectives from the top to the bottom is really gonna make that happen. And it’s really, you know, the first opportunity where we’ve had that invitation in a very accommodating way, right? This is truly inclusive. There’s something in this for the business. There’s something in this for customers there’s something in this for technical folks, and everybody gets some value out of it. Everybody gets tons of value out of it, but it’s not just, “Yeah, you need to look at DevOps, because otherwise, this tech thing is gonna get out of control.”

    O’Hanlon: Mm-hmm.

    Pereira: There’s messaging and there’s value in it for the business, and there’s something in it for the customers, and that’s unique and that is a perspective that we’ve never had before, but is starting to come to reality.

    O’Hanlon: Alright. Well, definitely, as I said before, we’ve been having the conversations on DevOps.com about value stream management and value stream mapping, and I firmly believe that, at the very least over the next year, we’re going to be hearing a lot more about value stream management. 

    So, Steve, Garima, Siddharth—thank you, all three of you, for bringing your expertise to today’s DevOps Ambassadors Insights Panel on value stream management. I appreciate it very, very much, and I do value your expertise, so thank you, all, for joining me today. I really do appreciate it.

    Pereira: Thanks for having us. Great to see you all.

    O’Hanlon: Alright.

    Bajpai: Thank you. Bye bye.

    Pareek: Thanks, Charlene. Goodbye, everyone.

  • Make GitHub Backups Part of Your Development Process

    Make GitHub Backups Part of Your Development Process

    GitHub’s platform is the largest host of source code in the world, hosting over 190 Million repositories. So much of the code we rely on every day is hosted there. You are, in all likelihood, using it. But what would you do if, one day, all of the project code GitHub stored for you disappeared?

    Just like all other cloud services, GitHub follows a cloud computing principle called “shared responsibility” for data protection and security. It essentially means that users of a cloud platform and providers of said platform split the responsibility for safeguarding data. In other words, users are just as responsible for mitigating risks on their side of the equation as GitHub is on theirs. It’s all laid out in GitHub’s terms and conditions:

    Taking this into account, there are a handful of things that can impact your ability to access to data in GitHub. Here are a few:

    Account Compromises

    In July of 2019, Ubuntu Security reported that the credentials for a company-owned GitHub account were compromised. These compromised credentials were used to create repositories, issues and more.

    In this case, critical infrastructure was decoupled from GitHub, and the breach wasn’t allowed to spread. However, Ubuntu had to restore various repositories and issue trackers to their previous state. When rolling back from an account compromise like this, backups are infinitely helpful, as they give you a previously good state to compare with the current state.

    Additionally, attackers often leave backdoors in compromised codebases. This can allow them to gain deeper access once the initial discovery and remediation process is completed. If you only have your infected codebase, it can be challenging to uncover all the infected files or identify possible vectors for future attacks. Restoring to a previous version of your code eliminates that threat.

    Ransomware Attacks

    Ransomware is the act of taking control of data and encrypting it so that only the attacker can unlock it. The attacker will usually ask for a ransom to unlock the files; otherwise, they may delete the data.

    In May 2019, ZDNet reported on a ransomware attack in which a hacker held various repositories hostage for a fee. The hackers modified the git histories to the point where the repositories were unusable, and they demanded payment within 10 days to reverse the changes.

    These attacks can create a severe disruption in operations. Developers will be unable to commit code, creating a complete stoppage in new feature development. Bug fixes and even support tickets (if managed through GitHub) could also be affected. However, restoring from a backup would allow a business to continue working.

    Service Downtime

    Depending on a third party always carries some risk, and GitHub is no exception. No service can deliver 100 percent uptime, but when your entire business (or codebase) depends on GitHub’s availability, you might want to mitigate that risk by having your own backups.

    For example, in June 2020, GitHub experienced a major, hours-long outage before stability returned. So, if all your work was stored in GitHub, you would have had to wait for access to be restored. This downtime can be devastating if it occurs during a crucial launch window.

    These are just three examples, and they happen more often than you think. A recent report by Oracle and KPMG found that 49 percent of IT professionals could attribute recent data loss to their failure to safeguard data. That’s essentially the same odds as a coin flip. Having continual access to all your work shouldn’t be left to chance. That’s why you need a backup strategy.

    Strategies for Backing Up Data in GitHub

    1. Managing Your Own Backups: This means you are responsible for all the infrastructure, business processes and ongoing repair costs to create the backups. You might think this will be a more cost-effective option, but the ongoing labor and maintenance expenses tend to add up quickly. It also means your team may use up cycles on something that is not part of the core business. What you make up for in control, you lose in time and spent resources.
    2. Using a Third Party to Manage Your Backups: Sometimes referred to as Backup-as-a-Service (BaaS), this involves outsourcing backup management responsibilities to a separate company. It removes the responsibility from your business, but it might seem more expensive upfront. In most cases, there is nothing to do after choosing a provider. They manage the entire process, from cradle to grave. This includes any API updates (which can happen often), implementation and ongoing maintenance. The drawback is that you lose control. Terms of service can change, or the data that’s backed up could change. And, not all BaaS solutions are transparent in what they do and how they access data.

    So, What’s the “Right” Choice?

    For most businesses, the sheer amount of work required to build and maintain backup software is a non-starter. Development cycles are too valuable to tie up with work not directly supporting the business’s product roadmap. You may get to a size where you think adding this competence makes sense; however, even some massive multinational corporations use some form of BaaS.

    Just make sure you do your research. Read reviews, speak to their sales or development team and confirm that they know what they are talking about. You need to determine if they have built a credible product. For many PaaS and SaaS tools, third-party backup applications are built by faceless companies with no track record. Considering the level of access they will have to your data, you want to ensure they are reputable with a proven history of successfully building backup software.

    Regardless of the method you choose, having a backup strategy in place is vital. Due to the nature of how GitHub works and how important your code repositories are to your business, the sooner you get something in place, the better.

  • On-Premises Software Delivery Tool: Build or Buy?

    On-Premises Software Delivery Tool: Build or Buy?

    The on-premises software market is four times larger than the SaaS market, but harder to reach. Every software vendor who wants to pursue it has to decide how they are going to package and deliver applications inside customers’ firewalls, automating away the complexity of keeping applications deployed and running. Should they build or buy a deployment tool (and how should they think about the features they’ll need beyond just deployment)? Both have their pros and cons, but as demand to reach the larger enterprise market grows, more software vendors are choosing to buy.

    Deploying apps on-premises used to be very different. Fifteen years ago, sysadmins and database administrators (DBAs) would manually configure servers and the applications that ran on them. When something broke in production, admins would often troubleshoot and fix by SSHing in to tweak a config file or setting. But now, particularly within agile development organizations, everything is done via automation (i.e., the evolution of DevOps and SRE). As a result, customers have lost patience with 150-page installation manuals and bespoke, hand-written scripts.

    Today, any ISV looking to modernize an existing on-premises app (or SaaS-only company considering creating their first on-premises offering) is, most likely, going to default to using Kubernetes because it abstracts away the technical overhead of keeping production-grade applications operating.

    With that in mind, software vendors should consider seven areas when deciding whether to build or buy a delivery and management solution. These include:

    Maintenance

    There are free or low cost resources if a software vendor wants to go it alone. Kubernetes and cloud-native technologies let capable engineers stand up a solution in months. But software vendors often overlook the long-term cost of keeping the tool up and running. There is a lot of tooling “around Kubernetes” the vendor has to build to successfully operationalize this. Sure, if you’re going to deploy to up to five customers, you don’t need formal tooling and processes that make it easy to engage whole teams of people (favoring a small build and manual effort), but when you want to go from five to 50, or 50 to 500 customers, you need advanced tooling (favoring buy).

    Cost

    Purchasing a ready-made solution comes with up-front costs, but building and maintaining a bespoke solution can be much more expensive. The obvious costs of building a deployment tool are design, engineering, support and operational costs. There are larger, long-term costs to consider, most notably pulling away one of your greatest assets, software engineers, to spend months (or years) on ongoing support needs. Redeploying your engineers results in opportunity and productivity costs, such as neglecting your core application(s) and stalling your business growth.

    Internal Resources

    You also need to evaluate your internal resources. Building a bespoke deployment tool is a commitment because the tool becomes a product in and of itself. Do you have a team with the available time and the technical skills? If your team isn’t large enough, you may incur costs for expanded headcount or contractors. If you don’t have added FTE costs, you may have a skills gap. When developing a custom deployment tool, most teams focus on the features that enable software distribution and installation, but they don’t consider management and troubleshooting features. In these situations, the FTE and skillset costs may be higher than paying for a comprehensive solution.

    Business Continuity

    Most homegrown solutions are created by an engineer or two who become experts in the systems they’re developing. This makes tons of sense for core business logic, apps that are constantly going to be improved and onto which new members will be onboarded. However, for ancillary systems that support the core business, these applications end up becoming a silo of knowledge. The question isn’t, “How much time will this take me to maintain after I build it?” but rather, “Who will maintain this after me, and who after them?” followed by, “How will they learn the intricacies of this system?” If the answer(s) to this question isn’t clear, this approach should be reconsidered.

    Beyond The First Team

    If you expect that your team will be the first, or one of the first, teams within your company to face the problem you are solving for (but you are certain others are sure to follow), then this decision is even more complex. A common customer delivery and management platform can increase cross-sell by simplifying the marginal effort for your customers to deploy a second, third, fourth, etc. application from your company. Buying will likely require a long-term relationship with a vendor dedicated to solving the problem. Similarly, building a solution will require extra investments. Beyond the initial technical/product requirements of abstracting a home grown solution to meet the needs of additional teams/architectures/products, there are also the other implications. The winning solution will win because it will evolve to meet the needs of multiple teams and have strong internal advocacy. Depending on the size and complexity of the company, this internal advocacy can be quite time consuming and require more than just engineering talents (generally, this is what customer success and account managers do 100% of the time).

    Target Environments

    Your application may need to be deployed in varied on-premises environments and may not have the time to adequately address all of them. If your team has not previously built an installer, they may not have the skillset to build out richer features in AWS, Azure, VMware and others, to solidify the customer experience and day 2 operations, beyond just bringing the application up. An AWS Cloud Formation template might get you out the door, but when the next customer says, “I need GCP,” you’d have to start over.

    It’s rinse-and-repeat for each of the target environments, which slows your time-to-market, potentially losing high-value sales. Conversely, if you look at ready-made solutions, you can select the best out-of-the-box experience for your customers.

    Control and Compatibility

    There are situations where your team must build the on-premises deployment tool themselves. For example, when the app requires so much last-mile configuration for each customer deployment that there just isn’t a way to standardize it. Or if you don’t have enough customers to justify paying for a commercial product, it might make more sense to do a quick sprint and hack something out.

    An off-the-shelf solution can significantly accelerate your time to market, but you have to order off the menu. Also, if you have access to your customer’s infrastructure, then building your own tool may make more sense. On the other hand, requirements such as faster time-to-market, an off-the-shelf comprehensive solution and lower costs over time may outweigh customization. If speed, lower costs and quality are important, a solution from a focused vendor who has thought through all of the technical and operational challenges, with no surprises or last-minute fixes, is the way to go.

    Competitive Advantage

    Your software engineering team may be your company’s greatest resource. But it’s far better to keep your engineers focused on innovation, adding value to your application and winning more customers through improved software. Coding a deployment tool is undifferentiated overhead. It does not make your business more successful. Installers and support tooling are table stakes to deliver a good customer experience, but the reality is that no customer is going to pay you more money for that capability.

    The Bottom Line

    Simplified access to the on-premises software market is one of the greatest opportunities for software vendors today. Kubernetes and cloud-native technologies make it easier to develop tools to package, deploy, manage and troubleshoot apps. But the job isn’t for everyone. The costs, both up-front and long-term, favor ready-made commercial applications for most software vendors. Consider your resources and customer needs to decide if it makes more sense to build or buy your solution.

  • Languages and DevOps: API-First

    Languages and DevOps: API-First

    One of the places development has made significant changes is in the land of APIs. These days, a program almost isn’t considered a proper application unless it offers APIs to do most of what can be done via UI (and even command line). This is a big deal for several reasons; first and foremost, the ability to subdivide the work and then test each subdivision. A proper API does one thing and does it well. Testing can validate that this statement is true for any given API. It also allows for distribution of processing. It is far easier to use a load balancer to scale APIs than it is to scale a monolithic application. And finally, it helps with portability, because with API-first, developer users don’t care what platform it’s running on, just that it can be reached via URL and responds as expected.

    Everything is a trade-off, and there are several places that API-first has issues. Integration testing is objectively harder. Loosely coupled applications, in general, are harder to test, and requiring multiple portions of an application be available to run testing becomes onerous, particularly as the layers of APIs start to build up. Second, security is another issue – particularly if you’re offering a public-facing API, but also for internal usage. Once an attacker has access to the network, internal-only APIs are no different to them than external. Smart IT departments design internal APIs with the same controls that external requires. Third is DevOps in general. API-first grew up in the agile/DevOps world, but because there are cascading dependencies, is actually harder to manage DevOps for than most monolithic applications. Finally, documentation – the bane of API-first’s existence. The number of poorly documented APIs vastly outnumbers the astoundingly well-documented ones. Even well-documented APIs, like Google Places, require some thought and testing to use well. Poorly documented ones – my favorite is, “It’s mysite.com/users – everyone knows what’s in there!” – are next to useless without a learning curve.

    With all of that said, what about languages? That greatly depends on what you’re doing. Users don’t care what language it is implemented in – as noted above, they care that the API can be reached via URI, and that it behaves as expected. So you do have options. Indeed, at this point in the evolution of APIs, the language options are so open that you’ll probably be choosing an API definition language instead of a programming language, and just instruct your definition tool to spit out the language of choice.

    Wait, what?

    For those who haven’t followed API development closely over the last few years, it goes something like this. APIs became standardized, then APIs became a big deal, then smart people started to think in terms of code generation á la ancient 4GLs. What if you could use a specification language to describe what you want to do, then pick your language and tell the tool to generate the API? What if you could do this for the entire API hierarchy? That’s where we are. I’m most familiar with OpenAPI and RAML, but there are a large number of specifications out there, and a larger number of tools to develop in them. Before my API friends freak out, of course this is an over-simplification of what happened. I only have 1000 words, max, and my editor would like less. Relax a bit.

    And then you generate.

    I’ll rely on my own experience here, but there are literally dozens of tools available to get you from idea to API interface (doing the work behind the interface is where you come in). With Swagger CodeGen, the tool I’m most familiar with, you specify the API, then generate server interfaces (they call them stubs), and generate client libraries. Finally, you develop what goes on behind the interface on the server side, and (you or others) use the API on the client side. When I say “generate,” I mean “Tell it your favorite language, and click generate.” It spits out the code you can then use to flesh out your API. It does not spit out the info to get the API into your DevOps toolchain, so you’ll have to do the work for that. Which means the more APIs you can generate in a batch, the less DevOps lifting you’ll have to do (more on that if/when we cover microservices). Since most languages can easily be placed into DevOps via their build tools, I don’t want to make this sound too onerous – security and testing will be a bigger problem, by far.

    API security is slowly getting automated and placed right into generated code, but all the usual bits, like certificate management, still have to be handled. The interesting thing, that hasn’t (yet) gotten a ton of attention, is security of clients. It is far harder to verify that the client library is secure when the purpose of that library is to grant access to your API. I expect we’ll eventually have to get there for some applications, but today, it appears to be largely individual orgs figuring out how best to lock down client libraries, and most of the limiting factors end up server-side.

    I’ve generated Python, Java and Ruby APIs via various toolsets, and considering the problem domain is still relatively small, they work fine. And it saves one piece of the puzzle … but does introduce another piece – coordinating client and server updates via DevOps.

    Keep making the apps that run the world, and use API generators to further standardization, spitting out whatever language you need on the backend, and a portable interface for clients.

    You. Rock.

    Whatever your language of choice, in the end, it will likely be your build tool that links it to DevOps. Your biggest concern for DevOps will probably be portability, at least in the near future. Meanwhile, keep rocking it. You’re doing the work that needs doing, even if it isn’t the part of DevOps everyone is currently buzzing about.

  • Logz.io Adds Prometheus Service to Observability Portfolio

    Logz.io Adds Prometheus Service to Observability Portfolio

    Logz.io announced today the general availability of a cloud service, based on Prometheus, that can collect and analyze metrics alongside its existing log management service.

    Jonah Kowall, CTO, Logz.io, said the goal for Prometheus-as-a-Service is to provide IT teams with a centralized approach to analyzing data collected from instances of Prometheus running on local servers and clusters. While Prometheus is relatively easy to deploy, it’s not designed to scale horizontally, which means IT teams have to deploy multiple instances of Prometheus on a fleet of clusters and servers. The Logz.io service makes it easier to aggregate and analyze all the data those instances of Prometheus collected, said Kowall. That approach not only offloads the storage of those metrics, Kowall said, it presents an opportunity to correlate that data against logs and traces that Logz.io already collects.

    Logz.io, as part of that effort, is also making it possible to upload a Grafana dashboard, typically employed to visualize Prometheus data as a JSON file, to Logz.io, which will then convert the Grafana dashboards to dashboards based on the Logz.io user interface.

    While Prometheus is closely identified with Kubernetes, IT teams are starting to deploy the open source monitoring tool more widely to collect metrics. A recent Gartner report predicts that, by 2025, half of new cloud-native application monitoring deployments will rely on some form of open-source software to instrument applications. As part of that transition, Prometheus has already emerged as a de facto standard, Kowall noted.

    The primary focus, at least initially, is still on monitoring IT platforms, but as IT teams start to correlate metrics, logs and traces, they begin to benefit from additional context, otherwise known as observability, Kowall said.

    Observability, of course, has always been a core DevOps principle. However, as more applications become instrumented using open source tools and agent software, it’s becoming less costly to achieve. DevOps teams will still need to instrument each application, but the costs associated with collecting metrics and other data continues to decline. Most providers of monitoring tools and observability platforms are moving toward adding support for open source tools and agents to collect data alongside the proprietary tools and agent software they already make available. It will be up to each DevOps teams to decide whether to rely on open source tools or continue to rely on proprietary software that may be able to surface deeper insights or is simply easier to deploy.

    Regardless of the path chosen, the rise of microservices-based applications with numerous dependencies is forcing the observability issue. Microservices-based applications may not crash as frequently as a monolithic application because they are, ideally, designed to degrade gracefully in the absence of a single point of failure. The challenge is that, once they do degrade, it’s difficult to pinpoint the source of the issue; this may result in applications running sub-optimally for an extend period of time.

    The good news is that DevOps teams should now have the tools required to diagnose and resolve those issues long before any end user notices.

  • Why Developers Should Care About Resilience

    Why Developers Should Care About Resilience

    Recently, a friend reminded me of a joke we used to have when we were both developers at a huge software corporation (we won’t mention names, but back when printers were a thing, you probably owned one of theirs). We didn’t develop printers. We developed performance testing and monitoring tools.

    We were the dev team, which was completely separate from the QA team and from the Ops team (yes, I’m that old – we didn’t even call it DevOps back then). Support was part of a completely different organization and “business” was something that happened in an altogether different galaxy. That is, until a customer had an issue.

    When an important customer had an issue, we definitely felt it at R&D. Managers called hourly to check on status, and we would have intense conversations with the support team who would then, in turn, have intense discussions with the customer. If we, the software engineers developing the product, ended up on the same call with both support and the customer, it was a sure sign that things were bad. You knew that millions of dollars were on the line, and, consequently, that pressure was high. But most of all: you knew that we would do anything we could to solve the problem, because we needed those millions of dollars to be paid, so that we could keep getting paid. And so, we kept the coffee flowing.

    This is where the joke part comes in. We used to have a system for managing support tickets, collecting information, tracking time and assigning it to the relevant software or support engineer, so everyone would know who was in charge. Today, you probably use Zendesk, ServiceNow or one of their alternatives. We used a home grown system, of course.

    Our joke was this: the purpose of the system was not to track and measure and propagate the handling of customer issues. The purpose of the system was to indicate someone else is now responsible for solving the problem. Or, if you would rather paraphrase one of my favorite Douglas Adams jokes, the issue is now under a Somebody Else’s Problem field.

    Because solving customer issues is difficult and frustrating, and being pressured by business could ruin a software engineer’s day. This is because when you’re a happy code monkey, and all you want to worry about is writing beautiful code and releasing cool features and drinking some coffee, solving a customer issue gets in your way.

    You end up spending too much time sifting through endless logs, crossing your fingers and trying to reproduce an issue that only happens at 2 a.m. on a Friday, usually when a blue zebra is passing by. Spending your time on a heated support call is less fun than, say, drinking coffee quietly with your friends.

    As R&D managers, we understand that our developers want nothing to do with solving customer issues. We want to make sure they are motivated and happy and efficient, and that they are free to challenge themselves by learning the latest buzz-worthy technologies and thinking up ways to introduce those into our product in a way that will make our customers happy.

    Additionally, as R&D managers, we are far more likely to understand the business impact of having too many support tickets or having support tickets that take too long to resolve. We know that the loss on such cases is twofold:

    1. The customer that faced an issue and is waiting for a solution may become frustrated, and the company may lose their business.
    2. Our engineers are busy solving that customer issue instead of delivering new value to the business.

    Therefore, as R&D managers, it is our responsibility to make sure that our engineers have the tools to handle these issues as efficiently as possible when they come up. It is our responsibility to make sure they are motivated, and that they understand the business impact of solving a customer issue as quickly as possible. And whenever possible, it is our job to make sure that they make our application as reliable and stable as possible to begin with.

    That ensures that we have test coverage, and the right knowledge, practices and tools to ensure that we have as few customer issues as possible to begin with.

    Happily, changes in dev culture usually means that “support” and “business” and even “customers” are not as far away as they used to be. When developing a SaaS platform, your developers often have an almost intimate, real-time ability to track what users are doing and how it impacts their experience. The culture of DevOps has done a lot for breaking the silos that used to separate the dev team from the support team, and to make sure that all members of the team engage and understand how the code they write and the issues they resolve help our customers.

    I hope you already found the way to motivate your team to solve such issues, and that they have enough time to drink coffee. If you and your team are still struggling, we suggest bringing in a live debugging tool that can help them solve customer issues even faster, by making sure they have all the data they need at their fingertips, and that they can instantly reproduce any issue in a live, remote environment. Live debugging tools give your engineers a developer-friendly experience, and also gives them the power to reduce issue resolution time, too. Now, all you need to worry about is that your engineers have enough coffee.