Category: Enterprise DevOps

  • Why a Unified Enterprise DevOps Platform is Necessary

    Why a Unified Enterprise DevOps Platform is Necessary

    As a senior DevOps consultant working with large organizations over many years, I have been involved in more than 70 digital transformation projects, with hundreds of applications. While many things have changed over time, one thing has been a consistent source of problems that presents a formidable impedance to these transformations. The diverse number of DevOps platforms and tool variants used in those platforms results in a chaotic software delivery environment. From the point of view of any one application, almost any choice of DevOps platform and toolchain is fine, to an extent. But when you stand back and look at the larger picture, an enterprise with many variations of DevOps platforms and tools is at best chaotic; at worst, it is a gross impediment to overall organizational agility – which undermines the very purpose that DevOps is meant to serve.

    I relate to the quote “You don’t need another CI/CD automation tool. You have too many already.” The aggregate set of platforms and tools is like a Frankenstein’s monster with many incompatible parts resulting in an awkward looking product that is hard to control, replicate and evolve.

    DevOps Platform Blockages

    Success with DevOps requires frictionless interplay between the people, processes and technologies that need to cooperate for continuous deliveries to happen effectively. All three of these factors are impacted when an organization has too many variants of DevOps platforms.

    People factors – It starts with leadership. Business leaders want to leverage the values of continuous flow of software improvement made possible by DevOps. However, the flow is bottlenecked when the services of the organization must be delivered through a disparate combination of unconnected pipelines, each with separate processes to monitor, manage, troubleshoot and maintain.

    Process factors – Continuous delivery works because the DevOps platforms enable automation of processes to remove bottlenecks in flow. Process management and improvements are seriously slowed down when the details of automating the orchestration, execution and monitoring for software version management, integration, testing, security assurance and deployment processes are not standard between different DevOps platform variants.

    Technology factors – Often application development teams have their own favorite DevOps platform and a chosen set of tools. However, when application development teams in the same organization choose different DevOps platforms and tools, sharing knowledge, process data and practices becomes difficult. These incompatibilities slow down the overall evolution of continuous delivery for the organization.

    Requirements for a Unified DevOps Enterprise Platform

    Culture barriers, process problems, and DevOps platform “wars” are factors impacting the progress of digital transformations. A new unified enterprise DevOps platform would accelerate time-to-value and realize efficient, safe operation for digitally transforming enterprises and organizations. The following are general requirements for a unified enterprise DevOps platform.

    ● A new unified approach that supports end-to-end capabilities for continuous delivery of software, as a managed service, for multiple streams of pipelines is needed.
    ● Unified capabilities should include visibility, orchestration, integration, security, compliance, governance, traceability and management of flow for application value streams across multiple pipelines.
    ● Businesses need unified metrics that show business value, costs of the value stream and timing for end-to-end continuous delivery flow across multiple pipelines.
    ● A new unified approach is necessary to simplify and accelerate toolchains that support new cloud-native and cloud-adapted applications, infrastructures, machine learning, commercial off-the-shelf and open-source software.

    What This Means

    Digital transformations are an imperative for modern enterprises and organizations to remain competitive. However, the transformations of many enterprises and organizations are being seriously impeded by having too many DevOps platforms and tools. A new unified enterprise DevOps platform is needed to resolve the technical problems of broken, leaky, “Frankenstein” toolchains and processes.

  • Ambassador Insights: Value Stream Management

    Ambassador Insights: Value Stream Management

    Value stream management is the next critical evolution for enterprises and organizations that want to further mature their efforts toward smart, efficient and effective software development and delivery. The need for ongoing software innovation and delivery is becoming a top priority for all businesses today, thanks to the new digital demands organizations are facing from their employees and customers.

    Value streams help ensure that software and services under development deliver on the promise of value for employees or customers. But it’s not as easy as it seems; plenty of enterprises still struggle to know exactly what value is being derived from their software development efforts.

    In our third episode of Ambassador Insights we talk with DevOps Institute ambassadors Siddharth Pareek, the Global DevOps Center of Excellence Leader at NatWest Group; Garima Bajpai, the founder of Canada DevOps Community of Practice; and Steve Pereira, founder of Visible, about how value stream management can help enterprises on the next step of their software development journey.

    The video is below, followed by a transcript of our conversation. Enjoy!

    Transcript

    Charlene O’Hanlon: Hey, everybody, welcome back to TechStrong TV. I’m Charlene O’Hanlon, and I am here now with a group of very, very smart people in the value stream management space. They are part of the DevOps Institute Ambassadors, and they are here for our monthly installment of the DevOps Institute Ambassadors Insights Panel Discussion.

    This month, the topic is on value stream management, so like I said before, these guys are experts in the field, and I’m sure we’re gonna get some really insightful information from them. Thank you, all three of you, for joining me today.

    We have Siddharth Pareek, who is the Global DevOps Center of Excellence Leader at NatWest Group. Siddhartha, thank you so much for joining me today.

    Siddharth Pareek: The pleasure’s all mine.

    O’Hanlon: We also have Garima Bajpai, who is the Canada DevOps Community of Practice—is that correct, is that your correct title, Garima?

    Garima Bajpai: Yes, I’m the Founder for the Canada DevOps Community of Practice, and thank you for the introduction.

    O’Hanlon: Alright. Well, thank you for joining me today. And last but certainly not least, we have Steve Pereira, who’s the Founder of Visible. Steve, thank you so much, it’s great to see you again.

    Steve Pereira: Thanks. Great to see you, it’s great to be here.

    O’Hanlon: Alright, great. So, let’s talk value stream management. You know, it’s something that we’ve been talking a little bit about here and there on DevOps.com over the last couple years, but it seems like in the last year or so, especially, it has taken on more steam, if you will. It’s definitely more significance, a lot more people having the conversation about value stream management and its role in organizations, but I think it’s probably important that we at least start at the beginning, level set a little bit, and maybe, Steve, we can start out with you. Can you just kinda maybe loosely define what value stream management is and what it means to organizations?

    Pereira: Yeah, absolutely. That’s a great starting point. I think folks in the DevOps space are familiar with the origins to a degree where we had agile kind of be wildly successful in boosting development performance, right, and then all of a sudden, we hit this block where it’s like, okay, well, we can’t really go any faster unless we look at the bigger picture, right, and we start to involve Operations.

    Well, that’s where I see value stream management now is that DevOps has reached this point where we’re seeing massive performance improvements, we’re seeing a lot of positive outcomes, and we’re hitting the boundaries of that where all of a sudden, the bottlenecks and the opportunities are elsewhere at a further zoom point. So, if we now zoom out again and look at a larger sphere of influence, there’s a ton of opportunity outside of DevOps where value stream management is really the recipe for success. It’s a holistic operational capability that allows you to tie together this big picture view of your organization—and not just software development, but value streams are everything in your business, right? Your entire organization is just a collection of value streams delivering value to customers.

    So, the ability to organize, visualize, measure, and manage all of that in a very, let’s say, low touch, high automation way is going to be a game changer for organizations of all kinds, whether they are primarily software driven or building and delivering software, or if they are supply chain health care logistics, it really doesn’t matter. It’s gonna touch everything, because it has to, to drive the level of performance that we want and that end to end level of productivity.

    O’Hanlon: Okay, alright. Garima, have you, in your position, you’re obviously very deeply embedded in the DevOps space, have you seen kind of an increased interest in value stream management over the last year, especially in the DevOps space?

    Bajpai: Yes, I have, and I actually come from a large enterprise background and I work with complex systems, largely monolithic environments, which are transitioning into cloud native applications.

    Now, when we talk about these large enterprises, value stream mapping, to me, has been a tool in the toolbox of value measurement. So, I would actually reflect from the standpoint where Steve had left. So, value stream mapping, to me, is as much an art as it is science. So, what Steve was mentioning, the scientific aspect of value stream management, which is basically leading to more productive, cost-efficient organizations where we can bring speed and scale.

    But it has also an element of creativity where we bring the discovery part of the value, right? So, how do we ensure your value stream management is able to discover new value for the organization? That’s the first point. The second point is, visualization of that value is also important. So, how your value stream management is able to map all the complex value streams which you have in an enterprise and bring it together to visualize that value in one way. So, that’s the second thing.

    I also wanted to bring another point, which I have experienced when working with large organizations. When you think about a data-driven approach towards value, there is no unified process to combine the value in one way, to build consensus in the organization. There are various functional silos and complex structures, and there is a maze of organizations which we have to bring together for consensus building. And that is where value stream management is pretty useful.

    And the fourth aspect I would like to bring from a creative aspect is, we also combine value stream management with applied data science to produce the future of the organization, the future state, defining the future state.

    So, these are the four elements I would like to bring into the conversation, probably changing the conversation a bit as well, so, let’s see how it goes. [Laughter] 

    O’Hanlon: [Laughter] Alright, great. So, Siddharth, kind of expanding on that, is your organization currently using value stream mapping or value stream management, and if so, have you guys noticed any changes in when you’re using it versus before you started using it?

    Pareek: I’ll put it this way. We’re at the starting phase of the journey of adoption of these value stream mapping concepts. And I could easily answer and say that what Steve and Garima said, I’m done with it. But I’ll add something to the flavor so that the audience understands value stream mapping more.

    What you are doing when you’re talking about DevOps is, you’re talking about reducing failures. We are talking about releasing early, we’re talking about less failures, less bugs, all those things. What is really the customer’s, the end user who is really worried about it, really concerned about it, maybe not. They are probably concerned about the customer satisfaction. They’re talking about how much reviews they’re getting on their product on different platforms, or maybe more.

    And what is happening is, there is a gap between these two things—what developers are delivering and what the customer is expecting. And that is where the role of value stream mapping comes from, where it bridges the gap between the IT side and the business side, which is clearly visible to everyone across the stream. And what it does is, it tells you a similar story for everyone. When you ask a Dev guy what are the values you’re delivering, he will not talk about incident reductions or chain failure occurrence or DORA metrics, all those things. They will talk about, we are delivering what businesses expect of us.

    Coming back to the story of why, how much it is done within the organizations of ours, I think this needs to be picked up and probably come up in the later conversations. Why it has not been done so much, because of the problems that exist within the organizations.

    So, in short, we are starting the journey, but miles to go.

    O’Hanlon: Okay, alright. So, it sounds like value stream management is a great idea and organizations really should be considering it, but it might not be as simple as just flipping a switch or saying, “Hey, let’s do value stream management.”

    So, what are some of the things, then, that may be tripping up organizations as they start on their value stream management journey? And Siddharth, since your organization is kind of in that place right now, what are you seeing as kind of the roadblocks, if you will, to value stream management in your organization?

    Pareek: I mean, we talk about value stream mapping, we talk about a holistic view of the place on the left-hand side where the idea generates to the place on the right-hand side somewhere, where the value is delivered to the customer. And over there, software development teams or a testing team has a small role to play. 

    The bigger roles are your governance, your security, your compliance. How can you visualize them? How can you make, you know, process re-engineer them? For example, the NatWest Group which I worked for, they have huge compliance laws that need to be followed. Can just a simple value stream exercise streamline them? Maybe yes, maybe no. And why I say either of them is, because it depends upon the deep-rooted problems that could be solved or could not be solved going forward.

    O’Hanlon: Alright, great. Steve, I know your organization deals a lot with companies that are going through the value stream management kind of adoption journey. So, what are you seeing as some of the things that organizations are having trouble with as they start out using value stream management or as, even in their consideration phase?

    Pereira: Yeah, there’s a lot going on. There’s tons of complexity. There’s a huge spectrum of circumstances. Certainly, Siddharth really touched on a major one, which is the existing requirements in an organization that enforce the status quo, right? The things that keep us where we are, are these powerful forces of, like, things need to be done a certain way and we expect them to follow process. So, thinking of how to improve process or things that will affect the flow of value through the organization always get pushed to the wayside, right? Because it would be nice to have that, but we gotta keep everything rowing as is, right? We can’t risk that and we can’t ever forget about the fact that nothing can fall through the cracks in the status quo.

    But that’s where we see that drift, you know, the slow drift into legacy and calcification and friction is that. That doesn’t mean that you’re moving forward, right? It doesn’t mean that you’re going to make progress. And the promise, here, of value streams is really in re-orienting the entire organization to be driving vastly improved value flow, but also, as Garima mentioned, innovation, right? It’s not only just making things better, it’s making better things and it’s making different things, right? It’s this idea of thinking in terms of value and orienting towards value allows you to re-imagine your entire organization, and that is kind of the promised land. 

    But to get there, you have to get over the realities of the org chart, right? The realities that we structure our organizations based on legacy hierarchies where everyone has to fit in and report to somebody and then they have reports and you have to go up and over through the org chart. If you look at an org chart and you map the flow of value to the org chart, it’s just this wandering path that goes all over the organization and no wonder it’s inefficient, no wonder we struggle to deliver value and really make meaningful improvements.

    So, I think I would say demoting the org chart in terms of importance and focusing more on the streams and the interactions between the streams, starting to see those streams, and then aligning our teams and aligning our activities and our operations to those streams is no easy feat, but we can start really easily by just seeing it, right? Just seeing what’s going on, just observing the current state, that lays the groundwork.

    So, I would say that there can be a lot of defensive reactions to the idea that the org chart is not that important to customers, to value delivery. That could threaten people who are really invested in that hierarchy and their position in it. But ultimately, it does bring everybody together in a much more meaningful way to think in terms of streams.

    The other challenge, I think, is that, when we think about surfacing gaps and surfacing issues, a lot of cultures means that people get pointed at, right, and it’s, they’re scapegoating and there’s people, individuals who are responsible or those challenges, and we know that that’s not true. But the cultures that are in legacy trouble, are in these calcified states and not succeeding with transformation efforts, it’s often because that’s the mindset. The mindset is, we need to find the person accountable and force them to tackle the problem. It’s not our problem, it’s their problem, because I have my problems and someone’s trying to scapegoat me for something, so it’s only fair it’s going around, right? 

    And the level of visibility and the metrics and the measurements that you can get from value stream management can be used or abused, right? You can use those measurements to draw all kinds of conclusions, and in the wrong culture, it can be weaponized. That’s not to say that there’s not a ton of value and a ton of promise there, it’s just, if it’s misapplied or it’s done in a way that’s more kind of tailoristic or legacy founded, you’re potentially doing harm by surfacing all of this reality that people tend to ignore and just go about their business.

    O’Hanlon: You bring up some good points, and obviously, it doesn’t sound like value stream management, adopting it, is a walk in the park, but obviously, there are clear benefits, otherwise organizations really wouldn’t be considering it.

    I’m wondering if you could maybe kinda choose one or two things that maybe organizations need to do to derive the value from value stream management. Garima, I’d like to start with you. What are some of the things that, as I said, organizations really need to do to kinda derive that value from value stream management? Considering all the things that Steve just said, it sounds like there’s really an uphill battle for a lot of organizations.

    Bajpai: Right, and I think we will start from what Siddharth as well as Steve has mentioned that, coming from a large enterprise background, I see that there’s a lot of valueless complexity in large organizations, which quickly leads us to job security. So, there is a lot of organizations which are struggling with this problem.

    Now, to disable this problem, I think what we will necessarily have to do is, first of all, nobody wants to do a useless job. We need to be sure that we have inculcated behavioral aspects into value stream mapping. And, you know, looking from a bird’s eye view perspective, there are three things which we need to necessarily do.

    The first and foremost thing is empowering people and building accountability where you can actually visualize that in your value stream map, everybody is accountable for a value, you know? Nobody wants to do a useless job, right?

    The second part of this whole problem is making the value stream mapping as efficient as there is a loop that is basically, you’re incrementally building the value, you’re not trying to kind of big bang this approach of doing your value stream mapping and engaging everybody to kind of ensure that we decompose this complexity in the organization.

    And the third aspect, which is really very important, is that, first set the strategy. Whenever you are trying to do a value stream management, what is the objective? There can be four objectives—cost, you might need to ensure speed, you might need scalability in the organization, or you want to introduce new value into the value stream. So, there are four possibilities. So, what are your key objectives? Just define that strategy before you prioritize or you do any kind of decomposition of the value stream, from a value stream management perspective.

    O’Hanlon: Alright. Siddharth or Steve, do you have anything that you’d like to add onto that?

    Pareek: Yep. One thing is, and the organizations started very wrongly is, to confuse value stream mapping with the process mapping, to think both are somehow a similar exercise. However, people need to understand, process mapping talks about a very micro view of what things are happening, whereas value stream mapping talks about the macro view.

    And why that is important to know is, if you go into the process mapping stage, you may optimize one component of your whole floor, which is good for that component, but if you look holistically, it may create chaos for other components as well. 

    So, let’s suppose, you know, you will have streamlined a manual process of delivering certain products at a faster rate. You’ve improved that process, but what you’ve done is, maybe for other components, you have created a long stack of inventory to clear it out. And that is where this value stream mapping would come across. Because you’re going one level above the thought process, or from micro to macro. So, that is one proof that people make it happen.

    The second thing that makes it happen is, people go with jumping on the value stream mapping exercise or sessions they need to have. The first and primary thing is, they need to get the buy-ins of those stakeholders, because after the exercise, this scenario, the output would be very scary for people, accept and to action upon. So, they should have a good acceptance that yes, we agree to what happens after these sessions. Two things. I think that’s it from my side.

    O’Hanlon: Okay. Steve, any thoughts?

    Pereira: Yeah, I think that one thing that I want to clarify for the audience is that we’re talking a lot, I mean, this session is about value stream management and we keep talking about value stream mapping. And it speaks to the starting point, you know? It speaks to how you actually start to make traction with this and get going is understanding the big picture and it starts with mapping. And for me, it starts with mapping outcomes to obstacles and key objectives in the organization.

    So, starting with clarity, and we’re talking about the difficulty involved here and how complex this is. Really, we’re getting so much better at adopting these techniques, because we’re starting to realize that you can’t get there with agile, you can’t even get there with DevOps, and we struggled for decades trying to make this work. The value stream thinking approach, the holistic approach, the fact that this is the entire organization and it’s entire ecosystems that are involved in this. The flow of value is larger than we imagined initially, right? It’s not software development, it’s not even Development and Operations, right? It affects everything in the organization, but that perspective puts you decades ahead of where we were stuck in the mud.

    And so, just adopting that way of thinking and stepping back and saying, “Okay, well, what’s really going on here? What are we dealing with in the current state?” That’s gonna put you in this really strong position to adopt things like value stream management. But value stream management to me is the act of managing the value stream. It’s not software. There are tools that are available to do that job and we’re in early days of that, but management is the act of managing value, right? It’s visualizing, it’s tracking the flow, it’s understanding the flow of value through the organization.

    So, that really, to me, starts with mapping and this baseline view. And I’ll echo everything that the other folks said—bringing in the right perspectives from the top to the bottom is really gonna make that happen. And it’s really, you know, the first opportunity where we’ve had that invitation in a very accommodating way, right? This is truly inclusive. There’s something in this for the business. There’s something in this for customers there’s something in this for technical folks, and everybody gets some value out of it. Everybody gets tons of value out of it, but it’s not just, “Yeah, you need to look at DevOps, because otherwise, this tech thing is gonna get out of control.”

    O’Hanlon: Mm-hmm.

    Pereira: There’s messaging and there’s value in it for the business, and there’s something in it for the customers, and that’s unique and that is a perspective that we’ve never had before, but is starting to come to reality.

    O’Hanlon: Alright. Well, definitely, as I said before, we’ve been having the conversations on DevOps.com about value stream management and value stream mapping, and I firmly believe that, at the very least over the next year, we’re going to be hearing a lot more about value stream management. 

    So, Steve, Garima, Siddharth—thank you, all three of you, for bringing your expertise to today’s DevOps Ambassadors Insights Panel on value stream management. I appreciate it very, very much, and I do value your expertise, so thank you, all, for joining me today. I really do appreciate it.

    Pereira: Thanks for having us. Great to see you all.

    O’Hanlon: Alright.

    Bajpai: Thank you. Bye bye.

    Pareek: Thanks, Charlene. Goodbye, everyone.

  • ServiceNow Acquires Intellibot to Gain RPA Capabilities

    ServiceNow Acquires Intellibot to Gain RPA Capabilities

    ServiceNow’s acquisition of Intellibot, a provider of a robotic process automation (RPA) platform, this week is part of a larger effort to automate the development of applications that are driving digital business transformation within enterprise IT environments.

    Earlier this month, ServiceNow unfurled a set of revamped low-code tools that both professional and so-called citizen developers could employ to build applications to modernize various business processes. However, many existing business processes are either manual or paper-based. An RPA platform enables organizations to create bots that mimic, within an application environment, the way those processes are currently managed.

    Josh Kahn, senior vice president for Creator Workflow Products at ServiceNow, said ServiceNow envisions the Intelligent RPA platform as a mechanism to automate rote processes that, for one reason or another, don’t manifest themselves via a set of application programming interfaces (APIs) that application developers can easily invoke.

    Each IT team, of course, will need to decide when it makes more sense to employ bots to digitize an existing workflow versus reengineering it using a set of custom applications infused with machine learning algorithms and natural language processing (NLP) capabilities. As digital business transformation initiatives become more urgent, the application development backlog within many enterprise IT organizations that require those capabilities is steadily increasing.

    However, in some cases, an RPA platform can reduce that backlog by surfacing a set of visual tools to automate a process that doesn’t require a lot of developer expertise to effectively digitize.

    In general, Kahn said ServiceNow is making a case for building a wide range of applications on top of the same software-as-a-service (SaaS) platform many internal IT teams already employ for IT service management (ITSM). That approach enables IT teams to streamline processes as the number of custom applications that need to be both developed and, ultimately, managed, steadily increases over time, Kahn said.

    The developers of those applications might be professionals using a low-code tool instead of writing additional code to quickly build an application, or it might be a “power user” that has enough understanding of software development best practices to build an application. Regardless of who builds the application, the rate at which applications are built using low-code tools is clearly about to accelerate.

    Kahn said the next step for ServiceNow will be to embed the Intellibot RPA capabilities in the ServiceNow platform sometime after the deal closes in the second quarter. In fact, one of the reasons ServiceNow opted to acquire Intellibot is because, as a smaller startup vendor, it is easier to incorporate their RPA platform than it would be to reengineer a much larger RPA platform. However, ServiceNow remains committed to integrating with other third-party RPA platforms that are already being employed by its customers, Kahn noted.

    Regardless of how or at what rate applications are developed, the number of core technologies required to build them continues to expand. The issue IT organizations will need to come to terms with is the degree to which they want to stitch all those tools together themselves, or rely on a platform to do it for them.

  • Languages and DevOps: To The Cloud!

    Languages and DevOps: To The Cloud!

    Cloud and DevOps are almost a match made in heaven. Cloud started truly gaining steam in the age of DevOps and Agile, and has adapted – or grown up in it – well, depending on your viewpoint. If not for the combined issues of data gravity and data (im)mobility, we might have gone even farther than we did into the cloudy world.

    While the breadth of possible cloud apps means a wide variety of languages is in use, Node rules this roost. Every conceivable language is used in cloud development – because, in the end, they’re just servers – but Node is the go-to language of today. The broad use and appeal of JavaScript make it an obvious choice for enabling full-stack development, and the pool of developers is larger than any other language today. Other languages are used, but not in the numbers that Node is. Python is the challenger, and a surprising number of PHP sites still exist, but forward-looking development aims at Node or Python. Considering the distance from the hardware, there is less call for C in cloud than there is in other areas of development, but it isn’t gone, since interfacing to the OS is also often done in C.

    Targeting cloud is not so much about language as it is DevOps tooling, though. All of the major Git companies have cloud-centric extensions (this is the core of GitOps) that raise the level of “fully automated.” Cloud environments, by their nature, provide APIs for infrastructure, and the Git vendors (and many, many others) have hooked into those APIs to make deployment an integrated part of DevOps at a level of detail that other environments (with the exception of containers) require a lot of work to match.

    The negative to cloud-centric development is lock-in, of course. Creating an entire DevOps toolchain and process that supports vendor X will not work directly with vendor Y. Tools can help with this issue, but the real solution is in our next installment – container development.

    The part of development that arguably benefits the most from cloud-centric development is testing. It is easier to spin up a test environment as-needed and spin it back down when testing is done – no need for a spare set of hardware or a stand-alone VM server; just redirect the DevOps ecosystem and spin it up, then account for data (scrubbing, copying, etc), and you’re off. Codify the steps for data, and you have a re-spinnable environment for test-on-demand.

    The part of cloud that is arguably the weakest is security. Let’s face it, far more of your infrastructure is publicly (or semi-publicly, if running in a VPC) exposed than in a data center. There are far more places to lock down, and a sometimes-bewildering number of options to secure. But the flexibility of pay-as-you-grow makes it an appealing option, and there are tools to help manage all those lockdown points. Just make sure that you’re not the weak point in the shared security model.

    You. Rock.

    Cloud makes a lot of things easier, and Node makes development of any step in the stack easier by using the same underlying language on the server that is used in browsers and in many mobile development platforms. Node “feels” more right for web development, at least because one connection, one call … even if it makes long-running processes require a bit more work.

    Whatever you choose, pay attention to how deeply invested in the cloud vendor’s proprietary interfaces you are, because, like all technical decisions, one day you or your replacement will have to get around those interfaces. And keep making the software the org needs.

  • Eclipse Foundation Creates Adoptium Working Group for OpenJDK Binaries

    Eclipse Foundation Creates Adoptium Working Group for OpenJDK Binaries

    The Eclipse Foundation announced today a working group that will establish a governance framework for an Eclipse Adoptium project that enables developers to download compatible runtimes based on OpenJDK source code.

    Members of the Adoptium working group include Alibaba Cloud, Huawei, IBM, iJUG, Karakun AG, Microsoft, New Relic and Red Hat.

    OpenJDK is a free and open-source implementation of the Java Platform, Standard Edition that is provided by Oracle. AdoptOpenJDK was created in 2017 to provide open source, high-quality Java runtimes based on OpenJDK. Since then, AdoptOpenJDK has been downloaded more than 240 million times.

    The Eclipse Foundation is now continuing that effort under an Eclipse Adoptium project that aims to become the primary source for OpenJDK-based binaries. The Eclipse Adoptium working group has negotiated with Oracle to create an OpenJDK Community TCK License Agreement (OCTLA) that will enable it to run compatibility tests to ensure conformance to the Java specification.

    Mike Milinkovich, executive director of the Eclipse Foundation, said the Adoptium WG will provide the infrastructure, marketing, community building and developer education required to ensure timely releases and adoption of Eclipse Adoptium runtimes.

    OpenJDK is a reference implementation used by multiple vendors to create their own binaries, which are only guaranteed to be compatible with the standard until the next update arrives; updates are released approximately every six months. The Eclipse Adoptium project will make it simpler for developer teams to identify which binaries are compatible with specific implementations of OpenJDK. Ultimately, the goal is to encourage developers to build applications using frameworks that make use of new OpenJDK features as they are added to the platform versus continuing to rely on older binaries.

    At the same time, however, the Eclipse Foundation is also looking at making it simpler for enterprise IT teams to rely on long term support (LTS) licensing agreements that would be based on Eclipse Adoptium runtimes, Milinkovich noted.

    Despite the plethora of programming languages developers use and even after 25 years, Java remains the dominant language for developers working in enterprise IT organizations. Over the last few years, OpenJDK has gained traction as organizations adopted an open source implementation of the specification that enabled providers of frameworks for building applications to reduce licensing costs.

    It’s not clear at what rate enterprise IT organizations are replacing proprietary editions of Java with OpenJDK. However, in the wake of the economic downturn brought on by the COVID-19 pandemic, reliance on open source software has increased to reduce costs. At the same time, many organizations are looking to build microservices-based applications using Java frameworks that are based on OpenJDK runtimes.

    Regardless of the underlying platform employed, as developers are asked to build and deploy applications faster, most of them will rely on the programming language they already know rather than take the time to master another.

  • Open Source Vs. Proprietary DataOps

    Open Source Vs. Proprietary DataOps

    Core DataOps concepts are making their way into data engineering teams and, from there, into the broader enterprise. Data engineers are retooling how they create data products, and much of this work revolves around creating data pipelines.

    DataOps pipelines offer the kind of observability that traditional data integration and ETL processes don’t or can’t. They allow you to continuously integrate and test new data sources and deliver data in streaming or batch contexts with higher levels of quality and reliability than traditional, siloed approaches. They can also support machine learning efforts by preparing data for training, testing and deployment.

    DataOps pipelines shorten the cycle time between a data consumer’s request and how quickly and reliably the data producer can fulfill it, all while observing service quality throughout its life cycle. The challenge teams face when building pipelines is how to construct them. The two approaches, as I see it, are do-it-yourself using open source, or through commercial product acquisition.

    Open source certainly has its vocal adherents and some visible successes, but there are trade-offs to taking the open source route.

    Building Or Buying Your Pipeline

    One advantage of open source is in its flexibility and availability. Open source licenses, excluding the SSPL, gives users incredible freedom over what they can do with the software. If you have the skill, you can compose a DataOps pipeline that can take any data, enrich it and route it to the right place. That flexibility, though, is also a downside. While you can do anything you want, you also have to do it. Open source projects like Kafka, Pulsar, Spark, Airflow and Flink don’t know anything about the data they’re handling. That’s up to the developer.

    This may not sound like a problem, but today’s data engineers are handling dozens of data types in hundreds – or even thousands – of different formats. If you add in operational data, you’re also looking at data flooding in from firewalls, containers, SNMP traps and HTTP sources. And that’s just what’s coming at you. You also need to fetch data from object stores, multiple activity hubs and other messaging sources. No open source project natively supports the variety and volumes of data required in a modern DataOps pipeline. Every new data source you add means starting from primordial components rather than higher-level abstractions, adding time and undifferentiated effort to your work.

    You’ll also need essential features like per-source backpressure, support for a range of protocols and data stores, role and permission management and so on. That’s a lot for overworked data engineering teams to build and maintain over the life of a pipeline.

    There’s no ‘easy button’ for proprietary products, either. Many commercial products only support a handful of sources and destinations and may have excessive licensing costs related to data ingest rates or per-connector licensing fees. Also, they can be challenging to scale, from an operations perspective. Seeing a cool drag-and-drop interface may look great during the demo, but it quickly becomes a cluttered mess when you’re looking at thousands of similar data processing pipelines.

    The advantages to commercial products center around the features enterprises need, like governance, management and security. They may also integrate with other components more readily than open source. Finally, having someone to call when you need support is also essential when running in production, although open source core companies also offer support.

    Take a Higher-Level View

    As you’re evaluating DataOps pipeline options, keep two things in mind. First, regardless of which path you choose, ensure your chosen technology offers observability and monitoring features. DataOps is about reacting to change faster, and you can’t react if you don’t know what’s going on. Building observable DataOps pipelines ensures you are delivering a valuable data product to the range of data consumers in your enterprise.

    Next, get past the technology. DataOps is more a people discipline than a technology practice. A key component of a successful DataOps implementation is aligning metrics and incentives across teams. If the staff running the DataOps pipeline are disconnected from the business outcomes their data products inform and influence, they’re not incentivized to run the pipeline as a mission-critical infrastructure component. It’s key to align all of your teams with shared metrics and incentives.

    DataOps Determination

    Before you decide on your DataOps pipeline implementation, there are some things you should do.

    • Determine who your customers are and what their expectations are around data delivery, enrichment, user experience and observability requirements. Are you only serving data scientists, or are you also serving infrastructure and operations, business intelligence and marketing? Knowing your customer allows you to make better decisions.
    • Test the software in a real-world environment. Everyone dislikes being told to perform a proof of concept before you buy/build, because POCs are time consuming. Remember – they’re less time-consuming than picking the wrong tool and looking foolish later.

    Engineering teams sometimes choose technologies that are in fashion rather than selecting the right tool for the job. Taking a pragmatic view of performance, manageability and engineering effort, as well as involving a wide array of stakeholders, will give you a higher chance of success with DataOps.

  • Oracle Updates Java Development Platform

    Oracle Updates Java Development Platform

    Oracle announced today the availability of Java 16, which adds 17 enhancements to a software development kit that is now being updated every six months.

    Georges Saab, vice president of development for the Java Platform Group at Oracle, said while most Java development teams are still using Java 8 or earlier editions of the platform, the expectation is that many of them will soon move forward, as updates are made available in smaller increments that are easier to test and vet.

    At the same time, Oracle is committing to make available long term release editions of Java that won’t be updated as frequently. While there are many organizations anxious to take advantage of new capabilities as quickly as possible, Saab noted there are just as many that value stability more than innovation.

    The latest edition adds support for finalized pattern matching for instances and records, as well as other language enhancements that were previewed in Java 14. Developers can also employ a packaging tool to ship self-contained Java applications that will make it easier to deploy in modern IT environments, said Saab.

    Additional capabilities now available include the Vector application programming interface (API), the Foreign Linker API (JEP 389), and the Foreign-Memory Access API that are all available in incubations as well as a sealed classes capability that is being made available as a preview.

    In general, Saab said, one of the primary reasons many organizations don’t tend to update Java more frequently is the frameworks they rely on to employ it at a higher level of abstraction are still optimized for older versions. However, as Java updates are delivered in a more modular fashion, it will become easier for the providers of those frameworks to keep pace.

    It’s not precisely clear to what degree IT teams are employing Java to build next-generation applications based on microservices. However, a recent survey published by VMware, found 95% of respondents have either already used the open source Spring framework, based on Java, to either containerize an application (65%) or plan to do so (30%).

    A quarter of a century after its initial development, Java is still the dominant programming language relied on to build enterprise applications. In many instances, the lack of tools for building Java applications for Kubernetes platforms was a significant inhibitor of adoption for the cloud-native platform. In the last six months, however, the number of tools for building and deploying Java applications on Kubernetes clusters has increased.

    Despite the proliferation of alternative platforms for building and deploying applications on Kubernetes platforms, the appetite for retraining legions of Java developers is low. Most developers will rely on the programming tool they already know, especially when it comes to building mission-critical applications for the enterprise. That may mean some of those applications might not be as fast as they could be, but, given the level of risk, there’s always going to be a natural tendency toward being safe rather than sorry.

  • Tackling Audit Compliance as Code

    Tackling Audit Compliance as Code

    Do the words, “It’s audit time!” make your stomach sink? If so, you’re not alone. Assisting with evidence collection for compliance audits around PCI DSS, SOC-2, ISO 27001, NIST and HITRUST is a drain on DevOps teams’ time and resources at companies of all types and sizes – time and resources that could be better spent driving innovation.

    The good news is, vendors have taken notice, and are delivering new tools and platforms that use automation to eliminate repetitive manual tasks and dramatically speed the audit process. This is entirely in line with the DevOps philosophy of tackling various IT tasks and requirements “as code” to remove tedium and improve efficiency and orchestration. In fact, as this movement to improve the compliance process picks up steam, I believe it’s entirely likely the next evolution is the use of open APIs and online communities where engineers can build and share software to automate a wide variety of evidence collection for compliance.

    But before we can get there, let’s understand where we are today and assess new opportunities going forward.

    Audit Support: DevOps’ Favorite Task

    As keepers of cloud environments, DevOps teams typically spend hundreds or thousands of hours each year gathering evidence for compliance audits. This tedious, manual labor takes significant time away from their primary job responsibilities. Large enterprises often have multiple, overlapping audits each year, with little break from the grind of evidence collection. Companies that make heavy use of the public cloud face additional challenges; the increased complexity means that evidence collection is more difficult, and the fast pace of software development and change management in the cloud (compared to more traditional monolithic application updates) makes it difficult for compliance to keep up. This contributes to an overall lack of clarity and confidence in the compliance process at the leadership level.

    Audit Prep as Code

    When I ran a cloud engineering team at a previous job, we realized the vast majority of repetitive, manual labor involved in evidence collection could be automated to save time and free up resources for other business needs. This automation also improved the completeness and accuracy of the evidence gathered, ensured that important areas weren’t overlooked and allowed for more regular assessment to prevent “drifting” out of compliance in the fast-paced cloud world. While automation helps with multiple aspects of compliance audits (such as evidence-to-control mapping and cross-walking evidence across common control frameworks), it’s the automation of evidence collection that primarily impacts DevOps.

    In this scenario, automated collectors crawl your public cloud environments and connected SaaS systems to rapidly gather technical evidence such as specifics on user access, encryption of data at rest, key management, network segmentation, firewalls, vulnerability scan reports and more. The results are transformed into formatted reports ready for internal review, analysis and auditor review/approval. This, alone, can save thousands of hours annually for typical enterprises facing multiple audits. Even better, the data is considered more complete and accurate from a compliance perspective, because it includes time stamps and other metadata to ensure capture and processing integrity.

    As mentioned, this is well aligned with the broader “X as code” mantra that underlies many DevOps automation initiatives – and why not? What could possibly be a more repetitive, manual task than capturing, renaming, organizing and describing screenshots?

    The Scripting Curse

    When confronted with evidence collection, an engineer’s first thought is usually, “I can write a script for that!” Scripting is a form of automation, but it has some limits in this scenario. First, scripting and doing the necessary QA shouldn’t be your day job; there are better things to spend time on. Second, given the explosion of cloud infrastructure and SaaS apps, there can be a lot of scripts to create. In fact, it’s arguably a never-ending task. Finally, while you understand your scripts, who’s doing the next audit? If it’s not you, are you going to explain and train someone else on all the work you’ve done? Scripting elements of evidence collection is an improvement over doing it manually, but is still only one piece of the larger puzzle.

    Compliance is a Community, not an Island

    So what’s the best way to automate evidence collection? Multiple vendors are tackling the challenge, from cloud platform providers to established GRC vendors and cloud compliance startups – and new tools have already begun to hit the market over the last few months.

    But as discussed, a primary challenge is that there is a lot of complexity in most cloud environments, and tools that approach the puzzle from only one direction – say, a particular cloud platform – still leave a lot of potentially relevant systems untouched. In fact, when you take all of the variations possible with cloud infrastructure into account, this might, ultimately, be a challenge that is best tackled at the DevOps community level.

    Consider this: adherence to industry standards for data handling, privacy and protection is not a competitive issue. We are all better off if everyone shares a basic commitment to complying with these IT security and compliance standards. Viewed from that perspective, the more we can lower the barriers to achieving this compliance, the broader the adoption will be.

    What I hope to see is a DevOps community that coalesces around automating audit readiness. At the engineering level, this could entail sharing of custom collectors. For platform and SaaS vendors, it means ensuring open APIs that allow capture and extraction of relevant data. And the compliance tool vendors must design their collection engines, evidence libraries and usage policies such that customization and sharing is possible. Similarly, automation tools must not only pull data from standard systems, but support pushing data from custom systems or legacy infrastructure. In my view, this bottoms-up approach is the best way to stop wasting DevOps’ time on audit prep, while still allowing enough autonomy to do what works best for their specific environment.

    In this future, then, perhaps an upcoming compliance audit won’t produce a collective groan from the DevOps team – or at least, generate a quieter one.

  • Measuring the Value of DevOps-as-a-Service (DaaS)

    Measuring the Value of DevOps-as-a-Service (DaaS)

    The late great W. Edwards Deming, statistician, professor, author, lecturer, and consultant said, “In God we trust. All others must bring data.” The point is, if you want people to trust your information, then you need to present measured data. This is true for countless circumstances. As a DevOps consultant, I am often asked by my clients what data should be used to show the value of their DevOps capabilities, in a way that is acceptable to the stakeholders of those DevOps capabilities. A lot of excellent research results, published by DORA and others, describes metrics that demonstrate the value that DevOps provides to application development. More and more enterprises are embracing the idea of providing DevOps capabilities, such as pipelines, tools, hardened containers and tech stacks, as a centralized service to support their many application teams. The question then becomes, “What metrics should be used to show the value of DevOps-as-a-Service (DaaS)?”

    Before measuring the value of anything, you need to know two things: “What is it?” and “Why is it important?”

    What is DevOps-as-a-Service?

    In high-level terms, DaaS is DevOps capabilities made available to the users of those capabilities through portals and APIs. There is no standard set of services; DevOps capabilities offered vary from organization to organization. In my SlideShare, DevOps-as-a-Service (DaaS) Values, I illustrate a generic blueprint for DaaS.

    The DaaS blueprint has four layers, as follows:

    • Stakeholders – There are three types of stakeholders: business, app-dev users and DaaS teams.
    • Service access layer – This layer consists of user portals, APIs, and access security.
    • Service layer – The service layer consists of user services and management services. User services include things such as repositories, CI/CD pipelines, sandboxes, test data, configuration data, guides and training. Management services include service catalogs, issues and change requests, SLOs, SLIs, governance policies, experience monitoring and automated tasks.
    • Resources layer – This layer includes application stacks, infrastructure resources, tools, infrastructure code and topology data.

    Why is DaaS Important?

    The value of DaaS, as perceived by different stakeholders, can be summarized as follows:

    • Business – DaaS makes the business competitive.
    • App-dev Users – DaaS improves application development.
    • DaaS team(s) – DaaS improves management of DevOps capabilities.

    What Service Level Objectives (SLOs) Demonstrate the Value of DaaS?

    The SLOs need to be chosen to best fit the requirements of each application. The following are examples.

    Business value can be demonstrated by:
    ROI based on tracking cost savings relative to a benchmark comparing prior non-DaaS, trend cost per developer and per release.
    SLO agility of services – Lead time for changes, and frequency of releases.
    SLO reliability of services – change failure rate, time to restore services.
    SLO governance, compliance and security scores.

    App-dev user value can be demonstrated by:
    SLO availability of DaaS, service level indicators (SLIs) depend on service components and architecture.
    SLO service response time.
    SLO experience scores.
    SLO change request response time.

    DaaS team value can be demonstrated by:
    SLO scope of applications – number of apps supported, number of users per month.
    SLO number of requests served per month.
    SLO critical events per month.

    How Does an Organization Develop DaaS Metrics?

    Start by picking an application that is supported by DaaS. Do not try to boil the ocean and measure all applications at once. Then, appoint a DaaS SLO/SLI team to work on metrics development for the application. The team should include stakeholders from business, app-dev and DaaS support teams associated with the application.

    DaaS SLO/SLI team members determine the SLOs of DaaS that each class of stakeholder values the most, and then decomposes the services to identify sources of data that will form the most important service level indicators (SLIs).

    With the SLIs in hand, instrumentation, telemetry and analysis requirements can be determined and used to design monitoring agents, communication, analyzers, alerts and dashboards.

    Then, roles, policies, playbooks and responsibilities will complete the solution. Test the solution for the chosen application. As you gain confidence in the solution, adapt and expand the solution for use with other applications.

    Continue to monitor the progress and performance of the SLO metrics, and continuously improve them with experience and as newer technologies become available.

    What This Means

    DevOps-as-a-Service provides many valuable capabilities to business, app-dev users and DaaS teams. Metrics that demonstrate value to each class of stakeholder should be determined for each application that consumes the capabilities offered by each unique DaaS. A DaaS SLO/SLI team can determine the best metrics and system design for the implementation for an example application. The SLOs can be adapted and evolved for other applications in the organization, and used as a basis for continuous improvement.

  • IBM Deepens DevOps Insights on Mainframes

    IBM Deepens DevOps Insights on Mainframes

    IBM today extended its effort to enable IT organizations that have mainframes to embrace DevOps best practices by updating the analytics tools it provides.

    Tom McPherson, vice president for IBM Z Software, said a version 6.0 update to the suite of IBM application discovery and delivery intelligence (ADDI) for IBM Z tools surfaces more granular insights by leveraging a new version of Wazi Analyze that now analyzes code immediately after a change is made. That capability is critical, because most of the workloads running on a mainframe are highly tuned. Any issue that is likely to occur can usually be traced directly back to the latest software update made to the environment, McPherson said.

     

    Wazi Analyze also can now be deployed as a set of containers, which McPherson said means IT teams can opt to run the analytics application on a platform other than the mainframe.

    ADDI also has been updated to improve Assembler support by making it easier to analyze multi-level dynamic assembler macro calls. A set of application programming interfaces (APIs) for feeding that data back to a continuous integration/continuous delivery (CI/CD) pipeline has also been added.

    Finally, IBM revealed it signed a licensing agreement with ITP Software Systeme GmbH, Munich (ITP) to enhance application analytics capabilities. That capability will be added to a future release of DevOps for IBM Z Hybrid Cloud, which includes ADDI and IBM Wazi Developer, to enable organizations to determine how best to modernize legacy monolithic applications as they transition to a microservices-based, cloud-native computing environment.

    Naturally, not every monolithic application is going to be converted into a series of microservices. However, McPherson said, it’s clear there is a need for analytics tools that will enable IT teams to more easily determine which elements of an application are better suited to being reconfigured as microservices that could be deployed on, for example, a Red Hat OpenShift cluster.

    For the last several years, IBM has been encouraging IT organizations that have mainframes to embrace DevOps best practices as part of a larger effort to position mainframes as another type of distributed computing platform. The overall goal is to lower the total cost of mainframe ownership by reducing the need to rely on specialists to manage mainframe application environments. Each year, more of those specialists retire. IBM wants to make it more feasible for organizations to replace those specialists with IT generalists, like those that typically manage distributed computing environments. At the same time, management of distributed computing environments is becoming more automated, thanks to the rise of DevOps.

    Of course, mainframe processes have been highly automated for decades. IT teams managing mainframes may still rely primarily on waterfall-based processes to deploy applications, but the underling infrastructure is designed to scale up and down on demand as application workloads require. Convincing IT teams that, for decades, have employed the same processes to manage mainframes often requires a significant amount of cultural change.

    Regardless of motivation, however, the winds of DevOps change are now blowing through mainframe environments just as fiercely as they have through the rest of IT.