Category: DevOps Practice

  • The 5 Core Principles of Lean Management

    The 5 Core Principles of Lean Management

    Lean project management is rooted in cost-saving manufacturing methodologies. The ultimate goal of lean methodology is to reduce waste. Companies in all industries have adopted lean principles into their own processes, but lean is most suited toward physical manufacturing, engineering and related industries.

    Regardless of your company’s size or the activities of your organization, lean project management principles can be extraordinarily helpful in optimizing your everyday operations.

    Below are the five core principles that guide lean thinking for project managers.

    Identify Value

    Defining the value of a given product is crucial. Before beginning any manufacturing project, or any project for that matter, the amount of value a finished product will generate must be assessed accurately. How much customers expect from a given product must also be ironed out.

    Value questions that lean teams will ask:

    • What do customers want?
    • How will they benefit from our product?
    • How much are they willing to pay for the benefits of our product?

    Lean teams can, for example, estimate how much customers are willing to pay based on related data points, then set costs low enough to reach an agreeable price. The goal is to eliminate all waste in every process. Any area where value is not added—or worse, cost is exceeding value—must be addressed early on using lean management principles.

    Map the Value Stream

    The value is the end goal of the project. Once you’ve established value as a definite endpoint, you can begin mapping the way to get there (or value stream). Every single process and every single step involved in getting to the end product is part of the value stream. From gathering the materials needed for production to coordinating efforts with PR, every related process is represented in the value stream.

    Kanban is one of the most popular ways to map value streams. Tasks are broken down and are posted in a “just-in-time” fashion. That way, each task is taken one at a time and only done when it is absolutely necessary to complete.

    Using Kanban flowcharts and other diagrams, managers can set up visual aids for value streams, mapping every single step onto a single page. By mapping the flow of activities, lean project managers can easily identify waste and inefficiencies with the process, allowing them to ameliorate these issues swiftly.

    Create Flow

    By mapping the value stream, project managers can spot waste. Any tasks that are done too early, too late or too often creates waste. By analyzing the map and removing waste, lean project managers can create the optimal flow of activities.

    Lean project managers check for possible bottlenecks and delays. If there are any inefficiencies present, tasks may have to be reordered. Sometimes, project managers may even decide to append value-adding tasks to improve processes.

    Ideally, every step should flow into the other, beginning at the factory (or organization) and arriving smoothly as an end product to the customer.

    Employ a Pull Approach

    Many organizations use a push approach, stockpiling supplies in anticipation of demand. Lean organizations, however, employ a pull approach, wherein production increases as demand increases. Customers can “pull” directly from the manufacturer, getting what they need in an on-demand fashion.

    The pull approach requires extremely accurate market research and exceptionally efficient dissemination of information within the organization. Also, all processes must be streamlined; otherwise, it will take too long to ramp up production to meet demand.

    Seek Perfection

    At its core, lean management is about continuous improvement. Lean management requires a thoughtful approach to every action. Optimization must be part of the corporate culture itself. Every member of the organization must be working toward removing waste, adding value and delivering quality. There is always room for improvement in lean organizations.

    Conclusion

    Lean organizations are removing waste. Lean project managers ensure that activities within the organization flow properly, without interruption, delays or bottlenecks.

    Identifying value means finding out what your product will do and how much customers will pay for it. Value streams are then mapped, following every step in production. Project managers ensure that tasks flow and processes run smoothly. Market data is collected during development and these data points drive production. The most important principle that guides lean thinking is to seek perfection. The lean management philosophy demands a dedication to continual improvement.

    — Ellie Martin

  • 5 Practices That Will Help Drive Successful Robotic Process Automation Deployments

    5 Practices That Will Help Drive Successful Robotic Process Automation Deployments

    Enterprises have begun to invest considerably in robotic process automation (RPA) in recent years. RPA is one of the critical technologies that enterprises now are leveraging to automate their business processes to make their businesses more agile and efficient. When properly deployed and managed, RPAs can automate everything from mundane tasks (such as organizing files) to more significant functions, such as user provisioning to resources and more.

    While RPAs do promise to help businesses move more quickly and with more agility, a company’s RPA strategy needs to be executed effectively across the organization. In my recent discussions with CIOs and others implementing RPA, the following practices repeatedly were mentioned as essential to RPA success.

    Understand the Process Being Automated and How RPA Impacts Other Operations

    One of the most common mistakes made is not thoroughly understanding the process being automated and the potential implications of automating the process. An example would include a business process that is time-sensitive and automated, which throws off the workflow timing. It’s possible the automation moves things along too quickly.

    Another example would be not taking into consideration the nature of the data being accessed. If it’s regulated or sensitive data, there are security implications that must be managed. If the bot potentially will process sensitive or regulated information, the bot should be controlled appropriately. RPA technology is an IT function, and at the very least IT and security should be made aware of and potentially involved in the oversight of the bot. It’s much better to inform them and bring them in early than them finding out later about an army of bots managing and manipulating information.

    Consider Refining the Process Before Automating

    After the process being automated is understood, evaluate the process for potential optimization. Sometimes processes that have been underway in organizations for years are more than just inefficient; they’re actually antiquated. Look for ways the process can be made more effective; consult all stakeholders when necessary or warranted; and consider refining the process if possible. There’s no sense in automating an inefficient process.

    Document, Test and Catalog Your RPAs

    After the RPA is built, make sure to document its functionality and how its code works and examine its functionality. The RPA is now ready to be used. It’s important to continually monitor the RPA for potential abuse or something gone wonky. Send all the documentation relating to the bot to the IT department.

    As the Bot Owner, Know You’re Responsible for Your Bot

    Should the bot do something unexpected or unwanted, prepare to answer for it as the bot, or business process, owner. Make sure the bot is managing risk within its acceptable scope and that it’s not given too high a level of access. While you as process owner may have administrative rights to an application, does it make sense for those rights to be transferred to the bot? Perhaps it does, and perhaps it doesn’t. It’s a risk management call and will vary from situation to situation. Just be aware that, as the owner, you also own the risk.

    Remember Your Bot May Live Beyond Your Tenure

    In one, two or five years from now you may not be in your current job position. It’s essential that the bot is constructed for this eventuality. Beyond the proper documentation and cataloging of the bot mentioned previously, put into place the appropriate identity and entitlement management of the bot. Down the road, someone may want to enhance the bot with machine learning or AI capabilities. For when this happens, it’ll be essential to have the bot well-documented as well as its entitlements well-defined.

    Consider Centralizing the RPA Deployment Process

    While some enterprises have chosen to undergo organizationwide automation efforts, others are dipping their toes in tepidly with small RPA pilots. Others still haven’t decided on how to incorporate bots within their organization, so it’s happening organically and haphazardly as individuals and departments decide to make a go of it on their own.

    If an organization is large enough, it may make good sense to put into place an RPA center of excellence. It will provide a repository of policies and strategies for RPA deployment, help catalog bot capabilities and dependencies and help drive overall RPA efficiency as the organization becomes more and more automated.

    These bots are here to stay—in fact, they are going to grow in numbers. Technology research and advisory firm Gartner expects global spending for RPA to reach $2.4 billion in 2022, up from $680 million last year. That’s an increase of 57 percent annually. RPAs help drive efficiencies and remove mundane, low-risk tasks and workflows so that humans can focus on higher value and higher cognitive tasks.

    — George V. Hulme

  • How CI/CD Can Fast-Track Next-Gen Media Rollouts

    How CI/CD Can Fast-Track Next-Gen Media Rollouts

    Using CI/CD can help digital service providers move as quickly as over-the-top content providers

    Online video consumption has been increasing tremendously with rapid change in consumer expectations to have seamless viewing experience across various digital devices. To capture this growing demand, it is critical for digital service providers (DSPs) to deliver fast-track rollout of next-gen media services. Moreover, to counter the growing influence of over-the-top content providers (OTTs), DSPs must deliver media services that are high quality, right the first time and consistent, along with creating competitive advantage through speed to market. However, unlike OTT players, DSPs are challenged by extremely long development and rollout timelines for new services and offerings, primarily because of vendor-specific hardware and software applications that do not support rapid changes.

    Figure 1: Rapidly changing landscape of consumer video consumption.

    Embracing an effective continuous integration and deployment (CI/CD) setup can enable DSPs to achieve the same service innovations and delivery agility that OTT providers are offering. An important characteristic of an effective CI/CD setup is to create a fully automated deployment that is easy to operate and visualize. The setup also should have remote accessibility to ease parallel deployment at multiple geographic locations and a robust monitoring toolchain to monitor each step in the deployment process.

    However, implementing a CI/CD setup with these characteristics for next-gen media landscape comes with its own challenges; it requires end-to-end CI/CD orchestration, which is operationally complex. This involves stitching together different types of video services such as live TV streaming, VoD, electronic program guide (EPG), 4K/HDR, user personalization and profiling. These services need to be rolled out in multi-geographic locations and multi-tenant environments, catering to multiple vendors. The service must offer a seamless viewing experience across all digital platforms including TV, mobile, tablet and desktop.

    In setting up such complex processes, cross-functional and cross-geographic teams own different parts of the process, which can create gaps in design, infrastructure setup, configurations and eventually deployment. The CI/CD pipeline created can be costly and ineffective.

    Setting Up Effective CI/CD

    One-click automated delivery pipeline tool for predictable rollouts/rollbacks

    Create a one-click automated delivery pipeline tool that can facilitate automated change requests (CRQ) to initiate CI/CD jobs. The tool triggers the entire deployment of all the app components in the respective tenants across multiple geographies.

    Key considerations in setting up a one-click automated delivery pipeline:

    • Track CRQ approvals over systems such as JIRA to trigger the automated delivery pipeline.
    • Build pipeline as a code with intermediate CI/CD stages using open source tools such as Jenkins.
    • Design a master job to accept component and configuration details. This performs multi-job configuration and triggers contextual deployment/rollback simultaneously across tenants.
    • Interconnect different aspects of networks, infrastructure, components and applications to run the entire process smoothly.

    Setting up an automated delivery pipeline provides greater accuracy and predictability by reducing the human error factor.

    Automated snapshots, pre- and post-validation scripts to ensure consistency

    Create a fully automated snapshot management process that includes automated snapshot creation, rollback and deletion across respective tenants. Automated snapshots before deploying the new version of application facilitates rollback to the previous consistent version with zero downtime, if any failure occurs.

    Ensure automated multiple validation checks at every deployment stage to ensure consistency. These checks include validation of network connectivity, memory usage, CPU utilization, running applications, error and debug log files, license files, binary file size and configuration files. These validations can be done via tools such as groovy, ansible, Jenkins, python or shell scripts with proper libraries that match industry standards.

    Auto-scaling tool to optimize resource usage

    Ensure the application always has the right amount of capacity to handle traffic demands. This requires integrating an auto-scaling tool with predictive machine learning (ML) models and change request process to dynamically scale up/down the infrastructure.

    Key considerations in setting up auto-scaling tool:

    • Identify external and internal influencing factors to analyze system throughput.
    • Leverage insights derived from a monitoring dashboard to manage changes in current payloads.
    • Leverage triggers from a ML model to plan future expected system throughput.
    • Create automated configurable scripts to support dynamic scaling based on the inputs received from change request and ML model.

    Creating such tool enables better infrastructure availability with efficient cost management.

    Tool-agnostic and on-demand environment creation framework built on open source

    Leveraging open source tools provide flexibility and control over the entire exchange-to-exchange deployment process. Using customized framework with a layered approach makes the delivery pipeline tool-agnostic. It gives flexibility to easily replace/plug in any tools in the DevOps chain without much dependency.

    This ensures increased reusability, as the same toolchain can be used for multiple environment setups. Further, a customized framework dynamically handles the resources and utilizes the infrastructure effectively.

    Service assurance dashboard for stability and monitoring end-to-end processes

    A service assurance dashboard can provide glimpse of end-to-end CI/CD performance in a single window. It provides real-time graphical insight into failures/blockages in the pipeline, highlighted in the context of the overall flow. The dashboard should dynamically fetch data from the deployment pipeline for respective components and monitor how the upgrade/rollouts are processing. It should provide insight into resource usage, which can trigger dynamic auto-scaling and give a visual comparison of previous firmware with the new one.

    Implementing such a service assurance dashboard can provide greater visibility and help in easy diagnosis and error prevention.

    Benefits

    • Accelerated time to market with nearly 60 percent reduction in waiting time with parallel deployment across multiple tenants.
    • Improved predictability with significantly reduced fallouts and downtime/failure rates.
    • Optimized cost with approximately 50 percent lower IT spends in the deployment process and resource usage.

    This article was co-authored by Sumit Thakur, manager, strategic insights at Prodapt; Vijay Ramachandran, senior project manager at Prodapt.

    — Lakshmiprasad A

  • DevOps Chat: Doing Agile, with AgileCXO’s Jeff Dalton

    DevOps Chat: Doing Agile, with AgileCXO’s Jeff Dalton

    Agile has a bit of a longer history and subsequent maturity than DevOps. Despite this, how can an organization say that it is in fact an “agile organization”? What does that even mean?

    Our guest on this DevOps Chat is Jeff Dalton, founder of Broadsword consulting, author of “Great Big Agile” and founder of AgileCXO. Jeff shares with us his latest venture, which is building AgileCXO into a worldwide movement to certify that organizations are, in fact, practicing Agile with best practices. Have a listen.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hello, everyone, it’s Alan Shimel, devopsy.kinsta.cloud. You’re listening to another DevOps Chat. Today’s DevOps Chat is, I think, gonna be a good one. We have with me author, founder Jeff Dalton of the AgileCxO. Jeff, welcome.

    Jeff Dalton: Good morning, Alan. Thank you for having me, sir.

    Shimel: Absolutely. My pleasure having you. Talking off mic, we both lamenting freezing down here in Florida, but, you know, first world problems. It’s gotten into the 50s; we’re breaking out the cold weather gear.

    Dalton: Yeah. Right?

    Shimel: There was a time in my life where, if you told me it was 50s in January, February, I’d be out running around.

    Dalton: Go to the beach?

    Shimel: Yeah, pretty much. But, anyway, such is life when you move to Florida, Jeff. But let’s talk a little bit about the AgileCxO. First of all, let’s assume some folks in our audience are not familiar at all with it. Talk to us. What is it?

    Dalton: Sure. AgileCxO.org is an organization that we started about two years ago and it was based on the notion that leaders really struggle with how to successfully create Agile culture in their organization and it was based on a series of about 200 assessments we did of Agile organizations to try to really figure out why it wasn’t taking hold in the way leaders wanted it to. And the conclusion we came to was really simple, really, is that Agile leaders weren’t being Agile themselves and they really didn’t know how to be. They need help with that, so there’s lot of reasons for that and we talk about that in the book, “Great Big Agile,” but that’s how we started about two years ago.

    And, since then, we’ve developed a series of models for performance and for evaluation and a series of training courses. We have partners in China and Japan and India and the US now, delivering these services to their customers, so we’re pretty excited about it, and, now that the book’s come out, we’re even more excited about it.

    Shimel: Absolutely. So, Jeff, for some of our folks out there, they hear people say, “I wrote a book,” right? And you know what? I’m blessed to have a lot of friends who’ve written books; I’ve contributed to several books. And people say, “Oh, yeah, that guy wrote a book.” They don’t really understand what goes into writing a book, though. I mean, right? I mean, it’s one thing to write a blog article or something like that. Writing a real book is work, though.

    Dalton: It sure is.

    Shimel: Share with our audience a little bit, what was the effort required here _____ _____ —

    Dalton: That’s a great question. I’ve never been asked that, but it’s funny. I started my book with the—you know, you have the acknowledgements in the front of it and the very first line was “Writing a book is much harder than I thought it was.” [Chuckles]

    Shimel: Yeah, it is. No, it is. I mean, there’s no doubt.

    Dalton: You know, I’d written a lot of articles, lot of long articles. You know, I write a lot for Cutter IT Business Journal and for Better Software magazine and per week. And I’ve written quite a few articles in my life and I’ve written quite a lot for e-books, but what I really wasn’t prepared for was the sheer volume and effort associated with telling the story in completeness. You know, it’s one thing to kick one idea out in a blog post or to kick a series of ideas out and explain ’em, but to really explain a system of ideas and provide supporting information and research and really tell a convincing story, from start to finish, is a massive effort that I didn’t even do alone. I had help. I had—you know, my wife’s an editor and she really helped out, and I had other people on our team really help, providing me with information and reviews.

    And that doesn’t even consider the publisher, who kept coming back with demands for more information and better clarification and really kept me on track there. And then there’s artwork and covers and indices. And I know, at the very end, they’re like, “Oh, you need a glossary”; I’m like, “Oh, okay.” So it’s a massive, massive effort, but it’s also a really cool effort because what I’ve learned from writing the book, one, is that you don’t really know what you’re talking about until you’re forced to flesh it out in writing.

    Shimel: Yeah. For sure.

    Dalton: And it’s really helped me working with my customers and clients because I’m able to really articulate what I think. And this is important because the book is about a new idea that people aren’t talking about that much. And a lot of people are talking about Agile—a lot of people—and there’s a lot of fantastic books about Agile. We didn’t wanna write another one. We thought that Ken Schwaber’s book on Scrum is beautiful and doesn’t need much elaboration and there’s books on DevOps that are very well done, and we didn’t really wanna write another one of those. I wanted to write about something new, so thinking through that entire story required a lot of effort and time and much more than I expected, but it finally comes together. At some point, you get in a groove and then it just becomes a mission. And we completed that mission in October, so.

    Shimel: Good for you—

    Dalton: A book came out the other end, so it’s pretty exciting stuff.

    Shimel: Absolutely. But, you know, what’s kinda different in your case, Jeff, is the book really serves as the bible, the how-to, the map of this whole AgileCxO, as we were talking about. Agile in a vertical, I think, is how you mentioned it. Agile in a vertical way, not the usual horizontal, if you will.

    Dalton: Right. Yeah, one of the things I talk about in the book early on is really trying to position the different industry models a little bit more succinctly because we hear about process models or behavioral models and frameworks, and there’s a lot of ambiguity in the industry about the differences between these things. “What’s the difference between Scrum and Agile?” is something you hear people talk about a lot and “What’s the difference between ISO and CMMI and PMBOK and all these different techniques and models out there?” And we categorize these things into three different categories in the book: what we call “why-ability” models—why are we doing stuff?; “what-ability” models—what am I supposed to do?; and “how-ability” models—how am I supposed to behave?

    So you take something like an ISO or a CMMI or a PMBOK, these are squarely what-ability models—you must do these things; there’s a list of things that you’re supposed to do—but gives very little guidance on how to do it, what kind of behaviors are necessary. And a lot of the Agile frameworks are really great why-ability models—you know, “These are Agile values and here’s why you should do these things”—and they’re very light on the “what” because they wanna be more empirical and they wanna have teams learn and improve and change, as it makes sense. But there isn’t a lot of detail for a leader to say, “How do I make an Agile organization?”

    So we coupled that with the notion that scaling Agile is really about scaling vertically, from teams up to senior management, not so much scaling horizontally, which is what we see with things like SAFe or LeSS. These are all great models and they’re necessary models, but the conversation’s really been focused on horizontal scalability, but nobody’s really talking about vertical scalability and the changing of leadership. Though we talk about being change agents and changing companies, when we do Agile transformations, but what we’re not doing so well is changing leadership.

    Shimel: No.

    Dalton: And that’s the primary impediment. I mean, if you look at any major study, like the VersionOne study—people talk about that one a lot—they’ve been saying for years, they’ve been raising this flag for years, that leadership is an impediment. And Cutter IT Business Journal just did a whole series of articles—I wrote one of them—about the problems with Agile leadership, so nobody’s really been addressing it, other than in the trade magazines. So we decided to write a book about it, and we’ve had a lot of experience. I spent a lot of years at Ernst & Young, doing leadership coaching, and we’ve had a lot of experience working with senior management, so we said, “Hey, they need help understanding this,” and so we wrote a how-to book for them. And that’s what it is. It’s called Great Big Agile and it’s actually filled with lots of pictures and it’s in color and it’s really designed for a senior manager to use as a reference.

    Shimel: Fantastic. And we’ll get into where the book may be available, how some of our listeners listening in, if they wanna grab a copy, where they can, but I think the other thing I wanna convey to the audience, Jeff, is that this book doesn’t exist in a vacuum. It exists—

    Dalton: Right.

    Shimel: Right? It exists within the larger universe, as you mentioned, of other Agile books and DevOps books and a lot of, as you call them, these horizontal frameworks and so forth, but the book itself has kinda served as, again, a blueprint for an entire school of thought, if you will, that is now taking shape worldwide, with people who are offering training and coming in and assessing organizations and helping them to achieve this type of vision, right?

    Dalton: Right. Well, you know, consultants, especially management consultants, have, for years, been talking about leadership reengineering and reengineering the organization. If you’re reading anything from Peter Drucker or any of those guys, they’ve been talking about it for years. And I remember, the decades I was with Ernst & Young, years ago, we always talked about this and we always did engagements, working with customers, to help improve leadership performance. What we didn’t have is a model that describes how to do it. We sat down with leaders and we said, “Well, who do you wanna be?” and, “I’m a smart guy. Let me help you.”

    In a lot of ways, we need the same types of models and assessment and training that we’ve been championing for lower-level teams—Scrum certification, for instance, SAFe certification, DevOps cert and training. These things are considered normal coursework and certifications for the average worker now. We don’t have anything similar for leadership, and that’s what this model is all about, is giving leaders a chance to really learn and have a how-to model for how to improve performance and to get their whole organization certified so that they can be known as a really progressive Agile organization.

    Shimel: And we spoke off mic, Jeff—you know I’m one of the co-founders of the DevOps Institute—so I have more than a little bit of experience and insight into what it takes to have sort of a worldwide network of people who are offering training, education, assessments, based upon, you wanna call it, “curriculum” or “knowledge,” that you’ve sorta codified, if you will. That’s a big job, right? It’s more than just Jeff Dalton.

    Dalton: Right.

    Shimel: And so share with us, if you can, a little bit, what’s the organization like?

    Dalton: Sure. Well, we have a team. This didn’t start from scratch. I own another company called Broadsword that’s been around for almost 20 years and we’ve been reasonably successful in the performance improvement market. We work with various models, including Agile frameworks but also CMMI and ISO and PMBOK and Scrum and XP and SAFe. And we work with all of those models and we work with companies—largely, federal government contractors of pretty good size—so we have a lot of experience working with Fortune 500 companies all over the world. We have customers in China and Hong Kong and India and Japan and the US and then some in Europe and South America. We’ve been doing this pretty successfully for about 20 years and we’ve built a large network of CEOs and leaders, especially CIOs and CTOs, of organizations all over the world. And so that’s where we started, and so Broadsword has 15 people that have been focusing on other models and other training, by other organizations, for years.

    And so what happened is is we were doing some assessments of couple of really large companies, two years ago, and we realized that the real impediment was this whole notion of vertical traceability, we call it, or vertical expansion of agility from teams up to senior management. And we started noting that with a lot of our clients. And so we said, “There’s something needed here. The industry really needs help with this, so let’s create a model that we can use with our clients.” Then we started getting calls from other companies, saying, “Can we use your model?” and other consultants around the world, and so we started—so it started organically, this network of a couple of hundred CEOs and CTOs and CIOs, which is where we came up with the name of the organization—“CXO.”

    And so it really happened organically because of demand, so what we did is we shifted about half of those folks over to the organization to work on that and we hired a few folks. We have a chief operating officer; we have a CTO; we have a partner manager; we have model developers. So we have a team of folks that are really dedicated to this, to make it work, and we’ve outsourced things like marketing and bookkeeping and all that kind of stuff, so we can focus on the important things. But it’s a group of really senior, very experienced people, so we’re really pushing to try to make it happen.

    Shimel: Yep. Hey, Jeff, we’re starting to run short on time, so I wanna start putting a bow around some of these things.

    Dalton: Sure, yeah.

    Shimel: First of all, people listening out there—and our audience is pretty diverse, worldwide, but can you give me either the one big reason, or the top three reasons, why AgileCxO is something they really need to take a look at?

    Dalton: Sure. Well, the biggest reason is that agility and self-organization are unavoidable—they’re coming—and it has nothing to do with Agile. Interestingly, Agile and DevOps are a reaction to a worldwide trend. If you take a look at the Gallup study that’s done every year on people’s faith in leadership and institutions, what clearly is being defined in their study—and it’s a worldwide study—is that people are losing faith in traditional leadership. They’re losing faith in the kind of leadership that Harvard and Stanford are pumping out every day; they’re more interested in self-determination and self-organization.

    This is happening in banking, education, software development—the military has even suffered from this or if “suffering” is the right word. All kinds of industries, universally, are moving, more and more and faster and faster, towards this notion of self-determination and self-organization. It’s a worldwide trend. And you see a lot of chaos in the world and that’s really what it’s about—it’s about a massive upheaval in the traditional norms of leadership.

    And so it’s inevitable that, if you wanna maintain young performers that are of high quality and you wanna have a successful organization long term, that you’re gonna need to adopt Agile culture, top to bottom, in your organization. So you’re gonna have to completely reengineer the way you run your companies, or else you’re just not gonna be able to attract talented workers. So that’s just inevitable. The data tells us that that’s coming, so that’d be the number one reason to look at this.

    The second reason is that leaders are really struggling with understanding what Agile means, what self-organization means and how to actually achieve it. And they’re getting a lot of pushback from the lower levels of their organization to be more Agile, but, if you’ve ever worked on an Agile or a DevOps team, you know that, the further upstream you push, the harder the swimming gets. There’s a reason for that—we call that an “organizational type mismatch,” where I’ve got a command-and-control infrastructure, from CEO down to basically supervisor level, in an organization, and then I’ve got all these people saying, “I wanna self-organize and be more Agile.”

    So that clash of cultures is extremely energy-dependent—in other words, burns a lot of energy in your company—to the point that your costs are overcoming your profits. So that would be the second reason to do it, is it’s a how-to guide for how to avoid that cultural, organizational type mismatch.

    And, finally, I think the third reason is that it helps employees be a lot happier, and happy employees are more productive and happy employees stick around longer and happy employees, team members, are happier to work there and create less friction, less resistance. So it’s good for everybody. So all good reasons to check out the book, to check out AgileCxO. And if you are a consulting firm or a company that really wants to help companies achieve these three goals, become a partner and join us and join us on our mission. We’d love to have you.

    Shimel: Very cool. Jeff, last part—and it really segues off what you just said—where can people get more information?

    Dalton: Well, two places that I would recommend you do that. One, you can buy the book on discount at Amazon, just “Great Big Agile”—go to amazon.com. You can also buy it from any other book retailer, if you like Barnes & Noble better, or from Apress—apress.com is who distributes it for the publisher—but Amazon is probably everybody’s go-to for where to get the book, so I’d definitely take a look and read at that.

    And then agilecxo.org. If you go to agilecxo.org, you’ve gonna have at your fingertips samples of the model, lists of the training, videos to watch. We do our own podcast, similar to yours, Alan—the Agile Leadership Podcast. And what we do there is we interview CIOs and CTOs of state and government, primarily, around the country, so we’ve got Texas and Kentucky and Florida and Michigan and Wisconsin and CIOs from really large, 2,000-, 3,000-person IT organizations that are trying to adopt Agile. And we get their perspective and the things they’re struggling with at the Agile Leadership Podcast. So all that’s available at agilecxo.org and feel free to stop by.

    Shimel: Excellent. Jeff, we’re about out of time. First of all, thank you very much for sharing with us today the insights and more about AgileCxO. Continued success with it. We’ll watch it go and maybe we have you back on the show, couple months, and see where things are.

    Dalton: Thanks. I would appreciate that. And, while we’re talking about that, Alan, we’d love to have you on our “Agile Leadership Podcast.”

    Shimel: Any time. Just let me know when and a dial-in or log-on and I’m there. Hey, stay warm down there.

    Dalton: I’m doing my best. I think I may have one degree on you down here.

    Shimel: Yeah, maybe. But it’ll warm up soon enough and I’ll have to get—you know, I like _____—usually, I go to Islamorada fishing.

    Dalton: Right. Yeah.

    Shimel: Key West partying and Marathon somewhere in between, so—

    Dalton: _____ not gonna lie, we do a lot of fishing around here.

    Shimel: Oh, I’m sure you do; it’s good fishing. Anyway, Jeff Dalton, the AgileCxO organization and author, thanks for being our guest here today on DevOps Chat. And this is Alan Shimel for devopsy.kinsta.cloud and DevOps Chat. Have a great day, everyone. We’ll catch you soon on the next DevOps Chat.

    — Alan Shimel

  • Query Your Network Like a Database

    Query Your Network Like a Database

    Network process automation is an often-overlooked component of DevOps. Incorporating DevOps philosophies toward networking, or DevNetOps, can lead to even greater process improvements and more flexible deployments. Fortunately, it’s becoming even easier to automate network tasks with evolving standards and open data models that are allowing more programmatic access and insight to the network infrastructure. The most dramatic shift, however, is now being able to write queries about your network devices and topology as though you were querying a database!

    When rapidly deploying application updates, network flexibility is key. Policy changes or configuration updates are often required. Network operators can best serve DevOps processes when they can programmatically check the status and/or repair key issues from these changes. Here are some example queries that are commonly useful across the entire network infrastructure:

    • Do all distribution layer access links in my network have redundant paths?
    • Are all Border Gateway Protocol (BGP) sessions currently established with configured peers?
    • What are the nearest neighbors of a down device?
    • Are any device interfaces intended to be operational currently down anywhere in my network?

    Note that we are moving away from looking at individual devices, one at a time, to querying about network issues and status across all devices in one simple query. The challenge has been to create a structured data model of network details that was normalized across all vendors and device types that single programmatic query could leverage. The goal is to be able to query about the status of an interface port on a Juniper switch as well as a Check Point firewall, for example. Converting the detailed text files that configure each device into such a normalized data model is becoming easier with established standards, such as OpenConfig. OpenConfig has defined a common syntax and schema for network-oriented applications across vendors and applications.

    Accessing such a network schema is now extremely efficient with the widespread adoption of the GraphQL query language. Network data models are going to be graphical data structures by design, and GraphQL is ideal for building flexible data queries in minutes to such schemas. GraphQL was developed by Facebook and made an open source project in 2015. It has now been adopted by hundreds of development-minded organizations as a more efficient API model, regardless of how the underlying schema is structured.

    GraphQL is already being used widely in DevOps organizations aside from network-oriented applications. GraphQL is extremely intuitive and queries look like a JSON object without the data included. Queries simply return a JSON object with the specifically requested data. From there, it is a small matter of a few lines of code to embed GraphQL queries into a scripting language (such as Python) to analyze, process or format the results. If you aren’t using GraphQL in your DevOps coding, you should be!

    The full process looks like the diagram below (Figure 1), where text files can be parsed and normalized into an open graphical database schema that represents the current network state. The GraphQL queries can be constructed, along with a Python script (Figure 2) to format results, as shown in the third column. The following example returns both the desired admin state of every interface port in the network, along with the current operational state (up or down). The Python script compares the two states and alerts on any discrepancies.

    Figure 1

    In the example query results from Figure 1, consider that the alerts are reported from both an F5 load balancer (atl-bl01) and a Juniper router (sjc-ce01). Writing code that could parse such disparate device types and text files—without a normalized database schema, based on something such as OpenConfig—would be extremely complex. However, a simple GraphQL query can apply across the entire network. The main portion of the Python code that was used to embed the above query, compare the results and display the table, is shown below:

    Figure 2

    Network operations are becoming an increasingly important part of DevOps processes. Automating network tasks is greatly simplified when you can view your network infrastructure as a database. Embedding queries that confirm network policies and health checks into Python scripts can be trivial when leveraging open data models and query languages such as GraphQL. This will enable a new convergence of agile development and network operations for the first time.

    — David Erickson

  • Chef Achieves Multiple Compliance Mandates

    Chef Achieves Multiple Compliance Mandates

    Chef today announced it will make it easier for organizations that adopt its IT automation framework to stay compliant with mandates such as the Federal Information Processing Standards (FIPS), Secure Technical Implementation Guidelines (STIG) and the Center for Internet Security (CIS) certification.

    John Snow, senior software development engineer and federal content lead for solutions engineering at Chef, said it’s incumbent on the provider of an IT platform or application to make sure their offerings comply with widely required mandates. FIPS compliance, for example, not only applies to federal agencies, but also any organizations that interacts with those agencies.

    Specifically, Chef said it has achieved FIPS compliance along with providing an implementation of STIG profiles for RHEL 7 and Windows Server 2016 in Chef InSpec and achieving a (CIS) certification for Amazon Web Services (AWS) Foundations Benchmarks Level 1 and 2 in Chef Automate. Chef claimed it is the first vendor to achieve CIS certification across AWS, Microsoft Azure and Google Cloud Platform.

    Snow said Chef was able to achieve that goal in part because of the time and effort it put into developing a parser that enabled the company to approach compliance as a programmable extension of its development process. That translation process is required because most compliance mandates are written for auditors by auditors, not by IT operations teams. Chef will also leverage that capability to ensure its platforms remain current with any future updates to the compliance mandates its supports, said Snow.

    Chef has been making a case for treating compliance as code for several years now. Any given compliance mandate typically consists of a set of controls that need to first be put in place and then tested and validated. Rather than trying to achieve that goal relying on manual processes that don’t scale, Chef contends achieving compliance should become a natural extension of best-in-class DevOps processes.

    It’s not clear yet to what degree compliance may become integrated within DevOps. But recent advances in DevSecOps suggest it’s only a matter of time before responsibility for implementing compliance controls “shifts left” along with security control. Compliance teams may define what controls will need to be put in place. But responsibility for implementing those controls as part of any quality assurance process is moving toward DevOps teams. Developers, however, will take on that responsibility only when a means to programmatically implement those controls is readily available.

    The long-term economic implications of that shift may prove to be profound. Organizations of all sizes incur massive costs both implementing compliance controls and having them audited by a third party. As the compliance process becomes more automated, not only does it become easier to implement controls at scale, but the documentation required to pass an audit is generated automatically. Given that most auditors charge per hour for their services, the savings derived from limiting the time and effort associated with preparing for and then passing an audit could be substantial. The challenge and opportunity is educating the DevOps team on not just how much money might be saved for the company, but also how much of that money might be reinvested in advancing DevOps adoption.

    — Mike Vizard

  • API Security: A Key Part of the Bigger Plan

    API Security: A Key Part of the Bigger Plan

    Standing as the “fuel” powering the customer-driven platform revolution, application programming interfaces (APIs) are the new “it thing” amongst operating systems.

    APIs are responsible for how apps communicate with each other and have become key components in many digital transformation strategies. Serving as a set of tools for building software application, they have empowered many businesses to increase their value. However, as they have risen in popularity, concerns have risen over the adequacy of measures being put in place to ensure security for businesses using APIs.

    The Open Web Application Security Project (OWASP) is a global non-profit organization committed to improving application security and makes it a primary mission to ensure software security is a priority for all. OWASP regularly issues a top 10 standard set of rules that are applied to security policies across various platforms. The most recent list presents notable changes that reflect the increase in dependency on APIs, with a focus on how they can both help and hinder security efforts.

    Why Should Businesses Care About API Security?

    Every application needs APIs. It is difficult to find an organization today that doesn’t have its own app or uses an app to complete critical business activities. They contain programming standards, instructions and protocols that allow applications to communicate with one another. Essentially, APIs serve as a bridge that ensures proper and consistent communication between two systems.

    API security is a critical part of API development because millions of applications communicate back and forth daily. With the cyberthreat landscape constantly evolving and the cloud API market predicted to experience a 20 percent CAGR by 2022, it is easy to understand why it is so imperative that companies develop and continually implement thorough API security measures.

    Most APIs do have some governance from management tools, which can offer a range of control features aimed at making APIs more reliable, secure and controllable. However, security should also be top of demand in the design phase of an API and formulating controls that can be implemented across the organization.

    Arming Your Organization with the Right Security Tools

    Systems that host public APIs feel the impact of heavy traffic on a daily. Some of that influx is, unfortunately, not authentic. To avoid overburdening the legitimate users and protect their system, companies need to evaluate behavior to flag potential problematic traffic.

    API security works in the same manner by using rules and algorithms that evaluate client sessions. APIs ask simple questions that gauge how clients are behaving, what they are doing and whether there are unusual error rates or repetitive behavior in short time periods.

    Machine learning is often used to answer these questions and to identify and deter malicious API client practices. Standard web approaches typically don’t function properly with APIs, simply because hackers work to continuously be ahead of the game. They know that standard DoS attacks don’t work, so they distribute hacking attempts across bots that hide alongside legitimate traffic to sneak them through the system, undetected.

    To find these bad eggs and stay on top of the large amount of them, it’s necessary to set up a machine learning-based system that have a deep understanding of API traffic, with a thorough understanding of API keys, access tokens and what the typical request context is on any payload.

    How Can You Prevent Attacks Right Now?

    There are several actions you can take immediately to protect your system from potentially damaging attacks. Begin by using HTTPS, if you’re not already, to ensure proper authentication and authorization. Also, adjust your software development life cycle (SDLC) to include rigorous API security testing and validation with a primary focus on input validation.

    Also important: Make sure all network servers are running on regularly patched OS versions that are stable and include carefully configured security groups. Implementing role-based access control and VPC isolation across environments also can help improve responses to prevent attacks.

    Finally, having a pre-documented response policy for security incidents will help your company establish a secure development environment that is properly maintained at all times.

    Routine API audits and testing can help your organization continually improve the API development and ensure your business is protected against DDoS attacks and malicious bots, while still letting legitimate API traffic through. These techniques will help you validate API requests and sort through which are legitimate and which are not, to completely eliminate or reduce API attacks.

    — Eltjo Hofstee

  • Dev and Ops: Not from Different Planets

    Dev and Ops: Not from Different Planets

    At a conference, I heard a speaker remark that “Dev people are from Mars while Ops people are from Venus.” That’s catchy and captures what a fair number of managers believe instinctively. But is it really true? I’m not convinced.

    We all know that DevOps enables us to deliver software faster, with fewer bugs and much less downtime than traditional approaches. But DevOps is new, complex and often challenging for teams to embrace. Developers often guard their tools and operations shake their head each time they hear about some new asset bundling package they have to support. I think the first few meetings of Dev and Ops probably do a lot to build the notion that they are from different planets.

    In my experience, however, the opposite is true. To borrow a different popular catchphrase, they are more “brothers (or sisters) from a different mother.” When you examine the makeup of Dev and Ops people, you find more similarities than differences. Here are some examples of their common ground:

    • Consistency and Standards. Both Dev and Ops are technical endeavors that place a huge premium on things being “precisely” correct. Whether coding a service or setting up an AWS instance, “in the ballpark” doesn’t cut it. For this reason, both camps try to establish standards for how they work. Whether keeping Dev teams all using the same environment or Ops trying to keep Dev, testing and products all running the same, both need to benefit from building consistent environments.
    • Coding Skills. Whether coding or scripting, Devs and modern Ops teams need to be comfortable in a variety of highly technical tools. Build scripts, deployment, automated testing and reporting all require some coding skill.
    • Common Tools. If you look at the DevOps ecosystem, you’ll increasingly see tools that are intended for both Dev and Ops tasks. Dev people, for example, use Git for reusable code; Ops uses Git as well for configuration of things such as Docker. Also, Dev uses Jira to track bugs while Ops uses Jira to track tasks.
    • Shared Metrics. While their focus may be different (code versus pipeline), both Dev and Ops share many metrics of success—consistent delivery, fewer errors, responsiveness to customers and uptime, to name a few.
    • Test-Driven. Dev and Ops both embrace the “fail fast” mantra. They want to know if a change will break anything as soon as possible, long before it ever impacts a customer. Both often start with automated tests, which are run on Dev or local instances, to verify the changes are non-breaking.
    • Love of Automation. Neither Dev nor Ops enjoy manual, error-prone work. Both gravitate to tools that automate routine tasks, reducing variances and errors.
    • Star Wars. Finally, both groups are avid “Star Wars” fans. Just watch how they open automatic doors using the force or name servers or other things with sly “Star Wars” references (such as “Secret Rebel Base”). In fact, next time you’re in a DevOps meeting and somebody is wondering if Dev and Ops can get along, bring up Jar Jar Binks and marvel at the immediate—there will be a completely concordant reaction from both sides.

    So, are Dev and Ops from different planets? I don’t think so. I see Dev and Ops as being like those twins separated at birth, who, when reunited decades later, laugh the same, have the same twitches and both married spouses with the same first name. And in the best DevOps shops I have seen, that’s a really good thing.

    — Joshua Platt

  • The Power of a Game: What I Learned from Facilitating The Phoenix Project Simulation

    The Power of a Game: What I Learned from Facilitating The Phoenix Project Simulation

    I recently had the opportunity to become certified to lead a simulation based on the best-selling IT book, “The Phoenix Project.” The Phoenix Project simulation is not a product as much as it is an experience. The realistic scenarios, challenges and pain points embedded in the game built a creative tension in a safe environment and allowed everyone (including me) to discover new insights into applying DevOps to improve flow throughout the value stream—from the business request through IT and to customers.

    DevOps and Agile practices are built on the back of Lean principles that go back decades before the Toyota Production System. As someone who attempted to apply Lean to IT long before DevOps came into existence, I have always said that DevOps is about creating an environment where it is safe to observe, experiment and learn based on outcomes. I was fortunate to meet with Gene Kim many times while he was writing “The Phoenix Project.” We always talked about Lean thinking, Lean systems and Lean tools. Those discussions eventually morphed into The Three Ways—the core principles of DevOps.

    Alignment with the Business Changes the Feel of Work

    The simulation put people in a position where it was easy to see and feel the impact of not being aligned with business and not being coordinated with all the elements of it (AppDev, engineering, IT Ops, change management, etc.). In each round, the team was able to reflect and learn from the experience and experiment with new ways of working. What we all experienced was profound: The DevOps principles, systems and tools are effective only when the team directly experiences the frustration of a broken work system and they work together to solve the problem.

    It was amazing to see how quickly people took on the personas of the functional role they had been assigned. The woman who took on the role of retail operations—you may remember Sarah in “The Phoenix Project”—became rather aggressive as she demanded results. The CISO—John in the book—constantly nagged people about SOX-404 issues. And of course, the CEO was a royal pain to everyone. Although we all knew this was a simulation game, everyone reported the stress and tension they felt as we embarked on round one with no real plan of how we were going to transform into a DevOps team.

    You Can’t Improve What You Can’t See

    Among the chaos of trying to get the work done, it became instantly apparent that the team had no visibility of all the work or priorities. True to life, there was more work that our people could handle so we needed a way to see the work, compare it to our capacity to do the work, identify trade-offs and then work with the business and other stakeholders to set priorities. Once the team was able to accomplish this (through some trial and error) we all could feel the alignment of purpose, see the smiles on people’s faces as we created value and feel what collaboration between silos is really like.

    This discovery was just the tip of the iceberg. As a coach and facilitator, I was able to ask the team questions that positioned them to reflect on what had just happened and see within themselves additional changes that needed to take place to improve the quality, speed and volume of our deployments. Some of the other topics, which were experienced and later expanded on, include:

    • A recipe for coordination among silos.
    • Making time for improvement, learning and technical debt.
    • DevOps without Value Stream Mapping can be hazardous to your health.

    At the end of the day, DevOps is about creating an environment where it is safe to observe, experiment and learn based on outcomes. The Phoenix Project Simulation creates an environment to give everyone that experience. From there, they return to work and begin their journey.

    — Mike Orzen

  • LightStep Unfurls Dynamic Tracing Service

    LightStep Unfurls Dynamic Tracing Service

    LightStep announced a stand-alone dynamic tracing tools capability that visualizes patterns in code based on 100 percent of the data available.

    Company CEO, Ben Sigelman, said LightStep Tracing differs from other approaches to tracing because it does not rely on a small sample of data that usually doesn’t wind up providing much actionable insight. Instead, LightStep Tracing makes it possible to search and visualize distributed traces across 100 percent of the data made available with a software-as-a-service (SaaS) application.

    Unveiled at the QCon London 2019 conference, LightStep Tracing is based on functionality originally developed in the application performance management (APM) platform, dubbed LightStep [x]PM, that is optimized for microservices-based applications. LightStep Tracing, however, does not require DevOps teams to implement LightStep [x]PM to take advantage of distributed tracing.

    Distributed tracing is becoming more widely employed because that capability makes it much simpler to visualize patterns indicative of a potential performance issues. Sigelman built one of the first instances of distributed tracing during his tenure at Google. Known as Dapper, the project now analyzes more than two billion transactions per second. Sigelman subsequently co-created the OpenTracing Application Programming Interface (API) standard that is now made available under the auspices of the Cloud Native Computing Foundation (CNCF).

    According to Sigelman, one of the most common mistakes many DevOps teams make today is overvaluing a distributed trace as a data source. While access to distributed traces are required, the real value comes from being able to dynamically inspect 100 percent of all the traces available in a way that visually surfaces patterns, added Sigelman. DevOps teams should be careful what conclusions they draw from analysis based on samples of data because outliers that could have a significant impact on application performance can be easily missed, cautioned Sigelman.

    LightStep Tracing provides DevOps teams with access to dynamic system diagrams that visually summarize the bottlenecks, as well as real-time performance histograms enable organizations to understand what defines normal application behavior in their environment. Sigelman said that DevOps teams, for the first time, can also capture shareable snapshots of system behavior that explain performance anomalies based on statistical summaries of thousands of relevant traces. Within minutes, Sigelman said LightStep Tracing can be employed to instrument and visualize service-to-service interactions across web, mobile, microservices, serverless and monolithic applications.

    One of the more frustrating aspects of DevOps is that most teams will spend far more time looking for the source of the problem, versus fixing it. But until the source of a problem can be identified, the reputation of the IT organization in the eyes of the application owner continues to suffer. There’s also an unfortunate tendency for DevOps teams to spend an inordinate amount of time in “war rooms” trying to proof that whatever element of the IT environment they manage is not part of the root cause of the problem. Rather than wasting all that time at the expense of the reputation of the IT organization, perhaps the time has come to discover and visualize issues hopefully long before any application owner even becomes aware they exist.

    — Mike Vizard