Tag: feedback loop

  • The Role of DevOps Feedback Loops in Your Value Stream

    The Role of DevOps Feedback Loops in Your Value Stream

    Feedback loops are found in all systems, organic or inorganic. They propel the recirculation of hot or cold air to raise or lower temperatures; they drive the idea exchange that helps businesses fine-tune operating strategies. Yet, nowhere are feedback loops more important but also misunderstood than in software delivery.

    Attaining the “holy grail” of software development—gaining maximum efficiency and maintaining velocity while reducing or eliminating wait times and waste—involves a variety of value stream activities that generate complex feedback loops. These loops are not sequential. They must operate in tandem and in sync with one another within a constrained environment or delivery flow will be disrupted. Understanding, mapping and optimizing this “ecosystem” is essential to ensuring all feedback loops are timed appropriately and no software activity is thrown off schedule.

    Misperceptions of Software Feedback Loops

    Much confusion arises among business leaders regarding the role of feedback loops in software delivery because they are also invaluable in other business activities. There, feedback loops operate in both open and closed systems. A system, whether open or closed, is a set of elements that take inputs through a phased transition to the emergence of an output.

    For example, assume a bank is developing a new depositor service, and although it announces the service will be available soon, it does not solicit specific feedback on how the service will work. Even if it receives random feedback from the public, it is coming from outside the controlled environment. That’s an open system. If the bank hosts customer focus groups and monitors the outcomes of specific variables without any opportunity for outside influences to impact the process, the activity is taking place in a closed system.

    The use of a closed system, whether in business or software development, embraces the principles of systems thinking—the shift in mindset away from linear to circular. With systems thinking, there are a half-dozen defined components, one of which is feedback loops. Understanding feedback loops is an essential part of value stream management to achieve the goal of delivering value to customers faster.

    Feedback loops can have a very positive or negative impact on software flow because awareness of and action on effective feedback impacts all parts of the system. But, before software leaders and teams can address their feedback loops, they must identify them in the value stream and understand the causality of each loop. As you identify feedback loops, patterns start to develop. These patterns can be direct or indirect, and feedback may need to be amplified or throttled, automated or removed altogether based on the loop’s causality.

    Optimizing the System

    Right now, feedback loops are likely disrupting software delivery flow and impeding velocity for millions of companies around the globe because software professionals, from project managers to software engineers, don’t understand them. Even though they recognize that software activities are taking place and that one activity may impact others, they don’t adequately consider the fact that information and material flow are interconnected, as are request flow, development flow and operations flow. They don’t perceive that software delivery is like a three-dimensional game of chess, with the pieces moving around the board multi-dimensionally.

    They also haven’t analyzed their feedback loops in detail so they can adjust based on reality, not perception or plan. Even if they have mapped the flow and the loops, most are unaware of the severe negative consequences even a seemingly minor issue can cause.

    Follow a Real Example

    Here’s an example that mirrors real-life scenarios I have witnessed time and again. Assume that for a new feature request, Epic 1 has entered Intake. The project gets into the Design phase and other work starts moving across the organization. Since everything is interconnected, the User Story ends up in Backlog waiting for the Epic to be completed. Two more User Stories are completed but the Epic has not moved, so they all sit in Backlog.

    Once the Epic clears Design, the User Stories start moving to the Plan phase and the Epic can go straight to Development. Soon, User Stories are In Progress and Material Flow kicks in. Somebody is working on a sprint, and someone else is working on those two pieces of code to make the change or implement a story. The DevOps pipeline is flowing, but not as efficiently as it could.

    Let’s assume instead that teams and managers now understand the feedback loops, and they have been automated to the greatest extent possible. User Stories are automatically moving from Backlog to In Progress once the Epic enters Development.

    When code goes from Build to Test, if a defect is discovered, automated feedback loops can connect to the right User Story and move the code back to In Progress. The developer fixes the defect and it goes right back to In Progress. The company meets its release window.

    In addition, once these elements are connected, and everyone understands these loops, management gains traceability across the value stream, which makes everyone, including customers, happy.

    Beyond the Basics

    The effort described here may sound like a lot of work to the uninitiated, but in practice, the effort more than pays for itself. That’s a great thing because it doesn’t stop here. In our example, as in virtually every software project, there is a lot of valuable human feedback beyond what I have mapped. It could be the information flow from a product manager through a product owner and on to someone tasked with the development portfolio. Or it might flow from testers to the product owner.

    Companies don’t have to map all this flow, but they should understand and address its potential impact. Furthermore, as project specifics, customer mindsets and other variables change, teams should adjust accordingly. That may mean creating, amplifying, throttling or automating feedback loops—or removing them altogether.

    Once organizations have attained a solid footing, they can explore more options. For example, tools, especially with integrations between and among them, can propel and automate the feedback loops, but they should be aligned, from a systems perspective, with the organization’s project workflow.

    Final Thoughts

    Streamlining flow through more efficient feedback loops is only a piece of the puzzle, but it opens up new opportunities such as more easily recognizing and removing waste. It enables organizations to release software faster, which is an imperative for digital transformation.

    Even if they don’t develop software in-house, firms that use enterprise software, from Salesforce to SAP S/4HANA, often have teams that modify it. Plus, customers and end users are all clamoring for a no-touch, self-serve digital experience. If a firm doesn’t provide, its competitors will.

    In mastering the science of delivering quality code at the right level of velocity, companies propel greater innovation, deliver ROI faster, ensure higher quality with lower costs and deliver products that better fit their intended audience and use. They also position themselves to be value stream leaders, and that pleases everyone from the board room to the sales floor.

  • How DevOps Orchestration and Feedback Loops Can Enhance Your DevOps Pipeline

    How DevOps Orchestration and Feedback Loops Can Enhance Your DevOps Pipeline

    DevOps orchestration and feedback loops can significantly impact and improve your organization’s delivery pipeline. In this post, we’ll define and illuminate the importance of DevOps orchestration and feedback loops, and how together they can make a difference in your pipeline.

    DevOps Orchestration

    Although people widely use the terms “automation” and “orchestration” interchangeably, their scopes vary with significant process differences between them. DevOps automation automates a single function or a task in a DevOps pipeline that is routine and monotonous. Orchestration describes a string of such automated tasks consolidated into a streamlined workflow. In environments where automation would not work because of barriers among the groups, orchestration can be used to help organizations solve issues despite such barriers. Hence, DevOps orchestration simply brings together already existing automation networks into a cohesive, consolidated platform.

    The Need for DevOps Orchestration

    DevOps teams deal with constant navigation between development and operations. It would be highly beneficial if a single platform could pull together the tools teams typically use as various automation pieces from multiple DevOps toolsets. This is where DevOps orchestration services come into the picture, as orchestration enables teams to achieve that.

    The primary goal of DevOps orchestration is to ensure seamless and quick delivery of new builds into production. It also aims to achieve streamlined and error-free software releases. Different organizations use various coding standards, guidelines and policies. DevOps orchestration plays a major role in efficiently managing and ensuring adherence to such policies. DevOps orchestration also is pivotal in properly defining and maintaining third-party software and their multiple versions across different environments.

    DevOps Monitoring

    Effective DevOps monitoring is vital. It is the key to understanding and evaluating various performance metrics in the production environment. When done right, DevOps monitoring can provide visibility into different critical aspects such as deviations in performance, application functionality issues and service availability problems. In any DevOps environment, the ability to identify and pinpoint such issues proactively can be highly beneficial, and DevOps monitoring enables executives to do so.

    Feedback Loop

    Feedback loops are the essence of any process improvement and DevOps is no exception. Typically, such loops initially function with an organization’s policies and progress all the way through implementation, policy adjustments (if any) and usually ending with reimplementation. They are not limited to specific process cycles in a DevOps environment—for example, it’s applicable to CI/CD practices, infrastructure monitoring and application development

    Types of Feedback Loops

    Broadly, there are two categories in feedback loops: reinforcing feedback loop and balancing feedback loop. People refer to the former as “a loop of accelerating change.” In a reinforcing feedback loop, increased value in the first entity results in an increase in the second entity, which in turn further increases the value of the first entity. Conversely, in a balancing feedback loop, value increase in one entity results in a decreased value in the second entity, which consequently decreases the value of the first entity.

    Significance of Feedback Loops

    One of the major capabilities that differentiates and carves a unique niche for DevOps from the rest of its predecessor methodologies is its feasibility to incorporate feedback from customers as well as other systems in the pipeline, including the monitoring system. The feedback obtained helps to bridge the gap between the software’s function and customer expectations. It also provides valuable insights about what you can do to improve the build of the software and ultimately enhance its functionality and reliability.

    Feedback loops are particularly crucial when it comes to continuous testing. To carry out continuous testing effectively, building automated tests is only a job half done; the rest depends on the visibility of the test results and how you can utilize them to enhance the existing process. To do that, you need comprehensive feedback of your software’s performance across various stages of the software development life cycle (SDLC), starting right from development and continuing all the way to post-production. Implementing the right feedback loops will help you gather such detailed feedback.

    Thus, feedback loops serve as the difference between merely speculating how an end-user might consume a product and methodically improving the existing workflow to better suit the end-user requirements through feedback.

    — Kavin Elango

  • The Importance of Instant Feedback in Analytics

    The Importance of Instant Feedback in Analytics

     A feedback loop is essential for every part of the modern DevOps environment. From continuous integration and deployment to application development to infrastructure monitoring to team dynamics, feedback is the most important piece of the DevOps puzzle.

    But feedback is of little to zero value if it is not timely. And in the modern development world, that means instant or as near to instant as possible.

    In the past, prior to the current world of web and mobile development, feedback was a closed loop between the developer and the application. Often aided by tests or a compiler, desktop developers could spot issues and follow trails to solve problems without worrying about end users seeing the flaws or getting caught in a bad deploy. There were no deploys, unless you consider pushing a new version out on disc or CD a deploy.

    As the world moved to web-based software and mobile software, new components came into play including cloud, containers, test-driven development and constant iterative deployments. All these led to a new need to monitor systems far beyond the single machine with operating system ecosystems prior to the web revolution.

    Before fully understanding the value of instant feedback, we first need to define what feedback is. When we are running multiple, divergent systems, such as a CI/CD server, containers for our applications or separate containers for our databases, information will be pushed to us in the form of feedback. Without some sort of analytical engine, this is simply raw data.

    Raw data often has little value. To those working directly with it, there may be something to be gleaned without manipulating it further, but for those abstracted from direct interaction with raw data, it is important to have some sort of buffer between the information we are looking for and the data in the raw.

    Taking raw data and putting it into a usable form is the role of the analytical engine. It’s important to note here that not all data is necessary to all people. For example, the sales department has no need of a dashboard showing CPU usage over time. At the same time, the operations team likely doesn’t need to see the burn rate of the sales team. Getting the information is the first step. The next is to get the correct information to the right people in a timely way.

    The importance of instant feedback is compounded by the development style adopted by the modern application development team. We want our web-based and mobile applications to be deployed consistently, if not constantly, as well as securely and with stability. When an application enters the world, we want to know if the deploy occurred correctly, if there were errors, and how many resources were necessary.

    If the team needs to wait for even a few minutes, it can be detrimental to business. The concepts of high availability and constant uptime for users has made it so every deployment needs to be tested and retested, but all this needs to be done in a short time frame.

    Modern, instant feedback allows everyone to see what is successful (and what is not) with enough time to adjust before a slight issue becomes a major meltdown. The analytical dashboards are feeding this information, converting the raw deployment and application data and presenting it as valuable feedback. This helps the modern DevOps adherent to streamline processes and continue to deliver a better product to customers and end users.

    It’s easy to see the past was a time when analytics and feedback was of little consequence, but as we continue forward, with faster deployment and development cycles, the role of instant feedback will continue to increase and better, faster solutions will need to grow to meet that need.

    — Geeta Schmidt

  • DevOps Storytime: The Line, The Loop and The Cube

    DevOps Storytime: The Line, The Loop and The Cube

    In a time not so long ago, but long enough to be before the mainstreaming of magical ideas such as “configuration as code” or “platform as a service (PaaS),” in a period where the enterprise IT shops were ruled by the one-two punch of Six Sigma and ITIL, there were three magical artifacts sought by the people who spent their working days lubricating the gears of the consulting, development and operations factories throughout the lands of corporatedom.

    The three mystical objects consisted of The Line, The Loop and The Cube. Each were first envisioned as possibilities by “automation zealots” as they roamed the corporate cubicle farms in vain attempts to provide enlightenment to their enterprise brethren. As the quest to drive evermore efficacy out of technology shops of all shapes and sizes has progressed, strategists, designers, architects and developers have been slowly creating these possibilities into being. Much like the legend of the Deathly Hallows, if these once-mythical objects come fully into being, the enterprise that can behold all three of the artifacts can create the nigh invincible product/technology factory as it becomes the master of time.

    Continuous Integration – The Line

    In conceptual land, The Line was just what it sounds like: a singular line, never bending and never deviating from its course. It connected the space from development to production. The idea of The Line has been around a long time—long enough for many shops to now have working implementations running as standard operating procedure. But it wasn’t always this way.

    circlesquareIn reality land, The Line was a patchwork of open-source frameworks, glue code, xml files and a few engineers unlucky enough to be tagged with role of “Build Master.” Woven together by the Build Master, these transformative robots and would take coders’ output, compile it into a workable build that merges all of the developed functionality, run and validate test harnesses against it. Once validated, the newly minted build then would be promoted to the higher-level environment, ultimately moving from dev to test, test to stage and, lastly, from stage to prod.

    The tools to build and manage The Line have been through many iterations since it first came into use and many shops have progressed far enough into the advanced territories of The Line to call themselves masters of the once-mythical Line. The shops exploiting The Line are now so prolific and the tools have advanced enough that it’s no longer an onerous task to get change-resistant enterprises to try it out.

    The Line has transformed from mythical artifact to concrete reality so much so that the believers now outnumber the holdouts. The end result is that shops not embracing the basics of continuous integration risk losing credibility in the eyes of their staff and any potential recruits.

    OODA is a Way of Being – The Loop

    When someone starts talking about “getting the right information to the right person at the right time,” that person is talking in fairly vague terms about the the long sought-after grail of The Loop. The Line is a good start, but the robot watching the line is only as smart as the humans who created and configured it. This is one of the many reasons to chase The Loop.

    What is the use in “shifting left” if you can’t get the alerts and metrics to the people who care? How can the metrics from “shifting left” be acted upon if they can’t be pushed into the view of those who can act upon them? The Loop is the answer, because it is the mechanism to determine many things beyond “the right person.” The Loop knows how to get messages to all interested and responsible parties. The Loop has mechanisms to account for an ever-fluid organizational structure and backup delivery methods for backup delivery methods.

    All enterprises have feedback loops. The key differentiator for moving toward mastery of The Loop is the degree to which you have automated the and abstracted the mechanisms inside the feedback loops.

    Many people and enterprises are resigned to life without The Loop. The signs are easy to see if you know where and how to look. Are mail distribution lists and org charts always a pain in your organization? Are executive-led fire drills the norm when systems are down? Are system alerts dropping on the floor rather than getting to the first responders and application owners?

    If you answered yes to any of the above, your enterprise has not committed to developing and leveraging The Loop. Basic “loop-like” alerting functionality is commonly available from alerting vendors, but true mastery of The Loop cannot be achieved without robust identity management and synchronization technology. Without this critical integration point, alerts and information flow can only be as effective as your manual processes that update the myriad data islands in each and every alerting system within your enterprise.

    When enterprises grow, being able to have the enterprise robot army seamlessly drive information flow without requiring sieve-like manual processes is also the key to tightening the enterprise OODA (observe, orient, decide and act) loop. Done with full mastery, one could imagine the possibility of acquiring the last of the three magical grails and maybe the most sought after: The Cube.

    Perceiving the Matrix – The Cube

    The story of The Cube begins with a question: “What defines the so-called ‘right’ information?” When enterprise technology and design evangelists talk of The Cube they often wax poetic and describe it as all things to all enterprise people at every level within the organization. It provides not only the integrated matrix of the fully faceted enterprise from an organization perspective, but also extends beyond operational information rights into decision rights, affinity groupings and cross-functional teams.

    cubeWe have begun to see promising beginnings of The Cube pop up within medium and large enterprises through popularizing algorithms and outfitting teams with ubiquitous mission control-style dashboards and screens in team areas. But it is only when these dashboards can integrate and harmonize wide varieties of development, operational and business KPI data that we will be able to claim that we have delivered full and timely context (i.e., “the Right Information”) to the staff sailing the enterprise ship.

    The hard part of creating The Cube is not so much the technology required to achieve its development and integration. The really difficult part is coaxing the organizational will to define and maintain the KPI taxonomies for each organizational unit and the integration points between them. Given that a fully mastered Loop doesn’t necessarily understand how KPIs, infrastructure and applications interrelate and affect each other, the Loop will forever be but a required stepping stone to The Cube.

    Will a Unicorn be Required for Unification?

    The current landscape of open-source and vendor tools to wield and tie together these ungainly enterprise models may not, as of yet, be sophisticated enough to both integrate and simplify the provisioning and cultivation of The Lines, The Loops and The Cubes. The sultry, siren-like call of the Unicorns, however, will likely beckon the cloud builders of today and tomorrow to create not just development to deployment pipelines, but also fully integrated feedback loops with intelligent understanding for how the data connects to form perspective.

    It is odd to think about, because with the idealized future state so close to actual possibility, it begs a new set of questions. Hordes of IT professionals around the world have been working to find faster and more fool-proof ways to deliver meaningful perspective into the hands of decision makers and first responders. We are so close now to that mission being accomplished that there are a few instances of the perspective itself not only being generated by “the robot” but also being handed to “the robot” to take the necessary remediation steps.

    The ending we are approaching, much like the beginning, circles around a question: “If we truly automate all the things, will there be any more work to do?”

  • Four key vectors for closing the DevOps feedback loop

    Four key vectors for closing the DevOps feedback loop

    A question I am asked frequently about DevOps – and a perennial challenge to all DevOps slide artists – is what exactly we mean by the idea of ‘closing the feedback loop’

    Q. What do you mean when you mention ‘closing the feedback loop’? Ops monitor our systems today, and alert devs when they are needed (next step, devs carrying pagers!). Is this the same Ac?

    Yes, this is definitely part of the equation. Feedback from operations (and more) to development (and more) – through alerts, monitoring, log entries, service tickets, or verbally – is essential. Good fast feedback helps, for example, to:

    • connect dev and ops more closely to user experiences
    • establish shared insight to replace fragile tribal knowledge
    • foster ownership from both teams in the entire service
    • provide rapid response to problems or new requirements
    • encourage faster experimentation and a tolerance for failure

    Feedback loops in smaller organizations tend to be relatively simple – just lean over the cubicle and tell your counterpart what is going on, or program the monitoring system to send you an alert. However for larger organizations the multiplicity of teams, volume of events, separation of duties, and variety of feedback can make this harder to handle.

    In larger organizations then, I see four key vectors used in closing the DevOps feedback loop:

    • Machine to Person (M2P)
    • Person to Person (P2P)
    • Machine to Machine (M2M)
    • Person to Machine (P2M)

    I have tried to illustrate these 4 vectors below (and not being any kind of graphic artist, I would love any feedback on this image):

    FeedbackLoops

    Machine to Person (M2P)

    This is more or less the alert cycle the original question posed – a machine process (performance monitoring, for example) picks up an error situation that it cannot identify it as a ‘known resolvable’ issue, so forwards it to a person to take charge. Machines can resolve many problems automatically with intelligent orchestration – rollback a release, increase capacity, modify a configuration, migrate a workload, restart a service – but sometimes a person needs to fix hard and/or infrequent problems.

    Person to Person (P2P)

    This is just the classic ‘hot line’ feedback, where application users (internal or external) talk directly to someone in IT (e.g. help desk staff). This can be the most powerful feedback of all. Not only can it help identify defects, but it can also identify opportunities for improvement – from new features to entirely new services. However, to be useful this personal feedback must be transferred into development or operations process – e.g. to update configuration scripts, or as input to future release planning.

    Machine to Machine (M2M)

    This includes automated problem resolution (e.g. monitoring detects a performance bottleneck, triggers a provisioning engine to add additional capacity), but also includes data mining that feeds back ‘real-world’ production data to build more realistic test and QA environments, or to inform more accurate capacity plans. This feedback makes prod more stable, makes QA and test more realistic, reduces human and process errors, and helps to build scalability into applications from the start of the SDLC.

    Person to Machine (P2M)

    This can take many forms, such as end-users logging problems into a self-service portal or Service Desk, or rating a mobile app, or complaining online. Automation can trigger resolution for known problems, so end users can address problems automatically; or feedback can reach product owners to resolve common user complaints. However, P2M feedback may waste resources and increase technical debt, as solvable problems are fixed repeatedly rather than permanently; or user complaints are ignored.

    Final Tips

    These four vectors of active feedback are just a starting point, but describe a framework that enterprises can work to when looking to close and accelerate your feedback loops. Remember too that in a DevOps mode, feedback needs to be faster, it needs to be more accurate, and especially in a large organization it needs to connect not just with dev and ops, but with other stakeholders too – business owners, test and QA teams, security teams, and of course end users.

    What are your feedback loops? Do you cover all four vectors? Do you have more that I am missing? Please let me know in the comments below – as always, I would love to hear from you.