Tag: OODA

  • DevOps Storytime: The Line, The Loop and The Cube

    DevOps Storytime: The Line, The Loop and The Cube

    In a time not so long ago, but long enough to be before the mainstreaming of magical ideas such as “configuration as code” or “platform as a service (PaaS),” in a period where the enterprise IT shops were ruled by the one-two punch of Six Sigma and ITIL, there were three magical artifacts sought by the people who spent their working days lubricating the gears of the consulting, development and operations factories throughout the lands of corporatedom.

    The three mystical objects consisted of The Line, The Loop and The Cube. Each were first envisioned as possibilities by “automation zealots” as they roamed the corporate cubicle farms in vain attempts to provide enlightenment to their enterprise brethren. As the quest to drive evermore efficacy out of technology shops of all shapes and sizes has progressed, strategists, designers, architects and developers have been slowly creating these possibilities into being. Much like the legend of the Deathly Hallows, if these once-mythical objects come fully into being, the enterprise that can behold all three of the artifacts can create the nigh invincible product/technology factory as it becomes the master of time.

    Continuous Integration – The Line

    In conceptual land, The Line was just what it sounds like: a singular line, never bending and never deviating from its course. It connected the space from development to production. The idea of The Line has been around a long time—long enough for many shops to now have working implementations running as standard operating procedure. But it wasn’t always this way.

    circlesquareIn reality land, The Line was a patchwork of open-source frameworks, glue code, xml files and a few engineers unlucky enough to be tagged with role of “Build Master.” Woven together by the Build Master, these transformative robots and would take coders’ output, compile it into a workable build that merges all of the developed functionality, run and validate test harnesses against it. Once validated, the newly minted build then would be promoted to the higher-level environment, ultimately moving from dev to test, test to stage and, lastly, from stage to prod.

    The tools to build and manage The Line have been through many iterations since it first came into use and many shops have progressed far enough into the advanced territories of The Line to call themselves masters of the once-mythical Line. The shops exploiting The Line are now so prolific and the tools have advanced enough that it’s no longer an onerous task to get change-resistant enterprises to try it out.

    The Line has transformed from mythical artifact to concrete reality so much so that the believers now outnumber the holdouts. The end result is that shops not embracing the basics of continuous integration risk losing credibility in the eyes of their staff and any potential recruits.

    OODA is a Way of Being – The Loop

    When someone starts talking about “getting the right information to the right person at the right time,” that person is talking in fairly vague terms about the the long sought-after grail of The Loop. The Line is a good start, but the robot watching the line is only as smart as the humans who created and configured it. This is one of the many reasons to chase The Loop.

    What is the use in “shifting left” if you can’t get the alerts and metrics to the people who care? How can the metrics from “shifting left” be acted upon if they can’t be pushed into the view of those who can act upon them? The Loop is the answer, because it is the mechanism to determine many things beyond “the right person.” The Loop knows how to get messages to all interested and responsible parties. The Loop has mechanisms to account for an ever-fluid organizational structure and backup delivery methods for backup delivery methods.

    All enterprises have feedback loops. The key differentiator for moving toward mastery of The Loop is the degree to which you have automated the and abstracted the mechanisms inside the feedback loops.

    Many people and enterprises are resigned to life without The Loop. The signs are easy to see if you know where and how to look. Are mail distribution lists and org charts always a pain in your organization? Are executive-led fire drills the norm when systems are down? Are system alerts dropping on the floor rather than getting to the first responders and application owners?

    If you answered yes to any of the above, your enterprise has not committed to developing and leveraging The Loop. Basic “loop-like” alerting functionality is commonly available from alerting vendors, but true mastery of The Loop cannot be achieved without robust identity management and synchronization technology. Without this critical integration point, alerts and information flow can only be as effective as your manual processes that update the myriad data islands in each and every alerting system within your enterprise.

    When enterprises grow, being able to have the enterprise robot army seamlessly drive information flow without requiring sieve-like manual processes is also the key to tightening the enterprise OODA (observe, orient, decide and act) loop. Done with full mastery, one could imagine the possibility of acquiring the last of the three magical grails and maybe the most sought after: The Cube.

    Perceiving the Matrix – The Cube

    The story of The Cube begins with a question: “What defines the so-called ‘right’ information?” When enterprise technology and design evangelists talk of The Cube they often wax poetic and describe it as all things to all enterprise people at every level within the organization. It provides not only the integrated matrix of the fully faceted enterprise from an organization perspective, but also extends beyond operational information rights into decision rights, affinity groupings and cross-functional teams.

    cubeWe have begun to see promising beginnings of The Cube pop up within medium and large enterprises through popularizing algorithms and outfitting teams with ubiquitous mission control-style dashboards and screens in team areas. But it is only when these dashboards can integrate and harmonize wide varieties of development, operational and business KPI data that we will be able to claim that we have delivered full and timely context (i.e., “the Right Information”) to the staff sailing the enterprise ship.

    The hard part of creating The Cube is not so much the technology required to achieve its development and integration. The really difficult part is coaxing the organizational will to define and maintain the KPI taxonomies for each organizational unit and the integration points between them. Given that a fully mastered Loop doesn’t necessarily understand how KPIs, infrastructure and applications interrelate and affect each other, the Loop will forever be but a required stepping stone to The Cube.

    Will a Unicorn be Required for Unification?

    The current landscape of open-source and vendor tools to wield and tie together these ungainly enterprise models may not, as of yet, be sophisticated enough to both integrate and simplify the provisioning and cultivation of The Lines, The Loops and The Cubes. The sultry, siren-like call of the Unicorns, however, will likely beckon the cloud builders of today and tomorrow to create not just development to deployment pipelines, but also fully integrated feedback loops with intelligent understanding for how the data connects to form perspective.

    It is odd to think about, because with the idealized future state so close to actual possibility, it begs a new set of questions. Hordes of IT professionals around the world have been working to find faster and more fool-proof ways to deliver meaningful perspective into the hands of decision makers and first responders. We are so close now to that mission being accomplished that there are a few instances of the perspective itself not only being generated by “the robot” but also being handed to “the robot” to take the necessary remediation steps.

    The ending we are approaching, much like the beginning, circles around a question: “If we truly automate all the things, will there be any more work to do?”

  • Enough monitoring. Act!

    Enough monitoring. Act!

    Or How I learned to stop worrying and love the auto-remediation

    These days StackStorm – and other approaches to auto-remediation and event driven automation more generally – are getting broadly adopted.

    Everyone from Facebook through Netflix and WebEx (yeah users!) are not just better “managing” their 2am pages (although that’s cool too PagerDuty) – they are exterminating them.  

    And as they adopt StackStorm for remediation inevitably a discussion emerges about monitoring vs. fixing.  Why?  

    At some level it is because of the excellent work that has been done recently in response to monitoring sucks.  Thanks Sensu, thanks SignalFX, NewRelic, DataDog, AppDynamic, Splunk, the entire ELK stack, and the other >150 monitoring solutions I’ve left out.  

    But at another level it is because of OODA tells us that all that observing and orienting is only as good as the decisions and actions you take.   

    OODA

    The thing is, even though folks gradually are coming to realize that good event-driven automation INCLUDES the D in it – and does not purely rely on the smarts of monitoring since with event driven automation it is possible to run a routine that validates events before acting upon them, most people still miss the fundamental insight of ODDA.  And it is an insight that is crucial especially if you, like those users above and many folks in the world of active cyber-defense, are going hand to hand with smart actors and otherwise are in a world in which what goes wrong is changing.  

    Here is that perhaps non-intuitive insight that, again, I’ve learned from folks that live the OODA against tricky actors (human and the entropy of scale and creeping complexity) everyday:  

    Make your false positives and the creation and control of your actions cheap and you can move fast to fix, way before a root cause analysis (RCA) would typically have completed.  

    Some of the tips that even the wikipedia on ODDA repeats for actual war fighters utilizing the ODDA is “embrace the chaos” – don’t wait to know what is happening before your act because the pace of your actions is itself a key determinate in whether you triumph or not.  

    One way to think of it is that you have a budget from O through A and the total of those steps is your Mean Time To Recover (MTTR).  

    If you spend all of your MTTR budget on O and O then your D and A will be too little, too late.

    Returning to the world of infrastructure and DevOps – here are some drivers pushing us all to focus more on the D and the A and to obsess less over the O and the other O:

    • Docker and containers and cloud itself.  Now compute can be ephemeral (sometimes).  Same deal with lots else that can be software defined.  Even storage! (hi Nexenta ;))
    • Imperative stuff exposed via APIs and declarative systems.  Yes, this is a StackStorm plug. U R surprise?  Check it – when Netflix knows that there is a happy API there that will do the care and feeding of their Cassandra reliably even when AWS is doing an AWS entropy dance, they can obsess slightly less over RCA and more over making sure StackStorm has the resources it needs to fix and fix and fix and fix again before any of us start noticing Netflix acting strange.  (You want to see how Netflix auto-remediates their Cassandra environment?  Hands on tutorial with code samples here. )
    • Infrastructure as code.  Without infrastructure as code the cost of changing the integrations and automations that live inside of event driven automation can be astronomical.  In the world of legacy run book, needing to tweak an automation to respond to RIAK burps instead of Cassandra burps is a bunch of perfectly performed mouse clicks (intentional irony) OR a nice set of proprietary code.  In our world – all the glue, the rules, and the workflows are themselves just code.  
    • And – sorry to extend the point but this is crucial – the code is YAML or maybe just whatever scripts you have lying around or maybe the configuration management system you have already deployed.  A system like StackStorm can start with the bunch of scripts you already have – those that, along with a lot of human activity comprise your D and A – and ingest them. (yum, yum, Java, Python, Bash and Ruby)   And then those scripts, those configuration management systems, all of it plus your monitoring and more are there for you to tie into a system that automates the entire OODA loop.  And then the good stuff happens because you have incredibly fast MTTR (and even proactive MTTR, preventing the outage before it occurs).  
    • Workflow.  With workflow you can see what is happening.  Especially if exposed via FLOW (another pitch, I know).  But not just FLOW and not just StackStorm.  The return of workflows is real and widespread.  Read a good overview of many of the projects published this Spring on devopsy.kinsta.cloud here.  
    • Feedback loop  Maybe goes without saying – but once you automate your entire OODA and have it as code – you can iterate better on it.  And that’s especially true if you take every single transaction from O through A and log it.  How could you do it? Don’t worry, works out of the box with StackStorm.  Or just set up a unique execution ID on every event and pass that through to each interpretation of the event via your rules engine and then keep that ID as you get the multiple workers in your highly resilient workflow system to start executing and then use that ID as well when doing post-hoc reporting; and don’t forget the APIs, GUIs and CLI as well.  Or just grab StackStorm – it is open source after all.

    In conclusion – OODA is a great frame of reference.   However we in DevOps seem to be overly obsessed with the O and the O whereas those that first articulated OODA – those folks using OODA in combat – focus at least as much on D and A.  

    To me this not only implies do the D and A faster and better (via automation) – it also implies that we may not need yet another monitoring project.  The world can stop reinventing monitoring for now, we have 157 projects at least.

    Instead – let’s turn to figuring out how to lower the cost of false positives and the cost of automating ever changing environments – so we can iterate faster and faster – staying farther ahead of the entropy monster and the real monsters trying to steal and corrupt what we build.  

    If you like this sort of discussion and want to learn how Facebook, WebEx, LinkedIn, and coming up in a few weeks – Netflix is doing it – you may want to join the Event Driven Automation Meet-up which I help organize with my friend Brian Sherwin, LinkedIn SRE and author of the remediation system Nurse.  Netflix has been kind enough to agree to host the next one at their new offices nearby to their old offices in the South Bay here in the Bay Area.  Even if you don’t think you can make it, we try to share content via this meet-up and whenever possible we also stream the sessions.  

    Also, at the risk of being “too vendory”, StackStorm is the leading open source project and company dedicated to event driven automation and specifically auto-remediation – we stormers and the broader StackStorm community are here to work with you, to support you, to get you good code and to pick your brain and to ask for your PRs, feedback, integrations and automations (aka packs).  So grab some code, join our Slack based community, and let’s get going.