devopsy.kinsta.cloud

Author: Abdul Salaam Khurram

ShiftLeft & DevOps: A Narrative Within a Narrative?

I procrastinated watching the season finale of “Westworld” just because I ended up reading a day later how Dolores ended the episode! I don’t know who to blame: my inquisitiveness or that guy who couldn’t hold himself. It kind of feels stale when you know the ending.

Finally, I finished weeding through the multiple narratives across the episodes, and the flashback narrative merged seamlessly with the current one. That got me thinking about using that as an analogy and maybe understand whether the concept of ShiftLeft is really actually a “narrative within a narrative” of #DevOps.

Ok, so what is ShiftLeft?

I came across the term at my fav go-to source, @devopsy.kinsta.cloud, where Alan wrote on how much to Shift Left and its diminishing returns, and another very old article on the continuous testing approach, which primarily seeks to test early and often, to catch defects, yeah, early!

Then it was thrown at me in the context of identifying repeatable activities, often manual with consistent results, performed by the Support & Operations team, and moving that work left from Level-3 all the way to L-0. Empowering the help desk, I would say!

In both the scenarios, the underlying objective was to catch defects and or reduce life support activities early in the cycle (below) which costs more and more as we move up the food chain, which in turn makes a release and subsequent maintenance cheaper.

As Alan also referred in the same article to the IBM Study, it’s almost 100 times more expensive to fix an eventual production defect which cost about six times more if caught at Implementation stage or about 15 times more if found during testing. Thus, “Early identification and prevention can improve reliable software delivery” would be the mantra.

Now, to the crux of this article, Is ShiftLeft not what we aim to achieve via DevOps?

DevOps basically creates a collaborative, non-siloed cultural change to bring both Development and Operations together early in the game.

Throw in a bunch of tools to facilitate CI/CD, pull in the security guys early, expand automated testing coverage, include change and release—who are the catalysts, as I call these functions in my earlier article here—squeeze in a few monitoring tools and dashboards and voila! You have a complete and mostly successful DevOps team to dish out the most appetizing feast of quality software repeatedly.

A table for two (Dev & Ops) has become a party of six! Yaaay!!

So, from DevOps standpoint, ain’t we doing the same as ShiftLefting (If I may coin the word) with the resulting cost effectiveness, more efficiently?

We do when we as a team make shorter and frequent sprints; gain insight, not just on how to code better or leaner but also by learning from the masters in Operations who run that software on the infrastructure they monitor and support daily. Or perform the VA scan early, or push a scenario as unit test instead of regression, and/or raise flags on potential capacity challenges.

So again, isn’t DevOps team already doing that ShiftLefting as part of its charter?

There is a limit to moving things left, but then, if your organization is still doing wireframes under a waterfall or is in the process of setting up a DevOps team, then this approach could be a stop-gap arrangement, an eventual merging of a narrative within a new narrative.

But then my argument would be, Why stop-gap and why not a full frontal? Why not embrace the DevOps model, which comes preinstalled with all the ShiftLeft possibilities?

Even if we are aware how an episode ends, procrastinating raises the risk of being left behind in the race to deliver quality, cost-effective product quickly.

Technology is redefining businesses: Transportation is an app and soon probably will be driverless; banks are rebranding as tech companies in FinServ business; a fridge can reorder milk; trash cans can be Wi-Fi-enabled! Ah, the internet of things. Is Alexa listening?

My other relevant posts are here and here.

Thank you for the time.

— Abdul Salaam Khurram

May 17, 2017
DevOps- Is [Ops] operations contribution equally Important?
I started my technology writing journey with a 2 part blog (One & Two) on Service Delivery Excellence, before jumping into the DEVOPS bandwagon, but now as I read across all related discussions, I end up feeling that the good kid is always the DEV and OPS are the unnecessary tag along.

We always talk about how the end goal is to make quicker releases, fixated towards adding value with continuous deployments, insinuating that OPS is the roadblock towards that goal, or that collaboration is the name of the game, and OPS should stop being confrontational, etc, etc.

Today, I am on the other side of the fence and will sit on judgment on the question of if any changes are needed at all for the OPS teams to engage themselves into the DEVOPS culture.

The lifecycle framework, defined by ITIL, COBIT or MOF (looking at the ‘Microsoft Operations Framework’ with renewed interest after my initial days of GRC stint) is quite similar across all phases of IT Service Delivery and consistent with the DEVOPS principles of stable, scalable, lean and quicker delivery of value to the user community.

ITSM’s broadly references
- Focus on services rather than systems
- Process oriented management approach
- Continual Improvement as part of their Deliver, Support and Operate lifecycle,
Then, the questions arise:
- Does the Service Delivery Framework (SDF) changes?
- Do we have to revisit the 3P’s (policies, processes & procedures) which are working well, as we move on to align ourselves with DEVOPS?
Historically in almost all large organizations, Operations [OPS] has always worked in silo, minimal trust of the development & testing teams to deliver a high quality change with a very suspicious view of all changes, as understandably they are responsible for the stability and maximizing availability of the Production environment with a consistent user experience.

On the other hand Developers [DEV] are under greater stress to deliver features and fix defects at a much faster pace due to many reasons. Shorter coding windows, cutting corners on testing cycles, deferring performance/ regression issues ( yes it happens!), and of course your team’s efforts are not just measured but the survival depends on the count of production runs rather than on quality, innovation or improvements.

“Throwing a dead cat over the wall” is how someone wrote of the handover of new code to OPS, I wouldn’t agree with that analogy, as OPS will at least expect it to land on three legs, a DOA would be thrown back with the same impunity!

The business value, Operations add in terms of availability and support to the customer far exceeds the risk of adding half-baked features in the ‘need for speed’. On the other hand it’s also imperative that we work towards a goal of a collaborative approach to a stable, sustainable and faster delivery of features to remain competitive.

I define DEVOPS as “combining a right mix of interdependent variables like People, Process and tools for fast and quality software delivery”, and Operations contribution towards this effort spans across all the variables adding immense value towards a successful initiative, with new insight of the challenges and shared concerns in improving the processes and delivering better software.

DEVOPS is not just about creating scalable infrastructure on the fly, or implement newer tools for rapid and automated deployments, or even about creating a SWAT response around monitoring. It is equally about envisaging a collaborative model, ensconced in proven principles and practices in order to add business value by delivering quality software rapidly.

So, what should OPS do differently to be an effective contributor of the DEVOPS team?

To start with, OPS should grab the advantage of being involved at the outset of the planned lifecycle by contributing with constructive inputs and using their expertise in:
- Assessing Infrastructure impact and capacity review.
- Influence design decisions & operational needs.
- Review all Service commitments and SLA’s
- Security and entitlement management
- Highlight any potential changes to operational procedures.
- Update monitoring and alert mechanisms
- Improve automation.
- Upgrade run-books; training of the service desk, share knowledge.
OPS also can keep a strict overview in the issues faced during the deployment/testing journey of the release and measure it against their own operational readiness thresholds.

The best outcome of this collaboration, in my opinion is that there will be no need for a ‘Warranty’ or ‘Initial support’ period by the DEV teams. OPS will be fully aware of the journey and has partnered to ensure the ‘Cat’ has passed the minimum health checkup and also is fully aligned with the changes to the support model from day one.

In fact the ‘Cat’ will no more be chucked over the wall, rather will formally pass through immigration!

Thank you for the time.

Visit my blog http://askhurram.com/

Read On, Live On.
October 7, 2015
Release Management, a catalyst for DevOps
We have discussed in my last post “DEVOPS and Continuous Change” on the benefits of doing shorter and swifter runs instead of managing a few ‘Dinosaur’ releases. We have also touched upon the way to go about multiple releases by identifying component /functional dependencies without impacting functionality and the expected savings of time, resources and cost, reaching the goal of rapid delivery of software.

It’s common to find a formal Release Management (RM) team in large organization, with a lot more of medium and some small ones following suite. We will not touch upon the benefits of having a core RM team helping plan and manage the complex journey of change from the handover of code by the Application teams, packaging and deploying across testing environments by Operations, validating testing signoffs, risk analysis, impact assessment and ultimately ensuring a successful Production deployment.

Rather, let’s delve deeper to understand how this Confounding or the Third variable called RM fits into DEVOPS. In my opinion a very important actor which neither belongs to DEV nor OPS!

I would call Release Management a catalyst in true sense, which enables change without itself undergoing such change, a neutral team dedicated in managing the change by brokering a formal hand-off between DEV and OPS.

RM teams traditionally have been the most underrated, with the DEV and OPS hogging the credit of every successful run to production. They converge the business understanding of new features or fixing a functionality break with the Development team’s new version of code release or the Operations expectation of infrastructure upgrades which will also provide an enhanced user experience advantageous to business and development.

So, back to my question, where does RM team fits in the cozy DEVOPS selfie?

Wouldn’t it be incorrect to ignore RM from the DEVOPS equation?

I would take the liberty of stating that the DEVOPS movement in an organization should be owned, driven and determined by RM, adding Change/Configuration and Release management expertise to the existing cross-functional mix of the DEVOPS team.

RM should be driving this collaborative approach to rapid delivery, bringing together the right mix of people from both sides and ensuring efficient use of resources, empowered and with executive oversight supporting hard decisions.

They would be able to add value in this rapid delivery process, by not just tasking them with managing a release cycle but empowering and enhancing their capabilities. RM team under the DEVOPS banner should also be involved in these broad points below:
- Prioritizing the book of work, creating trade-off between Business needs and Development delivery.
- Understanding complex business requirements along with adhering to geographical and geopolitical needs and rules.
- Distributed Set of applications, architectural complexities, interdependent and independent components and dependencies,
- Complete awareness of the infrastructure landscape, scalability, capacity planning, migrations.
- Release planning, calendar management, process adherence, true agile sprints, optimal resource utilization, Contingency or Rollback strategy planning.
- Post deployment/release lessons learnt, retrospection, streamline or integrate processes after every run.
- Ensuring consistent message is sent across that the DEV and OPS are not adversaries rather, partners in this collaborative approach of software delivery.
- Management reporting along with any risks assessed or highlights deficiencies.
This take is based on the many DEVOPS discussions which I have come across where there is no mention of RM or is being assumed as ‘part of DEV and works with OPS’.

My disagreement on this thought process is not about to which group it essentially belongs or will be part of, but in assigning an inconsequential role to RM in the whole release journey when in fact they should be the drivers!

Thank you for your time.

Visit my blog for earlier posts http://askhurram.com/

Read On, Live On.
May 19, 2015
DevOps & Continuous Change

A remark by a colleague while waiting for the coffee machine to complete its cycle started my train of thought. “Should we have multiple minor releases or just do a few major ones in a year?”

In large organizations, due to many factors, the turnaround time for a single successful release is quite extensive; but if the conceptual change which we are talking about bringing whilst using the #DEVOPS methodology would not only reduce these long cycles but deliver quicker and quality software.

I have already discussed in my earlier article “DEVOPS: Getting our organizations aligned” on the traditional approach and how the combination of this two (DEV & OPS) has been proven beneficial in various organizations and how many are still hesitant in embarking on that path.

Continuous Change which encompasses both Continuous Integration and Delivery is paramount to achieving this objective. To attain the objective of delivering rapid and consistent value to the stakeholders regularly, we have to have Continuous Change happening.

I am sure we have already moved out from the Mainframe era to a more Distributed systems state. But unfortunately we have yet to let go of that culture where we pile up components on a single quasi Distributed Application, instead of creating smaller independent, yet interdependent systems.

So, back to the same question; shall we go with a few Dinosaur sized, labored releases or make multiple short and swifter runs? In my opinion, we should strive towards the leaner model. Easier said than done you say! Agree. If I have to, then I would go about it this way:

In an environment having a suite of Distributed Applications, start with Analyzing the components which make up the Major release, List out the changes (fixes and new features) which are planned, asses functional and technological impact, Identify dependencies, upstream/downstream connectivity’s, architectural deficiencies, list out the identified subset of components or group of components by functionality, which can go as an independent release.

I am resisting the urge to say ‘standalone’ as with the current set of complex intervened collection of applications, it would be an incorrect statement. It would rather be a subset of components which can be independently upgraded. Now the big task is to review and decide if this subset or group would add functional value to the overall suite of applications without the rest of components going in.

Once identified, the success of this release working as expected would be in the extensive testing (Unit/ Regression/Performance) of these components, and as we gain confidence in this split and deploy procedure, we can start with, prioritizing and scheduling critical must have features or bug fixes which also should be backward compatible with other components lagging behind. We can even come up with scenarios where these components are not just getting updated frequently, but would also have multiple independent tracks. Of course it goes without saying the testing and signoff process is strictly followed albeit in a shorter window.

Deployments to production have always been a source of heartburn to the OPS teams, we have often seen finger pointing and firefights to contain change fails. In order to maintain stability and maximize availability, stricter controls and reduced deployment windows are put in place. In promoting frequent and smaller component level changes, OPS will also be in control of what’s going in, and smaller component level releases can be rolled back quickly within the Green-Zone window if the change is not working as expected. Though I would categorize this rollback, as a process fail than a deployment fail, and should be reviewed very seriously with right mitigation plan and lessons learnt retrospection, Continuous learning after every successful or failed sprint ensures competent and confident releases.

Another major factor impacting multiple runs of Production releases is the long Green-Zone windows, by moving to the quicker and leaner deployments cycles, these requests can be substantially reduced, and can eventually aim to reach a stage where changes are deployed online without extensive downtime.

Similarly, frequent maintenance activities when there are no planned deployments, effectively has the same risk on environment unavailability. Analyzing and moving to a state where these activities are conducted online without a Green-Zone would alleviate the maintenance downtimes and ensure seamless availability, adding stakeholder value.

I have not touched on the Continuous Integration or the Automation of the deployment process for obvious reasons, we can’t achieve the objective of Continuous Change without having the Integration process in place where incremental changes to code are build into a package swiftly and seamlessly, even a sanity or basic testing might be integrated to ensure the quality of the builds. Same goes for automating the deployments, develop & integrate tools required for rapid deployments. We have seen instances where automating a single activity or revisiting a process has created substantial time savings.

While it’s easy to propose changes to the Release Process or advocate rapid deployment cycles, we have to ensure adequate checks and balances are in place, compliance adherence and audit trails are created.

Now to the most important measure of this article, Cost Savings!

Proponents of the Dinosaur Releases would argue that the common activities which entail a single release like the builds, Testing, Release Management, OPS availability or reduced GZ would cut costs and multiple runs to PROD would logically increase costs.

But it’s definitely the other way around, we wouldn’t need to feed the Dino anymore, it will be a cost worthy rapid delivery with optimized resources, a leaner process where DEV and OPS are working together to ensure quality value added deliverables are made available to business faster.. This cultural change and collaborative effort also creates new vistas of growth and provides a perfect platform to excel. The cost benefit analysis would eventually be in favor of Continuous Delivery.

Opportunity cost is another variable which has not been considered; an innovation or a new feature available to customers quickly with reduced time to market cycle will provide the much needed edge in these competitive times. I recall some conversations where a few must have’s were pushed out due to various reasons, frustrating the business.

DEVOPS was never meant to replace the traditional approach to software development; rather it is an efficient use of the existing resources within a collaborative model to rapidly deliver quality software products. A major cultural shift in our approach is required and to reach that goal we need to embrace the shades of grays, many a times over 50 (sorry, couldn’t resist) when it’s not truly black or white.

Thank you for the time.

Visit my blog http://askhurram.com/

Read On, Live On.

February 24, 2015