devopsy.kinsta.cloud

Author: Joe Senner

  • Rugged DevOps: Good, Bad & Ugly

    Rugged DevOps: Good, Bad & Ugly

    The Good: DevOps has been a success for Dev.

    With higher expectations, better processes, more management focus and new tools like advanced PaaS and containers, developers are churning out more and more software faster than ever. Success! Or is it?

    The Bad: DevOps has been a success…for Dev.

    For Dev teams, this may result in pats on the back and high fives. But for Ops, this can look like one thankless tidal wave of work after another. The same software velocity that is praised in Dev is currently overwhelming their counterparts in Operations. And frankly, many DevOps tools released in recent years happen to benefit Dev much more than their counterparts in Ops. (My pet theory is that developers at IT automation firms are acting and thinking like…developers – focusing their creative energies on solving Dev problems.) A similar problem holds true for IT Security teams who want to build quality in by “shifting left” – see  https://devopsy.kinsta.cloud/2015/12/10/getting-rugged-devops-right/ . From the vantage point of IT Operations and Security teams, they are doing more work than ever but without any more human or IT automation resources. To add insult to injury amid their increased workloads, many IT Ops teams have to invest countless hours (away from family and personal time) in training courses and migrations to reorient their processes to fit into shiny, new Dev-centric “DevOps” tools and paradigms. 

    The Ugly.  

    Work. Overwork. Glitch. Search. Find. Rework. Repeat.

    A quick glance at IsItDownRightNow.com will show you which IT Operations teams are in a living hell at the moment. The other thing you will notice is that these IT Ops teams are from some of the most reputable companies on the planet. They are talented teams but stuck in rework cycles of web performance problems, security vulnerabilities and outages.

    These problems get the attention of everyone. Operations…and Dev…and Security…and customers…and Finance…and auditors and the press.

    No one likes rework. And no one likes the panicked fire drills that cause panicked rework. So what causes those rework cycles?

    Drama-free IT. Visibility Cures Chaos.  

    Much of this chaos can be tied to miscommunication. Boring? Maybe. Simple? No. Part of miscommunication is cultural. (Hint: OrcaConfig’s “How to Screw Up DevOps” white paper covers the cultural aspects of miscommunication). But miscommunication is also cured by automation solutions that promote transparency and visibility.

    If we want drama-free IT we need IT automation solutions that eliminate miscommunication and missed assumptions between Dev, Ops and Security.

    What better way to do this than with full transparency and visibility?

    Bird’s Eye Views:

    Many IT automation tools cater to SysAdmins who are interested in managing configurations of individual servers. That’s great if your focus is on individual severs. But IT Operations, eCommerce teams, Application Owners, Compliance & Security teams and IT Managers often care more about the health and performance of their application ecosystem (including related applications, middleware, databases and configurations that tie them together). So there is often a mismatch between the tools that SysAdmins and Developers have versus what IT Ops needs.

    Operating at Enterprise Scale Requires at-a-glance Visibility:

    IT Ops teams need bird’s-eye views of applications, databases, middleware and ecosystems to promote ease of ongoing operation. They need to be able to see, at a glance that a configuration compliance problem exists and they need to know right away the location and nature of the error. Large scale IT Ops teams do not have the luxury of toggling back and forth between multiple reports and multiple screens searching for that non-compliant, broken needle in an ecosystem haystack. If they are faced with a glitch or an outage, they need to find and fix it right away. One size does not fit all. Dev, Ops and Security each have their own interests and their own need to view their application through a lens that is most meaningful for them. Multiple ecosystem views eliminate guesswork between Dev, Ops and Security and save precious time in the find-fix cycle.

    Forrester Research spells it out well in their 7 Habits of Rugged DevOps white paper, Habit 1: Increase Trust And Transparency Between Dev, Sec, And Ops:

    “I&O pros pride themselves on understanding the interconnectedness of the hardware and software environment, and they are unnerved by complexities that poorly governed delivery practices create because the firm rates I&O pros on their ability to improve the performance and stability of the production environment. To maintain the environment, they must reduce the chances of an outage or interruption of service, and they must reduce the MTTR when an issue does occur.“

    Understanding that interconnectedness depends greatly on how much intuitive ecosystem visibility we provide to our colleagues in IT Operations. But it’s not just IT Ops that benefits from visibility, Dev and Security do too.

    Developers are able to write more secure code when they get early, in-context feedback. As David Mortman, distinguished engineer and Dell chief security architect, said, “Security issues are product quality issues, and no one wants to write buggy code.”

    Even better than efficiently finding and fixing errors is of course to prevent them in the first place and to proactively eliminate the damage they cause. Here too, the answer is better visibility. With true application ecosystem visibility, Dev and Ops teams would be able to model relationships between applications, middleware and their configurations so users will know that “changing this, affects that”.

    My teenage son has better visibility via his smartphone as to his driving route and weather conditions than I have for my multibillion dollar IT operation.

    Visibility into the Future:

    Ecosystem visibility is a key to communicating between Dev, Ops and Security and running complex application environments in the present. But what about the future?

    Release automation tools and custom scripts are excellent at deploying changes across large environments. Unfortunately they can be just as efficient at deploying typos and other mistakes across large environments, causing the chaos that we discussed earlier. Here again Operations teams would benefit from improved visibility – the ability to preview and pre-validate proposed changes across the entire ecosystem well before those changes are deployed. This is the real promise and savings of ecosystem visibility. Better visibility, with more use-case lenses cuts out the miscommunication that causes too many performance glitches, security vulnerabilities and outages. This chaos takes a toll on our IT Operations teams, our employers and our customers.

    What Next?

    We are soon entering a new season with new budgets. The temptation to “buy something” will be great. In the spirit of Rugged DevOps my recommendation is to take a full, bird’s eye view of your SDLC and to shift Ops and Security leftwards, building their critical considerations into your processes and into your IT automation tool selection.

  • Dev loves Open Source. Ops needs Scale. Now What?

    Dev loves Open Source. Ops needs Scale. Now What?

    Crossing the divide from Dev-Ops to DevOps:

    At the heart of the divide between Dev and Ops are the very different needs and constraints of each organization. Traditionally their day-to-day lives are very different. So are their war stories. And so are the metrics that are traditionally used to judge Dev and Ops. Predictably, the two organizations can end up internally competing with one another over (among other things) resources and IT automation tool selection. Part of the driving force behind the DevOps movement is to reduce this unproductive internal competition. Even with the increased levels of trust and communication that DevOps fosters, IT Operations teams will still have different criteria for IT automation than their colleagues in Dev.

    My company is considering implementing (or has just implemented) an open source configuration management tool. That’s great for Dev. But as far as I can tell my job, and my team’s job, is about to get a LOT harder

    ~ VP IT Operations

    Dev loves Open Source:

    Take configuration and release automation as example. Like so many other areas of enterprise software, many developers naturally gravitate to open source options. It’s no wonder that 90% of code in modern applications is open source – after all, open source provides developers with

    • Immediate gratification: Just Google it! Choose the right key words to describe your situation and you might find a good match.
    • Community: Don’t reinvent! Find out what have others done.  Opinions and suggestions might be plentiful across your favorite user groups. Maybe someone has solid sounding advice. Maybe someone even has specific settings or scripts that they have used. If so, you might be in luck.
    • Free: Enough said.

    But while Dev is drawn to prebuilt open source components that they can then modify,

    Ops has completely different needs including scale, stability, support, security and speed.

    Let’s look at each of these in turn:

    • Scale: It’s great that the code works in Dev and in Test. In Operations, I need it to work everywhere, all the time. And we need to add 10s of thousands of servers with no new resources.
    • Stability: Don’t force me to rebuild my infrastructure to be stable in a production environment.
    • Support: Someone, somewhere needs to be on the hook for this software, with enforceable SLAs. This cannot be left to chance. I need a phone number. Now.
    • Security: Failure is not an option. Our financial future and reputation is on the line. We cannot allow our customers to become vulnerable due to security issues. According to a study by Forrester Research, consider:
      • “One out of every 16 downloads of open source component requests are for a component with a known vulnerability.”*
      • “31% of companies have had or suspect a breach in an open source component, gaining control over open source risks is essential to improving security.”*
    • Speed: We need quick time to market and low total cost of ownership that comes from quick learning curves.

    In the context of configuration and release automation, can we satisfy both Dev and Ops?

    Yes. Here’s why:

    A new generation of application configuration automation solutions have been introduced in recent years. (To be transparent my company, OrcaConfig is one of these).  These solutions cater to Ops’ need for fully supported, central, secure control, ecosystem visibility, workflow automation and compliance enforcement – all at enterprise scale and without relying on open source, community driven scripts.

    But what about Dev’s needs?

    I will answer a question with two questions.

    1. Does Dev need open source?
    2. Or do they want the benefit that open source often brings?

    If those sound like product requirements document (PRD) type questions, they should. In an era with tightening constraints on budgets and headcount, selecting the wrong tool based on incorrect assumptions is a major self-inflicted wound.

    These next generation DevOps solutions provide Dev with application-centric modeling (after all it is all about the application), templated workflows (immediate gratification) and versioned full stack configurations (time and money savings) while providing accountable, knowledgeable support (community is great. Community and support is even better.)

    While there are no silver bullets in the drive to DevOps success, my hope is that we can dispense with the open source “requirement” and instead focus on needs and results.

  • 5 Steps for Black Friday, Cyber Monday and Holiday Readiness

    5 Steps for Black Friday, Cyber Monday and Holiday Readiness

     5 Steps for Black Friday, Cyber Monday and Holiday Readiness

    Ho. Ho. Ho. It’s that time of year again. Seriously. It is.

    It’s too late for April Fools but it’s definitely not too early for Holiday ecommerce readiness planning. In fact now is the perfect time to get your ecommerce house in order well before the planned lockdowns of your critical web based applications.

    Balance is important in any holiday ecommerce planning. On the one hand, you obviously couldn’t and shouldn’t make too many significant changes to your ecommerce environment too far into November. But you also need to actively seek out and evaluate new processes and new tools that will improve your customers’ and your employer’s experience during the upcoming holiday season. Making too many or too few changes can land you in hot water. So for starters, perhaps we can agree that a reasonable goal is to avoid earning hall of shame status at http://www.isitdownrightnow.com/

    Christmas Downtime can have multiple root causes but can often be traced back to

     

    1)      Sins of Omission: Lack of proper planning for the anticipated burst in traffic on Black Friday, Cyber Monday and the entire holiday season and

    2)      Sins of Commission: Really frustrating situations where you were trying to upgrade, trying to improve, trying to modernize but sure enough, “changing this breaks that”. You don’t want to spend the holidays remediating one problem after another. Do you?

    Rudolph the Remediation Reindeer from OrcaConfig

     

    The focus of the remainder of this article is to help you with the latter – those seemingly innocuous, “routine” changes to your ecommerce app and all of the related apps, databases and configurations in your ecommerce ecosystem.

    So what’s a DevOps team to do?

    Life is tough enough dealing with “unknown unknowns.” Let’s not make it any tougher by failing to plan for the “known unknowns”. So here are a few keys that we have picked up over years working with e-tailers and retailers who prioritize Christmas ecommerce:

    1. Blackout Periods: Make sure your automation tools know you’re in a blackout period. Whether you use homegrown tools, commercial automation tools, or a combination thereof, users should not be allowed start automated jobs against Production systems during the holiday blackout without approval.
    2. Role Based Access: If your automation tools don’t support blackout periods (and probably even if they do), now is the time to double check that the role based permissions in each tool are set up correctly and adhere to your organization’s guidelines. The most well intentioned engineer can accidentally select Production as a target environment for an automated job, and if they have permission to do so, it can cause an accidental catastrophe during a business critical time.
    3. Restrict Direct Access: Most organizations have an initiative to lock down all their Production servers from direct access, and if you don’t it’s time to add it to the docket. An engineer logging directly into a server to make a seemingly innocuous configuration change to one application can easily bring down another application in the process. Now not only do you have an outage, you have no way of knowing why the outage occurred and who caused it.
    4. Monitor Changes, Not Just Performance: Don’t just monitor the performance of your Production systems, monitor everything you possibly can including configuration changes. Especially if your engineers still have direct access to Production systems, it is imperative you have the tools and processes in place to monitor for configuration changes to the environment.
    5. Experiment Now, Well Before Show Time: All year you’ve been going to conferences, chatting with your colleagues, and in general hearing about all the new tools you should try out. Now is the time to do it! Plenty of vendors offer free trials of their solutions. Some even offer free, full featured trials (ahem). Make good use of your time now by evaluating new time-saving, error-proofing solutions before the calendar catches up with you.
  • DevOps-in-Name-Only? Don’t do DOINO

    DevOps-in-Name-Only? Don’t do DOINO

    Let’s not be ‘that’ guy. And let’s not be ‘that’ organization.

    You know, the one where “DevOps” is some magic pixie dust that we can sprinkle around, and then pat ourselves on the back for being forward-thinking, modern IT managers. It’s actually harder than that. Harder but simpler. Harder because DevOps involves humans. Simpler because pixie dust doesn’t actually exist. (I prefer software instead.)

    There is thankfully a lot of good “devops” software out there. We are pretty proud of ours too but we fully admit that paying closer attention to human factors of DevOps helps ensure that you get the most from your software whether you buy it or build it.

    Respect

    Any “DevOps” organization worthy of that title needs respect and understanding between Dev and Ops. Each brings their own talents, mindsets and constraints. Those other guys generally aren’t trying to make your life difficult just because they can, or because they don’t care. They are making your life difficult because they may not fully understand your environment, your pressures, your management and even the metrics that define ‘your side’. Or maybe they have pressures and constraints of their own. This isn’t a matter of touchy-feely kumbaya. Getting to know and respect each other is actually just good business.

    Communication

    A DevOps organization should also have good communication. Note that I didn’t say lengthy communication or countless tedious meetings. Just good communication. I also didn’t specify whether that should be verbal or written. It just needs to be good and it needs to fit your organization and your environment. Whether you are a multi-national organization with datacenters across the globe or if your entire IT department can fit in one room,

    1. Focus on sharing what is important for your colleagues to get their jobs done.
    2. Make clear what you need or what you need to know from them so that you can do your job well.
    3. Don’t hide that key nugget or even that bit of bad news under mountains of data. Over-sharing is as destructive as withholding key info.
    4. Say ‘thanks’, ‘good job’ and ‘sorry’ every now and then to your colleagues in Dev/Ops.

    Process

    A) People learn by doing, more than being talked at. B) What gets measured, gets done. So find something that works for your organization and your culture to nudge the DevOps mindset that you are seeking. Maybe it’s a job rotation program (even for a couple of hours per week). Maybe it’s paid company lunches. Or working in the same physical area. Perhaps it’s having Dev report on traditional Ops metrics and vice versa. Or maybe it’s sharing the risk and rewards of holding Dev and Ops accountable for combined DevOps metrics. The point is to foster team performance in a we’re-in-this-together mindset.

    DOINO

    The bad news is that #DOINO (DevOps-in-name-only) is worse than useless. DOINO can cheapen and make a joke of a pretty good concept that should be helping IT teams accomplish more and enjoy their jobs while doing so.

    The good news is that you don’t have to be perfect at all of the human factors of DevOps to make real progress. Your improvements may be sparked by formal cross-functional meetings, informal sharing of more beers or even (my favorite) by clever new software that automates the hell out of tasks you had previously taken for granted. The important thing is to just get started.