devopsy.kinsta.cloud

Author: Don Macvittie

  • Shhhh! It’s a Secret

    Shhhh! It’s a Secret

    In one of our regular back-and-forth breaks, Lori and I were talking shop a week or two ago, and she shared an excellent report by GitGuardian with me. Frankly, none of the information in the report was a huge surprise for me–Git is filled with information that companies probably don’t want out there, but that is nearly impossible to track proactively. That’s what GitGuardian tries to do for you, but honestly, I’ve not yet dug into them, so caveat emptor. Or, read someone else’s opinion to help you decide if they’re a good idea – I’m just going to talk about the report.

    The key takeaway is that you are asking your DevOps teams to move faster and faster. Putting a section of code they are working on into a public repository is an easy way to “take work home” or share it, especially during lockdowns. It is also a good way to hand over critical information to anyone searching for it—about your infrastructure, or even critical bits of data like SSL keys or credentials. And with the current level of cyberattacks against nearly every industry, I’d say there are people looking for that information, no matter what industry you are in.

    This is not really a DevOps-specific issue–credentials and such in code has been a problem for IT for as long as I’ve been working in the field. At some point, any automated system needs to perform AAA, and that means the data required is stored somewhere. A multitude of approaches have been attempted to resolve the problem technologically, but none have been a silver bullet. Yet. Eventually, someone will figure out a way to make it work; then a whole new attack vector will arrive, probably aimed at the tool that is settled on to protect this very data.

    This is a personnel/policy issue that is exacerbated by the speed of Agile and DevOps. Your best solution, at this time, is to write a clear policy that contains consequences for putting secret data into a public repository. We’re short on “consequences” right now, preferring to say, “Right, let’s make sure that can’t happen again,” but in the case of putting secrets out in public, it is a different level of ‘bad’ than a coding error getting checked in. Those secrets can actively be used against the organization, and if the not-to-be-shared data is clearly delineated, any violation can be viewed as intentional, be it because of convenience or malfeasance.

    According to the report, there are millions of bits of privileged information out there. Not all of them are from enterprise IT shops—but some are. And some might be from your organization. Set expectations, get feedback. Make it clear you do not want to stop progress on delivery of systems, but you also don’t want to implement a system with a built-in back door for attackers to use. It is easy enough to use alternative delivery methods for those few bits of data, and put the bulk of the source out where it is needed.

    And keep rocking it. Toward Data Science says there are 128 million public repositories out there (Q1, 2021). Think how many lines of code are in those repositories, and how many secrets potentially are, too. Just make certain yours are not included in the list. No sense keeping your next great solution from running because your critical data leaked.

  • The Forces Massing Against the Data Center

    The Forces Massing Against the Data Center

    We’ve come a long way over the last few decades. A ton of stuff about IT has changed, even in the last five years. And it is true in IT that the only constant is change.

    We like to have control of our environment to provide the most stable platform for user satisfaction that we can and simplify troubleshooting when things do go wrong. But even such a simple desire has been stripped away in the flood of changes we face. To be certain, some areas of IT within most enterprises are very stable environments that only change after careful consideration, but much of the rest of IT–both within and outside of the enterprise–has become the wild west again.

    And there is a lot of good in this cycle. Trying new things solves old problems and drives innovation, there can be no doubt. The changes we’re making address things like, “There was never enough testing,” or “We don’t have the security staff,” which means our apps are better because of the wave of changes we’re currently riding. The cloud is a standard option for new development even in the most stodgy enterprises, and some enterprises are moving older apps to the cloud as time permits.

    That last bit is going to force some hard choices in the future. As many of you know, I spent several years writing exclusively about storage, so when I see something happening in the storage space, I get interested. A former coworker of mine recently wrote about Kubernetes storage options from an angle I hadn’t considered before, and it drove me to look deeper into what’s going on today.

    In case you missed it, storage–specifically, data gravity–has tethered a large number of applications to enterprise data centers. The app needs access to the data; if the data is in the data center, the app has to be there and have secure, ready access to the data somehow. Cloud and containers offered some options/workarounds to deal with this issue, but they’ve lagged far behind the rest of the agile infrastructure.

    That is changing. Higher-speed, distributed datasets are here and are being used. I have not followed the space closely, but my immediate thought is, “They have to be proven.” They’re going to have to show that they’re as stable and secure as the DBMS in the data center and that you can get your data out of them no matter what. In the end, data is a company resource–a very valuable company resource. In some cases, the data a company owns is worth more than the company’s products. So, total control and security of that data is imperative and must be proven on the ground.

    But it will be. Because there is a need. And that will be the final step in making data centers far less critical for the average enterprise. Some enterprises already don’t need a data center, but most still do, largely because of data control and the resulting data gravity. Apps can be shifted from cloud to cloud or cloud to on-premises, or from cloud to hosting via Kubernetes, almost as an afterthought. But data is far less agile. In the end, a couple terabytes (or petabytes, depending on dataset) of data aren’t just flung over to a new cloud. Distributed, though; copies could be done in parallel, at high speed, making them far more mobile.

    Vendors already recognize that subscription models are easier to sell, with far fewer user maintenance headaches. IT enjoys not having to go through huge checklists to make sure every bit of supporting software is up-to-date with the right version–in a hosted environment, it’s done for you. Enterprises already recognize that a fair number of applications can be cloud-native and are putting them out there. Slowly, over time, more and more applications–even data-heavy applications–will be viable in the cloud for your average organization. The bottleneck, as I see it, will be cloud storage costs. Cloud storage is both more cryptic than other cloud costs and more expensive. Ingress, egress, storage, instance … all these things add up, and the more instances required to achieve parallelism, the more they cost. Until that issue is resolved, don’t expect any mass migrations. But migrations will come, eventually. Kubernetes is preparing to do its part and make data easier to use and move; cloud vendors will just have to find a way to rationalize the bandwidth implications with the costs we, the users, are charged.

    But over time, it will be less profitable to produce hardware and software targeted at individual enterprises. We’ve already seen some moves to hosted models that should open your eyes to the direction your vendors are headed–Atlassian and pretty much the entire source code scanning industry, for example, come to mind.

    For most organizations that have been around awhile, the key to making the move will come from internal motivations. They’ll have to figure out how to manage containers with virtual storage internally, in their data center, before they’re comfortable throwing that architecture out onto the public cloud. It’s a good path, but for the short to mid-term, data gravity will slow or even halt the “To The Cloud!” charge for core applications.

    For most large and established organizations, the ROI just isn’t there to move tons of data around. The core systems that work will be kept and may be improved, and only when they need to be replaced will things like a full-on container architecture be considered. But, again, this is likely to be an internal architecture, to start, (and often, will be forever) simply because that’s where the important data is.

    For newer organizations, the opposite will be true for the short to mid-term. As the data lake grows in the cloud, the costs to keep it there grow in a non-linear fashion, eventually biting into profitability. This group will also want to understand how to run a fully agile container infrastructure internally, complete with virtual data access.

    Whichever group you belong to, consider adding on-site Kubernetes or local cloud skills to your talent repertoire. Those are the skills that overlap; the juncture where all of these paths seem to cross, and the agility that full-on container automation with storage offers will not negatively impact the organization.

    And keep rocking it. We’re enjoying your apps. Thank you for keeping them humming along!

  • Some Technical Debt is Self-Resolving

    Some Technical Debt is Self-Resolving

    But don’t count on it.

    Fellow devopsy.kinsta.cloud contributor Bill Doerrfeld recently wrote, “Enterprises Face Growing Technical Debt,” a great article about trends in technical debt, and it got me thinking about some of the things that have been going on in the industry over the last few years.

    As I’ve mentioned before, the changes underlying our systems combined with the need to keep systems that existed before those changes alive are making for quite the debt ratio, and that debt is coming due, probably sooner than you think.

    We’ve got enhanced ways of tracking how much time technical debt is consuming built right into our CI/CD tools, but there are inherent weaknesses in those tools. The ability to track time spent on issue X, and to categorize issue X as debt, requires a solid categorization process and relies on staff to keep close track of their time. Right or wrong, the industry has currently (maybe permanently) moved away from tracking time in small increments and, instead, most organizations track results. But unless you know the percentage of time spent keeping a system that is outside your main-line development toolsets alive, you can’t determine how bad technical debt really is.

    There are some bright spots though, and while I (and others, to be sure) have been sounding the alarm about technical debt since DevOps teams were all allowed to pick their own toolchains (glad that’s mostly over with), we need to be aware that counter-forces will help alleviate technical debt issues, also.

    First is the trend of making things simpler. Let’s face it, on the DevOps side, Python became popular at an insane pace. There is a very good reason for that. Shell scripts written by master scripters to achieve a goal are impossible for even good IT folks to follow. What you can’t follow will take much longer to fix–because you have to break it apart, figure out the parts, parse through massively complex pattern matching, etc. Python simplified all of that. If you write a Python script to do the same thing as a bash script, a far higher percentage of IT will be able to glance at it and go, “I know what this does.” This trend exists across IT (bug dashboards are another good example at a much higher level), and as easier-to-understand tools are increasingly part of the mix, the old and arcane will be less of a debt.

    Second is that, while I have raised warnings about things like the growth in API usage and CI/CD toolset complexity, the fact of the matter is that both of these things reduce the amount of code that we, internally, are maintaining. Steps automated, be they by calling someone else’s API, or calling your CI/CD vendor to handle build or deploy steps, are less steps for us to worry about (as long as they work, of course–have trusted partners).

    But we still need to be aware of technical debt, because all of the things we’ve mentioned in the past—the increasing complexity of our environments, staff turnover, team-specific tools and languages—are driving up debt.

    Check with your CI/CD vendor to see what their offering for tracking debt looks like. Other measures that don’t require tracking actual time spent on individual activities include keeping a tally of the number of bugs. And a surefire way to get a great feel for where your weaknesses are is looking for the software no one wants to work on, or the software that someone warns the new people not to touch. That’s where your debt load is the heaviest. IT is full of tinkerers;”If it ain’t broke, don’t fix it …” only applies in IT if “…because no one here knows exactly how it works,” also applies.

    Ask your teams to list which tools in their area (be it silos or DevOps teams) need to be updated and/or replaced, and then to rank them. Then take that data to team leads, and ask them to decide where your limited hours and money should go. Provide data on relative debt that you gathered above, so a comparison of team rankings to debt cost can be made.

    And then, set aside time to alleviate debt. It doesn’t have to be massive, just keep moving forward on it. You set aside time for all sorts of things, even a few man-hours a week on debt will multiply. It will start slow, but once you are spending less time each week fixing “product X,” some of that time can be added to technical debt resolution, increasing what can be done. Look for low-hanging fruit; a small app or script that eats up an outsized amount of time, or a system that is insanely expensive to keep, while the market has moved past what it actually is used for in your department, etc.

    Then enjoy the fruits of your labor, and put those astounding skills and that extra time to use rocking even more new apps for your users.

  • Yes, You Do Need SCA

    Yes, You Do Need SCA

    We often go to restaurants and treat ourselves to unfamiliar and exotic foods made with ingredients we’re only vaguely aware of. A chef and their team (or a manager and their crew) are our vouchsafe that what’s in there isn’t deadly. Most of the time, that works out just fine; but, very rarely, we end up paying the price. The same is true for food brought home from the supermarket, though for lesser illnesses it is harder to track because they’re not clustered like restaurant customers. While it is indeed exceedingly rare for the results to be fatal, the CDC estimates that around 3,000 people die of food-borne illnesses in the U.S. each year, and millions more are made ill. The U.S. has incredibly safe foods, and yet, we still have these illnesses. Simple things like leaving ingredients out too long, or consuming food sourced from a contaminated production facility can cause problems, even in an otherwise safe environment.

    Our code is similar—how it is created and consumed—and increasingly falls in this category. We rely upon a ton of sources for the systems that run our networks, and, quite often, there is no one to tell us how safe our sourcing is, or if changes are happening beneath that escape our notice.

    This is why we absolutely must have software composition analysis (SCA). If no one else is going to stand up and verify the safety of all that code we are pulling in, then we need to at least be aware of what is there. We need to understand the inter-relationships of pieces of underlying software, both the known and the unknown. Some libraries are so ubiquitous that you are using them and likely don’t even know it. But that’s not good management; knowing what you have and taking care of it are increasingly becoming a core part of IT. Knowing what alternatives are available in case something goes wrong with your preferred (or automatic) inclusion is imperative for continuity, making SCA a critical part of continuity planning along with the fact that it’s just good business practice.

    The good news is, if you are using any of the major code scanning tools from the DevSecOps space, you either already have SCA available to you, or you can buy it as an add-on from the same vendor. Hit their website and find out.

    Your systems are out there, every day, serving the needs of your users. Protect those users from unforeseen circumstances like vendor issues, open source changes, security vulnerabilities and nested dependencies. While the vast majority of open source is safe and secure, we use so much of it—and the tools we use link in so much of it—that it’s almost certain there is a bomb hidden somewhere in your code. Think of it like the food-borne illness example above: Would you eat at a restaurant that said “Yeah, 90% of our food is safe, but we’re not sure about the other 10% …”?

    You’re building astounding things, and keeping the world moving. Don’t stop, but don’t let your guard down, either. Know what is running in those apps, what its vulnerabilities (both security and downtime) are and what you can do to mitigate those weaknesses–so you can keep making astounding things rather than spend your time fixing what is already there.

  • Make a Plan for Test Automation

    Make a Plan for Test Automation

    The amount of testing that we could be doing is massive. Most of us don’t look at testing across the spectrum and all-inclusively, but let’s do that for a second.

    We have functional testing at the code level, which is reasonably well automated already, if your shop is using such automation. Then we have integration testing, which is a superset collection of API testing, code integration testing, usability and regression testing. Integration testing is automated, and sometimes that’s done well, but this massive space is still growing. Next up would be security testing, which covers much of the same code that other testing is interested in from a strictly security focused perspective. Then, there is actual usability testing, which is more important for some areas than others—embedded needs far less usability testing than web apps, for example—and has different levels of automation at this point in time.

    Really, when you think about it, the complexity of this environment is massive. We not only need to test an array of different functionalities from an array of different perspectives, but we have to do it on an array of platforms. Testing on Windows is different than testing on Linux; testing in the cloud can use different tools than testing in-house; clients can be on any given platform, depending on the application and target market … we really do have a beast, here.

    And testers have always been behind, or have not covered things well. This is not a ding on testers, this is the result of the complexity at hand and the fact that, in many instances, one line of code can go wrong in a bunch of different ways.

    Enter automation, which is growing rapidly in the world of continuous integration (CI). Automated testing tools range from broad to very focused, and all help facilitate testing. The DevOps increase in development means more testing, and testing was already behind. This has been discussed pretty thoroughly in the past, but it finally feels like we’re getting to a point where we have viable solutions to the problem. If testers become test developers that review results of automated tests, and through CI those automated tests are increasingly worked into the process of development, we’ll see testers getting better coverage and quality seeing a continued increase.

    But that does mean you should keep your eyes on the market as it continues to grow, and pick tools that offer the most functionality that is most useful to your test team. And get them up and running. More testing means more work setting up tests. Even no-code test solutions for UI require someone to record the actions that are necessary to complete the test, and that takes someone’s time. But the return is long-term testing that doesn’t require a person until a red flag is raised by automation.

    I’m all for it. Both testing and security have the most to gain from automation, because they’ve been the most behind historically, and corner-cutting has cost organizations dearly. So, let’s get with it; check out some new test tools (there are plenty listware articles giving you a place to start), and let’s make great software better!

  • A Funny Thing Happened On the Way to API Management

    A Funny Thing Happened On the Way to API Management

    As API use has exploded across IT departments large and small, the need to manage APIs as independent entities—both APIs themselves and calls to them—rapidly became apparent. We have been through this cycle several times before–ORBs, DCOM, object repositories … the list goes on.

    There are several competing goals when introducing some management structure into formerly as-needed remote calls occurs. We want to publish and advertise connections in the hopes we can avoid (or at least minimize) reinventing the wheel, while we also want to make access easier for app developers that are trying to use our APIs. Meanwhile, we want a centralized location where ops can monitor API usage, security can review APIs for compliance and “approved” APIs can be advertised or even hosted. Finally, we want to give internal developers access to approved external APIs that have been deemed appropriate and safe for the organization to use.

    As with previous iterations of this process, these goals, taken together, prove to be huge. There is a lot there to try and cover in an ever-changing API landscape. It is worse in this iteration, simply because of the proliferation of external APIs. Prior to REST, each iteration had more APIs, both internal and external, than the one before, and REST is keeping with that pattern, though now that we can look back, the external bit is a hyperbolic curve, not a line.

    So, it is interesting and cool to see the market fragmenting a bit. The entire package is a huge order that a few companies have taken on; I’ll leave it to you to decide if they’ve handled it well enough for your org. Others have taken one of two approaches—centralizing API access to external services that have been vetted and interfaced with by the vendor or enabling internal API developers and users.

    Now, we have a couple of manageable chunks to look into. I would even go so far as to say most vendors come from one or the other of these worldviews, and you should compare their “tilt” to your organization’s needs before purchasing an API management product. I would much prefer if we separated these markets into ‘API management’–those doing (or trying to do) it all, ‘API integration’–those simplifying and enabling access to approved APIs like Salesforce or ADP and ‘API enablement–those making writing, publishing, securing and advertising internal APIs easier. But I don’t get to make these calls, so we’ll have to watch it unfold.

    If your need is to streamline and protect your organization’s access to external platforms, and you use many popular services, start with API integration. If you need to get a handle on APIs that you are creating and using internally, start with API enablement. If you need both, and a centralized store of approved APIs regardless of source, look to API management. APIs have become central to our development efforts, and there will only be more of them in the future, so it is worth deciding where your org could use help and set out to research your options.

    Of course, you’re rocking it. DevOPs isn’t letting DEVops poke too many holes in the firewall; DEVops is cranking out solutions and devOPs is keeping them alive. This is just another tool to keep things on an even keel as more and more of the organization’s development is API-based.

  • Cattle, Not Kittens – Revisited

    Cattle, Not Kittens – Revisited

    The early days of DevOps were filled with pronouncements that we needed to treat our servers as cattle, not pets (kittens). For a year or so, the refrain was everywhere. And we did. We detached servers from dedicated hardware, we made spin up and spin down so easy that we just replaced malfunctioning servers (which has caused its own problems), we have even made servers designed to spin up, do one thing well, then spin back down until needed again – something unthinkable even a decade ago.

    And we’ve reaped benefits. Servers suck up less attention from the DevOPS side, and standardization has made it easier to code for them, moving things along and allowing members of another team to pitch in when there is a crunch because of familiarity.

    But IT is filled with people, and something happened in the transition. In hindsight, we should have seen it as a possibility, but things changed so much, so fast that I don’t think anyone was considering this possibility.

    Our DevOps tools/toolchains/processes became our pets (kittens).

    Now the special snowflake that requires specific knowledge and has to be treated with kid gloves is the DevOps step, not the servers it uses. Like pet servers, a fragile step that is working is avoided like the plague, when it should be shored up. Like pet servers, a process that is locked into a specific tool, OS, or platform is left that way, even when other options are a better solution. Like pet servers, the person who “knows all about” that is left to manage the step that should, by all rights, be easily understandable by anyone on the DevOps team.

    It’s time to start treating DevOps like cattle, not kittens. Some steps (like some servers) will be more difficult to make less pet-like and more cattle-like, but if history is any gauge, the payoff is worth the investment.

    We don’t have a cross-toolset language, other than doing things in the various shells and calling the shell scripts from a given toolset (not the best solution, just an example of where we’re at), but that shouldn’t stop us from standardizing and genericizing as much as possible. Less fragility, more reusability and a broader support team is good for everyone.

    You all are knocking it out of the park. We – all of us – are spending more time at work and at home in the digital space than ever before, and, generally speaking, the tools “just work.” That’s all on you. Keep kicking it. But standardizing to keep things running smoothly just makes sense. You know your org; it is possible that the easiest route to cattle is to standardize on a single toolchain (perhaps one or two toolchains with reusability documented) possibly to shove a bunch of steps into scripts and share amongst whatever tools each team chooses (I, and history, strongly discourage this approach, but neither I, nor history, has to live in your environment. There might be a good reason for this, so I include it here).

    All I’m saying is, don’t re-implement every step over and over if you can avoid it, and don’t make steps so arcane that you need a specialist (not internal, and definitely not external). Keep rocking it, and get those cattle moving!

  • Our API Mess is Coming

    Our API Mess is Coming

    APIs are the satisfaction of a long and deep need to be able to create consistent and reliable integrations between disparate systems, operating systems and datasets. As we started to use REST-based APIs, we also realized they fulfilled a previously little-addressed gap in automation. Frankly, they are a force multiplier in DevOps delivery.

    Early efforts at system-independent APIs were not great, but we learned lessons and are steadily getting better. While REST is the default solution for APIs, it probably isn’t the end of the road, as we continue to learn and improve. That’s what we in tech do – get excited about an idea, drive it to broad use and then iteratively improve it.

    And that’s the root of the problem. We have been, and are, saturating our applications with APIs. Internal, external, external-that-rely-on-another-external. The thing that comes to mind every single time I look at an API dependency map for even a medium-sized application is “spaghetti code.” Those lines are going all over, quite frequently returning to themselves.

    And we’re going to have to maintain this creation. Maybe not us, individually, but us as in IT and DevOps teams. The only constant is change, and far too many of us are not only failing to plan for API changes, but failing to even think about them. Because things are generally going swimmingly and most IT managers and DevOps teams have more urgent things to think about.

    But risk management – particularly broad risk – is an essential part of future planning. If you lose access to external API X, what will you do to keep your application running? If you lose an entire internal system that hosts API Y, how can you recover in a timely manner? While most organizations will identify and repair single points of failure, the solution is a pool, and the impact of losing Z servers in the pool needs to be understood, if nothing else – questions like, “At what point would user experience be impacted?” need to be asked, and a plan formulated to keep that from happening.

    We are already seeing issues created by using external APIs. All of them are predictable – an API going offline, or even disappearing completely; an API interface or attendant data format change; a change to the algorithms underlying the API, causing a change in data results, etc.

    API usage is far easier than it has historically been – and companies are working hard to make it easier to access APIs, both yours and others’. This is a good thing, as it means less time spent figuring out and tweaking API calls and request data. But it means our API dependence will only grow, and the number of lines in our dependency graphs – each representing some amount of risk – will only grow.

    “But we’re doing things in the cloud, so we can auto-scale, and failures are no longer and issue for us!” some shout. While that can help some scenarios, it doesn’t help others – if a vendor goes under and takes an API you use with it, or a preferred vendor (or inside component) becomes a security risk, you’ll still need to take steps. Knowing what and where your weaknesses are before they become issues is important, and worth your time.

    Most of you are killing it, cranking out solutions faster, iterating to make them better, using APIs to extend functionality and reduce delivery times. I’m just saying, stay aware of the technical debt you’re creating, because debts always come due, and risks eventually become issues. So know what’s a risk in your infrastructure, and have a plan to react when it does cause problems.

  • Dear Vendor: Letter to Technology Vendors

    Dear Vendor: Letter to Technology Vendors

    While I was writing my “Dear Management …” post, something was going on behind the scenes that inspired me to write another. We all have things we wish to say to vendors; this open letter picks some of the big points that they seem to invariably struggle with.

    Dear Vendor,

    We want to use your products, otherwise we wouldn’t even be writing to you. But depending upon who you are, and what year it is, you are struggling to meet our needs.

    We understand that your insistence that your new product or acquisition will solve all of the problems we don’t have is probably executive-driven, but give us the benefit of the doubt. We know our environment, and just because we use you for one thing does not automatically imply we want you to constantly push this other thing at us. By all means, tell us about your entire product line, but then back off. We will let you know if anything appears to fulfill a need.

    While we’re chatting, can we discuss pricing? Not just cost, but the byzantine structures you create, time after time? If you are going to use 15 different charges and fees to jack up the price there, we’re paying the bill – we’ll see it. You’re far better served by being forthright. Do you have any idea how many stories like, “I am not funding your Ferrari with your commission,” stories (this was actually said; it was hilarious – we went with a competitor) are out there? And they keep occurring. Just this week, the phrase “Your pricing is bonkers!” was submitted to a company. We want a good product at a fair price, not a cheap price up front that will drain all of our budget over time.

    Speaking of good products, let’s talk about security. Security issues with your product are our issues more urgently than they are yours. Don’t mess around; when notified, quantify the threat and notify us immediately, with suggested remediations until you can get a patch ready. Holding off until you can patch puts us in the unenviable position of being vulnerable and, worse, not even knowing we’re vulnerable. Use good coding practices so we’re not all caught off guard, and waive those secure coding requirements for no one. Interns and C-level execs are not professional coders (interns aren’t yet, and C-levels no longer are), and any code by these two groups needs to be reviewed by someone that knows your systems before they are sent to us for deployment. Don’t force bad updates on us. Do your best not to give us bad updates at all.

    Finally, on the topic of forcing things on us – guess what? We have a need for a specific tool in a specific environment. Telling us that starting next month your services are all “moving to the cloud” does not suit all of our needs. We get it, you want that data. It’s all the rage right now. You should understand that, if you stop serving our needs in your quest for our data, we’ll go elsewhere, eventually, no matter how reluctantly. Cloud is a great platform, but most IT tools need the versatility to run where we need them, not just on one platform out of many.

    And, as a side note, your AIOps generally isn’t. We know it’s the buzzword du jour, but honestly, if it’s basically the same product with some logic or a dashboard strapped on, that’s not AIOps, it’s normal product improvement. When you have deep data collection and analysis that uses trained algorithms, then you can join the crowd screaming AIOps. Until then, hush. You sound silly, and it undermines the rest of your messaging.

    Just to reiterate: We have a need, show us how you fill it, and let us decide if that suits our requirements. Then tell us about your other offerings and back off. We’re smart enough to smell vaporware, stubborn enough to resist your desire to collect all of our data and wise enough to judge whether or not you and the product you’re pushing this week will be around in a year or two. You serve a purpose, and we’re grateful you are doing this stuff so we can focus on our business. But please, make life easier on us, not harder. And if one more of you sells a product we don’t need to the C-suite, we may explode. That Sword of Damocles that you raised over us will inevitably fall, and your company’s reputation will suffer for it. But we’ll pay the price for trying to implement something we didn’t need in the first place.

  • Dear Management: Letter to DevOps Leaders

    Dear Management: Letter to DevOps Leaders

    My occasional reminder that “Change is good, but predictability is better,” is in the form of a message to DevOps upper management. I thought it would be fun to write, and would offer a different perspective and drive than normal. Hope you enjoy!

    Dear Management,

    Too much of anything is bad. No matter the topic, once too much of it is present, problems crop up. That is just as true in DevOps as any other topic.

    We need some level of standardization, because too many different toolchains is bad. If there is a tool learning curve before team A can assist team B, we are not taking advantage of standardization. In fact, we’re creating a well of technical debt that someone is going to have to clean out. Are we really going to maintain separate toolchains for each project on top of the variety of languages and targets we already have to manage? For years? Is that the most productive use of our time? Do we even have an exit plan to replace that dying open source project that is the core of one of our apps?

    So let’s do some level of standardization on toolchains, shall we? That is the most stable route to standardized tests and easy-to-troubleshoot environments.

    But let’s not get too standardized, either. Because too much of anything is bad, and too much standardization is stagnation.

    We have never seen an environment nor talked to peers and heard them say, “Oh yeaaahh, our environment is perfect! We made everything work smoothly, and all metrics are headed in the right direction while development and deployment have gotten easier.” Those people don’t exist, because in our age of near-constant change, those environments don’t exist.

    In spite of standardization, we need to keep trying new tools, options, environments and policies to increase our quality and deliverability. Give a team the OK to ignore standard X and bring in something they think is better suited to their needs; then see if you can adopt it more widely.

    In short, we want standardization to make our jobs easier, but we don’t want stagnation. We don’t need fifty languages, twenty CI tools, and half a dozen security scanners. We need a few good ones, and the ability to look around at new products/ideas in the space to find better ones.

    In fact, that app that no one is allowed to mess with? Let’s rewrite it, so it is newer and more maintainable; and, while we’re at it, let’s try some of these new and different products and processes. After all, we’re biting the bullet re-writing, we may as well take the extra step, and we might find something useful to the broader collection of DevOps teams.

    And trust us when we tell you a tool or process is the right fit. Ninety percent of the time, the buzzword of the day that you saw on Twitter is either not ready for production yet, or is already in use at our org. It’s our job, so we’re paying attention. And buzzwords are marketing lingo aimed at you, to force us into using something we’ve already looked at.

    You’re doing a good job, we just think we could be doing better. Give us standards, along with room to challenge them a bit. Of course some of us won’t be happy when our tool isn’t the one chosen to standardize on, but we’ll pull together in the end, if you remind us that we’re working to put the org on a more stable and maintainable footing.

    Thank you,

    More of your DevOps employees than you think.