devopsy.kinsta.cloud

Tag: software complexity

  • Why We Still Need Specialists

    Why We Still Need Specialists

    A few years back, I wrote here that specialists—particularly infrastructure specialists—were not going anywhere.

    I think it is worth updating that prediction and reminding you all that specialists across the bulk of DevOps subdomains are still necessary—even if you have broken down their silos.

    Spend time in any of the variety of places that DevOps folks hang out and listen to/read what they actually say. When they express their sentiments as a real-time activity, the idea that we made one team responsible for most, if not all, of an application’s life cycle seems rather brutal.

    In most industries, we have increasing specialization as complexity increases. Even if it was allowed, most carpenters are not qualified to do plumbing or electrical work. Oh, most know the basics and could do it in a pinch, but they aren’t the expert that the plumber or electrician is. IT, for the last many years, has tried to do the opposite, with a decent rate of success. But practicality wins out over rigidity to make that success.

    We still have specialists. And you should ignore people that suggest they need to go away. The situation is already bad enough without having a person you can turn to for help with a harsh issue in this language, or on that piece of networking gear. While some of this—language specialists are the easy example—is merged into a DevOps team, some of them—network admins, for instance—only partially are. The networking within/between containers is often handled by the DevOps team (using one set of specialties), while the networking that allows all of the organization to run is still handled by operations. But I can’t think of an org I deal with regularly that doesn’t house these specializations somewhere.

    This is one of the things that has traditionally held back low-code/no-code solutions—they assume these specializations and will require specialists in whatever the underlying technologies are when things inevitably go wrong. The same is true (to a lesser extent) of DevOps, AI and automation. Someone has to know what is going on underneath—they must have specialized knowledge—in order to fix it when things go wrong.

    Remember, you are asking your DevOps teams to be both Jack of all trades and master of some. That is a recipe for failure, but DevOps is designed to withstand or overcome failure in short order, so I only mention this so you understand why new hires seem wild-eyed at times and old hands seem jaded—there is a lot going on that they’re mastering and remastering.

    So, stand up if you’re one of the “sailing against the wind” crowd. Be proud you’ve managed to thrive in an environment with a ton of specialist parts and make sure you know who to ask for help when something goes wrong and you end up out of your depth. Even if the answer to “Who can I ask for help?” ends up being “Learn it for yourself and then be prepared to help others,” you’ll rock it either way and the rest of us who have struggled through it feel your pain and respect your work. So, thanks—for those who’ve never experienced it, but rely upon your work.

  • Organized Architecture

    Organized Architecture

    I’ve mentioned IT pendulums plenty of times before, but today I’m going to use them to explore a different angle on IT: The age of containers and microservices. It’s time to ask yourself if the current architectural design of the overall product offering of IT is the best one you can afford to offer.

    Agile and the whole “developer is king” movement brought us a whole lot of independent app teams running off and doing their own thing. While the core of that movement was short-lived, we got a lot of good and bad out of it. The good included breaking the traditional “We change at glacial rates” process that some IT shops had—because a team absolutely wants the tools that will make their job easier, not the ones pre-approved by committee or whatever process a given org used. And the bad included the very things those of us who’ve seen this type of thing play out before warned about. Suddenly, we had critical apps and infrastructure relying on tools that one dev used in his hobby development. Sometimes that was good, and for sure it happened before Agile came along, but now we had an avalanche of it. Stuff that hadn’t been through security; often code/libraries/apps that no one outside the team (or the individual) knew was running in the organization. Some of that stuff is still happening.

    And then we hit true uptake of microservices. That drove us to break things up in non-traditional ways. A lot of good came of that process, but a lot of complexity did, too. Complexity shared across applications with a huge spike in API calls and operations with a spike in deployed processes and routing. Perhaps the worst thing to come out of the combination of the two is “Who cares why it failed? Just spin up another!” That movement is mostly dead (thank goodness), but it only died because of the number of high-profile “Oops” issues that it caused. It allowed attackers to go unnoticed; it caused paying customers to spread security details across the web; it caused users to suffer crashes that would have been fixed in any sane development environment … and some of those are still out there. Cases where there’s a script to restart an app or service every X hours because it crashes randomly. Not acceptable. Make all the excuses you like, but if you don’t know what’s causing the crash, you’re wrong to leave the app or service running.

    And we introduced a huge amount of complexity. Now we have double the infrastructure—one physical, one virtual (container or cloud). We have applications spread across a lot more systems (be they instances or physical systems) and often across platforms; even across data centers. We live in a very cool time, with some sweet stuff going on. But complexity is a very real problem we need to solve. One app/app portfolio may have limited complexity, but we still allow teams too much leeway in defining the environment the application lives in. We need to offer a happy medium—a world where there are options, but those options are defined, manageable, and secure.

    So we’re likely to see a resurgence in the importance of enterprise architecture teams again, with their focus being on organizing the mess we’ve made. Indeed, I know this is underway in a couple of Fortune 500 companies I stay in touch with. And I know of one organization that isn’t doing this and has all their UI functionality running through a single container … for multiple applications. That is no better architecture than each using a completely different UI toolset. So, I expect they’ll want coordinated solutions also. Just so something like that can’t happen again.

    You all are doing it, day in and day out. As I often say, just don’t let this bit drag you down. Confront it head-on, approach standardization with “Yes, where it is good; no, where it is not” as your credo. Security is the big one, but maintainability is another. Even with tools like Ansible at their disposal, managing 50 different environments is a bit much for ops if it could be done with five. And keep rocking it. Through a pandemic, meeting increased demand, giving users more and more. Thanks once again, from all of those users who don’t know to thank you.

     

  • Some Technical Debt is Self-Resolving

    Some Technical Debt is Self-Resolving

    But don’t count on it.

    Fellow devopsy.kinsta.cloud contributor Bill Doerrfeld recently wrote, “Enterprises Face Growing Technical Debt,” a great article about trends in technical debt, and it got me thinking about some of the things that have been going on in the industry over the last few years.

    As I’ve mentioned before, the changes underlying our systems combined with the need to keep systems that existed before those changes alive are making for quite the debt ratio, and that debt is coming due, probably sooner than you think.

    We’ve got enhanced ways of tracking how much time technical debt is consuming built right into our CI/CD tools, but there are inherent weaknesses in those tools. The ability to track time spent on issue X, and to categorize issue X as debt, requires a solid categorization process and relies on staff to keep close track of their time. Right or wrong, the industry has currently (maybe permanently) moved away from tracking time in small increments and, instead, most organizations track results. But unless you know the percentage of time spent keeping a system that is outside your main-line development toolsets alive, you can’t determine how bad technical debt really is.

    There are some bright spots though, and while I (and others, to be sure) have been sounding the alarm about technical debt since DevOps teams were all allowed to pick their own toolchains (glad that’s mostly over with), we need to be aware that counter-forces will help alleviate technical debt issues, also.

    First is the trend of making things simpler. Let’s face it, on the DevOps side, Python became popular at an insane pace. There is a very good reason for that. Shell scripts written by master scripters to achieve a goal are impossible for even good IT folks to follow. What you can’t follow will take much longer to fix–because you have to break it apart, figure out the parts, parse through massively complex pattern matching, etc. Python simplified all of that. If you write a Python script to do the same thing as a bash script, a far higher percentage of IT will be able to glance at it and go, “I know what this does.” This trend exists across IT (bug dashboards are another good example at a much higher level), and as easier-to-understand tools are increasingly part of the mix, the old and arcane will be less of a debt.

    Second is that, while I have raised warnings about things like the growth in API usage and CI/CD toolset complexity, the fact of the matter is that both of these things reduce the amount of code that we, internally, are maintaining. Steps automated, be they by calling someone else’s API, or calling your CI/CD vendor to handle build or deploy steps, are less steps for us to worry about (as long as they work, of course–have trusted partners).

    But we still need to be aware of technical debt, because all of the things we’ve mentioned in the past—the increasing complexity of our environments, staff turnover, team-specific tools and languages—are driving up debt.

    Check with your CI/CD vendor to see what their offering for tracking debt looks like. Other measures that don’t require tracking actual time spent on individual activities include keeping a tally of the number of bugs. And a surefire way to get a great feel for where your weaknesses are is looking for the software no one wants to work on, or the software that someone warns the new people not to touch. That’s where your debt load is the heaviest. IT is full of tinkerers;”If it ain’t broke, don’t fix it …” only applies in IT if “…because no one here knows exactly how it works,” also applies.

    Ask your teams to list which tools in their area (be it silos or DevOps teams) need to be updated and/or replaced, and then to rank them. Then take that data to team leads, and ask them to decide where your limited hours and money should go. Provide data on relative debt that you gathered above, so a comparison of team rankings to debt cost can be made.

    And then, set aside time to alleviate debt. It doesn’t have to be massive, just keep moving forward on it. You set aside time for all sorts of things, even a few man-hours a week on debt will multiply. It will start slow, but once you are spending less time each week fixing “product X,” some of that time can be added to technical debt resolution, increasing what can be done. Look for low-hanging fruit; a small app or script that eats up an outsized amount of time, or a system that is insanely expensive to keep, while the market has moved past what it actually is used for in your department, etc.

    Then enjoy the fruits of your labor, and put those astounding skills and that extra time to use rocking even more new apps for your users.

  • But Wait, There’s More! The Trouble With DevOps Tools

    But Wait, There’s More! The Trouble With DevOps Tools

    DevOps tools today increasingly are no longer out-of-the-box ready for use. And that’s an aggravating issue

    Imagine you bought a new power tool—a sweet battery-powered drill/screwdriver that did everything you needed.

    Imagine it looked as though it ran on its own, but when you got it out of the box, you discovered that it didn’t come with a battery.

    Imagine you got the battery, and only after the battery was installed you discovered that it needed a specialized wrench/chuck to install and remove bits. And you had to go buy that.

    Imagine you got the chuck and were all set to go, only to discover that it would slip and normal drill bits wouldn’t turn unless you had a specialized adapter.

    toolsThat’s what our tools environment in DevOps has become. And we need to demand better.

    If your software requires X and Y and Z to be useful—whether you are commercial or open source, whether X and Y and Z are a million bucks or zero—state it before install.

    We’re setting up an entirely new environment, stretching things a bit for an internal project, and honestly, the amount of “Oh yeah, this is required, too” that we’re finding out after install is insane. We have done our research; we should not be getting surprised by “But wait! There’s more!” type installs.

    For an industry that insists on plug-and-play for all OSes and every odd bit of hardware ever dreamed up, we sure let software steamroll us. Vendors need to be clear about what will be required to run the tool and what will be required to run it effectively before we waste our time. Because there are some dependencies we’re not willing to accept, for usability reasons and/or security reasons. And if, under the covers, the tool does exactly the same thing as another tool … then what, exactly, is the value-add? It better be good.

    We’re just one DevOps group—and a small one at that—but we have decided for this project not to use tools that dupe us into thinking they’re usable out of the box, then immediately require us to install something else. Or at least we’re trying. This has gotten so prevalent that in a couple of areas, we’re struggling. After all, the software still has to be built.

    It’s a good thing drill manufacturers don’t think that way. Maybe there is hope.

    Meanwhile, we’ll keep trying to do what you all are doing: kicking rear and keeping systems running.

  • DevOps Introduces Complexity. Really

    DevOps Introduces Complexity. Really

    The larger the project, the more resources are generally dedicated to it. That is true of the vast majority of fields of human endeavor. Consider building the San Francisco Bridge versus building a crossing over a stream. Or World War II versus the Spanish Civil War. The resources dedicated scale with the size and complexity they need to support.

    This is true of IT projects also. A four-line script does not require the attention that a highly available web app with dynamic data does. It is a pretty simple concept, we all are aware of it, and methodology doesn’t much change it.

    The problem is that we are often terrible about acknowledging this concept when it comes to internal support projects. We don’t see a compiler as a tool in need of regular maintenance and locking down, and largely it isn’t. There definitely are cases where the newest compiler/interpreter should be implemented, but except for massive change such as Python 2.x to 3.x, there isn’t a driving need to have the newest and shiniest.

    It’s Complex—and Growing More So

    What happens when we apply this relaxed attitude to DevOps? A mess. The reason is simple: The automation side of DevOps is generally implemented with tools. Lots of tools. Tools that require maintenance, troubleshooting, interfacing to other tools … It gets complex fast.

    And where on the calendar is time for these things set aside? Almost always the answer to that question is “nowhere.” But these aren’t compilers and interpreters. If your Bugzilla DB goes bad, you have a major problem. Since compilers and interpreters are generally read-only—changes aren’t made to them except to upgrade—it generally requires a disk-level error or serious human failures for them to stop working as planned. That’s not true for things with constantly changing datasets. And many DevOps tools have constantly changing datasets.

    In an increasing number of organizations, the DevOps infrastructure should be listed as the largest application portfolio the organization manages. And that is where you need to be.

    Make DevOps Tools a Project

    It is that simple. Tooling is a massive bit of overhead that seems to be not so big because it streamlines the development/deployment process. In short, the tools save enough time that their cost seems smaller than it is. But that cost will not go away, and once the pipeline is fully automated with a stable toolchain, keeping it running is a cost IT will have to continue to bear. Don’t just treat it like something you can check into when it fails; give it the same treatment that you use it for. Make certain you have backups—if possible, make your DevOps environment totally dynamic—so a failed instance of X can just be replaced by a new copy. You can’t “just spin up” complete and accurate datasets, but the backups fill that void.

    It’s More Than Just DevOps

    Stay on top of tool updates and improvements, just as you would for any mission-critical software. Because once your software factory is dependent upon the DevOps toolchain, it is mission-critical software. Learn how changes will impact the environment you are using the tools in.

    And it’s more than just the tools themselves. The DevOps environment is highly complex with a lot of variables that can have a negative impact. The move from RHEL/CentOS 6.X to 7.X is a good one. Sure, it’s better by objective standards, but there is enough change that it breaks a lot of things. Mostly the change in networking, but libraries and tools also. In fact, I helped someone work through inter-process communications (IPC) issues caused by the upgrade. This is a stumbling block for the DevOps team in a RHEL/CentOS environment, even though the DevOps team may not be the one choosing to upgrade.

    It’s All Good, Though

    So there is a cost. The cost is not small. But the benefits still far outweigh the costs, and I’m not arguing otherwise. What I am arguing is that it needs to be a cost that is accounted and planned for. “When it breaks, we’ll fix it” is too relaxed for the toolchain that will run your software factory. Once it’s right, take the time each week to keep it right. Set up the policies, processes, and time to keep it clipping along smoothly.

    And keep kicking rear. This is just a cost of doing business, much the same as database maintenance. Make sure you’ve got the time set aside, so your other projects can keep moving the org forward, and all is good.

    — Don Macvittie

  • Software Complexity Driving DevOps Automation

    Software Complexity Driving DevOps Automation

    The drivers behind increasing software complexity also are the drivers behind expansive DevOps automation. As the saying goes, “Software is eating the world.” DevOps is gobbling up software, too, engulfing it in practices that are necessary to make it thrive under the pressure of the current environment.

    That environment and those drivers are of keen interest to many. DevOps authority Andy Vitus, partner at Scale Venture Partners, has identified four drivers in particular that are creating the software complexity pushing DevOps automation.

    Multiple Distributed Software Locations

    According to Vitus, by moving software from one server to multiserver to multiple data center deployments, the industry has made software more elaborate. Applications now execute in a distributed manner and in parallel using multithreading inside multiple processor cores; among many servers running multicore processors; and in and between geographically dispersed data centers. This makes software more powerful and reliable, though it demands more DevOps automation to produce it.

    Simply incorporating the ability for software to run in parallel using multithreading adds intricacies to the development process. To keep both the workload and the number of man hours needed during development manageable, automate as much of the additional coding duties and testing as possible.

    “Running applications across servers and data centers requires that you ensure consistency in the data layer,” which will then need new databases to handle the load, says Vitus. Depending on the database, the developer must rewrite the software to be able to use it.

    The entanglements don’t end there.

    With mobile applications, mobile devices and back-end servers both store data. “You have to be a lot more careful about knowing what version of the data is the true one, unless you’ve got a synchronization layer built in,” says Vitus. Either way, programming annoyances can be introduced.

    Getting Lost in Translation

    Expanding development from a single programming language to multiple languages, even for the same project, has made software more convoluted. In legacy development shops, programmers typically used one language for everything regardless of whether it was the best language for their creations. These days teams are applying choice languages such as C++, Python and Go to the tasks to which they are best-suited. “This increases the complexity of security and patch management as you use six, seven or eight different languages all with their own vulnerabilities,” says Vitus.

    While adopting several languages certainly adds to patch management and security headaches, understanding how each language can and must be compatible with the others for them to work together is an entirely new challenge. “The ways that they handle variables and time stamps changes over time, and you have to ensure that different services can still talk to each other,” says Vitus.

    Stacks of Stacks

    Moving from a simple three-tier stack to a distributed microservices architecture catches software development in another snag, Vitus says. The database layer has grown to include in-memory and other database types. As development breaks the application layer into microservices, each potentially with its own data store, the two layers combine to create this new stack.

    Millions of Endpoints

    Finally, moving from deploying servers to deploying servers, mobile and Internet of Things (IoT) takes software down new paths of mostly uncharted territory, says Vitus. “Today you deploy software changes to a back-end server that must talk to a mobile client using iOS or Android or to IoT devices such as dishwashers and light bulbs in smart homes,” he says. “You could be deploying code to millions of devices.” This certainly affects the construction of the code for use on so many different platforms.

    Conclusion

    These drivers help create a magnified need for DevOps automation. Considering the costs of hiring engineers and technicians to do the same development work manually, the world will be watching companies that are pushing for increased automation to displace those labor costs. “To the extent that you can offload some of that to regression tests, you end up saving, and the cost of building a new software product goes down,” says Vitus.