devopsy.kinsta.cloud

Author: Don Macvittie

  • DevOps Future: Have a Break Plan

    DevOps Future: Have a Break Plan

    A big tenet of DevOps is, “Automate all of the things.” It’s generally a good policy, and definitely allows us to meet the delivery goals of DevOps. Taking the full-on, all-new technology case, an org could run full automation on the dev side and deploying with an application release automation tool to a fully automated container management system—fast, efficient, programmable, even portable. This level of automation absolutely enables DevOps.

    Until things break.

    Ignore the well-known and easily overcome cases such as, “That container is no longer responding,” and focus on the larger infrastructure environment. Do you have a plan for getting builds out if DevOps has taken you to many builds a day and your entire Jenkins environment just crashed? The easy answer is, “Switch to another Jenkins server,” but if you are big enough to have more than one, workload may be an issue, as might be the Jenkins environment itself. The usual example of Jenkins for client builds and separate Jenkins environment for server builds is a good example. If the client environment drops, you can’t just move it to the server build environment. Test tools and other tools for clients will have to be moved over, and almost certainly the build environment will have to be modified.

    Along those same lines: What if, instead of a few containers dropping, your container management system fails? (It’s software, it will happen.) What’s the plan then? Depending on the type of failure, it could take your entire application or application portfolio down with it. In most organizations, downtime is not something taken lightly, so what is the plan to return to normal function? Waiting until you’re in the middle of a disaster caused by a single bad disk, failed memory chip or an attack exploiting the infrastructure is a bad plan. We’ve been doing this for decades, yet we know software fails, and sometimes it fails spectacularly.

    Once the first rush of DevOps is through, processes are streamlined and automation is basically in place, it’s fair to take a deep breath and plan for how to overcome the break and total failure of each tool in your toolchain. Yes, those tools will change and you’ll have to do a little work each time you add/change one, but what is your preference: a slightly slower adoption of DevOps after the low-hanging fruit is gathered or confront a failure of infrastructure proportions without a plan?

    Long before computers, Benjamin Franklin is credited with having said, “By failing to prepare, you are preparing to fail.” That credo has been repeated often since. There is truth in that statement with any form of automation. That which is automated eventually goes awry. You can plan for it, or suffer the stress of trying to figure out what to do while it’s an emergency. I suggest having a break plan.

    — Don Macvittie

  • Containers as Disposable Servers?

    Containers as Disposable Servers?

    There is a belief in the space where containers meet DevOps that a service hosted in a container crashing is no big deal, and you can just spin up a new one. It’s not a pervasive belief, but it is common enough that we’ve all heard/read it somewhere.

    I think this view of the world is, at best, incomplete. There are easily verifiable cases where it just isn’t reality. A good example is the Cloudbleed bug, where ignoring errors and spinning up a new instance to meet load demands was one of many things that combined to make a sensational (if low-risk) failure.

    Another good example is shopping carts and payment systems. All the small-business owners I know would balk at a shopping cart of payment processing system that could crash mid-transaction, and the provider shrug and say, “We spun up another instance; have the customer try again.”

    Are there scenarios where crashing containers simply being replaced might be acceptable? I would argue no. They’re software errors. The world has not changed; errors are still bad. I can invent “but if you …” scenarios as well as the next person, but the fact is, if your software is crashing—be it container, VM, hardware, firmware or whatever—you did something wrong and you can’t know the extent of the issues the crashing creates without digging into the problem. At which point, you may as well fix it.

    I’ve written my share of bugs, like any developer. And I’ve fixed my share of bugs, like any developer. Ignoring them is a recipe for disaster, when a seemingly benign bug is suddenly exposed to have major issues.

    Schedule time to fix issues. Rushing forward with new features and products is cool, but keeping the customer base happy and safe is far cooler when revenue time comes. If containers and DevOps are saving you the time they’re supposed to be, you have a little extra time to get in there and fix issues, so use it productively.

    — Don Macvittie

  • ‘Let the Developer Choose the Language’: Too Much of a Good Thing

    ‘Let the Developer Choose the Language’: Too Much of a Good Thing

    As more and more enterprises drive into the DevOps space, this is less of an issue, but you still hear stories that make one cringe. The idea of DevOps (and, by extension, agile) is to get products to market faster and spend less time managing/fixing. Some take the idea to be freedom to choose—and it is to some extent, as long as an organization remembers that it can be (and often is) an increasing management nightmare to pile on product after product. DevOps already demands a fair number of new products to manage, so standardizing those products is imperative to the long-term success of DevOps in the organization.

    Technically, each team could choose its own toolchain, programming languages and data stores in a vacuum, ignoring the rest of the organization. But that is a terrible direction to take in almost every scenario. Yes, we have a lot more choices for languages and environments, and most of DevOps is still in development stages, so there are a ton of products to choose from. But because you can have them all in production doesn’t mean you should. I still see some pundits saying, “Let the developer choose the language.” That’s fine, as long as part of “choose the language” is “with our current architecture as the default.” Five languages or three CI/CD tools or several ARA tools is overhead—overhead to maintain, to keep skills current in and to troubleshoot. And it is overhead that isn’t going away without a project to replace it.

    I’ve written about this before, and yeah, this is the agile/DevOps technical debt. It happens more than you’d think, and yes the product gets to market fast, but if the life expectancy of the product in question is more than a year, you’re going to be maintaining it quite a bit. People who point to microservices or serverless and say, “Each could be in a different language!” have never worked in a long-term maintenance setting, I would guess. It’s hard enough to eliminate redundancies that had to happen in the normal course of business/application growth; making it worse by creating redundancies based on developer whim is just a bad idea.

    That could be my experience talking. I’ve lived through the “No, we can’t do that because the backend is written in (insert obscure toolset), and it won’t integrate with (insert technology du jour)” conversations. Both as a developer and as a manager. It creates future problems that didn’t have to be.

    Dalibor Siroky hinted at this problem in point number three of his “Five Biggest Ways to Fail at DevOps” article, but I thought it needed a bit of an expansion. Again.

    Keep kicking it, bring in new tools that add value, but don’t just build redundant chains because Developer Don is most comfortable with the programming language emoji.

    — Don Macvittie

  • Persistent Storage: You’ll be Hearing More

    Persistent Storage: You’ll be Hearing More

    As containers take over the IT environment, a problem that has existed since virtual machines (VMs) became popular has become more urgent: how to keep persistent storage, and keep it consistent.

    Just as a business designed for foot traffic of 50 to 100 people per hour does not work well when it sees 500 to 1,000 people per hour (think Black Friday in most places these days), storage designed to hold data that persists across instances and even installations often struggles in containerized environments.

    There are a couple of reasons why this is so.

    First, we now treat our servers as disposable resources. If one dies, we leave it dead and simply spin up another. Normal processing keeps the dead server for later evaluation, but many shops do not follow this best practice. The issue has to do with servers that die when writing to persistent storage—what was the state of the write? Did it get interrupted mid-transfer? Luckily, applications can be designed to account for this, and the increasing prevalence of transactional processing in applications that are not based on a database management system (DBMS) shows that people are doing just that.

    Second, the number of server instances hitting those persistent stores is higher in a containerized environment—they are smaller, lighter and faster to spin up and their container density is generally higher than VM density. Contention issues go from zero to full-on when an app goes from “one server is writing to this persistent store” to “two servers are writing to this persistent store.” And often there are hundreds of servers writing to the persistent store. As mentioned, the stop-gap is to implement some form of transactional processing to prevent writes and overwrites.

    Persistent Storage: It’s Got Issues

    The problem is, we end up with transactional processing written into dozens or hundreds of server instances to protect a small number of persistent disks. This is not optimal, and at a minimum increases software complexity.

    That’s why you’ll be hearing more. Here are a few places working on simplified but powerful persistence ideas:

    {code} Rex Ray

    Originally part of EMC, {code} is Dell’s open source group and it has led the charge on the container storage interface (CSI) that facilitates persistent storage. Examples the company has implemented include databases and Minecraft, but the technology works for anything that must be stored on disk.

    Container Orchestration Engines and Cloud Providers

    All of the well-known container orchestration engines (COE) and infrastructure-as-a-service (IaaS) cloud vendors have methods to handle persistent disk—some more transactional than others and some more work than others. Expect this process to become standardized (and possibly for that standardization to take the form of CSI above) in the next year or so. If persistent data is critical to your container environment, it is worth looking at how each solves the problem, as some of the solutions are procedural (such as Docker’s recommendation), and thus are somewhat portable across COE solutions. The weakness of IaaS providers is portability between systems, but that is true of all IaaS procedures/solutions at this time.

    Virtualization Engines

    Both VMware vSAN and Microsoft NTFS have systems for providing persistent disk. While much of this is the simple case of “disk that doesn’t disappear when the container does,” that is the bulk of persistence needs. Expect the support to continue to expand, perhaps even going so far as to implement 100 percent transactional shares.

    Storage Array Vendors

    Storage array vendors have useful tools for this area. While what you use to access their storage is not too terribly large a concern to these vendors, they do have a vested interest in seeing that their storage meets the needs of the changing enterprise. As such, they are delving into methods to help make persistent storage available and consistent. This is not a huge stretch for the vendors, as the systems they produced were always intended for multiple access. Where they lag (and where Rex Ray steps up) is in interoperability/interchangeability of vendor solutions.

    Conclusion

    For those with controlled access to persistent/shared storage, these issues are not that big a deal. For high-volume or high-turnover DevOps installations, though, persistent storage is another link in the chain of reliability that must be protected. Keep kicking it, and hopefully before your need becomes urgent, a solid system will be in place to manage and protect persistent stores.

    — Don Macvittie

  • No-Ops: It’s Been 5 Years

    No-Ops: It’s Been 5 Years

    As I hinted in this post, the state of no-ops is not anywhere near no-ops for most use cases. The closest we have today would be software as a service (SaaS), which handles the majority of operations for us, but still requires some interaction by operations staff.

    The next closest I can think of is the extreme form of hosted. My company uses a hosting provider to handle its website and blog, and the provider does a ton of the heavy lifting. But it would be a fabrication to imply that this mean no operations were performed. The simple fact is that backups, software updates and user management all are in our operational ballpark.

    About That Five Years …

    Why did I mention five years? That’s when the “No-op flame war” erupted. PCWorld has a pretty good summation of it, with links to the blogs in question. The short version is Netflix claimed No-Ops (later it became pretty clear the company just renamed a DevOps-reduced operations group), and Etsy responded with, “That’s because you’re doing it wrong.” Then people jumped on the “no-ops is not a good phrase” bandwagon, and it was … fun to watch.

    But Again, That’s Not the Point

    We too often get caught up in buzzwords. What we want to do is reduce operational overhead significantly per app. This reduces time to delivery, and it frees up time to work on new apps that are useful to the business or spur new business. If we can operationally manage three times the applications because we’ve reduced operations overhead, then we have time to do more. One of the things we haven’t done real well in the past is phase out technology. Perhaps with the free time we could reduce the operational overhead caused by five different solutions to the same problem across our application domain, offering even more time to work on new functionality or product lines.

    Don’t Get Sucked In

    I would argue that both Netflix and Etsy were “doing it right.” You start from where you are, and you streamline both operations and development. You start to ignore boundaries where the ROI is measurable, and you make your environment better/stronger/safer/more agile, whatever the business needs. And then you iterate again. Who cares what you call it?

    No No-Ops

    If No-Ops is coming, it is a loooonnnnggg way off. That’s okay. Because we are improving. You may iterate through improving a zillion times, with different goals each time, implement advanced monitoring and recovery, and wake up one day to realize that you’ve reached “No-Ops.” But as long as you are improving the cycle time/maintainability of the environment you work in, worrying about no-ops is pretty much a waste of time in the current environment.

    But You Knew That

    Here at the end, I’m kind of repeating what most of you already know. But when the hype hits, it doesn’t hurt to have a reminder. Most of you are keeping your business running and cranking out new apps as you’re able, and doing so at a faster pace than you were five years ago while others were arguing about no-ops. So keep doing that. Push Agile/DevOps further on each project. The more time you free from repetitive work, the more you have for doing/learning cool new stuff. So keep cranking, and one day you’ll have apps that are near no-ops.

    — Don Macvittie

  • The Role of Operations in a No-Ops World

    The Role of Operations in a No-Ops World

    It seems that every once in a while, we in high tech get wrapped up in the next thing that will eliminate work and/or jobs. Sometimes, as is the case with DevOps, we get something that reduces redundant work and actually makes our life easier in the long run. Most of the time, however, we don’t. XML was going to end programming, Java would make everyone a programmer, SOAP meant non-programmers could piece together chunks of code and create complex systems … the list goes on and on.

    Today’s trip on the merry-go-round is for Ops and DevOps. The love in some quarters is “No-Ops”—no operations necessary at all! The data center will run itself! But where is there meat to these claims, and where is it just not possible with today’s technology? Let’s take a look.

    SaaS is Today’s No-Ops Option

    If you want a nearly complete no-ops solution today, sign up for software as a service (SaaS). You’ll still have to manage users and access—the normal AAA stuff—but the application is built and managed by someone else. That’s as close to no-ops as we have in today’s world.

    Any other solution requires a fair amount of ops, no matter what we are told. Installing servers—be they in the data center on hardware, in VMs or in cloud instances—still requires work. Even if you have a tool such as Ansible or Kubernetes cranking hard for you and doing most of the repetitive heavy lifting, you are still tying the pieces together and making things work correctly. Networking is often left to operations/DevOps teams, and making servers into cattle requires operations heavy lifting, even if you are that automated. Apps break and troubleshooting is still a largely manual exercise, with automated help in figuring out what went wrong, but the fix still heavily in the hands of operations.

    Hints at an Future Sans Operations

    As is usually the case with these claims of ending some portion of the IT workload, there is some underlying benefit if you don’t get caught up in the hype. Remember that early in cloud some were telling us that there was no longer any reason for a local data center. Of course, there are a billion reasons, and each environment is different. But there is no denying cloud has opened up how we look at server deployment. The same is true for no-ops. There are a lot of reasons that it will not actually be no-ops for an entire project (let alone an entire data center), but there are huge benefits to gain from the idea.

    Serverless

    In serverless, you develop the code behind and API, then expose the API. The system is designed to run the code only when requests come in. Application firewalling is generally mandatory and built into the system. This partially a security feature, and partially the location used to determine what code needs to be called to service a given request. The idea in the cloud is that you only pay while your API is running, and all of the operations parts of the call are taken care of for you. That’s true, if you accept that, in serverless, developers are doing operations as part of development—app firewall configuration, even the simplified version of it that serverless uses, is definitely operations, not development. And troubleshooting between serverless and server-based apps that call the API is going to fall—at least to some degree—on operations. It must, since operations are the people who have network and internetwork knowledge.

    Serverless offers some great options for deployment, though, that undeniably reduce operations overhead. For things such as cloud-bursting, the benefit is pretty undeniable. It makes pay-as-you-grow cloud into a seriously granular proposition, and can eliminate the kind of bottlenecks that all code has by spinning up a new instance for each connection.

    Container Management

    With an increasing amount of functionality folded into container management tools, the ability to auto-scale applications (or, at least, instances of a part of an application) is a big step toward no-ops. Without manual intervention, the container management tool can expand or contract the application’s footprint in response to connections, response times, etc. The built-in routing, when it fits the needs of the organization, limit the amount of networking required to facilitate this scaling also. As time goes on, expect more automation to be folded into container management to make at least the maintenance of applications easier. The advertised ability to perform rolling upgrades on applications while keeping them live is a huge operational bonus too. It’s not precisely no-ops, but it certainly is less-grey-hair-ops.

    Dev/Test/Deploy

    Application release automation, the growth of continuous test/delivery/deployment and configuration automation tools all reduce ops as well. They don’t make for no-ops, but they do facilitate the concepts, reducing operations overhead while advancing the concepts of DevOps. Expect these tools to continue to grow. Most configuration management/application deployment tools work with a selection of networking devices/appliances already, expect that trend to continue. Again, it’s a reduction in ops—not quite no-ops, but a great step in that direction.

    And More

    We’ve looked at a couple of really cool technologies that forward the reduction in operations theme beyond what is listed here. Automation of auditing and full-on automation of security testing (combining code analysis with real time data to get better coverage than today) are just two examples. Keep an eye on the market as these toolsets mature.

    Keep Kicking It

    Meanwhile, Ops isn’t going away. Stay the expert while expanding your skills. As I (and others) have said before, a tool that reduces the hands-on aspects of operations is not a threat, it’s an opportunity. That freed-up time will be useful for all those projects you wish you could do, but didn’t have the time. Or, even better (we can debate that statement later), all of those things the business wishes you could do but you don’t have time.

    In the end, you are keeping the org running, and connected to the world. Embracing the parts of no-ops that make your job easier just makes sense. Don’t fall for the hype; just pick up the good bits, and keep rocking it.

    — Don Macvittie

  • The Cloud and DevOps: Like Peanut Butter and Jelly

    The Cloud and DevOps: Like Peanut Butter and Jelly

    Cloud is practically designed for DevOps. No matter which cloud provider you are using, each has set up systems designed to spin up/spin down on demand, has the ability to run pretty much what you need and offer APIs/command lines to achieve your DevOps goals on the infrastructure side.

    That means that the infrastructure side of DevOps can be handled by careful choice of base image, some scripts to configure network and other supporting infrastructure, and some scripts to install what’s needed once the instance is running and protected.

    Not All Roses …

    The problem is one of breadth and complexity. Yes, cloud is the perfect place to spin up DevOps, at least on the infrastructure side and, once you have a Jenkins instance, on the coding side, too. But the complexity masked by “spin up the infrastructure side” can be daunting. Different sets of APIs for servers, storage and networking, and sometimes different APIs for security versus accounts … It can get snarled quickly.

    So, What to Do?

    What can you do to make life in cloud automation/DevOps easier? There are numerous options available to you to simplify the operational part of cloud DevOps and allow you to focus on discussing issues and resolving them.

    First on the list is the growing amount of automation of cloud automation. Sound crazy? It’s not—just my way of expressing that cloud vendors provide you with APIs and tools such as Puppet are getting better at wrapping those APIs for you, so you can focus on problems, not what is supposed to be the solution.

    This idea is extended by companies that are tying disparate APIs together into a unified whole in one segment or another. One example of this is CloudCoreo, whose tools can do security audits of cloud deployments. The company takes the time to wrap the various security-related APIs and let you pick what is important for your team to monitor.

    There also is an entire market segment growing up of both consultants and VARs that will help an enterprise move to the cloud. Most of these companies focus on one vendor or another, but all share a desire to help make the shift to the cloud easier. By way of examples to get you started, consider Neoxia (in the EU) for GCE, Accenture (in North America) for AWS or Atmosera (in North America) for Azure. (Note: I have never used the cloud services of any of these companies; I’m merely offering you a sampling of what is available. Check your cloud providers’ partners page for more partners.)

    Another option that will reduce, but not eliminate, cloud-based complexity is to use containers to set up the application. The container can then be run on whatever cloud platform suits your needs (or in your data center, if that’s the preference). Any instances spun up still must be protected and given access, but most of the networking, security and instance complexity can be handled at the container management level.

    In fact, if you implement container management and choose for interoperability, a container management tool such as DC/OS can let you choose at deployment time where to put the container.

    Seeing what enterprises are starting to do with containers and entire build/test environments, it is possible that very soon you will be able to just kick off a job, have it build containers that will run your DevOps processes, run tests, deploy to other containers and then go away once the relevant data from them has been saved. The containers used along the way, by taking advantage of a multitarget container management platform, could be anywhere.

    To Sum It Up: Cloud is Good for DevOps

    The issue at hand—as it has been for DevOps and cloud for a while—is the rate of change. Here we’ve discussed a cross-section of ways to make DevOps in the cloud easier and more complete. But no doubt we’ve missed some great ideas and products, because there is so much going on.

    The point is, tools are coming available to make it easier. Don’t write a billion miles of scripts that are useful for only one provider if you can avoid it. Look into these options (and others that will crop up) to lighten your load of technical debt moving forward, and make getting DevOps roaring in the cloud easier.

    And as always, keep rocking it. The change has been constant, and you’re still cranking. That says a ton.

    — Don Macvittie

  • With DevOps Culture, It’s Mind Over Matter

    With DevOps Culture, It’s Mind Over Matter

    There is quite a bit of conversation about tools versus culture, and sometimes, those constantly involved in the back-and-forth forget that not everyone is so steeped in the finer points of the conversation.

    Tools is the easy part. There are a ton of tools available to optimize IT operations, from small bespoke products such as the Antsle Private Cloud Server to CA’s broad suite of Dev/Test/DevOps/ARA tools. There are plenty of places to find lists or charts of tools, and I’ll leave you to them, just mentioning the two I’ve written about in the not-too-distant past: Xebia Labs’ Periodic Table of DevOps, and CA/Automic’s Continuous Delivery Landscape.

    My youngest just built a Cube-solving robot, so this picture inspired me.

    But what of culture? When pundits and old hands talk of culture, what exactly are they talking about?

    Well, it still varies a bit from organization to organization, but there are some definite underlying foundations.

    Increased Cross-Team Communications

    Even if an org is full-on DevOps, there is still the requirement of cross-team communications to make sure everyone understands what is going on.

    One of the things that this does not make abundantly clear, I will belabor a bit (sorry, old hands). Communications is not just talking. It is not just email exchanges, it is not just sitting in another team’s meetings while researching your current problem on your phone.

    It can be these things, but it is a breadth of options. People are using Slack channels to keep everyone updated, common notifications or even pub-sub systems to make certain information about the state of IT, operations and builds are getting wide distribution. Increase communication, not chatter.

    Organizational Change

    You hear that a lot, and it can mean complete overhaul of an IT organization, but in an enterprise setting, it rarely does. The reason is simple. There may only be one or two Cisco specialists, or one or two EMC specialists, and these people don’t properly belong on one or two product/project teams in the long term. The same goes for security and a whole further swath of operations team members.

    So in the enterprise, what “organizational change” often means is moving those responsible for development and core deployment closer together on small, product- or portfolio-based teams. Networking staff, security staff and infrastructure staff will join the teams as needed, but are not part of the core team.

    Management Change

    This is another often spoken-of mantra that easily can be misunderstood. In short, managing a small, agile, entire product lifecycle team is not the same as managing a pool of resources that pop from project to project and are either (a) not involved in the operations portion at all, or (b) not involved in the development portion at all.

    For a great set of perspectives on this topic, I’ll recommend our partner The DevOps Institute. It has an eBook and a certification course (DevOps Leader) on this topic.

    Decentralization, Yet Increased Oversight

    This is often a confusing pair of messages we send when talking about what DevOps culture change does. It really is not that complex—pundits just assume everyone already knows, and, honestly, those knee-deep in DevOps do know.

    Decentralization, as taught in the DevOps Leader course mentioned above, is about driving as much decision-making as possible down to the team. When teams are built around a product or group of products, then those with the most pertinent information about what is best for the product quite often can be made within the team, speeding decision-making and implementation.

    Increased oversight is an acknowledgement of two things. First, there is more information available in an agile environment, and that information is available more frequently. Standup meetings offer some, while milestones offer other data. Second, tools are starting to allow rollup of information in a uniform fashion for CIOs, CTOs and business leaders to look at. This means they can quickly see the status of projects they have a vested interest in and provide feedback into the process. To go from, “Oh, this roadmap item is going to take a lot more work than we thought,” to, “Great, we have a plan, let’s keep moving,” is shorter when there is more information available to the decision-makers on both the business side and the IT side.

    There is no doubt plenty I have missed, but these seem to be the core things that those of us discussing this topic a lot assume others know, and that those just looking into DevOps really don’t. Hopefully it will help someone push through the clutter and come to a decision on DevOps in general, or culture changes in particular.

    — Don Macvittie

  • Sometimes It is More Than Culture, It’s About Gains

    Sometimes It is More Than Culture, It’s About Gains

    Almost since the beginning, there has been a tug-of-war between whether better toolchains or increased communications were “real” DevOps. The answer to that question is, Yes.

    Better tools undeniably speed the delivery process and streamline operations. Anyone who claims differently is not living in the real world. Jenkins alone can make your development world a better place, and it is but one tool—with hooks into like a million more, but still one tool.

    Increased communications speed the delivery process and streamline operations, if they are frank and honest. A developer who knows that his choices for included dependencies can make life easier or more difficult on maintenance is far more likely to include that consideration in their choice, for example.

    Gains Are In the Doing

    But the rock-star gains that some shops have seen from DevOps? Are a bit of both, and something else. The sudden improvement of 4x delivery frequency most often comes from no longer avoiding that thing you don’t talk about, or the legacy piece of software that eats an increasing amount of time just being kept alive because it’s necessary.

    And I would argue that is the best place to start. I’ve touched on this tangentially before, so I thought I might be more direct.

    You have it. That thing in the process that is a bottleneck. No one talks about it. There are reasons—some because a few years ago you tried to resolve it and couldn’t; some because the bottleneck is a natural outgrowth of some requirement; some because “it’s always been that way,” But there is that step (or more often three or four steps) in the develop/test/deploy cycle that consumes a huge chunk of time to do little work.

    There are also those tools/processes so old that they consume an inordinate amount of resources. Most shops more than a few years old have them. I’ve worked in a couple of 100-plus-year-old companies and frankly, these bits of outdated software were part of the genotype of the org because they’d been in IT so long.

    Manual processes are known to take time, simply because they’re manual. Development itself, even when broken into smaller pieces and implemented piecemeal in the manner of Agile, still takes time. Testing, too, is still (or again, depending upon your background) a largely manual process. And security checks are also in a ton of shops.

    These are known. And if you’re approaching the problem from the “find the bottlenecks,” I would exclude them. Better tools or more automation can help all three, streamlining who does what can help test and security, but these aren’t the places that I’m thinking of.

    Aging Support Software

    I worked in a shop once that spent an inordinate amount of time keeping a “database” alive that was in existence before RDBMS. Yep. There was mission-critical software that used the database, so it was maintained as well as possible internally. Internally because the vendor no longer existed.

    Several analyses were done, and each time it was decided that the DB couldn’t be gotten rid of without a major rewrite of the systems it supported. But as time went on more and more resources were poured into those systems and the database. Eventually, it was forced from above that the database had to go. The estimate to do so came in at an insane number and the project was dropped. Meanwhile, an increasing amount of resources were being poured into the DB and the systems that supported it. All were outdated, and all required special skills that were also outdated. Soon the org was paying exorbitant rates for the few people who could work on the system to do so via contract.

    Finally, with hard dollars attached to the preservation of this system, a forced upgrade was a viable project, and it went forward.

    This is all pretty standard enterprise stuff, it’s kind of the way IT has functioned in the past. But had IT management gone to business units with improvements that could be offered during the upgrade, all of the resources poured into not upgrading over the years could have been used for new product/feature development. Identifying the fact that replacement was inevitable and wouldn’t get cheaper for waiting is the type of outdated software issue I’m talking about.

    Bottlenecks

    From the bottleneck side, there is often a “this monster log file must be reviewed by a human” step on the way to production, too. Think about simple automation. What is the human scanning for? Talk to those responsible and see if they can offer guidelines concerning how they would train a new person to review the file. Then write a small script to do it—not as a final solution, but to show that a person doesn’t have to if the review is well-bounded. People won’t trust it, so run it in parallel for a while, and learn from what it misses, point out what it finds. Long term, eliminate the delay of someone pouring over logs looking for the same things over and over. Take time to make it possible. It won’t be a ton of time, but it will reap gains. Just talking to the team that must accept the log output might reap gains.

    There isn’t a need to immediately install 15 DevOps tools or reorg all of IT before starting your DevOps journey. Just look for the places resources are being invested to little long-term gains, or where the process is held up in a manner easily eliminated. And keep rocking it.

    — Don Macvittie

  • Finishing the Pipeline: Monitoring, Tools, Automation and …

    Finishing the Pipeline: Monitoring, Tools, Automation and …

    It is true that DevOps has grown to be the beast in the room. I’ll co-opt the “cloudwashing” term to call it “DevOps-washing” because it seems every tool you’ve ever used to develop and deploy applications now has a DevOps angle to it. For example, the Jenkins update center is kind of overwhelming. While I can’t be certain, there’s a chance that our new coffee maker has a plugin for Jenkins.

    But it’s also true that DevOps really is coming of age. The success of Jenkins is largely based on CI/CD, and the tools built both up and downstream from it are folding in more and more of the DevOps process, making life easier—if not exactly easy—for IT.

    Even with platforms, the choices are growing and the automation is, too. Whether one of the major cloud vendors, where deployment can be fully automated or hyperconverged infrastructure and where deployment can be fully automated, we’re moving toward a future of not caring about physical infrastructure but rather, “Can I get a connection to it?” Of course, it’s not that simple, but running Docker or Kubernetes on hyperconverged infrastructure means we have on-demand deployment, and the cloud does much the same.

    Tooling for monitoring is also getting better, whether you are monitoring user experience or a particular API’s CPU usage. For the most part, we are not yet to the point where automation can deal with the majority of alerts from monitoring, but that boundary is being pushed daily. We certainly can tool our systems—both the infrastructure and the app sides—to offer us more data about application health than we’ve been able to get in the past. From physical hardware all the way up through containers we have a wealth of options for monitoring, and application monitoring is getting even more attention than infrastructure monitoring.

    Heading Down the Trail: Monitoring and More

    Which leaves one wondering where this is bound. Let’s assume that the areas that are a bit behind—like over-arching network configuration from ARA tools—catch up. They will, and that’s pretty obvious, so I’m trying to remove the short-term roadblocks to talk about what we might be planning for and facing moving forward.

    If we can assume a place to deploy that has reasonably unlimited capacity—such as the cloud, where cost is the limit, or hyper-converged, where capacity is a limit but expandable in chunks—and we can assume that ARA is running the Dev/Test side, then pairing it to a preferred server configuration (be that VM or container, depending on app, group, company, etc), and pushing out the deploy, then we’re left with the monitoring piece, and automating the responses to monitoring issues. That is growing, too.

    Right now, the standard in the DevOps world for a server that is malfunctioning in some way is to kill it and start a new instance. This often masks real-world problems, and if it appears to “fix” the problem, reduces the priority of resolving what went wrong. This is, in a nutshell, the real error for Cloudbleed. They were seeing servers spew errors, when one appeared to core, they simply started another. It allowed a simple programming error to persist over months and become a security issue.

    So the next great leap will be to find a quick way for monitoring to determine exactly where problems are occurring, and force a developer to fix it. There are tools that can auto-generate bug reports, but they’re rudimentary at this point in time, and we need them to be more advanced. There will also always be the trade-off of how much in resources you use for monitoring when the point of the app has nothing to do with monitoring. And like so many things, the hidden cost of keeping that monitoring up to date will be a technical debt—one of many that the DevOps community seems slow to acknowledge.

    There Always Seems to be More

    On the “time we waste” side, the fight over internal/external needs to end. The fact is that internal cloud isn’t going anywhere anytime soon, and if you’re running in containers, you don’t have to care. My company is currently looking at a tool that can run hundreds of containers in a very (physically) small space. We can deploy to the cloud or deploy there, as long as we have container templates. It doesn’t matter what the business need forces us to use; containers really will run pretty much everywhere. It does no one any good, harping about internal/external. A deployed app is a deployed app.

    Now, talking about the differences in networking between internal/cloud, external/cloud, and container manager (regardless of where it sits): That is a discussion worth having.

    — Don Macvittie