devopsy.kinsta.cloud

Tag: container management

  • It’s Funny How We Forgot About Container Sprawl

    It’s Funny How We Forgot About Container Sprawl

    …Because it sure as heck did not forget about us.

    We are (thankfully) past the days when everything was a container. At least, most of us are. Just like anything else, some orgs will continue that approach for reasons both great and terrible. And all of us still use far more containers than we need, I would say. “One container, one function,” works if you are talking about one whole application, and maybe sometimes for busy parts of an application, but it certainly isn’t needed to host a couple of low-use APIs, for example. My favorite example is separate instances of the same exact application for different teams (not business units – that is a good method to separate business units).

    Thankfully, technology is once again saving us from ourselves and our misdeeds. Processing power is no longer an issue, meaning we can spin up a million containers and keep them running until our public cloud bill arrives. So, it’s not about processing power. I cannot remember the last time that an org was bandwidth-limited, so it’s not about the network. The biggest issue is still security. More containers means a bigger footprint, which inevitably creates more vulnerabilities. Right behind that is container inventory management. What is running where, what images are available and what state are they in, which running containers have drifted from their definition, etc.

    I’m a developer that has sometimes done support for the apps I develop. That means that I laugh at your “Our images do not drift because we’re agile and build-driven.” Of course they do, because troubleshooting is real. The real question is, can you get them back into alignment when the ticket closes? The DevOps toolchain is pretty darned good at that if you let it. Changing a running image and telling no one is a terrible idea that still happens nearly every day across our industry. But technology has brought us drift monitoring. We can be notified of changes as they happen. For those in highly sensitive spaces like FinTech, changes can be blocked in production, but for most of us, we want to know when it is happening and who made the change. And we can get that today.

    Technology has also brought us some wonderful scanning tools from SBOM to SCA to Registry. These tools can give us a list of things for security to worry about in our images. In order to reliably generate these lists, we still have to know what images we are keeping and where—but choose a reliable registry or two, fit them into your DevOps processes and scan on commit. Simple and thorough—as long as everyone uses them.

    Thanks to policy and regulation,  container management tools can offer you a detailed list of not just what instances they’re hosting, but metadata like what image they came from. That helps manage inventory, and by now we are all building hierarchies of images from base to application specific.

    So, we can tell what is running, what images are available to run, how far running images have drifted, what known vulnerabilities exist in both our images and our instances and how compliant we are WRT containers. Dunno about you, but it seems technology has our back yet again.

    You are cranking it out, keeping the lights on, and dreaming of more you could be doing. Use the tools we already have to make it easier and the resulting systems more secure and reliable. You won’t regret it. And once again for those of you whose users never say it: Thank you.

     

  • What We Don’t Know is Dangerous

    What We Don’t Know is Dangerous

    Pop quiz: How many database management systems (be they RDBMS or NoSQL or even flat file) do you have running in your overall organization? Not actual databases, but management systems?

    How many containers do you have that are exposed to the internet, directly or indirectly? Not container management systems, because you’re only running a couple of those and they’re all relatively new, but actual containers?

    The questions could go on and on. Not too long ago, because there was an identifiable cost associated with it, cloud inventory management became a popular thing – finding and removing cloud instances, services and storage that were no longer necessary became a part of normal operations for organizations that had even a mid-sized cloud presence.

    But now, we have sprawl all over the place. So let’s extend inventory management to all software, and get rid of what isn’t needed on a regular basis … because it does indeed build up quickly.

    Yes, there are a few companies out there trying to do just this, and I encourage you to check them out. I have not, because our inventory at Ingrained Tech is manageable without that level of overhead. Our development and ops time is limited, so we keep a tight rein on what is out there and the maintenance it requires.

    But for larger organizations, the more you have out there, the larger attack surface and potential data sieves you have in place. So make sure if it isn’t needed, it’s gone. In the age of containers, this actually requires a fair amount of investigation; first, a list of all containers part of an ongoing project (and in even a mid-sized company, “They’re all together for each app,” won’t create this list because random containers are out there, too). And then a list of all containers not associated with a known project that can be gone through and usage determined.

    It’s not glamorous. It’s a bit of a slog. It’s in the organization’s best interests, though, and the slog should motivate your teams to put a policy in place to manage container life cycles. Like notes in Docker files that say when the container is no longer needed, or what it is needed for, so future you can easily get rid of it without stress.

    If you are in a containerized environment, this problem is growing almost by the day. Seriously. On our last project, we had 12 active containers before we pared it down to the five we needed long-term. The others were R&D along the way or tools we were using before focusing on a different method of achieving the same goals. Since we’re a small team, we just cleaned up when the project was coming into focus, but a larger org needs to be more focused on cleaning things up as they go.

    Meanwhile, those assets are letting customers get to your org, and allowing employees to manage the systems. Keep rocking it, and just remember to pick up, particularly after cutting-edge new projects, but also occasionally as just a normal cleanup step.

  • DevOps and Containers

    DevOps and Containers

    “Works on my machine!”

    We’ve all had that co-worker. I have had one at pretty much every company I’ve worked for. At several jobs, it devolved into a running joke.

    There is a belief that containers alleviate this issue. And they may—or they may simply mask the issue. The choice is up to IT.

    Inevitably, something that works on one machine and not another can be traced back to on-the-fly configuration changes. If “we use containers for DevOps” means that developers have full control of the container environment, then it must also mean that any configuration changes be codified into a documented container image that can be spun up repeatedly and even recreated in a Dockerfile if needed.

    If, instead, it means that devs have a playground in containers and, once they get it going, they can chuck it into production—even if you don’t say it that bluntly—then you are making things worse with containers, not better. At least in the days of “works on my machine,” the issues had to be resolved because the only time you heard that phrase was if something didn’t work on the machines it needed to be deployed to!

    As a developer, it is tempting in both container and virtual environments to upgrade a package or tweak a setting without documenting it. But a clean DevOps environment needs to be able to build the product from Dockerfiles, not from a pre-existing image. Yes, you can just clone the existing image, but then there is no documentation of what has actually been changed and it takes a lot more work for those following you and debugging future problems.

    So take the time; keep an up-to-date Dockerfile (or whatever equivalent script is used to build your containers). It is not that significant an investment to stop for a minute, open a file and type a few lines. And it pays off in spades if, in the future, someone is trying to figure out what the heck is going on in there. Consider it self-documenting configuration. I’m not a fan of “self-documenting code,” but for config, it is pretty clear if you go, “Oh, we needed version 10.2, not version 10.1; let me update the init script.”

    There are a fair number of shops out there that modify things running in production, and the world hasn’t ended yet—though that may well be how it happens. So, a good practice that will help those trying to follow behind you is clearly not a necessity. But each of us has gone back after a year and said, “Why did we do that this way?” which was (or wasn’t) then answered with, “Oh, here is exactly how the environment was configured.” But it might help answer the ‘why.’

    Your apps are powering the world—or your specific apps are powering your little piece of it. And you’re kicking it just keeping them running. Now, go make them better! Consolidate changes, manage configuration through Dockerfiles, Ansible, init-scripts, whatever combination you use … And keep rocking.

  • The Role of Containers in Business Transformation

    The Role of Containers in Business Transformation

    Containers hold new promise for business and digital transformation. Yet many differences exist between monitoring containers and monitoring other elements of infrastructure, such as VMs, storage, memory and compute. Understanding those differences is vital for ITOps managers as they navigate their businesses’ digital transformation.

    Interestingly, the differences are as much technical as they are organizational. Companies need to explore these differences and consider questions such as, how to ensure that a container-based project performs when it goes into production, and what changes you must make to infrastructure to ensure you have enough compute and storage power to support the containerized applications your development team is going to launch.

    The essence of container management is to be prepared for the changes coming from application development, which means understanding resource and infrastructure as it is today to estimate the effect of those changes tomorrow.

    Here are four differences ITOps managers need to consider with container management as they move their digital transformation journeys forward.

    Containers Versus VMs

    The signature feature of containers is they can hold and run an entire application all in isolation. VMs offer operating system level isolation, but at the cost of a much larger footprint; to begin with, each one has its own copy of the entire operating system. VMs are also far less portable than containers, especially when moving among private and public clouds and data centers. As a step toward digital business transformation, most companies are looking for DevOps tools and for ways to deploy their applications to the cloud, whether for QA or production. Organizations rolling out new applications are also gravitating to a container-based architecture from the start.

    Advantages of adopting containers include:

    • Agility: Instantiating a new container goes as much as 20 times faster than instantiating a new VM. Startup is measured in seconds rather than minutes and once deployed, containers scale up and down rapidly in response to load.
    • Density: Most containers are megabytes in size rather than gigabytes. With their smaller footprint, containers allow for as much as five times greater density than VMs.
    • Cloud-readiness: Containers can be deployed to any cloud (private, public, virtual or physical) which opens up the possibility of using the same containerized application in hybrid and multi-cloud environments.
    • Blast radius: As software developers incorporate and release changes more quickly, they look for ways to control potential damage from bugs. A problem in a container is less likely to cause widespread damage and is easier to isolate and repair than a problem in an entire VM or physical server.
    • Cost savings: Every VM requires a hypervisor and an operating system (OS), which can involve licensing and fees. Containers run side by side, sharing the OS kernel of a single physical machine while holding isolated copies of specific library versions they depend on. Architecturally, VMs tend to contain more components and require greater allocation of resources. Containers, on the other hand, tend to hold a specific component or a single micro-service, minimizing resource overhead.

    Containers, DevOps and Real-time Monitoring

    A company adopting DevOps to speed up deployment of its software products automates as many development processes as possible and relies heavily on automation scripts to move code along, from continuous integration through continuous deployment and continuous delivery (CI/CD/CD) to production. Along the way, multiple developers must check in on new changes. Automated processes, such as health checks and unit testing, take place and then the build is deployed to the next environment, such as integration or QA. With each set of changes, ITOps needs the answers to several questions:

    • How will this affect performance?
    • What will be the impact of the changes when we deploy?
    • Do we have enough compute, storage, memory and networking to support the changes?

    In a traditional environment, developers have plenty of time to answer those questions. But if they’re deploying modern, containerized, cloud-based applications, that window tends to be much smaller which can lead to the kind of guesswork that makes ITOps uneasy. That’s where container monitoring comes in. When ITOps can see real-time and historical analytics of containers and their hosts, across physical, virtual and cloud environments, they can create performance benchmarks to help them make informed choices about infrastructure.

    Microservices Get Added into the Mix

    Microservices are loosely coupled and independent services, each with their own release schedule and lifecycle. Developers are able to make small changes to elements of these applications which allows them to mitigate the risk of negative business impact by reducing the blast radius around those changes. In case of failure in a single change, there’s less to repair and less impact on surrounding, dependent services. We’re seeing developers move away from packaging a single application per container to packaging a single service of an application per container. This allows them to make changes more frequently and keep up with business demands.

    Multiple Silos

    The goal of DevOps is to bring application development and traditional ITOps closer together. However, container monitoring and management changes things. Traditionally, areas such as storage, network bandwidth, VMs and physical hosts that require monitoring have all been in the ITOps silo, an area that developers don’t typically own. Developers write code and build software in the application development silo and then hand the software off to the ITOps silo to run. However, things have changed and now developers aren’t only writing code, but also owning and using containers in the application development silo. Once the software and containers leave this silo, their consumption of resources like compute, storage, network bandwidth and memory will also affect the ITOps silo. That’s why monitoring and management is as necessary for containers as it is in areas like storage, VMs and physical hosts.

    While container technology may be new for the traditional enterprise, the impact it will have on business transformation cannot be ignored. Containerization brings increased simplicity, consistency and portability to production environments, enabling faster IT delivery and increased performance, arming businesses with a new agility and allowing them to rapidly respond to changing customer needs. In fact, according to 451 Research, application containers will be a $2.7bn market by 2020.

    The businesses that embrace this opportunity will be taking a vital step towards transforming into bustling digital enterprises that can deliver innovative products and top-notch customer experiences. To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Yinghua Qin

  • The Role of Operations in a No-Ops World

    The Role of Operations in a No-Ops World

    It seems that every once in a while, we in high tech get wrapped up in the next thing that will eliminate work and/or jobs. Sometimes, as is the case with DevOps, we get something that reduces redundant work and actually makes our life easier in the long run. Most of the time, however, we don’t. XML was going to end programming, Java would make everyone a programmer, SOAP meant non-programmers could piece together chunks of code and create complex systems … the list goes on and on.

    Today’s trip on the merry-go-round is for Ops and DevOps. The love in some quarters is “No-Ops”—no operations necessary at all! The data center will run itself! But where is there meat to these claims, and where is it just not possible with today’s technology? Let’s take a look.

    SaaS is Today’s No-Ops Option

    If you want a nearly complete no-ops solution today, sign up for software as a service (SaaS). You’ll still have to manage users and access—the normal AAA stuff—but the application is built and managed by someone else. That’s as close to no-ops as we have in today’s world.

    Any other solution requires a fair amount of ops, no matter what we are told. Installing servers—be they in the data center on hardware, in VMs or in cloud instances—still requires work. Even if you have a tool such as Ansible or Kubernetes cranking hard for you and doing most of the repetitive heavy lifting, you are still tying the pieces together and making things work correctly. Networking is often left to operations/DevOps teams, and making servers into cattle requires operations heavy lifting, even if you are that automated. Apps break and troubleshooting is still a largely manual exercise, with automated help in figuring out what went wrong, but the fix still heavily in the hands of operations.

    Hints at an Future Sans Operations

    As is usually the case with these claims of ending some portion of the IT workload, there is some underlying benefit if you don’t get caught up in the hype. Remember that early in cloud some were telling us that there was no longer any reason for a local data center. Of course, there are a billion reasons, and each environment is different. But there is no denying cloud has opened up how we look at server deployment. The same is true for no-ops. There are a lot of reasons that it will not actually be no-ops for an entire project (let alone an entire data center), but there are huge benefits to gain from the idea.

    Serverless

    In serverless, you develop the code behind and API, then expose the API. The system is designed to run the code only when requests come in. Application firewalling is generally mandatory and built into the system. This partially a security feature, and partially the location used to determine what code needs to be called to service a given request. The idea in the cloud is that you only pay while your API is running, and all of the operations parts of the call are taken care of for you. That’s true, if you accept that, in serverless, developers are doing operations as part of development—app firewall configuration, even the simplified version of it that serverless uses, is definitely operations, not development. And troubleshooting between serverless and server-based apps that call the API is going to fall—at least to some degree—on operations. It must, since operations are the people who have network and internetwork knowledge.

    Serverless offers some great options for deployment, though, that undeniably reduce operations overhead. For things such as cloud-bursting, the benefit is pretty undeniable. It makes pay-as-you-grow cloud into a seriously granular proposition, and can eliminate the kind of bottlenecks that all code has by spinning up a new instance for each connection.

    Container Management

    With an increasing amount of functionality folded into container management tools, the ability to auto-scale applications (or, at least, instances of a part of an application) is a big step toward no-ops. Without manual intervention, the container management tool can expand or contract the application’s footprint in response to connections, response times, etc. The built-in routing, when it fits the needs of the organization, limit the amount of networking required to facilitate this scaling also. As time goes on, expect more automation to be folded into container management to make at least the maintenance of applications easier. The advertised ability to perform rolling upgrades on applications while keeping them live is a huge operational bonus too. It’s not precisely no-ops, but it certainly is less-grey-hair-ops.

    Dev/Test/Deploy

    Application release automation, the growth of continuous test/delivery/deployment and configuration automation tools all reduce ops as well. They don’t make for no-ops, but they do facilitate the concepts, reducing operations overhead while advancing the concepts of DevOps. Expect these tools to continue to grow. Most configuration management/application deployment tools work with a selection of networking devices/appliances already, expect that trend to continue. Again, it’s a reduction in ops—not quite no-ops, but a great step in that direction.

    And More

    We’ve looked at a couple of really cool technologies that forward the reduction in operations theme beyond what is listed here. Automation of auditing and full-on automation of security testing (combining code analysis with real time data to get better coverage than today) are just two examples. Keep an eye on the market as these toolsets mature.

    Keep Kicking It

    Meanwhile, Ops isn’t going away. Stay the expert while expanding your skills. As I (and others) have said before, a tool that reduces the hands-on aspects of operations is not a threat, it’s an opportunity. That freed-up time will be useful for all those projects you wish you could do, but didn’t have the time. Or, even better (we can debate that statement later), all of those things the business wishes you could do but you don’t have time.

    In the end, you are keeping the org running, and connected to the world. Embracing the parts of no-ops that make your job easier just makes sense. Don’t fall for the hype; just pick up the good bits, and keep rocking it.

    — Don Macvittie