devopsy.kinsta.cloud

Tag: Portability

  • Build Once, Deploy Anywhere

    Build Once, Deploy Anywhere

    This late in the game, it seems stunning to me that I have to write this, but there are plenty of you who aren’t there yet—so let’s try a different approach.

    A huge part of DevOps is reducing redundancy. We have cut the fat until it hurt, then refocused and cut some more. There is a blog here about burnout, but we’ll save that for another day. The point is, we don’t do the same thing over and over unless there is a quantifiable benefit—like rebuilding and testing new iterations or spinning up base images to start a new project.

    We insist that our applications run in as standardized an environment as possible, with images from a repository and standardized software installed.

    And then we promptly put it into a unique environment where deployment is completely different. Yes, I’m talking about which cloud or which data center; if it’s deployed via a data center, then hardware, virtual machine or container? The work to standardize the important bits and leave the less impactful bits to teams to decide has been astounding on the Dev side of DevOps (for those new here, what I call DevOps), but have been far less effective in DevOps. Even companies that claim to be pure cloud generally have two or more cloud vendors, each of which is doing things differently.

    So, with all of that said, I can say what I should not have to say after all of these years:

    Make containers standard. I don’t care which flavor and management tool you use, but add in that portability layer. It will protect you from a future where your preferred hosting environment (be it cloud, local or hosted) becomes hostile or overly expensive—or simply ceases to exist. And it will standardize the rest of your DevOps processes. Yes, your preferred cloud vendor offers Kubernetes, and no, that’s not sufficient. Kubernetes offerings today are like Microsoft in the 1990s. Embrace and extend to build the lock-in barrier higher. You do get some great functionality from the cloud vendors I pay attention to, but processes become non-standard. Better to limit platform-specific work to “spin up a server” and have everything else be reusable whereever you are deploying. The benefits here are far-reaching; there will be configuration and security issues that are different when you move from cloud to cloud or cloud to data center or data center to cloud, but the core of DevOps will remain standardized. There are tools to help with baseline cloud security and risk management, so those in addition to using Kubernetes as a portability platform will make current work less onerous, and massively increase portability from deployment location to deployment location.

    You’re short of staff. It does not take a rocket scientist to predict that this situation will not change. Adopt the things that will reduce your workload—like DevOps standardization on Kubernetes (or OpenShift or whatever) as a deployment platform—and keep rocking it. You can’t wave your hand and get more staff or hours in the day, but you can implement containers as the sole deployment option to save time. And use those hours to solve other company problems—because that’s what we do, solve problems for the business. And keep smiling, because you’re rocking it. If you weren’t, they’d find someone who could.

  • Next Up: Portable CI

    Next Up: Portable CI

    One thing that we’re very good at in IT is breaking new ground, coming up with ideas that work astoundingly well, getting broad adoption, then standardizing them for interoperability. When a new technology (looking at you, cloud) has problems with standardization and interoperability, we find a way around vendors’ reluctance to give you freedom of choice. For cloud, for instance, we found another great technology – containers – that greatly limits our lock-in to a given cloud vendor’s fare. Of course, whether standards or workarounds are the answer we arrive at, there are always wrinkles. Vendors are certainly not motivated by interoperability; they want your regular cash infusions, and interoperability could be more of a drain on them than a feed of new customers. Particularly in today’s environment, where vendors sometimes think that doing “to” customers is an acceptable substitute for doing “for” customers. We have to work around things like “embrace and expand,” which is still very real, and deploying containers to cloud A versus cloud B still has a bit of vendor-specific work, but we have enough to get it done.

    As continuous integration (CI) has become critical to an increasing number of organizations’ IT processes, the time has come to do the same with CI tools. Standardization is a good first goal, but it is more difficult here than in many areas, because the topic is so broad. An app might be built in one tool and tested in another, or it might be built via the use of shell scripts while tested with a UI tool, etc. It is not an easy solution to build, but it is an idea whose time has come – particularly with the cloudification of the space. We now have full-on CI systems that are in the cloud, and which hook you in with custom instructions. We’ve already had one large DevOps vendor essentially tell their entire user base, “We’re moving you to the cloud, unless you pay these inflated fees,” and others that never offered options for traditional architectures at all.

    Yet, we know apps move from in-house to cloud (and though it is uncool to mention, move back), and organizations are motivated by cost containment and mobility more than by what their chosen vendor wants them to do.

    So, it is time for some bright company or OSS team to put together a standard and a tool that implements said standard. It will be a lot of work, to make portability between CI tools that came from widely disparate roots – some from build, some from test, some even from deployment. But we’ll be needing it. A while back, everyone was a Puppet shop. Now, it seems, everyone is an Ansible shop. In simple terms, everyone will inevitably be a [something else] shop, because nothing lasts forever. So, a tool providing CI mobility between tools will definitely have a market – smaller today, but eventually large. We don’t need Yet Another CI Tool, we need that portability.

    I’m sure it’s coming. History does offer glimpses of the potential future, and we’ve seen this cycle before. I’m just saying, the time is now, not five years from now. We can target the OS of our choice and the cloud vendor of our choice, using containers to simplify the differences in host OS/vendor. The ability to use the CI tool of our choice is the logical next step.

    I haven’t heard of a project or company working on this yet; if you have, drop me a link in comments or email (I prefer comments, so others can look at your suggestion too, but whatever works for you), and I’ll take a look. It would be cool to support this kind of effort, when it eventually comes about.

  • Languages and DevOps: Traditional

    Languages and DevOps: Traditional

    For quite a while now, all the cool kids have been going on and on about “full stack development,” “API first,” and a ton of similar methodologies/practices. And yet, a huge swath of applications are still targeted at a single system to do a specific job. There are fat clients, there are internal servers, there are standalone apps. All are still in development, and the hype machine doesn’t account for them much, though they’re pretty pervasive.

    We still have to maintain them, or write yet another. They’re not going anywhere, because there are some problems for which they are the right solution. So let’s talk about them.

    For clarity, I’m defining traditional as, “Targets a specific OS/hardware platform, and is not primarily web-based.” We’ll exclude mainframe, because the toolsets and integration with DevOps are still different enough to warrant their own blog.

    Online forums are full of people trying to re-target traditional apps (perceived Windows 10 issues caused many to want to move from Windows, and the growing Linux trend of making distros into beta – or even alpha – testers has many wanting to move from Linux), while others are just trying to keep them running as the world changes underneath their feet.

    Easy Integrations

    While these dev tools are the ones that Agile grew up around, that makes them well-integrated into Agile and DevOps. The feeds to/from compilers and various build tools are the same as other spaces we’re looking at (or, more precisely, the others are the same as traditional), so integration of AppDev is generally not an issue. Integration of Ops is harder, but exists. The various DevOps tools with an emphasis on operations (I like to call it devOPS as opposed to DEVops) are available to traditional development, and you can certainly deploy through them, but they’re like the second cousin to “full stack.” Good Enough rules the day, and a little work will be required. Thankfully, tools like Ansible and Puppet smooth the way a bit for deployment, not caring much if you’re deploying a traditional app or a web service, as long as you tell them what to do to successfully deploy.

    You got to move it, move it…

    Portability is increasingly becoming an issue. The rise and fall of “preferred” Linux distros, the costs of being a Microsoft shop … there are a lot of reasons to want to move your app around. The best first step to this (and the only step some will need, IMO) is containerization. Put it in a container, and your underlying hardware can run on whatever OS the corporation settles on, while your apps will run in containers on the OS they need. For most organizations, this will only be a stop-gap; a way to quickly move servers away from one OS and onto another, but the apps will have to move fully to the new OS, eventually. Microsoft has Windows Subsystem for Linux, which can help move from Linux to Windows, and Linux has Wine to help run Windows apps on Linux. Both have come a long way toward interoperability, but neither is perfect – or even easy. It will take a bit of work, but far less than rewriting for the target OS top-down. Tools like Cygwin are also available to get your Linux app running on Windows if you don’t want to use Microsoft’s solution. In the end, you are going to have to modify code, but you are porting an application, so no surprises there.

    Consider the future

    A place I worked in the late 90s/early 2000s had an OS/2 machine that was critical infrastructure. We were not allowed to touch the box. I think if the admins had their way, we wouldn’t have been allowed to look at the physical box. Of course, we virtualized it just as soon as we could in a step similar to my “containerize” advice above. Now is a good time to consider writing your apps with an eye toward portability, so you can remove this type of system from the equation. That RHEL 5, or Windows XP app should really be modernized or eliminated at this point. The tools for cross-platform development are out there, from Mono to Java to Xojo; platform-dependent bits can be isolated for rapid retargeting. It’s up-front work that pays off in spades when the time comes to change the targeted platform.

    You. Rock.

    Whatever your language of choice, in the end, it will likely be your build tool that links it to DevOps. Your biggest concern for DevOps will probably be portability in the near future. Meanwhile, keep rocking it. You’re doing the work that needs doing, even if it isn’t the part of DevOps everyone is currently buzzing about.

  • Portability Shmortability: Most Applications are Going Nowhere

    Portability Shmortability: Most Applications are Going Nowhere

    I’m a huge fan of the idea of taking our super-duper applications and deploying them where it makes most sense. I always have been. Those of us in application development spent forever bemoaning the fact that our applications weren’t all that portable between operating systems. In fact, looking back through the history of computers, you find there has always been one roadblock or another to stop portability.

    First there was byte order and instruction set, and we’ve largely overcome those. Internationalization wreaked havoc on us all for a while, and we overcame that. We’re still pretty mightily struggling with OS-dependent things such as windowing systems, but browser-based apps have largely overcome those also. Java takes a lot of heat for a lot of reasons, but it did improve portability of apps written in it. Maybe not “Write once, run anywhere,” as was often claimed—things such as file locations were still OS-specific—but still a lot closer than compilers that were deeply tied to the operating system they ran on. Virtualization and containers have enhanced portability in another dimension, letting the developer take the OS with the app.

    We have overcome a ton of issues, and advancing technology has introduced a ton more. Now we have to either choose mobile targets or find a tool we like that can generate code for wildly different mobile systems from a single source. DevOps really isn’t helping with this process—though it will, eventually. By automating the build process, the target will be switchable at the targeting end, meaning our app—given a tool that can multi-target—can be built for whatever mobile platform the cool kids are playing with today, and the ones that are considered “cost of entry” can be generated in the same run.

    So far, so good, right? We find problems, we overcome problems, DevOps helps us overcome them faster with a wider selection of outputs …

    And then we hit the data.

    Vendors in a variety of spaces would love to paint you a picture where it doesn’t matter. It does. The mass of your data, wherever it resides, is a pull like gravity to new application development. And it should be.

    Some will try to explain to you that in the age of APIs, it doesn’t matter where the data actually resides: You can get it.

    But it really does matter, and this is not Utopia. Even if you ignore the performance implications of data being latency-infested miles away from applications that need to use it non-stop, those APIs are only a viable option if the data can be secured and protected. By definition, an API is an advertisement that whatever data/service it fronts can be found right here. For applications that traditionally are data center apps, it might be possible to expose your customer file or other critical data via API and then lock down access to the API by IP …

    But then we quit ignoring the latency problem. If every bit of data an app needs has to flow from San Francisco (or Tokyo, or Bejing, or London, or wherever) to Seattle before the calling app can do what it needs to and then has to be sent off again to Albuquerque (or Rostov, or Madrid, or wherever) the delays will, by definition, be more than they would be if the app sat with the data and had to transfer over a 1GB (or 10GB) connection between servers before sending it to users.

    This means most data-heavy apps, no matter the level of agile or DevOps, aren’t likely to move. Vendors with a stake in the process will trot out the one or two customers they convinced to make such a move to tell you how great it is, but performance math and security sense tell you it is only great until things get busy or that exposed API is used by bad actors to vacuum user data.

    Some applications have smaller datasets and can move the data with the app. Other applications can be architected not to care where the data is. A few don’t need a database at all. Move these applications to whatever platform makes sense. Make the CD for them retargetable.  Most serious business applications rely too much on core data to fall into these categories. And thus, they aren’t moving, unless someone comes up with a viable way to move a ton of data between platforms that is safe and fast and effectively translates where necessary.

    So, that’s what we need to see next in the march of portability. We currently have containers that can be run pretty much anywhere and languages that will work on pretty much any platform. We need data tools that can alleviate the current set of non-portability/latency constraints and get data where it is needed, when it is needed, securely.

    Unfortunately, I don’t see that happening soon. So addressing the issue, “Where do we target our new application?” will have to start with the question, “Where is the bulk of the data it needs?” At least for now.

    Most of you knew that, because you’re living it. I just thought you might have a use for a third-party article about it. Because someday, you will have this conversation with someone who isn’t living it. Enjoy.

    — Don Macvittie

  • DevOps and Data Portability

    DevOps and Data Portability

    You have a classic car. You’re in the middle of restoring it, and it looks mighty sweet. Paint is polished, wheels are stock, hubcaps (or spokes) are shiny, engine is completely restored to original spec.

    But it has no gas tank.

    A bit of a stark comparison, but that’s where we’re at with application release automation (ARA)—at least in terms of portability. And where we’re likely to stay.

    Hold on, I’m a Fan

    Before the DevOps aficionados get out the torches and pitchforks, let me be completely clear that I’m a huge fan of ARA, and explain again why.

    Tools such as Jenkins allow DevOps teams to automate the entire build/test stack—or, as much of it as their organization requires. That is a huge step forward, for all the reasons we’ve covered in the past, including the one I think is often overlooked or underemphasized. With frequent build/test feedback, the code is still fresh in the developers’ head when they learn of problems. Most of the time, they are fixing before moving on to the next task, which is a huge plus compared to waterfall, where many problems show up long after the developer in question has forgotten what the code is doing because they’re off on other things.

    Likewise, tools such as Puppet, Ansible and Chef (to name a few) allow DevOps and security to set standards via script, and use those scripts to deploy over and over. This has all the benefits we’ve discussed, too—speed of delivery, reduction of human error, repeatability that becomes stability, etc. This gives operations the ability to answer, “Is server X set up to do this new thing?” by looking in deploy scripts and quickly updating things if the answer is no, but X will be required going forward. That’s one of my favorite bits of Application Provisioning.

    Riding the Gap

    For those unfamiliar, ARA simply bridges the gap between DEVops and devOPS. Tying code being released to platforms and configurations it will be released under. It has the ability to rapidly re-target a release to a new environment, and it can do predictable and repeatable releases, offering the benefits of application provisioning for the actual release segment. Most tools do this by tying together your favorite build tools and provisioning tools, and giving Ops a control panel that can manage the process end-to-end.

    The benefits are huge. Particularly for Operations and Security (who have another place to standardize and insert essential monitoring), but also for development because it offers one further step in testing and by doing so reduces the surprises six months down the road, when you’ve forgotten what

    Bubblegum += 10

    Is in your code for.

    So don’t take my intro as offensive. I am a fan. But I’m a realist. There is no panacea.

    It’s About the Data (as Always)

    The two weakest parts of most ARA systems are server provisioning and data issues. Server provisioning is setting up OS/hardware, and many ARA systems tend to assume you magically did elsewhere, but this problem is only an issue in highly physical environments and there are tools to handle the problem; they’re just not integrated well with most ARA systems. Data issues are the bigger problem.

    It’s cool that we can drag and drop a couple boxes, and magically switch our deployment from inside the DC to a cloud provider in the time it takes the ARA to build/deploy. But we all know that’s not how the world works. Now you have an application, complete with database that is correctly configured and security that does AAA solidly … and no data.

    “Just move the data, then!” the choir sings. No, that’s really not how it works in an enterprise. Data itself is an object of security and compliance, and some source databases are so massive that moving them is a nearly impossible chore in and of itself. Amazon tips its hat to this fact with its import/export service.

    This was the reason why I gut reacted to the “No more physical data centers!” crowd back in the midst of the hype with an eye roll. The amount of data stored in data centers is staggering, and often not portable at all. Add to this the fact that such data isn’t going to reduce, and we have a quandary.

    What About Moving Forward?

    Sometimes, it is necessary to accept that the current state is not changeable, but take steps to improve the future state. I don’t see this as a reasonable approach in this case, because the rules of data aren’t changing in a way to change future state in a positive way. We’re generating more data every day, and that data will end up stored somewhere. Be careful where you store it, think long-term strategic. Having ARA doesn’t magically make that data portable, so make sure that where you store it today, while it’s small, is where you’ll want to store it in five years, when it is a massive, largely immobile asset.

    What to do … What to do

    Use ARA to multitarget all of those apps that are portable without terabytes of data, and make sure data generation/collection/analysis apps are on the architecture you will want them on moving forward. ARA offers a ton of other benefits, so data portability isn’t a deal breaker—or even a significant negative, since you probably already knew some datasets weren’t going anywhere—but don’t expect miracles out of it. It can’t make your large datasets portable, so don’t expect it will. And use it to target the platform your organization feels is most appropriate for apps that will generate or collect a lot of data.

    — Don Macvittie