devopsy.kinsta.cloud

Author: Don Macvittie

  • Next Up: Portable CI

    Next Up: Portable CI

    One thing that we’re very good at in IT is breaking new ground, coming up with ideas that work astoundingly well, getting broad adoption, then standardizing them for interoperability. When a new technology (looking at you, cloud) has problems with standardization and interoperability, we find a way around vendors’ reluctance to give you freedom of choice. For cloud, for instance, we found another great technology – containers – that greatly limits our lock-in to a given cloud vendor’s fare. Of course, whether standards or workarounds are the answer we arrive at, there are always wrinkles. Vendors are certainly not motivated by interoperability; they want your regular cash infusions, and interoperability could be more of a drain on them than a feed of new customers. Particularly in today’s environment, where vendors sometimes think that doing “to” customers is an acceptable substitute for doing “for” customers. We have to work around things like “embrace and expand,” which is still very real, and deploying containers to cloud A versus cloud B still has a bit of vendor-specific work, but we have enough to get it done.

    As continuous integration (CI) has become critical to an increasing number of organizations’ IT processes, the time has come to do the same with CI tools. Standardization is a good first goal, but it is more difficult here than in many areas, because the topic is so broad. An app might be built in one tool and tested in another, or it might be built via the use of shell scripts while tested with a UI tool, etc. It is not an easy solution to build, but it is an idea whose time has come – particularly with the cloudification of the space. We now have full-on CI systems that are in the cloud, and which hook you in with custom instructions. We’ve already had one large DevOps vendor essentially tell their entire user base, “We’re moving you to the cloud, unless you pay these inflated fees,” and others that never offered options for traditional architectures at all.

    Yet, we know apps move from in-house to cloud (and though it is uncool to mention, move back), and organizations are motivated by cost containment and mobility more than by what their chosen vendor wants them to do.

    So, it is time for some bright company or OSS team to put together a standard and a tool that implements said standard. It will be a lot of work, to make portability between CI tools that came from widely disparate roots – some from build, some from test, some even from deployment. But we’ll be needing it. A while back, everyone was a Puppet shop. Now, it seems, everyone is an Ansible shop. In simple terms, everyone will inevitably be a [something else] shop, because nothing lasts forever. So, a tool providing CI mobility between tools will definitely have a market – smaller today, but eventually large. We don’t need Yet Another CI Tool, we need that portability.

    I’m sure it’s coming. History does offer glimpses of the potential future, and we’ve seen this cycle before. I’m just saying, the time is now, not five years from now. We can target the OS of our choice and the cloud vendor of our choice, using containers to simplify the differences in host OS/vendor. The ability to use the CI tool of our choice is the logical next step.

    I haven’t heard of a project or company working on this yet; if you have, drop me a link in comments or email (I prefer comments, so others can look at your suggestion too, but whatever works for you), and I’ll take a look. It would be cool to support this kind of effort, when it eventually comes about.

  • Languages and DevOps: Mainframe

    Languages and DevOps: Mainframe

    In true DevOps fashion, I nearly forgot to cover the mainframe after saying it warranted its own blog!

    There is a lot going on right now in the mainframe space, so the question of mainframe language choices has several different answers. We’ve got Linux running on the mainframe, we’ve got cross-compiled open source capable of running on the mainframe natively, we’ve got mainframe-based containers and we even have traditional mainframe tools being ported to other platforms (hello, COBOL on Linux!) so that development can happen on smaller platforms but target the mainframe.

    Taken in the order they came available, we have traditional mainframe development, in languages like COBOL and RPG, right on the mainframe in standard LPARs. This has gotten easier over the years, but is still basically the same development environment that was used in the 1980s, with prettier UIs.

    Then we have Linux LPARs available that allow for development right on the mainframe. There are issues and differences from other Linux development environments, but not as many as you’d think. This approach gives a broad selection of development tools on a Linux platform, making it an appealing choice.

    Next came containers, which can then run Linux in a container with the container offering the interfaces that Linux expects to find. This approach has the benefit of being more like development on other Linux platforms, but interaction with core services can still be platform-specific at times. This approach offers the breadth of open-source languages that came naturally with containers plus the few that have been ported over to z/OS.

    Then came speed porting. The biggest weakness of the above options was that it took too long for vulnerabilities to be addressed and new functionality to be moved over to the mainframe. The fact is that porting a language took time, and wasn’t as rapid as the response for the core Linux kernel. There has been a movement to improve the dev tools that create the ports, and now response time to CVEs or improvements in the kernel is much better. A side effect of this movement has been increased availability of languages and tools that are used by DevOps to get things done.

    Finally, we have recently received support for traditional languages on other Linux platforms. Most notably, given the number of lines of code out there, COBOL is now officially supported on Linux, meaning you can develop on the desktop without a ton of overhead, and then deploy to the mainframe. This has the effect of enabling DevOps processes and mentality for more traditional languages. This is still new, and the differences between developing on one platform and deploying on another will have to be worked out, but it is a step in the right direction.

    So what should you use?

    Mainframes are generally a more controlled environment than most DevOps teams are used to. The work required to support and update a given language, keeping it and its libraries up-to-date is more complex than just a generic Linux system, and that gives rise to very focused shops in the mainframe space. Generally speaking, you will be told what to use, or more to the point, which languages are supported. If you get input into the process, recommend Python, as it has the biggest pool of developers and libraries out there at the moment. But chances are, the mainframe team already has a language in place that they want DevOps teams to focus on, and in that case, use what’s available.

    No matter which language you end up using, API-wrapping mainframe functionality is a time honored tradition at this point, make the mainframe do the heavy lifting and return just what your application needs. Think of it as a big, smart database that can pre-process results for you at high speed, in high volumes.

    And keep rolling.

    Mainframes have been around forever. They’ve been pronounced dead dozens of times, and yet the security, stability and locality of a mainframe keeps them in the mix. Data gravity helps too – the data is there, the routines to do processing on the data are there, handing that data off to another platform for presentation isn’t that big a deal, and the mainframe just keeps ticking along. Because (at this point) generations of developers have made it tick. Keep that tradition alive if you’re in a mainframe shop; they are literally the core of systems in the places they are in heavy use.

  • Languages and DevOps: Recommendations

    Languages and DevOps: Recommendations

    In light of the points brought up in these articles, what is my advice for which language to use on a given project? The same advice I have always given development teams: Use the right tool for the job. My father was a woodworker, and drilled that concept into me. I brought it to development, and it is true. The difference is that “the right tool for the job” is a much more complex question in an existing IT environment than it is in woodworking. Whether you need a chisel or a screwdriver is immediately obvious. Whether Python or Go is the right tool … not so much. Things to consider:

    Does the tool have a broad developer base?

    This is the most straightforward question on this list. Can you get people to work in the language? I worked at a place that still had RPG III code running on a mainframe. We used to joke that those two retired developers in Australia were getting rich from maintaining it – and we still did some new development in the language. It was a bad choice that we moved away from. Maintaining the old while developing a new platform made sense. Packing any new functionality into a tool about which few knew the ins-and-outs, and even fewer were being trained for? Made no sense.

    Does the tool specialize in, or have modules/add-ons that customize it for your target?

    While at the 50,000-foot level, you can theoretically do any computing job in any language, that doesn’t make your preferred language a wise choice. Making certain the tool supports what you need to do is paramount. This is the closest to “right tool” thinking – does it do the job?

    Is the developer base growing or shrinking?

    While this is more important for longer-lived projects, you do want that developer base mentioned in the first point, above, to still be around. This extends to add-ons, also. If you choose ASP.NET today, will it be a solid choice tomorrow? (I’m not saying it won’t, here, just picked one framework to illustrate the point with).

    Does the tool do what you want, or will you be working around faults to get there?

    Honestly, you can – and people have done it – write websites in C. That doesn’t make it the right choice. You’ll end up spending more time getting around the fact that C is ill-suited to this task than you will developing the actual site. Assess the language with regard to how well it actually does the job you need to do. This is a different take than the second point, above – this is the “just because you can, doesn’t mean you should” part.

    Does the tool fit in your environment?

    This is the one that far too many struggle with. New development teams want to bring the tool they’re most comfortable with, because it reduces development cycles in the short term. The issue here is simple. It is far easier to manage an environment that makes a conscious effort to keep our burgeoning number of tools, languages and libraries as controlled as possible. Introducing Go to develop servers, just because someone is excited about it, makes no sense if the entire environment is Node on the server side. Use the environment you have so you can maintain things more easily. Everything is more thoroughly vetted – from which add-ons to include through security – with the language an organization uses the most; take advantage of that. Start with, “We use Java in 70% of applications,” and then deviate from there only if you are moving toward a goal of moving off of Java (again, not saying you should, just an example).

    As a New DevOps Person, Which Should I Learn?

    Not so long ago, the ubiquitous advice to new developers was, “Learn Java!” It was everywhere, doing everything, even when it wasn’t the best tool for the job.

    Similarly, that answer is now “Learn Python!” Of all the languages out there, it is the one used the most. I do not have data to back the following statement up, but I will bet most Dev people have used Python on one or more projects already, and a fair number of Ops people have, too. If Java is the past, Python is the future.

    That said, check out companies where you would really enjoy working and see what they’re doing. Generally speaking, they’re not looking to hire Python developers if they are doing 70% of their coding in Go. I can’t recommend tailoring your learning to one or two organizations’ environment(s) (unless you already work at one of them), but consider the popular languages that have been listed in this series and learn the ones your target organizations use first.

    All of these languages have free/cheap options for learning them. No one is stopping you from learning Go; indeed, there are websites whose purpose is to help you learn by doing. The same type of learning is available for most languages, with implementation details varying a bit. There are a wealth of great books available for all of them, also, if that is your preferred method of learning a new language.

    In the end, learning a new language will never be a negative. It will help you be aware of the list above, and provide meaningful input in the inevitable “Is this the right tool?” conversations.

    So, start with Python and then work through others. With one side note: if you lean toward the Ops side, know what different Ops tools require, and consider learning some of those, also. There is one prominent Ops platform that still uses TCL, for example, and we all know about CICS. So be aware of what might be needed, and at least be familiar with that.

    And Keep Rocking It

    Modern IT environments are unbelievably complex. From rights management to storage, to data gravity, to a wide array of languages and appliances – you and your teammates are keeping it all humming. Keep doing it. Learn new languages before they become necessary to your job, but never think you’re falling behind. If the company is still clipping along, you are the right person for the job.

  • Languages and DevOps: Network

    Languages and DevOps: Network

    It is fun to discuss the changes in network operations over the last couple decades. We’ve come full circle, and, honestly, ended up in a better place. Network specialists were super command-line wizards who kept us going, with few GUIs in sight. Everyone thought the next iteration would be GUIs, so network operations could be accessible to more than a few command-line specialists. “Everyone” was mostly right; we did head in that direction. It was optional, and many network operations friends pointed out that you could still do more, faster with the command line. Before GUIs had the opportunity to catch up with command lines, though, automation came along. And then DevOps. And finally,* configuration-as-code. Pretty quickly, the ability to do everything through an individual product’s GUI was determined to be a weakness. Network operations personnel were dealing with more vendors to handle the networkification (it is too a word! I just made it up) of common development activities, like some aspects of security. And now, APIs and command lines rule the juncture of DevOps and networking.

    *Finally – so far, at least

    And that shows in languages. Shell scripting and Python run this show, with little competition from any other language. We could get into a massive discussion about whether shell scripting is programming, but we won’t, because at least in networking, it most definitely is considered to be programming.

    One time, when we were tracking down a problem that involved all of IT, I was sitting with the serious network geeks and marveling at their ability to make everything they wanted happen from the command line in an instant, and their ability to parse command output. I’m no slouch when it comes to networking, but they easily put me to shame. I got to chatting with a few of them, and the consensus was that they loved computing but hated programming. Networking made sense to them. This explains why networking DevOps people are continuing to use Shell scripting and adopting Python at the same time.

    Python and Shell

    Shell is a given in networking. There are a lot of extant Shell scripts for doing all sorts of things, and the need to automate to keep up with Agile makes shell scripting a logical choice.

    Python enters the picture because it is “common ground.” It is intuitive enough for most networking people to do what they need to get done, and known or knowable to most developers. When it comes to usage, Python has overtaken Shell; (don’t take my word for it, check out the excellent work of the NetDevOps survey on GitHub, or just glance at the graphic below, created by them) in fact, Shell and Python command so much of the network development space that you are probably better off just making sure you’ve got those down, and focusing your free time on other issues like completely different networking commands, or even skills for data center versus cloud. The ROI would be better.

    But Wait, There’s More

    The key operational changes recently have been around GitOps and reporting back into the DevOps toolchain. People no longer want your appliance or application to have a pretty GUI, they want it to report in to the tools that provide automation and a larger-picture GUI. Many vendors are there already, but many are not yet. That creates a need for tools – products, open source projects, or internally developed – to do that integration while waiting for the vendors to catch up. These, again, are well-served by Python, though other languages that can handle APIs are viable, too.

    We’re headed for a world where systems detect anomalies and react to rectify them, but we’re not there yet – not on a strategic scale, at least. And we probably won’t be for some time, as automating massive network changes is a scary proposition and will take time to use these systems for detection, monitoring what they could have done, before organizations are willing to let them “just do it.”

    Carry On

    Meanwhile, NetDevOps is getting folded into DevOps proper, and in many organizations, using infrastructure-as-code or GitOps, is being deployed and managed like any other part of the life cycle. So we are moving along. For my network peeps, your job will continue to change, but it certainly won’t go away. You’re essential to smooth-running systems, and while the network may become even more virtual, it will still be networking. And under that virtual network will be … a physical network. So keep kicking it, and carving a path from users to apps, while protecting those apps and ensuring they have adequate resources. Oh, and if I spin up another instance at midnight, you’ll hop on and make sure it’s got traffic directed to it, right? RIGHT?!

  • Languages and DevOps: Can’t be Contained

    Languages and DevOps: Can’t be Contained

    The evolution of platform is an interesting study. From physical to virtual to cloud to containers, the progression has been steady, and each successive platform has benefits for both Dev and Ops that make them appealing. In the case of containers, portability is the word.

    As easy to fold into your DevOps toolchain as cloud, yet able to be deployed wherever you need, containers have absolutely taken the industry by storm. While all cloud vendors offer container options these days, their “value add” capabilities are thinly veiled attempts at vendor lock-in, so most organizations have avoided them.

    Development languages are as diverse as they are for physical or cloud; you can pretty much find containers hosting every language imaginable. That makes perfect sense, since one use of containers is to keep an otherwise-obsolete environment alive. That’s certainly not the primary use of containers, but it is an important one. The catch with containers is that the primary languages depend greatly on usage. A container that is persistent and intended to last months or even years will tend toward a different set of languages than one that is ephemeral and intended to last only minutes.

    The big weaknesses of containers – security, persistent storage, and management – have been addressed more and more as container usage has grown. Complexity of environments is now the big issue, but that is not inherent to containers; merely enabled by the ease with which new machines can be spun up.

    For longer-term containers, Java and C# are still the languages of choice, though others are certainly making headway. A combination of how new container technology is, and the extra layer between applications and hardware in the current popular container solutions make C/C++ less popular than on physical machines. They are still used because like other targets, containers still have operating systems that are linked to more easily in these languages.

    Shorter-lived containers tend to use languages that grew up in the Agile environment – from Python to JavaScript to Go. The ability to quickly crank out the code and get to test is useful for Agile and DevOps environments.

    Hooking into DevOps toolchains is at least as easy as it is in a cloud environment. There are even a fairly large number of companies that have moved to spinning up a container to build, then destroying it, only to do it all again on the next build. This is not mainstream yet, but the ability to completely configure the container and its loaded toolset makes it an appealing option.

    The kicker – one that cannot be overstated – is that it is simple and easy to get developers running with a copy of a complex system that once would have required several physical boxes. I, personally, have worked on a seven-plus server client system that I had a complete copy of on my (admittedly beefy) development laptop. It wasn’t the most performant, but it was enough to do unit testing.

    The security weaknesses of cloud come along with containers too – there is a lot more than just configuring an IP to directing traffic, for example – but mitigating tools and processes are available.

    This pundit truly believes that containers are the future of DevOps. They have the best of previous iterations, while offering mobility that just didn’t exist in other solutions. The ability to move an entire infrastructure from one deployment target to another (data center to cloud, for example), or from one cloud vendor to another makes this option the king for organizations that require (for business objectives, compliance or finance reasons) the ability to move their applications quickly.

    Meanwhile, rock on.

    Containers are the cutting edge of wherever it is we are headed. They are more agile than any other solution, they do not limit programming or networking options, can be run wherever the organization needs them and can be modified or redeployed easily. There is more complexity, but most of it has been obfuscated; only when things go terribly wrong is the complexity really an issue. Development for containers, with a few exceptions, is development for the target OS. That makes it easy to keep the organization’s systems preferences while enabling developers. All good. And many (I daresay most) of you are already taking advantage of them. Keep rocking it. Use what the org needs, and don’t forget to check every once in a while to make certain what is in use is what needs to be in use.

  • Languages and DevOps: To The Cloud!

    Languages and DevOps: To The Cloud!

    Cloud and DevOps are almost a match made in heaven. Cloud started truly gaining steam in the age of DevOps and Agile, and has adapted – or grown up in it – well, depending on your viewpoint. If not for the combined issues of data gravity and data (im)mobility, we might have gone even farther than we did into the cloudy world.

    While the breadth of possible cloud apps means a wide variety of languages is in use, Node rules this roost. Every conceivable language is used in cloud development – because, in the end, they’re just servers – but Node is the go-to language of today. The broad use and appeal of JavaScript make it an obvious choice for enabling full-stack development, and the pool of developers is larger than any other language today. Other languages are used, but not in the numbers that Node is. Python is the challenger, and a surprising number of PHP sites still exist, but forward-looking development aims at Node or Python. Considering the distance from the hardware, there is less call for C in cloud than there is in other areas of development, but it isn’t gone, since interfacing to the OS is also often done in C.

    Targeting cloud is not so much about language as it is DevOps tooling, though. All of the major Git companies have cloud-centric extensions (this is the core of GitOps) that raise the level of “fully automated.” Cloud environments, by their nature, provide APIs for infrastructure, and the Git vendors (and many, many others) have hooked into those APIs to make deployment an integrated part of DevOps at a level of detail that other environments (with the exception of containers) require a lot of work to match.

    The negative to cloud-centric development is lock-in, of course. Creating an entire DevOps toolchain and process that supports vendor X will not work directly with vendor Y. Tools can help with this issue, but the real solution is in our next installment – container development.

    The part of development that arguably benefits the most from cloud-centric development is testing. It is easier to spin up a test environment as-needed and spin it back down when testing is done – no need for a spare set of hardware or a stand-alone VM server; just redirect the DevOps ecosystem and spin it up, then account for data (scrubbing, copying, etc), and you’re off. Codify the steps for data, and you have a re-spinnable environment for test-on-demand.

    The part of cloud that is arguably the weakest is security. Let’s face it, far more of your infrastructure is publicly (or semi-publicly, if running in a VPC) exposed than in a data center. There are far more places to lock down, and a sometimes-bewildering number of options to secure. But the flexibility of pay-as-you-grow makes it an appealing option, and there are tools to help manage all those lockdown points. Just make sure that you’re not the weak point in the shared security model.

    You. Rock.

    Cloud makes a lot of things easier, and Node makes development of any step in the stack easier by using the same underlying language on the server that is used in browsers and in many mobile development platforms. Node “feels” more right for web development, at least because one connection, one call … even if it makes long-running processes require a bit more work.

    Whatever you choose, pay attention to how deeply invested in the cloud vendor’s proprietary interfaces you are, because, like all technical decisions, one day you or your replacement will have to get around those interfaces. And keep making the software the org needs.

  • Languages and DevOps: API-First

    Languages and DevOps: API-First

    One of the places development has made significant changes is in the land of APIs. These days, a program almost isn’t considered a proper application unless it offers APIs to do most of what can be done via UI (and even command line). This is a big deal for several reasons; first and foremost, the ability to subdivide the work and then test each subdivision. A proper API does one thing and does it well. Testing can validate that this statement is true for any given API. It also allows for distribution of processing. It is far easier to use a load balancer to scale APIs than it is to scale a monolithic application. And finally, it helps with portability, because with API-first, developer users don’t care what platform it’s running on, just that it can be reached via URL and responds as expected.

    Everything is a trade-off, and there are several places that API-first has issues. Integration testing is objectively harder. Loosely coupled applications, in general, are harder to test, and requiring multiple portions of an application be available to run testing becomes onerous, particularly as the layers of APIs start to build up. Second, security is another issue – particularly if you’re offering a public-facing API, but also for internal usage. Once an attacker has access to the network, internal-only APIs are no different to them than external. Smart IT departments design internal APIs with the same controls that external requires. Third is DevOps in general. API-first grew up in the agile/DevOps world, but because there are cascading dependencies, is actually harder to manage DevOps for than most monolithic applications. Finally, documentation – the bane of API-first’s existence. The number of poorly documented APIs vastly outnumbers the astoundingly well-documented ones. Even well-documented APIs, like Google Places, require some thought and testing to use well. Poorly documented ones – my favorite is, “It’s mysite.com/users – everyone knows what’s in there!” – are next to useless without a learning curve.

    With all of that said, what about languages? That greatly depends on what you’re doing. Users don’t care what language it is implemented in – as noted above, they care that the API can be reached via URI, and that it behaves as expected. So you do have options. Indeed, at this point in the evolution of APIs, the language options are so open that you’ll probably be choosing an API definition language instead of a programming language, and just instruct your definition tool to spit out the language of choice.

    Wait, what?

    For those who haven’t followed API development closely over the last few years, it goes something like this. APIs became standardized, then APIs became a big deal, then smart people started to think in terms of code generation á la ancient 4GLs. What if you could use a specification language to describe what you want to do, then pick your language and tell the tool to generate the API? What if you could do this for the entire API hierarchy? That’s where we are. I’m most familiar with OpenAPI and RAML, but there are a large number of specifications out there, and a larger number of tools to develop in them. Before my API friends freak out, of course this is an over-simplification of what happened. I only have 1000 words, max, and my editor would like less. Relax a bit.

    And then you generate.

    I’ll rely on my own experience here, but there are literally dozens of tools available to get you from idea to API interface (doing the work behind the interface is where you come in). With Swagger CodeGen, the tool I’m most familiar with, you specify the API, then generate server interfaces (they call them stubs), and generate client libraries. Finally, you develop what goes on behind the interface on the server side, and (you or others) use the API on the client side. When I say “generate,” I mean “Tell it your favorite language, and click generate.” It spits out the code you can then use to flesh out your API. It does not spit out the info to get the API into your DevOps toolchain, so you’ll have to do the work for that. Which means the more APIs you can generate in a batch, the less DevOps lifting you’ll have to do (more on that if/when we cover microservices). Since most languages can easily be placed into DevOps via their build tools, I don’t want to make this sound too onerous – security and testing will be a bigger problem, by far.

    API security is slowly getting automated and placed right into generated code, but all the usual bits, like certificate management, still have to be handled. The interesting thing, that hasn’t (yet) gotten a ton of attention, is security of clients. It is far harder to verify that the client library is secure when the purpose of that library is to grant access to your API. I expect we’ll eventually have to get there for some applications, but today, it appears to be largely individual orgs figuring out how best to lock down client libraries, and most of the limiting factors end up server-side.

    I’ve generated Python, Java and Ruby APIs via various toolsets, and considering the problem domain is still relatively small, they work fine. And it saves one piece of the puzzle … but does introduce another piece – coordinating client and server updates via DevOps.

    Keep making the apps that run the world, and use API generators to further standardization, spitting out whatever language you need on the backend, and a portable interface for clients.

    You. Rock.

    Whatever your language of choice, in the end, it will likely be your build tool that links it to DevOps. Your biggest concern for DevOps will probably be portability, at least in the near future. Meanwhile, keep rocking it. You’re doing the work that needs doing, even if it isn’t the part of DevOps everyone is currently buzzing about.

  • Languages and DevOps: Traditional

    Languages and DevOps: Traditional

    For quite a while now, all the cool kids have been going on and on about “full stack development,” “API first,” and a ton of similar methodologies/practices. And yet, a huge swath of applications are still targeted at a single system to do a specific job. There are fat clients, there are internal servers, there are standalone apps. All are still in development, and the hype machine doesn’t account for them much, though they’re pretty pervasive.

    We still have to maintain them, or write yet another. They’re not going anywhere, because there are some problems for which they are the right solution. So let’s talk about them.

    For clarity, I’m defining traditional as, “Targets a specific OS/hardware platform, and is not primarily web-based.” We’ll exclude mainframe, because the toolsets and integration with DevOps are still different enough to warrant their own blog.

    Online forums are full of people trying to re-target traditional apps (perceived Windows 10 issues caused many to want to move from Windows, and the growing Linux trend of making distros into beta – or even alpha – testers has many wanting to move from Linux), while others are just trying to keep them running as the world changes underneath their feet.

    Easy Integrations

    While these dev tools are the ones that Agile grew up around, that makes them well-integrated into Agile and DevOps. The feeds to/from compilers and various build tools are the same as other spaces we’re looking at (or, more precisely, the others are the same as traditional), so integration of AppDev is generally not an issue. Integration of Ops is harder, but exists. The various DevOps tools with an emphasis on operations (I like to call it devOPS as opposed to DEVops) are available to traditional development, and you can certainly deploy through them, but they’re like the second cousin to “full stack.” Good Enough rules the day, and a little work will be required. Thankfully, tools like Ansible and Puppet smooth the way a bit for deployment, not caring much if you’re deploying a traditional app or a web service, as long as you tell them what to do to successfully deploy.

    You got to move it, move it…

    Portability is increasingly becoming an issue. The rise and fall of “preferred” Linux distros, the costs of being a Microsoft shop … there are a lot of reasons to want to move your app around. The best first step to this (and the only step some will need, IMO) is containerization. Put it in a container, and your underlying hardware can run on whatever OS the corporation settles on, while your apps will run in containers on the OS they need. For most organizations, this will only be a stop-gap; a way to quickly move servers away from one OS and onto another, but the apps will have to move fully to the new OS, eventually. Microsoft has Windows Subsystem for Linux, which can help move from Linux to Windows, and Linux has Wine to help run Windows apps on Linux. Both have come a long way toward interoperability, but neither is perfect – or even easy. It will take a bit of work, but far less than rewriting for the target OS top-down. Tools like Cygwin are also available to get your Linux app running on Windows if you don’t want to use Microsoft’s solution. In the end, you are going to have to modify code, but you are porting an application, so no surprises there.

    Consider the future

    A place I worked in the late 90s/early 2000s had an OS/2 machine that was critical infrastructure. We were not allowed to touch the box. I think if the admins had their way, we wouldn’t have been allowed to look at the physical box. Of course, we virtualized it just as soon as we could in a step similar to my “containerize” advice above. Now is a good time to consider writing your apps with an eye toward portability, so you can remove this type of system from the equation. That RHEL 5, or Windows XP app should really be modernized or eliminated at this point. The tools for cross-platform development are out there, from Mono to Java to Xojo; platform-dependent bits can be isolated for rapid retargeting. It’s up-front work that pays off in spades when the time comes to change the targeted platform.

    You. Rock.

    Whatever your language of choice, in the end, it will likely be your build tool that links it to DevOps. Your biggest concern for DevOps will probably be portability in the near future. Meanwhile, keep rocking it. You’re doing the work that needs doing, even if it isn’t the part of DevOps everyone is currently buzzing about.

  • Languages and DevOps: Web Development

    Languages and DevOps: Web Development

    Web development has changed a lot since DevOps first hit the scene. The languages used have shifted, where languages are used has shifted, the likely back-end hosting environment has changed – but DevOps has folded these issues in and just kept plugging along.

    We’re going to approach this from two angles – what languages you’re likely to use, and what DevOps offers. This is not a “Best Billion Web Development Languages for 2021!” article; there are plenty of those out there, and you are more than welcome to read them. This is more about using these languages in a DevOps environment, which has become standard for web apps more than any other type of development.

    We’ll start by observing that your web app is, more than likely, going to have at least three parts – HTML, CSS and JavaScript. This is not news to those of you doing it daily, but we’re just setting the bar, here. Other languages may be used to generate or populate any/all of the above, but, in the end, these three languages get sent to the browser, so will be part of the equation.

    Backends add an incredible amount of complexity to this equation. I’ve done Ruby backends that did most of their work in C++ and Java; I’ve done pure Java backends that spit out the required three components; I’ve done Node backends that had very little actual HTML/JavaScript source code, but generated a ton of both – it’s all there. And the hosting environments for backends have changed. When DevOps started, most hosts were virtual machines (VMs) or hardware, whereas now we’re mostly running cloud instances or containers. Those backends have to be configured and managed, too, along with all of the dependencies that your application requires. When you think about it, we really have created quite a messy environment.

    But DevOps has risen to this challenge.

    I’ve long been an Atlassian fan, and find them to be a great tool for managing the agile side of DevOps. Add in Ansible, for the operations side, and you have a good thing going. I recently looked into GitLab, and GitOps – through GitLab – offers a compelling alternative to Atlassian’s offerings; more so since Atlassian made some rather unpopular changes right about the time I was wrapping up my look at GitLab.

    The key is that Git holds all the code. By adding in the scripts to spin up those increasingly complex backends and secure the entire infrastructure, you get an “App in a (virtual) box.” No matter what DevOps toolchain you use, tying this in is simple, because it already works well with Git and associated tools.

    Developers tend to care more about the development side of the equation (imagine that!), so let’s talk a bit about what’s going on out there.

    PHP is hanging on solidly, with a bulk of applications and (perhaps more importantly) libraries to keep it in the running. PHP is one of the original “generate an HTML page” languages, and is slowly being displaced. It isn’t a bad choice for your web projects, but it isn’t the only – or at this point, the most popular – choice.

    Java is still somewhat popular, and while it offers good support for web development, it is … heavy? Is that the word? It’s something for this type of development, and has ceded ground to other languages when it comes to pure-play web development. The skills are transferrable to other targets, so it is a good broad-use language. Like the other serious players in this space, there are web-centric libraries and frameworks that make Java hum. I’m a fan of Spring, but the options are pretty broad.

    JavaScript wins this race today. It is used on both client and server, there are a ton of developers and DevOps people available who can be productive in JavaScript from day one, and it is increasingly used in other contexts – meaning skills could transfer to other projects. Like any language, there are trade-offs, but the portability aspect makes the trade-offs things that many are willing to work to overcome. Libraries and frameworks abound to make JS development faster and more dynamic. Personally, I like Node for server-side development, and Angular for clients that need more than straight HTML/CSS offers (probably because it’s the one I’ve used the most).

    Python is definitely a contender in this space, as it is in most others today. With Django (or whatever your framework of choice), it functions well for web development, and its popularity has given it a host of support libraries to get things done. Developers are easy to find, and skills are broadly applicable, as Python is used in most areas of Dev and Ops at this point.

    Windows aficionados are still using C#, and it does have good hooks into Microsoft DevOps tools, so any Microsoft shop should include it in evaluations. I’ve done C# on Azure for web apps, and for me it was … different, but not broken. In a Microsoft shop, C# will be familiar, and at this point, most developers are familiar with web clients, so merging the two is only a tiny bit of a learning curve, but not a major one.

    Ruby does seem to have ceded its spot to Python. It is unlikely you’ll find a steady flow of ruby developers to fill those positions, but it is a highly intuitive language that you can train Java, and even Python developers in, if necessary. Ruby is well-supported for web development; indeed, it feels more natural using Ruby for web work than command line tools (at least to me, YMMV). There are web frameworks to make Ruby more intuitive in web development, and jRuby offers deep integration with Java to make use of already existing Java classes.

    All in all, JavaScript should be the strong contender for a new project today, but any of these tools will serve the need. They are all well-supported, and DevOps tools start with integration to most of these languages. Do what’s best for your environment, and keep rocking it.

  • Languages and DevOps: Embedded

    Languages and DevOps: Embedded

    Every once in a while, it is a good idea to take stock of the environment and the market around us to determine if what we are doing is what we should be doing. I’m a developer, not a prognosticator, so these observations are simply based on my experience in the development world, along with a frank assessment of what’s changing, to try and offer some advice to my fellow developers. This week I’ll talk about embedded, but we’ll hit a bunch of application targets over the coming weeks.

    Embedded development is, generally speaking, more low-level than most other modern programming practices. Considering the goal of embedded is to interface with the hardware, this is no surprise.

    C/C++

    And that brings us to the king(s) of the embedded hill. C, and to a lesser extent, C++ rule this roost. You will see a lot of talk about Python, and we’ll get to that in a minute, but for historical and performance reasons, C is still your best bet. If your DevOps tools support your Make variant – Ant, Maven, or whatever – it can be plugged directly into the DevOps process. Testing will still take extra steps. Since deployment is most often handled directly, that does not generally need extra steps, but, in some cases, it might. A lot of general guides claim C/C++ is “easy” to learn. I disagree with this assessment; newer languages tend to be more intuitive while retaining some of C’s better features, but it is, at this time, necessary to embedded development.

    There’s a Reason (or Many) for That …

    There are a lot of conflicting factors going into where embedded is headed in the future, so we’ll try to address them here.

    Historically speaking, C/C++ have had better performance and smaller footprints than interpreted (even bytecode) languages. They are easier to code in than Assembly and supported by a ton of libraries to get work done; they struck the right balance for embedded. The weight of these developers and the legacy source code they created are part of the reason C/C++ are still on top.

    But three things have caused a shift that will open up room for other languages to make headway into this space. Embedded devices are getting faster, and storage is getting both smaller and cheaper. That means that a less-efficient means is more viable. Arduino and other hobbyist embedded platforms have also come of age. It is now viable to pick up an embedded board and start prototyping over the weekend. These embedded environments are well-supported by newer languages, and, historically speaking, today’s hobby builders are tomorrow’s startups. This is pulling other languages into the mix far more frequently, particularly our next entry.

    Python

    Python is popular in almost every category of programming. It has been around long enough to lose the mantle of “language du jour,” and should be considered based on its strengths and weaknesses. Given that the newer wave of prototype systems is well-supported by Python, it is easy to maintain, and has a broad selection of libraries (often, ironically, written in C), it has gained clout in the embedded space. It doesn’t perform as well as C/C++, but modern embedded devices are of a class that this may not be as important (early embedded devices were high-performance tools, while IoT often has devices that don’t require anything like near-real-time), and hardware improvements make this less of a concern. Python’s popularity makes it relatively easy to work in to your DevOps toolchain, with targeting of embedded devices making deploy and test require extra steps … but that’s true for all of these languages, as it is a targeting feature, not a language issue.

    Assembler/Assembly

    In the end, we need machine code. Even bytecode (or p-code, if you prefer) interpreted languages execute their instructions in hardware, eventually. Because performance and space have long been issues in embedded development, assembly language has been a preferred tool for pieces of the development pie. While few developers use assembly language for everything, most embedded developers use it, at some point, to modify libraries or create targeted subroutines. Certainly not a “popular” language among developers, most of us buckle down and just do what we need to. Like C, assembly is normally built using a Make variant. These are easily integrated into DevOps toolchains to get reporting and results up to those dashboards that run a company’s agile efforts.

    Others

    There are a wealth of other languages in use on embedded platforms, most notably C#, but also newer languages like Rust. The environment is changing, and as the ramifications of terabyte SD cards and multi-core embedded boards start to sink in, they are worth watching. We are not that far off from having a desktop-quality computer in a board only a little larger than a credit card. That changes a lot of things. More room for data storage and processing, and the need for tools that are easier to use in these contexts.

    Libraries

    All languages use libraries to make development easier. We are past the days when coders wrote all of the functionality of their project. The tools that bring libraries into a project need to be incorporated into the DevOps toolchain. For C/C++, the mechanism is largely manual, and can be automated with scripts or with the build tool. For newer languages, tools like NPM will need to be hooked into the toolchain. The ease of library dependency management is one of the appealing aspects of newer languages, and making it easier for a developer to hook in libraries also makes it easier for DevOps tools to make use of library management functionality.

    You. Rock.

    Whatever your language of choice, in the end it will likely be your build tool that links it to DevOps. Maybe we should do a series on those, too; comparing when to use Gradle versus Ant, for example. Until then, keep rocking it. All development depends upon embedded in the end, so you’re holding up the foundation.