devopsy.kinsta.cloud

Author: Don Macvittie

  • ‘Best’ is Relative, but Starts at the Top

    ‘Best’ is Relative, but Starts at the Top

    Your aging mom needs a new car. You are picking it out for her because she quit paying attention to cars long ago. All you ever buy is Fords. You also always buy trucks. Do you pop off and buy your Mom an F350? Not unless she actually needs an F350 (highly unlikely, but aging moms are as varied as any other group of people, so maybe). Why? Simply because you are buying a vehicle to solve a problem: Your mother’s need for a new car.

    Ford F350
    Ford F350

    Setting the parameters of decision-making is the CIO’s role. Or at least a part of it. IT has struggled with justification since its inception. Far too often we are justifying choices that are made before justification is carried out. Browser wars, OS wars, language fights … all have marred IT practically from the day that an IT staff became necessary.

    The fact of the matter is that people have preferences, and organizations do, too. If you have 300 instances of Oracle database running, you have investment in staff, training, licenses, etc. that you can reuse if Oracle is a choice on the next project. Contrast this with “the next project” containing a purchased package that only works with SQL Server. The two are natural tensions that need to be figured out.

    In far too many cases we zero in on what an individual thinks is best or “right,” and not on the needs of the organization across projects. On the development side, Agile has made this more of an issue than it has been in a long time, maybe ever. Simply put, the faster the better is the rule for Agile, but long-term ramifications of that approach can be large. If internally developed apps vary from NoSQL to SQL server to Hadoop to Oracle and you’re having to maintain all of the above, both in Ops and in Dev, you may have delivered quickly, but the accrued cost—call it the mortgaged technical debt—can be expensive over the long term.

    And the way that we end up with multiple solutions to a given problem is by allowing champions to drive their point home in a vacuum. While it is great when an employee says, “I know a [needed library, tool, app] that will solve this and I can have it running in a day,” there is a larger picture that needs to be considered.

    DevOps definitely gives us the opportunity to review these choices and put a stake in the ground. Sometimes the internal champion is right, and that’s the tool to use in this scenario. Sometimes they’re not, and an already implemented tool/library/app should be used. But the conversation needs to happen.

    I wrote about some of this back when I worked for F5. It was amazing to me the number of customers who purchased F5’s very expensive gear, then set it up as a load balancer and went looking elsewhere for WAN optimization or security solutions, when the BIG-IP could do the job. That’s a good example of waste, but just the tip of the iceberg.

    When implementing DevOps on the Ops side, take a look at what you have. If you’re writing the same script for three different targets, it is probably time to go find out why. The reasons might be perfectly acceptable, but they might not be. If purchased apps really only support one product for AAA, then that product is probably required, even if it means propagating logins across to the corporate standard AAA tool.

    But the process to make certain these decisions are being made intelligently, with a focus on standardization but adaptability to handle exception cases, is driven from the top. Stressing that first choice is X, but Y and Z are acceptable alternatives for projects of priority three and above is important. Stressed from the top it says, “We have standards, and we all need to try to adhere to them to keep workloads down as much as possible, but we have a business to support, and will make exceptions where business value is added.”

    It’s not always easy. Not politically, and not technologically. Sometimes the solution available internally isn’t a perfect fit for a given problem, but long-term, the work required to make it solve the problem is less than the expense (in duplicated expertise and licensing fees, let alone duplicated DevOps scripts) to drop an easier solution in. IT staff are smart folks and fully capable of making those decisions, but the direction to do so must come from the top. Without it, some will aim for standardization over speed of implementation, some the opposite. Inevitably, many will aim for what is easiest for them.

    I have worked in two industries where data went from paper in filing cabinets to microfiche to mainframes to client/server to virtualization and/or cloud. Don’t assume what you’re doing only matters for now. Make the rules, ask the tough questions such as, “What do we have in-house that could fulfill this function?” and expect honest answers.

    And if you’re going to do calculations such as TCO or ROI, make sure the team knows you want the numbers to lead to the solution, not the other way around. Because we all have seen the case where the solution changed how the numbers were calculated, and that just wastes everyone’s time.

    So step up, Mr. CIO. Get Mom the Focus or a Honda—whatever will suit her needs. Not what you’ve always done, and not what is easiest for you to do. Set the rules so the team knows what decision criteria are, and make them clear so there is no miscommunication. And finally, give your DevOps team the room to make consolidation recommendations. If they’re seeing duplication of effort, let them call attention to it, and have a process to resolve their concerns. In the end, a simple, “No, that duplication is necessary because of X,” or “Yes, we should consolidate,” needs to come out of that process for each recommendation they make.

    And if your Mom is that one that drives an F350, send me a picture of her by the truck. I’ll talk to the editors about getting it posted.

    — Don Macvittie

  • Datadog Monitoring and JIRA Integration

    Datadog Monitoring and JIRA Integration

    Operations uses monitoring to provide early detection of issues with systems/servers/networks. Among those monitoring options is Datadog, a software-as-a-service (SaaS) cloud service/application monitoring tool that allows Operations to be notified of issues with their applications in near-real time. Development teams, meanwhile, use JIRA to track tickets and manage overall agile development.

    But what if your product is the very thing that Operations manages? The sweet spot for DevOps is that location where the app is internally developed, and managed by Operations, because the entire stack is under IT management.

    That’s why Datadog and JIRA are now integrated.

    Datadog can monitor an infrastructure and application on a variety of levels at a variety of locations in the infrastructure stack, from disk availability to response times to memory usage. Datadog has the ability to monitor and emit alerts when things are abnormal or out of specified ranges, or they outright fail.

    JIRA is Atlassian’s Agile project management software, with issue tracking and all of the usual agile tools in one place.

    From Datadog Monitoring to JIRA ticket.
    Configuring Monitoring’s alerting to JIRA.

    Now, Datadog can create JIRA tickets, update them as more information becomes available and populate them with relevant information from the running systems.

    Say, for example, one cloud instance runs low on memory and passes over a threshold. Datadog detects the issue and opens a ticket in JIRA that shows what system and when, as well as whatever other information Operations has defined. That ticket is now in the system like any other, and a developer can grab it and look at memory use for the portions of the overall application that run on the instance in question.

    According to the press release, these are the three automations included in this integration:

    1. Creating JIRA tickets when performance issues are detected within Datadog.
    2. Populating JIRA tickets with pertinent information about identified issues.
    3. Updating existing JIRA tickets when new information surfaces.

    I’m an analytical person, so my first thought was, “How cool,” and my second was, “But what if it spams JIRA with non-development issues?” Since Operations configures what goes to JIRA, and Datadog supports alert rollups (multiple alerts in one or two notifications), it is possible to keep such spamming under control. An overzealous Operations staff initially might configure the reporting to send too much, but as experience grows, this should become a viable way to automate bug reporting. In particular, tagging of Datadog-generated events should show quickly what alerts are creating too many tickets, and give the team opportunity to change the way tickets are created—or, if it turns out they’re created because of a serious application-impacting error, raise the priority of the fix.

    Which brings us to the updating status part. In Datadog, operations can leave a comment on an issue with an “@jira-update” tag and JIRA will get notified of the comment. That means Operations can raise awareness of an issue that may seem mundane but has larger impact than is obvious, or add detail to a JIRA ticket if needed, all from within Datadog.

    It is interesting that Datadog is also integrated with BitBucket, another Atlassian product, so it is possible that opened tickets can be marked as “in test” when an integration build includes reference to the ticket. In short, BitBucket integration means that Datadog’s monitoring is aware of commits and checkouts, and Operations staff will be able to correlate that to issues that occur. Adding in JIRA integration means that Operations sees the commits, gets the alerts and updates JIRA. The wealth of information is such that you’ll want to use their various filtering techniques just to make certain you’re getting the important stuff.

    There are some areas that are intriguing to me, such as the impact of a particularly complex set of queries/reductions on a big data platform. I haven’t tried the Hadoop integration, but it appears that this agent would be able to communicate that a given reduction is the source of the problem to the JIRA interface. Even if these complimentary integrations aren’t there yet, you can feel it coming to where “This reduction caused systemwide issues” is what is generated in JIRA. That’s pretty cool.

    Overall, the integration of Datadog and JIRA is a DevOps dream come true, tying Operations closer into Development, and allowing Development changes to be tracked closer in Operations. After all, the point is to keep the systems running as close to peak efficiency as possible, and automate as much as can be. The issues spawning from Datadog alerts into JIRA tickets also pretty much guarantees increased need for development and operations to talk, as problems that crop up by the automated system are discussed and researched.

    If you’re dedicated to a cooperative environment for Dev and Ops—even if you aren’t calling it DevOps—the Datadog/JIRA integration is worth a look.

    — Don Macvittie

  • FLAP, Part 3: Full Data Center Provisioning

    FLAP, Part 3: Full Data Center Provisioning

    In the first two parts of this series (Part 1, Part 2), we covered server provisioning and application provisioning. Knowing a bit about each of these tools is useful (although one blog is hardly a thorough evaluation of strengths and weaknesses—I strongly suggest you do more research), knowing how they work together is the point of full layer automated provisioning (FLAP). Automating one or the other helps, but automating both—and integrating that automation—brings us to the level of “Oh, engineering needs a new test environment, and it has to be internal …” and, with a push of a button, it is delivered. That should be the long-term end goal for your organization, so staff time is free to attend to some of the many other issues that plague any given IT organization.

    The thing is, these tools combine in both better and worse ways. Some are designed to work together as if one product; some take some work to get going but the combination adds value once connected; and some combinations are painful to put together and almost never are worth the effort.

    One thing you’ll note is a repeated discussion of OS support for combined solutions. The reason we address it here, even though we have limited words and it’s already been mentioned in product write-ups, is simple: The limitations of these combinations are compounded. That is, if the server provisioning tool doesn’t support Windows, then the complete solution does not, no matter what the application provisioning tool does. This applies both ways, and is an important consideration for most shops, so we mention it here.

    Tools Built to be Together

    First off, we’ll take the easy ones. These are the combinations that were designed to work together, and do so pretty well. They vary in maturity level, but otherwise, they’re good options if they fit into your environment.

    Razor and Puppet

    This is the easy one. Razor was built to be the server provisioning part of Puppet. While the solution set is relatively new, and it shows in the installation phase (which has several manual configuration steps both before and after Razor install), it is maturing rapidly, and the two dev teams work pretty closely together. This is one of only two solution sets that uses a single UI for all provisioning—in this case the Puppet UI. Support for operating systems is broad, including all versions of Linux and Windows to boot. (get it? To boot?) If your data center is highly heterogeneous, or you are already a Puppet shop, it is worth looking closely at this combination. You just have to keep in mind that it is improving at a rapid rate, so those things you don’t like likely will change soon.

    Satellite + Puppet + Foreman

    As of version 6.0, Satellite is a collection of tools knitted together that include Puppet with Foreman. This means that Red Hat has a vested interest in making certain the two tools work well together. And they certainly seem to. While the operating systems support is limited to Red Hat OSes, the solution is pretty rock solid, and like Razor and Puppet, it is improving at a rapid rate since the switch to this new platform. The UI is independent of the tools underneath, and this is the other solution that offers a “single pane of glass” for all installation activities. The one catch with Red Hat, as mentioned before, is that the functionality of underlying systems is only supported when used with Satellite. How that fits into your organization’s plans is up to you.

    Close Enough to be Second Cousins

    There are some combinations that, while not officially interdependent, have a long history of mutual support, and thus make them strong combinations. These tools tend to come with straightforward directions to make them work together, and those instructions generally work well. Once paired, they can manage complete automation in a relatively integrated fashion.

    Note that these combinations are not even a little exclusive. They are pairings the project teams saw made good sense, and have grown out of closeness. Both sides of these solution sets interoperate with other tools (see the next section).

    Ansible + Cobbler

    Ansible and Cobbler both trace their genesis to Red Hat, and they share some employees/team members—and have for a long time. That makes it natural that the two are closely integrated. Before Ansible started discussing all of the other server provisioning tools out there, it already had Cobbler integration.

    As one would expect, there is Ansible setup, Cobbler setup, and then integration. But once combined, Cobbler can do for bare metal, Ansible can do for cloud, and together they provide a full-stack solution. Since both support most operating systems, the final solution does, too.

    Juju + MaaS

    Juju and MaaS have interoperated for a long time. Both are projects of Canonical, so it is no surprise that their interoperability continues. Years ago this author used them together and it was … difficult. But that was years ago, and just in the last year both projects have made strides in documentation and ease of use. Like most solutions in this category, there are separate installations and then integration configuration. Since both support a large number of operating systems, this solution can manage the full data center also.

    Their traditional home, and the one you’re most likely to be comfortable with them in for the short term, is Ubuntu-heavy shops. But they do support most platforms out there, so it’s worth the time to look into them even for non-Ubuntu shops.

    The Extended and Mixed Families

    Then there are the ones that have not forged solid one-on-one relationships, some because they are forging a separate path and some because they want to interoperate with everyone. Note that some of the above systems meet this category—Puppet is integrated with most of the server provisioning tools out there—but due to space considerations, we’re not going to go over those products again here. This list contains both application provisioning and server provisioning tools not mentioned above.

    Chef

    Chef has put the bulk of its focus on cloud and virtualization, with the notable exception to this trend being in AIX, where it does support server provisioning. “The bulk of its focus” is not absolute, though, and that’s on purpose. Chef has had presenters at the annual conference talk about using it with server provisioning tools, so it isn’t unaware of the integrated solution space, just that’s not where their focus has been.

    SaltStack

    The Salt crew is happy to support everyone. While they don’t talk loudly about whom they are integrated with, the product is in use with most major server provisioning tools out there. As mentioned in the SaltStack write-up in part two of this series, cloud integration is pretty broad also, making Salt perhaps the most versatile of the application provisioning tools with regard to full data center automation.

    Stacki

    Originally built to interoperate with SaltStack, Stacki (disclaimer: this author’s employer) has moved on to support most of the availableapplication provisioning tools. The catch with Stacki is that some are supported with sample integrations and video tutorials at the open source layer, while others require a license and engagement of Stacki core engineers to obtain support. So there is a video showing Ansible and open-source Stacki working together and talking about the integration, but it would require a paid license and negotiation to get SaltStack or Chef with Stacki. Because of the way Stacki is designed, it is perhaps the easiest of all these products to integrate—simply add the appropriate “Pallet” and tell Stacki to include it in installations. It is getting the correct pallet that requires licensing for some platforms.

    Conclusion

    Since any server provisioning tool that allows users to customize the install can have the agents for a given application provisioning tool be the last step of a given install, these overviews are at least a little bit arbitrary. To be precise, they are easier to use in the given configurations, but all application provisioning tools can be forced to interoperate with any of the server provisioning tools listed.

    The point of this series was really to help those looking into full data center automation quickly get up to speed on what’s available in the marketplace, and as much information as a few 1,200-word blogs will allow to help shorten the list to ones relevant to your organization. It is by no means complete, and there are issues and concerns with each of the products that only further research will bring to light, so don’t stop here.

  • FLAP, Part 2: Application Provisioning

    FLAP, Part 2: Application Provisioning

    In part 1 of this series we talked about server provisioning tools. In that post, we mentioned some of the more prominent application provisioning tools on the market with regard to integrations. This post will focus on the application layer of full layer automated provisioning (FLAP).

    Application Provisioning Who’s Who

    Application provisioning technologies most likely are familiar: Puppet, Chef, Salt, etc. We’ll take a look at each, how they compare in the areas of control mechanisms, operating systems support and ease-of-use features (integration with server provisioning tools is mostly covered from a server provisioning tool perspective in part 1 and part 3 of this series, because integration must be managed in the server provisioning tool). A blog is too short to go much deeper than that, but this will help you determine which ones should be considered in your environment. Information here is valid as of the day this blog is written, but the market is moving fast, so don’t hesitate to check at the provided links for improvements. Because this topic is a popular one, don’t hesitate to look for up-to-date information provided by other sources, either.

    It is worth mentioning that all of these tools are moving to support configuration of network equipment. For a guy who spent years worrying about programmability and interoperability at a networking gear vendor, that is very cool news. I haven’t had time to actually try any of that support yet, so I’ll leave evaluations to you.

    A word of advice: This entire marketplace has cutesy-naming disease when it comes to really important concepts. While you are grimacing at “recipes” or “grains” or whatever, learn that terminology. It may be cutesy, but the heart and soul of these products hides behind those names and the details of what concepts they encapsulate. During your evaluation period preferably—but certainly once one is chosen—you should get those terms and their meanings down pat. It will make your life much easier. The same is true with the variety of scripting formats. They’re all communicating basically the same information, but so do English and Japanese—knowing one doesn’t automatically makes you an expert in the other. Take the time to truly understand the scripting language for your tool of choice.

    Ansible

    Ansible is a full-app provisioning tool that allows users to install and customize applications through a combination of a command line and scripts. The tool is well-documented, easy to install and works very well at its designed task. Ansible offers support for Red Hat and Windows running on Docker, OpenStack and AWS. Notably missing is VMware support, but since the machines are available via IP and the OSes are standardized, VMware virtuals can be managed/orchestrated with Ansible.

    The open source version of the tool comes with command line interfaces and management is handled via command line and editing scripts. The premium add-ons such as Tower include UI, analytics and programmable APIs.

    Since the purchase by Red Hat, it is reasonable to expect an increased focus on RHEL for this tool and a commensurate reduction in non-Red Hat OSes. Not that this is assured, but history says it is a likely long-term result. If you’re a Red Hat shop, Ansible is a good place to start—though looking at Satellite—and trying to predict how Red Hat’s various open and add-on provisioning tools will evolve and emerge—is a good idea for any evaluation at this point in time.

    Chef

    Chef is one of the original application provisioning tools out there. It is managed through command line and scripts in the open source version, and through these plus a GUI in premium. Unlike Ansible, many Chef APIs are bundled with the open source product, only APIs for premium services like analytics and reporting are premium.

    Chef supports most variants of Linux and Windows, with most cloud and virtualization vendors available as platforms. In terms of completeness of support, it really is up toward the top of the list.

    If you have a largely heterogeneous environment, Chef is a good tool to look at for automating application provisioning—be it for deployment, refresh or continuous delivery.

    Juju

    Juju advertises itself as an orchestration layer above the other products in this space. While there is a small amount of truth to that claim, most of what you can do in terms of orchestration with Juju can be done with the others. In fact, Puppet has kind of found a home in entire complex multi-server systems. Juju comes with GUI in the open source version, but the product does not have a traditional API.

    Juju supports Ubuntu and Windows, and is interoperable with most of the major cloud providers. Deep server provisioning integration is really only provided for MaaS (another Canonical project), but users have made it work with other server provisioning tools. Juju scripts (charms) are primarily written in Ruby, though other languages are definitely possible. For Ubuntu shops, Juju would be the item I’d place on the top of the application provisioning list. For Windows shops, it would definitely be worth considering. For Red Hat shops, other solutions are probably more appealing.

    Puppet

    Puppet should need no introduction to readers of devopsy.kinsta.cloud. The standard in application provisioning, it uses scripts and command lines to deploy applications with a pretty broad base of user support that equates to lots of user shared scripts. Other vendors in this list also have vibrant communities contributing scripts, but Puppet, by sheer weight of numbers (and marketing), has a larger selection available.

    Supporting all major operating systems, Puppet’s documentation is solid. The only quibble is that you have to be careful when coming from a search engine about whether you’re on the open source documentation or the Professional Edition documentation. Puppet comes with APIs, though the GUI is only available in the commercial version. Puppet is the progenitor of Razor, the server provisioning tool, and while it has been used with all of the provisioning tools in the first iteration of this article, expect it to work best and provide the most value with Razor going forward.

    For highly heterogeneous shops, or shops that include multiple server provisioning tools, Puppet is a solid bet. While it will work fine in other situations, the value-add of a more focused product is worth considering if your environment maps to another tool’s strengths.

    SaltStack

    Salt is an interesting member of the application provisioning crowd. They don’t share name recognition with some of the other players, and yet have a respectable user base. Their product is certainly comparable to the others in this market, and the user community is pretty active. Not being deeply associated with any of the server provisioning tools, SaltStack has been integrated with most of them.

    The Salt Master runs on almost all flavors of Linux, and can manage all major Linux distributions plus Windows, OS X and Solaris.

    Like all of these other tools, SaltStack can manage any machine that is accessible over the network and is running a supported operating system., Additionally, the dev branch has Salt Cloud built in to the Salt Master, which spins up cloud images, automatically puts them into salt and starts managing their configuration. Expect that this type of functionality will become the norm for the market (Juju does some of this also, and others have varying levels of cloud support)—it just makes sense, and it compliments server provisioning. There is a tiny bit of overlap in the VM space, but then you can choose whether to use server provisioning or Salt Cloud to handle those instances.

    For highly heterogeneous shops that wish to use best-of-breed (actually “best fit” is a better name) server provisioning along with a solid application provisioning tool, SaltStack is a heavy hitter worth considering. If you have a more specialized environment focused on a single OS, SaltStack will work, but its advantage of broad support for both OS’s and server provisioning tools will not come into play.

    Satellite

    Satellite is Red Hat’s provisioning solution. Unlike most of the others in this list, it is a cross between server and application provisioning. In actuality, it is that “layer above provisioning” that Juju positions itself as. Under the covers, Satellite (as of release 6.0) is using Foreman for server provisioning and Puppet for application provisioning. The reason it still makes this list then is because it does do application provisioning, and does orchestration too. You can use it to spin up MariaDB or OpenStack, and it is a marketed and supported product, so it fits the list.

    As one might wholly expect, Satellite supports RHEL. It can manage instances of RHOSP and RHEV, along with VMware and Amazon, and does bare metal through Foreman also. Satellite is deeply tied to Foreman and Puppet, so if you have issues using this tool set, it’s not going to be your first choice. Because Satellite extends and improves Red Hat Network access, for Red Hat shops it should be No. 1 on the list of viable options, but if you use a lot of non-Red Hat machines, including (as of version 6) Fedora and CentOS, it is less appealing. It used to be that Spacewalk was the viable option for those CentOS and Fedora installs, but now they are separate code bases, meaning you have to maintain two different systems. At that point, it is worth looking at other options available on the market.

    The newer version 6 architecture is designed to provide a platform moving forward that is more powerful and adaptable, so expect to see good things from it if you are a Red Hat customer, possibly even cancelling out the disadvantage of no support for CentOS and Fedora.

    Summary

    All of these products are very good in their best environment, and most of them will give you pain in their very worst environment. The question is, which environment do you have? While only employees can answer that for a given environment/tool combination, hopefully there are some pointers here—and elsewhere—that can help make the decision.

    In the third and final installment, we’ll look at the combination of server and application provisioning to offer full stack automation.

  • FLAP, Part 1: Server Provisioning

    FLAP, Part 1: Server Provisioning

    We have reached the state where end-to-end data center provisioning is a reality. It’s early yet, so there is a certain amount of work involved, but today a DevOps team essentially can turn on the automation and move on to other DevOps priorities. It takes some effort, depending upon the solution chosen, but the end result is an infrastructure that provides for installation of operating system (OS), hardware (if needed) and apps, along with configuration of the whole, at the push of a button.

    There are a number of solutions ready for your data center today, and this three-part series will explore them. In the first part, we’ll talk about server provisioning—the part that takes you from uninstalled server or virtual to a functioning OS configured for your environment. The following two articles will cover application provisioning and merging the two to achieve FLAP. (Yes, I made up FLAP. Use it or not, as you see fit.)

    Disclaimer: I work for a vendor in this space. While I do not believe that impacts my objectivity, I will note where my employer is mentioned, and leave it to the reader to decide if I’m blinded to some items/issues based upon my association.

    Server Provisioning: What and Who

    A server provisioning system takes a physical or virtual server from nothing to fully installed OS. This layer of FLAP is fundamental to automating the entire process. If the system has to be installed manually, using tools and scripts to install applications just doesn’t offer the overall benefit as when the entire process is automated.

    For this discussion, we will be sticking with the major provisioning tools out there. To be sure there are more, and I have information about them, but for this short blog post, we’ll stick with the ones that hold something special—market share, history, integrations, associations—that make them a good long-term bet. Our list is alphabetical, and does not show preference by ordering.

    Cobbler

    Cobbler is one of the original server automation tool sets, and definitely has the history and stability to warrant your attention. For shops that use a lot of Fedora and CentOS, Cobbler in combination with Spacewalk is particularly appealing, as we’ll talk about later. Support includes most Linux distributions. Users have created ways to include Windows in the list of installable OSes, but there is not yet an official solution for Windows.

    Cobbler is integrated with Ansible, Spacewalk, Saltstack, Puppet(user created), and older versions of Satellite. There is some amount of concern about Cobbler’s future growth, simply because Red Hat moved Satellite off of Cobbler with release 6.0, though Red Hat still supports Cobbler. It is too soon after the split to predict how this will fall out, but currently it seems to have motivated the Cobbler core team to do more. Users concerned should check GitHub analytics to see what the trends are for committers and commits at the time they are evaluating (good advice for all of these products, but more pertinent for one where there is a question of its future).

    Cobbler’s hardware support is better than most, and in terms of application support it is a pure-play server automation tool; there is no support for applications at all—application provisioning is expected to deal with that aspect.

    Foreman

    Foreman is an open source project that is currently enjoying the sponsorship of Red Hat, but support is not limited to Red Hat OSes. In fact, Foreman will install just about any flavor of Linux, and can target most physical/virtual/cloud platforms. Using WDS, Foreman can also install Windows.

    One of the big strengths of Foreman is its interoperability—it doesn’t much care which application provisioning tool you are using. It is integrated into Satellite (with Puppet), and has support for all of the other major application provisioning tools. This means that no matter what you’re using to get your apps installed, Foreman is worth looking at for getting your servers installed.

    Like most products out there, Foreman provides CLI, API and GUI. They’re relatively stable in relation to the market, simply because Foreman implemented them earlier than most competitors, and is constantly improving them.

    In short, if you are in an environment that has a wide selection of OS, architecture and application requirements, Foreman should be at the top of your short list. For more focused solutions (in terms of OS, whether or not cloud, or application provisioning tools), it makes more sense to compare specific features/functionality with some of the more focused tools in this list.

    MaaS

    MaaS was the sleeping giant of the automation world, but it does indeed look as though it is awakening. MaaS is part of Canonical’s Ubuntu project, so there is plenty of backing, but historically its goals have been modest and its documentation more so. That changed some time over the summer/fall, and MaaS is showing increased support for OSes, better documentation and a more focused attempt to get the word out. If you are primarily an Ubuntu shop, it is well worth considering.

    Like most of the products in this offering, MaaS uses PXE to get the machines into its system, and then uses predefined images that have been imported (Ubuntu images don’t need importing) to install a machine from bare metal/virtual to working OS. MaaS proclaims support for CentOS, RedHat, SuSE, Ubuntu and Windows. For those unfamiliar with the space, that’s an impressive list. But the last time this author used MaaS, it was still only Ubuntu, so I cannot comment on the level/quality of support for each of the OSes. Due diligence should involve testing with those OSes your organization needs.

    The weakest part of some installers is non-standard hardware, such as RAID controllers, specialized network cards and the like. MaaS supports these components and includes UCS support in its packaging. Directly mentioning UCS support is only to be found with MaaS and Stacki, so if you’re a UCS shop, it is worth considering.

    In addition to CLI, MaaS touts an API and a GUI. These are important tools in the DevOps sense, because the API lets you tie MaaS into your automation architecture, and the GUI helps find issues specific to MaaS and its domain. Particularly interesting is the collection of logs that show install results and operations results clearly for each server.

    The one major drawback to MaaS is its limited application provisioning support—it is well-integrated with Juju, but none of the other big names in the space.

    In short, for Ubuntu shops using Juju, this is the tool to put at the top of your short list. For everyone else, the tool is viable, but should be considered against the features/functionality of other tools.

    Razor

    Razor is Puppet’s newer bare metal installer. It is young yet, and there are still issues to be worked out, but the large cadre of developers that are dedicated to Puppet say that Razor will continue to improve as time moves on. At current it directly supports most flavors of Linux and Windows installations. The catch to Razor is on the other end: It is designed to work with Puppet, and support for other application provisioning tools is low—indeed, is limited to user-designed solutions. Needless to say, because it is sponsored by Puppet, it is expected that official work in this area will remain slow, as it is not a priority.

    Razor does not have a dedicated GUI; instead, it uses Puppet’s GUI to report results. As with integration, this is okay for Puppet shops but pretty much eliminates Razor from consideration for other shops. (More on this in a future blog in this series.)

    Because Razor is new, there are a couple of serious caveats with the product. First, install is one of the worst in the market, with a lot of pre-install steps, a lot of install steps, and configuration of an array of underlying systems by hand. Second, each currently installed server that you do not want Razor to re-install must be entered into Razor manually on the command line. This is so limiting in a generalized data center environment (where Puppet shines) that you can expect this limitation to be resolved quickly.

    In short, for Puppet shops that do not see their application provisioning tool changing, this is the tool to put at the top of your short list. For other shops, it’s worth looking at competitors first, simply because Razor requires Puppet.

    Stacki

    Stacki is an open source project sponsored by StackIQ (as promised, this is my employer). Having roots that stretch back more than a decade, Stacki specializes in RedHat/CentOS installations with additional support for clustered software. As of this writing, CoreOS and Ubuntu have been added to Stacki’s list of supported OSes, and expect that list to grow over time.

    The biggest appeal of Stacki is the variety of ways to get systems under management. All of these apps will auto-detect a new server on the network, grab it and install it. Stacki adds the ability to list just the servers that should be clean-installed in a spreadsheet, the next time each of those machines boots they will be installed, but no other machine will. While there are ways in all of these systems to simulate this functionality—you can add exceptions to the “install all machines that boot on this subnet” rule via the command line in Razor, for example—using a spreadsheet to affirmatively manage what machines are to be installed and how gives an offline bulk way of managing installations and re-installations.

    As of this writing, Stacki does not have an API, and only with commercial add-ons can a GUI be had, but expect one or both of these things to change as the community begins guiding development efforts. Stacki is optimized to spin out hundreds of servers in minutes or hours instead of weeks or months, utilizing a peer-to-peer parallel installer to service installations faster as more servers come online. While it works as well as others in smaller environments, it really comes to its own when there are a large number of servers under management. On the initial installation front, while the other systems in this list require hand-installing and/or configuring prerequisites and sub-systems, Stacki comes as an ISO with a UI that asks a few questions and configures everything.

    Stacki has been used with most (Ansible, Puppet, Saltstack) of the major application provisioning tools this series of blogs will consider, making it versatile on the application side of the equation.

    Overall, if you’re a Red Hat shop that uses CentOS a lot—particularly a larger shop—Stacki should be the first stop on your short list. For everyone else, the tool is viable, but should be considered against the features/functionality of other tools.

    Stay tuned for part two of this series, which will cover application provisioning.

  • Counterpoint: Right-Picking the First Project to Go DevOps On

    Counterpoint: Right-Picking the First Project to Go DevOps On

    Lori and I were discussing DevOps and surveys the other day, and in what must be described a rare occurrence, we disagreed on the best path to undertake your first DevOps project. Her take is here , and well, you’re about to hear mine.

    Lori proposes that agile projects are the best for a first DevOps implementation. Her rationalization is that by doing DevOps where it will come into play more quickly and frequently, the value of DevOps will be readily on display to add weight to requests for more funding/staffing to move on with a wider DevOps implementation.

    This is all true, and certainly is worthy of consideration. Indeed, in a perfect world we would do all of the DevOps work as part of the overall project plan, hand-in-hand with developers, so that “finished” meant completely ready to repeatedly deploy, upgrade, and manage. Some shops are headed in this direction, and a very few shops that stand to gain the most from DevOps – highly dynamic environments like cloud providers that need to provision/change/delete systems at a rate that makes most of us think of alcohol – have already gotten to that point. But for your first project, such cooperation is unlikely.

    There is a lot to learn in your first DevOps project. The primary perceived benefit of agile development is more iterations, faster. Are you seeing my concern yet? You’re trying to figure what/where/when for the very first time – not to mention the big ‘who’ part – and meanwhile iterations are flying by. For some organizations, there are enough automation bits already in place that this may be a feasible idea for a first project, but for most, let’s not fail to deliver, or worse, rush to catch up and deliver a massive failure, just because we wanted to be fast.

    Learn first, rush later. That’s a reasonable motto to live by in DevOps. If an iteration of an agile project is bad, the dev team talks about issues in a stand-up, refactors the workload, and works on another iteration without deploying the broken code. In DevOps, if an iteration is bad, unless you have an entire mirror of your production environment, all of the customers will see it. That’s not good for selling the benefits. Indeed, it is the opposite of good.

    So Lori is right, you could indeed gain some great mind-share by doing it right the first time in a high-iteration project, but doing it wrong in that same environment will kill the chances you can get the opportunity to do it more and better.

    I’m not a cautious person, but I am realistic. When the cost of failure is high, and the probability of failure due to learning curve is accentuated, pick a project with timelines that will allow you to do an excellent job of showing off the power of DevOps. That means giving you time to digest the new processes, procedures, automation tools, and testing goals. Agile reduces that time, and while an agile project may fit your needs perfectly, chances are, you’re better off approaching the first project with extra time built in to learn.

    Of course, neither Lori or I mentioned the real killer in first DevOps projects (though we both have in blogs before now). That’s complexity. Don’t go for the complex project first. The more there is going on, the more lines of control, automation tasks, hidden dependencies, and just flat-out gotchas you’ll find… And the greater the risk to successful delivery.

    Even if you’re not trying to garner the attention of those holding the budget purse strings, successful delivery is certainly a key indicator to ability/willingness to implement more DevOps projects. And if making it all automated with lines of communication that facilitate rapid recovery of errors, and reduce call times are your goals… well success in the first project, even if not hugely visible, is important.

    If you are lucky enough to be able to roll DevOps right into the development effort of an agile project, and grow the DevOps side as the dev team is adding functionality on the development side, by all means it is a worthy attempt, even for a first project. But if you are automating an agile project already well along in its lifecycle, think twice, consider the timelines for releases, and the issues that have to be considered for that particular project, before you commit to investing your initial (presumably limited) DevOps resources there.

  • It’s About What Works.

    It’s About What Works.

    Enterprises have been working through production delays and operational inefficiencies for decades. Don’t get sucked in too deeply to the ever-present mantra of “this time is different”. There is a lot of source material out there from people in a different kind of operations that have already been through the same wringer as DevOps presents. You just have to know where to look and remember in the end “it’s about what works.”

    SkyIsFalling

    I was talking to Lori about my concept for this blog, and she sent me a link to a pretty good (dated, but relevant in light of DevOps) PDF that introduces quite a few of the concepts. But let me pull from personal experience to try and show how little uniqueness is enshrouded in DevOps.

    Like DevOps… In the past.

    I was hired into a utility company a long time ago to take them from a run-of-the-mill utility to a fully implemented Automated Meter Reading utility. No utility in the country had gone to 100% automated for meter readings for a variety of reasons, some technological, some fear of change, some organizational. For the organizational part, imagine how many meter readers it takes to read 350,000 meters… Now imagine reorganizing without all or nearly all of those jobs. And imagine they type of jobs – automated meter reading operations and high-tech meter repair – that would replace them. It was a tall order.

    My role was strictly datacenter. I was given a mostly empty chunk of DC real estate, and told to build. When I walked in, there was a server for collecting the readings of the system that won the prototype phase, and some modems to read some meters that were already automated when the project was first conceived.

    While the engineering manager, operations manager, and myself overcame our various technological issues, others were dealing with the organizational issues. When I joined the company there were less than one percent of meters being read electronically. When I left the project two years later we were at 80% and the groundwork was laid to reach 100%. The organizational changes had been run through, both in terms of staffing issues, and in terms of resource utilization in IT. The transformation took a long time because physical hardware had to be installed on each meter.

    But reading errors rapidly became an exception. Cases where there was too much snow and the meter reader had to estimate? A thing of the past. Estimation was now reserved for exception cases, as normal operational procedure covered the vast majority of meter reads, even in blizzard conditions. And cost went down. While the initial investment was extremely large for a utility of that size, the ROI was solid, and rapidly that money was being recouped.

    What did they have to do? They had to conquer the three issues DevOps faces. Automate complex, error prone operations, make organizational changes that will serve the long term interests of operations, drive disparate groups to work together and understand issues impacting the overall project, and constantly monitor the ROI of the overall project.

    For that project, we were lucky in that there was more than buy-in from executive management, their was strategic priority. We got what we needed as long as we showed we were on track. DevOps is unlikely to achieve that level of support unless C-level execs can see the ROI and how it will help the organization going into the future. That’s the topic of my next blog, but luckily we won’t need that level of buy-in.

    …Is DevOps Today.

    Get a project manager, give them an operational process that is either done so frequently as to warrant special attention or is error prone enough to warrant special attention, and tell them to figure out the steps, who is involved when, and come up with a plan to automate it that can be achieved in a reasonable amount of time.

    Then set guidelines for what you will measure to prove success, and make certain you start measuring them in the current state if you aren’t already. Measurements are useless without a baseline.

    For an oft-repeated process, reduction in the time it takes to complete the process OR in the manhours required to complete the process is a good first measure. For error prone processes, measure rates of error.

    It is that easy to get started. While the PM will require some time of employees, if you choose the project well, the PM will be the only one devoting significant amounts of time to it… And you will come out the other end having a story to talk about. Hopefully having a success story, but even if the results are not enough to warrant further investigation, you’ll have a solid reason why.

    Are we increasingly hearing about nifty cool tools and whiz-bangs? Yes. And if you have the funds, go explore them. But always remember, first came DevOps, only then did tools get developed. Some will help you, none are strictly necessary, and DevOps rarely gets an overflowing budget.

    So hop on in there. You have done this a zillion times in each of the given areas of IT – how many standardized VM images or automated spin-up scripts do you have laying around for example? The only difference here is that you are doing it across more of IT, and you’re being more formal about it.

    And if the results of your first project show little improvement in your chosen measurements, you will have to look closely at the project, the PM, the organization, and the measurements. Each of these will lend itself to different solutions, it’s up to you to decide what to do for your company at that point.

    But likely, if you chose the project well, you’ll be off and running with little risk and much knowledge gained.

    And the PDF Lori sent me? It is about Business Process Management. And if you look closely at the steps, you will find that BPM was doing for the business pretty much exactly what we’re doing for ourselves now – reducing turnaround time, saving money, and improving customer (in our case the business user) experience.

  • Network Engineers as Programmers? Not so fast.

    Network Engineers as Programmers? Not so fast.

    It is interesting, as the triplet new waves of hype evolve – cloud, SDN, and DevOps – that we are repeatedly hearing The Voice of Reason say things like “All network engineers need to be programmers”. While this is certainly not the only voice that demonstrates what I’m talking about, an example of it is Kirk Byers’ commentary on Network Computing’s site. (http://www.networkcomputing.com/data-centers/programming-an-essential-skill-for-network-engineers/a/d-id/1297898) I do not know Mr. Byers, and want to be clear, he’s not the only one portraying this wisdom, so the reference is merely that – a reference to the syndrome I discuss here.

    In DevOps, certainly having staff members that understand both application development and network engineering is a bonus. Being one of those fabled unicorns, I also know that they’re few and far between. I’ve worked with some great ones over the years that “just get it”, and they certainly add value to an automation project that involves networking. But there is a reason that there are few of us that understand both.

    I’m a programmer. I’m a very good programmer. In the course of my career, I’ve also become very good at strategy and even technical marketing. I learned networking because of my career path. When I chose to gear my career at new challenges, there were many. From datacenter administration to IT management, from storage and servers to networking to security, from cloud to embedded. I’ve enjoyed every aspect of my career, but I’m still ‘a software guy’ at heart… Just one that knows about networking, storage, and servers at a deep level.

    The thing is, my knowledge is not some magical formula. There are people in your enterprise that know all of this, and not all of them are software people.

    SquarePegI’m just suggesting we stop and think about it. When making the move to DevOps, there is a need for a certain set of skills. Those skills absolutely do not have to be vested in a single individual. There’s more overhead if you’re coordinating efforts, but think about what is going into place – automation scripts (of one form or another, no matter what tools you use) that have to have a big initial push to get developed, and then, unless your environment is in perpetual flux, just get used and occasionally modified.

    This does not require retraining of the entire networking staff. In fact, in maintenance mode, those scripts are not generally going to require regular work. Indeed, the largest changes to these scripts will occur as part of a pre-planned project that can take them into account.

    Looked at another way, developers sometimes need to deal with networking issues… Does that mean all application developers need to be retrained as network engineers?

    And there’s a risk. If you decide all of your network staff needs to learn how to code effectively in whatever language and/or tool you settle upon, some of them are likely to leave. Because they’re network engineers and not application developers for a reason. Just as I am at heart a developer, they consciously chose network engineering. For the amount of maintenance required after the initial push to get the automated portion of DevOps into place, one network engineer who is also interested in development is plenty. It doesn’t have to be the whole team. If it did, then all network engineers would need to be security engineers also, no?

    It’s a wave. It will wash over your IT shop in stages, cresting with the workload required to get the automation part of DevOps running, then receding, leaving ripples, but not a ton of water behind. Treat it as such. Get the help necessary to fill the gaps, put a team together that can address all the issues, then let them all go back to their roles, with added (and reduced) responsibilities that DevOps brings. Let developers do the development, working in tandem with network engineers. Find the network engineer that is most interested in development, and have them sit with the developers, working it out, getting everything right and learning what the scripts do. Then let that person handle maintenance, and let the developers roll back until the next major project needs teamwork.

    In the context of network engineering, system engineering, and application development, it’s a project, not a lifestyle change, to automate things. We know how to do projects, they don’t generally involve upheaval and retraining of entire teams.

    But if you’re still convinced you need the unicorn that can do it all, give me a call. That is, after all, what my firm does – bring knowledge both broad and deep to the table to help you solve problems – but honestly, most organizations don’t need someone like me nearly as much as they need a process, a list of attainable goals, buy-in from IT management for the manhour investment, and a plan.

  • DevOps’ weakness and what it does best

    DevOps’ weakness and what it does best

    Every great movement has its weaknesses, be the movement social, technical, or political, there are always ways in which it falls short and steps must be taken. DevOps is not some special case that is pure, and if you’ve worked in it, you know how very true that statement is. From meetings that are a tug-of-war to failures that aren’t readily traceable or automation that spawns problems across a datacenter, there are definitely weaknesses in the process.

    But like any good movement, the benefits are perceived to be worth the work to minimize the weaknesses. If stability, agility, and security can be served by the standardization and improved communication of DevOps, the perception is accurate.

    HammerNailWhile it takes a commitment of man-hours, the meetings can be gotten through with disparate interests being satisfied, failures can be traced and changes made to make certain it doesn’t happen again, and cascading errors can be prevented with careful analysis and planning. Even if such a cascading error does crop up, once it has occurred you can take steps to keep it from recurring.

    The problem that is the bane of DevOps is not one of these things, it is a simple problem encountered in every IT shop. Complexity. The volume of complexity in IT is astounding, and it is difficult to take complex systems and effectively automate them without a massive investment in man-hours and possibly budget also.

    While there are plenty of readily available examples of the levels IT complexity, I will draw upon my recent experience at F5 Networks to offer one single example. F5’s BIG-IP is a stellar platform for all types of traffic optimization and more. From security to image minification, BIG-IP can do it. The thing is, offering the breadth of possibilities that BIG-IP does introduces complexity. The number of configuration objects and the number of parameters that can be configured for those objects is daunting.

    But F5 saw that this was a problem (real or perceived), and invested the man-hours to make life easier for customers. Of course they’re a business, so it wasn’t some great altruistic move, but that’s not the point. When they released version 1.0 of iApps, their deployment automation and configuration streamlining solution, the time to deploy the network resources for a typical app was reduced to a fraction of manual deployment man-hours.

    The idea is to set some defaults for all of those parameters and provide a UI to ask how to configure the remaining elements. This allows IT to set the parameters that are predefined policies and less flexible application-to-application, while allowing the more adaptable parameters to be defined at deployment. It also allows easy replication of existing deployments.

    The only drawback to the original deployment scenarios was its lack of true automation. Over time F5 has added API interfaces to build and deploy iApps, easing that crucial issue.

    The benefit is that F5 gear is easier to use – particularly with their selection of pre-built iApps that you can use out of the box – but it came at a cost in man-hours. Simply put, the complexity was removed via iApps, but when you need it, you can always dip back to the traditional way of implementing cascading objects.

    Am I selling you F5 gear? No. I’m simply offering one example of extreme complexity created by adaptability that was overcome and the result was a huge positive. And pointing out it took effort to get there. Dedicated effort. Complexity is just another surmountable obstacle, if you address it correctly.

    It takes investment. You have to set the time aside. This stuff doesn’t happen on its own, and too many enterprises try to do DevOps in their spare time. Yes, everyone is crazy busy, but peeling off some hours today to dedicate can reduce man-hour overhead of your systems moving forward. You need to look at complex processes and break them down to the smallest chunks you think you’ll need, then automate those.

    This approach has the added benefit of making management of devops implementation easier. Meetings are much shorter if the part you are talking about is focused on a single task, utilizing a subset of the infrastructure.

    If you’re going to do it, set aside the time. Do it right, and reap the benefits.

  • Coaching DevOps Baseball

    Coaching DevOps Baseball

    Coaching DevOps Baseball

    Every technology gets surrounded by hype during its maturation, and DevOps is certainly not immune to that phenomenon. If you’ve spent more than five minutes looking into it, you know that, because the current discussion sounds something like this:

    “All right, there’s other stuff, but it’s just support, so I want you all to go out there and practice the important part! You only win the game if you score runs, and you only score a run if you run from third to home, so get out there and run from third to home! You’ll be the best team ever on the only part that matters, running it in!”

    baseball

    “What? That hitting the ball and making first, second, third base? That’s supporting stuff, doesn’t really matter, master running from third to home, that’s how you score runs, and runs win games! Oh sure, some hotshot can hit a home run, but then the team relies on the guy who can get home runs, so forget about it, just focus on running third to home!”

    Yes, the vast bulk of punditry sound like this right now, and yes, it is as damaging to your DevOps efforts as this coach would be to a baseball team.

    And yes, you need to care about all the other parts of the game – because it isn’t baseball if you only do one part, and it isn’t DevOps if you only do one part. No matter how much the coach thinks that part is the important part. After all, in the game, you’ll never get the chance to run third to home if you’re not able to handle the preceding parts of the game.

    So when you read pundits at the moment, pay more attention to what they minimize than what they consider the end game. The benefits of putting scripts and configurations into version control, building automated testing, taking the time to write automation scripts you’ve been thinking about off and on for years (and putting those into VC also), all of these things are not only necessary prerequisites to the nirvana some foresee if only you do DevOps “right”, but they also hold benefits all their own. Harness your hotshots to automate what’s hard, and make that automation adaptable enough that it can change with the times. The “right” way to do it is methodically. Indeed, I would add the Etsy slogan “measure everything”. You can’t tell how DevOps is impacting the org if you’re not measuring things.

    And doing it methodically is actually pretty easy. Pick a team or (my preference) a project, and tell them to do it. Set standards for keeping each change to configs in version control, demand that they automate as much as possible, but leave points in the process for human checking, require that automated tests are complete and run successfully before the code is considered complete (and set a standard for maintaining tests in the future, since that’s where most orgs have issues). Involve security deeply, because a failed drive, or a server dropped from a bad config can be recovered quickly, a security breech not so much. Then take what you learn, and apply what works well for your org again, perhaps to a larger team or project.

    The whole point of DevOps is to improve IT’s responsiveness to changing business needs, increase operational capacity, and reduce human error. It’s not to achieve some goal set by people outside of your org, it’s to do it within the confines of your org.

    So go hit some home runs, run the heck out of the first three bases, make IT more efficient and effective. Worry about things like “IT Transformation” when they come up and make sense, not when you’re just learning the rules of the game.