devopsy.kinsta.cloud

Tag: toolchain

  • Monitor Toolchain Health, Too

    Monitor Toolchain Health, Too

    A funny thing happened on the way to being agile. We adopted a ton of tools to make us more responsive. They really do help us deliver software quickly and efficiently, and they have improved IT processes along with our specific process improvements.

    They also created a toolchain that resembles areas of responsibility of old. That makes perfect sense; we still need to store source code, build applications, deploy somewhere, worry about security, etc. And we need to do all of that faster, which is where automation and the toolchain really shine.

    But you need to keep in mind the overall impact of the toolchain upon the development process. Now the organization is dependent upon those tools. We went from slow and steady to fast and with a large series of steps along the toolchain. Things are better, we’re delivering quality software at a faster pace, but in some ways, the process risks becoming more fragile. If a tool in the chain shows signs of trouble, you need to be aware of it and on top of the issue almost immediately. The price of ignoring issues in the process can be catastrophic disruptions in software delivery. Losing a CI tool is akin to shutting down customs at an international airport: The source of the problem is big, and the downstream problems are show-stopping.

    So while you are tooling your applications, make certain you are tooling the build chain, also. Know when a build that always takes ten minutes suddenly spikes to triple that. Know when a test suite delivers non-breaking errors at an increased rate. Know when space is low on a shared drive that the build tools use. Tool everything, because the software delivery toolchain is now the core of application development and deployment.

    Tooling these systems can also offer a warning of problems in a given application. It is possible that the increase in build time is because there are a lot of previously unused libraries being included … and that those libraries have not been properly vetted by security. It is possible that the disk space on that shared drive has PII test data that needs to be cleansed. There is a lot of good that can come from over-tooling the build chain, and unless you have a bit of tooling that slows the process, there is no downside. It won’t take too long to build a list of what to watch, and modern DevOps tools have enough reporting and APIs that it won’t take long to begin watching those items, either.

    That is where the people part comes in. Don’t ignore the tooling once it is in place. Build a process to regularly review results and take action when necessary. No tooling is useful if the results are ignored; sadly, we have a long history of spectacular failures that started with, “We saw indications, but didn’t think it was important at the time …” So act on results, even if that just means flagging them for review in the next reporting cycle—just don’t ignore them.

    And keep rocking it. The organization thrives on the software you build, deploy and support. It is a massively complex system that has IT staff at its heart, so keep it thriving and watch for indicators of illness in the core toolchain.

  • Managing Risk

    Managing Risk

    We have built some beautiful toolchains that crank out a finished product on the fly without needing anything close to the level of intervention that was historically required. The most advanced organizations on an automation journey could change a line of code and then wait for the new version to hit production without doing a thing. I say “could” because I don’t know of a single org that does this; there are, of course, checks in place and it’s not single-line changes but projects that release. But the infrastructure is there that would allow them to do this. It’s a high probability that one day we all will.

    The problem is that most organizations only look at risk when building these toolchains and don’t bother with contingency plans. But over the long life of any given toolchain—toolchains that are made up of many moving parts—there will inevitably be the need for contingency plans.

    And it’s certainly not only toolchains—remember when popular libraries were purposely corrupted by their author to make a point? And unless you’ve done your best to ignore it, you’re also aware of faked versions of popular libraries that insert code before continuing on to the original…

    Most organizations do not consider a scenario in which critical libraries are suddenly unavailable or a scenario where a tool is crippled by movement in the market—a vendor making harsh licensing changes, the underlying tech making changes that impact the tool, etc. While it is comforting to think that open source protects us from many changes (and it forking does), that is not complete coverage that entirely protects our systems and toolchain. We need to know what is absolutely critical and how the organization will keep delivering in case that absolutely critical element goes away.

    When I have done disaster planning—of which this is an adjunct—I have done so with a vague generality in mind. We have plenty of work to do, so all I ever wanted was a list of critical parts and an idea of how we would adapt without them. I worked at a place that kept a backup data center in case of total regional disaster—the backup data center was intentionally located on the other side of the country—but most of us don’t need that kind of DR. Certainly not if we can go, “This OSS library is used in 60% of our apps; if we have to, we could adapt this other one,” instead of building infrastructure.

    For OSS dependencies, we have two great options that work together. Generate SBoMs, then correlate usage from those across the entire app portfolio. Then set up and maintain a local repository with point-in-time copies of those libraries that are deemed critical.

    A similar, “Our CI tool is critical, but we could easily port processes and YAML to this other one,” is enough, too—it is absolutely not going to be easy if you ever have to do it, but “We looked at this six months ago and we know what to do,” is far better than “Oh crap, now what?”

    You built it, you operate it, you are rocking it. Just take some time to protect it and you are covered.

  • It’s a Toolchain!

    It’s a Toolchain!

    I was talking with some friends the other day, and we got to discussing how inaccurate the word “toolchain” is. We pundits–and even a lot of marketers–use the word toolchain regularly and, while it’s not a terrible encapsulation of what we’re doing, it is, at best, a hasty generalization. There are a lot of reasons for this.

    The biggest reason is that “your toolchain” can have meaning when talking to a single DevOps team, but an organization rarely has just one. And increasingly, even single teams will use different tools depending upon the project they’re currently focused on. There are a lot of languages, targets, environments, specialized tools, frameworks, etc. that are unique and interdependent. Any organization that actually ends up reaching “one toolchain” will keep that honor only until the next merger. Or until the next project starts up. Because in the end, IT is about solving problems and while some things like OSes get standardized, others do not, as they are focused on solving those problems.

    Then we have the variations. Even ‘one’ toolchain is really not the same chain of tools; it is the same CI/CD tool managing a pool of others. This has really interesting connotations for the idea of the “weakest link in the chain,” whether we’re talking security, compliance or testing, because the weakest link might depend on which tools are spawned this run. To take an easy example that doesn’t overlap with the biggest reason above—targeting containers for test and virtual machines (VMs) for deployment. Some organizations do this type of thing regularly due to licensing or space issues. Two different deployment steps in ‘one’ toolchain. There are more instances like this than you would think. “This project uses make, that one uses cmake” is an example of the type of scenarios we’re talking about. These minor variations are handled by what gets called from CI.

    Finally, most of the real-life organizations I stay in touch with are both project-based and are constantly evolving. That makes both of the above scenarios the norms, not the exceptions. While they would love to have one stack and one toolchain for all projects, no one realistically sees that happening anytime soon. Some organizations that crank out precisely one type of application for one type of audience (dynamic web apps for a particular market, for example) can get to a single DevOps toolchain, but most do not.

    A windbag like myself claiming that we’re using the wrong term—and that toolchains should always be plural when having these discussions—does not make it important. I do think it matters though because what is useful for one toolchain is not for the next. The oldest, most established and, arguably, most thorough code analysis tools are for C/C++. Talking about how we have a solution to help add testing to your toolchain is not terribly useful if the point is to extoll the virtues of those code analysis tools and the audience is full of people from primarily Python or Go shops.

    Right now, important tools for you to consider are those that make your work easier. Consolidation should be high on your list of priorities; we have far too many DevOps tools. Merging should be a consideration. Yes, Python and Ruby have different needs, but if there is a single tool that can offer build or test for either based on information contained in the pipeline, grab it so two toolsets don’t bifurcate your build process.

    And keep kicking it. Seriously. In the grand scheme of things, we’re entering a bit of a rough patch in IT, but you all have been managing to keep the org online and functioning for plenty of rough years. Don’t stop now. Your users would thank you if they weren’t annoyed by that spelling error on the button of the “About” page. So I’ll do it for them. Thank you.

  • Atlassian Extends Jira’s Reach Into DevOps Workflows

    Atlassian Extends Jira’s Reach Into DevOps Workflows

    Atlassian today added a range of capabilities to its Jira project management application that make it easier to manage software development projects, including a toolchain page that simplifies tool discovery for any given project.

    In addition, Atlassian is including a Discover tab on that page that makes it possible to track activities and set priorities by providing more context around the goals for any given project.

    Atlassian said it also plans to add scope change insights later this month and, in the first quarter of 2023, auto-detection of stuck issues will be added to an existing ability to predict when application development projects will be completed.

    The company has also added a pull request activities feed to the Code tab in Jira to make it easier to compare and contrast issues, identify anomalies and discover opportunities to move work forward along with a Code insights view for tracking branch, commit, pull request and deploy activities.

    The Releases tab in Jira now surfaces feature flag status alongside development, build and deployment statuses. There’s also an ability to automatically generate release notes in the Confluence wiki application provided by Atlassian. Finally, in 2023, Atlassian is committing to making available a dark mode option for running Jira that some DevOps teams prefer.

    Suzie Prince, head of product for DevOps at Atlassian, said all these Jira updates are extensions of an Open DevOps initiative the company launched earlier this year that integrated Jira with a wide range of DevOps tools. The goal is to make it simpler for DevOps teams to track project progress without necessarily having to deploy, for example, a separate analytics application. As part of that effort, Atlassian also revealed Jira has been integrated with Compass, an extensible catalog launched earlier this year for tracking software components and is available in beta.

    In effect, Atlassian is making a case for Jira to become a software development project tracking hub by automatically pulling data from various other DevOps tools. Rather than requiring a project manager to send multiple requests for updates to individual developers and manually enter data, Jira’s integrations with various tools can both pull relevant data and apply analytics in a way that makes it simpler to identify issues that are adversely impacting deadlines.

    Those insights can then be shared more easily with both development teams and other stakeholders in the business that have a vested interest in making sure project deadlines are met. It can also make it simpler for application development teams to justify the need for additional resources.

    Project management is, of course, one of those thankless tasks that doesn’t always get the appreciation it deserves. However, as it becomes easier to automatically collect data about the status of any given project, the task itself should become less intrusive. Hopefully, as those tasks become more automated, the friction that often exists between application development teams and project managers will also subside.

  • Focus on What You Can Do

    Focus on What You Can Do

    Here we are in the second half of 2022 and the writing is on the wall. No matter how much budget you have had in the last few years, be it large or small, it is going to shrink. This is cyclical, and should not be a surprise. The amount of budget shrinkage may be larger, but budgets have fluctuated since before any of us were born. I would argue that they fluctuate in IT more than any other part of the business, simply because much of IT is still viewed as overhead and, honestly, we’re expensive. Our salaries, our equipment, our software and our SaaS cost a lot, making them juicy targets when corporate belts have to tighten.

    So let’s talk about what to do for the next year (or five) while things finish sinking and then start to recover. New projects will be few and far between but improving what an organization already has deployed costs little. And the “Do it now” mentality of Agile and DevOps has helped in a ton of ways, but honestly leaves a trail of “could be better” implementations behind. Implementations that you can use iterations to improve. My “Look at me! I’m a pundit too!” recommendations would be:

    1. Review
    2. Consolidate
    3. Evaluate
    4. Iterate

    Review what you have across the entirety of the org. I’ve made this recommendation before—possibly even as recently as six months or so ago—but it bears repeating. We had a phase where teams and even individual developers made choices for a product or product line. This created a lot of overlap. List tools used at each step of the DevOps chain; for each project/DevOps toolchain.

    Consolidate. In the short term, costs need to go down and be sustainable once they do. Better to eliminate software or SaaS redundancy than that other kind of redundancy. This step also means simplifying the corporate DevOps infrastructure. So compare products doing the same job and choose one. This isn’t as easy as pundits make it sound; each product brings something unique to the table. But you have to make the hard choices and reduce maintenance costs and hours.

    Evaluate. This is not hard to believe for anyone who has worked at multiple organizations, but in case anyone missed it, no environment is perfect. They all have strengths and weaknesses. The best staff in the world, given unlimited time and budget, will do at least one dumb thing. Then after a year, a lot of the smart things they did look dumb too—because the market and/or the technology have moved on. Look at each step of your environment—from which libraries/modules are included to the deployment steps and run-time monitoring. Figure out what could be better and then organize by cost and schedule them out.

    Iterate. Start back at the review stage and do it again, because if you think you got it all the first time, you’re wrong. Or you are a heck of a lot smarter and more thorough than the rest of us, one of the two.

    This will keep the team improving the corporate IT environment while outside factors keep budgeting down. This cycle is different from others because it is unclear how IT staffing will be impacted. While it is looking more and more like IT will have the same impacts as usual, at the start of this downturn that wasn’t at all clear. Either way, improving the environment without million dollar projects is useful in hard economic times. My company is currently reconsidering our choice of frameworks and how we’re still using a test tool as DevOps management tool, for example.

    Keep the lights on, keep rocking the servers and apps running the org and focus improvements around factors you can’t change. But keep making it better, no matter what’s going on around you.

     

  • Lock Down Your Toolchain

    Lock Down Your Toolchain

    We have done amazing things with Agile and DevOps, increasing IT responsiveness to levels that most people would not have believed and our business counterparts only dreamed of even a decade ago. Think about it—we can check in a single source file and kick off a chain of events that involves half a dozen or a dozen software tools that collectively shepherd an entire application through the development process to deployment. Through all of that, only deployment might require human interaction, depending upon corporate policy. In many places, even deployment does not require a person. Just development.

    We didn’t get to this point by moving slowly, and sadly, we did not get to this point by acting in a security-first manner. Thanks to the security-toxic mentality that DevOps was born in, we have had to go back and try to bolt on security where it was ignored initially. But we’ve largely managed that process. We have tools that handle privileges, monitor connections, scan source, look for vulnerabilities in support software, even scan entire container definitions before the build.

    Know what most of you still don’t have? A truly secure toolchain. This is bad policy. A knowledgeable attacker is going to go after one of the aggregate locations. Places like container management servers that give them access to a lot of resources. Or the toolchain, which gives them access to every bit of software the organization develops. If an attacker wants to do code injection without detection, the toolchain in charge of builds and deploys is it.

    And yet, for ease of use and keeping up with iterations, many of you have credentials for the toolchain stored in easily accessible places so that whichever tool is running the show has access to credentials for each piece of the toolchain. I wish I could say “No one keeps that data in flat files anymore,” But alas, I cannot.

    One of the reasons it is not sufficiently secured is that build toolchains fall somewhere between Dev security and Ops security. That means our teams that came from a development background are not focused on security for the build chain, and our teams/team members that came from an ops background are not focused on the toolchain selected and often initially deployed by developers.

    Which brings me to my recommendation. Put security professionals on the toolchain to verify it is locked down at every step. There are a lot of places for the toolchain to be insecure, not just poorly stored credentials. And the toolchain touches all of IT’s crown jewels.  So make sure it isn’t wide open. If your organization doesn’t have dedicated security staff, bring in consultants to review the toolchain and make recommendations. There are some great security teams for hire out there that will be happy to lend a hand.

    And keep rocking it. Your teams are building and deploying at a rate never seen before—even if DevOps is “slow” at your org, it is still likely faster than its predecessors—for doing anything. So keep working miracles, and lock down your castle’s the inner keep.

  • At Some Point, We’ve Shifted Too Far Left

    At Some Point, We’ve Shifted Too Far Left

    Those of us involved in DevOps have a tendency to see the world with blinders on. It is rather easy to fall into the “If all you have is a hammer, everything looks like a nail” trap. We have used the phrase “shift left” with the attitude that this is the solution to every problem in IT.

    Just for clarity, it is not. That is as blunt and unambiguous a statement as I can make. DevOps is great and has helped in many areas, but it is not the only solution, and not every problem is a DevOps toolchain problem.

    In the end, much critical work in IT is not DevOps-related, nor can it be impacted by DevOps. I’ll mention two areas that are obvious and talk in-depth about one of them.

    First off, no matter how far down the Agile/DevOps trail you travel, the wagon you are riding in is separate and distinct from the DevOps process. If you have a data center, all that hardware is still, in the end, hardware that must be configured and managed. For the “But we’re 100% in the cloud!” crowd (who say that loudly and forcefully enough that I suspect they’re trying to convince themselves that this is a good thing)—in the end, you are still using a network and computers to connect. The massive work-from-home migration due to lockdowns made it abundantly clear that IT is needed to set up and support both. Some of the horror stories coming across ops and DevOps social media were … ugly. Employees don’t want to be—and should not need to be—IT specialists to get their job done.

    The other easy one is security. Yes, insert security into development and DevOps workflows. Yes, make operations staff more aware of security. Yes, automate security as much as possible because we’re chronically short of security staff. But, in the end, lots of security is external to DevOps. While Kubernetes security is absolutely DevOps, password management, for example, absolutely is not. And that’s just one of many, many examples.

    So I am going to ask (but not answer) the question: Where is the line? How far left is too far? The answer is yours to decide with regard to your organization and the tools/environment that exist in your organization. In a primarily cloud environment, the answer to this question is far different than in a primarily data center environment. It is rare that any organization be purely one or the other at this point in time, so there are similarities, but few exact solutions we can name.

    Security is akin to a cost center. It is actually insurance, but few orgs view it that way. And even if they did, it would not make much difference. Much as my father once declared that he and Mom were “insurance poor” and canceled half their policies, most orgs want to contain the costs of security whether they admit it or not. When ‘security merged into DevOps’ hits the ears of people trying to stretch budgets just as far as they can, bad things tend to happen.

    DevSecOps implements security where it should have been all along—at the development stage—but it does not replace other security, it just makes our apps more protected. There are cases where security testing hours are seriously reduced, but since testing in all forms is the first thing to go when staffing is tight, again, it is not replacing something that was happening—it is making it possible for us to do what we should have been doing all along.

    It doesn’t help that our toolchains have taken on lives of their own—at many organizations, staff are required to maintain them—because they, too, are apps that DevOps struggles to help. Once you’ve invested time and money in building that edifice, you want to use it as much as possible. So, many orgs want to shift left everything in IT, right into the toolchain octopus.

    You know your org better than anyone. You are the ones making it work day in and day out, in all of your environment’s glorious complexity. Start talking about what is too much; when adding a step in an already complex toolchain needs to be justified. Push back against everything needing to shift left (where appropriate, I’ll say again for clarity, DevOps has done amazing things for us, but too much of a good thing is our living nightmare and, historically, is tomorrow’s technical debt). Keep rocking it, and make sure you are rocking the right things in the right places.

     

  • Update Those Ops Tools, Too

    Update Those Ops Tools, Too

    We’re all pretty well settled on the idea that the rate of change in IT has been on the increase over time. From the advent of VMs to Agile to DevOps to containers, the movement has been faster—and the same is true for internal development.

    I’m going to approach this topic from three perspectives instead of my usual two. Kind of like breaking the fourth wall in cinema. We’ll discuss this movement just a bit from the perspectives of my two preferred bits, DevOps and DevOps. But for this blog, we’ll also mention DevOps—the infrastructure, tools and processes that enable the other two.

    For DevOps, languages—both updates to those currently in use and the addition of new ones—bring with them a nearly automatic update to tools used. Favorite libraries/modules, IDEs, build tools—these are all looked at and updated more frequently today than at any point in history, by my estimation. For teams that do not change languages or libraries much, source code analysis (SCA) scanning is increasingly helping to take a closer look at all those included tools and filter them based upon impact (normally security impact, but other impacts on occasion, too—like project activity and stability).

    For DevOps, the toolchain is still new enough that evolution is causing regular change, and that change is bringing with it new tools and utilities to solve the problems of this existing infrastructure. In fact, I’ve long thought the DevOps part needed a bit of a cooldown on the change and new tools/utilities front, personally.

    And finally, DevOps has gone a bit sideways in this regard. Every person—whether you call them ops, admin or DevOps—responsible for troubleshooting machines and apps has a collection of tools. But the divergence of platforms has created a divergence of tools. It is possible to work without a dedicated set of tools—and, indeed, many DevOps engineers do. Others have a folder. Meanwhile, there is another dedicated set of DevOps engineers that have a PCMCIA or USB full of useful tools that is always with them. Some have both. Of course, what tools are included in all of this varies widely also. It is an interesting artifact of our growth in IT that developers tend to share or borrow the best libraries/modules/APIs across an organization, but DevOps is more independent. As long as DevOps team members are getting the job done, no one is all that worried about which tools they used to troubleshoot. They do tend to share if asked, but with little physical evidence of what tools they used, it doesn’t come up often.

    So, what am I getting at? Look for ways to enable DevOps tool sharing on an ongoing (or at least regular) basis. An internal social media channel dedicated to the vendor your teams use to communicate, brown bags, a maintained rating system “Why I think fdisk rocks” type of thing, etc. The more power there is at the fingertips of every DevOps member you have, the better. Given our history, building an “approved” list is likely a bad idea, but having that central communication location that security can also peruse to look for red flags is not a bad idea. Of course, they are likely too busy, but worst case, such conversations can help in forensic analysis later. Not being cynical, just living IT in the 21st century.

    And keep kicking it. More and better tools will help keep all those sweet apps and servers clipping along, so there is really only upside here. Give your teams the tools and revel in your considerable uptime.