devopsy.kinsta.cloud

Author: Don Macvittie

  • Convert COBOL Easily? Yes, Please!

    Convert COBOL Easily? Yes, Please!

    I was reading Mike Vizard’s article about IBM previewing an AI tool to convert COBOL to Java and mentally cheered this move. No, seriously—massively cheered this move.

    There are a lot of uses of large language models (LLM) right now, and many of them are not all that exciting, in my opinion. This one, though? This is huge. In the course of my career, I have been involved in three projects to convert COBOL and one to convert FORTRAN, with some projects that required interfacing with RPG III systems. In all of these cases, the costs were prohibitive. Two of the three COBOL projects never made it past estimates because the cost of resources who know both COBOL and Java were prohibitive, as is the complexity of most mainframe applications as they were modified over the years. The Fortran project never got past resourcing, and the RPG III interfaces were just that. We had to pull the data as it was presented by RPG III systems and only act on our copy, with a separate process used to update the core systems. Why? Because modifying the RPG was impractical, but we could be agile. It was … painful, to put it mildly.

    What all of these projects shared was the lack of a skilled workforce. And what IBM is offering is an LLM that acts as that workforce. If you get the application into Java, there is a large pool of developers that can then troubleshoot it. While the environment issues will still exist and people skilled in tools like JCL are increasingly hard to find, that part can be rebuilt in the new environment. (One question I have about IBM’s project is, “Can it convert to Java for a Linux LPAR/VM?” I suspect that this iteration converts within z/OS, but it would be cool to see increased options in the future.)

    Honestly, this trend will only increase. While IBM’s use of LLM is aimed at moving COBOL to Java, there are an incredible number of lines of Java out there to train on, so soon enough, there will be other conversions in and out of Java. And that’s all good news. We recreate the wheel a lot. The longer-term implication of AI-assisted conversion is that we won’t have to. Think about it: We could feed in a mathematical routine in Fortran and get the language of our choice out the other side.

    Way back in the day, a collection of compilers named TopSpeed let you develop in any of several languages, even mixing and matching to create applications that used each language (and developer) for their strengths. This is the TopSpeed dream writ large. (Disclaimer: I was a huge TopSpeed fan. Bought their intro kit and wrote systems mixing C/Pascal/ASM without having to mess with things like parameter sequences. Yeah, I was cool.)

    We have had too few developers for a very long time. Now, we are about to have a world where developers can focus on truly unique functionality, and the rest is generated based on the mass of implementations we already have. Think in terms of sorting algorithms—99% of the sorting problems out there were mastered decades ago, and yet we frequently see people re-implementing, even today. Now we can tell a system, “Generate a tree sort,” and just make calls. Every time we implement a new language, this type of thing has to be re-implemented, and often it is re-implemented because we don’t know about the available solutions. In the future, we will have solutions at our fingertips.

    And as with the other benefits of automation, we’ll do less busy work and add more business value. While there will be some pain, I think in the long term that is a huge benefit. And you’ll still be there, making things go zoom.

  • What Are You Securing?

    What Are You Securing?

    Every once in a while, I think it is important to look at the available technologies and tooling and ask, “What are we trying to do?” In the case of information security, we need to ask, “What, exactly, are we trying to protect?”

    The answers to that question are seriously limited. Availability, reputation, data. Since reputation is a business mapping of the other two, that leaves us with availability and data. Of the two, data is more important. Our systems being down absolutely costs us money, but leaking customers’ private data or our corporate secrets, like source code or technology designs, is worse in the long run.

    The problem with these observations is that if we are here to protect the data, and data is the more important piece of security, then why do we have far better application security than data leak prevention? Almost all of you have policies and processes to run an application through the wringer and get an “as secure as possible” result at the end. But data? The purpose of applications is to expose it. And in the process, we risk exposing too much of it or allowing tons of it out of a valid endpoint. I remember in the early days of login-based websites, there was a spectacular leak that let an attacker get hundreds of valid user credentials per page submission (there were a surprising number of these for a couple of years there; this one was at a college). We are facing much the same thing via APIs these days or even via pages meant to make access easier for users.

    Locking all of this behind logins is a useful step, but to use one of the many phrases we’ve stolen from military science, it is only a part of ‘defense in depth.’

    Once it is agreed that data is one of the two key things we need to protect, then the focus should be on how to do so. There are some amazing DLP products, both standalone and integrated into other technologies, that are worth looking into. There are also some great advances in automated response under the guise of AI/ML that should be watched closely because they are changing rapidly. Limiting the maximum number of responses allowed from a query is almost mandatory for application stability and for data leak prevention. And it isn’t that hard to implement.

    I assure you that ne’er-do-wells—particularly state-sponsored and corporate espionage hackers, but also amateurs—are using AI to make attacks more successful and less detectable. The least you can do is respond in kind, with some of the tools that can now do things like notify you about low-and-slow data leak attacks.

    Keep focused on what the goals are. Reiterate them often, and keep rocking it. That’s a lot of data splashing around the organization that your team is responsible for; make sure it’s as safe as possible.

  • Low-Hanging Fruit, 2023 Edition: Part Two

    Low-Hanging Fruit, 2023 Edition: Part Two

    Last time, we discussed setting up a comparative inventory system for your growing API footprint. The idea is that as security catches up to new technology deployments, enterprises will have to step up their game and implement those new technologies. API security is currently the biggest need because of exposure to the world, but not far behind it is container security, which we’ll discuss here.

    The point of container security is to make certain that your images are as secure as possible. Container security looks at source container definitions, added libraries and configuration options inside the container definition files ( or for those of you who prefer them, Docker files).

    A Tempting Target

    We are spinning up a bewildering number of containers these days, and having source images–particularly if they come from the outside–is a tempting target to attackers. Imagine if an attacker can hit the base container definition that all of your “golden images” come from and just insert a call to download a library. Soon, your entire application infrastructure could be a tool for that attacker to do as they wish. And if they’re cautious, they might get away with it for months (as many have).

    The Definition of Containers is Security

    So there is absolutely an imperative to ensure that the definition of containers is security–all of your containers. And there is an imperative to know what software is installed via container definition and what software is actually running on container instances. Much like our discussion about static versus dynamic API scanning, bad actors can modify a running container, so drift detection is half the problem, even while scanning registries and definition files has more appeal from the blast radius perspective.

    There are a lot of tools and vendors that range from simply scanning the files and generating an SBOM to complete container security by scanning the images, using the SBOM to perform software composition analysis, using data feeds to determine which software in the SCA is a threat, and how urgent or broad that threat is and monitoring deployed containers for drift. The more of these an organization implements, the more configuration and maintenance effort, but the payoff is security at a more fundamental level. The whole appeal of a container architecture is recoverability–if things go terribly wrong, we can redeploy easily. But if the source files hold the vulnerabilities, redeploying is temporary at best and likely would warn attackers that you’re onto them.

    Be Proactive

    So a more proactive approach is required. Obtain and install tools that will help you protect your container infrastructure, be they sold as container security, cloud workload protection, SCA or something else. The key is locking down this avenue of attack at the beginning of the software supply chain, not where you get the tools.

    And again, keep rocking it. As I said in the other post on low-hanging fruit, we need this because you all are serving the organization’s needs with the best solution for the job. Now that security technologists understand your new problem domain, let’s get tools into place to continue the protection of all that shiny new stuff.

  • Low-Hanging Fruit, 2023 Edition: Part One

    Low-Hanging Fruit, 2023 Edition: Part One

    We’ve been cranking it out, haven’t we? Generating cool new toys for the business, creating new infrastructure to host those new toys and trying our best to secure all this new stuff while maintaining both our sanity and our cadence. And we’re doing it. Kind of.

    One thing we’re naturally lagging behind on is security. While we don’t purposely shortchange security in our DevOps processes these days, it always takes a bit to figure out the best way to secure new technologies after they’re implemented. And we’re implementing a ton of new technologies. So our security staff (and increasingly, DevOps/SRE teams) is rushing to play catch-up. One area where this is screamingly apparent is APIs. We’re spewing them across the organization, often without cataloging or responsibility. This makes sense, as in many cases, the API is just an evolution of the function call. “I need a service, it is a good generic service, let me throw it up and we can use it from wherever the client ends up deployed,” type of thing. The problem is that leaves a ton of attack surface not truly protected. Because without cataloging or responsibility, there is no way for security to know about many of these APIs.

    API Scanning Technology

    That brings us to today’s topic. If you haven’t done so yet, it is past time to look at API scanning technology. It comes standalone and bundled with pretty much any API security product on the market and is increasingly bundled with other development security or even development tools. And the whole point is to get a list of APIs. Oh, most commercial implementations do something with that list–include it in static testing, drive it into WAFs, whatever—but just having the ability to inventory what is running on your network is reason enough to stop doing less important stuff and figure out what your exposure is. And the tools make it easy. They come in two flavors, and we should consider both.

    Static API Scanning

    The first type digs through source code and API specs to come up with the list of what is supposed to be running on the network. The big weakness of this method is that it only knows what it is told. This is static API scanning.

    Dynamic Scanning

    The second watches network traffic for API calls and then documents them, possibly even generating API specs for you. The big weakness of this method is that it only discovers what is being accessed on the network while it is watching traffic. Currently, unaccessed APIs are not caught, nor are ones on unreachable networks. Unsurprisingly, this is dynamic scanning.

    As usual, the two together are more powerful than apart. By implementing both technologies, an organization can work toward a hardcore comparison of what should be running on the network versus what is running on the network. This goes beyond detecting unreported APIs, it also means that zombie APIs or runtime drift can be discovered by comparing what is there to what should be there.

    If you’re not doing this, consider adding it to the toolbox. Setup is not onerous, and the reports are lists of APIs and parameters that can be reviewed at leisure. Some commercial products that offer these two functionalities take it quite a way beyond list generation, and if comparisons are made, work hard to understand just how much each toolset can do.

    And keep rocking it. We need more tools to lock down the new perimeter precisely because you and your team are tackling new technologies and platforms to better serve the business. We are increasingly seeing security tooling catch up with the broad selection now available, so go implement them.

  • A Distributed Workforce is the New Normal

    A Distributed Workforce is the New Normal

    The pandemic emphasized and improved what we have known for decades: A team does not need to be centrally located to be productive. This varies based on the people involved and roles being fulfilled, but honestly, lots of people can work just fine from home, a coffee shop, the pool …

    And while we could debate the wisdom and usefulness of lockdowns forever, they did drive organizations that had never considered distributed work to actually take the plunge. Many are returning to centralized organizations, but a significant number of employees have moved to where they would like to live to work from home, and some of those employees will not be back.

    I normally avoid reference to any contentious societal discussion, but the lockdowns and pushback directly impact us in IT. The circle that was our network had already expanded to include public cloud, hosting providers and SaaS providers. Now it includes all of those plus some–perhaps only one or two, perhaps many–home offices. And that is unlikely to change, no matter how hard an organization tries. If nothing else, motivated individuals will want to work from home after hours, and if they are salaried, the organization will support that approach—meaning we in IT will have to support that approach.

    A deep look at what that implies is necessary. One of the things that having a sudden “work from home or shut down” mandate imposed upon companies did was make security an afterthought. For the first couple of months of 100%-remote work, we saw crazy numbers of access misconfigurations that caused breaches. That’s just the easy one. Your source is in a publicly accessible repository. Are the tools to access that source secure? Are you sure? What about SaaS? Is there a method other than user-generated passwords required to access each SaaS? Do you even know what SaaS systems business users have adopted? Does the organization use password management? It is necessary since SSO isn’t covering everything, and really can’t.

    The same is true for accessibility. Management teams started with, “Let’s get people working,” because the alternative to people working was layoffs or even corporate bankruptcy. If you paid any attention at all in that timeframe, you heard the horror stories of IT going to employees’ houses and setting up or troubleshooting corporate desktops that went home. It was an ugly time, and only afterward did organizations start to consider options for handling communications. Is it policy to have someone go to a nearby coffee shop when their connection is down, or is that a day off? Can they work without connectivity and then upload afterward? Most organizations still don’t have coherent policies in this regard, allowing teams to decide. In fairness, not every employee is great at working from home, so it is possibly the best option to have team leads deal with failed communications issues … But a corporate policy or guidelines should still be developed, and technical solutions like satellite backup should also be considered.

    Organizations rushed to WFH and then rushed to return-to-office (RTO). Your job is to make sure that, with all of the rushing, things are stable and secure. You’ve been rocking it all along—don’t let shifting corporate priorities drag you down now.

  • Automation or Concatenation?

    Automation or Concatenation?

    We’ve been in a period of mergers and acquisitions for a while in the DevOps and DevSecOps spaces, and it seems to be heating up even more. Combined with products expanding to cover a more broad segment of the software development life cycle (SDLC) and strategic partnerships to cover more of the SDLC, it is starting to look like our automation efforts are going to be increasingly driven by vendors. Those efforts will be more concatenation than automation–though technically the same in perception, they vary wildly in implementation, and we should be planning for it.

    Build Versus Buy

    We have always struggled with build versus buy, and we have always struggled with best-of-breed versus overarching solution set. The entire hype of Every. Single. Vendor. Claiming to be a platform is partially driven by the latter argument. Putting “platform” in your marketing does not make a vendor better at those things outside its historical strengths, though, so that movement is doomed to fail for most organizations. Some were already moving to be a platform and had assembled–by M&A or by internal development–an impressive platform already, and those were successful because they were good at a breadth of things. The best of them are continuing this trend. I try to avoid plugging vendors here, but I think an example is in order in this case.

    Akamai’s recent purchase of NeoSec is the poster child for vendors doing the platform thing super well but not resting on their laurels. I’ve followed Akamai’s security offerings for years, and the platform has impressive functionality, but NeoSec is exactly what they needed. It plugs a couple of holes in their security offerings and brings a whole customer base into the Akamai fold. That’s what I’m talking about when I refer to the good ones.

    The problem is the acquisition and expansion of offerings trend is starting to gain a lot of steam, and not every vendor jumping on the bandwagon does the job as well as Akamai does.

    This leaves enterprise IT staff responsible for DevOps and DevSecOps – even straight security staff – with a new set of issues. Best-of-breed is taking a hit again, and overarching solutions are on the rise. The difference is the breadth that some of these new “platforms” offer is massive. And we know that even the good ones will have better and worse parts of the overall solution. So IT now has to start considering what vendors call a feature and how it individually compares to the competition. This will require more up-front work for evaluators. Now, a larger (and growing) list of “features” that were once “products” has to be considered piece-by-piece against the competition, and organizations must evaluate the feasibility (and cost) of implementing two (or even multiple) massive platforms to get the most critical functionality needed.

    As of this writing, point solutions that do one thing extremely well are holding up, but the trend is obvious. Those markets will be subsumed into larger offerings that may not do as well at other parts of the overall problem. Evaluators will have to be more careful and consider their options with an eye to the larger infrastructure and problem domain. Do you want a single vendor doing all of the steps of DevOps or responsible for every aspect of application security? That will be an individual decision and, if not, how the final solution/architecture looks will be a more complex discussion.

    Integration and Automation

    In our quest for integration and automation, many vendors are responding with the concatenation of tools to solve the automation part. You’ll be stuck figuring out if they’re doing it well enough for the org.

    But that’s not a problem you can’t resolve. You’ve been rocking it through staffing issues, lockdowns, growth and shrinking… Just keep rocking it. Choose the solution that makes the overall best sense for your organization, get it running and take a break; head to the beach for some relaxation time. Or whatever you do to clear your head – because the next problem will still be there when you get back, and it’s better to be relaxed when you tackle it.

  • Defining ‘Best’

    Defining ‘Best’

    (Note: For the last few years, I’ve been sparse with naming vendors in blogs. This one will have more than usual as examples.)

    We all have a ton of tools at our disposal. From installed tools to products to open source, we can solve any given problem in fifteen different ways. Several years ago, I wrote a blog about using what you have to get the most out of your purchases. There are several “platforms” that solve a lot more problems than just what you are currently using them for. At the time, my shining example was F5’s BIG-IP. That thing can do everything: Networking in your data center and probably cooking your breakfast, too. At the time, most people used the box to handle load balancing but then were concerned about the cost. Duh? I mean, you have a tool that can solve a fistful of problems … and you’re using it for load balancing. I mean, it’s very good at load balancing, don’t get me wrong. But I feel like quoting Marvin from The Hitchhiker’s Guide to the Galaxy. “Here I am with a brain the size of a planet and they ask me to pick up a piece of paper. Call that job satisfaction? I don’t.” (Interestingly, at the rate that generative AI is advancing, it is only a question of time before a machine throws that quote out there.)

    The thing is, investors and the hype machine told all of your vendors three things. First, that they needed a recurring revenue stream. Second, that you wanted everything as-a-service and third, that they needed to be a platform. That means that those fifteen SaaS products you’re on the hook for now want to solve all of your problems because “they’re a platform.” Some of them do a kick-arse job of it and you might very well have a SaaS license that covers whatever you need to do today.

    This is screamingly obvious for some vendors. If you’re an Akamai customer, for example, the breadth of things you can do through them is stunning. Most customers do not seem to take advantage of that to the extent they should. This is no different than that BIG-IP in your DC; it’s just as-a-service. You’re already paying for both. You should make the most of them.

    Other vendors might surprise you but  they have done the same through M&A or expanding their “platform” from marketware to reality. I have hinted at this in the not-too-distant past but will say it point-blank here. Know what your products are capable of. Tomorrow you might need some of that functionality, and knowing if it’s already in the organization (but not being used) is a big deal. Also, if the “platform” already fits into your organization, then choosing a ‘best’ solution leans toward that product line already–less integration is the best integration.

    And keep those lights on. You may have more tools than you need, but you’re managing them and keeping the organization online. You should be getting kudos for that, so I’ll offer them. Thank you. I replaced my phone this weekend and had to reconnect everything and to all of those orgs I had to log back into, reinstall or reconfigure, let me say: I am grateful for each IT person there. Even my bank, who honestly pisses me off—I’m still grateful for their IT.

     

  • Monitor Toolchain Health, Too

    Monitor Toolchain Health, Too

    A funny thing happened on the way to being agile. We adopted a ton of tools to make us more responsive. They really do help us deliver software quickly and efficiently, and they have improved IT processes along with our specific process improvements.

    They also created a toolchain that resembles areas of responsibility of old. That makes perfect sense; we still need to store source code, build applications, deploy somewhere, worry about security, etc. And we need to do all of that faster, which is where automation and the toolchain really shine.

    But you need to keep in mind the overall impact of the toolchain upon the development process. Now the organization is dependent upon those tools. We went from slow and steady to fast and with a large series of steps along the toolchain. Things are better, we’re delivering quality software at a faster pace, but in some ways, the process risks becoming more fragile. If a tool in the chain shows signs of trouble, you need to be aware of it and on top of the issue almost immediately. The price of ignoring issues in the process can be catastrophic disruptions in software delivery. Losing a CI tool is akin to shutting down customs at an international airport: The source of the problem is big, and the downstream problems are show-stopping.

    So while you are tooling your applications, make certain you are tooling the build chain, also. Know when a build that always takes ten minutes suddenly spikes to triple that. Know when a test suite delivers non-breaking errors at an increased rate. Know when space is low on a shared drive that the build tools use. Tool everything, because the software delivery toolchain is now the core of application development and deployment.

    Tooling these systems can also offer a warning of problems in a given application. It is possible that the increase in build time is because there are a lot of previously unused libraries being included … and that those libraries have not been properly vetted by security. It is possible that the disk space on that shared drive has PII test data that needs to be cleansed. There is a lot of good that can come from over-tooling the build chain, and unless you have a bit of tooling that slows the process, there is no downside. It won’t take too long to build a list of what to watch, and modern DevOps tools have enough reporting and APIs that it won’t take long to begin watching those items, either.

    That is where the people part comes in. Don’t ignore the tooling once it is in place. Build a process to regularly review results and take action when necessary. No tooling is useful if the results are ignored; sadly, we have a long history of spectacular failures that started with, “We saw indications, but didn’t think it was important at the time …” So act on results, even if that just means flagging them for review in the next reporting cycle—just don’t ignore them.

    And keep rocking it. The organization thrives on the software you build, deploy and support. It is a massively complex system that has IT staff at its heart, so keep it thriving and watch for indicators of illness in the core toolchain.

  • Identify Needs First

    Identify Needs First

    In several projects of late, I have noted a trend in otherwise rockstar IT departments: They identify an area that they wish to implement or approve and then set out to address it. So, “We need a new CI tool,” spawns a project to choose and implement a new CI tool, for example.

    But there are so many details that these projects are missing from the start that should have been put right into the charter/project kickoff documents.

    Let’s start with, “What CI tools do we currently have in place?” This is the 2020s. How is it possible we are still starting these projects as if there was no solution in the virtual building already doing the job? We need to start from, “Here are the tools that can do this job that are currently running in our environment.” Only then should we move on to, “Here are other options to consider.” But I will humbly suggest that if there is a single tool that can solve your issues, it should be used–and if, of all of the tools running in the organization, none of them can solve all of your issues, then a new product should be chosen that can.

    The same is true for things like alerting. When I am analyzing a given vendor’s solutions, I ask about how many places they can raise issues–it is important that a vendor asking you to purchase their product can report and alert to whereever the organization needs it to. So that could mean Jira or ServiceNow, Slack or email, centralized logging—all of the above and sometimes more (SIEM for security events, for example) need to be supported. But when a given organization is implementing? Choose a reporting method.

    The most common is, “We do Slack, but we also do email. And, of course, we coordinate though issue management like Jira …” Stop. Seriously. This is not about, “Oh, they made it easy, so we just implemented every option.” This is about, “We need to keep track of conversations and resolutions.” If two people are discussing an issue over Slack while a separate team is discussing over email, you’re failing to communicate. Even if there is linkage to ServiceNow or Jira and that is where conversations should happen, we all know that final resolution—but not troubleshooting steps—is what ends up there. So offer one robust and widely available alerting mechanism and let people run with it.

    That type of redundancy is all over IT. Organizations that would not implement per-team email solutions (not servers, solutions) will happily allow for per-team DevOps toolchains. Why? What, exactly, is different? The purpose of a mail server is to deliver mail. Choosing which mail vendor/service to run with is based upon how a given product does so and what additional services come with it. How is that different than, “The purpose of the DevOps toolchain is to deliver applications, and selection is based upon what vendor/product best serves the way our organization does so?” There isn’t a difference; we imagine one. We latch on to one feature that we must have to keep our current favorite, and the compromise is that the next team over can keep their tools, too, because they must have some other feature.

    Evaluate what the organization as a whole needs, identify tricky bits like must-have features and determine if that really is a requirement. If it is, ask why it isn’t a requirement for every project not using that tool.

    And keep moving those apps along! You are building and deploying more apps more securely than ever before. Continue to sanitize the build and delivery process, and continue to crank out apps that make the business hum. But don’t make it part of your job to support fifteen tools that all do the same job. Streamline and improve. And keep rocking it.

  • Must You Collect All That?

    Must You Collect All That?

    It is interesting that we no longer consider teams that super-tool their apps as being slightly ‘off’ or, working in a field where systems were mission critical, even life-and-death. Now, we all are collecting and reporting data points in real-time across not just the SDLC, but across the entire environment. It is wild the amount of data available to the average DevOps team about what is going on with their applications. But it’s not all being used.

    No, seriously, it is unreal. Let me offer two anecdotes that explain why I know you’re not using it all.

    Most of my regular readers know that I worked at F5 for years, the first couple on the DevCentral team, doing the Dev part. Their flagship product, the BIG-IP, collected an incredible amount of super-granular data. I am not exaggerating when I say an incredible amount. They had data points from the wire up to the traffic routes through the load balancing algorithm to nodes. And it was (and likely still is) all available to users via APIs. Whatever data was needed about traffic passing through the BIG-IP or the nodes behind the BIG-IP and the connections in between users and those nodes, you could get.

    One of my projects was to offer a simplified interface to the APIs. They were complex and granular, and customers wanted fast and easy, so I was publishing code that worked as an intermediate layer.

    Note: F5’s APIs, like everyone else’s, are far more advanced today than they were then, and it is easier to do what you need to do because they took the “create a user layer” part seriously. It’s complex machinery so it still has complex APIs, but accessibility is way up from those early days.

    Anyway, BIG-IP collected so much data that I could not name all of the fields, let alone when each one was pertinent. I gave up trying after a while, because if a board-level engineer needed a data point, the idea at the time was “Why not expose it?” so ‘number of connections to a given node’ under this algorithm was hidden among things like ‘microseconds to connect to DNS on that connection’ (I’m making those up; it’s been years). Some were terribly useful to end users and others were not. All were accessed in the same way, and learning by experimenting was the only real way to tell the difference.

    Users did not need all of that data. Users did not need the complexity and obfuscation created by supporting all of that unnecessary data.

    My other example is a massive system of interconnected IoT devices that were centrally connected/controlled with a private, specialized network to manage them (actually, there were several different networks on several different mediums, and that matters where data points are concerned).

    When setting this system up, we estimated about half a million IoT devices with capabilities that ranged from passive monitoring to active controllers of physical devices. Four different communications mechanisms, a couple dozen different devices, every major (and a couple minor) databases and command-and-control software on every major OS, managing from a handful to hundreds of thousands. And all of these were super-tooled because the IoT network was mission-critical. Our databases and the infrastructure to support/protect them were a major part of our data center budget because of the amount of data.

    The amount of data was a flood. The management software was developed by different companies with different transports and endpoints in mind, meaning each had a different dataset. Once all of the networks were running satisfactorily, our next immediate project was to create a unified dataset for reporting and troubleshooting. I was in charge of IT, but the top-level person–because each of those half a million devices was a physical install–was a physical engineer. We fought non-stop about what was important. I was trying to map and convert datapoints between systems; he was trying to maximize data available for reporting up. He often said, “Well no, we don’t need that today, but we might.” This put us in a zone where “number of devices on this system” was on par with “attenuation of the communications line to the nodes on this physical subnet.” Totally unworkable, and our compromise was to put together a small IT team to put a layer over all that raw data and offer an intelligent interface side-by-side with data dumps.

    We are there again today. I hear many of my friends in the enterprise space saying, “Because of ML, we might need that data and storage is cheap.” I disagree. Storage is an overhead that is wasted if you are not actively using the data. We are past the “all data is good data” craze; it’s time to adapt. Useful data that expands insights is all you need tooling for. Report data that will help solve problems, and if the team finds tomorrow that it needs a data point that is not currently collected, it can be added in.

    There is room for debate here–the rate of change for ML systems (please stop calling them AI; they’re not, yet, and we’ll need that word when they are cognizant) means that there is a reasonable argument for “all of the data.” And I’m not proposing limiting data collection. Honestly, application security specifically and enterprise IT security overall has benefitted from datapoint sharing between formerly separate market spaces. If the SCA scan shows X framework is used, then the IAST can narrow down tests to those that look at X and not its competitors, for example. But honestly, don’t drown yourself in a data lake when all you need is a fish tank where you can see all the guppies.

    And keep rocking it. Even if you entered IT two years ago, you are now a seasoned vet and things have changed–beyond all of our expectations, I’d say. Yet you’re still plugging away, keeping the enterprise running. Don’t stop—just try to keep the data hoarding to a reasonable level and make sure the data you have is being used, whether as useful feeds to an ML tool or as consumable information for other people/systems.