devopsy.kinsta.cloud

Tag: IoT

  • Defining DevSecOps for IoT

    Defining DevSecOps for IoT

    The Internet of Things (IoT) has brought plenty of benefits to companies and consumers, allowing businesses to track inventory in their supply chain, landlords to measure their tenants’ energy consumption, and pet owners to find a lost cat. It’s become a significant part of the Internet infrastructure, too. The number of global IoT connections grew by 18% in 2022 to 14.3 billion active IoT endpoints, with no end in sight.

    But for all the technology’s advantages, it also presents new security challenges. IoT technology uses network connections to facilitate communication between devices and systems. When devices are IoT-enabled, they can collect and transmit data in real-time, enhancing efficiency, safety and convenience across various applications and industries. All that enables great technological feats… and gives bad actors new opportunities.

    IoT device scale and diversity: Each device, from smart refrigerators to industrial sensors, operates with different technical architectures and security capabilities. This diversity makes it difficult to implement a one-size-fits-all security strategy. The sheer number of devices connected to the Internet makes it challenging to secure every endpoint against potential breaches.

    Proliferation of security vulnerabilities: IoT security vulnerabilities typically arise from inadequate security design and practices. Many IoT devices lack robust security features due to cost constraints or oversight, leaving them susceptible to attacks. Devices could be shipped with default, easy-to-guess passwords, or without the capability to be updated or patched, making them easy targets for hackers.

    IoT ecosystem complexity: IoT ecosystems typically involve multiple interconnected devices and systems, each adding layers of complexity in terms of management and security. Managing these systems requires careful coordination and a clear understanding of how information flows and is processed within the network. This complexity often results in security being an afterthought or being inconsistently applied.

    Decentralized network devices: As security threats evolve, IoT software must adapt through regular updates to address new vulnerabilities. However, the decentralized and widespread nature of IoT devices complicates the updates’ delivery. Ensuring all devices are consistently updated is challenging when devices are geographically dispersed or lack remote update capabilities.

    Enter DevSecOps

    Most organizations that are cognizant of security are familiar with the concept of DevSecOps and likely employ people in that role. DevSecOps introduces a security-first mindset to the DevOps process, ensuring that security and compliance measures are integrated from the start. The main objective is to create a collaborative environment where the security, development, and operations teams work together to address security issues, reducing the potential risks and vulnerabilities.

    As in other software development projects, it’s important for programmers and system designers to work with the security team and DevSecOps in particular.

    Incorporate automated security testing tools into the CI/CD pipeline: This helps address security throughout the IoT development process. These tools automatically scan for security vulnerabilities in the code from the earliest stages of development through to deployment, making it easier to identify issues before production.

    Ensure all IoT devices are securely provisioned: By securing these devices from the point of manufacture, it becomes harder for unauthorized users to gain access or compromise the device.

    Implement MFA: Multi-factor authentication (MFA) requires users to provide two or more verification factors to gain access to IoT devices, adding an additional layer of security beyond just username and password. This authentication method is particularly important in IoT environments, where devices often collect or disseminate sensitive data.

    Establish regular update processes: Establishing a process for regular firmware and software updates is critical in maintaining IoT security. These updates often contain patches for recently discovered vulnerabilities that could be exploited by attackers. Ensuring that IoT devices are always running the latest version of software helps protect them from such threats and maintains system stability and performance.

    Plugging In

    The integration of DevSecOps principles into IoT development is useful for addressing the myriad security challenges posed by the diverse landscape of IoT devices. By embedding security practices at every stage of the IoT development lifecycle, from design through deployment, organizations can significantly enhance the security and integrity of their IoT ecosystems.

    This approach not only mitigates risks such as unauthorized access and data breaches but also supports the seamless and secure operation of IoT devices. As the IoT continues to evolve, embracing DevSecOps will be crucial for ensuring that innovations in this space are both robust and reliable, safeguarding the digital and physical assets they control.

    Photo by Jorge Ramirez on Unsplash

  • DevOps Strategies for Connected Car Development

    DevOps Strategies for Connected Car Development

    The concept of smart, connected vehicles is no longer science fiction but a day-to-day reality. These vehicles, empowered by internet connectivity and a suite of digital technologies, offer unprecedented convenience, safety and entertainment features. 

    But as the cars on our roads become smarter and more interconnected, the complexity of developing their software systems multiplies. DevOps is emerging as an indispensable strategy for organizations in the connected car industry, bringing together software development and IT operations to streamline, optimize and accelerate the development life cycle. 

    This article briefly reviews connected car technology and explores several DevOps strategies that can support connected car development. We’ll discuss how DevOps can manage the complexity of automotive software, accelerate development speed without compromising safety and provide the essential tools for efficient building, testing and deployment in a highly regulated environment.

    What Are Connected Cars?

    Connected cars are vehicles equipped with internet access, allowing them to share data with devices both inside and outside the car. Essentially, they offer a multitude of features that make the driving experience more convenient, safe and enjoyable. While this technology is still in its nascent stages, it’s rapidly evolving, and soon, connected cars will become a staple on our roads.

    These hyper-connected vehicles are part of a broader ecosystem that encompasses a network of physical objects—everything from other vehicles and traffic lights to pedestrian crosswalks and parking spaces—all connected by technology.

    The potential of connected cars is immense. They promise to transform our lives in numerous ways, from reducing traffic congestion to improving road safety and revolutionizing the way we travel.

    Key Features and Systems of Connected Cars

    Infotainment Systems

    An integral part of connected cars is the infotainment system. This comprehensive system provides both entertainment and information tailored to the driver’s needs. From navigation and music streaming to voice commands and hands-free phone calls, these systems are designed to enhance the driving experience while ensuring safety.

    Infotainment systems also offer a wide range of services, such as real-time traffic updates, weather forecasts, and points of interest. They can connect with a smartphone, allowing users to access apps, make calls and send messages without taking their eyes off the road.

    Telematics

    Telematics uses GPS and onboard diagnostics to record and transmit data about the car’s performance, location and usage. It allows for real-time tracking of the vehicle, making it particularly useful in fleet management and stolen vehicle recovery.

    Telematics data can also be used to assess driver behavior, monitor fuel efficiency, and predict maintenance needs. For instance, if the car’s engine starts showing signs of trouble, the telematics system can alert the driver and recommend a nearby service center.

    Advanced Driver-Assistance Systems (ADAS)

    Advanced driver-assistance systems (ADAS) are systems that help the driver in the driving process. These systems use sensors, cameras and radar to monitor the vehicle’s surroundings and assist the driver in various ways.

    ADAS can provide features like adaptive cruise control, lane-keeping assist and automatic emergency braking. These systems not only enhance driver comfort but also significantly improve road safety by reducing the risk of accidents.

    Vehicle-to-Everything (V2X) Communication

    Vehicle-to-everything (V2X) communication is a key technology in connected cars. It enables vehicles to communicate with any entity that may affect the vehicle and vice versa. This includes other vehicles (V2V), infrastructure (V2I), pedestrians (V2P) and networks (V2N).

    V2X communication can improve traffic flow, reduce accidents and enhance fuel efficiency. For instance, connected cars can receive information about road conditions, traffic jams or accidents ahead in real-time, allowing the driver to take appropriate action.

    Over-the-Air (OTA) Updates

    Over-the-air (OTA) updates are another key feature of connected cars. Similar to how you receive updates on your smartphone, OTA updates allow car manufacturers to update software, fix bugs or add new features remotely.

    OTA updates can improve vehicle performance, enhance security and extend the vehicle’s lifespan. They can also add new functionality, allowing the car to stay up-to-date with the latest technology without the need for a physical visit to a service center.

    Why DevOps is Essential for Connected Car Development

    Addressing the Complexity of Car Software Systems

    The connected car is a complex ecosystem of software systems. These vehicles have numerous systems that communicate with each other, the driver and the outside world. Managing the development of these systems can be a daunting task, and this is where DevOps strategies come in.

    DevOps aims to shorten the system development life cycle and provide continuous delivery with high software quality. This methodology is particularly suited to the complex software systems of connected cars, as it encourages a holistic view of the development process, ensuring that all components work together seamlessly.

    Moreover, DevOps helps to manage the complexity of car software systems by automating tasks, reducing errors and improving efficiency. The use of automated tools for configuration management, deployment and monitoring means less manual work, fewer mistakes, and quicker problem resolution.

    Enhancing Development Speed Without Compromising Safety

    One of the greatest challenges in connected car development is the need for speed. In this fast-paced industry, companies are under pressure to develop and deploy new features quickly to stay competitive. However, this must not come at the expense of safety, as any faults in the system can have dire consequences.

    With DevOps strategies for connected car development, it’s possible to enhance speed without compromising safety. The DevOps approach emphasizes a culture of rapid and frequent deployments, coupled with rigorous testing to ensure quality. By integrating development and operations, it enables teams to work together more effectively, leading to faster development cycles.

    Furthermore, the use of automation in DevOps allows for quicker detection and resolution of issues. Automated testing tools can run thousands of tests in a fraction of the time it would take a human, ensuring that any faults are identified and fixed before they can cause harm.

    Building, Testing and Deploying Efficiently in a Regulated Environment

    The automotive industry is heavily regulated, with strict safety and emission standards that must be met. These regulations add an extra layer of complexity to the development process, as they must be considered at every stage of design, testing, and deployment.

    DevOps strategies for connected car development are well-equipped to handle this challenge. By implementing a continuous integration/continuous deployment (CI/CD) pipeline, teams can ensure that all changes are tested and validated against regulatory requirements before they are deployed. This not only ensures compliance but also saves time and resources by preventing issues from reaching production.

    Additionally, DevOps encourages transparency and visibility across the development process, making it easier to demonstrate compliance to regulators. Through the use of version control systems, every change can be tracked and traced, providing a clear audit trail.

    DevOps Strategies for Connected Car Development

    Version Control for Connected Car Development

    In any software development project, version control is vital. It allows developers to track and manage changes to their code, making it easier to collaborate and preventing conflicts. In the context of connected car development, version control becomes even more critical due to the complex and interconnected nature of automotive software systems.

    Version control enables multiple teams to collaborate on projects without stepping on each other’s toes, using techniques like branching and merging. Furthermore, version control systems provide a history of changes, which is invaluable for debugging and understanding how the system has evolved over time. This can be particularly useful in a regulated industry like automotive, where there may be a need to provide an audit trail.

    Continuous Integration and Continuous Deployment (CI/CD)

    CI/CD is a core DevOps practice that involves integrating changes to the project regularly and automatically testing those changes to detect and fix problems early. The aim is to provide rapid feedback so that if a defect is introduced into the code base, it can be identified and corrected as soon as possible.

    For connected car development, a robust CI/CD pipeline is essential. It allows for fast-paced development without compromising quality or safety. By automating the build, testing and deployment processes, teams can ensure that new features and updates are delivered quickly and reliably.

    Automated Testing and Quality Assurance

    Given the complexity of connected car systems and the potential consequences of a malfunction, rigorous testing is of paramount importance. However, manual testing is time-consuming and prone to error. That’s where automated testing comes in.

    Automated testing is another key DevOps strategy for connected car development. By writing test scripts that can be run automatically, teams can save time, reduce the likelihood of human error and ensure comprehensive coverage of the system.

    Moreover, automated testing tools can be integrated into the CI/CD pipeline, allowing tests to be run every time changes are made. This provides immediate feedback on the impact of those changes and ensures that any issues are identified and addressed as early as possible.

    Infrastructure-as-Code (IaC) for Connected Car Environments

    The final piece of the DevOps puzzle is infrastructure-as-code (IaC). IaC is the practice of managing and provisioning computing infrastructure through machine-readable definition files rather than physical hardware configuration or interactive configuration tools.

    In the context of connected car development, IaC can be a powerful tool. It allows teams to create and manage their development, testing and production environments in a consistent and repeatable way. This not only enhances efficiency but also ensures that the environments are configured correctly and securely.

    Moreover, by using IaC, teams can automate the creation and teardown of environments, saving time and reducing the risk of human error. IaC also supports version control, enabling teams to track changes and roll back to a previous state if something goes wrong.

    Conclusion

    In conclusion, connected cars represent a significant advancement in the automotive industry, but they also bring a host of development challenges. By adopting DevOps strategies for connected car development, companies can navigate these challenges and unlock the full potential of this exciting technology. From managing complex software systems to ensuring rapid and safe development, DevOps offers a robust and flexible approach that is tailored to the needs of the connected car industry.

  • Raspberry Pi 5: Faster, Better, Stronger — Spendier

    Raspberry Pi 5: Faster, Better, Stronger — Spendier

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    In a cheeky extra post this week: Everyone’s favorite single-board ARM computer, the Raspberry Pi, has a new generation coming soon. Compared to the ’4, RPi5 has double the performance, quadruple the base RAM and far more capable I/O. (more…)

  • Splunk Adds Hub to Extend Observability to Edge Computing Platforms

    Splunk Adds Hub to Extend Observability to Edge Computing Platforms

    Splunk today launched a platform for ingesting data from sensors, industrial equipment and internet-of-things (IoT) platforms so that it can be analyzed alongside data collected from IT environments.

    Announced at the company’s .conf23 conference, the Splunk Edge Hub extends the reach of the troubleshooting platform Splunk provides that increasingly involves various types of edge computing platforms.

    Tom Casey, senior vice president for products and technology at Splunk, said Splunk Edge Hub is designed to be deployed in those edge computing environments and reduce the total amount of data that needs to be transferred back to an instance of the Spunk analytics platform running in a data center or in the cloud.

    That approach makes it possible to remotely monitor everything from environmental conditions to identifying anomalies indicative of an outage that might be prevented. The overall goal is to unify visibility across IT and operational technology (OT) environments as these systems and applications become more integrated, said Casey.

    It’s not clear how quickly that integration is occurring, given the historic cultural divide that has existed between IT and OT teams. But with the rise of digital business transformation initiatives, the data collected from edge computing platforms is increasingly being incorporated into a wide range of applications deployed at the edge, in local data centers and in the cloud.

    In general, observability tools are being used to not only improve processes but identify and block suspicious activity. Of course, OT teams have been monitoring edge computing platforms for decades, but observability platforms make it feasible to interrogate logs, metrics and traces to hopefully discover issues before there is a disruption to services. The challenge, of course, is that not everyone on either the OT or IT team necessarily knows how to shape the queries required to discover the root cause of a potential issue. The Splunk approach essentially leverages the skillset of IT professionals that have mastered its platform to address that issue; something most OT professionals are not going to be able to do using their existing tools and platforms.

    One way or another, DevOps teams will soon need to be able to remotely diagnose edge computing issues as the number of workloads deployed on these platforms expands. IT teams are not going to be able to travel to the physical location of every edge computing platform every time an issue arises. In many ways, the rise of edge computing will force a transition to observability tools that will enable IT teams to more proactively manage IT environments rather than responding after a metric that OT teams are monitoring has already exceeded its threshold.

    In the event there is still an issue, observability platforms should also reduce mean-time-to-remediation (MTTR) as visibility into IT environments, including edge computing platforms, is improved.

    It’s not likely that OT and IT teams will ever be completely unified, but as observability platforms continue to evolve, there will be much greater opportunities for collaboration.

  • A DevOps Guide to IoT Technology

    A DevOps Guide to IoT Technology

    Internet-of-things (IoT) is a term used to describe the network of physical devices—from everyday household items to sophisticated industrial tools—connected to the internet, sharing and collecting data. With the advent of cheap computer chips and the ubiquity of wireless networks, it’s possible to make anything a part of the IoT. Add to that the exponential growth in artificial intelligence and machine learning, and the IoT is a truly significant part of our lives.

    The IoT allows devices to communicate and interact over the internet, and they can be remotely monitored and controlled using technologies many of us have never heard of, such as MQTT, QUIC and AMQP. The proliferation of these devices has led to an exponential increase in the amount of data we can gather, analyze and use to make more informed decisions. We’ve already seen the transformative potential of IoT in our homes, workplaces and cities, and it’s only set to increase as more and more devices come online.

    At the heart of this technology is the concept of seamless connectivity and communication. However, the rapid development and deployment of IoT devices pose significant challenges. This is where the DevOps approach comes in, offering a solution to these challenges by improving efficiency and effectiveness in the development and operation of IoT systems.

    Importance of DevOps in IoT Development and Deployment

    In the context of IoT, DevOps is crucial. IoT devices are often deployed in the field, in homes or in other environments where physical access is difficult or impossible. As such, the ability to update and manage these devices remotely is a critical necessity. DevOps methodologies enable this remote management, providing mechanisms for continuous integration and continuous delivery (CI/CD) that make it possible to push updates to devices regularly and reliably.

    Moreover, given the pace at which IoT technology is advancing, it’s crucial to be able to adapt quickly to changes and new developments. In this regard, DevOps is invaluable. By bringing together development and operations, it allows for more agile response to changes, faster issue resolution and overall better performance.

    The Role of DevOps in IoT

    Speeding Up IoT Development Cycles

    One of the key benefits of using a DevOps approach in IoT development is the ability to speed up development cycles. With traditional development practices, the time between identifying a requirement or an issue and implementing a solution can be lengthy. In contrast, DevOps practices, such as continuous integration and continuous delivery, enable rapid prototyping, development, testing and deployment of new features and updates.

    With the DevOps approach, developers and operations teams work together from the beginning of a project, collaborating closely to ensure that the software being developed is robust, reliable and meets the needs of the business. This cooperative approach, combined with automated testing and deployment processes, significantly reduces the time to market for new features and updates.

    Ensuring High-Quality, Reliable IoT Applications

    Another significant advantage of utilizing DevOps in IoT development is the increased quality and reliability of the applications being developed. With traditional software development methods, testing and quality assurance often occur after the development process. This can lead to issues being identified late in the cycle, which can be costly and time-consuming to fix.

    On the other hand, DevOps practices emphasize continuous testing throughout the development process. This means that issues can be identified and addressed much earlier, leading to higher-quality, more reliable software. In the context of IoT, where devices may be deployed in critical applications such as health care or transportation, the importance of this cannot be overstated.

    Enabling Scalability and Flexibility in IoT Systems

    Finally, DevOps plays a crucial role in enabling the scalability and flexibility of IoT systems. As the number of IoT devices continues to grow exponentially, the ability to manage and monitor these devices efficiently is becoming increasingly important. DevOps practices, such as infrastructure as code (IaC) and the use of containers and microservices, provide the tools necessary to scale and adapt IoT systems as needed.

    With IaC, the infrastructure required to run an IoT application is defined and managed as code, making it easy to replicate and scale. Containers and microservices, on the other hand, allow applications to be broken down into smaller, independent components that can be scaled individually, providing a high level of flexibility and efficiency.

    Implementing DevOps in IoT Development

    Integrating CI/CD in IoT Application Development

    Continuous integration/continuous delivery (CI/CD) is a fundamental DevOps practice that plays a vital role in IoT development. It involves automating the processes of software delivery and infrastructure changes. With CI, developers frequently merge their code changes into a central repository where automated builds and tests are run. CD takes this a step further by automating the release of these changes to the production environment.

    For IoT, integrating CI/CD can help maintain the quality of software, increase efficiency, and speed up the process of delivering software updates or new features to devices. It also ensures that any changes made can be easily rolled back, reducing the impact of any potential issues.

    Applying IaC for IoT Infrastructure Management

    Infrastructure-as-code (IaC) is another key DevOps practice that is highly relevant to IoT. IaC is the process of managing and provisioning computing infrastructure through machine-readable definition files rather than physical hardware configuration or interactive configuration tools.

    In the context of IoT, IaC can simplify the process of setting up and managing the complex infrastructure required for IoT applications. With IaC, developers can quickly and reliably replicate infrastructure setups, making it easier to scale IoT applications as the number of connected devices grows.

    Using Containers and Microservices in IoT Application Architecture

    Containers and microservices are two more DevOps practices that can significantly benefit IoT development. A container is a standalone executable package that includes everything needed to run an application, including the code, runtime, system tools, system libraries and settings. Microservices is an architectural style that structures an application as a collection of small, loosely coupled services.

    In the context of IoT, using containers and microservices can greatly simplify the process of developing, testing and deploying IoT applications. Moreover, they can enhance the scalability and flexibility of IoT systems, making it easier to adapt to changing requirements and scale as the number of connected devices grows.

    Challenges of Implementing DevOps for IoT and How to Overcome Them

    Dealing with Device and Platform Diversity

    One of the main challenges in implementing DevOps for IoT is dealing with the diversity of devices and platforms. IoT devices can range from simple sensors to complex machines, each with its own capabilities, requirements and operating systems. Moreover, these devices may communicate using different protocols and have different connectivity requirements.

    The key to addressing this diversity is to adopt a platform-agnostic approach. This involves developing software that can run on any device, regardless of its hardware or operating system. Containerization can help achieve this by providing a consistent and reproducible environment for running applications.

    Ensuring IoT Security and Privacy

    Security and privacy are significant concerns in IoT, and they can be particularly challenging in a DevOps context. The increased frequency of updates and changes that come with the DevOps approach can potentially introduce new vulnerabilities. Moreover, the interconnected nature of IoT devices means that a security breach can have far-reaching consequences.

    To address these challenges, it’s essential to integrate security practices into the DevOps process. This is often referred to as DevSecOps. This approach involves continuously monitoring and testing for security vulnerabilities throughout the development process rather than treating security as an afterthought.

    Managing IoT Data Volume and Complexity

    The sheer volume and complexity of data generated by IoT devices can also pose a challenge in a DevOps context. Traditional data management and analysis tools may not be able to handle this volume and complexity.

    To tackle this, it’s important to utilize modern data management solutions that can handle large volumes of data. Additionally, employing advanced analytics and machine learning techniques can help make sense of complex data and extract valuable insights.

    Coordinating IoT System Updates and Maintenance

    Finally, coordinating updates and maintenance for a large number of IoT devices can be a significant challenge. This is particularly true in a DevOps context, where updates and changes are frequent.

    To address this, it’s crucial to have a robust system for managing and tracking updates. Automated deployment tools can be invaluable in this regard, allowing updates to be pushed out to devices in a controlled and coordinated manner.

    Conclusion

    In conclusion, implementing DevOps in IoT development can bring significant benefits, including faster development cycles, higher-quality, reliable applications and improved scalability and flexibility. However, it also presents unique challenges that need to be effectively addressed. By understanding and properly implementing the principles and practices of DevOps, we can harness the full potential of IoT technology and drive forward innovation.

  • Must You Collect All That?

    Must You Collect All That?

    It is interesting that we no longer consider teams that super-tool their apps as being slightly ‘off’ or, working in a field where systems were mission critical, even life-and-death. Now, we all are collecting and reporting data points in real-time across not just the SDLC, but across the entire environment. It is wild the amount of data available to the average DevOps team about what is going on with their applications. But it’s not all being used.

    No, seriously, it is unreal. Let me offer two anecdotes that explain why I know you’re not using it all.

    Most of my regular readers know that I worked at F5 for years, the first couple on the DevCentral team, doing the Dev part. Their flagship product, the BIG-IP, collected an incredible amount of super-granular data. I am not exaggerating when I say an incredible amount. They had data points from the wire up to the traffic routes through the load balancing algorithm to nodes. And it was (and likely still is) all available to users via APIs. Whatever data was needed about traffic passing through the BIG-IP or the nodes behind the BIG-IP and the connections in between users and those nodes, you could get.

    One of my projects was to offer a simplified interface to the APIs. They were complex and granular, and customers wanted fast and easy, so I was publishing code that worked as an intermediate layer.

    Note: F5’s APIs, like everyone else’s, are far more advanced today than they were then, and it is easier to do what you need to do because they took the “create a user layer” part seriously. It’s complex machinery so it still has complex APIs, but accessibility is way up from those early days.

    Anyway, BIG-IP collected so much data that I could not name all of the fields, let alone when each one was pertinent. I gave up trying after a while, because if a board-level engineer needed a data point, the idea at the time was “Why not expose it?” so ‘number of connections to a given node’ under this algorithm was hidden among things like ‘microseconds to connect to DNS on that connection’ (I’m making those up; it’s been years). Some were terribly useful to end users and others were not. All were accessed in the same way, and learning by experimenting was the only real way to tell the difference.

    Users did not need all of that data. Users did not need the complexity and obfuscation created by supporting all of that unnecessary data.

    My other example is a massive system of interconnected IoT devices that were centrally connected/controlled with a private, specialized network to manage them (actually, there were several different networks on several different mediums, and that matters where data points are concerned).

    When setting this system up, we estimated about half a million IoT devices with capabilities that ranged from passive monitoring to active controllers of physical devices. Four different communications mechanisms, a couple dozen different devices, every major (and a couple minor) databases and command-and-control software on every major OS, managing from a handful to hundreds of thousands. And all of these were super-tooled because the IoT network was mission-critical. Our databases and the infrastructure to support/protect them were a major part of our data center budget because of the amount of data.

    The amount of data was a flood. The management software was developed by different companies with different transports and endpoints in mind, meaning each had a different dataset. Once all of the networks were running satisfactorily, our next immediate project was to create a unified dataset for reporting and troubleshooting. I was in charge of IT, but the top-level person–because each of those half a million devices was a physical install–was a physical engineer. We fought non-stop about what was important. I was trying to map and convert datapoints between systems; he was trying to maximize data available for reporting up. He often said, “Well no, we don’t need that today, but we might.” This put us in a zone where “number of devices on this system” was on par with “attenuation of the communications line to the nodes on this physical subnet.” Totally unworkable, and our compromise was to put together a small IT team to put a layer over all that raw data and offer an intelligent interface side-by-side with data dumps.

    We are there again today. I hear many of my friends in the enterprise space saying, “Because of ML, we might need that data and storage is cheap.” I disagree. Storage is an overhead that is wasted if you are not actively using the data. We are past the “all data is good data” craze; it’s time to adapt. Useful data that expands insights is all you need tooling for. Report data that will help solve problems, and if the team finds tomorrow that it needs a data point that is not currently collected, it can be added in.

    There is room for debate here–the rate of change for ML systems (please stop calling them AI; they’re not, yet, and we’ll need that word when they are cognizant) means that there is a reasonable argument for “all of the data.” And I’m not proposing limiting data collection. Honestly, application security specifically and enterprise IT security overall has benefitted from datapoint sharing between formerly separate market spaces. If the SCA scan shows X framework is used, then the IAST can narrow down tests to those that look at X and not its competitors, for example. But honestly, don’t drown yourself in a data lake when all you need is a fish tank where you can see all the guppies.

    And keep rocking it. Even if you entered IT two years ago, you are now a seasoned vet and things have changed–beyond all of our expectations, I’d say. Yet you’re still plugging away, keeping the enterprise running. Don’t stop—just try to keep the data hoarding to a reasonable level and make sure the data you have is being used, whether as useful feeds to an ML tool or as consumable information for other people/systems.