devopsy.kinsta.cloud

Tag: security

  • Unused Code: Proving a Negative

    Unused Code: Proving a Negative

    We have a ton of code that is no longer being exercised. We really don’t talk about it much, but it’s out there. From feature flags shutting off code to API versioning making an interface obsolete, it’s just hanging around out there. Like an albatross. Waiting for someone to misuse it. Or, to address the more pedestrian concerns, confusing people new to the codebase. But that code is not contributing a thing.

    Easy example. You are on version 3.2 of your API. Version 2.1 is not called by anything, but the code is still hanging out there. It is not that 2.1 cannot be called, but that it is not currently called. The problem with that is that new devs have to work through what’s current, and since 2.1 and 3.2 probably share 95% of their code bases, what portion of that code base is unused? And the API is sitting there, waiting for someone to penetrate your network and find malicious uses for it.

    Proving a Negative

    But here is the problem: Identifying that a bit of code–be it an API, a method, just a few lines in an if statement, whatever–is never exercised is terribly difficult and, in highly complex environments, borders on the impossible (it is never impossible, assuming IT owns all of the code that could exercise that bit). The old adage about proving a negative absolutely applies. We can–and testing tools in both the functional and security spaces do–generate a list of code that is exercised, or identify code that isn’t being exercised. But not a list of code that is never exercised. That functionality in test tools is super useful, because it helps prioritize issues that are being used a lot and leaves the ones not seeing a ton of traffic for later resolution.

    But I’ll say that in most organizations, that’s no longer far enough. The cruft in your code causes problems, and you need to clean it up. If you’ve worked in IT for any length of time, you’ve been through the “Oh, that bug is because the developer called something no longer used” scenario at least once. And it is a straight-up waste of time.

    So here is my proposal. Test tools generate a list of what isn’t called from systems under test. Pick one–security testing or functional testing–and require it for any bit of code, no matter how small, to go into production. That means that the system knows about all source code and thus knows definitively if a bit of code has been exercised.

    Clean Up as You Go

    Is that a tall order? Yes. Testing isn’t free. But given a scenario where you reduce technical debt by eliminating excess code, I think it will pay for itself. And that is not counting the security posture improvement that reducing the code base implies.

    You’re running the systems that present your organization to the world, so you may as well do some due diligence on yourselves. Keep making cool new stuff and upgrading the old, but clean up as you go with some form of monitoring, this or a more procedural one, so the quality is high and the excess is low.

  • Steady On. We Still Have a Job to Do

    Steady On. We Still Have a Job to Do

    It is all the rage to talk about how significantly large language models (LLMs) and generative AI will change every niche and corner of IT. And change IT it will. But the hype cycle is at its highest right now, and I’d caution against getting too worked up about it. We’re in for a few interesting years, and the spaces I cover (DevOps and security—with a more application-centric perspective but with ops experience) are definitely going to benefit from the various AI bits being worked in. All testing—including AppSec, but all testing—will benefit both in test generation and results analysis/filtering. In fact, I’ll go ahead and say that the question in testing will no longer be, “How much time to we have to devote to it?” and will rapidly become “How many systems’ resources do we have to dedicate to it?” It will still be a long process—AI isn’t speeding up the cumulative time it takes to run thorough testing (be it functional or security). It will make that the bottleneck though, because test generation will at least be LLM-assisted, and will eventually become completely generative AI-created while at the same time, results will get more and more filtering and analysis. A minor result here plus a minor result there will automatically be detected as an extended vulnerability. There are a ton of other possibilities, but these two seem pretty obvious when I look across the testing technology space. More testing and better results analysis are on the horizon—without requiring more staff. Of course, you can do so much more, which would require the organization to add staff to follow up on results, but that’s nothing new; testing finds issues that are interruptive and take man-hours to fix—we all know of bits in the application portfolio that we should probably fix but that aren’t high enough priority to do so, for example. But fixing is different than finding. I have seen demos in security scanners that offer auto-fix options for developers. Run the test during CI, create a ticket and let the developer see suggested code rework right in the IDE. It is early days for this, but if you look, you can find vendors doing it. And that’s really rather cool.

    But we’ve all heard the “this changes everything” storyline over and over. 4GLs, XML, Java and JavaScript were all going to eliminate developers. In fact, they all created more developers. AI has a better chance of actually removing parts of our workload than any of these technologies did, but we’re still in “watch and wait” mode. It seems obvious that grunt coding will fall to LLMs … And aside from keeping staff trained to manage source, that is a good thing. We use open source, modules and libraries to do much the same thing today. I see this as equivalent to entry-level writing jobs. We’ll have to transform from the current model, but the current entry level development model wasn’t ideal anyway, so lets make a better one. Testing capabilities will be greatly enhanced, but testing of all kinds was so far behind new code generation at most orgs that the impact will be all positive as testing catches up. AI-based test generation today can generate thousands of permutations for a given piece of code, and analyze the results to just have analysts look at the truly disconcerting or confusing test results. And the question will become, “How many servers do we want to dedicate to run this?” Which is different, but just means we have more tests running to make a more solid and/or secure product and environment.

    So, in summary, is change coming? Yes. And that is not significantly different than the last couple decades where change has become IT’s constant companion. For most of you, the way you do your job today is not the same as the way you did it even two years ago. ML/GAI/LLMs are just another iteration. The apps still have to be built, deployed and maintained. So until HAL comes along, keep kicking it. You’re keeping it together, and who knows what the future brings.

  • What Are You Securing?

    What Are You Securing?

    Every once in a while, I think it is important to look at the available technologies and tooling and ask, “What are we trying to do?” In the case of information security, we need to ask, “What, exactly, are we trying to protect?”

    The answers to that question are seriously limited. Availability, reputation, data. Since reputation is a business mapping of the other two, that leaves us with availability and data. Of the two, data is more important. Our systems being down absolutely costs us money, but leaking customers’ private data or our corporate secrets, like source code or technology designs, is worse in the long run.

    The problem with these observations is that if we are here to protect the data, and data is the more important piece of security, then why do we have far better application security than data leak prevention? Almost all of you have policies and processes to run an application through the wringer and get an “as secure as possible” result at the end. But data? The purpose of applications is to expose it. And in the process, we risk exposing too much of it or allowing tons of it out of a valid endpoint. I remember in the early days of login-based websites, there was a spectacular leak that let an attacker get hundreds of valid user credentials per page submission (there were a surprising number of these for a couple of years there; this one was at a college). We are facing much the same thing via APIs these days or even via pages meant to make access easier for users.

    Locking all of this behind logins is a useful step, but to use one of the many phrases we’ve stolen from military science, it is only a part of ‘defense in depth.’

    Once it is agreed that data is one of the two key things we need to protect, then the focus should be on how to do so. There are some amazing DLP products, both standalone and integrated into other technologies, that are worth looking into. There are also some great advances in automated response under the guise of AI/ML that should be watched closely because they are changing rapidly. Limiting the maximum number of responses allowed from a query is almost mandatory for application stability and for data leak prevention. And it isn’t that hard to implement.

    I assure you that ne’er-do-wells—particularly state-sponsored and corporate espionage hackers, but also amateurs—are using AI to make attacks more successful and less detectable. The least you can do is respond in kind, with some of the tools that can now do things like notify you about low-and-slow data leak attacks.

    Keep focused on what the goals are. Reiterate them often, and keep rocking it. That’s a lot of data splashing around the organization that your team is responsible for; make sure it’s as safe as possible.

  • Low-Hanging Fruit, 2023 Edition: Part Two

    Low-Hanging Fruit, 2023 Edition: Part Two

    Last time, we discussed setting up a comparative inventory system for your growing API footprint. The idea is that as security catches up to new technology deployments, enterprises will have to step up their game and implement those new technologies. API security is currently the biggest need because of exposure to the world, but not far behind it is container security, which we’ll discuss here.

    The point of container security is to make certain that your images are as secure as possible. Container security looks at source container definitions, added libraries and configuration options inside the container definition files ( or for those of you who prefer them, Docker files).

    A Tempting Target

    We are spinning up a bewildering number of containers these days, and having source images–particularly if they come from the outside–is a tempting target to attackers. Imagine if an attacker can hit the base container definition that all of your “golden images” come from and just insert a call to download a library. Soon, your entire application infrastructure could be a tool for that attacker to do as they wish. And if they’re cautious, they might get away with it for months (as many have).

    The Definition of Containers is Security

    So there is absolutely an imperative to ensure that the definition of containers is security–all of your containers. And there is an imperative to know what software is installed via container definition and what software is actually running on container instances. Much like our discussion about static versus dynamic API scanning, bad actors can modify a running container, so drift detection is half the problem, even while scanning registries and definition files has more appeal from the blast radius perspective.

    There are a lot of tools and vendors that range from simply scanning the files and generating an SBOM to complete container security by scanning the images, using the SBOM to perform software composition analysis, using data feeds to determine which software in the SCA is a threat, and how urgent or broad that threat is and monitoring deployed containers for drift. The more of these an organization implements, the more configuration and maintenance effort, but the payoff is security at a more fundamental level. The whole appeal of a container architecture is recoverability–if things go terribly wrong, we can redeploy easily. But if the source files hold the vulnerabilities, redeploying is temporary at best and likely would warn attackers that you’re onto them.

    Be Proactive

    So a more proactive approach is required. Obtain and install tools that will help you protect your container infrastructure, be they sold as container security, cloud workload protection, SCA or something else. The key is locking down this avenue of attack at the beginning of the software supply chain, not where you get the tools.

    And again, keep rocking it. As I said in the other post on low-hanging fruit, we need this because you all are serving the organization’s needs with the best solution for the job. Now that security technologists understand your new problem domain, let’s get tools into place to continue the protection of all that shiny new stuff.

  • Low-Hanging Fruit, 2023 Edition: Part One

    Low-Hanging Fruit, 2023 Edition: Part One

    We’ve been cranking it out, haven’t we? Generating cool new toys for the business, creating new infrastructure to host those new toys and trying our best to secure all this new stuff while maintaining both our sanity and our cadence. And we’re doing it. Kind of.

    One thing we’re naturally lagging behind on is security. While we don’t purposely shortchange security in our DevOps processes these days, it always takes a bit to figure out the best way to secure new technologies after they’re implemented. And we’re implementing a ton of new technologies. So our security staff (and increasingly, DevOps/SRE teams) is rushing to play catch-up. One area where this is screamingly apparent is APIs. We’re spewing them across the organization, often without cataloging or responsibility. This makes sense, as in many cases, the API is just an evolution of the function call. “I need a service, it is a good generic service, let me throw it up and we can use it from wherever the client ends up deployed,” type of thing. The problem is that leaves a ton of attack surface not truly protected. Because without cataloging or responsibility, there is no way for security to know about many of these APIs.

    API Scanning Technology

    That brings us to today’s topic. If you haven’t done so yet, it is past time to look at API scanning technology. It comes standalone and bundled with pretty much any API security product on the market and is increasingly bundled with other development security or even development tools. And the whole point is to get a list of APIs. Oh, most commercial implementations do something with that list–include it in static testing, drive it into WAFs, whatever—but just having the ability to inventory what is running on your network is reason enough to stop doing less important stuff and figure out what your exposure is. And the tools make it easy. They come in two flavors, and we should consider both.

    Static API Scanning

    The first type digs through source code and API specs to come up with the list of what is supposed to be running on the network. The big weakness of this method is that it only knows what it is told. This is static API scanning.

    Dynamic Scanning

    The second watches network traffic for API calls and then documents them, possibly even generating API specs for you. The big weakness of this method is that it only discovers what is being accessed on the network while it is watching traffic. Currently, unaccessed APIs are not caught, nor are ones on unreachable networks. Unsurprisingly, this is dynamic scanning.

    As usual, the two together are more powerful than apart. By implementing both technologies, an organization can work toward a hardcore comparison of what should be running on the network versus what is running on the network. This goes beyond detecting unreported APIs, it also means that zombie APIs or runtime drift can be discovered by comparing what is there to what should be there.

    If you’re not doing this, consider adding it to the toolbox. Setup is not onerous, and the reports are lists of APIs and parameters that can be reviewed at leisure. Some commercial products that offer these two functionalities take it quite a way beyond list generation, and if comparisons are made, work hard to understand just how much each toolset can do.

    And keep rocking it. We need more tools to lock down the new perimeter precisely because you and your team are tackling new technologies and platforms to better serve the business. We are increasingly seeing security tooling catch up with the broad selection now available, so go implement them.

  • The Security Pipeline

    The Security Pipeline

    Over the last few years, the ability to secure our applications has grown, and deep integration into the DevOps toolchain has, too. There are more tools doing more security checks protecting more of the infrastructure and source than there have ever been. The key is putting them to use intelligently.

    We now have the ability to secure the image an app is based upon, the images that are derived from that original—complete with supporting software. We can get feeds to tell us which bits of our included open source components are insecure and how to protect them; we can scan the source, the assembled application, the APIs and the network. We can put runtime protections in place in front of web apps and in front of APIs. We can even track the long-tail attacks like slow data extraction that are becoming more common as security tools master more obvious attacks.

    That is a great set of security capabilities. And automation puts every bit of it within reach. Your job is to pick the bits you need to increase the security posture of your environment and get them worked into your specific toolchain. This is not all that difficult, but nothing is free, so you will need at least some implementation time and some staff time to work through results. After all, no matter how automated, current solutions do end with a list of changes that need to be made to increase security. Some tools will put a runtime fix in place immediately, but honestly, that is a patch—it only stays in place until you can fix the underlying issue.

    Integrating all of the great security tooling with each other is another aspect that will require some time. Some vendors cover a massive amount of the security testing needed across markets and technologies. These companies show us that even if an organization would rather use best-of-breed, there is a lot of benefit in increasing integrations. For an easy example, consider this: If the software bill of materials (SBOM) generated during the build steps shows that a given framework is included in an application, then active testing via dynamic or interactive application security testing can include extra tests specific to that library.

    Integrations will take effort, but once they’re in place they will enhance application security. There is no perfectly secure application, but the tooling is available to make a given environment less of a target because it is too much work to attack. And that is what you should be shooting for.

    I’ve said before that making security part and parcel of your DevOps practices is mandatory at this point, but that was in reference to the great DevSecOps toolsets currently out there. I’m suggesting now that we include it all–from design to GitOps runtime drift management. Because we can, and the result is reduced risk to the organization.

    And keep rocking it. Honestly, you’re maintaining something worth protecting–that alone means you are successful, even if you aren’t aware of every issue that crops up. Issues are why you’re there, and the systems keep running because you are.

  • DigiCert Allies With ReversingLabs to Secure Software Supply Chains

    DigiCert Allies With ReversingLabs to Secure Software Supply Chains

    DigiCert today announced it has allied with ReversingLabs to integrate binary analysis and threat detection capabilities with a code signing service it provides.

    Deepika Chauhan, chief product officer for DigiCert, said the addition of these capabilities to the DigiCert Software Trust Manager service would make it simpler for organizations to operationalize a methodology for securing their software supply chains. The capabilities are designed to be embedded within continuous integration/continuous delivery (CI/CD) pipelines.

    In addition to being a certificate authority, DigiCert’s Software Trust Manager service automates code signing workflows. ReversingLabs, meanwhile, just added the ability to detect secrets exposed in application binaries to its Software Supply Chain Security (SSCS) platform. Those capabilities can identify secrets inadvertently left in applications as plain text, can identify secrets that can be discovered because of weak cryptography and also includes scripts that scan for malware and can generate software bills of materials (SBOMs).

    Chauhan said when combined with DigiCert Software Trust Manager, it becomes possible for the two companies to provide a comprehensive approach to detecting malware, software tampering, inclusion of secrets and certificate misconfigurations in any type of software.

    A recent ReversingLabs survey found 87% of respondents detected significant risks in their software supply chain in the last year, with software containing vulnerabilities (82%) followed by secrets leaked through source code (55%), malicious code (52%) and suspicious code (46%) posing the most serious risk to the business.

    A full 88% said software supply chain security is an enterprise-wide risk, but only 60% said their software supply chain defenses were up to the task. Nearly two-thirds (65%) of respondents, however, acknowledged their organization’s software supply chain security program wasn’t as mature as it should be. As a result, 80% are currently focused on improving security for the software supply chain, with 96% noting a more comprehensive approach to software supply chain security that detects more than vulnerabilities is needed.

    It’s not entirely clear just yet whether development teams or cybersecurity teams will be taking the lead in terms of driving the adoption of DevSecOps best practices to secure software supply chains. It is clear that securing software supply chains in the wake of a series of high-profile breaches is a much higher priority.

    Erik Thoen, director of product management for ReversingLabs, said as a result, it’s only a matter of time before development teams and cybersecurity professionals are forced to work more closely together than they have in the past.

    In the longer term, a wave of legislation in the U.S. and Europe will soon require organizations to embrace DevOps best practices to address increased potential liability that might stem from an application breach. A proposed Cyber Resilience Act currently being negotiated by the member states of the European Union seeks to require organizations that sell internet-connected hardware platforms to ensure both their devices and their software comply with cybersecurity best practices. In the U.S., meanwhile, a National Cybersecurity Strategy proposal put forward by the Biden administration seeks to hold organizations that collect data or build software more accountable for breaches.

    While both proposals are a long way from becoming law, it’s clear governments around the world are concluding the only way to ensure better cybersecurity is to require it.

  • DevOps Security: Your Complete Checklist

    DevOps Security: Your Complete Checklist

    In a fast-paced environment like DevOps, your security strategy needs to be even more agile. While moving fast is great for delivering application features and functionality, speed shouldn’t come at the cost of security.

    DevOps security incorporates security strategies directly into your DevOps pipeline, creating a culture of pre-secured software deployments that work seamlessly for the end user. However, this requires a complete shift in your operational policies and security mindset. You need to start viewing security as an integral part of your DevOps lifecycle instead of a separate department.

    Here’s a checklist of DevOps security strategies for your development teams to deliver secure and reliable applications faster:

    Your Complete DevOps Security Checklist

    1. Analyze your development process for security hazards.
    The first step in implementing a DevOps security strategy is finding and fixing loopholes in your existing development process.

    For instance, continuous delivery and development are key in a DevOps environment. However, if each change doesn’t go through a code review, it has the potential to break your application.

    Deploying changes without passing them through a security checkpoint is like driving at high speed on a road with many curves. Each curve (change) is a potential hazard, and the danger increases when the driver does not know a car is coming from the other side (bug).

    Therefore, you must turn the spotlight on your development process and find those curves.

    2. Create a common goal between development and security teams.
    Traditional roles of developers and security teams might look like they are at crossroads — Developers write code, and security teams find fault with their code. However, in a collaborative environment such as DevOps, security and development must work hand-in-hand to deliver a secure and seamless experience to end users.

    Your organization should have a common goal of integrating security directly into the development process. Therefore, encourage open and regular communication between development and security teams. Further, establish joint milestones and review them to ensure there are no silos between both teams.

    Once these teams get used to this collaborative approach, your deployments will become faster because fewer things will break.

    3. Integrate security procedures in the development phase.
    Automation is the lifeblood of DevOps security. Integrating scanning tools into development workflows is critical to your entire process.

    For example, automated tools can scan the code, network and infrastructure to identify security vulnerabilities and provide recommendations for remediation. With this, development teams can fix a few security concerns themselves before passing them on to the security team for a final check.

    4. Add observability to your pipeline.
    One of the biggest challenges of the distributed nature of DevOps operations is visibility. How do you efficiently monitor a large number of operation nodes spread across several different locations?

    Observability helps you do this by using logs, metrics and traces—i.e., the outputs of your application to determine internal application health. With observability tools, you can see much deeper into the operations of your systems and identify the exact details of any security concerns. You can also easily identify potential bugs and fix them before they cause problems.

    5. Make changes in small, incremental units.
    It is infinitely easier to test code in small units and deploy them as minor but incremental changes than handling large, unwieldy blocks of monolithic code. Deploying gradual changes means you can resolve vulnerabilities faster.

    6. Scan and secure your entire DevOps pipeline.
    A large part of the security hazards in your DevOps process comes from unsecured containers and operations nodes. According to a study from ThreatStack, 94% of organizations say containers represent a security risk.

    However, the risk is not limited to containers.

    Other services like APIs, third-party tools like Docker and imported code pose a potential security risk to your processes. This makes it necessary to scan all your operations nodes periodically for vulnerabilities. And once you’ve cleared them, remember to lock the door behind you by requiring strict credentials for access to your administrative dashboards.

    Ticking Items Off Your DevOps Security Checklist

    If you implement most of the security guidelines on this checklist, your organization will have a solid security foundation. However, DevOps security is not a one-time process.

    You will need to continuously integrate security protocols into each part of your DevOps life cycle. Furthermore, you will need to re-train your employees and invest in new technology that makes your security pipeline smoother, such as observability.

    However, since this entire process makes your applications more secure and compliant while shipping them to market faster, investing in DevOps security is well worth it!

  • DevOps Onramp: Security on the Rise

    DevOps Onramp: Security on the Rise

    I don’t normally advertise stuff in my blog, because that’s not what it is here for. It is here to offer you the thoughts and ponderings of someone like you, and advice you didn’t ask for. But today we’ll make an exception since I also blathered for Techstrong Group recently, and soon you’ll be able to hear it.

    If you are not familiar with it, DevOps Onramp is this week, and you can go and register to hear my session and the sessions of other great voices that are involved in DevOps. My session is about DevOps and Security, but there are a wide variety of sessions for you to choose from.

    I am trying to make the time to be involved in the conversation when my session airs, and it looks like my schedule will allow it, so if you are interested in DevOps security, I’ll likely be around to discuss it with you.

    I can’t speak for all of the sessions, but mine starts with the basics for newcomers and then covers more advanced topics as it goes on. The truth is that I was supposed to do two sessions–intro and advanced, but I started talking too fast, and the next thing you know, I crammed it all into one session, saving you time for some of the other great content that is available at the show.

    So (virtually) drop by, watch my session, see what else you’re interested in, and feel free to drop any of us presenters a line telling us what you think, good and bad. I’m sure the show organizers would be happy to receive feedback too, but I’m all about presenters.

    And keep rocking it. We’ll be back next week with our regular content (and no ads).

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    LinkedIn

    SiriusXM
    Associate Director for Software Delivery
    Deerfield Beach, Florida
    $157,000 or $250,000

    SimplyHired

    Nagarro
    Remote
    Senior Staff Engineer – DevOps
    $155,000 to $196,000

    Dice.com

    Engage Partners
    Davie, Florida
    DevOps Architect
    $130,000 to $160,000

    Indeed.com

    ServiceNow
    Orlando, Florida
    Staff DevSecOps Engineer
    $114,000 to $144,000

    Monster.com

    Swift Group
    Fort Belvoir, Virginia
    DevOps Engineer
    $80,000 to $180,000