devopsy.kinsta.cloud

Author: Don Macvittie

  • DevSecOps Implementation: Patch Management

    DevSecOps Implementation: Patch Management

    Patch management has been a thorn in IT’s side for decades. While it is undeniably necessary for security and stability purposes, the staggering array of patches that are available make managing them effectively nearly a full-time job – or a job better left to users. Few IT professionals want to be the “patch person,” and yet, nearly all IT professionals want their systems protected by the latest patches. No one wants to find out an intrusion was easily preventable via patching. It goes without saying that the SolarWinds attack has left quite a few IT departments scrambling for an alternate solution.

    The world of patch management has gone through several evolutions that we won’t cover in much detail here. Instead, we’ll focus on what’s available, what you should look for and where the market is headed. Suffice it to say that patch management as a practice has grown from an individual applying patches to servers and server apps while employees were encouraged to patch their equipment, to complex systems that manage the patching themselves, with IT oversight. Currently, patch management is often bundled with endpoint management, simply because many of the patches are going to end points (user machines and devices).

    Out of this growth have come several market differentiators that you should be aware of when looking into a way to manage the avalanche of updates the average organization is seeing.

    • Centralized or de-centralized? Increasingly, patch management is run on servers/cloud and pushes changes out to those machines/devices that need them, or agents on each machine manage updating themselves. Which is best for a given organization really varies. If a lot of machines are running at the edge of their capability, an agent running updates in the background might be too much to ask. In that scenario, a server-managed install environment is probably better. Conversely, if a centralized location to run updates from is considered an architectural weakness (because it requires access to the update pushing engine), agents might be the better plan.
    • OS support. We’re way past the days when single-OS solutions were considered useful. List the operating systems used in your environment, and choose a product accordingly. Most products on the market support Windows/Mac/Linux, but there are variances. First off, make certain your chosen product supports all three. Then ask, “Which flavors of Linux?” Make certain the flavors running in-house are supported by the software.
    • App support. There are a lot of applications out there. Which are supported by the tool? How much leeway does the customer have for adding applications by pointing the tool at update locations? Does it support FOSS updates? There are a lot of those too, and which ones matter. Make certain the tools the organization uses are supported or can be added.
    • Many tools will scan machines for update issues. Understand what your chosen tools scan for, and how the scans impact performance on the target machines.
    • Updates are not 100% perfect. There will inevitably be problems. Make certain the tool can roll back changes. Updating 50 core infrastructure machines, only to find, hours or days later, that the update breaks something demands roll-back capability.
    • Inter-relationships. Updating a piece of software can require that another piece of software or a driver be updated, which can demand that UEFI/BIOS be updated. Or the CMOS on a RAID controller. Understand how cascading updates are handled by the tool, and if the cascading updates can be packaged, so that a failure automatically rolls back the updates applied thus far.
    • Approvals. Speaking of driver and BIOS updates, what tools are available to allow a human to approve changes? No one wants willy-nilly updates to the drivers on core systems – or even to drivers on user systems.
    • Security of the tool itself. We’re in perilous times where IT security is concerned, and frankly, an update command-and-control server is an attacker’s dream machine – take it over, put your back-door update into the mix, and tell it to update every machine it manages. It cannot be understated: an update management tool must be as protected as possible – both by the customer locking down the installation and by the developer making sure the code is solid.

    Once the system is chosen and worked into your DevSecOps environment, updates will be more responsive, which leaves one more thing to consider – does it feed into your agile tools? If the update management tool notifies a project management tool like JIRA when it updates a server, then if code on the server breaks, the developers aren’t in the dark about changes, and will have breadcrumbs to help them track down the source of the problem.

    Imagine if manual updates became a rarity instead of the norm. There is a cost associated with this process, but most of the products on the market can be had for a reasonable price – if you waste a couple hours a month (or for some products, a year) on updating, they’re paying for themselves. Of course, your time is a sunk cost, and the cost of these is an operational cost, so there is an evaluation your organization will have to undertake. Given how busy everyone in DevOps and DevSecOps is, it is likely worth the investment for all but over-staffed organizations. What? There are a few!

    Find what suits your organization, and put it to use. Faster updates, less time investment. The idea is sound, you just need to figure out what suits your needs. And keep rocking it. You are managing updates already, this is just a way to make your life easier. And we could all use a bit of easier in a busy, complex IT environment.

  • DevSecOps Implementation: It’s About the Data

    DevSecOps Implementation: It’s About the Data

    It seems we are constantly barraged with, “So-and-so announced that their systems had been breached and 80,000 user records were stolen,” notices. In fact, https://haveibeenpwned.com/ lists ten billion hacked accounts.

    That is no surprise. Databases exist to pool related data together. Even NoSQL type databases, which some saw as an opportunity for wildly disparate data to be gathered and analyzed, almost exclusively contain related data sets. We generally don’t want or need to correlate, say, the number of pirates in the world with annual global temperatures, for example.

    One would think, with decades of experience under our belts, we would have a good solution for protecting that data. If it was a simple problem, we would. It is not.

    Databases are the primary target of most attackers. After all, getting access to your user database gives them account names through which they can try and gain elevated privileges. Getting a large amount of user/credit card information gives them a revenue stream by selling them off (though it is now so common that prices on the black market have become depressed, and this isn’t really profitable, given the risk versus the reward, anymore).

    And databases must, by definition, have broad access. A database does not exist in a vacuum; it exists to serve an app. For server-based databases, it almost always exists to serve multiple apps. It must be on the corporate network (even if that network is a virtual private cloud), and it must … offer up data. It has users that have greater privileges than your average customer, and is notoriously difficult to protect with strong protections like encryption.

    Encryption poses problems at multiple levels. It is still a bit slow, for starters. This is the nature of the beast – if it takes X microseconds to respond to a request, encrypted data will take at a minimum X+decrypt time (regardless of where decryption occurs). If encryption is done by column, index fields cannot be encrypted (because “nZ19?SDF” is unlikely to be in the same sort order as “ford explorer”, for example) Which means some data is immediately excepted from encryption. If encryption is handled by the database system, then the time overhead exists, and the database system must send the resulting data over your network … unencrypted. I was involved in a couple of projects that required encrypted databases – the one I can vaguely talk about was putting information about your corporate network onto Android and Apple devices. Needless to say, since phones and tablets are misplaced quite often, and most have weaker security than your average laptop, we needed information about critical infrastructure to be encrypted. It was … painful. We only managed it successfully because corporate networks are finite resources, so the amount of data we needed to store was limited, and we could prove performance wasn’t going to be the issue.

    So what can you do? Continue to follow attempts to adequately encrypt corporate databases. Eventually, someone will crack this problem in a way that is performant, can be debugged and reliably protects the data.

    Until then, protect your databases. Use the tools we’ve talked about in this series to monitor and protect the entire network, knowing that the database is a high-profile target if someone gets in. And use the other security tools available. The major database vendors offer DBMS security tools to evaluate your current status. Some will flag weak passwords, others will monitor for injection attacks (as will development tools discussed in the first few installments of this series). Lock down interfaces to databases – by user, by IP, whatever makes the most secure, but still usable, environment for you. Set up monitors for behavior outside the bounds of normal behavior – a query of 250K records should raise a flag, for example, since the vast majority of apps will only do a subset that can be paginated. Sure, it will probably be a false positive caused by someone hand querying, but the one time it isn’t, you’ll be glad the monitor was there.

    And protect your database backups – be they actual backups or a dump to test. Fewer organizations are using real, unfiltered data for testing, but it crops up in most orgs at one point or another, as testers are trying to look at issues that fake/masked data can’t replicate. For actual backups, those must be encrypted. To lose your customers to a hacker because you had an unencrypted database dump lying around, and the attacker didn’t even have to attack the DB, is downright shameful, and happens too often. Along these same lines, and this should go without saying, but it’s happened enough that I feel compelled,  do not copy large chunks of your database out to an unsecured cloud storage bucket. Some vendors have addressed this issue by not allowing public unsecured buckets, but you can still do this, depending on the cloud you are using. So, don’t.

    The database security market will be best served by being rolled into SIEM and/or XDR. And while I’m in full-on pundit mode, for this statement, I’ll predict that is where it is headed. Database security is as much about overall security as it is the database, so overarching systems are the best place to detect and defend from. So watch for moves in that direction. A holistic view will serve you better in the long run.

    Meanwhile? Keep rocking it. Treat your data like gold, because to hackers, it really is. And, keep running the world’s business. Celebrate your successes as much – or more – than you learn from your failures.

  • DevSecOps Implementation: EDR/XDR

    DevSecOps Implementation: EDR/XDR

    We mentioned host intrusion detection and network intrusion detection in an earlier blog, and mentioned firewalls a couple of times in passing. Let’s delve a bit into the history to understand how these tools’ functionality has evolved over time.

    On the host side, we had virus detection, in one form or another, from early in the evolution of the Internet. The idea being that, if an antivirus program could scan your files, it could help protect you from malicious applications. Applied to an organization, if antivirus applications could scan all users’ desktops, it could help protect the organization. And it did kind of work. Most organizations were protected, at one point or another, by their antivirus software. Then, they realized the need to see more than what simply scanning files could show. A file that looked innocent on disk could assemble malicious code while executing, so runtime environments also needed to be watched. Generally speaking, the same antivirus companies led the charge into runtime protection. But there is more to protecting systems than simply looking at running applications. Scripts, even database entries, could be made malicious, and activity over time could indicate a bigger problem than a virus – an intrusion. This problem was the reason host intrusion detection was born. The idea that all of your system(s) needed protection all of the time created a mostly agent-based environment that monitored several aspects of the running system to watch for suspicious activity.

    On the network side, firewalls were implemented early in the growth of the Internet to allow limited access to corporate resources over the network. If no one ever needed to telnet to the machines on subnet X, the telnet port for subnet X could simply be blocked at the firewall, and no sleep need be lost over attempts to connect. Almost immediately after their inception, firewalls started logging attempts to access ports, and this information was made available to network administrators to review. This had an interesting impact on the management of hosts, because now, the firewall could inform network administrators if one of the internal hosts was constantly attempting to do something it shouldn’t be – possibly indicating an intrusion or a bad employee (more often than not, it turned out that undertrained, curious employees were just poking around). But firewalls – whether using whitelist or blacklist methodology – were fragile and/or interruptive. They also simply plugged holes in the perimeter without being proactive. That is when some bright people realized that, if you monitored streams running through the firewall, you could detect attacks and intrusions and offer admins proactive information about potential threats. That was the genesis of network intrusion detection (at least, as with all of these, as much of it as I can cram into a blog).

    But as time has moved on, there was a need for more. Network intrusion prevention was tried, but most organizations don’t want automated systems cutting off connections for fear of impacting real users. While it is still around, it hasn’t gained all that much traction. The next step was to “connect the dots,” so to speak. If the HIDS on host A was detecting a suspicious pattern of behavior, it only made sense to have the HIDS on the other hosts in the network take a look for similar behavior. Command and control of cross-HIDS installs is at the heart of endpoint detection and response (EDR), which gathers information from multiple HIDS agents and correlates it to get a more holistic view and alert on activity that a single host might not have noticed (a super simplified example: if the same user account tries to get super-user privileges with a single attempt on a dozen machines. HIDS might not raise a high-priority alert for a single attempt, but EDR sees twelve attempts on twelve different machines, not just the one on a given box, and can raise a high-priority alert).

    Extended detection and response (XDR) extends this information amalgamation and analysis to the network side, including attempts to gain access to the network (in our simple example above, imagine if the same user account tried to ssh into machines it doesn’t have network access to).

    As of this writing, EDR and XDR are using machine learning (ML) to perform faster processing of all the data they are seeing, and to limit false positives while raising critical alerts up to human operators. At this point in the ongoing feud between attackers and security personnel, I’ll go so far as to call it state-of-the-art. For now. Other attack vectors and XDR-defeating attacks will no doubt come along, and will have to be answered, but for now, if you have a publicly exposed network, it is the best you can get. Vendors have moved both EDR and XDR analysis to the cloud, ostensibly to improve analysis. It does reduce the amount of maintenance customers must perform, but cloud usage comes with an operational pricetag that prospects should understand before signing on the dotted line. Most mid-sized or larger organizations don’t have a storage or processing power shortage, so operational efficiency is what you are really paying for, in most cases. Make sure it’s worth the cost, and you have a plan if you lose access to that ML engine in the cloud.

    Meanwhile, keep kicking rear. Mark each day with, “At least we’re not SolarWinds,” and, remember: you’re doing your best to protect the org in a dynamic and hostile environment. Celebrate each day without a breach as a success – because someone did try.

  • DevSecOps Implementation: SIEM

    DevSecOps Implementation: SIEM

    The world is filled with events. Our inbox floods with events that marketers really want us to pay attention to, while news feeds flood us with events they’re trying to raise above the background noise, but then, the dog barking interrupts our consumption of that information. Our family is, meanwhile, texting us about events on the family level that may be related to events on the national or world stage … meanwhile, social media is full of garbage information about events that may or may not be real, and that you may or may not care about.

    We ignore the vast majority of these inputs and move along with our day. The inputs that require our attention usually, but not always, get it. Sometimes, we fail horribly at filtering, judging importance, or both, and it has life-impacting ramifications.

    This is not too far from the state of security event management when security information and event management (SIEM) was born. Disparate feeds were coming in from all over; analysts – be they dedicated security folks or systems administrators wearing an extra hat – had to guess which of dozens, hundreds or even thousands of events were a real threat to the environment.

    The first step to gaining control of that environment was standardization and aggregation. Get the events into one place, with similar information available for similar events. That was where SIEM came from.

    Since then, the number of events coming in has continued to rise. Companies that were seeing hundreds of events a day are now seeing hundreds an hour, and SIEM vendors had to keep up. The complexity of attacks, and the ability to detect them, has gone up, and SIEM vendors have had to keep up. Tools available for feeding the SIEM and for acting on the aggregated data have evolved … and SIEM vendors had to keep up.

    At this point in time, SIEM is best characterized as a data aggregation platform with limited intelligence that aids in filtering of irrelevant events. It’s akin to turning off social media and telling those close to you to call if needed; it filters out the worst noise and elevates the most important messages.

    Other tools can analyze their own data and make their own deductions, but SIEM is where you bundle all security-related events in one place, so it is the logical place to filter the noise and raise awareness. But it requires a high-volume, adaptable datastore. Vendors have reached that plateau.

    Once volume issues were mastered, SIEM vendors turned to helping filter the insane volume of security event noise. Yes, it is a security event if Bob logged in on a new workstation, but it probably isn’t a noteworthy security event. Unless Bob’s new workstation is in a country where the company doesn’t do business, or Bob is still logged in on his own workstation, blissfully unaware of the other login.

    So, rudimentary filters were applied to the event data in the SIEM. This got rid of a huge amount of noise. Next came filtering events that, logically, looked like notable security events, but just weren’t. Something was scanning ports on the firewall. An internal firewall. Sounds bad, unless the security tools’ logs showed that an authorized user kicked off the scan. Then, it’s a simple matter of asking the user in question, “Are you scanning X?” and moving along. It’d be better for the system to lower the importance of the event based on the fact that it is an authorized user doing the scanning, with corporate-approved tools. Then, an analyst may not need to get involved at all (my security friends are cringing because, literally, they trust no one, and would point out that authorized users can misuse tools, too – so I’ll note that here).

    But the event volume kept going up, and the analysts were even more swamped. Between adding new reporting apps (HIPS alone can add thousands of reporting points to a SIEM), and increases in events on existing monitored points, it was ugly.

    Enter AI. This is when machine learning (ML) started becoming the norm in SIEM. Now, we want the system to avoid using hard-and-fast rules that must be maintained, but to use knowledge it gains by watching the flow of data and an analyst’s resolution to selectively raise/lower the priority of events. Now, the events that actually bubble up to analysts are ones that the ML engine sees as fishy, and the number is greatly reduced. False positives still occur, but not nearly at the volume they did before, as the system learns how to detect them.

    But wait! There’s more!

    All that data in one place with ML engines crawling over it was just too much temptation for data scientists and deeply knowledgeable security personnel. So, the ability to plot a course through seemingly unconnected security events and determine that this or that was actually evidence of intruder or attacker activity spread across the system came next. This is where we are now – the ability to connect the dots on events that analysts (or the system) might have filtered out to find patterns that indicate intrusions.

    But it comes with a price. It’s not cheap to set up, it’s not easy to train, and maintenance of a system that takes inputs from everywhere and tries to correlate the resulting dataset is … work. We’re not talking a little bit of data here, and, like the environment it comes from, we’re not talking a little bit of complexity. Do you need it? I would say yes. Even if you only use SIEM as a big data store, it will help you perform post mortems when (not if) you have a breach. It is worth the effort, unless you are tiny or have no Internet presence to speak of.

    Keep rocking it! SEIM is another tool to consider as you move to DevSecOps dominance, but again, just a tool that requires you to keep the org safe.

  • DevSecOps Implementation: Intrusion Detection

    DevSecOps Implementation: Intrusion Detection

    Originally, this series was just going to be four articles on the DevSec side of DevSecOps. There are many reasons for this, but primarily because that side is cleaner. The other reason is that these topics are beyond the work we were doing at Accelerated Strategies Group.

    But we’ve had a number of requests to continue, and I’ve used many tools in the space, so on we go.

    Intrusion Detection

    Intrusion detection (IDS) started because attackers were getting through firewalls, and, once inside, seemed to have free rein. Intrusion detection wanted to offer another option for detecting and stopping attackers by watching things inside the firewall for indications your organization had been compromised. Relatively quickly, intrusion detection was supplemented by intrusion prevention (IPS). The idea of prevention was to proactively make changes based upon known issues to keep intruders out. Intrusion detection offered information about what paths attackers were exploiting, and intrusion prevention would plug them. Note that IDS and IPS are often further subdivided into host-based (HIDS) or network-based (NIDS) systems. Since their role – detection of intruders – is the same, for this overview we will discuss them together. A serious deep dive into IDS/IPS should look at HIDS and NIDS separately.

    In the DevSecOps space, some have folded these into security monitoring, but they are still performing the same basic tasks – just much better. In fact, if you found intrusion detection to be limited and clunky to work with because of rules maintenance and fingerprinting issues, it’s probably time to take another look. Here’s why:

    Problems With Traditional Intrusion Detection

    The problems with pure rules-based intrusion detection were two-fold.

    First, rules were hard and fast.

    “Look for X and Y and Z. If all are present, raise an incident.” This tended to result in only two types of rules – ones that were too narrow, and thus brittle, or ones that were too broad and that buried IT/security staff in alerts.

    Second, those rules were outdated before they hit the network.

    The old adage “You are always fighting the last war,” (meaning you are doing what worked the last time you won) is more true in IT security than almost anywhere else. For a rule or a fingerprint to be assembled, it has to have been seen. That means we were perpetually protecting against previously successful attacks and doing little against emerging threats. Security vendors took a lot of steps before coming to our current solution, which is a great step in the right direction.

    Attacks against IT infrastructure share a lot of commonalities. Purely rule-based systems counted on tribal knowledge to code those commonalities into rules. It was clunky and imperfect. Templates helped, by adding common items based upon the type of rule you were creating, but, like rules, they tended to be static and old news.

    Enter Machine Learning

    Bright people with an eye on the ever-changing environment knew that we needed something that could adapt to multiple attack forms/vectors and draw the type of conclusions that security analysts were tasked with. The ability to reduce the number of alerts that get raised is pretty close to miraculous. Anyone with a web presence sees thousands of security events a day, as attack bots and scanners automate chunks of the attack matrix. Some kind of solution to weed out the false information is nearly mandatory, as no organization has the manpower to monitor at those levels and reasonably follow up on threats.

    Machine learning (ML) makes connections between alerts – that might not be a big deal in isolation – and other events that, together, make a case for human intervention. Added to fingerprinting and rules, it is a powerful addition. While a successful attack would require going through rules and making certain you covered the behavior/changes found in the new attack, ML can take that information in and adapt automatically.

    This is cranking through volumes of data we couldn’t even imagine a decade ago, and finding anomalous behavior, correlating it and alerting where necessary. How much it reports up the chain is generally configurable, allowing an organization to tighten the flow of alerts that analysts have to deal with.

    Go With the Flow

    More recently, flow evaluation has become a positive step in the monitoring process. If this connection/user moves from A to B to C, it might be an attack. This allows for another way to rate suspicious activity and raise it to an analyst’s awareness.

    An increasing number of vendors are moving the analysis portion of these activities into their hosted environment. This is a “great strength/great weakness” solution – putting the data points gathered from your system into their cloud allows them to correlate events and improve all customers’ responses. The other side of the coin is that you are paying them operational costs each month, and for most vendors, you’re charged based on how much of the service you use. This has budget implications. It also assumes you have connectivity to their service. If you don’t, for any reason, no analysis will be done on collected data. A careful rollout and contingency planning are in order for this scenario to be successful.

    Integration with security information and event management (SIEM) systems is ubiquitous. We’ll talk about SIEM next; I just wanted to include here that IDS/IPS vendors are feeding SIEM.

    IDS isn’t perfect, and there are a ton of other things out there to help, but it’s a step in the process, one you should consider as you gear up your DevSecOps to professional levels. Some of you are using the tools, but don’t have them integrated into your DevOps systems. This is a nudge to do so. If resolution of an attack automatically creates tickets for Dev to make the code (or ops to make the config) more secure, it absolutely is a plus to the organization responding proactively.

    Keep kicking it. These are just tools, and they’re useless without you. Figure out what your organization needs, and update those needs regularly as the environment (both attackers and architecture) continually changes around you.

  • Linux Foundation Exam for Certified IT Associates Available

    Linux Foundation Exam for Certified IT Associates Available

    A while back I wrote about an upcoming certification people receive from passing the Linux Foundation Exam. Now that it is available, I’m circling back as promised with more information.

    For those unfamiliar, this certification is specifically targeted at people who are newer to their IT career or just launching it. The goal is to offer an onramp for people to show their basic skills and demonstrate to potential employers their ability to get the job done or learn what is needed. It fits in well with their other offerings, giving Linux admins a path to use the foundation’s certifications to reflect progress in their career.

    I have information about the beta test, more information about what is stressed in the testing, and availability information, so let’s dive right in.

    First off, for those like me who have a friend that might benefit from using this test, it is available for purchase now and can be scheduled in January 2021.

    Who is the Linux Foundation Exam for?

    This exam is aimed at entry-level IT folks or those who are contemplating a move to DevOps. Though it is not targeted at help desk people aiming at systems admin jobs, it is suitable for them also. Its goal is to provide employers with proof of a minimum set of skills in IT basics, with (unsurprisingly) a strong emphasis on Linux. An interesting application that Lee Marer of Outcert, one of the Linux Foundation’s partners, pointed out to us was interest from K-12 and colleges—it could be a benefit to people who are recently out of school to already have something that shows they know their way around.

    Specifically, Lee thought the following groups would benefit from the test, and sees interest from these groups:

    • K-12/College: Already mentioned. A good way to offer practical benefit to coursework and offer prospective employers a hint at an inexperienced student’s capability.
    • Those moving to DevOps: Because a lot of DevOps is open source, a lot of DevOps depends upon Linux. This test shows those looking to move have the background required.
    • Those who are intimidated by the more advanced tests available from the Linux Foundation and others. My friend fits into this category. The amount of prep required to pass some tests is not on par with the practical knowledge they test for. This test offers a stepping stone that allows users to study the basics before being asked to advanced tasks such as (for an extreme example) parsing packets.
    • Employers: This exam proves a basic understanding of the topics covered, allowing employers to say “Must have passed this exam” in place of “Must have Doctorate and 15 years of experience.” (I joke, but it does offer employers a bar for help desk or entry-level DevOps.)

    What’s in the Linux Foundation Exam?

    Specifically, Dan Brown, senior manager, content and social media at the Linux Foundation, told us that the test would cover the following topics:

    • Linux Fundamentals (20%)
    • System Administration Fundamentals (20%)
    • Cloud Computing Fundamentals (20%)
    • Security Fundamentals (16%)
    • DevOps Fundamentals (16%)
    • Supporting Applications and Developers (8%)

    We had hoped to have a couple of sample questions for you, as an example of what to expect from the test and from those who took it. As of this writing, answers to those questions are not available. We’ll keep you posted.

    How Can I Take the Linux Foundation Exam?

    As mentioned above, it is available for purchase now (at a good price for an exam), and the scheduling of exams opens in January 2021.

    Tests are delivered remotely, with a live proctor and monitoring used to ensure it is an evaluation of the test taker’s knowledge, as opposed to the test taker’s ability to look up answers.

    This will be a point of contention, as it is in interviews and other exams. Simply put, the ability to look up what you don’t know and apply what you find to answer your questions/fix your problems is a critical skill at this point in IT. But for an entry-level test, I think that proving you have the basis to even understand what you are looking up is more important. So while I believe that unfettered internet access should be available during higher-level tests, for this test it makes perfect sense to me that monitoring will detect searches for answers.

    So Why Cover it Here?

    I think that this industry has developed a kind of psychosis. We talk about skills being more important than formal education and for entry-level positions, ability being at least as important as experience. Yet, job postings read like wish-lists. No, I don’t have five years’ experience in that three-year-old tech, thanks. The Linux Foundation Exam offers an avenue for organizations to demand basic competence without going crazy on a laundry list of items. For help-desk, for admin, for DevOps, this test would prove basic competency in a swath of Linux applications and utilities that would simplify the hiring process. I think we need that. I think we need that across the board, but starting with entry-level is good for me. Priced to be available to entry-level people, gauged to judge their basic knowledge, and from an authority on Linux, this might just be the answer at the bottom end of the scale.

    In fact, combining this test with a demonstrated ability to look up and comprehend answers would be all that is needed for most entry-level positions. Employers get assurance that they’re making the right hire, applicants don’t have to try and figure out which of those “requirements” is actually a requirement. Everyone wins. But only if we’re talking about it. So I’m talking about it, to people who’ve laughed at job postings before and know something like this could help: You. Keep kicking it, and consider this another tool to get co-workers who can kick it, too.

  • DevSecOps Implementation: Interactive Testing

    DevSecOps Implementation: Interactive Testing

    This is the fourth installment in this series on DevSecOps. Read the first installment, on static analysis, here the second installment, on source composition analysis, here, and the third installment, on dynamic scans, here.

    Dynamic testing looks at the running application, poking and prodding to see how it reacts to known vulnerabilities. A complete dynamic scan watches a test runner as it runs against the application and tries all points of entry it can find, normally a highly automated process that takes a while.

    Where dynamic scans fall short is in pinpointing the problem within the application. Knowing that things went wrong when a given CVE was exercised against the app is one thing, but knowing what went wrong and where it went wrong is much more useful to make development teams productive. This is where interactive application security testing (IAST) tries to lend a hand. Using tooling in the application, interactive scans can offer the exact module/source file/shared library (etc) that caused the error.

    Generally speaking, interactive tests are best run at a targeted subset of an application, finding details needed to resolve issues the organization is already aware (or suspect) exist. In our test environment, we use interactive scans to zero in on issues that were detected by static and/or dynamic scans.

    It is likely that the interactive testing market will be the one to continue to move down the line to full-on security monitoring and/or it will encompass a growing list of environmental issues—it is, after all, designed for a running application to get exercised and find the issues. It is not a stretch to see these tools move in the above directions, and indeed you can see the beginnings of these moves in different vendors.

    But for today, the things-you-want-to-look-for list is here. My usual disclaimer applies … Through my (and others’) work for ASG, we’ve turned up a lot of information about DevSecOps tools and what to look for. We don’t have room in these blogs to delve into the depths of that research, but I’m offering an abbreviated hit-list of things to look for when moving toward this type of product. There is a lot to look for, and please be aware that this list is just to get your journey jump-started.

    • Language/framework support: It doesn’t need to be as tightly integrated as language-dependent static scans, but the tool still needs to support your specific development efforts. Support for Ruby/RoR, for example, is important if that is your dev platform of choice.
    • Source of vulnerabilities: Like other security tools, understand what interactive testing tools cover and how it is covered. Currently, most of these tools count on an external test running tool to actually run the test, while the dynamic scan watches what is happening inside the application.
    • Toolchain integration and support: Can the tool be called from your build/test toolset? Does it have integrations to feed results back into your build/bug-tracking systems? While these tools offer good interfaces to see what is going on, the explosion in tools that DevOps brought is making us dashboard-weary. Make certain your centralized code quality information store can be used as the single pane (or pain, depending upon implementation) of glass for initiating scans and reporting results.
    • Test tool support: As mentioned above, most of these tools are watchers that rely on another source to run tests. Make certain that there is some manner of coordination between these two so the test runner can be kicked off from the dynamic scan or vice-versa. More manual steps aren’t needed here and just slow the process.
    • False-positive rate: Interactive testing still struggles with false positives, though these tools are getting better. Have an understanding about average false-positive rates of the product in question, then ask about customization options to reduce this number in your given environment without reducing security posture.
    • Cost to configure/maintain: As is probably obvious, tooling an application to look inside of it is a step beyond normal production needs. Make certain the overhead of adding interactive testing to a project and the cost of integrating are worth the results to be had.

    Vendors call it “interactive” because it watches the interaction of the pieces of code inside the app as it’s running. Many customers call it “interactive” because the “tool” used to run apps for testing ends up being people “interacting” with the application. Both are somewhat accurate, though automation is coming apace, as it is for everything software testing.

    Choose a product that has enough automation today to suit your organization’s needs, and is progressing to wow you. Of all the security tools out there right now, interactive holds the greatest promise for wowing us. Since it is happening at runtime and can see inside the application, everything from environment to installed applications on the host could be covered, and that’s where some vendors are headed. An integrated environment security test tool with deep knowledge/reporting that includes knowledge gained from static scans and source code analysis is on the horizon—and, given the direction of all of DevOps, making that solution fully automated and reporting back to build systems is coming also.

    Interactive won’t replace the other steps in DevSecOps, but it will replace some of their functionality—or more often enhance some of their functionality. If a static scan notes a potential vulnerability, dynamic could verify if it is a real vulnerability before a human ever sees a report on the issue, for example.

    I’ll end this as I’ve ended the others: You all are kicking rear and taking names in a tough IT environment that exploded into near-100% remote for most of you. Keep kicking rear and don’t forget security. It is more important as your organization becomes geographically dispersed because now applications that were “secure enough” for the internal network are shared publicly. Even with infrastructure protections such as VPN in place, more caution should be applied to those applications simply because they are more exposed than “must be on our LAN” allowed for.

    Most users forget that in the end, someone is running those servers and keeping things working. So from me, for all those who’ve never seen the inside of IT, thank you. And keep rocking it.

  • DevSecOps Implementation: Dynamic Scans

    DevSecOps Implementation: Dynamic Scans

    This is the third installment in this series on DevSecOps. Read the first installment, on static analysis, here and the second installment, on source composition analysis, here.

    One weakness of static analysis is its failure to account for environment and use. Running static analysis on a code base as the only check before production deployment is akin to looking at the engine of a VW Golf before assembly and pronouncing the vehicle fit to use. The way a vehicle runs after assembly, and the way it is—or, more precisely for dynamic scans, could be—used, matter to any product. That is where dynamic scans come in.

    Exercising the product as it runs is a big deal, and that is what dynamic scanning attempts to do. Automating the testing of applications by exercising inputs and watching the results, dynamic scans can detect a variety of issues that static analysis simply cannot. These tools are the source of a lot of the noise in DevSecOps because they’re testing a variety of scenarios with each run, and things that a dynamic scan sees as vulnerabilities might very well be known to not be a risk at all by the dev team. As with static analysis, this has been improving over time, and they’re well worth a look at this point.

    The best use for these tools is as gateways—before going to test, in test and before going to production, set a policy for promotion requirements and then run dynamic scans to determine if the application meets that policy, needs remediation or needs approval to go live with possible issues known.

    With all of that said, here is a starter list of things to look for when considering using dynamic scans:

    • Production Safe: These tools run tests that try a documented vulnerability to see how the application responds. The thing is, if you’re not very careful, this can cause serious problems in production. Since long-term the trend is to eliminate a pure test environment for all but the most stability-conscious organizations, knowing that the product you choose can run in prod without causing outages is an important step.
    • Vulnerabilities Tested: Most vendors offer a list of what they look for. Compare these lists to your organization’s needs and other vendor’s lists. Since choosing a product is unique to each org, I don’t have much in the way of buying advice, but here I will say if the product does not support at least the OWASP top 10, look elsewhere, simply because OWASP is both a recipe for you to prevent attacks and a handy list of easy vulnerabilities for attackers to look for. All of the vendors we looked at for our ASG work do support OWASP, so this might be non-advice.
    • Integration: Dynamic scans need to be run after the product is fully assembled. This is not to say complete, but the various pieces need to be together—the scans generally look at an application, not pieces thereof. This means they can be spawned at any point after build—or, more specifically, after post-build deployment to dev/test/prod. Make certain there is a place in the toolchain to call the scanner from and proper gateways to suit your organization’s needs. Part of that evaluation is how the application is kicked off: Can your build tool call it directly? Is it deeply integrated into your chosen environment, or will you have to do work?
    • Reporting and Gating: What level of reporting is available: Can dynamic reports and static reports be integrated to show that a potential problem reported by static scan is found to actually be a problem by the dynamic scan? Can an overall status report be found (à la C-level dashboards) to show the security posture of a given application? Can alerts be configured for apps that fail certain tests (so you can make informed decisions before going to production, for example)?
    • RBAC: This is applicable to all scanning, but I’ve chosen to include it here because for dynamic (and interactive), it’s a bit more important. Can you limit what an individual can see/do? In the end, dynamic scans are reporting on actual vulnerabilities in your codebase—only those who need to know to fix the issue should be able to see it; others should just see pass/fail overall for the app, not a list of known vulnerabilities in the app. The ability to run these scans should likewise be limited. Who/when is a corporate decision that is as much policy and process as it is the chosen tool, but if the scanner doesn’t support that policy, enforcement becomes more difficult.

    When you get to interactive scanning, it is time to consider how security issues traditionally have been handled versus how they should be moving forward. Most organizations hold security issues separate from other application development issues. It is time for that to end—and indeed, some organizations ended this process long ago, but too many are hanging on to the past. Security vulnerabilities that can result in a risk to the organization need to be run through the system as high-priority bugs. Because they are. We used to think that something making the UI look bad for 5% of customers was more important than a security issue impacting no one. A truly global internet and the rise of automation even among attackers make this a silly stance to take. No one is impacted by security issues at this second, but sometime in the next minute, everyone might be impacted. It’s like saying you don’t need scalability for Black Friday because the servers are handling traffic well right now (written in early November). The issues that dynamic scanners alert on are generally real (removal of false positives has come a long way in the dynamic space) and generally dangerous. If your scan tool can find it, assume attackers’ scan tools can, too.

    As with the other installments, I’ll thank you all for keeping the lights on in a really topsy-turvy IT environment—you are doing us all a service—and mention that all of your hard work is at risk if security at your organization isn’t taken seriously. We quickly identified the threat from the sudden opening of millions of RDP ports on the public internet, and most of you took steps to secure your own. Keep it going. Make certain the apps you are developing and deploying are as safe as you can make them. And keep kicking rear—you are the rock that’s keeping the suddenly dispersed company clipping along.

  • DevSecOps Implementation: Source Composition Analysis

    DevSecOps Implementation: Source Composition Analysis

    This is the second installment in this series on DevSecOps. Read the first installment, on Static Analysis, here.

    One of the better additions to security in recent years is source composition analysis (SCA). The purpose of SCA is to sit in the gap between static analysis and dynamic analysis to help you find issues introduced by included libraries, projects, etc. While effective, source composition analysis is only available on open source libraries and packages, which are the highest growth areas of external sources in any project. If we are purchasing a product to include in our application, we take our time and compare and contrast to make certain we have the right thing. If there is an open source option, we just drop it in. Particularly if the option is popular—little thought goes into using MySQL database in Python or its equivalent in other languages. If a Python project uses MySQL or MariaDB, it is needed, so it is grabbed and used.

    The growth of source composition analysis is a great boon to security. Having a tool that takes care of monitoring vulnerabilities in code that is included across the board, sometimes in very uncontrolled ways, means vulnerabilities are known before the app goes live or the new release is published. Security staff doesn’t have to try to ferret out what is included in projects across the org and then cross-reference those findings with the National Vulnerability Database (NVD) or project-specific lists. The tool takes a load off security’s shoulders and makes certain developers know the risks their open source software (OSS) usage represents. The best tools notify developers when including vulnerable open source and can be configured to require approval before a vulnerable OSS library/module can be included.

    With all of that said, let’s talk about what to look for. There are some similarities with our static analysis list, but some distinct differences also. Disclaimer: As mentioned in the Static Analysis blog, this is very high-level. The research we’ve done at Accelerated Strategies Group went much deeper than we can present in a series of blogs; this is just to help you understand the top-level issues and get started down the road to a solid DevSecOps plan:

    • Language Support: This goes beyond most users’ needs in static analysis. One of open source’s strengths is building off other projects. It is often the case that Python will call C modules and some Python projects even include Go libraries. This means your chosen development tools aren’t so much relevant for scanning as the development tools chosen by projects—and their included projects.
    • Depth of Checking: Which brings us to, “How deep does the tool go?” Nesting of open source libraries can be deep and complex, with the project an application uses directly just the tip of the ice-berg. Ideally, look for a tool that answers the, “Is this even ever called?” question. It’s a good start to alert on included OSS projects and it is far better to alert on ones where the vulnerable parts are called from somewhere in the hierarchy.
    • Where to Run: The market is slowly moving to support both cloud-based and local scans. Make certain your organization’s needs are met by the vendor’s offerings.
    • When to run: Some vendors bundle source composition analysis in with their static analysis IDE plugins; others have developers run the scan command line; others offer source composition analysis at the build step—when the app is together in one place; still others offer all of the above. Again, make certain the needs of your organization are met.
    • What Is Detected: As with most security tools, this is normally a base comprised of public sources (NVD, OWASP), with add-on detection from other sources. Know what those other sources are and what they offer: that is one major difference between most products.
    • Reporting: Make certain that vulnerability reporting has everything you need. Consider the possibility that it alerts on a vulnerable OSS library in source file insecure.go. All other uses of that vulnerability need to be visible in some way from that report. Removing the reference from insecure.go is useless if reallyinsecure.go also includes it and has an exception from management to keep the inclusion.
    • Reporting Integration: While scanning OSS usage and vulnerabilities the usage introduces is important, so also is the ability to see these results side by side with static, dynamic and even interactive scans so the overall security posture of the application can be evaluated.
    • Frequency Versus Cost: OSS vulnerabilities come in unpredictable patterns, with several dropped in a week after no important ones were seen the week before. You are going to want to run these scans frequently enough to feel comfortable with your security posture, but depending upon licensing, each run might cost your company money. Make sure you understand your needs and costs associated with meeting them.
    • Licensing: The newest feature to be added to these tools is the ability to list all licenses of all included libraries so that someone can review and see what risk is introduced to the organization from licensing. Many open source projects are licensed so that commercial use is limited; this tool puts all the various licenses together so an organization can evaluate litigation risk based upon license usage. The best tools allow you to also enter proprietary license information so that all licensing relevant to an application is reviewed together.

    These tools help improve security posture by automating what is an increasingly complex manual process and bringing in vulnerability sources that security (be they secure developers or dedicated security staff) would have to manually look for. They enable shifting OSS vulnerability resolution left, allowing your team to become more agile and proactive. Make certain the tool chosen is the right one for the organization.

    As I said in the last installment, you all are keeping the lights on in a truly difficult time, making a tough job look easy. Don’t let vulnerabilities that can be caught easily with these tools but might stay hidden without stop your great work. They are relatively easy to install and configure; once you have them, you have on-demand OSS vulnerability scanning and if you are like most organizations, OSS is a huge piece of your software infrastructure. Keep rocking it, and make sure you are covered for OSS vulnerabilities and licensing.

  • DevSecOps Implementation: Static Analysis

    DevSecOps Implementation: Static Analysis

    One of the things I’ve done for Accelerated Strategies Group recently is looking into DevSecOps toolsets. This is a fun area for me, as development and security fit together well in my mind. Having a separate security group is useful, even necessary in some scenarios, but letting developers crank out code and looking for vulnerabilities only after has always seemed terribly inefficient. Whenever I’ve managed developers, security training is just part of the package I bring. It’s far easier to write it well the first time (the equivalent of shift-left) than rewrite it.

    I won’t name names here—this is work I did for ASG, it is not mine to give away—but I will talk about DevSecOps today, and moving into the future. I’m planning three to four installments for this series, discussing the different aspects of DevSecOps tools.

    For clarity, we’re defining DevSecOps tools as “developer-enabled security tools.” This means that in some way, they help developers improve overall security posture while leaving them to be developers. We all know security has to shift left in every way if it’s to be useful in a DevOps world, but most organizations haven’t seriously started the move to shift security. Hopefully, this series helps.

    Depending upon which financial analyst firm you prefer, projections are that the DevSecOps market will grow 28% to 33% a year for the next few years. While this is good news for DevSecOps vendors, it implies very few organizations have systemically adopted DevSecOps, or there would not be room for that level of growth.

    Whether a traditional team or a DevOps team, the static analysis portion of DevSecOps is (generally) easy to set up and configure and most products offer feedback at both coding time and build time. IDE plugins have come a long way, some even offering advice as-you-code and others allowing developers to run on-demand static scans of their work. The industry as a whole has worked hard to lower the false-positive rates that made most organizations reluctant to deploy them broadly, and prioritization of issues is possible in the best products, making identification and remediation pretty easy. Try a pilot out: Pick a vendor and project, grab the IDE plug-in and check out what they can do. Most do not yet directly support code smells, but I suspect that is coming rapidly, as the systems are already designed to look at source and determine issues. Adding to the collection of possible issues will be a differentiator that quickly turns into a market requirement.

    Things to check for:

    • Language support: Do the languages you use have a scanner? While most of the tools on the market offer support for a breadth of languages, there are differences, so check the corner cases. This truly is a make or break decision—the best product on the market is useless to your org if it doesn’t support the languages you use.
    • IDE integration: This is pretty straightforward, are the IDEs your developers use day in and day out supported? Most of these tools offer command-line variants also, so it is possible that even without support directly in the IDE, the security functionality of a given product dictates that command-line is better. In that scenario, a process to ensure developers are even running scans is a good idea.
    • Vulnerabilities detected: At a minimum, support for whatever is currently in the National Vulnerability Database (NVD) should be required, along with OWASP compliance. Most vendors offer support beyond the NVD, so a bit of research to see what types of vulnerabilities are detected will be in order.
    • Click-through support: With IDE plug-ins, one of the great strengths of scanning is the ability of the developer to simply click on the vulnerability in the results list and go to the offending code. Make certain this works—and works as advertised.
    • False-positive rate: This has historically been a problem, but the industry has brought it down overall. Ask your prospective vendors to share their numbers. Our experience shows low single-digits for a percent, which is far better than the past, but still leaves room for improvement.

    The other part of static analysis is plugging static analysis into the build process. Most of these products have integrations into the most popular build management tools. Scanning the entirety of the application allows the static analysis to see a broader vision of the app and note issues with interactions between portions written by different developers. Reporting for build-time scans is generally at the application/team level, and the best tools will show results of development-time scans side by side with build-time scans if you want them to.

    Things to check for:

    • Language support: Again with the languages. It’s pretty simple here: The core of the application may be built using languages a given vendor supports but critical parts might not. Integration is where that becomes a real issue. Make sure what needs scanning can get scanned.
    • Integrations: Most of these tools offer automated scans through integration with build tools or manual scans that the build manager can kick off. Automation is preferred by a long shot, so check if your tool of choice is integrated into your organization’s build tooling.
    • Depth: We’re not going to get into source composition analysis (SCA) here—that’s for another blog—but we are going to observe that vulnerabilities can come from your source’s sources. How far does the scanning tool trace in a static analysis situation? Is it deep enough to cover a given application’s integrations and inter-relationships? While SCA will detect a lot that static scanning won’t, things such as “unsafe values sent to [insert external API]” should be available in static code analysis.
    • Reporting: This is where you’ll find the most variety in popular tools. Look at their reporting and click-through support to find what best suits your needs.
    • False-positive rate: Same as IDE integrations above—make certain it is low enough for your organization’s comfort.
    • Scan run location: This is becoming a differentiator in the space—do you want your scans run in the cloud? In the data center? Some mix of both? Check your vendors; there are significant differences in the level of support for these options.

    This is just the tip of the iceberg, of course. It is an entire market segment, and static analysis tools have been around a long time. Many organizations have not taken dev-time security seriously yet, even though we all know we need to; I’m just offering a place to get started.

    DevSecOps just makes sense. Static scanning is mature and generally well-integrated. This is a great place to start shifting security left. We get it: Developers don’t want to be security analysts or they’d have filled one of those many security openings. But IDE integration makes these scanning tools more interactive and less intrusive, showing developers how to do it securely as they code. This is great to me because good coding practices will become second nature with real-time, on-the-spot reminders and pointers. It also means that if the developer acts upon the tool’s recommendations, the code is developed more securely the first time and there will be no need to revisit it for security proofing.

    You’re all keeping the lights on in a terribly tough time. DevSecOps attempts to make it more likely that your astounding work isn’t interrupted. It just makes sense to get these tools to protect your organization’s portfolio and your hard work.