devopsy.kinsta.cloud

Author: Mike Rothman

  • The Dog Ate My DevOps Dozen

    The Dog Ate My DevOps Dozen

    As the new Techstrong Group team member, I wasn’t familiar with the DevOps Dozen awards. The basics: It’s a cool program that recognizes excellence across a number of categories in the DevOps world. The winners will be announced during our Predict 23 conference on January 12. I have to tell you that the submission process was interesting. 

    It’s not surprising that many of the submissions came in on the last day. Some within the last hour of the last day. But what was surprising were some of the excuses we got when companies inevitably asked for extensions.

    No one actually used the “dog ate our submission” excuse—mostly because, well, it’s an online submission process so, unless the dog swallowed a computer, that’s not really a thing anymore. It seemed that honesty prevailed this year. We got a bunch of folks that said they were working on a major launch; others said they’d been consumed with getting ready for KubeCon and asked for a little more time.

    We were somewhat sympathetic since we also were getting ready for KubeCon and totally understand the all-hands-on-deck push required to get ready for a major event. We also encountered some issues with our payment processor which prevented the completion of some submissions. We’ve fixed the issue and we’ve learned from that experience, so it won’t be a problem next year.

    In terms of the categories, unsurprisingly the top three that attracted the most submissions were “Best DevSecOps Solution,” “Best End-to-End DevOps Tool and Service” and “Best New DevOps Tool/Service Provider”. Given the move to integrate security into the DevOps environment and the move to integrate the variety of DevOps tools onto a platform, the popularity of those two categories was expected. 

    The next wave of categories included “Best Observability Solution,” “Best Cloud-Native Security Solution/Service” and “Best CI/CD Tool” to round out the top six. Again, less strategic and more tactical in this second group.

    If you are looking to go where the crowds aren’t, the least-crowded three categories in terms of submissions were “Best DevOps-Related Video Series (Video),” “Best DevOps Transformation (non-vendor)” and “Best DevOps Presentation of the Year.” Maybe those aren’t the sexiest, most buzzworthy categories, but the competition is minimal.

    So what did we learn from this year’s submission process? Try not to wait until the last minute and that the most competitive categories are those that deal with hot issues. Keep an eye out for the beginning of public voting, and we’ll see you at the awards ceremony during Predict 23!

  • DevOps World 2022: Developer and Security Links Protect Your Supply Chain

    DevOps World 2022: Developer and Security Links Protect Your Supply Chain

    Ever since the SolarWinds attack back in December 2020, software supply chain attacks have been top-of-mind for any company that builds software. The idea of endangering not just your organization by being attacked but also your customers really sharpens your focus.

    Another complicating factor is that applications use a combination of open software components and homegrown code to deliver applications. This software supply chain means your software can be compromised through no fault of your own but instead based on the components you use. The Log4j attack is an example of a component creating vulnerabilities in any software that uses it. So how do we defend our software? We need to think differently about where security starts and how security concepts can be integrated into software earlier in the development process where it interfaces with the rest of the technology infrastructure.

    Building a Secure Pipeline

    Modern software development is built on the CI/CD pipeline. It allows the application security testing and software composition analysis to happen way earlier in the integration process (shift left), and that allows you to identify and fix issues before the code gets anywhere near customers. Sounds simple, no? Just run your code through a pipeline, integrate security testing, identify issues, fix them, and deploy secure software.

    As with most things, what sounds simple in theory can be challenging to implement in reality. First, shifting left requires a collaboration between developers and security teams. Finding the security issues don’t matter if they don’t get fixed. To increase the likelihood of these issues getting fixed, the security team needs to provide context to the developers regarding the urgency of the flaw or vulnerability and some guidance on the best way to fix the issue. To be successful, security folks can’t just drop a report with hundreds of defects and vulnerabilities on the developers and expect that anything productive is going to happen. If this scenario reminds you of getting a report from your vulnerability scanner with hundreds of issues you won’t fix, well, that’s because it’s very similar.

    Second, security must add value to the development and architecture process by offering up infrastructure-as-code (IaC) templates and a toolchain that makes it easy to implement secure application stacks. Maybe that looks like a template library managed by a cloud security Center of Excellence, which can provide these tools across all of the application teams leveraging efforts.

    Combatting Supply Chain Attacks

    Finally, you can implement a security champions program, training a developer (or two) on each team about security issues and they can act as a security emissary to the developers. This provides scale, ensuring that the security team doesn’t become a bottleneck to fixing the security issues.

    Given the sophistication of the software, the attack surface of the code, and the velocity of the development process, combatting supply chain attacks requires tight and ongoing collaboration with the development team and shifting left to catch the security issues earlier in the integration process.

    Want to learn more about this and other related topics? Join CloudBees for DevOps World 2022, held in Orlando, Florida, at the World Marriott Center. Use discount code DW22 when registering and get access to all the fantastic offerings, including keynotes, sessions, training and other interactive activities. We can’t wait to see you there for the DevOps Remix!

  • Learn Something New Every (Cloud-Native) Day

    Learn Something New Every (Cloud-Native) Day

    Being the new guy at Techstrong Group, the first couple of weeks was spent trying to find things. Like the bathroom when I’m at HQ in Boca Raton or the login to the SurveyMonkey account so I could check on the status of our various in-flight research projects. 

    I also realize I have to learn the vernacular, especially around DevOps and this cloud-native stuff. To be clear, I’ve been researching DevOps and how to integrate security into the process (yes, DevSecOps) for a while. But this cloud-native terminology is something I need to learn more about.

    You’re probably a bit confused about why I’m perplexed by the term ‘cloud-native.’ My work in cloud security for the past 10 years gave me some opinions about vernacular. When I say “cloud-native,” I’m typically referring to the native services you use in your cloud provider. Like AWS Fargate or Azure Container Registry. Those services are “native to the cloud (provider).” 

    But I’m realizing that native cloud services are only a portion of the larger cloud-native revolution. Like DevOps, cloud-native is a state of mind, not a service you buy from a cloud provider. It’s about openness. It’s about portability. It’s about agility. It’s also humbling as I realize I have a lot more to learn.

    That’s why I’m looking forward to our CloudNativeDay event on August 10, 2022. I get an opportunity to dig in and learn about some topics germane to my new coverage areas. For instance, the session on cloud-native challenges in traditional enterprises caught my eye. I’ve been involved in born-in-the-cloud companies, so we didn’t have any installed base to consider. But how would you embrace new and disruptive technology when your company has extensive legacy infrastructure? I can’t wait to hear about those more traditional enterprises’ opinions and experiences.

    Observability is also high on my list of things to learn more about. I’ve been working with customers on advanced security monitoring strategies for decades, but looking at the application layer and associated infrastructure is a different animal. I suspect there will be some technical similarities (collect data, analyze data, alert on data, etc.), but the use cases are fundamentally different. 

    I’m also excited for the last keynote of the day. However, I’m a little biased since it involves an excellent conversation Mitch Ashley and I had with Donald Lutz of Taos (an IBM company), where we dug into some of the enterprise migration challenges he’s seen in the field. 

    They say that when you’re young, you know everything. But as you get older, you realize you don’t know anything. I’m definitely in the latter camp, which is why I’m not going to squander a great opportunity to learn—and neither should you. Register and check out the event. I’ll see you there.

  • Why Your DevSecOps Initiative Will Fail

    Why Your DevSecOps Initiative Will Fail

    When you’ve been in and around the security industry for long enough, you get used to the industry hype machine turning a cool innovation into, uh, meh. This hype cycle starts at the RSA conference each year, and folks like me look for new hot stuff on the show floor. For perhaps only the second time in 25 years, I wasn’t at the show in person, but it seems that DevSecOps was the preferred bandwagon on which every vendor jumped. Sigh.

    I’m not sure whether it’s my increasing age or this business that has made me cynical, but here we are.

    To be clear, from a customer’s perspective, there is great value in the concept of DevSecOps, as evidenced by the upcoming DevOps Connect: DevSecOps @ RSAC 2022 virtual event. Having these (sometimes) mortal enemies working together toward a common goal is great, in theory. But alas, in practice (and similar to what happened when DevOps hit the scene), it’s easier said (or displayed in a vendor booth) than done.

    Thus, I’ve come to a sobering conclusion: Your DevSecOps initiative will fail.

    Most organizations will talk a good game for a while and they may even try some stuff, buy some kit, send folks to training and the like. But in the end, despite all these efforts, you guessed it—they will still fail.

    Is this a foregone conclusion for every organization? Well, no. But this isn’t my first rodeo, and I’m pretty sure this one will end like the others—in failure.

    But let’s say you want it to be different this time. Really, you do. And you are willing to take some actions to give yourself a chance at success. If that’s the case, here are a couple of tips to increase your chances, however small.

    Culture

    First and foremost, DevSecOps is not a thing. It’s not a set of tools (no matter what the vendors tell you); you don’t buy it on a VAR line card. DevSecOps (like DevOps) is a cultural shift. Your teams need to work together. They need to collaborate. They need to have shared incentives. And they need to experience the consequences of not playing nicely in the sandbox.

    Cultural shifts are hard. People don’t like to change. They resist. They undermine. They sabotage. It all sounds nefarious, but they do. So unless you can change your culture, you’ve got no shot.

    Shift Left

    Another one of these security-marketing-terms-run-wild is shift left. It means you want to integrate security testing directly into the development process as early as possible. You should be running security checks in the developer’s IDE to ensure they don’t violate security policy (like hardcoding API keys in their source, for example) before they even commit the code.

    You must also integrate security testing into the CI/CD pipeline(s). Thus, you need tooling that works with your preferred pipeline.

    Security Champions

    Security testing will create angst for your developers. They will have false positives. They will have to fix things that they probably don’t understand. It will cause them to miss delivery dates. Of course, they should build it right the first time, but in their mind, they’ll see it as the security folks causing delays.

    The developers will be annoyed, and you’ve got to manage that annoyance—or have senior management quash any rebellions, but that’s not a long-term solution.

    On the other hand, you could establish a security champions program, which anoints (and trains) developers on each team to “represent” security within the development team. They help educate the developers on the tools and secure coding. They diffuse irritation when something goes wrong.

    A strong security champions program is one of the leading indicators of DevSecOps success.

    Break the Build

    Finally, the security team must have the juice to break the build if a critical issue is found during the deployment. We talk about this with many organizations and they nod their heads. But they are lying to me and to themselves. When it comes down to it, someone on the dev team calls someone in senior management and, all of a sudden, that critical issue is now an alert, not a failed deployment.

    Not being able to stop a build will kill your DevSecOps program. It’s like when a quality control issue forces someone to stop the assembly line. If you don’t stop the line, faulty products can result. In a security context, that means some critical vulnerabilities will be much more expensive to fix later.

    So, there you have it. If you can evolve your culture, integrate security testing into your toolchain, empower security champions and empower your team to stop a deployment, you’ve got a chance at a successful DevSecOps program.

    Unfortunately, you probably can’t do these things, and you’ll likely fail. In that case, I have some lovely property by a waterfall I’d like to sell you.

  • Top 5 AWS Security Mistakes: Leaky S3 Buckets

    Top 5 AWS Security Mistakes: Leaky S3 Buckets

    As we get ready to discuss our list of the Top 5 AWS Security Mistakes in the upcoming devopsy.kinsta.cloud webinar, we wanted to provide a preview of the type and depth of information we’ll be discussing. Since the most-talked about, and likely the most vulnerable, aspect of AWS security is inevitably those leaky S3 buckets, it’s a given we’ll tackle S3 first.

    In this article, we’ll dig into S3 buckets, explain how they become vulnerable, cover eight different ways they can inherently become open to the public and how to find and protect any public facing S3 data.

    The S3 Bucket

    S3 is one of the oldest services in AWS—so old that parts of it still support XML-based policies instead of the JSON you see everywhere else. S3 also has a lot of features that aren’t as commonly known or used anymore, such as allowing someone else to host put objects in your bucket yet still maintain ownership of them.

    Think of an S3 bucket as a server that can have subdirectories and files. Because S3 buckets can have a different root url (e.g. “dops.s3.amazonaws.com”) and have their own life cycle and settings, we tend to think of buckets as servers, not directories. Also, directories in S3 allow you to organize objects, but don’t have any distinct settings of their own, other than a name.

    AWS has a wide range of use cases it needs to support, which is why we have all these mechanisms. Sharing publicly, within an account, between accounts, hosting websites and so on, creates complexity. The good news is S3 always defaults to secure and private. The bad news is AWS allows human beings to use it (and therefore weaken security) and can be confusing to manage. For example, S3 supports both read and write permissions (and list, for buckets). It’s possible to have no public read but public write permissions, which could lead to people placing bad files in your directories.

    8 Ways AWS Data Becomes Public

    There are at least eight different ways an AWS S3 bucket can inadvertently become open to the public and be exposed to a data breach.

    • Bucket ACLs (Access Control Lists): This is an XML document that defines the first layer of access. By default, only the account owner has access, but this can be opened up to other AWS accounts or the public at large. Amazon recommends against using these at all, but it’s also the easiest way to make something public quickly, so we see it all the time.
    • Bucket Policies: These are super-flexible JSON policies that allow you to set things such as IP-based and other conditional permissions on a bucket. While this should be the primary way for managing public access, ACLs are the first tab and about three clicks to make something public. It’s this flexibility that leads to mistakes such as opening up to a wider range of IP addresses than intended or using a negative to block access to some IPs but inadvertently granting access to everyone else.
    • IAM Policies: These are the normal IAM permissions you use to control access throughout your AWS account. You can’t make a bucket public with them, since they only govern AWS users, but you can open up access to the bucket by authorizing another AWS service access, and then that service exposes the content.
    • Object ACLs: These are the primary object-level controls. It’s just like a bucket ACL and uses XML to allow access from other accounts or the public. Objects do not necessarily inherit bucket ACLs. You can totally make an object public even if the bucket is private.
    • Explicit IAM or Bucket Policy Statements: IAM policies and bucket policies can have explicit statements referencing the object, which will override the object ACL (if you own the object), since they are evaluated first.
    • Pre-Signed URLs: These object-level policies must be created using code (not the console) and provide temporary access for anyone with the URL. It’s used, for example, to share files for a few minutes or an hour for someone using your app to download a media file.
    • CloudFront Origin Access Identity: CloudFront is the AWS content delivery network and can serve as the front end to S3. You can create something called a CloudFront origin access identity to write an IAM policy that allows CloudFront to access the S3 content. If the CloudFront allows public access, it can access the S3 content and that won’t show in a bucket policy or ACL.
    • Cross Origin Resource Sharing (CORS) Policies: CORS is required if you use S3 in a website and don’t want the browser to break due to same site security settings. CORS in S3 won’t override an ACL or bucket policy but could mask public access in limited situations where the data is exposed in the web code through the authorized site.

    All put together, AWS evaluates IAM permissions first. Within those, the only one to make a bucket public over the web is the CloudFront Origin Access Identity. AWS then turns to the bucket ACLs and policies for any explicit deny statements. Then it looks at the object ACLs for public access. If you make an object public and the bucket policy doesn’t have an explicit deny, it is still public, but otherwise a good bucket policy will block everything. Besides, a lot of the problems are open bucket ACLs, not object ACLs.

    4 Ways to Find Your Public S3 Buckets and Protect Your Data

    What’s the best way to find public buckets? It’s relatively easy.

    1. Log into the console, click on S3 and look for the Public tag. AWS uses some advanced back-end math to evaluate all the bucket policies to figure out if something is public, which catches most of the fringe cases, but it does not show if the bucket is private and objects in it are public.
    2. Check CloudFront to see if you use any origin access identities. It’s easier to look here than in all your bucket policies (usually). This saves you from looking through object ACLs as well.
    3. Check any CORS policies, if you use them. This is the least common situation and I even hesitated to complicate this post by mentioning them.
    4. The bad news … there is no easy way to find public objects. Even finding them programmatically can be difficult if you have a large number. Also, changing a bucket ACL doesn’t cascade to the object ACLs so you need to run through and fix them one by one. However, if the bucket is locked down the attacker would need to know the full URL/name of the object to find it. It’s not perfect, but it’s better than nothing.

    That’s it. Although, if you know of any other fringe cases, please don’t hesitate to let us know. While conducting assessments, we find well over 90% of the problems are just basic bucket ACL and policy mistakes. Object ACLs are also an issue, but slightly lower risk since you need the exact URL to find them if your bucket is otherwise secured properly.

    If you are curious about the other 4 Top AWS Security Mistakes, sign up for the Aug 14 webinar.

    — Mike Rothman