devopsy.kinsta.cloud

Category: DevSecOps

  • WhiteSource Report Finds NPM Vulnerabilities Fixed Fast

    WhiteSource Report Finds NPM Vulnerabilities Fixed Fast

    WhiteSource today published a report that found most of the vulnerabilities that affect node package managers (NPMs), widely employed to deploy JavaScript applications, are addressed long before they are assigned a Common Vulnerabilities and Exposure (CVE) in the National Vulnerability Database (NVD).

    The report, based on an analysis of the vulnerabilities that WhiteSource tracks in its own vulnerability database and revealed at the AWS re:Invent conference, found more than 90% of vulnerabilities in NPM packages are fixed before the vulnerability is published in the NVD.

    Susan St. Clair, director of product management for WhiteSource, said that fact suggests that, overall, the open source community is staying on top of vulnerabilities as they are disclosed rather than waiting for them to be assigned a CVE number.

    The WhiteSource report comes on the heels of a breach of UA-parser.js, a widely employed NPM based on a JavaScript library that detects browser, engine, OS, CPU and device type/model from User-Agent data. Unknown malicious actors found a way to inject malware into the NPM that modified the library. That vulnerability has since been remediated.

    In fact, the WhiteSource report noted 83% of the CVEs assigned to the most widely used versions of vulnerable NPMs could be remediated before the publication of the CVE.

    In the wake of a series of high-profile breaches of software supply chains, there is a lot more focus on how open source software is actually secured. Commercial software products have a distribution list that they typically notify of a security patch. Open source updates, on the other hand, need to be tracked by developers that use that software. The level of attention individual developers might pay to any number of open source updates can vary widely.

    Many developers are also hesitant to immediately update their applications because they are not sure what impact that additional code may have on their application, said St. Clair. For example, their application may have dependencies that are no longer backward compatible with the updated code, she noted.

    Developers also need to determine the degree to which that vulnerability may affect their applications because they often don’t use every component of an open source library, so the fix being offered may not be required.

    Open source project maintainers often depend on their community to help uncover vulnerabilities and create the fixes as quickly as possible. When a vulnerability is detected in an open source library, the more common practice is to report it to the NVD even after the fix for that issue has been made available.

    In general, St. Clair noted the most important thing for the open source community to remember is to remain proactive when it comes to tracking vulnerabilities and then implementing DevSecOps best practices. The one thing that is certain is that a wide range of malicious actors also are tracking those vulnerabilities closely. It’s not uncommon to see a wave of exploits being created once a vulnerability is disclosed via the NVD.

    The challenge, of course, is that the more open source software that is employed, the more difficult it becomes for individual developers to stay on top of every vulnerability in an application portfolio that only continues to grow. On the plus side, vendors have pledged more than $10 million to help better secure open source software. Hopefully, the return on that investment will become apparent sooner than later.

  • A Pro Football Approach to Agile Vs. DevOps

    A Pro Football Approach to Agile Vs. DevOps

    Enterprises and professional football teams have a lot in common. The pressures of being more agile, more resilient while ensuring performance is felt both on and off the field. For CIOs, surviving and thriving requires a serious focus on operations tied to business mandates and growth.

    I am frequently asked questions around infrastructure and operations (I&O) with organizations looking to solve ITOps, DevOps and DevSecOps challenges.

    • How do we shift from legacy ITOps models to transformative digital operations?
    • How do we align process, people and tools for closer integration, consolidation and correlation?
    • How are we structured as teams across functions—ITOps, DevOps, SecOps?
    • How do we transform to analytics-driven operations at speed and efficiency?

    Historically, Agile and DevOps teams face off on the IT playing field as adversaries; both sides want to advance their plans, execute plays and reach the end zone. Disconnects happen when teams are not aligned, goals are not defined and resources are constrained.

    To effectively deliver digital services, teams must focus on the essentials of clear communication, unified culture and measures to ensure resiliency, performance and availability if they are to be effective.

    Traditional I&O teams often operate independently, and knowledge is not shared nor available to cross-functional teams. Attempts to respond and resolve critical incidents can take longer. With DevOps or even DevSecOps, the ideal is for groups to align across functions by breaking down silos of tooling and responsibility.

    In a hybrid ops environment, a lack of integration and coordination between DevOps roles can easily misalign efforts and priorities. Whether it’s hybrid in terms of IT environment or hybrid in support model, there are unique challenges that must be addressed. Not being able to huddle around the same data sets while troubleshooting incidents poses a huge challenge when implementing proper analytics comes into play. Legacy IT operations management (ITOM) tools simply cannot support DevOps’ use of cloud-native services nor support remote IT teams working globally.

    For successful DevOps and traditional Agile teams to coexist and be successful, IT leaders need to change existing mindsets and implement new practices in order to scale.

    • DevOps and ITOps groups have to feel supported by the entire IT organization.

    Often DevOps starts as an experiment with a small group of people working on one application. Everyone in IT, from the leaders to IT workers, needs to understand what the end goals are for integrating new methodologies. Deep realization sets in when DevOps finishes a project and pushes to production which is still supported in traditional ITOps models. It becomes obvious that operations needs to adapt to frequent change.

    • Plan and implement DevOps practices to avoid disruptions.

    With legacy systems still in practice for delivering digital services, DevOps teams cannot disrupt critical day-to-day operations by driving new support models without taking into account migration considerations. Instead, teams will need to work in a hybrid model using application development delivery methods.

    • Collaboration and planning tools are essential.

    The use of collaboration and planning tools alongside automation capabilities is essential in hybrid organizations. Start by automating alerts to generate tickets for the appropriate team(s) to handle versus having Level 1 support process all alerts independently.

    • Build or buy scalable data analysis systems.

    These systems will aggregate and correlate data across infrastructure and digital/cloud-native technologies to deliver contextual insights to problems and provide corrective steps to resolve them.

    Scaling and modernizing operations and creating agile practices is an ongoing process that needs constant planning, testing and adapting. Streamlining DevOps and ITOps processes using automation starts the process as teams begin to work together versus working in traditional silos.

    For winning football franchises, dynasties weren’t built in a single season; the same is true for your digital ops transformation. It takes challenging the status quo—taking a hard look at the deeply embedded ways ITOps has always been done—and reimagining the whole process focused on data-driven digital operations.

  • WhiteSource Adds SBOM Tool That Lists Vulnerabilities

    WhiteSource Adds SBOM Tool That Lists Vulnerabilities

    WhiteSource has added a software bill of materials (SBOM) tool to its portfolio that, in addition to capturing the components of an application, also surfaces vulnerabilities that should be addressed.

    Many organizations are becoming more rigorous about making sure SBOM are attached to every software development initiative in the wake of an executive order issued by the Biden administration that requires federal agencies to employ them. While not every organization builds applications for federal agencies, the fact that the federal government is requiring them in the wake of a series of high-profile breaches has led other organizations to review their software supply chain processes.

    An SBOM facilitates those reviews by making available a formal, machine-readable inventory of software components and dependencies that can be used to track their supply chain relationships and identify dependencies and hierarchical relationships.

    Ori Bach, executive vice president for product at WhiteSource, said WhiteSource SBOM takes the concept a step further by also identifying open source libraries. It will also automatically update the SBOM when changes are made to make it easier to identify unintentional or malicious content being installed during application builds.

    Finally, WhiteSource SBOM also provides remediation guidance that ensures updates won’t break the build.

    Bach said it’s already apparent cybercriminals have significantly increased their effort to compromise supply chains as part of an effort to compromise as many downstream applications as possible. In many cases, cybercriminals are now specifically targeting open source projects that create software that tends to be employed across a wide range of use cases.

    An SBOM in and of itself doesn’t make applications more secure. However, it can provide visibility into the components of an application that can then be used to inform DevSecOps best practices, which organizations are implementing more broadly as responsibility for application security continues to shift left toward developers.

    There are two primary specifications that organizations can use to create an SBOM. The Software Package Data Exchange (SPDX) specification is an open standard for communicating SBOM information including components, licenses, copyrights, and security references. Defined by the Linux Foundation, SPDX is now recognized as the ISO/IEC 5962:2021 international standard. The second is the Software Identification (SWID) format that is today widely used by publishers of commercial software.

    Regardless of the format used to construct the SBOM, Bach said there’s an opportunity to more closely align the increased visibility it enables with the application security goals of the organization building the application.

    The primary reason SBOMs are not as widely used as they might be by now has as much to do with inertia as it does with technology. However, as organizations start to expect an SBOM to be made available, it’s clear application developers and builders will need to find a way to automate the SBOM process as applications are continuously updated. The challenge, of course, is that applications are now being updated with greater frequency—thanks mainly to widespread adoption of DevOps best practices—that now need to be integrated with any tool for creating and maintaining an SBOM.

  • DevOps Teams Struggling to Keep Secrets

    DevOps Teams Struggling to Keep Secrets

    A growing number of organizations are suffering security incidents related to exposed secrets in DevOps CI/CD pipelines, according to a recent ThycoticCentrify report.

    The study paints a troubling picture: Only 5% of survey respondents said most of their development teams use the same secrets management processes and tools.

    The incidents run the gamut, from secrets published in the clear in public cloud code repositories to insecure third-party code to vulnerabilities in the organization’s own code or configurations.

    The problem is often made worse by security teams not knowing what to protect because development teams have their own cloud infrastructure or the ability to create internet-facing applications directly without security teams’ knowledge.

    “The reason why we see so many security incidents related to exposed secrets in DevOps is that using hardcoded passwords and keys is the easy path to getting things to work,” said Joseph Carson, chief security scientist and Advisory CISO at ThycoticCentrify. “We need to make it just as easy for DevOps to use secrets vaults so even when secrets get compromised they are no longer usable, valid or expired.”

    Carson said moving away from persistent privileges to just-in-time, on-demand privileges will reduce these types of misconfigurations and attacks.

    “Unfortunately, many developers continue to use hardcoded passwords or static keys that, when compromised, allow an attacker to gain control of the application resulting in either data exposure, ability to change the application configuration or move laterally around the network,” he added.

    John Bambenek, principal threat hunter at Netenrich, said the fundamental problem is that when organizations go to agile and DevOps, they remove all friction to moving code quickly—and that almost always includes removing security.

    “That means security is inherently reactive instead of being embedded into engineering, and that’s assuming security even knows what to protect in the first place,” he said. “It’s the natural consequence of removing security from the equation and expecting that it’ll just work on the backend. That approach hasn’t worked for decades; a JIRA board isn’t going to suddenly fix that problem.”

    From Carson’s perspective, secrets management is the ability to move away from hardcoded passwords or static keys to just-in-time privileges or one-time-use passwords so even when comprised they cannot be used.

    “Many privileged access management solutions that protected privileged access for years have extended functionality to developers to help move the value into DevOps so they can manage credentials for applications, databases, CI/CD tools and services without causing friction in the development process,” he said.

    Approaches like privileged access security helps enable API-as-a-service and provides instant availability of secrets, SSH keys, certificates, API keys and tokens.

    Secrets Management

    Bambenek added the problem isn’t choosing a secrets management process or tool, but rather that they aren’t in place at all.

    “Pick something that will keep keys and secrets out of public cloud repositories that developers will use that allows for quick and easy rotation of keys as the need arises,” he said. “The most important thing is that it’s the same tool everywhere that operates in a near-invisible process for developers. Four decades of software vulnerabilities have taught me that if you rely on the developers to do the secure thing, you’ll consistently be disappointed.”

    One Wrong Step

    Davis McCarthy, principal security researcher at Valtix, pointed out developers are not security practitioners by trade.
    “Developers are building new ideas, and no good idea comes without experimentation,” he said. “Experimenting with a new technology stack may lead to misconfigurations that unintentionally expose data.”

    McCarthy said shared SSH keys between users create a larger attack surface, and users will store secrets differently creating multiple attack vectors.

    If one user’s endpoint becomes compromised and their SSH keys are exfiltrated, it’s difficult to know what the first mitigation steps are. Between insider threats and malware, the wrong step could lead to a breach.

    “Whereas a software engineer may hold great knowledge of secure coding methodologies, web application attacks and encryption, they are still focused on executing their craft,” he said. “Organizations that do not build security into the development life cycle create a systemic risk that leads to failures in best practices.”

    McCarthy added organizations can centralize and embed secrets management into the tools already in use in CI/CD pipelines by creating an inventory, auditing it and frequently rotating secrets to sensitive environments.

    Bambenek said the short answer is to make sure the infrastructure of CI/CD pipelines is centrally managed and controlled without the ability of engineering teams to ‘roll their own.’

    “No security control will remain intact if someone can pick their own tools without any real governance,” he said. “CI/CD processes don’t require engineering teams to manage the backend of the deployment pipeline which means someone can ensure those functions are in place to begin with.”

  • Logz.io Extends Open Source Observability Platform

    Logz.io Extends Open Source Observability Platform

    Logz.io has enhanced its observability platform to include both unified dashboards and support for distributed tracing to monitor service performance enabled by the open source Jaeger agent software being advanced under the auspices of the Cloud Native Computing Foundation (CNCF).

    At the same time, Logz.io is making available a preview of an anomaly detection capability that leverages machine learning algorithms to build an artificial intelligence (AI) model to monitor data.

    The company has also added support for OpenSearch, a fork of the open source ElasticSearch software project that is no longer available via an Apache license. OpenSearch was created by Amazon Web Services (AWS) and Logz.io as an alternative search engine.

    Finally, Logz.io has added a security event management capability that makes it easier to classify, prioritize and collaborate on mitigation of cyberattacks as they are discovered.

    The Logz.io observability platform is entirely based on open source software. In addition, the company has made contributions to enhance both Jaeger and the OpenTelemetry project, including contributing an OTEL Span Metrics Processor that makes it easier to aggregate metrics.

    Logz.io CEO Tomer Levy said that, in general, it’s still early days as far as observability is concerned. As a concept, observability traces its lineage to linear dynamic systems. Observability at its most basic level measures how well internal states of a system can be inferred from knowledge of its external outputs. The overall goal is to make it easier to query machine data in a way that enables DevOps teams to proactively discover the root cause of issues before they cause further disruption. That’s a major advance over monitoring platforms that typically only provide predefined metrics to identify when a specific platform or application is performing within expectations. The issue, however, is most IT teams really don’t know what questions to ask to take full advantage of an observability platform, said Levy.

    However, as more organizations are exposed to observability platforms, it will become easier to advance both DevOps and DevSecOps as more platforms continue to leverage AI to provide remediation capabilities and automate processes based on the massive amounts of data collected via these platforms, said Levy. There is simply no way the average IT team will be able to manage all the microservices being strewn across a distributed computing environment without the aid of an AI model trained using observability data, he added.

    As DevOps continues to evolve, it’s more than likely that the best DevOps engineers are not going to want to work for organizations that have not invested in some form of observability. The odds of being consistently successful as a DevOps engineer decline sharply as IT environments become more complex.

    On the plus side, it should soon become a lot easier as the overall level of application instrumentation increases to manage those complex environments and to do so at a level of scale that doesn’t require a small army of DevOps engineers.

  • Top 10 DevOps Things to Be Thankful for in 2021

    Top 10 DevOps Things to Be Thankful for in 2021

    Thanksgiving has been observed in the U.S. since 1861 and was officially designated as an official holiday in 1863 by president Lincoln. Of course, the tradition itself traces its lineage to the early days of the Pilgrims.

    This past year, of course, has been one of recovery from a pandemic that caused the deaths of millions of people around the world. Vaccines have saved the lives of countless more, so there is much to be thankful for despite all our recent trials and tribulations. In that spirit, there are more than a few things for DevOps professionals, in particular, to be thankful for this year starting with:

    Digital business transformation: Organizations of all sizes have had to adjust their business models because of both major changes in buyer behavior and an ongoing supply chain crisis. DevOps teams are at the core of any major digital business transformation initiative, so the demand for DevOps expertise continues to outstrip the available supply. Going forward, it’s been predicted we’ll see more digital transformation in the next 10 years than in all the last 40 years combined. The vast majority of those initiatives would never see the light of day if it were not for the expertise of developers and engineers working hand-in-hand to build and deploy applications faster than ever.

    Low-code platforms: As the demand for applications to drive digital workflows spiked in the wake of the pandemic, professional developers relied more on low-code platforms to decrease the time required to build an application. While many complex applications will still be built using procedural code, it’s clear not nearly as many applications could have been deployed without relying on low-code platforms to build them. It remains to be seen how many of the next generations of low-code applications will be built by so-called citizen developers that may soon be flooding DevOps pipelines with application code.

    Microservices: As an architecture for building applications the core concept of employing loosely coupled services together to construct an application goes all the way back to when service-oriented applications (SOA) were expected to be the next big thing in the 1990s. Microservices have, of course, been around for several years themselves. However, with the rise of containers as a software artifact for building microservices, it’s only now becoming apparent that building an application constructed using microservices not only takes less time but is easier to maintain and also more resilient.

    Observability: As a concept, observability traces its lineage to linear dynamic systems. Observability in its most basic form measures how well the internal states of a system can be inferred based on knowledge of its external outputs. In the past year, a wide range of IT vendors introduced various types of observability platforms. These make it easier for DevOps teams to query machine data in a way that enables them to proactively discover the root cause of issues before they cause further disruption. That’s a major advance over monitoring platforms that typically only provide predefined metrics to identify when a specific platform or application is performing within expectations.

    Biden’s executive order: In the wake of a pernicious attack against IT service management software from SolarWinds, the Biden administration published a sweeping executive order that requires federal agencies to implement a wide range of security best practices. Foremost among them is guidance concerning the security of software supply chains; which resulted in many organizations reviewing their entire software supply chain to make sure similar breaches will not occur.

    DevSecOps: As an extension of any DevOps best practice, DevSecOps generally describes an overall shift left in terms of making developers more responsible for application security. While still in its relative infancy, DevSecOps is starting to gain traction as DevOps and cybersecurity teams look to bridge a cultural divide that is as wide as any in IT. The ultimate goal is to deliver more secure applications without reducing the speed at which they are being built and deployed.

    GitOps: At its core, GitOps is an operational framework that takes DevOps best practices used to automate application development and extends them to infrastructure. The basic idea is that the configuration software created to manage infrastructure-as-code is just another software artifact to be stored in a Git repository. The approach provides a means for IT teams to achieve a continuous delivery goal that, more than a decade after DevOps best practices were defined, remains elusive. It’s still early days as far as GitOps best practices are concerned, but this has been the year when GitOps become part of the mainstream DevOps conversation.

    Kubernetes: Simultaneously one the most powerful and most complex IT platforms ever to come down the proverbial enterprise IT pike, Kubernetes has emerged as a de facto standard for building and deploying container applications that can more efficiently scale up and down as required to consume IT infrastructure. The challenge (and the opportunity) now is to make Kubernetes much more accessible to mere mortal IT administrators that typically don’t have the programming skills of a site reliability engineer (SRE) and which are both hard to find and retain.

    GraphQL: Employed as a query language for application programming interfaces (APIs), GraphQL provides a more efficient means of accessing data than the REST APIs that have been employed by developers in the past. Originally developed by Facebook, GraphQL adoption is still in its infancy but it’s clear APIs based on GrapQL will one day be as plentiful as REST APIs. That’s especially critical for microservices-based applications that rely on APIs to access data across a highly distributed computing environment that requires a more efficient means for transferring data.

    Venture and Wall Street capital investments: Investment firms are helping fuel DevOps advances as they continue to make capital available for companies vying for their share of a market that is expected to see compound annual growth rates of 24.7% through 2026. There will, of course, eventually be a wave of consolidation as merger and acquisition activity inevitably ensues. In the meantime, however, there is a wide range of DevOps technologies that might not otherwise be as broadly adopted if it were not for all the cash being poured into DevOps companies.

    The most important thing, of course, is that so many DevOps professionals are still here to build and deploy applications. It’s been a trying year for all concerned; however, as the pandemic crisis continues to recede, we can all be thankful for the opportunities that lie ahead.

  • Are You Feeling the Hiring Crunch?

    Are You Feeling the Hiring Crunch?

    After years of experience, we can now definitively say one thing about information technology: There is always a hiring crunch, and it will always be better soon.

    I first noticed this trend in InfoSec years ago. Everyone was short-staffed and looking for available candidates. Everyone wrote their recruitment text like there was no shortage and they could ask for the moon and everyone was certain that this temporary staffing shortage would end soon.

    It didn’t. It still hasn’t. And it has spread. DevOps in general, development in particular are acting exactly the same these days. And it’s not going to get better.

    So, my advice? If you absolutely must have a person, write your recruitment copy to reflect the actual skills you must have. No bells and whistles, no “We always include…” no “But what if we…” Hire for what you need. Or hire with an eye to training what you need. And offer competitive pay. No one here will argue that IT is underpaid, but when there is a shortage of people, or more are leaving than coming in, then the cost of employees goes up. That’s just the way it is. If you are only hiring the ones you must have, then you can afford to pay them a bit more.

    Meanwhile, invest in even more automation. Let’s face it; again, we can learn from InfoSec. They are doing more with less staff—even if “more” is not “everything that everyone wishes they could”.  They have at least partially conquered their staffing woes by increasing the use of automation. DevSecOps is automating even more of the space, so we really will eventually get over the InfoSec staffing shortage.

    We can do this for all of IT, too—just follow their lead. Invest in automation, take the risk and turn on automated actions. Start small, but get started. Relatively high turnover and lower qualified applicant counts are likely to be the status quo for the foreseeable future, so automating processes as much as possible is not taking away jobs at this point, it is improving response in light of a lack of people for the available jobs.

    So, to sum up:

    • Hire when you can, but be focused. Don’t ask for experience in all three big cloud providers if you’re looking for an Azure person. Don’t ask for fifteen years of experience in Go for that entry-level position. Be realistic, be laser-focused and get the person you need in.
    • Train when you can’t hire. Let’s face it, there are going to be people who are massively familiar with application security but who have never done DevSecOps specifically or never used your DevSecOps solution. Train them—whether they are internal or hired with an eye toward training.
    • For non-critical, long-term needs, explore automation options. That includes both things that are being done manually today and things that are not being done at all. Find a way to get it done in the absence of hiring someone.

    And keep rocking it. You are already getting the job done—I’m just recommending ways to stay on top of it and deliver more while you try to find the additional people you need. A balance of automation and hiring will help you sleep at night and make the rest of the org certain that IT is on the ball.

  • Mastering the Shared Responsibility Model

    Mastering the Shared Responsibility Model

    It’s no secret that cloud-native application development is growing exponentially, with Agile development, IaaS and PaaS from providers like Amazon, Microsoft and Google, enabling innovation at a pace that is challenging for security to keep up with. 

    A global pandemic and the resulting remote work mandates have only accelerated this movement. And with this change in distributed work and agile application development comes greater security considerations—and a higher degree of risk to the business. Security in the cloud is a shared responsibility, but too many organizations find it difficult to fully understand their role in the security of their applications. There are many elements of securing cloud-native applications that cannot be delegated. They require careful attention as part of a risk-based application security program that extends from design to development throughout their application development life cycle.

    With the appropriate visibility, risk-based thinking and automation, cloud-native applications have the ability to be more secure than ever, while enabling faster development and delivery at the same time. But to get to this point, organizations need to understand their responsibilities. 

    Cloud infrastructure security and compliance are necessities for ensuring how applications and data are deployed and operated in the cloud. But what about what is in those applications? Applications are made up of lines of application/infrastructure code and open source libraries built by developers, DevOps, product managers, security champions and other roles. 

    How do we know the components that make up the application are effectively secured? It is too late and not enough to solely rely on a secure cloud posture. Reducing risk early in the development lifecycle will dramatically reduce costs and business risk. Some sources have assessed the cost impact of fixing a risk in early production to be four to five times as much as one uncovered during design, and up to 100 times more than one identified in the maintenance phase.

    Modern applications are delivered through sophisticated “plumbing” of secure cloud infrastructure to our device interfaces. Just as we require clean and safe water to transit the physical plumbing into our homes, we need our applications to remain secure and high-quality as they transit cloud services.

    The IaaS/PaaS Shared Responsibility Model

    The best way to think about security in this cloud-native world is by using a shared responsibility model, with IaaS customers only responsible for certain elements of the security of an application that lies on top of the infrastructure. 

    In the traditional shared responsibility model, IaaS vendors are responsible for the configuration, management and security of infrastructure elements such as networking, databases and compute engines, in addition to physical security, redundancy and more. But each of their customers is responsible for the security and privacy of everything on top of that, from design to code to deployment.

    The Traditional IaaS/PaaS Shared Responsibility Model

    The Customer Side of the Shared Responsibility Model

    What’s missing from this model are software design, code risk and CI/CD security and integrity—and these areas are the most difficult, time-consuming and require a well-thought-out and comprehensive application security program.

    With new approaches to application security, organizations can better understand the risk of their cloud-native applications and automate a significant amount of risk remediation activities at the design or coding phases before deploying to the cloud. This requires taking a broad but contextual view of AppSec across the entire SDLC—and the modern nature of attacks requires a complete view of the SDLC to understand where the most impactful business risk lies.

    Organizations must evaluate the full picture of their risk—from design to code to cloud:

    1. Visibility—Understanding your risk starts with visibility into your security and compliance risks with a real-time inventory of assets, application and infrastructure code behavior, developer knowledge, third-party security alerts, sensitive data and business impact.
    2. Secure by Design—You need to make sure that security is addressed at the design stage—before any code has been written or committed.
    3. Change Management—Organizations need to focus on the risks that matter early in the development process and based on business impact. Material changes to the applications and infrastructure should be carefully evaluated while other changes can be approved more quickly or ignored, thereby accelerating product delivery.
    4. CI/CD Security and Integrity—As an ever-running, highly-trusted and sensitive process, CI/CD is often overlooked. Many attacks in the recent past exploited this Achilles’ heel of the development cycle by embedding attackers’ code within native code commits or other automatic CI/CD processes. Organizations should ensure CI/CD operations are secure with integrity and security checks in place.
    5. App & Infra Risks Together—To properly secure your applications, you must correlate the risks of your applications and infrastructure in one platform.
    6. Automation—Automation must also be employed to limit mistakes caused by human error, ensure consistency in enforcement of policies and increase the speed of the DevSecOps process. Organizations must carefully manage this process, because automation can also introduce a new category of issues, such as race conditions, untimely triggering, incomplete state validation for edge cases, etc.

    Infrastructure-as-Code (IaC)

    The human element of risk has changed with IaaS and PaaS due to the power that cloud administrators now have. Instead of purchasing, configuring and deploying servers over a period of months, an admin can spin up a Kubernetes cluster or an S3 storage bucket, load customer data and expose it to the Internet with relevant identity and access management (IAM) policies in record time via infrastructure-as-code (IaC) using GitOps instead of ClickOps.

    The rise of IaC has enabled us to view, evaluate and respond to infrastructure changes in the same way we would any application code change. Organizations must apply the same governance rules and automation to review changes in infrastructure before they reach production, along with real-time analysis of cloud misconfiguration. 

    The main challenge is context! All IaC tools in the market today are looking at security misconfiguration as a single dimension (for example, an S3 bucket missing encryption; a cloud database missing SSL for incoming connections, etc.) and this creates a lot of noisy alerts. The risks of cloud infrastructure and the applications that run on top of it are inherently connected—it is harder to connect the dots and understand the true risk of changes than it is to accept alerts at face value!

    When securing your applications in the cloud, don’t rely on traditional tools and processes. Today’s technologies enable new approaches that help you understand and remediate risk in entirely new ways and make cloud-native applications more secure than ever.

  • Stopping Mobile Fraud With Automation and DevSecOps

    Stopping Mobile Fraud With Automation and DevSecOps

    Mobile fraud is a multibillion-dollar problem that’s only getting worse. One study showed that mobile click fraud alone rose 64% during the first few months of the COVID-19 pandemic. In fact, during the pandemic, about one in five mobile ad clicks were fraudulent. And that’s just one form of mobile fraud.

    Weaponized botnets, mobile malware, mobile Trojans and mobile cheat engines and dynamic instrumentation tool abuse—the methods that fraudsters use to defraud companies and consumers via mobile devices—are constantly expanding in number and scale. 

    Historically, organizations have fought against fraud through monitoring and data analysis. As transactions and data come in, analytics and AI work to discern legitimate events from fraudulent ones. These systems require training and a lot of data to become proficient at identifying fraud, and they can become very accurate. But fraudsters are always changing their tactics to evade these systems, and, as many of us have experienced, false positives are still a big problem. If you’ve travelled enough overseas, likely you’ve had your bank account or credit card frozen until you could call your financial institution to let them know that your purchases were genuine. 

    Another challenge with existing mobile fraud prevention solutions is that they are designed to protect networks and network resources first. As a result, while fraud is occurring and fraud prevention systems are learning, mobile end users are still being affected. Also, when a bad act is identified and a system enforces protections against fraud, these enforcement points typically block network traffic. That protects the network, but the consumer and their mobile app remain at risk. Once a fraudster compromises a mobile app, they may be able to attack through other vectors, even if one network is shut down. For instance, if a consumer unwittingly downloads and launches EventBot to their mobile device thinking it was a legitimate app, it may compromise multiple banking apps. If one bank shuts down its network after detecting fraud, EventBot may have other opportunities on the device.

    DevSecOps

    As this chart shows, once fraud takes place, damage to a brand, customers and the bottom line begins immediately. Even if it is detected and halted, the harm it has already caused can’t be undone. Mobile app publishers and developers need to stop mobile fraud where it starts, and that means implementing mobile fraud prevention inside the app within a DevSecOps framework. By blocking the technical methods used to carry out fraud via mobile apps, organizations can make massive inroads toward stopping fraud before it ever gets to the network level, preventing both customers and organizations from getting scammed.

    Why Traditional Security Approaches Don’t Work in Mobile Apps

    To achieve a multi-layered defense against fraud, a mobile app should include app shielding, anti-tampering, code obfuscation, data encryption (including strings, resources, preferences), jailbreak/rooting prevention and man-in-the-middle prevention (such as certificate pinning or certificate validation). Certainly, there are commercial SDKs and third-party libraries that can provide these functions, with some limitations. But these approaches typically still require extensive development work and introduce additional frameworks or programming language dependencies and incompatibilities. 

    They don’t solve the fundamental problem of implementing fraud prevention inside a mobile app—it’s too expensive and time-consuming.  The root of the problem rests in the way mobile apps are developed, released and updated in a modern agile organization. DevOps depends on processes that are dynamic, automated, agile, integrated, iterative and continuous. Mobile app security has traditionally been very different: Static with manual, line-by-line coding. This way of implementing manual security will never work in an agile delivery model.

    The promise of DevSecOps is built-in security, delivered as a fundamental part of a release process where security is designed into the process at each and every phase of the life cycle. The security model can be delivered and evolved atomically, iteratively and dynamically so that it adapts to fit the app.

    The future of fraud prevention lies in the mobile app, but to make this possible, the industry needs to apply AI and ML to automate fraud prevention and security implementation into mobile apps on time within an agile, fast-moving and iterative app life cycle and release process.

  • Prioritizing Scalability, Reliability and Security in Engineering

    Prioritizing Scalability, Reliability and Security in Engineering

    As digital products and services are more deeply embedded in critical industries and infrastructure and the implications of problems grow in scope, engineering organizations are renewing their focus on building platforms that are scalable, reliable and secure. More importantly, they are recognizing that this isn’t the responsibility of an SRE or CSO, it’s an ethos that must be carried each day by every engineer on the team. This is an important shift—from teams and cultures that valued speed of delivery above all else to teams and cultures that make high-quality systems the foundation of everything they do.

    This isn’t easy, and big cultural changes don’t happen overnight. There are many things that can go wrong when the team is operating in the legacy mode of prioritizing quick time-to-market but there are also some strategies and best practices for beginning the process of cultural change across your engineering organization.

    The Old Way of Doing Things

    You’re the engineer on call, and you get woken up at 3:00 a.m. by an alert that tells you an app is down. Groggy and annoyed, you look at the message and have no idea what service it’s referring to or where to begin.

    After nearly an hour, you can’t figure out who owns the service or how to get it up and running. So, you call that senior engineer on your team—you know the one—the person who gets called every time there’s a tricky question or problem. Thankfully, they answer your call in the middle of the night and help you bring the app back up. But in a month, the same scenario happens again; only this time, when you go to call that senior engineer, they’re no longer there. They quit to go work at a company that cares about reliability.

    Best Practices for Today

    How can you avoid this vicious cycle? Here are some best practices you can implement to begin shifting focus away from speed and toward scalability, reliability and security.

    1. Evangelize the Narrative That the Systems you Work on Matter

    At the end of the day, your systems are there to serve your customers and users. Leadership and teammates who embody these values are incredibly important. People should worry about letting their customers down, not about being yelled at by their manager or receiving a bad performance review.

    2. Set Explicit Expectations about Uptime

    When you say that your app needs to have 99.99% uptime, it doesn’t have the same impact as saying 52.6 minutes of downtime a year. Translate uptime numbers so they’re tangible—and on people’s minds every time there’s an outage.

    3. Don’t Sweep Problems Under the Rug

    After the dust settles from an incident, it’s important to get together and talk about it. Be transparent about what happened, which person or team made the error and what the implications to users were. The idea isn’t to point fingers or assign blame but to bring the team together so that you can learn, evolve and figure out how to prevent this from happening in the future.

    Here are some questions you can use:

    1. Who wrote the code?
    2. How was it tested?
    3. Should our automated testing have caught this?
    4. Who reviewed this code?
    5. Were they the right people to review the code?
    6. Why wasn’t it caught in the staging environment?
    7. Who released it and who went through the release validation checklist?

    Formalizing this process lets everyone on the team know that mistakes are part of the job and that there’s a standard way that problems are handled. This is a great way to help engineers feel free to do their best work without fear or avoidance.

    4. Define Individual Owners for Every Piece of the System

    If there’s any confusion about who owns what within your engineering org, you need to address that right away; it’s a massive problem. Not only do you need an owner for every part of the system, but they must also be empowered to fix issues and know that they’re the ones on the hook.

    5. Invest in Documentation, Runbooks and Logs

    The owner of each piece of the application is also responsible for making sure that documentation is up to date, runbooks are clear and easy to follow and logs are readily accessible. When you’re getting paged at 3:00 a.m. about a service you’ve never heard of before, these resources are key.

    6. Prioritize the Delivery Pipeline Like it’s a Key Feature

    You should always be thinking about how to build safeguards around your workflows, like automated testing or new tools. This may mean you need to go toe-to-toe with product managers to make sure the work required to build a robust delivery system isn’t getting kicked down to the backlog.

    7. Get Business Buy-In by Using Terms They Understand

    Unreliable code is technical debt. And debt is something all leaders understand, regardless of their technical abilities. Stripe published a survey in 2018 that found the average developer spends more than 17 hours a week dealing with maintenance issues, costing $300 billion per year globally. If you’re pushing to invest in SRE practices, those numbers might help you make your case.

    Culture comes first, but the right tools can reinforce the culture and drive accountability over time. Tools that help teams keep track of critical information such as service owners, documentation and runbooks can help tremendously. Additionally, having easily available dependency maps of the architecture are key to quickly understanding complex systems. Lastly, setting organization-wide standards for best practices and providing templates for spinning up new services that follow these can help ensure that these things are not forgotten.

    This article was based on a talk given by the author at Conf42: SRE 2021.