devopsy.kinsta.cloud

Category: Blogs

Citizen Development Program: Enabling Citizen Developers
In part one of this series, we made the case for embracing citizen developers to close the developer demand-supply gap while delivering true business agility. We outlined the foundational elements that are consistent across successful citizen development programs: 1) adopting a low-code application platform, 2) embracing the citizen development journey and 3) marketing (internally) citizen development successes.

We also discussed finding the right citizen developers – individuals with the right aptitude and attitude to be successful building applications using low-code tools. Now that we have identified the right use cases and the ideal citizen developer, we will focus on the key decisions for enabling (and empowering) citizen developers.

Enabling (and Empowering) Citizen Development

To enable citizen developers, you must focus on the individual and set them up for success through training, support, and recognition. Empowerment addresses the citizen development process, the life cycle of an application and, more importantly, reducing the organizational friction associated with both.

As you look to build out a citizen development enablement program, you need to answer the following questions about training, support and recognition of your citizen developers:
- Training
  - What is the training curriculum for citizen developers? Do learning paths differ based on developer category?
  - Is there organization-specific content that needs to be included in the training? If so, who owns the creation and maintenance of that content?
  - How will developer proficiency be assessed?
- Support
  - How will citizen developers obtain ongoing support?
  - Will you schedule office hours hosted by subject matter experts?
  - Will there be an online community?
  - Will citizen developers be aligned with a specific SME / pro developer?
  - How quickly do you want to scale the program?
  - What are the appropriate checkpoints / milestones associated with scale?
- Recognition
  - Will you formally recognize training accomplishments? Perhaps though gamification via badges?
  - How will you track, celebrate and reward successful applications?
  - Does demonstrated competency equate to increased developer access?
It is important to remember that the role of “citizen developer” is frequently an additional responsibility beyond an individual’s day job. As such, it is critical to educate them, support them and celebrate their wins in order to ensure success.

In order to empower your citizen developers, you need to consider both the application development process and the overall application life cycle. You need to answer the following questions about the process and the life cycle:
- Application Development Process
  - What is the overall objective for citizen development? What metrics will you use to track progress?
  - Does each application require a separate approval or are developers, once approved, able to build multiple applications?
  - What are the technical standards and best practices specific to your organization?
  - What is your development process? Who has authority over development, testing and deployment?
  - Is your core development team / center of excellence structured, resourced and funded to support citizen development?
  - How will citizen developers and pro coders collaborate on applications?
- Application Life Cycle
  - What is your application intake process? Who evaluates new app requests for platform fit, redundancy, cost, etc.?
  - Once an application is moved to production, who is responsible for the ongoing support and maintenance?
  - What is your process for rationalizing and deprecating applications?
As low-code platforms continue to evolve, the capabilities that facilitate the collaboration between the citizen developer and the pro coders are critical to deliver a successful application. It is not uncommon for a pro coder to collaborate with a citizen developer to deliver the last 5% to 10% of an application’s necessary functionality.

We have identified the right use cases and the right citizen developers in part one. We have enabled and empowered the citizen developers in part two. In part three of this series, we will explore establishing technical guardrails to protect the platform from the citizen developers, and the citizen developers from each other.

This post is the second in a three-part series that provides real-world insights on delivering business value from citizen development initiatives.
February 22, 2021
Protecting Data from Insider Threats
“Insider threat” can be defined as the threat an employee or a contractor will use their authorized access to, wittingly or unwittingly, do harm to an organization’s security. Certainly, as organizations become data-driven, embrace data democratization and release new applications at breakneck speed, the probability of an insider threat is rising sharply.

A common example of an insider threat is granting permissions to on-call SRE engineers who may need to access system tables across multiple databases, inspect contents of S3 buckets and make changes to the production environment to fix availability issues. Every two weeks, the composition of the team may change, and the composition of services and databases may vary dramatically.

While it is impossible to completely safeguard against insider threats, organizations can better protect their data by investing in best practices that (a) reduce the likelihood of such a threat playing out, (b) limit the blast radius from an insider attack and (c) help detect the attack as quickly as possible. The third point is especially important – the Ponemon Institute’s recent Cost of Insider Threats 2020 Global Report claims that it takes, on average, 77 days to detect and contain an insider attack.

A DevOps Framework for Protecting Data From Insider Threats

For security measures to be effective, they must be non-invasive, especially in agile organizations where DevOps teams are expected to push releases several times a day.

Implementation of Access Policies-as-Code

Implementing access policies-as-code automates workflows for user and service access authorization. Using this approach, manual policy practices are replaced with “code” (policy files readable by your team and by the systems that enforce the policies) so that the right people on your team can request, approve and publish policy changes instantly.

As an example, here’s how the above scenario may be implemented as code:

Engineers can add themselves to on-call status with a code push in GitHub to a Terraform script, with only the team lead authorized to approve this action. (See example 1, below.)

Example 1

Using frameworks like Sentinel, changes to production environments can be limited to only on-call engineers. (See example 2, below.)

Example 2

Similar policies can limit only those engineers who are able to read unencrypted content of S3 buckets by fetching keys from Vault. (See example 3, below.)

Example 3

The above examples demonstrate guardrails that allow engineers to perform their duties, while drastically reducing the administrative burden on leads, resulting in fewer errors. An additional benefit of this approach is that it forces a public review of all changes to policies, which makes the security model more robust.

Zero Standing Permissions

Zero Standing Permissions (ZSP) is a DevOps-friendly security tenet that requires any administrative privilege to be available to users only when needed, and only for as long as needed. It enables teams to configure access, via administrative privileges, to highly critical components of their infrastructure for a limited duration, with restrictions on the scope of those privileges.

Here are a few examples of how teams can implement ZSP for their databases:
- Create an SRE role on the database that can update and delete database tables and a private Slack channel of db-admins that can allow on-call engineers to assume this role.
- Implement a service that can be used to request and grant access via this SRE role for a default of 60 minutes. After that time, the access is automatically revoked and needs to be granted again.
- Further implement policies in BrewOPA that restrict an SRE from dropping tables and databases, which avoids any disasters.
Audit Trail for Privileged Activity

SIEMs are increasingly the nerve center of every security organization. While effective threat-hunting and incident response requires several years of expertise, DevOps teams can significantly bolster the security of their cloud platforms by investing in log collection and basic analytics for all privileged activity in their environment.

These tools include services such as the following:
- Session recording for all Linux boxes using Cmd
- Logging activity across databases with Data Cloud services
- Centralizing and collecting all environment logs using services like CloudTrail.
These logs can be visualized in tools like Kibana, where teams can look at activity pattern anomalies. These could include:
- Someone opening up an FTP connection to outside the organization
- Creating a manual copy of tables outside of the database
- An increased number of authentication failures across resources
While the concept of “least privilege” as a safeguard against insider threats to data has existed for some time, the advent of cloud-native and DevOps methodology has exacerbated the challenge of implementation. In recent years, security-as-code and infrastructure-as-code have emerged as good models for implementing security solutions, and are rooted in the above framework. The future encourages collaboration across various stakeholders, documents security policies for easy review and builds on existing tools and workflows. It will help limit insider threats, limit privilege sprawl and prevent “user fatigue.”
February 22, 2021
Gainful Employment

February 22, 2021
Amex, Adobe Join GitHub Intern Program

GitHub revealed today that Adobe and American Express have joined its Summer 2021 Fellowship program, a 12-week, hands-on curriculum where interns work alongside developers to contribute to open source projects hosted on the GitHub repository.

Vanessa Gennarelli, senior director of education at GitHub, said the overall goal of the program is increase the pool of developer talent with experience working remotely on an open source project that requires a fair amount of collaboration with other developers.

The program was developed in collaboration with Major League Hacking (MLH), which organizes hackathons for students. Each intern participates in a series of short hackathon sprints involving multiple open source projects over the 12-week period. GitHub originally created the fellowship program as a way to address the demand for online internships during the COVID-19 pandemic.

Those internships have provided open source maintainers with access to a pool of talent that can reside anywhere versus, for example, always trying to recruit individuals located in a specific geographic area, Gennarelli said. GitHub is also trying to increase diversity within the open source community, Gennarelli added.

Three quarters of open source maintainers (76%) that have worked with MLH interns reported their open source project was in a better state after working with MLH Fellows. Of eligible fellows who were matched for jobs with an MLH Fellowship partner company, more than 20% ended up receiving full-time offers at that company.

Mike Swift, MLH CEO, said the internships provide a way to partially close a gap in developer expertise that only continues to widen. Organizations are especially keen to find developers that have open source expertise, Swift added.

A survey of 3,400 developers and technology managers conducted O’Reilly Media on behalf of IBM finds 65% of respondents preferred skills related to the underlying open source technologies. The same percentage agreed that contributions to open source projects impress potential employers and result in better professional opportunities. In many cases, open source projects are among some of the most innovative development initiatives in the IT industry, Swift noted.

Of course, as open source becomes a primary driver for building software, more contributors are looking to get paid for their efforts. Much of that compensation comes in the form of organizations that hire developers for the express purpose of contributing to open source projects that they deem critical to their operations.

Regardless of how developers are compensated for their time and effort, it is apparent open source projects are the fountainhead from which many opportunities spring. Future success in any endeavor has as much to do with who someone knows as it does actual talent. The best place for any developer to expand their base of industry contact is to collaborate on an open source project. The more projects they contribute to, the greater the reach of their personal network and, by extension, their net worth, will become.

February 19, 2021
Cloud Misconfigurations Threaten Cloud Migration
If we’ve learned one thing during the COVID-19 pandemic, it’s that digital-native business models are essential to survival. That means cloud adoption is more important than ever before. Research shows that the public cloud market is expected to grow to $308.5 billion in 2021, an increase of 16% from 2020.

But while the cloud holds incredible potential, we have an outstanding security issue to address. That issue is cloud misconfigurations. Through 2025, 99% of cloud security failures will be the responsibility of the customer, quantifying the need for security professionals to turn their attention to security hygiene issues like cloud misconfigurations.

Digging Into the Cloud Misconfiguration Problem

To get to the root of the problem, we must fully understand the risk of cloud misconfigurations and how they differ from more traditional vulnerabilities. While traditional vulnerabilities can be patched, cloud misconfigurations can create vulnerabilities in otherwise secure applications and infrastructure. Imagine cloud infrastructure like a highway system and cloud misconfigurations as road hazards. They can be incredibly dangerous and lead to accidents.

As a specific example of cloud misconfigurations, let’s look at identity and access management (IAM). Poorly configured IAM, such as misconfigured roles or a lack of multi-factor authentication (MFA), can lead to compromised administrative accounts. If a threat actor hijacks a legitimate administrator account, they potentially can take full control of an entire cloud environment. Because IAM sits above the cloud infrastructure layer and all workloads and data within, once it is exploited a threat actor will often be able to circumvent your other security barriers, such as network segmentation, leaving you defenseless.

Cloud Misconfigurations Can Easily Scale

Where misconfiguration risk dramatically increases is through the adoption of cloud-native applications and practices like infrastructure as code (IaC) templates used by DevOps teams. These templates offer teams greater speed and scale for building and managing applications, but the downside is that misconfigurations can be unknowingly replicated from development environments to production environments (where sensitive data is stored) at greater velocity.

In short, the biggest risk to organizations right now is scaling these misconfigurations through the cloud. As the cloud grows in adoption and scale, we’re witnessing these misconfigurations scale right alongside it. Where before, a misconfiguration might have been limited to a siloed application or environment, today, that same misconfiguration can impact the entire organization without checks in place to catch it. Worse, many cloud storage buckets have logging disabled, meaning once threat actors are able to identify a misconfiguration and access an internal cloud bucket, an organization won’t even be able to see what data was accessed.

Understanding the Shared Responsibility Model

This all points to a higher-level, hygiene-related issue to keep in mind when securing cloud environments: responsibility. Particularly, organizations struggle to understand the shared responsibility model and how it applies within their own organizations. While some may fail to delineate between cloud provider responsibilities and their own, the major issue at stake is the shared responsibility between various internal teams that often goes undiscussed and undocumented until there is a security event.

When thinking about software-as-a-service (SaaS), to a much lesser degree the responsibility falls on consumers. However, when considering infrastructure- and platform-as-a-service (IaaS/PaaS) and all the moving parts such as network, user credentials, resource configurations, workloads, identity configurations and more, cloud consumers become responsible for much more. A key consideration to note is that one can never outsource accountability, no matter which cloud model is used. To put this more simply, if one puts data in any cloud provider, they are still accountable for that data.

Enabling a Holistic Cloud Security Strategy

When advising security and business leaders on how they can better secure their cloud environments and applications, I highlight what my team calls “The Big Cloud 5,” a set of recommendations developed to help organizations adopt a holistic cloud security model that accounts for proper security hygiene and shared responsibility.
- Gain awareness and deep cloud visibility
  The very first step to ensuring cloud security is understanding how teams are using cloud technologies, leveraging shadow IT and cloud provider APIs. This allows you to get situational awareness and make informed decisions today as well as in the future. This is not a one-time event, but something you’ll need to do continuously.
- Set guardrails to automatically prevent the most serious cloud misconfigurations
  Drawing lines in the sand around the most offensive (and potentially destructive) misconfigurations that should never exist in an environment is key to automating protection in the cloud. This will help keep templates and practices controlled, so that poor hygiene doesn’t inadvertently take root and spread. Think of this as your “dirty dozen.” What configurations should never exist in your cloud environments?
- Standards are the precursor to automation
  One can’t automate what hasn’t been standardized, and while there aren’t widely accepted security standards yet, key stakeholders in an organization must be in agreement about how to secure cloud infrastructures.
- Train and hire security engineers who code
  To fully leverage APIs, security teams must have engineers who know how to code and automate security processes. An assessment of skills that exist across your security team (e.g., knowledge of coding in the likes of Python or Ruby) can point to areas in training and hiring that need further investment.
- Embed security in the development pipeline
  Map out who, what, when, where and how your organization pushes code into the cloud. Once mapped, identify the least disruptive insertion points for security processes and tools, so that they can exist in as much of the development pipeline as possible.
If you feel your organization is a step behind on proper cloud security hygiene, know that you’re not alone. Earlier in 2021, the Cybersecurity and Infrastructure Security Agency (CISA) released a report highlighting the importance of strengthening security configurations based on the uptick in successful cloud attacks, often attributed to poor hygiene and the mixed use of computing devices in the remote work environment. While this is a growing problem, it is also a reversible trend. Best practices, cyber hygiene and a shared responsibility model can help companies safely and securely migrate to the cloud.
February 19, 2021
The Digital Transformation of Decision-Making
The ability to make effective business decisions has historically relied on factors such as experience, qualifications, talent and, in many cases, luck. Strategic and tactical choices are also informed by ‘borrowing’ the insight and knowledge of others. In many situations, ‘follow the leader’ can be an effective approach, and one need only study any sector of industry or commerce to see organizations everywhere mimicking each other, with competitive advantage based on relatively small acts of incremental innovation.

Today’s ambitious businesses have many more options at their disposal to inform decision-making than previous generations. The extent, for instance, to which businesses can now collect, store and analyze data has been transformed by improvements in technology. The result is many businesses are now sitting on enough data for advanced analytics to inform smarter, more efficient decisions.

The Role of Advanced Analytics

Stepping back for a moment, advanced analytics is a set of tools and techniques that use existing data to develop insights to inform business decision-making – in particular, improving the accuracy of predicting future events based on certain actions. In practical terms, it is particularly useful when organizations are planning to implement a new business activity, where it is employed ahead of legacy processes that simply can’t deliver the same levels of insight.

Online shopping is a prime example. When a consumer clicks on various items on a website, they will often be given other suggestions – sites often will present this information with the words ‘Frequently bought together,’ for example. This is based on what has already been viewed and uses advanced analytics to predict what else the shopper might be interested in. It’s just one of many ways retailers now rely on advanced analytics to intelligently cross-sell, and as many people will know, it’s very effective.

In a wider sense, advanced analytics can be used in a huge variety of scenarios, irrespective of industry, market or type of customer. In doing so, businesses stand to reap a number of potential benefits, including:
- Smarter decision making. By generating data based on strong logic and reliable information, advanced analytics can draw on both past and present data to make more reliable, more confident choices about what to do in the future.
- Focused knowledge. By understanding the factors that improve business outcomes alongside those that are negligible, leaders can focus their attention and investment on what is more likely to deliver the desired impact.
- Risk management. In an era of heightened risk, and the need for greater risk management, advanced analytics can identify risk based on available data, helping to inform business models that minimize risk, or help businesses make decisions based on insight rather than hope. This can also help improve governance and mitigate regulatory and compliance risk.
- Strategy automation. Because advanced analytics takes care of analyzing data, teams have greater flexibility and time to focus on wider strategic considerations, which can help make the business better.
Advanced Analytics in Action

Advanced analytics employs various techniques across an ever-increasing range of use cases. For example, data mining is applied to ensure organizations make use of the most relevant data, extracting what is useful from the raw data. This data can then be subjected to further analysis for a multitude of objectives, such as understanding customers better or building more effective strategies.

In addition, ‘big data’ analytics can be applied to both structured and unstructured data, and can be particularly helpful when the data generated isn’t in a proper format, or for organizations with massive data volumes to process.

Then there’s predictive analysis, where machine learning algorithms are applied to current and past data to identify relationships and patterns. The algorithms then build a prediction model that can inform decision making. In general, the more data that is available, the more accurate predictions will be. It can’t, however, deliver 100% accuracy, and random, unpredictable factors such as a mass healthcare crisis illustrate the obvious limitations on predictive analysis.

Taking this a step further, prescriptive analysis helps determine the best way to implement a business decision. Even when a decision is close to being made, there can be more than one possible implementation path to follow. Prescriptive analysis uses current and past data to suggest possible outcomes before actions are set in stone, recommending the best implementation options.

Advanced analytics is a rapidly growing niche of the technology industry, regularly described by industry experts and commentators as among the most important, transformational IT trends. Given the pivotal role of decision-making across every organization, this perhaps shouldn’t come as a surprise, and for businesses everywhere, it’s now only a question of when they apply it to improve their fortunes, not if.
February 19, 2021
CircleCI Adds Private Orbs to DevOps Toolchain

CircleCI announced today it is enabling IT reams to configure their own private integrations around its namesake continuous integration/continuous delivery (CI/CD) platform.

The company enables integrations with third-party services via reusable config packages, based on YAML files, known as orbs. Now, internal IT teams can create private orbs that they can securely reuse to invoke, for example, a frequently accessed microservice via the command line interface (CLI) and invoked via the CircleCI platform.

CircleCI CEO Jim Rose said the goal is to make DevOps teams more efficient at a time when the number of microservices organizations are reusing is increasing steadily. Rather than having to manually integrate each of those services every time they are needed, a private orb will consistently automate the process using the same template, said Rose. That approach, over time, will enable DevOps teams to more easily build and maintain an ecosystem of microservices, Rose added.

In general, Rose said, with more DevOps teams now working from home to help limit the spread of the COVID-19 pandemic, there is a greater need to automate as many tasks as possible to enhance overall DevOps productivity. Constructing an orb gives DevOps teams a relatively simple way of achieving that goal, Rose said. Today, there is often a small, elite group within DevOps teams that builds tools to automate processes in a way that the rest of the DevOps team can more easily consume. Private orbs make it easier for anyone on the DevOps team to create a reusable process, said Rose.

Many of those DevOps teams are, of course, attempting to accelerate digital business transformation initiatives launched in response to the pandemic. Those initiatives are generally based on applications being constructed using microservices that, ideally, developers will reuse across multiple projects. The CircleCI private orbs can be remotely invoked from anywhere via the CI/CD platform the company manages on behalf of DevOps teams that employ it.

It’s not clear to what degree IT organizations are shifting CI/CD platforms to the cloud in the wake of the pandemic. In theory, a cloud-based platform is more accessible to DevOps teams that, like most employees, are working from home. Even after the pandemic eventually subsides, many members of DevOps teams will undoubtedly be working from home more regularly.

In the meantime, a recent analysis of data from 11 million workflows on the CircleCI platform suggests that, after an initial spike of activity in the early days of the pandemic, DevOps teams settled into a more deliberate pace of application development after quickly adjusting to what has become the new DevOps normal. That new normal is likely to change again once COVID-19 vaccines become more widely distributed, but there will certainly be no returning to the way applications were built and deployed prior to the pandemic.

February 18, 2021
DevSecOps Trends to Know For 2021

For DevSecOps leaders, 2021 will be the year of the open source supply chain attack. It’s already starting, in fact. On January 7, security researchers at Sonatype identified three malicious Java components in the Maven Central repository. The components had identical names to reputable components. Then on January 20, the same research team found three more compromised packages in the node package manager (NPM). In all cases, these malicious components had nearly identical names to common Java and NPM packages.

The bad packages were downloaded thousands of times before they were removed, compromising the open source software supply chain. Such incidents are increasingly common. The practice of intentionally or unintentionally deploying poorly vetted code — we call it shadow code — is inevitable. Shadow code happens when developers include third-party software code in applications without approval or sufficient safety validations. Developers do this for the same reason that they use libraries and other components – it is quicker than writing the functionality from scratch. Often, these components offload compute-intensive or specialized features. Some examples are shopping cart software, payment systems or responsive design requirements. Sometimes, too, developers mistakenly include shadow code that is fake, but has a similar name (this is referred to as typosquatting), like the examples we described above.

Shadow code, plus a growing reliance on open source, magnifies the security risk and attracts more bad actors. Modern applications are composed of anywhere from 50% to 90% open source code, depending on the code audit. Rapid code iterations make it impossible for security reviews to police all code changes and dependencies. With many open source libraries containing hundreds of internal dependencies, even the best-automated code checking tools cannot keep up.

How Shadow Code and Supply Chain Attacks Play Off Each Other

The compounding risks of shadow code and supply chain attacks are one of the most pressing issues that will face DevOps and DevSecOps teams in the coming year. Security researchers at Sonatype tracked a 430% increase in supply chain attacks against 24,000 open source software components in 2020. The report blamed the growth of these types of attacks on two factors; first, DevOps teams are increasing code velocity to accelerate time to market. Second, security teams are responding to traditional zero-day vulnerabilities more quickly, shortening the window for exploits. Combined, the two factors encourage more supply chain attacks.

Attacking the Modern Extended Software Supply Chain

In modern web applications, most front-end and middleware application code comes from third-parties. Some is proprietary, but most code is open source, due to cost considerations. Open source component source code is a massive attack surface that is often poorly monitored, measured and inventoried. DevSecOps teams should create plans to reduce the impact of these attacks in 2021 and beyond.

Because supply chain attacks are often used either as launch pads for other attacks or to steal sensitive information, the business and financial risk from these compromises is considerable. In its annual “Cost of Breach” survey, IBM found that the average data breach cost $3.8 million and required 280 days to fully contain.

Supply chain attacks are likely an even more expensive subset of attack, due to several factors. Because they take advantage of trusted relationships, these attacks may go undetected for months or even years. Supply chain attacks also tend to be sophisticated and target an organization’s most valuable assets.

Evidence is emerging of coordinated attacks attempting to take advantage of supply chain weaknesses at the same time. Multiple attackers have targeted known users of vulnerable and end-of-life (EOL’d) versions of the online shopping platform Magento (aka Magecart attacks). We suspect that advanced hacking groups are either running the same scanning software packages to identify vulnerable targets, or are buying lists of viable targets via the dark web.

The Open Source Supply Chain Risk

The benefits of using open source software in building web and mobile applications is clear. Development teams, DevOps and DevSecOps teams can reuse code or libraries maintained by someone else to ship code faster and reduce the amount of code they must maintain.

Nowhere is this more of a risk than JavaScript and Node.js. The meteoric rise of Node.js as middleware has dramatically increased the number of packages and libraries that can be used by developers. NPM, the package manager for Node.js, is the most frequently downloaded piece of software in use today.

The rapid usage growth of Node.js and JavaScript has created a chaotic environment conducive to supply chain attacks. Attackers can inject code into trusted open source packages in a variety of ways. (To be clear, Java, Python, PHP, Golang and Ruby have the same potential supply chain vulnerabilities. But JavaScript is, by far, the largest target and it dominates the front-end and near-front-end, where user data is easiest to skim).

Some big, corporation-backed libraries, such as React (backed by Facebook), are well maintained. But many other commonly used libraries are maintained by under-resourced groups of developer volunteers. These unpaid maintainers have little incentive to do the “grunt work” of following rigorous security best practices, such as setting up automated code analysis or responding quickly to reports of security problems.

The cost of running all widely used open source JavaScript libraries through a proper code review would likely be in the tens or hundreds of billions of dollars. Because the maintainers of this code rarely profit from their work, they are not inclined to pay for and set up automated review systems. This is true even when maintainers work at large companies; the sheer amount of work behind maintaining a popular open source project can be backbreaking without major organizational support. Even code reviews and running code analysis tools is no panacea. Many popular open source projects change rapidly due to numerous and frequent code commits. So, a clean code review is no guarantee that the component remains safe a week or a month later.

For projects run by unpaid developers, commit access to core repositories is often poorly regulated, because maintainers are happy to get any help at all on their project. Reviews of pull requests may be superficial and miss well-obfuscated attacks. Worse, sometimes developers just get tired and stop maintaining their library. They may or may not notify users. Their ghost library may sit in GitHub or NPM, still downloading properly, but with no one paying any attention to bug reports or other suspicious activity.

The problem of abandoned or outdated code is often underestimated. According to a May 2020 study by Synopsys, 91% of commercial applications contain abandoned or outdated open source components. For these reasons, compromising open source libraries that have either been abandoned, are outdated or are lightly policed is a favorite tactic of malicious hackers looking to mount supply chain attacks – and that threat will only grow in 2021.

Learning to Live with Shadow Code and Supply Chain Risk

Getting rid of all shadow code is not an option. There’s just too much of it in use; halting the process entirely would negatively impact development cycles. Restricting developer behaviors in a heavy-handed manner usually ends badly (with more shadow code). Likewise, getting rid of the supply chain risk is impossible. Modern applications are too complex, with dependencies on open source components that themselves contain numerous third-party dependencies.

That doesn’t mean DevSecOps should do nothing. Developers can and should be educated about the risks of shadow code and supply chain attacks. Taking some simple steps to reduce the risk can go a long way. For example, ask developers to create simple checklists for every piece of third-party code they use to ensure the component meets the most basic risk standards. A checklist might include, “recently updated, responds to emails, has more than X number of downloads.” For open source code, use modern software composition analysis tools like Snyk and Sonatype that check open source libraries and can be dropped into the DevOps pipeline as part of the continuous integration process.

From the other direction, DevSecOps teams should attack the shadow code/supply chain problem by thinking about what application behaviors might indicate a compromise. To this end, teams should create a list of behaviors that are anomalous and create filters or notifications for those. For example, if a sign-up page starts taking consistently longer for users to complete, that might be a red flag for form jacking. As a final backstop, look at using machine learning systems to spot anomalous application behaviors. These systems can monitor all application behaviors, at scale, in a way that humans never can.

Combined, these defensive practices should reduce shadow code and supply chain risks for DevOps and DevSecOps teams. While thinking about these problems is not fun, a proactive approach to tackling this malicious duo will keep your development process and applications far more secure.

February 18, 2021
Building Trust Into Your Product by Design

In today’s world, every tool, every solution and every technology has the potential to do both good and bad. Good, if used according to the way it was designed, but bad if it falls into the wrong hands, or is manipulated or abused. As the ‘Peter Parker’ principle says, “With great power comes great responsibility.”

As the COVID-19 pandemic continues and consumers look to digital communications to maintain some sense of normalcy and connectedness during these desolate times, the onus is on us, industry leaders, to ensure that the products and services being developed to connect and communicate with disparate employees, customers, and partners aren’t weaponized by bad actors.

The best way to do this? By building trust into your design.

You Have to Define Trust to Know How to Build Trust

According to Merriam-Webster, the dictionary definition of trust is “one in which confidence is placed.” But the reality is that in today’s world, there are many ways trust can be perceived.

Think of the automotive industry, where trust can be tangibly seen in the growth and success of certain car brands over decades. There, trust can be seen as ‘brand loyalty,’ or customers who repeatedly purchase from the same brand because they trust the company and trust the product.

Trust in technology is the same. It has to be engineered and cultivated, but at the end of the day trust in the product must be mirrored by consumer trust in the organization. The two go hand in hand.

The best way to approach building and sustaining trust when developing new products boils down to three key things: integrity, third-party accountability and consistency. Integrity stems from delivering high grade products and solutions, in conjunction with demonstrating accountable, transparent communications at the organizational level. Third-party validation further underscores that integrity, and restates the quality of products and services being delivered from a third-party perspective. And finally, consistency; being able to deliver exemplary products and solutions on a continual, consistent basis – the most essential ingredient in maintaining and bolstering trust among customers.

Transparent Communication is Key

As we enter 2021, the threat landscape is fraught with bad actors looking to take advantage of the latest threat vectors and emerging technology. As our lives continue to transition from physical to digital, and as more personal information is stored online, companies have to prioritize proper and timely communication about cybersecurity incidents.

This brings us back to organizational trust being foundational for trust in a product. If you’ve seen the Netflix documentary ‘The Social Dilemma’ you know that social media is an oft-cited example of technology being misused or manipulated by bad actors. But after Twitter suffered a social engineering attack in July, the platform issued a swift update to its users. They were open, prompt and transparent in their analysis of the event, and highlighted not only the updated safety precautions put in place in response to the breach, but also underscored their ongoing commitment to trust and transparency within the Twitter community.

Incidents like these, though harmful in the short term, can be less detrimental if addressed and communicated in a timely, transparent manner. There’s a saying, ‘It’s not what happens to you, but how you react to it that matters.’ The same goes for proper breach disclosure within the tech industry as a whole.

Trust has Never Been More Foundational to the Business of Building

As we enter 2021 and the COVID-19 pandemic rages on, more companies than ever are finding themselves in the business of building. Healthcare providers are drumming up new ways to digitally engage with patients and providers outside the physical confines of a hospital. Financial services organizations are adding more immediate, personalized ways to engage with customers when, where and how they need it. Educators are creating new digital solutions to keep millions of children, learning from home for the foreseeable future, engaged. And retailers are rebuilding the in-store experience for customers entirely online.

With so much building going on around us, it’s important to keep in mind that the building process today requires not only outlining what you’re trying to build and defining how you intend to use it, but also analyzing how it can potentially be abused, manipulated or rendered ineffective.

Consistently running your products and solutions against the latest threat models is one way to ensure your products’ security posture is up to date with the latest attack vectors. Another idea is to incorporate ‘security champions’ into your product development process and teams, to act as mini consultants who evaluate potential cybersecurity vulnerabilities. Once you’ve incorporated methods like these, turning to a third party to assess and sign off on your product’s cybersecurity resiliency is a good way to validate your products’ security adherence and effectiveness in the face of ever-evolving threats.

Trust and Development Should go Hand in Hand

At its core, trust and development should go hand in hand. Developers don’t need to recreate the wheel every time – there are plenty of libraries available online to leverage and pull from. What’s most important is for developers to understand the libraries they’re using to build, and ensuring that proper security protocols are built in from the start.

Trust can seem like a huge lift, or an unnecessary distraction on the way to an MVP, product launch or update. I assure you, this is not the case. No matter how big the effort may appear, investing in trust – and the various methods, ways and changes that harden your products and services against abuse – will pay dividends down the line. Customers will trust your product and your company, and will, therefore, be more likely to remain loyal. If nothing else, keep in mind that building trust takes time, effort and consistency, while losing trust can happen in the blink of an eye.

February 18, 2021
DevSecOps May Be Key to Prevent Data Leaks
Data leakage has become something of a norm in the digital realm, despite the fact that cybersecurity budgets are growing and more engineers are assigned to these very issues. Nowhere does this seem more true than in the realm of health care, where breaches that expose sensitive patient data are, sadly, a fairly common occurrence.

A recent report by Approov illustrates how severe the problem has become. According to Approov’s All That We Let In research report, many of the mobile healthcare applications in use today are leaking sensitive patient data through APIs, potentially compromising millions of patients.

The study tested 30 popular mobile health (mHealth) apps, and the research estimates that some 23 million mHealth users are exposed, at a minimum. The average number of downloads for each app tested was 772,619. Analysts expect that the total number of users exposed by the 318,000 mHealth apps now available on major app stores is likely far greater. Among vulnerabilities detailed in the report:
- 50% of the records accessed contained names, Social Security numbers, addresses, birthdates, allergies, medications and other sensitive data for patients.
- 50% of the APIs tested allowed users (medical professionals) to access the pathology, x-rays and clinical results of other patients.
- 50% of the APIs tested did not authenticate requests with tokens.
- Of the 30 popular apps tested, 77% contained hardcoded API keys, some of which don’t expire, and 7% contained hardcoded usernames and passwords. Seven percent of the API keys belonged to third-party payment processors that warn against hard-coding their secret keys in plain text.
“Look, let’s point out the pink elephant out in the room,” said Alissa Knight, researcher and author of the report. “There will always be vulnerabilities in code, so long as humans are writing it. Humans are fallible. But, I didn’t expect to find every app I tested would have hard-coded keys and tokens, and all of the APIs to be vulnerable to broken object-level authorization (BOLA) vulnerabilities allowing me to access patient reports, X-rays, pathology reports and full PHI records in their database. The problem is clearly systemic.”

The report helps to illustrate a critical point: Sloppy code can lead to data leakage. “These findings are disappointing, but not at all surprising. The fact is, leading developers and their corporate and organizational customers consistently fail to recognize that APIs servicing remote clients, such as mobile apps, need a new and dedicated security paradigm,” said Approov founder and CEO David Stewart. “Because so few organizations deploy protections for APIs – that ensure only genuine mobile app instances can connect to backend servers – these APIs are an open door for threat actors, and present a real nightmare for vulnerable organizations and their patients.”

For organizations deploying apps using a DevOps methodology, it becomes very clear that much more must be done on the cybersecurity side to prevent data leakage. However, the general consensus is that data protection falls under the purview of CISOs and cybersecurity teams in the enterprise. Perhaps there is no better argument for adopting DevSecOps than the data leakage problems caused by insecure code and sloppy APIs. The report echoes that sentiment and offers several recommendations for developers creating and organizations using mobile applications. In other words, organizations should take several key steps to protect their customer data and sensitive resources, including:
- Address both app security and API security: Recognize that synthetic traffic to the API is an issue, and arises from bots and automated tools, not from genuine apps and legitimate data requests.
- Shift left and shield right: Secure the development process and harden apps, but ensure that runtime protection is also in place.
- Protect against X-in the middle attacks: Certificate pinning is critical, but often left undone because expired certificates can block apps and impact customer experience. However, when done correctly, certificate pinning does not impact either performance or availability.
- Improve visibility into controls: Organizations and developers need to monitor the effectiveness of the controls they implement and adjust them easily – both for compliance with legislative mandates and to sustain data security and privacy.
- Continuous testing: Penetration testing and static and dynamic code analysis should be performed regularly.
Interestingly, those recommendations fit squarely into the role that DevSecOps should take in the DevOps process.
February 18, 2021