devopsy.kinsta.cloud

Tag: GitHub repositories

Two Malicious npm Packages Aim to Steal Credentials and Other Secrets

Bad actors took over a npm maintainer account and have published two malicious packages designed to steal credentials, API keys, and other secrets from the computers of victims who download them from the repository.

Analysts with Sonatype’s Security Research Team wrote in a report that the two packages – sbx-mask and touch-adv – likely are more than test packages, with the attackers hijacking the publisher account to take advantage of the trust maintainers build with developers to steal valuable information, in this case, secrets that can include credentials, certificates, or API keys.

Sonatype is tracking the packages under Sonatype-2026-001276 and Sonatype-2026-001275, adding that the malware campaign is still active and under investigation. The attacks haven’t been attributed to a threat actor yet. Sonatype reported the packages this week to npm.

The malicious packages are only the latest examples of a rising trend of bad actors targeting open code repositories, not only npm but GitHub and others.

As Penta Security noted in January, “attackers often create repositories that appear to be legitimate projects and distribute malicious files alongside normal code, or they secretly insert malicious code into existing open source projects to target developers. The fundamental problem is that it is extremely difficult to block these attacks completely at the software level.”

Aikido Security, which offers an app security platform, wrote this month about a mass campaign being run in March by a threat actor known as Glassworm across open code repositories.

Fighting Back

Organizations appear to be pushing back at the rising tide of such software supply chain attacks, with a recent Futurum Group survey finding that 39% of those surveyed expect to increase spending on software security testing, followed by additional investments in securing APIs.

In the recent software supply chain campaign detected by Sonatype, the attackers’ reliance on trusted maintainer accounts rather than creating new packages or posing as popular libraries is one aspect that makes the threat stand out.

“The attackers appear to have taken control of accounts with established credibility in the npm ecosystem,” the analysts wrote. “This dramatically increases the likelihood that developers will install affected packages without suspicion. Additionally, the attackers employed multiple exfiltration techniques, including both webhook-based collection and direct email transmission. This dual approach increases the resilience of the campaign and suggests a level of planning beyond opportunistic attacks.”

They also noted the timing of the publication of the package, writing that multiple malicious versions popped up in a narrow window of time, which indicates that the exploitation was automated or access to multiple accounts was coordinated.

Going Deeper into Packages

According to the report, the sbx-mask package runs automatically through a post-install script to immediately execute its payload, which increases the likelihood of fast credential exposure, the analysts wrote.

For its part, touch-adv does not automatically execute, but rather when the application code is invoked, though not on install. It reads TRACE_ID, collects and formats environment variables, and then sends the exfiltrated data via a POST request to an email address, ethan@vm0.ai via agentmail.

“This distinction is notable,” the analysts wrote. “Rather than relying solely on install-time execution, attackers are increasingly embedding malicious code deeper within npm packages to evade detection. By delaying execution until runtime, these techniques make the malware less obvious during standard dependency inspection.”

Harvesting and Exfiltrating Data

The payload is embedded within application code, which executes either during installation or when the package is imported into an application. The malicious script then runs a series of actions that harvest and exfiltrate sensitive data. This data includes environment variables from the host’s system, sensitive credentials like API keys, authentication tokens, and cloud service secrets.

Once collected, the data is transmitted to endpoints controlled by the attacks, such as webhook services and hard-coded email addresses, in this case, ethan@vm0.ai via agentmail.

“Because environment variables are widely used to manage secrets in development and production environments, this type of attack can lead to serious downstream compromise,” they wrote. “Exposed credentials may enable attackers to access cloud infrastructure, manipulate CI/CD pipelines, or move laterally across systems within an organization.”

Targeting Developers

Developers and organizations that rely on npm packages as part of their software development lifecycle (SDLC) are the primary targets of the campaign, according to Sonatype. The high-risk targets include CI/CD pipelines and automated build systems, cloud-based deployment environments, and local developer machines with access to credentials and tokens.

“In these contexts, a single compromised package installation can expose secrets that grant attackers access to broader infrastructure, increasing the potential for downstream compromise across an organization,” the analysts wrote.

If an organization downloaded one of the packages, they need to remove it immediately and rotate credentials that may have been exposed. They also need to review system and CI/CD logs for unusual activity, with an eye on outbound network requests that could indicate data that’s been stolen.

Overall, they also need to reduce their reliance on long-lived credentials, according to Sonatype.

March 23, 2026
Attackers Testing New Strain of Shai-Hulud on npm: Aikido

Threat actors behind the virulent Shai-Hulud worm that wreaked havoc in open npm repositories toward the end of 2025 apparently are trying out a new strain that comes with slight modifications.

Security researchers with Aikido Security, who have been tracking Shai-Hulud for months, wrote in a report that was updated January 2 that there doesn’t appear to be an outbreak of infections from the latest strain, which was found in the npm package here, but which has since been removed.

The fact that there isn’t a major spread of the strain “suggests we may have caught the attackers testing their payload,” Aikido malware researcher Charlie Eriksen wrote. “The differences in the code suggest that this was obfuscated again from original source, not modified in place. This makes it highly unlikely to be a copy-cat, but was made by somebody who had access to the original source code for the worm.”

The npm package that includes the new Shai-Hulud strain reportedly was initially uploaded in March 2021 by a user dubbed “huquocdat,” though it wasn’t updated until December 28, 2025.

Stealing Secrets

The rapidly propagating worm emerged in September 2025 and quickly spread, creating tens of thousands of malicious repositories that affected hundreds of GitHub users and hundreds of compromised packages. The malware is used to steal sensitive credentials that include API tokens, SSH keys, cloud access keys, and environment secrets – securely encrypted values that function as environment variables within workflows.

The data is then exfiltrated from GitHub repositories controlled by the attackers.

Another version of Shai-Hulud – which arose in November 2025 and in repositories was described as “Sha1-Hulud: The Second Coming” – included the same features found in the first iteration, but also included new capabilities that made it more difficult to detect and faster to replicate, making it more dangerous.

Ashish Kurmi, co-founder and CTO of Step Security, wrote in November that the “malicious code that steals and publicly exposes developer credentials, marking one of the most significant supply chain incidents of recent months. The ‘Sha1-Hulud: The Second Coming’ attack demonstrates that supply chain security remains one of the most critical challenges facing the software development ecosystem.”

Cybersecurity firm Wiz said as much as 27% of cloud and code environments were affected by the second coming of Shai-Hulud.

Changes in Latest Payload

According to Aikido’s Eriksen, changes in the new Shai-Hulud strain include an initial file now called “bun_installer.js” and a main payload called “environment_source.js.”

There is also a new GitHub repository description – “Goldox-T3chs: Only Happy Girl” – when the malware leaks data to GitHub, he wrote.

In addition, the new names of the files containing the leaked data include “3nvir0nm3nt.json,” “cl0vd.json,” “c9nt3nts.json,” “pigS3cr3ts.json,” and “actionsSecrets.json.”

The malware also does a better job of handling errors when TruffleHog, a scanning tool that digs into code repositories to find secrets, passwords, and sensitive keys, times out, and comes with version-dependent package publishing.

“In the previous version, it tried to call bun to publish a package it had infected, which does not work on Windows,” Eriksen wrote. “It now handles that.”

‘Dead Man Switch’ is Gone

He added that “there’s a subtle, but important difference in the order of which data is collected and saved, which suggests an intentional change. … In the old version, it saved the ‘contents’ file first. Now it saves it last.”

There is good news, Eriksen wrote. In the iteration released in November, the malware included a “dead man switch” that tried to wipe the data from the user’s home directory if the data exfiltration failed. In the version found by Aikido, that dead man switch “appears to be gone,” he wrote.

The researchers also noticed a bug that the threat actors mistakenly put into their code. Regarding the file names, in the first case, it tries to fetch the file “c0nt3nts.json, but it actually saves the file c9nt3nts.json

January 5, 2026
Typosquat Supply Chain Attack Targets Go Developers

A backdoor that impersonates a widely used database module in the popular Go programming language can give hackers control of infected systems, according to a senior threat intelligence analyst with developer-focused platform provider Socket.

The malicious package, which the threat actor first published in November 2021 and remains in the Go Module Proxy, typosquats the legitimate BoltDB database module that is used by a broad range of organizations, including e-commerce company Shopify and Heroku, a cloud platform-as-a-service (PaaS) organization, Socket’s Kirill Boychenko wrote in a blog post.

The attack illustrates how a bad actor can exploit features in the Go Module Mirror, in particular the indefinite caching of modules, Boychenko wrote. In this case, once the threat actor published the initial malicious version – v1.3.1 – to GitHub, it was indefinitely cached by the Go Module Mirror service.

“While no prior cases have been reported publicly, this incident highlights a critical need to raise awareness of similar persistence tactics in the future,” he wrote. “With immutable modules offering both security benefits and potential abuse vectors, developers and security teams should monitor for attacks that leverage cached module versions to evade detection.”

Targeting the Supply Chain

The backdoor is another indication of the continued focus by cybercriminals on exploiting the software supply chain to launch their campaigns – including the planting of malicious packages in code repositories like GitHub, npm and Python Package Index (PyPI) – and highlights the ongoing popularity of the Go language.

Developer Nation, a developer community, said in a report in September 2024 that 11.5% of backend developers surveyed said they use Go, a language created by Google. The adoption rate has remained stable in recent years – JavaScript, Java (39%) and Python are still more widely used – but the size of the Go community has expanded along with that of the backend developer population.

Within the Go ecosystem, the BoltDB package is broadly used, with 8,367 packages depending on it, Socket’s Boychenko wrote, adding that “its extensive use across thousands of projects positions BoltDB among the most prominent and trusted modules in the Go community.”

He added that Socket has petitioned to have the malicious package removed from the module Mirror and reported the associated GitHub repository and account, which were used to distribute the package.

Google officials released a statement saying that the module was removed from both the Go module proxy and GitHub and that they’ve added it to the Go vulnerability database for those who think they may have been affected.

“We are addressing this through fixes like capability analysis via Capslock and running comparisons with deps.dev,” they wrote. “We want to thank Socket and the Go team contributors that detected the module and are addressing fixes. We’ll continue to work with the wider industry to raise awareness around common open source security issues like these and work being done through initiatives like SLSA and OpenSSF.”

Typical Typosquat

The malicious package is named github.com/boltdb-go/bolt, which closely resembles the legitimate BoldDB domain, github.com/boltdb/bolt, and – as with all typosquatting campaigns – is meant to trick developers into visiting the hackers’ site.

The backdoor “enables remote code execution, allowing a threat actor to control infected systems via a command and control (C2) server,” Boychenko wrote. “After the malware was cached by the Go Module Mirror, which the Go CLI toolchain downloads from, the git tag was strategically altered on GitHub to remove traces of malware, hiding it from manual code review.”

Immutability Challenges

A key to the attack was how the Go Module Proxy service works, he wrote. The service prioritizes what is cached by performance and availability, so once cached, the module version continues to be available via the Go Module Proxy, even if the original source is modified afterward. It’s a design that benefits legitimate uses cases, which was in this case exploited by the hacker to ensure that it continue to distribute malicious code even those the repository itself was changed.

Such immutability is used to ensure that any user that uses a tagged version gets the same bits, which prevents silent changes or overwrites after they’re published. It also has security benefits, including making sure that a compromised library can’t replace code that’s already been downloaded, he wrote. However, once a malicious version is published, it remains malicious in the cache.

Boychenko noted other reports that looked at the security risks from the caching in the Go Module Proxy, including in GitHub and a reverse-engineer community site.

Evading Detection

The Go typosquat package includes a remote access backdoor embedded among what in all other aspects is a legitimate database function. The malicious code connects to the C2 server at what Boychenko called an obfuscated IP address. The malware takes commands from the server and essentially gives the bad actor full control of the system.

It works to evade detection by distributed functions across multiple files within the malicious package, with one making the backdoor connection while another brings in the components that make up the obfuscated IP address. The malware starts up when a developer calls and includes a capability to automatically restart if it crashes, which gives it continuous access to the infected system.

“The threat actor’s use of a clean, unflagged IP hosted on Hetzner Online GmbH (AS24940) indicates a high level of operational security, suggesting that this infrastructure was procured specifically for this campaign to avoid premature detection and blocklisting,” he wrote. “Unlike indiscriminate malware, this backdoor is designed to blend into trusted development environments, increasing the likelihood of widespread compromise before discovery.”

Check the Package

Such attacks put an onus on programmers.

“To mitigate supply chain threats, developers should verify package integrity before installation, analyze dependencies for anomalies and use security tools that inspect installed code at a deeper level,” Boychenko wrote. “Ensuring that Go’s module ecosystem remains resilient against such attacks requires ongoing vigilance, improved security mechanisms and better awareness of how threat actors exploit software distribution channels.”

February 5, 2025
Fake Stars in GitHub a Growing Security Threat, Analysis Finds

GitHub for several years has had a problem with fake stars being used to artificially boost the popularity of code repositories, particularly those containing malware or used for scams. A recent report shows just how big a problem this is and how it rapidly expanded in 2024.

An analysis by Carnegie Mellon University, North Carolina State University, and developer-focused security platform firm Socket found that about 4.5 million stars on GitHub are inauthentic, a situation that the authors wrote is a “prevalent and escalating threat happening in a platform central to modern open-source software development.”

“Despite its wide prevalence and rising popularity, our findings have also revealed its shady nature: fake stars are at worst, used to spread malware in short-lived repositories, and at best, used as a short-term promotional tool, which does not bring long-term returns,” they wrote.

Stars on code repositories are like “likes” on social media, used to promote and increase the popularity of repositories and the developers behind them. The system can be gamed by using fake stars to inauthentically grow the presence of repositories and in recent years, cybersecurity researchers have found bad actors using fake stars to promote code repositories that contain malware, adding fuel to an accelerating trend of software supply chain threats.

A Problem Years in the Making

Checkmarx in late 2023 wrote that “the growing demand for artificially boosting one’s presence in this thriving community has led to the emergence of a massive black market, with online stores and chat groups openly selling GitHub stars.”

Researchers with Dagster, a data orchestrator vendor for data engineers, earlier that year wrote that those looking to buy GitHub stars didn’t need to go to the dark web; they found dozens of such services through a Google search. They found one repository with 24 fake stars attached.

The authors of the most recent report also found star-selling services on Google, selling stars for 10 cents to $2 a piece and promising to deliver the stars in hours or days.

“These bought GitHub stars … corrupt the already-limited value of the GitHub star count as a decision-making signal for stakeholders in the software supply chain and may pose a security threat to all GitHub users,” they wrote.

Surge of Activity in 2024

To better measure the problem, they built StarScout, a tool for detecting fake starring behaviors – like low activity and lockstep trends – across GitHub and found that such activities surged in 2024 and that most were used to promote short-lived malware repositories that were posing as pirating software, game cheats, or cryptocurrency bots.

They also wrote that the characteristics in profiles of fake stargazers are similar to those of average GitHub users but that many have “highly abnormal activity patterns.” In addition, fake stars used to promote the growth of a repository typically help for fewer than two months before becoming a burden, the researchers wrote.

Looking back five years, the use and impact of fake stars were limited until last year, when StarScout detected fake stars in 15.84% of repositories with about 50 stars.

The same month, Check Point wrote about a threat group, Stargazer Goblin, that was behind a network of GitHub accounts they called the Stargazers Ghost Network that distributed malware and malicious links via phishing repositories in GitHub and used techniques – including starring – to make them appear legitimate. The network, which comprised more than 3,000 such accounts, distributed such malware as Atlantida Stealer, Rhadamanthys, RisePro, Lumma Stealer and RedLine.

A Software Supply Chain Threat

“From a security perspective, our analysis of fake stars confirms the public perception that malicious activities are becoming increasingly common in the current software supply chain,” the university and Socket researchers wrote in their 18-page report. “What’s more, we showcase how scams and malware can be promoted through fake signals in GitHub, reflecting the issues seen on social media but in a unique high-stake scenario.”

The report adds to the growing body of evidence of the increasing use of fake stars for malicious purposes and puts greater pressure on developers to validate the source of software components before using them.

The researcher reiterated the “critical need for vigilance in assessing other repository signals beyond star counts” and also urged the development of better ways to signal the popularity of repositories and developers. In addition, there needs to be more research on spam, fraudulent, and other malicious activities in the software supply chain, particularly given the risk of social engineering attacks, they wrote.

January 7, 2025
More Than 3,000 ‘Ghost’ Accounts Spreading Malware on GitHub

GitHub and similar open-source code and project repositories have become a common target of cybercriminals looking to lure developers into unknowingly downloading malicious scripts.

However, a threat group known as Stargazer Goblin is using a novel method for distributing malware and malicious links through the Microsoft-owned GitHub platform that has included creating a network of more than 3,000 fake accounts that include phishing repositories, according to threat researchers with cybersecurity firm Check Point Research.

Some of those accounts are used to distribute the malware – which includes an array of threats, including Atlantida Stealer and other information stealers like Rhadamanthys and RedLine – and malicious scripts, while others are used to create an air of legitimacy around those fake accounts through such actions as starring (essentially “liking”), forking (creating a copy) and watching.

“The Stargazers Ghost Network changes the game by providing a malicious repository where a malicious link is ‘starred’ and ‘verified’ by multiple GitHub accounts, thereby supporting its legitimacy,” Check Point researcher Antonis Terefos wrote in a report, saying that such actions can help lure victims into downloading the content. “We are entering a new era of malware distribution, where ghost accounts organically promote and distribute malicious links across various platforms. Future ghost accounts powered by artificial intelligence could launch even more targeted campaigns, making it increasingly difficult to distinguish between legitimate content and malicious material.”

Bad Actors Turn to Code Repositories

GitHub, Python Package Index (PyPI), npm, and similar repositories are attractive to hackers who see them as a way of spreading their malware via software supply-chain attacks by implanting malware into legitimate software packages that are then used by developers and organizations.

In this case, the phishing templates and tags in the fake repositories are tailored to the interests of victims, including social media, gaming and cryptocurrency. That highly victim-oriented nature can add to the threats to targets including ransomware, stolen credentials and compromised crypto wallets.

The malicious GitHub repositories primarily target Windows users, though similar malware distribution methods can be used against Linux or Android users, which also have large user databases, according to Check Point.

Stargazer Goblin is running the Stargazers Ghost Network as a distribution-as-a-service (DaaS) that other bad actors can use to distribute their malware or malicious links via the phishing templates on GitHub. In addition, the researchers believe that what was found on GitHub is part of a larger operation that includes similar Ghost accounts on platforms like X (formerly Twitter), Instagram, YouTube, Twitch and Discord.

A Threat Two Years in the Making

The threat group has been running the network of bogus accounts since at least 2022 – and possibly earlier – with Check Point discovering an advertisement from July 2023 on the dark web that was written in both English and Russian and offered to pay others for particular services, such as $10 to start a repository with 100 accounts and $2 for providing an older “aged” repository, with discounts of purchases of more than $500.

The campaign has been effective, according to Terefos. A campaign that ran in January and distributed Atlantida Stealer racked up more than 1,300 victims in four days, with the malicious links possibly distributed through Discord channels. Atlantida steals user credentials, personally identifiable information, and crypto-wallets.

“The repositories targeted various types of victims who wanted to increase their followers on YouTube, Twitch and Instagram and also contained phishing templates for cracked software and other crypto-related activities,” he wrote.

Check Point estimates that Stargazer Goblin has earned more than $100,000 running the network, including $8,000 between mid-May and mid-June alone.

Separating the Jobs

The network also is set up to make it difficult to take the whole thing down. In some cases, the phishing template uses three Ghost accounts that have different responsibilities. One is the phishing repository template, another provides the image used for the template and the last serves malware as a password-protected archive.

“This structure and operational method enable Stargazer Goblin to quickly ‘fix’ any broken links that may occur due to accounts or repositories being banned for malicious activities,” Terefos wrote. “By distributing responsibilities across multiple accounts, the network ensures flexibility in replacing its compromised components. This minimizes disruption to their operations, allowing them to swiftly adapt and continue their malicious activities on GitHub.”

The way the network of malicious accounts is set up makes it difficult for most users to detect, though Check Point recommends keeping operating systems and applications updated, being cautious of unexpected emails or messages with links – particularly if they come from unknown senders – and improving the security awareness of workers.

July 25, 2024
Why Do You Need GitHub Backup?
You’ve probably heard the joke that there are two types of people in IT: Those who do backups and those who will start. Though it’s still valid, this joke has become less relevant to businesses and professionals. The IT industry has been increasing expenditures on security for years, and backup is a critical area. However, despite the growing awareness of the need for backups and the wide availability of modern backup solutions, the problem still exists. The number of security breaches is growing, and the topic of data security looks like an endless arms race. So what can organizations do?

First, let’s analyze some data and trends. Year over year, the number of cyberattacks, especially ransomware, is increasing. The year 2020 saw attacks intensifying due to the pandemic and the sudden shift to remote work, for which, unfortunately, not everyone was prepared. Once organizations got better at enabling effective work-from-home models, the hope was that everything could settle into a new normal. Unfortunately, the new norm became these increased attacks.

It’s a bit like the metaphysical yin and yang—two opposing but complementary forces. One drives the other. Suffice it to say that the estimated cost of the global damage caused by ransomware in 2021 is as high as $20B! And a ransomware attack occurs, on average, every 11 seconds around the world.

According to an Identity Theft Resource Center (ITRC) report, by the end of September 2021, there were more incidents of this type than during all of 2020. The authors of the report emphasized that they found 26 cases where cloud databases were unsecured. As a result, hackers were able to access confidential data belonging to 99 million people.

Git Threats: Are my Bitbucket/GitLab/GitHub Repositories Secure?

Back in May 2019, ransomware attacks impacted hundreds of repositories on GitHub, GitLab and Bitbucket. The attack wiped away all the data and left only one piece of information: The amount of ransom demanded and a method of payment. Eventually, most of the data was recovered but the cost and the amount of time spent recovering from the attack was enormous.

According to the Cisco Benchmark Study, approximately 40% of companies experienced downtime longer than eight hours due to a major failure. And 39% reported that at least half of their systems had been affected by a serious breach. According to IBM, the total average cost of a data protection breach on a global basis is as much as $3.86M, and the highest ransom demanded by cybercriminals in 2020 was $15 million. These numbers are scary. On the other hand, in most cases of successful ransomware attacks, the ransom is ultimately not paid; even if organizations do not pay the ransom, the mere fact that they fell victim and experienced failure or downtime can be very expensive.

GitHub as Backup Tool

GitHub provides many useful security tools. However, we must be aware of two things: What does GitHub backup really mean and what are its features? And what responsibility do cloud services providers have versus users’ responsibility for security? Let’s start with the latter.

Most cloud service providers (including Amazon, Microsoft, Google, IBM, Salesforce, GitHub or Atlassian) operate on the basis of the so-called shared responsibility model. Users of cloud services most often assume that providers are fully responsible for their protection. However, providers are primarily responsible only for ensuring the security and availability of infrastructure, software and access, period. Users are responsible for the security of the business data stored as part of these services. So users themselves have to take care of backup and disaster recovery plans and solutions. In short: Providers are responsible for the security of the cloud itself and users are responsible for data security in the cloud. And at this point, it is worth pointing out that, according to data from the UK’s CybSafe, in 2019, 90% of data breaches were caused by user error.

It can happen to any of us—a ransomware attack, phishing, loss of access to a GitHub account or an entire computer or simply overwriting another developer’s work in a repository. It affects both individual users and large corporations. In July of 2019, Ubuntu Security reported that the credentials for a company-owned GitHub account were compromised. Hackers used these compromised credentials to create repositories, issues, pull requests, etc. The attack did not cause a major crash, but repairing the damage took some work and restoring some repositories or issue trackers to the previous state was difficult.

There are some common features any good backup should have:
- Automation (i.e. daily backups)
- AES encryption with its own encryption key
- Versioning
- Long-term data retention
- Disaster recovery process
- Easy monitoring (audit logs, email notifications)
- Central management
- Multi-tenancy (to manage admins, privileges and roles)
- Scalability
Based on these criteria, GitHub (and other similar hosting services) cannot be considered a proper backup tool. GitHub is a great open source project, but its purpose is completely different.

How to Back Up Manually

Since the cloud service itself is not enough, how can you take care of GitHub backup? Well, you could do it on your own. And often, especially in small businesses, this is the preferred method. It may seem like a good idea at first. Knowing Git well, we can create appropriate scripts that create backups for us that give us full control over what is happening without having to pay anyone for it. But this is very risk and not profitable in the long run. Disk space costs money; our employees’ time to create scripts and perform the backup is another cost. And the most important thing, in this case, is that these scripts must be maintained and updated all the time, so they constantly generate additional costs. Any change in such a script may cause an error and, as a result, deprive an organization of a working backup. Of course you could also create an appropriate mechanism to test it, but that’s an additional cost. Manual GitHub backups do not end with writing a single script.

I’ve experienced this myself. In one project, backups were generated using such a script and then kept on our own servers. It was cheaper this way. But we did not have any validation of the created backup. One day, while updating the backup scripts, we did a manual test and it turned out that we had a serious problem—the backup was created correctly but its name did not change, so we kept overwriting the same archive constantly! The effect? It turned out that we didn’t have any copies older than one month! Luckily for us, they weren’t needed because the systems and databases were working properly at that time and there weren’t any incidents, but it makes me cringe just thinking about “What if …?”

All in all, manually writing backup scripts is not a very good idea for many reasons. The low cost of this solution is only theoretical; with time, the cost of maintaining this mechanism increases and, in practice, often only one or two people are responsible for it, which creates additional risk. Okay, you might have more control, but overall you are wasting time and money. The backup itself is also not enough, because you need a proper restoration solution for systems or data in the event of an incident. There’s also the question of how to create a backup of the script that creates the backup? Are you sure you cannot make better use of your employees’ working time?

How Much Does it Cost?

It is difficult to accurately measure this; it all depends on the level of complexity of systems, the number of repositories, the skills of your team, etc. However, you can estimate how much the lack of access to services or programmers’ downtime will cost. You can also calculate the development and maintenance costs of writing, testing, maintaining and updating your own backup scripts. The history of GitHub attacks and crashes shows that the data is usually recoverable, which is good news. On the other hand, the time when services are unavailable or when teams are unable to work can be very costly.

Third-Party GitHub Backup Tools

Another solution—and, in my opinion, a better one—is to use third-party backup software.

In fact, in the GitHub documentation, it is recommended:

“Backing up a repository: You can use the API or a third-party tool to back up your repository.”

If you can use the GitHub API, why do you need a third-party GitHub backup tool? Well, first of all, such applications are created by experts in a given field who have knowledge, experience and stay up-to-date on the current security trends. Second, these solutions are ready and available right away; you don’t have to spend time and money reinventing the wheel. Yes, there is a cost, but in the long run (taking into account the factors described earlier) the cost usually turns out to be much lower than a bespoke solution. Outsourcing backup management responsibilities may be incredibly beneficial and will allow us to focus on developing our business instead of managing GitHub repository backup all over again.

When it comes to backup, be smart. If external third-party tools allow us to develop our products faster and more efficiently, then we should not hesitate to use them. And never forget to implement proper backups and disaster recovery plans. Taking care of it today will help ensure a more secure future.
December 2, 2021
Using Netflix’s HubCommander to Automate GitHub Organizations
Netflix owes a great deal of its exponential growth to its phenomenal tech stack. Throttling its content through a single internal API, the company was able to deliver content agnostic of device type and quickly disrupt its contemporary competitors in the consumer entertainment industry.

We’re intrigued, then, when Netflix exposes some of its internal architecture for others to use. HubCommander is just one recent addition to Netflix’s growing collection of open source tools. It’s a ChatOps tool for GitHub management.

We’ve previously covered an open source approach to ChatOps, tracking how to bring free operational tools into the conversation for increased transparency and efficiency. In this article, we’ll resume this research by learning how Netflix has utilized HubCommander and delve into its open source iteration to see how others might leverage it to automate its GitHub organization.

Problems

Managing many GitHub repositories while working in a large organization can be frustrating; especially difficult is the task of managing permission levels between various user subsets. Not only are various user permission levels difficult to trace, but there is an intrinsic security problem to address as well, especially when working with external teams.

“One of the biggest challenges with using GitHub organizations is user management.” — Netflix

Project managers need to administer different organizations, and GitHub requires admin privileges to manage such repository settings; if onboarding thousands of employees, editing this manually could be an extremely tedious job.

Benefits!

At Netflix, HubCommander standardizes its user management approach, enabling automation for GitHub repo permissions across development teams.

“Management of many users on GitHub can be a challenge without tooling. We needed to provide enhanced security capabilities while maintaining developer agility. As such, we created HubCommander to provide these capabilities in a method optimized for Netflix.”

HubCommander thereby provides this missing link, granting admins the ability to quickly grant privileges. Easily to embeddable into chat, the HubCommander is made with ChatOps in mind. Therefore, privileged GitHub organization management tasks can be performed from the Slack channel, “without granting administrative or owner privileges to your GitHub organization members.”

Using HubCommander

To use HubCommander, first, you must have Python 3.5+, Slack, a GitHub organization, and a GitHub bot user with ownership level privileges.

Next, Slack channels with HubCommander installed can make the call !help to return a list of bot commands:

Source: Netflix

To create a repo, for example, you can simply use the command !CreateRepo:

Source: Netflix

Utilizing HubCommander also has security benefits as it reduces the number of GitHub API permissions granted per user—reducing attack vectors inherently increases security. Another point on security is HubCommander’s integration with Duo for additional authentication:

Source: Netflix

The total list of HubCommander features include:
- Repository creation
- Repository deletion
- Repository description and website modification
- Granting outside collaborators specific permissions to repositories
- Repository default branch modification
- Repository PR listing
- Repository deploy Key listing/creation/deletion
- Repository topics creation/deletion
- Repository branch protection enabling/disabling
- Enable Travis CI on a GitHub repo
- Safeguard commands with 2FA via Duo
Automating GitHub With ChatOps

If your code resides in multiple Git repositories, like Netflix, having a method to configure permissions across all entities in a convenient way becomes a necessity. With a bot to perform operational tasks on GitHub repositories, Netflix is benefiting from automatic manipulation and a wider team access, adding to its agile goals:

“The reduction in administrative overhead has significantly simplified our open source efforts.”

Netflix operates with three development channels (a core Netflix OSS, Spinnmaker and a skunkworks project for new ideas). Thus, having a consistent permissions model across organizations can decrease friction. Also, since Netflix collaborates with third parties, such as contractors or other open source contributors, HubCommander also acts as an effective way to maintain security.

Final Thoughts

Netflix recognizes a growing popularity in the ChatOps approach and cites other benefits such as team transparency, a timestamp for each event and a self-service nature—all reasons for adopting a ChatOps strategy into its developer tooling.

HubCommander appears like a good ChatOps tool for GitHub repository management. However, GitHub Actions (still in beta at the time of writing) is a new feature that will aid development workflows, offering automation for the GitHub operational duties themselves. This may spark alternative bots to be developed, especially in terms of workflows that respond to events.

For now, with open source utilities such as HubCommander, certain GitHub duties can be quickened, stimulating a culture of more efficient management to decrease redundant tasks. While shorthand for GitHub repository management might not seem at first glance as a priority, it could scale down operational headaches tremendously.

— Bill Doerrfeld
November 14, 2018