devopsy.kinsta.cloud

Author: B. Cameron Gain

Best of 2021 – Torvalds’ Bug Warning is a Lesson for Linux Users

As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the third in our series of the Best of 2021.

Linux does, occasionally, raise security concerns. While many users see it as the most secure, robust and versatile operating system available — that’s this writer’s opinion, as well — security precautions still have to be taken.

A recent, widely publicized case illustrated this point; Linux creator himself, Linus Torvalds, warned against the use of the Linux 5.12 release. He described a “nasty bug,” and wrote that the situation is a “mess,” due to the use of swap files when adding Linux updates. This nasty bug, in fact, had the potential to destroy entire root directories.

Some of the main takeaways following this “mess” include: tread very carefully when installing early Linux releases, especially those that involve swapping files instead of partitions, and especially, despite Linux’s well-known security advantages, avoid becoming complacent, because Linux security is not always foolproof.

Hence, while the “state of Linux security today is quite good, and has evolved in a positive way with more visibility and security features built, like many operating systems, you must install, configure and manage it with security in mind; that is how cybercriminals take advantage, [via] the human touch,” said Joseph Carson, chief security scientist and advisory CISO at Thycotic, a provider of privileged access management (PAM) solutions.

A Patch for Nastiness

As Torvalds noted a few weeks ago, “most people don’t use a swap file, but a separate swap partition and the bug in question really only happens when you have a regular file system, and put a file on as a swap.”

“The bad news is that the reason we support swap files in the first place is that they do end up having some flexibility advantages, and so some people do use them for that reason. If so, do not use [release candidate] RC1,” Torvalds wrote. “Thus, the renaming of the tag.”

After issuing the warning, Torvalds released a patch that he says prevents the bug from destroying swap file systems. However, it may have already been too late for early adopters of release 5.12. Ubuntu, a leading Linux distro, can swap files by default.

“It is nasty bug if you are still using swap files,” Carson said. “If you do still use swap files, then you could be impacted, resulting in potential data loss or a corrupted system.”

DevOps teams – or anyone else running Linux and installing patches, whether on multi-servers or on individual workstations – still need, of course, to follow strict best practices. “Like any operating system, security depends entirely on how you use, configure or manage the operating system,” Carson said. “Each new Linux update tries to improve security; however, to get the value, you must enable and configure it correctly.”

Linux Goodness

The fact that Torvalds was so forthcoming about the bug, as well as the level of transparency that the Linux kernel offers, also demonstrates one of the many reasons Linux remains popular. Given that the Linux kernel, in one variety or another, is used “not only in about 50% of the internet servers of the world, but also in a substantial part of all our smartphones, it is good to see this level of transparency at ‘root level,” said Dirk Schrader, global vice president, security research at New Net Technologies (NNT), which providers cybersecurity and compliance software.

“The security of Linux is based on its transparency; the ability to review the code of a distribution,” says Schrader. “Quite often forgotten is that transparency also involves talking about the mistakes, the errors, those nasty bugs.”

Citing National Institute of Standards and Technology (NIST) vulnerability database statistics, Schrader described how, compared to the Windows family of desktop and server operating systems, for example, the Linux kernel shows better results for overall vulnerabilities. The number of vulnerabilities have also declined over the past four years, while Microsoft’s operating systems do not display the same trend, according to NIST’s national vulnerability database.

Since Linux’s famous kernel is open source and transparent, it is possible to extrapolate that there are a greater number of potential vulnerability watchdogs compared to those monitoring vulnerabilities in closed systems. Some may argue that Microsoft has been, at times, less successful at detecting vulnerabilities and issuing much-needed patches.

However, Linux users still must remain vigilant.

“Still, for any of the Linux distributions, anyone using the early release candidates — RC1 in particular — should make sure that their own development or build process is undergoing change control, so that no mishaps will transfer the nasty bug into a production environment,” said Schrader.

December 21, 2021
NS1 Touts a Common Delivery Platform for Devs and Ops

NS1 has extended the capabilities of its DNS and network management platform with the release of NS1 Connect. The platform is designed to serve as a delivery platform that integrates developer and network operations teams’ tasks for the creation and management of applications across highly distributed cloud-native and on-premises environments.

While DevOps teams typically rely on dozens of tools for the deployment of software and the management of the network connections on which the software runs, NS1 was created to—among other things—reduce the number of different tools DevOps teams require by improving visibility, automation and management for these processes.

In this way, NS1 Connect provides a unified API framework to automate DNS, traffic optimization and network services management across heterogeneous networks, Sanjay Ramnath, vice president of products at NS1, told devopsy.kinsta.cloud. NS1 Connect’s reach covers internal and external applications deployed via on-premises, cloud or hybrid infrastructure. This enables NetOps and DevOps teams to integrate NS1 solutions into their infrastructure and CI/CD pipelines with lower complexity and enables NetOps teams to accelerate management and minimize the risk of misconfiguration, Ramnath said. “Prior to this, teams had to deal with complex manual processes or disparate API frameworks that slowed down their ability to develop, deploy and deliver applications.”

In his talk, Your Business Depends on Eliminating Boundaries Between Applications, Users, Infrastructure, and Data during INS1GHTS 2021, NS1’s annual user conference, Ramnath described how a number of NS1 products and services are integrated into a single platform under the NS1 Connect umbrella. They include cloud-native network services for DNS, DHCP and IPAM (DDI), edge-to-cloud networking for NS1’s managed DNS, dedicated DNS, private DNS and DNS for China and application traffic automation for VPN traffic steering, pulsar active traffic steering and general traffic management. Other commonly used network services are provided as well, including IP address management that can be centralized across the network, said Ramnath.

The idea was to remove many of the roadblocks DevOps teams face when seeking to deploy on highly distributed infrastructures. “The goal here is to remove some of those boundaries and barriers that are talked about on the infrastructure side,” says Ramnath. “[This helps to make sure] that our control plane is centrally provided and centrally consumable as a cloud solution and that our products can be deployed wherever they are needed in a zero-touch way, in a lightweight fashion, making sure that we are aligned with cloud-native operations.”

The need to maintain low latency across distributed infrastructures is also key. As David Coffey explained during the conference in The Future of Application Traffic Intelligence and Automation session, low latency is essential for creating “that great experience for users.”

“Low latency is not an attribute of this feature, it is a top, tier-one priority for people to accomplish, but it’s an incredibly complicated aspect, given all of the frameworks that are out there—the tools, the DevOps capabilities and the components required to support and deploy these modern-day applications,” said Coffey. “That’s what enables application companies to deliver that low latency and treat it as an attribute in the system; not having to go out of their way to create this. We want to enable all developers to embed this into their system—and we can provide them with a degree of success for delivering that great experience to their customers and to their networks.”

July 19, 2021
Rocky Linux Emerges as a CentOS Replacement

Rocky Linux was created to offer the community an alternative following Red Hat’s recent decision to shift its focus away from CentOS—the open source version of Red Hat Enterprise Linux (RHEL). However, the Linux distro’s popularly can also be attributed to how its creator sought to offer a superior alternative based on lessons learned during CentOS’ development—and less as a perceived backlash to Red Hat’s controversial move, said Gregory Kurtzer, Rocky Linux’s creator, who is also founder and CEO of Ctrl IQ.

“As one can probably imagine, there were a lot of lessons learned from the founding of CentOS—we did a lot of things correctly, but we made some mistakes, as well, that set into motion some of the difficulties the project went through,” said Kurtzer. “I was not—nor am I now—upset at Red Hat or anyone else. It’s important to understand that companies have a legal responsibility to do what they need to do. But I will say, this is why open source communities should not be controlled by a single company, especially if that company has a competing product to the open source project.”

In addition to applying experience gained from the CentOS project to help “recreate the initial goals of CentOS in a new project,” Kurtzer said he was prompted to create Rocky Linux following Red Hat’s decision to end-of-life (EOL) CentOS. The move “affected my company, our customers, as well as the entire industry,” said Kurtzer.

“To be blunt, my knowledge—in terms of doing it myself is severely dated. Luckily, after I made the announcement that I was considering recreating CentOS, thousands of people reached out to me wanting to be part of the project and contribute,” said Kurtzer. “From that, our development team and open source community was born.”

Additionally, CentOS Stream offers value to the community by allowing “us and others” to take part in developing the upstream Enterprise Linux standard, said Kurtzer, who added that his team will continue to debug Rocky Linux so that solutions are properly integrated into the Enterprise Linux Standard.

“The necessity was absolutely clear, so I founded Rocky Linux and the Rocky Enterprise Software Foundation (RESF),” Kurtzer explained. “Together with our contributors, sponsors and partners, we all have united for a common goal: to create an open source community that will provide us all with an enterprise operating system to stand the test of time, attrition and commercial agendas.”

Interest in the development of a CentOS alternative was strong when Kurtzer first floated the idea of the project to the community. Since its release in June, the total number of ISO downloads is 56,000, according to Rocky Linux Tier0 mirror data.

“We have zero visibility as to any other ISO downloads from any of our other mirrors, but if I were to guess, I’d say it would be about two to three times more, in total,” said Kurtzer. “When we all get a moment to breathe, it will be fun to see how many people are using Rocky Linux, and take in all the feedback the community has provided,” he said.

With this team of developers, Kurtzer said it took about seven months to build the general availability (GA) release. “We didn’t just jump right into building the operating system because the goal was to create an infrastructure where many contributors and developers can take part without putting the security of the product at risk. Building that infrastructure took about four months,” Kurtzer said. “Once the infrastructure was completed, we spent about two months building the first-release beta candidate and then spent the following month ensuring that we did everything correctly.”

July 13, 2021
China’s Open Source Activity Surged in 2020

Activity among China-based open source project contributors and creators surged in 2020, as Alibaba, Huawei, Tencent and other leading Chinese tech companies are playing an increasingly important role in the open source development community worldwide.

According to the recently released Open Source Contributor Index (OSCI) ranking, as of March 2021, Alibaba, Huawei, and Tencent collectively ranked in the top 20 worldwide for the first time. Rankings are based on the total number of contributors and contributions on GitHub. Microsoft, Google and Red Hat ranked first, second and third, respectively.

Alibaba, as well as China-based Pingcap, came within striking distance of reaching the top 20 worldwide open source ranking for 2020 measured in activity; the total number of forks, commits, stars, pull requests, issue comments and other metrics, according to the GitHub 2020 Digital Insight Report released in April.

Authored by X-lab in conjunction with East China Normal University, Kaiyuanshe, Shanghai Open Source Information Technology Association and other unnamed “scientific research institutions and open source community institutes,” the X-lab report revealed Pingcap’s eponymous Pingcap (a distributed relational database) and Alibaba’s AntDesign (a Microsoft React framework) activity metrics of 210 and 193, respectively, in 2020. Meanwhile, jwasham (an open source study plan detailing how to become a software engineer) ranked 20th worldwide with an activity metric of 234.

Open Elastic’s Kibana, recently forked by Amazon Web Services (AWS) and Kubernetes ranked first and second worldwide with activity metrics of 698 and 602, respectively.

Torsten Volk, an analyst for Enterprise Management Associates (EMA), says the increased popularity of and activity in China-based open source projects are proof that the concept behind open source software actually works.

“This is the classic ‘one plus one adds up to more than only two’ theory, and that applies to most companies anywhere in the world. And if it was possibly to gain advantages by only using open source code but not contributing, companies anywhere else in the world would have done so right from the start,” says Volk. “Remember that the core idea behind open source software is not to give out your secret sauce for other companies to enjoy for free, but to spend the least possible amount of funding on building commodity code.”

What, if anything, China-backed open source projects means for the community worldwide — as well as China’s influence in the industry in general – remain to be seen. Tencent, which entered the top 20 in March ranked by contributors in the OSCI study, for example, has become a leading supplier of components for machine learning (ML) databases in, for example, Kubernetes and cloud-native environments.

“This is great for everyone, and there is no benefit for anyone to purposefully not contribute to a project they benefit from. The benefit comes from your own contributions, in terms of reputation and credibility gain, across your industry. This motivates others to also contribute, as there is little to no benefit in keeping your own code secret while self-serving from the GitHub buffet,” says Volk. “As in the case of Kubernetes, your own company’s differentiation needs to come from the manner your proprietary apps [integrate] on top of this platform, or from the way you are able to integrate Kubernetes clusters with machine learning databases and other elements higher up the stack. And that’s why open source works.”

May 5, 2021
Solving Cloud-Native Challenges in the Rush to the Cloud
The shift to cloud-native environments away from traditional data center infrastructures continues unabated, but security and complexity challenges remain a struggle for DevOps teams.

These were two of the main takeaways in a survey and analyst report published by the IT consultancy firm Flexera.

The survey, based on the responses of 750 IT professionals, revealed that 92% of the organizations represented in the survey have a multi-cloud strategy, 80% have a hybrid cloud strategy and only a minority (45%) have integrated data between cloud environments.

The top challenges, according to the survey results, are:
- security (81%)
- managing cloud spending (79%)
- governance, lack of resources/expertise and compliance (75%)
- comprehending cost implications of software licenses (55%).
The solution for these challenges largely involves — in addition to implementing the right DevOps culture and practices — adoption of the right tools and platforms, especially for security. However, first-generation cloud-cost management tools “simply are not cutting it” for modern architectures, said Asim Razzaq, CEO of Yotascale, which offers a cloud resource optimization platform.

“Today’s modern architectures are dynamic, and ownership is transient and not always well-defined. The greatest strength of containers and Kubernetes is the ability to scale on-demand and share cloud resources among multiple applications,” said Razzaq. “Tracking cloud costs in modern architecture requires a strong ability to automate cost attribution — or tag management — for those dynamic resources, and the ability to understand usage at the application level, even for containers that are sharing resource instances.”

First-generation cloud management tools are, in this way, unable to provide the information DevOps teams require to understand how to automate cost attribution for modern architectures, said Razzaq.

“[DevOps teams] can’t break down costs into any level of meaningful detail for engineering and finance teams to digest and act upon,” said Razzaq. “Conversely, newer approaches that are focused on Kubernetes cost management lack the ability to understand and break down the costs of non-cloud-native workloads.”

Indeed, tools are often frustrating to engineering teams because of their static custom reporting, broad and generalized recommendations and no effective workflow for effective collaboration between finance and engineering, Razzaq said.

“The biggest source of frustration is the inability to drill down and manage costs in a way that matches the way that these teams do business. There may be 10 teams, each with five production applications running in the cloud, for a particular account,” Razzaq said. “Engineers need to see, at the application level, what their resource utilization, costs and cost anomalies are in order to make the best cost/performance decisions for their applications. Without that, they’re flying blind; having to defend their cloud usage without any supporting data.”

As an example, a misconfigured Kubernetes cluster can “cause a massive spike in costs,” Razzaq explained. “Without detailed cost and utilization reporting at the team and application level, an engineer can spend weeks digging through spreadsheets and logs trying to track down the source of the anomaly. That’s weeks of their time that they aren’t [using to] be productive on new features.”

Razzaq said there are a number of solutions DevOps teams can adopt to address these issues, including automated cost attribution and tag management, actionable recommendations and comprehensive observability capabilities.

Automated cost attribution and tag management “is the foundation of accurate and detailed cloud cost management,” said Razzaq. “Engineers need tools to help them set tag management policies, identify untagged instances and update tags quickly and easily in their cloud environment,” he said. “With this in place, cloud-native workloads can be automatically tagged as they are dynamically created, ensuring that cloud costs are always clearly attributed to the right teams and applications.”

Actionable recommendations at the cloud-instance level with closed-loop feedback also are an important process “so that engineers can see exactly why the recommendation is being made, and what the cost impact will be of taking that action,” Razzaq said. “The ability to provide feedback on the recommendation and dismiss recommendations that don’t align with business needs ensures that new recommendations are relevant,” Razzaq added.

Finally, tools that provide a unified view of both cloud-native and traditional workloads that have been shifted to the cloud help to ensure “that engineering and finance have a single source of truth for their cloud costs that enables them to make smarter decisions for their business,” Razzaq said.
April 19, 2021
Red Hat Survey Reveals More Confidence in Open Source

A recent Red Hat survey revealed how DevOps teams increasingly depend on open source to support their shift to cloud-native environments and digital transformations, and to maintain security in these highly distributed environments.

Based on a survey of 1,250 IT professionals, Red Hat maintains that the survey respondents — from organizations that were Red Hat customers — were unaware that Red Hat sponsored the study to help avoid “biased or influenced responses,” Red Hat said in a statement.

During the past two years, the number of organizations that deemed open source necessary for their digital transformations increased 11% to 53%, representing the majority of the respondents’ organizations.

The vast majority (87%) of the respondents considered open source to be “more secure” than or “as secure” as proprietary software, while 84% indicated that enterprise open source “is a key part of my organization’s security strategy.”

While the survey reflects open source’s importance for security for IT professionals in DevOps, its support of the overall business mission is key, as well, says Momodou Jaiteh, application security consultant at nVisium, an application security provider.

“Information security is about reducing and mitigating risk to enterprises, from technology, people and process perspectives. It helps enable the business to operate securely in the best interest of its shareholders and customers,” Jaiteh said. “Thus, security typically plays a supporting, [but] nonetheless critical role in the success of the business, especially in a digitally connected world.”

Open source software often helps businesses reduce overhead cost in delivering services to their customers through reduced time to market, Jaiteh said. One major additional advantage is “visibility and input from a lot of different people,” Jaiteh added.

“Greater visibility often translates to more observing eyes on all aspects of the open source software, which often translates to better security. This is even better for mission-critical software that deals with sensitive information, such as cryptographic algorithms, etc.,” Jaiteh said. “It is not a surprise to see open source used more in infrastructure modernization, because as we move to DevSecOps models of software delivery, infrastructure often is the bottleneck. Therefore, solving for infrastructure helps developers focus on aspects that are unique to their business.”

While survey respondents ranked improving collaboration between IT and business stakeholders to achieve a digital transformation as their highest priority, implementing cloud technologies or infrastructure ranked second. Infrastructure modernization is thus seen as a prerequisite for a full digital transformation, said Dirk Schrader, global vice president, security research at New Net Technologies (NNT), a cybersecurity and compliance software provider.

“Open source plays a vital, pivotal role in almost all digital trends driving any industry sector today,” Schrader said. “Cloud computing or containerization is rooted in open source technology, many development libraries are open source, and the market for open source solutions has been a model for servitization; the changes in one’s business model away from selling a product to selling a service, or an integration of service, and product that focus on additional value-add for the customer.”

For security, the main advantages consist of being able to review, inspect and audit “each and every piece of code,” Schrader said.

“The ability of open source to be a base for services makes it the natural companion for any digital initiative,” Schrader said.

April 1, 2021
Python Remains Popular in Programming Language Study

A ranking by analyst firm RedMonk reflects on Python’s continued popularity as a leading enterprise programming language for a number of reasons. According to a recent study by RedMonk, JavaScript also remains widely popular as the web development programming language of choice for a significant number organizations.

In many ways, Python’s strong showing in the ranking — drawn from GitHub pull requests and Stack Overflow data — indicates its long-term viability among DevOps developers.

“One lack of change that is worth surfacing is Python’s number-two spot,” Stephen O’Grady, a RedMonk co-founder and analyst writes in The RedMonk Programming Language Rankings: January 2021. “Java was extremely hot on Python’s heels – and was, in fact, closer to the number-one ranking than to PHP behind it – but Python’s ability to defend its new high ranking is notable.”

Python’s prolonged high ranking is notable because of its versatility, Rachel Stephens, a RedMonk analyst who is responsible for the RedMonk Language Rankings’ design, told devopsy.kinsta.cloud. “The breadth of use cases — and associated libraries that support those use cases — means there are a number of possible entry points for developers. The number of toeholds for the language within the enterprise helps its ubiquity,” said Stephens.

There also remains much discussion about what accounts for Python’s continued popularity beyond the scope of the study. A popular subject in forums, including Reddit, Python is typically described as highly versatile, relatively easy-to-learn and a simple, yet elegant language. While not cited in the RedMonk study, some software engineers and coders — such as this writer — appreciate how Python can make more intuitive sense, and is also less rigid compared to, say, a classic language such as C++. While JavaScript is largely used for web apps and scripts, of course, Python could arguably be the most popular language outside of the sphere of web development, thus accounting for its second-highest ranking in the study.

As such, “due to the accretive nature of our methodology, movement within the top-ranked languages is an infrequent occurrence,” Stephens said.

“[However], Python overtaking and sustaining its ranking over Java was a big deal. That said, we remain strong believers in the future of Java and the Java ecosystem, and we know there is a vast amount of enterprise Java that isn’t captured in the metrics we use,” Stephens said. “We don’t view Python’s ascendence as a zero-sum gain over the use of Java.”

For JavaScript, “in spite of all of the competition from up-and-coming languages, all the discussion of fragmentation – and even criticisms of JavaScript the language itself – it remains remarkably popular,” O’Grady said.

“Simply put, JavaScript remains – its detractors notwithstanding – a force of nature like no other within the industry, and there are no indications in the data that this is likely to change any time soon,” O’Grady added.

Meanwhile, in the book Hardcore Programming for Mechanical Engineers, published by No Starch Press, JavaScript trails Python in other rankings, while Python is also more popular than JavaScript for other reasons than those described in the RedMonk study. Python, for example, is cited as the ‘The Most Wanted Language,’ based on a Stack Overflow survey.

While Python is not the fastest language, as measured in benchmarks listed in Hardcore Programming for Mechanical Engineers — Go consistently outperforms it, for example, in execution times — Python’s popularity remains very high overall, according to the book’s author.

“These results are not surprising; Python is an extremely versatile and productive language,” author Ángel Sola Orbaiceta wrote. “Writing code in Python is a delight, and its standard library is well-equipped: for just about anything you want to do, Python has something ready to help.”

March 29, 2021
Cassandra Update Boosts Recovery Speeds

Apache Cassandra is a row-oriented NoSQL database created by developers working at Facebook. With elasticity as one of its key features, open source contributors have added new streaming capabilities to Cassandra 4.0 for its beta launch. These improvements have been shown to enable the database to stream data between nodes at speeds up to five times faster than in the past.

What this means is that if a node on a network crashes, for example, the data can be transferred that much more quickly to another replacement node. Conversely, during peak load data times, different nodes on the cloud or on-premises servers can more readily and quickly manage potential data overflows for improved operations elasticity, Cassandra’s contributors say.

Elasticity—or lack of—was often seen as a weak point in previous Cassandra releases. Indeed, elasticity improvements, geared for DevOps taking advantage of distributed microservices environments are “really a key point,” said Josh McKenzie, a Cassandra committer and member of the project management committee. McKenzie is also the lead of open source strategy and vice president of software engineering for DataStax, which has historically funded the bulk of Cassandra’s contributions and support.

The new streaming capability also could potentially serve as a welcome feature for database management, especially in highly distributed stateless environments of Kubernetes and microservices, said Clive Longbottom, an analyst for Longbottom Associates.

“On the first one, this seems like an excellent update. It makes the ‘eventual consistency’ of Cassandra less of an issue, and the more guaranteed timescales of recovery across a range of scenarios is welcome,” Longbottom said. “I would hope that such a major change will get more to consider Cassandra for such data work—it can be, when implemented correctly and the right jobs put on it, an effective data platform.”

The new streaming speed is made possible because the data is able to bypass the use of the servers’ CPUs, McKenzie said. “Being able to get anywhere from four to five times the throughput when bootstrapping a new node on recovering from failure or scaling clusters up and down means the process is now a totally different ballgame.”

An example of a six-node cluster setup.

For DevOps teams, this means Cassandra’s modifications—including its now less complex and shorter-in-size code configurations—make for faster streaming speeds when bootstrapping a new node or during disaster data recovery. “Running repair and dealing with [server failures] means you don’t have to rebuild data so much and you can now run at line speeds as fast as your network,” McKenzie said.

The developers also sought to reduce the number of hours spent on performing what used to be an especially onerous task of transferring data to and from different database sources in a network. Multi-cloud and mixing and matching legacy and cloud environments further complicated the process.

In other database platforms, the process of doubling database capacity can take five to seven months. Building in part on steady improvements past releases have offered, Cassandra 4.0 now can complete the process in just a few hours, McKenzie said. “I love hearing about how people can now run Cassandra with a script and it just takes maybe six hours on a Saturday morning with one person and one with one script instead of a whole weekend,” McKenzie said.”So it’s just really doubling down on those unique strengths that make Cassandra such a good fit for exactly where we are in the world today: in the state of always scrambling with massive influxes of data.”

The committee members also sought to extend the timeline of the final beta release. Instead of adding more incremental improvements while attempting to accelerate the release cycle, Cassandra’s 4.0 contributors sought “to really take our time, making sure that we were hammering down and smoothing out all the dents,” McKenzie said. “We wanted to get things to where we have a really really stable platform to iterate on very rapidly.”

July 27, 2020
The EU’s Plans to Build a Wall Around GDPR-Protected Data

An ambitious German- and French-lead European Union (EU) initiative could wrestle tighter management, and eventually, control of data from commercial cloud providers.

The project, dubbed as “Gaia-X,” is also intended to offer firewall-like protection for data sources from the EU. The initiative could have repercussions on how DevOps teams from outside of the EU manage data from European users, as well as from operations they might have that are physically based in EU member states.

Without specifically citing U.S. cloud giants such as Amazon Web Services (AWS), Google Cloud Platform (GCP) or Microsoft Azure, German and French government officials said the initiative will allow organizations’ cloud data to benefit from EU sovereignty and security. Bruno Le Maire, France’s minister of the economy and finance (Ministre de l’Économie et des Finances), said during a televised Franco-German conference last week, for example, that Gaia-X would offer significantly more data protection for data stored on cloud services from EU organizations and users.

Despite how General Data Protection Regulation (GDPR) mandates rules EU and international firms with user data from EU member state sources must follow—to avoid the risk of heavy fines and other penalties—the EU still lacks sovereignty in the application of GDPR, Le Maire said. Data stored on a U.S. cloud service provider’s server in the U.S. or overseas, for example, is still accessible by U.S. law enforcement officials in some cases under the Clarifying Lawful Overseas Use of Data (CLOUD) Act.

Without disclosing technical details about the kinds of cloud services Gaia-X might offer, the initiative will adhere to four principles: “openness, interoperability, transparency and trust,” Le Maire said.

“We are not China. We are not the United States—we are European countries with our own values,” Le Maire said. “We have our own economic interests that we want to defend.”

Over 20 organizations will form Gaia-X, including AtosBosch, Deutsche Telekom, SAP and Siemens. A prototype of the service should be available by end of year. While specific cloud services the organization will offer remains undisclosed, government officials announcing Gaia-X said it will offer artificial intelligence (AI)-aided meta analysis of data in such sectors as healthcare and smart city management. The use of AI services for data needs would also meet stringent privacy and ethical programming mandates, the officials said.

In a statement, the German Federal Ministry for Economic Affairs and Energy (Bundesministeriums für Wirtschaft und Energie) said:

“Predominant data silos, a lack of standards and the limited transparency of data infrastructure offers are today major obstacles to the application of artificial intelligence. To overcome these obstacles would be very costly and for this reason, the diverse know-how available in Europe is not pooled and used efficiently.”

Build That Wall

While created to ostensibly protect user data from the European member states and to enforce non-compliance with major penalties, accessibility by governmental authorities outside of the EU, for example, can remain outside of EU control. As mentioned above, for example, data storage that is ostensibly GDPR-compliant yet is on a physical AWS or another cloud provider’s server remains subject to access by U.S. law enforcement in certain circumstances.

GDPR compliance is thus often considered a starting point for EU data sovereignty. In many ways, the EU’s initiative, pending the final details about Gaia-X’s infrastructure and the services it will offer, will serve as a firewall built to protect EU.

Clive Longbottom, an analyst for Quocirca, agreed.

“By creating a total data platform that is GDPR compliant, uses have the choice: build directly on that platform and not have to jump through hoops to prove compliance,” Longbottom said. “Those who choose not to, or due to geographical position, cannot use the platform directly can export data into Gaia-X and only have to prove compliance for that part of the platform outside of the environment. So, yes, EU data can be completely airlocked into Gaia-X – or can just make life a lot easier for users.”

Instead of serving as an alternative to AWS, Gaia-X, based on the information the EU officials have communicated so far, is more of a data handling environment that may just have its own cloud components, Longbottom said.

The idea, based on the available information “is to present an environment where data can be handled firmly within the needs of GDPR,” Longbottom said. “Now, there are fundamental difficulties around this, such as with data caching in the network, but we won’t let inconvenient realities get in the way of political perceptions.”

The Gaia-X is thus “really meant to be a platform for data exchange that meets stringent requirements within the EU,” Longbottom said. “I would hazard that it will include those working outside of the EU that want to deal with countries, companies and entities within the EU, accepting data feeds from pretty much any cloud platform. It will probably be that once the data is accepted into the Gaia-X environment, it will then be deemed to be secure under GDPR—and that will not make it necessarily secure under GDPR end-to-end.”

Cloud Alternative

A main selling point of the project is that once data adheres to Gaia-X compliance, DevOps could, in theory, benefit from more seamless data transfers between parties within the environment, as well as transparency about how and where it is stored.

“Gaia-X is a good idea that will make life easier for EU members to share data across a trusted environment and allow externals to feed data into a known, highly standardized environment where they then do not have to prove GDPR compliance across Gaia-X itself,” Longbottom said.

However, whether or not Gaia-X might one day compete against AWS, GCP or Azure remains to be seen. “If the EU can get more cloud platform vendors to join in and show how their platforms are GDPR compliant from the ground up, more EU companies will want to use such platforms where GDPR compliance is a given,” Longbottom said. “Meanwhile, most will stick with the big public platforms and use Gaia-X as a downstream GDPR data handling environment.”

June 19, 2020
How GitOps Can (Sometimes) Save DevOps

A lot has been written about GitOps since its release by Weaveworks more than a year ago as a “single source of truth for the whole system” for using Git for Kubernetes deployments on the cloud.

Aptly described by Alexis Richardson, Weaveworks co-founder and CEO, as “90% best practices and 10% cool new stuff,” GitOps largely offers a framework for DevOps CI/CD deployments on Kubernetes. This was something very often seen as missing among the many Kubernetes management distributions on offer—hence, its popularity.

The tool is obviously not of use for organizations that have not adopted Kubernetes, but in many ways has enabled organizations’ DevOps to more efficiently take advantage of Kubernetes by running Git in a more disciplined and focused way.

“The idea behind GitOps is to maximize the time developers spend on productive coding versus worrying about working with the latest source code, merging their saved code back into the repository and continuously supporting problems that come up during the application build process,” Torsten Volk, an analyst for Enterprise Management Associations (EMA), said.

GitOps offers DevOps a way to use its version control capabilities to archive changes to application deployments in a way that allows them to be recalled at a later time. This can come in handy when an architecture fails and it is necessary to access and reinstate a previous version of an application deployment or update prior to the incident. All changes are tagged and archived in a way that each developer knows when and how their changes to an application were changed.

“Ideally, each developer would check in their code and receive a link to the running application with his or her latest changes applied, as well as with all the other changes that were checked into the repository,” Volk said.

The Adoption Checklist

Volk offered the following elements he said DevOps should have in place before adopting GitOps:

— All infrastructure and application configurations, including databases, middleware and machine learning models, need to be defined as code, Volk said.

“Everyone must accept the basic concept that code is only complete when all components for a successful build are checked into Git,” he said. “CLI-driven, ad hoc deployments are no longer acceptable.”

— The DevOps group has to be able to detect changes to application code that previously might have been able to bypass Git. There must be thus be a process in place to bring these changes back under GitOps control, Volk said.

“The IT Ops/DevOps teams need to make operational changes to the app exclusively via changes to the code checked into Git,” he said. “This means that tasks such as scaling, failover, creating maintenance windows, etc. as well as urgent troubleshooting tasks have to run through Git. The latter is especially tricky, as the enterprise will not accept longer mean time to detect (MTTD) and mean time to repair (MTTR) due to GitOps.”

— Declarative management practices have to be used in Dev and Ops. “This has proven to be a critical roadblock,” Volk said. “Kubernetes developers, as part of Dev and Ops, still often prefer the control of imperative management over the promise of defining their desired end state and letting the software deal with how to get there.”

For DevOps teams that have the prerequisites in place as described above to use Git to run Kubernetes deployments on the cloud, then GitOps deserves at least a look. However, for those organizations that adopt GitOps for the right reasons, it is not designed as a one-size-fits-all solution and organizations will need to modify it to fit their needs. Its implementation, for example, will almost certainly different compared to how Weaveworks described its functioning after using it for two years to run Kubernetes and Prometheus telemetry databases on a cloud platform.

DevOps will, thus, need to be actively involved in GitOps deployment and maintenance to make sure it meets organizations’ individualized needs. When first describing GitOps, Weaveworks’ Richardson wrote: “Fair warning: this is what works for us, and dear reader, you may disagree.”

— B. Cameron Gain

October 23, 2018