devopsy.kinsta.cloud

Tag: container orchestration

AI-Driven Microservices: The Future of Cloud Scalability

Picture a Black Friday sale pushing e-commerce infrastructure to its absolute limits. Even with cloud auto-scaling configured in container orchestration platforms, throttling begins as the system struggles to keep pace with surging traffic. The scaling mechanisms react only after thresholds are breached, leading to mounting delays. Manual resource allocation cannot keep up, resulting in server throttling, degraded user experiences and potential revenue loss. This scenario, which I have encountered multiple times in my eight years as a software engineer, represents one of the most common challenges in scaling microservices effectively.

Today’s microservices architectures are increasingly complex, demanding more sophisticated scaling strategies than ever before. Traditional methods often fall short when faced with modern enterprise workloads, where traffic patterns can fluctuate unpredictably. I have been paged numerous times due to exhausted throttling limits, requiring manual scaling interventions during off-hours — a painful and inefficient solution to a growing problem.

Human Error

The evolution of microservices scalability tells an interesting story. In its early days, scaling was predominantly manual, requiring engineering teams to constantly monitor and adjust resources during high-demand periods. This approach was not only labor-intensive but also prone to human error. Organizations relying on dedicated on-premises data centers faced particular challenges with systems introducing latency and demanding constant oversight during peak traffic periods.

For smaller companies depending on in-house infrastructure, the absence of robust auto-scaling capabilities presents a critical challenge. These setups typically require significant investment in both hardware and skilled personnel to manage dynamic workloads, often leading to inefficiencies during peak usage. Without sophisticated auto-scaling mechanisms, organizations face a difficult choice: Under-provision and risk performance bottlenecks or over-provision and waste valuable resources.

Limitations

Even with modern auto-scaling in cloud platforms, the limitations are clear. Scaling remains largely reactive, with additional servers spinning up only after demand spikes are detected. This lag leads to temporary throttling and performance degradation. During peak times, over-provisioning results in wasted CPU and server utilization during subsequent low-traffic periods.

The inadequacy of threshold-based auto-scaling becomes particularly apparent during high-traffic events like holiday sales. Engineers often find themselves on-call to handle performance issues manually, adding operational overhead and delaying service recovery. These systems lack predictive capabilities and struggle to optimize cost and performance simultaneously.

AI solutions

AI offers a solution to these challenges. Through my experience with cloud-native platforms, I have seen how AI can transform scaling capabilities by incorporating predictive analytics. Instead of waiting for problems to occur, AI-driven systems can analyze historical patterns, current trends and multiple data points to anticipate resource needs in advance.

This innovation has particular significance for smaller enterprises, enabling them to compete effectively with larger organizations that have traditionally dominated due to superior infrastructure capabilities. By integrating AI models, companies can optimize resource allocation, reduce latency and lower operational costs without requiring extensive infrastructure investments.

AI-Driven Microservices: A Fundamental Shift

Looking ahead, AI-driven microservices represent more than an incremental improvement — they are a fundamental shift in how we approach distributed systems. The future lies in systems that do not just react to demand but anticipate and prepare for it. While unexpected scenarios will always exist in distributed systems, AI-driven scaling significantly reduces their frequency and impact.

The key to success in modern cloud environments is not just about having more resources — it is about using them more intelligently. As we continue pushing the boundaries of what is possible with microservices architecture, the integration of AI into our scaling strategies becomes not just advantageous but essential for building truly resilient, efficient and scalable systems.

January 21, 2025
Extending GitOps to the Enterprise
GitOps is a relatively new term, but it continues to gain momentum as more organizations embrace the paradigm. In DevOps, GitOps lands on the software engineering side of the development and operations continuum. Even though GitOps is more likely to be adopted by smaller organizations, larger enterprises can successfully implement GitOps with just a few tweaks to the overarching framework.

Let’s start at the beginning. Wouldn’t it be great if we could combine DevOps and GitOps approaches and use the Git distributed version control system as the ultimate source of truth? Then, when there’s a dispute over the correct state, people know where to go for the correct version.

In GitOps, the system’s desired configuration is stored in a revision control system such as Git. Any difference between the desired state stored in Git and the system’s actual state indicates to DevOps teams that not all changes have been deployed. These changes can be reviewed and approved through standard revision control processes such as pull requests (PRs). When a PR is approved and merged to the main branch, an operator software process is responsible for changing the system’s current state to the desired state based on the configuration in Git.

GitOps doesn’t require a particular set of tools, but the tools must:
1. Operate according to the desired system state stored in Git.
2. Detect differences between the desired and actual states.
3. Perform required operations on the infrastructure to synchronize the actual and desired states.
In an ideal implementation of GitOps, manual changes to the system are not permitted and all changes to configuration must be made to files stored in Git. The infrastructure and operations engineers’ roles in a GitOps model shift from performing infrastructure changes and application deployments to developing and maintaining GitOps automation and helping teams review and approve changes through Git.

GitOps is great for non-critical environments, but enterprises are less likely to adopt GitOps due to a number of challenges facing GitOps users.

Challenges with GitOps

GitOps is applicable only to a subset of the software development life cycle (SDLC). This is important because GitOps tools are sometimes marketed as a one-size-fits-all solution that will solve all release problems—this is simply not true. First, GitOps requires that your deployment artifacts are already there. This means that tasks such as
- Compiling code
- Running unit/integration tests
- Security scanning
- Static analysis
are not a concern of GitOps tools and are assumed to already be in place. GitOps also doesn’t address promotion of releases between environments.

Auditability

A default GitOps process is not fully auditable because of force push. Force push essentially allows users to remove any unwanted blocks of commits from Git history on the central repository.

Business Approvals

PRs are a typical approval model used in GitOps. Developers make changes, create PRs and then an approver may accept that PR. This would be recorded and a change would be deployed. A PR-based approval model assumes peers are already using Git and are familiar with PRs.

Generally this works great until we need to include business approvals. Business personnel may not be familiar with Git or PRs. If a team needs business approvals, they have to build a process on top of GitOps. This drastically increases lead time.

Governance

GitOps is dependent on a pure PR-approval based model, one in which the PR approver is responsible for reviews. After a PR is approved, there is no way to enforce enterprise policy rules. Enterprises need to control what end users can do on specific clusters to ensure those clusters are in compliance with policies to meet governance, legal requirements or enforce best practices.

An Enterprise Approach to GitOps

However, there are ways to solve this problem at the enterprise level.

Alerting on Drift Detection

Enterprise customers get alerted whenever there is a drift in their repository and, based on that, customers can choose to approve or reject the drift. If rejected, the merged PR should be reverted to the previous state to ensure that Git is in sync with the desired state of the cluster.

Use a Pipeline for Promotion of Releases Between Environments

To promote an artifact between environments, users would have the ability to use a pipeline with deployment stages, performance monitoring and log analysis/approvals.

Let’s say the developer commits the source code and the GitOps deployment happens in the QA environment. The PR can be automatically approved upon meeting the policy standards set by the company. Then it can be pushed to a production environment.

Governance

There are many reasons why developers and platform engineers integrate policy-as-code into their current GitOps processes. For one, this strategy helps to accelerate application development and deployment because it helps solve many of the change management hurdles that slow development pipelines.

On the development side, policy-as-code helps app developers understand and abide by the company’s configuration, security and compliance policies. For example, a developer may not remember—or may have no reason to know—when deploying a load balancer onto Kubernetes in AWS is or is not sanctioned. Policy-as-code solves this problem automatically.

The ability to run policies during GitOps deployments and adhere to those policies as a part of the deployment and upon any violations—alert, audit, notify and automatically roll back the changes from Git to the previous successful compliant state so that Git remains in sync with the cluster.

GitOps + AIOps

There is a notion that once GitOps is in place, everything will work seamlessly with no problems. Not so much; it is important to continually monitor deployment state to ensure everything stays as it should. That is where AIOps comes into picture.

GitOps enables DevOps and development teams to see any manifest change, infrastructure change or artifact change. Post-deployment, the GitOps operator can instantly understand the changes in the service health, see what caused those changes and automate the process of investigating them using AIOps. If any severe anomalies are found, the operator can determine the current state of the system and, should anything differ, revert to the previous compliant Git commit.
December 14, 2021
4 Container Orchestration Security Concerns
Container orchestration is a must-have technical skill, according to 51% of respondents to the 2021 Upskilling survey from DevOps Institute. So, it’s no surprise, then, that container technology is becoming increasingly popular. However, container environments are far more complex and dynamic than other infrastructure, making container security essential. Security ensures containers are running as intended, protecting the infrastructure, software supply chain, runtime and everything in between.

If you’re looking to use container-based applications within your software infrastructure, you may want to keep in mind the security risks that come with this technology. To help, I asked DevOps Institute Ambassadors to weigh in on the security concerns they have with container orchestration. Here’s what they shared:

Anshul Lalit, head of technology and transformation, Kongsberg Digital

“One of the most common security concerns with container orchestration is that it shares the host operating system and the host kernel. Thus, it raises the concern that a host OS or kernel vulnerability could potentially be taken advantage of and exploited in a container. An example of this is the recent attacks on Linux’s mmap syscall. Though you can avoid this by using defined best practices, it’s worth noting that any compromise in the host OS could compromise containers in that host. This scenario is in contrast to a virtual machine (VM), in which the guest OS and the host OS and kernel share no common binaries and are thus inherently separated.

Similarly, a rogue process, a vulnerable container image that is running, and a security vulnerability in the orchestrator can all provide an attacker a means to get their foot in the door. For example, if a container is designed to run with root privileges, an attacker running a vulnerability scanner on the orchestration host will quickly find the vulnerable container.

Users should follow best practices for securing the containers running in a container orchestration platform by providing a Linux security module, for example, that can enforce a policy that will protect against a rogue container or user from extracting sensitive data from the host system.”

Supratip Banerjee, solutions architect, Principal Global Services

“Containers are creating new challenges for cloud security. While these structures are dynamic and automated, new threat vectors are also present for precisely the same reasons containers are useful. DevOps and SecOps teams must be aware and prepared for these possible issues:
- Containers, unlike virtual machines, share the host operating system on which they execute. If settings are not correctly maintained, the host and containers may be exposed to security concerns.
- Orchestration automation adds a degree of complexity which can lead to misconfigurations that overprovision access and, as a result, an expanded attack surface.
- Containers are constructed using pictures kept in public or private repositories; they are frequently reliant on other images, and a single flaw in any of them might spread across thousands of containers.
- Because containers are ephemeral, IP address-based security measures are ineffective and complicate forensic investigations.”
Parveen Arora, co-founder and director, VVnT SeQuor

“If your project(s) involves the use of containers, you will have decomposed your application into its various constituent services, which, while beneficial, isn’t necessarily as secure as if you are using VMs. Since container platforms like Docker borrow code libraries from the host, they’re fundamentally not as secure as VMs. Vulnerabilities in a container environment can therefore allow malicious software to break out of a container and infect the host environment.”

Vishnu Vasudevan, head of product engineering and development, Opsera

“The security challenge, especially when it comes to container orchestration, is people think they can spin up their own image or they can download the image from the cloud and start using it. It’s not as easy as everybody thinks. There are vulnerabilities being introduced every minute. You need to have a security tool and processes that can automatically scan for every container that is being splintered. Furthermore, a team needs to create shared practices around blue-green deployments or rollbacks of the container changes that are being pushed into production.

From the pandemic alone, we have seen significant cloud movement, but on average, 50 vulnerabilities are introduced every day. In fact, there are people just waiting for loopholes where they can execute malicious scripts. And some people, especially people who are in mining, try to gain visibility into vulnerabilities within cloud accounts and penetrate them to run huge containers for their own cloud mining purposes. On average, it can take 280 days to even detect any vulnerabilities or a hack into the system.

Companies using containers must consider threat vulnerability management as part of their orchestration strategy. Do you think the team is experienced enough to scan 50 vulnerabilities a day and fix them? Probably not, since release velocity is shortening a release cycle to one or two weeks for modern application initiatives. This creates a lot of technical debt that can accumulate rapidly, which is why threat vulnerability tactics and processes must be considered when moving to the cloud.”

Learn more about container orchestration and similar topics by registering for an upcoming SKILup Day.
October 7, 2021
Catch Emerging Security Risks Earlier by Leveraging Kubernetes Audit Logs
When a Kubernetes deployment suffers a security breach, it can be difficult for security teams to diagnose and identify the source of the intrusion and its impact. Each Kubernetes cluster may host multiple applications, and each application may be composed of dozens or even hundreds of interacting microservices deployed as pods.

The complex and dynamic communication between the applications’ components, as well as the automated Kubernetes orchestration activities, makes it difficult to understand the structure of each application and differentiate between normal behavior and malicious behavior from the observed network activity. The ephemeral nature of pods, which may exist only for moments to perform an activity before they are disposed of, makes this task even more difficult. A pod that was vulnerable may be gone by the time security teams are investigating a breach, or resume normal operation as the attacker pauses its abuse or pivots or escalates the attack to another Kubernetes resource.

K8s is by nature dynamic, flexible and scalable, and it is this same advantage for cloud native application developers that becomes a challenge for security professionals. How can one understand the source of a breach, let alone how it propagated through the system, without deep understanding of the architecture of the application?

One solution to the problem of tracing breaches is to examine the Kubernetes audit logs since they record all cluster-administration activity, even for pods that may have existed for seconds and are now gone. But K8s audit logs are complex and often convoluted, requiring deep expertise to understand. Therefore, security teams need to not only increase their proficiency with K8s audit logs, but also to apply automation and software tools to the K8s logs to detect suspicious behavior after the fact and even during their occurrence.

In fact, there are many potential threats that can be discovered when observing audit logs for suspicious behavior. They include:
- Stolen or misused credentials, enabling hackers to gain access to K8s-based clusters or pods.
- Misconfigured Rules Based Access Control (RBAC), enabling lateral attack propagation, privilege escalation and unauthorized data access or manipulation.
- Exploitation of vulnerabilities in the Kubernetes API Server, enabling bypassing of authentication, authorization, admission control or validation of cluster administration requests. This lets users gain access to privileged and sensitive resources.
- Violations of security policies diverge from compliance requirements and best practices.
Detect Breaches in Near Real Time by Watching K8s Audit Logs

There may be a vast time span between an initial breach and the detection of the compromised application or the unauthorized access of secure data and private information. Security teams must seek to close the gap between a breach attempt and its detection as much as possible, while still being able to diagnose the causes of a breach after it has been detected and assess its scope and impact.

Since K8s deployments can be vast and intricate, machine learning and artificial intelligence are preferred tools to monitor audit logs and flag suspicious behavior. An automated tool could continuously scan logs to learn what normal behavior looks like in the company’s clusters, and then flag network behavior that is unusual or suspicious for security teams to qualify, prioritize and investigate. This methodology would enable security teams to investigate possible security violations in near real time to protect and secure the applications.

In addition to detecting security risks, security tools that leverage K8s audit logs should provide reporting that is easy to understand and consume by security teams. By providing the correct user experience, companies can make their K8s logs accessible to more than the hard-core experts, effectively opening a new vista of security possibilities for more security professionals. The difficult learning curve of Kubernetes will no longer be a wall between K8s pros and newcomers with less experience, relieving pressure from understaffed security teams.

Step Up to Security and Compliance Needs

As Kubernetes moves toward the mainstream, with roughly 71% of Fortune 100 companies using Kubernetes as their main container orchestration tool, companies must also look toward compliance with standards and government regulations designed to protect financial transactions and private or personal information such as PCI, GDPR and HIPPA. The penalties for violating these standards and regulations can be severe, so companies should proactively adopt K8s audit log monitoring as a way to identify breaches earlier, to limit their impact on customers and data, and to document the behavior of the application and would-be hackers for possible legal inquiries. An effective audit log strategy will be vital in determining what happened in the case of a breach.

Leverage Audit Logs for Real-Time Security Observability in K8s

Many tools exist to secure the perimeter of the data center and to identify possible security risks during application development. The benefits of analyzing and leveraging Kubernetes audit logs at runtime have been relatively overlooked. K8s audit logs are a prime way to comprehend the behavior of any cloud-native application orchestrated by Kubernetes, as well as the best way to comprehend how the Kubernetes infrastructure itself can be abused by an attacker to compromise the applications it is hosting. As such, security teams have a responsibility to deploy tools and gain a proactive, real-time view into K8s audit logs in order to minimize the time between breach and remediation and to validate compliance conformance at runtime. Such a thoughtful approach to Kubernetes security will go a long way to protect your deployments from harm.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or Aug. 2020.

— Nitzan Niv
March 13, 2020
Building Hybrid and Multi-Cloud Architectures for Analytics and AI
We are in the early stage of the data revolution. Today’s conventional wisdom states that hybrid latency prevents you from running analytic and machine learning workloads in the cloud with the data on-prem. As a result, most companies either do not burst their workloads into the cloud or have to copy their data into a cloud environment and maintain that duplicate data—and even this can take days or weeks.

Counter to conventional wisdom, you can create a high performance hybrid/multi-cloud data platform for analytics and machine learning by using data orchestration technology. Container orchestration technology (e.g. Kubernetes) enables containers to run in any environment agnostic to the hardware that is running the application, similarly data orchestration technology enables active working sets and data to be orchestrated on demand. This enables enterprises to create a compute agnostic, storage agnostic and cloud agnostic stack to serve all their data driven workloads, including analytics (e.g. Apache Spark, Presto, Apache Hive) and AI (e.g. Tensorflow, PyTorch).

An enterprise hybrid cloud analytics strategy should burst compute that spans on-prem and cloud data stores. By bringing the hot data to the analytics and machine learning applications, the performance is the same as having the data co-located in the cloud. Also, the on-prem data stores will have offloaded the computation and minimized the additional I/O overhead.

Specifically, a data orchestration system should:
- Abstract data access across storage systems. Today, data is siloed in various storage systems and deployments. For example, lots of on-premise data is stored in systems such as Apache HDFS, Ceph, EMC Isilon and IBM Cleversafe, and data in the cloud is stored in systems such as AWS S3, GCP Store, Alibaba OSS, etc. Historically, there have been many attempts to unify all data into a single storage deployment with limited success. As a result, we’ve seen more of a trend around embracing these data silos by abstracting the data in various storage systems/deployments. To do this, the data orchestration system needs to abstract and virtualize across various storage deployments/deployments by server side API translation, which enables users to run existing applications without modifying any code. The system also needs to provide a single global namespace to enable apps and users to access all the data in their environments without the need to understand which storage the data is stored in.
- Bring data locality in a separated compute and storage environment. While cloud vendors and leading companies in data-heavy industries have adopted a decoupled compute and storage architecture, it can still be very challenging to implement this type of architecture, especially when it comes to bursting on-premises data into the public cloud. A data orchestration system resolves this challenge by providing caching functionality that enables the system to automatically orchestrate the hot data to be closer to compute while at the same time keeping cold data in a cheap storage. Because each workload may have a different usage pattern, the system must have a policy that enables customized policies best suited for those specific workloads. Moving forward, it is also very critical to have policies across various clouds to enable a multi-cloud environment.
- Provide security for data in motion and at rest with secure access methods. A data orchestration system plays a critical role in the success of an organization’s data strategy. While open source software is important and can help in avoiding potential vendor lock-in, it is also critical to provide necessary security features in this environment given the system touches a lot of data in the organization.
Besides empowering data driven application developers, a data orchestration system also brings value to data and infrastructure engineers. It de-risks vendor lock-in by providing organizations with flexibility at the infrastructure level. Transitioning to different storage systems (including cloud storage), adopting another application framework, or even having a hybrid or multi-cloud environment are all possible without the need for many different technologies or incurring large development costs.

— Haoyuan Li
October 17, 2019