devopsy.kinsta.cloud

Author: Gadi Naor

Measuring Success in DevOps: Top Four DevOps KPIs

There is no news in DevOps concept. Many companies have integrated DevOps to improve and accelerate software development and help drive their digital transformation, and that number continues to grow exponentially. There are now entire tool ecosystems, methodologies and transformation models—as well as endless resources—available to guide companies along the DevOps journey.

But DevOps success can be difficult to measure. DevOps isn’t a formal framework; it’s more of a culture and a set of practices, and there is limited guidance available to ensure you’re doing it properly or accurately measuring your successes and failures. DevOps also looks different in every organization—no two DevOps shops will be the same.

However, despite the nebulous definition of what a DevOps organization looks like, there are some core key performance indicators (KPIs) that should be common to all DevOps environments: asset management, monitoring, continuous integration/continuous deployment and continuous security.

Asset Management

In this context, asset management means measuring what is automated and how effectively it is controlled. It doesn’t matter if you are using tools such as Puppet, Chef, Ansible or Terraform; what matters is how effectively you use them. These tools allow you to automatically provision bare-metal or cloud assets, as well as join a load balancer and receive traffic. However, you can achieve similar results from Kubernetes or chaining lambda functions. Automation is really about controlling assets to achieve your business goals.

Whether you’re using on-premises virtualization or cloud workloads, taking the time to streamline your processing cycles down to the minimum hardware required can significantly reduce monthly costs. The higher the percent of your servers, containers, services, etc. are automated, the less time you’ll spend fixing and maintaining your infrastructure. Some of your applications may be uniquely configured to your particular organization, and therefore cannot be automated. However, if applications cannot be automated, it might be time to retire them and build next-generation applications.

Monitoring

Should you be monitoring all of your resources? Yes. Should you also filter out the noise? Absolutely. Monitoring agents should be deployed to all of your servers, and if your PaaS tools and other providers have integrations, you should be using them and parsing out unneeded data. The only thing worse than not monitoring anything is monitoring everything without proper filtering.

Good monitoring should always be coupled with utilizing the information acquired. Capturing logs, parsing them and using tools such as ELK, Datadog or Sumo Logic, for example, give you a large kit that can not only capture and glean useful information from a tremendous amount of data, but can also make decisions and perform predictive analyses. You might learn your stack is operating inefficiently, or not scaling fast enough during peak utilization periods.

Disparate systems should be aggregated into a common tool, lest you spend valuable time combining information across systems or pulling information from data warehouses to get your insight. Let the tools do that work for you. You should also be using tools such as Logstash to pull and understand application, database and other logs in your stack for meaningful data. If not, you’re limiting your insight into the service’s health.

Continuous Integration/Continuous Deployment

Continuous integration and continuous deployment (CI/CD) is the difference between releasing software once a quarter (or year) and performing dozens of releases to production within a day.

But it’s one of the most difficult indicators to measure, since every environment is different. It is important to understand how many code commits don’t make it past your integration environment. These blockages could be the result of poor programming practices, poor code quality or poor testing. However, if features stay in dev or integration for too long, there is surely an underlying issue. In most cases, it centers around automated testing.

So, how do you measure good testing? This is one of the most difficult questions enterprises face, especially when automated testing is added after monolithic applications are already written. Essentially, good testing is about creating good business logic and integrating testing that meets most use cases, while covering each function and class. Immediate feedback on failed or successful tests expedites the automation process. The less automated testing is built, the more QA has to test by hand. Failing to integrate good business and integration testing slows down the entire CD part of the pipeline significantly. Also remember shooting for 100% test coverage is unrealistic—over 80% should be sufficient.

It’s also important to remember applications that are not in a pipeline and must be manually tested will never keep up with the pace of cloud-born innovation. If you are dealing with a giant, monolithic legacy app, it’s time to design its replacement (starting with testing). Pipelines should also have notifications built into them, with chaining logic to react to failures. Tools such as Jenkins, CircleCI, TravisCI and most code repository solutions, such as GitLab, offer rich feature sets to aid in pipeline creation and management.

Continuous Security

The easiest way to ensure security compliance is to shift left and resolve security issues in the development stage. Too often, security is applied at production, which means it’s not part of the environment’s end-to-end process. Securing applications and networks at the development level will give you more confidence that applications will interoperate properly at the production level.

After shifting left, make sure you continuously deploy and monitor your clusters, nodes and pods in a secure manner. Ideally, you have a tool that provides a real-time summary of your cluster’s compliance and security status by looking at workload security and governance checks, cluster worker node CIS Kubernetes Benchmark checks, cluster Ingress controller security best practices and istio security checks.

DevOps Success Never Ends

Success in measuring your core DevOps KPIs is not about getting to full automation immediately—it’s about understanding your biggest issues, continuously securing them and improving them over time. DevOps has no end; success requires continuously measuring your progress and improving your environment with the most effective and up-to-date tools. Take control, implement thorough and meaningful monitoring, deliver with confidence and do it securely.

— Gadi Naor

August 21, 2019
The Hybrid Cloud and Why DevOps Teams Need Unified Stack Observability

The modern data center is constantly evolving to keep up with the demands of always-on, anywhere computing. Long gone are the days of monolithic, on-premises data centers. Today, the computing modus operandi is that of the hybrid cloud, which, according to Statista, will be worth close to $50 billion in 2018—just about double what it was in 2014—and will grow to almost $92 billion by 2021.

As more and more DevOps and security teams are integrating hybrid cloud environments and serverless architectures, they need to gain visibility and insight into the increasingly distributed workloads provisioning and supporting their applications. Without a single pane of glass across their hybrid clouds, Dev, Sec and Ops teams cannot quickly discover vulnerabilities or identify and respond to attacks.

The emerging best practice to the cross-cloud visibility conundrum is unified stack observability. Unified observability is the ability for Dev, Sec and Ops to share the same application-aware monitoring environment for vulnerabilities, threats or attacks across all workloads supporting their applications. The idea behind unified observability is that each team can look at the different stack layers as one—with an app-aware, single-pane, top-down view. For Dev teams, unified observability means that they can see and define all the third-party services connected to their applications; for Ops, it means they can see and monitor all workloads. And for Sec teams, it means the ability to set security policies for detecting and blocking attacks at each layer of the app stack by enforcing policies uniformly. There are a few recent trends accelerating the need for unified stack observability.

The Accelerated Need for Unified Stack Observability

Recent computing and application development trends are changing the way Dev, Sec and Ops teams develop and manage their distributed applications across hybrid cloud infrastructures. These changes include the increased use of third-party services, an overabundance of purpose-built security tools and the increasing need to address the shared cloud responsibility model, among others.

An Uptick in Third-Party Code Use

Third-party code increasingly is being used in apps and is one of the critical attack vectors exploited by hackers. SafeCode research notes that 78 percent of companies are running part or all of its operations on open source software, while 66 percent create software for customers built with open source. These stats have nearly doubled since 2010. The very practice of code-lifting, or pulling open source code from different repositories, which has enabled developers to share innovation and collaborate across architectures and levels, is turning to be one of the biggest attack vectors for hackers to exploit. Running on many different functions, open source applications and code borrowed from third-party services increasingly are becoming more vulnerable and therefore require a need for constant observability and monitoring.

Even though such code is usually vigorously tested for vulnerabilities during the build and deploy stages, it can—and in some cases is—compromised in production. The global Magecart credit card skimming incident is an indicator of this: The hackers were able to infiltrate the third-party components and exploited them to gain credit card information, resulting in 800 e-commerce sites being compromised. To avoid such distributed vulnerabilities, Dev, Sec and Ops teams need to constantly index third-party functions and services, and scan for possible vulnerabilities or attacks. Such monitoring requires unified observability at the app code level.

The Growing Overabundance of Purpose-Built Security Tools

Increasingly enterprises are leveraging multiple cloud providers while still running their own data centers, either on-premises or as private clouds. A 2017 report from Forrester Research points to 89 percent of enterprises using at least two clouds, while 74 percent use three or more public clouds. Despite a wide industry effort lead by global leaders such as VMware to provide seamless cross-cloud orchestration and management, most clouds remain in their own environments with a set of purpose-built management and security tools.

Then there is the traditional security approach, which can get in the way. According to Cybersecurity Insider’s “2018 Cloud Security Report,” only 16 percent of organizations believe the capabilities of traditional security tools are sufficient to manage security across the cloud. Eighty-four percent said traditional security solutions either don’t work at all in cloud environments or have only limited functionality, meaning that more tools with specific functionality must be integrated within the cloud. On-premises, private and public clouds remain siloed for most team, stressing an overtaxed security resources across organizations.

Shared Responsibility Can Be a Shared Burden

Even though the shared responsibility model has been well-documented, securing public cloud environments can be a burden to security and IT teams as their responsibilities continue to grow, while available talent is short to find. The traditional shared responsibility model assumes that the cloud vendor secures the hardware and software of the cloud itself, while the customer is responsible for the security of their assets within the cloud.

With the shared responsibility model, most security teams are unsure about how to do their part. Gartner reports that through 2022, at least 95 percent of cloud security failures will be the customer’s fault. This can include anything from a failure to monitor to a failure to adequately secure the correct elements. This can also include customer misconfiguration, mismanaged credentials or insider theft. Additionally, The “2017 IBM X-Force Report” noted a 424 percent jump in breaches related to misconfigured cloud infrastructure, largely due to human error. These gaps in security leave security teams with their organization, data and apps vulnerable.

App-Aware Stack Observability: The New Modern Security Mandate

The biggest distributed security blind spot facing modern Dev, Sec and Ops teams today is for them to monitor for vulnerabilities or attacks at each layer of the stack—at the application/third-party code-use level, their distributed workloads and the network each in isolation. Unified application-aware stack visibility and observability is becoming the only way to fully see and understand potential risks in today’s increasingly distributed environments, effectively addressing some of the biggest security challenges faced by modern Dev, Sec and Ops teams.

— Gadi Naor

September 24, 2018
Serverless: What You Need to Know

There are clear benefits to serverless architecture, with many enterprises already making or planning to make this change. Like any paradigm shift however, the move to serverless carries both benefits and risks. Here’s what you need to know.

AWS pioneered the serverless space with its release of AWS Lambda back in 2014, with the other major cloud providers following: Azure Functions as a service, Google Functions and IBM Cloud Functions (Apache Openwhisk). Later entrants in this space took the form of container-based, fully-managed serverless frameworks such as Kubeless.

The Good, the Bad and the Use Cases

Some of the most common serverless use cases today include building batch processing like image compression, cloud job automation, serving IoT devices from edge to the cloud and building single page applications (SPA).

On one hand, cloud-hosted serverless functions eliminate the overhead associated with managing actual infrastructure and introduce significant operational cost savings through efficient utilization of the underlying infrastructure. This allows developers to develop in their own comfort zone, using their language of choice: Python, JavaScript and, depending on the platform, even Go, Java, C#, PHP or others.

On the other hand, delegating the infrastructure control to the provider means that serverless introduces new set of operational blind spots and observability challenges. It also complicates the placement and management of deep security controls, when compared to container, virtual machine (VM) or even bare-metal workloads.

Serverless is Not the Answer for Everything

Whether building new applications or migrating existing ones, serverless runtimes have significant restrictions. Cloud-hosted functions-as-a-service restrict the runtime duration of a function, require a “warm-up” time for function invocation and don’t guarantee local state preservation across function invocations. This means that application logic that is purely time-bounded, involves short functional transformation of input into output and can handle a slow starting time would fit the serverless paradigm perfectly.

Migrating state-oriented applications or stateful microservices into serverless functions is a different story. This will typically make use of a database—either managed or as a service—or make use of an application cache such as Redis or object store such as S3 to store state across requests.

While cloud-hosted serverless removes the burden of patching and maintaining infrastructure, the ephemeral nature of functions makes it extremely challenging when it comes to establishing perimeters and controls around sensitive data in flight and at rest.

Build, Ship and Run Serverless

Serverless, as young as it may be, is extremely promising. Adoption choices should be balanced around platform maturity for developers, testing, operations and what security and regulations require.

If we take as an example what Docker did for container packaging, with its Container Image format, this is exactly what functions and serverless need: a common language for packaging. A common tool for building, shipping, running, operating and securing function-based workloads would serve as common ground for the variety of platforms already out there.

From a security perspective, unification around secret consumption, service-to-service authentication and authorization between first and third parties, function workflows and access whitelisting, observability, security network monitoring, access policies to the network and access policies to data—all these would lower the friction of adoption.

The Cloud Native Computing Foundation recently released version 0.1 of Cloud Events, the communication message format for function and event triggering. This emphasizes how far serverless is yet from maturity and cohesive standards.

From a security perspective, Type A serverless workload protection systems must be able to inspect the serverless workload runtime. Type B workload protection systems are limited to what the serverless platform provider APIs make available for out-of-band data-oriented security analysis.

Cloud-hosted functions makes Type A protection challenging to achieve, whereas fully managed serverless frameworks (for example, Kubeless), enables enterprises to adopt serverless and open the door for new developer experience. These also accelerate innovation velocity and, with security tools designed for microservices and modern virtualization technologies, can maintain full visibility and runtime control over network and data.

Securing Functions as a Service: A New Arsenal of Security Tools

Functions, just like any other web-based microservice implemented using containers, VMs or bare-metal server, are not free from the well-known top 10 OWASP web application vulnerability list.

Functions, however, strongly rely on non-web, event-based communications and event-based networking channels. These, on public cloud providers, prevent inserting security controls to detect exploitation attempts or data exfiltration and enforce well-defined workload policies. This requires new arsenal of security tools that understand microservices, scale horizontally and coexist with the existing security operations and security stack.

The ephemeral nature of functions-based workloads makes them more difficult to exploit, and it is definitely harder to persist a threat. Yet third-party code vulnerability or open source components gone bad can result in sensitive data flowing into the wrong hands, landing in a misconfigured object store.

As serverless requires persistent data storage in external services, a scenario in which circumventing a running function to leak secrets makes that sensitive data one hop away from an attacker is not unrealistic.

Controlling it all together: What private information can be processed, what sensitive data is in flight and through which event grid channels it’s being received and sent, along with how functions are invoked by API gateways and which API calls which functions make to internal and external services. This is where risk, regulation and modernization collide.

Bottom Line

Despite significant adoption and constantly growing interest, serverless is still in its infancy. When it comes to standards, operations, testing and orchestration tooling, workflows and security, a flood of water must yet pass under the innovation bridge of cloud providers, with AWS Lambda leading the adoption curve, Azure FaaS making giant innovative leaps forward and community-driven frameworks built on top of Kubernetes.

Innovation aside, serverless creates huge operational and security blind spots. Understanding those risks and having proper introspection and inspection capabilities is what should dictate the adoption and the choice of the platform.

— Gadi Naor

June 14, 2018