Qualys today announced it is offering a completely free, 60-day version of its cloud-based remote endpoint protection solution. The product is fully featured, with nothing turned off in the free version. It’s also cloud-based, so there are no worries about hosting the solution. This comes at a perfect time, with so many workers working remotely.
“The Qualys remote endpoint security solution allows security teams to leverage the lightweight Qualys Cloud Agent to:
Complete an up-to-date inventory of all their remote endpoint hardware and the applications they are running
Gain a real-time view of all critical vulnerabilities impacting the operating system and applications posing a risk to these devices
Remotely patch these systems within hours from the cloud with one click of a mouse without using the limited bandwidth available on VPN gateways
Realize visibility in device hygiene by tracking common misconfigurations that leave the endpoints exposed to exploits.”
DevOps.com spoke with Qualys President Sumedh Thakar for more information on this new free solution. That conversation can be heard below:
Here’s a screenshot of the web-based interface of the solution:
What happens after the 60 days you ask? Qualys did not say definitively what the plan is for that, but I suspect that will depend on what happens with our economy and the work-from-home trends. In the meantime, if you are looking for a remote endpoint solution for the enterprise, this is a great chance to try one free for 60 days.
Good developers are in demand—great ones, even more so. With software development employment in the U.S. expected to grow by21% by 2028, the need for engineers is only going to surge further.
But this doesn’t mean that the labor landscape will become uncompetitive: Now more than ever, individuals with a passion for software are entering the field, and organizations will only be looking for the best. As the amount of people going into software development goes up—with global numbers expected to reach27.7 million by 2023—it will become increasingly important to demonstrate your skills and dedication to the profession.
The gap between a good developer and a great one is bigger than you may think. Here’s what the best developers do to excel at their jobs and lead in their field.
The Best Software Developers Know How Their Skills Fit Into the Big Picture
While a good coder handles the responsibilities they have within their role, the best coders make sure they see the big picture. They take the time to understand the foundations of how businesses work, including revenue, profit, bottlenecks and the strategies of departments other than their own.
This might mean investing extra time into general business self-education or holding meetings with other departments, but great programmers know it’s worth it in the long run. They recognize the importance of grasping how the cogs of an organization come together to keep things running—and exactly how they fit in.
By having this 360 view, they can hold influence in areas outside their remit and clearly match the business’ mission to their own work. Valuable coders will use their problem-solving mindset to propose solutions to company-wide problems and put themselves in the shoes of other team members.
Rest and Relaxation Is Vital for Productivity
Far from the stereotype of frequently pullingcaffeine-fueled all-nighters, the best developers know that, like the powerful systems they work with, they also need some downtime to recharge.
Great coders challenge the stereotype of having a bad diet and inactive lifestyle and prioritize their own health and well-being. They recognize that as soon as you pass the threshold of50 hours of work a week, you’re no longer productive. In fact, working too much can be counterproductive, as you’re much more likely to make mistakes.
Great developers know when to shut down their computer and spend time with their families, do some exercise, spend time in nature and eat a healthy meal. This time to recharge is essential in order to go back to work refreshed and ready to tackle any problems with a clear and rested mind.
Continuous Learning Is Crucial for Progression
Ideally, all developers should be making efforts to advance their education in the field they’re in. As a developer, if you stop studying, you’re bound to get stuck at one point or another. The best coders make this a special focus in their lives, and make sure they’re always up to date with the latest developments in the technologies they work with.
It could be a new coding language, new frameworks or a new technique. Great coders are constantly making a concerted effort to advance their own education. Ultimately, they are always asking “why,” and are unafraid to experiment using their newfound skill.
Developers can easily find online courses to supplement formal education. Taking part in hackathons is another way to boost extra learning, as well as keeping up with technical blogs and podcasts.
Great developers also follow other passions outside of coding. For example, the logical nature of software development allows coders to excel at learning other skills, such as languages, playing an instrument, martial arts or even flying a plane. It enables them to exercise the same part of their brain while enjoying something fun and non-work related.
Personal Projects Show Dedication and Independent Thinking
Coders know that the best way to learn how to build great products is through practice. Even if the project ultimately fails, building a product will always provide important lessons to apply the next time.
Side projects that a developer might work on include personal tools, open-source libraries, a startup idea or a freelance project. These personal projects allow programmers to be in the driving seat: They are forced to solve problems independently instead of asking for help, equipping them with skills that might otherwise take longer to gain.
Engaging in independent work also allows developers to explore and utilize the new languages and frameworks that they gain in their personal education efforts. When developers that have spent time working on their own projects try to prove themselves in the job market, they will stand out against those that have remained largely within the confines of their professional role.
Ultimately, the vast majority of developers are in the game because they have a passion for building exciting and useful products. This makes it easier to find those who go the extra mile to really hone their existing skills and learn new ones. But, like in any profession, there will always be a certain percentage that remains a cut above the rest. Take a holistic approach to your career in software development with this advice and raise your game, and chances of landing that dream job.
If you run down the laundry list of major industries–transportation, retail, healthcare, automotive, financial services–you’ll see that the major players in each have essentially become software companies. What they do and how they serve their customers rests entirely on their software working perfectly. Underpinning this is a massive digital transformation that has shifted organizations away from the old, static ways of enterprise IT monoliths. Instead, today’s businesses are working in more dynamic IT environments–agile, multi-cloud and, especially for new digital services, based on containers and microservices. All of which means embracing more IT complexity than ever before.
Kubernetes Brings More Data to the Table, but that Impacts Observability
A key part of this more complex environment is the soaring adoption rate of Kubernetes over the last few years. In a survey of 5,000 enterprises, 40% said they were currently running Kubernetes in production and over 80% reported that they’d adopted Kubernetes over any other container management system. Another recent survey revealed that 68% of CIOs are already using containers and a total 86% expect to deploy them over the next year. Enterprise IT is increasingly gravitating to containers and especially to Kubernetes for orchestrating their containerized applications.
What drives organizations toward containers and Kubernetes is the flexibility given to development teams to release services, features or apps faster. But faster doesn’t necessarily mean better or higher quality, unless you also apply cloud-native architectural patterns, include quality gates into your delivery pipeline, embrace new deployment models such as canary or feature toggling and provide full visibility and short feedback loops from production back to engineering.
Kubernetes itself provides great visibility into node or pod health. Kubernetes even ensures the health of your pods through its built-in health check mechanisms. Thanks to components such as service meshes, you also get live information about dependencies between services running in your pods. This data can, in theory, provide organizations with the necessary insights into their environments to understand whether their organization is releasing better value faster to their end users and unlocking new business value.
The reason I said these are theoretical benefits is because, in practice, all that extra data doesn’t necessarily mean more visibility. Often it can mean less visibility–and with that, less insight into how the different elements of your environment may be impacting speed of production deployments, overall system performance and the end-user experiences. I’ve seen Kubernetes environments that are packed with literally billions of interdependencies between applications, containers, microservices and clouds. Asking human beings to keep track of that level of data manually is an impossible task now. And relying on traditional, manual approaches to manage and monitor IT environments shortchanges the value that organizations should be getting out of Kubernetes.
Kubernetes Environments Need Full Stack Observability—that Means Needing Deterministic AI
Getting the full potential out of Kubernetes means having observability across the full technology stack and those billions of interdependencies. But as more organizations are discovering, that can’t happen without AI. Specifically, a deterministic AI engine. In fact, in a 2019 survey, 88% of CIOs said they were looking to AI as a critical solution for overcoming IT complexity. As long as Kubernetes is fueling complexity, overcoming that complexity means making deterministic AI an integral part of any Kubernetes environment.
A deterministic AI applies specific algorithms to detect system anomalies and apply its collected dependency information toward detecting the root causes behind those anomalies. When that deterministic AI engine ingests critical Kubernetes events, such as state changes and workload changes, it provides visibility into the Kubernetes stack’s interdependencies. That AI-powered, full stack observability matters because it gives IT new capabilities for their environments, including instrumenting and mapping containers, managing and allocating container resources and drilling down into container runtimes at work. All of that is possible due to the level of observability that you can only get with deterministic AI; traditional metrics and dashboards just can’t compete.
By providing full stack visibility and interdependency mapping across a Kubernetes environment, deterministic AI is empowering IT teams to identify, in real time, the root cause of degradations in business service availability–from reliability and resource usage to system performance and user experience–faster than they ever could manually. That level of automated speed and visibility also makes it possible to immediately resolve those issues at the source before they can impact the end user. The automation provided by deterministic AI also works its way into DevOps’ production pipelines, accelerating the release and feedback cycles of new deployments and allowing teams to more quickly incorporate user feedback into the next iteration for even better experiences.
In short, a deterministic AI engine makes life better for both IT and the customers they’re aiming to serve. Deterministic AI takes work that used to be time-consuming, tedious and counterintuitive when done manually, and performs it fast and automatically. That then frees up the teams that used to do that to focus on work that is more satisfying and drives new business value.
AI-Powered Observability Relies on Quality Data and Quality Monitoring
All that said, a deterministic AI’s ability to provide reliable, precise full stack observability in a Kubernetes environment relies on it being given consistently quality data it can process. The algorithms are dependent on the data that’s ingested by the AI engine. If there’s no standard monitoring capability being used across the board, then you invite a scenario where different teams bring their own preferred monitoring tools into the fold. The bigger a Kubernetes cluster gets, the more diverse the set of monitoring tools can be. And the greater the diversity of monitoring tools, the more inconsistent the data being fed into the deterministic AI engine.
The way out of this predicament is to leverage a single monitoring tool as a self-service capability into Kubernetes clusters, one that can support the diversity of technologies that are introduced into the stack and provide a baseline standard of system and event data that developers, IT administrators and the deterministic AI all lean on. In other words, when you feed consistent, quality data into the AI, you make it smarter. And a smarter deterministic AI engine provides the full stack Kubernetes observability that organizations need.
Building a Positive Feedback Loop Between Kubernetes, Deterministic AI and Observability
Kubernetes and deterministic AI are a natural fit. Kubernetes provides a platform for managing and instrumenting containerized applications. The deterministic AI provides a tool for mapping out the Kubernetes stack, offering insights into interdependencies and degradations where they arise and resolving them before they can affect end users. The data from Kubernetes makes AI better, and a smarter deterministic AI engine means more precise observability across your technology stack.
It’s a self-reinforcing feedback loop that eliminates the usual pitfalls of container-induced IT complexity. Instead, it creates new standards of visible, consistent and quality data that organizations can leverage for faster and better deployments–and in turn, better user experiences and greater business value.
To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or August 2020.
The original focus for Kubernetes was very much on stateless applications, which don’t rely on coordination between instances or sharing data between client sessions. In practice, that usually means that we can stop and start our stateless applications without data loss or impacting on client connections.
If all applications were stateless, the world of application deployment and management would be super simple, but unfortunately that’s not all use cases. The second class of applications are stateful, where the application is retaining data or state across its entire lifecycle. Whilst older monolithic applications such as SQL databases fit into this paradigm, there are also many architectures where the individual elements making up a particular service are actually clustered together in some way. They may be sharing state data between them, or they need to be synchronized together.
One of the fundamental features of Kubernetes is that the scheduler can move pods around between agents, which works fine for stateless applications but not so well for stateful, which tend to like their storage and networking to not change during operation, unless handled in a very specific operational way. In order to try to handle the complexities of stateful applications, Kubernetes 1.5 added StatefulSets, which gave us the concepts of stable network and storage. But this doesn’t solve all of our problems, since Kubernetes still doesn’t know what’s actually going on inside the pods.
Typically these kinds of applications have a set of lifecycle states which require logical ordering of actions in order to maintain operation. Stateful applications tend to have coordination between their elements, so, in general, if we were to just start and stop instances then we may impact on the correct behavior of the application—we may need to wait for re-balancing of data, or there may be multiple steps involved in an upgrade operation. This also applies to many operations that you are likely to perform such as scaling, and these actions also tend to be unique to the application, so a Cassandra cluster is managed differently from a Kafka cluster. That domain-specific knowledge of how to administer this class of applications isn’t captured in the paradigms that Kubernetes gives us out of the box, so how do we automate and package that knowledge?
The solution to this problem is the Operator pattern. Operators encapsulate operational tasks in code, so we can orchestrate application lifecycle actions using Kubernetes APIs. Operators encode domain-specific knowledge about the lifecycle actions required for the application—how to scale it, how to upgrade it, how to recover from failure scenarios and so on.
When we install an Operator into our cluster, we will get a controller, which is the code that manages the lifecycle of the particular thing and where the domain-specific knowledge is encapsulated. We’ll also get a bunch of custom resource definitions, which extend the Kubernetes API to address new resource types, so these add a new type which you can then put data into and instantiate.
There are now many different ways to build Operators. The first one to emerge was the Operator Framework, which originated at CoreOS, but since then a whole range of different ways to build Operators have appeared, including Kubebuilder, Metacontroller and KUDO. These all take different approaches to building Operators, from SDK’s requiring Go expertise and deep Kubernetes knowledge to the KUDO approach of a polymorphic Operator, which can be configured very simply by using YAML.
For cluster administrators managing large clusters, this proliferation of development methodologies and management interfaces can be problematic. The class of applications that Operators were developed to manage are almost always part of a larger application stack, with dependencies between them. If we have a lot of Operators written in different ways running on our clusters, how can we ensure they will interoperate with each other, and how do we validate and test them?
When we depend on other services, we primarily care about two things: how to access and use it, and what it does. The KUDO project is working on the Kubernetes Operator Interface (KOI) to address this, enabling Operators to easily compose with each other based on a well-known set of CRDs. This interface is not just about connection strings and secrets, either; KOI aims to answer the question of “What does this Operator actually do?” By exporting an Operator’s behaviors as a CRD, tooling such as CLIs and GUIs can be written that better abstract the tasks that users need to perform rather than having to understand a library of custom resources.
Defining behavior is one thing, validating it is another. How do we write standardized tests that confirm an Operator actually does the thing it says it is going to do? In order to try and address this problem, the KUDO project has also developed a test framework, Kuttl, which allows Operator developers to write tests to prove correct behavior. Kuttl has wider use cases outside of the KUDO project, with the ability to test conformance for all kinds of Kubernetes resources, so this is in the process of spinning out into a separate project.
These kinds of efforts around standardization, definitions and testing will enable us to build ecosystems of Operators that naturally compose with each other, no matter the implementation—allowing users to build application stacks they can be confident will inter-operate consistently, rather than a constellation of unrelated Operators strung together by assuming the right configuration values. Collaboration and cross-project working are key to the future of the Operator ecosystem, and there are now more and more collaborative efforts emerging across the industry with these goals in mind.
To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or August 2020.
This article was co-authored by Gerred Dillon, principal engineer for cloud native applications at D2iQ. Gerred builds tooling for large scale workloads on top of Kubernetes. He is the creator of the Kubernetes Universal Declarative Operator and assists with the Kubebuilder subproject, SIG API Machinery, and CNCF SIG App Delivery.
Atlassian announced today it will make cloud-based versions of Jira Software, Confluence, Jira Service Desk and Jira Core available for free as part of an effort to assist organizations whose employees are working remotely to help combat the COVID-19 coronavirus pandemic.
Robert Chatwani, chief marketing officer for Atlassian, said given the current unprecedented situation the company has decided to increase the number of free versions of its software, including Jira project management software, which is widely employed by software development teams.
The company already makes available free versions of its Trello, Bitbucket and Opsgenie software. Atlassian is hoping that once the current COVID-19 coronavirus crisis subsides, a much broader base of end users including ones from business units will have gained some firsthand experience employing its project management applications and collaboration tools, said Chatwani. Previously, Atlassian revealed it has a goal of having 100 million active users of its software.
Chatwani said many organizations are finding that having employees working from home is the new normal for the foreseeable future. While regions in Asia appear to be recovering from the initial COVID-19 coronavirus outbreak, that does not appear to be the case in other regions anytime soon. Once organizations re-engineer their workflows, they may discover some knowledge workers are more productive working remotely, he noted.
Less clear right now is whether having most or all employees working remotely is a temporary measure in the furtherance of a business continuity strategy or simply an inevitable future of work that, because of the current crisis, is arriving much earlier than expected.
Of course, not every employee is psychologically tempered to work from home. Software developers, for example, may be more accustomed to working in isolation than project managers. Chatwani said it will be important for organizations to establish new routines to maintain organizational cohesion, including determining when development teams should launch sprints to build elements of an application. Managers also may want to designate “office hours” during which anyone can contact them online to make sure employees feel they are accessible, he added.
Organizations should also get used to the idea that children may be both seen and heard while online meetings are occurring, given the fact that millions of children are now being schooled at home, Chatwani noted.
Many organizations to one degree or another have been relying more on remote workers. There are even some companies that don’t have physical headquarters—all the employees already work remotely. Some of those organizations have even gone so far as to define a time zone around which the organization will standardize on regardless of where employees are physically located.
Whatever the strategy employed, organizations that hope to succeed in this new climate need to go well beyond ensuring remote network connections are available and secure—they need to find a way to either maintain their existing processes and culture to make sure that whatever product or service they provide continues to be built or delivered in a way that allows the organization to survive and hopefully continue to thrive.
When a Kubernetes deployment suffers a security breach, it can be difficult for security teams to diagnose and identify the source of the intrusion and its impact. Each Kubernetes cluster may host multiple applications, and each application may be composed of dozens or even hundreds of interacting microservices deployed as pods.
The complex and dynamic communication between the applications’ components, as well as the automated Kubernetes orchestration activities, makes it difficult to understand the structure of each application and differentiate between normal behavior and malicious behavior from the observed network activity. The ephemeral nature of pods, which may exist only for moments to perform an activity before they are disposed of, makes this task even more difficult. A pod that was vulnerable may be gone by the time security teams are investigating a breach, or resume normal operation as the attacker pauses its abuse or pivots or escalates the attack to another Kubernetes resource.
K8s is by nature dynamic, flexible and scalable, and it is this same advantage for cloud native application developers that becomes a challenge for security professionals. How can one understand the source of a breach, let alone how it propagated through the system, without deep understanding of the architecture of the application?
One solution to the problem of tracing breaches is to examine the Kubernetes audit logs since they record all cluster-administration activity, even for pods that may have existed for seconds and are now gone. But K8s audit logs are complex and often convoluted, requiring deep expertise to understand. Therefore, security teams need to not only increase their proficiency with K8s audit logs, but also to apply automation and software tools to the K8s logs to detect suspicious behavior after the fact and even during their occurrence.
In fact, there are many potential threats that can be discovered when observing audit logs for suspicious behavior. They include:
Stolen or misused credentials, enabling hackers to gain access to K8s-based clusters or pods.
Misconfigured Rules Based Access Control (RBAC), enabling lateral attack propagation, privilege escalation and unauthorized data access or manipulation.
Exploitation of vulnerabilities in the Kubernetes API Server, enabling bypassing of authentication, authorization, admission control or validation of cluster administration requests. This lets users gain access to privileged and sensitive resources.
Violations of security policies diverge from compliance requirements and best practices.
Detect Breaches in Near Real Time by Watching K8s Audit Logs
There may be a vast time span between an initial breach and the detection of the compromised application or the unauthorized access of secure data and private information. Security teams must seek to close the gap between a breach attempt and its detection as much as possible, while still being able to diagnose the causes of a breach after it has been detected and assess its scope and impact.
Since K8s deployments can be vast and intricate, machine learning and artificial intelligence are preferred tools to monitor audit logs and flag suspicious behavior. An automated tool could continuously scan logs to learn what normal behavior looks like in the company’s clusters, and then flag network behavior that is unusual or suspicious for security teams to qualify, prioritize and investigate. This methodology would enable security teams to investigate possible security violations in near real time to protect and secure the applications.
In addition to detecting security risks, security tools that leverage K8s audit logs should provide reporting that is easy to understand and consume by security teams. By providing the correct user experience, companies can make their K8s logs accessible to more than the hard-core experts, effectively opening a new vista of security possibilities for more security professionals. The difficult learning curve of Kubernetes will no longer be a wall between K8s pros and newcomers with less experience, relieving pressure from understaffed security teams.
Step Up to Security and Compliance Needs
As Kubernetes moves toward the mainstream, with roughly 71% of Fortune 100 companies using Kubernetes as their main container orchestration tool, companies must also look toward compliance with standards and government regulations designed to protect financial transactions and private or personal information such as PCI, GDPR and HIPPA. The penalties for violating these standards and regulations can be severe, so companies should proactively adopt K8s audit log monitoring as a way to identify breaches earlier, to limit their impact on customers and data, and to document the behavior of the application and would-be hackers for possible legal inquiries. An effective audit log strategy will be vital in determining what happened in the case of a breach.
Leverage Audit Logs for Real-Time Security Observability in K8s
Many tools exist to secure the perimeter of the data center and to identify possible security risks during application development. The benefits of analyzing and leveraging Kubernetes audit logs at runtime have been relatively overlooked. K8s audit logs are a prime way to comprehend the behavior of any cloud-native application orchestrated by Kubernetes, as well as the best way to comprehend how the Kubernetes infrastructure itself can be abused by an attacker to compromise the applications it is hosting. As such, security teams have a responsibility to deploy tools and gain a proactive, real-time view into K8s audit logs in order to minimize the time between breach and remediation and to validate compliance conformance at runtime. Such a thoughtful approach to Kubernetes security will go a long way to protect your deployments from harm.
To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or Aug. 2020.
Tens of thousands of people and hundreds of cybersecurity vendors descended on San Francisco at the end of April. While the RSA Conference was the primary draw, there are a number of peripheral events that go on throughout the week as well, such as the Cloud Security Alliance Summit, the CIO/CISO Interchange, BSides San Francisco and the Qualys Security Conference.
I was able to attend some of the Qualys Security Conference and I had the opportunity to see Alex Mandernack, security solution architect for product management at Qualys, presenting on the need to shift left when it comes to cybersecurity.
Mandernack outlined some of the issues organizations face today when it comes to security. Migrating to the cloud and embracing DevOps culture means more ephemeral, elastic environments and compressed development time frames. He pointed out that these trends make security more challenging, but also described the opportunity they present to change our approach to streamline and improve security.
The Traditional Pipeline is Broken
The traditional model of application development is much more linear. Each app team builds their own image—often in a silo or vacuum disconnected from other teams. Penetration testing is conducted on a sporadic or infrequent basis, and eventually the app is deployed. Vulnerability scans are run on the app in production, and issues or deficiencies are reported back to the developers to address or incorporate into the next build.
The process is slow and cumbersome. It also has the potential to introduce redundancies that waste resources and make the organization less efficient. As companies embrace DevOps and move to the cloud, the traditional development pipeline simply doesn’t work—especially not for security.
DevOps, and Cloud, and Containers — Oh My!
The cloud and DevOps fundamentally change the development lifecycle. Development time frames are compressed, and the process is a self-feeding, iterative circle rather than a repeating linear approach. The traditional approach of identifying and resolving security issues after an application is deployed won’t work in a DevOps environment because by the time issues are identified and fixed, the app itself may have already moved on to the next version.
Cybersecurity and the development pipeline have enough challenges with the cloud and DevOps, but there is also a newer technology trend that exacerbates the problem—containers. Containers are designed to get code quickly and to operate fast. They are also built for scalability—spawning and spinning down hundreds or thousands of containers constantly to keep pace with demand.
There is a diverse array of tools and platforms associated with cloud, DevOps and containers. Organizations often employ a complex collection comprised of some combination of AWS, Azure, Docker, Kubernetes, OpenShift and others. The result can quickly become a black hole when it comes to security—which is why it’s important to push security left into the CI/CD pipeline.
Shift Left to Help Security Teams Do Better
In a DevOps culture, developers and security teams need to think about security sooner. It’s important to get the security tools into the process earlier. Thankfully, with the DevOps lifecycle and the tools commonly used within a DevOps environment, it can be done.
For starters, companies should build a golden image. Developers shouldn’t have to patch and update the base operating system before they even start developing. The golden image takes a baseline operating system such as Ubuntu or CentOS, and completely patches and configures it so everyone has a standard, consistent image to develop on.
With the golden image in place, the next step in shifting security left is to implement testing and scanning in the CI/CD pipeline. Put vulnerability information at the fingertips of developers as they’re developing and implement vulnerability gates in the pipeline to ensure that applications that don’t comply with policies and meet minimum requirements are not allowed to be deployed.
New Role of the Security Team
Part of shifting security left includes providing tools that make security self-service for the DevOps team. Automating actions with scripting, APIs and CI plugins ensures that security is simplified and streamlined to provide value for developers.
In this shift left scenario, the security team has to redefine its role in the process. The security team is still vital, but rather than being the police or a roadblock in the development process, the role of security focuses on verifying and auditing the process. The security framework should have dashboards and live data that enable the security team to monitor activity and identify trends that may need further attention.
Better Security, Faster
There is often conflict between developers and security teams, but there doesn’t need to be. Mandernack stressed that developers don’t want to create insecure apps either.
Shifting security to the left into the CI/CD pipeline streamlines the whole process. Ultimately, the result is higher quality, more secure applications that are developed faster.
Kubernetes and Docker are two of the most commonly heard buzzwords in modern DevOps conversations. Docker is a tool that enables you to containerize and run your applications, and Kubernetes provides you with a platform to orchestrate or manage these containers—since managing thousands of containers manually using the Docker CLI would be a practical nightmare.
However, simply running thousands of containers and managing them through Kubernetes is not enough. You have to observe and analyze them properly to ensure that your services are up and running optimally. This process is called Site Reliability Engineering (SRE), a term initiated and popularized by Google. Observability and analysis are a major element of SRE. It can be subdivided into the following three areas:
Monitoring allows you to extract numerical metrics from your applications and resources that can later be visualized and analyzed to present the current state of your resources. Once the metrics are extracted, they can be used to set up alert rules, facilitate profiling and debugging, and make better SRE decisions.
Logging enables developers to debug their containers in case of a failure. Containers are short-lived, and so are logs. Kubernetes and Docker do provide a native way of browsing container logs, but it is very limited in its functionality. As a result, a centralized log plane is a must in any container-oriented environment.
Tracing allows you to debug services running on a network and follow a request trail until the source of a problem can be determined. In a microservices architecture, when multiple services/containers send requests to each other to perform one business task, a proper tracing solution is necessary.
This article will list and describe the most effective open source tools for monitoring and analyzing services running as containers.
Prometheus
Prometheus is the first tool that comes to mind when discussing open source monitoring solutions. It is very popular in the development community, and it is a graduate of the CNCF project, which focuses on the monitoring side of SRE. Originally created and open sourced by SoundCloud, Prometheus simplifies the process of pulling numerical metrics based on a time series from a given metrics endpoint. It was built to monitor highly dynamic container environments.
Prometheus is subdivided into three elements: Prometheus server, Alertmanager and exporters. Exporters are stand-alone, independent processes/containers that can be run on your target resource to generate and export metrics via a metrics API. The Prometheus server then takes care of service discovery as well as pulling metrics from the exporters to be stored in the Prometheus DB and later used for visualization or alerting. Alertmanager is responsible for setting up alert rules, analyzing the data in the Prometheus DB and sending alert messages to multiple receivers if a certain rule is triggered. There are a huge number of exporters, listed here, which are both officially supported by Prometheus and maintained by the community.
Prometheus has become the industry standard for monitoring cloud-native architectures. While it is known for its simplicity with service discovery, its ease of use, its alerting capacity and its integration with Kubernetes, its polling architecture is not ideal. Currently, the metrics endpoint has to be reachable by the Prometheus server. However, there is a pushgateway implemented in Prometheus that enables metric pushing rather than polling. Another downside of Prometheus is that it doesn’t scale well. This problem can be fixed in the Thanos adaptation of Prometheus.
Related tools and technologies: Grafana, Cortex, Thanos, Prometheus Exporters, Alert Manager, Istio, Prometheus Operator.
Grafana is an open source metric analytics and visualization suite. It allows you to create custom dashboards using data taken from multiple sources, such as Prometheus, Elasticsearch, MySQL, Postgres and Redis. Furthermore, Grafana has its own alerting system and a role-based access control (RBAC) system for software. As a data visualization tool, Grafana is famous among Prometheus users for enabling them to effectively visualize the metrics stored in Prometheus. There’s a wide array of official and community-built custom dashboards for various data sources in Grafana, allowing users to easily set up dashboards (found at this link) and get on with monitoring. Grafana offers another associated product called Loki, which aggregates logs in Kubernetes and integrates well with the Grafana UI.
Related tools and technologies: Loki, Prometheus.
Figure 2: Kubernetes Resources Dashboard in Grafana. Source: Replex.
Elastic Stack
Elastic Stack is a group of open source products from Elastic designed to help users search, analyze, and visualize data from any type of source, in any format, in real time. This product was previously recognized as the ELK stack, where each letter of the acronym stood for one of the company’s main products: Elasticsearch, Logstash and Kibana. Elastic Stack provides both monitoring and logging solutions with the help of its big data database, Elasticsearch.
In order to aggregate logs, people tend to use Elasticsearch for storage, Logstash or Fluentd for log streaming, and Kibana for visualization. Fluentd is not part of the Elastic Stack, but it is widely used with Kubernetes to stream logs in place of Logstash, the tool provided by Elastic Stack. Similarly, Metricbeat is used to scrape metrics and set up visualizations on Kibana. The enterprise version of Elastic Stack ships with X-Pack, a set of additional tools that enables reporting, alerting and role-based access control (RBAC), among other features. By default, the Elastic Stack GUI, Kibana, does not support RBAC. You have to use the previously mentioned enterprise version of Elastic Stack to enable it.
Related tools and technologies: X-pack, Metricbeat, Logstash, Kibana.
Figure 3: Discover View in Elasticsearch. Source: XpresServers.
Sensu Go
Sensu Go is a telemetry and service health checking solution for multi-cloud monitoring at scale. It allows you to gain visibility into servers, containers, services, applications, functions and connected devices across any public or private cloud. Sensu can be run side-by-side with Prometheus to get the best out of both solutions, or it can be run natively without Prometheus. Sensu works best alongside Prometheus, since exporting application level metrics to Prometheus requires loading the Prometheus SDK to your application codebase and exposing a metrics endpoint. The endpoint is then scraped and stored in the Prometheus Server. This may sound like a lot of work—and, sometimes, it is. Sensu avoids this complexity by using a sidecar concept. A Sensu agent is deployed alongside your application. The agent constantly collects and exposes metrics to the Prometheus server without your having to change the application codebase.
Sensu works without Prometheus too. It can operate natively in Kubernetes, where it has its own server to store and visualize metric data exposed by the previously mentioned Sensu agent.
Sysdig has two open source products: Sysdig Inspect and Falco. Here, we’ll focus on Inspect, which monitors and captures the container processes running in your system and allows you to drill into those processes for post-incident forensics. This enables you to analyze your application performance, troubleshoot errors and monitor any processors that could be misbehaving. Furthermore, if your system is breached, Sysdig allows you to understand how the breach occurred and what data was taken in the process. Sysdig Inspect is a very powerful tool that focuses on the performance tuning and security investigation of your system.
Related tools and technologies: Grafana, Sysdig, Sysdig Falco.
Jaeger is an end-to-end distributed tracing solution that was open sourced by Uber Engineering. It allows you to monitor and troubleshoot transactions in complex distributed systems. In modern microservice architectures, the majority of operational problems fall under the umbrellas of networking and observability. When a service failure occurs, you have no idea how the requests have gone from service to service over the network to complete a single business transaction. This makes debugging extremely difficult. Currently incubating under CNCF, Jaeger uses tracing to enable root cause analysis, performance and latency optimization, and distributed transaction monitoring. Jaeger works out of the box with Istio, a popular service mesh implementation open sourced by Google.
Related tools and technologies: Prometheus, Jaeger, Zipkin, Istio.
These tools are widely used in the tech industry, and they all have their benefits. Most of these solutions, however, require skilled implementation and ongoing manual maintenance that can be a burden for DevOps teams and a distraction from the business. There’s no one solution that can cater to all of your requirements, since each tool focuses on one or two specific aspects of observability and analysis. By mixing these tools together, you can derive a unique solution for your individual business needs.
For ease of comparison, the chart below outlines the features offered by each of the tools discussed in this article.
Zoho today announced it is making available for free a bundle of software-as-a-service (SaaS) applications dubbed Remotely through July 1.
Raju Vegesna, chief evangelist for Zoho, said Remotely is a humanitarian gesture to help mitigate the impact of a global coronavirus epidemic. Remotely comprises a set of productivity and collaboration applications that users can access in as little as five minutes.
Included in Remotely are productivity, collaboration and project management applications as well as tools that automate the migration of data into the Zoho cloud platform.
Now that many local governments are recommending that organizations allow employees to work from home to help limit the spread of the coronavirus outbreak, many IT administrators are scrambling to provide remote access to applications at scale. It’s one thing to provide access to a small number of employees who might travel intermittently, but it’s quite another to support an entire remote workforce for some unknown period of time.
Vergesna, however, said that at this point the challenges associated with making that transition have a lot more to do with overcoming cultural issues than they do the technology. Much of the transition to Zoho SaaS applications has been automated, he said.
It remains to be seen to what degree the coronavirus outbreak might accelerate the transition to cloud applications. Many IT administrators are still managing local instances of Windows servers to deliver applications because the time and effort to migrate to SaaS applications have been substantial. Making those on-premises applications available remotely at scale is not only challenging from a technical perspective, but it’s also expensive.
Of course, Zoho is far from being the only provider of SaaS productivity and collaboration applications. However, it remains to be seen whether other providers will follow Zoho’s lead. The hope each of these vendors has is that once the coronavirus crisis passes many of the organizations that take advantage of their software to enable employees to work from home will start paying to use that software.
In the meantime, the economic impact of the coronavirus is likely to be staggering. Conferences are being canceled and face-to-face business meetings are being shifted online. It’s hard to say how many of those meetings are likely to stay there for good. At the very least, many more end users are about out to discover just how rich SaaS application environments have become. In many cases, IT teams will stitch together workflows across a range of SaaS applications using application programming interfaces (APIs). The challenge they will face is acquiring the programming skills required to construct those workflows.
As an industry, we spend a lot of time talking about the process and technology that make up a good DevOps practice. We have a strong focus on the development, IT operations, QA” and security roles. We debate tools and automation strategies. We wax philosophical about DevOps culture.
What is either lacking entirely or simply glossed over in that conversation is what I would consider the backbone of any good DevOps team–the people.
You might assume by “people” I mean their skills with the technology or their expertise with a certain process. Those two things are important, but all things being equal, the people aspect of DevOps I am referring to is really more simple–or even more basic–than that. It all starts with the culture that is defined by the lead.
As with any cross-functional discipline, teamwork is essential to the adoption and success of your DevOps practice. At its core, DevOps is about collaboration among people from a variety of teams and departments. It brings together multiple disciplines, people and processes with the common goal of high quality and high efficiency output. DevOps relies so heavily on teamwork that without strong leadership, your DevOps culture and the success of your practice are destined for a rocky road at best.
A strong DevOps leader will need to bring together people in roles that have either been traditionally siloed or are now being done by fewer people. This is no easy feat, however building trust, earning respect, communicating effectively and motivating everyone toward the common goal is paramount to the success of the DevOps team.
These traits–trust and respect, communication and the ability to motivate–are not unique to DevOps leads, rather they are the key characteristics that make up any strong leader. By putting them in the context of DevOps, the importance of each in the process and culture is irrefutable.
Trust and Respect
Paul Zak, the founding director of the Center for Neuroeconomics Studies and a professor of economics, psychology and management at Claremont University, led a study to better understand the neuroscience of trust. His team found that the effect of trust on work performance, engagement and loyalty were huge. Respondents indicated that employees with high trust had 106% more energy, were 76% more engaged and 88% said they would recommend their company as a place to work to family and friends.
The numbers might not come as a surprise, but how you achieve high trust might. First and foremost, be vulnerable. Do not be afraid to ask for help. This not only exhibits the trust you have in the people on the team, it also demonstrates the sign that you are secure in yourself as a leader.
The trust factor for a DevOps lead is multifaceted. The team must trust the lead enough to also trust his/her vision for the process. By showing vulnerability and leaning of the subject matter experts from each discipline for input, you start to build rapport with the team and the process becomes owned by the different functions involved. By asking for help, you invoke the natural tendency to cooperate. That cooperation leads to mutual trust. And by earning trust, you also earn their respect.
Communication
As with any cross-functional team, communication should be viewed as critical an element as your DevOps tools. A successful DevOps lead will be able to understand and speak the languages unique to each discipline. By understanding the nuance and meanings of the words used by the IT operations teams, QA, security and development, the lead can easily help communicate goals, set expectations and establish rules of engagement. Through daily communication, the team will be able to reduce friction and become far more effective as a group.
Motivating Toward a Common Goal
Former Boston Bruins coach Harry Sinden said it best, “Putting on the same jersey doesn’t make a team. You’re still a collection of individuals until you find a common goal.”
The DevOps lead who motivates the team toward that common goal will find success. But how do you do that? To find out what motivates the people on your team you need to talk to them and get to know them. Involve them in developing the overall strategy. Seek their input and make sure they feel they have ownership of the outcome.
Set milestones and celebrate accomplishments. Reward success. Provide feedback and coaching in the moment–both good and bad.
The team that feels like they are part of something are no longer a collection of individuals.
In the end, when you are assessing your DevOps practice and culture, make sure you take a long hard look at the people. They are truly the backbone of any DevOps practice. The need to technical prowess and experience should not be underscored, however, make sure you look for someone who has those skills plus all of the necessary attributes of a true leader. The success of your DevOps practice may depend on it.
