devopsy.kinsta.cloud

Category: Blogs

  • Seven Effective Tips to Get the Most Out of your DevOps

    Seven Effective Tips to Get the Most Out of your DevOps

    It seems that every firm is trying to include DevOps in their software delivery cycle these days. This is understandable. According to the 2019 Accelerate State of DevOps Report, companies using DevOps are more likely to achieve elite performance in software development and delivery.

    Companies such as Google and Amazon can be deploying thousands of updates a day. These rapid updates improve company performance and result in the provision of better software solutions for the end-user. 

    In this article, we will look at seven effective tips to get the most out of your DevOps so you can maximize success. Before we look at some of these tips, let’s briefly define DevOps.

    What is DevOps?

    DevOps is a term for a group of management practices that aim to quicken the deployment of changes to a system while ensuring high quality delivery. It incorporates management theories such as agile infrastructure and agile development. One of the most influential books on the topic, which resonated with a lot of influential business leaders in Silicon Valley, is “The Lean Startup” by Eric Ries.

    At its heart, DevOps boils down to the idea that you should be able to implement changes quickly, so you can release a superior product to your customers faster. This ability to bring a superior product to market fast, gives companies that can achieve this edge an advantage over their competitors.

    Review Your Organizational Structure

    Collaboration is the bond that allows software companies to deploy and execute changes fast. Companies that are experts at DevOps classically reduce the obstacles to collaboration.

    In traditional IT organizations, departments are siloed. They interact with each other only when necessary. Hence, they have limited knowledge of each other’s work. This model presents a linear process of developing, testing and deployment. 

    To reduce these silos, you need to first analyze your organizational structure, and identify obstacles to collaboration. This starts by ensuring management is on board with your vision. You need to:

    • Review the job description of managers and create KPI that support collaboration.
    • Create remuneration packages that foster closer collaboration between departments.

    Addressing structural issues that impede collaboration between departments will help. You also need to put in place a system of communication between departments.

    Fortunately, there are useful tools to smoothen continuous collaboration between different lines of business. Channels of communications should not be limited to sending emails or conference calls. Remember, DevOps requires continuous and fast collaborations to ensure fast deployment.

    Using unified communications (UC) is an effective way to foster collaboration in a company. Unified communications integrates phone calls, email, chat, document sharing, conferencing and other platforms necessary to exchange information within the organization. Importantly, you can also access much of this information in one place.

    Reduce Operational Hurdles for Deployment

    The way an update is deployed will differ by company. There is usually a system in place to discuss targets and a series of checks in place before deployment. For example, department heads will come up with an idea. They will be developed, then reviewed by quality assurance before operations deploy it.

    This can be a cumbersome and time-consuming process, which slows down the speed of development. The problem for many companies is that changes, both large and small, need to go through the same process of quality assurance before deployment.

    Source: Tech Beacon.

    While it makes sense to have this process in place for major updates, it should be possible to push smaller updates faster. Two things need to happen to achieve this:

    • Empower staff to make decisions independently while giving them the opportunity to learn from mistakes without risk.
    • Reduce the administrative hurdles to deployment by creating a ranking system for reviewing updates. This will allow developers to push small updates independently, or with minimal oversight.

    Empowering staff to push changes enables you to speed up DevOps.

    Plan in Small Batches

    Planning in small batches is important in promoting DevOps in an organization. With small batch testing, the cycles of testing hypotheses, acquiring feedback and going at it again can be done within a very minimal time frame. It is fast and efficient—simple. 

    This feedback might feel obvious. Perhaps the better question to consider when planning in small batches is: How small is small? Let the INVEST concept in agile development help you out.

    • Independent: Planned small batches should be independent of other parts.
    • Negotiable: Every batch should be iterable and negotiable upon receipt of feedback from deployment.
    • Valuable: Each one should add value to the project and the end-users.
    • Estimable: Batches should contain estimable information for scope determination.
    • Small: Small should be small enough to be processed within a few hours up to a few days.
    • Testable: A planned batch should be testable and can be monitored continuously.

    Results of deployment, testing and monitoring of each batch should then be deployed at least once a day. Also, in planning small batches, different departments should be kept in the loop. This will greatly benefit DevOps efforts in the organization.

    Create Space for Collaboration

    Even without DevOps, fostering collaboration between departments is imperative. You can formalize collaboration through regular meetings. However, departments are usually situated in separate office areas, floors or even different buildings. Think about having to walk to a conference room on a different side of the facility just for a meeting.

    This kind of setup is an example of a bottleneck for collaboration in an enterprise. And with collaboration being the heart of DevOps, creating a company culture and office space conducive to teamwork is of utmost importance.

    SpaceX is an example of a company that promotes collaboration throughout the organization through company culture and physical space management. Offices are open, and each person can see what everyone else is working on. This promotes motivation for employees to work passionately. Also, everyone has access to camera feeds from different parts of the factory, which inspires curiosity about what others are doing. Because of this culture, employees become dedicated and excited to accomplish results.

    Take Advantage of Microservices

    In processing complex IT systems, it is more beneficial to break bulks to smaller, more digestible pieces. This is where microservices come in.

    Microservices or microservice architecture is an architectural style wherein a complex system is broken down into smaller services. Rather than treating a system as a whole, it is perceived as a collection of manageable pieces.

    Teams simultaneously working on parts of a system as a whole makes it more prone to breakdown. Utilizing microservices makes monitoring, maintenance and testing easier. They are also loosely connected to each other which means fixing wouldn’t much affect the other parts. Moreover, each microservice is releasable independently.

    As an example, Wix started out with a monolith which was useful at first when they were only handling 1 million sites. However, interdependencies and poor quality coding resulted in downtime which is ultimately bad for business. Now that they shifted to using microservices architecture, they quickly scaled up reaching 70 million sites.

    Maintain Continuous Monitoring

    DevOps promotes fast and continuous integration and deployment. A fast-paced environment is more susceptible to problems. Something at any point can fail within the software development life cycle. Continuous monitoring helps you identify mistakes fast. It ensures the optimal performance, reliability and health of the software.

    Setting monitoring targets, like development milestones, can improve this process. A development milestone is an overarching category that can give an insight into your team’s actual performance. Monitoring tells you the performance of your DevOps implementation.

    A method to consider is synthetic monitoring wherein your application undergoes simulated customer interaction. This way, you can monitor the exact action and performance of your software before they get released to actual users.

    Practice Change Management

    DevOps employs continuous integration, continuous deployment and continuous monitoring. It is a cycle of fast-paced changes. Change management is essential to keep the system run like clockwork.

    It might be easier said than done because changes have effects on various stakeholders. This includes your customers as the end-users, security, regulatory compliance and company policies. With this, changes within the development, testing, deployment and monitoring should be cohesively managed.

    Change management is intrinsically reactive. It usually happens because of feedback received after deployment. But with DevOps, change management should be proactive. Instead of managing large-scale changes, processes are broken down into smaller parts with clearly defined scopes. Furthermore, anticipated changes are scheduled, scaled and communicated across the board. Possible risks and problems are efficiently managed because of this.

    Summary

    No system is perfect nor exactly the same. The key to a successful DevOps implementation is assessing what works best for your organization. DevOps is not a standardized process. It is not a system that will work for every company. Rather, DevOps is a theory of management.

    Improving DevOps at your firm will involve a system of continuous review and improvement. You need to understand what is working, the problems and what you can improve. This can all be done through continuous processes with the help of automation and collaboration. No matter the size of your software development projects, these simple tips and tricks will help you make the most out of your DevOps.

    — Sam O’Brien

  • Shift-Right Testing: The Emergence of TestOps

    Shift-Right Testing: The Emergence of TestOps

    The concept of shift-left has been a popular trend in continuous testing practices for a while. We are now beginning to see shift-right practices as an emergent trend in testing. 

    Shift-right entails doing more testing in the immediate pre-release and post-release phases (i.e. testing in production) of the application lifecycle. These include practices such as: release validation, destructive/chaos testing, A/B and canary testing, CX-based testing (e.g. correlating user behavior with test requirements), crowd testing, production monitoring, extraction of test insights from production data, etc. 

    Shift-right not only introduces such new testing techniques, but also requires testers to acquire new skills, make aggressive use of production data to drive testing strategies and collaborate with new stakeholders, such as site reliability engineers (SRE) and operations engineers. 

    In shift-right of testing, and in the increased collaboration with operations disciplines, we see the evolution of a new discipline in DevOps that we call TestOps. 

    In this blog, we will discuss the various shift-right testing trends and practices, what’s driving them, the new skills and collaboration required so that TestOps benefits can be realized. 

    Emerging Trends in Shift-Right Testing 

    In a recent survey of continuous testing conducted jointly by Capgemini and Broadcom, testing in production was ranked as the number one practice (with 45% respondents) either currently implemented or planned (see Figure 1). In addition, 39% of the respondents mentioned the use of analytics from operations to determine or optimize test coverage.

    Figure 1.

    When asked how customers measure the effectiveness of continuous testing processes, production data and user feedback and adoption of new functionality were ranked number one and number two respectively (see Figure 2).

    Figure 2.

    Finally, in response to how customers test their digital applications to ensure that they meet real world requirements, almost all of the responses were related to production data (see Figure 3). Use of production data for designing test strategy and chaos testing emerged as new methods compared to similar survey done a year ago.

    Figure 3.

    The survey data clearly indicates that customers are actively practicing (or considering adoption of) shift-right testing techniques. This report has a chapter (with my contributions) dedicated to shift-right that discusses the techniques and recommendations for its adoption, as well as relevant customer case studies.  

    What’s Driving Shift-Right Testing Trends? 

    The goal of shift-right testing is to ensure correct behavior, performance and availability over the production of an application.

    There are several drivers for adoption of shift-right testing trends.

    Customer Experience (CX) Is a Key Quality Metric for Digital Applications

    Unlike classic testing, this takes into account real-world users and their experiences. An application with perfect traditional quality scores (such as FURPS) may still suffer from poor CX if it fails to delight the customer. CX is measured using various metrics such as Customer Effort Score (CES), Net Promoter Score (NPS), Customer Satisfaction Score (CSAT), etc. While it is possible to shift-left some of these measurements to some degree, most CX measurements are deduced from systems in production (or close to production). Examples of CX-based testing include the following: 

    • Requirements validation based on real user journeys, behavior and feedback.
    • Deriving functional and performance test scenarios based on the above.
    • A/B testing and canary testing, to experiment how well customers like (or dislike) changes. 
    • Crowd testing, to better understand real-world user experience.

    In a World of Agile Delivery, It may Not Be Possible to Test Everything Before Release to Production

    Time to market (or lead time for changes) is a top business imperative for digital applications, which is why QA/testing is considered a major bottleneck in continuous delivery. While it is possible to optimize testing effort and cycle time using shift-left practices (such as model-based testing, change-impact testing, test process and execution automation etc.), those approaches may still take too long (or too much effort) during the shrinking release cycle time. In some cases, the exact usage patterns may not even be fully understood before release. The idea is to learn usage patterns from production and use it for better testing strategy within shift-left. 

    Complex and increasingly distributed systems using microservices (using thousands of components) and cloud-native techniques that allow releases at a very granular level make it more and more difficult to test fully in pre-production environments. This is complemented by the fact that since such releases are so granular, they may be backed out easily (rollback) in case they cause problems.

    Hence, enterprises test to assure “good enough” quality (to ensure timely release) and rely on fast remediation (or rollback) to address defects or problems that arise.

    Also, CX measurements (as described above) often provide real-user feedback in shades of gray, unlike the black-n-white pass/fail measurement of classic testing. Which means it is not possible to perfect pre-production test suites to ensure thorough CX coverage. 

    A Hundred Percent Reliability (or Quality) Is Often an Unrealistic Goal

    Related to the above, one of the key principles we learn from the DevOps discipline of SRE is that 100% reliability is not only unrealistic, but often too expensive to accomplish. SRE establishes the concept of service level objectives (SLOs) and error budgets to quantify acceptable risk tolerance in production systems. The same principle applies to testing and overall quality. See my related blog for more on this subject

    Some Validations Are Difficult to Run in Test Environments

    These include large-scale performance tests when the test environments are not appropriately sized (or configured) relative to production environments. 

    Another example is destructive or chaos testing. While it is possible to run isolated chaos testing in test environments by using techniques such as service virtualization (to simulate failure in dependent components), it is difficult to execute for large-scale destructive tests. For example, Netflix does a significant amount of their testing in production.

    Similarly, enterprise monitoring to gather insight about real-world usage patterns and failures is only possible in production. Though we advocate turning on monitoring in test environments (as part of shift-left), such monitoring only helps to do localized monitoring (e.g. of the specific app or system being tested), and that too under test conditions. However, performance test scenarios that have been developed by testers can be re-used in production (shift-right) as synthetic monitors to measure application performance and regression.   

    Lastly, as an extension of monitoring, testing of some AI-based applications may be difficult without access to production data. Some machine learning algorithms, for example, need to be constantly refined based on real world data. While these algorithms are developed using a limited set of training and testing data, they need to be monitored and tuned, based on its performance in production. 

    Developers and Testers Need Easier Closed-Loop Feedback from Production

    Easier closed-loop feedback from operations allows developers and testers to better understand the behavior of the app, predict the success/failure of releases and respond to production incidents (e.g. reduce MTTR).

    Interpretation and use of voluminous production data is often difficult and tedious for development and test teams. Use of shift-right techniques allows developers and testers to access production data more easily in a form that is easily consumable and actionable–by leveraging the right insights from AIOps disciplines. This data collaboration is a key tenet of the DevOps paradigm. For example, synthetic monitors (which are typically created by developers and testers in the dev/test stages) provide means for developers to get monitoring feedback in a form that they are familiar with. 

    In addition, automated root-cause analysis of defects/incidents in production can provide developers and testers insight on failure patterns and trends, their sources (when production data is correlated with development data) and what’s causing them, and how they may be better prevented. 

    Similarly, data usage scenarios (e.g. from database logs and RR pairs from service transaction logs) can be used for generation of relevant test data.  

    In essence, shift-right testing practices are necessary to create a continuous feedback loop from real user experience with a production application back to the development and testing process. 

    The Evolution of TestOps Discipline

    It is clear from the customer survey data and drivers discussed above that there is an evolution toward shift-right practices. 

    Just like we did with shift-left testing, in order to make these practices sustainable and to realize the benefits, we need to also evolve tester use cases, skills, tools and collaboration patterns with people and disciplines to the right of the application lifecycle, i.e. operations. In other words, a new discipline needs to be established—we call this new discipline TestOps—a sub-discipline within the broader DevOps context. 

    Though the term TestOps (like all the X-Ops sub-disciplines within DevOps) implies the collaboration between testing and operations, it isn’t simply about shift-right. Why? Because with DevOps, operational disciplines themselves have shifted left (such as shift-left of monitoring, configuration management, SRE, etc.). Hence TestOps—true to the principle of continuous testing—refers to better collaboration with operations disciplines across the DevOps lifecycle.  

    Let’s consider some of the key practices in TestOps shift-left (see Figure 4). 

    Figure 4.
    • Early reliability testing: this includes static scalability analysis of the system architecture, as well as performance and stress testing.
    • Configuration quality and testing: this includes testing of “as-code” (anything “as-code” can and should be tested as such, just like application code) environment and deployment configurations, as well as deployment testing (to test the correctness of deployments) and rollback testing (to ensure that rollbacks can be done quickly and successfully).  
    • Early monitoring: includes monitoring of systems early in the lifecycle (e.g. in development and test environments) as well as development and testing of synthetic monitors before they are deployed to production.
    • Early AIOps insights: this includes early use of AIOps techniques (in the dev/test environments) to glean testing and quality insights, for example for defect and release risk prediction.

    Below are some of the key TestOps shift-right practices (see Figure 5).

    • Chaos engineering: this includes testing the reliability of the system through the injection of controlled failure scenarios.
    • A/B testing: consists of a randomized experiment with two (or more) variants to study which are more effective with users.
    • Canary releases: consists of techniques used to reduce the risk of introducing a new software version in production by gradually rolling out the change to a small subgroup of users, before rolling it out to the entire platform/infrastructure and making it available to everybody.
    • Synthetic monitoring: includes monitoring techniques using an emulation or scripted recordings of transactions (that reflect typical user behavior).
    • CX testing and analytics: includes extracting of CX data to understand customer experience as well as to derive insights from it (such as customer problems, new feature requirements, etc.).
    • Other testing in production: includes performance tests and testing of AI-based applications.
    • Testing insights from AIOps data: as described above, a rich variety of testing insights can be gleaned from operations data.
    Figure 5.

    Key TestOps Skills and Tools

    Let’s consider the new skills required for the key TestOps practices. This includes the following: 

    • Data analytics skills: Since most of the production data is both voluminous, diverse and often temporal, testers must now pick up data analytics so that they quickly glean insight from such data, as well as perform correlations with other datasets to identify actions. Advanced analytics skills (such as predictive analytics or machine learning) are also necessary to predict events (e.g. release quality). While these analytics skills are well understood within the operations community (with the increasing adoption of AIOps), these are relatively new for developers and testers. 
    • CX skills: As we discussed, CX is now considered a key metric for measuring quality in production. New types of CX testing include A/B and canary tests as well as crowd testing. Like other operational data, CX data is also typically voluminous and often unstructured. While CX teams specialize in such principles, testers need to acquire sufficient skills to glean insight from the CX process and data, and collaborate effectively with such teams.
    • Monitoring and operations skills: Understanding of monitoring principles (e.g. creation, testing and deployment of monitors and use of data from monitors) is a key skill required for TestOps. Similarly, testers must also understand operations principles (incidents, failures, alarms, MTBF, MMTR, configuration definitions, etc.) so that such information can be leveraged for testing purposes as well as for collaboration with ops teams.
    • Reliability skills: Reliability is also another key operational quality attribute that testers need to be familiar with. This includes chaos/resiliency testing, deployment and rollback testing, and configuration testing. The discipline of SRE has evolved within DevOps to address reliability, and many of its principles overlap with TestOps. 
    • New tools skills: In addition to using traditional testing tools, TestOps also requires the use of additional tools such as monitoring, data analytics (for production and CX data), CX testing (such as A/B and canary testing) and reliability testing. Technology innovations that support testers with these new capabilities include RunScope (a lightweight API monitoring tool that can be used for monitoring both in test and production environments), Blazemeter (for performance testing for CX and reliability), Optimizely for A/B testing. For data analytics, technologies such as automation.ai simplify the work of data aggregation and modeling so that testers can more easily extract actionable insights from data collected across the lifecycle. 

    Key TestOps Collaborations

    In addition to new skills, testers must now learn to collaborate better with additional stakeholders as alluded to above. These include: 

    • Collaboration with data science and AI/ML engineers: This is necessary for testers to tweak or build out new algorithms and models required to perform the analytics needed for TestOps.
    • Collaboration with CX engineers and teams: This is necessary for testers to monitor the appropriate CX metrics and data, and to study the impact of their testing/QA activities on CX.
    • Collaboration with Operations engineers and teams: This is fundamental for TestOps to not only support all the shift-right testing activities, but also to gain access to production data that is required for TestOps analytics. This is analogous to how testers collaborate with developers as part of shift-left. 
    • Collaboration with SREs: As mentioned, there is a fair degree of overlap between SRE and TestOps principles, and hence collaboration with SREs is key for testers. See my blog on how testers and SREs can work better together to improve reliability. 

    Conclusion

    Hopefully, this article has provided readers with some insight into the emerging discipline of TestOps. As we build increasingly distributed microservices and cloud-native systems, and as the discipline of SRE matures, we expect to see more of shift-right testing practices that enrich TestOps.

    — Shamim Ahmed

  • GigaSpaces to Provide Free Go-Live Services

    GigaSpaces to Provide Free Go-Live Services

    GigaSpaces, a provider of a platform for processing analytics data in real-time, will be making available professional services for free during the duration of the COVID-19 pandemic as part of a Go Live initiative to reduce the total cost of acquiring and implementing its platform.

    Yoav Einav, vice president of product for GigaSpaces, said the economic impact of the COVID-19 pandemic shows it’s apparent organizations of all sizes will need to reduce their IT costs. At the same time, many organizations will need to keep IT projects moving forward as nascent digital business strategies now become business continuity strategies, he said. Most of those business continuity plans will be contingent on providing a better digital customer experience based on real-time analytics.

    GigaSpaces has been building up its professional services capability for the past several months and plans to tap into the expertise of third-party IT services providers to make sure there is enough available expertise in the right place to meet anticipated Go Live demand.

    GigaSpacesIt’s still unclear to what degree IT projects will be impacted by the COVID-19 pandemic. Undoubtedly, many IT projects are being paused or outright canceled, especially in vertical industry segments that have been among the hardest hit. Organizations that are moving IT projects forward, however, should expect to IT vendors to help reduce the total cost of implementing those projects by providing free additional services, as in the case of GigaSpaces, or at the very least much more flexible payment terms.

    There are, of course, some vertical industry segments where organizations are building new applications that are highly relevant. One GigaSpaces customer, for example, is racing to build an application that will make it easier to find critical healthcare supplies that are frequently out of stock when needed most, noted Einav. Other organizations are simply moving to reduce costs by offloading queries from traditional databases.

    In many cases, organizations will be evaluating which platforms to employ to build these applications not just on the raw performance attributes of the platform but also how quickly those platforms can be implemented. As such, the robustness of the IT services ecosystem, including its overall DevOps acumen, will become a more critical evaluation criterion.

    In the meantime, a major transition away from legacy batch-oriented applications toward applications that process data in near real-time that has been gaining momentum now for several years should continue. Most organizations are now going to require the sub-second performance of a financial services application. However, visibility into supply chains in near real-time is becoming a bigger requirement, especially when many of the organizations that make up the supply chain are potentially not as economically resilient as they once were.

    Less clear is the rate at which the transition to near real-time applications will now occur. However, at a time when competition between organizations is about to become more fierce than at any time in recent memory, many organizations are about to discover how big a difference modern applications can really make.

    — Mike Vizard

  • Filling the Skills Gap for Effective DevSecOps

    Filling the Skills Gap for Effective DevSecOps

    With the rise of DevSecOps comes a whole new need for training and upskilling. It isn’t a secret that the cybersecurity skills gap will only continue to grow. The security skills gap combined with the rise of DevSecOps has introduced hidden challenges organizations must address.  

    The Rise of DevSecOps 

    While DevOps accelerates agility and scalability of organizations, it may also expand risks of threats—particularly if security is an afterthought in an organization’s cloud strategy. In a recent survey, 52% of companies admitted to scaling back security measures to meet a business deadline or objective. Similarly, 68% said their CEOs demand that DevOps and security teams prioritize accelerating business processes. 

    Even when organizations race to release updates despite security consequences, the threat landscape is becoming more complex, faster and automated. In the third quarter of 2018, FortiGuard Labs noted almost 34,000 new malware variants—a 40% increase over the second quarter and a 126% increase over the first quarter. Cyber criminals now use advanced technologies such as artificial intelligence (AI) and swarm technology—as well as DevOps itself—to create single-use malware that targets a particular organization across several points of the attack surface. 

    Thus, companies have realized that security must be woven into the DevOps approach–hence the birth and rapidly increasing adoption– of DevSecOps.

    The Security Skills Gap

    At the same time that organizations are embracing DevSecOps, the paucity of skilled IT talent persists. A report from 451 Research on the IT skills shortage notes organizations are facing talent gaps across a range of IT specialties, including database administration (31%), general network administration (36%) and server/systems administration (43%). 

    The dearth of seasoned cybersecurity professionals looms even larger, with an estimated skills gap of just under 3 million individuals. Understandably, 53% of respondents to a report from ESG admitted to a significant shortfall of cybersecurity skills at their organization. These figures represent the general cybersecurity personnel needed to support and secure traditional network environments, to say nothing of DevOps.

    Overcoming the Hidden Challenges

    In DevOps, any security implementation that hinders speed will be seen as a threat to their primary objectives. Traditional IT teams and DevOps teams are often at odds here. IT will suggest the use of security tools that DevOps sees as causing bottlenecks, which runs counter to their primary objectives. However, while DevOps may be highly proficient at building applications, they often lack the expertise and skills to do so securely.

    An answer to this challenge is to add a cybersecurity specialist to each DevOps team to create a DevSecOps team. This DevOps security specialist (or team of specialists) can guide application developers through the shared responsibility model, helping them stay on track with both development and security requirements. They’re also there to provide strategies for consistent security policies across and between all their cloud workloads and services, all while protecting the DevOps mission of reliability and high performance.

    With DevSecOps in place, the team can choose, implement and manage tools that will better equip them to meet the goals of speed and security. Take, for instance, the use of security solutions that are offered as a service (SaaS) or web application firewalls, which can auto-scale. That allows publicly facing web apps to grow as needed, without compromising security. The right tools can also be eased into deployment with minimal effort. Some even have built-in functions that cover security during deployment, maintenance, scaling and for all the fine-tuning that needs to take place throughout ongoing use and development.

    Once an organization has augmented DevOps with DevSecOps, teams can integrate security from day one of each new project. DevSecOps team can also develop the necessary cloud security playbooks and ensure that those guidelines are followed. DevSecOps can even directly affect the bottom line when they help prevent violations against regulatory requirements and the fees and penalties that come with them.

    Closing the Gap

    The fact remains, though, that good skills are hard to find. This makes it harder to have a DevSecOps team in place. Applications being built in or migrated to the cloud need to be protected against new threats that propagate across workloads—and from cloud platform misconfigurations at the user interface and application programming interface levels. 

    Addressing this challenge requires a concerted effort on the part of both the private and public sectors to develop just-in-time training and education programs to develop cybersecurity skills that are critical for every organization, and furthermore educate for cloud security and DevSecOps practices. More and more organizations are in need of training programs that focus on the cloud security skills necessary to upskill their current and future workforce.  

    Failure to understand and implement a security strategy can make an organization susceptible to policy and enforcement gaps, as well as risk to business continuity and digital operations, which can determine whether or not your organization is able to thrive in today’s digital landscape.

    Proactive DevSecOps

    As organizations evolve from DevOps to adding a DevSecOps practice, the talent shortage continues to loom large. This shortage could pose significant security risks to your organization. But, you can play an active role in bridging the skills gap by identifying training programs and certifications available for your teams focused on security. Employees completing these trainings and earning relevant certification can then work with IT security and DevOps teams to provide the security that enables the speed you need to remain competitive.

    — Lior Cohen

  • CircleCI Surfaces More DevOps Insights

    CircleCI Surfaces More DevOps Insights

    CircleCI today announced it is making it easier for DevOps teams to determine the amount of resources being consumed by its continuous integration/continuous delivery (CI/CD) platform to help make application development teams more efficient.

    Kunal Jain, senior product manager for CircleCI, said via the application programming interface (API) that CircleCI exposes DevOps teams can now determine, for example, which workflows are taking the longest to run or what tests might be creating bottlenecks into the CI/CD pipeline.

    Armed with that insight DevOps teams can also optimize how credits to employ the CircleCI platform are allocated across their DevOps teams, added Jain. That capability should make it easier for DevOps teams to predict how much they are spending per month to run their CI/CD platform, said Jain. Many more IT organizations may opt to access CI/CD platforms using a software-as-a-service (SaaS) approach that limits the need for them to invest in their own IT infrastructure or in some cases even manage the platform.

    DevOps teams have always been sensitive to efficiency but given the current economic climate, minimizing costs is becoming a higher priority for many organizations. As such, DevOps teams need more insights into, for example, how to best break up workflows to keep CI/CD platform costs to a minimum, said Jain. In fact, Jain says the total cost of operating a CI/CD platform is likely to now play a bigger role in selecting a platform as IT leaders demand more visibility into DevOps processes.

    Of course, some DevOps teams may find themselves competing for credits based on how efficient they are. Teams who are more efficient than others could also allow IT leaders to reallocate credits as they best see fit based on the progress being made. In a previous era where compute resources were largely considered to be a nominal cost, many DevOps teams may find access to IT infrastructure might no longer be taken for granted.

    CircleCI claims it runs more than 30 million independent jobs run every month. The company last year committed to exposing more API endpoints to help DevOps team better understand the languages, architectural patterns, code testing patterns, delivery success rates and team practices being employed. Like most providers of CI/CD platforms, CircleCI has made a commitment to making available a much broader range of actionable insights to DevOps teams via its API.

    At this point, it’s unclear what the impact the economic downturn brought on by the COVID-19 pandemic might have on DevOps adoption. Undoubtedly, many organizations will put a hold on some application development projects as they reassess their priorities. On the other hand, with more people working from home, digital business transformation initiatives that require new applications are likely to become a higher priority as organizations seek to reinvent how they engage their end customers. Those projects may even be accelerated as part of a larger business continuity strategy. Regardless of the number of application development projects being launched, the scrutiny of how IT resources are being allocated and consumed is about to increase substantially.

    — Mike Vizard

  • Bridging the Network Automation Skills Gap

    Bridging the Network Automation Skills Gap

    With the introduction of network programmability and APIs for everything from management systems to devices themselves, the modern network landscape has evolved to being managed like software. This shift from hardware-focused networks to software-centric functions has had a profound effect on network management techniques and skillsets required to keep pace with the changing ecosystem.

    A recent EMA survey notes that skills gaps associated with network automation are an issue for 96% of enterprises. Even respondents with significant network automation initiatives in place felt their solutions stretched the abilities of their network teams. Network automation can expose skills gaps where network engineers might lack experience with new tools and software development. At the same time, application development teams, who likely would be included in building an in-house network or automation software solution, will also lack networking skills. Hybrid skills with both development and networking expertise are rare and available only at a premium.

    This journey to network automation can be difficult, especially given the required technical skills that are not in abundance within many organizations. Simplifying network automation and empowering a broader set of participation through capabilities to easily create and manage automation workflows presents a key opportunity for organizations to not only mature efforts around network automation, but to also execute against broader business digital transformation objectives.

    Four Keys to Democratizing Network Automation

    The key to democratizing network automation and bridging the skills gap is to provide NetOps teams with the ability to easily design and build network automation without having to re-tool, build custom code or learn specialized software skills. There are four key elements required to effectively democratize network automation in the enterprise.

    Easy Onramp for Non-Developer

    Automation is typically accomplished through writing code in a particular programming language. For a person without any programming skills (or even minimal skills), they must first learn how to code before they can even attempt their first automation task. This is very difficult in terms of time and interest. Additionally, many network engineers see the amount of work it would take to learn to write code, and become disinterested. There has to be a new set of tools and methodology to empower these people to automate their work without the long road of learning to become programmers. It’s important to invite as many people to the automation table as possible including IT, security, cloud and NetOps teams, to streamline business processes.

    Capture Network Tribal Knowledge

    Within the organization, there is already existing embedded tribal knowledge that is incredibly valuable. By harnessing and transferring that knowledge into an automation workflow without a lot of programming, it provides an accelerated path to network automation. The goal is to leverage traditional network automation and management practices (Scripts, CLI) for re-use across the organization.

    Network engineers with years and years of experience have a valuable set of processes and procedures they’ve built up inside their heads, and this wealth of information is a valuable asset. Unfortunately, this information is rarely documented and is normally passed on by activity and word of mouth. This is the type of tribal knowledge that needs to be captured, preferably through automation workflows that can be used in a task, documented in the workflow and modified over time to improve it or change it as the infrastructure changes.

    Integrate DevOps Principles

    Augment network automation with CI/CD pipelines and API services. These are well known solid principles that are already used in other domains outside of networking and pay big dividends when applied to the automation of complex networks. When we look along the horizon of the network automation journey, these same CI/CD practices will be adopted by future networking teams and allow them the same advantages of reliability and velocity within the network.

    Enable Cloud Architect Participation 

    It is important to have networking people that can understand the networking constructs of the cloud, as well as cloud teams that understand how it integrates and operates within the network. Enabling cloud architect participation in network automation will allow for the real-time feedback and control of cloud-based applications and infrastructure critical to modern networks.

    Cloud networking is increasing in complexity and will soon require the skills and experience of network engineers to help solve the problems of cloud networking. Traditionally, cloud and networking teams have been separated because their technology domains have been siloed and there was very little interaction required between the teams. As technologies like SD-WAN have been adopted, the line of responsibility between cloud and network teams has blurred. Cloud and networking teams will need an automation platform that can be used by both to create workflows and automate between both domains. Business technology decisions are forcing these two groups together, and they must have a way to use their own tools, but also have a platform to collaborate automation between them.

    As applications and services are becoming more complex, distributed and require connectivity and policy enforcement across a diversity of domains (such as cloud, SD-WAN, data center, wireless and network applications), the management of these network concepts requires us to re-think how we have traditionally managed networks. As network automation plays an increasingly larger role in enterprise networks, a byproduct of the complexity is a skills gap that threatens to slow or stymie enterprise initiatives.

    By making the onramp to network automation easier for non-developers through capturing tribal knowledge, integrating DevOps principles and enabling cloud architect participation, enterprises better ensure a successful migration to network automation across multiple domains.

    — Rich Martin

  • Comparing Three Approaches to Multi-Cloud Security Management

    Comparing Three Approaches to Multi-Cloud Security Management

    Moving to a multi-cloud environment can bring many advantages, but it also brings huge challenges in managing both the everyday operations and security of multi-cloud infrastructures.

    To meet these challenges, several approaches to multi-cloud management have arisen. The most common can be roughly grouped into three categories: Cloud Management Platforms (CMPs), Infrastructure as Code (IaC) and cloud-native abstraction. The differences between these approaches reflect the differences between top-down and bottom-up cloud management, as each management process suits a different set of companies.

    In this article, we’ll take a look at all three approaches, explain what the advantages and disadvantages of each are and then explore whether new approaches to multi-cloud management are required.

    CMPs

    CMPs are the highest level approach to cloud management—the approach that is furthest away from the native code that your clouds run on. The rise of CMPs has come as the result of a simple observation: public cloud services are redundant and overlapping, and can therefore be managed through an abstraction layer that hides the complexities of what is happening at a cloud-native level.

    By using a CMP, admins do not need to understand the differences between AWS and Microsoft public clouds (for instance), but instead use a consistent interface to manage both.

    This has some huge advantages, primarily when it comes to improving cloud security. CMPs allow IT operations teams to implement a common security layer within multi-cloud environments, and apply the same identity and access management (IAM) processes across all the clouds they are working with. 

    This is particularly useful with cross-cutting applications that interfaces with multiple, critical portions of your cloud storage to help simplify, and which requires highly complex user access policies. This is ultimately one of many reasons why security breaches of cloud-based data are so rare, as 99% of businesses in the UK that rely on cloud-based software have never experienced a security breach. 

    IaC

    IaC is a second approach to multi-cloud management. This approach arose in response to utility computing and second-generation web frameworks, which gave rise to widespread scaling problems for small businesses.

    Administrators took a pragmatic approach: they modeled their multi-cloud infrastructures with code, and were therefore able to write management tools that operated in a similar way to standard software.

    IaC sits in between the other approaches on this list, and represents a compromise solution. It gives more fine-grained control over cloud management and security processes than a CMP, especially when used in conjunction with SaaS security vendors whose software can apply a consistent security layer to a software model of your cloud infrastructure. This is important because SaaS is growing rapidly in popularity, with 86% of organizations expected to have SaaS meeting the vast majority of their software needs within two years. 

    On the other hand, IaC requires a greater level of knowledge and vigilance than either CMP—or cloud-native approaches. If admins are knowledgeable enough to write code that can accurately (and reliably) model multi-cloud infrastructures, IaC can be a powerful way to improve DevOps security. These software models, however, will require constant monitoring to ensure that changes in the cloud-native layer do not introduce vulnerabilities into management tools.

    Finally, IaC can be significantly more efficient, when it comes to computing and network resources, than a CMP. Because IaC software runs closer to the cloud-native layer than (most) CMPs, it can execute common tasks much more quickly.

    Cloud-Native Abstraction

    The final approach to multi-cloud management is to manage your environment using only cloud-native tools. This approach has grown in popularity recently, as developers grew increasingly comfortable with working with containers and Kubernetes directly. Cloud storage providers have responded by offering increasingly complex tools designed to work directly on the cloud-native layer, and the storage provider you use should come with these tools.

    Cloud native application management has several advantages, and several disadvantages. The most obvious advantage is that cloud-native tools give admins a much greater level of control over their multi-cloud environments than the other approaches on this list. Because these tools run natively, they are also typically faster and more efficient than the other approaches we’ve mentioned above.

    On the other hand, saving computing time comes at the cost of increasing the workload for administrators. If you are working across a large number of clouds, managing each separately using each cloud’s native applications can quickly turn into a full-time job. In addition, managing multiple and complex access and usage policies at a cloud-native level can make it difficult for admins to keep on top of how their IAM processes are running, and increases the risk of mistakes. 

    Is It Time For a New Approach?

    All of the approaches above are a different solution to solve the same problem, namely the challenges caused by rapid adoption of multi-cloud technologies. Each, as we’ve seen, has a primary advantage when compared to the others. But in truth, none of the approaches above represents a completely adequate solution to managing security in multi-cloud environments.

    The choice of which approach to take must therefore be a pragmatic one. For companies who are blessed with the knowledge, talent and time to manage multi-cloud infrastructures at a cloud-native level, this is likely to be the best solution, especially at a time when cloud waste is booming. If you take this approach, however, you should also be aware that implementing it will require putting in place thorough and rigorous code auditing processes in order to ensure that any small IaC elements that run on top of cloud-native tools do not introduce common security vulnerabilities into your network.

    For small to medium size businesses, or those running hugely complex multi-cloud systems, CMPs will be a safer bet. The reduced efficiency of these services–both in terms of computing resources and capital investment–is likely to be more than made up for by the reduced load on the admins who spend their days working with them, and the reduction in the number of mistakes that are made.

    Ultimately, it’s apparent that a new approach to multi-cloud management is required. One that combines the flexibility and speed of cloud-native with the ease of CMPs. But at the moment, it’s difficult to ascertain what that will look like.

    — Gary Stevens

  • How to Secure Online Coding Platforms

    How to Secure Online Coding Platforms

    Threat Modeling for Online Coding Platforms

    The evolution of DevOps teams and a greater reliance on cloud-based computing has completely changed the coding process. Now, with Integrated Development Environments (IDEs), coding can be done entirely online. This is convenient, but are online IDEs secure? To answer this, we will focus on two popular cloud-based IDEs: AWS Cloud9 and Visual Studio Online.

    In online coding, the IDE is rendered inside your browser, the browser JavaScript engine uses WebSockets to initiate an Secure Shell (SSH) connection to your linked device in the background—e.g. virtual private server (VPS)—providing you with the familiar terminal interface to execute commands. The environment—meaning the linked cloud VM/VPS or SSH-accessible device—includes tool configurations (like which tokens are included, or cloud configuration files), a copy of the source code, compilers and other tools that you want to use.

    Figure 1. Local versus cloud-based IDE. Source: Trend Micro.

    The main difference between local and cloud-based coding platforms is where the bulk of the environment resides. In cloud IDEs, the environment is a virtual machine instance inside your cloud provider. In terms of security, this means you are delegating trust to the cloud provider, but you are responsible for preventing any backdoors from opening or introducing misconfiguration issues to the VM. In the case of using your own device with AWS Cloud9, you are responsible for configuring the device securely.

    Linked Devices/VMs Are Not Always Secure

    What is running on the backend often dictates whether linked devices and VMs are secure. As previously described, an SSH-linked device is necessary to use these online coding platforms. In the case of Visual Studio Online (still in preview mode as of this writing), we can find a Visual Studio (VS) Code server running on the machine.

    The code server itself is a Node.js application to which your browser will connect. You can also download the whole ~/.vscode-remote folder and run the server inside your local environment.

    As the owner of the linked device, you have permission to promote yourself to root and install or configure whatever you think you will need. You also have, by default, both Docker and Git pre-installed.

    In the case of AWS-hosted Cloud9, the situation is a little more complicated. The backend necessary for platform communication is located inside the linked device while the frontend remains hosted on a different location. But, like in VS, you can also promote yourself to root.

    This brings us to our first security concern: How secure or private are linked devices/VMs?

    The linked device contains confidential information—access tokens, application configurations, source code, etc., which should be protected against unauthorized access.

    By default, the platform is secured by the provider. However, you should keep in mind that, together with your root access availability, you are also responsible for preventing misconfiguration issues, especially when using third-party plugins. AWS does not provide plugin support for plugins not developed by them.

    There are possible misconfigurations you might make that can lead to security issues. For example, if you set your IDE to be accessible from outside, either intentionally for ease of sharing, or unintentionally, that change can have major consequences.

    In addition, it is not common for access tokens to be stored in encrypted token vaults. A lot of them can be viewed via plaintext configuration files. Without additional security measures, your access tokens can end up exposed to outsiders.

    Once unauthorized access is acquired by cybercriminals, they can compromise your code for their own gain. An example would be the supply chain attack against a software company that modified software updates with malware.

    Browsers Can Introduce Malicious Extensions or Vulnerabilities

    The next security concern is the browser itself. As online coding platforms are accessed from web browsers, you should be very careful when accessing these sites from public, non-domain, shared or unprotected computers.

    Malicious browser extensions are a well-known phenomenon. While an untrusted computer possibly infected with malware presents an obvious risk, it is also possible for an attacker to steal code using a malicious web browser extension.

    Code Extensions and Plugins Can Also Contain Malware

    The main advantage of Visual Studio Online and, generally, the VS Code platform, is the number of extensions available. This in and of itself is yet another possible attack surface.

    Let’s imagine a malicious VS Code extension—a useful-looking extension with an embedded backdoor. The lack of permission checking (such as disk access, network access, process access, etc.) for extensions during installation or use, becomes a security problem. The extent of security checks during extension publishing is limited to having a valid publisher ID and a few image-related restrictions. This means you have to trust the extension developer entirely.

    No Software Application Is Error-Free

    The ever-present possibility of vulnerabilities in any kind of software must be considered as well. In the case of online coding platforms, they can be affected by web-class vulnerabilities since IDEs are web applications. For example, a vulnerability that allows an attacker to execute their own JavaScript code can have control of the IDE or remote linked devices.

    A recent vulnerability affecting the Visual Studio Live Share extension, which is also available for VS Code, shows that we can expect to see another vulnerability affecting online coding platforms in the near future.

    Securing Cloud IDEs Is a Must for DevOps

    Threat modeling allows us to understand the different factors at play that can affect the overall security of a computing environment. Cloud IDEs should be no different from other software, if not more critical to secure.

    Here are some recommendations for each of the security issues we raised:

    • For linked device security — Install trustworthy software. Keep that software updated. Don’t indiscriminately open ports to the internet.
    • For browsers — Work inside trusted and secured environments. Try to avoid using shared computers. Install browser extensions from trusted vendors only.
    • For VS Code extensions — Avoid installing extensions from unknown sources or authors.
    • For vulnerabilities in general — Make sure your environment is updated to the most recent version.

    — David Fiser

  • Coronavirus Economic Shutdown: Analysts Hit the Brakes on Global IT Spending Estimates

    Coronavirus Economic Shutdown: Analysts Hit the Brakes on Global IT Spending Estimates

    By most accounts, 2020 was set to be a pretty good year for U.S. gross domestic product and enterprise IT budgets. That was before the novel coronavirus, and the subsequent economic shutdown that is designed to blunt its spread and avoid overwhelming the nation’s health system. Analysts are now slashing their economic growth rates and enterprise technology spending estimates.

    Previously, research firm IDC estimated that real GDP in the U.S. would reach 2.4%, but downshifted that to a “pessimistic scenario” of 1.4%. While IT spending dropped from an expected 5.1% year over year growth to a pessimistic scenario of 1.3%. The base scenario for GDP growth is now 2.0% and for IT spending 4.3%.

    “The pessimistic scenario is not a worst-case scenario,” said Stephen Minton, vice president in IDC’s Customer Insights & Analysis group. “Things are moving so quickly that we need to constantly recalibrate our assumptions and expectations, but the pessimistic scenario reflects an IT market in which weaker economic growth translates into weaker business and consumer spending across all technologies over the next few quarters. Things could get worse, but hopefully not.”

    Earlier this month, Forrester analyst Andrew Bartels wrote in this post that the U.S. and global technology market spend will slow to 2% this year.

    “That assumes the U.S. and other major economies have declined in the first half of 2020 but manage to recover in the second half. There is also a 50% probability that U.S. and global tech markets will decline by 2% or more in 2020 if a full-fledged recession hits,” Bartels wrote.

    Not all areas in technology will be impacted equally. Forrester expects there will increased demand for cloud infrastructure services, specialized software, communications equipment and telecom services for remote work and education.

    However, Forrester expects computer and communications equipment to potentially decline 5% to 10%, while technology consulting and systems integration services spending could be flat to down 5%, should companies really belt-tighten.

    “Software spending growth will slow to the 2% to 4% range in the best case and will post no growth in the worst case of a recession. Tech outsourcing and telecom services will hold up better, though contract revisions could cause spending to go down, as well,” stated Bartels.

    S&P Global Ratings also predicts slowing IT spend. According to its update, Global IT Spending Set To Slide As Coronavirus Hits Hardware Sales, S&P Global economists cut their global GDP growth rate by more than half to 1% to 1.5% from 2.8%—in just two weeks.

    “We believe the spread of the coronavirus will hurt enterprise and consumer IT spending across the globe with particularly bleak ramifications for the hardware and semiconductor segments. However, we expect some of the deferred spending to return gradually in the latter half of this year through heavy government stimulus in the U.S., China, and elsewhere,” wrote S&P Global Rating Analyst Andrew Chang.

    S&P Global Ratings anticipates IT spending to decline 3% year over year, compared their prior estimate of 2% to 3% growth. S&P Global Ratings expects some of the biggest hits to hardware, with smartphone revenue falling to negative 9% from up to 2% growth, and PCs to negative 9% from a previous range of negative 3% to a positive 4% growth.

    While enterprise technology budgets are being slashed, the need for business-technology isn’t going away. In fact, it’s accelerating as more knowledge workers shift to working from home, more remote collaboration and keeping legacy systems running and moving forward with digital transformation efforts.

    — George V. Hulme

  • How to Secure Your Kubernetes Cluster on GKE

    How to Secure Your Kubernetes Cluster on GKE

    Google Kubernetes Engine (GKE) is easy to get going with, but requires additional security controls. The documentation is hard to grasp as there are many features and changes that tie to specific Kubernetes versions that will require using beta feature enablement, as well as some out of date documentation that can catch people out.

    There is also an outstanding bug that we have raised on Kubernetes that is hit by GKE. This is an edge case situation if you end up running pods directly and not deployments, and only happens if you enable pod security policies (which we recommend doing for security reasons).

    If you are looking at production and have sensitive workloads, we advise that you implement everything inside of this article.

    Securing Your Kubernetes Infrastructure

    When you deploy a default GKE cluster with no additional options provided, you’ll get some sensible security defaults:

    1. Basic authentication is disabled.
    2. Encryption at rest for the Operating System (OS) image is enabled.
    3. Client certificate issuance is disabled.
    4. Secure Operating system (Using Container-Optimized Operating System; COS).
    5. VM Integrity monitoring to protect from malware/rootkits.
    6. Restricted Service Account.

    So, why did Google enable these things by default? Essentially it means:

    • You can only authenticate via Google Cloud Platform (GCP) Single Sign On (SSO) to the Kubernetes API. That means, no basic authentication or client certificate can be used.
    • The OS is locked down and container optimized with a limited and read-only root filesystem with integrity checks. Integrity monitoring of the OS is enabled, to protect against rootkits and malware.
    • The service account that the Kubernetes nodes are using is restricted and will only have privileges to access cloud services it needs, such as Stackdriver for logging and monitoring. 

    The things you need to make sure you enable:

    • Enable secure boot to validate the authenticity of the OS and kernel modules. (If you decided not to use the default OS, then it might cause issues).

    Using the virtual trusted platform module (VTPM) to sign kernel images and the OS, means that authenticity can be established. It also guarantees that nothing has been tampered with, and kernel modules have not been replaced with ones containing malware or rootkits inside.

    • Enable intranode visibility so you can see the data flowing between pods and nodes.

    Making sure all traffic is logged and tracked between pods and nodes will help you identify any potential risks that may arise later on. This isn’t necessarily something you need to do for development, but, something you should do for production.

    • Put the Kubernetes API on a private network.

    • Put the node pool on a private network.

    • Provide a list of authorized networks that should be allowed to talk to the API.

    Making your nodes and Kubernetes API private means it isn’t subject to network scans that are happening all the time by bots, hackers and script kiddies.

    Putting it behind an internal network that you can only access through a VPN is also good, however, this is a much more involved process with GKE and isn’t as simple as a feature flag like the others. You can read about what is involved here.

    Securing Your Application Containers

    Again, Google enables some default features to get you started. However, there are still gaps that you will need to fill. 

    What do you get?

    • Managed certificates
    • Role Based Access Controls (RBAC)

    What you will want to enable:

    • Pod Security Policies. This is Beta and requires you to inform Google that you’re wanting to run the cluster creation in beta mode:

    • Network Policies.

    This sounds great, but what does it actually mean? Basically, some of these roles are split into two, one is for application development teams to own and the other is for the cluster administrator.

    Application Developer

    • Set network policies at an application level so you only allow the right level of access, i.e., AppA can talk to AppB but AppC can’t talk to either of them.
    • Have encryption to your service endpoint for your desired domain, i.e., https://www.mydomain.com.

    Kubernetes can be a bit of a learning curve, there are technologies that make it simpler in terms of dependencies such as helm, that will allow you to deploy application dependencies with pre-defined deployments. But there is no real substitute from understanding the main components of Kubernetes;  Network Policies, Ingress, Certificate Management,  Deployment, Configmaps, Secrets and Service resources.

    The main security components are network policies, secrets and certificate management. Network policies will allow you to control the traffic to and from your applications. Secrets are base64 encoded, so there is no real security in terms of how they are stored; therefore making sure the cluster administrator has enabled secret encryption, (as mentioned further down), will add that additional layer.

    Certificate management will make sure the traffic to your service is encrypted, but if you’re communicating between services, then you should also add TLS between your applications. Having the cluster administrator install something like cert manager, will allow an easier way to encrypt between services. There are also services like Istio, but as that product does a lot more than just certificates, it can add more complexity than necessary.

    Cluster Administrator

    • Have the ability to control what users can do and in what namespaces (RBAC), i.e., maybe Bob the developer can do deployments, (create, update, delete), but can’t view secrets in namespace our-teams-app1-dev.
    • Implement a deny all network policy for both egress and ingress (outbound and inbound). This can be controversial, as it can catch teams out, so making sure you’re communicating this is key. It will, however, force teams to define network policies to get their application working.
    • Force an application deployment security stance with Pod Security Policies. This means preventing containers running in the cluster escalating privilege, mounting in devices (including binding to the host network) on the nodes or running more privileged kernel system calls.

    You want to make sure that development teams can’t deploy insecure applications or make attempts to escalate their privilege, mount in devices they shouldn’t or make unnecessary kernel system calls. Pod Security Policies offer a way to restrict how users, teams or service accounts can deploy applications into the cluster, enforcing a good security posture.

    RBACs and Pod Security Policies go hand in hand. Once you define a good Pod Security Policy, you then have to create a role that references it and then bind a user, group and/or service account to it, either cluster wide or at a namespace level.

    Note: GKE uses a webhook for RBAC that will bypass Kubernetes first. This means that if you are an administrator inside of Google Cloud Identity Access Management (IAM), it will always make you a cluster admin, so you could recover from accidental lock-outs. This is abstracted away inside the control plane and is managed by GKE itself.

    We recommend the below to be a good PSP.  This will make sure that users, service accounts or groups can only deploy containers that meet the criteria below.

    If you wanted to create a role to use the defined PSP above, then it would look like something below, this is a cluster role as opposed to a standard role. To then make this enforced on say all authenticated users, you would then create a role binding to apply to the “system:authenticated” group.

    Remember that as this is cluster wide, any applications that may need more privileged permissions will stop working, some of these will be things Google adds into kubernetes; such as kube-proxy, that runs in the kube-system namespace.

    You can read more information on RBAC and PSP’s on kubernetes.io.

    Securing Sensitive Data and Application Cloud Services

    We’ll break this down into two.

    1. Encrypting Your Secrets in Kubernetes

    The recommendation for encrypting secrets using Google Clouds KMS service, is to have segregated roles and responsibilities, and to define a new Google project outside of the Google project that will host Kubernetes and applications. This is to make sure the encryption keys and, more importantly, the key that is signing the other keys (envelope encryption) isn’t residing in the same project that could potentially get compromised.

    For encrypting secrets you need to:

    • Setup a new Google project to implement segregation of duties (if necessary).
    • Create the master key ring and key inside that project.
    • Only allow the GKE service account in the Kubernetes hosted project access to the key in the dedicated key project.
    • Only allow just the right permissions (encrypt and decrypt) of the service account.

    The documentation on how to do this can be found here. But the main things to remember are:

    • The key has to be in the same location (zone) as the cluster (this is to reduce latency or loss of a zone).
    • Access permissions have to be correct for the service account.
    • The binding is a KMS IAM policy binding.

    Once this is setup you can pass the path to the key, to the “gcloud container clusters create” command:

    Note: If any of the above is incorrect, you will get a 500 internal server error when you go to create a cluster. This could be the path is incorrect, the location is wrong or the permissions are not right.

    2. Consuming Cloud Service Inside of Kubernetes

    There are four different ways to allow application containers to consume cloud services. All of these have limitations in some way, i.e., being less user friendly and unautomated for developers (making developers wait for the relevant access to be provisioned), or have less visibility and more complexity in tying together auditability for Kubernetes and cloud administrators.

    1. Workload identity (this is still in Beta and is the more long-term direction Google are going in): Require a constant process of managing google IAM as well as Kubernetes Service Accounts to tie them together. This means managing Google IAM roles, policies and service accounts for specific application service lines as well as Kubernetes Service Accounts. It does, however, improve auditability.
    2. Google Cloud Service Account keys stores as secrets inside Kubernetes namespaces (where the application will be living): Similar to the above, but without the binding. This means provisioning a service account and placing it as a secret inside of the applications namespace for it to consume natively. This has the downside of not having full auditability within Google Cloud and Kubernetes.
    3. Use something like Vault to broker between Cloud and Applications: Using Vault provides an abstraction to cloud and will generate short lived access keys to applications. However, there is still a secret required to be able to speak to the Vault service, hence, the permissions are still the same but just abstracted down one. It also disjoints audibility between Google Cloud and Kubernetes.
    4. Using the default service account node role of GKE: Much simpler and more developer friendly, but more risky. It would mean allowing applications to use the default node service account and modifying the role to cater for all the service policies applications would need, increasing the scope and capability of the node service account to most Google Cloud services.

    Having Good Visibility; Current and Historic

    Note: As of today, there is no access transparency on when Google accesses your Kubernetes cluster (Google Access Transparency). This could be problematic for a lot of organizations who want assurances around the providers data access.

    When the cluster is provisioned, all audit logs and monitoring data is being pushed to Stackdriver. As the control plane is managed by Google, you don’t get to override or modify the audit format, log or extend it to your own webhook.

    It does mean that you can search your audit logs for things that are happening inside of Kubernetes in one place, for example; to query all events against a cluster inside of a specific Google project against your user ID, you can do the below:

    From this point on, you could take it further and add additional security rules such as setting custom metrics to alert when cluster admin changes are made or specific modifications are happening on roles (cluster-admin roles).

    Summary

    Security is something everyone wants but, as you can see, it can be quite an involved process to achieve. It also requires domain knowledge to get enough context to assess the risk and what it might mean to your applications and business.

    Security without automation can slow down productivity. Not enough security can put your business at risk. Enabling security features that require Beta feature enablement may also not be suitable for the business and only General Availability features are acceptable, which compromises on security.

    As a general rule, hardening your clusters and enforcing secure ways of working with Kubernetes, containers, cloud services and your applications will get you the best outcome in the end. There may be frustrating learning curves, but as the industry matures, these will slowly be remediated.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or August 2020.

    — Lewis Marshall