devopsy.kinsta.cloud

Tag: shift left

  • Building a Security Feedback Process for DevOps

    Building a Security Feedback Process for DevOps

    The last few years have seen some major slip-ups in the security space among all major cloud providers, resulting in uncertainty and speculation. That’s understanding; cloud security is an extremely complicated subject as enterprises build and deploy applications faster than ever before to keep up with business requirements. Most of the security issues that occur in the cloud today are around compromised accounts, identity theft, malware apps and, of course, data breaches, which are business-critical and showstoppers, in many cases.

    Given the dynamic environment of DevOps and the nature of the cloud, it’s a complex problem to address and fix. The most effective way is to embed security, or what is usually talked about as the shift-left of security, in the IT pipeline. If DevOps is the culture of collaboration to reduce risk by the size of change, DevSecOps is the automated security control built into each point to build a sort of self-defense mechanism.

    DevOps is not about tools; it’s about philosophy and process. One of the most important processes to include to succeed at DevOps is a feedback loop that reinforces constant collaboration.

    This is certainly true when it comes to security-related information. In a healthy DevOps organization, security problems that are detected at any point in the delivery pipeline are communicated quickly to all stakeholders, who can, in turn, respond accordingly.

    In this post, we will look at the steps involved in building a security process that will ensure the right information reaches the right people quickly and that teams collaborate without duplicating efforts.

    Automate Your Way to Better Feedback

    Automation is the first step, and a no-brainer when it comes to DevOps. Without automated security tools for code analysis, configuration management, access control, vulnerability management and secrets management, it is very difficult to scale security in a dynamic DevOps environment. Manual efforts in the cloud are doomed to fail in many cases, as environments change rapidly. Hence, automate. The recommendation is to automate every security control and the entire feedback process as much as possible.

    Manage Roles Right and Control Their Accesses

    Identity and access management has become the linchpin in cloud security. Since it’s very easy to give people the wrong access, it becomes important to figure out who can access what resources. Access to information and resources must be secure, yet easy to obtain. All access should be given to a group rather than an individual. Mapping security access to the organization’s hierarchy is an effective way of establishing a communication loop. The right information reaches the right group of people. It is important to be able to answer questions such as the following to ensure the communication loop is efficient:

    • Who can delete this VM?
    • What are the resources a certain user has access to?
    • Who has service accounts access?

    Tools that show a relationship between users, groups, roles, permissions and resources help answer these questions.

    Encrypt Secrets, Manage Their Keys

    Another important step to building an effective communication cycle is controlling access to application-level secrets. Security can be improved by tightening information flow between relevant groups, which can be accomplished through creating and controlling access to secrets using principles of least privilege. Access keys to those secrets can be kept in a centralized secrets management tool or a key management tool. The rotation of these keys can be automated as often as required.

    Designing for Failure and Proactively Managing Vulnerabilities

    Though security personnel may cringe at the mention of the word “failure,” designing for failure means thinking ahead and having a plan B. The cloud is dynamic by design and things happen at an incredible pace, often resulting in issues of elasticity, configuration and cloud power.

    Context becomes an important parameter for managing vulnerabilities because there are multiple vulnerabilities at any given time. Automated vulnerability management and configuration management solutions are available to take the complexity out of protecting cloud environments. Automation capabilities and continuous monitoring provide deep insight, intelligent recommendations and continuous protection of the cloud, thus ensuring effective management of information in the feedback cycle.

    Log Everything — Everything

    Just embedding security to shift left is not enough to ensure security. It is important for all relevant collaborating groups to know the status when some code is deployed. The primary source of visibility and feedback comes from logs. Effective logging strategy is the paramount attribute of security design. Logging should be enabled everywhere possible: data access logs, identity logs, admin activity logs, logs within the cloud environment, OS, network platforms, etc. Logs, though cumbersome and volume-heavy, are non-intrusive and a critical visibility tool. You need to know what’s going on before deciding to do something with the event itself, which brings us to the next step: monitoring.

    Monitor Those Events

    Testing departments often seem paralyzed with too many flags, which prevent them from prioritizing critical issues. Having log and event data is an important first step, but the absolutely necessary next step is to relay them into an automation strategy that will separate the unimportant, voluminous issues from business-critical priorities. Automated monitoring for events triggered makes the feedback cycle more effective. Otherwise, logs can become noise—a copious amount of noise.

    Alerts and Notifications

    We embed security in the code. We manage access and roles. We turn on the logs and monitor them. An event occurs. Now, who do we alert? An effective alert notification policy will ensure that notifications are sent to the right group of individuals, depending on the notification filters—the level of criticality and relevance of the issue type. Notification policies specify the conditions that trigger a notification that will be sent to the recipient. Notifications can be sent by email, SMS, voice, push messages, etc. A recipient can then acknowledge the alert, which indicates to other users that someone is working on the issue, thus avoiding duplication of efforts.

    Conclusion

    To conclude, automating security controls and feedback cycles enables a productive DevOps ecosystem by exposing vulnerabilities early in the product life cycle. Automation, streamlined communication and appropriate response to priorities ensure availability, network connectivity and resilience. At the same time, they reduce the possibility of a data breach and duplication of efforts. At the heart of everything is the ability to deliver code in a secure and reliable way to meet ever-evolving business needs with agility.

    To see how Symantec can help maximize security in a DevOps environment, check out a free trial of Symantec Cloud Workload Assurance.

    This sponsored article was written on behalf of Symantec.

    — Twain Taylor

  • Shift Left Without Fear: The Role of Security in Enabling DevOps

    Shift Left Without Fear: The Role of Security in Enabling DevOps

    It’s the age of DevOps, and we all want to shift left, which refers to the idea of performing processes earlier in the CI/CD cycle. That includes security checks and audits. By starting security operations earlier in the delivery pipeline, it becomes easier to find problems, and teams have more time to address them before pushing code into production.

    The challenge, of course, is building a shift-left security operation that allows you to perform security operations early without slowing down delivery speed. Let’s take a look at how to square that circle.

    Defining Shift-Left Security

    A popular term in DevOps culture is “shift left.” But what does shift left mean? Shifting left refers to the effort by a DevOps team to implement measures to ensure application quality at an early point in the software development life cycle. In the case of application security, this means implementing measures to ensure security concerns are taken into consideration while the application is being developed, rather than at the end of the process.

    The overall goal of shift-left security is to identify and resolve potential vulnerabilities earlier in the development life cycle, when these issues are less expensive to fix. Doing so will help to mitigate the risk potential security concerns pose to both the delivery schedule and your end -users.

    DevSecOps: Integrating Security Into the Development Process

    Let’s be honest: Faster and more frequent development cycles aren’t going away. Slowing delivery speed to analyze and bulletproof the application at the end of the development process is no longer a viable format for implementing security. DevOps and CI/CD need to be embraced and enabled by the security staff.

    For security teams to enable the DevOps process, collaboration is key. Through this collaboration, the process of DevOps transforms into something known as DevSecOps. In DevSecOps, security personnel work closely with the rest of the DevOps organization to help those who create and deliver the application to gain a better understanding of what is needed to address security concerns as early in the development life cycle as possible.

    The following tips serve as a jumping-off point for implementing this collaboration:

    • Make security the responsibility of the entire DevOps team: Impress upon the development and operations staff that security is now the responsibility of the entire organization. Simply addressing potential vulnerabilities post-development will eat up too much time and increase the risk that something will be missed. Buying into this process is an essential part of implementing successful collaboration between the security team and the rest of the organization. An example of this would be impressing upon developers to code with security in mind.
    • Facilitate regular discussions about application security throughout the development process: It is the responsibility of the security folks to keep security considerations at the forefront of a developer’s mind while they develop new features. Use the agile process to your advantage. Backlog meetings can involve security folks to ensure the creation of user stories includes consideration for security concerns. Your daily standup meetings are another place to include security personnel. This will provide security staff with an opportunity to speak up when the work described by the developer requires—or might require—considerations related to security that have yet to be addressed. Doing so will help to ensure the ball is not dropped, allowing for potential issues to be rectified before they even exist.
    • Use test automation and continuous integration to your advantage: When it comes to addressing security concerns throughout the development process, continuous testing is your best friend. And, in cases where it makes sense, the DevOps team should integrate automated test scripts for the detection of security and into their continuous integration process. Such tests can include developer-written scripts to test for vulnerabilities in the code base and tools for static code analysis. Another part of your continuous testing strategy could involve the use of tools for infrastructure monitoring.

    The data produced by these strategies can then be analyzed by security folks. And from this analysis, recommendations can be made to the DevOps team regarding the path forward to close potential gaps in security before they ever reach production and can be exploited.

    Understanding the Role of Security Personnel in the Development Process

    Security folks are often not coders, and development personnel are often not security experts. And while it may be frustrating at times, the solution to managing this disconnect is in the collaboration described above. Situations will arise where developers need to develop with various technologies to build a quality product. It’s possible that these technologies will lend themselves to security concerns that require security expertise. The job of security personnel shouldn’t always be to fight back against the use of these technologies. The security team should act as facilitators—take what development and operations staff intend to do and foresee the potential impacts that innovation will have on security—then take these potential impacts and provide insight into what needs to be done to reduce the risks.

    Conclusion

    In the past, security often was considered as an afterthought in the development of an application and its infrastructure. The movement toward faster development cycles and more frequent application releases has forced development organizations to think about security differently. Nowadays, so much can change in an application in a short time. This has forced DevOps organizations to integrate security into the development pipeline to make security a continuous consideration in the development life cycle. With this integration, the responsibilities of security personnel have shifted. Security teams now need to be enablers of the DevOps culture, empowering DevOps processes with security in mind.

    To see how Symantec CWA can help you shift left without fear, sign up for a free trial to see how it can help secure your cloud.

    — Scott Fitzpatrick

  • Tech Pros: Plan for Failure to Breed Success

    Tech Pros: Plan for Failure to Breed Success

    Veteran tech pros will tell you: Failure does not always mean failing at something; it often means failing to do something. Failure can be unpredictable and accidental, but usually, a lack of preparedness causes the most significant incidents. Data centers go down, huge bursts of bot activity generate too many requests for a network to handle and every systems engineer knows that one day—probably sooner rather than later—AWS will go offline. Not preparing for the worst increases the likelihood of the outages, attacks, breaches and downtime that keep tech pros up at night.

    While this spans every aspect of technology, recently there has been a surge in app development failure planning through something increasingly known as “plannable catastrophes.” This pragmatic approach to failure involves considering catastrophes during the application design stage—anticipating disasters on the front end so an app can still perform even under duress. Preparing for catastrophes in this capacity can decrease the amount of time an app is down and give companies the upper hand if an unprepared competitor is experiencing a similar issue simultaneously.

    Plan for Failure

    The first step in planning for failure involves designing a way for processes to fail gracefully—for example, by switching the processes to a degraded service option rather than failing. If one service goes offline, the rest of the application will continue to work. A real-world example of this is enabling users to log in and view documents even if the search process is offline. Another way to plan for failure involves building in request buffering, so the components servicing the requests do not become overloaded. This is designed to prevent possible bottlenecks, especially if there is a spike in requests.

    The second step is to detect failure conditions quickly, so you know when to switch over to the degraded service option. Having full visibility is essential to developing early warning signals for system issues.

    Business Considerations

    Every time an application is down or running slowly, there is a direct impact on the business, either in lost productivity or lost revenue. It’s crucial to bring business context into monitoring performance metrics and to prioritize issues based on business impact, ultimately helping ensure that end users or customers have an optimal experience.

    It’s also key to include endpoint monitoring tools to quickly see if all the functions are online and load times are within range. Monitoring spikes in load or processing times, as well as resource utilization, is also crucial. Together, this will allow you to detect failure conditions quickly, ultimately helping you know when to switch over to the degraded service option—hopefully via automation.

    Creating a Holistic View of a Cloud Environment

    How does planning for failure come to fruition in app development and deployment? The answer lies in creating a holistic view of a cloud environment. Implementing this holistic view from the start of the process involves knowing that an application is going to fail—not if, but when—and planning accordingly. Top best practices include:

    • Shift Left: Integrate operations and monitoring considerations in application design and system architecture, coding, testing and preproduction. This helps both development and operations teams identify issues early, cut down on troubleshooting app issues later and react quickly to keep applications up and running even during spikes in demand or service failures.
    • Adopt an application performance monitoring (APM) tool: Full-stack visibility, from the end user through to the applications and infrastructure, is essential to having the ability to spot the early warning signs of a potential service disruption. To create that view, you should select an APM tool that can monitor the performance of the application holistically and allow you to design meaningful performance metrics and alerts. Implementing a curated collection of easy-to-use tools that seamlessly provide full-stack performance management without the cost or complexity of traditional APM solutions is key.
    • Be pragmatic in app development: Leverage APM tools in the staging or dev environment to measure the effect of variables on performance before the app goes live. Use APM tools to set up baselines at each phase, creating valuable metrics to flag when issues are about to take place and to detect failure before an app goes completely offline. Understanding the app’s baseline activities and response time enables tech pros to identify if the app is performing at the level it was designed. Tech pros can leverage platform capabilities to meet and exceed initial goals.

    Conclusion

    Failure is inevitable. Tech pros must ensure end users continue to have the best possible experience in the application and that business goals and service level agreements (SLAs) are met even amid crises. Preparedness comes in the form of taking preventative measures to help avoid and mitigate failure and having a plan in place to get back online as quickly as possible if preventative measures fail.

    Shifting left, building failure mitigation into the design phase and implementing an APM tool helps create a holistic view of the cloud environment and achieve these goals. Planning and building for failure help tech pros avoid and mitigate potentially detrimental issues, ultimately using failure to breed success.

    — Melanie Achard

  • Embedding Accessibility into the DevOps Process

    Embedding Accessibility into the DevOps Process

    Calls for accessibility and inclusion are a defining characteristic of current times. Persons with disabilities (PwD), including vision, hearing, cognitive and physical impairments and others, are demanding equal access to virtually all public services and forums they interact with, from transportation to banking and self-service terminals in airports—and increasingly, digital (web and mobile) services.

    Ensuring accessibility can help enhance brands and is good business practice. The process can seem daunting but integrating accessibility into DevOps doesn’t have to be overwhelming. There are several simple steps for getting started.

    Understanding Your Organization’s Unique Accessibility Motivators

    Knowing what accessibility elements you need to test for depends on numerous factors, including your industry and where you are located.  For example, U.S. government organizations are subject to Section 508, while telecommunications providers must comply with the Communications and Video Accessibility Act (CVAA). Different scenarios require a nuanced approach to accessibility testing.

    The risks of digital inaccessibility are growing far beyond government websites, with the number of Americans with Disabilities Act (ADA)-based web accessibility lawsuits increasing 181% between 2017 and 2018. Since the beginning of this year alone, several well-known brands including Domino’s Pizza and Beyoncé have been sued for web accessibility breaches, bringing the issue to the forefront.

    E-commerce sites are often more concerned with lost revenue opportunities, as the average blind person is known to abandon several companies per month due to web inaccessibility. For transaction-oriented sites, inaccessibility translates directly to lost business. Whatever the motivators, building empathy can be a great way to give this mission meaning for developers. Organizations can do this by staging and having developers participate in “empathy events” where simulations are offered and developers can experience first-hand what it’s like to interact with a given website using the same assistive technologies that people with disabilities rely on regularly.

    Set Achievable Interim Milestones

    Agile DevOps teams emphasize catching defects as early as possible in the software life cycle to ensure efficient, high-quality rollouts. Unfortunately, accessibility testing is often relegated to the end of the process or performed as user-acceptance or usability testing. This type of testing is often manual and can be resource-intensive and expensive.

    Chart showing the process for building in accessibility into a product

    Organizations only became aware of issues upon hearing from end users encountering problems—or, worse, taking legal action to gain equal access to web-based content and services.

    Getting from a manual process to a fully automated and integrated process takes time. We recommend setting achievable, interim milestones that start small and build on early success. Organizations may not know what to tackle first, but some commonly shared examples of achievable first milestones are accompanying all digital videos with subtitles and/or sign language translation (for the hearing impaired) and ensuring that all new UIs are clear of any issues identified by automated tools. Experts and trainers can offer valuable advice on addressing the basics and then building on the success.

    Implement Tools

    Today’s developers have easier access to more comprehensive accessibility testing tools than ever before. Currently there are tools available for all the major platforms (web, Windows, Android and iOS) and many of these have been open-sourced, making them richer every day. Interactive testing functionality can enable development teams to thoroughly assess the accessibility of products early on in the cycle, while libraries allow these tests to be run regularly as part of the development test automation. A good tool will provide developers with simple remediation guidance and allow them to embark on a learning journey to increase their knowledge and expertise.

    Measure, Improve and Automate

    Ongoing measurement is the key to understanding whether your accessibility transformation is taking hold. Leading metrics will focus on the change in development practices. These changes will give early indication of end user outcomes. An example of a leading metric is “percentage of new UI code that is clean from an automated testing perspective.”

    End user outcomes, while lagging, are important to ensure that the activity is leading to real change. An example of an outcome might be the number and the change in the accessibility-related call center inquiries. A reduction would indicate that fewer accessibility issues are making their way into production. Metrics are strongest when they are presented in a manner that directly correlates to business outcomes (such as customer satisfaction), as these tend to resonate the most with business executives, which helps build the broadest internal support base.

    Once you’ve had some early successes, it’s time to expand, and you can keep tackling issues incrementally. Organizations should also focus on continually shifting accessibility testing “further left” earlier in the development cycle to enhance quality while maintaining velocity. The sooner you detect and fix accessibility issues, the less likely these are to become more deeply baked into the software, making them more difficult and time-consuming to undo.

     chart showing building in accessibility earlier in the product development cycle

    Organizations also should consider replacing interactive and manual accessibility tests with fully automated tests. While manual testing cannot be completely eliminated, automated unit testing and integration testing enable developers to test for accessibility as they code (not after the fact), while automated regression testing ensures live web content and applications stay accessible after changes are made. Automated regression testing is critical for organizations wishing to ship software often while maintaining quality—in some industries such as financial services, a disproportionate number of websites have experienced some level of accessibility regression.

    Conclusion

    DevOps teams are focused extensively on maintaining exceptionally high levels of software quality combined with velocity. We’ve become obsessed with minimizing the number of bugs that make it into production but may be overlooking one of the key attributes of high-quality software—accessibility.

    In most cases, functional testing does not cover accessibility testing. While functional testing determines whether a system is working as expected (i.e., certain inputs generate certain outputs), accessibility focuses on ensuring usability for PwDs as a non-functional attribute. With estimates that the current number of PwDs in the U.S. nearly equals the total populations of California and New York combined, accessibility testing needs to be prioritized in the same way as functional testing. Otherwise, you are shutting these people out.

    In summary, organizations that demonstrate a commitment to accessibility through their digital products and services will have a natural advantage in an increasingly competitive market space. Getting started can be relatively simple, and you should focus on motivating development teams to achieve and overachieve, emphasizing continual improvement rather than perfection from the get-go.

    — Dylan Barrell

  • How to Design DevSecOps Compliance Processes to Free Up Developer Resources

    How to Design DevSecOps Compliance Processes to Free Up Developer Resources

    Anything that gets rolled out quickly in the technological world is a major step forward—as long as it doesn’t compromise security.

    DevSecOps is able to provide compliance processes far more effectively than before, and without jeopardizing security while also freeing up developer resources.

    How do you direct your IT team to keep your applications and customers safe without completely draining your resources in the process?

    The answer: automation.

    Why is Security Vital to DevSecOps?

    To take full advantage of the agility and responsiveness of DevOps, you must bring forward an IT security team through the entire life cycle of your apps.

    In the past, when software could take months or years to complete, it was perfectly acceptable to bring in the security team in the final stage of development. But in the days of DevOps, a security team must get consulted from day one due to rapid and frequent development cycles.

    An effective DevSecOps system can be completed in days, so the outdated approach to handling security during the development phases also must change.

    How Did Security Radically Change DevOps?

    It is important to remind developers that security is a shared responsibility that must be integrated from end to end to guarantee a smooth and functional application. The mindset is so important that many have changed from using the term DevOps to DevSecOps to stress the importance of the need to build a very sound security foundation.

    If you are already making the transition to the cloud with AWS, chances are you have already implemented DevOps strategies into your interface so you can test new virtual machines (VMs) and deploy applications quickly without waiting for infrastructure to get manually configured.

    For those who prefer the old-school method of developing applications, neglecting security until the end leaves their applications exposed. One of the most common security issues with most SMEs is misuse of security during beta tests, which is basically akin to leaving your front door wide open when you go to sleep at night. It’s just a terrible idea.

    Making the transition from DevOps is not a luxury but a necessity for developers who want a secure environment, even during the dev/test period.

    Designing Compliance Processes for IT

    Even if you add “security” to the term DevOps, it means little without properly adding the processes you use into the building of the applications.

    Security must be built into the processes by design, or you are not correctly taking advantage of what DevSecOps means—and your developer resources will get neglected, too.

    Like the name implies, security must be in the center of development and operations. To maintain compliance, you must enforce a shared responsibility between AWS and the customer:

    • AWS is responsible for security of the cloud, including the hypervisor and hardware.
    • Customers are responsible for security in the cloud. They do this by securing your applications and network with services through a third-party vendor.

    The best examples of DevSecOps in the cloud take this philosophy to heart.

    Automation is Critical to Compliance Processes

    A “security first” mentality is good , but are you really taking action with your compliance processes to free up developer resources?

    The answer for many developers is, “Not quite.” Automation is essential to DevSecOps, yet so many systems lack the right tools. This is especially unfortunate for organizations that have the necessary skills and procedures in-house to implement built-in security processes directly into the application by design.

    For other organizations, automation is only achievable by leveraging their services from a security integrator, who can then provide services to help define processes and roles, unveil security tools and set up automation procedures.

    Steps to Establish Automation on DevSecOps

    Now that you know automation is crucial to the success of any system, here are a few basic steps you can take from day one to start building the security processes into your design:

    1. Consider a “shift-left” mentality when adding new processes, ones that incorporate design thinking from customers’ security needs and remain consistent. Steps such as threat-modeling storyboards build cyber resilience into the application before the first code is even written.
    2. Consider a cloud-based vulnerability scanning solution. It will help you define standards for the AMI as well as validate adherence to those standards.
    3. Implement an audit, logging and monitoring solution to track changes to the in-use AMIs. It helps combine account activity and AWS resources from the cloud with data related to security incidents. The in-use AMI changes can do everything from providing ongoing insights to allowing other security controls and having influence over network security.
    4. View the AWS Marketplace catalog. Along with what an internal staff or a systems integrator can help automate, there are a number of other terrific resources out there for automation. The software listings are all from independent software vendors that make it straightforward to locate, test, buy and deploy software that runs effectively on your AWS environment. Like other AWS services, the providers are billed based on consumption, so you pay only for what you need.

    By asking yourself the important question, “What am I trying to accomplish with my security controls?” you can get a variety of answers regarding what measures you need to take for compliance processes.

    Final Thoughts

    Speed, speed, speed. Developers today simply have no time to waste when it is time to launch new applications and get them out the door.

    You have little time to burn on extra features, but you also can’t afford leaving your systems exposed and vulnerable to attacks. That’s why DevSecOps uses superior security processes to free up developer resources by utilizing as much automation as possible.

    — Sam Bocetta

  • Survey Finds Wide Gap Between DevOps Adoption and Success

    Survey Finds Wide Gap Between DevOps Adoption and Success

    A recent HBR survey shows there is work to be done to advance DevOps in organizations

    A survey of 654 subscribers to the Harvard Business Review found that while 86 percent of respondents said it is important for their organizations to be able to build and deploy software quickly, only 10 percent said they are successful at it.

    Conducted by Harvard Business Review Analytic Services on behalf of Google, the survey found nearly half (48 percent) of respondents said their organization always relies on DevOps methodologies when deploying software, while another 21 percent rely on DevOps selectively. An additional 8 percent said their organization plans to adopt DevOps practices.

    Among those respondents employing DevOps practices, the top benefits cited were increased speed to market (70 percent), productivity (67 percent), customer relevance (67 percent), innovation (66 percent) and product/service quality (64 percent). Despite those benefits, however, most organizations still have a significant amount of work to do in terms of implementing best DevOps practices—the three biggest hurdles organizations face are organizational silos (50 percent), legacy technologies (49 percent) and resistance to change (46 percent).

    Melody Meckfessel, vice president of engineering for Google Cloud Platform (GCP), said these issues are precisely why Google is sharing how it operates in the form of site reliability engineering (SRE) principles, which are a prescriptive set of guidelines for implementing best DevOps practices. Not every organization needs to implement those principles to the same level, but just about every organization can benefit from adopting a set of well-defined engineering principles, said Meckfessel.

    The challenge many organizations face is the fact that most of the IT staff are not engineers by trade. Most IT staffs are made up of administrators who tend to have limited programming skills. But Meckfessel said even IT organizations that don’t necessarily have a lot of engineering expertise still can benefit from automating operations based on the SRE principles.

    Many of those organizations will need to slow down the rate at which they are building applications today to be able to speed up tomorrow. It takes time for organizations to master the cultural change required to implement DevOps practices. But once they begin to master DevOps, the benefits to the organization compound quickly. Meckfessel said two of the most important attributes of DevOps is adopting an immersive approach to training and establishing a no-blame culture within the IT organization when it comes time to conduct post-mortems. That latter goal is especially difficult to achieve sometimes, because most interactions between developers and IT operations teams historically have been less than positive.

    Meckfessel also noted it’s critical now for organizations to incorporate cybersecurity into its DevOps processes as well, which requires not only additional training for developers but also closer alignment with cybersecurity teams.

    It’s now only a matter of time before the complexity associated with emerging technologies such as microservices based on containers force many organizations to dive deeper in DevOps. The real challenge now is charting a path of least resistance that also generates the maximum level of benefit for all concerned.

    — Mike Vizard

  • Fail Fast: How Shifting Security Left Speeds Development

    Fail Fast: How Shifting Security Left Speeds Development

    How DevSecOps can help organizations increase security, move faster and save money

    With developers under constant pressure to create more software in less time, the last thing you need is for your code to fail at the end of the development lifecycle. No developer wants to create inoperable or insecure code.

    We know that most software products reach the market with flaws or vulnerabilities. On average, more than 85 percent of all software applications have at least one vulnerability, and more than half of flaws remain unaddressed for one to three months after first being discovered.

    The sooner a developer can identify, view and correct a flaw, the more efficient it becomes to fix in the software development life cycle (SDLC). That is the underlying philosophy of DevOps. Despite the somewhat negative connotation, failing early in development can be beneficial in deploying software to market more quickly.

    The most advanced organizations today employ a DevSecOps model, which tends to incorporate more frequent security scans, incremental fixes and faster rates of flaw closures into the SDLC. More importantly, it allows security teams to shift left to test software products against security standards earlier in the development process.

    According to Gartner, more than 80 percent of development teams will implement DevSecOps by 2021. In 2017, that number was around 17 percent.

    In Veracode’s latest “State of Software Security” report, we discovered companies that adopt a DevSecOps approach to development are addressing software vulnerabilities more than 11 times faster than companies that use a more traditional software development method. In the process, DevSecOps companies are innovating faster, saving money and becoming more efficient in how they identify and resolve new risks.

    By integrating developers with IT operations and focusing everyone on making better security decisions, DevSecOps can help deliver safer software with greater speed and efficiency.

    Introducing security earlier in the development process creates a sort of blueprint for teams to build within the specifications of major IT regulation and helps reduce risk by fixing vulnerabilities before they can be exploited. Think of it like building an apartment complex according to city codes with inspectors integrated in the construction crew.

    In practice, DevSecOps can add some friction and hinder the development process. Traditional tools for testing code and assessing application security risk simply weren’t built for the speed that DevOps testing requires.

    To make DevSecOps successful, development teams need to start with automated testing tools, as relying on manual processes can’t possibly keep pace with accelerated development timelines. Tools that can be used in an integrated development environment (IDE) are key, as they allow developers to integrate security into their workflow rather than having to launch a new environment whenever they need to test code. Solutions that check for flaws during the coding process enable developers to address vulnerabilities early on when fixes are the most cost-efficient. And because DevSecOps is equally concerned with security when software is in production, development teams need tools for testing applications after release. Development teams, meanwhile, must be comfortable with changing the way they work.

    Business leaders must recognize the importance of integrating security with development and communicate that priority to development teams. New regulations such as the EU General Data Protection Regulation (GDPR), which can fine companies up to 4 percent of their revenue for failing to disclose processes related to data breaches, are now in place. All businesses must be motivated to operate with a security-first mentality that permeates through the business, including within development teams. Veracode data shows that customers taking advantage of DevSecOps’ continuous software delivery are closing their vulnerabilities more quickly than the typical organization.

    DevSecOps doesn’t always require security personnel to be physically involved in the development process. Companies can and should train developers on basic security techniques and offer security resources to help developers become responsible for the security of the software they build.

    As the DevOps movement has unfolded, security-minded organizations have recognized that embedding security design and testing directly into the continuous software delivery cycle of DevOps is necessary. This is the genesis of DevSecOps principles, which offer a balance of speed, flexibility and risk management for organizations that adopt them.

    — Mark Curphey

  • Don’t Forget to Automate Cloud Security

    Don’t Forget to Automate Cloud Security

    Cloud has been around long enough, and we’ve had enough breaches stemming from unsecured cloud (mostly data objects, but other items, too) that we should know better. I cringe whenever I see a security headline along the lines of “… Unsecured S3 Bucket.” Would you publicly expose a server in your data center and not lock it down? No.

    So, what is the problem? Frankly, “cloud” now consists of so many separate-but-integrated offerings that it can be daunting, even improbable, for security staff to lock it down. Add in dozens or even hundreds of instances/storage locations/routers, and you have a separate data center with different requirements—at least, that’s what it essentially is for your security team. Then add agile and DevOps, spinning up new infrastructure and changing the parts at a rapid pace, and you have a fast-moving, complex data center environment. Yet, if you are a typical organization, your security staffing hasn’t significantly changed, nor do you have requirements for securing the physical infrastructure. In short, same group of individuals, a ton more work to do.

    Before it was purchased by VMware, I had the pleasure of working with CloudCoreo, and truly liked what I saw. The problem in highly automated environments, be they DevOps or just automated, is that a lot of parts are moving as the system is maintained/updated. Some of those parts are changing security settings in a complex (data center-size complex, in many cases) environment. For security teams to keep up and be part of DevOps—or just keep up with automation—quite often corners must be cut.

    That’s bad for all of us—security staff, the company and customers. If anything, when the cloud environment grows complex, security needs to be more cautious, not less. And yet, the number of data leaks/exploits that involve “misconfigured” security settings in the cloud says we have a problem.

    So what to do? One option is to check out security control and auditing software from the likes of RightScale, Dome9 or Evident.io. (DISCLAIMER: I have not used any of these tools. They were chosen because each shares attributes with CloudCoreo, which I have used.)

    The ability to scan and apply policy to a broad array of cloud artifacts—from users to back-end storage—is essential in allowing security to keep up with DevOps. There is just too much change in a fully automated cloud-based environment to expect security to keep up without the same level of tools as developers and ops get. “Shift left” doesn’t stop misconfigurations, but scanning can detect them, and some tools can even auto-fix those misconfigurations.

    In the end, protecting systems and data is what the security team does. Bringing them along on the DevOps grand tour and offering them the same type of automation opportunities as the rest of IT is benefiting from is mandatory. It is not “would be nice to have” to give security the chance to keep up; it should be a requirement.

    And you might still have rogue S3 buckets that security doesn’t know about that might be misconfigured, but any that are paid for by the corporation will be available to security and those automated tools and, thus, will be locked down.

    And if security isn’t enough of a reason for you to want to implement one of these tools, “Come for the compliance, stay for the security.” Most solutions offer auditing that helps with compliance.

    — Don Macvittie

  • DevOps Chat: Shift Ops Left, Shift Dev Right with Kristian Stewart, IBM

    DevOps Chat: Shift Ops Left, Shift Dev Right with Kristian Stewart, IBM

    In this DevOps Chat we speak with Dr. Kristian Stewart of IBM’s Cloud Event Management team. Kristian has some keen insights into the role of dev and ops and how they interact. Kristian calls it Shift Ops left, shift dev right. An interesting way of looking at things.

    As usual the streaming audio is immediately below and the transcript of our conversation is below that.

    Audio

    Transcript

    Alan Shimel:               Hello, everyone, this is Alan Shimel, devopsy.kinsta.cloud, here for another DevOps Chat. Got a really hot chat today and my guest joining me is Kristian Stewart of IBM’s cloud event management team. Kristian, welcome to DevOps Chat.

    Kristian Stewart:         Thanks very much for having me, Alan.

    Alan Shimel:               And I just wanna make sure I got it right – it is Kristian Stewart, right?

    Kristian Stewart:         Last I checked.

    Alan Shimel:               Okay. Well, sometimes my funny French accent throws people off, but, anyway – and, Kristian, I mentioned you’re working with IBM in cloud event management, but give our audience a little bit behind that. What exactly does that mean?

    Kristian Stewart:         Well, let me give you a bit of a background ’cause there’s some history here, in particular with respect to the topic of the podcast, which talks about operation tools. So I’ve been with IBM for ten years. I’m one of the lead architects for a suite of IT service management tools, and my own specialization is in event management, as part of IBM’s Netcool brand, okay? So I’ve been in this industry for 18 years.

    I worked for a small start-up back in the late ’90s – the company was called “Micromoves,” and we were focused on introducing tools for disciplined fault management into telcos, into communication service providers, during the dot-com boom of the late ’90s. And then, in the early ’00s, we pivoted and successfully sold into enterprise, finance, retail, as their IT infrastructures became larger and more complex and as they matured, from an IT operations perspective. So we were acquired in 2006 and we started to work towards applying interesting mathematical and analytical techniques, for increasing the value provided by our tools, and applying machine learning and problems to this space. So that was my background up until about three years ago.

    So what do I mean by “cloud event management”? Well, more recently, as part of the movement of the industry that we serve, as they move into cloud and their investment in cloud has increased, I’ve been working with teams to build software-as-a-service offerings  capabilities we had, till that point, only available on-premise, so it includes alert notification tools, runbook automation tools, and cloud event management. And so our particular interest in DevOps is twofold, right? So our transition itself, from going to shrink-wrapped client service software, that was originally deployed on-prem, has gone through to the development of cloud-native, cloud-scale, public cloud resident offerings, and with our guys embracing paradigms like micro-service architectures, 12-factor apps, and, of course, DevOps.

    And one of the biggest challenges or one of the biggest changes our teams have had to endure is that they now have to operate these deployments themselves, right? So they’ve gone from shrink-wrapped, “Give it to the client. Now it’s their problem,” – well, kind of – through to “Now we’re hosting this for you.” So we’ve had folks from development and QA transitioning to operations roles, and they’re required to apply software engineering skills and work with app developers to ensure that we’ve got robust capabilities in private cloud. And we’re providing 24/7 ops to support these offerings.

    But we find ourselves in a fairly unique position because we’re running operations but as a service product that themselves are designed to help people responsible for running operations for their own apps and services. So it’s just sort of a renewed empathy for our end users, especially as we start to angle our offerings towards teams that are embracing DevOps. Does that answer your question?

    Alan Shimel:               Yeah, and then some. [Laughs] But good stuff, Kristian. Thank you very much. So wanted to talk a little bit today bringing this back home to DevOps, right? And one of the trends that we’ve been seeing over at devopsy.kinsta.cloud recently is, really, around kind of putting the “ops” in DevOps, right? ‘Cause we’ve spoken now to a few folks in the DevOps space from both the vendor and practitioner side, who are really emphasizing the role that ops plays in DevOps. And one of my things about it, though, Kristian, is I get the opportunity to speak to a lot of people like yourself, a lot of people who are heavily invested in DevOps, but I also get a lot – the chance to talk to a lot of people who aren’t and are new DevOps or just dipping a toe in the water and planning their DevOps migration and transformation.

    And one of the things that I try to remind people is that “Don’t leave your common sense at the door,” at the DevOps door, right? Bring it with you, number one. And, while you’re bringing something with you, it doesn’t mean everything you’ve done in ops to this point and all the lessons learned and best practices accumulated get thrown out – you know, the baby with the bathwater. With DevOps, we evolve. We’re not necessarily revolutionizing; we’re evolving. And so building on best practices and time-tested ops practices and methodologies and processes is probably a better solution than, you know, scrapping the whole thing and starting from scratch. And who has the opportunity to do that anyway? What do you say to that, Kristian?

    Kristian Stewart:         Oh, I agree completely. And, you know, amongst our clients, we’ve got a whole array of clients, from telcos to finance to retail, right? You name it. Across those clients, we have people on varying points on the curve of maturity, with respect to adopting DevOps. But what I say to those clients, as they look at those paradigms, is that the business objectives of operations teams haven’t changed substantially, certainly since I’ve been in the business. There’s a lot that has changed, but the business objectives continue to be measured by KPIs like mean time to repair, mean time between failures, and pressure on costs, right?

    So disciplines – so, within a, let’s say, a mature enterprise that have invested heavily in ITIL, they adopt disciplines like event and incident management, and that leans on a key set of capabilities, which are equally applicable when you transition to DevOps. You know, the proximity of your developers to your operations and the smoothing of the processes and the interactions between those organizations, if anything, are enhanced by some of the traditional methodologies and tools. So, you know, to list some of them, you need tools which can consume events from highly heterogenous environments; you need to minimize the amount of management noise that’s presented to the people who have to respond to events; you need to be able to integrate with other systems; you need to be able to help pinpoint probable causes, drive efficiency in operations, do things like apply machine learning and advanced analytics to gain insight from your event and incident data.

    So, if you look at the evolution of these practices, historically, right, ITIL emerged in parallel with the practice of event incident management. It didn’t dictate it. You know, these standards emerged as – sorry, event and incident management emerged as de facto practices, driven by the availability of practical tools to solve real problems, and they still solve those problems. You know, if you look at the way that operations has evolved, look at telecom from the late ’80s and early ’90s, right?

    So, in a telco network operations center back then, you might have had dozens of very highly-skilled people, _____ network engineers, with training on dozens of different management tools, and these guys and girls were expensive, right? I mean, their time cost a lot of money, and then the introduction of event and incident management tools helped transition from that to having perhaps the same number or even more operations staff of a substantially lower skill. And so the first thing event management and incident management did was to try and lower the cost of entry for running an efficient operations team.

    But now, if you look at the changes in IT management now – so I was chatting to a large enterprise just the other day, and, ten years ago, they – so these guys have always been driven by those same metrics – MTTR, MTBF, cost, right? A few years ago, their job was to provision physical hardware, static middleware, and the management of that stuff from an operations perspective. Now these guys are having to provide platform-as-a-service tools, distributed runtime container and orchestration systems, and they still have responsibility for the existing infrastructure – the network, the physical servers,  the legacy monolithic applications. And all that still requires management. So, mainly, they’ve got the challenge of managing hybrid cloud, coupled with extremely tight expectations on time-to-market that working practices like DevOps and Agile software development allow for.

    So I’d say that event and incident management tools are as relative now, if not more so, than they ever were because, when you shift to practices like DevOps and site reliability engineering, you know, back to the olden days where you have ops employing of hundreds of you know, low-skill operators, best practice is transitioning to one where you have a substantially lower number of substantially higher-skilled operations staff. Right? So these guys and girls are highly skilled, but they’re time-poor. You know, your typical operations professional, in an organization that’s embraced DevOps, wanna be concentrating on automation, on liaising with dev early on in the release life cycle, on prepping for new rollouts. They don’t wanna be grappling through log files and sifting through performance metrics to find out what happened just so they can restore a broken service.

    And I think the eventual goal of the kind of tools that we provide – event and incident management – isn’t just to help the operator find the needle in the haystack; I think it’s to hand them the needle. And that’s useful, whether you’re a low-skilled junior operator in old-school operations or a time-poor, hyper-technical site reliability engineer with 20 years’ experience. Alan, you still there?

    Alan Shimel:               Yes. Kristian, I’m sorry. I had a little internet issue. So all really good, but, you know, one of the things I wanted to touch on, Kristian, is the subject of empathy, as part of kind of your DevOps mantra, right? Sometimes, people give cultural things short shrift, but empathy is important in DevOps. And what I wanted to ask about was the idea of “How do we get developers?” Right?

    For a long time, the ops people have sort of had to empathize with developers are doing and what they’re doing in terms of development to help in deployment and stuff, as part of DevOps, but now, putting that on the other foot, right, getting developers to empathize with what the ops folks are dealing with, as part of this DevOps equation. I mean, I’ve heard of some organizations where they’ll actually embed developers on the ops teams for short periods of time so that they get empathy the hard way, if you will, right?

    Kristian Stewart:         Yeah.

    Alan Shimel:               They gotta live it. What do you think about something like that?

    Kristian Stewart:         I think, in terms of getting the development professionals involved, I think – so you mention role rotation, right? And I think that’s one tactic. You know, I like to think of this as much as “shift-right dev,” as it is “shift-left ops,” right?

    Alan Shimel:               Yeah.

    Alan Shimel:               Good. Good.

    Kristian Stewart:         And, actually, you know, it reminds me of the situation in shrink-wrapped software development, maybe 20 or 25 years ago, between development, who were the coders, and QA. Right? So there was sort of a brick wall, prior to Agile software development, where it was throwing a pile of code over the wall. And I think Agile, with that shift-left testing, really helped to remedy that, to the extent that I see, today, developers and quality assurance engineers working really, really closely together and not depending on, you know, reams and reams of documentation to communicate with each other.

    So, I think, with DevOps, I think a similar thing is happening, and I think you need shift-right dev, as well as shift-left ops. So operating an application or a service, I think, has to be a shared responsibility, and I do think that dev need to get skin in the game. So that’s gonna include dev as an escalation point, when an incident occurs. It can also include role rotation. But, for mutual benefit, I think they both need to use the same operations tools, right, so that they see what each other are doing, when an incident or a critical event occurs in the environment. And then they’re motivated to improve the use of those tools for themselves, as well as for the sake of operations.

    I think, you know, a 3:00 AM incident notification, you know, like a page at 3:00 AM, is a great motivator. So, for ops people, it’s an incentive to get runbooks and automatic incident responses written so that service restoration is as quick as possible and, preferably, automatic. With dev, it’s an incentive to make their services or applications operable, right, so they need to get the right amount of instrumentation into the app so that ops can manage an incident or, better still, that some script can, so that neither of them get the 3:00 AM escalation. So I think there’s a – with all due respect to my dev and ops colleagues, there’s a carrot-and-stick approach there.

    Alan Shimel:               Mm-hmm. Yeah, that makes perfect sense. Kristian, I want you to repeat what you said before. We need to shift ops left and shift devs right or did I have that backwards?

    Kristian Stewart:         Yeah, no, yeah, that’s what I said. Shift-right dev and shift-left ops, right?

    Alan Shimel:               Got it.

    Kristian Stewart:         So get them to have a degree of overlap with respect to their responsibilities. And, of course, what they do in response to those responsibilities can be quite different. You know, let me see if I can think of an example. So, for devs, under those circumstances, when they’re being involved in responding to an incident, it can be stuff as simple as being really clear in their application logs.

    So, if you think about events that are generated by, say, a container-based app deployed on Kubernetes, okay, so the event and incident management system is gonna be presented with logs from the app code; they’re gonna be presented with logs from middleware; logs from Kubernetes itself; logs from, potentially, external technologies and monoliths – NFS mounts, databases, legacy apps, network; and they may also be getting alerts from monitoring systems that are watching performance metrics for the running container, so transaction request rate between micro-services, CPU utilization, response time.

    So, if devs can make sure that something as simple as the concept of criticality or severity is included in the logs, that really helps with filtering _____ ops figure out whether or not something’s really a problem. You know, for example, is this event classed as a trace message? A debug message? A warning? An error? Is it known that this is service-affecting? Right?

    Alan Shimel:               Yeah.

    Kristian Stewart:         And, you know, how better to motivate devs to implement those sorts of practices than to make them consume log messages that are completely uninterpretable, right?

    Alan Shimel:               Agreed. Agreed. Hey, Kristian, we’re over time already, but, as I told you before we started, the time here goes quickly. But, you know, boiling all of this down, the last – whatever – 18 minutes down, really, I wanna make the point of – and I loved what you said about ops shift left, dev shift right – that our ops and devs can work together. And, as you said, they each will have different reactions with different roles and responses to given incidents or given stimuli, but they can work together and we can build on what ops has built over the last 50 years of the IT industry. And –

    Kristian Stewart:         Mm-hmm.

    Alan Shimel:               Right? We don’t leave that at the door of DevOps; it’s part of it. And one thing I want – you know, if we can ask people to take that with them. And you agree that’s key takeaways here for them?

    Kristian Stewart:         Absolutely. Couldn’t agree more.

    Alan Shimel:               Fantastic. Hey, Kristian Stewart, IBM, thank you for joining us on this episode of DevOps Chat. Love to hear more in the future about what you and the IBM team are doing around cloud and hybrid and some of the other stuff kicking around. And we’ll have you back again soon, but, for now, thanks for being our guest this episode.

    Kristian Stewart:         Alan, it was an absolute pleasure.

    Alan Shimel:               Great. Hey, this is Alan Shimel, everyone, for devopsy.kinsta.cloud, DevOps Chat. Thanks for joining us and we’ll see you soon on the next chat.

    — Alan Shimel

  • First-Time Quality for Software on a Blockchain Network

    First-Time Quality for Software on a Blockchain Network

    There is probably more trust in the world today than at any other point in history—not necessarily by choice, but because of our use of software to carry out almost every type of transaction. The use of blockchain networks brings visibility to those networks of transactions: Information about an asset that was once held by separate owners is changing to the lifetime history of an asset or transaction and is shared between all stakeholders.

    For software developers, this means ensuring the integrity of a blockchain network is paramount. To that end, three key interesting aspects regarding continuous testing are emerging:

    • New approaches to business networks require the ability to test supply-chain business processes and transactions.
    • Reliance on shared technology ecosystems requires the ability to test APIs from multiple perspectives and to simulate the behavior of consensus.
    • Increased dependency of APIs requires the ability to test integrations at scale without affecting distributed ledgers in which any interaction is immutable (unchangeable) and final.

    Project teams face an unprecedented demand for first-time quality while minimizing the impact of testing on the team’s delivery. To respond to this challenge, test teams are enhancing their capabilities so they can test:

    • Apps that integrate with a blockchain network;
    • Chaincode that manages the execution of transactions on a blockchain network; and
    • Provision and scalability of a blockchain network.

    In addition to these new capabilities, proven continuous testing techniques and tools can be adopted to help the team deliver quality software into their blockchain ecosystem.

    Test Entire Ecosystems

    Using a blockchain network means an organization is redefining the structure of its business networks and processes, connecting distributed supply chains with a single transaction backbone. Projects need testers to minimize the risk of business disruption by validating every business process works across all stakeholders on the blockchain network.

    Test State Early—Shift Left

    In a blockchain ecosystem, the new paradigm is state-based communication, not message-based communication. The behavior of the state of an asset is influenced constantly by market, internal and regulatory changes, so projects increasingly are adopting Agile and DevOps practices to respond as quickly as possible. Shifting testing to the left on an Agile project means the team can test early, and those tests can then be reused continuously. Providing early, iterative feedback on quality at the time of development means fewer problems are found late in the life cycle, where their business impact can be severe.

    Establish Technical Trust

    One of the great things about blockchain technology is the potential to create greater transparency throughout an ecosystem.  Trust is embodied in the transaction itself, which gives an important new role for digital systems of trust. Regulators and owners of a blockchain network must ask testers to validate the codification of contracts, compliance and certifications, which are then deployed into the blockchain network and embodied within the business transactions.

    Use of blockchain technology will fundamentally change business processes and enable new opportunities. Success depends on the level of trust stakeholders have in the blockchain network and the blockchain ecosystem.

    However, a blockchain ecosystem is an environment shared among different organizations, each of which has a stake in—and dependence on—the overall system. This means each organization must trust the other stakeholders to fully test any software that interacts with the blockchain ecosystem in a manner that will not impact the reliability of the overall system. As such, a blockchain ecosystem that has a culture and reputation of technical trust between the dependent stakeholders is more likely to be successful. Continuous testing brings trust throughout the blockchain ecosystem.

    To establish technical trust in a blockchain ecosystem, ensure these four key things:

    1. Your team can test when needed, not just when possible.
    2. Required technical environment works as required to support the business flows.
    3. Business processes and transactions run as expected.
    4. Solution scales to the expected usage.

    Benefits of Continuous Testing and ‘Shift Left’

    Enabling teams to shift testing to the left means projects can test when needed, not just when possible. Continuous testing means an organization can deliver capabilities faster, respond quickly to ongoing changes and realize the value of transforming applications to utilize the power of a blockchain network.

    Watch the webinar replay on how to implement quality checks as early as possible, run tests as often as possible and integrate automated testing with release management to minimize the complexity of deployment.

    About the Author / James Hunter

    James Hunter jpg

    James has spent a large part of his career architecting and implementing large-scale, distributed systems in the finance, public and defense sectors. He now leads a global business segment responsible for defining technology strategy then developing and implementing solutions for both IBM and clients. Mentoring and encouraging future technical leaders is important to James. He works with students of all ages and speaks in primary schools to universities to evangelize on the benefits of technology and the positive impact software can have on our society and the environment. He has a degree in Management Systems and he is a Chartered Fellow of the British Computer Society. Connect with James on LinkedIn/Twitter