devopsy.kinsta.cloud

Category: DevOps Practice

  • What Developers Really Think About Pentesting

    What Developers Really Think About Pentesting

    A developer in their natural habitat is often spotted in a state of deep concentration, coding awesome features to tight deadlines. Feature-building is often our favorite part of the job, and really, it’s the fundamental outcome of the software development life cycle (SDLC). 

    However, as we have discussed before, many of us are still prioritizing features over security best practices. After all, in most organizations, it is set up to be someone else’s job and adequate security training for us is limited. Penetration testing and static analysis scanning tools (SAST) are just part of the overall process to mitigate security risks, operating rather independently from what we do — until the code bounces back to us for hotfixes, of course.

    It’s at that moment that many developers think, “Do the pentesters hate me?”

    These interactions often define a team and culture. The concerning thing is, the lack of communication, understanding and overall collaboration has created tension, at least on the side of the developer.

    Think about it: Imagine you’ve spent a few hundred hours sculpting a marvelous statue, and then someone comes along with a hammer and starts smashing bits off it after they’ve told you its foundations are not up to scratch. That’s the perceived dynamic between a tester and developer — the latter has their software darlings slaughtered by an outsider that hasn’t labored through the process with them; instead, they’ve extended the workload and delayed the satisfaction of shipping code. 

    Having moved into the security space long ago, I can see both sides of the story. And no, pentesters don’t hate developers. The pentester is in all likelihood, overworked and under a lot of pressure. As such, a constant stream of common security bugs that could be quite easily fixed at the code level take up time, resources and headspace away from the really serious issues. 

    I always saw pentesters as kind of like parents. They want you to do well, and when you don’t they’re not mad, just disappointed.

    Now that I’ve put that (perhaps slightly unfair) image in your mind, let’s explore this a little deeper. What has caused this world view among developers?

    Resentment for Being Told What to Do

    Nobody likes feeling as though they’ve done a bad job, or that someone doesn’t like their work. Sadly for developers, when static analysis and pentest results come back to them, it can feel like a report card. They’ve been given low grades, but at the end of the day, their bosses assess them on the features they’ve built and the time they’ve delivered them, not whether there were vulnerable elements in the software or not.

    For the poor pentester, this is a case of “don’t shoot the messenger.” It’s nothing personal — they are tasked with finding bugs, and they found them. Granted, at a person-to-person level, maybe some pentesters are grumpier than others, but they’re not (or shouldn’t be) out to crucify development teams. It would be far easier for both teams if they were on the same page with what constitutes security best practice. Developers are not expected to be perfect; realistically, the testing team is there to protect them from shipping vulnerable code.

    Minor Issues vs. Higher Priorities

    It’s true, a developer’s highest priority will always be the building of features, and in this crazy world of rapid digitization, it will have to be done at speed. While some coders have a personal interest in security and secure coding, the general sentiment is that security is “someone else’s problem,” which inevitably includes pentesters. 

    Most common vulnerabilities are indeed minor issues to remediate — once known, the fixes are simple to execute for things like cross-site scripting (XSS) and SQL injection. The problem is, many developers don’t realize they’re introducing them in the first place, and these seemingly minor issues are the small window of opportunity an attacker needs to cause devastating problems for a company. According to Akamai, between November 2017 and March 2019, SQL injection vulnerabilities accounted for 65% of all web-based attack vectors. For a vulnerability that has had a known fix for more than 20 years, that is a sobering statistic. 

    Some pentest teams do assist in the remediation of security bugs, but others will provide a report of the bad news and expect developers to work through hotfixes, even if they have moved onto a different project by the time this happens. And in some cases, the development team may be faced with a report that includes bugs they can’t (or shouldn’t be expected to) fix. It still has to be part of the findings, and again, not taken personally.

    The “happy medium” for this would be pentesters, security personnel and development managers acting in more of a mentor role to ensure the team has what they need in terms of effective training and tools, giving individual coders the best chance to succeed and code securely from the very beginning of the SDLC. Both teams really should be meeting half-way to ensure security is considered from the start, as part of a healthy DevSecOps practice. 

    Lack of Knowledge vs. False Positives

    Static analysis is an element of the security process in the SDLC, and static analysis scanning tools (SAST) play a fundamental role. And yes, false positives are an issue with these and other types (DAST/IAST) of scanners. It’s an annoyance in what is already a slow process, requiring manual code review and putting pressure on both developers and pentesters alike. Pentesting personnel have taken time to meticulously set up custom rules to avoid inaccurate readings and provided company-specific guidance, yet some false readings slip through and end up in front of a head-scratching developer. 

    This process isn’t perfect, but the other problem is that many developers lack enough knowledge to mitigate a lot of common vulnerabilities on a consistent basis. With security training rare in tertiary education, and on-the-job training varying in its effectiveness, it stands to reason that there may be some overconfidence at play as well (and it’s not their fault, we as an industry need to get better at equipping them with what they need).

    Remediation Tasks

    Sometimes, there is an assumption by overworked engineers that pentesters are just hanging in the wings, waiting for the moment to strike by testing an application and raining on the development team’s parade. They are over-testing, they’re nitpicking, they’re creating extra work. 

    The only problem with that is they, too, are overworked (more so, in fact, the cybersecurity skills shortage is at dire levels and getting worse) and simply don’t have the time to test without reason. They are not the sole decision-makers in prioritizing testing; it could have been requested by senior leadership, a customer, as part of a security audit or even determined as a result of a bug bounty program. 

    For a developer, being pulled off current feature-building sprints to work on security fixes is annoying — especially if it’s not their work. Perhaps a previous team did the last update, or another vendor. However, security is everyone’s problem. That doesn’t mean every developer has to take ownership of security bugs as though they have made them all themselves, but they do need to come to the party in terms of security being a shared responsibility. 

    Where to from Here?

    Sometimes, a mindset shift can be all it takes to make significant headway in solving a problem. We’ve talked about the rather frosty reaction a developer has to less than favorable pentest results, but what if they could turn it into a challenge? Perhaps they could think of the pentester as a friendly competitor; someone they can beat at their own game. After all, a security-aware developer that can eliminate common bugs as they write code is going to make their job much more difficult. By contrast, a developer with no focus on security is going to be comprehensively bested by their pentester counterparts when they can easily break their software. 

    Pentesters and developers may not be joined in harmony 100% of the time, but their relationship can be vastly improved when an organization addresses security as a key priority, and empowers teams with the right knowledge and tools to succeed — especially developers. It comes down to whether a company-wide, positive security culture is a priority, and if we are to fight the (currently) losing battle against common vulnerabilities, it absolutely should be.

  • Knowledge Is Power

    Knowledge Is Power

    I was doing some research over the weekend, and it became abundantly clear that you can find an Ansible script to do pretty much any repetitive task in your infrastructure. This is good news for Ansible users, and the pool of available scripts will, no doubt, continue to grow as critical mass effect kicks in.

    My concern is for those that find a script that does what they want/need, and drop it into their infrastructure. This approach may be a good short-term fix, but is full of potential peril. Even a cursory read of a complex script is insufficient.

    We all know the watchwords of the day are “speed” and “agility,” but take some time to test and get to know those scripts. When (not if) they fail, you’re going to want a clue of what is going on in there to track it down, and you certainly want to understand what is being pulled into your infrastructure. While the community is likely to catch malicious scripts relatively quickly, the inclusion of questionable sources and cases where your infrastructure causes a script to have issues are more likely to make it to your desk.

    The easiest validation cycle is to:

    • Read it for understanding. Know what it is doing, why it is doing it and what it is including.
    • Run it in an isolated environment. A pod, a VPN, whatever, just somewhere that you can control ingress and egress.
    • Scan the results as you would anything in production. Make certain sources are valid for your organization, make certain inclusions are safe and validate that any vulnerabilities in the resulting systems are known and acceptable to your organization.
    • If you can, do runtime testing upon it to catch any real-time changes. Most “changes after install” type of software is either well known or not included, so this step is more peace of mind than mandatory.

    If your thought was “takes too long, not necessary,” the risk is on you. But on the list of weaknesses in DevOps, that mentality is definitely in the top three. “Takes too long” is an organizational concern that is different everywhere. “Not necessary” is only true until it isn’t. As an IT professional, having at least a passing familiarity with the environments you are creating is part of the job. Fail to do it at your own risk.

    Much like there are trusted sources for the various types of repo we access, trusted playbook developers are a list you can curate. I know of a few that, in a pinch, I’d be willing to drop a script from into my architecture with only a cursory review, then give it a more thorough going over later when there is more time. But I know that because I’ve read their scripts, and I get that they are also careful.

    While I have Ansible scripts on my mind because that’s what I was looking into, this advice applies to any and all automation scripts that pull from outside sources. Python and Node do a ton of this too, but only highly complex projects pull from a lot of disparate sources. Ansible and other infrastructure automation tools pull from a variety of sources almost by definition (because infrastructure is complex), so they deserve more attention.

    In the end, automating all the things changes nothing about responsibility. So, take the time to make certain you know what is going on in your environment, and why. If you are really motivated, take the time to sweep through and remove any references you are certain you don’t need — but nested dependencies make the ROI on this type of work minimal, in my experience.

    Meanwhile, you are keeping a ton of servers running and users happy. Keep rocking it, and keep looking for solutions like automation scripts to make you more responsive — just don’t relinquish your authority for the datacenter to random submitter on the internet — because responsibility will still be yours.

  • How DevOps Tools Are Contributing to Automation Initiatives Across the IT Landscape

    How DevOps Tools Are Contributing to Automation Initiatives Across the IT Landscape

    Prolific advancements in IT, specifically in agile methods and culture over the years, has led to burgeoning needs for a more holistic end-to-end software delivery approach. Agile development is an umbrella term that encompasses numerous incremental and iterative software development practices — including Kanban, SAFe (Scaled Agile Framework), Scrum, Extreme Programming (XP), Lean Development, among others.

    Each methodology works on its own unique approach. However, the core commonalities they share lie mainly in their values and vision. The methodologies integrate continuous feedback from iteration to refine and deliver a successful software system. They include various forms of continuous evolution, such as continuous integration, planning and testing, of both software and project. Furthermore, one of the most significant aspects of agile methodologies is their keen focus on empowering workforces to work together and collaborate more efficiently, thus simplifying and speeding up decision-making processes.

    Advancement in agile development is what has birthed the DevOps culture. The technology is born from the rising need to adapt to increasing software velocity and throughputs generated through agile methods.

    Understanding DevOps

    The DevOps market consists of a set of practices, designed to integrate and automate processes between IT and software development teams, enabling them to create, test and deploy software in a quicker and more efficient manner.

    The term DevOps, which was coined in 2009 by Andrew Clay Shafer and Patrick Debois, is a combination of the words “development” and “operations.” It is also indicative of a cultural shift in the IT sphere, which bridges the gap between historically siloed teams, such as operations and development.

    The DevOps framework essentially adopts a culture where the critical focus is on facilitating collaboration between the business, operations and development teams. Although tools are considered as pillars of the technology, DevOps is designed to deliver continuous value for customers, empower organizations to provide solutions of the highest quality at swift rates, and bring about the automation of the entire business process spectrum, from building to deployment of a product or application.

    Shifting Trends Toward Automation in Software Development Processes and the Subsequent Impact on DevOps Adoption in the IT Space

    The software development process consists of several integral parts, among which testing is a crucial task. It allows developers to monitor the functionality of the software, monitor and register performance, and detect any aspects that require remediation. However, with software projects becoming increasingly more complex as the years progress, and with project development life cycles accelerating, conventional methods of manual QA (quality assurance) testing fall short in meeting testing objectives within the desired time frames.

    Consequently, software developers are shifting their focus toward automated workflows and testing tools, in order to expedite testing routines without compromising on consistency and completeness in QA processes.

    DevOps practices are significantly dependent on automation to ensure frequent and timely deliveries across platforms. Studies have shown that nearly 85% of IT leaders claim that automation is a critical aspect of their DevOps strategies. This is attributed largely to the fact that automation empowers infrastructures to successfully field constant changes in codes that are associated with DevOps tools. 

    Furthermore, automation allows the effortless scaling of environments and unburdens skilled workforces from mundane and repetitive tasks, enabling them to focus on more integral aspects.

    Considering a process perspective, automation in testing involves the addition of test automation actions and tools to regular software development regimens. For instance, new builds delivered to a repository can be automatically subjected to automated testing procedures using a number of prescribed tools, with little to no need of developer intervention. The results are recorded with care, then compared to previous test runs and eventually submitted for review to developers. 

    Depending on the outcome, the software can then be cycled back to the developers for further remediation or approved for deployment. This is especially relevant in terms of the DevOps framework, which relies on continuous integration/continuous delivery (CI/CD) pipelines.

    CI/CD pipelines are a prominent outcome of developer operations, allowing developers to validate the quality of the software with minimal intervention, and deploy apps and software to customers in a quicker manner.

    CI/CD platform CircleCI, for example, has recently raised over $100 million in Series E funding, in an effort to boost the development of their DevOps product suite. This move is indicative of CircleCI’s proficiency in delivering an automated approach to DevOps practices, including tools for companies to create, test and deliver software.

    How Is the Current Pandemic Impacting Digital Development?

    The COVID-19 pandemic has asserted a massive influence on the industrial landscape, driving workforces across the globe into remote-only work set-ups. With millions of people working from home for a seemingly indefinite period, enterprises across the globe are rallying to accelerate automation efforts in order to more efficiently deliver digital business applications to engage their employees from remote workspaces.

    According to various studies, nearly 73% of IT operations and DevOps teams are working toward maintaining and expediting initiatives for digital transformation throughout the current crisis, holding the belief that the value of digital offerings is set to witness a positive growth trajectory in the social distancing era.

    DevOps industry players such as JFrog are working toward providing enterprises and teams with access to cloud-based DevOps technologies, free of cost, particularly those which are undertaking active research and response efforts to combat the COVID-19 pandemic. These DevOps technologies include Pipelines, Xray and Artifactory, among others.

    While the exact impact of the global crisis on the DevOps environment in particular and the IT space in general is yet unclear, it is becoming more prominent that the strong focus on the prevention of such crises in the future — both in and out of the IT realm — is likely to set a great precedent for the growth of the DevOps industry in the years ahead.

  • Why DevOps and Federal Agencies Need Each Other

    Why DevOps and Federal Agencies Need Each Other

    In times of crisis, the U.S. government needs to bring in tech tools fast to help solve the problem.

    First, now is a vulnerable time and weaknesses are exposed in telework infrastructure as the government rapidly shifted to remote work. Cyberattacks are up as adversaries try to exploit the weaknesses. Second, federal agencies are also overloaded with traffic, to the point that their technology is reaching a breaking point.

    DevOps and DevSecOps may be the key to solving these issues for government agencies. Both, DevOps and DevSecOps, ensure delivery of a product is fast, secure and constantly evolving, which is vital to helping the U.S. government maintain continuity during the ongoing pandemic and set up future stability.

    The Air Force experienced these benefits firsthand when its internal DevOps team pushed out at least 10 updates a day, sometimes up to 30, and sustained that agile standard for some time. This agility, security and speed allows the Air Force to focus on the end-users and mission owners.

    If the government doesn’t have DevOps, it will lack the foundation needed to build a strong tech stack and work with other emerging tech solutions.

    DevOps Use Cases in Government

    With the government workforce spread out geographically, government networks are experiencing an increase in users plugging into virtual environments. Continuous patches and updates keep government systems and networks secure, functional and able to handle the load that comes with a surge of users. DevOps ensures the government has the tools and processes in place to support growth and prevent network failure under the additional stress of onboarding thousands of users each day.

    The government also needs tools that can help the distributed workforce maintain continuity, especially when it comes to security, operations and maintenance. Employees need correct access to do their jobs, but agencies must maintain control over what data is shared and how to confirm users are who they claim to be. Access management is vital in remote environments to ensure tighter control over who has privileged access.

    A modernized approach to privileged access tools means developers can hit targets of continuous delivery to improve systems and security. Otherwise, exploitable vulnerabilities are exposed to bad actors. Solutions for secure privileged access, especially across a distributed workforce, are key to scaling tech across government.

    With a hasty shift to remote work also comes change management, which is already difficult, not to mention the additional strain of a pandemic. In a crisis, federal agencies can’t wait for in-house solutions that take a long time to develop. The government needs commercial tech that can help build remote infrastructure.

    Federal Market Opportunities

    As the government responds to COVID-19, federal agencies are in search of technologies and tools that can support their missions. Many agencies are feeling immediate strain on their IT systems as part of the shift to telework, and have received reallocated and additional budget — with broad latitude on how they spend — and procurement flexibility to modernize their systems. This opens up opportunities for tech companies, like those in the DevOps and DevSecOps space.

    Tech companies that already work in the federal market know that while the government procurement process can be lengthy and complex, it can be well worth the effort for stable revenue. In times of economic uncertainty, working with the federal market can help companies build recession resilience.

    Another part of the opportunity now is that the government needs flexible and scalable technologies that can help them respond to the COVID-19 crisis, but first, they need DevOps tools for a strong foundation and infrastructure.

    The government needs DevOps and DevSecOps now more than ever, and there are commercially successful tech companies that can help with exactly that today. In return, the government can provide a sustainable source of revenue for emerging DevOps companies. Working together can be a win-win for these tech companies and federal agencies, and can help move us toward a more innovative, safer and healthier future.

  • Measuring DevOps Performance

    Measuring DevOps Performance

    As IT becomes increasingly central to our organizations, it is increasingly important to improve our ability to deliver innovations efficiently and safely. DevOps is a movement to reimagine the way we deliver software, with an emphasis on delivering value to end users through automation and collaboration. In the midst of complex changes to complex processes, it’s easy to lose sight of the most important point: Our “improvements” must deliver actual improvements. Measuring the performance of a software delivery team is the basic foundation on which you can assess the impact of changes.

    One of the main contributions of the State of DevOps Report has been to focus consistently on the same key metrics year after year. Although the questions in their survey have evolved and new conclusions have emerged over time, the four key metrics used as benchmarks have remained in place:

    1. Lead time (from code committed to code deployed)
    2. Deployment frequency (to production)
    3. Change fail percentage (for production deployments)
    4. Mean time to restore (from a production failure)

    The book “Accelerate” provides a detailed explanation of each of these metrics, and why they were chosen; those points are summarized here.

    The first two of these metrics pertain to innovation, and the fast release of new capabilities. The third and fourth metrics pertain to stability, and the reduction of defects and downtime. As such, these metrics align with the dual goals of DevOps, to “move fast, and not break things.”

    These also align with the two core principles of lean management, derived from the Toyota Production System: “Just in time” and “Stop the line.” “Just in time” is the principle that maximum efficiency comes from reducing waste in the system of work; and that the way to reduce waste is to optimize the system to handle smaller and smaller batches, and to deliver them with increasing speed. “Stop the line” means the system of work is tuned not just to expedite delivery, but also to immediately identify defects to prevent them from being released, thus increasing the quality of the product and reducing the likelihood of production failures.

    Lead time is important because the shorter the lead time, the more quickly feedback can be received on the software, and thus the faster innovation and improvements can be released.  The book “Accelerate” revealed that one challenge in measuring lead time is it consists of two parts: time to develop a feature, and time to deliver it.

    The time to develop a feature begins from the moment a feature is requested, but there are some legitimate reasons why a feature might be deprioritized and remain in a product’s backlog for months or years. There is a high inherent variability in the amount of time it takes to go from feature requested to feature developed. Thus, lead time in the State of DevOps Report focuses on measuring only the time to deliver a feature once it has been developed.

    The software delivery part of the lifecycle is an important part of total lead time, and is also much more consistent. By measuring the lead time from code committed to code deployed, you can begin to experiment with process improvements that will reduce waiting and inefficiency, and thus enable faster feedback.

    Deployment frequency is the frequency of how often code or configuration changes are deployed to production. Deployment frequency is important since it is inversely related to batch size. Teams that deploy to production once per month deploy a larger batch of changes in each deployment than teams who deploy once per week. All changes are not created equal. Within any batch of changes there will be some which are extremely valuable, and others that are almost insignificant.

    Large batch sizes imply that valuable features are waiting in line with all the other changes, thus delaying the delivery of value and benefit. Large batches also increase the risk of deployment failures, and make it much harder to diagnose which of the many changes was responsible if a failure occurs. Teams naturally tend to batch changes together when deployments are painful and tedious. By measuring deployment frequency you can track your team’s progress as you work on making deployments less painful and enabling smaller batch sizes.

    Change fail percentage measures how frequently a deployment to production fails. Failure here means that a deployment causes a system outage or degradation, or requires a subsequent hotfix or rollback. Modern software systems are complex, fast-changing systems, so some amount of failure is inevitable. Traditionally it’s been felt that there’s a trade-off between frequency of changes and stability of systems, but the highly-effective teams identified in the State of DevOps Report are characterized by both a high rate of innovation and a low rate of failures. Measuring failure rate allows the team to track and tune their processes to ensure that their testing processes weed out most failures before they occur.

    Mean time to restore (MTTR) is closely related to the lead time to release features. In effect, teams that can quickly release features can also quickly release patches. Time to restore indicates the amount of time that a production system remains down, in a degraded state, or with non-working functionality. Such incidents are typically stressful situations, and often have financial implications. Resolving such incidents quickly is a key priority for operations teams. Measuring this metric allows your team to set a baseline on time to recover, and to work to resolve incidents with increasing speed.

    In 2018, the State of DevOps Report added a fifth metric, system uptime, which is inversely related to how much time teams spend recovering from failures. The system uptime metric is an important addition for several reasons. First of all, it aligns with the traditional priorities and key performance indicators of sysadmins (the operations team). The number one goal of sysadmins is keeping the lights on or ensuring that systems remain available. The reason for this is simple: the business depends on these systems and when the systems go down, the business goes down. Outages are expensive.

    Tracking system uptime is also central to the discipline of site reliability engineering (SRE). SRE is the evolution of the traditional sysadmin role, expanded to encompass web-scale or cloud-scale systems where one engineer might be responsible for managing 10,000 servers. SRE emerged from Google, who shared their practices in the influential book Site Reliability Engineering. One innovation shared in that book is the concept of an error budget, which is the recognition that there is a trade-off between reliability and innovation, and that there are acceptable levels of downtime.

    According to the Site Reliability Engineering Book, Chapter 3, “Put simply, a user on a 99% reliable smartphone cannot tell the difference between 99.99% and 99.999% service reliability. With this in mind, rather than simply maximizing uptime, Site Reliability Engineering seeks to balance the risk of unavailability with the goals of rapid innovation and efficient service operations, so that users’ overall happiness — with features, service, and performance — is optimized.”

    The State of DevOps Report shows how these five metrics are interrelated (See Figure 1). The timer starts on lead time the moment a developer finishes and commits a feature to version control. How quickly that feature is released depends on the team’s deployment frequency. While frequent deployments are key to fast innovation, they also increase the risk of failures in production. Change fail percentage measures this risk, although frequent small deployments tends to reduce the risk of any given change. If a change fails, the key issue is then the mean time to restore service. The final metric on availability captures the net stability of the production system.

    Figure 1: Correlation between the five metrics.

    How the Five Key Software Delivery and Operations Performance Metrics Tie Together

    Together, these metrics constitute a team’s software delivery performance. The goal of any DevOps initiative should be to improve software delivery performance by strategically developing specific capabilities such as continuous delivery and the use of automated testing.

    How your team measures these capabilities is another challenge. But “Accelerate” makes a compelling argument for the validity of surveys. Automated metrics can be implemented over time, although the mechanism to do this will depend on how you do your deployments. Salesforce production organizations track past deployments, but it’s not currently possible to query those deployments, so you would need to measure deployment frequency (for example) using the tools you use to perform the deployments. Salesforce publishes their own service uptime on Trust, but that gives no indication of whether critical custom services that customers build on Salesforce are in a working state or not.

    Surveys provide a reasonable proxy for these metrics, especially if responses are given by members of the team in different roles. Guidelines for administering such surveys are beyond the scope of this book, but your teams’ honest responses are the most critical factor. Avoid any policies that could incent the team to exaggerate their answers up or down. Never use these surveys to reward or punish; they should be used simply to inform. Allow teams to track their own progress and to challenge themselves to improve for their own benefit and for the benefit of the organization. As it says in the Agile Manifesto “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”

    Metrics provide a reliable, long-term indicator of how your software delivery team is performing. They open the door for your team to experiment with different approaches and assess their impact using a common standard. The key metrics described here are important because they emphasize end-to-end performance, and thus incent teams to focus on collaboration towards this common goal. Balancing velocity with reliability is critical, thus these metrics should be viewed together, to ensure that one goal is never emphasized at the expense of the other.

  • AWS Aligns With Slack to Foster Dev Team Collaboration

    AWS Aligns With Slack to Foster Dev Team Collaboration

    Amazon Web Services (AWS) and Slack Technologies have formed an alliance through which they are moving to transform the way development teams collaborate.

    Under the terms of the multi-year agreement, Slack will migrate voice and video calls to the Amazon Chime real-time communications service.

    In addition, AWS and Slack are integrating Amazon AppFlow integration with Slack, which enables data to be transferred securely between Slack and AWS services such as Amazon Simple Storage Service (Amazon S3) and Amazon Redshift. The two companies are also promising to make it possible to transfer data bi-directionally between multiple Slack channels and AWS services in a single flow.

    AWS and Slack have also integrated the AWS Key Management Service with Slack Enterprise Key Management (EKM) to distribute and control cryptographic keys. That capability builds on an instance of EKM for Slack’s Workflow Builder automation tool that was released last month.

    Finally, the two companies are committing to make the AWS Chatbot service accessible from directly within Slack.

    Slack, which already makes extensive use of the AWS cloud, has emerged as a de facto collaboration tool within many organizations that have embraced best DevOps practices. Eron Kelly, head of product marketing for AWS, said this alliance will make it easier for development teams to employ a wide range of communications tools at scale.

    Many DevOps teams already rely on Slack Calls to launch video and voice calls whenever a subject being discussed using text messages requires a deeper level of interaction. As reliance on Slack Calls increases, the software development kit (SDK) for the AWS Chime service that Slack will invoke will ensure the quality of the video and voice calls, said Kelly.

    In many ways, DevOps processes revolve around a mix of asynchronous and synchronous communications. The challenge has been to find a way to enable DevOps teams to use those tools in a way that enhances rather than detracts from productivity. Managing multiple Slack communications channels can easily become overwhelming, especially if developers are working on multiple projects at the same time.

    Ideally, DevOps teams will eventually become more adept at determining, for example, what messages, such as status updates, are best delivered asynchronously versus urgent requests that need to be made via a communications medium such as Slack.

    In the meantime, the COVID-19 pandemic has clearly increased reliance on communications services delivered via the cloud. With most members of DevOps teams working from home, tools such as Slack have become a virtual substitute for walking over to the next cube in an office to resolve an issue. Once development teams return to the office, it will be interesting to see to what degree tools such as Slack have become the default means of communication, even with fellow employees who may be only a few feet away. In fact, such tools may play a critical role in helping organizations adhere to social distancing policies even after employees return to the office.

    As philosopher Marshall McLuhan once observed, the medium is the message. Today, however, those messages increasingly are being delivered within a stream of real-time communications that is becoming more challenging to manage.

  • COVID-19, the New Normal and the Indisputable Importance of Mobile App Security

    COVID-19, the New Normal and the Indisputable Importance of Mobile App Security

    As the United States emerges from COVID-19 lockdown, it’s not back to business as usual. COVID-19 remains a very serious risk, and until a vaccine or treatment arrives, we will all need to remain vigilant. Further, it’s also certain that the new normal post-COVID-19 world represents a fundamental transformation in how businesses operate, with mobile business models taking center stage.

    For most businesses, their storefront was their most important asset for awareness and revenue generation prior to the pandemic. Having a good physical location was critical to bringing in foot traffic and allowing customers to easily discover them. In the new normal, however, digital location has become far more important as consumers are increasingly turning to their mobile devices. So, it’s critical for businesses to make apps that are easy to find and use in order to continue generating revenue going forward.

    The combination of the pandemic and growing mobile usage has made developing and updating apps not just a nice-to-have marketing tool, but a necessary task for business survival. An enormous number of previously casual mobile users are now depending on mobile apps for their banking, shopping and other day-to-day transactions. As such, businesses need to redesign their apps to ensure they’re easy and intuitive to use so they don’t lose those potential customers. This requires developers to iterate even faster to deliver an engaging, glitch-free experience.

    Unfortunately, in the mad rush to ship apps as fast as possible, security often gets short shrift in favor of features and functionality. According to the Verizon Mobile Security Index 2020, even before the pandemic hit, 43% of app developers said they knew they were cutting corners on security to “get the job done.” 

    Features Trump Security … Until They Don’t

    Now with business models hinging on mobile, developers can no longer rest on “features first, security later” mentalities. The struggle here is that implementing mobile app security is hard to do. It takes time, often extending the development cycle, and it’s expensive. Even if development teams are committed to implementing security, they may lack the skills to do so — good iOS and Android security engineers are scarce and in high demand.

    Eventually, however, poor security will bring consequences. Cybercriminals operate much like nimble startups, searching for opportunities, creating minimal viable malware and then continually improving it to become more effective. Case in point: the EventBot malware for Android that was discovered in April. It masquerades as a legitimate app, such as a banking app or other popular consumer that, once installed, harvest unprotected data from other apps on a device. EventBot also can intercept text messages (SMS) sent to the device, which enables it to capture the identity verification codes used by multi-factor authentication solutions. With access to these codes and the unprotected user credentials found in the app, hackers can easily launch account takeover attacks on tens of thousands of unsuspecting mobile app users.

    What’s more, the new normal post-COVID-19 world is causing a huge increase in app usage, cybercriminals and hackers believe that the time is ripe to start exploiting known mobile app security flaws. For example, it’s not a secret that most mobile apps lack basic encryption. Similarly, most apps can be tampered with, repackaged and distributed on non-official app stores. With malware like EventBot that can be embedded in popular apps that users already trust, the game for mobile app exploits has been upped dramatically.

    EventBot is just one example of why it’s critical for app developers to encrypt all app data (including the strings, resources and in-app preferences that are stored on the device), obfuscate code and shield apps from tampering and reverse engineering efforts. Doing so will stop EventBot and other potential attacks, and prevent cybercriminals from using apps as Trojans for additional attacks.

    Solutions to the Security Development Challenge 

    If a development team decides the best route is to implement security themselves, they should first make sure they have the appropriate skills in house. If so, a good start is to address each of the OWASP Mobile Top Ten vulnerabilities. 

    Other development teams turn to security software development kits (SDKs), which they will integrate into their apps to provide security. This reduces the scope of the coding but still requires developers to have extensive security experience, and it’s critical to vet SDKs before integrating them, because rogue and vulnerable SDKs are a serious problem in the mobile app industry.

    A final option is security automation through artificial intelligence. It’s fast, taking just minutes to completely secure an app without any coding, and compared to manual coding, it’s inexpensive. But, as you should do anytime you’re outsourcing security, do your due diligence to make sure the platform truly secures the app without introducing additional vulnerabilities.

    The new normal has elevated the importance of mobile apps as the primary way customers interact with businesses, and cybercriminals are taking note. So, in the race to provide an engaging, intuitive experience for customers, don’t neglect their safety. Focusing on features and functionality at the expense of security might pay off in the short-term, but the long-term consequences could be grave. Identify the security development model that will work best for your team, but make sure you implement security quickly. Cybercriminals will not wait for you to prepare before launching new, even more devastating attacks.

  • 3 DevOps Security Best Practices Your Organization Can’t Afford To Ignore

    3 DevOps Security Best Practices Your Organization Can’t Afford To Ignore

    CI/CD pipelines are at the core of daily operations for many businesses today. These processes, when set up correctly, help to keep the delivery process consistent by automating many manual tasks and providing visibility into how the software is being worked on.

    DevOps is also the place in your technology stack where your infrastructure has access to many different resources, from development and production environment to analytics keys and code signing credentials. The more resources your pipelines have access to (secure secrets, proprietary code, databases, etc.), the more important it is to keep your CI/CD system secure.

    In this article, we talk about the different parts of DevOps security for CI/CD pipelines and highlight a number of ways to improve on these aspects. We usually divide security practices into three parts that you can address with different kinds of solutions:

    1. Secure pipeline configuration
    2. Code and Git history analysis
    3. Security policy enforcement

    All three aspects are equally important. Let’s take a deeper look at these categories and ways to address them.

    Secure Pipeline Configuration

    It is possible to use your CI/CD pipeline configuration to make security issues less likely to happen.

    First, safely store secrets that you use in your pipelines for connecting to databases and third-party services. Depending on the CI platform you use, you may have the option to use encrypted-at-rest environment variables, or to use the contexts feature. Contexts are used to provide access to environment variables across projects. Their use can also be restricted to specific security group members as defined by the organization’s administrator.

    Another option is to use a third-party solution to dynamically fetch secrets from their secure storage for your jobs. To ensure that none of your credentials become exposed to those who shouldn’t see them, you should never check these into your repository in plain text, even if the repository is private.

    Second, for sensitive files like code signing keys, I recommend adding an additional layer of isolation between your encrypted files and your repository for added security. Keep them encrypted in your repository and store the decryption key in the environment variables or contexts, only decrypting them inside your CI/CD jobs when they are needed. This makes it even more difficult to expose or leak these highly sensitive files.

    Third, you never want to leave your CI/CD environment running without monitoring it. Therefore, make sure that the containers and VMs used for your pipelines are destroyed after jobs that require sensitive information finish running. 

    Fourth, pay extra attention to how your CI/CD system handles builds for pull requests that come from forks of your repository. If your build stages require secrets to work, forked pull requests might gain access to them through your pipelines. 

    By addressing these four aspects you can significantly reduce the risk of security incidents due to your CI/CD pipeline configuration. Next up, code and Git history analysis.

    Code and Git History Analysis

    We all love Git for its comprehensive history of changes, being able to look through the history of the project with a simple CLI command and see who has most recently changed a line of code, among other benefits. This also means that any sensitive information that makes it into the Git history can be accessible to attackers if a Git repository is exposed, even if the most recent state of the repository doesn’t contain the secrets anymore.

    Trufflehog and GitLeaks are two tools that can help you identify secrets that have been committed to the codebase so that you can deactivate and replace them. These two tools will also scan your Git history for traces of secrets that your team might have added to the repository in the past.

    Once your Git history is clear of secrets, you can proceed to the next level: making sure that your current revision doesn’t contain any vulnerable dependencies.

    Static Application Security Testing (SAST) techniques can look through the application in your commit and analyze its dependencies. If any dependencies contain any issues or known security vulnerabilities, your commit will be marked as insecure and won’t be allowed to proceed to deployment.

    Dynamic Application Security Testing (DAST) techniques go one step further and spin up a copy of your production environment inside your CI job in order to scan the resulting containers and executables. The dynamic aspect helps the system catch dependencies that are being loaded at launch time, for example, as those won’t be caught by SAST.

    Security Policy Enforcement

    Some security aspects can’t be statically checked based on known vulnerabilities, but rather are specific to your particular company, and therefore need to be codified as policies. These can take the form of automated or manual compliance checks.

    Certain tasks such as reviewing the list of all accounts that have access to your repositories, or making sure your onboarding and offboarding processes are in sync, will be manual. We recommend that you set reminders to perform these regularly.

    Some tasks actually can be automated quite easily. For example, third-party services can provide a convenient way to codify a set of rules that will be matched against in your CI pipeline, e.g., proving compliance with the regulations that govern your data. If there is a mismatch, the build will fail.

    Some of the Important Parts Can’t Be Automated

    Keeping your CI/CD pipelines secure is more important today than ever before as automating your CI/CD practices requires more access to development and production environments, analytics keys and code signing credentials than in the past. The hard thing about security is that if you only get one of the three categories right, you won’t be 33% secure. You will be 0% secure until you get all the bases covered.

    With the approaches outlined above, you’ll be in a much more secure position than you would be without them. However, some of the aspects that matter the most for CI/CD security can’t be automated or detected with an automated policy. These aspects are shaped by the behavior and culture within your team.

    A DevSecOps Mindset

    Your team is your most valuable set of security researchers because they know your applications best and constantly work on them. Have you implemented a clear process for your teammates to report security issues if they find any, and do you dedicate time to fix the security flaws they find? Do you celebrate those who report issues openly? In many companies, the answer isn’t an immediate “yes” to these, and it should be.

    How long does it take for someone on your team to report suspicious activity? Do they need to post in multiple Slack channels to get attention, or is there a streamlined process to get an issue like this looked into? How do you treat those who report false positives — do you blame them for wasting the team’s time, or do you hold a blameless post-mortem so that everyone on your team learns from the mistakes? All these questions can make or break your security-minded development culture.

  • Review, Review, Review

    Review, Review, Review

    Ossification is an issue in IT, and always has been. It is borne largely out of staff and management knowing what to expect from existing software/infrastructure, and the deep product knowledge that allows teams to make existing systems do what is needed. But this becomes a series of rocks that the ship of progress smacks into.

    Contrary to popular belief, DevOps and other new methodologies are not immune to this problem. Quite the contrary, we’ve spoken with people that say “We’re a Jenkins shop,” which sounds exactly like “We’re an Oracle shop” of a decade ago. The wave just ushered in new tools, it didn’t change tendencies. OK, it did more than usher in new tools, but it didn’t change tendencies.

    So, it’s time for a friendly reminder: Review what you are doing, review cloud usage, review tools in the toolchain, review methodologies, review security posture (which has suffered in this environment for far too many shops) and review customer satisfaction.

    Many organizations used the push of Agile and DevOps to update stagnant parts of the infrastructure and development toolsets. Don’t let them become stagnant again. Just as you don’t want change for the sake of change, you don’t want to stay with one tool or methodology just because it is comfortable.

    This isn’t some easy task that I ⁠— or anyone else, no matter how much they try to tell you differently ⁠— can dictate to an organization. There isn’t a handy list, because no two organizations are exactly the same. Not in toolsets, not in culture and not in appetite for risk. The only thing we can offer is guidance to look closely at where you are, what’s available in a given area ⁠— Application Release Orchestration, for example ⁠— and evaluate the available tools just like you were making a fresh choice and did not have a tool in place. Then do the same for processes. Is this the best process for what we’re trying to achieve? Are there tweaks we can make to it? Should we completely change the process to take in changes in environment that have happened since we adopted this process?

    Finally, if you don’t have someone responsible for monitoring cloud costs, appoint someone. Get them the tools, help them understand where the money is going and what cloud-based server usage is. Could these servers be spun down outside of business hours? Could we follow the sun better by staggering cloud data centers in which cloud servers are active, rather than hosting them always in one location or leaving them on 24/7 in multiple locations? Is one team using a cloud feature no one else is? If so, how does it apply, and are the other teams not using it because they don’t need it, or is there some other solution they’ve adopted?

    Basically, question everything. We’re launching a new project that will be a simple dynamic app. Make a few selections from a limited list, get a summary of what you should do and why. It is all really quite simple. Should we host in a web page or make it a standalone mobile app? If mobile app, what tools for development and what targets? Integration of Git — should it be local or on one of the big providers? Do we want any team member to trigger builds on commit or do we want someone managing branch merging before-hand?

    These are all things we already do on other projects, but we’re digging in and seeing what could be better. What could we improve, and what would the rewards for that improvement be? It takes a bit of time and some discussion. We’re confident we’ll burn through it in the next day or so and come up with a plan for development and architecture that best suits the app, but is informed by where we’re at now. It’s been a fun exercise, to be honest. If nothing else, it gave us a strong reminder that any given cloud provider’s “simple” pricing is not that simple, and we need to pay more attention to app characteristics versus fees.

    You’ve rocked it this far, even kept the lights on during the great shift to working from home. Don’t stop improving the environment though, find innovative ways to make it better and keep clipping along.

  • Using this Time to Optimize Your Dev Team’s Workflow

    Using this Time to Optimize Your Dev Team’s Workflow

    As Andy Warhol once said, “They always say time changes things, but you actually have to change them yourself.” This more than ever resonates with the change that companies have found themselves experiencing of late. The more the world embraces digital methods of work, the more that change becomes paramount to a successful workflow. It is crucial to adapt, whether that means throwing out the old or bringing in the new — or a mix of both. 

    The current global pandemic has forced companies to reconsider how they operate, and it’s no different for their development teams who now find themselves distributed. So the question becomes: How do you go about using this time as an opportunity to optimize your team’s workflow?

    By making the necessary changes, you can not only increase productivity, but also make your team happier. As the saying goes, “Happy devs, happy life.” Let’s look at automation, independence and understandability as key components of your digital transformation. 

    What’s Killing Your Team’s Productivity 

    The Problem: Too Much Bureaucracy

    Being too dependent on management or your colleagues is often a bottleneck at many companies, especially larger traditional ones. When we were all in offices, this problem could be mitigated more easily — but today, things are shifting toward more ownership, and having everyone distributed means you need to be able to get the job done yourself.

    The Solution: Shift-Left

    Bureaucracy shouldn’t be in place just for bureaucracy’s sake. While the goal of those processes are typically risk mitigation and governance, DevOps advocates have been arguing that modern tools and techniques allow shifting-left much of that burden directly onto developers. Use this time to empower developers to build in alternative safeguards relying on automation, self-service and measurement instead of change boards and forms.

    The Problem: Status Meetings

    Status meetings are the bane of all developers’s lives. Nobody will argue that they’re not important, but who wants to sit through meetings when you can be spending time elsewhere? — for instance, at your desk, sipping a nice cup of coffee while working on your code and finding that elusive bug.

    The Solution: Asynchronous Communications

    What used to be done with lots of in-person status meetings doesn’t always translate well to a video conference — water cooler talk just isn’t the same via teleconference. While it’s important to try and maintain that personal touch, it’s even more important to enable asynchronous communication methods as much as possible — both free-form (such as email) as well as structured (such as ticket management). Moving to this new mode of operation requires developers to be more independent. This is another place where automation can play a huge role, formalizing tribal knowledge into scripts and tools, making them easily accessible for all.

    The Problem: Lack of Data

    How many hours have you spent staring into the great abyss of code, hoping and praying that you’ll be able to deploy soon? You’re not alone, we’ve all been there, done that. Waiting for deployments is par for the course of living life as a dev. The source of this long wait? A small, missing piece of data. And this need for data creates what I like to call the engineer’s dilemma: Is it better to fly slow but have the data I need, or fly fast but perhaps be making worse decisions due to a lack of data?

    The Solution: Understandability

    Understandability tools allow quick access to data in real time. This results in shorter, more efficient workflows, which in turn improves team velocity and code quality substantially. Understandability is not the same thing as observability. Think of it like this: Observability helps teams quickly identify where a problem is, but you still need a tool that can quickly fetch the data in order to solve the problem. In other words, if data is the new oil, go find the understandability tool that you think is the best drill.

    Conclusion

    Change is not a concept most like. But now that it’s happening anyway, it creates an opportunity to introduce a healthier, more productive workflow. Grab that cup of coffee and ask yourself: What change would benefit us most? Find it, implement it and watch as your distributed teams become a smooth-sailing ship in no time.