devopsy.kinsta.cloud

Category: Blogs

  • Stopping Mobile Fraud With Automation and DevSecOps

    Stopping Mobile Fraud With Automation and DevSecOps

    Mobile fraud is a multibillion-dollar problem that’s only getting worse. One study showed that mobile click fraud alone rose 64% during the first few months of the COVID-19 pandemic. In fact, during the pandemic, about one in five mobile ad clicks were fraudulent. And that’s just one form of mobile fraud.

    Weaponized botnets, mobile malware, mobile Trojans and mobile cheat engines and dynamic instrumentation tool abuse—the methods that fraudsters use to defraud companies and consumers via mobile devices—are constantly expanding in number and scale. 

    Historically, organizations have fought against fraud through monitoring and data analysis. As transactions and data come in, analytics and AI work to discern legitimate events from fraudulent ones. These systems require training and a lot of data to become proficient at identifying fraud, and they can become very accurate. But fraudsters are always changing their tactics to evade these systems, and, as many of us have experienced, false positives are still a big problem. If you’ve travelled enough overseas, likely you’ve had your bank account or credit card frozen until you could call your financial institution to let them know that your purchases were genuine. 

    Another challenge with existing mobile fraud prevention solutions is that they are designed to protect networks and network resources first. As a result, while fraud is occurring and fraud prevention systems are learning, mobile end users are still being affected. Also, when a bad act is identified and a system enforces protections against fraud, these enforcement points typically block network traffic. That protects the network, but the consumer and their mobile app remain at risk. Once a fraudster compromises a mobile app, they may be able to attack through other vectors, even if one network is shut down. For instance, if a consumer unwittingly downloads and launches EventBot to their mobile device thinking it was a legitimate app, it may compromise multiple banking apps. If one bank shuts down its network after detecting fraud, EventBot may have other opportunities on the device.

    DevSecOps

    As this chart shows, once fraud takes place, damage to a brand, customers and the bottom line begins immediately. Even if it is detected and halted, the harm it has already caused can’t be undone. Mobile app publishers and developers need to stop mobile fraud where it starts, and that means implementing mobile fraud prevention inside the app within a DevSecOps framework. By blocking the technical methods used to carry out fraud via mobile apps, organizations can make massive inroads toward stopping fraud before it ever gets to the network level, preventing both customers and organizations from getting scammed.

    Why Traditional Security Approaches Don’t Work in Mobile Apps

    To achieve a multi-layered defense against fraud, a mobile app should include app shielding, anti-tampering, code obfuscation, data encryption (including strings, resources, preferences), jailbreak/rooting prevention and man-in-the-middle prevention (such as certificate pinning or certificate validation). Certainly, there are commercial SDKs and third-party libraries that can provide these functions, with some limitations. But these approaches typically still require extensive development work and introduce additional frameworks or programming language dependencies and incompatibilities. 

    They don’t solve the fundamental problem of implementing fraud prevention inside a mobile app—it’s too expensive and time-consuming.  The root of the problem rests in the way mobile apps are developed, released and updated in a modern agile organization. DevOps depends on processes that are dynamic, automated, agile, integrated, iterative and continuous. Mobile app security has traditionally been very different: Static with manual, line-by-line coding. This way of implementing manual security will never work in an agile delivery model.

    The promise of DevSecOps is built-in security, delivered as a fundamental part of a release process where security is designed into the process at each and every phase of the life cycle. The security model can be delivered and evolved atomically, iteratively and dynamically so that it adapts to fit the app.

    The future of fraud prevention lies in the mobile app, but to make this possible, the industry needs to apply AI and ML to automate fraud prevention and security implementation into mobile apps on time within an agile, fast-moving and iterative app life cycle and release process.

  • Building on Terraform: Evolution, not Revolution

    Building on Terraform: Evolution, not Revolution

    Terraform is widely adopted among both enterprises and service providers who embrace infrastructure-as-code (IaC) as their model for infrastructure provisioning automation in DevOps environments. And while the technology has limitations, innovations seeking to unseat Terraform are highly fragmented and have seen tepid adoption. This points to a market that’s settling on a de-facto standard, so rather than trying to replace Terraform with the next great innovation, infrastructure operators should instead find ways to extend and augment Terraform to suit their unique needs. Let’s explore several of the better-known shortcomings of Terraform, some drivers for the rise in new alternatives to the technology, and what users need to consider when contemplating a different path.

    Without a doubt, Terraform is big. Across the big three public clouds, the number of Terraform provider installs tells the story: At the end of August, there were 457 million provider installs for AWS, and around 72 million each for Azure and GCP. Yes, compared to longer-in-the-tooth infrastructure configuration management tools like Ansible, Chef and Puppet, IaC as a category is still early. But, the trend is clear: IaC is becoming widely used at scale, and Terraform is the IaC platform most enterprises go with.

    It’s also no secret that there are shortcomings to Terraform. A quick search on Reddit and Hackernews unearths the litany of complaints, some more valid than others: 

    • It’s tricky creating a Kubernetes cluster and then adding a resource to it
    • Challenges passing values through a config or dumping a pure representation of an elevated config
    • Cluster deletion is a pain
    • An overall lack of visibility into what’s happening during the provisioning process
    • The need for glue code to drive everything is a drain on developer productivity
    • Lingering challenges with backward compatibility
    • Continuous update challenges and the need to refactor
    • Scaling across multiple workspaces and multiple cloud accounts
    • Rollbacks can be tricky, especially when dealing with immutable infrastructure

    Terraform’s parent company, HashiCorp, of course, steadily works to address these and other pain points. But the whole thing highlights an interesting question: Why is Terraform so popular?

    Terraform is the only IaC solution that meets the needs of developers who want an IaC tool that preserves state, fits well into their CI/CD pipelines, is intent-based and gives them the right level of granularity. Terraform, even with its challenges, is better than the alternatives in these regards.

    Here’s why: 

    Terraform was, from the start, designed for and targeted to developers. It maintains state and supports iteration, making IaC a powerful model for automating infrastructure. PaaS solutions, by contrast, are not robust enough for most developers who do not need or want all of the infrastructure decisions abstracted away. Most cloud management platforms suffer from the least common denominator problem, and even though the technology has improved in this regard, it’s a solution that developers generally abhor for its granularity; it’s geared more to the needs of IT management. Configuration management tools, as an alternative, are not intent-based, so flow becomes too complicated.

    It’s interesting to note that state (maintaining the mapping between the resources that you’ve represented in your definition files and the actual physical resources that are in the cloud provider) was added into Terraform, helping resolve the issue of infrastructure drift. Terraform also realized that developers don’t necessarily need a UI. These tangible examples of the developer-centric design ethos at HashiCorp point to why it is so widely adopted.

    So, what’s driving the interest in new approaches to infrastructure automation? The proliferation of domain-specific use cases and an ever-present need to reduce time-to-value are the leading reasons. Domain-specific automation options are becoming more dominant over OOB solutions because they take too long to get to automation; one tool does not fit all. Domain-specific automation addresses issues such as security automation, AIOps, DevSecOps, security compliance, integration with container scanners—not just generic orchestration. People are looking for value very fast, which can be found only with domain-specific automation—and therefore, fragmentation happens as different solutions emerge to address those cases.

    There are several interesting alternatives to Terraform. Terragrant is used along with Terraform to expand its capabilities to multiple workspaces. It allows you to use the same template across many environments. Pulumi is another alternative that has achieved a measure of adoption and visibility. It provides native code support which addresses the problem with YAML file development complexity. Another more ambitious undertaking is the Crossplane open source project. It takes a Kubernetes-centric approach to creating a universal, cross-cloud control plane. It looks good conceptually, but it’s a fundamentally different approach, and adoption is nascent.

    Turning to one of the Terraform alternatives isn’t necessarily a bad idea, but is fraught with landmines. In the end, this choice is less about technology and more about adoption and ecosystem momentum. Vendor and professional investments in tools, training, and skill sets surrounding Terraform is a formidable moat, and it cannot be easily replicated. Tools that would win over Terraform would have to win adoption, and achieving adoption means delivering all the enterprise-grade features and support that Terraform users currently enjoy. Chef and Puppet are examples of how this could happen. Both were successful technologies, but they were slow to react to the challenge presented by Terraform. Ultimately, these projects suffered from a legacy mindset and were overtaken by Terraform’s more focused, simple solution.

    Most of the issues with Terraform lie in managing it at scale. When you have many workspaces and templates, it becomes hard to manage; it’s also a challenge that templates cannot be generalized among environments, resulting in a lot of template redundancy. Also, Terraform still needs help handling Day 2 operations and visibility into drift detection. 

    When considering ways to extend Terraform to address a broader set of use cases, you should first consider how well things will interoperate across your environment. If you have multiple clouds, multiple service layers and multiple cloud vendors, this is probably the most important factor. Also, investigate any new technology’s ability to support the things about Terraform that are critical to your Day 2 operations. Things like secrets and state come to mind. You’ll want enough management and monitoring in the new solution without it being a new burden to carry. You should be able to use your existing Terraform files, and you should be able to support all the versions of Terraform that you run. Compliance with governance and audit are key, and the new solution should support Terraform for both your on-premises/air-gapped environments and your cloud environments. Ideally, anything you add to Terraform would be open source, eliminating new complications from cross-vendor lock-in.

     

    Terraform use case 

    Architecture for how 8451 evolved its Terraform deployment

    For all its shortcomings, Terraform remains the best option on the market for DevOps organizations, and there are ways to manage it with fewer headaches and without developers wasting so much time writing glue code or reinventing the wheel over and over again. One way this happens is by integrating Terraform with meta-orchestration tools that allow Terraform to become more flexible in managing environments and workspaces at scale. The open source Cloudify project is one example.

    Yevgeniy Brikman gave a talk two years ago at HashiConf that has aged well. At the time, DevOps was still in the Stone Age. Today, it’s probably in the Bronze Age. Those of us building IaC today are being asked to deploy infrastructure that looks like modern cities, but the tools we have to work with remain relatively primitive. What we’re building takes too long to provision and it looks bespoke and temperamental at best or kludgy and fragile at worst.

    It’s a story repeated over and over: Teams try to build their own, homegrown solution first as a wrapper on top of Terraform. Ultimately, they realize that it’s unsustainable as a lack of engineering resources becomes a bottleneck. We need solutions that augment—not replace—what they’ve already built and what works. That’s almost always Terraform. With Terraform, DevOps organizations get a proven IaC option that’s targeted to the needs of developers, with a substantial ecosystem and market uptake. When considering its shortcomings, it’s best to embrace an evolution strategy, and not start from ground zero.

  • Era Software Adds SaaS Edition of Log Management Platform

    Era Software Adds SaaS Edition of Log Management Platform

    Era Software has unfurled a software-as-a-service (SaaS) edition of its platform for collecting and managing log data that is based on a time-series database it built.

    At the same time, the company has revamped the user interface of the underlying EraSearch log management platform that IT teams can deploy themselves to make it simpler to troubleshoot IT issues.

    Stela Udovicic, senior vice president of marketing for Era Software, said the EraCloud edition of the log management platform provides a more accessible way to access a log management platform based on an object storage system that can handle any type of log data.

    There’s also an indexing engine based on machine learning algorithms that enables the platform to scale better than rival platforms, she noted.

    Available via the Amazon Web Services (AWS) marketplace, EraCloud collects, indexes and analyzes log data from cloud applications, Kubernetes and IT infrastructure platforms to provide a centralized means of collecting data at a lower cost. In addition to being able to continuously monitor billing costs, an EraSearch cluster can now be provisioned in minutes.

    Era Software, previously known as EraDB, earlier this year raised an additional $15.25 million in financing to drive development of a log management platform that provides an alternative to open source platforms, such as Elasticsearch, that don’t scale as well. The result is a higher total cost as the volume of log data that is less efficiently managed via Elasticsearch steadily compounds. As part of that effort, the company also supports a subset of the Elasticsearch application programming interface (API) to facilitate transitioning from Elasticsearch.

    Many organizations today already have terabytes of log data they are trying to manage and store, noted Udovicic.

    It’s not quite clear how many IT organizations are considering alternatives to log management platforms. However, as the amount of log data that IT teams want to be able to store over a longer amount of time increases—in part to facilitate greater observability—the pressure to contain the cost of retaining all the data is also rising.

    Observability, in general, requires combing logs, metrics and traces—a specialized form of logging—in a way that makes it easier to identify the root cause of an issue by querying data. It differs from monitoring in that observability is achieved using a predefined set of metrics that track whether a platform or application is performing within expectations.

    It may be a while before most IT teams generally achieve observability, but the journey typically starts with finding a more efficient way to collect log data in both proprietary and open source formats. Rather than having to normalize all that data, Era Software is making a case for analyzing that data in whatever format it happens to be available.

    One way or another, a much larger percentage of the applications deployed by IT teams in production environments will have to be instrumented to achieve observability. The issue now is determining how best to make all the data collected from those applications available to DevOps teams at a reasonable cost.

  • Successful Software Development in a Hybrid Environment

    Successful Software Development in a Hybrid Environment

    The desire to work from home or from anywhere has only increased as the COVID-19 pandemic lingers. Employees cite a number of reasons why they prefer to skip the office, including cutting commuting times, being more productive, caring for family members and exploring new locations. People are increasingly comfortable with remote interactions and the new technology tools that have sprouted to support the work-from-anywhere trend. On average, employees said they want to work from home 2.5 days a week, according to a monthly survey of 5,000 Americans, while 73% of employees seek hybrid work environments where they have some flexibility in where and when they work. 

    The rapid spread of the COVID-19 Delta variant also prevented employers from enforcing a full-time return to the office. As a result, the new normal for today’s workforce will remain hybrid at best—a few days in the office and a few days working at home. We are a workforce in transition. For many companies, this has put an unnecessary strain on their digital transformation initiatives.

    A question remains: How can companies continue to innovate and execute on their software development initiatives with their employees separated from their teams and offices? Here are some of my recommendations for helping organizations to continue their software development projects in today’s hybrid work environment:

    Inclusive Environments

    Hybrid workforces require a highly inclusive environment. As a nearshore IT partner that works with hundreds of clients remotely, we ensure that our teams have the same access to managers, HR and the technology necessary to succeed on projects and to productively engage with clients. For example, we have learned that the best way to be inclusive is to take either an all-virtual or all-in-person approach during client meetings. It’s also critical to train your managers to provide the same advancement opportunities to employees regardless of whether they are in or out of the office. From the executive team on down, everyone needs to support employees, regardless of where they are. The same goes for onboarding new employees and partners—a mostly virtual process over the past 18 months. Increasing online mentoring and onboarding platforms to connect employees and to manage client projects while the workforce is hybrid is another significant approach we have taken to ensure everyone feels included and engaged. 

    Collaboration and Knowledge-Sharing 

    Employees were abruptly forced to transition away from the office where they were used to frequent impromptu meetings and lunchroom chats. For many, this shift had a negative impact on brainstorming, collaboration and knowledge-sharing and thus, innovation. We give our teams the flexibility to use different technology tools to make up for what they miss most in an office environment—regular, casual human interactions. Our teams use a variety of communications tools for frequent check-ins to connect with clients and their co-workers. As a result, our people say they are more engaged with team members and clients than ever before. They are doing more impromptu chats and “instant” meetings with colleagues and clients as if they were in the office. We also migrated to video conferencing as our communication channel of choice. Video conferencing tools increase engagement and allow team members to join virtual meetings anywhere from their preferred devices. The ability for teams to interact, share and collaborate wherever they are is essential to developing cutting-edge teams and software. 

    Trusting Cultures

    According to the Harvard Business Review, 34% of employees felt that their supervisors “expressed a lack of confidence in their work skills,” and an even more significant number of workers reported feeling that they needed to be constantly available to prove themselves. Sentiments like these highlight a lack of trust, impacting development outcomes. Employers need to trust that their employees are working diligently and consistently throughout the workday or whenever they can successfully complete the tasks at hand. For example, working remotely allows individuals to work when they feel most productive. Organizations need to recognize individual productivity cycles and understand that different team members may be productive during different times of the day, including outside of normal business hours. 

    Being transparent, forthcoming and open about expectations are keys to building trust with employees. Successful digital transformations and innovative software development are only possible with the right team and people. From a client’s perspective, these are the types of IT partners they want on their side. Technical aptitude and an environment where all employees feel valued, trusted and motivated to do their best are critical for developing effective digital solutions.

    Employees have made it clear that they don’t want to be back in the office full time. For many organizations, a return to normalcy centers on developing a hybrid workforce that can work effectively with digital development partners. Creating the right environment and strengthening employee relationships are keys to ensuring software development projects can continue successfully in today’s new normal.

  • What Developers Need to Know About Edge Computing

    What Developers Need to Know About Edge Computing

    More and more companies are living “on the edge” as they move from “I’ll think about edge computing” to “I don’t think our business can optimally grow and compete without edge computing.” Developers will play a key role in helping organizations use the edge, but their effectiveness will depend on having edge-aligned processes, products and people in place.

    Edge computing takes place near the physical location of either the user or a data source. It’s one way that organizations can use and distribute a common pool of resources across multiple locations. The edge model provides users with faster, more reliable services while expanding the potential of hybrid cloud computing for organizations. Customers and businesses alike will benefit from the way in which edge computing optimizes the use and application of artificial intelligence and machine learning by running data-intensive applications closer to the data source.

    While it is in many ways a question of architecture, applications are key to effectively exploiting the edge. Here are some of the things developers—and their organizations—need to know.

    Edge computing and cloud computing are not mutually exclusive. The best solutions will leverage both platforms. For example, centralized computing might be used for storage and compute-intensive workloads, while edge computing would be used to exploit data at the source, in near-real-time. Infrastructure and company culture need to be in place to support this, but developers must be thinking in terms of building applications that can cross and bridge computing platforms.

    Consistent tooling is essential. Developing for legacy IoT was very specific and required specialized tools and skills. The modern edge, in contrast, is part of a technology continuum, and developers need tools that they can use across that continuum, including:

    • Provider/enterprise core: The “non-edge” tiers, owned and operated by public cloud providers, telco service providers or large enterprises.
    • Service provider edge: Tiers are located between the core or regional data centers and the last mile access.
    • End user premises edge: Tiers on the end-user side of the last mile access can include the enterprise or the consumer edge.
    • Device edge: Standalone (non-clustered) systems that directly connect sensors/actuators via non-internet protocols. This represents the far edge of the network.

    Containerization enables flexibility, scale: Linux container technology wasn’t built for the edge, but the model fits perfectly with edge requirements for modularity, segregation and immutability. 

    Keep an “open” mind. Open APIs open up new opportunities. Developers using open APIs can access real-time data programmatically, making it possible to build applications that can leverage data as it’s “born.” Likewise, a focus on open source development tools and platforms will help to ensure that organizations can continue to leverage the edge as the model evolves. 

    What’s different is the devices. Developing for the edge is just like developing for any other platform–except that developers must write consistent and secure code that can be seamlessly copied to all device types. 

    Edge computing may not start with developers, but developers will play a huge role in organizations’ ability to leverage the edge. A focus on flexibility, openness and the future will help ensure that developers build applications that fully exploit and expand the edge advantage. 

  • Sales Incentives

    Sales Incentives

  • Building and Leading Product Teams in Uncertain Times

    Building and Leading Product Teams in Uncertain Times

    The world seems to be changing faster than ever, and there’s no reason to believe it will slow down anytime soon. The pandemic has compounded this sense of constant flux, with customer priorities shifting overnight. Against this backdrop, product teams need to constantly evolve their development work without losing focus or feeling they need to react to every small change in the market. 

    This requires team leads to throw out their old playbooks and develop a new strategy that allows them to deliver continuous innovation without creating chaos for their teams or losing sight of their north star. Road maps are shrinking from years to months, and are now being created with the understanding that they may become outdated or redundant at any time.

    Building and maintaining an organization that can not only survive but thrive in these conditions is a daunting task. It requires a structured, programmatic approach that also supports agility and adaptability. Those requirements may feel at odds, but they actually are not. Here are five tactics to help your product strategy evolve with the times. 

    Keep That “Day One” Mentality

    It’s important to keep the soul of your product top of mind. It helps to retain a “day one” mentality which means constantly reminding yourself why you started in the first place and what problem you set out to solve. That founding problem statement should serve as your guiding light, especially through times of rapid growth and change.

    Customer requests are, of course, a high priority and should always be thoughtfully considered. But ask yourself if the request is aligned with that day one problem you set out to solve. If it’s not, work with the customer to identify the underlying issue they’re really trying to solve for versus focusing on the symptom they’re experiencing. 

    Cultivate a Culture That Thrives on Change Without Succumbing to Chaos

    It takes a combination of strong organizational capability and the right culture to build a team that can thrive on ambiguity. Strong organizational capability is achieved through talent, architecture and execution skills. 

    Great product teams build an architecture that supports iteration and are able to execute toward their north star in small, rapid steps. Team leaders should build a culture that takes failure in stride and learns from it to improve each subsequent iteration in the product journey. 

    Change is inevitable, but chaos is avoidable. Leaders must create an environment that feels psychologically safe and supportive, so that team members feel confident in changing priorities and making rapid decisions. 

    Build an “Innovation Factory” or a System that Reduces the Potential for Chaos

    Innovation and agility require well-planned foundational resources. Following are the key steps that can help put these building blocks in place.

    • Think of velocity as a part of your product. Teams should be broken into small groups to autonomously drive the product forward, but in such a way that the overall experience feels seamless to end users.
    • Democratize the product concept phase by creating conditions for a large number of ideas. Everyone involved should be inspired to contribute, including employees, partners and customers.
    • Create a system for product development excellence. Fully automate your product development life cycle by approaching developer infrastructure as if it were a service, with clear service-level objectives within a CI/CD framework. Digitize each step in the product concept, development and deployment journey, with a focus on measurement, learning and iteration in every phase. Ask who and how many people are building versus approving at each stage and ruthlessly optimize and automate where possible. 

    Bring a Service Mindset to Everything you Do

    If you’re building a SaaS product, as many startups today are, you’re no longer delivering a product in the traditional sense. You’re delivering a service, and you’re responsible for the entire experience from start to finish. This means you’re in full control over when to release new features and how you measure user engagement. You can see where the friction points are, and where customers are delighted. We have a philosophy at my company of “less input, more output”—simplicity is the name of the game, where the output and value your service creates for customers is the only measure that matters. 

    The outcomes required may change as customer priorities evolve and new requirements flow in, but adapting quickly and effectively should just be a matter of (re)connecting well-built services within your platform to prioritize and address those changes. Leverage the cloud ecosystem in a way that allows you to safely bet on other players and partners who are moving as quickly as you are. Focus on making sure your foundation is strong; one that can provide a service at scale even as the outcomes customers want change over time. This allows you to stay laser-focused on the task at hand: Maximizing value for users. 

    Incorporate Users in the Product Development Cycle and Learn as you Go

    The cloud has transformed the way organizations build products and integrate feedback. The entire life cycle is digitized and measured. Product teams can move at a much higher velocity and in smaller increments, with the ability to make near-real-time adjustments along the way. But the greatest advantage a cloud environment offers is that we can now include users in the entire product life cycle, every step of the way. It gives us the ability to measure outcomes quickly and take action on the information that immediately benefits customers.

    It sounds counterintuitive, but product teams can prepare for the unknown. There’s no way to see what’s over the horizon, but with the right people, processes and tools in place, there’s no reason that uncertainty should derail you. There has never been a more exciting time to be an innovator or a builder. Meet the uncertainty ahead with open arms and a strategic eye.

  • Prioritizing Scalability, Reliability and Security in Engineering

    Prioritizing Scalability, Reliability and Security in Engineering

    As digital products and services are more deeply embedded in critical industries and infrastructure and the implications of problems grow in scope, engineering organizations are renewing their focus on building platforms that are scalable, reliable and secure. More importantly, they are recognizing that this isn’t the responsibility of an SRE or CSO, it’s an ethos that must be carried each day by every engineer on the team. This is an important shift—from teams and cultures that valued speed of delivery above all else to teams and cultures that make high-quality systems the foundation of everything they do.

    This isn’t easy, and big cultural changes don’t happen overnight. There are many things that can go wrong when the team is operating in the legacy mode of prioritizing quick time-to-market but there are also some strategies and best practices for beginning the process of cultural change across your engineering organization.

    The Old Way of Doing Things

    You’re the engineer on call, and you get woken up at 3:00 a.m. by an alert that tells you an app is down. Groggy and annoyed, you look at the message and have no idea what service it’s referring to or where to begin.

    After nearly an hour, you can’t figure out who owns the service or how to get it up and running. So, you call that senior engineer on your team—you know the one—the person who gets called every time there’s a tricky question or problem. Thankfully, they answer your call in the middle of the night and help you bring the app back up. But in a month, the same scenario happens again; only this time, when you go to call that senior engineer, they’re no longer there. They quit to go work at a company that cares about reliability.

    Best Practices for Today

    How can you avoid this vicious cycle? Here are some best practices you can implement to begin shifting focus away from speed and toward scalability, reliability and security.

    1. Evangelize the Narrative That the Systems you Work on Matter

    At the end of the day, your systems are there to serve your customers and users. Leadership and teammates who embody these values are incredibly important. People should worry about letting their customers down, not about being yelled at by their manager or receiving a bad performance review.

    2. Set Explicit Expectations about Uptime

    When you say that your app needs to have 99.99% uptime, it doesn’t have the same impact as saying 52.6 minutes of downtime a year. Translate uptime numbers so they’re tangible—and on people’s minds every time there’s an outage.

    3. Don’t Sweep Problems Under the Rug

    After the dust settles from an incident, it’s important to get together and talk about it. Be transparent about what happened, which person or team made the error and what the implications to users were. The idea isn’t to point fingers or assign blame but to bring the team together so that you can learn, evolve and figure out how to prevent this from happening in the future.

    Here are some questions you can use:

    1. Who wrote the code?
    2. How was it tested?
    3. Should our automated testing have caught this?
    4. Who reviewed this code?
    5. Were they the right people to review the code?
    6. Why wasn’t it caught in the staging environment?
    7. Who released it and who went through the release validation checklist?

    Formalizing this process lets everyone on the team know that mistakes are part of the job and that there’s a standard way that problems are handled. This is a great way to help engineers feel free to do their best work without fear or avoidance.

    4. Define Individual Owners for Every Piece of the System

    If there’s any confusion about who owns what within your engineering org, you need to address that right away; it’s a massive problem. Not only do you need an owner for every part of the system, but they must also be empowered to fix issues and know that they’re the ones on the hook.

    5. Invest in Documentation, Runbooks and Logs

    The owner of each piece of the application is also responsible for making sure that documentation is up to date, runbooks are clear and easy to follow and logs are readily accessible. When you’re getting paged at 3:00 a.m. about a service you’ve never heard of before, these resources are key.

    6. Prioritize the Delivery Pipeline Like it’s a Key Feature

    You should always be thinking about how to build safeguards around your workflows, like automated testing or new tools. This may mean you need to go toe-to-toe with product managers to make sure the work required to build a robust delivery system isn’t getting kicked down to the backlog.

    7. Get Business Buy-In by Using Terms They Understand

    Unreliable code is technical debt. And debt is something all leaders understand, regardless of their technical abilities. Stripe published a survey in 2018 that found the average developer spends more than 17 hours a week dealing with maintenance issues, costing $300 billion per year globally. If you’re pushing to invest in SRE practices, those numbers might help you make your case.

    Culture comes first, but the right tools can reinforce the culture and drive accountability over time. Tools that help teams keep track of critical information such as service owners, documentation and runbooks can help tremendously. Additionally, having easily available dependency maps of the architecture are key to quickly understanding complex systems. Lastly, setting organization-wide standards for best practices and providing templates for spinning up new services that follow these can help ensure that these things are not forgotten.

    This article was based on a talk given by the author at Conf42: SRE 2021. 

  • Google Makes Fuzz Testing Integration Tool Available

    Google Makes Fuzz Testing Integration Tool Available

    Google today made available a ClusterFuzzLite tool that makes it simpler to incorporate fuzz testing into a DevOps workflow.

    Fuzz testing refers to an automated software testing technique that involves providing invalid, unexpected or random data as inputs to an application to discover issues such as potential memory leaks.

    Jonathan Metzman, a senior software engineer for Google, said ClusterFuzzLite is a lighter-weight version of a tool that Google makes available to its internal development teams. It is not a fuzz testing tool itself, but rather a tool that makes it easier to surface the results of a fuzz test within a DevOps workflow. Once those results are surfaced, it then becomes easier for developers to address issues before an application is deployed in a production environment.

    Core capabilities enabled by ClusterFuzzLite include continuous fuzzing, sanitizer support, corpus management and report generation tools. The goal is to make fuzz testing more accessible to a broader range of developers as part of an overall effort to embrace DevSecOps best practices, noted Metzman.

    ClusterFuzzLite currently supports GitHub Actions and Google Cloud Build, but Metzman said it is easily extensible to other DevOps platforms. It is designed to run as part of a DevOps workflow by adding a few lines of code to a continuous integration/continuous delivery (CI/CD) workflow.

    Google has already made ClusterFuzzLite available to open source projects, but is now providing additional access for commercial developers. Since its release in 2016, over 500 critical open source projects have integrated into the Google fuzz testing program for open source software, resulting in over 6,500 vulnerabilities and 21,000 functional bugs being fixed.

    In the wake of recent guidelines issued by the U.S. National Institute of Standards and Technology (NIST) that were themselves in response to an executive order issued by president Biden, the number of organizations using fuzz testing is expected to increase now that NIST has determined that fuzz testing is a minimum standard requirement for code verification.

    For several years, Google has been making a concerted effort to improve the security of open source software. The challenge for all developers is finding a way to build more secure applications without slowing down the rate at which they are deployed. Fuzz testing makes it possible to identify issues much earlier in the application development life cycle versus just before an application is deployed.

    Less clear, of course, is whether developers are starting to use these tools on their own or if the increased adoption is because of a concerted effort by their organization to embrace DevSecOps best practices. Regardless of the motivation, however, the overall state of application security should continue to improve as it becomes simpler for developers to invoke more advanced testing tools.

    In the meantime, organizations of all sizes are reviewing how they construct software in the wake of a series of high-profile security breaches. In theory, at least, increased focus on software supply chains should prevent issues from arising after deployment when they become much more costly to remediate.

  • Akamai Brings Web Application and API Security Together

    Akamai Brings Web Application and API Security Together

    Akamai Technologies, Inc. this week launched a service that consolidates the process of securing both web applications and application programming interfaces (APIs).

    Amol Mathur, vice president of product management and strategy for Akamai, said with the launch of App & API Protector the managed security services provider is making it easier for IT teams to embrace DevSecOps best practices via a converged web application and API (WAAP) service rather than using separate tools and services to secure them individually.

    The service is also designed to continuously discover API requests that are then automatically inspected for malicious code. Optional API security controls can be enforced based on policies defined by internal IT teams.

    In addition, the company provides access to a multidimensional threat scoring model, dubbed Adaptive, that combines data collected by Akamai with data and metadata from each web and API request to help organizations better understand which threats require immediate attention. Security events are also continuously analyzed using machine learning algorithms to deliver highly accurate policy-by-policy tuning recommendations that can be implemented via a single click.

    Akamai security researchers also make use of machine learning and data mining techniques to continuously analyze over 303TB of daily attack data that is used to automatically update protections. Alternatively, organizations can opt to manually evaluate that data themselves to minimize any unexpected impacts those updates might have on their applications.

    Other capabilities include built-in bot mitigation tools that are informed by a directory of more than 1,500 known bots, integrations with the Akamai command-line interface (CLI), Terraform or scripts used to drive automated pipelines spanning a continuous integration/continuous delivery (CI/CD) platform.

    Akamai

    A recent report published by Akamai found that between January 2020 and June 2021 there were more than 11 billion total attempted attacks made over an 18-month period. The most common attack vector was SQL injection (SQLi) at 6 billion attacks, followed by local file inclusion (LFI) with 3.3 billion attacks and cross-site scripting (XSS) with 1.019 billion attacks.

    The report also found credential stuffing attacks reached more than one billion attacks and peaked between January 2021 and May 2021. Distributed denial-of-service (DDoS) attacks reached a peak of 90 in January of 2021.

    Akamai has been making a case for outsourcing the management of web application and API security via the content delivery network (CDN) it created. Its CDN provides a layer of isolation between web-facing applications and the rest of the enterprise. The challenge organizations now routinely face is that, as they move to deploy cloud-native applications based on microservices, responsibility for security has become more decentralized. The security services provided by Akamai create an opportunity to better manage application security via a single pane of glass at a time when there is more focus than ever on securing software supply chains, said Mathur.

    It’s not clear to what degree organizations are going to rely more on managed security services as part of a larger effort to embrace DevSecOps best practices. However, at a time when application security expertise is hard to come by, it’s becoming apparent that, in many instances, relying on an external service provider to secure application environments is the simplest way to quickly improve the overall security posture of any organization.