devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

Mastering the Shared Responsibility Model
It’s no secret that cloud-native application development is growing exponentially, with Agile development, IaaS and PaaS from providers like Amazon, Microsoft and Google, enabling innovation at a pace that is challenging for security to keep up with.

A global pandemic and the resulting remote work mandates have only accelerated this movement. And with this change in distributed work and agile application development comes greater security considerations—and a higher degree of risk to the business. Security in the cloud is a shared responsibility, but too many organizations find it difficult to fully understand their role in the security of their applications. There are many elements of securing cloud-native applications that cannot be delegated. They require careful attention as part of a risk-based application security program that extends from design to development throughout their application development life cycle.

With the appropriate visibility, risk-based thinking and automation, cloud-native applications have the ability to be more secure than ever, while enabling faster development and delivery at the same time. But to get to this point, organizations need to understand their responsibilities.

Cloud infrastructure security and compliance are necessities for ensuring how applications and data are deployed and operated in the cloud. But what about what is in those applications? Applications are made up of lines of application/infrastructure code and open source libraries built by developers, DevOps, product managers, security champions and other roles.

How do we know the components that make up the application are effectively secured? It is too late and not enough to solely rely on a secure cloud posture. Reducing risk early in the development lifecycle will dramatically reduce costs and business risk. Some sources have assessed the cost impact of fixing a risk in early production to be four to five times as much as one uncovered during design, and up to 100 times more than one identified in the maintenance phase.

Modern applications are delivered through sophisticated “plumbing” of secure cloud infrastructure to our device interfaces. Just as we require clean and safe water to transit the physical plumbing into our homes, we need our applications to remain secure and high-quality as they transit cloud services.

The IaaS/PaaS Shared Responsibility Model

The best way to think about security in this cloud-native world is by using a shared responsibility model, with IaaS customers only responsible for certain elements of the security of an application that lies on top of the infrastructure.

In the traditional shared responsibility model, IaaS vendors are responsible for the configuration, management and security of infrastructure elements such as networking, databases and compute engines, in addition to physical security, redundancy and more. But each of their customers is responsible for the security and privacy of everything on top of that, from design to code to deployment.

The Traditional IaaS/PaaS Shared Responsibility Model

The Customer Side of the Shared Responsibility Model

What’s missing from this model are software design, code risk and CI/CD security and integrity—and these areas are the most difficult, time-consuming and require a well-thought-out and comprehensive application security program.

With new approaches to application security, organizations can better understand the risk of their cloud-native applications and automate a significant amount of risk remediation activities at the design or coding phases before deploying to the cloud. This requires taking a broad but contextual view of AppSec across the entire SDLC—and the modern nature of attacks requires a complete view of the SDLC to understand where the most impactful business risk lies.

Organizations must evaluate the full picture of their risk—from design to code to cloud:
1. Visibility—Understanding your risk starts with visibility into your security and compliance risks with a real-time inventory of assets, application and infrastructure code behavior, developer knowledge, third-party security alerts, sensitive data and business impact.
2. Secure by Design—You need to make sure that security is addressed at the design stage—before any code has been written or committed.
3. Change Management—Organizations need to focus on the risks that matter early in the development process and based on business impact. Material changes to the applications and infrastructure should be carefully evaluated while other changes can be approved more quickly or ignored, thereby accelerating product delivery.
4. CI/CD Security and Integrity—As an ever-running, highly-trusted and sensitive process, CI/CD is often overlooked. Many attacks in the recent past exploited this Achilles’ heel of the development cycle by embedding attackers’ code within native code commits or other automatic CI/CD processes. Organizations should ensure CI/CD operations are secure with integrity and security checks in place.
5. App & Infra Risks Together—To properly secure your applications, you must correlate the risks of your applications and infrastructure in one platform.
6. Automation—Automation must also be employed to limit mistakes caused by human error, ensure consistency in enforcement of policies and increase the speed of the DevSecOps process. Organizations must carefully manage this process, because automation can also introduce a new category of issues, such as race conditions, untimely triggering, incomplete state validation for edge cases, etc.
Infrastructure-as-Code (IaC)

The human element of risk has changed with IaaS and PaaS due to the power that cloud administrators now have. Instead of purchasing, configuring and deploying servers over a period of months, an admin can spin up a Kubernetes cluster or an S3 storage bucket, load customer data and expose it to the Internet with relevant identity and access management (IAM) policies in record time via infrastructure-as-code (IaC) using GitOps instead of ClickOps.

The rise of IaC has enabled us to view, evaluate and respond to infrastructure changes in the same way we would any application code change. Organizations must apply the same governance rules and automation to review changes in infrastructure before they reach production, along with real-time analysis of cloud misconfiguration.

The main challenge is context! All IaC tools in the market today are looking at security misconfiguration as a single dimension (for example, an S3 bucket missing encryption; a cloud database missing SSL for incoming connections, etc.) and this creates a lot of noisy alerts. The risks of cloud infrastructure and the applications that run on top of it are inherently connected—it is harder to connect the dots and understand the true risk of changes than it is to accept alerts at face value!

When securing your applications in the cloud, don’t rely on traditional tools and processes. Today’s technologies enable new approaches that help you understand and remediate risk in entirely new ways and make cloud-native applications more secure than ever.
November 17, 2021
Plutora Enhances VSM Platform with Release Insights

Plutora has enhanced its value stream management (VSM) platform to provide dashboards that enable DevOps teams to better optimize application delivery.

Jeff Keyes, vice president of product marketing and strategy for Plutora, said dashboards added to the platform will make it possible for application release managers to leverage predictive analytics to manage application delivery in a more progressive way rather than always having to react to an unexpected event.

To enable that approach, Plutora has revamped its release insights dashboard that surfaces problem areas and hot spots in addition to adding analytics that track both software builds and larger project initiatives.

That latter capability is especially critical for executives that currently find it challenging to keep track of multiple software development initiatives, especially those that may have fallen behind schedule without senior leaders being aware of the implications, said Keyes.

While VSM platforms are typically used by senior IT and business leaders to keep track of how software deadlines are impacting key performance indicators (KPIs) for the business, Keyes said it’s become apparent that application release managers need access to more advanced analytics to optimize application deployment schedules. The Progressive Release Management capability surfaces insights via dashboards in the Plutora platform; that makes it simpler for application delivery managers to optimally orchestrate allocation of resources.

Release managers, of course, already have access to a wide range of platforms for automating the application delivery process. However, Keyes said, those platforms don’t provide enough insights into how to optimize application delivery across a DevOps workflow made up of disparate tools and platforms. Plutora enables release managers to employ value stream flow metrics and more traditional DevOps metrics to reduce cycle times and improve overall DevOps throughput.

Keyes said once organizations are provided with that insight, they should also be able to reduce the overall fatigue level many DevOps teams currently experience by removing friction within those workflows. In fact, that fatigue level is a major reason why there is so much turnover in many DevOps teams, he noted.

It’s not clear how many organizations have a dedicated application release team, but optimizing the software delivery process is a major challenge for almost every DevOps team. While many organizations routinely make use of continuous integration (CI) tools to build applications faster, continuous delivery of applications has proven to be a more elusive goal. This is largely because each platform on which applications are deployed tends to be unique. As a result, many DevOps teams still rely on spreadsheets to manually manage application deployments.

However, as the pace at which applications are being built continues to accelerate, it’s clear that application delivery is becoming a bottleneck. VSM platforms provide a means of applying analytics to optimize a process that can then be used to make better-informed decisions that might convince more organizations to automate application delivery. That assumes, of course, that the organization had the foresight to invest in a VSM platform capable of surfacing those insights in the first place.

November 16, 2021
Nobl9 Adds Hydrogen Platform to Surface Technical Debt Issues

Nobl9 today unveiled a Hydrogen platform that makes it simpler for DevOps teams to identify technical debt issues as they manage service level objectives (SLOs) as code.

Hydrogen enables DevOps teams to define SLOs using a simple wizard or by embedding code within their application as part of a DevOps workflow. That code is then used to automatically collect data from platforms such as Datadog, New Relic, Prometheus, Splunk and others with requiring any additional instrumentation.

Nobl9 COO Kit Merker said that approach also makes it possible to surface technical debt that inhibits achieving those goals via the Hydrogen platform. Armed with those insights, it then becomes easier for DevOps teams to focus on reducing or eliminating that technical debt because Hydrogen automatically generates a ticket in the Jira project management application—widely used by DevOps teams—to make sure the issue is addressed. Otherwise, in the heat of a DevOps sprint, the issue is likely to be forgotten as the application development project continues to move forward, noted Merker.

The ability to identify that technical debt is an outgrowth of the ability to manage SLOs as code that is enabled by the core Nobl9 software-as-a-service (SaaS) platform. Hydrogen extends that capability by automating workflows whenever a specific service shows indications that it cannot meet objectives because of a dependency on a legacy platform that needs to be upgraded, added Merker.

There is also a service health dashboard that organizes SLOs into logical groups that are mapped to features via labels, which Merker noted makes it possible for multiple stakeholders to easily identify which issues affect a particular service most. Hydrogen can also trigger a variety of automated workflows to prompt additional investigations that escalate as services worsen.

Overall, Hydrogen is designed to provide engineering teams with more accurate context that can be used to prioritize which issues should be addressed first during any technical debt planning cycle, said Merker. Hydrogen can enforce code release policies based on technical debt budgets that are enforced via integrations with continuous integration/continuous delivery (CI/CD) platforms.

Technical debt generally refers to any code change that is required to optimize a service that is not immediately addressed. The more legacy platforms there are in an IT environment, the higher that technical debt tends to become. At a time when more organizations are building and deploying microservices-based applications, that technical debt, over time, tends to adversely affect applications that are much more dependent on external services than a traditional monolithic application. A single technical debt issue can have a cascading impact on multiple microservices-based applications that would, otherwise, would be difficult to detect.

The challenge, of course, is that technical debt doesn’t always get the attention it deserves when most of the IT team is focused on rolling out the next great application. However, as application environments become more complex it may only be a matter of time before all the technical debt that is being accumulated becomes unsustainable.

November 16, 2021
Stopping Mobile Fraud With Automation and DevSecOps

Mobile fraud is a multibillion-dollar problem that’s only getting worse. One study showed that mobile click fraud alone rose 64% during the first few months of the COVID-19 pandemic. In fact, during the pandemic, about one in five mobile ad clicks were fraudulent. And that’s just one form of mobile fraud.

Weaponized botnets, mobile malware, mobile Trojans and mobile cheat engines and dynamic instrumentation tool abuse—the methods that fraudsters use to defraud companies and consumers via mobile devices—are constantly expanding in number and scale.

Historically, organizations have fought against fraud through monitoring and data analysis. As transactions and data come in, analytics and AI work to discern legitimate events from fraudulent ones. These systems require training and a lot of data to become proficient at identifying fraud, and they can become very accurate. But fraudsters are always changing their tactics to evade these systems, and, as many of us have experienced, false positives are still a big problem. If you’ve travelled enough overseas, likely you’ve had your bank account or credit card frozen until you could call your financial institution to let them know that your purchases were genuine.

Another challenge with existing mobile fraud prevention solutions is that they are designed to protect networks and network resources first. As a result, while fraud is occurring and fraud prevention systems are learning, mobile end users are still being affected. Also, when a bad act is identified and a system enforces protections against fraud, these enforcement points typically block network traffic. That protects the network, but the consumer and their mobile app remain at risk. Once a fraudster compromises a mobile app, they may be able to attack through other vectors, even if one network is shut down. For instance, if a consumer unwittingly downloads and launches EventBot to their mobile device thinking it was a legitimate app, it may compromise multiple banking apps. If one bank shuts down its network after detecting fraud, EventBot may have other opportunities on the device.

As this chart shows, once fraud takes place, damage to a brand, customers and the bottom line begins immediately. Even if it is detected and halted, the harm it has already caused can’t be undone. Mobile app publishers and developers need to stop mobile fraud where it starts, and that means implementing mobile fraud prevention inside the app within a DevSecOps framework. By blocking the technical methods used to carry out fraud via mobile apps, organizations can make massive inroads toward stopping fraud before it ever gets to the network level, preventing both customers and organizations from getting scammed.

Why Traditional Security Approaches Don’t Work in Mobile Apps

To achieve a multi-layered defense against fraud, a mobile app should include app shielding, anti-tampering, code obfuscation, data encryption (including strings, resources, preferences), jailbreak/rooting prevention and man-in-the-middle prevention (such as certificate pinning or certificate validation). Certainly, there are commercial SDKs and third-party libraries that can provide these functions, with some limitations. But these approaches typically still require extensive development work and introduce additional frameworks or programming language dependencies and incompatibilities.

They don’t solve the fundamental problem of implementing fraud prevention inside a mobile app—it’s too expensive and time-consuming. The root of the problem rests in the way mobile apps are developed, released and updated in a modern agile organization. DevOps depends on processes that are dynamic, automated, agile, integrated, iterative and continuous. Mobile app security has traditionally been very different: Static with manual, line-by-line coding. This way of implementing manual security will never work in an agile delivery model.

The promise of DevSecOps is built-in security, delivered as a fundamental part of a release process where security is designed into the process at each and every phase of the life cycle. The security model can be delivered and evolved atomically, iteratively and dynamically so that it adapts to fit the app.

The future of fraud prevention lies in the mobile app, but to make this possible, the industry needs to apply AI and ML to automate fraud prevention and security implementation into mobile apps on time within an agile, fast-moving and iterative app life cycle and release process.

November 16, 2021
Building on Terraform: Evolution, not Revolution
Terraform is widely adopted among both enterprises and service providers who embrace infrastructure-as-code (IaC) as their model for infrastructure provisioning automation in DevOps environments. And while the technology has limitations, innovations seeking to unseat Terraform are highly fragmented and have seen tepid adoption. This points to a market that’s settling on a de-facto standard, so rather than trying to replace Terraform with the next great innovation, infrastructure operators should instead find ways to extend and augment Terraform to suit their unique needs. Let’s explore several of the better-known shortcomings of Terraform, some drivers for the rise in new alternatives to the technology, and what users need to consider when contemplating a different path.

Without a doubt, Terraform is big. Across the big three public clouds, the number of Terraform provider installs tells the story: At the end of August, there were 457 million provider installs for AWS, and around 72 million each for Azure and GCP. Yes, compared to longer-in-the-tooth infrastructure configuration management tools like Ansible, Chef and Puppet, IaC as a category is still early. But, the trend is clear: IaC is becoming widely used at scale, and Terraform is the IaC platform most enterprises go with.

It’s also no secret that there are shortcomings to Terraform. A quick search on Reddit and Hackernews unearths the litany of complaints, some more valid than others:
- It’s tricky creating a Kubernetes cluster and then adding a resource to it
- Challenges passing values through a config or dumping a pure representation of an elevated config
- Cluster deletion is a pain
- An overall lack of visibility into what’s happening during the provisioning process
- The need for glue code to drive everything is a drain on developer productivity
- Lingering challenges with backward compatibility
- Continuous update challenges and the need to refactor
- Scaling across multiple workspaces and multiple cloud accounts
- Rollbacks can be tricky, especially when dealing with immutable infrastructure
Terraform’s parent company, HashiCorp, of course, steadily works to address these and other pain points. But the whole thing highlights an interesting question: Why is Terraform so popular?

Terraform is the only IaC solution that meets the needs of developers who want an IaC tool that preserves state, fits well into their CI/CD pipelines, is intent-based and gives them the right level of granularity. Terraform, even with its challenges, is better than the alternatives in these regards.

Here’s why:

Terraform was, from the start, designed for and targeted to developers. It maintains state and supports iteration, making IaC a powerful model for automating infrastructure. PaaS solutions, by contrast, are not robust enough for most developers who do not need or want all of the infrastructure decisions abstracted away. Most cloud management platforms suffer from the least common denominator problem, and even though the technology has improved in this regard, it’s a solution that developers generally abhor for its granularity; it’s geared more to the needs of IT management. Configuration management tools, as an alternative, are not intent-based, so flow becomes too complicated.

It’s interesting to note that state (maintaining the mapping between the resources that you’ve represented in your definition files and the actual physical resources that are in the cloud provider) was added into Terraform, helping resolve the issue of infrastructure drift. Terraform also realized that developers don’t necessarily need a UI. These tangible examples of the developer-centric design ethos at HashiCorp point to why it is so widely adopted.

So, what’s driving the interest in new approaches to infrastructure automation? The proliferation of domain-specific use cases and an ever-present need to reduce time-to-value are the leading reasons. Domain-specific automation options are becoming more dominant over OOB solutions because they take too long to get to automation; one tool does not fit all. Domain-specific automation addresses issues such as security automation, AIOps, DevSecOps, security compliance, integration with container scanners—not just generic orchestration. People are looking for value very fast, which can be found only with domain-specific automation—and therefore, fragmentation happens as different solutions emerge to address those cases.

There are several interesting alternatives to Terraform. Terragrant is used along with Terraform to expand its capabilities to multiple workspaces. It allows you to use the same template across many environments. Pulumi is another alternative that has achieved a measure of adoption and visibility. It provides native code support which addresses the problem with YAML file development complexity. Another more ambitious undertaking is the Crossplane open source project. It takes a Kubernetes-centric approach to creating a universal, cross-cloud control plane. It looks good conceptually, but it’s a fundamentally different approach, and adoption is nascent.

Turning to one of the Terraform alternatives isn’t necessarily a bad idea, but is fraught with landmines. In the end, this choice is less about technology and more about adoption and ecosystem momentum. Vendor and professional investments in tools, training, and skill sets surrounding Terraform is a formidable moat, and it cannot be easily replicated. Tools that would win over Terraform would have to win adoption, and achieving adoption means delivering all the enterprise-grade features and support that Terraform users currently enjoy. Chef and Puppet are examples of how this could happen. Both were successful technologies, but they were slow to react to the challenge presented by Terraform. Ultimately, these projects suffered from a legacy mindset and were overtaken by Terraform’s more focused, simple solution.

Most of the issues with Terraform lie in managing it at scale. When you have many workspaces and templates, it becomes hard to manage; it’s also a challenge that templates cannot be generalized among environments, resulting in a lot of template redundancy. Also, Terraform still needs help handling Day 2 operations and visibility into drift detection.

When considering ways to extend Terraform to address a broader set of use cases, you should first consider how well things will interoperate across your environment. If you have multiple clouds, multiple service layers and multiple cloud vendors, this is probably the most important factor. Also, investigate any new technology’s ability to support the things about Terraform that are critical to your Day 2 operations. Things like secrets and state come to mind. You’ll want enough management and monitoring in the new solution without it being a new burden to carry. You should be able to use your existing Terraform files, and you should be able to support all the versions of Terraform that you run. Compliance with governance and audit are key, and the new solution should support Terraform for both your on-premises/air-gapped environments and your cloud environments. Ideally, anything you add to Terraform would be open source, eliminating new complications from cross-vendor lock-in.

Architecture for how 8451 evolved its Terraform deployment

For all its shortcomings, Terraform remains the best option on the market for DevOps organizations, and there are ways to manage it with fewer headaches and without developers wasting so much time writing glue code or reinventing the wheel over and over again. One way this happens is by integrating Terraform with meta-orchestration tools that allow Terraform to become more flexible in managing environments and workspaces at scale. The open source Cloudify project is one example.

Yevgeniy Brikman gave a talk two years ago at HashiConf that has aged well. At the time, DevOps was still in the Stone Age. Today, it’s probably in the Bronze Age. Those of us building IaC today are being asked to deploy infrastructure that looks like modern cities, but the tools we have to work with remain relatively primitive. What we’re building takes too long to provision and it looks bespoke and temperamental at best or kludgy and fragile at worst.

It’s a story repeated over and over: Teams try to build their own, homegrown solution first as a wrapper on top of Terraform. Ultimately, they realize that it’s unsustainable as a lack of engineering resources becomes a bottleneck. We need solutions that augment—not replace—what they’ve already built and what works. That’s almost always Terraform. With Terraform, DevOps organizations get a proven IaC option that’s targeted to the needs of developers, with a substantial ecosystem and market uptake. When considering its shortcomings, it’s best to embrace an evolution strategy, and not start from ground zero.
November 16, 2021
Successful Software Development in a Hybrid Environment

The desire to work from home or from anywhere has only increased as the COVID-19 pandemic lingers. Employees cite a number of reasons why they prefer to skip the office, including cutting commuting times, being more productive, caring for family members and exploring new locations. People are increasingly comfortable with remote interactions and the new technology tools that have sprouted to support the work-from-anywhere trend. On average, employees said they want to work from home 2.5 days a week, according to a monthly survey of 5,000 Americans, while 73% of employees seek hybrid work environments where they have some flexibility in where and when they work.

The rapid spread of the COVID-19 Delta variant also prevented employers from enforcing a full-time return to the office. As a result, the new normal for today’s workforce will remain hybrid at best—a few days in the office and a few days working at home. We are a workforce in transition. For many companies, this has put an unnecessary strain on their digital transformation initiatives.

A question remains: How can companies continue to innovate and execute on their software development initiatives with their employees separated from their teams and offices? Here are some of my recommendations for helping organizations to continue their software development projects in today’s hybrid work environment:

Inclusive Environments

Hybrid workforces require a highly inclusive environment. As a nearshore IT partner that works with hundreds of clients remotely, we ensure that our teams have the same access to managers, HR and the technology necessary to succeed on projects and to productively engage with clients. For example, we have learned that the best way to be inclusive is to take either an all-virtual or all-in-person approach during client meetings. It’s also critical to train your managers to provide the same advancement opportunities to employees regardless of whether they are in or out of the office. From the executive team on down, everyone needs to support employees, regardless of where they are. The same goes for onboarding new employees and partners—a mostly virtual process over the past 18 months. Increasing online mentoring and onboarding platforms to connect employees and to manage client projects while the workforce is hybrid is another significant approach we have taken to ensure everyone feels included and engaged.

Collaboration and Knowledge-Sharing

Employees were abruptly forced to transition away from the office where they were used to frequent impromptu meetings and lunchroom chats. For many, this shift had a negative impact on brainstorming, collaboration and knowledge-sharing and thus, innovation. We give our teams the flexibility to use different technology tools to make up for what they miss most in an office environment—regular, casual human interactions. Our teams use a variety of communications tools for frequent check-ins to connect with clients and their co-workers. As a result, our people say they are more engaged with team members and clients than ever before. They are doing more impromptu chats and “instant” meetings with colleagues and clients as if they were in the office. We also migrated to video conferencing as our communication channel of choice. Video conferencing tools increase engagement and allow team members to join virtual meetings anywhere from their preferred devices. The ability for teams to interact, share and collaborate wherever they are is essential to developing cutting-edge teams and software.

Trusting Cultures

According to the Harvard Business Review, 34% of employees felt that their supervisors “expressed a lack of confidence in their work skills,” and an even more significant number of workers reported feeling that they needed to be constantly available to prove themselves. Sentiments like these highlight a lack of trust, impacting development outcomes. Employers need to trust that their employees are working diligently and consistently throughout the workday or whenever they can successfully complete the tasks at hand. For example, working remotely allows individuals to work when they feel most productive. Organizations need to recognize individual productivity cycles and understand that different team members may be productive during different times of the day, including outside of normal business hours.

Being transparent, forthcoming and open about expectations are keys to building trust with employees. Successful digital transformations and innovative software development are only possible with the right team and people. From a client’s perspective, these are the types of IT partners they want on their side. Technical aptitude and an environment where all employees feel valued, trusted and motivated to do their best are critical for developing effective digital solutions.

Employees have made it clear that they don’t want to be back in the office full time. For many organizations, a return to normalcy centers on developing a hybrid workforce that can work effectively with digital development partners. Creating the right environment and strengthening employee relationships are keys to ensuring software development projects can continue successfully in today’s new normal.

November 15, 2021
Building and Leading Product Teams in Uncertain Times
The world seems to be changing faster than ever, and there’s no reason to believe it will slow down anytime soon. The pandemic has compounded this sense of constant flux, with customer priorities shifting overnight. Against this backdrop, product teams need to constantly evolve their development work without losing focus or feeling they need to react to every small change in the market.

This requires team leads to throw out their old playbooks and develop a new strategy that allows them to deliver continuous innovation without creating chaos for their teams or losing sight of their north star. Road maps are shrinking from years to months, and are now being created with the understanding that they may become outdated or redundant at any time.

Building and maintaining an organization that can not only survive but thrive in these conditions is a daunting task. It requires a structured, programmatic approach that also supports agility and adaptability. Those requirements may feel at odds, but they actually are not. Here are five tactics to help your product strategy evolve with the times.

Keep That “Day One” Mentality

It’s important to keep the soul of your product top of mind. It helps to retain a “day one” mentality which means constantly reminding yourself why you started in the first place and what problem you set out to solve. That founding problem statement should serve as your guiding light, especially through times of rapid growth and change.

Customer requests are, of course, a high priority and should always be thoughtfully considered. But ask yourself if the request is aligned with that day one problem you set out to solve. If it’s not, work with the customer to identify the underlying issue they’re really trying to solve for versus focusing on the symptom they’re experiencing.

Cultivate a Culture That Thrives on Change Without Succumbing to Chaos

It takes a combination of strong organizational capability and the right culture to build a team that can thrive on ambiguity. Strong organizational capability is achieved through talent, architecture and execution skills.

Great product teams build an architecture that supports iteration and are able to execute toward their north star in small, rapid steps. Team leaders should build a culture that takes failure in stride and learns from it to improve each subsequent iteration in the product journey.

Change is inevitable, but chaos is avoidable. Leaders must create an environment that feels psychologically safe and supportive, so that team members feel confident in changing priorities and making rapid decisions.

Build an “Innovation Factory” or a System that Reduces the Potential for Chaos

Innovation and agility require well-planned foundational resources. Following are the key steps that can help put these building blocks in place.
- Think of velocity as a part of your product. Teams should be broken into small groups to autonomously drive the product forward, but in such a way that the overall experience feels seamless to end users.
- Democratize the product concept phase by creating conditions for a large number of ideas. Everyone involved should be inspired to contribute, including employees, partners and customers.
- Create a system for product development excellence. Fully automate your product development life cycle by approaching developer infrastructure as if it were a service, with clear service-level objectives within a CI/CD framework. Digitize each step in the product concept, development and deployment journey, with a focus on measurement, learning and iteration in every phase. Ask who and how many people are building versus approving at each stage and ruthlessly optimize and automate where possible.
Bring a Service Mindset to Everything you Do

If you’re building a SaaS product, as many startups today are, you’re no longer delivering a product in the traditional sense. You’re delivering a service, and you’re responsible for the entire experience from start to finish. This means you’re in full control over when to release new features and how you measure user engagement. You can see where the friction points are, and where customers are delighted. We have a philosophy at my company of “less input, more output”—simplicity is the name of the game, where the output and value your service creates for customers is the only measure that matters.

The outcomes required may change as customer priorities evolve and new requirements flow in, but adapting quickly and effectively should just be a matter of (re)connecting well-built services within your platform to prioritize and address those changes. Leverage the cloud ecosystem in a way that allows you to safely bet on other players and partners who are moving as quickly as you are. Focus on making sure your foundation is strong; one that can provide a service at scale even as the outcomes customers want change over time. This allows you to stay laser-focused on the task at hand: Maximizing value for users.

Incorporate Users in the Product Development Cycle and Learn as you Go

The cloud has transformed the way organizations build products and integrate feedback. The entire life cycle is digitized and measured. Product teams can move at a much higher velocity and in smaller increments, with the ability to make near-real-time adjustments along the way. But the greatest advantage a cloud environment offers is that we can now include users in the entire product life cycle, every step of the way. It gives us the ability to measure outcomes quickly and take action on the information that immediately benefits customers.

It sounds counterintuitive, but product teams can prepare for the unknown. There’s no way to see what’s over the horizon, but with the right people, processes and tools in place, there’s no reason that uncertainty should derail you. There has never been a more exciting time to be an innovator or a builder. Meet the uncertainty ahead with open arms and a strategic eye.
November 12, 2021
Prioritizing Scalability, Reliability and Security in Engineering
As digital products and services are more deeply embedded in critical industries and infrastructure and the implications of problems grow in scope, engineering organizations are renewing their focus on building platforms that are scalable, reliable and secure. More importantly, they are recognizing that this isn’t the responsibility of an SRE or CSO, it’s an ethos that must be carried each day by every engineer on the team. This is an important shift—from teams and cultures that valued speed of delivery above all else to teams and cultures that make high-quality systems the foundation of everything they do.

This isn’t easy, and big cultural changes don’t happen overnight. There are many things that can go wrong when the team is operating in the legacy mode of prioritizing quick time-to-market but there are also some strategies and best practices for beginning the process of cultural change across your engineering organization.

The Old Way of Doing Things

You’re the engineer on call, and you get woken up at 3:00 a.m. by an alert that tells you an app is down. Groggy and annoyed, you look at the message and have no idea what service it’s referring to or where to begin.

After nearly an hour, you can’t figure out who owns the service or how to get it up and running. So, you call that senior engineer on your team—you know the one—the person who gets called every time there’s a tricky question or problem. Thankfully, they answer your call in the middle of the night and help you bring the app back up. But in a month, the same scenario happens again; only this time, when you go to call that senior engineer, they’re no longer there. They quit to go work at a company that cares about reliability.

Best Practices for Today

How can you avoid this vicious cycle? Here are some best practices you can implement to begin shifting focus away from speed and toward scalability, reliability and security.

1. Evangelize the Narrative That the Systems you Work on Matter

At the end of the day, your systems are there to serve your customers and users. Leadership and teammates who embody these values are incredibly important. People should worry about letting their customers down, not about being yelled at by their manager or receiving a bad performance review.

2. Set Explicit Expectations about Uptime

When you say that your app needs to have 99.99% uptime, it doesn’t have the same impact as saying 52.6 minutes of downtime a year. Translate uptime numbers so they’re tangible—and on people’s minds every time there’s an outage.

3. Don’t Sweep Problems Under the Rug

After the dust settles from an incident, it’s important to get together and talk about it. Be transparent about what happened, which person or team made the error and what the implications to users were. The idea isn’t to point fingers or assign blame but to bring the team together so that you can learn, evolve and figure out how to prevent this from happening in the future.

Here are some questions you can use:
1. Who wrote the code?
2. How was it tested?
3. Should our automated testing have caught this?
4. Who reviewed this code?
5. Were they the right people to review the code?
6. Why wasn’t it caught in the staging environment?
7. Who released it and who went through the release validation checklist?
Formalizing this process lets everyone on the team know that mistakes are part of the job and that there’s a standard way that problems are handled. This is a great way to help engineers feel free to do their best work without fear or avoidance.

4. Define Individual Owners for Every Piece of the System

If there’s any confusion about who owns what within your engineering org, you need to address that right away; it’s a massive problem. Not only do you need an owner for every part of the system, but they must also be empowered to fix issues and know that they’re the ones on the hook.

5. Invest in Documentation, Runbooks and Logs

The owner of each piece of the application is also responsible for making sure that documentation is up to date, runbooks are clear and easy to follow and logs are readily accessible. When you’re getting paged at 3:00 a.m. about a service you’ve never heard of before, these resources are key.

6. Prioritize the Delivery Pipeline Like it’s a Key Feature

You should always be thinking about how to build safeguards around your workflows, like automated testing or new tools. This may mean you need to go toe-to-toe with product managers to make sure the work required to build a robust delivery system isn’t getting kicked down to the backlog.

7. Get Business Buy-In by Using Terms They Understand

Unreliable code is technical debt. And debt is something all leaders understand, regardless of their technical abilities. Stripe published a survey in 2018 that found the average developer spends more than 17 hours a week dealing with maintenance issues, costing $300 billion per year globally. If you’re pushing to invest in SRE practices, those numbers might help you make your case.

Culture comes first, but the right tools can reinforce the culture and drive accountability over time. Tools that help teams keep track of critical information such as service owners, documentation and runbooks can help tremendously. Additionally, having easily available dependency maps of the architecture are key to quickly understanding complex systems. Lastly, setting organization-wide standards for best practices and providing templates for spinning up new services that follow these can help ensure that these things are not forgotten.

This article was based on a talk given by the author at Conf42: SRE 2021.
November 12, 2021
Akamai Brings Web Application and API Security Together

Akamai Technologies, Inc. this week launched a service that consolidates the process of securing both web applications and application programming interfaces (APIs).

Amol Mathur, vice president of product management and strategy for Akamai, said with the launch of App & API Protector the managed security services provider is making it easier for IT teams to embrace DevSecOps best practices via a converged web application and API (WAAP) service rather than using separate tools and services to secure them individually.

The service is also designed to continuously discover API requests that are then automatically inspected for malicious code. Optional API security controls can be enforced based on policies defined by internal IT teams.

In addition, the company provides access to a multidimensional threat scoring model, dubbed Adaptive, that combines data collected by Akamai with data and metadata from each web and API request to help organizations better understand which threats require immediate attention. Security events are also continuously analyzed using machine learning algorithms to deliver highly accurate policy-by-policy tuning recommendations that can be implemented via a single click.

Akamai security researchers also make use of machine learning and data mining techniques to continuously analyze over 303TB of daily attack data that is used to automatically update protections. Alternatively, organizations can opt to manually evaluate that data themselves to minimize any unexpected impacts those updates might have on their applications.

Other capabilities include built-in bot mitigation tools that are informed by a directory of more than 1,500 known bots, integrations with the Akamai command-line interface (CLI), Terraform or scripts used to drive automated pipelines spanning a continuous integration/continuous delivery (CI/CD) platform.

A recent report published by Akamai found that between January 2020 and June 2021 there were more than 11 billion total attempted attacks made over an 18-month period. The most common attack vector was SQL injection (SQLi) at 6 billion attacks, followed by local file inclusion (LFI) with 3.3 billion attacks and cross-site scripting (XSS) with 1.019 billion attacks.

The report also found credential stuffing attacks reached more than one billion attacks and peaked between January 2021 and May 2021. Distributed denial-of-service (DDoS) attacks reached a peak of 90 in January of 2021.

Akamai has been making a case for outsourcing the management of web application and API security via the content delivery network (CDN) it created. Its CDN provides a layer of isolation between web-facing applications and the rest of the enterprise. The challenge organizations now routinely face is that, as they move to deploy cloud-native applications based on microservices, responsibility for security has become more decentralized. The security services provided by Akamai create an opportunity to better manage application security via a single pane of glass at a time when there is more focus than ever on securing software supply chains, said Mathur.

It’s not clear to what degree organizations are going to rely more on managed security services as part of a larger effort to embrace DevSecOps best practices. However, at a time when application security expertise is hard to come by, it’s becoming apparent that, in many instances, relying on an external service provider to secure application environments is the simplest way to quickly improve the overall security posture of any organization.

November 11, 2021
Data Theorem Adds Runtime Protection Enabled by Observability

Data Theorem, Inc. this week added an Active Protection offering to its portfolio of application security services that makes it possible for DevOps teams to embed observability and runtime defenses in their applications via a software development kit (SDK).

The Data Theorem cloud services are based on Trustkit, an open source framework the company created to make it easier to employ secure socket layer (SSL) to encrypt connections and Analyzer Engine, its runtime analytics engine.

The company is now extending its services to make it possible to embed observability capabilities by collecting logs and traces. Those logs and traces are then used to enforce runtime security policies that protect applications from known attacks from malicious sources and other common attack vectors as defined by the Open Web Application Security Project (OWASP) Foundation. It also prevents violations of encryption levels, authentication types and authorization rules along with a variety of custom rule checks, including preventing broken object-level authorization (BOLA) attacks.

Doug Dooley, Data Theorem COO, said Active Protection can be applied across any application stack, including microservices applications based on cloud-native frameworks. In contrast, existing approaches to securing application runtimes based on web application firewalls (WAFs), runtime application self-protection (RASPs) or endpoint detection and response (EDR) agents cannot be applied to cloud-native applications based on containers and serverless computing frameworks.

Active Protection solves that issue because it is designed to be applied via the Data Theorem software development kit (SDK) or using application extensions made available via a serverless computing framework or as a container sidecar.

Dooley noted that the Active Protection service is also designed to be fully integrated with DevSecOps workflows that are increasingly being driven via continuous integration/continuous delivery (CI/CD) platforms to enable continuous, automated security inspection and remediation. The company claims to have detected more than one billion application incidents and is currently used by more than 8,000 applications.

Developers don’t generally set out to build and deploy insecure applications. The issue is most developers don’t have a lot of security expertise, so it’s easy for mistakes to be made. Organizations looking to embrace DevSecOps best practices need to find a way to achieve that goal in a way that developers can easily embrace. The more automated the checks for security issues become during the application development process, the more likely it is developers will make additional efforts to remediate vulnerabilities long before an application is ever deployed in a production environment. In effect, DevSecOps is easier to achieve when addressed from the bottom up rather than via an edict from above without any real implementation plan in place.

It’s still early days as far as DevSecOps is concerned, but as best practices for DevOps continue to evolve, there may come a day when security is just another gate that needs to be passed within the context of a larger DevOps workflow. In the meantime, DevOps teams and cybersecurity teams should put aside their differences and work together as the number of attacks launched against applications increases.

November 11, 2021