devopsy.kinsta.cloud

Category: Doin’ DevOps

From Cloud Chaos to Cloud Smart With Multi-Cloud Services

Organizations that embrace multi-cloud services experience real results and a heap of advantages—including greater resiliency, agility and data sovereignty. This is why it’s no surprise the majority of enterprises today are using two or more public clouds, with 81% reporting in a recent survey that they plan to adopt a multi-cloud strategy by 2024.

But despite the benefits, it’s also been proven that multi-cloud can be complicated and even overwhelming. It requires an entirely new approach to optimize workloads, simplify development and tackle a growing pattern of operational complexities. This has been described as cloud chaos, where greater cloud choice has led to a massive spike in complexity. Most report that building new apps is slow and cumbersome, and managing their entire app portfolio across disparate clouds is difficult and expensive. No wonder, when each cloud requires its teams to use proprietary tools that are siloed and incompatible with one another. Meanwhile, for the average employee, getting fast, secure access to critical apps from anywhere is imperative, yet it’s often a challenge. All this complexity has broad repercussions on the business, from managing costs to attracting and retaining top talent.

Tackling this multi-cloud complexity needs to start with creating a layer of abstraction that spans various, diverse cloud environments and doesn’t hinder access to each cloud provider’s unique services. However, organizations must take it a step further to solve the challenge of how best to build, manage, govern and optimize apps and workloads across a multi-cloud environment.

Multi-Cloud Services

Multi-cloud services—a fast-emerging category of IT services—are the key to helping organizations implement a multi-cloud strategy that simplifies and streamlines development, operations, networking and security across clouds. A multi-cloud service provides a consistent API, object model, identity management and other core functions across clouds. And, it has one or more of the following characteristics:

1. Runs on a single cloud but supports interactions with at least two different clouds
2. Runs on multiple clouds and supports interactions with at least two different clouds
3. Runs on any cloud or edge, even in disconnected mode, and basic operations are fully automated

Multi-cloud services can be used to standardize cloud infrastructure and operations into one platform to reduce the complexity of individually building or consuming the equivalent native services from multiple clouds. In this model, public clouds, data centers, private clouds and edge locations are all verticals, and multi-cloud services are horizontals that provide functionality across these environments.

Let’s take a closer look at the five most common categories of multi-cloud services and when to use them.

1. Application Services

Application services allow teams to use core services in a standardized manner across clouds. These core services include observability, replication, backup, and restoration. By implementing application services, enterprises can increase their speed of innovation and make their technologies easier to use. Examples of application services include developer tooling, databases, modular development platforms, artificial intelligence, machine learning and serverless and CI/CD capabilities.

2. Infrastructure Services

If your organization chooses to run its applications as VMs, containers, or both, using infrastructure services is the right move to standardize management and operations across the clouds. Different types of multi-cloud infrastructure services include the ability to manage and observe compute, storage and network services leveraging VMs or containers, infrastructure automation and Kubernetes management solutions. When applied, infrastructure services can improve resiliency, security, performance, interoperability and portability.

3. Security Services

When using multiple clouds, an issue enterprises run into is that each individual cloud has its own security tools and approaches, making it difficult to ensure your enterprise is protected against cyber threats. That’s where multi-cloud security services come in – centralizing security operations and secure software supply chains. Some examples of multi-cloud security services are network detection and response, endpoint detection and response, next-gen antivirus and secure access service edge. These services enable organizations to achieve cohesive visibility, context and control by providing a single interface.

4. End-User Services: Access to Data and Services

No matter how big your enterprise is, or how many applications you use, it’s necessary to provide the right level of access to applications and data. But this can prove difficult when you have a multi-cloud infrastructure. Multi-cloud end-user services, which consist of virtual desktops, device management and end-user application delivery, will provide your organization a uniform way for all employees to have quick, secure access to your applications, all while simplifying device management.

5. Data Plane

In the past, workloads were restricted to a single data plane contained within a single public or private cloud or edge location. Now with the rise of multi-cloud services, enterprises can run workloads in the multi-cloud data plane. A prime example of workloads running in the multi-cloud data plane is a distributed application that runs in multiple cloud provider environments. Part of the application might run in the private cloud and the other services are deployed in a public cloud. The multi-cloud data plane is where the containers, services and data exist and where user interaction occurs.

From Cloud Chaos to Cloud Smart

The desired destination of every multi-cloud journey should be to move from a position of cloud chaos to one of cloud smart. Being cloud smart means taking an architected and planned approach to multi-cloud and digital transformation. As more enterprises adopt multiple clouds, multi-cloud services will be key to becoming cloud smart by helping to reduce complexity and risk and proving the freedom to select the right cloud for all situations based on technical and business requirements.

October 4, 2022
Game Developers: 5 Strategies for a Great User Experience

The gaming industry has exploded since the start of the COVID-19 pandemic. Many people turned to social and casual gaming to entertain themselves at home and interest is expected to continue to rise. According to PwC, the global gaming industry is forecast to increase revenue from $214 billion in 2021 to $321 billion by 2026. With a booming and competitive market, gaming companies must deliver a seamless experience to their end users. A recent survey found that 47% of gamers demanded fast performance, citing it as the most important aspect of gameplay. If users encounter poor performance such as glitchy graphics, crashes or freezes, long load times and in-game purchasing errors, they’ll move on—damaging the company’s brand and revenue. Companies need to have the right technology in place before things go wrong. Here are five strategies game developers must focus on to provide the best gaming experience for end users.

1. Developers Should Increase Processing Speeds

Gaming is a demanding application for developers. Instant response time—delivering processing speeds measured in milliseconds—is required for any gaming experience, even when there are substantial usage spikes. It can be a challenge as developers have to account for vast numbers of reads and writes against massive data volumes from millions of simultaneous players interacting in real time and sophisticated fraud detection rules.

2. Personalize the User Experience

Personalization helps a game attract a broader audience by offering in-game features that match their preferences. It is an essential way for game developers to retain and monetize their users. With so many different games available for users to play, engaging them for a long time will determine a game’s success. To offer this type of personalized experience, developers need to integrate data silos from different sources, where they continuously update all available historical data from their system of record with data streaming in from the digital edge.

3. Scale Systems Horizontally

Gaming applications need to store, access and retrieve enormous amounts of data, such as player profile information, game performance, statistics and rankings and session information, for millions of users. Developers need to be assured their system can scale out easily and quickly to handle any load at any time, without the need for coding or other manual processes. Relational databases aren’t the best fit for gaming as they are built to scale vertically. On the other hand, modern data platforms scale horizontally across servers, clusters, and data centers to ensure consistent and predictable performance.

4. Prevent Gaming Fraud

As companies seek ways to monetize gaming through microtransactions, where they offer in-game purchases for additional features, the threat of malicious activity increases. According to TransUnion’s 2022 Global Digital Fraud Report, the U.S. gaming industry had the largest percentage of annual fraud growth last year, increasing 60.3% from the previous year. With more players and financial transactions, companies must take their fraud prevention activities to the next level to ensure smooth and secure transactions. Developers need to continuously update the rules engines for their fraud detection systems. They can do this by processing huge amounts of streaming and historical data that they can feed into machine learning algorithms.

5. Implement AI/ML Gaming Systems

Gaming companies should turn to artificial intelligence (AI) and machine learning (ML) systems to improve the gaming experience for their players. The technology can power interesting enhancements to games. For example, it can be used to turn speech into text to document and report bad behavior or to support an in-game assistant that helps players based on their habits, preferences, and abilities. To provide these experiences, developers need to feed their AI/ML systems an unending amount of data to deliver better results.

The stakes for gaming companies have never been higher. With the gaming market expected to reach three billion global users by 2023, companies must provide the best gaming experience to gain players and protect their reputations. Players demand flexibility, speed and reliable performance. That means delivering smooth gameplay with no hiccups.

October 3, 2022
9 Best Practices for Web Hosting Security

A good website hosting service is crucial if you aim to provide the best quality services through your brand’s website. Whether you are setting up a virtual store, website or blog, the advantages of a good web hosting service are immeasurable.

For a web hosting service to be great, it needs to be a safe and stable environment for your web surfers. Having good data storage capabilities with enough security to avoid breaches is also a good idea. Here, we will highlight the top web hosting best security practices you should emulate.

Best Practices You Should Follow for Secure Web Hosting

Today, the internet has 1.88 billion websites distributed over many servers. A web server can host multiple sites in one physical machine. However, depending on the hosting type, if hacking occurs on one website, it can have a devastating effect on all other sites on the same server. Therefore, you need to take your website security seriously as a business owner.

Do Regular Backups

Websites contain lots of crucial or sensitive information on them. Loss of this information may cause significant setbacks in your company’s daily plans and operations. The best way to resolve these issues is to include backups in your hosting strategies. You can back up your information at any time or at different frequencies, such as daily, monthly or weekly.

Most websites can create backups automatically, although some require more attention. A few web hosts limit how often you can back up your data. The backup frequency will depend on your chosen plan; a plan with a high price tag usually offers unlimited backups daily.

Reading the conditions outlined by your hosting provider in detail is essential to understanding the full extent of the backup options available. The conditions will also give you more insight into all the restrictions associated with that backup service. Check if the provider offers a simple restoration point since there is no point in backing up your data if you can’t retrieve it.

Malware Scanning

Being able to scan for malware is a part of every good hosting plan by default. Certain brands of web security, such as SiteLock, offer this service. If you note your provider’s website is flashing a protection seal, that’s an indication that the host has already taken protective measures against any virus or malware.

Many software systems can quickly help you scan for threatening elements. Some of the most well-known ones may include rkhunter and ClamAV. Checking the rating for how efficient this software is and performing regular full scans on your server is key to keeping things secure.

Network Monitoring

Network monitoring is a critical part of any web hosting operation. By continuously observing the network for performance and security issues, web hosts can identify and resolve problems before they cause major disruptions. Besides, it helps detect and prevent attacks by hackers.

By monitoring traffic patterns and identifying unusual activity, web hosts can often head off attempts to exploit vulnerabilities in the system. As a user, ensure that you monitor your network activity daily. This practice allows system admins to identify any attack or potential malware spread between servers instantly.

Firewall and TLS

Firewall and Transport Layer Security (TLS) certificates are both essential for all hosting types. TLS encryption ensures that anyone trying to intercept your data can only see incomprehensible characters. TLS technology protects your sensitive data. It may include information like bank account details or even customer information. Besides TLS, you can use web application firewalls (WAF) to filter and monitor your HTTP traffic. WAF helps defend and protect web applications from bad actors.

Website Access Restrictions

Business information is critical to every business; thus, restricting access to unauthorized users is vital. Most web hosts allow authorized users to connect to the servers and the applications within them using advanced authentication methods, such as two-factor authentication or biometric verification. These methods provide an extra layer of security that helps prevent unauthorized access and data breaches.

Also, it helps to ensure that only authorized users have access to sensitive data and systems. Avoid malicious intruders from accessing your resources by deactivating the privilege of logging in at the user root stage.

Prevention of DDoS Attacks

Distributed denial-of-service (DDoS) attacks interrupt activities on your website by overflowing its resources with devastating traffic, making it unavailable to your clients. Using robust security options is fundamental to protecting your content and brand against these malicious attacks.

SQL Prevention

Hackers can use SQL to drive their rogue codes to your website code. This method is one of the oldest and is still extensively used in cybercrime. Websites that use databases from SQL are more likely to receive these attacks. To prevent this, you must equip your team with a handy cheat sheet so that they can defend against these attacks.

Nevertheless, providers can prevent SQL attacks using different measures. They ensure you get regular firmware updates, including software, plugins, and themes. This can help them configure the web application firewall, investigate vulnerability to prevent occurrences of SQL attacks, and apply cross-site scripting.

Software Updates

Most applications, including the CMS, require regular updates. These updates contain remedies to many security risks. Companies that manufacture these software systems have patches to provide solutions to any security hole. Even if it may sound like a small step, it’s crucial to update your software to enhance the safety of your website.

Using Strong Passwords

Most people who can access crucial information on your website need to have a strong password. If a hacking attempt occurs, the staff and the administrators must consider changing their passwords. The website must possess a working password-strength policy, and every person must comply.

Conclusion

Successful websites have thousands of people visiting and leaving their data. As the owner, you should take responsibility for this personal data and protect your users. You can use web proxy services in conjunction with the best practices outlined above to defend your website. By opting for quality IPRoyal proxies, you’ll be taking the right step to ensure the safety of your site and all of the information contained within. Remember that you can always compare proxy providers to choose the best service for your need.

Pair this with the correct actions to improve your security measures’ effectiveness, and you’ll always be at your best game to focus on your business or content.

September 30, 2022
DevOps World 2022: A DevOps Retrospective

With the seminal IT conference DevOps World 2022 in Orlando September 26 to September 29 upon us, I think about the pivotal industry conferences and events critical to my path to the cloud, DevOps and cloud-native. It brings me to this brief retrospective, highlighting some of the most significant influences that still guide my path today.

I highlight these for you as conferences, books, emerging development tools, and changes in infrastructure delivery all played critical roles in my completely rethinking of how we create software through DevOps.While leading IT for a global R&D industry consortium from 2010 to 2016, I was unknowingly in the middle of a significant shift from designing network protocols to entirely moving to the cloud, creating self-service development portals and automating the development process.

This journey began partly when a friend and colleague, Alan Shimel, creator of the then-new DevOps.com site, said, “Check out this thing called DevOps … and you gotta read this book!” Looking back at my Amazon order history, I see I first purchased The Phoenix Project in November 2013. The book immediately became the main topic of discussion among my engineering team. While we didn’t label it as a DevOps tool then, Jenkins established the basis of our DevOps toolchain across multiple software-based research projects. Other technologies, including Artifactory, Maven, Git, GitHub, Bamboo and OpenShift, for example, quickly followed, enabling us to spin up development environments for many new software projects. In less than a year, we completely retooled IT from a significant bottleneck to becoming the easiest path to creating new software projects by equipping teams with DevOps, development tools, environments and cloud platforms.

Jenkins World (now DevOps World) helped me understand the central roles CI and CD played in creating automated workflows and delivering software more quickly from development into test environments. The first AWS re:Invent helped me rethink the cloud, from a data center scarcity mindset to making compute, storage and bandwidth resources harnessed through automation and a credit card. I wrote my first post about DevOps in 2014, The DevOps Journey, where I did my best to make sense of precisely what DevOps is.

As I go back and reread that post, I’m surprised how much of those thoughts still apply in 2022. Today, I continuously read many technical books, learn from thought leaders and attend influential conferences, including this year’s DevOps World. I wonder what will change how we think about and create software again in the next decade!

If you can, join us in Orlando for DevOps World. Techstrong.TV will be broadcasting live from the event; if you can’t make it, we’ll keep you updated on all of the developments announced there. Also, join us in November for our own DevOps Experience virtual event, which will highlight DevOps Everywhere: The Edge and Beyond. Hope to see you soon!

September 22, 2022
Why You Need to Empower Your Developers

In recent years, the demand for software developers worldwide has increased. However, in 2020, when the pandemic hit, many CIOs pulled back considerably on their IT spending. This trend has quickly reversed. In 2022, worldwide IT spending is expected to reach $4.4 trillion. Additionally, according to the U.S. Bureau of Labor Statistics, employment in software development is projected to rise by 22%.

Although organizations across several industries are increasing investment into IT, the supply of technology talent has struggled to keep pace. With the demand for creativity, developers understand their importance and can better dictate their terms — including higher salary compensation and enhanced benefits. Organizations that don’t meet these expectations risk losing coveted talent.

Remote work continues to be a top priority for many developers, along with great company culture. To create a positive work environment for developers, business leaders must offer a culture that makes tech employees feel a sense of belonging and purpose while working remotely. Many developers also want opportunities for growth and the ability to provide agile and innovative toolsets to help increase productivity and collaboration.

Let’s discuss some of the ways modern organizations can still empower developers and how business leaders can build a workplace culture that attracts some of the best developers in the business.

Provide a Supportive Digital Workplace

Open source usage has increased significantly in the workplace and continues to be a formative resource for developers. The open source community provides many new technologies that developers seek and even require in order to work at the top of their abilities. For example, open source provides some of the most popular programming languages available, including Javascript, C+++, and Python. The presence of multiple languages makes it easier for developers to write code and produce high-quality work efficiently.

Open source communities help developers build their skills and allow them to customize their technology stacks, as a result allowing them to optimize their productivity and learn different techniques from other open source projects. There is little to no debate that open source is critical to the tech industry and developers. Embracing open source can increase satisfaction and create a better workforce overall.

Inspire Collaboration Among Software Developers

In the past, office perks such as free food and collaborative social areas were used to attract new employees and retain current employees. However, in the remote work world, these can no longer remain a priority. Instead, business leaders should prioritize the holistic professional experience that their organization provides its employees. For developers, in particular, building this experience falls on CIOs and IT leaders to ensure they are providing workplace satisfaction, modern technology and flexible processes. To do so, they need to focus on tools that promote productivity and communication.

Most employees are familiar with the workflow software and collaboration tools that many workplaces provide. Without these offerings, remote workers would feel confused by their workload at the worst and adrift from their co-workers at best. For developers, however, these general collaboration tools won’t cut it, as they are poorly suited for engineering workflows. General collaborative software doesn’t offer the ability for deep customization, nor do they adapt natively to deploy on-prem or in the private cloud. As a result, developers have little to no use for these tools.

And that lacking sense of utility makes a crucial difference. Developer burnout is at an all-time high and inefficient workflows can keep teams from performing at their best. Or, worse, it can frustrate individual employees, leading to lower retention rates and less attractive offerings for prospective employees. Business leaders looking to offer developers an improved working environment should provide a more integrated, developer-oriented tech stack.

Empowered Software Developers are Productive Developers

While the market has changed, for many developers, fragmented tools remain a top productivity challenge. Investing in modern tools and programming languages while also removing tech complexities are the keys to improving productivity and simultaneously creating a better developer experience. This is why organizations must implement open source applications so developers can have complete visibility and control over the entire stack.

With tool fragmentation causing a number of distractions, developers can become frustrated and spend time responding to every individual incident. This poses the threat of a developer losing their love of the work. By doing away with fragmentation from old technology and selecting relevant solutions, business leaders allow developers to do what they do best: Develop software.

When organizational leaders empower their developers, they in turn find their product teams are more productive and feel more tied to the organization as well as their customers and each other. Thus, what is best for developers is best for the organization at large.

September 20, 2022
Optimizing Prometheus Implementations With Managed Services

Prometheus is a leading metrics-based open source monitoring solution and time series database used across the IT community to collect metrics from systems, applications and services. According to the 2021 Cloud Native Survey from the Cloud Native Computing Foundation (CNCF), the use of Prometheus in production has reached 65%, an increase of 43% year-over-year.

“As cloud-native stacks have grown and matured, observability has become crucial for visibility into application metrics, performance, alerting, and insights,” said Chris Aniszczyk, CTO of CNCF. “Prometheus is a key element of this, enabling organizations to build a foundation of observability.”

If you are using Prometheus, it is probably because of its many advantages including reliability, easy initial implementation and its open source nature. However, in addition to these benefits, Prometheus also brings several challenges if you are deploying and maintaining the solution on your own.

Facing the Four Main Prometheus Challenges

Prometheus users face four primary challenges that can be alleviated by the right service provider:

Challenge One: Support

Prometheus users face the dilemma posed by any open source tool: There is no vendor behind the solution to deliver technical support. It is easy to take tech support for granted, but when you are working with a technology that is not supported, even a small issue can become a substantial problem. A commercially-backed product can provide a critical level of expertise you do not have in-house, and take on responsibility for keeping the system up and running.

Challenge Two: Maintenance

The greatest challenge you’ll likely face with Prometheus is the amount of time your team must spend configuring and maintaining the system.

Just one example of how Prometheus drains your resources is the use of third-party exporters to expose metrics in the Prometheus format. These integrations are designed by members of the open source community an do not come with any support; upgrades are hit-or-miss.

Exporters are difficult to maintain. First, there are often multiple exporter options and choosing the right one for each of your services is time-consuming. Second, you can easily have hundreds of different services running in your environment and your team can spend hours every day making sure the exporters are working properly.

Another maintenance issue involves PromQL, the Prometheus query language. PromQL is a custom query language, and it is very complex and difficult to learn. Having to rely on your one PromQL expert for all Prometheus queries across the organization is simply not practical or sustainable.

With all these maintenance requirements, you will have some team members spending all their time managing and maintaining a Prometheus environment, eating away at resources that could be used to focus on business needs and innovate new applications.

Challenge Three: Scalability

Users tend to deploy Prometheus in silos, with a separate server for each cluster. Over time, you may have tens or even hundreds of Prometheus servers in your environment that you are not aware of. This leads to scalability issues regarding consistency, maintainability and a global view of your metrics.

In addition, storage is a major hindrance to scaling Prometheus, because the system typically stores metrics for about two weeks. If you need to analyze metrics older than a couple of weeks, you run into a serious problem.

Challenge Four: Correlating Data

Data does not have much value if you cannot correlate it and, unfortunately, Prometheus does not provide correlation for metrics to other elements in your environment like processes or cloud regions. This makes it difficult to pinpoint the source or impact of a problem.

Options for Solving Prometheus Challenges

Options are available to help you with Prometheus but most do not solve the four major issues outlined above. The following are your options:

APM: APM vendors all have some level of support for Prometheus, but this is an add-on to their platform. They offer an ingest tool that can transform Prometheus data into a format you can query natively within the APM platform. In reality, they are not helping you utilize Prometheus but instead are forcing you to work within the constraints of their system.

Cloud tools: Some cloud hyperscalers offer solutions to support Prometheus. These tools can help with deployment and infrastructure maintenance, but you are still stuck doing all the other configuration and maintenance work.

Managed service: The preferred option is a Prometheus managed service, which can alleviate all four of the main challenges posed by open source Prometheus. The right optimized managed service provider delivers support, maintenance, long-term storage and data correlation.

Must-Have Capabilities for Your Prometheus Managed Service Provider

The following are essential capabilities you should look for when choosing a Prometheus managed service provider:

Prometheus compatibility: You need an optimized managed service provider whose platform is designed to be natively compatible with Prometheus, consuming data in the Prometheus format and allowing you to run PromQL queries on the data.

Simplified PromQL: To eliminate the reliance on PromQL experts for every query, a managed service provider should simplify the use of PromQL, without breaking compatibility.

Exporter management: The right optimized managed service provider will be able to streamline the selection, deployment and management of exporters, saving you a significant amount of time and effort.

Automated discovery: Tracking and optimizing all your services can be challenging. For this reason, a Prometheus-managed service should be able to automatically discover the services running in your environment and help guide you with configuration.

Metric Storage: One of the shortcomings of Prometheus is the lack of long-term storage. Consequently, a Prometheus managed service provider should provide a database for long-term storage of metrics, so you can perform comparative analyses of the data, from month to month and even year-to-year.

Prometheus compatibility: If you already use open source Prometheus, your teams are used to using things like PromQL and Grafana dashboards. You should select a managed service provider that is Prometheus-compatible so your team can use the same dashboards and queries they have always used to view the data.

Conclusion: Optimized Valuable Tool

Prometheus is not a standalone solution. But it is an optimized valuable tool to collect metrics for performance analysis and it is appreciated by your architects, developers and IT Ops pros. While Prometheus may have some limitations, it is worth finding a solution to these issues; that solution is a managed Prometheus service. The right managed service provider can give you the best of both worlds—all the benefits of Prometheus without the time, resources and expense needed to run it effectively.

September 16, 2022
Heat Cooks Twitter DC | AI Will Kill All Humans | Patreon Layoffs, CSAM Claim

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Twitter is in a “non-redundant state” thanks to a hot summer, AI is likely to eliminate us, and Patreon fires 80 staff amid nasty allegations (TW: Child safety). (more…)

September 15, 2022
Where Should I Store My IaC?
The most successful software companies rely on repeatability, auditability and simplicity when building solutions. The emergence of infrastructure-as-code (IaC) has empowered developers to apply these practices to infrastructure allocation. What are the best practices for storing IaC? Should IaC code be stored with the application and feature code? Should it be stored separately in its own application-specific repo? Or should all IaC code for all applications in an organization reside in the same repo?

Let’s explore these questions, determine the tradeoffs between each solution and create guidelines that consider your organization’s needs.

The Problem With Storing IaC With Feature Code

The naive option for storing IaC is to keep all IaC code within the application repository that consumes this IaC code as part of its CI/CD pipeline. It’s easy to set up, doesn’t require complex permission allocation and will be fully visible to developers who work on this codebase, thus making it easy to retrieve context. This ease is why 20% of polled developers state they always store feature code and IaC in the same repo. However, there are a number of problems with holding IaC code with feature code.

Software development best practices include minimizing redundancy. Storing all IaC code within the dependent application’s codebase violates best practices by requiring the infrastructure-as-code to contain not just application-specific allocation information but also all foundational infrastructure. Organizations may classify common infrastructure shared across multiple applications and ecosystems as foundational infrastructure. Examples of such base, foundational infrastructure include Kubernetes clusters and log storage accounts.

In addition, storing IaC code with feature code introduces risk. Allowing developers admin-level access to all deployment scripts needed to bring code to production violates the principle of least privilege and makes lateral movement easier for attackers who manage to hijack a developer account.

Organizations benefit from a degree of secrecy regarding their deployment processes; the deployment pipeline and its configurations are bespoke and attackers who have a mapping of the services an organization uses can more easily architect supply chain attacks. Storing IaC with feature code makes it so that a code leak reveals all CI/CD configurations for not just the application, but the whole organization. The last thing an organization should do is perform an attacker’s reconnaissance for them, but that’s exactly what happens in the above situation.

Storing infrastructure-as-code alongside product code may be easier to set up initially, but violates the principle of least privilege and zero-trust principles. This method of storing code introduces unnecessary risk–there must be a better solution.

Storing All IaC Code Together is Not the Answer

Since storing IaC with feature code in a common repo isn’t the optimal solution, what about storing all IaC in a dedicated IaC repo? This would allow DevOps and security teams to implement granular access controls and make code redundancy easily identifiable and preventable. However, this organizational structure introduces its own problems.

When all IaC is stored in a separate, shared repository, it becomes difficult to match feature changes to IaC changes. This scenario introduces friction to the development processes, puts a damper on developer velocity and increases tension between development teams, DevOps teams and security teams.

Storing all IaC in a common repository gives developers virtually no control over security decisions affecting their workflows. Between the lack of ownership and the additional steps needed to modify any code-specific deployment scripts, storing all IaC together prevents developers from quickly changing IaC and it unnecessarily hurts the organization.

IaC Falls Into Two Categories: Base and Feature-Specific

IaC and other CI/CD scripts ultimately fall into two categories: base and feature.

Base deployment scripts may be described as code used broadly by an organization for, among other things, routine deployment actions or to interface with shared resources. Separating this code empowers organizations to allocate base IaC ownership and maintenance responsibilities to their DevOps team, which is well suited to maintain shared infrastructure.

Feature IaC scripts are the bits of code used to provision and manage resources associated with specific software. Developers frequently interact with the deployment scripts tied to their deliverables; changing release steps, modifying gateways, updating server configurations and many other needs necessitate different handling protocols for feature IaC versus base IaC.

Now that we understand the difference between the feature and base IaC, we can define proper management strategies for each.

The Solution: A Hybrid Approach for Storing IaC

Best practices call for categorizing feature IaC and base IaC differently and allocating the responsibilities to separate teams to maximize security while not adding unnecessary friction to the development and release process. Base IaC should be managed by DevOps teams, while feature-specific IaC should be handled by the code owners of the managed software.

For your organization, this starts with inventorying the organization’s systems and services. Gaining visibility into how these resources are shared helps inform business decisions regarding IaC code storage, helping minimize redundancies within deployment protocols. Visibility is integral to defining which IaC is foundational to software deployed by the organization versus IaC which is specific to certain applications, libraries and other software deliverables.

Helping DevSecOps Teams Manage IaC and More

Best practices don’t mean much if organizations can’t enforce them. Some concluding tips:
- Track resources, including registries, repositories, builds and cloud/IaC deployments
- Implement granular access controls, enabling the appropriate allocations of ownership depending on whether the resource is meant to be shared or application-specific
- Provide visibility into the use of resources which empowers organizations to adhere to the principle of least privilege without adding unnecessary friction to the development process
- Provide a comprehensive asset inventory by integrating directly with the components of your organization’s software supply chain
These integrations allow complex insights to be derived that are simply not possible with other security tools. In addition, these integrations allow for constant monitoring for security updates, thus reducing the time between a zero-day being discovered and patching the vulnerability.
September 15, 2022
Embrace Change in Production to Improve MTTR

DevOps creates value through constant deployment changes. When done right, more frequent deployments allow teams to continuously improve their products or services while making it easier to manage the effects of the change. One way to do this is with continuous integration and continuous deployment (CI/CD). CI/CD helps to reduce the risks to each build and allows teams to release valuable features quickly and safely. But CI/CD alone is not sufficient for understanding change and service impact in production. The recent DEJ State of IT Performance Report found that change is the largest source of production issues. A full 76% of all performance problems can eventually be traced back to changes in the environment but 67% of organizations don’t have the ability to identify changes in their environment that caused performance issues.

In a modern IT environment, different teams are constantly integrating new features and new technologies and renewing their stack. With many moving parts, it’s impossible to mimic the production environment when testing new features in pre-production. By the time you’ve implemented and tested new code, your branch will be different from what is in production.

Here’s the reality we’re working in:

● Integrating testing in CI/CD pipelines will not keep bugs out of production. Some organizations don’t do true CI/CD, meaning some processes are not even tested and automated in the pipeline.
● User configuration errors are common. Here’s a consolidated list of configuration errors that brought down production.
● Change anxiety is real: Ops folks get nervous when hitting the deploy button, even for small changes.

Changes are inevitable in DevOps, but organizations can protect themselves from a performance incident or an outage by integrating change in production safely with a few simple steps.

Build Data Connections

Today, data is kept in silos, which is why it’s time-consuming and challenging to make associations and leverage them to find the root causes of performance issues. DevOps teams need to connect their data in meaningful ways to troubleshoot complex issues in distributed systems. Building the “connections” foundation is critical to understanding the state of a system and how the pieces interconnect and interact.

How well an application performs in production is related to a combination of code, infrastructure and operations. For instance, when an incident occurs, teams should quickly know if that issue is related to code, infrastructure or operational activity. Understanding these connections and correlating the data cannot be performed manually. We have hit the wall in linking together data silos; we just don’t scale as software does. Teams need to consider a monitoring and analytics approach that connects operational data together from the start and models the causal links between the data for real-time root cause diagnosis.

Understand Change Impact to Avoid Outages

How well can your DevOps team identify the key changes made to your production environment? If your answer is “not very well,” then you’re missing critical data that helps to speed up troubleshooting. The lack of visibility into changes and how they impact production makes it impossible for operations staff to identify the cause—the very thing that SREs care most about.

Changes are made to the code or the cloud infrastructure, either directly or through infrastructure-as-code (IaC) templates, for a variety of reasons. The risk of misconfiguration and performance issues increases when these operational activities are not continuously monitored and correlated across the stack.

Metrics don’t tell the whole story. Logs don’t pinpoint where to act. Simply collecting these data types leaves many companies struggling with knowing what is important and what’s not when it comes to real-time troubleshooting.

Having a shared “system of record” for changes for all the different streams of activities increases visibility and control of all changes in real-time. Teams can see pre-incidents and be aware of increased risks when changes are made, especially during high-velocity change periods. By seeing how a code change can impact things down the line, developers can proactively troubleshoot and debug code.

Implement Real-Time Root Cause Diagnosis

In the same DEJ report, organizations shared that 66% of MTTR is spent on the ‘diagnose’ stage. This correlates to the 67% of teams that are unable to identify the changes in their environment that caused performance issues.

With a shared “system of record” for changes, DevOps teams can troubleshoot incidents in real-time, dramatically cutting the time spent on the “diagnose” stage of MTTR. This is especially effective when teams can see all the changes and other events that are happening to the system in a single, searchable/filterable real-time feed; this timeline would provide the potential causes of issues.

The best tool would allow you to start from an effect and work backward, chaining multiple events overlaid with observability data to determine the causes quickly. Building the causal chain from the end helps teams hone in on the connected change and answer each “why” in a straightforward way. This capability makes it easier and faster to prioritize relevant changes and resolve performance degradations or outages with actionable context.

Communicate Everything

During diagnosis and resolution, it’s important to see what’s being done without having to ask on Slack, wait for each other or unknowingly interfere with remediation actions. The best option would be leveraging the same timeline to capture all the context of an incident at the moment it occurs, and at any moment in time during diagnosis and resolution, to avoid reverting each other’s changes.

While an incident postmortem brings teams together to gather data and discuss the details and learnings from it, this process is often inefficient and time-consuming. Each team member involved in the incident response effort would have to recollect what they observed, what actions they took and what they communicated to other team members. What if a system could capture all the actions and communications during an incident response effort? Wouldn’t it be nice to just summarize your thoughts in one sentence and be done with the reporting?

When preparing for planned and unplanned changes that can impact the bottom line, DevOps teams need to consider these steps to gain true observability of their operations. Focus on finding the connections in the data to explain cause and effect. Understand change impact —as those changes are being made —to avoid outages. Capture and communicate activities as seamlessly as possible and identify and capitalize on opportunities to automate the workflow.

September 14, 2022
The Definition of Done: Why it’s Important for Success

Establishing clear communications and expectations across software organizations is essential to the success of any project. A company is less likely to experience unexpected delays or other unpleasant surprises if everyone is on the same page. To ensure all team members are synchronized and to eliminate unnecessary ambiguity, engineering teams should create a “Definition of Done” to guide projects.

What is the “Definition of Done”?

In Agile software development, the definition of done is a working agreement between teams that defines a set of standards to identify something as “done.” Because Agile is an iterative approach to software development, sometimes it is unclear when a task is complete. Product managers often report that something is ready for delivery when it’s “Dev done” and not “QA done.” In other words, the code has been written, but no documentation or tests have been created or executed. For other parts of the software organization, code without QA or documentation is not “done.” Intuitively, the definition of done exists to eliminate confusion about what “done” means across an organization.

How to Create a Definition of Done for Your Software Organization

For some software organizations, the definition of done is a static set of guidelines and benchmarks established at the start of a project. For other organizations, including here at Allstacks, engineers have the latitude to amend the definition of done as they go. That approach empowers an engineering team to make decisions that will produce the best possible results.

When establishing a definition of done within an engineering organization, consider these tips:

● Communicate criteria in a way that the whole software organization can universally understand.
● Ensure that the definition of done addresses both tactical and technical requirements.
● Create a living document for alignment that can be amended because of unforeseen circumstances, like losing a team member or a significant market shift.
● Consider that the robustness of the document will vary by team maturity, and specific teams might need an addendum for special requirements. For example, bigger enterprise IT teams frequently need more detailed documentation compared to two-person shops.
● Establish an understanding among all teams that, until all criteria are met, the user story, task or project is not “done.”

A definition of done acknowledges the shared responsibility between the frontend, backend and QA. Beyond those roles, other stakeholders might contribute to defining “done,” like Scrum masters, board members or business analysts, depending on the organization and its structure. Further, the product owner may also define “acceptance criteria” with goals targeted toward the end-user experience that the engineering team must also address as part of their work.

How a Definition of Done Can Help Teams be Successful

The definition of done helps a software organization maintain alignment on projects and keeps everyone on the same page. Misalignment can have heavy consequences. For example, it can initiate premature conversations with customers before a product is really ready, create friction between internal stakeholders and/or risk other go-to-market initiatives. When the definition of done encompasses defined steps for a project down to the task level, Agile teams can create more cohesion across an organization and eliminate unforeseen delays.

A definition of done outlines each task in a software engineering project. Noting the completion of each task can be an indicator of whether a project is on track to meet established deadlines. By noting the time it takes to complete each task, an engineering team can measure the velocity of a project, and that can help protect due dates or raise a warning flag about the need to shift timelines.

The Definition of Done Helps Support Software Team OKRs

Objectives and key results (OKRs) are a goal-setting framework to help senior leadership focus on what matters to the organization. By defining software engineering OKRs, senior developer leaders can keep their teams focused on aligning engineering goals and activities to business outcomes while also giving them room to experiment.

Creating engineering OKRs involves taking the organization’s objectives and crafting actionable, generalized engineering objectives while identifying the metrics and milestones that drive progress. Translating the organization’s objectives for the engineering team helps them to understand exactly how their work impacts the company.

One of the key benefits of the OKR framework is giving the engineering team a shared language and vision for success. However, without an established definition of done, OKRs will have a lot of unintentional gaps (and teams will have a lot of misunderstandings). Start with a definition of done to ensure that each team knows what their part is and when it’s officially ready for handoff to another team. Teams will pass the baton smoothly across the organization and there will be fewer obstacles in the race toward release in the market.

A Definition of Done Can Help Organizations Grow Smarter

In a small startup with a handful of software employees, establishing a shared understanding of timelines and goals is easy. As software organizations grow, casual chats at the water cooler aren’t enough to keep an organization in alignment. All team members must understand and have input on goals and objectives and those goals must be in writing and accessible to everyone. The benefits of establishing a definition of done include creating a shared understanding and unified language for software delivery, ensuring that new employees have access to tribal knowledge and process expectations, and detecting and removing false agreements between teams to create real consensus. By creating a definition of done early in an organization’s history, company leaders can ensure alignment even as more people join.

September 14, 2022