devopsy.kinsta.cloud

Author: Bill Doerrfeld

Popular iOS Apps Request Excessive User Data Permissions

Most apps require some sort of user data to function properly. Google Maps needs location data to offer routing services. Twitter needs photo library access to upload a photo, and so on. Many users are quick to allow access to personal information without hesitation. However, in recent years, it’s become apparent that many apps are collecting (and leaking) an excessive amount of personal data, resulting in serious privacy issues. This has led Apple to give user privacy a heightened focus in recent iOS updates and marketing campaigns.

Often, user data is being collected in the background for legitimate use cases, such as tailoring personalized user experiences and refining app performance. But, many apps are also monetizing user data by reselling it to third-party advertising data brokers without explicit user consent. More seriously, recent studies show that commercially available “anonymous” data can be de-anonymized. If location data is exposed, it could put at-risk people in actual danger. These are nuanced privacy violations that U.S. regulators have yet to respond to and address.

To shed light on the data that apps are collecting, a recent study conducted by Jamf analyzed the iOS permissions requested by nearly 100,000 popular apps on the App Store. The study found the four most-requested mobile app permissions to be photo roll, camera, location data and microphone. The findings also revealed that popular apps are consistently requesting access to data that does not serve the application’s core purpose.

Users should consciously be aware of the data they allow apps to access. Yet, the onus is also on business leaders and app developers to limit unnecessary data collection and reduce the potential for data overexposure. In the wrong hands, access to device hardware, such as a microphone or camera, could be used to steal sensitive material. And, if leaked, personally identifiable information (PII) exploits could be disastrous to the business.

Below, I’ll review the key takeaways from the Jamf study and consider if the constant barrage of permissions requests is exacerbating privacy issues during the application development process.

Four Most-Requested App Permissions

The Jamf research found the top four types of app permissions were photo library access, camera access, location data and microphone access. In addition, other popular areas included calendar information, contacts and Bluetooth. Less-requested app permissions included voice processing, health data and local network access. The study arranged apps by their App Store categories, such as business, education, entertainment, news, shopping, travel and other areas.

1. Photos

By far, the most requested data access type is photos. At least half of apps across all categories request access to user photo libraries. Naturally, this request is most common among photo and video apps such as YouTube, FaceApp or Splice; 96% of such apps request access to photos. This is followed by 87% of shopping apps (Amazon, Shop, eBay, etc.), and 84% of social networking apps (Facebook, Instagram, Twitter, etc.).

At issue here is that the level of intended photo library access can be highly variable. For example, granting access to an entire camera roll along with decades of old personal photos just to upload one picture grants a rather invasive privilege. This has led Apple to enhance privacy controls—users can now choose to allow access to select images only or their entire roll.

With these new constraints in mind, app developers should take steps to respect user privacy. Be aware of potential photo limitations and design apps to only collect necessary photos for the task at hand.

2. Camera

The next-most requested permission is for camera access. This is most common among photo and video apps, among which 90% request camera access. This is also common among shopping and social networking apps (83% each) and business apps (75%) like Zoom, Slack or WebEx. Similar to findings about photo library access, enabling camera access is ubiquitous. However, it is also one of the most dangerous permissions to grant indefinitely. If a bad actor obtained account access, this vulnerability could enable them to spy on users.

Again, developers should consider introducing camera functions only when it’s essential for the task at hand. Obviously, it’s unethical to invoke camera access when users are unaware. Even camera-related bugs can have extreme repercussions. For example, a 2020 lawsuit against Instagram argued the app was spying on users. Due to its threat potential, high-security facilities have disabled camera hardware entirely.

3. Location

Location tracking is another permission that can be very useful but is, at times, overprescribed in modern app development. Location data includes cellular, Wi-Fi, GPS and Bluetooth data used to determine a user’s approximate location. The categories that collect location data most often include shopping and food and drink, both at 81%. Social networking and photo and video app categories follow at 72% and 68%, respectively.

Nowadays, iOS privacy settings offer a check against tracking location data in the background. iOS users can now choose between: ‘Allow While Using App’, ‘Allow Once’ or ‘Don’t Allow’.

When an app tracks data in the background, a blue indicator now displays in the status bar to increase user awareness. iOS also reminds users when apps use significant amounts of location data and offers ways to turn location services completely off. Android has followed suit with an ‘Allow only when the app is in use’ function, too.

Location tracking feels a bit creepy. In the wrong hands, it could easily be leveraged for nefarious purposes. Out of all the permission-granting options, location data presents much more of a physical safety concern for the end user. From the user’s standpoint, they should only enable access to location tracking while using an app. From a developer’s standpoint, they should be extremely careful when collecting and storing such PII to ensure regulations aren’t broken.

4. Microphone

Lastly, access to the user’s microphone is the fourth most commonly requested permission. The survey found 69% of social networking apps, 64% of photo and video apps and 41% of both business and productivity apps (Asana, Google Calendar, TimeTree, etc.) all want to be handed the mic.

Here, potential privacy concerns are similar to those involving camera access. Bad actors could potentially activate the mic to listen to private conversations, record ambient noise and sell findings to advertising brokers. To increase user awareness, iOS 14 now requires an orange dot to display, indicating when the mic is in use.

Other User Privacy Concerns

User behavior tracking and app-to-app information exchange is another ongoing area of privacy concern. Companies often resell such information to data brokers, which is then used to refine highly targeted ad campaigns on third-party sites. This is how your search on Google results in product advertisements popping up in your Instagram feed. Apple is just starting to counter this practice with their App Tracking Transparency feature. In iOS 14.5, app devs must ask whether they can track your activity.

Recent analysis has confirmed that commercially available consumer data, though labeled as “anonymized,” can successfully be used to piece together an accurate picture of a user by corroborating behavior on other apps. For example, users were able to track and pinpoint behavior that resulted in the outing of a catholic priest on Grindr. Smartphone app data could similarly be used by predators to spy on celebrities or politicians, posing a tangible security threat.

Another cause for concern is metadata associated with photos. People often enable GPS-embedding into images to help organize their photo memories. If this is inexorably linked to photos, it essentially permits location data sharing to parties with photo library access who might not otherwise have location tracking permissions.

From Big Data to Big Problem

Perhaps the most curious finding of this report is that all apps are requesting permissions that seem irrelevant to their core value proposition. For example, why do 87% of shopping apps need access to a user’s entire photo library? Do 46% of food and drink apps really need access to a user’s personal contacts to place a food delivery order? Do navigation apps really need your personal calendar information to calculate a route downtown?

The answer is not a pretty one. There has been a business imperative, for years, to track and collect large amounts of user data. There is undoubtedly an argument to be made for enhancing the user experience with innovative features and the role of big data in AI. However, this ongoing practice continues to creep into ethical gray areas. The fact is, the more personal data an app collects, the more potential privacy issues and security threat vectors are created in the process.

Requesting permissions to on-device hardware and data collection is not always necessary. Thus, to stay safe and limit data overexposure, don’t overprescribe data collection. If it’s not fundamental to your app’s core function, don’t collect it, and don’t store it. Think twice before requiring many device permissions, and regularly audit app permissions requests to keep them to a minimum. As iOS and Android introduce new platform guidelines to elevate user privacy, developers should be prepared for users to grant fewer and fewer data collecting permissions. This is their right.

It would be great if tech companies invested in honest, ethical business models that didn’t rely on harvesting and reselling user data to shady advertisers. And for those that mine user data but say it’s anonymous, I would point to reports that show anonymous data is nothing but pure myth.

August 30, 2021
Majority of Software Engineers Want Remote Work Options

The pandemic brought unprecedented change to the digital ecosystem, impacting users, companies and software engineers alike. COVID-19 accelerated ongoing digital transformation and forced many organizations to innovate rapidly. Newfound reliance on digital experiences increased demand for software programmers, exacerbating demand in a market already short on talent. In this era, work-from-home (WFH) became commonplace and most engineers moved toward full-time remote work positions.

Having had a taste of freedom from corporate settings, most developers now want remote work options in a post-pandemic future. A significant amount also desires to work 100% remotely. As a result, many companies have instituted permanent work-from-home policies. Yet, they are still grappling with the repercussions of a fully remote workforce. For example, there are differences of opinion about whether to adopt location-based salary adjustments and whether or not those remote workers should be granted unique employee benefits.

The 2021 State of Remote Engineering Report, conducted by Terminal, surveyed 1,100 software engineers in North America and explores how the pandemic has affected the remote developer workforce; showcasing both the positive and negative effects of remote work on engineers. Interestingly, the report exposes international talent’s declining interest in physically moving to the U.S. as domestic conditions worsen and remote work becomes more globally accessible. Below, we’ll explore the key takeaways from the study to posit how this new normal is influencing the productivity, morale and recruitment of software engineers in North America.

The New Normal: Remote Work

The new normal has dramatically shifted the status quo. Impressively, findings show that 86% of engineers are currently working 100% remotely. This is in stark contrast to the pre-pandemic numbers, which showed 19% working remotely 100% of the time. Many of these employees prefer to keep these new working arrangements and standards in place going forward. The study found that 80% of engineers surveyed want options for working from both home and the office, and one out of three want to work remotely 100% of the time.

There are many reasons why engineers prefer the flexibility of remote work to the traditional workplace. For one, there is no need to commute, thus saving time and energy. Engineers also cite an improved work-life balance and appreciate the increased autonomy. It also appears that many organizations handled the shift to remote work effectively—74% of engineers say their companies adapted well to remote work. Due to these positive results, 76% of engineers would be happy working fully remote 60% of the time or more.

Remote work not only benefits software engineers but could also bring productivity enhancements to help achieve business goals. Without workplace distractions, 70% of engineers surveyed reported they are more productive when working from home. Yet, this new normal isn’t without its drawbacks. As the traditional corporate firewall diminishes, CISOs struggle with a dilemma: striking an ideal balance between granting new IT freedom and locking down systems and restricting access to adopt a zero-trust mindset.

Issues With the Pandemic and Remote Work

Of course, there are some other drawbacks associated with full-time remote work. A lack of day-to-day interaction with coworkers is a challenge cited by 58% of participants in this study. For 39% of engineers, it’s harder to collaborate or feel part of a team in this new dynamic. Burnout or lack of motivation and home-based distractions can also be barriers for remote-working engineers. The loneliness of at-home work can take its toll; 38% say their mental health has gotten worse amid the pandemic.

Companies also face challenges recruiting within this new paradigm. The talent shortage remains a big issue for many companies trying to rapidly deliver new software; 64% of engineers say there is a software engineer shortage and 65% say this gap holds back technical development. Though there is a clear skills gap, most companies are not offering adequate benefits coverage to entice full-time workers. Of these benefits, 23% of engineers would like a home office stipend for utilities, internet and other expenses; 22% also want more flexible work hours and 11% desire new technology and productivity tools. And 26% of engineers also ranked support for mental health as one of their top three sought-after benefits.

The report also addresses some inefficiencies associated with hiring engineers for remote positions. For 59% of engineers surveyed, recruitment involves far too many interviews. Many of us working in IT can likely attest to being approached by LinkedIn scouts with irrelevant offers. As such, 40% of engineers say they are often interviewed by people who don’t understand the technology. Other recruitment issues include generic interviews and long delays, both of which could frustrate candidates. It appears companies must fix their hiring practices or risk losing potential talent.

The Era of Employee Choice

If anything, the State of Remote Engineering report drives home the imperative to consider full-time remote work as a viable option for prospective hirees. To fill engineering labor shortages, some U.S.-based companies have historically flown in international talent. Yet, the desire to move to work in the U.S. is declining. In fact, 39% of engineers have no interest in moving to the U.S. for a job. Engineers cite crazy politics (56%), gun violence and safety concerns (55%), racism (47%) and mishandling of the coronavirus (51%) as reasons not to move to the U.S. The idea that employees must remain in physical proximity with a company’s headquarters is changing; thus, companies will have to go where the existing talent already lives.

In addition, today, engineers have more bargaining rights than they did in the past. “The era of employee choice is here,” said Clay Kellogg, CEO, Terminal. If we’re in a seller’s market, engineers are the ones to profit. Increased bargaining rights could mean remote engineers may demand equal pay, no matter their location. The study found that, on the whole, engineers are not in favor of location-based salaries; 49% say they would only accept a location-based salary if their cost of living dropped by 20%.

In addition to offering competitive wages, teams are finding creative ways to maintain culture and avoid burnout. For example, 55% of engineers think virtual get-togethers like happy hours and social activities help stave off burnout. With all these findings in mind, companies should consider instituting a work-from-anywhere policy wherever possible and provide additional benefits and support for remote workers to attract more engineering talent.

The State of Remote Engineering surveyed over one thousand software engineers across the U.S., Canada and Latin America. For more granular details, consider viewing the full report here.

August 24, 2021
What It Will Take To Bring Cloud Gaming to Fruition?

The gaming industry is booming. There are 211 million people gaming in the U.S., and it’s one of the fastest-growing forms of entertainment on the planet. In the age of Twitch streaming, gaming has even upstaged traditional sports viewership. On average, young gamers between the ages of 18 and 25 spend a full hour more watching online gaming than watching traditional sports.

Gaming has also become a way of connecting with others—65% of players say they play with others online or in person. Some software development teams even encourage gaming in the workplace to increase productivity. For this reason, there has been a massive shift toward online cooperative games, RPGs and sandboxes like Minecraft. Also, games are becoming more cloud-based, with purchases, downloads and game saves occurring in the cloud. Still, many people lack the hardware necessary to run complex games locally. All these factors are harbingers of a coming shift away from PC and console-based processing.

I recently met with Noam Salinger, product manager, Granulate, to learn about the future of gaming and how software teams in the gaming industry can prepare. To Salinger, “cloud gaming is the next step.” The move away from consoles and toward cloud-based, remote processing will have huge ripple effects. Lowering the barrier to gaming could introduce millions of more players into the ecosystem and further balloon the industry. However, cloud gaming presents significant hurdles. The transition will require a substantial reduction in latency, 5G networking and highly efficient computing optimizations.

Roadblocks to Cloud Gaming

“Cloud is an integral part of any gaming company,” says Salinger. More and more functionality associated with gaming is being pushed online. There are already online game marketplaces, cloud-based account management and remote game saves. This is especially prevalent with mobile gaming. But, what about the actual processing for computationally-intensive games—could these workloads also move to the cloud soon?

This kind of cloud-based gaming could soon come to fruition. Yet, Salinger foresees some major roadblocks to cloud gaming:

1. The Software Components

A complete jump to cloud gaming will require having the proper software infrastructure in place and getting vendors on board to support it. This is perhaps the biggest obstacle to cloud gaming in the software industry, said Salinger, “It’s still not mature enough.”

2. The Business Components

Furthermore, gaming platform companies might refrain from cloud gaming due to fears of it cannibalizing their existing hardware sales channels. On the other hand, cloud gaming presents a big opportunity to take on more of a subscription-based model, such as allowing a user to play any game they want from an extensive suite of games. Akin to streaming services like Netflix, Hulu or Peacock, we could see similar platform styles emerging in gaming. “They will have no choice but to participate at some point,” said Salinger.

3. The Network Components

As most gamers can attest, online games can easily suffer from network issues that negatively affect gameplay. Salinger predicts that the ubiquity of 5G networks will be necessary to propel the cloud gaming market forward. 5G could be a universal method to enable real-time connections for a broader base of end-user gamers.

Even if software developers optimize game processing speeds, a poor network could add enough latency to data throughput to cancel out those gains. 5G could expedite this process, but it’s taking a lot longer than expected. Some estimates place 5G finally arriving at full throttle sometime in 2023, after more antennas are constructed and pre-existing physical infrastructure is upgraded.

4. The Latency Components

Another significant roadblock to cloud gaming is figuring out real-time remote computing on the game vendor side. “This is where the latency aspect of things really comes into play,” said Salinger. Since game clients will be making many remote calls, optimizing cloud CPU utilization will be vital to reducing the response time to requests.

Cloud gaming will require parallel running processes that anticipate user action. Similar to the way modern web applications simultaneously trigger multiple backend requests for authentication, geolocation, user databases and so on, real-time cloud games will require many different processes running in parallel. Servers could allocate resources before requests come in, using predictive analysis of usage patterns to trigger components in advance of user behaviors.

To enable intelligent resource allocation, Salinger describes how an OS-agnostic kernel module could be in charge of resource allocation, continuously adapting to changes in the application workload. Low-level resource allocation at the CPU level, the allocation memory, I/O, and the sequence of execution must be optimal for each specific application, he described.

This sort of process is already being applied in production by some ad tech companies to enable real-time ad bidding. For example, by optimizing cloud CPU utilization, Perion is able to perform faster, process more responses and use fewer machines to perform the same task—thereby reducing the roundtrip time for each request. Similar principles could be used to advance cloud gaming.

Rendering sophisticated virtual environments is highly resource-intensive, and high-performance graphics cards are expensive. Video games are set to only increase in complexity with new AR/VR modes. Cloud gaming could help respond to rising computational needs and replace the need for local computing hardware. In the process, this move could bring in more gamers, disrupt the console wars, and nullify age-old PC-versus-Mac compatibility issues.

Moving to a cloud-based approach is also arguably more aligned with today’s fast-paced software release cadences. Pushing more frequent patches can help resolve bugs faster. But it also could aid in retaining engagement by introducing new levels and objectives continually.

The rise of the gaming ecosystem can’t be ignored. But as it stands now, there are still significant challenges to wider penetration of cloud gaming. Cloud computing optimizations to reduce latency, mainstream adoption of 5G and significant re-platforming of software infrastructure are just some of the many issues that will need to be tackled before truly cloud-based gaming experiences advance to the next level.

August 4, 2021
Making Blockchain-Based App Dev More Accessible

Blockchain has many potential uses cases outside of cryptocurrency. Established sectors such as financial, automotive, supply chain, and mobility could soon move to adopt decentralized applications and distributed ledgers. The technology has even been hailed as the ideal foundation to maintain a connected economy once humans go interplanetary.

Recently, spectators note a swelling interest in decentralized, peer-to-peer, autonomous development approaches. Eyeing the excitement around cryptocurrency and blockchain, many developers are eager to start experimenting with their own blockchain-based applications. This incoming disruption could remove the middle man and give power back to people, says blockchain and cryptocurrency entrepreneur Max Kordek.

However, building your own blockchain from scratch is incredibly difficult. So, I recently met with Max Kordek, co-founder, Lisk, to understand how to make blockchain-based app development more accessible. According to Kordek, there are many benefits to creating your own blockchain, but simplifying the creation process will be necessary to empower new startups and enable this new paradigm to flourish.

Potential Use Cases

The decentralized, tamper-proof nature of blockchain has disruptive potential to remove central authority and give more data control to the collective hive. “For me, it’s all about removing the middle man,” said Kordek. “It gives power back to the people.” Cryptocurrency is obviously the most widespread use case; Bitcoin is proof that a blockchain-based system can see mass adoption. But there are many other use cases outside of cryptocurrency where similar record-keeping is important.

For example, the automotive industry could link data across manufacturers or share data between vehicles. NFTs are using blockchain to represent one-of-a-kind digital art objects. The airline industry is looking to blockchain to synchronize partner ecosystems. Blockchain is also transforming online gaming by substantiating in-game value systems.

From insurance to supply chains to transportation, most industries hold a potential blockchain use case. But blockchain isn’t constrained to only terrestrial scenarios. Information systems engineers foresee blockchain powering future in-space economies. With significant funding going toward asteroid mining and Mars missions, an interplanetary humanity seems inevitable. Compared to Bitcoin, “Ethereum is a better choice for the interplanetary economy,” said David Vandervort. Gathering transactions in blocks and pinging satellites could enable monetary transactions from planet to planet. “An interplanetary financial system would have to be built on blockchain technology,” Kordek said.

There is ripe potential in countless areas. But who will be leading this shift? According to Kordek, we may see corporations augmenting existing structures with blockchain. However, established enterprises will find it challenging to unshackle themselves from legacy systems. Instead, new startups are more likely to have a bigger say in the blockchain revolution. “It’s much more realistic that an entirely new startup adopts blockchain and structures their entire business around blockchain,” said Kordek.

The State of Blockchain Development

After Bitcoin was established, other developers forked the project and changed parameters to introduce new cryptocurrencies, such as Litecoin and Dogecoin. Eventually, Ethereum came along, which was unique as it enabled developers to build apps with smart contracts. However, “the developer ecosystem was horrible at the beginning,” explained Kordek. “It was the wild west of development.” To this day, it’s still very complicated to develop on Ethereum, he added. Plus, since it is open source and immutable, it’s vulnerable to security threats. “It’s like the biggest bounty program in the world—if there’s one bug, contracts are drained.” Ethereum also has scalability issues — it can only run 30 transactions per second, equating to 1 million transactions per day.

For these reasons, it may be better to build your own blockchain. However, doing so is a difficult task. From a research perspective, this requires a lot of theoretical knowledge, Kordek explained. Blockchain also introduces new maintenance concerns—since the infrastructure is always on, it is unstoppable, making patching a bug or pushing an update tenuous at best. If hundreds of millions of dollars are dependent upon a stable network, one mistake could have drastic repercussions. The high development and QA roadblocks make prototyping blockchain-based systems from scratch challenging. “Experimentation is essential to bring an industry forward,” said Kordek. “It’s important we simplify blockchain development so more experimentation can happen.”

Open-Source SDKs Make Blockchain More Usable

So, how do we make blockchain more accessible so that experimentation can occur? Well, a lot of progress has already been made via ready-made open source language-specific SDKs. For example, there’s Polkadot (Rust), Cosmos (Go) and Lisk (JavaScript). Kordek is an advocate for language-specific libraries that fit a team’s programming preferences. Out of these, he favors JavaScript, as it’s generally easier to learn and already supports many web-based interfaces.

These projects are not about the smart contracts themselves; they’re about improving your own blockchain creation process. According to Kordek, going down this route grants more control in the early days. If something is broken, you can fix and push and update. If tokens are hacked, you could roll back and adjust things, he said.

As companies get bigger and bigger, the technology could have a centralizing effect. “How can we somehow bring power back to people?” asks Kordek. Blockchain is one profound answer, empowering new entrants to alleviate problems across industries — it could even help us boldly go where no one has gone before.

August 3, 2021
Multi-Cloud Is Not About Lift and Shift
Multi-cloud is often framed from the perspective of “lift and shift.” In other words, an organization takes a computing environment previously running on one cloud and transfers it over to a different service provider. Cloud migration may make sense for cost savings or performance benefits and such a shift may be necessary when migrating on-premises data centers to a public cloud.

In theory, lift and shift sounds like it would empower cloud consumers with more choice in the marketplace. However, according to John Gentry, CTO of Virtana, lift and shift is quite rare in practice; it may not be the true driver behind multi-cloud strategies. Moving resources between clouds can result in costly egress fees. Plus, culturally-ingrained cloud preferences become challenging to shake. Unique configurations and add-on software-as-a-service (SaaS) make each cloud more nuanced, requiring new skills and procedures with every migration.

Instead, Gentry advocates for a “fit-for-purpose cloud” strategy. This attitude is less about moving similar workloads between clouds but, instead, pairing particular types of workloads with the appropriate cloud types. According to Gentry, using multiple clouds simultaneously could still help organizations strategically align their digital resources and reduce dependency on a single cloud service provider (CSP). But, the real value of a hybrid multi-cloud setup lies in matching computing environments to different business layers.

Fit-For-Purpose Cloud

At first glance, the ability to move between clouds appears to be driving a new multi-cloud economy where cloud strategists can swiftly migrate resources and optimize intercloud utilization. However, when organizations attempt to shift, they usually experience many hidden costs, Gentry warned. These often pop up around data egress. Or, new data storage costs may take a surprising toll, Gentry added. “A lot of companies look at it from a computing perspective and they don’t account for the storage footprint.”

DevOps in the cloud also introduces new responsibilities and processes that may differ between clouds, continued Gentry. From unique storage operations to managing the life cycle of a virtual machine, each cloud carries unique configuration methods. This makes it more difficult to abandon established procedures in favor of a different computing arrangement.

Nevertheless, Gentry recognizes that we’re starting to see a maturing effect of the multi-cloud concept, especially among large enterprises. “Companies have a multi-cloud strategy—it’s less about moving between clouds than about picking a cloud that has specific advantages for a given workload,” he said. In a fit-for-purpose cloud approach, different workloads correspond to different layers of the business. For example, an organization may require a local data center or private cloud for high-security data. Certain applications may benefit from a distributed global public cloud, whereas other workloads could reap performance benefits and cost savings from a cloud-adjacent edge model. “These specific areas may not have anything to do with one another other than supporting the overall business,” explained Gentry.

Matching Cloud Types to Business Units

So how do we match the workload at hand to the ideal computing arrangement? Gentry identified three emerging cloud types:
- Private cloud: A private core for particular functions that can’t be replatformed or rearchitected. Business areas such as financial records or patient care hold sensitive data that must be stored properly to meet regulations and SLAs.
- Public cloud: End customer-facing, where sensitive data is not an issue. Public distributed cloud is ideal for delivering a highly available front end for customer-facing digital experiences.
- Cloud adjacent/edge: When an environment is kept in a cloud adjacent facility with relatively local geographies. This may sit alongside the public cloud or connect to a front end in a cloud.
For example, Gentry described how a large health care company uses a hybrid multi-cloud setup to support various business divisions. The organization is keeping traditional patient care workloads exclusively on-premises; due to regulatory reasons, it will never see the inside of a cloud. Yet, they are simultaneously building the next generation of applications in a public cloud, using Google Cloud to scale a clinical trial division. Simultaneously they began using Microsoft Azure for their back office. This scenario highlights one peculiar advantage of late-stage cloud adoption, noted Gentry: “They had the advantage of being able to take a more thoughtful approach.”

Gentry described another case of an organization performing genomic sequencing. For data-heavy applications like this, where there is a high dependency on input and output, syncing with a public cloud could be extremely costly and introduce latency. For them, the most cost-effective method was to use a cloud-adjacent or edge model alongside a public cloud.

How to Leverage Fit-For-Purpose Cloud

One situation that may warrant cloud migration is a complete replatforming, noted Gentry. He described one early cloud adopter that rushed into AWS, only to find it wasn’t cost-effective for their needs. Architects then had to make tough decisions to determine what would be repatriated back to local data centers and what should remain in the cloud. “Their only motive going from cloud to cloud was, ‘We made a bad choice and we want to move.’”

Clearly, to match cloud use with business requirements, you must choose the appropriate cloud. AWS is widely adopted for elastic computing and offers a wide product catalog. GCP is offering feature innovation and cost reduction to compete and is going after un- and under-penetrated regions. Azure, on the other hand, will offer licensing advantages if you are a Microsoft shop.

Secondly, to unite IT around a hybrid multi-cloud strategy, you must remove the adversarial state between traditional ops and cloud ops. Classic ops is typically very safety-oriented, using a slow approach with endless tests in pre-production. Traditional ops may produce yearly updates. This is directly opposed to the new cloud ops approach, which emphasizes agility, speed and constant deployment. “They sit in different silos,” Gentry explained.

To solve this dilemma, companies must share technical skillsets between the three main cloud types. “DevOps must become a continuum for all types,” Gentry said. In addition to cultural change, DevOps tools must evolve to monitor workloads across a hybrid multi-cloud estate. This will require integration to tie together tools and deeper observability.
August 2, 2021
Higher Mobile App Stability Begets Higher User Ratings

In the competitive mobile market, application stability is imperative. Downtime leads to poor user experiences, which stunts growth and revenue. Apps that have poor stability suffer the consequences. In fact, new studies show that just a 1% lower stability ranking can lead to nearly a one-star drop in App Store ratings.

Bugsnag recently released its 2021 Application Stability Index (ASI) report, Characteristics of Leading Mobile Apps. The study took 260 top-performing mobile apps across industries and reviewed each’s stability and end-user satisfaction. The study found that apps with solid stability tend to rank much higher in the App Store and maintain increased user engagement. The report also demonstrated that developers are pivoting to weekly releases to quickly address bugs, progressing from the bi-weekly standard for software updates.

Below, I’ll review the specific details of the ASI and the report and consider how stability varies throughout industries. Hopefully, the results here will help other software teams benchmark their app’s performance and release cadence against top-performers within their sector.

Stability = User Satisfaction

When measuring uptime and availability, five nines, or 99.999% uptime, is the golden target for most teams. This report measured stability as the percentage of apps sessions that were crash-free. The median across market-leading apps was 99.80%.

What’s interesting is that apps with more nines in their stability score achieve markedly higher App Store ratings. Apps that scored less than 99.00% had a median rating of 4.15, whereas apps with 99.99% to 100% measured uptime had a median rating of 4.70. The distribution of ratings was extremely wide for low-performing apps, compared to higher average scores for higher stability.

What do these results say? Users expect uninterrupted sessions. Frequently occurring stability issues can damage the consumer’s opinion of an app, leading to dissatisfaction and harsh reviews. The more stable an app is, the better chance it has to drive consumer success.

Stability by Industry

The study also exposed stability differences between industries. B2B SaaS apps ranked relatively high, at 99.85%. This is likely because B2B software must meet service-level agreements (SLAs). Such services have fewer (yet more high-paying) customers to support, the report speculated. Mobile apps directly tied to the exchange of money, such as e-commerce (99.86%), travel and hospitality (99.90%) and financial and banking (99.85%) also ranked consistently high in stability.

Coming in second were mobile apps in the business-to-consumer (B2C) (99.82%), consumer goods (99.80%) and health and wellness (99.78%) categories. Though these areas require decent availability to remain viable, they may not need such stringent SLAs. In such scenarios, non-stop availability isn’t as directly tied to end revenue.

Lastly, apps with the worst stability were found in the mobile games (99.71%) and media and entertainment (99.65%) categories. Mobile gaming had the worst ratings of all industries; mean stability equated to over a full day of outage per year. Increased downtime in both categories is perhaps due to more problems with frequent media updates. These areas also tend to have a lower average revenue per user (ARPU), which may affect how teams value ongoing support.

Deployment Frequencies

According to the report, highly stable mobile apps tend to release at least once per week, replacing the bi-weekly norm for updates. Nearly all industries are pushing at least four releases per month, aside from consumer goods and travel and hospitality, at 2.5 and 4 respectively. E-commerce takes the cake with a mean of seven releases per month.

Nowadays, progressive delivery is made possible due to increased emphasis on CI/CD. Therefore, rapid deployment frequencies are the norm. Releasing new versions more quickly can help appease users with new features. It’s also helpful to quickly patch bugs and decrease mean time to recovery (MTTR) for major incidents.

Users Love Stable Apps

Throughout the pandemic, digital reliance soared—84% of consumers said applications were a lifeline to normality, found a recent AppDynamics report. In this climate, mobile applications are critical to delivering valuable interactions.

Yet, user expectations have evolved. Now, it’s readily apparent that stability directly impacts consumer confidence. The ASI findings behoove engineers to edge closer to the five-nines goal. The more nines you have, the better off you are.

“The results of our second ASI report highlight the need for better proactive error monitoring and stability management to help engineering teams build healthier apps that deliver superior customer experiences,” said James Smith, SVP of the Bugsnag Product Group at SmartBear.

It’s good to reiterate that this study examined a handful of leading apps from each industry. So, these scores shouldn’t be construed as inclusive of the entire app marketplace, only as benchmarks for the top mobile app performers.

July 23, 2021
Could Full-Stack Observability Help Lessen IT’s Burden?

The year 2020 was a landmark year for many reasons, other than the obvious. The IT world saw the first time that cloud service revenues dwarfed spending on data centers. Across the board, budgets have been increasing around cloud adoption. And in this ‘new normal,’ digital experiences are slated to remain just as crucial of a touchpoint as ever. In fact, new findings show that for 85% of users, digital applications are absolutely essential to their daily lives.

Rising digital demands mean reliance on the cloud is here to stay. But if organizations don’t right-size their cloud workloads, expenses could get out of control. Plus, maintaining world-class user experiences requires a keen insight into the overall performance of apps, systems and solutions. If teams aren’t continually monitoring application performance and business KPIs, the bottom line could suffer.

I recently met with Joe Byrne, AppDynamics Regional CTO, to discover what full-stack observability (FSO) means for today’s cloud-heavy applications. According to Byrne, DevOps and SRE teams should be maintaining a holistic view of the entire pulse of the application—including non-functional metrics (NFM) alongside traditional security and functional monitoring. Doing so could reduce time spent on root cause analysis and help ensure code is continually meeting business objectives.

What is Full-Stack Observability (FSO)?

In the classic definition, observability focuses on measuring the internal status of a system based on external outputs. Application performance monitoring (APM) typically tracks telemetry around logs, metrics and dependencies. IBM, for example, calls the observability trend “a natural evolution of APM data collection methods.”

Full-stack observability (FSO), said Byrne, takes this one step further to track business-related key performance indicators (KPIs) alongside additional, more traditional software metrics. “FSO helps understand that software is performing optimally and that it is meeting business objectives,” said Byrne. In this way, FSO takes a holistic view to determine not only if an application is functionally sound but if it’s meeting its end goals.

FSO would consider critical performance actions, such as user experience telemetry. For example, Byrne explains, an e-commerce website could find value in tracking performance levels such as login time, check out time and payment initiation. Tracking such metrics is necessary to pinpoint roadblocks and create actionable steps to remediate holes in the user journey, said Byrne. Increased visibility here could also help engineers quickly respond to issues to better optimize the user experience. In addition, aggregating information at a higher level—to view total products sold per day or total revenue per day, for instance—could inform overall progress.

How FSO Could Reduce IT Stress

“We’ve reached a point where people have moved away from brick and mortar and are going online,” said Byrne. These online spaces are more omnipresent — and valuable — than ever. The AppDynamics 2021 App Attention Index found that the average digital consumer uses 42 applications daily. Furthermore, 85% of people now say digital services are a critical part of how they get by in life.

Mounting digital reliance has resulted in rising digital complexity. Therefore, said Byrne, companies have been moving to the cloud for the ease of scalability, elasticity and the capability of cloud functions. Cloud-native abilities like serverless, Kubernetes, identity and access management (IAM) and other add-on services are helpful perks to avoid reinventing the wheel.

But of course, lifting and shifting all your current architecture to the cloud is costly; plus, it won’t necessarily eliminate application bugs. The user journey is increasingly vast, and broken touchpoints along the way could hinder performance. This is complicated by increasing third-party dependencies. Yet, more often than not, it is the application’s custom code that is having the issue, explained Byrne.

Perhaps the most significant benefit of full-stack observability is a reduction of remediation time. “If you don’t have perspective, it can take a long time to run down an issue,” said Byrne. “The cost of finding a root cause can be so expensive.” Having a quick window into network, application and infrastructure hiccups is necessary to quickly address issues and save potentially lost revenue.

Shifting Performance Monitoring Left

An Amazon study found that every 100ms of latency costs companies a 1% loss in sales. Knowing the reality of high user expectations, software companies must be especially proactive in how they scale to meet the needs of consumers. As 72% of users believe application problems are the brand’s responsibility, the onus is on companies to avoid negative brand reputation.

To spot performance hangups early on, Byrne encourages a shift-left attitude toward observability around business outcomes. To date, the shift-left mantra has focused more on code validation and security—necessary elements to rapidly deploy software updates. Now, adding performance testing could be the next iteration to further correlate each software push with end business outcomes.

“You want to see and validate as much as possible before production,” said Byrne. “Performance testing is a key piece in the overall success of an application.”

This could involve testing the load time for a specific endpoint—if a key action doesn’t meet a certain threshold, then it would not be promoted. Incorporating performance monitoring within a CI/CD process is a burgeoning practice also referred to as BizDevSecOps.

There’s No Going Back

The new normal is upon us, and it’s hard to imagine going back to any semblance of a world where there’s a decreased reliance on technology. From airline bookings to banking and grocery orders—software applications are now intrinsic to nearly every facet of modern life. To compete in such an economy, finely-tuned modular customer experiences that continually optimize business outcomes will truly set companies apart from the rest.

In short, to maintain high software standards, it’s essential to break down operational silos and work together as a team—the DevOps creed. End-to-end ecosystem visibility could help enable that, ensuring, in Byrne’s words, that “applications perform in a way that meets the needs of the business.”

July 22, 2021
Big Gains Are Made In The ‘Last Mile’ of DevOps

A new study by mabl found a correlation between complete, fully automated test coverage and a significant increase in customer happiness and DevOps maturity. In contrast, those with moderate DevOps maturity and nearly automated testing coverage tend to experience a sharp decline in end-user satisfaction.

Mabl recently released its 2021 State of Testing in DevOps Report, a study that highlights the state of DevOps adoption and quality testing throughout the software industry. According to Dan Belcher, co-founder, mabl, the study results clearly show that true business returns are made in the last mile of DevOps adoption.

I recently met with Belcher to hear his viewpoint on the major takeaways. Below, we’ll explore these and other insights from the 2021 State of Testing in DevOps Report.

State of DevOps Maturity

Nearly one in four teams have a mostly mature DevOps methodology and automated process. Still, a significant portion, 52%, are still aspiring or striving to fully embrace a mature DevOps model. A small percentage, 11%, have a fully automated DevOps software development pipeline in place. The bulk of teams are still using manual deployment processes and disparate frameworks, limiting the potential for full automation.

Interestingly, non-technical barriers remain the most common inhibitor to realizing a completely automated development pipeline, found the report. Out of these obstacles, reluctance to change was the most common barrier, cited by 29% of respondents. Other challenges to DevOps adoption include budgetary constraints, DevOps not being a company priority and internal politics.

When added up, non-technical roadblocks account for 82% of the problems that hinder complete DevOps adoption. On the one hand, this depicts the cultural challenges associated with introducing DevOps in an organization. On the other, it could signify the wealth of DevOps packages and open source tooling available to manage areas like automated deployment and version control. The report found GitHub, Jenkins, Gitlab, Azure Pipelines and BitBucket among the five most commonly used DevOps Tools.

Although it’s growing rapidly, continuous delivery is still an emerging trend; almost half of developers still deploy code less than once a month. Only 15% of teams are deploying code daily or multiple times per day. This echoes findings from CDF’s State of Continuous Delivery Report, which pegged this figure at 11%.

Fully Automated Testing

Testing is a critical component of the software life cycle. With the need to ensure functional applications and high-grade security, automated testing is helping teams discover more errors earlier in the development process. According to the report, the most popular test types include UI or functional tests, unit tests, regression tests, integration tests and end-to-end tests. Non-performance tests and visual testing ranked lower. Naturally, the teams that have implemented fully automated pipelines are automating more types of these tests.

In general, it appears test automation aligns with DevOps maturity. The report found that those with fully automated testing experienced significant net customer experience gains. For example, 72% of teams that were automating regression testing indicated their customer happiness is pretty good or amazing, whereas this figure sank to 6% at companies not automating regression testing. And 80% of teams with high test coverage report high customer happiness, whereas only 30% of teams with low test coverage report high customer happiness. Nonetheless, much has yet to be fully automated—similar to the state of continuous deployment, the bulk of teams rely on a combination of automatic and manual testing approaches.

In terms of tooling preferences, Selenium ranked as the most popular option for functional testing, with 41% of developers using it. The report also found migration toward Cypress, Smartbear and other functional testing tools. Interestingly, the report also uncovered a noticeable lack of adoption when it came to automated API testing—only about one out of three are automating their API tests. Postman, SoapUI, internal tools and ReadyAPI are among the most popular tools for running manual API tests.

As I’ve covered before, APIs are surging in use across digital transformation initiatives. Web APIs are an increasingly common mechanism for delivering data to partners and consuming third-party software-as-a-service (SaaS). Full API testing coverage will be essential to audit the entire end-to-end digital experience. Low-code API testing is one way to reduce barriers to auditing the highly technical area of integration.

Key Takeaways

DevOps success is a ‘last mile’ problem. There is much variance in DevOps maturity and testing coverage. It’s pretty flat until you reach that last point, when breakthrough results occur. “Success in DevOps is a ‘last mile’ problem,” said Belcher. “The difference between 90% and 100% of the way there is dramatic.”

Quality really matters. Though much tech discussion focuses on increasing delivery speed, code quality is just as important. “This isn’t about speed at all costs, it’s about innovation with velocity and quality,” said Belcher. “Quality engineers are just about as key to success as developers writing code.”

DevOps success is less about technology and more about people. As the report found, the majority of the roadblocks to DevOps maturity are non-technical. “The things holding us back, as an industry, are more about leadership and figuring out our processing than introducing more technology,” said Belcher.

Breaking down role silos is still fundamental to DevOps. Some see DevOps as instrumental to opening collaboration between roles. Simultaneously, there is the idea that new tooling dashboards will arm individual developers with complete ownership to develop, test and deploy software (known as full cycle development).

Belcher believes that the full cycle momentum has been misinterpreted to mean developers alone have full accountability. Instead, he frames developers as a pitcher on a baseball team; leading the game, but utterly incapable without the rest of the team. Similarly, disparate roles should be working together, said Belcher. Reaching that ‘last mile’ of DevOps is not only about CI/CD or full cycle development; it’s about working together as a group.

I’ve only scratched the surface of many findings from The State of Testing in DevOps. The report queried 600 quality engineers and software development professionals and offers loads of additional insights on the state of DevOps, testing, CI/CD and more. You can view the full results here.

July 7, 2021
The State of Continuous Delivery in 2021

One of the hallmarks of DevOps is to improve the delivery of software. Therefore, mature DevOps systems employ continuous delivery (CD), the process of releasing bite-sized software changes on a regular basis. Amid high demand for digital innovation, streamlining this release pipeline is becoming more and more crucial for enabling overall software fluidity.

So, what is the state of CD among today’s development teams? The Continuous Delivery Foundation (CDF) recently released its 2021 State of Continuous Delivery Report, a study that helps establish baseline continuous delivery metrics. Below, we’ll review some key takeaways from the report and discover mean rates for things like deployment frequencies, recovery times and average lead time for code changes.

Benefits of Continuous Delivery (CD)

Before we look at the broad metrics surrounding CD adoption, let’s consider: what are the benefits of CD? A big net benefit of CD, the report noted, is that it helps realize feedback earlier in the software development cycle. A lean, CD-driven development approach could help quickly discover features that won’t succeed, which avoids wasting precious development resources.

CD also increases overall delivery speed for small software chunks. In an era where microservices architecture is all the rage, this process is a necessity for continually initiating many disparate updates. For example, Netflix utilizes Spinnaker to deliver hundreds of microservices and thousands of daily deployments.

With these results, organizations can reap increased responsiveness and agility. CD can enable fast feedback to changes, which can help businesses adapt to unforeseen conditions, such as zero-day exploits. An investment in CD can also bring net positives to developer team dynamics, according to the report.

Continuous Delivery Statistics

Software performance can be indicative of overall organizational performance. So, how should an organization measure its software delivery success? Well, the book Accelerate outlines four crucial metrics to watch: lead time for changes, deployment frequency, time to restore service and change failure rate. The CDF report establishes current industry averages for three of them.

In terms of deployment frequency, the study found that the highest percentage of developers (31.3%) release once per week to once per month. A large percentage of developers, 27.3%, release every month to six months. Only 10.8% of developers are elite performers, releasing multiple times per day. This indicates that rapid release cycles are still maturing across organizations.

The average lead time for changes is more variable than you might think. It takes 28.3% of developers one week to one month to go from committed code to running code successfully in production. For 27.3% of developers, this lead time is increased, from one month to six months. Only 5.74% of developers say it takes less than an hour, indicating that a truly on-demand delivery mechanism remains out of reach at most organizations.

On the other hand, developers are much quicker to act when it comes to restoring services. The survey showed 34.4% of respondents said it takes them one hour to one day to respond to an unplanned outage. A significant number, 15%, respond to incidents in less than one hour. Still, about 50% of outliers are responding somewhere between one day and more than six months.

Decreasing the mean time to recovery (MTTR) is imperative to respond to broken applications, quickly address security vulnerabilities, and meet evolving customer expectations. As a result, MTTR has become a de facto metric for Site Reliability Engineers (SREs) to watch closely.

Factors That Affect Software Delivery

Other factors may influence software delivery agility, such as industry type and programming language preferences. Out of the many industries attempting to refine their delivery models, the report highlighted retail as an industry that ranks high in both speed and stability metrics. Other sectors, such as financial services, government and energy ranked high on stability yet received only average scores for speed. Industries like telecommunications received poorer overall scores.

Programming language choices may also dictate overall software delivery performance. The study found that shell scripting languages, Go, JavaScript, PHP and Scala were among the top five programming languages, based on the combined software delivery performance data for organizations using each language.

DevOps Aims to Improve CD

Continuous delivery is vital to responding quickly to changes and enabling iterative, progressive delivery of new features.

Another facet of modern CD that can’t be understated is the widespread prevalence and reliance on open source tools. CDF open source packages like Jenkins, Jenkins X, Spinnaker and Ortelius remain vital to enabling CD processes for numerous developer teams.

The 2021 State of Continuous Delivery Report provides a baseline of standard continuous delivery practices across companies and sectors. The findings are based on SlashData’s Developer Nation survey, which reached 19,000+ respondents from 155 countries from late 2020 to early 2021. One area where reporting is lacking is change failure rates, which we hope future studies will benchmark. For more insights, you can download the report here.

July 2, 2021
Report: The State of Cloud-Native Application Security

The cloud brings tremendous capabilities in terms of increased deployment fluidity and automation. Along with cloud adoption has come the use of cloud-native tools built specifically for developing applications for this domain. However, cloud-native tools carry some nuanced security concerns, such as misconfigurations, known vulnerabilities and leaked secrets. As such, 83% of organizations recognize security as very important to their cloud-native strategy, according to a recent report from Snyk.

Snyk’s State of Cloud Native Application Security report surveyed hundreds of IT professionals on their cloud-native security concerns. Below, we’ll pick out the top takeaways from the report, identify the most common trouble areas for cloud-native application security and see if there’s a correlation between deployment automation and higher degrees of application security.

The Move to Cloud-Native

Teams are moving to cloud-native tech, empowering DevOps with software-driven architecture and infrastructure-as-code (IaC). In this new cloud-native paradigm, 58% of production workloads are deployed as containers, 21% now are serverless and 50% use some form of IaC within the deployment process.

Organizations are moving to cloud-native technology for a variety of reasons. For one, containerized software affords increased speed — 68% of organizations moved to containers for increased deployment velocity, the study found. This is followed by ease of management (at 67%) and reduced cost (43%). There is also the security imperative for adopting cloud-native tools; 36% of respondents cited security as a primary reason for moving production applications to containers.

Top Cloud-Native Security Concerns

While cloud-native technologies such as containers, Kubernetes, serverless and IaC enable more rapid development, they also bring unique security issues. The report found that misconfigurations and known unpatched vulnerabilities are by far the most common type of incidents among cloud-native environments. In fact, 45% of organizations have experienced an incident arising from a misconfiguration, followed by 38% due to known unpatched vulnerabilities.

Other common cloud-native security incidents include secrets leaks, failed audits and malware. Interestingly, the study also found that data leaks by insiders are more common within organizations with higher degrees of cloud-native adoption. According to the report, 38% of organizations with high cloud adoption have suffered data leaks by an insider, whereas the figure is cut in half, at 17%, within organizations with low cloud adoption. Secrets like API keys must be managed carefully, especially as cloud-native tools force more dependencies. “The need for effective management of these kinds of artifacts is a key differentiator from the more centralized pre-cloud era,” the report noted.

Automating Security Testing

End-to-end deployment automation shows promise, yet is still at an early stage of maturity for most development teams. Though 95% of organizations use some kind of deployment automation, only about one in three organizations have an entirely automated deployment pipeline.

Organizations with increased deployment automation tend to embrace a higher degree of security testing, as well. Highly automated pipelines are twice as likely to use security testing throughout their development lifecycle, found the report. Since incidents most commonly arise from misconfigurations and known vulnerabilities, automated scanning could help identify many cloud-native issues, comparing production code against databases of known exploits.

So, when does this security testing occur? More often than not, it happens within the CI/CD pipeline. Over 60% of organizations perform security testing within the CI system. This is opposed to testing source code repositories or local IDEs and CLI tools early on in the development process. In terms of testing frequency, there is a wide range of executions. For those groups with high deployment automation, 70% test security daily or more frequently. Automated security testing seems to be functioning well, as 72% of fully automated teams find and fix critical vulnerabilities in under one week.

The report also exposed an interesting disconnect with regard to security testing ownership. Only a small percentage of security engineers (less than 10%) feel developers are responsible for the security of the cloud-native environment and applications, whereas 36% of developers stated they are responsible for security. These figures may demonstrate the security onus shifting left toward the development side. Or it could underline changing attitudes as full-cycle development becomes more realistic. Regardless, teams will have to find clarity on roles to avoid some difficult conversations!

Cloud-Native Security

With the shift toward cloud-native strategies, security standards are rising to respond to new issues like misconfigurations. To address mounting application threats, increased automation could help shift security left—this and other reports clearly demonstrate a correlation between fully automated deployment pipelines and an increased security testing regimen.

As a result of cloud-native adoption, 58% of organizations have increased their security concerns since adopting cloud-native tools. Part of this involves moving to a zero-trust approach across all infrastructure, whether internal or external facing. The report revealed that 58% of organizations are increasingly concerned with misconfiguration, 52% are increasingly concerned with insecure APIs and 43% are increasingly concerned about known unpatched vulnerabilities, while 41% are concerned about secrets leaks. Hopefully, these benchmarks will help you see how your organization stacks up in comparison to others.

Snyk’s State of Cloud Native Application Security report questioned 600 experts in development, security and operations on cloud-native adoption and security practices. For more insights, view the study in its entirety here.

June 22, 2021