devopsy.kinsta.cloud

Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

  • The Benefits of Crowdsourced Testing

    The Benefits of Crowdsourced Testing

    Charlene O’Hanlon and Rob Mason from Applause talk about how crowdsourced testing is a highly effective method for testing applications and its benefits to DevOps teams. The video is below followed by a transcript of the conversation.

    O’Hanlon: Hey, everybody. Welcome back to TechStrong TV. I’m O’Hanlon, and I’m here now with Mason with Applause. Rob is a return to TechStrong TV. Rob, welcome back. So good to see you again.

    Mason: Thank you, and thanks for having me on again.

    O’Hanlon: All right, so we had a great conversation, what was it, a couple of months ago?

    Mason: Yeah, a little while ago.

    O’Hanlon: I think so. Yeah, yeah, so time flies when you’re having fun, I guess. But, at that point, I did invite you to come back on so we could talk about the topic of continuous testing, which I think is something that is gaining a lot of traction in the DevOps space, and it’s certainly garnering a lot of conversation. Sorry, not continuous testing, crowdsource testing. Oh my goodness, my brain this afternoon, what am I going to do? Sorry about that. Yes, crowdsource testing, and it, too, is something that a lot of DevOps teams or a lot of organizations are considering as an alternative, I guess, or maybe a complement to the way that they’re doing testing these days. You are the expert in the field, so I want to talk to you about it and kind of get a download, if you will, on crowdsource testing. What you’re seeing in the space right now, who’s talking about it and what organizations, how they could benefit from it?

    Mason: Right. Thank you. Yeah, so Applause is the largest crowdsource testing company in the world. _____ [inaudible due to audio distortion] quality testing for some of the largest companies in the world, like Google, PayPal, Walmart, Amazon, Facebook, a lot of those different types of companies, and we do that digital quality testing by using a crowd. Applause actually started off as uTest about a decade ago, _____ the name uTest, so anyone could test. Basically, if you go to uTest.com, you sign up to become a tester. Today, that community is over 750,000 people from 233 different countries around the world, and they’re making money from testing products from the company, or the companies I mentioned before, as independent contractors. They bring their own computers, they decide what they work on, what they don’t work on, they bring their gear, they decide when they want to work.

    You start off by going to uTest.com, you sign up, and then we put you through all sorts of training. We have what we call uTest Academy, lots of great videos, training courses. It’s all free. You don’t have to pay anything for it. Joining the community is free. Once you join the community, you start to level up your skills as a tester, and you become better and better at writing great bug reports, with _____ the attachments, the steps to reproduce them, learn now to execute test cases, how to do payment testing or accessibility testing. We train you on all sorts of things around testing because we know there are not enough tests going on. We know there is not enough quality in our products today. It could always be better, and so ‒

    O’Hanlon: Yeah ‒

    Mason: Go ahead.

    O’Hanlon: No, no, you go ahead.

    Mason: I mean, with how fast the technology landscape is changing, you guys see it firsthand every day, it’s very easy to switch from one bank to another or from one app to another, and so it’s very important that that first experience is first class. More and more big brands are very much aware and very in tune with those first experiences and the ongoing experience of their users, so we use a very large community of real people in the real world, around the world, every country you can imagine. You know, 233 countries, as I mentioned. They level up over time to become more and more expertise. As they progress in their learning and participation, they get invited into more projects, so they get opened up to more ability to earn money, basically. When they tell us ‒ voluntarily, of course ‒ about themselves more, what devices they have, do they have an iOS device, android device, what kind of laptop they have, they get invited to more projects.

    The more they tell us, the more that we can match them to projects. We use machine learning algorithms to match testers to projects based on past experience, so we’ve got a data science team, and machine learning algorithms that match based on performance. It’s not based on demographics or race or ethnicity, it’s all based on what you can do, and what you can do is behind a screen. The machines don’t know. It’s based on did you turn up on time for the project, did you execute ‒

    O’Hanlon: Yeah.

    Mason: ‒ _____ well, were your bugs well written? It’s all based on pure scoring, which is great. Some of our projects do require physical presence, it will be like a test for Starbucks, and you have to go in and buy a coffee and prove that you can order it online and pick it up, and all that sort of stuff. But, most of them are remote, and you can work wherever you are, from your couch with your dog sitting on your lap, and still execute great testing for the customers. That’s what the community is really a lot like. Some of the projects do have demographic requirements, but mostly those are in the medical space or in fitness. For instance, we’ll be looking for a specific age class for testing online fitness classes for some of our customers, and so sometimes there are demographic requirements. But, for the most part, it’s pretty wide open for anyone to participate. What’s nice is in some of these communities, or sorry, in some of these countries, it’s hard to find jobs. Especially during COVID, we had about 50,000 coming in per month during COVID.

    O’Hanlon: Wow.

    Mason: That’s trailing off now with less people coming in, but with all the people that were out of work, you can make money at Applause. Some people do it as a moonlighting job on the side, just to earn some extra money, maybe some play money. Some people make a living out of it. As an independent contractor, again, they make their own time and everything, but they’re making a living off of Applause doing testing for these customers full time. Again, anyone can do it, so if you’re in a country where maybe women can’t get jobs because it’s old-school and it’s sexist there in that country or what have you, or you’re in a demographic that it’s harder to find work, uTest doesn’t care, like bring your skills, bring your abilities, and anyone can get work. If you look at our community, we’re incredibly diverse across, again, all those countries, but also in our global community, we have a high ratio of women in the community.

    Not everyone, obviously, tells us what gender they are, but 35 percent of our global community are female. We have top-rated testers in the community, 41 percent of our gold-rated testers, that’s the highest level tester we have, 41 percent of those are women, 40 percent of our silver testers are women, 40 percent of our bronze testers are women, so you can just see that across the board it’s, again, based on performance. It has nothing to do with gender or anything else, which just levels the playing field for a global economy, and for global testing of enterprise products. I’m sorry. I’ve been talking ‒

    O’Hanlon: No, that’s awesome. I love to get the download, so it’s like you kind of take it from there and just run with it. Wow. That’s really cool stuff. I think that the last time we spoke, you guys were working on a couple really cool projects, but it seems to me that this would be a great technology, or a great service, if you will, for organizations that really don’t have maybe an in-house testing environment, or maybe organizations that do have an in-house testing environment but really kind of want to go that extra mile and make sure really that all their bases are being covered. How much of your customer base, then, are those companies that maybe already have testing abilities in house but they’re just looking to kind of up their game, if you know what I mean?

    Mason: I would say probably almost all of them have QA ‒

    O’Hanlon: Really?

    Mason: ‒ of some sort, but the difference is QA in a lab with a relatively small team is very different than real world QA. If you have some simulators or some real devices in the lab, some browsers based in your lab, it’s very different than browsers around the world. Probably a lot of the enterprise companies have their sites localized into different languages. Probably in your lab in your local team, you may not have all those native language speakers, so that’s _____, so localization is one. Currency testing, payment instrument testing, all those things that happen in the real world, you know, poor networks, flaky networks, different flavors of Samsung devices or android pixels, all that sort of stuff. A lab test is great, and it’s a great way to make sure the base foundations of your applications are good on your mobile apps, but it’s very different when you get out in the real world, and that’s where the customers judge you.

    They don’t judge you for your lab experiences, they judge you when you get out in the real world, that’s important. Almost all of our enterprise customers have their own QA teams, but they admit, and these are big companies ‒ it’s Google, it’s Amazon, it’s Facebook ‒ they admit that they don’t have a big enough team, even at their size, to cover what they really need to do to test their applications. You need a company that can bring resources from around the world, speaking all the different languages, all the different currencies, all the different types of devices. I think the last time we counted, there were about four million different devices in the hands of this community, because everyone ‒

    O’Hanlon: Wow.

    Mason: ‒ has got a phone _____ [inaudible due to audio distortion], what have you, so you take all those devices and you bring them to bear on some of these huge problems of the large enterprise, and it changes the dynamics. It’s daunting if you want to build a product and launch it globally and have it be successful, and not know that it’s going to work the first time. Even if you get a great, large outsourcing firm based in India or over in Europe, they’re based in one area, in one region. They all speak probably the same language, a very similar language, they’re in the same area, the same currency. How do you get that global test? The answer is crowdsource testing. There is no other way to do it. It’s just not practical any other way. If you have to manage those masses of people, like how do you make sure that it’s not just total chaos, and that’s what Applause has really specialized in for more than a decade, is making sure that we give great quality in what we give back to our customers, it’s actionable, it’s precise. Every bug report from us has to have an attachment and a video, it has to have reproduction steps.

    Again, we don’t just do bugs for customers, we do all sorts of testing, like we test smart phones, we test smart TVs, smart cars, IoT, AR/VR. We do machine language, data collection, data collection from machine learning ‒ I said machine language, but machine learning ‒ so all sorts of things we do for our customers using this global community. When you think about it, I have 750,000 people I can bring to bear on your problems to improve the digital quality of your applications. Sometimes you need more data, sometimes you need _____ [inaudible due to audio distortion], sometimes you just need a lot of people to bang on a website, sometimes you need people to go deep and look for those hard-to-find bugs. It varies greatly, so we cover pretty much every area that you could expect of digital quality testing with that crowd, and we _____ [inaudible due to audio distortion] so you don’t have to think about it. That just allows us to scale, but what we’re really proud of is just how successful our community has been, how diverse they are, how broad and how deep they get.

    They are true experts, and a lot of these people in the community, some of them are just people moonlighting after working some other kind of job, but some are true QA experts that have been in the industry for decades and have just chosen this new way of life, because it’s flexible. You always work at home, or wherever _____ [inaudible due to audio distortion]. You can be on the beach and doing the job. It doesn’t matter to us, as long as the job gets done. It is the new world, and as COVID removed some jobs for some people, at least temporarily, we got an influx, but also people are now looking for that new way of working. We see the industry moving out of the offices, and that’s what we’ve been at Applause for a decade. Our community, the people doing the testing for us, have been outside of offices since we started, and so it is the natural way for us. It’s like welcome to our party, but now you have to figure out how to work ‒

    O’Hanlon: Right.

    Mason: ‒ _____ [inaudible due to audio distortion], and we’ve been doing that _____ for a long time.

    O’Hanlon: That’s very cool. On average, how many testers do you have on a particular project?

    Mason: It varies greatly, like sometimes we’re doing a global payment instrument test that’s making sure something like an android pay or an Apple pay works across the globe when they’re rolling out a new feature. There, we have to engage a lot of people to do that, to make sure that all the different banks, all the different currencies work around the world. Sometimes, it’s a point release, a patch release from a customer with a new version of their mobile app, they made some targeted changes, they’ve told us what they are, some regression testing around the area, a specific testing in that area, so it’s a smaller team, so anywhere from a handful of people to thousands of people at times. We don’t generally go much larger than thousands on a single project unless there is a very specific need. We do have some that spike up there in very large numbers, but, for the most part, we’re in the tens to hundreds range, less so in the thousands on a given, specific, what we call test cycle.

    For most of our customers, in fact all of our customers, it is subscription-based at this point, so they’ll sign up and have us test a digital asset over a long period of time, and so those assets go through their cycles. The first release is different from a patch release versus a new, major update, a new, major feature. Again, the size of those teams vary over time, and we can pull as many as we need into it. If there are different skill sets, like maybe a customer has a mobile app and now they’re focused on accessibility, and so they’re going to do a release 100 percent focused on making their application more accessible, we’ll bring in an accessibility group from our community and have them focus on accessibility. These are people not only that know accessibility testing well, but they’re people with real accessibility limitations, so it’s real people with real disabilities testing your application to make sure it works in the real world, because there is nothing more frustrating than trying to use an application that claims it supports you and then finding out it doesn’t.

    O’Hanlon: Right, right. Well, it sounds like great stuff, and you guys have been doing this for ten years or so?

    Mason: Yeah, just over 11, coming on 12 now.

    O’Hanlon: That’s great. How did you guys come up with this idea? I mean, I think it’s really kind of fascinating because it really does address a real problem that is in the industry, and that is being able to test all the things, if you know what I mean. Is this something that your founders just decided one day like, “Well, you know what? The best way to do it is let’s just ask everybody to do it, and everybody to help us out.”

    Mason: Yeah, our founders were engineering managers from the past, they were engineers, and they had the experience of building and testing applications in their own labs, and then going out in the real world and finding out that things were different when they got out to the real world. They said, “We have to change the model. The model isn’t going to work, and it’s only getting worse over time.” They said, “We have to go and turn this model upside down,” and they were far ahead of their time when they launched the company. Now there are some other much smaller companies out there doing some similar things, but they’re much more niche. But, they saw what was coming. The real world matters, and the differences are just ballooning and you just cannot keep up, even as a large company in the lab, so great foresight there. It took a lot of work to get to the point where you can manage that many people and have it be successful.

    Obviously, if you just put random people on a project and say, “We’ll do stuff for you,” you’re going to get terrible results and just waste your time. Obviously, places like Walmart and Facebook and others like that will not tolerate that, so you have to have a very good system and process around it. We have a special enterprise grade platform that our customers and community interface on, so we do software development to make our own platforms to allow those groups to interact with each other, and then we’ve got a great team of community management around them, recruiting people, training them, helping them learn how to be great testers. Ultimately, we’re making careers for people. We’re helping people make livings working for themselves wherever they want to be.

    O’Hanlon: That’s great, and we’re going to need to wrap this up here very soon, but have you heard of any stories of folks that joined the platform to do testing here and there, and ultimately decided that, “Hey, this is what I want to do for a living. This is what I want to do all the time,” and they ended up actually doing a career change because of this?

    Mason: Absolutely. In fact, we’ve hired several of those.

    O’Hanlon: Oh, really?

    Mason: It’s not generally the model, to hire from the community, because the community is our model, but sometimes they rise up so far and they’re such stars in the community, and we’re looking for additional help in managing the community, or elsewhere in our business, that we find them and bring them in. A lot of our great employees we have at the company are people that we found through the community, because they’ve come in and shown what they can do. There are others that have gone off and moved on to full-time positions in companies. Some of them do have full-time QA positions, and then work for us moonlighting. Some of them, like my daughter is in the community, anyone can be in the community. My daughter is in the community, and she makes a few bucks on the side and buys herself snacks or whatever _____.

    So, it really is a very diverse community of everyone who is just getting started or learning about testing, giving an appreciation for what it takes to make digital quality products, and what companies go through to test them, so I think it’s a great experience for everyone. I encourage every single person to sign up at uTest.com. Again, it’s free. We pay you, not the opposite, so you can make money. All you have to do is experience some digital quality products, give us good feedback, and help us make these products better for the planet, basically.

    O’Hanlon: That’s awesome. Well, Rob, thank you so much for walking me through this. I love the idea of pretty much crowdsourcing anything, because I truly believe in the collective intelligence of the populations. Anything that we can do to kind of encourage that and really take advantage of that at the end of the day, really capitalize on that collective intelligence and the ability of crowds to really do some good, I think is good for everybody. Thank you, again, and it was great having you on TechStrong TV again, and hopefully we’ll see you again in the future.

    Mason: My pleasure. It was great talking to you.

    O’Hanlon: All right, everybody, please stick around. We’ve got lots more TechStrong TV coming up, so stay tuned.

    [End of Audio]

  • Dependability is Crucial for Real-Time Experiences

    Dependability is Crucial for Real-Time Experiences

    Millions of people around the world experienced a major disruption after an October 2021 outage saw Facebook, WhatsApp and Instagram go offline. A few days prior, the popular workplace collaboration platform, Slack, also went down for some users. And in the last few weeks of 2021, Amazon’s AWS caused considerable global chaos after significant networking issues in its busiest data center, US East 1, disrupting streaming services such as Disney+ and Netflix.

    Downtime can be costly, causing organizations to lose $5,600 per minute and up to $300,000 per hour, on average, according to Gartner. But it’s not only financial and reputational loss that businesses need to worry about. Outages also seriously downgrade—and in the case of real-time experiences, completely disrupt—the customer experience.

    At a time when consumers are demanding more live and interactive digital experiences, organizations can ill afford disruption or downtime, as customers quickly move to other services that ‘just work’.

    It’s not just the capacity within a real-time system that matters, but also globally distributed failover capabilities when something does go wrong. While organizations could try to build their own failover mechanisms, it’s extremely difficult to failover so services continue seamlessly, regardless of the status of its underlying infrastructure. Simple failover systems not only take longer to resume but typically suffer data loss during the time that a region is disrupted. Businesses need to know that even if, say, AWS suffers an outage, they won’t be affected.

    This is why it’s vital that organizations get it right when looking to develop their own real-time infrastructure. Managing tens of thousands if not hundreds of thousands of concurrent connections while synchronizing data in real-time means that if anything does go wrong it’s immediately obvious to users. If tech behemoths like Amazon and Facebook can’t get it right, what chance do organizations with far fewer resources have when building and managing their own real-time infrastructure?

    The Steps to Success With Real-Time Infrastructure

    Regardless of whether it’s for synchronized multi-user collaboration, audience engagement or instant updates, developers have a hard enough time developing even the most basic of real-time infrastructure, let alone making sure it’s dependable. There are four key elements to achieve dependability: Predictable performance, data integrity, reliability of service and high scalability.

    • Predictable performance: When we think of performance we usually imagine more computing power or speed. But when it comes to real-time, it’s not just about minimizing latency and bandwidth, it’s about having a predictable system. If developers have global latencies and bandwidth within certain operating boundaries, it provides certainty in uncertain operating conditions. This makes it easier to design, build and scale features around this certainty.
    • Data integrity: As we all know, the internet can be an unreliable network. Users abruptly disconnect and reconnect all the time, resulting in data that gets lost, is delivered in the wrong sequence or that is sent multiple times. As a result, organizations must ensure their infrastructure can guarantee high integrity without sacrificing anything else.
    • Reliability of service: The live nature of delivering updates in real-time requires total reliability. Outages can cause distrust and annoyance from users. Organizations could find their reputation taking a hit or customers quickly jumping ship if reliability proves to be a recurring issue. To prevent this, organizations should develop infrastructure that’s fault-tolerant and redundant at both a global and regional level so they can deliver a great experience even if there are multiple failures.
    • High scalability: More often than not, delivering an experience in real-time can result in unpredictable traffic; for example, a huge number of people wanting sports updates for a World Cup game. If thousands of devices connect over a short period of time but capacity can’t scale up quickly enough, then an outage will occur. To avoid this, developers must use infrastructure that can be elastic and meet changing levels of demand. Otherwise, the user experience is seriously affected and real-time capabilities crash when the network is swamped with connections it can’t cope with.

    While organizations often build their own infrastructure with the best of intentions, it’s easy for them to run into trouble. Most lack the internal expertise, resources and time; plus, going it alone often results in a huge technical debt. It’s also a massive undertaking, due to the amount of ongoing complexity and cost involved.

    Third-Party Partnerships Enable Dependability

    Unless organizations have the resources, finances and expertise of the likes of Amazon or Google, then building your own real-time infrastructure is just setting yourself up for failure. Instead, they should save on development cycles and focus on generating revenue more quickly by offloading their real-time infrastructure needs to a third party. This will mean organizations can design without limitations and focus on solving the challenges that really matter to them while putting in place real-time infrastructure that is as dependable and future-proof as possible.

  • Credit Karma Looks to Open Source Distributed App Platform

    Credit Karma Looks to Open Source Distributed App Platform

    Credit Karma has launched an application platform, dubbed Talon Polly, that makes it simpler to deploy microservices across multiple runtime environments.

    Richard Pounder, principal software engineer at Credit Karma, said the consumer financial services provider expects to make Talon Polly available as an open source project next year.

    Written in the Rust programming language, Pounder said Credit Karma created Talon Polly to provide its developers with a language-agnostic platform for building and deploying applications. Talon Polly can be deployed as a sidecar using containers or as a process running within a software environment.

    While there is no shortage of frameworks for building and deploying software, Pounder said Talon Polly is designed to make it easier for development teams to reuse code and components across multiple applications using, for example, a contract-based library included in the framework. That approach also enables Credit Karma to ensure a common set of security policies are enforced across those application environments, he noted.

    Ultimately, the goal is to enable developers to spend more time writing business logic rather than mastering the nuances of multiple application development frameworks created for specific application environments, said Pounder.

    It’s not clear to what degree other frameworks may emerge to address the same issues at Talon Polly. Odds are good that other organizations that need to deploy distributed applications built using multiple programming languages have faced similar challenges. The Cloud Native Computing Foundation (CNCF) has also adopted Dapr, a set of application programming interfaces (APIs) created by Microsoft that are deployed as container sidecars to make it simpler to build distributed applications, as an incubating project.

    Regardless of approach, the rise of distributed microservices-based applications is creating a significant management challenge for DevOps teams. Each microservice is built by different development teams that prefer one programming language over another for any number of reasons. Deploying and updating those microservices across a distributed computing environment requires more sophisticated orchestration of workflows, given all the dependencies that exist between microservices.

    Of course, it’s not quite clear yet the degree to which microservices-based applications will supplant legacy monolithic applications. Most new applications are being constructed using microservices to make applications more resilient. However, the number of legacy monolithic applications running in existing IT environments means it could be years before microservices-based applications will ever outpace them. As such, DevOps teams will be managing a mix of monolithic and microservices well into the second half of this decade.

    The paradox is that while microservices-based applications are more resilient because calls to services can be rerouted in the event of a disruption, the overall application environment becomes more complex as the dependencies between distributed microservices increases. As such, many organizations are opting to use microservices to build applications only when a certain level of resiliency is required or it’s been determined that a function likely to be used by multiple applications is more easily accessed via a microservice that has its own API.

    One way or another, going forward, it’s not so much a question of whether microservices will be employed but rather to what degree and across how many applications.

  • Unreliable Server Scare | Information Batteries | ARM IPO PDQ

    Unreliable Server Scare | Information Batteries | ARM IPO PDQ

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: We worry about chips failing randomly, we ponder a new way of thinking about workload shifting, and we grok Arm’s IPO.

    1. Smaller Features Lead to Random Failures

    First up this week: Could cosmic rays be causing increasing numbers of errors in cloud servers? As chips get faster and more densely packed, the worry is that undetected bit-flips could cause serious problems.

    Analysis: How can you mitigate the problem?

    As workloads grow in number and complexity, this is becoming a real issue—not merely theoretical. (See also: DevSecOps worries such as Rowhammer.) Think about where the critical data are and how to detect random errors. Even consider running important workflows in lockstep pairs.

    John Markoff: Tiny Chips, Big Headaches

    Imagine for a moment that the millions of computer chips inside … the largest data centers in the world had rare, almost undetectable flaws. … As the tiny switches in computer chips have shrunk to the width of a few atoms, the reliability of chips has become another worry for the people who run the biggest networks. … There is growing evidence that the problem is worsening with each new generation of chips.
    …
    Both Facebook and Google have published studies describing computer hardware failures whose causes have not been easy to identify. … Facebook researchers noted that some processors would pass manufacturers’ tests but then began exhibiting failures when they were in the field. … Smaller transistors, three-dimensional chips and new designs that create errors only in certain cases all contributed to the problem … according to Google.
    …
    Intel executives said they were familiar with the Google and Facebook research papers and were working with both companies to develop new methods for detecting and correcting hardware errors. [But] computer engineers are divided over how to respond to the challenge.


    Have we hit a wall? So asks Thutmose V:

    For more than 50 years, semiconductor manufacturers have been trying to keep up with Moore’s Law, which … has effectively served as the target for semiconductor engineers to push them forward. … The size of some parts of the transistors is now 10 nanometers or less. 10 nanometers is about equal to the width of 100 silicon atoms—100 atoms is not very many.
    …
    When things are that small, does quantum randomness start to be noticeable? Normally if the odds of something failing is a trillion to one, you don’t need to worry. But if you have 10 billion transistors operating at a clock speed of 3 billion times a second, a one in a trillion chance of failure is nowhere near good enough. … It is a miracle these things work as well as they do.


    We’ve known about the problem for years, but only now is it becoming reality, says dsgrntlxmply:

    We observed certain network failures to be caused by corruption of an on-chip routing table memory in a SoC that lacked parity protection for these tables. The failure rate was low, but was higher in units that were deployed at higher altitudes. … The problem was eventually mitigated by software table sweeps, and in newer SoC generations, by parity in critical tables.

    These Single Event Upsets are said to be caused by muons generated by high energy cosmic ray interactions in the upper atmosphere. Third semester undergrad physics.

    2. Mind-Bending Speculative-Compute Research

    Many of us are familiar with delaying a workload until a time when servers are idle—or when energy is cheap. But some fascinating research asks: What if we could predict a future workload and perform it in advance?

    Analysis: “Information batteries” can be cheaper than Li-ion

    The researchers argue that pre-caching even parts of a workload’s output when energy costs are low—or even negative—can save far more money than traditional grid-scale storage such as batteries or pumped hydro. For example, speculatively transcoding video versus generating it on the fly, or predicting future machine learning training workloads.

    Caitlin Dawson: A New Way to Store Sustainable Energy

    Renewable energy has an intermittency problem. … What if surplus renewable energy could be stored as computation? … That’s the thinking behind “information batteries,” a new system proposed by … Barath Raghavan, an assistant professor in computer science at the USC Viterbi School of Engineering … and Jennifer Switzer, a Ph.D. student from UC San Diego.
    …
    When renewable energy is available in excess, it is used to speculatively perform computations. … The stored computed results can then be used later when green energy is less plentiful. … For instance, every day, YouTube data centers transcode more than 700,000 hours of videos to different resolutions. Many of these computations are predictable and can be performed at a time when there is excess green energy.
    …
    [But] computations that are completed in advance do not need to match exactly with the computations completed at a later time. … The challenge, said the researchers, is determining what computation to perform, where and when, and how … to efficiently retrieve the results later. … It is only possible in some workloads and in some contexts. But Raghavan believes with improved prediction and integration into large systems, the technology points towards a promising future alternative.


    This isn’t just batch computing with a fancy name. cognizant_ape explains:

    On AWS we do a lot of batch processing when the data centers are low on traffic. This is just the standard SPOT instances which you can trigger once the cost goes below a certain amount and then end when it goes about a certain amount.

    The big issue is being effective with I/O and setup time. The article seems to be getting ambitious to the point of a distributed compiler with a flexible optimization metric.


    Cut to the chase. It hinges on prediction, says bradley13:

    Predicting loads is easy, but predicting the computations that will make up those loads? Not so much. … Imagine: Most employees start work in an hour — let’s predict which database queries they will generate. Your success rate better be high.

    3. Confirmed: Arm will IPO, Not be Sold to Nvidia

    SoftBank and Nvidia have bowed to the inevitable. Arm, designers of ubiquitous chip cores, will IPO sometime in the next 12 months.

    Analysis: The only sane choice, but ecosystem risk remains

    As I’ve said before, it seemed everyone was against the idea of Nvidia owning Arm. It put the kibosh on Arm’s open licensing stance, dooming the ecosystem to scraps from the table as Nvidia grabbed first dibs on everything. Now regulators must turn their attention to the very real risk that Nvidia will still acquire Arm anyway—via the stock market.

    Richard Waters, Arash Massoudi, James Fontanella-Khan and Antoni Slodkowski: $66bn sale of chip group Arm to Nvidia collapses

    The deal, which would have been the largest ever in the chip sector, was set to give … Nvidia control of a company that makes technology at the heart of most of the world’s mobile devices. A handful of Big Tech companies that rely on Arm’s chip designs, including Qualcomm and Microsoft, had objected to the purchase [arguing] that Nvidia would get an unfair advantage by having first rights to Arm’s technology.
    …
    Jensen Huang, Nvidia’s chief executive, hoped to use Arm’s processor designs to cement his company’s growing role in data centres. … Nvidia abandoned its pursuit of Arm at a board meeting on Monday, said a person familiar with the discussion.
    …
    The deal’s failure brought an immediate management shake-up at Arm, with chief executive Simon Segars replaced by Rene Haas, head of the company’s intellectual property unit. … In the UK, where politicians view Arm as a strategic national asset, attention is set to shift to whether the company will be listed in London [but] Haas said … no decisions had been made on where Arm would be listed.


    Reminding us why this matters to DevOps, here’s Timothy Prickett Morgan:

    Now there is no question that Arm Holdings will be going public. … The plan is to do so before the end of [its] fiscal year in March 2023. … Arm Holdings will be returning to its IP licensing roots, having spun off some uncore business back into SoftBank, and seeking out new markets as it has been able to do with hyperscalers [and] cloud builders.
    …
    When Intel was worried about AMD Epyc chips, what it really needed to worry about was boosting price/performance to keep Arm CPUs out of the hyperscaler and cloud builder datacenters. [But AWS] has Graviton Arm server chips on a pretty regular schedule, and you can bet Masayoshi’s last yen that Graviton4 will debut at re:Invent 2022 this year.


    But what happens next? Pun it up, HereAndGone:

    If Softbank is going to take ARM public, Nvidia will find it much cheaper simply to acquire a controlling interest. … Looks like a duck, walks like a duck, but cheaper than a duck—while ducking regulatory issues. This plan is not quacked up to solve what it pretends to solve and will likely prove most fowl.

    The Moral of the Story: Fall seven times and stand up the eighth


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Nicole Geri (via Unsplash; leveled and cropped)

  • The Evolution of Incident Management

    The Evolution of Incident Management

    Have you ever thought about the history of incident management?

    If you’re an SRE, you might be so caught up in the day-to-day work of managing reliability and responding to incidents that you never take time to step back and reflect on the evolution of your role and your responsibilities. And that’s a shame because SREs didn’t invent incident management concepts and strategies on their own.

    On the contrary, the way SREs think about incident response, structure incident management teams and rank the priority of incidents owes much to incident management strategies developed in the offline world decades ago. To understand fully what it means to be an SRE today, you have to appreciate this deep history of incident management and response.

    So, let’s take a look at that history and examine where modern incident response concepts originated.

    Historical Problems in Incident Management

    Societies have always had incidents, of course. Fires, floods, infrastructure breakdowns and similar crises have been happening for millennia.

    For most of history, however, humans lacked an efficient, purposeful way to manage these sorts of incidents. Response efforts were ad hoc, and their effectiveness owed more than a little of their success to sheer luck.

    Particular challenges included:

    • Lack of effective, consistent communication between stakeholders.
    • Varying organizational structures which made it hard to identify leaders, coordinate response efforts and delegate tasks.
    • Inconsistent response strategies.
    • Differing approaches to assessing the priority of incidents.

    Historically, organizations may have been able to handle incidents well enough if the incidents required response from only one, small group. But the more stakeholders involved, the harder it was to respond quickly and effectively.

    Putting Out Fires: The Birth of the ICS

    Things began to change for the better when stakeholders started thinking about better ways to put out fires—literally.

    By the 1960s, fire chiefs in California realized that they were struggling to respond effectively to the wildfires that broke out every summer. Each year brought worse blazes than the last, with more land burned and more buildings lost. The Laguna fire of 1970 brought matters to a head and was the catalyst for a new approach to incident response for fire agencies.

    After assessing what was going wrong, the fire chiefs determined that it wasn’t a lack of equipment or personnel. It was poor coordination among the various firefighting agencies that responded to blazes. Lacking a clear chain of command and a systematic approach to firefighting, the agencies struggled to deploy their resources rapidly and, thus, effectively.

    To fix the problem, California fire chiefs developed what became known as the Incident Command System, or ICS. The ICS defined a hierarchy for incident response with an incident commander at the top. It also defined several categories of incident response processes including operations, planning, logistics and finance. And it established a consistent set of terms that stakeholders can use to describe their actions during incident response, which makes it easier to communicate clearly.

    Although the ICS was initially conceived to fight fires, it became the de facto standard for organizing incident response strategies of all types.

    From ICS to NIMS

    The history of incident response doesn’t end with the ICS. A new chapter began in the early 2000s when the U.S. federal government developed an even more comprehensive approach to incident management called the National Incident Management System, or NIMS.

    NIMS was born in the wake of the September 11, 2001, terrorist attacks, which underlined the importance of efficient communication not just between different agencies of the same type (like fire departments), but of entirely separate organizations. To achieve this, NIMS expanded upon the principles of the ICS.

    In addition to adopting most of the incident command principles and practices included in the ICS, NIMS included standards for coordinating the distribution of resources. It also embraced the concept of the emergency operations center, which is in some ways similar to a network operations center in the digital world.

    In some respects, NIMS resembled a compliance framework (although to be clear, that’s not what it is). It includes fourteen management principles, similar to compliance controls, that organizations must implement to manage incidents using a NIMS approach.

    Incident Management Today

    Obviously, putting out forest fires and responding to terrorist attacks is pretty different from dealing with data center failures or a buggy application deployment. ICS and NIMS weren’t designed for site reliability engineering or IT teams specifically.

    Still, the influence of ICS and NIMS on the way SREs think is clear enough. Terminology like “incident commander” comes from these frameworks. So do concepts like shared accountability for incident response processes and the importance of involving all stakeholders—not just technical teams—in incident response.

    ICS and NIMS may not be acronyms familiar to most SREs. But they should be, because they are the historical sources of the incident management philosophies that form the foundation for SRE work today and offer valuable lessons for any SRE on the job today.

  • Heterogeneous Hardware Needs Universal Software

    Heterogeneous Hardware Needs Universal Software

    As hardware advances and diversifies, we’re entering what many see as a new golden age of computer architecture. However, the idea of an ever-expanding hardware landscape can be daunting for software developers, because while hardware diversity is wonderful for price and innovation, it can lead to spiraling software complexity on the back end. So many software tools are specific to the hardware they run on that a world where CPUs, GPUs, FPGAs, ASICs, DSPs and more are freely intermingled feels overwhelming; an invitation to interoperability headaches and spiraling tech overhead. The question on the minds of developers is how to navigate this constantly changing ecosystem to take advantage of the new world of hardware acceleration without driving themselves crazy in the process. 

    The challenge for developers is that, if we look historically, every time a new piece of world-changing silicon shows up, it often comes with a new software stack of tools. With heterogeneous hardware innovation moving at a nearly untrackable pace already, keeping up with the software end is beginning to feel untenable. In response, a new philosophy is emerging: We should be able to have a common set of tools, libraries, frameworks and compilers that can address a wide variety of accelerators in a “silicon agnostic” way. 

    Anyone doing DevOps work today should be asking, “Are the tools we are adopting ready for the heterogeneous hardware future?” Rather than using proprietary stacks exclusive to individual vendors’ concepts of how acceleration should work, we should be thinking about abstraction layers that allow us to move to open software stacks that work across multiple vendors and architectures. 

    This philosophy is central to a number of movements—oneAPI being one example—of asking for libraries, compilers and analysis tools that are more versatile. It’s incumbent on any future-conscious developer to analyze the tools they’re using and ask if they are preparing for the future or locking them into a single solution that excludes you from other innovations. 

    When it comes to navigating the changing developer landscape, it’s particularly important to look at the multiple-vendor piece of the puzzle. One of the signature strengths of computing and electronics is its ability to interoperate. Without compatibility between platforms, we end up locked into single solutions that don’t take advantage of the diverse landscape of innovation available to us. If systems are built from the ground up with mixing and matching in mind, we don’t have to change our software tools to adapt to new technologies and we can use the same languages, libraries and analysis tools we are used to. 

    It’s important we take on this work now because hardware diversity is not just at an all-time high, its momentum is only growing. In the past when the hardware landscape was more limited, switching to different toolsets for different CPUs was a feasible, if inelegant solution. Today’s computers, however, have computational capabilities from multiple vendors. If my tools don’t stretch across all of its ecosystems, I end up in a complex mess very quickly.  

    A particularly relevant example can be found in supercomputing, one of the most exciting and fast-moving areas of innovation in the developer community. What is the most exciting thing about supercomputing? It’s heterogeneous, allowing multiple architectures to take on specialized hardware acceleration for the different tasks for which they are best suited. The Aurora supercomputer, for example, is planned to achieve up to 2 exaflops of performance—a staggering number (the first supercomputer I helped design had 1 teraflop—or 2 millionths of Aurora’s performance). The amazing capabilities of these machines are only possible because of advancements in heterogeneous memory technologies, storage capabilities, sophisticated CPUs and GPUs all working together seamlessly.  

    The advancements we’re seeing in supercomputers, as always, will ultimately filter down into lower-level systems. In the near future, regardless of the type of work you do, enormous computing power is coming online. We need to make sure that our libraries and tools can take full advantage of it when it arrives.

    Two great places to learn more about activities to help software be open, multi-vendor and multi-architecture for heterogeneous programming are:  https://sycl.tech/ and https://www.oneapi.io. There are numerous online tutorials, a link to the SYCL book (free PDF download) and much more.

  • Taking a DevSecOps Approach to API Security

    Taking a DevSecOps Approach to API Security

    An application programming interface (API) is a data transfer approach that enables services within an application to talk to other applications (or other services within a system). Essentially, it enables Service A to talk to Service B in a uniform way. Companies benefit from APIs on a daily basis without even thinking about them. APIs are a way for companies to gain market share from competitors and monetize applications in new and exciting ways. For example, every time someone signs on to their favorite social media platform or shops on Amazon, the application is utilizing APIs to fetch and update the data with which they are interacting.

    Discovering and securing APIs is a never-ending battle for IT teams across all industries. APIs, for all their benefits, open new attack surfaces targeting a company’s data via vulnerabilities and exposures. In Gartner’s recent report, API Security: What You Need to Do to Protect Your APIs, the analysts estimated that for 2021, 90% of web-enabled applications will have more surface area for attack in the form of exposed APIs rather than the user interface (UI), up from 40% in 2019. This turned out to be accurate.

    Traditional Approaches to API Security

    Traditionally, APIs are protected by enterprise API gateways and content delivery networks (CDNs). Although these approaches hedge against most brute force and denial-of-service (DoS) attacks, a more nuanced and sophisticated attack can breach these protections. The traditional approaches to API security are littered with limitations.

    Companies typically use two types of APIs – customer-app-facing APIs also called “north-south APIs” and internal APIs called “east-west APIs.” The external type is exposed to all public internet traffic. External developers can make requests and receive responses to these customer-facing APIs and take advantage of the functionality and data of an application while bypassing the user interface (UI). Internal APIs, however, are exposed only to intra-application traffic. For example, the cart service may call the user service via an internal API during the checkout process of an e-commerce application.

    Customer-facing APIs tend to be a bit more stable, visible, and targeted at a developer audience. The largest challenge of these north-south APIs is ensuring that this layer is secure every second of every day. This open and accessible API is designed to be utilized by developers outside the publisher’s organization. The goal is to leverage a community of technical audiences and increase usage of the organization’s data and services. Occasionally API publishers encourage developers to create applications that expand on the core business.

    While internal developers may call north-south APIs for their application development, the more common east-west API calls between services often drive much of the incremental feature development and new innovations. These APIs may or may not be accessible by developers outside the publishing organization. However, increasing numbers of east-west APIs are native in the public cloud so the attack surface can easily be exposed publicly. Further, changes to these east-west APIs are made constantly — as fast as a company’s DevOps and CI/CD practices allow. Unfortunately, this speed of change makes mistakes, errors, and security oversights more likely. These APIs and changes are less visible, which leaves companies at risk of discovering vulnerabilities only after a hacker has breached private data.

    Security Breaches of APIs

    Data leaks through all APIs, but companies can’t protect what they don’t know. The moment APIs are exposed on the internet, they create attack vectors for hackers to extract data. If some of the largest companies in the world have insecurities throughout their codebase of APIs and services, we can bet every other company does too.

    Security breaches of APIs are so dominant because APIs are such a ubiquitous method of passing information. Data breaches come in many forms, but they almost always affect company revenue and reputation. New API breaches and vulnerabilities are discovered almost daily. Capital One, Microsoft, T-Mobile, Symantec, McDonald’s, Instagram, Salesforce.com, and Venmo have all experienced major API breaches in the past few years.

    The DevSecOps team or security specialists should make ongoing risk-based decisions on when certain issues should be mitigated, and where in the CI/CD pipeline those concerns should be tested. However, in key scenarios, active protection is required to prevent a security incident from occurring or escalating. IT teams need to use tools that add this layer of protection without requiring humans to be involved.

    The traditional approaches to API security, although helpful at reducing the attack surface, are too slow to keep up with the ever-evolving tactics of bad actors. Only with a clear and full-stack view of an application’s security can IT manage risk and enforce appropriate security policies. Without better telemetry and observability, every security decision will be a shot in the dark and critical holes in API security will be missed.

    Consistently and Constantly Analyze

    Security processes and tooling should constantly analyze the cloud environments, API gateways, logs, web and mobile applications, as well as source code for changes that might create vulnerabilities. Once IT is able to identify the changes to their system, they can confirm that authentication, authorization, encryption, access and verification meet the standards set for the team.

    Active protection without human intervention is the only way to achieve real-time enforcement of security policies and protect against attacks as they occur. Ideally, a security solution should automatically discover and inspect for API security vulnerabilities, while humans analyze the situational risk and prioritize the work.

    With active protection in place, when a security incident occurs, IT will be able to rapidly leverage critical information to appropriately identify the components of the system affected and the contributing factors that led to the event, and protect the data.

  • Linux Foundation Survey Sees Rise in SBOM Use

    Linux Foundation Survey Sees Rise in SBOM Use

    A global survey of 412 organizations published today by the Linux Foundation found nearly half of respondents (47%) producing or consuming software bills of material (SBOMs), and more than three-quarters said they expected to produce or consume them in 2022.

    SBOMs have become a bigger area of focus in the wake of a series of high-profile software supply chain-related breaches. The breaches involved application components compromised by malware that had been surreptitiously inserted during the application development life cycle. The Biden administration has gone so far as to issue an executive order requiring all agencies to provide SBOMs for all the commercial and custom software they use, but as of yet a deadline for compliance with that order hasn’t been set.

    In general, the Linux Foundation survey found that a large percentage of respondents (82%) are already familiar with the term SBOM. Respondents cited various benefits of having an SBOM, including making it easier for developers to understand dependencies across components in an application (51%), making it easier to monitor components for vulnerabilities (49%) and making it easier to manage license compliance (44%).

    However, the survey also makes it clear there is much SBOM work to be done. A full 62% of respondents, for example, are looking for better industry consensus on how to integrate the production/consumption of SBOMs into their DevOps practices. Respondents also cited a need for consensus on the integration of SBOMs into their risk and compliance processes (58%).

    More than half (53%) also said there is a need for better industry consensus concerning how SBOMs will evolve and improve.

    The survey also makes it apparent that organizations are more aware of the role SBOMs play in securing supply chains. A full 80% of organizations are aware of the White House’s executive order on improving cybersecurity, with more than three quarters (76%) considering making changes as a direct consequence of the order. Overall, the survey found the top consideration organizations focus on when evaluating open source software is security.

    Stephen Hendrick, vice president of research at the Linux Foundation, said 2022 is likely to be remembered as the year of the SBOM. There are several standards for building SBOMs, including an open source software package data exchange (SPDX) specification backed by The Linux Foundation and the Joint Development Foundation that is now recognized as the ISO/IEC 5962:2021 international standard. There is also CycloneDX, a lightweight SBOM standard based on an open source project that originated via the Open Web Application Security Project (OWASP) community and a software identification (SWID) standard for tags that some organizations use to create SBOMs.

    SBOMs are, essentially, a tool for listing the components that make up an application and are used inconsistently. However, when a critical vulnerability is discovered, they can sharply reduce the amount of time required to discover all the software components that need to be remediated. The challenge is, with the rise of containers, it has become much simpler to rip and replace software components in a way that may not always be reflected in the SBOM. Hendrick said in time, advances in technologies such as machine learning algorithms should make it easier to instantly reflect those changes in an SBOM.

    Regardless of approach, it’s apparent DevOps teams, like it or not, are about to be held more accountable for the provenance of every component employed across their entire application portfolio.

  • Continuous Security: The Next Evolution of CI/CD

    Continuous Security: The Next Evolution of CI/CD

    Following a wave of high-profile cyberattacks and the White House’s release of the executive order on improving the nation’s cybersecurity, how to build a successful cybersecurity program has never been so hotly debated as it is now. From SolarWinds to the Colonial Pipeline attacks, these events demonstrate that defensive measures alone cannot mitigate cybersecurity risk, and any cybersecurity program must begin where nine out of 10 breaches begin—the code. 

    The adoption of CI/CD in recent years as an extension of Agile and DevOps has enabled organizations to embrace digital transformation and ensure code is always being delivered that matches the most current business needs. Unfortunately, this has further siloed security as processes failed to keep pace. While there are a number of point solutions to address the ever-expanding domain that encompasses application security—from traditional pentesting and vulnerability scanners to monitoring of the individual components of the DevOps toolchain like containers—too many tools results in a DevSecOps “hairball” that leaves organizations inundated with data, yet without actionable information. 

    Moving Beyond Secure DevOps

    As organizations continue to evolve their delivery processes, security has found its role in secure DevOps (SecDevOps) programs to ensure the execution of security processes at every step of the software development life cycle (SLDC). By continually managing security practices, policies and debt in existing pipelines, this approach ensures that all stakeholders—from the C-suite to the developers and AppSec managers—have the information they need at every step of development to share responsibility in delivering secure software.

    However, these programs, while a good starting point, have left security requirements and checks in PDFs, Excel documents, and emails failing to capture the same benefits of security integration into the SDLC that Dev and Ops have commanded. Without SecDevOps orchestration, the information often lags the pipeline, creating new bottlenecks and uncertainty. SecDevOps orchestration gives security and development teams the confidence to know they’re following the right policies and eliminates the manual work that creates drag on the SDLC. With the automation of the application security process, enterprises get consistency and insight into their program that allows their focus to stay on shipping code while knowing it meets their organizational security standards. 

    Adding Continuous Security

    With CI/CD, organizations are able to rapidly meet the most current needs of the business as they rapidly evolve, but without a continuous approach to security as well, once again, security is siloed off. Organizations must be able to adapt their security requirements for changes both internal (changing databases, different software versions, etc.) and external (compliance requirements, changing threat landscape, etc). We need to take CI/CD one step further and also have security integrated continuously. 

    By extension, continuous security (CI/CD/CS) is the philosophy of continuously shipping software that meets the most current security standards for the business, accounting for internal and external change throughout the SDLC. 

    To effectively adopt a CI/CD/CS, organizations must focus on three key principles:

    • There is no perfect code, therefore there is no perfect security—Good application security manages cyber risk in line with overall business strategy. When you have the ability to accept risk within the risk tolerance of the businesses, you know the right times to stop, and the right times to carry on because other controls are in place. Don’t let perfection be the enemy of the shipped. 
    • Bite-sized application security—DevOps isn’t about moving fast and breaking things, it’s about moving efficiently and fixing things at the right time. Deliver the right application security information at the right time to the stakeholder responsible for executing it. This ensures better collaboration so stakeholders know where, when and with whom to direct their attention. 
    • Micro-feedback loops—Leverage orchestration to get real-time feedback on policy compliance before it creates bottlenecks down the line. This enables you to respond to changes both internal and external as quickly as possible, creating an iterative cycle of information for future decision-making. 

    By continually managing security practices, policies, and debt in existing CI/CD pipelines, this approach allows a scalable application security program to be deployed in any SDLC so it can continually evolve to always be in lockstep with today’s business.  

    Different organizations have different risks they need to account for, which means security must be aligned to business strategies and priorities in order to dynamically respond. With end-to-end integration in the SDLC, continuous security supports CI/CD to improve productivity and speed time-to-market, while reducing the risks that might impact a particular business. Software is inherently impermanent and organizations need to be able to continuously balance security, technical and business priorities to ensure they are maintaining their focus on what matters most: Delivering value to customers and shareholders. 

  • Improving Software Security in 2022

    Improving Software Security in 2022

    The recent Log4j vulnerability showed just how quickly a security bug could disrupt not just an industry, but the entire world.

    Organizations, especially federal agencies, will always find themselves at some level of risk, but they can also do more to mitigate those challenges. In November 2021, the Biden administration issued a directive through the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA) to fix hundreds of software and hardware vulnerabilities. 

    While this effort created an immediate call-to-action to patch known security risks, organizations from all industries must constantly manage known and unknown threats. A study released in July 2021 found it took organizations an average of 205 days to fix critical vulnerabilities, a timeframe that provided bad actors a wealth of opportunity to conduct serious damage.

    Agencies must adopt a more proactive approach to cybersecurity by improving software code quality. Let’s look at how to do that.

    An Ounce of Prevention

    Shaking up the status quo does tend to raise a few eyebrows, but the truth is that security programs should be in a constant state of continuous improvement. Today, our security programs work in an overly reactive state that relies too heavily on mitigation once a threat has emerged.

    Emphasizing a preventative approach may not be widely understood outside the security team, especially when an agency has a relatively clean security record. It might be seen as something that isn’t broken and, therefore, doesn’t need fixing. In this instance, getting leadership buy-in at all levels of an agency is important.

    Some pertinent points security executives should emphasize to change security culture inside their organization include:

    • The time and cost savings achieved through preventative measures, such as role-based training and related tools, as opposed to the potential cost of a critical incident.
    • Finding and fixing software vulnerabilities as code is written keeps releases on time with fewer showstoppers from the security team.
    • Preparing for and preempting potential security risks from the development team to release saves time and money overall. 

    Shifting Left of Left

    Shifting left has been popular in Agile and DevOps environments for over a decade. It involves testing small software components as early as possible rather than waiting until the end of the sprint.

    To create more secure code, organizations need to start left of left, eliminating common vulnerabilities as early as possible to create a safer user experience. 

    Starting left of left is a developer-first concept and requires organizations to get serious about uplifting their engineering cohort, placing emphasis on the creation of high-quality code. Security-aware developers are worth their weight in gold, and they need support in the form of job-relevant, hands-on training in secure code as well as the ability to provision the right tools. The opportunity to be mentored by more experienced developers will also foster an environment where code is crafted with a security-first mindset and the precision required to take software to the next level.

    One key area that often gets overlooked is the user experience; particularly with regard to how users access information.

    Security misconfigurations accounted for 21% of cloud-based data breaches in the past year, and amateur-hour errors (like storing passwords in plaintext, for example) resulted in serious productivity and customer trust losses. To avoid these errors, aim for a secure user experience that weaves tight security into a flow that makes sense. Adding more barriers—complex password requirements, a CAPTCHA, a horde of flesh-eating zombies—can turn users away. On the other hand, getting too permissive with security measures renders the entire point moot.

    A successful, secure user experience needs to weave tight security into a flow that makes sense, presented in a way that doesn’t detract from anything compelling about the software. 

    Improved Developer Upskilling

    Developers, of course, want to write secure code, but they often lack those skills or need a refresher course. Meaningful training too often gets overlooked as the day-to-day needs of organizations allow for little time to improve skills.

    In working with developers, we’ve found that about 75% prefer structured on-the-job learning instead of opening a manual. They would rather learn by doing and want training focused on practical applications, something that most current training programs are missing.

    Look for any and all opportunities to upskill your developers. They are on the front lines when it comes to stopping vulnerabilities. Provide them both the time and right-fit resources, understanding that these efforts will pay considerable dividends in the future.

    Government agencies, in particular, face inherent challenges in keeping their systems secure. They often must manage outdated systems with few financial resources and fight for top talent in an incredibly competitive market. There is no panacea for removing vulnerabilities, but taking a proactive and preventative approach that emphasizes the customer experience with highly skilled developers can help agencies take significant steps forward.