devopsy.kinsta.cloud

Category: DevOps Practice

  • Akamai: Buying Linode | Firefox: Not OK | Gone: Google Vaccine Mandate

    Akamai: Buying Linode | Firefox: Not OK | Gone: Google Vaccine Mandate

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Linode gets bought by Akamai, Firefox market share is “measly,” and Google brings staff back to the office.

    1. Akamai + Aker = ‘Synergy’

    First up this week: Why is Akamai buying Linode? Whatever the reason, it’s a big payday for its solitary shareholder: Not far off a billion dollars.

    Analysis: There goes the neighborhood?

    The 100 lb gorilla of edge computing and CDNs is promising not to mess with Linode’s winning open source cloud formula: No change, steady as she goes, etcetera. But how many times have we heard that before?

    Here’s local scribbler Paige Gross: Linode’s $900M acquisition by Akamai is a ‘game changer’

    A household name in the Philly tech scene … CEO Christopher Aker founded the company in 2003, and it’s been bootstrapped ever since, growing to about 250 employees. … Aker has owned 100% of Linode since he founded it.
    …
    Aker said the deal brings the company closer to solving evolving challenges “as cloud services become all-encompassing, including compute, storage, security and delivery from core to edge. … We’ve been talking about ‘alternative cloud’ — alternative to the big guy — and now we’ve sort of validated ourself in this market.”
    …
    The Old City company maintains “well over a million accounts” but has anywhere around 150,000 active users of its platform at any particular time. Linode’s HQ, the bank building at Third and Arch Streets, remains. … For clients and Linode employees, there won’t be too much noticeable change [Linode] says.


    Why does Akamai want Linode? Jonathan Greig asked the CEO:

    Akamai CEO Tom Leighton [explained] that Linode has great customer support and is already in 11 locations, which Akamai is going to “dramatically” expand. … Leighton also noted the September 2021 acquisition of Israel-based Guardicore, a cybersecurity company that offers a micro-segmentation solution to reduce the potential attack surface of corporate networks, secure applications, and meet compliance standards.
    …
    According to Leighton, the company is expecting the cloud compute category — which includes edge applications, its net storage business, and Linode — to reach “well over half a billion dollars in 2023. … We were pioneers in edge computing and now we’re taking a big step forward in cloud computing with Linode.”


    But here’s a worried-sounding zoid.com:

    Well good for them but I hope [Akamai] don’t screw it up. Linode is a great hosting company.

    2. Firefox isn’t OK

    Last week, a journalist innocently asked, “Is Firefox OK?” And people lost their minds.

    Analysis: No, not OK—it’s a big problem for DevOps and the web.

    We need Firefox and Gecko. A Chromium monoculture is bad for the web and bad for DevOps. (Before you ask: No, the existence of WebKit doesn’t change things, thanks to Apple’s ridiculous policies). Much as I tried to say something nice about Firefox last month, its market share is, frankly, pathetic.

    Matt Burgess: Is Firefox OK?

    In the two decades since Firefox launched from the shadows of Netscape … the browser has slid to less than 4 percent of the market—on mobile it’s a measly half a percent. … Its fate also has larger implications for the web as a whole.
    …
    Each year Google pays Mozilla hundreds of millions of dollars in royalties. … In its 2020 financial results … Mozilla listed its total revenue as $496 million. … The Google-Mozilla deal was last renewed in 2020 and is expected to expire in 2023. … Mozilla and Firefox acknowledge that for its long-term future it needs to diversify the ways it makes money.
    …
    Firefox still matters: … The browser market is dominated by Google’s Chromium codebase and its underlying browser engine, Blink. … Microsoft’s Edge Browser, Brave, Vivaldi, and Opera all use adapted versions of Chromium. Apple [has] its WebKit browser engine [but] other than that, Firefox’s Gecko browser engine is the only alternative.


    Firefox, how do we love thee? Let not_count_zero count the ways:

    I still use Firefox because:
    a) I remember Internet Explorer and the standards wars. …
    b) I don’t trust Google. …
    c) Most sites work well.
    …
    I do have questions though. … They’re currently getting around a half a billion‽ What are they doing with it? … This seems really off—really, really off.


    But heed a deeply worried Thom Holwerda:

    The decline and potential demise of Firefox is a massive problem that everybody seems to be kind of tiptoeing around—too afraid to acknowledge that if Firefox were indeed to disappear, we’d be royally screwed. We’d end up right back where we started 20 years ago, with Chrome being the new IE6.
    …
    The situation is especially dire for … Linux users, developers, and distribution makers. … Gnome Browser or KDE’s Falkon can barely be taken seriously, and … are based on Apple’s WebKit, which isn’t a great position to be in.

    3. Google’s Return to Work Plan Drops Vaccine Mandate

    Google is planning “celebrations” to welcome back employees after being fully remote. And in practice—unlike at IBM—this doesn’t mean a vaccine mandate.

    Analysis: Vocal minority wins—or do they?

    It was just three months ago that Google said it “firmly stands behind our vaccination policy.” What changed? It’s unlikely to have been due to a tiny minority of staff publishing a manifesto.

    Jennifer Elias: Google relaxes mandates

    The company will no longer require vaccination as a condition of employment … as it prepares to bring workers back to the office … at least three days a week for a “hybrid” work model. … The company [had previously] told employees that they must comply with vaccine policies or they’d face loss of pay and eventually loss of employment.
    …
    According to Google spokesperson Lora Lee Erickson … the company dropped the requirement … after removing the Jan. 18 deadline it had set for employees to either get vaccinated or get exemption approval. She declined to provide further details on the policy or the reasons for the reversal.
    …
    Google Real Estate and Workplace Services VP David Radcliffe [said Google] is preparing to begin its 30-day transition period to the hybrid work week if conditions continue to improve. He said his team is planning “celebrations” to welcome back employees. … Unvaccinated employees who are approved to enter offices will still need to follow additional protocols, including testing and wearing a mask.


    It’s too simplistic to say the anti-mandate brigade are also anti-vaxxers. For example, raxxorrax:

    I am vaccinated and HR knows this. … But if they officially demanded a certificate, I think I would have called it quits. Many people don’t have that option [but] there are consequences for such unreasonable demands.
    …
    I understand a mandate for vaccinations in certain occupations relating to health. It is something you have to personally accept if you want to work in the field.


    Although a few others are way over at the end of the bell curve. Greg Jinkerson:

    Don’t get excited, [Google is] still evil and stupid. … Now they’re backing down for some reason, [but] this is barely anything notable as far as a rollback of restrictions. [It] is not any kind of encouraging news.

    The Moral of the Story: Failure is the foundation of success—and the means by which it is achieved.


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Aleksey Kuprikov (via Unsplash; leveled and cropped)

  • Top Nine Skills for SREs to Master

    Top Nine Skills for SREs to Master

    It’s easy to talk at a high level about what site reliability engineers do: They ensure that IT systems achieve availability and performance requirements.

    But which skills, exactly, do SREs need to do to perform their jobs? That’s a more complicated question.

    To answer, let’s look at the top nine site reliability engineer skills that modern SREs (or aspiring SREs) should master. Although SRE skills may vary from one team to the next depending on the types of systems managed and the main types of reliability challenges faced, virtually all SREs need a core set of standard skills that allow them to understand and manage the type of complex, distributed systems they will have to support at the typical organization today.

    Here’s a breakdown of top SRE skills.

    Networking Expertise for SREs

    The network plays a pivotal role in connecting modern, distributed environments. As such, it’s often the culprit when something goes wrong—a lesson that Facebook, for example, learned when a networking problem brought down its entire global infrastructure.

    Situations like this are why SREs should master networking concepts. Even if their organization also employs networking engineers, site reliability engineers need a deep understanding of networking themselves to know when the network is the root cause of an incident and how to resolve network-caused issues effectively.

    Linux and Unix

    If you come from a Windows background but you want to be an SRE, there’s no getting around it: You’ll need to learn how to work with Linux and other Unix-like systems in addition to Windows.

    That’s because, even at organizations that don’t rely heavily on Linux servers, you’re likely to find that Linux and Unix concepts are deeply embedded within other systems that you have to work with. Most public cloud management tools follow the conventions of Linux CLI tools, for example. So do systems like Docker and Kubernetes, even if you run them in a Windows environment.

    Cloud Computing

    Like Linux and networking, cloud computing is another category of skill that modern SREs can’t live without.

    The reason why is almost self-explanatory: Around 90% of businesses use the cloud, and you can’t manage reliability for cloud environments very well if you don’t understand cloud architectures, cloud networking, cloud data storage, cloud observability and so on.

    CI/CD Pipelines

    SREs don’t typically help to develop software, but they nonetheless need a deep understanding of how software is written and deployed—which, at most organizations, today, is a process that happens via a CI/CD pipeline.

    It’s hard to engineer reliability if you don’t know how to address reliability problems that emerge from application source code or deployment processes. Understanding how CI/CD processes work and which tools drive them is key for virtually every SRE today.

    Quality Assurance and Software Testing Automation

    SREs also don’t usually help to test software pre-deployment. That task falls to developers or quality assurance engineers.

    Nonetheless, understanding how software is tested—and how to use test automation to speed tests and expand test coverage—is a vital SRE skill. After all, the more thoroughly and efficiently your team can test software, the greater your chances of catching reliability problems pre-deployment, when they are easier to fix and pose a much lower risk to the business.

    Security Engineering and Response

    Securing is another domain that SREs don’t “own,” but where they nonetheless require significant skills.

    Indeed, good reliability engineering makes security a priority, and vice versa. SREs who don’t understand security fundamentals are at risk of implementing reliability solutions that are effective from a reliability standpoint, but not necessarily secure.

    DevOps

    Although SREs are not DevOps engineers, SRE and DevOps are closely related domains. SREs at most organizations today will be expected to understand DevOps concepts and, in many cases, work alongside DevOps teams.

    So, plan to master DevOps skills as part of your site reliability engineer skills acquisition strategy.

    Incident Management

    Perhaps the single most important type of skill for SREs to learn is incident management. Although many roles may participate in incident response, SREs usually take the lead in organizing the incident response team, communicating with stakeholders and devising the best strategy for resolving each incident as quickly as possible.

    This means SREs should know how incident response roles are structured and understand incident response concepts. They should also be familiar with incident response platforms that automate the complex processes required to ensure rapid, effective incident resolution.

    Managing Postmortems

    In addition to overseeing incident response, SREs may be tasked with managing post mortems. Knowing how to run a postmortem—as well as when a postmortem is necessary, and when it makes sense to use a “blameless” postmortem approach—is an essential SRE skill.

    The list of SRE skills could certainly go on. Above are only the most fundamental types of skills SREs will need for most modern environments. But if you’re just starting out on your journey to becoming an SRE, the nine skill domains described above are a good place to begin acquiring the knowledge you’ll need to excel in an SRE career.

  • Can Regulatory Mandates Secure Software Development?

    Can Regulatory Mandates Secure Software Development?

    Software companies have a long history of delivering incomplete and insecure products. This happens for a couple of reasons. Fast time-to-market has always been a business priority, taking precedence over security, especially as DevOps has become the norm in software development methodologies. And software, by nature, is easy to update and change, so some defects or oversights could be allowed to slide through with the thought that “We can always fix it later.”

    Unlike car manufacturers that put a premium on including safety features upfront (or face the heavy hand of regulators), software makers have typically emphasized speed to market as a way to stay competitive.

    But the other shoe often drops on that approach in the form of costly recalls, extensive patch programs, cybersecurity breaches, and other issues. The fix-it-later mentality in software has become increasingly challenging in recent years with the emergence of connected/IoT devices that are harder to keep current and have a greater impact. Connected closed-circuit surveillance cameras, for example, are frequently hunted and exploited on the internet to spy on people. Internet of things (IoT) devices operating at dams or nuclear power plants can now be hacked (as seen at the Bowman Avenue Dam in New York, or three utility companies in Ukraine hit by BlackEnergy malware). And, across the board, the cost of data breaches continues to rise.

    A White House executive order (EO) from 2021 is also putting pressure on software manufacturers, DevOps and DevSecOps professionals to improve quality, with a mandate for U.S. federal agencies to enforce secure software development through their contracts and acquisitions as part of efforts to secure the software supply chain. The core difference between this effort versus previous attempts such as FISMA, is the visibility into the software development process.

    The attention being paid to secure software development is a step in the right direction, but ultimately it’s still up to the industry rather than regulatory bodies, to change the mindset toward building secure software from the ground up.

    Technology is Faster Than Mandates

    The EO represents a shift in the landscape, mandating not just secure products, but secure development practices. Although it applies officially to federal civilian agencies and their contractors, federal regulations and other government mandates often become global standards, as has happened with the General Data Protection Regulation (GDPR), the EU’s privacy and security law.

    But regulatory mandates aren’t enough. They often move too slowly to be a real solution, especially in a software development environment that is moving steadily towards more agile, fast-paced environments where continuous integration and continuous delivery (CI/CD) is the norm. The timelines for new regulations typically require a two-year window for implementation; if a company already has an applicable product in use, it could be up to five years before they’re required to be in full compliance, such as in the payment industry. By that time, the key elements of the industry best practices have already changed. Simply put, technology moves too fast for regulatory bodies to keep up.

    What’s more, the regulations themselves could ultimately fall short if they are perceived to have a high cost burden of compliance. This is specifically true if the technology used in the software development life cycle can’t keep pace with the needs of compliance. In support of the EO, the National Institute of Standards and Technology (NIST) has released Draft Special Publication (SP) 800-218, and, as is routine, accepted open public comments. The feedback from industry consortiums in these situations often includes considerations about the costs of implementing new regulations and can sometimes lead to softening of some aspects, arguing, for example, that the new regulations put an unfair burden on small and mid-sized companies. The final set of regulations can sometimes leave the door open for ways to address compliance without addressing the issues that led to the regulation in the first place.

    For example, the California Consumer Privacy Act (CCPA) is widely hailed as the nation’s strictest set of privacy protections, yet it still has its weaknesses. The most commonly cited weakness is that the law makes it extremely difficult for consumers to take advantage of the new regulations when they feel they have been wronged.

    Industry Must Make it Happen

    Ultimately, it is up to software manufacturers and the industry to improve the security of their software. They should adopt best practices to build secure procedures into the design, development, testing and validation of new software, as well as its deployment and use. NIST’s Secure Software Development Framework is one example of guidance, being drawn from recommendations from groups like BSA, OWASP, and SAFECode. Industry groups, such as the International Organization for Standardization (ISO), could adopt standards for development and certification similar to the universally accepted UL standards for electrical products.

    The rising demand for secure development requires a substantial shift in attitudes toward how software is made. A lot of these things can be more easily implemented if organizations invest in the adaptation of new technologies faster.

    The threat landscape has become too sophisticated and pernicious for the problem of insecure software to be ignored. Fix-it-later is no longer a valid option if organizations want to protect their networks and, especially, their data, which has become the lifeblood of operations in every sector. While regulatory mandates help, the job ultimately falls to the software industry itself and its consortium of like-minded people bridging the gap between developers and DevOps teams of competing companies to ultimately deliver what’s best for the consumers.

  • Best Practices to Improve your Engineering Sprints

    Best Practices to Improve your Engineering Sprints

    What is the purpose of sprints? We break work into sprints so we can more easily monitor, estimate and predict what we want to ship and when. Over the years, engineering teams have gotten used to moving quickly from one sprint to the next, but I think it’s important to be more thoughtful in how we plan and execute sprints.

    Running effective and successful sprints can define whether or not your organization is able to ship high-quality software fast and often. It also gives teams more flexibility and the ability to quickly change something that is not working or, on the other hand, to iterate on a successful technology. 

    After almost 20 years as an engineer, manager and as a CTO, I have found there are several best practices that improve the sprints that are part of my team’s software development life cycle. 

    Sprint Planning

    When planning a sprint, there are several things to keep in mind.

    Make Sure the Work Aligns with Business Priorities

    There are always a number of elements an engineering team can focus on during any given sprint. It’s fairly easy to lose sight of the business’ end goal. Make sure to gut-check to make sure that the sprint goals align with current business priorities.

    Ensure the Workload is Balanced From the Start

    This seems like an obvious statement, but make sure team members have equivalent workloads so you avoid burnout. It is also important to understand how different engineers on the team work—some people work best by selecting their own tasks with items that are unassigned, some pick up unfinished tasks as they finish others and some prefer to be assigned certain tasks because they have specialized skills. 

    Don’t Underestimate the Time That Interruptions/Additions Take 

    Along the same lines as the above, don’t overlook this for several reasons. Without adequate, data-driven insight into the actual state of each team member’s workload, it is almost impossible to gauge what’s happening in real-time. While business-critical sprints are assumed to take priority, side projects, Slack interruptions and meetings can easily consume an engineer’s day. Make sure to proactively check that each member has enough true deep work time without becoming burned out. 

    Don’t Underestimate the Time it Takes to get Things to a Done State

    Sometimes engineering teams only look at the development stage, but the full cycle required for the software to get to the end user might take longer. Allotting the right amount of time to get the job done across all phases of development will help avoid unexpected surprises. 

    When You Get to Mid-Sprint

    Monitor Unplanned Work

    Initial planning doesn’t necessarily take into account the bugs and fire drills that will likely arise after the sprint starts. Pay close attention to any mid-sprint additions and adjust as necessary. There may be team members who have cleared their sprint queue and are capable of taking on this work, but being aware of team members’ workloads at every stage is critical. 

    Do Daily Stand-Ups and Raise Flags as Early as Possible

    This will help anyone encountering roadblocks so a small issue doesn’t turn into a huge bottleneck. If you look closely at your PR cycle time, you will find a ton of insights about where roadblocks often happen. Using historical data during sprints can also go a long way toward helping identify potential bottlenecks, allowing you to proactively avoid friction by delegating reviews or setting out additional time for approvals. 

    Get Ahead of Burnout

    The tech industry has a turnover rate of 13.2%—pretty high when compared to other sectors. But the reality is that it can be really difficult to identify burnout without hard data. Don’t wait until people start speaking out about being overworked: Burnout prevention should be a full-time focus. Before sprints, make sure people aren’t coming in with high levels of fatigue and make sure that overtime doesn’t routinely increase throughout.

    Use Sprint Progress Estimations to Reprioritize

    Each sprint starts off with a set workload, but in reality, every sprint is dynamic. Take a look at each day’s progress and evaluate if your team is ahead of or behind schedule. If you are on track to finish a majority of the work, you can anticipate a successful sprint end. If not, don’t worry; you can always recommend that your team recalculate priorities. Engineering managers can take this time to load-balance some of the heavier work and shift goals to the next sprint as needed. Staying flexible while maintaining a clear view of what is actually happening during the sprint, will inevitably result in success. 

    At the End of the Sprint

    Do a Thorough Sprint Retrospective 

    As a team leader, you should constantly be thinking about how things are going and how people are feeling. If you can, use hard data to look at things like time allocation, PR throughput and any items added mid-sprint. Taking notes throughout the sprint will help remind you of any issues that came up, and you can discuss ways to improve for next time. Each sprint is a chance to learn; the more you learn, the better the sprint and the better the product will be for end users. 

  • Agile and DevOps for Kiosks

    Agile and DevOps for Kiosks

    Alan Shimel talks with Sam Zietz, the CEO of Grubbrr and Touchsuite, about how software for kiosks can use Agile and DevOps to keep up with a wide range of hardware profiles. The video is below followed by a transcript of the conversation.

    Announcer: This is Digital Anarchist.

    Alan Shimel: Hey, everyone. Welcome to another segment here on TechStrong TV. I’m really happy to be joined by a good friend of mine today. His name is Sam Zietz. Sam is the founder, CEO of GRUBBRR. You may not know the company but Sam’s going to tell you all about it. Hey, Sam. Welcome to TechStrong TV.

    Sam Zietz: Al, thank you for having me on. Excited to see you again.

    Alan Shimel: Yeah, it’s been – well, with COVID it’s been a long, strange trip. Sam, we’ve never had you on TechStrong TV though. We’ve never really talked about the company. Why don’t we start with that. Give people a little background on GRUBBRR and maybe a little of your own background.

    Sam Zietz: Sure. So GRUBBRR is the leader in self-ordering technology. When you talk about self-ordering technology you think kiosks, mobile ordering, online ordering. And in reality, this was going to naturally occur over the next 5, 15 years. That’s before COVID hit. COVID came about and acted as a huge accelerant. So if you think about how we used to go to a bank and there used to be a teller, that teller got replaced over a 5 to 15-year period of time by online banking and ATMs.

    We used to go to the airport, and in the airport we used to see a ticketing agent. That ticketing agent got replaced by a bank of kiosks or QR codes on your smart phone. Same thing was going to happen in many verticals like restaurants and hospitality and retail, and it’s going to be a combination of kiosk, mobile and online, and that online will be the company’s website or a third party delivery service like an Uber Eats, a Grubhub, or a Postmates. What ended up happening through is COVID, and now all of a sudden nobody wanted to interact with a cashier and cashiers didn’t want to interact with consumers either.

    This was all of a sudden a hazardous position and people didn’t want to shop any longer for that type of work. So that accelerated the move towards self-ordering technology. Then Joe Biden was elected president, and one of this four pillars was this $15.00 an hour minimum wage, and regardless of where you fall out politically, whether you think it will be successful or not, he has, and whether it’s passed or not the minimum wage has increased. And right now we’re in a labor shortage on that end of the market for a combination of factors, many of which is people don’t want to come back and do those types of jobs.

    And as a result, you’re seeing especially in restaurants that they can’t even open their doors many times because they don’t have the employees. So for all of these reasons we’ve accelerated the movement towards self-ordering technology.

    Alan Shimel: Yeah. I mean, Sam, this all falls under what we call digital transformation, right? And digital transformation, as you mentioned, was well underway before COVID, but we’ve probably accelerated four to seven years of digital transformation in the last year, year and a half. And the move towards automation, which is what this is, was again well under way. Quite frankly, when we look at should we automate everything – this is a question for academics – should we automate everything, should we automate only what should be automated or should we not automate everything for the sake of preserving jobs, and whether you believe in a Star Trek future where you get a basic, whatever they call it, wage where they send you money or not, quite frankly, if a machine can do it better they don’t take sick days, they don’t steal, purposely anyway.

    Hopefully not. They’re a better employee in some regards. So if there’s a machine that can do the job better, a futurist or technologist will say by all means the machine should do the job then because that frees up people to do hopefully higher-level work, more valuable work. Now are there some people that can’t do higher level, more valuable work and what do we do with them as a society – that’s probably a discussion for another day. But the fact is that today we have the technology, to quote an old TV show, to do this, and that’s really what you guys are doing there at GRUBBRR.

    Sam Zietz: Yeah. And historically this should’ve occurred years ago. It really was restaurant owners said, “Hey, you know what? We’ve been doing it the same way for the last 20 years,” and because labor was so cheap and because the minimum wage was so low, instead of investing in technology like almost every other industry has they said, “You know what? Let’s just throw cheap labor at it,” and that’s what occurred. COVID has exposed those businesses that had not invested in technology, and quite frankly rewarded those companies that had been proactive.

    They were extremely successful during the pandemic, the ones that had digital, they had an online presence, they had these self-ordering applications already in place. They were hitting record numbers. Everybody else either exposed or permanently shut down as a result. And by the way, these aren’t jobs that anybody wants.

    Even pre pandemic, the average turnover for a cashier in this country was 150 percent annually. I think Panera put out a stat before COVID that they were at 100 percent. And they were bragging that they were at 100 percent. So they were proud of the fact that every one of their employees –

    Alan Shimel: It was only 100 percent.

    Sam Zietz: Yeah. So these aren’t –

    Alan Shimel: There’s no doubt. Look, I think we’re both saying the same thing. This is the way it was going. We just had an accelerant take place. So let’s talk now about GRUBBRR. So you guys are filling that void by developing software solutions that allow for self-ordering. And whether people are ordering online via the internet for like a third-party delivery or in person, it’s still a software-empowered transaction. Like everything else, software is leading the world.

    In your case though, Sam, the challenge is that even with self-ordering it comes in a lot of form factors. It comes in a lot of shapes and sizes. And how do you make a software that is shape and size, form-factor independent? I imagine that’s got to be a challenge.

    Sam Zietz: Yeah. Well, we’ve got talented dev people, so from my perspective it’s easy. They do all the hard stuff.

    Alan Shimel: You didn’t write too many lines of code. I get it. But the challenge is there.

    Sam Zietz: We’re agnostic on the hardware side, so we work with all of the leading kiosk manufacturers too. They bring us in for the software. Why? Because nobody’s going to buy a kiosk by itself. A kiosk by itself doesn’t do anything. It’s the software that does it. Just like your Dell computer doesn’t do anything.

    It’s the Microsoft and the other applications that you put on it, that’s the reason you’re buying the computer. So it works the same way on the kiosk. We’re agnostic. So it’s funny, we built this beautiful restaurant of the future in our headquarters here that people can come in and they can see everything in a working environment, but then after they realize we want a kiosk or we want KDS or we want mobile, whatever it is, we take them next door into our showroom and there we have everything.

    In there they can decide do we want a wall mount, do we want a tabletop, do we want freestanding, do we want a 15-inch, a 21-inch, a 24-inch, a 32 – you know, whatever they want. Do we want it in a portable fashion where it’s truly mobile? All of those options are there, but the software at the end of the day, the core that’s running it, it’s the same. It’s the same software. And you’re right, that’s exactly what we spent our time building, was software regardless of form factor that will work the same.

    And the key to this is that somebody can operate it with no training. So you don’t need to – our competitors in the kiosk space, for the most part, fall into two categories. There are POS companies that turn their POS system sideways and say now it’s a kiosk. Well, that doesn’t work. Or they’re writing an app on an iPad, and that clearly doesn’t work either. So really the advantage in which you were talking about online and now going in store with a kiosk, one of the great things about that is the marketing team and the brand owners can now curate that user experience much like they do with the online experience.

    So you can control and provide that consumer the absolute best experience, where before you had a cashier and maybe you taught them at the end to say, hey, at the burger place, do you want fries with that? Okay, and more people buy fries because the cashier asked them if they wanted fries. However, if I’m ordering a salad the likelihood that I’m going to buy fries is pretty low. However, I might take a fruit cup, I might take a healthy treat.

    So being able to, again, pair offerings to the items that that consumer would like increases the average ticket and improves the experience for that consumer. You also eliminate all the errors, right? Because now you’re putting it in yourself. How many times have you gone and said, “Listen, I’d like a hamburger – I’ll have a cheeseburger with extra ketchup, no pickles,” so they go, “Got, it extra pickles, no ketchup.” You’re like, no.

    So there’s a whole host of reasons why that experience is better and could be faster, because maybe you could only afford to put one cashier in there, but you could put a whole slew of kiosks in there for a fraction of the price of one cashier. Well now you eliminate wait times and you can redeploy personnel into the actual prep of food and improving consumer experience.

    Alan Shimel: Yep. There’s another issue I wanted you to touch on, Sam, and that is really one of the inherent advantages of modern software and whether it’s an app, think of it as an app, or it’s just portable software that’s not particularly tied to – hardwired, if you will, into the form factor of hardware, the ability to update on the fly. Right? You change your menu items, you change your price, your running specials, or I mean really, really big functionality – we’re going to start accepting PayPal or Apple Pay or Google Pay or whatever the – we’re going to start taking bitcoin.

    I don’t care how you want to pay for it. But right? We need to update that. By having it all dependent on software, not burned into the silicon, we do that as well. How does GRUBBRR handle that? Like the whole update process?

    Sam Zietz: Yeah. So first off, we can update everything in real time as well as – so we can also be notified before even the client knows that there’s an issue with software. So maybe their internet went down because their provider – there was a problem, whatever. Many times we’re calling them and saying, “Hey, your internet’s out.” Oh, wow. Somebody was in the closet and unplugged the router or whatever and, boom, it’s back up.

    But you’re talking about an entire ecosystem. So because we have the ecosystem instead of a bunch of disparate pieces of technology in there, you’re absolutely right. You make one change. So price would be an example. If we want to change up a price it changes on a kiosk, it’ll change in the POS if they’re using a POS. It will also change on their digital signage, so the menu board above.

    Great. We want to change the price. Now let’s say you’re out of avocados. Boom. Now guacamole comes off the menu, so now it’s not even on the menu board if you don’t want it to be. If you run happy hour, great. At 5:00 we change the price, at 7:00 we change it back.

    There’s none of that whiteout or cardboard being take up with those changes. We’ve also integrated –

    Alan Shimel: [Inaudible comment]

    Sam Zietz: – AI and machine learning into it, so based on weather, time of year, time of day, order frequency, items will move up or down on the menu. Because we all go left to right. So what you want to do is you want to feature the most popular items or the items that that consumer’s most likely to order at that point of time to the left so that they can then make their decision quicker and move on in the ordering process.

    Alan Shimel: Excellent. Sam, we’re running low on time. I did want to mention, I know you guys recently announced a deal with Samsung, I believe, which is probably one of the largest – I’m going to assume one of the largest kiosk producers in the world – to be powered by GRUBBRR. Do you want to tell us a little bit?

    Sam Zietz: Yeah. So Samsung is obviously a very large company. I think they’re the sixth most recognizable brand in the world. They announced on June 30th that they’re entering the kiosk space. I would bet that in short order they will probably be the 800-pound gorilla in the kiosk space. What they’ve really done is they’ve democratized it, bringing that price standpoint down so it’s affordable to everybody.

    So you’re going to have a lower price. But traditionally kiosks were kind of hobbled together. They were a shell with a whole bunch of other pieces of technology kind of plugged in to make it look like it was one thing, but it was really five or six things. Samsung has truly built one slick unit that has it all built, which has allowed them to eliminate points of failure so the unit will be more successful. Obviously nobody builds screens as effectively as they do.

    In fact, they also have it so from touching it you can’t catch bacteria or COVID or any of that. But the exciting thing for us is, again, as great as a piece of software is, no one is going to buy a kiosk unless it does something, and that’s where we come in. We provide the true value proposition that powers it, and by buying the kiosk with our software you end up reducing OPEX or labor or at least solving for labor when you can’t even find it. More importantly, you increase your revenue.

    Typically, our clients see a 12 to 22 percent uplift in average ticket because people will buy more as a result of that curated experience, and we improve customer experience. So those are the value propositions there, but it’s exciting. When you partner with somebody like Samsung it’s not just here in the US. It’s worldwide. And they’ve got unbelievable reach.

    Alan Shimel: Oh, yeah. We’ve dealt with them. I mean, just another example, no pun and no irony intended, but software is eating the world. Hey, Sam, you know what we didn’t tell people? Where can get they go get more information about GRUBBRR?

    Sam Zietz: Our website’s probably the best place, GRUBBRR.com, G-R-U, double B, double R dot com. So, yeah.

    Alan Shimel: Excellent. Alright, man. Hey, say hello to the family. Best of luck with GRUBBRR. Maybe we’ll be ordering somewhere on some GRUBBRR software soon out there. Check it out. Full disclosure. I know Sam a long time, maybe 20 years. Happy to see him doing well with this and playing in the software game. Sam, be well, and we’ll speak to you soon. Come back on and keep us posted.

    Sam Zietz: Alright, Al. Thank you again for having me on.

    Alan Shimel: Alrighty. Sam Zietz from GRUBBRR here on TechStrong TV. We’re going to take a break. We’ll be right back with our next guest.

    [End of Audio]

  • IBM ‘is Ageist and Sexist’ | IBM Mainframe-aaS | IBM Vaccine Mandate

    IBM ‘is Ageist and Sexist’ | IBM Mainframe-aaS | IBM Vaccine Mandate

    Think! Welcome to Big Blue Week here at The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: IBM’s employment practices scrutinized, IBM z/OSaaS breaks cover, and IBM encourages staff back to the office.

    1. ‘Dinobabies’ and ‘Dated Maternal’ Workforce Woes

    First up this week: IBM’s defense in an employment age discrimination lawsuit looks to be shredded. Internal emails “appear to show high-level discussion about paring the ranks of older employees.”

    Analysis: Not a good look

    According to one of the plaintiff’s lawyers, unnamed “top IBM executives were explicitly plotting with one another to oust older workers … to make room for millennial employees.” On the face of it, they seemed to be doing the polar opposite of the Age Discrimination in Employment Act.

    Noam Scheiber: Making ‘Dinobabies’ Extinct: IBM’s Push for a Younger Work Force

    Top IBM executives were directly involved in discussions about the need to reduce the portion of older employees … sometimes disparaging them with terms of art like “dinobabies.” … Documents made public by a Federal District Court … show executives discussing plans to phase out older employees.
    …
    The newly unsealed documents, which quote from internal company emails … show top IBM executives specifically emphasizing the need to thin the ranks of older workers and hire more younger ones. [One] email refers to IBM’s “dated maternal workforce,” an apparent allusion to older women, and says: “This is what must change. They really don’t understand social or engagement.”
    …
    Adam Pratt, an IBM spokesman, [claimed,] “IBM never engaged in systemic age discrimination.” [He] said that some of the language in the emails “is not consistent with the respect IBM has for its employees” and “does not reflect company practices or policies.”


    Yikes, ageism and sexism? Experienced staff are valuable, says MrBanana:

    I did some time at IBM [and] remember being invited to a meeting with a new group, headed by a new “thinker,” who had something to present to myself and a colleague. We walked past the foosball table to the meeting room. Naturally the room had no chairs, and was covered in multi-coloured Post-it Notes. Once they had gone through their mood boards and the non-linear presentation of this new idea, I turned to the other old geezer in the room. “Didn’t we look at something like this 8 years ago?”

    “Yeah, we worked out that compute power just to run the UI for one session wouldn’t be available for a couple of decades.” The children were somewhat disappointed, and (fortunately) we were never asked back to their romper room.


    But experience can be both a blessing and a curse, learned aahortwwy:

    I experienced both sides of this in five years at a company in hypergrowth. … I learned so much at that job: … In year 1, I was determined to accomplish X and ignored the skeptical comments of some colleagues who’d been at the company longer than me. I failed and wasted some time. In years 4 and 5, new colleagues were determined to accomplish X and ignored my skeptical comments.

    2. IBM Wazi as-a-Service Confusion

    “Mainframe” is not a dirty word. It’s secure, vertically scalable, and it put the VM in VMware. Now, what IBM Wazi did for on-prem LPARs, Wazi aaS promises to extend to hosting z/OS instances.

    Analysis: Mainframe-as-a-service reinvented

    Mainframes and cloud-native DevOps need not be incompatible. To modernize legacy apps using hybrid cloud instances for dev/test—and even prod—can be a big productivity boost. But isn’t it weird that IBM spun out its mainframe-as-a-service zCloud offering only a few months ago?

    Mike Vizard: IBM Shifts Mainframe App Testing to the Cloud

    While many workloads over the years have been migrated off mainframes or simply retired, the applications that run on z/OS today are still among the most mission-critical that organizations run. … IBM this week revealed it has added the ability to set up testing environments on the IBM Cloud for z/OS mainframe applications.
    …
    IBM plans to make IBM Wazi as-a-Service (Wazi aaS) for z/OS generally available in the second half of this year, [which] will enable DevOps teams to self-provision an instance of a z/OS virtual server on a secure private space running on IBM Cloud. … At the same time, it will become easier to modernize mainframe applications by making them more accessible to other applications via a set of secure APIs … using a cloud-native set of DevOps tools that run in a Kubernetes environment.
    …
    Modernizing mainframe applications is a less complicated endeavor than migrating them. The challenge is that building, deploying, maintaining and securing all the APIs required to do so represents a significant amount of effort.


    OK, boomer, why are we talking about 1960s mainframes on DevOps.com? Here’s ArchieBunker:

    Mainframes are different beasts altogether, almost alien when you compare them to PC hardware. You have binaries written for the S/360 system in the 1960s? Good news you can still run them natively without a recompile.

    You think virtualization and containers are a hot topic? IBM was running multiple OS instances inside a VM in the 1970s. … The engineering is incredible.


    But z/OSaaS? It’s déjà vu time for Beleagured Greybeard:

    “Cloud Managed Services for z” AKA “zCloud” has been around for over 10 years. … IBM offloaded it with the Kyndryl spin, so its odd that they now want to re-invent it.
    …
    If it’s a strategic platform, why didn’t they hold onto it instead of spinning it off to Kyndryl? Sounds like yet more daft leadership decisions.

    3. IBM’s Return to Work Plan has Vaccine Mandate

    IBM wants to “keep employees safe” as they return to office work after being fully remote. In practice, this means a vaccine mandate.

    Analysis: IBM Digital Health Pass is divisive to vocal minority

    Although there are exemptions on medical or religious grounds, IBM has promised to place dissenters on unpaid leave. As you might expect, the policy has a few employees up in arms.

    Cassie Werber: How Salesforce, IBM, and HSBC are approaching return to work

    When the pandemic hit, and IBM moved 95% of its workforce to fully remote, it was pleasantly surprised to discover no drop in productivity. … Still, they planned to return to offices.
    …
    The company saw this as a way to keep employees safe. … One example … is its development of the IBM Digital Health Pass as an alternative to vaccine cards or paper vaccine certificates, which they use to make sure all the staff that come into their offices are vaccinated.
    …
    Arvind Krishna, IBM’s CEO, meanwhile came up with what he called the IBM Work From Home Pledge. [It] included commitments to flexibility, kindness, sensitivity to family, setting boundaries, and acknowledged video fatigue. It concluded with a commitment to “be connected.”


    Tricky to “be connected” if you’re not allowed back in the building. Anne D’Innocenzio, Heather Hollingsworth, Tali Arbel, Mae Anderson, Ken Sweet, Matt O’Brien and Dee-Ann Durbin:

    IBM software engineer Justin Albano said his company recently notified employees that it would be revoking badge access to workers who aren’t vaccinated or haven’t proved that they are. “We’re not able to enter an IBM office at this time,” said Albano.
    …
    He said some workers have successfully sought religious exemptions but “my thought process was I shouldn’t have to explain to my company my faith” to get out of a policy that’s “immoral in its foundation.”


    What’s the alternative? With a perspective, it’s Chris Haney:

    Or, get the vaccine. Protect yourself and the people around you. You people really are detached from reality.

    The Moral of the Story: Those who know do not speak; those who speak do not know.


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Marcin Wichary—cc:by; leveled and cropped (ask your parents)

  • IBM Automation Insights: Improving Observability and App Performance

    IBM Automation Insights: Improving Observability and App Performance

    Observability offers deep visibility into the internal state of complex systems in order to quickly identify and troubleshoot performance issues to keep up with customer expectations and business requirements. Observability enables better application performance monitoring (APM). So, how can DevOps teams gain full observability into modern applications? 

    On Feb. 23, experts from IBM and GitLab will come together at the IBM Automation Insights virtual event to discuss how to effectively observe, manage and maintain applications across platforms and how to improve application performance.

    The virtual event has been curated to create interactive sessions, led by DevOps experts who will deep dive on the latest innovations, conduct demos and share approaches that tackle challenges.

    IBM Automation Insights features a speaker lineup of distinguished engineers, IT leaders and executives, including:

    • Cosmo Schillaci, DevOps business leader at IBM
    • Jennifer Velasquez, Americas DevOps leader at IBM
    • Laurel Dickson-Bull, product manager at IBM
    • Kurt Dusek, senior solutions architect at GitLab
    • Jamie Coleman, software engineer at IBM

    Attendees will learn how Instana and GitLab can enable DevOps teams to accelerate CI/CD pipelines and how to build applications easily using cloud-native Java applications and microservices locally, with containers, and for OpenShift/Kubernetes.

    There will be a cross-platform panel discussion in which attendees will get a chance to interact with the speakers, ask questions on any of the strategies presented and discuss several topics, including cybersecurity, optimization for observability, vulnerability tool proliferation and more.

    For more information and to register, please visit the IBM Automation Insights website.

  • CircleCI Adds CI/CD Service for macOS Development

    CircleCI Adds CI/CD Service for macOS Development

    CircleCI today added a dedicated continuous integration/continuous delivery (CI/CD) edition of its namesake platform for developers building applications on the macOS platform.

    Jim Rose, CircleCI CEO, said this offering is aimed at developers that either want dedicated resources to speed up their build processes or have to address compliance and security requirements that prevent them from using shared infrastructure via the existing CircleCI cloud service.

    Most of the developers building applications on macOS platforms are creating applications that will ultimately be deployed on Apple iOS devices, noted Rose. Given the popularity of those devices, the number of organizations using CI/CD platforms to accelerate creation and updates of those applications has increased, Rose added. CircleCI estimated there are now more than 30 million developers building applications on the macOS platform that would benefit from a CI/CD platform running on dedicated bare-metal infrastructure accessed via the Amazon Web Services (AWS) cloud. CircleCI’s dedicated host provides secure access to 50% more cores and more than twice the storage capacity than is available via a CI/CD platform running on top of virtual machines.

    Other capabilities provided by CircleCI include the ability to monitor the performance of apps being developed for multiple platforms in the same pipeline. Many applications deployed on iOS are also deployed on other platforms such as Android, Windows or Linux, noted Rose.

    CircleCI also provides access to testing tools that the company recently added to its portfolio of DevOps tools. Those testing tools are especially critical given the stringent requirements developers face when trying to get approval to make their iOS applications available via the Apple Store, said Rose.

    Developer productivity is, of course, a major issue for organizations trying to strike a balance between automating software development tasks and the art that goes into building applications. While developers appreciate automation, they don’t necessarily want to feel they are working in a factory. Given the demand for their expertise, many developers will simply jump ship when they feel software development processes have become overly onerous. While many managers tend to believe they have a handle on measuring developer productivity, there is often too much focus on, for example, how many lines of code are written versus the business outcomes achieved.

    A recent CircleCI survey also found that more than half (52%) of respondents don’t allow their software developers to choose their own tools. While there is a clear need for a consistent approach to building and deploying applications, organizations will find it challenging to attract and retain the best developers if they don’t allow them to experiment with different tools.

    One way or another, however, the rate at which developers can build and deploy applications is rising as DevOps best practices are adopted. It’s not clear that organizations are going to be able to absorb all that software but, in the wake of the COVID-19 pandemic, the appetite for mobile applications that drive digital business transformation initiatives has never been greater.

  • The Benefits of Crowdsourced Testing

    The Benefits of Crowdsourced Testing

    Charlene O’Hanlon and Rob Mason from Applause talk about how crowdsourced testing is a highly effective method for testing applications and its benefits to DevOps teams. The video is below followed by a transcript of the conversation.

    O’Hanlon: Hey, everybody. Welcome back to TechStrong TV. I’m O’Hanlon, and I’m here now with Mason with Applause. Rob is a return to TechStrong TV. Rob, welcome back. So good to see you again.

    Mason: Thank you, and thanks for having me on again.

    O’Hanlon: All right, so we had a great conversation, what was it, a couple of months ago?

    Mason: Yeah, a little while ago.

    O’Hanlon: I think so. Yeah, yeah, so time flies when you’re having fun, I guess. But, at that point, I did invite you to come back on so we could talk about the topic of continuous testing, which I think is something that is gaining a lot of traction in the DevOps space, and it’s certainly garnering a lot of conversation. Sorry, not continuous testing, crowdsource testing. Oh my goodness, my brain this afternoon, what am I going to do? Sorry about that. Yes, crowdsource testing, and it, too, is something that a lot of DevOps teams or a lot of organizations are considering as an alternative, I guess, or maybe a complement to the way that they’re doing testing these days. You are the expert in the field, so I want to talk to you about it and kind of get a download, if you will, on crowdsource testing. What you’re seeing in the space right now, who’s talking about it and what organizations, how they could benefit from it?

    Mason: Right. Thank you. Yeah, so Applause is the largest crowdsource testing company in the world. _____ [inaudible due to audio distortion] quality testing for some of the largest companies in the world, like Google, PayPal, Walmart, Amazon, Facebook, a lot of those different types of companies, and we do that digital quality testing by using a crowd. Applause actually started off as uTest about a decade ago, _____ the name uTest, so anyone could test. Basically, if you go to uTest.com, you sign up to become a tester. Today, that community is over 750,000 people from 233 different countries around the world, and they’re making money from testing products from the company, or the companies I mentioned before, as independent contractors. They bring their own computers, they decide what they work on, what they don’t work on, they bring their gear, they decide when they want to work.

    You start off by going to uTest.com, you sign up, and then we put you through all sorts of training. We have what we call uTest Academy, lots of great videos, training courses. It’s all free. You don’t have to pay anything for it. Joining the community is free. Once you join the community, you start to level up your skills as a tester, and you become better and better at writing great bug reports, with _____ the attachments, the steps to reproduce them, learn now to execute test cases, how to do payment testing or accessibility testing. We train you on all sorts of things around testing because we know there are not enough tests going on. We know there is not enough quality in our products today. It could always be better, and so ‒

    O’Hanlon: Yeah ‒

    Mason: Go ahead.

    O’Hanlon: No, no, you go ahead.

    Mason: I mean, with how fast the technology landscape is changing, you guys see it firsthand every day, it’s very easy to switch from one bank to another or from one app to another, and so it’s very important that that first experience is first class. More and more big brands are very much aware and very in tune with those first experiences and the ongoing experience of their users, so we use a very large community of real people in the real world, around the world, every country you can imagine. You know, 233 countries, as I mentioned. They level up over time to become more and more expertise. As they progress in their learning and participation, they get invited into more projects, so they get opened up to more ability to earn money, basically. When they tell us ‒ voluntarily, of course ‒ about themselves more, what devices they have, do they have an iOS device, android device, what kind of laptop they have, they get invited to more projects.

    The more they tell us, the more that we can match them to projects. We use machine learning algorithms to match testers to projects based on past experience, so we’ve got a data science team, and machine learning algorithms that match based on performance. It’s not based on demographics or race or ethnicity, it’s all based on what you can do, and what you can do is behind a screen. The machines don’t know. It’s based on did you turn up on time for the project, did you execute ‒

    O’Hanlon: Yeah.

    Mason: ‒ _____ well, were your bugs well written? It’s all based on pure scoring, which is great. Some of our projects do require physical presence, it will be like a test for Starbucks, and you have to go in and buy a coffee and prove that you can order it online and pick it up, and all that sort of stuff. But, most of them are remote, and you can work wherever you are, from your couch with your dog sitting on your lap, and still execute great testing for the customers. That’s what the community is really a lot like. Some of the projects do have demographic requirements, but mostly those are in the medical space or in fitness. For instance, we’ll be looking for a specific age class for testing online fitness classes for some of our customers, and so sometimes there are demographic requirements. But, for the most part, it’s pretty wide open for anyone to participate. What’s nice is in some of these communities, or sorry, in some of these countries, it’s hard to find jobs. Especially during COVID, we had about 50,000 coming in per month during COVID.

    O’Hanlon: Wow.

    Mason: That’s trailing off now with less people coming in, but with all the people that were out of work, you can make money at Applause. Some people do it as a moonlighting job on the side, just to earn some extra money, maybe some play money. Some people make a living out of it. As an independent contractor, again, they make their own time and everything, but they’re making a living off of Applause doing testing for these customers full time. Again, anyone can do it, so if you’re in a country where maybe women can’t get jobs because it’s old-school and it’s sexist there in that country or what have you, or you’re in a demographic that it’s harder to find work, uTest doesn’t care, like bring your skills, bring your abilities, and anyone can get work. If you look at our community, we’re incredibly diverse across, again, all those countries, but also in our global community, we have a high ratio of women in the community.

    Not everyone, obviously, tells us what gender they are, but 35 percent of our global community are female. We have top-rated testers in the community, 41 percent of our gold-rated testers, that’s the highest level tester we have, 41 percent of those are women, 40 percent of our silver testers are women, 40 percent of our bronze testers are women, so you can just see that across the board it’s, again, based on performance. It has nothing to do with gender or anything else, which just levels the playing field for a global economy, and for global testing of enterprise products. I’m sorry. I’ve been talking ‒

    O’Hanlon: No, that’s awesome. I love to get the download, so it’s like you kind of take it from there and just run with it. Wow. That’s really cool stuff. I think that the last time we spoke, you guys were working on a couple really cool projects, but it seems to me that this would be a great technology, or a great service, if you will, for organizations that really don’t have maybe an in-house testing environment, or maybe organizations that do have an in-house testing environment but really kind of want to go that extra mile and make sure really that all their bases are being covered. How much of your customer base, then, are those companies that maybe already have testing abilities in house but they’re just looking to kind of up their game, if you know what I mean?

    Mason: I would say probably almost all of them have QA ‒

    O’Hanlon: Really?

    Mason: ‒ of some sort, but the difference is QA in a lab with a relatively small team is very different than real world QA. If you have some simulators or some real devices in the lab, some browsers based in your lab, it’s very different than browsers around the world. Probably a lot of the enterprise companies have their sites localized into different languages. Probably in your lab in your local team, you may not have all those native language speakers, so that’s _____, so localization is one. Currency testing, payment instrument testing, all those things that happen in the real world, you know, poor networks, flaky networks, different flavors of Samsung devices or android pixels, all that sort of stuff. A lab test is great, and it’s a great way to make sure the base foundations of your applications are good on your mobile apps, but it’s very different when you get out in the real world, and that’s where the customers judge you.

    They don’t judge you for your lab experiences, they judge you when you get out in the real world, that’s important. Almost all of our enterprise customers have their own QA teams, but they admit, and these are big companies ‒ it’s Google, it’s Amazon, it’s Facebook ‒ they admit that they don’t have a big enough team, even at their size, to cover what they really need to do to test their applications. You need a company that can bring resources from around the world, speaking all the different languages, all the different currencies, all the different types of devices. I think the last time we counted, there were about four million different devices in the hands of this community, because everyone ‒

    O’Hanlon: Wow.

    Mason: ‒ has got a phone _____ [inaudible due to audio distortion], what have you, so you take all those devices and you bring them to bear on some of these huge problems of the large enterprise, and it changes the dynamics. It’s daunting if you want to build a product and launch it globally and have it be successful, and not know that it’s going to work the first time. Even if you get a great, large outsourcing firm based in India or over in Europe, they’re based in one area, in one region. They all speak probably the same language, a very similar language, they’re in the same area, the same currency. How do you get that global test? The answer is crowdsource testing. There is no other way to do it. It’s just not practical any other way. If you have to manage those masses of people, like how do you make sure that it’s not just total chaos, and that’s what Applause has really specialized in for more than a decade, is making sure that we give great quality in what we give back to our customers, it’s actionable, it’s precise. Every bug report from us has to have an attachment and a video, it has to have reproduction steps.

    Again, we don’t just do bugs for customers, we do all sorts of testing, like we test smart phones, we test smart TVs, smart cars, IoT, AR/VR. We do machine language, data collection, data collection from machine learning ‒ I said machine language, but machine learning ‒ so all sorts of things we do for our customers using this global community. When you think about it, I have 750,000 people I can bring to bear on your problems to improve the digital quality of your applications. Sometimes you need more data, sometimes you need _____ [inaudible due to audio distortion], sometimes you just need a lot of people to bang on a website, sometimes you need people to go deep and look for those hard-to-find bugs. It varies greatly, so we cover pretty much every area that you could expect of digital quality testing with that crowd, and we _____ [inaudible due to audio distortion] so you don’t have to think about it. That just allows us to scale, but what we’re really proud of is just how successful our community has been, how diverse they are, how broad and how deep they get.

    They are true experts, and a lot of these people in the community, some of them are just people moonlighting after working some other kind of job, but some are true QA experts that have been in the industry for decades and have just chosen this new way of life, because it’s flexible. You always work at home, or wherever _____ [inaudible due to audio distortion]. You can be on the beach and doing the job. It doesn’t matter to us, as long as the job gets done. It is the new world, and as COVID removed some jobs for some people, at least temporarily, we got an influx, but also people are now looking for that new way of working. We see the industry moving out of the offices, and that’s what we’ve been at Applause for a decade. Our community, the people doing the testing for us, have been outside of offices since we started, and so it is the natural way for us. It’s like welcome to our party, but now you have to figure out how to work ‒

    O’Hanlon: Right.

    Mason: ‒ _____ [inaudible due to audio distortion], and we’ve been doing that _____ for a long time.

    O’Hanlon: That’s very cool. On average, how many testers do you have on a particular project?

    Mason: It varies greatly, like sometimes we’re doing a global payment instrument test that’s making sure something like an android pay or an Apple pay works across the globe when they’re rolling out a new feature. There, we have to engage a lot of people to do that, to make sure that all the different banks, all the different currencies work around the world. Sometimes, it’s a point release, a patch release from a customer with a new version of their mobile app, they made some targeted changes, they’ve told us what they are, some regression testing around the area, a specific testing in that area, so it’s a smaller team, so anywhere from a handful of people to thousands of people at times. We don’t generally go much larger than thousands on a single project unless there is a very specific need. We do have some that spike up there in very large numbers, but, for the most part, we’re in the tens to hundreds range, less so in the thousands on a given, specific, what we call test cycle.

    For most of our customers, in fact all of our customers, it is subscription-based at this point, so they’ll sign up and have us test a digital asset over a long period of time, and so those assets go through their cycles. The first release is different from a patch release versus a new, major update, a new, major feature. Again, the size of those teams vary over time, and we can pull as many as we need into it. If there are different skill sets, like maybe a customer has a mobile app and now they’re focused on accessibility, and so they’re going to do a release 100 percent focused on making their application more accessible, we’ll bring in an accessibility group from our community and have them focus on accessibility. These are people not only that know accessibility testing well, but they’re people with real accessibility limitations, so it’s real people with real disabilities testing your application to make sure it works in the real world, because there is nothing more frustrating than trying to use an application that claims it supports you and then finding out it doesn’t.

    O’Hanlon: Right, right. Well, it sounds like great stuff, and you guys have been doing this for ten years or so?

    Mason: Yeah, just over 11, coming on 12 now.

    O’Hanlon: That’s great. How did you guys come up with this idea? I mean, I think it’s really kind of fascinating because it really does address a real problem that is in the industry, and that is being able to test all the things, if you know what I mean. Is this something that your founders just decided one day like, “Well, you know what? The best way to do it is let’s just ask everybody to do it, and everybody to help us out.”

    Mason: Yeah, our founders were engineering managers from the past, they were engineers, and they had the experience of building and testing applications in their own labs, and then going out in the real world and finding out that things were different when they got out to the real world. They said, “We have to change the model. The model isn’t going to work, and it’s only getting worse over time.” They said, “We have to go and turn this model upside down,” and they were far ahead of their time when they launched the company. Now there are some other much smaller companies out there doing some similar things, but they’re much more niche. But, they saw what was coming. The real world matters, and the differences are just ballooning and you just cannot keep up, even as a large company in the lab, so great foresight there. It took a lot of work to get to the point where you can manage that many people and have it be successful.

    Obviously, if you just put random people on a project and say, “We’ll do stuff for you,” you’re going to get terrible results and just waste your time. Obviously, places like Walmart and Facebook and others like that will not tolerate that, so you have to have a very good system and process around it. We have a special enterprise grade platform that our customers and community interface on, so we do software development to make our own platforms to allow those groups to interact with each other, and then we’ve got a great team of community management around them, recruiting people, training them, helping them learn how to be great testers. Ultimately, we’re making careers for people. We’re helping people make livings working for themselves wherever they want to be.

    O’Hanlon: That’s great, and we’re going to need to wrap this up here very soon, but have you heard of any stories of folks that joined the platform to do testing here and there, and ultimately decided that, “Hey, this is what I want to do for a living. This is what I want to do all the time,” and they ended up actually doing a career change because of this?

    Mason: Absolutely. In fact, we’ve hired several of those.

    O’Hanlon: Oh, really?

    Mason: It’s not generally the model, to hire from the community, because the community is our model, but sometimes they rise up so far and they’re such stars in the community, and we’re looking for additional help in managing the community, or elsewhere in our business, that we find them and bring them in. A lot of our great employees we have at the company are people that we found through the community, because they’ve come in and shown what they can do. There are others that have gone off and moved on to full-time positions in companies. Some of them do have full-time QA positions, and then work for us moonlighting. Some of them, like my daughter is in the community, anyone can be in the community. My daughter is in the community, and she makes a few bucks on the side and buys herself snacks or whatever _____.

    So, it really is a very diverse community of everyone who is just getting started or learning about testing, giving an appreciation for what it takes to make digital quality products, and what companies go through to test them, so I think it’s a great experience for everyone. I encourage every single person to sign up at uTest.com. Again, it’s free. We pay you, not the opposite, so you can make money. All you have to do is experience some digital quality products, give us good feedback, and help us make these products better for the planet, basically.

    O’Hanlon: That’s awesome. Well, Rob, thank you so much for walking me through this. I love the idea of pretty much crowdsourcing anything, because I truly believe in the collective intelligence of the populations. Anything that we can do to kind of encourage that and really take advantage of that at the end of the day, really capitalize on that collective intelligence and the ability of crowds to really do some good, I think is good for everybody. Thank you, again, and it was great having you on TechStrong TV again, and hopefully we’ll see you again in the future.

    Mason: My pleasure. It was great talking to you.

    O’Hanlon: All right, everybody, please stick around. We’ve got lots more TechStrong TV coming up, so stay tuned.

    [End of Audio]

  • How the Cybersecurity EO Impacts DevOps Teams

    How the Cybersecurity EO Impacts DevOps Teams

    Eric Greenwald, general counsel for Finite State, talks with Mike Vizard about how the executive order for securing software supply chains issued by president Biden will impact DevOps teams. The video is below, followed by a transcript of the conversation.

    Announcer: This is Digital Anarchist.

    Mike Vizard: Hey, guys. Thanks for the throw again. We’re here with Eric Greenwald who is the General Counsel for Finite State, which specializes in cybersecurity around embedded systems. Eric, welcome to the show.

    Eric Greenwald: Thanks very much. Very happy to be here.

    Vizard: The Biden administration is going to save us all from ourselves. We’ve got this whole effort around software supply chain going on. NIST is now trying to define what a critical component of a software supply chain exactly is. That has been a mystery to many of us for a long time. So maybe you could sort this out for us a little bit. What exactly do they mean and what are they looking for?

    Greenwald: Sure. Well it’s a little bit complicated because I think the conventional understanding of what is critical software is a potentially very broad and malleable concept. What NIST is trying to do is narrow the focus a little bit mostly so that it can phase the implementation of the executive order. I think if you look at some other efforts at imposing standards for software security or security in general that there’s been a rollout and the standards weren’t necessarily drafted in a way that made sense or was easy or low cost to implement, and so what they’re trying to do is avoid those pitfalls and do a rollout that starts with a narrower scope but then eventually expands I suspect eventually to pretty much the majority of the universe of software products that the government buys.

    Now where is it going to stand when the executive order actually begins implementation? That’s what this initial definition is meant to establish. But if you look through what NIST has put out so far, it’s not 100 percent clear, and I think we’re going to get some additional guidance both from NIST and also the executive direct some other departments and agencies to put out some additional guidance. The NIST definition that they put out back on the 25th of June includes a proposed set of categories of software that would be defined as critical under the executive order, but it’s not a definitive list.

    I believe it’s CISA from DHS that’s charged with putting out a definitive list somewhere down the line in the implementation of the executive order. I can’t remember the exact date. But in another 30 or 60 days we’ll see a definitive list of those categories from CISA.

    Vizard: Well, are they doing this out of their concern for us all or are we going to get in trouble at some point? Is somebody going to call me up as a developer and say, “Hey, you’re not protecting these critical components and you’re about to be fined or something bad is going to happen”? So what’s the stick that goes with the carrot.

    Greenwald: So the first stick that gets wheeled in is by the Office of Management and Budget, that is going to use it to compel the departments and agencies that are buying software from vendors out there in the private market that they can only buy software that meets the definition of critical software if it has passed the technical requirements that will eventually be promulgated under the executive order. So the first line of people potentially getting in trouble are the departments and agencies for failure to actually implement the EO’s requirements. Now indirectly, what’s going to happen is as the departments and agencies attempt to enforce the standards under the executive order is that they will select software vendors based upon whether those vendors are meeting the technical requirements. So there’s this trickle-down effect that again it’s going to go through those vendors that are selling directly to the government, but then it will also impact all the various companies out there that are providing components to those vendors that sell to the government.

    So it’s not just going to impact vendors who sell directly to the government. What’s more, it’s going to go even further beyond that because I think especially if people start to think that the standards promulgated under the EO are sensible, are a good way to actually impose tighter security requirements, you’re going to see them adopted more broadly and made mandatory not just for those companies that are selling to the government. But you might see industry groups adopting them, you might see a lot of companies as they are buying software telling their vendors, “Hey, we want you to meet these requirements that have been promulgated under the EO. They’re not required by law to, but we’re requiring you under our contract to meet them.” And you might even see insurance companies start requiring them as a prerequisite for obtaining cyber insurance coverage.

    Vizard: I’m not a lawyer, but sometimes when I do look at these regulations I do notice there’s a common theme between them, almost to the point where I feel like somebody kind of copy and pasted something from one into the other. Is that how things work? People kind of look at one agency and say, “Well thanks for taking the lead on this,” but then they just implement it into their regulatory framework?

    Greenwald: So the best analog to offer on that is if outside of the cybersecurity realm is environmental standards, particularly auto emissions standards. You look at California that leads the way in auto emission standards that it imposes on automobiles sold in California. You see a lot of other states adopting those standards as the model. Not all states, by any measure. But the interesting thing is that car manufacturers can’t just meet those standards to sell automobiles in California.

    They’re going to meet those standards for all automobiles sold in the United States and probably other parts of the world too. Switching to an analog from the security world, you look at the NIST framework. The NIST framework, all voluntary. Nothing about it is mandatory. It’s also not specific technical requirements. So it’s a rough analog.

    But if you look at what happened with the NIST framework you see all sorts of contracts that require compliance with the NIST framework. You see other industry groups or even – and again, in the insurance industry, a lot of insurance companies want to know before providing you with cybersecurity insurance coverage do you meet the basic requirements or the guidelines promulgated under the NIST framework. So I think to your copy and paste point, I think yes, it definitely does happen. And my first hope is that the technical standards promulgated under the EO will be sensible and actually promote better security. My second hope is that if they do they get adopted beyond just procurement for the federal government.

    Vizard: Do you think that the compliance process can be automated as we go forward or is this going to be a lot of manual reviews of code? We’re all hoping or looking for – compliance-as-code is kind of the buzzword of the day in the DevOps space. Do you think we can code our way through this?

    Greenwald: So I’m going to avoid getting too deep into technical issues because I’m a lawyer first and a technology guy second, but I’m more technology policy than engineering, for example. My feeling is that the only way we’re going to be able to implement the technical requirements under the EO in a manner that is efficient and effective, and not too costly I should add, is by automating the process. I will note that part of the approach that Finite State takes to this is automating those software reviews, the scanning for vulnerabilities and assessments of what is actually in the software because that is – number one, doing it once manually is labor intensive. Under the EO, my expectation is that you’re going to see requirements to do it repeatedly, and I think the ideal is doing it with every update or at least even if the updates have a long periodicity doing it more frequently than the updates.

    Or let me just say this – I think that the companies that are able to run their security checks on their software regularly are going to be at a competitive advantage as they approach procurement officers in the federal government to buy their software. And the same thing I think is going to apply more broadly across the industry, that the more frequently you can do these security scans the greater your competitive advantage. And there’s just no way to do it frequently without automating it.

    Vizard: We talk a lot about DevSecOps in our world these days. I wonder if this is just going to force that issue where I might as well start biting this bullet now because one way or another the government’s going to kind of drive me towards it whether I like it or not, so maybe now’s the time.

    Greenwald: Yes. Absolutely in two things. The first is that the EO specifically calls for the promulgation of technical standards for secure development of software. The second is, to your point, my view, what I advocate for companies that are looking at the EO and saying, “Hm, where’s this going, who is this going to apply to?” that if you sit around and wait for it to become mandatory for your company and/or your products, that you’re going to be way behind the power curve and that these companies that are looking at this need to start developing a plan, an aggressive plan, to develop securer software development ops, to implement a software bill of materials – and again, we’re going to see very soon where the technical standards for the software bill of materials will land.

    So people I think will be able to start jumping on that soon. And third, having transparency about your security measures, whether it’s software development or testing, that are accessible to anybody buying your software or using your software. And just to add a last point on there, that being able to do that on a repeated basis where the test results can be validated is going to be very important for companies who, again, want to maintain a competitive advantage in the marketplace.

    Vizard: And I suspect that just because I wrote my code before the rules were actually implemented won’t be much of a benefit to me because they’ll expect me to go back and fix stuff that’s in production.

    Greenwald: That’s absolutely right. So number one is that for any software that the federal government procures following the date that the EO is promulgated, which was I forgot now, like early May, any software procured after that date forward has to meet the EO’s requirements, again, if it falls under the critical software definition. I’ll say something about that in a moment. But the other is that eventually the EO expects OMB to impose those same requirements on critical software that is legacy, that predates the promulgation of the EO.

    As a reference, quick note on if your product meets the definition of critical software. As I noted initially, the definition is being drawn somewhat narrowly. Frankly, I think it’s fairly broad as put out by NIST just a week ago. But again, it’s the initial phase and the definition, the scope, is going to get broader, and I would expect it to end up being quite broad at the end of the day. What day that is, is hard to say. But a key to note is that in the frequently asked questions in the NIST definition that they issued last Friday, they specifically say if a procurement officer wants to apply the EO’s requirements to something that doesn’t meet the definition of EO critical software, they can go ahead.

    There’s nothing that prohibits them from doing so. And if I’m a procurement officer and I’m thinking, okay, I’m buying software today. I know that eventually it’s going to have to meet the EO’s requirements. Why am I going to buy something where I’m worried about it meeting the EO requirements? I’m going to buy something today that is going to give me no headaches down the road in terms of whether it actually meets those requirements.

    And what’s more, if you’re buying software, you’re looking at 10 different products and three of them meet the EO’s requirements, to me that’s an indication that those products are safer. And so I think that you’re going to see procurement officers for a number of reasons gravitating towards those companies whose products meet the EO’s technical requirements.

    Vizard: Yeah. I guess I’m a little more worried about how granular it might get, because we have all these libraries and microservices that people build and then they deploy, and the definition of how that thing may become critical could change over time and what I thought was inconsequential suddenly is critical.

    Greenwald: Well again, and as I said at the top, to me if you’re trying to establish a conventional definition of what’s critical, it’s really hard to circumscribe that narrowly. Anything that has access to a system that you might consider critical, should be considered a critical component. If you’re thinking about what’s the conventional understanding of critical, yes, it’s enormously broad. Again, NIST is trying to start a little bit more narrow and expand later, but I don’t think anyone can feel confident if they have a product, unless it’s something that where there’s just no argument that it could ever be construed as critical, that they should be looking at the EO requirements and figuring out a way to comply with them – well I was going to say comply with them now – we don’t have the requirements yet.

    The technical requirements haven’t been promulgated. But as we start to see which way the wind is blowing, where these are likely to land, companies should be looking to develop a plan to meet those requirements where we expect them to land. And again, not just those companies that sell software to the federal government. So the granularity of the definition, to me I don’t want to go so far as to say as it’s irrelevant, but I think that software vendors of all stripes, whether it is the obviously most critical component of an operating system or something that’s not so obvious, that they should all be thinking about how they’re going to comply. There’s a separate question about granularity, and that is how granular are the technical requirements themselves?

    That remains to be seen. I don’t have a good sense as to how specific and how technical NIST and the other agencies are going to get. But I have a lot of confidence in NIST because they have very good experts in house and they also have spent a lot of time developing relationships with the private sector, making sure they understand what the private sector needs, what really hurts the private sector – and I should say hurts them unnecessary, because there’s no way we get to a more secure software landscape, whether for the federal government or more broadly, without enduring some pain and some cost. You can’t wish your way into better security.

    Vizard: Cool. What’s your best advice to folks then right now? What should they be thinking about, what should they be doing, and how do I get ready?

    Greenwald: So the first thing I would say is if you don’t already have a software bill of materials for your products, that you should start moving in that direction and quickly. As I said, I think it’s just in another week or so that we’re going to see the standards for an SBOM promulgated. When that happens, it’ll give you a very specific target to go for. But there’s no reason not to start now in moving towards that. So that’s something that I think people can start working on today. I think it’ll be a little farther in the future before we start seeing again the specific technical standards being promulgated.

    But I would be, number one, keeping an eye on what’s happening with the various bits of guidance and rules and regulations that are coming out of the EO implementation process, so tracking it closely. Second, if you’re looking to what you can do right now, start looking at generally accepted practices, for example, for secure software development. Because I strongly suspect that where NIST lands is going to be pretty close to what are generally accepted best or very good practices with respect to secure software development, and the same thing for security testing. So we’re a little early in the process to be able to say do A, B, and C specifically.

    Again, within a week or two we should have a little bit of that specific A, B, or C with respect to the software bill of materials. It’s going to take a little bit longer before we get those technical specifics for software development and testing. But again, I think that we can expect NIST to hue to what are already generally accepted as good or best practices in those arenas.

    Vizard: Alright. Like it or not, secure software is on the way. Eric, thanks for being on the show.

    Greenwald: You bet. Thank you for having me.

    Vizard: Alright. Back to you guys in the studio.

    [End of Audio]