devopsy.kinsta.cloud

Category: Features

DevOps Chat: Crowdsourced Continuous Testing with Rainforest’s Derek Choy

Scaling your testing is the scourge of many software teams. While test automation and continuous testing have made great strides, it still is difficult to automate every QA test. Also, engineering resources who can script these tests are difficult to find. Enter Rainforest.

Rainforest has a unique crowdsourcing solution to continuous testing. It combines crowdsourcing talent with automation to deliver a high level of quality assurance on a continuous basis.

In this DevOps Chat, we speak with CIO Derek Choy as he explains to us why Rainforest in some ways can be the Uber of QA. As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

Transcript

Alan Shimel: Hey, everyone, it’s Alan Shimel, DevOps.com, and you’re listening to another DevOps Chat. Today’s DevOps Chat features a first-time guest on DevOps Chat. He’s Derek Choy, CEO at Rainforest, or Rainforest QA, as some may know them. Derek, welcome.

Derek Choy: Thank you. Just a little correction: I’m CIO at Rainforest.

Shimel: What did I call you?

Choy: Responsible for primarily the technical side of the functions here.

Shimel: Did I call you “CEO”?

Choy: I think you did, but I just want to make a little correction there.

Shimel: You know, it’s 2019; that’s my prediction.

[Laughter]

Choy: I don’t know that I would support that prediction, but thank you. [Laughs]

Shimel: Well, I _____ – CIO’s enough. But, anyway, Derek, thanks. Yes, it’s Derek Choy, CIO at Rainforest. So, Derek, you know what? I didn’t make the mistake on purpose; it was an honest mistake. Good. That leads us – why don’t we talk a little bit about Rainforest for our audience who may not be familiar?

Choy: Absolutely. So Rainforest is a testing platform. What we do is we help our customers with their software testing. You know, as you know, we’re moving – a lot of companies are moving towards a rapid development cycle, CI/CD, frequent releases and stuff. Testing becomes much of a challenge for a lot of our customers because now they not only have to do testing, but they have to do it fast. They have to have accurate results and stuff.

So Rainforest is actually a crowd-based testing platform that allows our customers to simply input test cases in plain English, so no Selenium or any sort of programming language, and then we can actually execute them as if they are automated test. We’re fully integrated into any CI/CD tools, like Jenkins or CircleCI and so on, so you can actually trigger a Rainforest test. Typically, we can return results in 30 minutes or so, and you can actually test across many different platforms, like different browsers, different OS’s, and the same for mobile as well. And so that allows our customers a wide variety of options, in terms of where they want to run their plain-English test cases and when they want to run it – it’s on demand. And we typically return results in 30 minutes, fully integrated into an automated release cycle.

Shimel: Great. Excellent. And, Derek, how did it come to be that you became the CIO of Rainforest? Let’s hear a little bit your own personal journey.

Choy: Cool. Oh, thank you. So I have been always involved in the product engineering side of things. I started off my career earlier with Accenture. Spent a number of years at eBay. And then my last company at Aria Systems, building SAS-based billing system, also running their engineering and product function. So, coming to Rainforest, I brought my experience and knowledge in this space, to help build the engineering and the product organizations. Now here, besides product engineering, I’m also responsible for our professional services team and our client success team. So essentially kind of helping build out product and making sure that our customers are successful in using that.

Shimel: Absolutely. So, Derek, you know what? Here we are in December. We’re looking at 2019.

Choy: Mm-hmm.

Shimel: I probably have spent the last three weeks – if I tell you I’ve spoken to dozens of vendors and C-level execs at large organizations, and, to me, the real story of 2019, as well as what’s emerged as the story of 2018, is around – you know, you wanna call it “digital transformation” and “DevOps,” _____ say great, but, really, down to these two things: companies are – they need scale in what they’re doing today, right?

Choy: Yeah, absolutely.

Shimel: When we’re talking about data and computation and transactions or whatever the metric is that’s driving your company, everybody wants to do it at huge scale, like unprecedented scale that, 20 years ago, you would’ve went – you know, I’m just reminded of Carl Sagan saying, “Billions and billions of stars,” _____.

Choy: [Laughs] Yes, that’s right.

Shimel: That’s the scale. And the only way to achieve that kind of scale is automation.

Choy: Yes. Absolutely.

Shimel: So these are two kind of twins, co-joined at the hip – scale and automation.You can’t have scale, I think – well, you can’t handle scale without automation.

Choy: Absolutely. Yeah. And – sorry.

Shimel: Go ahead. No, no, go ahead.

Choy: I –

Shimel: You know, to me, that’s it right there. That’s the crux of it.

Choy: Exactly. And, Alan, I would say that I’ve never been more excited, right, than where we are right now. I think we are definitely at a very exciting time and, just to kinda set the context and maybe kinda recapping a little bit of what you just said, right, the world is very different now, even compared to five years ago. Right? A while ago, you would hear companies having a quarterly release. Some would even have an annual release, if you remember, especially the bigger companies, right? You know, release cycle was a lot longer, takes a lot more time to get things out to the market. Today, release cycles are typically maybe in a month, right? Maybe two weeks and, in some cases, days or even multiple times a day. Right?

And exactly to you – and the interesting thing is that the world has got accustomed to that and has come to expect that, frankly. Right? Think about the apps that you’re using. If they come back to you and say, “Well, you know what? We update our apps once every six months,” or something, you’re like, “Wait, six months? Wow, that’s unheard of.” Right? “I was expecting a weekly update or at least monthly or something.” So the world has actually comes to expect that.

Now the challenge, exactly as you pointed out, is how do companies deal with that? Right? How can we scale our organization through automation, among other things, right, to make sure that we can keep up with that pace? I think that that is the challenge with a lot of technical leaders, is we understand what the companies are looking for, understand what the consumers and our customers are looking for, right? But how do we do that? Right?

And there’re different aspects of it, Alan. I think that there is definitely a development aspect of it, right? How do we scale development? How do we do rapid development, Agile, sprint? You know, all these things. Things have evolved a lot over the last, I would say, ten years, in terms of faster development. There’s a lot _____ –

Shimel: On the development side?

Choy: Exactly. And there continue to be more evolution on that, which is super exciting. Now one thing that hasn’t really, at least in my opinion, gone as far in this particular context is testing. Right? If you think about testing, we’ve had automated testing for a long time – 10, 20 years, maybe even longer, right? We’ve always had manual testing, right?

Now, when you think about testing, that’s – which is a critical part of the release cycle. Of course, right? You cannot push things out without some sort of testing at least, right? There hasn’t really been a lot of innovations and evolutions in testing. We’re still trying to do automated testing; we’re still doing some manual testing; we’re still doing it kinda the same way, so to speak. There’re new automated testing language, but they all kinda do similar things, right? And they all have similar –

Shimel: Absolutely. And, Derek, let me stop you there for a second.

Choy: Yeah.

Shimel: To me, here’s the real crux in the issue.

Choy: Yeah.

Shimel: We talked about that we’ve certainly seen, let’s call it, a “revolution” in development acceleration, rapid development over, let’s say, 10 years. I’ll say even 20 years, right? _____ _____ _____ around for about 20 years.

Choy: Agreed.

Shimel: Do you consider testing part of development? Or did DevOps come along and say, “Wait a second, it’s all one continuity here, with development, testing, operations, security – you wanna put security next to testing, whatever – but it’s one continuum”?

Choy: Yeah. It’s definitely – I completely agree with that. It is definitely one continuum, meaning that they are all part of the overall workflow, right, to make automation possible, to make a fast release process possible. You cannot do it without any one of those components, right?

Shimel: No. But they haven’t kept pace with the changes that we’re in – what we consider when we talk development.

Choy: Yes.

Shimel: Where many of us think, you know, of coding.

Choy: Exactly.

Shimel: That’s the issue right there.

Choy: That’s the issue, exactly as you pointed out, right? All the components of that whole continuum hasn’t really kept pace, right, with the developments that we’ve made on coding, right? On the actual coding of the product. And I think that that’s starting to show or actually has been starting to show, in terms of how other components of that, testing in particular, has been frankly dragging down the overall speed of being able to push things out faster and so on, so forth. Right?

If you talk to – and you do that all the time. If you talk to a lot of CIOs and CTOs and so on, frequently, you’ll hear challenges with testing. Right? And most people will tell you, “Yeah, I’m okay. I get it going,” but very few people –

Shimel: “It’s good enough.”

Choy: Yeah, it’s just good enough, but it’s not great, right?

Shimel: No, it’s not.

Choy: Yeah.

Shimel: And, Derek, I’ll go one further with you. What I see is this, is that, when we talk about testing – and _____ _____ the same thing with security, right? Is we’ve got twin challenges.

Choy: Yes.

Shimel: I can’t do it fast enough, so I’m constantly behind, running as fast as I can, but I can’t – just not fast enough.

Choy: Exactly.

Shimel: And then, number two, the train won’t wait for me.

Choy: Yeah. Exactly.

Shimel: If I can’t finish my testing, “Well, you did it good enough.”

Choy: Yeah. “Let’s go.”

Shimel: “We’ve gotta release. We’ve gotta release this hour,” or, “We’ve gotta release this week.” You didn’t do – dirty testing? That’s okay; it’s probably secure anyway. And, boom, out the door it goes, right? And that’s the dilemma, the enigma of testing, is “I can’t do it fast enough, but, if I don’t do it fast enough, they don’t wait for me.”

Choy: Exactly. Right. And there is a reason for that. Exactly, right? Like we talked about earlier, because customers actually expect releases, right, whether you’re ready or not, right? They are expecting releases in a very frequent cycle. Right? And so companies therefore cannot wait. Right? It is not acceptable to go back to the customer and say, “You know what? My release is delayed and therefore you have to wait another week or two.” In the past, that used to be the common thing to do, right? Nowadays, that’s not acceptable.

And so then what happen, exactly like you said, is that people compromise, right? They’re like, “Oh, you know what? Testing is not done. Okay, well, you know what? It should be fine. Let’s just go.” Right? “Security testing is not complete, but we’ve done a portion,” right? “Okay, well, let’s just go.” Right?

The downside, however, is that – and you’re seeing it frequently; some people are just not paying attention to it – is that there is a high price to pay for that, right? You’ve seen a lot of security loopholes out there, right? And, yes, the release is out, right, but then at what cost, right? If people found a loophole and there is a data breach and costs companies lots of moneys and reputations and things like that.

And equally with software quality as well, right? Things don’t work. Even bigger companies, you’ve heard companies like Apple, for example, struggle with that. Google, right, struggle with that. And they release something and people like, “Wow, this is junk,” right? “And I don’t want to upgrade,” and you hear that all the time, right? And it tarnish your reputation.

Now bigger companies have, obviously, a lot more buffer to kind of weather these kind of quality problems, right? You know, whether you like it or not, you’re gonna continue to use Apple, right? Or Android or whatever. But then smaller businesses may have less tolerance for that, right? And customers of those businesses, if they’re not happy with your quality, they’ll move somewhere else.

So I think it’s very important for technical leaders to figure that out and not just kinda brush it aside and say, “You know what? It’s not important. If quality’s not good enough, let’s just go. It’s fine. We’ll find bugs in production; we’ll fix it.” Personally, I don’t think that that’s the right approach and I think that it’s gonna hurt businesses down the road.

Shimel: So you know what I call that? So that’s the double-edged sword of MVP.

Choy: Yeah.

Shimel: Right? ‘Cause you talk to these people and they say, “Don’t worry. We’re just doing an MVP. It’s a minimally viable product. We know it’s not 100-percent tested. We know it’s not a 100-percent baked. We know it doesn’t have everything, but we’ve gotta get an MVP out there, god darn it.” Right? [Chuckles] And so –

Choy: Yeah.

Shimel: And we appreciate – look, I’ve been – I founded, cofounded several venture-backed companies. I’ve been on pushing the VP of engineering, saying, “Hey, we promised our customer we had a new release coming out before Thanksgiving, man, and it has to get out. It has to get out. I don’t care if it’s fully tested. Get it out.”

Choy: Yeah.

Shimel: It’s not – you know, MVPs were being chewed to death.

Choy: Yeah. I – [laughs]

Shimel: So that “minimum” – that “M” in MVP keeps getting bigger and bigger and bigger. Right?

Choy: Exactly. Right, exactly.

Shimel: And more and more critical.

Choy: Exactly. I think we continue to push the boundary of MVP. You know, I think that – and that’s really the challenge, right? So, you being kind of the leader of the business, right, you’re not wrong because customers really need it, right? “In order to move my business forward, I really need the release. I really need to get that feature out. I really need to get ahead of the market.” That’s not wrong. Right? And I think that that’s the needs of the business; that’s the market; that’s what people expect today. The challenge that we technical leaders needs to solve – CTOs, CIO, VP of engineering, VP of product – what we need to solve is “How can we support the business in doing that but also maintain the quality?” Right?

And so, as a leader, in my opinion, going back to the business and say, “You know what? I can’t actually make your delivery date. You will have to wait another two weeks, four weeks, six weeks,” or something – that’s gonna be a tough sell, right? But, at the same time, releasing it without quality assurance, without security and all these stuff, is also not the right approach.

So I think what we leaders needs to figure out is “How can we do better, right, in QA? How can we do better in security?” All these other things that are probably less mature than development itself, coding itself, right, how can we do better in these areas so that we can also support the business in shipping this MVP? Whatever the definition of this MVP is, right? We can support a business in shipping the MVP because that’s very critical to the business but, at the same time, maintain the level of quality and the level of security and, among other things, right, that we are proud of, right? That we want to have, right?

And I think that that’s the biggest thing that we need to figure out in 2018 and definitely going into 2019, right? How fo businesses figure that out? How do we implement processes and tools and technologies that will allow us to do that?

Shimel: Yep. So, Warren – Warren? I’m sorry. Derek, fantastic discussion. Let’s bring it home a little bit to Rainforest.

Choy: Sure.

Shimel: What can you do to help?

Choy: Yeah, so Rainforest has a very – takes a very different approach to QA, right? You know, so the traditional way is, obviously, automated test with Selenium and all these different Appium, all these different technologies, or we can go manual testing and perhaps building an AMI offshore – that’s very, very typical as well. Rainforest looks at it and say, “Well, neither one are the best approach,” right? And therefore we come up with a testing platform, you know, that is powered by the crowd. And we allow our customers to actually put plain-English test cases on our platform. We can execute these tests without actually building an AMI offshore, but, at the same time, you don’t have to worry about the downside of automation, having to hire QEs and spend a lot of time writing Selenium scripts and things like that.

And so we offer a solution that, at least we think, solves problems on both sides, right, and also allows you to hook it up into your CI/CD process so that you can actually push this MVP out in an automated fashion, fully kinda connected with all the rest of the continuum, right, and be able to actually ship code faster but with higher quality.

Shimel: So let me ask, so, at some level, though, you are reintroducing or relying on human element?

Choy: Right. Correct.

Shimel: So why don’t you have the same scale problems that others do?

Choy: That is an excellent question. So I would say that there are two elements, right? So, when we look at the human elements, we look at it from a crowdsourcing basis, right? So think about Uber, for example, right? If I were to hire my own team of drivers, that would be a little bit challenging, but then, leveraging the crowd and scaling it based on demand, we’re able to actually solve a lot of the human aspect of the problem. We can have testers that come in whenever they like to come in and we have enough testers to support the test that our customers would like to do. Right?

And now the challenge there and probably a followup question then is “Well, but what about the quality of these people? Do they know what they’re doing?” Just like a Uber driver, right? How do you know that that guy actually drives well, right? So we actually developed a bunch of AI algorithms that actually analyze the work that they do and also take in feedback from customers, based on their former – the previous tests that they have done. And a combination of that with a lot of data mining and AI analysis helps our customers determine whether that test result is correct or not. If it’s not correct, we will engage additional testers to make sure that the tests are consistent and results are correct.

So we have refined that algorithm over the years and now we have achieved a very high level of accuracy that, based on our data, is showing us that the accuracy’s actually higher than having your own manual QA team. And so you get better results, without actually hiring an AMI, either onshore or offshore. You can actually run your test on demand – typically, you’ll get results in 30 minutes or so. Right? And it’s fully integrated into your CI/CD cycle. And the best part is you can do all these things and achieve all these benefits without having to hire QEs to write automated scripts and having to maintain it, and so on, so forth.

Shimel: Excellent. So, Derek, I get the outsource. I get the army of people doing it. But, on the back end, are you doing some automated testing as well?

Choy: Yes, we are. And so what we’re also doing is we’re developing additional AI algorithms to actually learn from these human executions. So what we do is we have lots of human executions that are happening as we speak. In our environment, we have 80 million, 90 millions of tests that have already conducted on our platform, so what we do is we actually do machine learning through these data and learn from previous executions and develop algorithms so that, instead of actually having to send it to a human tester, we can actually achieve it with a robot.

Very similar to the approach that Uber is taking. Today, human drivers are driving people around, right? In the back end, they’re developing these self-driving cars so that, in the future, we don’t really need a human drivers anymore. Very, very similar approach, from a QA standpoint. Today, we’re sending testers, but then, in the future, we’ll have these robots that will execute these tests based on what they have learned from these human testers.

And tomorrow is already here for us today because we’re already doing that and a number of our customers are already using these automated AI algorithms. So they just need to write their test cases in plain English and they’ll see the tests executed by robots automatically, without having to worry about building automations or anything like that. So that’s kind of our approach to solve the QA problem.

Shimel: So Rainforest, the Uber of QA.

Choy: I don’t know that marketing will approve that line, but –

Shimel: No, I don’t think _____ _____ –

Choy: – personally, but, yes. Yes.

Shimel: It does remind me – you know, when – you’re out in San Francisco, we’ve all been in venture-backed world.

Choy: That’s right.

Shimel: There was a period there where every pitch was “We wanna be the Uber of,” right?

Choy: Yeah. Exactly.

Shimel: Uber of this or the Uber of that and _____ _____ to be disruption, or crowdsource in this case, but fascinating take on solving an age-old problem.

Choy: Mm-hmm. Exactly.

Shimel: Interesting to see. Just – Rainforest is, what, about two-and-a-half years, three years old now?

Choy: Yeah, it’s just a little bit longer. We’ve been – our total existence, probably about six, seven years. We’ve kinda pivoted a little bit at the beginning, so our current product is about three to four years old.

Shimel: That’s when I think I first became aware of it.

Choy: That’s right.

Shimel: And, look, testing and continuous testing is such a – I mean, it’s a hot area. There’s a lot of different solutions vying in the market, but this one is really kinda unique, so it’ll be interesting to watch how it happens.

Choy: Yeah.

Shimel: Derek, as I promised you before we started, the time goes really fast here. We’re way out of time, but, hey, man, I wanna thank you for being a guest on DevOps Chat. And maybe we’ll catch up with you again in the new year and find out how things are going.

Choy: That will be awesome. Well, thank you, Alan, and have a wonderful holiday to you and everybody listening. And, hopefully, I’ll talk to you again soon.

Shimel: Absolutely. Derek Choy, CIO still, not CEO, of Rainforest. Thanks for being our guest on DevOps Chat today. Have a great holiday and thanks very much.

Choy: Thank you. Buh-bye.

Shimel: Buh-bye. This is Alan Shimel for DevOps.com and you’ve just listened to another DevOps Chat.

— Alan Shimel

January 14, 2019
DevOps and Retail: Transforming Brick-and-Mortar to Brick-and-Click

Brick-and mortar-retail is rapidly transforming with the digital age. As physical storefronts turn digital, it’s safe to say that traditional retail is quickly becoming a mesh of “brick-and-click.”

So, how are retail giants surviving in this new landscape? Interestingly enough, the remaining brick-and-mortar retail are reinventing themselves by applying helpful DevOps principles.

In this post, we’ll peek into cases of innovation and horror stories from historically brick-and-mortar retail chains and brands. From Target, to Balian Group, Sears and other companies, we’ll discover the importance of DevOps in this new brick-and-click multi-channel retail environment.

Target

In an effort to increase agility, Target engineering teams embrace something they call “the Dojo”—immersive, six-week training sessions with hands-on DevOps and Agile coaches. As Target CIO Mike McNamara recently told the Enterpriser’s project: “The Dojo has been fantastic in getting teams engaged with agile and DevOps, removing the natural resistance and fear of change, and then supporting the team through the changes while maintaining productivity. It’s been a huge success for Target.”

The result of these sessions is increased comfort with methods including containerization and tools that enable continuous integration and continuous deployment. McNamara also noted their grassroots approach; they encourage open source activity and listen to their engineers when they petition the use of new tech.

“Our engineers have fully embraced DevOps,” he said.

Bailian Group

For another case study, take Bailian Group, a large Shanghai retailer that operates 6,000 grocery and department stores. By the time Bailian entered the online marketplace, it suffered from a severe lack of agility, non-native web apps and underutilized server assets.

“Our transition from traditional brick and mortar to omnichannel business presented a great opportunity but an equally large challenge,” noted Lu Qichuan, director of IaaS and Cloud Integration Architecture at the Bailian Group.

Qichuan and Bailian realized that to stay relevant, they had to cut long development cycles and embrace “new cloud application development tools and processes that foster a CI/CD and DevOps culture and increase innovation and time-to-market,” he said.

They ended up implementing a private cloud using OpenStack. Now, midway through their cloud transition, they estimate to have 10 million virtual machines by the end of 2020.

Sears

While the previous two examples had positive outcomes, digital transformation has not gone over well for other retailers. Sears declared bankruptcy in October 2018.

In its final years, Sears was in a sticky situation. To the chagrin of former CEO Eddie Lambert, surveys found their customers preferred to shop in stores as opposed to online. Disregarding customer desires, however, Sears attempted a big data play with its “Shop Your Way” program, under which Sears gathered user data to then sell. The program floundered, resulting in consumer irritation and loss of sales as reported by Business Insider.

Writing for InformationWeek, Emily Johnson reminded us that evaluating DevOps using the wrong metrics will ultimately lead to failure, citing customer satisfaction as the No. 1 goal.

In its final years, an abrasive Sears leadership halted any sense of DevOps to flourish. A disengaged Sears CEO Lambert created “a climate of fear,” according to Busines Insider reports. It appears the retailers that stay afloat encourage agility and intrapreneurship—this means encouraging innovation, not fear.

DevOps Aids the New Omnichannel Retail Benchmark

Brick-and-mortar are evolving into omnichannel retailers that bridge physical and virtual spaces. Take IKEA’s augmented reality app, which visualizes items in a house. From Disney, to Startbucks, Nordstroms and elsewhere, physical storefronts embrace digital offshoots of their core model to augment the traditional retail experience.

Surprisingly, physical storefronts aren’t completely dead. They still offer conveniences for urban environments with high walking scores. See the emergence of Amazon physical locations for ironic evidence of their benefit. Other storefronts and pop-up shops have taken on a museum-like quality. The flagship Funko store, for example, creates impressive displays and interactive exhibits, pulling in flocks of weekend shoppers that want more than a dull e-commerce shopping cart.

DevOpsGroup similarly describes a “reverse showrooming” movement; consumers research products online to then purchase in person. This enables brands such as ASOS, which is emphasizing social media and free worldwide deliveries, to stay competitive.

“Retailers have been forced to re-evaluate their business strategies and understand how new technologies will affect their offering,” according to the DevOpsGroup website.

DevOps Turns Bricks to Clicks

Online shopping has sufficiently disrupted retail, and brick-and-mortar must reinvent themselves to stay relevant. An open-minded leadership and a commitment to fast development, dynamic scaling, uncompromised availability and open source technology underpin a successful evolution.

Of course, this increased reliance on technology necessitates a hardened look at how development and operations communicate, and brick-and-mortar retailers must adopt the behaviors of tech companies to retain agility.

This may come in many forms: new user-facing digital experiences, microservices for composability in niche omnichannel environments, containerization to package code that scales well, distributed data stores, heightened user data security or smart incident management techniques, to name a few.

Traditional retailers, then, should dissuade their doubts against agile. DevOps can aid retail by reducing redundancy, cutting costs and streamlining production and distribution.

— Bill Doerrfeld

January 8, 2019
The Limiting Factors of Microservices Also Apply to FaaS

In enterprise development and deployment, the pattern is easy to detect. A big monolithic application is targeted for re-architecting as a microservices architecture. People excited about microservices get together and break up the design to make it far less monolithic and far more loosely coupled.

If your organization is lucky, it is at this stage that those responsible realize that breaking a monolithic application down too far is bad for complexity, performance, monitoring and maintenance—pretty much everything re-architecting is supposed to fix.

Far too many organizations don’t realize this until the implementation phase, though. First, a metric ton of little tiny microservices are deployed, then the organization realizes that functionality such as database access can be applied much more efficiently through one larger service that handles all interaction with the database for the app, rather than through five or 10 smaller services that replicate code and end up calling each other anyway.

Function as a service (FaaS) has much the same issue. It is an astounding solution for some uses, but in the end, users must consider what access the function requires to other parts of the system and what task is being achieved. The early uses of FaaS are well-documented; longer running independent jobs such as image or video processing were a bit early hit, for example, particularly for services that have highly varying traffic rates. But other uses, such as handling bottlenecks in traffic spikes (hello holiday shopping season!) or out-of-band operations processing, are less documented at this point. Entire applications have been designed from the ground up in serverless, but they are relatively few in terms of overall applications out there.

FaaS is highly suited to DevOps, because upgrading the function is as simple as redeploying code. There are no concerns about OS or library version, because those are largely beyond your control. But that is as big a negative as it is a positive. There is no flexibility in OS version or library inclusion because those are beyond your control—for the most part, anyway.

As with the vast majority of new technologies, expect adoption of serverless to grow and the hype to grow more quickly. It is definitely in the market where we have seen massive shifts—virtualization and then containers really did live up to the hype in terms of usage—but it needs to have a driving reason for adoption. Look closely at what your application and its DevOps toolchain are doing before hopping into the deep end. For some uses, FaaS is the fastest/easiest/lowest-maintenance solution. But that does not make it the Swiss army knife that virtualization and containers have proven to be. Indeed, if you think persistent storage is painful under containers, try it in a FaaS environment—as just one example.

FaaS is useful in many scenarios, but just remember where/when/why you are adding FaaS into your DevOps processes, and take advantage of on-demand processing with few scaling limitations while avoiding seeing everything as a FaaS problem because FaaS is the FaD.

— Don Macvittie

December 21, 2018
Using Trade Studies in Vetting Tool Stacks
Adding to your tool stack should entail more thought than, “Start your free trial today!” It’s great when there is an application need and cloud-based tools are easily grabbed with a few clicks. However, surveying alternatives before making a tool stack change can prevent costly mistakes. Trade studies can help vet best-fit tools for evaluation.

Short for trade-off, trade studies originated in the aerospace industry as a decision-making aid. Teams of engineers (or developers) usually generate multiple proposed technical solutions for any non-trivial problem. A little searching will find the term multiple criteria decision analysis, with three popular techniques: the analytic hierarchy process, the Kepner-Tregoe method and the Pugh method.

For DevOps teams, I lean toward variants of the Pugh method because it avoids creating an overly complex weighting and scoring scheme. Pugh analysis has gained popularity among Six Sigma aficionados for the same reason. Multi-disciplinary teams find the process easy to understand and the results just as useful as more algebraic methods taking longer and needing more input.

Let’s look at three steps in a practical approach: setting criteria, ranking best-fits and starting evaluation.

Setting Trade Study Criteria

Most tool stack disasters result from discovering a tool doesn’t meet all the requirements as the project unfolds. In my previous life, our IT team went all-in on a big-name ERP application partly because it featured web-based portals for suppliers. My e-business team discovered those portals required specific port numbers be opened at the firewall, which corporate security immediately rejected. We coded a way out, using Adobe ColdFusion querying the database directly and securing it with LDAP sign-on, but it took months.

You could search for the “information I could have used YESTERDAY” gif or you can spend time setting the right criteria. Pugh analysis starts with a matrix of criteria rows versus candidate tool columns. What most sources won’t tell you about Pugh analysis is a trick in generating that criteria list:
- Start with must-have features, with current requirements and anticipated future requirements.
- Research tools (data sheets!) for near-fits against those must-have criteria and create the matrix.
- Add all the features prominently listed for each of those near-fit tools to the criteria.
- Search again for more near-fit tools matching features on the extended list, adding to the matrix.
A common mistake is stopping after the first two bullets. Almost every trade study I’ve done discovers more tools fit the must-have list but didn’t emphasize all those features on their website. It’s a bit time-consuming, because every time you find a feature of interest you must go back and recheck all the other tools on your near-fit list for it.

Adding non-requirements? You’ll miss innovation opportunities outside the Venn diagram intersection. My recent client study on marketing reporting tools found one touting support for a dream-list feature. Digging into that tool found it may not support two must-have requirements but does support the others.

Ranking Best-Fits

That’s why the trade study criteria list should visually differentiate must-haves from extended features. I usually shade the extended feature rows, then plot how near-fit tools stack up against all criteria. You can certainly add weighting to features for a more sophisticated analysis, but I find a simple must-have versus nice-to-have usually suffices.

Now start scoring. A strict Pugh analysis uses a plus-same-minus scoring system. Plus means the tool has a clear advantage on that feature, same is same, and minus means that tool is at a disadvantage compared to the others. I sometimes simplify that to just X for a tool having the feature, blank for it doesn’t. This is mostly a data sheet comparison, and you may want to inject other information such as found reading reviews.

What you’ll find is a best-fit list, and it probably won’t be perfect. You’d like to see a couple tools capturing most or all the must-haves, and a few more capturing some of them and extended features for contrast. As an example, here’s the trade study matrix for that marketing reporting tool assessment I mentioned with the vendor names omitted.

Starting Evaluation

Notice this doesn’t have cost information yet. Cost biases technical decisions if applied too early, and feature coverage doesn’t necessarily correlate with cost. Vendors also don’t price their stuff the same way. Some price by developer seats, some by data source integrations, some by monthly tracked users (MTUs) on the web.

With a best-fit list in hand, it’s time to discuss the findings with your team and add information for evaluation. Confirm the scoring, check the vendor pricing models, ask the vendors about unclear feature support and download those free trials to see how they work. Consider how to cover those uncovered must-have features, or if those even are must-haves if few tools support them. Maybe swapping out another tool in the tool stack gets better coverage. Think about innovation those extended features may enable.

Want simpler, faster, actionable analysis without a massive RFP? Trade studies do that. I think it’s valuable to have one person on your team be the keeper of trade studies, someone skilled at interviewing for requirements and refereeing debates during evaluation. You may want to bring in a consultant skilled at competitive research and analysis to help develop your process. In any case, adapt your approach until trade studies help you make better selections—there is no one right answer.

— Don Dingee
December 19, 2018
Today: Predict 2019 Virtual Summit
Predict 2019 Virtual Summit is finally here! The summit is happening today, Tuesday, Dec. 18 at 10 a.m. Eastern. Don’t miss the opportunity to hear from Mirco Herring, Rich Mogull, John Willis, Chenxi Wang and other industry luminaries as they offer their DevOps and security predictions for 2019.

If you have already registered, then go here to join livestream starting at 10 a.m. Can’t make the live event date? No worries! Register for the event, and we’ll send you the replay link after the event. Click here to download the program guide.

Predict 2019 is a free virtual summit chock full of information and education for DevOps and security professionals presented in a virtual reality environment. It is a true virtual conference, replete with theaters to view the keynotes and presentations, a conference lobby to chat and socialize and a virtual trade show floor with virtual exhibitor stands equipped with video, downloads and even prizes for participation. And did I mention it’s free?

This year’s event offers up presentations and sessions from the biggest names in DevOps and security, as well as speakers from well-known companies who’ll be sharing what they’ve gleaned from their DevOps and security initiatives.

Here’s what the virtual format of Predict 2019 has to offer:
- Complete control and access over every aspect of your digital experience.
- Live and on-demand viewing from your home or office.
- Attend and interact using the device of your choice (laptop, smartphone and tablet).
- Digital downloads and presentations for all the sessions.
And, perhaps most importantly, you will have unlimited playback post-event for six months! (This is for attendees who register by December 18 only!)

Predict 2019 is here today, at a computer or mobile device near you. Visit the website for more information or to register for this daylong event. Registration—did we mention it’s free?—also includes access to the event on-demand. To register, click on the image below or this link.Predict 2019 will feature keynotes from a lineup of DevOps and Security IT executives and industry luminaries including:
- Mirco Hering sharing his predictions for 2019—Will AIOps take over DevOps?
- Dr. Chenxi Wang shares her predictions on future of cybersecurity in the cloud computing era.
- Rich Mogull’s offers up DevSecOps Predictions for 2019.
- John Willis speaks on the on Future of DevOps in 2019 and beyond.
In addition, Predict 2019 will include a number of featured panels with topics designed to provide even greater value to attendees. Panels include:
- Making DevOps More Ops-friendly – Charlene O’Hanlon (host), Damon Edwards, John Willis and Andi Mann
- The Business Value of Value Stream Mapping – Jayne Groll (host), Michael Orzen and Jeff Keyes
- Ubiquitous Open Source Makes for Security Challenges – Alan Shimel (host), Cindy Blake, Rami Sass and Baruch Sadogursky
- Cloud Security in 2019 – Tony Bradley (host), Chris Wysopal, Sam Bisbee and Jo Peterson
- Continuous Testing Equals Quality – Alan Shimel (host), Anand Akela and Lubos Parobek
Register Today at https://predict2019.com/#join-us

— saleem
December 18, 2018
The Top 5 Ways DevOps Fails – and How to Prevent Them

The working hypothesis for a Gartner report on how to avoid failure with your DevOps effort is that, unless the trajectory changes, “Through 2023, 90 [percent] of DevOps initiatives will fail to fully meet expectations.” The reason? You could boil it down to management not fully understanding DevOps tenets and nuances—things such as targeting business value, measuring success, meeting cultural/organizational challenges, fostering collaboration, understanding the inherently iterative process and managing for realistic expectations.

DevOps is about people from multiple disciplines coming together to solve problems streamline workflows, develop new processes, break down silos to develop business value. “A lot of people fixate on the tools,” said George Spafford, senior director analyst, Gartner and co-author of the report. “They identify best-of-breed, lay out their toolchain and say, ‘Ta-da!’”

The tools are important, but they are not the difficult part of DevOps. They are not a key obstacle. To succeed with DevOps, you often have to do something much more difficult: Change mindsets. Gartner identified these five top reasons for the failure of DevOps initiatives:

1. Doing DevOps for DevOps’ Sake

It’s important for DevOps initiatives to be grounded in generating business value. If the main goal is to achieve DevOps, you won’t be able to demonstrate sufficient value to sustain support. Failure to identify a business goal and focus the effort on attaining that goal can shut down the DevOps initiative in only a few months. Identify business value by working closely with business stakeholders on a project-by-project basis to identify opportunities or obstacles and define the business value that will be developed. Avoid the temptation to rush DevOps implementation by taking too large a leap. Trust the DevOps process.

2. Don’t Ignore the Culture

DevOps can be intimidating in the early stages. Done right, it usually involves organizational change. People need to understand from leadership that this is not optional. They also need to understand what DevOps is and ultimately how it will affect them. Otherwise, what they imagine will tend to be much worse. It’s also of chief importance to communicate the business goals, both for instituting DevOps and the business value that you hope to generate. Senior leaders must recognize that DevOps teams represent “a fundamental shift away from traditional command-and-control hierarchical management models,” according to the report. “To improve agility, decision-making must move to where the information is.” Relaying information up and down the chain of command to make a decision isn’t feasible with DevOps. The teams need to be empowered to make decisions themselves. For that to work, key DevOps team members need to be fully aware of senior management’s strategic priorities.

DevOps thrives in an open, honest environment where a frank exchange of perspectives is discussed, analyzed constructively by the group leading to decision-making. Many DevOps initiatives fail because they give short shrift to selecting people with the right temperament to help manage this process. Gartner suggests choosing people who you would describe as being team players, trustworthy, motivated, accountable, smart, experienced and effective communicators. Other words that come to mind include responsible, high emotional IQ, fair-minded, open-minded, aboveboard, not prone to casting blame, respectful and collaborative. In the end, for DevOps to work, you need to foster the type of environment people from multiple disciplines, with multiple agendas, can come together in a small team and get things done.

3. What We Have Here is a Failure to Collaborate

DevOps should not be limited to just IT and Ops. For it to succeed in generating business value, DevOps teams must work with other groups and stakeholders. This work “requires a systemic perspective and involvement, not uncoordinated silos,” according to the report. That said, while there is no magic number of DevOps team members, some have recommended five to 10 people. When you get into the range of 20 people, you may find that you DevOps effort is less effective. (Aside: It is possible to have more than one DevOps team.)

In considering collaboration, don’t forget senior management. Look to gain the support of an executive who can help champion the effort. Finally, you know that DevOps is working when DevOps team members are collaboratively compromising deferring to each other’s needs. The trick is to solve multiple problems with every decision.

4. DevOps is Iterative

Too many companies get caught up in the notion that they can launch DevOps in a single bound. According to the report, historically, traditional transformation approaches have a high failure rate. “DevOps involves too many variables for this type of approach to be successful in a large IT organization,” Gartner noted. “An incremental, iterative approach lets organizations focus on continual improvements and avoid the risks” of what is often termed the “big bang” approach. DevOps by its very nature is focused on continuous improvement with an iterative process, rolling out smaller improvements more quickly. Any thought of launching it all in one fell swoop misses the point and shows a distinct lack of understanding about the principles of DevOps.

5. Set Realistic Expectations

DevOps and DevSecOps are enjoying the huge level of interest and popularity among enterprises. When that happens, the hype tends to overinflate expectations about the outcomes. Gartner advises that you manage expectations by agreeing on objectives and success metrics. Establish your starting point with whatever metric you adopt, then pursue the goal iteratively. According to the report, “DevOps is not a one-time effort; rather, it’s about trying over and over again.”

— Scot Finnie

December 18, 2018
Containers: All Your Base are Belong to Us

The world is still changing. No surprise there, but the direction and velocity have been modified by the growth of container usage. Back in 2015, we at DevOps.com showcased a survey we performed that said container usage was staged to take off. That it did, with a vengeance. No matter the category—pilot programs, test, production—it seems to have a greater penetration each year.

And why not? There are a whole collection of benefits to containers—smaller footprint than VMs, more portable than cloud instances, more agile than … well, any of the alternatives. And the weaknesses—the use (and abuse) of IPTables and persistent problems with persistent storage—are being addressed at a rapid clip.

But It’s Not All Roses

Sorry to say that there is, as always, a fly in the ointment. We have to identify weaknesses in new architectures to plan for them and overcome them. And container infrastructure is not immune to this process. Nothing is perfect.

The first problem is with overall security. Your container management system is now the center of the world. A single point of penetration that could yield dozens to thousands of infected servers. It is like a magnet to ne’er-do-wells. Protect it, lock it down, follow advice on Container Journal and Security Boulevard about how to keep your system as protected as possible.

That was the easy one. We have a ton of experience protecting important central systems.

The more difficult ones all revolve around your physical infrastructure and where it meets your virtual container infrastructure. Look at all the boxes in your data center and the services they provide, then map out how those services integrate with your containers. Once you’ve mapped them, work the mappings into your DevOps projects. Load balancing and storage provisioning are two areas in which what a traditional DC does is duplicated inside the container environment, and where that processing is to occur should be a conscious choice about what is best for the application, not a decision based on what is easiest/in front of you.

It is an (arguably) bigger issue with storage. Just because the container can grow or shrink its internal storage does not make it the right place to position a fair selection of your data. Think about what will be needed in the future—with or without the application that this container or container set represents—and make a conscious choice about what is ephemeral data that can go away should an instance die versus what is needed long-term. Work those choices into DevOps processes. It can be a pain to access persistent data that is remote to a container, so make the painful parts automated via your DevOps processes.

Both of these problems are compounded in most (but not all) cloud environments. Too much cloud functionality is designed to lock users in. While that has been a standard practice in IT for decades, the problem with containers is that they are inherently portable. Build them to be so. Yes, lock-in functionality always brings benefits, but mobility lets a team choose where best to deploy a container system with each inflection point (purposely being vague here—a highly agile/DevOps shop could have daily inflection points, while a single team just piloting containers in their DevOps environment may only have them on occasion when something large changes with the system). Should something change at a preferred cloud vendor that makes them no longer preferred, the fewer modifications required to move to another platform the better. So building default container/container management into the DevOps toolchain is preferred to building in vendor-specific options when a choice is available. Storage is an easy one to point at and say “choice might not be available.” Allocating disk is not very portable between platforms, but if that’s one of only a handful of things that must change, portability is still an option.

In short, containers are coming, and they’re coming fast. Build them into your DevOps toolchain as portable solutions that automate the worst of the speed bumps. Make conscious choices about where infrastructure-based functionality should reside, and implement based upon those choices.

And keep rocking it. If moving from any other environment (with the arguable exception of cloud), containers will be faster to spin up and to spin down. Just address the issues before they hit you, and all will be well.

— Don Macvittie

December 13, 2018
Culture, Not Changing Much in DevOps

One of the things that has bemused me from day one of the DevOps movement is how culture is viewed with an odd dichotomy. It is simultaneously DevOps’ greatest strength and its greatest weakness.

Of course, there is a simple and reasonable explanation for this phenomenon. DevOps wants to change culture. Cultures that are resistant to change are typified as “bad” and those that are not are typified as “good.” It is never that simple, but that’s how the portrayal goes.

After a decade, you would think we’d have massive IT culture change supporting DevOps initiatives at one level or another (somewhere on a scale from pilot project to corporate standard) pretty much everywhere.

But data seems to indicate that this is not the case. Disclaimer here: I fall more on the “automation” side of the “What is DevOps, really?” question. I don’t think that matters here, since I’m pointing at data from the DORA “State of DevOps Research Report,” which is 1,900 of you all speaking, not me.

Image Credit: DORA State of DevOps Report

Most of the coverage on this report is by people who have bet their entire career on DevOps—a solid bet, but one that can lead to stilted coverage, which takes all the positives and hoists them high. And there is a lot of positive to be found. But I’ve noticed a trend of not looking at the data with a critical eye. The bit that I’d like to focus on today is what DORA labels “Firmographics”—a breakdown of respondents by title.

Only 27 percent of those who responded were on a DevOps team. Presumably because Google is a sponsor of DORA, it included Site Reliability Engineering with DevOps teams, and that’s a fair inclusion, since SREs by definition are doing DevOps.

Consider that data point representing “culture change” in a world where the respondents are, by self-identification, doing DevOps. Some will argue this is the wrong way to look at it, but one of the issues that culture people bring up constantly is breaking down silos and organizing around product or product portfolio instead of development and operations. But it seems more as though organizations are knocking holes in silos and communicating through them—which means we should pay more attention to that style of DevOps and quit harping on organizational change. Again, this fits well with my world view, which has always been “That specialist in a shared bit of hardware is not moving to a DevOps team.” The EMC specialist (for example) adds value to the entire org because of their domain-specific knowledge, not because of their deep understanding of the entire DevOps toolset. Once you’ve carved out exceptions for storage, networking and platform specialists, moving everyone to a DevOps team is pretty moot.

I won’t conjecture why only 27 percent are moved into dedicated teams, nor why that number did not go up from 2017. I have a predilection toward “because it’s not necessary,” but likely there will be two camps when this discussion gains steam because more reports show a low rate of change—Group A will be, “Because those people are resisting change,” while Group B will be, “Because those people don’t feel they need change.” Guess what? That will be the same group of people, and that angle of discussion will be irrelevant. The important bit will be, “Are you completing everything you need to get done, and what is your structure?” That’s where the meat of the discussion will need to occur. Because by deduction, 73 percent of people participating in the DORA survey are doing DevOps while not being on a dedicated (eg: reorged) team.

And that’s perfectly okay, as long as they’re achieving the goals, that’s what matters. The how is useful when deciding to move in a certain direction, but delivering is the thing IT should—and generally is these days—be gauged on.

— Don Macvittie

December 11, 2018
Reducing Poverty in the Age of Automation

In 1966, more than half of the people in the world lived in extreme poverty. I don’t mean public housing, food stamps, subsidized school lunch, free health clinic poverty. I mean the type of poverty in which there’s no electricity, no running water, no toilets and open sewers, and most children never see a doctor or the inside of a classroom. Back then, people living in extreme poverty had a daily diet of gruel; not rice and beans, mind you, but mushy, flavorless, boring gruel. Violence was rampant. Infant mortality was high and living to 65 was more the exception than the rule.

While it’s true that there were pockets of prosperity to be found in the Third World, for most people, if you didn’t live in Australia, Japan, North America or Europe, the odds were your life was not that different from a peasant living in the Middle Ages, except that maybe you had access to a TV or radio in the village square.

Cheap Technology Contributes to Declining Rates of Poverty

But things changed. In 1997, the number of those living in extreme poverty dropped to 42 percent of the people in China and India. By 2017, the number dropped again, this time to an amazing 12 percent in India and .07 percent in China. Now, this doesn’t mean that everybody in India and China rode around in Toyota Camrys and spent their evenings gazing at a 52-inch high-def TV, but it does mean that just about everybody went to school, saw a doctor and had some choice in what they ate for dinner. It might not seem like a lot, but when you consider that their great grandmothers could barely read a newspaper, it’s a big deal.

This progress didn’t happen by magic, and there’s a good argument to be made that it didn’t happen because of politics. My opinion is that it’s due in good part to the proliferation of cheap technology brought about by automation.

The Toothbrush as an Agent of Change

Small pieces of cheap technology can make a big difference. Take the toothbrush, for example. Not that long ago, a personal toothbrush was a luxury item for many. Today there are machines that make them automatically by the dozens, for pennies in material cost. Anybody can buy one and those who can’t many times can get one for free. Think what it must be like for a 7-year-old child in a classroom trying to learn how to read while in the throes of dental pain. There wouldn’t be a lot of learning happening.

Studies have shown that a leading indicator of good health in the long term is the practice of good dental hygiene on a daily basis. Giving a child a toothbrush and some toothpaste early in life makes a big difference. Children who brush their teeth regularly fare better. A toothbrush might seem like a trivial consumer good, but it is actually a high-impact technology that has lifted many people out of poverty.

Faster Credit Reporting Means Less Poverty

Another example of a small piece of automation that’s improved the lives of billions is credit reporting. Thirty years ago it took a few days to get the credit rating of a company, and getting the credit rating of a sovereign state could take longer. Today, getting a credit report takes seconds. Automated credit reporting accelerates financing, which not only makes it easier for a consumer to get a home mortgage or car loan, but also for sovereign states such as India to get large-scale, industrial goods such as bulldozers. More bulldozers translates into more roads; more roads lead to more commerce and more commerce leads to prosperity for more people. It’s a pattern that’s difficult to ignore. Just take a look at the economic growth that accompanied the expansion of the U.S. interstate highway system in the last century. In this century, the same growth pattern is underway as China grows its One Belt, One Road (OBOR) transportation infrastructure on a global scale.

The Cellphone: A Tool for Prosperity on a Global Scale

Which brings us to the cellphone. The first mobile phone cost $4,000. Today, you can buy a prepaid, throwaway one with service for around $50. Given the technical infrastructures required to build the phone and networking infrastructure necessary to provide service, selling a fully functional cellphone profitably at $50 is a near miracle. And, its impact goes way beyond a simple, “What time will you be home for dinner?” conversation. There’s a big difference between living in poverty and living in poverty yet having a cellphone. A cellphone puts a person on the grid even when living in a hut in a remote village in Niger. Having a cellphone means being able to solicit paying work. It means having access to simple bank services. It means being able to get medical care in an emergency. And, a cellphone can connect users with online learning resources that can enhance the educational experience for those living in parts of the world where access to secondary education is rare. It’s no wonder the Gates Foundation is focusing on programs that use cellphone technology to address issues in poverty stricken areas.

Not only has cheap technology reduced the cost of cellphones to nearly free, it also has reduced the cost of dental crowns through 3D printing technology and international trade through automated navigation systems for ocean-going vessels. Cheap technology also has brought about free online college courses and built-in writing assistants for email clients that have had the surprising effect of giving people better command of the written word.

The Fall of Global Death and Destruction

Our technological achievements allow us to grow more food, educate more people and reduce the level of violence to an all-time low. Remember, 20 million Russians were killed in World War II. That’s 5 million people a year over from the beginning of hostilities in June of 1941 until the war’s end in August of 1945. We’ve not seen slaughter in such numbers since that time. Yet, news of today’s conflicts is broadcasted on a global scale, so it can seem as though millions are dying everywhere. But the actual number of people killed in armed conflicts today comes nowhere close to the number casualties that were the result of the world wars of the last century.

As the Numbers Go Down, the Risks Become Greater

Things are getting better for more people and we technologists have been significant contributors to these improvements. Yet, new risks are on the horizon that could stifle the upward trends. One of the biggest potential dangers is a worldwide pandemic. While cheap air travel has removed the barriers of international travel for more people, it also has made it easier for a highly contagious disease such as Ebola to spread faster.

And then there’s always the risk of nuclear catastrophe. The number of nuclear weapons has decreased, but technology has made possible nuclear devices that are the size of a suitcase. Whereas it takes the resources of a sophisticated sovereign state to create and deliver missiles capable of a nuclear strike across continents, all that is required for a non-state belligerent to deliver a nuclear blow is a small device hidden in the trunk of a car driven across an unguarded frontier. It’s a remote possibility but a possibility nonetheless, and one that can have a catastrophic outcome of unimaginable dimensions. Fortunately, for better and worse, we in technology have created planetary surveillance systems that make it possible to guard against such an occurrence.

Considering the Big Picture

Technology grows in complexity every year, particularly for those of us in DevOps. The rate of innovation can be overwhelming. We’ve got to know more to do more. As such, many of us give most of our attention to the details that make the technology work. Sometimes we lose sight of the big picture. But, here’s something to consider: As weird as it may sound given current perceptions, there’s a good argument to be made that, given the increasing rate of technological innovation and the reduced costs of mainstream technology produced by such innovation, we might eradicate extreme poverty in our lifetime, if not the lifetime of our children. In the past such thinking was considered grandiose. But, as the $50 cellphone demonstrates, it’s a goal we can look forward to achieving.

Authors note: The source of the data on the reduced rates of poverty worldwide described in the opening of this article comes from the book, “Factfulness” (pages 52-53) by Hans Rosling , Anna Rosling Rönnlund and Ola Rosling.

— Bob Reselman

December 7, 2018
DevOps Chat: DevSecOps with Signal Sciences’ James Wickett

James Wickett is the man to go to for DevSecOps. The founder of the Rugged DevOps movement, which has merged into the DevSecOps group, James is one of the most knowledgeable people on the subject of DevSecOps.

In this DevOps Chat, James shares his views on what’s on the horizon for DevSecOps and the most important things we can do to make our teams more secure. Have a listen as James gives us his take.

As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

Transcript

Alan Shimel: Hey everyone, it’s Alan Shimel, DevOps.com, Security Boulevard and you’re listening to another DevOps Chat. I’m really happy to be joined on this DevOps chat by one of my friends from the Austin boys, James Wickett. James, welcome.

James Wickett: Hey, thanks Alan. Really glad to be here.

Shimel: It’s my pleasure to have you here. So for those of you who may not be familiar with James, you know, James is really one of the lynchpins in the, for my money, the best DevOps community in the world and it’s the folks over in Austin with Ernest Mueller and James and Karthik and Boyd and just a whole bunch of great people coming out of Austin. But when James isn’t busy building the best DevOps community around, he also works at Signal Sciences and of course James was also kind of the founder, prime mover behind Rugged DevOps and now DevSecOps, as well as with Shannon and those folks. So, I hope I didn’t make you blush, but –

Wickett: That’s all right. We loved having you come down to Austin and we’re pretty proud of the – you know, it feels like the InfoTech community here in Austin is really close knit and there’s a real family dynamic to it, to a certain sense, the good kind of family. I know that can be different for different folks, but you know, it’s very nice that we’ve been able to do that for so many years and we’re excited for bringing DevOps days Austin, continuing to come back year after year, and your partnership with that has always been great, Alan, so we –

Shimel: Yeah, no, it’s my pleasure. I don’t say it lightly, especially this past year and a half, two years, I have probably been to more DevOps days and DevOps events around the world, not just here in the US. And Austin’s a special, special place for it. James, as we were talking off camera a little bit, you know, over here at DevOps.com and the rest of the media op sites, we’re doing an event called Predict 2019 and it’s a virtual conference. It’s more than just kind of like a video like this where it’s actually a virtual conference where the sponsors have booths and there’s a whole session auditorium and everything, but the idea behind it is really to put on our crystal ball hats and take a look at, hey, what does the near term future look like? You know, if we’ve been in tech long enough, we know better than to make a long-term future guess.

Wickett: That’s right, that’s right.

Shimel: Right? Who knows? But, near term we’ll try. So, I wanted to spend a couple minutes with you and talk about what do you see 2019, the next 12 to 18 months. What’s factored in on the DevSecOps scene, on the DevOps scene, on AI Ops, on you know, just what we live in day to day, James?

Wickett: Yeah, yeah, that’s right. Yeah. Well, that’s a good question because when you think about DevSecOps and where it is today, I think, I always hope that we’re further along than we are, and the data still shows that like we have some ways to go. There is a SANS survey on secure DevOps and kind of the joining of just security and DevOps and where it is in the current state of the world and Signal Sciences, along with several other sponsors, are happy to kind of be a part of that survey and that research, but one of the things that is pretty astounding out of that whole piece is that 95 percent of securities responsibilities or 95 percent of security people reported that their main kind of thing they were doing was supporting legacy applications.

So, and it’s real telling that we see this – we’re seeing really in security and DevOps kind of the same problem we had with Dev and Ops, you know, ten years ago, right, this whole throw it over the wool mentality, you know, different groups value different things. Developers valued feature speed and deployment ability, I guess. Operations valued stability, reliability, things like that, and they always felt like they were at odds. Well, security comes in as sort of this third wheel here and what does that do to the mix?

And I always – I’m really happy like to be spending time with some of our customers at some events and I get to hear stories of people doing things that are radically different than securities done over the last decade, but to say that that’s not – to say that that’s ubiquitous would be false. Like that is just not where we’re at right now and I think it goes back to that quote, right, and I think this helps us with this 12 to 18 month prediction piece. Like the future is here, it’s just not widely distributed. Right?

So, I can see what leading high performing companies are doing and I can just say, next 12 to 18 months is like more people doing some of that stuff, and I’m excited about that.

Shimel: I agree with you. You know, James, this is something that I, you know, I’ve always kind of felt is that, you know, we certainly suffer from shiny trinket syndrome and we always want to know what the next great thing is or what’s the next big thing. But what we often forget about is the whole world isn’t a green field. It would be so great if we only had green fields and new apps to work on.

Wickett: That’s right.

Shimel: But we live in a world of legacy apps, right? We’re going – no one’s throwing out their mainframes. No one’s going to just throw away all of these applications that have been built. Excuse me.

Wickett: Yeah, we live in the brown field, Alan.

Shimel: Right, we live in a very, very brown field and so to me, should we be focusing near term at modernization, right, and how are we – because at the same time we have these legacy apps, James, we have such new infrastructure possibilities with, you know, so how do you containerize and Cubernettisize and make servo lists, all of this brown field application and at the same time, I really think that’s an opportunity for security.

Wickett: That’s right, yeah.

Shimel: To come in and let’s do it right. Right?

Wickett: Right.

Shimel: Let’s fix something that really wasn’t set up right the first time.

Wickett: Well, and the inverse or the – there’s another layer to that problem is that we’re seeing people in the security space that are almost cutting edge for like what we would call like DevSecOps and people that are doing great, but they’re all coming from like large enterprises, which is a really weird thing because innovation from the DevOps space came from like small, medium sized companies that were able to be nimble and be fast, but when you’re at a certain size, you don’t have security talent on hand. You don’t have anybody on staff and you’re also – you have a different set of concerns. Right? Your concerns are staying in business for the next several quarters.

Shimel: Right, keeping the lights on.

Wickett: Right, when we’re seeing people like Shannon Lietz over at Intuit or we’re seeing Topo over at Capital One, right, or Chitra Elango over at Fannie Mae, right, we’re seeing these stories and the way that people are piecing together pipelines and they’re tying all these security things into these DevOps movements. They’re all coming from big companies and I don’t think there’s – I think there’s definitely a connection there. We’re going to see some of that security wisdom and kind of learning of how to do DevSecOps come from the other end of the spectrum, which I think is a really cool thing.

If you’d have asked me three years ago, I would not have predicted that at all.

Shimel: No. So I’m going to tell you, along those same lines, James, to me, this may wind up being the hidden gem or the hidden legacy of DevOps. When I first launched DevOps.com and you were around, you remember, you know, besides everyone yelling about making a commercial, there was this huge argument raging between people about startup DevOps versus enterprise DevOps, you know, could enterprises do DevOps, was it a unicorn thing? And you know, some of the – I always felt that the right answer was, there’s only one DevOps. Right? There’s not a startup DevOps and an enterprise DevOps.

But, different organizations have different things. I think what we’ve seen over the course of evolution is that yes, DevOps really kind of took hold in the small midsized businesses out of necessity, but enterprises shaped and molded it and made it work at these large enterprises, and in doing so, created opportunity for the security folks, the QA folks. Right? All of these other silos, right, that maybe you didn’t have in a small company, right, because everyone wears like seven hats.

Wickett: That’s right, that’s right.

Shimel: Right? But, at the enterprise level, it’s giving it a chance to flourish and to – it’s like a giant petri dish, right, and so it’s great and it’s giving security people a chance to get in here. So –

Wickett: Well, and you look at, you know, you have a vision of like, hey, let’s put this for, this is not just for startups and west coast companies. This is something that can hit across – and I love that. I love going to middle of America, DevOps Days events like the Kansas City one was quite _____.

Shimel: Kansas City was pretty cool, yeah. Right.

Wickett: And I liked it because it’s people at larger companies wrestling with these problems, trying to make this happen and I’ve been lucky enough over the last several years of my career to do a lot of green field type stuff, but that’s not the yeoman’s work. The real tough work that we’ve got to do is DevOps transformations in these large enterprises. And you and Jean Kim through the DevOps Enterprise Summit we see stories of these types of transformations happening and seeing security being brought into that. I mean, it’s just–it’s really pretty cool.

Shimel: Absolutely. So, now not to prod you along here, James, but –

Wickett: Yeah, prod me. Prod away.

Shimel: I see it, so we’ve had a chance to build a little bit of a foundation. Right? We see these things happening at the enterprise level. Kansas City, Chicago’s another great example. Big companies, right, talking about their DevOps and their DevSecOps. How do we build on this momentum, James? What shape does that take? As security people, what do we need to do to move that needle?

Wickett: Yeah, that’s a good question. Let me – can I take two answer on that and –

Shimel: Sure, take three if you want.

Wickett: Sure, all right, good. Well, I think there’s – let’s start with the first answer. I think we’re still developing a mental model or some sort of platform of like how the delivery pipelines work. I mean, of course you can go to any number of talks, you can see how our people have assembled them, the tooling that they’ve put in place, but security has a lot of success whenever they can look across the – whatever the software delivery value chain and say, “All right, how are we delivering the software and where is our build system or our development process and then how do we deploy and deliver all the way to the customer?” And look across the broadly and find discreet tools that they can add in those places, and not just tools for like software, but like practices or ways to transform. Take the ISC squared, CISSP manual and rip out a good – a lot of the waterfall bits, but figure out like were does this change, like in the modern world? And I think that’s – I’d say that’s like the first approach that I would like to say.

I think philosophically, I would – I’ll kind of answer you with a situation that I was posed with recently. A friend of mine, he works at a large enterprise company. They are not cloud, just now thinking, “Hey, maybe we should do this cloud thing.” And which I – some people that are listening to this are like, “What?” But that’s just the way it is. People are –

They have so much investment in the data center that like now it’s like they finally kind of said, “All right, we have some new stuff that we’re building. We think the cloud is where we should do it and we’re hearing these things like Cubernettis and servals and containers and whatever.” Right? “So we’re trying to figure out what direction we take with all this.”

So now some of these companies, legacy type companies, are putting resources towards it. Well, this guy is a security guy and he had them kind of come in and say, “You’re going to DevOps the thing and DevOps is going to be the way.” Right? Not really – this is a company where maybe they’re not even really sure what DevOps really means. Right? Which is okay. They’ve heard it. They’ve got some of the vision and they’re interested.

He’s very concerned. He says, “James, what do I do about this? I know you like the DevOps and stuff like that, but you know, what do I do from a security perspective.” I give him an answer. I didn’t feel great about it. I gave him a, you know, I don’t know, find ways to inject stuff, I don’t know. I went home, thought about it, woke up the next morning and I was like, “I got to call him back.”

So I gave him a call and I was like, “Hey, the real answer to your question is never say no ever again. Never say no ever again. Security has lived the world of no for so long and you have to throw that away. Like your new answer has to be yes, you can have yes with qualifiers, yes with whatever, but you’ve got to find ways, creative ways to say yes and kind of the cloak and dagger, blame it on compliance, slowing down things for whatever, you can’t do any of that stuff anymore and you’ve just got to throw out all the security gets to be a blocker.”

And I said, “Because every organization that I see high performing, what we would now call like DevSecOps teams, but security and DevOps working together, they sort of rip tear up the security playbook and they say, if you are a blocker, everyone in the organization is going to route around you. But if you play ball, like you get to be a part of the solution. And security has to find a way to add value inside of that.”

And so, and I said, “Every single one, you talk to any person who’s undergone the transformation and working in any of these companies to Hulu to Netflix to Inuit or whatever, you know, or all the ones that we see at DevOps Enterprise Summit, they’re always finding ways to say yes and be part of the solution and not to see themselves as a blocker. So I think that’s a fundamental like philosophical shift that security has to undergo.

Shimel: Absolutely, and you know, just to turn it around on you a little bit, I’m going to be the Jewish guy spreading the good news too, right, the gospel. We started off this conversation early on and you said, “Well, we still have so much more to do and I wish we would be further along.”

Wickett: Yeah.

Shimel: You’re right, but someone much smarter than me once told me, James, change, especially in technology, is 99 percent of the time evolutionary, not revolutionary.

Wickett: That’s right.

Shimel: But, when we look back over a period of time, we’re often astonished what that evolution – how that evolutionary change adds up. So the fact, that little piece that you espouse right there about not saying no anymore, and yes, we can and you know, whether you qualify it, caveat it whatever, but yes, we can.

Wickett: Yeah.

Shimel: It’s so powerful. It really is. It’s so powerful. Look, I spent 2001, I’ve spent 18, 19 years hearing now from people, and – or being part of the no too, to be in all honesty.

Wickett: Right, yeah. I mean, that’s –

Shimel: You don’t want to wind up in jail, do you?

Wickett: I got my ISC squared books up there. They look at me with an evil eye.

Shimel: Yeah, so you know, it’s refreshing. It’s like liberating to take that attitude. How do we spread this gospel though? What more – what better way can we do it? I know you’re involved in DevSecOps.org and DevSecOps Days and I’m obviously too and at RSA this year, we’ll be doing our thing again. But what more could we do in 2019, James?

Wickett: Oh man, that, I don’t know the answer. I don’t know the answer to that question yet. I mean, I think there’s some pragmatic things. I think we continue to say the story side. I relate to what’s going on with security right now to what happened with DevOps in the beginning and if you rewind to like 2009, 2010, or like let’s say the first three years, right, DevOps equaled chef or puppet, like straight up, like that’s what it was. And so what was that? That was operations was finally able to be developers, to act like developers, to shift left. Right?

They participated in agile type practices with infrastructure, revolutionary, really cool, really great. But then there was also another movement that happened at the same time and that was the idea that like we needed the instrument a lot better. We need to do monitoring. Now, people talk about observability. There is a big push of understanding like what your systems are actually doing. So, and that was, then there’s kind of the whole like monitoring conference, there’s monitorama, there’s monitor movement and the monitorama in response to it.

There’s some great products have come out of that space. We see things like Data Dog and before them Cloud Kick and other folks. Well, I relate all that to security because if you look in the first several years of this security and DevOps kind of joining forces, I know people have been talking about it for many years, but over the last couple years, what are the things that everybody has kind of gone around? It’s the whole idea of shift left. So you see great companies, _____ types, Nick, other folks like that, that are focusing on, you know, putting all of your development testing, you know, figuring out what binaries, talking about software supply chain, bill of materials, all that sort of stuff. We’re still missing the push right.

Now, I mean, I work at a – or a shift right type thing and so like, Signal Sciences, we sit at that end of the spectrum, so this is a bit of a self-serving type statement, but I mean, you need to have sort of insight into sort of what your application is doing and when we started the company, we were singularly focused on one question and that question is, do you know right now if you’re under an attack with any degree of certainty, we’ll say. You don’t have to know 100 percent, but with any degree.

I ask people that question like in conferences and whatever and almost zero, like I’d say just one or two people per, y, every six months or whatever, says yeah, they know. Most people have no idea, like what’s actually going on in their site. And we see this like when was the last time you read the headline that said, like, you know, we lost a million account credentials, but we stopped them. We’ve been following them through our system, we saw that they were starting to leak data and so we turned it off, but we didn’t get it off soon enough, but they didn’t get 100 million, but they got a million. Right? You never see that news story. Right? You see the news story of like _____ we lost it all and we lost it all three months ago. We had no idea until it started being traded on the market. Right?

So, like we lack a critical insight and you know, operations had the same problem, right, didn’t understand what was going on with the systems, performance-wise, everything. Security is quite similar to that. So, being able to instrument and see and do monitoring and have that sort of security observability is like an important aspect to where I think we need to go. So, we’ve done a lot of the shifting left similar to ops in the early days, and I think we’re going to see some more movement on the shift right part.

Shimel: It also sounds though like opportunity there for some better tooling.

Wickett: Yeah.

Shimel: Right? I mean, a lot of our security tools, you know, from vendors really aren’t made for that, right? I mean, you know, we have so much – so you know, the fighting point in the security vendor space is, should we put more focus on security response versus prevention? Right? Threat intelligence, all of these things, and you’re right. We don’t have that analog of a Data Dog, a Signal FX or that, for the most part, in securities.

Wickett: Yeah, ’cause like I see that across like – so for Signal Sciences, like we provide that for web applications, but that’s also stretching across other boundaries that you need to look at, and a more fuller picture, and I remember whenever we were launching – when Ernest and I were doing cloud stuff at our first just kind of thing over at National Instruments, we were trying to find vendors and just straight up like operational monitoring vendors and I talked to like – that was my job. That was one of my first parts of the things.

It’s like, okay we built the thing that – we have this whole thing we called Pie that spun up our system, much like Chef and Puppet, but then we said, all right, what are we going to monitor this stuff with? And I started calling and monitoring vendors and getting demos and stuff and it’s like, at the end of it, I was like, I just need one that has an API. Like just a good working API that can like admit _____ _____ and connect, you know, ’cause this is early days. Right? And we ended up finding a Cloud Kick and they ended up getting bought and sold and we moved later on to Data Dog, but it was so hard at the time to find that. And I feel like security vendors are similar in that space, that they struggle from where like when – and we’re not alone in this. There’s other people in our space that are doing thing that are helpful for people, but emitting data back to all the systems that you work with, like the Slack and JIRA and whatever, but also just being part of like the whole tool chain approach. And I think that’s a story we’re going to see continue to be developed and companies that are adopting that and embracing the idea that they’re a piece in the overall tool chain, like I think those companies are going to continue to do well.

Shimel: I know that you can have like just standalone security monitoring and so I think security monitoring has to be part of your larger monitoring.

Wickett: That’s right.

Shimel: Two or three other points I wanted to hit on, get your opinion on. So, just like DevOps, same analogy, tools are important. Culture trumps tools in a lot of places and we can have the best tools in the world, but until the security people’s mindset is changed and we’ve touched on it in this talk, you know, not saying no anymore and all of that, so how do we go about changing that mindset? Well, one piece of it, James, is at the organizational level, right, we embrace security.

We make them part of the team. Another part of it though is at the community and industry level and we mentioned it now a few times and a lot of my friends mention it, and this is the whole what are security people taught. And I’m not knocking ISC Squared or SANS or any – we’ve got some –

Wickett: Let’s knock them. No, yeah.

Shimel: I’ve got friends on the board there. You know, I got to be careful. But, I mean in all honesty, and again, I’m not singling any one of them out, but when security people are being taught to say no, it’s very hard for us to come in and say, “Don’t say no.”

Wickett: Yeah.

Shimel: Right? So, do we, you know, you know I’m part – I’m one of the cofounders of DevOps Institute and we have a DevSecOps class there that you know, some of my more enlightened friends have helped us with. You and Ernest have done the DevSecOps thing. You guys do _____ and stuff like that. But those messages, I don’t know if they’re finding their way into, let’s call is mainstream security training, ISC Squared, CISSB exam, the SANS certifications and not just those two. There’s a half a dozen other very well-known ones. How do we, you know, is 2019 the year we make a beachhead there?

Wickett: I don’t know. I don’t know. I’m an optimist, Alan, so I would love to say yes to that question. I would love to say yes, but when you think about it, like it’s – security has a very tough job. They’ve worked themselves into a scenario that they’re beholden to compliance and standards and outside bodies of influence and then they also are inside of an organization trying to affect change, but they find themselves at a very like inequitable distribution of labor. So like you have a hundred developers. You have ten operations folks. You may have one security person. Right? That’s a recipe for a problem. They’re all going to have different values.

So, I don’t know if we’re just going to automatically fix the culture in 2019. I will say tools influence – tools have an influence on culture and that you know, in the CANS model or CLAIMS model for DevOps—or CALMS, I guess people like to call it as well, but for the M, for measurement and sharing, I think those are the two pieces that can really help influence the culture in a more positive light.

And so, you know, security has, you know, we have stocks and we have like we – we remit all the data and have people taking concern of this, but we’re not – if you’re not creating – this is something else that I try to influence among my security friends. It’s like, if you have any tooling that you’re buying and implementing whatever, but it’s not creating a feedback loop to developers, then it’s like worthless. You should _____. Right? If it’s just going to a sock and someone’s doing it like, that’s not good. Right? There’s no DevOp telemetry out of what’s actually happening.

And so I think finding products that can help or whether it’s like open source testing tools or you know, instrumentation on the right or detection on the left or whatever, but like creating those like, I don’t know, able – we’re able to have a sort of virtuous cycle of feedback loops to development. I think that’s going to be key. But yeah, I’m going to say like no, 2019 is not going to be the year just because of – just because of the way humans are. I would like – but I think that it’s going to get better.

Shimel: I think it will get better. Here’s what I think we need to do, James, and you have got to do in this.

Wickett: All right.

Shimel: You and I are both wired into this community. We know people on the ISC Squared board. We know SANS instructors. We know the folks at SANS. We know all of these places. If one of the missions of the DevSecOps.org group, team, community, DevSecOps Days, what I do with my stuff, is to make sure we reach those people and say, “Hey, we represent a very large piece of this community and you’ve got to listen to us. It’s like a voter registration drive then. And – but you’ve got to lead it, right? And so, I’m going to put the onus back on you, James, and say you can make this a reality. You –

Wickett: All right. Well, Alan, let me tell you how I am working on that. So I do have somewhat of a plan here or an idea here. So, I’m working with several others, Shannon Lietz, Ernest Mueller, John Willis and we’re working on the DevSecOps handbook, which is going with IT Revolution, and we’re trying to tell those stories of like how DevSecOps is transforming at large companies, small companies, helping people see the new path forward. So hopefully yeah, that’s –

Shimel: In 2019?

Wickett: That’s – the – what is it called, the like the draft galley copy, like we’re hoping to have some of those available in 2019 at the DevOps Enterprise Summit _____ either in London or in Vegas, so yeah, it depends on – you know, it’s a – they run a real great shop over there, so it depends on the schedule for editing and all that stuff.

Shimel: Well, that’s a revelation. Good stuff, man.

Wickett: So hopefully 2020, 2021, like we’ll start to see the industry really shift. You know, I do hope that – I always hope that things happen sooner than they do, but –

Shimel: It _____ before it’s time either, my friend, right?

Wickett: That’s true.

Shimel: So James, I told you this was only going to be 15 or 20 minutes. I’ve got a feeling we’re at like twice that already.

Wickett: Yeah.

Shimel: But, it’s okay. I think it was a great conversation. People will enjoy it. Predict 2019 folks, you can go to predict2019.com. James is, of course, with Signal Sciences and you should definitely check out their site and what they’ve got going on. And also just a quick reminder, RSA conference is here March 4 this year. And we are doing our DevOps Connect DevSecOps Days at RSA Monday March 4, so if you’re going to RSA, come a day early, come in at the Moscone Center, Shannon the whole crew _____.

Wickett: Yeah, if people are listening to this and they haven’t been to that, like, you’ve got to go. Like, I think that’s a very valuable single day track that’s really helpful. And also, some of the sponsors and vendors back there, like some of the really forward-thinking folks, I mean, it’s the – it’s really like the part of RSA that I feel like has really kind of anchored that for me in my experience at RSA. It’s like “Oh, okay. Here’s the crew. Here’s what we’re trying to do.” I can see the message going out, and I love the fact that kind of the vision, did you decide to start that or is that –

Shimel: That was me.

Wickett: That’s – I think that was good because –

Shimel: DevSecOps Days, I just – you know, I – so I have a relationship with RSA for 15, 20 years and I originally went to them with it. This is the fifth year we’re doing it. I’m really pumped. I mean –

Wickett: Yeah, well, and thank you for that because basically like we – I’d say one of the ways we make the transition is by going to security events and kind of helping make this happen.

Shimel: Yeah, raising the flag.

Wickett: Yeah, and you – that is an outpost for DevOps over at RSA, RSA of all conferences. So –

Shimel: Yeah, if we can do it there, we can do it anywhere.

Wickett: I agree. I agree.

Shimel: All right. Hey, James Wickett, Signal Sciences, thanks for being our guest on this DevOps Chat. James, thanks for all you do too for the community and everything else. Keep up the great work and we’ll speak to you soon.

Wickett: All right, thanks Alan.

Shimel: All right. This is Alan Shimel for DevOps.com Security Boulevard. You’ve just listened to another chat. Have a great day everyone.

— Alan Shimel

December 4, 2018