devopsy.kinsta.cloud

Tag: kubernetes

Predictions 2020: Ushering in 2020 Data Predictions

2019 brought us more data organizations running more advanced analytics, AI and ML workloads than ever before. 2020 is the year where we’ll see a spike in both the number of technologies and data teams that support these types of workloads internally. We’ll see AI and analytics teams merge into one as the new foundation of the data organization, focused on areas such as moving to the cloud while maintaining on-prem Hadoop, “Kubernetifying” the analytics stack and Hadoop compute. These are the trends I believe we’ll see come about in 2020.

Rise of the Hybrid Cloud

We’ve been hearing people talk about the hybrid cloud for the past three years now. For the most part, that’s all it’s been talk—but in 2020 it gets real. We are seeing large enterprises refusing to add capacity on-prem to their Hadoop deployments and instead invest in the public cloud. But they are still not willing to move their core enterprise data to the cloud. Data will stay on-prem and compute will be burst to the cloud, particularly for peak demands and unpredictable workloads. Technologies that provide optimal approaches to achieve this will drive the rise of the hybrid cloud.

One Machine Learning Framework to Rule Them All

Machine learning with models has reached a turning point, with companies of all sizes and at all stages moving towards operationalizing their model training efforts. While there are several popular frameworks for model training, a leading technology hasn’t yet emerged. Just like Apache Spark is considered a leader for data transformation jobs and Presto is emerging as the leading tech for interactive querying, 2020 will be the year we’ll see a front-runner dominate the broader model training space with pyTorch or Tensorflow as leading contenders.

Kubernetifying the Analytics Stack

While containers and Kubernetes works exceptionally well for stateless applications such as web servers and self-contained databases, we haven’t seen a ton of container usage when it comes to advanced analytics and AI. In 2020, we’ll see a shift to AI and analytic workloads becoming more mainstream in Kubernetes land. Kubernetifying the analytics stack will mean solving for data sharing and elasticity by moving data from remote data silos into K8s clusters for tighter data locality.

Hadoop Storage (HDFS) is Dead but Hadoop Compute (Spark) Lives Strong

There is a lot of talk about Hadoop being dead, but the Hadoop ecosystem has rising stars. Compute frameworks such as Spark and Presto extract more value from data and have been adopted into the broader compute ecosystem. HDFS is dead because of its complexity and cost and because compute fundamentally cannot scale elastically if it stays tied to HDFS. For real-time insights, users need immediate and elastic compute capacity that’s available in the cloud. Data in HDFS will move to the most optimal and cost efficient system, be it cloud storage or on-prem object storage. HDFS will die but Hadoop compute will live on and live strong.

AI and Analytics Teams Will Merge Into One as the New Foundation of the Data Organization

Yesterday’s Hadoop platform teams are today’s AI/analytics teams. Over time, a multitude of ways to get insights on data have emerged. AI is the next step to structured data analytics. What used to be statistical models has converged with computer science to become AI and ML. So data, analytics and AI teams need to collaborate to derive value from the same data they all use. This will be done by building the right data stack—storage silos and computes, deployed on-prem, in the cloud or in both, will be the norm. In 2020 we’ll see more organizations building dedicated teams around this data stack.

Talent Gap Will Inhibit Data Technology Adoption

Building the stacks that enable data technology into practice is hard, and this will only become more obvious in 2020. As companies discuss the importance of data in their organizations, they’ll need to hire the data, AI and cloud engineers to architect it. But there aren’t enough engineers who have expertise in these technologies to do that. This super-power skill is the ability to understand data, structured and unstructured, and pick the right approach to analyze it. Until the knowledge gap closes, we’ll continue to see a shortage of these types of engineers—many companies will come up short on their promises of “data-everywhere.”

China Is Moving to the Cloud on a Scale Much Larger than the US and Will Leap Frog From On-Prem to Massive Cloud Deployments for Advanced Workloads

Over the past five years, while enterprises in the U.S. have been moving in leaps and bounds to public clouds, enterprises in China have been investing mostly in on-prem infrastructure primarily for data-driven platform infrastructure. 2020 will be the inflection point where this changes. China will leapfrog into the cloud at a scale much larger than the U.S. by adopting the public cloud for new use cases, bursting in the cloud for peak loads and over time move existing workloads. Public cloud leaders in China will see dramatic growth that might outpace the growth of the current cloud giants.

2020 is the year where the rubber meets the road when it comes to advanced analytics and AI. Companies that back the types of technologies that enable and support this kind of data and workloads will emerge as leaders in the space. On the other side, companies that structure their data teams to meet the requirements of the new data stack will emerge as leaders as well. I’m excited to see how advanced analytics and AI opens up new and innovative applications and use cases.

Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

— Haoyuan Li

December 5, 2019
DevOps Chats: Canary Deploys Using Istio, with Autodesk

Spinnaker Summit 2019 Preview: Canary deploys give us a window into how new code deploys perform in production on a limited basis. The technology holds great promise in helping us learn the positive or negative effects of deploys without putting the more extensive set of microservices, application functions and beyond at risk.

How do you route traffic? How do you apply a virtual service manifest? Is there a quick way to perform a smoke test to find server errors? When comparing metrics from new code to old, how do you know which results are good or bad? There is a lot to know and learn about how to deploy and use Canaries.

Our DevOps Chat guest Omar Al-Hayderi, engineering manager and principal engineer at Autodesk, is giving a talk on “Canary Deploys with Istio: Lessons Learned” at the Spinnaker Summit 2019. His talk is on Sunday, Nov. 17, at 1:30 PM PT.

As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

Transcript

Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Omar Al-Hayderi. He is engineering manager, formerly principal engineer at Autodesk. And Omar is gonna be speaking at the Spinnaker Summit 2019 in San Diego. His talk is on Canary deploys with Istio. It’s happening on Sunday, the 17th of November, 1:30 p.m.

Omar, welcome to DevOps Chat podcast.

Omar Al-Hayderi: Thanks for having me, Mitch.

Ashley: Awesome to have you here. I’m excited to hear about your talk. But first, tell us a little bit about yourself, introduce yourself, and what you do at Autodesk. I think we know about Autodesk, maybe what part of Autodesk that you work in.

Al-Hayderi: Right, yeah. So, I’ve actually gone through a lot of change recently. I was working at a startup called PlanGrid, and we were a small construction software company four years ago, grown a lot since then. I started off at that company as sort of a backend generalist. You know, there was like 25 engineers, really, at the company, and have grown a lot since then, since the acquisition. We got purchased at about 400 employees.

In that time, I’ve sort of evolved naturally into more of a DevOps infrastructury engineer role, and—yeah, it was kinda the best four years of my life. Really learned a lot there. Post-acquisition, I moved into a principal engineer role on the infrastructure team. We then went through sort of a re-org and I became the engineering manager of the back end platform team. And now, that team, we sort of specialize in building a lot of tooling around our sort of DevOps and infrastructure cloud.

Ashley: Mm-hmm, great. Well, great experience to bring with you to Autodesk. It sounds like you’re working on some fantastic things. How did you come to decide that you’re gonna talk about Canary using Istio at the Spinnaker Summit? Was that something you chose to kinda bring in and start up at Autodesk, you joined a team already using it? What’s kinda the background on how you kinda came to this place to wanna talk about it?

Al-Hayderi: Yeah. So, I was sort of, you know, dipping my toes into Spinnaker at the early stages of it, and I was really fascinated by the tool. I loved using all sorts of features and playing around with it. And at PlanGrid, actually, before we were even acquired, we did yearly hack weeks, and we got to sort of move off of our regularly scheduled program and work on whatever we felt like, just had fun.

And one of the things that recently came out for Spinnaker was this idea of Canary deployments. So, I experimented with it, never got it off the ground, fast forward one year later to the next hack week, Kayenta, the Spinnaker Canary component, was a lot more built out and user friendly. So, I managed to get it off the ground then.

We sort of have a monolith API at PlanGrid. And so, a huge problem we had was, we would do weekly releases, and after the release, stuff would break and people would get paged and we’d have to roll back, so.

Ashley: [Laughter] Vicious cycle.

Al-Hayderi: Common story I’m sure everyone’s familiar with.

Ashley: Mm-hmm.

Al-Hayderi: So, I really wanted to find a way to solve that, and Canary just seemed like a great way to do that. So, after the hack week, I sort of kept working on it full time, and I started doing, you know, presentations to the rest of the company. Because Canary is sort of this new idea that a lot of people aren’t familiar with. I started, you know, being active in the Spinnaker Slack channels, and it actually turns out that an ex-PlanGrid employee moved to Netflix to be a lead on the Spinnaker open source software from there.

Ashley: Ah, interesting.

Al-Hayderi: Yeah.

Ashley: Small world.

Al-Hayderi: Yeah. [Laughter] And he noticed I was talking a lot about it and he brought up the point that they’re running the Spinnaker Summit in November and he asked if I wanted to talk. And I’ve actually never given a talk at a conference.

Ashley: Oh, excellent.

Al-Hayderi: Yeah, and I do a lot of talks internally. So, this is something I was super excited about and, yeah, I jumped on it.

Ashley: Great thing to do for your career, too, and it sounds like those internal talks have set you up well to do this.

Al-Hayderi: Yeah.

Ashley: So, how did you go about deciding, is this gonna be kind of an overview of how to do it? Is it going to be, “Here’s the metrics to watch” and the value that comes out of it is a combination of that? Tell us a little bit about what part of this you’re—kinda what take you’re taking in your talk.

Al-Hayderi: Yeah. So, public speaking was honestly never my strong suit when I was younger. And some great advice I got was that, when you’re giving a speech to a bunch of people, you wanna tell it like a story.

Ashley: Mm-hmm.

Al-Hayderi: So, since then, I’ve really focused on storytelling for even these tech talks I give internally.

So, the story I’m trying to tell here is how we leveraged Istio to finally solve some of the deep technical problems of running Canaries.

Ashley: Mm-hmm.

Al-Hayderi: And once we got it out there, sort of the growing pains and, you know, first obstacles we hit and problems we’ve got. I’m hoping people come out of this with a better idea of how to actually get this running in your infrastructure and some common pitfalls to avoid.

Ashley: Mm-hmm. Great. Well, tell us a little bit of that story. Kinda get us started down that path.

Al-Hayderi: Totally. So, the first thing I did was, I went in and learned exactly what Canaries are. And the basic point of it is that you want to expose a small amount of traffic to something new. You want to analyze the deltas and some sort of metrics or something between the new and the old. And then you want to make a decision if this new is ready for the majority of traffic.

Ashley: Mm-hmm.

Al-Hayderi: So, there’s a lot that goes into that. Anyone that’s worked with Spinnaker on production scale knows that managing all these pipeline definitions is very cumbersome, and trying to, if you end up changing something, especially changing production pipelines, it can be really dangerous.

So, a lot of the things we learned about our, you know, how we tested this, even just testing the Canary process in our sort of R&D Dev staging environments and the problem there, even though we got confident with how the pipelines worked, it was very difficult to get a signal on how to actually tune your Canaries. And when I say tune, I mean, when you’re comparing metrics from the new and the old, how do you know if it’s bad or how do you know if it’s a good decision to move forward? What are the thresholds, what metrics are you looking at? A lot of our iterations were focused on that.

Ashley: Interesting. Now, I’m curious if—are you approaching this problem of using Canary to understand the service mesh or the services itself, or are you trying to also understand kinda Istio and it acting as a sidecar proxy and how that’s performing coordinating traffic load balancing across that? Is it one or the other or both, or what part of the problem were you looking to solve?

Al-Hayderi: Right. So, actually, we were kind of blocked on the fact that we couldn’t route traffic, you know, with small granularity coming in from the Internet. And we actually only use Istio for its API gateway feature at the moment.

Ashley: Oh, okay, okay.

Al-Hayderi: Yeah, there are some blockers right now, at least in our version of Istio, that we’re running around connecting over SSL to Redis. So, that’s blocked having sidecars or application pods. So, when we do canaries, we’re just canarying end user or ingress traffic into our monolith.

Ashley: Okay.

Al-Hayderi: Yeah.

Ashley: Interesting. So, tell us the story. How did you, you went about implementing it—what did you learn from it?

Al-Hayderi: So, I mean, first off, we had our sort of snowflake-y pipeline that would deploy our own little service and change traffic and compare metrics. So, I guess the first part about it was how do we route traffic and what we actually do is, in a single pipeline, we’ll deploy containers to a new supergroup and we’ll, that’s using the V1 provider of Kubernetes.

Ashley: Mm-hmm.

Al-Hayderi: And then we actually used the V2 provider of Kubernetes to apply a virtual service manifest. Virtual services are just a kind of way of routing ingress traffic based on some set of rules. So, if the host header matches this and the path has this Regex, route it to this service.

What you can also do is, when you match a rule based on some ____ path prefix is route to several services based on weights. So, we first experimented with how are we gonna get a small amount of traffic, yet a meaningful enough amount of traffic to get a good enough sample size of data.

So, a lot of our, at the start, what we were doing was just sort of like load testing and seeing, “Okay, is 5 percent of the traffic to the Canary deployment enough? How about 10?” et cetera, et cetera. We also looked at some common best practices around Canaries and how they recommend running baseline deployments. So, what we would actually do is deploy two new deployments. So, one with the new code on a small sized server and one with the old code on a brand new, deployed small server. That way, you sort of take away a lot of variables from the experiment, like long running processes, et cetera.

So, we got there, got traffic routing correctly. Now, the next part we had to do was metrics, right? How do we analyze the delta between the two deployments? At this point, Kayenta only supported Datadog, at least out of the metrics providers that we used.

Ashley: Okay.

Al-Hayderi: So, what we really wanted to do, you know, going back to the whole point of getting Canaries out was stop deployments that are just plain broken, right? So, kind of getting, like, a cheap, easy way to get smoke tests. And so, we were just looking for server errors.

So, as we were trying that, the problem is, is that the server error metric that Istio gives you will only send metrics of a server error happens. So, between two deployments, if you only get two server errors on one, that will trip the Canary as a failure. It doesn’t give you the option to aggregate metrics, so we couldn’t do things like check error rates. So, that just really never worked.

And actually, in my talk, I go into sort of the—this was kind of a big, big problem the first time we actually released this into production is that Canaries would just fail a lot for no reason, right? And Canaries running for an hour long, you know, our release engineer is sitting there waiting for an hour and then has to trigger it again just because, you know, one server error happened on the Canary deployment.

Ashley: Well, it sounds like maybe one of the lessons you learned or concept you came up with is, don’t cast a big net. Start out with sort of higher, larger grain, most fundamental issues, kind of a ____ kinda concept. Start there, get those—get that quality fixed, improved, so you’ve got those things working better, probably tighten the net a little bit more and filter out some more specific things that are causing issues. Does that sound like the approach that you learned how to take?

Al-Hayderi: That’s exactly it, and actually, what I would recommend is, the first time you give out Canaries, don’t have it actually fail or roll back the deployment if the Canary was unsuccessful.

Ashley: Mm-hmm, okay.

Al-Hayderi: What we actually ended up doing was actually running this for a month and just gathering data, seeing which metrics worked, which metrics didn’t, and mapping up Canary reports with what we actually saw as problems in production. That way, that gave us more information on which metrics to use. We ended up going with average latency and 95 P latency and once that was out, we started actually aborting and rolling back deployments and then we started to get a lot of great value of this tool.

Ashley: Mm-hmm. Yeah, that’s actually very similar to kind of a quality process, right? You don’t solve all problems at once. You start, you prioritize, you filter, you work done—you know, improve it one step at a time, rather than just throwing a canary out there and seeing what happens, which sounds like you’ll get results, but you don’t know what or why, right?

Al-Hayderi: Yep, exactly.

Ashley: You’ve gotta dig into it and be a little more thoughtful about it, it sounds like.

Al-Hayderi: Yep.

Ashley: Good. Anything else? Any kinda other big things you’re expecting to talk about as part of this?

Al-Hayderi: So, another big thing we learned was, even after this, when we had, you know we were confident that our Canaries were failing when they should and passing when they should. The problem we got was, when a Canary failed, people didn’t really know what to do.

Ashley: Mm-hmm.

Al-Hayderi: Because, you know, when a deployment fails, we have a lot of data and we have a lot of observability into that. The Canary is getting a lot less traffic, and it’s hard to know exactly what failed just by looking at the Spinnaker UI Canary report.

Ashley: Mm-hmm.

Al-Hayderi: So, we then, you know, sort of invested more into our—we used New Relic for our APM and we piped our Canary into that so that we could actually see transaction data and the delta between it and the baseline metrics to see, you know, which end point, for instance, was misbehaving or was some new database core utilizing this—that helped us a lot, too.

Ashley: Mm-hmm. Interesting. Well, it sounds like you’re using—think about this in a systemic way, not just Canaries and Spinnaker, but also you mentioned Datadog, New Relic, a number of tools that you’re using as part of your environment, which all have to come together to figure it out, right? It’s not just one thing.

Al-Hayderi: Yep.

Ashley: Well, good. I wish you the best in your talk. The folks that would go to this talk, are they gonna be, tend to be software developers, engineers, architects, operations focused on the DevOps team? Who tends to be more interested in this than others?

Al-Hayderi: I would say more the DevOps operational people. People who work with the Spinnaker pipeline definitions at their company would get the most value out of this. But, you know, back when I was just a developer, I heard a talk about Canaries, and that sort of triggered me to bring this in. So, I mean, anyone who’s interested in resiliency and, you know, release confidence would get value out of this talk.

Ashley: Okay, excellent. Very good. Well, thank you for being on the podcast, Omar.

Al-Hayderi: Thank you very much, Mitch.

Ashley: It’s been great to have you, Omar Al-Hayderi, who is engineering manager, formerly a principal engineer, at Autodesk. He’s gonna be speaking at the Spinnaker Summit 2019 in San Diego. The dates for that is November 17th through the 19th, and Omar’s talk is on Sunday, the 17th, at 1:30 p.m.

So, thank you, all of you, for joining us and listening to this episode of DevOps Chat podcast. This is Mitch Ashley with devopsy.kinsta.cloud. You’ve listened to another DevOps Chat. Be careful out there.

— Mitchell Ashley

November 13, 2019
DevOps Chats: Debugging Spinnaker Apps, With Salesforce

Spinnaker Summit 2019 Preview: Debugging production issues in any environment can be challenging, and Spinnaker has its production learning curve. Problems aren’t always replicable in a smaller environment, and debug messages can be verbose and confusing to triage what’s happening.

Our DevOps Chat guest Chuck Lane, Salesforce lead software engineer, is giving a talk on “Debugging and Profiling Spinnaker Applications Live” at the Spinnaker Summit 2019. In Chuck’s talk, you’ll learn skills like remote JVM debugging, custom profiling builds and the magic of figuring out what’s going on with a multithreaded microservice using htop.

Chuck’s talk is on Saturday, Nov. 16, at 3:40 PM PT. Spinnaker Summit 2019 is Nov. 15-19 in San Diego.

As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

Transcript

Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Chuck Lane. He’s a lead software engineer with Salesforce.com. He works on Sales Cloud, which is the Salesforce you and I use, know and love, or our CRM capabilities. Chuck is talking at the Spinnaker Summit 2019 in San Diego, and his talk is about debugging and profiling Spinnaker applications live. And I think live means, like, live while they’re running, in production, and see what’s going on. We’ll explore what that is. This talk is on Saturday, November 16th, at 3:45 p.m. in San Diego. Chuck, welcome to DevOps Chat.

Chuck Lane: Thank you. It’s nice to be here.

Ashley: Excellent. Awesome to have you here. Would you start by just introducing yourself? Tell us a little about you and what you do at Salesforce.

Lane: Yeah. So, my name is Chuck Lane. I am, as you mentioned, a lead software engineer. And basically, my chief responsibility at Salesforce is to help bring Salesforce into using Spinnaker for our public cloud based deployments. So, as we make a transition to try to move away from first party architecture and move over into the public cloud, one of the technologies that we’re using to do that is Spinnaker and I’ve established myself as a subject matter expert in Spinnaker. And so, I kinda help to bridge the gap between the traditional development and deployment cycle and what that looks like using Spinnaker in a containerized world.

Ashley: Okay, great. You used a term there I don’t know if I’m familiar with. You mentioned something about moving from first party architecture to the public cloud. What’s that mean about the environment you’ve been in that you’re, and what you’re moving to?

Lane: Yeah. So, basically, what I mean there is that, historically, Salesforce has owned their own data centers, and we’ve deployed to those data centers. We are moving, embracing in a big way public cloud architecture, whether that’s through AWS, whether that’s through GCP, or any of the number, any other number of cloud offerings.

And so, as opposed to doing what some people would call a lift and shift where you just take your software that’s meant for, to be run on hardware that you own and move it into the cloud, we’re doing a fundamental re-architecture of the software to make use of all of the great things that cloud architecture allows us to take advantage of.

Ashley: Mm-hmm. Fantastic. That’s interesting to hear a little bit about your own evolution at Salesforce. So, your talk is about using Spinnaker, you know, really about debugging and finding out problems that are happening in production, but sometimes you might struggle with trying to replicate it into a smaller environment to get the same problems working.

Can you tell us a little bit about what sort of led you down this path to figure out how to do this? Was it a big problem that was happening or something you saw repeated over and over that there wasn’t a good solution to and you figured out how to do that? How did this come about?

Lane: Yes. So, basically, as you’re taking workloads and migrating them over, I mean, when—you know, when you just start with a small subset of workloads, you know, the path is relatively straightforward. And, you know—hopefully, anyways—and you don’t run into too many issues that can’t kind of easily be solved.

But the reality is, as you transfer more and more workloads over to the public cloud, that’s—you know, the devil’s in the details, right? So, that’s really where things can pop up where you’re hitting various limits or software isn’t performing in the way that you would expect it to perform. And these are the situations where we—where it can be very beneficial to jump into production software and really, you know, sometimes slap a debugger on it and see exactly what it is that it’s doing that differs from kinda what you expect. You know, and that can help you kinda tailor your workloads in such a way that you can make things run more smoothly.

Ashley: Mm-hmm. I know sometimes using debuggers, enabling them—that can be helpful, but it also can be too much information, trying to sort through what all is happening, trying to find what should you be looking at. What are some of the challenges you’ve found by turning on debugging or using debuggers?

Lane: Yeah. Well, so, I mean, there definitely is that problem that you’re saying just as far as too much information that’s coming out. One of the nice things about Spinnaker and the way the services are written is, you have a very fine grained tuning over which libraries inside of Spinnaker you tell it to print out and debug information.

Ashley: Mm-hmm.

Lane: So, if we know that there is a hiccup with one of the data binding layers or a hiccup with one of the authorization layers, then we can, through the config files, we can—and Java settings—we can really target that explicit directory, or I’m sorry, that explicit library and say, “Give me all the debug information that you have.”

Ashley: Mm-hmm.

Lane: But, you know, failing that, I mean, there are definitely times when it’s been advantageous and the best course of action is just to go down and actually look at the code and see what it’s doing. And, again there, it’s best to do that under—you can’t do it in production, usually, but what we can do is simulate a load that’s similar to what we would see in production and really, really take a look at what’s going on to help us identify those algorithms that might be o event squared instead of a login or something like that.

Ashley: Mm-hmm. Now, are there some specific techniques—I know in your description of your talk, when you talked about using remote JVM debugging, custom profile builds, even using htop, you know, a UNIX command or a LINUX command to help you figure out what’s happening with multi-threaded microservices. It sounds like a variety of different approaches that you’ve kind uncovered and learned that you can use to figure out what’s happening.

Lane: Yeah. And so, I mean, in general, we like to run as closely to the open source build as we can, the ones that are provided by Spinnaker, which is ultimately provided by Google. But there definitely are circumstances where we need an extra tool, be it htop, be it something like Glowroot, which is a Java application profiler where what we’ll do, then, is we’ll go in and we’ll build a custom image using the Spinnaker images as a base and then tack on those additional libraries and put them into the Spinnaker ecosystem and then launch them up just to see what additional information we can get out of there.

And so, you know, one scenario where that was really useful to us, we ran into a scenario where cloud driver, which is—cloud driver is the main tool that talks back and forth to all of our different cloud infrastructures.

Ashley: Mm-hmm, mm-hmm.

Lane: Sometimes calls to cloud driver were taking upwards of, well, two to three minutes to respond. Now, you know, timeouts are set in such a way that if it doesn’t hear anything back in about 30 or 60 seconds, then it just disregards that load and, you know, so that created quite a bit of problems for us.

By building a custom cloud driver image that had htop in it, we were able to see that the majority of the processes that were running were actually running basically commands to reach out to the Kubernetes clusters and get a list of all of the name spaces that are in the Kubernetes cluster. And, talking it over with some of the Spinnaker developers, what we found is that if you have 15 or so clusters that you’re connecting to Spinnaker, then that’s not so much of a problem to go and query each of those to get the list of name spaces. But as you scale up, on the order of 400 or 500 different clusters the way that we do, then a lot of the delays and a lot of your time can be just essentially Kubectl calls waiting to return to you the lists of name spaces that you need to go and scan.

So, we implemented a workflow based solution for that, basically, where we let our teams know that when they’re creating their clusters, they should use Terraform to go ahead and create the clusters—I’m sorry, the name spaces that they’ll be using. And then, if they need to use a name space after the fact, then we provide a pipeline that they can use that will dynamically add in an additional name space that their cluster will start to scan. So, that saves us from scanning all the clusters all the time, which was causing quite a bit of performance bottlenecks.

Ashley: Yeah, I would imagine that would have some overhead, maybe a lot of overhead if you’re doing that frequently. So, it sounds like a way to both reduce overhead, but also to get the information faster.

Lane: Exactly, exactly—yep, yep.

Ashley: Mm-hmm. Very cool. So, in your talk, are you going to be doing any demos? I know there’s always the demo Gremlin, or are you gonna be just showing, you know, talking about what some of these techniques are?

Lane: So, I plan on doing some demos. As far as—

Ashley: That’s really cool.

Lane: Yeah. You know, I may do the, what’s the—the Easy Bake Oven a little bit as far as, “Here’s the behavior and here’s what we’ve found in code.” But from what I understand, a lot of that stuff can kinda be dependent on Internet connectivity at the actual site. So, if it’s something where we have good Internet connectivity, then by all means, I plan on walking through a couple of debugging scenarios, as close to what we would do in real time as possible.

Ashley: Yep. Well, you know, someone somewhere—it’s been a while ago—gave me the great advice of, you know, “Have your live demo and have your disconnected demo ready in the background.” [Laughter] So, you can always at least show something locally if you can’t get on the net. So, if you’re depending upon the network—I’m not sure if you are for your demo, but always a good lesson, right?

Lane: Sure, exactly.

Ashley: Great. Are there any other kind of lessons learned, common mistakes, or mistakes that you or others might have made along the way, kinda so there’s hard things you learned by trial and error that you plan on sharing?

Lane: I mean, well, there are—whew. You know, I’ve been working on Spinnaker for a couple of years. So, there’s definitely been a lot of hard lessons learned, here. But honestly, what I would do is, I would encourage anybody who wants to get into Spinnaker to not just hang around the Slack channel, because the Slack channel does have a tendency to get overrun with, you know, just kinda people posting their stack traces and just saying, “Hey, has anybody ever seen this before?”

And, you know, I’ve gotten a lot more success by going through the commits, looking at the people who actually authored the code, and then reaching out to them directly with more than, “Hey, can you explain this to me?” but rather, you know, “Hey, I see what you did here and I see what you did here. I’m running into problems with these lines. Do you have a different approach, or is there something that I can be doing differently?”

And the other thing that I just can’t overstate is the value of being a member of one of the special interest groups.

Ashley: Hmm, interesting.

Lane: So—yeah. So, I’m a member of the Kubernetes V2 special interest group that’s lead by Eric Semene and Ethan Rogers from Armory, Eric’s from Google. And it has, you know, it has been just an absolute wealth of information and, you know, honestly, I don’t know if we ever would’ve gotten nearly as far as we had without those two people.

So, you know, yeah, I would just say that the community is really friendly and we always welcome new members who are ready to learn.

Ashley: You know, I think both of those are great suggestions, and I really appreciate that you made those. Because, one, things like the Slack channels on projects, open source, those can be a bit intimidating. Sometimes they’re not approachable, because there’s just so much noise so much happening on it and people reaching out for help like you’ve talked about, “Here’s a stack trace I’m trying to figure out.”

But also, your recommendation of reaching out to the code authors, you know what, it’s—people like to help each other. And, you know, it might seem like, “Hey, the people who wrote this aren’t gonna have time to bother with me”—they love to hear from people that are using their stuff to talk about it and help them out, but also hear about how they’re using it, and of course, they’re always looking for ideas and feedback and stuff like that. It sounds like you’ve had that kind of an experience.

Lane: Yeah, absolutely. I mean, and you know, the big thing is just, you know, as a coder, it’s easy to tell the people who are coming to you and just kinda wanting you to fix it for them and the people that are coming to you that have really tried to tackle it themselves. And, you know, I can’t speak highly enough about the latter rather than the former. I mean, you know, just give it your best shot and when you get stuck, reach out to somebody and it’s, you know, it can be immensely valuable.

Ashley: It’s kinda like going to a foreign country. Wherever you are, if you’re American going somewhere else or vice versa, everybody appreciates you trying to speak the native language, and at some point, when they see you struggle enough and how far you can go, they’re glad to help you and, you know, speak in your language.

So, same kind of thing with helping people with code. If you’re gonna just fob your problem off onto the developer of it—not appreciated so much. But they appreciate that you tried to take it as far as you could. As you mentioned, go look at the code and figure out what’s going on.

Lane: Yeah.

Ashley: You’ll get immense respect, you know, even if you aren’t a coder at that level, at that level of software developer, it’ll mean a lot to the developers.

Lane: Absolutely, absolutely.

Ashley: Well, hey, I think you’re gonna have a fascinating talk, and I love that, you know, this idea of trying to figure out what’s happening in production and some of the techniques that you’ve come up with and developed and have experienced and the fact that you’re sharing those with others. I’m curious, do you contribute any code to the Spinnaker project in any areas? Are you primarily a practitioner user of it?

Lane: So, I have. I’ve got a few small RPRs that have been pushed through, but really, the bulk of my commits right now have actually been to the Spinnaker website, the documentation. So, you know, I don’t know Java or Groovy or Kotlin, maybe, as well. Some of the patterns they use are a little bit—I come from a .net world, so they’re a little bit foreign to me.

But yeah, I’ve definitely written a number of different documentation pages and I found that that’s a great way to kinda get in and I’ve even got some PRs that are coming ups soon that aren’t doc related. So, yeah, hopefully, you’ll see my name more.

Ashley: Excellent. Well, you know what, documentation is important, too. There’s some folks doing talks at Spinnaker Summit that, you know, kinda ran into that point where, “Hey, there’s documentation for doing it one way, but not under this set of configurations or software.” So, that’s a contribution, too, so congratulations for being a part of the community and for sharing, also, your experience at the Summit.

So, Chuck, appreciate you being on the podcast today.

Lane: Oh, thank you. It’s been an absolute privilege. Thank you so much.

Ashley: Absolutely. Fantastic. I wish you all the best with your talk, I’m sure it’ll be great, and hopefully folks listening to this podcast will draw some more interest and bring folks to listen to you.

So, I’d like to thank our guest today, Chuck Lane. He’s lead software engineer at Salesforce.com, so you can imagine the environment he’s working in. There’s some super good lessons that Chuck’s bringing to the table. He’s gonna be talking at Spinnaker Summit 2019, which is in San Diego, November 15th through the 19th. His talk is debugging and profiling Spinnaker applications live, and his talk is on Saturday, November 16th at 3:45 p.m.

I’d also to thank you—you, our listeners—for joining us today. This is Mitch Ashley with devopsy.kinsta.cloud. Have a great day and be careful out there.

— Mitchell Ashley

November 12, 2019
DevOps Chats: Spinnaker Extensibility, With Armory

Spinnaker Summit 2019 Preview: Most open source software isn’t one size fits all, and Spinnaker is no different. While it captures the most general use cases of software delivery, most organizations find that it lacks some features they need to automate their entire delivery pipeline, such as integrations with internal tooling or compliance systems.

Out of the box (figuratively), Spinnaker provides Run Job and Webhook stages that teams can use to build custom integrations to help fill any void and enable adoption by a wide range of users.

Ethan Rogers, staff software engineer with Armory, joins us on DevOps Chat to share a preview of his Spinnaker Summit 2019 talk on Spinnaker extensibility. Ethan shares how Run Job and Webhook stages are used to build custom stages to capture any number of use cases with no code and simple configuration. He also covers situations when it’s necessary to write code.

Ethan’s talk, “Making Spinnaker Your Own,” is on Saturday, Nov. 16, 3:45 PM, at Spinnaker Summit 2019 in San Diego. Separately, Ethan appears on the “State of the Kubernetes V2 Provider…” panel Friday, Nov. 15 at 2:30 PM.

As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

Transcript

Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Ethan Rogers. Ethan is a staff software engineer and also a speaker at Armory. He’s gonna be talking at the upcoming Spinnaker Summit 2019 in San Diego. Now, Ethan’s gonna be talking about making Spinnaker your own, that’s his talk, and then he’s also gonna be on a panel about the state of Kubernetes V2 Provider, plus improved rollout strategies for deploying Kubernetes using Spinnaker.

So, let’s get started. Ethan, welcome to DevOps Chat.

Ethan Rogers: Thank you so much for having me. I’m really excited to be here and get to talk about Spinnaker.

Ashley: Wow. Absolutely. Great topic. I know you’re passionate about it, so fantastic to have you. Would you just introduce yourself? Tell us a little bit about you, what kinda work that you do and a little bit about Armory, the company that you work for?

Rogers: Yes. So, like Mitch said, my name is Ethan Rogers, I’m a staff software engineer at Armory. I’ve been actually working on the Spinnaker project for about the last three years. I’ve been at Armory for a year and a half, and then a year and a half before that, I was working for my previous company where we were, you know, I was part of a team for delivery and developer experience. And, you know, we were using Kubernetes for this new project that we were working on and looking for a way to deploy and got involved in Spinnaker super early on. So, I’ve kind of been a part of the community ever since then and it’s been a really great experience so far.

Yeah, so, that is actually kind of how I found Armory. Closer to the end of my time at my last job, I met the founders, DROdio, Ben and Isaac, and they had just—eh, within probably six to nine months started Armory. And we kinda hit it off. I had a really good relationship with the founders and it was kind of an opportunity to go keep working on Spinnaker full time, which is something I was interested in.

So, Armory, just a little bit about us—we’re bringing Spinnaker to the enterprise. So, what we’re doing is taking open source Spinnaker and, you know, on top of offering stuff like support and training, we also offer layers on top of open source Spinnaker, various different extensions and features that make it more aligned with what the enterprise is looking for today.

Ashley: Mm-hmm. Very good. Well, that explains, I know there’s yourself, but there’s also some other folks that are speaking at the conference.

Now, your topic, you gave a nice introduction about how you got involved with Spinnaker—how long ago was that, when you started using it?

Rogers: Yeah, it was about three years ago, kind of before the—like, right when the first version of the Kubernetes provider landed. I think, at the time, I didn’t realize how early we were. It kind of explains a lot of the PRs I had to submit when I first got involved.

Ashley: I’ll bet, I’ll bet. Do you write any code, submit any code or are you primarily on the usage end of Spinnaker?

Rogers: Yeah, so, I’m actually a developer by training. I got my degree in computer science, and transitioned pretty early on in my career into the ops side of things. So, I’m pretty fluent in both. I actually do write a lot of code. These days a lot less, just because of the role I’m in. At Armory, I do a lot more higher level stuff like BitTorrent, but I still try to get a few PRs submitted every week. I lead a couple of initiatives in the Kubernetes provider itself, so I do a little bit of everything.

Ashley: Well, I won’t ask you which is the dark side, ops or developing—no, I’m just kidding. It’s DevOps, it’s all together now, right?

Rogers: [Laughter] Yep.

Ashley: Okay. So, let’s jump into what you’re gonna be talking about. And, of course, we’re not gonna cover it all here, but I know that you’re focusing on kinda how open source, you usually don’t just use it out of the box, if you wanna use that term. I know it’s free and downloaded.

Rogers: Mm-hmm.

Ashley: But that you have to do some customizations to kinda get it working in your environment, and we’re talking about Spinnaker here. What’s your experience working with customers? What’s their expectation when they start to use Spinnaker and what do you have to kinda help them get started with?

Rogers: Yeah. So, the interesting thing about Spinnaker open source is that it’s actually, like, a very small part of what Spinnaker is at Netflix. For anyone listening who doesn’t know, Spinnaker was a project that was born out of Netflix. They built it, it was the next iteration in deployment technology from a project called Asgard.

Ashley: Mm-hmm.

Rogers: And one of the driving principles of Spinnaker was making it extendible. And the primary reason for that was because Netflix has a lot of kind of internal workflows and things that they like to—that make their development experience better, that don’t necessarily apply to the broader community.

And so, when you start using open source Spinnaker kind of out of the box, there are a lot of things that I think most companies want, right? The same reason that Netflix has those, the things that don’t apply to the community but work for them, every other company has something like that.

Ashley: Mm-hmm.

Rogers: So, when I was at my previous company, we had this workflow around our Docker images where we would add labels to our Docker images and then we wanted those labels to show up in the Kubernetes manifest where those images were deployed.

So, that was something that wasn’t available in Spinnaker out of the box, but we were able to use some of the extension methods to actually go grab that information and inject that stuff, which is kind of the basis of the talk I’m giving around Run Job.

Ashley: Okay, great. Well, let’s explain that, then, because I know you’re gonna be talking about Run Job and about Webhook Stages. What is Run Job, what is Webhook?

Rogers: Sure, yeah. So, just kind of, at a high level, Run Job and Webhook are ways that you can extend Spinnaker to kind of make it—to do the things that it doesn’t do out of the box that you need it to.

Ashley: Mm-hmm.

Rogers: There are more complicated ways to extend Spinnaker that involve actually taking the code and writing Java code, but we found that the majority of users or operators, I should probably say, are not super familiar with Java. So, that’s a pretty—it’s fraught with friction, I would say.

Ashley: Yeah, it’s a whole other ball of wax, right?

Rogers: Yeah, exactly. Yeah. So, the Run Job stage and the Webhook stage give kind of a lower friction way of extending Spinnaker. So, the Webhook stage is a stage within Spinnaker that lets you call out to some API. There’s the, you know, I select it as a stage in my pipeline via drop down and I configure it there. There’s also what we call preconfigured Webhooks or custom Webhooks that allow you to actually configure this as part of Spinnaker’s configuration, and then it shows up like it’s a native stage.

Ashley: Mm-hmm.

Rogers: So, under the hood, it’s still calling out to APIs. It’s as if you’ve configured it in line, but it shows up and, to the end user, actually looks like it’s part of Spinnaker itself.

Ashley: Very nice. So, what are some of the most common uses of that to extend integrations into other things? Is it internal third party tools, internal custom software people who built, is there, you know, typically—the typical use cases for this?

Rogers: I think that we usually end up seeing people integrating more with internal systems than, say, hosted services like GitHub or other—I don’t know, GitHub’s probably the easiest example. New Relic is another one that we see.

But I think where, in most cases, Spinnaker has integrations for the public stuff out of the box, but obviously, if the tool’s internal to your company, you don’t have that. So, that is usually where we see it. So, I think there was one company that I had talked to that was, they had this internal integration testing suite or load testing suite that they wanted to call out to, and they used the Webhook stage to make that call and kick off those tests.

Ashley: Mm-hmm. Great, great. Now, I understand you can do a lot of this without writing code, is that correct?

Rogers: Yeah. Yeah.

Ashley: So, what does that entail? What kind of a—what’s the configuration process look like to tie, let’s use that example that you gave of someone that wrote their own test framework of some kind that they wanna call out to in a Webhook stage?

Rogers: Yeah. So, I guess I should probably back up and clarify my answer a little bit.

Ashley: Okay.

Rogers: So, at some level, you will need to write code. The question is whether or not you actually have to write that into Spinnaker itself.

Ashley: Okay.

Rogers: One of the, I guess we can call it native extensions, which is essentially using Spinnaker as a library and writing Java and doing some spring magic to make it happen.

Ashley: Mm-hmm.

Rogers: So, if you were to use a Webhook or a Run Job stage, for that matter, you would have to write a little code. But the benefit of that, if it’s a Webhook the service, whatever service that you’re trying to connect to can be written in whatever language that you want, right? So, if it’s Go, if it’s Python, if it’s Ruby, if it’s—you know, whatever you wanna do, the service is running in the API as the contract that you expect.

Ashley: Mm-hmm.

Rogers: And so, with the Webhook stage, all you’re doing is making a call. You know, you could argue that writing a cURL command is writing code, but I think a lot of people are familiar enough with that, that it’s a pretty low overhead.

Ashley: Mm-hmm.

Rogers: So, if you can make a call in your terminal, you can make a call from Spinnaker with the Webhook stage.

The Run Job stage is actually a little more interesting, because what it allows you to do is run a container as if it were a script that you might be running. So, a really common pattern for doing custom logic in CI systems specifically is, I have a container, the container has all my dependencies and it has a script that I need to run. And we just run it like a one shot, you know—once this script finishes, if it fails, the pipeline fails. If it succeeds, the pipeline continues. And then, like, the world is really open, the world is your oyster at that point, because you can write the script in whatever you are most comfortable writing, and then you can run that within Spinnaker.

One of the more interesting things that we did with the Run Job stage is that you can actually output—you can output these special markers in your logs and Spinnaker will go grab those and inject them back into the pipeline so that you can use them in downstream stages. So, maybe your job is going off and collecting some information for you. You can actually use that in, like, say, your deploy stages or a downstream Run Job stage or Webhook stage as, like, arguments you pass down.

Ashley: It’s almost used like the log is like an asynchronous message bus, to make it more complicated than it is, but a super simple version of that.

Rogers: Yeah, yeah.

Ashley: Some down stage version where you could look at what happened before it and take appropriate action.

Rogers: Yeah. Yeah, and that’s really—you know, we heard, the motivation kind of behind that feature is, we heard a lot of users, like, you know, they wanna be able to extend Spinnaker, but they don’t wanna write Java code, you know? And containers, for the Run Job stage specifically, containers are obviously super prolific in our industry today. I’ve built my career off of Kubernetes and containers, so.

Ashley: Mm-hmm.

Rogers: Containers are kind of like the perfect medium for being able to do this, because you can contain everything that you need—[Laughter] you can contain everything that you need in this container, and then, because Spinnaker can run containers in Kubernetes, it’s kind of like a natural fit, right?

Ashley: Mm-hmm.

Rogers: We can kind of take advantage of the fact that it doesn’t matter what your code is, the interface that we need is a container and then the logs, and we can get all of that information out.

Ashley: I’m curious—this is totally a side question, but did you learn anything in college about containers and that kind of thing in software architectures, or is this something that you really started in after you graduated?

Rogers: I wish I had learned something in college about this. So, I actually, I graduated college in 2013, and got my start writing PHP and doing a lot more software development. But in the back of my head, I’d always been like—I am very interested in servers.

Ashley: Mm-hmm.

Rogers: So, writing the code was interesting to me, but it always seemed like the people with all the power and all the really, people that I looked up to, they were the guys, like, in a shell, deploying software, checking, like, “Is this server overloaded?” It was very much not DevOps, but it was something that really interested me.

Ashley: Sure, yeah.

Rogers: And I—you know, I kind of, I spent probably three years as a software developer without having done that. But once I transitioned kind of into this CI/CD space, that really gave me an opportunity to focus more on ops, while doing some development and, you know, we picked up Kubernetes at my last job, and that’s where it all kind of kicked off.

So, I think we started on Kubernetes 1.4, and that’s really where I learned everything I know about containers and orchestration platforms and stuff like that.

Ashley: Well, it puts you in a really great position, because you can speak to both sides, both audiences, if you will, you know, developers and operations folks.

Rogers: Yeah.

Ashley: And, you know, my experiences before kinda containers and all of this, all the DevOps environment, some of the best developers were engineers who really knew how to write code, but they knew the full stack down to the operating system and the network, right? They kinda knew enough of all of it. Maybe there were really good parts of it, too, besides writing code.

Rogers: Mm-hmm.

Ashley: There’s a lot of, I don’t know, power, if you will, in understanding that full stack and how it works.

Rogers: Yeah. I found that the more you understand about the stack, the better positioned you are to take advantage of it, right, and to actually write better code. You can go back and probably look at the birth of DevOps and see this, but engineers that spend the majority of their time writing code and then throwing it over the wall are more likely to write less performant code, right? Code that is not aware—you know, we wanna write really abstract code, but at some level, you have to be aware of what type of environment you’re running in.

So, if you’re running on a machine that has 128 cores and hundreds of Gigs of RAM, it doesn’t really matter how inefficient your algorithm is, because it’s gonna run fast. You know, it all depends on scale, but … I’ve found that engineers that understand kind of the cloud and the environments they’re running in and what the actual, you know, where are your network boundaries and what are your actual constraints are writing better code, they’re shipping more frequently, they’re iterating—like I said, they’re iterating more quickly and that all of that is super important.

I think the other thing that I’ve found is that it’s a growing process, right? I kind of like to look at it as—well, at least in my personal experience, I’ve found that I have a hard time grasping, kind of, the next level of what I’m doing until I’ve fully understood the previous level. And I think that’s really true for a lot of people, right? Until you’ve got a solid grasp of programming, it’s kind of hard for you to imagine what it would be like to understand ops, right? To understand how your program is actually running on servers.

Ashley: Yeah. I would describe it as kinda context. It helps to know when you understand the environment of what you’re operating in and what you’re doing, what you’re building is going to operate in. And vice versa, right? If you’re in operations or security now, with DevSecOps, understanding now how software is being built. So, it’s the great thing about the accessibility of resources around the cloud and contemporary software architectures and all of that.

Rogers: Yeah.

Ashley: Well, we’re running out of time. I wish we had more time to invest in this, but the great thing is that you have your talk coming up, so that’s where you’re really gonna get into more details about that. And Ethan, I thank you for being on the podcast.

Rogers: Of course, man. Thanks for having me.

Ashley: Absolutely. Ethan Rogers, staff software engineer and also speaker from Armory, he’s speaking at Spinnaker Summit 2019, which is on November 15th through the 19th in San Diego. Now, he has a topic that we were just talking about, “Making Spinnaker Your Own,” that’s on Saturday, November 16th, 3:45. He’s also on the panel about the state of Kubernetes V2 provider. That is on Friday, the day before—the 15th at 2:30.

So, thanks to you, all of our listeners, for joining us for this episode. This is Mitch Ashley with devopsy.kinsta.cloud, and you’ve listened to another DevOps Chat podcast. Be careful out there.

— Mitchell Ashley

November 12, 2019
DevOps Deeper Dive: SUSE Sets DevOps Course

There was a time when SUSE was considered to be the second-biggest force driving the adoption of Linux in the enterprise, after Red Hat. Then came an odyssey to rival any epic written by Homer. A few short years after moving its corporate headquarters from Germany to Oakland, California, the company was sold to Novell, which was then later acquired by Attachmate. Three years later, Attachmate was acquired by Micro Focus. Three years after that, SUSE (as an arm of Micro Focus) acquired distributions of OpenStack and the Cloud Foundry platform-as-a-service (PaaS) from Hewlett-Packard Enterprise (HPE).

That brings us to 2018, when SUSE was spun out of Micro Focus to become an independent company once again, headquartered in Germany once again. A year later Melissa Di Donato, formerly chief operating officer (COO) of SAP, was named CEO of SUSE. A few months after that, SUSE announced it would withdraw its distribution of OpenStack from the market to focus its efforts on emerging cloud-native opportunities stemming from the rise of Kubernetes.

For SUSE, those opportunities arise from a distribution of Kubernetes deployed on its Linux operating system, which also serves as the foundation for its container-as-a-service (CaaS) environment, along with a version of Cloud Foundry deployed on top of Kubernetes.

Brent Schroeder, global CTO for SUSE, said his company’s strategy is being driven by an acute need for increased flexibility. IT environments are inherently becoming hybrid as more workloads move to the public cloud, and many organizations will want to leverage a common substrate based on Kubernetes for building and deploying applications. In some instances, Kubernetes will be deployed on top of virtual machines, while in other use cases Kubernetes will be deployed on bare-metal servers. IT organizations will then decide to deploy a CaaS or PaaS depending on their requirements, said Schroeder, noting that regardless of the path chosen, IT modernization initiatives will create a significant opportunity for SUSE.

“Virtual machines, bare-metal and the cloud will be a huge catalyst,” said Schroeder.

DevOps, of course, is playing a major role in driving those transitions, which Schroeder said has become a major focus for SUSE as organizations think through how to not just build and deploy applications faster, but also secure and govern those processes. That latest version of SUSE CaaS Platform 4, for example, added support for Cilium open source project for securing network connectivity between containerized application services.

Of course, SUSE is not the only IT vendor making this journey. Red Hat was the first to deploy a PaaS environment on top of Kubernetes and since then has been on a mission to obliterate the distinction between a CaaS and a PaaS environment. As an arm of IBM, Red Hat has also significantly expanded its market research as IBM builds out a hybrid cloud computing strategy based on the Red Hat OpenShift platform.

Of course, Red Hat’s acquisition by IBM might create new opportunities for SUSE. After all, more than a few of the IT organizations that opted to embrace Red Hat OpenShift had plenty of opportunities to consider alternatives from IBM. Now that the acquisition has been finalized, some of them may be reconsidering their options.

It’s too early to say when SUSE might arrive at its ultimate destination. When venture capitalists buy a company such as SUSE the goal is to either take the company public or recoup the investment by selling it to another entity. Whatever the outcome, it’s clear SUSE intends to leave its mark in the brave new world of containerized applications.

— Mike Vizard

November 11, 2019
Mario’s Adventures in Tekton Land
Tekton is an open source, cloud-native CI/CD pipelines engine. The project has its roots in the Knative project, where about a year ago it started under the name “Knative Pipeline,” with the aim of providing an extension to build Knative’s own “source to image” service. About eight months ago the project was reborn as Tekton and it moved out of the Knative space to its own GitHub organization and it joined the newly formed CD Foundation with a new mission: becoming the common denominator for CI/CD in the cloud-native space. But how is Tekton going to achieve its new aspirations?

For it to be successful, we wanted Tekton to be intuitive, reliable and easy to operate and maintain. This was discussed in the Tekton working group, and we thought it would be useful if we started using Tekton ourselves as part of our day-to-day jobs, using it to provide CI/CD services to the community.

Diving into Tekton

Tekton pipeline aims to provide a set of components used to create a full-fledged cloud-native CI/CD system on top of Kubernetes.

The core project of Tekton (tektoncd/pipeline) defines a set of Kubernetes custom resources (CRD) as standard constructs for creating CI/CD pipelines. The following is a brief introduction to these CRDs:
- Task: A sequence of commands (steps) that are run in separate containers in a pod.
- Pipeline: A collection of tasks executed in a defined order.
- PipelineResource: Inputs (e.g. git repo) and outputs (e.g. image registry) to a pipeline.
- TaskRun: Runtime representation of an execution of a task.
- PipelineRun: Runtime representation of an execution of a pipeline.
Those pipeline building blocks can be reused, version controlled and curated in a catalog (tektoncd/catalog) that embeds best practices.

In addition, the Tekton community works on providing more components:
- Tekton Triggers (tektoncd/triggers) to be able to act upon events (for example: GitHub webhooks, CloudEvents and more).
- Tekton CLI (tektoncd/cli), which provides a command-line interface focused on the Tekton APIs and features (like getting the logs of your run, starting pipelines and more).
- Tekton Dashboard (tektoncd/dashboard) which provides a simple dashboard (à-la-kube-dashboard) to view pipelines and logs from your browser.
Re-Plumbing Test Infrastructure

We designed Tekton to be intuitive, reliable and easy to operate and maintain. We started using Tekton ourselves as part of our day-to-day job, using it to provide CI/CD services to the community. But we didn’t start from a blank canvas.

Tekton inherited a few features from the Knative ecosystem, including the CI setup. This was great as it allowed the project to relatively easily kick-start its CI using tools shared with several other projects in the Kubernetes ecosystem (including Kubernetes itself)–a combination of Prow, Tide, Boskos, a few tools from k8s.io/test-infra and several bash scripts contribute to tasks such as running E2E integration test.

Deployment of the various CI services is manual and the same is true about their configuration. Releases are crafted mostly manually. Since Tekton is made of several projects, we set up tektoncd/plumbing repository to host the shared CI/CD infrastructure: prow configuration, shared test scripts and tools, test docker images. Mario adventure started!

Mario’s Adventures

We started to work on what we are calling the Dogfooding project, which is basically using Tekton to build Tekton. And like a ball of wool and a cat, the more we thought about it, the more questions we had. How is Tekton going to integrate with the existing prow-based CI infrastructure? How are we going to test changes to the plumbing repo in isolation? How are we going to deploy the instance of Tekton used to run Tekton CI pipelines? How are we going to do all that with no disruption to our CI service?

Testing code continuously is essential infrastructure, even more so for cloud-native software, and re-plumbing all at once could leave us without running water. We decided to proceed with baby steps, reducing the area of impact as much as possible and starting our journey tackling the release automation work first.

Here is an excerpt of the work we did on this journey:
1. Run our releases using Tekton. That was the initial work, using Tekton to at least release itself, even if it gets triggered manually.
2. Integrating Tekton and Prow. Prow is able to create Tekton resources since April of this year (#1 1888 if you are curious), although it took a few iterations to be able to use it on our own CI.
3. Looking at solutions without using Prow at all. This one was (and still is) the trickiest:
- - We needed something that is able to react based on events (from GitHub), this is where the tektoncd/triggers project comes into place.
  - Prow comes with a UI and tooling to capture and store logs. This piece is missing from the Tetkon project as of today. We do have the tektoncd/dashboard project but it’s early days, there is no support for authentication yet.
Mario’s Adventures at KubeCon

Mario’s adventures will continue at KubeCon in San Diego. If you’re attending the conference and you’re curious to know more, come and listen to our talk, or watch it on the CNCF youtube channel after the conference.

This article was co-authored by Andrea Frittoli is a developer advocate at IBM and an open source enthusiast. He is a contributor to the Tekton project and he has contributed to OpenStack for several years. He’s a strong advocate for transparency in open source. Andrea has previously been a speaker at FOSSASIA, FOSS Backstage, OpenStack summits, Open Source Summits, Devoxx Days and various meetups.

— Vincent Demeester
November 8, 2019
The NVMF Boogie: How Kubernetes and NVMF Will Rock
I recently wrote a blog about how Kubernetes creates a global computing grid that delivers unprecedented compute flexibility. Then I wrote a blog about the importance of going fully Kubernetes-native. In today’s blog I would like to drill down into the role that the NVMe over Fabrics (NVMe-oF or just NVMF) standard will play in enabling Kubernetes infrastructure flexibility.

What is NVMe?

NVMe is a protocol that dictates how a CPU, moves memory via the PCI bus to a storage device. NVMe communicates over a set of rings (per CPU) where commands get submitted from any CPU to the underlying NVMe device. The design of NVMe eliminates intermediate layers between the CPU and the storage device. NVMe devices consist of a controller, queues, namespaces, namespace IDs and the actual storage media with some form of an interface.

Storage media can be grouped into sections called namespaces with an ID. In the context of NVMF, namespaces provide a way to enforce access control for the disks/consumers. Namespaces are analogous to an OS partition, except the partitioning is done in hardware by the controller and not the OS (you can still have OS partitions on namespaces). Some NVMe namespaces might be hidden from a user (e.g. for security isolation). A controller connects to a port through queues and a namespace through its namespace ID. A controller is allowed to connect to multiple namespaces and a namespace is allowed to be controlled by multiple controllers (and thus also, multiple ports).

Imagine smearing out this NVMe device across multiple computers and you get to the next important concept, a storage fabric.

What is NVMF?

When you put a network between the PCI bus and the storage device, you use NVMe over Fabric (aka NVMe-oF or simply NVMF). NVMF enables fast access between hosts and storage systems over a network. Compared to iSCSI, NVMF has much lower access latency, in practice adding only a small latency difference between local and remote storage. NVMF delivers a breakthrough in throughput and seek time relative to traditional device attached storage:

Source: Samsung.

This is possible because NVMF uses RDMA to efficiently access the memory of another computer without using the CPU or operating system of either. RDMA is a transport protocol that bypasses the kernel stack, offloading the workload to network hardware. This means the transport layer takes place in the RDMA NIC itself, so data is transferred directly to or from application memory without having to involve the normal OS-level network stack, achieving low latency with low CPU consumption.

Of course, NVMF performance is gated by the networking infrastructure used, whether Ethernet, Infiniband or Fibre Channel. The November 2018 final ratification of the NVMe/TCP transport specification has accelerated adoption of NVMF because it allows storage fabrics to be deployed on standard network hardware.

NVMF can be used with white label storage systems (JBOD), feature-rich block storage systems (SAN), and storage systems built with previous generation (SATA) devices. This means data centers can simultaneously leverage low latency devices while also using more traditional storage subsystems. NVMF also supports using storage class memory as a cache in the data center compute hierarchy.

Why is NVMF Important?

Think of NVMe as a replacement for SCSI, and NVMF as a replacement for iSCSI and iSER. NVMF has several distinguishing technical attributes that add up to huge performance scalability at the storage infrastructure layer:
- Protocol Efficiency: The NVMe-oF protocol is designed specifically for efficient memory transfers at scale. Designed from the ground up for scalability, the NVMe-oF protocol dramatically reduces the internal locking needed to serialize I/O and improves the efficiency of interrupt handling. NVMe-oF also supports MSI-X and interrupt steering, which prevents bottlenecking at the CPU level and enables massive scalability as systems scale. Combined with the large number of cores in modern processors, NVMe-oF delivers massive parallelism to connected devices which increases I/O throughput and lowers I/O latency.
- Command Set Efficiency: NVMe-oF has a streamlined and simple command set that uses less than half the number of CPU instructions to process an I/O request that SAS or SATA does, providing higher IOPS per CPU instruction cycle and lower I/O latency in the host software stack. NVMe-oF also supports enterprise features such as reservations and client features such as power management, extending the improved efficiency beyond just I/O. NVMEOF devices can approach DDR4 performance levels and, with SCM, become an integral component of the computing memory hierarchy.
- Queue Parallelism: With 65,535 parallel queues, each with a queue depth of 65,535 entries, NVMe-oF supports massive parallelism. Every CPU has a private queue to communicate with a data device and is capable of achieving high speeds because there are no locks between queues that live on separate cores. Because each controller has its own set of queues per CPU, I/O throughput increases linearly with the number of available cores.
- Integrated Security: NVMe-oF includes a tunneling protocol that provides security features produced by the Trusted Computing Group (TCG) and other related communities. Features planned for NVMe-oF devices and systems include simple access control, data at rest protection, crypto-erase, purge-level erase and others. Additional information on NVMe-oF security is available in the Trusted Computing Group and NVM Express Joint White Paper: TCG Storage, Opal and NVMe.
NVMe-oF is designed not only to solve the limitations of scale-out storage, but also to lay the foundation for a storage fabric that dynamically shifts as workloads demand.

What Makes NVMF Important to Kubernetes?

Kubernetes applications implicitly need the infrastructure flexibility enabled by NVMF. NVMF and Kubernetes makes distributed computing within sites easier.

The key point of my Kubernetes-native blog was “(Kubernetes-native) applications break apart into their constituent elements and can deploy and scale across the latency pyramid where they fit best. The elements dance across the infrastructure following their constituents in what I imagine as a containerized ‘ballet’ focused on delivering the best possible user experience at the lowest possible latencies.”

However, multi-cloud, multi-site and edge deployments introduce significant additional complexity for organizations operating Kubernetes at (or beyond) enterprise scale. Business needs are pushing IT teams to move workloads between sites, clouds, WAN edge and rapidly be able to clone, burst and then destroy workloads anywhere around the world as needed. Yet Kubernetes deployment across multiple regions, sites, clouds and WAN edge remains problematic.

OpenEBS is the solution for latency sensitive stateful Kubernetes workloads that need universal data agility. A NVMF-based disaggregated storage physical fabric mates perfectly with the disaggregated data agility software infrastructure offered by OpenEBS. NVMF allows data to physically float across a single storage fabric while OpenEBS allows stateful workloads to be loosely coupled across multiple storage infrastructures. And the OpenEBS MayaStor project natively integrates to NVMF to simplify the developer experience.

Together, NVMF and OpenEBS become the dance floor for those Kubernetes-native application components to do their infrastructure-aware (and infrastructure-intelligent) ballet. Given its performance and scalability, however, perhaps this is better called the NVMF Boogie.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

— Charles Piercey
November 7, 2019
DevOps Chats: Continuous Delivery at Airbnb

Spinnaker Summit 2019 Preview: Airbnb is rapidly moving from a monolith Ruby on Rails application to a distributed SOA/Kubernetes architecture in Kubernetes. The new architecture uses self-service codified pipelines and easy webhook integrations, scale adoption and collaboration across the company. Even though continuous integration isn’t new to Airbnb, every team now needs to be able to scale CI across 100’s of containerized services in AWS EC2.

Software Engineer Brian Wolfe co-led the decision to move to Spinnaker and build in more automation. At one year into the project, Airbnb has 40 services in production with many more to follow.

This episode of DevOps Chats features a preview of Brian’s talk, “Scaling a Migration to Continuous Delivery (Airbnb)”. Brian’s talk is on Saturday, November 16 11:00 am, at Spinnaker Summit 2019 in San Diego.

As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

Transcript

Mitch Ashley: Hi everyone. This is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today I’m joined by Brian Wolfe, who is a software engineer at Airbnb. Our topic is a talk that he’s gonna be delivering at Spinnaker Summit 2019 in San Diego. The topic of that talk is scaling of migration to continuous delivery. Continuous delivery is a hot topic right now. That’ll be happening on Saturday, November 16th at 11:00 a.m.

Brian, welcome to DevOps Chat.

Brian Wolfe: Thanks so much, Mitch. It’s really an honor to be talking with you today, so.

Ashley: Well I’m more honored to have you on. I appreciate you taking the time. Tell us a little bit about you. Introduce yourself, tell us what you do and a little bit about–I think we know Airbnb–but tell us about what part in Airbnb that you work in.

Wolfe: Cool. Yeah, so I assume most people who listen to the podcast know what Airbnb is, but we’re a worldwide community of hosts and guests. We bring people together to have really local experiences. I’m gonna skip the rest of that little spiel.

Ashley: [Laughs]

Wolfe: But I’ve been at Airbnb for about three-and-a-half years, and most of that time I’ve been kind of on the operational tooling side, and so a lot of that was observability works, so looking at your metrics and traces and logs, and then performance works, so like figure out how to make Airbnb–monitor Airbnb performance, make it faster and that sort of thing. And so but for the last year I’ve been the tech lead on our continuous delivery team, which is kind of an ambitiously named team. We’re not there yet. But the goal of this team was to really up level how we deliver software at Airbnb.

And so my experience is mostly looking at how we’re making the transition from a monolith which was written in Ruby on Rails to an SOA over the last two years and kind of continuing on into 2020. And part of that is how do we scale how we deliver our software. It used to be a more core team who understood how to scale our monolith. And now every team kind of needs to know how to deliver that software, how to scale it and how to keep it working.

Ashley: Now, is the move to an SOA, and so we mean service-oriented architecture, correct? Is that what you’re referring to?

Wolfe: Yeah.

Ashley: Is the move to that kind of architecture what prompted you to–or order Airbnb to–invest so heavily in this automated continuous delivery? Or was that kind of happening in parallel? Gives a little bit of idea of context of how this kind of came together.

Wolfe: Yeah, so it definitely happened because of the move to SOA. And if you look at what our processes were before, it was very consolidated. Most people were applying the same code base. And so you kind of could have practices that worked for that one code base. But as we moved to SOA we now have hundreds of services that are being deployed, and you need to be able to scale those best practices across all of those services. And so that requires that you add these automation pieces so that humans can make fewer mistakes.

Ashley: That makes a lot of sense. I imagine you have a kind of continuous integration happening at the front end of this, right, leading into your continuous delivery?

Wolfe: Absolutely. So we’ve had continuous integration for a long time. And so all of our unit tests and that sort of thing and all of our builds are happening in continuous integration. That’s been the case for at least three or four years. And we have fairly mature tooling around that. And we actually did a migration to our new platform there fairly recently that containerizes all of our builds and all of our MCI so it kind of runs in a more uniform manner.

Ashley: Mm-hmm. Is that moving to Kubernetes? Is that the change you’re referencing or something else?

Wolfe: So even stuff that’s not running in Kubernetes, so as you can imagine Airbnb is about 10 years old, and so we have a lot of technical history. And so some of that stuff is gonna be running on bare EC2 instances, and some of that is running inside Kubernetes at this point. And so along with the move to SOA we’re also migrating to Kubernetes. And that migration is progressing rapidly. But there’s still gonna be a lot of stuff that has to be built on EC2 and running on EC2. But the builds themselves can happen inside containers.

Ashley: I know we don’t think of Airbnb as something that’s built 10 years ago and evolved over time, right? We think of it as [Crosstalk] – [Laughs].

Wolfe: Yeah, yeah. It has a surprising amount of history for such a ____ company, you know?

Ashley: Well it’s an interesting, you know, it’s an interesting situation that you are involved in, because you’re not talking about a monolith application that was built 30 years ago or 20 years ago. It’s something fairly recent, Ruby on Rails, you know? So it’s still a contemporary programming language, something that everybody is familiar with. But even you have to go through your own kind of architectural and now continuous delivery evolution of that technology.

Wolfe: Absolutely. I think it’s kind of surprising that even at a company that’s ten years old you have what we consider legacy parts of our stack. I think that happens quickly with a field that evolves as rapidly as ours. And if you look at how continuous delivery has evolved in the last ten years, it’s been a pretty big shift.

One of the big things we’ve seen is, you know, we had a really good solution for deployment if you look in 2013-2014 time frame with our system and deploy board. But if you look at it now, it looks a lot like a really good CI system, like a really good continuous integration system. And the parts around actually having an organized pipeline that automates the deployment out, we don’t really have that in house yet. And so that’s really why we’ve been adopting Spinnaker as a solution.

Ashley: Interesting. Well tell us a little bit about were you involved in the choice of bringing in Spinnaker? Was that happening while or before you joined Airbnb?

Wolfe: Mm-hmm. So the decision for adopting Spinnaker happened last year. And so it was a discussion between myself, my manager and then Jing Jing and Jens, who are on the team. And it was largely about do we want to keep investing in deploy board, which was our existing solution. It was deeply integrated with all of Airbnb stack. Had a lot of stuff in it for how we deploy MonoRail, which is our big Ruby on Rails application.

Now, do we want to build automation into that, or do we want to bring in something new? And so we did a big research project to look at what are all the available options out there, and what would those give us, and how could we make those work at Airbnb. And at the end of that we decided that we should not invest more in our internal solution and should instead bring in Spinnaker and customize that to kind of encode Airbnb opinions into that via extensions and then use that as our platform for deployment moving forward.

So we’re still early days in that. We’re about one year in. We have about 40 services onboarded right now. These will be all production services, some of the really critical services out of Airbnb. But considering we have thousands of services at Airbnb, there’s some work to do to kind of make it the standard across.

Ashley: Mm-hmm. Excellent. Well interesting. Tell us some more about I know in the description of your talk, you talk about that you’re gonna discuss how self-service codified pipelines and easy web hook integrations help you scale adoption and collaboration across the company. Tell us some more about what that is.

Wolfe: Yeah, so one of the really harder parts about working at a company that’s growing as fast as Airbnb is its really hard to match the needs of everyone. And so you want to make everything self-service if you can so that, you know, I’m creating a standard pipeline that you know I deploy one thing, I do some AB comparison, I roll it out to my next environment, I do another maybe canary analysis rollout, and so on.

So I can come up with standard components that people can use. But then you have stuff that people have built over time. And so our research team has regression detection that is really specific to them. And they think it provides really high signal if some functionality regression has occurred because you have things like search ranking and the actual layout of the results in the response package, and you need to make sure those things don’t change if you don’t intend them to change.

And so they built a service to actually do that regression detection, and they want to integrate that with their deployed pipeline. And they want that to just be something that automatically happens every time that they run a deploy. You deploy to staging, you call this service with some arguments, you wait for that regression test to pass, and then if it fails you present some custom UI that says, “Okay, this is what failed. This is why. And this is how you can go learn about it some more.” And you move onto the next stage or you fail the pipeline and figure out what’s going on.

What we wanted to do is make it really easy for teams to plug in services like this, ‘cause they do provide the most value. The way we’re doing that is by integrating, actually, with our interface definition language that we use internally. And so we actually extend Spinnaker to just provide a little web hook stage that people can reuse and call out to their services. But then they get a nice user interface on top of that. And so they can kind of provide just a base level of functionality within Spinnaker. And it’s kind of a minimal amount of work for us to onboard a new type of stage. Does that kind of make sense?

Ashley: Yeah. I was just gonna ask you about that. Sometimes it can be a real challenge to introduce such a fundamental tool, like this is fundamental to your software development and delivery process and what kind of things that you can do to minimize the disruption, lower the bar in the area of entry for the teams. And a lot of it depends, too, on how much your organization does software process similarly, or is it very decentralized and everybody kind of does their own thing. It’s such an application if it’s been in a monolith kind of state. I imagine there’s a lot more similarities in how people work at Airbnb. But you’ve gotta still figure out how do you make this easy, right?

Wolfe: Yeah. So I think we’re lucky in the sense that right now there’s really one deployment tool that people use, and that’s deploy board. And now we’ve introduced a second deployment tool. And our strategy here is actually we make the experience when you’re first onboarding actually look a lot like deploy board.

And so Spinnaker lets you customize the user interface. And so we actually have a panel within Spinnaker that shows a view that actually looks really similar to the view that you get in our old tool. But then it provides all this power under the hood when you actually start your deploy. And so that’s been pretty valuable for us for onboarding because it doesn’t look that dissimilar, I would say a little rough around the edges, to be honest still.

It’s, you know, we spent a lot of time optimizing that flow for deploy board. People get a little bit confused sometimes. But just having that initial experience look similar has made onboarding a lot easier. When we first proposed using Spinnaker, people were like, “I can’t figure out what’s going on in the UI. I don’t know what I’m doing here. What are you guys thinking?” But just by providing that similar viewpoint people are like, “Oh, I know what to do. I’ll just click on this button, which looks the same as we saw before, and that should start my deploy doing the right thing.”

Ashley: You know, never underestimate the value of an easy-to-use user interface or process or whatever.

Wolfe: Especially just having that familiarity, I think, it’s–Jens on my team has been a continual advocate for really pushing on, like, make this easy to transition from our old tool to our new tool. And I would say that there are some usability gaps that we’ve had to come across. But they’re getting better, I would say.

Ashley: How far along in the rollout process of this are you? Are you in the still kind of beginning third of it, working with some of the early teams that are adopting? Are you now moving into more broader adoption of it? Where are you kind of in that process, that maturation process?

Wolfe: Yeah, so we’re kind of in the–we’re calling it a beta stage. So in alpha stage we were really hitting some early adopters, people who were willing to take some risks. Now we’re onboarding kind of the big customers, the ones who are pretty demanding and making sure that this actually works for them. And so what we’re doing is we’re measuring how much do we prevent, and so how many rollbacks are we preventing, how many incidents are we preventing, and making sure that Spinnaker, as the platform we’re currently providing, is actually delivering business value.

We’re limiting our onboarding throughout 2019 to about 50 teams. And at the end of 2019 we’ll have a really good idea of what works and what doesn’t. And in 2020 we’re going to open up the adoption to the full company. Kind of at the same time we’re going through a lot of scaling exercises to make sure that we will actually be able to scale to the full company.

Ashley: And give us an idea of the 50 teams, what are usually the size of those teams? What do they range?

Wolfe: Yeah, so the smallest teams will be probably about six or seven developers deploying a service. The largest team that we have onboarded has about 45 developers. And then we’re aiming for one of our front-end services, which is more monolithic, and so that’ll have several hundred developers working on it. But then there’s kind of an operational expert team that manages what that deployment looks like.

Ashley: Mm-hmm. Well that’s a pretty good range in different sizes of teams. So I’m sure–and parts of the application–so I assume they bring different challenges with them. How do you measure success from your decision of both going down the path of implementing continuous delivery and Spinnaker? You talked about business value. You also talked about preventing how many rollbacks and incidents that also happened. But how do you demonstrate to the people that said we’re gonna spend this money having you go implement this, and tell them that it was worth it?

Wolfe: Totally. That is a great question. And so one of the big motivators for us was actually for regression prevention. There are kind of two aspects for CD. One of them is increased productivity for developers. And then another one is having more vigorous processes for rollout. And we were definitely focused on the vigorous processes for rollout and having automated canary analysis by default for every service. And so that’s kind of the route that we’re pursuing, really, in 2019 in terms of proving the business value.

Ashley: Mm-hmm.

Wolfe: In 2020 we’re gonna have more metrics and more ability to say something around how productive we have made our engineers, or at least that we haven’t made the story harder. But right now it’s proven a little bit harder to quantify that. So we get people saying, “Yeah, we love it. We love that we can kind of ignore the deploy process now.” But aside from having a CSAT style score where you’re just saying, you know, rate how productive you are on a scale of one to ten, it’s been pretty hard to quantify the productivity gains we’ve seen with Spinnaker.

Ashley: Well, that is often a challenge, right, for us in the software world in implementing technology, and as you’ve talked about some of the regression testing issues, maybe even incidents in the old process compared to the new, hopefully there are some metrics you can also use from the old methods to the new and to give you some places to start, at least, to benchmark those things.

Wolfe: Certainly for regression detection we have. We have strong evidence that automating this pipeline–and so what I really think of Spinnaker as is a way to automate your run book. Just the fact that it is automated now has reduced the number of regressions that have gotten out to production. We have very strong evidence of that at this point.

Ashley: Well and hopefully you can sing some of the praises, the good things of what the teams have learned so far with folks that are adopting it as you take on some of the bigger challenges like the larger teams do.

Well now, I appreciate you coming on the podcast. It’s been fascinating learning about what you’re gonna talk about. And I wish you the best in your presentation at Spinnaker Summit.

Wolfe: Thanks so much, Mitch. This has been a pleasure. And I hope to meet you at some point soon.

Ashley: That would be fantastic. I would enjoy that very much. I’d like to thank my guest today, Brian Wolfe, software engineer at Airbnb for sharing with us his experience with Spinnaker and how they are migrating to a continuous delivery model and scaling that up in the organization. Brian’s gonna be talking at the Spinnaker Summit 2019, which is November 15th through the 19th in San Diego. His talk is on Saturday the 16th at 11:00 a.m. And I believe you’re gonna be doing that with Jens Vanderhaeghe? Is that correct?

Wolfe: Yes, that is correct.

Ashley: Awesome. Good to have you both up there. Well thank you everyone for joining us today. I’d like to thank our listeners for taking the time to check out the podcast and hear about this talk. This is Mitch Ashley with devopsy.kinsta.cloud. And you have listened to another DevOps Chat with my guest. Be careful out there.

— Mitchell Ashley

November 6, 2019
Microsoft Extends Azure Reach to On-Premises IT Environs

Microsoft today at its Ignite 2019 conference previewed an Azure Arc management service that extends the reach of the Microsoft cloud to on-premises IT deployments.

Azure Arc will be made available on both local Windows and Linux servers as well as Kubernetes clusters or an instance of Kubernetes that Microsoft is making available on Azure Stack for on-premises IT environments. With these announcements, Microsoft is, for the first time, extending its hybrid cloud computing strategy beyond systems running a version of the Azure operating environment on an on-premises IT environment.

Jason Zander, executive vice president for Microsoft Azure, told conference attendees Azure Arc creates an opportunity for IT organizations to extend the best DevOps and cloud management practices employed on Azure to on-premises IT environments. Rival cloud service providers, he noted, are only paying lip service to hybrid cloud computing.

In addition to extending the Azure management plane to on-premises IT environments, Microsoft today extended its portfolio of edge computing platforms capable of running instances of Azure Stack to include previews of commercial series and rugged series for Azure Stack Edge. Configurable with graphical processor units and field-programmable gate arrays (FPGAs) capable of running machine learning algorithms in harsh environments, the ruggedized instance of Azure Stack Edge even includes a portable server designed to be carried in a backpack.

To complement those systems, Microsoft is also previewing a range of additional services for Azure Stack, including a fully managed, real-time data ingestion service designed to stream millions of events per second from any source to build data pipelines the same way the company does on the Azure cloud; a cloud database management service for Microsoft SQL Server and Postgres; and an Arc-enabled application programming interface (API) management platform.

While Microsoft generally would prefer organizations to adopt Azure Stack as an on-premises extension to Azure cloud, not many organizations have the robust modern server infrastructure in place required to run Azure Stack. Azure Arc is an effort to extend the reach of Azure to multiple types of edge computing platforms without requiring organizations to deploy yet another operating system. Alternatively, organizations can employ a diverse range of Kubernetes distributions, of which there are now more than 70 that have been certified by the Cloud Native Computing Foundation (CNCF). Most organizations at this stage have yet to standardize on a single distribution of Kubernetes.

It’s unclear what the appetite is for extending the management plane Microsoft created for its Azure cloud to on-premises environments. However, as the scope of the hybrid cloud management challenge facing organizations grows as various new forms of edge computing manifest, Microsoft clearly sees an opportunity to increase the scope and reach of its cloud platform strategy. Each edge computing instance is essentially a smaller instance of a cloud platform for which most IT organizations will not have the tools required to manage at scale. In fact, many IT organizations might not even attempt to deploy workloads at all on those platforms without something equivalent to Azure Arc to manage and secure those environment first being in place.

— Mike Vizard

November 4, 2019
Breaking Up and Making Up: Composable Persistent Storage for Kubernetes

Kubernetes is rapidly emerging as the standard for application management in the data center. Kubernetes defines applications in a container model for deployment and delivers most of the software-as-a-service features of virtualized infrastructure at a lower cost and with less performance degradation. In this article we’ll cover the intersection of Kubernetes and scale-out data-intensive applications, what problems Kubernetes solves for those applications and why composable infrastructure provides the remaining piece for a total solution.

For heavy-duty data-intensive applications, such as Spark, Kafka, Aerospike, Vertica, Cassandra and Elasticsearch, ML-model training/service with TensorFlow, Kubernetes promises a common application orchestration solution. These applications are termed scale-out because they allow additional performance and capacity to meet any use case simply by adding more nodes. The challenge introduced by these applications is managing large production deployments of each application, compounded by managing multiple applications each with their own configuration requirements. Kubernetes provides an answer here for managing these disparate data-intensive, scale-out applications with a common container orchestration platform.

Scale-out applications are distinguished from typical enterprise applications in three ways. First, scale-out applications take a divide-and-conquer approach to problem solving: if you have a bigger problem simply deploy additional capacity and performance by adding nodes. Second, since performance and data capacity are simply increased by adding building blocks, you can exploit cheap commodity-off-the-shelf (COTS) nodes, avoiding the money pit that is the case in big iron hardware purchases required by traditional scale-up applications. Third, the data-intensive applications typically integrate an optimized storage stack within the application itself that effectively uses the cheap local storage available within the COTS nodes. The last point has become a problematic area for scale-out application deployment as overprovisioning is now common to overcome the difficulty in predicting workload behavior at initial deployment time.

Composable infrastructure tackles the last problem presented by embedded local storage for data intensive applications. Looking back over various transformations in the data center, one thing the legacy enterprise storage vendors got right was separating compute and storage. Both during acquisition and throughout the application lifecycle, separating persistent storage components from compute nodes running the actual application yields opportunities for cost efficiencies and flexibility.

Composable infrastructure brings the capability of separating compute from storage for these scale-out applications, which sport their own embedded optimized storage software stack because it does not dictate a new storage architecture, it simply says “take your existing servers with embedded local storage and disaggregate them”–placing the storage on the ubiquitous high-speed low-latency fabric you’ve already deployed for your applications: Ethernet. From the application perspective in a composable infrastructure, storage still appears local and the application runs as it did before.

In this way, applications run unchanged and the infrastructure they run on can be expanded, reduced or replaced in real-time. This is true software-defined infrastructure where infrastructure is treated as code. Being able to modify the node configuration ratio of compute and storage dynamically avoids the issues of purchase time lock-in of possibly under-provisioned nodes (which has simply led to overprovisioning). Composable infrastructure eliminates the downside of purchasing static resources that define the resources available before the workload is run.

Kubernetes launched not long ago in 2015. Many of the early adopters of Kubernetes focused on stateless applications that did not depend on persistent storage to deliver useful services. It is only recently that support for persistent storage emerged (an obvious requirement for data-intensive applications). After a couple of approaches to persistent storage integration, the key players in Kubernetes (and other container orchestration platforms) settled on an orchestration-independent persistent storage API, the Container Storage Interface (CSI). The CSI provides a standard method for storage providers–and composable infrastructure providers–to define and attach persistent storage to containers.

Why should you be interested in composable infrastructure when considering how to deploy your data-intensive applications on Kubernetes? Because Kubernetes only provides application and container orchestration–it does not fundamentally change the way the application works. These data-intensive applications were defined to exploit, and are most efficiently deployed, over COTS node and commodity storage. Each application still has its own storage stack, each carefully optimized for their particular workload. It is likely that deploying storage appliances, or software-defined storage solutions that aggregate pools of HDDs or SSDs, under data-intensive applications will only result in degraded performance and unnecessary costs.

A composable infrastructure solution is only useful as an underlayment for a Kubernetes deployment if it meets several requirements.

First and foremost, the composable infrastructure must scale to 100,000s of devices in support of the applications being deployed on top of the platform.

Second, the composable infrastructure platform must not destabilize or diminish Kubernetes capabilities and scale accordingly with commodity networking in mind. The composable infrastructure solution must share a similar design philosophy with Kubernetes, allowing it to scale. Kubernetes implements a distributed desired state model where work is distributed amongst the mostly autonomous nodes to reach a desired deployment configuration. Such an approach provides scaling on the order required by hyperscalers. You must ensure the composable infrastructure solution you choose provides a similar scalable approach.

Third, Kubernetes emphasizes a configuration state that is durable in the face of any individual node failure or network partitions. Durable configuration state in the face of large, highly interconnected commodity component deployment is critical for correct automatic operation at scale of business-critical applications, and your composable infrastructure solution must similarly support this.

Fourth, Kubernetes defines a model for application deployment in terms of groups of containers called Pods. For a manageable persistent storage solution, the support for CSI and Kubernetes must provide a native Pod view of the storage bindings since the Pod (the managed set of containers) is the proxy for an application node instance and the unit of expansion and migration.

Kubernetes has ushered in an exciting new chapter in data center architecture by providing a lightweight, richly featured application software orchestration platform. Think of Kubernetes as the ideal software application orchestration platform. What is missing is the scalable hardware orchestration solution that supports the dynamic model for application orchestration that Kubernetes defines. A feature rich, well-integrated composable infrastructure platform provides the needed hardware orchestration to enable the application migration and elasticity promised by the Kubernetes solution.

— Brian Pawlowski

November 4, 2019