devopsy.kinsta.cloud

Tag: Spinnaker Summit 2019

  • DevOps Chats: Demystifying Spinnaker VM Baking, with Salesforce

    DevOps Chats: Demystifying Spinnaker VM Baking, with Salesforce

    Spinnaker Summit 2019 Preview: As you use an open source tool like Spinnaker, the more you develop best practices and learnings. These are useful to share with others internal to the enterprise and other open source users.

    Jing Vergara, principal software engineer at Salesforce, joins DevOps Chats to share a preview of her upcoming talk “Demystifying Spinnaker VM Image Baking and Deployment.” Jing shares with us what to include, and not to include, when baking a VM image, how to avoid configuration drift and how to deploy to multiple cloud providers using Cloud-init files. Jing also shares how to use open source Packer templates with Ansible, Chef and Docker, and how to build your own Packer templates.

    Jing Vergara’s talk is on Sunday, Nov. 17th at 10:45 AM PT. Spinnaker Summit 2019 is San Diego on Nov. 15-17.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Jing Vergara, who is principal software engineer at Salesforce. Now, Jing is going to be talking at the Spinnaker Summit in San Diego in November, and she’s gonna be talking about a really interesting topic, “Demystifying Spinnaker VM Image Baking and Deployment.” Now, don’t think we’ve changed into a baking show. We’re baking VM images, and that’s what Jing’s gonna tell us a little bit about, that whole process—pipelining and creating VMs and baking them. [Laughter] So, her talk is on Sunday, November 17th, at 10:45 a.m.

    Jing, welcome to DevOps Chat.

    Jing Vergara: Thanks, Mitch.

    Ashley: Great to have you.

    Vergara: Thank you for the opportunity to have me talk here.

    Ashley: Well, I’m honored you’re on. Thank you for being here. Tell us about yourself. Tell us a little bit about what you do and maybe what parts of Salesforce product or code that you work on.

    Vergara: Sure. So, I’m part of the continuous deployment team at the company. We offer Spinnaker as a Service for all of the organizations within the company. And I help teams onboard their delivery pipelines into Spinnaker.

    Ashley: Oh, interesting.

    Vergara: Yep, and as part of my job, I also build custom extensions to Spinnaker to support their internal business needs.

    Ashley: So, you’re kind of the Spinnaker service center expert person or group that everybody goes to for help implementing Spinnaker. Is that close?

    Vergara: Yeah—very, very close, yes. So, my role within the team has been changing, but yeah, it has transformed into that one. Another thing that I do for Salesforce is that I represent Salesforce at a couple of the Spinnaker special interest groups for stakes insured so that I could learn more about upcoming features of Spinnaker, so we can leverage those in our roadmap as well so that we know we can offer it to the business.

    Ashley: Mm-hmm.

    Vergara: And then we also contribute back to the ________ and let them know what would and would not work for us as users of Spinnaker.

    Ashley: Well, and I noticed in your bio on the Spinnaker site, you have a Master’s in Computer Science from the University of Illinois at Urbana-Champaign. I used to live not too far, from there, though I didn’t go to that school. [Laughter]

    Vergara: [Laughter]

    Ashley: That’s a very strong C.S. degree, and it sounds like you’re putting it to good use with Spinnaker and DevOps style deployment with these kinda pipeline creation.

    Vergara: Yes, right.

    Ashley: Excellent. Well, tell us a little bit about what—so, we’re talking about your talk. So, describe for us a little bit about what are some of the things that are maybe a little mystical that need demystifying about Spinnaker VM image baking and deployment. What are some of the things people don’t know that you usually have to help them understand with, “Here’s how it works and here’s how we do it”?

    Vergara: Sure. So, for companies that are traditionally in a bare metal world where you don’t use VM images, it’s kind of a definite shift in the mindset, right?

    Ashley: Mm-hmm.

    Vergara: So, the way I usually explain it to them is using the baking metaphor, right? So, when you talk about baking, you think of putting the ingredients together like flour, salt, what have you, right? You form a dough, shape the dough, then you cook it. You start with a base ingredient—like, again, the flour—and you mix it up.

    So, in the VM world, when you bake an image, you start with that base OS image. You add your other ingredients, which means installing your applications or your agents to run on that VM. And then you have your standard or common configurations and possibly some data, right?

    Ashley: Mm-hmm.

    Vergara: You add them all up in the VM, you create a snapshot. So, that snapshot, you can now call it the VM image. So you can save this image into a store, and once you have that image, you can use it for deployment. So, deployment, it’s basically building or provisioning one or multiple VMs out of that same image.

    Ashley: Okay. Now, leading into this, the thing that kinda kicks it off is that, I’m not sure how it works at Salesforce, but is it something like a Jenkins process that then kicks off this creating of the image? How does that work?

    Vergara: Right. So, we have our continuous integration system. So, we have a few and we’re standardizing that at Salesforce as well, right? So, we have those artifacts built by those CI systems. We have our Docker images so they get stored, we put them into the cloud, and then we have triggers. So, we listen for those changes. So, Spinnaker supports a Pub/Sub subscription that triggers the pipeline. So, then, once we have those architects, we can start the baking process.

    Ashley: Great. Excellent. Well, that makes a lot of sense. Think of it as all of the pieces of software, OS on up, that are ingredients, right? Just elements of what’s gonna go into that image.

    Now, I imagine this is something that’s happening frequently if you’re doing a lot of deploys, right? Is this happening every time that you do a deploy or just as part of the integration process, and then it may go through a deployment process, too, or how frequently would you see this running?

    Vergara: Yeah, so it’s for every code change that gets checked in, right? Ideally, you would want that change to go all the way, so for continuous—again, for continuous delivery and deployment, that’s the ideal process. Check in, generate the artifacts, and deploy that artifact, create a baked image out of it, and then go and deploy it.

    Ashley: Okay, great. So, this is—

    Vergara: Right, so, but—yeah, yeah, right. But like, when we say delivery and deployment, of course, you know, there’s release management and change management policies at every company that says, “Okay, this is actually good for production deployment,” right? So, some of them are continuous deployment up to development or test environments, but for production, we still want to have that gate system in there.

    Ashley: Mm-hmm, yeah.

    Vergara: And Spinnaker actually supports it really well, you know? It’s got those manual judgment stages that you can add into the pipelines to get those kinds of approvals.

    Ashley: Yeah. Well, we’ve had some other folks talking about some of those stages like Run Job and Webhook and some of the capabilities that are built in.

    Vergara: Yes.

    Ashley: Easy to extend as well in Spinnaker. So, I think one of the things that you’re planning on talking about in your talk at Spinnaker Summit is maybe some lessons learned or best practices that you’ve figured out from doing this a lot at Salesforce. Any thoughts on that you want to share with us about the kind of best practices you’re gonna get into in your talk?

    Vergara: Sure, yeah. So, some of the best practices that I’d like to share with you is more around the baking and deployment. So, for example, when you do baking and never, ever bake secrets during bake time. [Laughter]

    Ashley: Yeah. No tokens, no keys, no certificates, no—

    Vergara: Exactly. Yes, yes. So, that’s very, very important. So, those kinds of sensitive data, you can actually perform that during the deployment stage. There is a trap stage there during deployment where you can go and, you know, connect to a secret store, like, Vault, HashiCorp’s Vault, right? So, you can pull down those secrets and install it on the resources or the VMs, right? So, that’s the safer way to go about it rather than baking it during bake time.

    And then another thing as well, you know, as much as possible to avoid configuration drift, right? So, if you have any common or standard configuration, be it your application configuration or your OS configuration, right, do it during the baking step, right? So, as much as possible, limit those VM configurations towards them. Again, you can do it during deployment.

    Ashley: Mm-hmm.

    Vergara: And then, for companies that support multiple cloud providers—so, for example, you need to go and deploy to, say, Amazon Cloud or Google Cloud or other clouds that are available and supported by Spinnaker, you can use Cloud-init files so that it’s not tied to one particular provider and you don’t have to repeat your bootstrap steps during that same time. So, you write your script once, your Cloud-init file, and then you can use it in those different cloud providers, so that’s, I think, that’s very, very helpful.

    Ashley: Now, is that capability something that’s built into Spinnaker if you write your script corrects, and I think you said init file for each—

    Vergara: Cloud-init.

    Ashley:—each cloud, Cloud-init.

    Vergara: Yes.

    Ashley: Is that something that, it’s a feature if you know about, you can use it up front and design that into how you’re writing your scripts?

    Vergara: Yeah, so, there’s—there are fields within the Spinnaker deploy stage where you can put that in. So, in, I think in GCEVMs, you can use the user data field in the deploy stage, and then in AWS, there’s the metadata field.

    So, the field kind of differs by provider, but it’s the same concept, right?

    Ashley: Mm-hmm.

    Vergara: So, you put it in there, you plug it in. And then, during the bootstrap process of those VMs, I think the OS has the support Cloud-init, though, so when they choose an OS, make sure it’s got the Cloud-init, and then Cloud-init will pick up those files and then launch the steps or commands from what you have in those Cloud-init scripts.

    Ashley: Mm-hmm, great. What are some of the things that people that are maybe new to Spinnaker or new to creating VM images and doing baking, what are some of the typical struggles someone might have that would be helpful to come to your talk for?

    Vergara: Right, so I experienced the same thing when I started out with Spinnaker. I’m like, “Oh, this is all new to me! How do I go about this?” [Laughter] So, one of the things that I’m gonna share in the Summit is, like, “Hey, you know what? Spinnaker actually uses Packer, it’s a HashiCorp tool that actually performs the baking behind the scenes.

    Now, Spinnaker offers a set of Packer templates, right? So, those are the very basic ones. And it actually varies per provider. So, you can only use one at a time. So, if you’re building for, say, GCP, then use the GCP Packer template. And if you’re using AWS, you have to use a different AWS Packer template.

    Ashley: Mm-hmm.

    Vergara: So, and then, if you have a more advanced use case, you can actually customize and build your own Packer template, so that’s one of the powerful teachers of Spinnaker as well, right? So, if you end up—like, you know, so for in-house, you use Chef or Ansible or Puppet. There are Packer templates that can use those kinds of provisioners, but you have to go and deploy those into Spinnaker.

    So, at the talk, I’m gonna share how you can deploy those custom Packer templates. So, there are a couple of ways on how to do it. So, one is the simpler way, but it’s kind of limited. Those Packer templates are stored like Kubernetes secrets in Spinnaker, so it’s got a limitation in size. So, if you have complex scripts or Packer templates, then you have to deploy it as a sidecar container.

    Again, I’m gonna share those tips at the Summit and give more details around it.

    Ashley: Mm-hmm. Yeah, folks may be familiar with Packer, too, it’s an open source project in and of itself for creating images, machine images.

    Vergara: Yes.

    Ashley: Great. Are there some other struggles that you find folks typically have, or may have when they’re new to this?

    Vergara: What else? So, Spinnaker by itself mostly focuses on releasing software changes, but it has a very flexible pipeline management system. Like, at Salesforce, we actually use it for building resources such as load balancers, DNS records, topics, queues. You can actually create custom job stages to perform those.

    Ashley: Mm-hmm.

    Vergara: So, we found it really awesome, because we can rebuild a full environment carrying down all of those existing resources, running the same set of pipelines that we can deploy together with those infrastructure resource setup. So, we can build it up right away, right, and then it also answers our scaling requirements. Because, you know, large scale enterprises such as Salesforce, you know, we have to build all of these resources and deployments. We’re talking about thousands of servers across different regions and geographical locations.

    Ashley: Mm-hmm. You know, sometimes tools, open source, or even commercial tools that are highly flexible and very powerful, sometimes with that comes complexity, so it helps to have some guidance, some experience mentoring from someone like yourself that’s gone through that process and can help you kinda get through it and understand how to use it, how to get things done, right?

    Vergara: Right. So, Spinnaker has a Slack channel. So, you know, when people like me or others who are new to Spinnaker have any questions, we go and ask in that channel. There’s a few channels there that are specific to certain features. So, for example, if you’re struggling with templatizing pipelines, then you go into the Pipelines as Code channel, right? Or if you have questions about how to use Kubernetes in Spinnaker, then you use the Kubernetes Slack channel.

    That was very, very helpful for us, especially in the beginning when we were trying to figure out how things are working.

    Ashley: Mm-hmm, I’ll bet, I’ll bet. How long have you been using Spinnaker, then, at Salesforce?

    Vergara: We started our journey last year.

    Ashley: Okay. [Cross talk]

    Vergara: Late last year, so almost one year now. [Laughter]

    Ashley: Wow, okay.

    Vergara: Yeah.

    Ashley: Wow. I bet a lot has happened in that one year. I know one of the things that you’re planning on also talking about is some contribution ideas for those that might want to help make Spinnaker better. And those are things that you—ideas you had or lessons you’ve learned implementing Spinnaker there, I assume?

    Vergara: Right. So, some of the contribution ideas that we have is with regards to a new effort called Kill. I’m not sure if you have interviewed the speakers.

    Ashley: I haven’t yet, but I may. [Laughter]

    Vergara: Okay. Yeah, so I’ll let them—you know, I won’t steal their thunder.

    Ashley: Don’t steal their thunder—okay, great. [Laughter]

    Vergara: [Laughter] Right, so, it’s one of those, they’re focused more around the managed delivery, so that’s when. Another contribution idea is centered more around each of the providers. So, again, if you’re building for one provider, it’s a different, totally different stage. So, for example, like GCEVM, right? So, they have their own deploy stage for that. And then AWS, they would have their own deploy stage for that. They’re kind of not, like, at par right now. Like, you know, they focus on certain things.

    I think another contribution there is trying to add more options or features into that, so it can fully support that provider. And so, for example, for GCE, I think one of the features lacking there, if I remember correctly—or they might have fixed it. But it was, say, so if you have a persistent disk and you want this particular persistent disk, then it’s not supported. It always creates a new one. So, it doesn’t give you an option to actually reuse an existing one that you’ve built earlier. So, things like those. I think that would be important.

    Same thing with AWS, right? So, there’s some features for EC2 VMs during deployment that could be added in there as well.

    Ashley: Great. I’m wondering, too, thinking about your talking that’s coming up, who are the kinds of folks that would be, get the most out of it, coming to your—obviously, people that wanna perform this function, but is that usually, is it developers, is it DevOps engineers, is it CI/CD automation specialists? Who are folks that would get the most out of your talk?

    Vergara: It would be around the Spinnaker administrators within their companies. So, if they want to learn how to configure Spinnaker to enable features for baking and deployment, then they would be the ones who would learn the most out of it. But I would say, like, all the DevOps engineers who want to figure out how they actually use Spinnaker for baking and deployment as well. Like, how do they set up their pipelines?

    I’m gonna give a demo on how it actually works from start to finish so that they can have a visual. Sometimes, you need that visual to say, “Oh, yeah! Oh, that’s how it works!” Right?

    Ashley: [Laughter] Mm-hmm. Well, that guidance is very helpful. And I can see, too, what you’re saying about the folks that are obviously administering Spinnaker in that deploy process are ideal to attend, and of course, other DevOps engineers automating and wanna know how this part of it works certainly would be good to go as well.

    Well, I wish you the best of luck. I know you’ll do real well. You obviously know, have learned a lot and know a lot from your experience at Salesforce. So, I wish you the best.

    Vergara: Well, thank you. Thank you, Mitch, again, for the opportunity for having me talk here.

    Ashley: Well, absolutely. Thank you for being on. I wanna thank my guest today, Jing Vergara, who’s princ—I can’t say it [Laughter]—principal software engineer at Salesforce. There, I got it out. Now, Spinnaker Summit 2019 is November 15th through the 19th in San Diego, and Jing’s talk is on Sunday, November 17th, at 10:45. It’s titled, “Demystifying Spinnaker VM Imaging Baking and Deployment.”

    So, thank you for joining us today for this DevOps Chat podcast. This is Mitch Ashley with devopsy.kinsta.cloud. Have a great day, and be careful out there.

    — Mitchell Ashley

  • DevOps Chats: Canary Deploys Using Istio, with Autodesk

    DevOps Chats: Canary Deploys Using Istio, with Autodesk

    Spinnaker Summit 2019 Preview: Canary deploys give us a window into how new code deploys perform in production on a limited basis. The technology holds great promise in helping us learn the positive or negative effects of deploys without putting the more extensive set of microservices, application functions and beyond at risk.

    How do you route traffic? How do you apply a virtual service manifest? Is there a quick way to perform a smoke test to find server errors? When comparing metrics from new code to old, how do you know which results are good or bad? There is a lot to know and learn about how to deploy and use Canaries.

    Our DevOps Chat guest Omar Al-Hayderi, engineering manager and principal engineer at Autodesk, is giving a talk on “Canary Deploys with Istio: Lessons Learned” at the Spinnaker Summit 2019. His talk is on Sunday, Nov. 17, at 1:30 PM PT.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Omar Al-Hayderi. He is engineering manager, formerly principal engineer at Autodesk. And Omar is gonna be speaking at the Spinnaker Summit 2019 in San Diego. His talk is on Canary deploys with Istio. It’s happening on Sunday, the 17th of November, 1:30 p.m.

    Omar, welcome to DevOps Chat podcast.

    Omar Al-Hayderi: Thanks for having me, Mitch.

    Ashley: Awesome to have you here. I’m excited to hear about your talk. But first, tell us a little bit about yourself, introduce yourself, and what you do at Autodesk. I think we know about Autodesk, maybe what part of Autodesk that you work in.

    Al-Hayderi: Right, yeah. So, I’ve actually gone through a lot of change recently. I was working at a startup called PlanGrid, and we were a small construction software company four years ago, grown a lot since then. I started off at that company as sort of a backend generalist. You know, there was like 25 engineers, really, at the company, and have grown a lot since then, since the acquisition. We got purchased at about 400 employees.

    In that time, I’ve sort of evolved naturally into more of a DevOps infrastructury engineer role, and—yeah, it was kinda the best four years of my life. Really learned a lot there. Post-acquisition, I moved into a principal engineer role on the infrastructure team. We then went through sort of a re-org and I became the engineering manager of the back end platform team. And now, that team, we sort of specialize in building a lot of tooling around our sort of DevOps and infrastructure cloud.

    Ashley: Mm-hmm, great. Well, great experience to bring with you to Autodesk. It sounds like you’re working on some fantastic things. How did you come to decide that you’re gonna talk about Canary using Istio at the Spinnaker Summit? Was that something you chose to kinda bring in and start up at Autodesk, you joined a team already using it? What’s kinda the background on how you kinda came to this place to wanna talk about it?

    Al-Hayderi: Yeah. So, I was sort of, you know, dipping my toes into Spinnaker at the early stages of it, and I was really fascinated by the tool. I loved using all sorts of features and playing around with it. And at PlanGrid, actually, before we were even acquired, we did yearly hack weeks, and we got to sort of move off of our regularly scheduled program and work on whatever we felt like, just had fun.

    And one of the things that recently came out for Spinnaker was this idea of Canary deployments. So, I experimented with it, never got it off the ground, fast forward one year later to the next hack week, Kayenta, the Spinnaker Canary component, was a lot more built out and user friendly. So, I managed to get it off the ground then.

    We sort of have a monolith API at PlanGrid. And so, a huge problem we had was, we would do weekly releases, and after the release, stuff would break and people would get paged and we’d have to roll back, so.

    Ashley: [Laughter] Vicious cycle.

    Al-Hayderi: Common story I’m sure everyone’s familiar with.

    Ashley: Mm-hmm.

    Al-Hayderi: So, I really wanted to find a way to solve that, and Canary just seemed like a great way to do that. So, after the hack week, I sort of kept working on it full time, and I started doing, you know, presentations to the rest of the company. Because Canary is sort of this new idea that a lot of people aren’t familiar with. I started, you know, being active in the Spinnaker Slack channels, and it actually turns out that an ex-PlanGrid employee moved to Netflix to be a lead on the Spinnaker open source software from there.

    Ashley: Ah, interesting.

    Al-Hayderi: Yeah.

    Ashley: Small world.

    Al-Hayderi: Yeah. [Laughter] And he noticed I was talking a lot about it and he brought up the point that they’re running the Spinnaker Summit in November and he asked if I wanted to talk. And I’ve actually never given a talk at a conference.

    Ashley: Oh, excellent.

    Al-Hayderi: Yeah, and I do a lot of talks internally. So, this is something I was super excited about and, yeah, I jumped on it.

    Ashley: Great thing to do for your career, too, and it sounds like those internal talks have set you up well to do this.

    Al-Hayderi: Yeah.

    Ashley: So, how did you go about deciding, is this gonna be kind of an overview of how to do it? Is it going to be, “Here’s the metrics to watch” and the value that comes out of it is a combination of that? Tell us a little bit about what part of this you’re—kinda what take you’re taking in your talk.

    Al-Hayderi: Yeah. So, public speaking was honestly never my strong suit when I was younger. And some great advice I got was that, when you’re giving a speech to a bunch of people, you wanna tell it like a story.

    Ashley: Mm-hmm.

    Al-Hayderi: So, since then, I’ve really focused on storytelling for even these tech talks I give internally.

    So, the story I’m trying to tell here is how we leveraged Istio to finally solve some of the deep technical problems of running Canaries.

    Ashley: Mm-hmm.

    Al-Hayderi: And once we got it out there, sort of the growing pains and, you know, first obstacles we hit and problems we’ve got. I’m hoping people come out of this with a better idea of how to actually get this running in your infrastructure and some common pitfalls to avoid.

    Ashley: Mm-hmm. Great. Well, tell us a little bit of that story. Kinda get us started down that path.

    Al-Hayderi: Totally. So, the first thing I did was, I went in and learned exactly what Canaries are. And the basic point of it is that you want to expose a small amount of traffic to something new. You want to analyze the deltas and some sort of metrics or something between the new and the old. And then you want to make a decision if this new is ready for the majority of traffic.

    Ashley: Mm-hmm.

    Al-Hayderi: So, there’s a lot that goes into that. Anyone that’s worked with Spinnaker on production scale knows that managing all these pipeline definitions is very cumbersome, and trying to, if you end up changing something, especially changing production pipelines, it can be really dangerous.

    So, a lot of the things we learned about our, you know, how we tested this, even just testing the Canary process in our sort of R&D Dev staging environments and the problem there, even though we got confident with how the pipelines worked, it was very difficult to get a signal on how to actually tune your Canaries. And when I say tune, I mean, when you’re comparing metrics from the new and the old, how do you know if it’s bad or how do you know if it’s a good decision to move forward? What are the thresholds, what metrics are you looking at? A lot of our iterations were focused on that.

    Ashley: Interesting. Now, I’m curious if—are you approaching this problem of using Canary to understand the service mesh or the services itself, or are you trying to also understand kinda Istio and it acting as a sidecar proxy and how that’s performing coordinating traffic load balancing across that? Is it one or the other or both, or what part of the problem were you looking to solve?

    Al-Hayderi: Right. So, actually, we were kind of blocked on the fact that we couldn’t route traffic, you know, with small granularity coming in from the Internet. And we actually only use Istio for its API gateway feature at the moment.

    Ashley: Oh, okay, okay.

    Al-Hayderi: Yeah, there are some blockers right now, at least in our version of Istio, that we’re running around connecting over SSL to Redis. So, that’s blocked having sidecars or application pods. So, when we do canaries, we’re just canarying end user or ingress traffic into our monolith.

    Ashley: Okay.

    Al-Hayderi: Yeah.

    Ashley: Interesting. So, tell us the story. How did you, you went about implementing it—what did you learn from it?

    Al-Hayderi: So, I mean, first off, we had our sort of snowflake-y pipeline that would deploy our own little service and change traffic and compare metrics. So, I guess the first part about it was how do we route traffic and what we actually do is, in a single pipeline, we’ll deploy containers to a new supergroup and we’ll, that’s using the V1 provider of Kubernetes.

    Ashley: Mm-hmm.

    Al-Hayderi: And then we actually used the V2 provider of Kubernetes to apply a virtual service manifest. Virtual services are just a kind of way of routing ingress traffic based on some set of rules. So, if the host header matches this and the path has this Regex, route it to this service.

    What you can also do is, when you match a rule based on some ____ path prefix is route to several services based on weights. So, we first experimented with how are we gonna get a small amount of traffic, yet a meaningful enough amount of traffic to get a good enough sample size of data.

    So, a lot of our, at the start, what we were doing was just sort of like load testing and seeing, “Okay, is 5 percent of the traffic to the Canary deployment enough? How about 10?” et cetera, et cetera. We also looked at some common best practices around Canaries and how they recommend running baseline deployments. So, what we would actually do is deploy two new deployments. So, one with the new code on a small sized server and one with the old code on a brand new, deployed small server. That way, you sort of take away a lot of variables from the experiment, like long running processes, et cetera.

    So, we got there, got traffic routing correctly. Now, the next part we had to do was metrics, right? How do we analyze the delta between the two deployments? At this point, Kayenta only supported Datadog, at least out of the metrics providers that we used.

    Ashley: Okay.

    Al-Hayderi: So, what we really wanted to do, you know, going back to the whole point of getting Canaries out was stop deployments that are just plain broken, right? So, kind of getting, like, a cheap, easy way to get smoke tests. And so, we were just looking for server errors.

    So, as we were trying that, the problem is, is that the server error metric that Istio gives you will only send metrics of a server error happens. So, between two deployments, if you only get two server errors on one, that will trip the Canary as a failure. It doesn’t give you the option to aggregate metrics, so we couldn’t do things like check error rates. So, that just really never worked.

    And actually, in my talk, I go into sort of the—this was kind of a big, big problem the first time we actually released this into production is that Canaries would just fail a lot for no reason, right? And Canaries running for an hour long, you know, our release engineer is sitting there waiting for an hour and then has to trigger it again just because, you know, one server error happened on the Canary deployment.

    Ashley: Well, it sounds like maybe one of the lessons you learned or concept you came up with is, don’t cast a big net. Start out with sort of higher, larger grain, most fundamental issues, kind of a ____ kinda concept. Start there, get those—get that quality fixed, improved, so you’ve got those things working better, probably tighten the net a little bit more and filter out some more specific things that are causing issues. Does that sound like the approach that you learned how to take?

    Al-Hayderi: That’s exactly it, and actually, what I would recommend is, the first time you give out Canaries, don’t have it actually fail or roll back the deployment if the Canary was unsuccessful.

    Ashley: Mm-hmm, okay.

    Al-Hayderi: What we actually ended up doing was actually running this for a month and just gathering data, seeing which metrics worked, which metrics didn’t, and mapping up Canary reports with what we actually saw as problems in production. That way, that gave us more information on which metrics to use. We ended up going with average latency and 95 P latency and once that was out, we started actually aborting and rolling back deployments and then we started to get a lot of great value of this tool.

    Ashley: Mm-hmm. Yeah, that’s actually very similar to kind of a quality process, right? You don’t solve all problems at once. You start, you prioritize, you filter, you work done—you know, improve it one step at a time, rather than just throwing a canary out there and seeing what happens, which sounds like you’ll get results, but you don’t know what or why, right?

    Al-Hayderi: Yep, exactly.

    Ashley: You’ve gotta dig into it and be a little more thoughtful about it, it sounds like.

    Al-Hayderi: Yep.

    Ashley: Good. Anything else? Any kinda other big things you’re expecting to talk about as part of this?

    Al-Hayderi: So, another big thing we learned was, even after this, when we had, you know we were confident that our Canaries were failing when they should and passing when they should. The problem we got was, when a Canary failed, people didn’t really know what to do.

    Ashley: Mm-hmm.

    Al-Hayderi: Because, you know, when a deployment fails, we have a lot of data and we have a lot of observability into that. The Canary is getting a lot less traffic, and it’s hard to know exactly what failed just by looking at the Spinnaker UI Canary report.

    Ashley: Mm-hmm.

    Al-Hayderi: So, we then, you know, sort of invested more into our—we used New Relic for our APM and we piped our Canary into that so that we could actually see transaction data and the delta between it and the baseline metrics to see, you know, which end point, for instance, was misbehaving or was some new database core utilizing this—that helped us a lot, too.

    Ashley: Mm-hmm. Interesting. Well, it sounds like you’re using—think about this in a systemic way, not just Canaries and Spinnaker, but also you mentioned Datadog, New Relic, a number of tools that you’re using as part of your environment, which all have to come together to figure it out, right? It’s not just one thing.

    Al-Hayderi: Yep.

    Ashley: Well, good. I wish you the best in your talk. The folks that would go to this talk, are they gonna be, tend to be software developers, engineers, architects, operations focused on the DevOps team? Who tends to be more interested in this than others?

    Al-Hayderi: I would say more the DevOps operational people. People who work with the Spinnaker pipeline definitions at their company would get the most value out of this. But, you know, back when I was just a developer, I heard a talk about Canaries, and that sort of triggered me to bring this in. So, I mean, anyone who’s interested in resiliency and, you know, release confidence would get value out of this talk.

    Ashley: Okay, excellent. Very good. Well, thank you for being on the podcast, Omar.

    Al-Hayderi: Thank you very much, Mitch.

    Ashley: It’s been great to have you, Omar Al-Hayderi, who is engineering manager, formerly a principal engineer, at Autodesk. He’s gonna be speaking at the Spinnaker Summit 2019 in San Diego. The dates for that is November 17th through the 19th, and Omar’s talk is on Sunday, the 17th, at 1:30 p.m.

    So, thank you, all of you, for joining us and listening to this episode of DevOps Chat podcast. This is Mitch Ashley with devopsy.kinsta.cloud. You’ve listened to another DevOps Chat. Be careful out there.

    — Mitchell Ashley

  • DevOps Chats: Plugins Extend Spinnaker for the Enterprise

    DevOps Chats: Plugins Extend Spinnaker for the Enterprise

    Spinnaker Summit 2019 Preview: Secrets management is vital to any vibrant DevOps and GitOps toolchain. As part of its open source maturation and thanks to the contribution of our DevOps Chat guest, Spinnaker now has a secure secrets management capability enabling GitOps, taking non-application secrets out of plaintext in hal config files.

    Cameron Motevasselani, software engineer at Armory, is giving his talk “Spinnaker Plugins: Extending Spinnaker for the Enterprise” on Sunday, Nov. 17, at 10:45 AM PT.

    In addition to his contributions to Spinnaker’s new secrets management facility, Cameron shares the work in progress to create an extensible Spinnaker plugin system. The current work is an early MVP to be followed by the implementation of PF4J and concrete extension points as part of Spinnaker stages.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Cameron Motevasselani—Motevasselani, thank you [Laughter]—a software engineer. He’s also a speaker at Armory. He has a couple of talks that are at the upcoming Spinnaker Summit 2019 in San Diego. First one—I guess this first one is on the plugin system and the second one is on secrets management.

    So, Cameron—welcome to DevOps Chat.

    Cameron Motevasselani: Hi. Thanks for having me.

    Ashley: Glad to have you on. Well, tell us a little bit about yourself. Introduce yourself and tell us a little bit about what you do at Armory and what Armory does.

    Motevasselani: Oh, totally. Yeah. So, I’m Cameron Motevasselani. Thanks for the quick introduction. Let’s see, I’ve been at Armory for about a year now. I joined and hopped feet first and started working on this secrets management project. So, that was pretty cool. And then, now, recently, I’ve been working on this plug-ins system. Yeah, that, I think, about sums it up.

    Ashley: Great. What does Armory do? Yeah.

    Motevasselani: Yeah, Armory, we basically have an enterprise offering for Spinnaker. So, we have some extensions that we’ve written on top of Spinnaker, so we don’t fork and have to maintain all that, we just—we extend Spinnaker and provide some extra features that way.

    Ashley: Very nice. Now, the way you phrased that, you’ve been working on secret management and the plugin. Does that mean you’re actually contributing, writing code and contributing it to the open source, or are you primarily an implementer of what’s already been built?

    Motevasselani: Oh, no, that’s a great question. Yeah, for the secrets management as well as plugins, we both, we’re contributing directly to open source for both of those projects.

    Ashley: Awesome. Excellent. Well, let’s start with secrets management. I mean, I think we all kinda have the idea of where we’re gonna keep our keys, our passwords, certificates, all that good kinda stuff. We don’t want that heading into the source code management system or somewhere in the cloud or in someone’s software. Tell us a little bit about the secrets management part of Spinnaker and what you’ve been doing, what you’re gonna talk about.

    Motevasselani: Yeah, totally. So, secrets management really came up as we started talking with customers about their Spinnaker setups and how they deploy and manage the configurations for Spinnaker.

    As you know, Spinnaker has the keys to the kingdom, essentially. It has your cluster keys and your credentials in order to deploy cloud resources that cost you money, right? And those are running—it runs the code that your organization arrives on. You can’t have that in source control. That is not, not good to have.

    Ashley: No-no—a big no-no, though a common mistake. [Laughter]

    Motevasselani: That’s right. Right. [Laughter] I think GitHub even yells at you now, if you have keys—

    Ashley: Oh, does it? [Laughter] Nice.

    Motevasselani: – in there. So, they’ve been adding nice little security features here and there. People would like to move towards this idea of GitOps so that you can manage your configuration via pull requests. It’s a great idea, and a lot of people can’t do that, because the keys are quite literally in the Spinnaker configurations.

    So, we pull those keys out and are essentially referring to them with our secrets management solution. So, you can use different backing stores such as S3 buckets or Vault and store your secrets there and then refer to them in the Spinnaker configuration that way.

    Ashley: Mm-hmm. Excellent, great. So, in your talk, you’re gonna be walking through what you’ve been doing, the functionality of it, how to implement it via GitOps?

    Motevasselani: Yeah, so—that’s right. The functionality, how it differs from, there are other solutions coming into open source now that tackle a slightly different use case. So, discussing the differences there as well as the future, essentially, of secrets management.

    Ashley: Okay. Very good. Great. Anything else to tell us about that before we move into the plug-in system?

    Motevasselani: Yeah. You know, secrets management, it is about the secrets for Spinnaker in particular. It’s not actually tackling the application secrets themselves.

    Ashley: Oh, okay.

    Motevasselani: So, I just wanted to make that clarification. And that’s at the current time. It’s possible that we might have a story for that in the future, but the talk that I’ll be giving is around Spinnaker secrets, not application secrets.

    Ashley: Okay. So, this is just what’s required to run, operate, manage the Spinnaker environment, not your application secrets?

    Motevasselani: That’s correct.

    Ashley: Okay, great. Well, that’s an important clarification. Great. Well, let’s talk about the new plug-in system. What is new about it, other than being new code?

    Motevasselani: Yeah. So, the plug-in system is, we’ve been developing it as a very—it’s been a very alpha, MVP type feature. So, right now in open source, we do have a method for users to create custom stages in Orca. There is quite a bit of knowledge and expertise that you have to have in order to do that, so I don’t quite recommend doing it right now. But we’re building on this plug-in system and we’ll be testing it out with users hopefully in the near future.

    So, right now, there’s currently no UI, but we’re working on a UI and then that should be getting open sourced as well in the future.

    Ashley: Now, that sounds like kind of a tough assignment. How do you create a plug-in system that anybody can do anything they want with, because that’s probably one of the requirements is, you want a lot of flexibility. How do you go about architecting and designing that so what you’re building can be extensible and you’re not gonna have to re-do it in a year?

    Motevasselani: Yeah, that’s such a great question. Because we are going with this MVP model of just getting something working and then getting feedback on it to iterate, we actually are already planning on deprecating this first, let’s say, method of loading plugins into Spinnaker. We’ve already been doing some internal revs on it and we’ve been working with the community, Google and Netflix in particular, on this project. So, we are going to be moving away from our current implementation into another full-fledged plug-in framework, and we’ve chosen PF4J for that.

    Ashley: Okay. Now, I’m not familiar with that framework. Can you tell us a little bit about that?

    Motevasselani: Yeah, so the way we currently did it—and hopefully, this isn’t giving too much of my talk away [Laughter]—but the way we’re doing it right now is essentially shoving JARs into a ClassLoader for Orca. Well, via Quark. So, these are different Spinnaker services and libraries that make up Spinnaker.

    Ashley: Mm-hmm.

    Motevasselani: Right now, it’s a pretty dumb way of doing it, just adding JARs onto a Classpath in order to load it onto the Spinnaker spring application context.

    Ashley: Makes sense. Pretty easy way to do it.

    Motevasselani: Yeah, exactly. But like you’re saying, there’s no—it’s very open ended right now. We are going to, as we move to PF4J, define concrete extension points within Spinnaker so that there’s actually not a sprawl of extension points, there’s clearly defined extension points throughout the code.

    Ashley: Mm-hmm.

    Motevasselani: That will help us iterate on this feature and make sure it’s a good solution as we continue developing it.

    Ashley: Okay, and this is the PF4J that’s on GitHub? I think it’s an Apache license, right?

    Motevasselani: Yeah, that’s correct. I believe it’s Apache. That’s correct. And so, moving into PF4J will give us a lot of really nice features that our current implementation doesn’t have, such as using a different ClassLoader for every plug-in so that it can have its own dependencies, et cetera.

    Ashley: That makes a lot of sense. Are there any unique challenges for implementing a plug-in system in an environment like Spinnaker? You know, this is software to create software, software to deploy software. You know, it’s not an application in and of itself, it’s something that you’re gonna make a key part of your kinda tool chain, your workflow pipeline. I wonder if there’s any unique challenges that come about in trying to build this.

    Motevasselani: Yeah, it’s interesting, because we have been looking at, you know, as you develop something new, you look at prior art in order to determine how to build what you’re building, I guess. [Laughter]

    Ashley: Mm-hmm.

    Motevasselani: For lack of a better word.

    Ashley: Technical speak, I know. [Laughter]

    Motevasselani: That’s correct. [Laughter] So, looking at something like Jenkins or Eclipse that both have plug-in systems, same with WordPress, those are all kind of monoliths, right?

    Ashley: Mm-hmm.

    Motevasselani: Spinnaker is definitely many different microservices composed into one application. Essentially, we’re just kind of turning it into a platform, kind of, by making it extensible and pluggable.

    But that is, to answer your question, a major challenge is, you have all these different microservices. You know, depending on how you deploy and update your Spinnaker installation, they might be on different—I don’t wanna say different versions, but if you’re deploying master all the time, you might have a different set of microservices in production compared to if you’re all running the same release version.

    So, deploying plug-ins into that environment is not an easy task and how you get the versioning done is important, as well as rollbacks.

    Ashley: You kinda have a recursive problem here, right? Because you have to version control the plug-in manager, which then version controls of what plug-ins can work with it, c?

    Motevasselani: Yeah, exactly. Plugins might rely on some functionality that is in one version of a microservice that’s not in another version, and if you need to do a rollback, how does that work, et cetera. That’s correct.

    Ashley: Well, you kinda took my breath away for a moment. You scared me when you said, you mentioned the WordPress plug-in system.

    Motevasselani: [Laughter]

    Ashley: Please, don’t do that! [Laughter]

    Motevasselani: I do think, though, that WordPress is an example of a project that has done very well—

    Ashley: They have.

    Motevasselani:—because it has such as good ecosystem of plugins.

    Ashley: It’s come a long way. It has come a long way. It’s much more, I don’t know, not self-sustaining, but kinda auto updating much better than it used to be. [Laughter]

    Motevasselani: Right, right. And we do hope to take learnings, right, from past projects, let’s say, as well. [Laughter]

    Ashley: Absolutely. I don’t mean to dis WordPress, you know? You gotta every once in a while. Well, tell us a little bit about more about your talk. What are you gonna cover, then? Is this more how to use it, or you’re gonna lay out kinda where you’re going in this next version and you’re looking for feedback? What are you trying to accomplish or get out of it? What would you like to have happen in the talk?

    Motevasselani: Yeah, you know, that’s a really good question. I think there’s a lot of stuff to cover, all the way from our journey running and creating this project, working with the community and those efforts, to showing users, “Hey, this is actually a nice and easy way of modifying or extending Spinnaker’s functionality.”

    The idea here is to—there’s kind of a few ideas here with this project, because we really want to lower the barrier of entry for users, particularly in enterprises, to be able to create extensions for Spinnaker. The method we chose to do this at first was a way that most users interact with Spinnaker, and that’s creating pipelines and defining stages.

    So, let’s say an enterprise has some custom infrastructure at their business running their data centers, running their cloud. If they want to, let’s say, interact with that in a native—in a Spinnaker native way, you would want to have a stage that their users can just choose. So, rather than having to learn how Spinnaker works on a very deep level, you can implement this nice, simple interface and get it going.

    So, at the end of the talk, it would be great for the audience to have an understanding of how to create and implement their plugin on their own.

    Ashley: Oh, excellent. Well, then, hopefully, you can, you’re creating an audience and a community to go out and try your new plug-in system and the variations of it.

    Now, you talked about using plug-ins as extensions to Spinnaker. Do you also see this plug-in system being kind of an integration interface to third party tools, maybe other open source, other commercial tools, or is it more things that just extend the functionality of Spinnaker?

    Motevasselani: Oh, that’s a really good question. So, there is a push to move Spinnaker to be a lean core and an expansive ecosystem, I believe, is the term being used, and this goes directly towards that goal. There’s certain functionality in Spinnaker, the core, Spinnaker core, that doesn’t quite make sense.

    For instance, there’s some stages that are specific to companies. Don’t wanna call Gremlin out, I think they do great work. There’s a Gremlin stage in Spinnaker that should probably be a plug-in. So, hey, Gremlin guys, hit me up. [Laughter] But not quite yet, maybe for the end plug-in system.

    Ashley: It’s that evolution of software, right?

    Motevasselani: Exactly.

    Ashley: I mean, that’s how you get to this point where you need to do that, you need to spin things out, so it’s part of its growth.

    Motevasselani: That’s right, that’s right. Yes. It would be really cool if you could have a Spinnaker installation where you can essentially pick and choose which cloud providers you want to have installed and running. If you only—you know, if you interact with, let’s say, both Google and Amazon’s cloud, maybe you don’t need Oracle’s cloud software to be on your cloud driver. Or maybe you want to have a cloud driver in each of those data centers—sorry, in each of those cloud accounts that only has the code necessary to deploy to that cloud environment.

    Ashley: Mm-hmm.

    Motevasselani: So, then you have a smaller attack surface, just leaner and smaller binaries, et cetera, et cetera.

    Ashley: Great. Well, I think, unless there’s anything you wanna add, I think you’ve given a really comprehensive look to kinda what that talk, both talks are gonna be about.

    Motevasselani: Yeah, that’s—you know, that’s hopefully, just getting the community excited and get some ideas flowing, I think, would be great with those talks.

    Ashley: Well, and I also wanted to just throw in my little thanks just to appreciate what you do and all the other contributors to the projects in software. And I always encourage people who are just using open source, and there are folks at the Spinnaker Summit who are implementing it, not writing code for Spinnaker, but talking about it at a summit like this is another way to contribute to the community. So, great to talk with you who are actually writing code as well.

    Motevasselani: Yeah, thanks. Nice talking with you, too, Mitch.

    Ashley: Great. Well, it’s been my pleasure. You’ve listened to another DevOps Chat podcast. I’d like to thank today’s guest, it’s Cameron Motevasselani, software engineer at Armory, and he is speaking at Spinnaker Summit 2019. The dates for the conference are November 15th through 19th in San Diego, and Cameron has two talks. His talk on the plugins is Saturday, November 16th, at 2:30 p.m. And then he’s also talking bright and early about secrets management—boy, you’re really gonna wake ‘em up here—Sunday morning at 9 a.m., Cameron, so. [Laughter]

    Motevasselani: [Laughter]

    Ashley: So, you know, maybe you serve mimosas at the morning talk. [Laughter] So, it’s been my pleasure to have Cameron on, and thank you to all of our listeners today for joining us. This is Mitch Ashley with devopsy.kinsta.cloud. You’ve listened to another DevOps Chat podcast. Be careful out there.

    — Mitchell Ashley

  • DevOps Chats: Debugging Spinnaker Apps, With Salesforce

    DevOps Chats: Debugging Spinnaker Apps, With Salesforce

    Spinnaker Summit 2019 Preview: Debugging production issues in any environment can be challenging, and Spinnaker has its production learning curve. Problems aren’t always replicable in a smaller environment, and debug messages can be verbose and confusing to triage what’s happening.

    Our DevOps Chat guest Chuck Lane, Salesforce lead software engineer, is giving a talk on “Debugging and Profiling Spinnaker Applications Live” at the Spinnaker Summit 2019. In Chuck’s talk, you’ll learn skills like remote JVM debugging, custom profiling builds and the magic of figuring out what’s going on with a multithreaded microservice using htop.

    Chuck’s talk is on Saturday, Nov. 16, at 3:40 PM PT. Spinnaker Summit 2019 is Nov. 15-19 in San Diego.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Chuck Lane. He’s a lead software engineer with Salesforce.com. He works on Sales Cloud, which is the Salesforce you and I use, know and love, or our CRM capabilities. Chuck is talking at the Spinnaker Summit 2019 in San Diego, and his talk is about debugging and profiling Spinnaker applications live. And I think live means, like, live while they’re running, in production, and see what’s going on. We’ll explore what that is. This talk is on Saturday, November 16th, at 3:45 p.m. in San Diego. Chuck, welcome to DevOps Chat.

    Chuck Lane: Thank you. It’s nice to be here.

    Ashley: Excellent. Awesome to have you here. Would you start by just introducing yourself? Tell us a little about you and what you do at Salesforce.

    Lane: Yeah. So, my name is Chuck Lane. I am, as you mentioned, a lead software engineer. And basically, my chief responsibility at Salesforce is to help bring Salesforce into using Spinnaker for our public cloud based deployments. So, as we make a transition to try to move away from first party architecture and move over into the public cloud, one of the technologies that we’re using to do that is Spinnaker and I’ve established myself as a subject matter expert in Spinnaker. And so, I kinda help to bridge the gap between the traditional development and deployment cycle and what that looks like using Spinnaker in a containerized world.

    Ashley: Okay, great. You used a term there I don’t know if I’m familiar with. You mentioned something about moving from first party architecture to the public cloud. What’s that mean about the environment you’ve been in that you’re, and what you’re moving to?

    Lane: Yeah. So, basically, what I mean there is that, historically, Salesforce has owned their own data centers, and we’ve deployed to those data centers. We are moving, embracing in a big way public cloud architecture, whether that’s through AWS, whether that’s through GCP, or any of the number, any other number of cloud offerings.

    And so, as opposed to doing what some people would call a lift and shift where you just take your software that’s meant for, to be run on hardware that you own and move it into the cloud, we’re doing a fundamental re-architecture of the software to make use of all of the great things that cloud architecture allows us to take advantage of.

    Ashley: Mm-hmm. Fantastic. That’s interesting to hear a little bit about your own evolution at Salesforce. So, your talk is about using Spinnaker, you know, really about debugging and finding out problems that are happening in production, but sometimes you might struggle with trying to replicate it into a smaller environment to get the same problems working.

    Can you tell us a little bit about what sort of led you down this path to figure out how to do this? Was it a big problem that was happening or something you saw repeated over and over that there wasn’t a good solution to and you figured out how to do that? How did this come about?

    Lane: Yes. So, basically, as you’re taking workloads and migrating them over, I mean, when—you know, when you just start with a small subset of workloads, you know, the path is relatively straightforward. And, you know—hopefully, anyways—and you don’t run into too many issues that can’t kind of easily be solved.

    But the reality is, as you transfer more and more workloads over to the public cloud, that’s—you know, the devil’s in the details, right? So, that’s really where things can pop up where you’re hitting various limits or software isn’t performing in the way that you would expect it to perform. And these are the situations where we—where it can be very beneficial to jump into production software and really, you know, sometimes slap a debugger on it and see exactly what it is that it’s doing that differs from kinda what you expect. You know, and that can help you kinda tailor your workloads in such a way that you can make things run more smoothly.

    Ashley: Mm-hmm. I know sometimes using debuggers, enabling them—that can be helpful, but it also can be too much information, trying to sort through what all is happening, trying to find what should you be looking at. What are some of the challenges you’ve found by turning on debugging or using debuggers?

    Lane: Yeah. Well, so, I mean, there definitely is that problem that you’re saying just as far as too much information that’s coming out. One of the nice things about Spinnaker and the way the services are written is, you have a very fine grained tuning over which libraries inside of Spinnaker you tell it to print out and debug information.

    Ashley: Mm-hmm.

    Lane: So, if we know that there is a hiccup with one of the data binding layers or a hiccup with one of the authorization layers, then we can, through the config files, we can—and Java settings—we can really target that explicit directory, or I’m sorry, that explicit library and say, “Give me all the debug information that you have.”

    Ashley: Mm-hmm.

    Lane: But, you know, failing that, I mean, there are definitely times when it’s been advantageous and the best course of action is just to go down and actually look at the code and see what it’s doing. And, again there, it’s best to do that under—you can’t do it in production, usually, but what we can do is simulate a load that’s similar to what we would see in production and really, really take a look at what’s going on to help us identify those algorithms that might be o event squared instead of a login or something like that.

    Ashley: Mm-hmm. Now, are there some specific techniques—I know in your description of your talk, when you talked about using remote JVM debugging, custom profile builds, even using htop, you know, a UNIX command or a LINUX command to help you figure out what’s happening with multi-threaded microservices. It sounds like a variety of different approaches that you’ve kind uncovered and learned that you can use to figure out what’s happening.

    Lane: Yeah. And so, I mean, in general, we like to run as closely to the open source build as we can, the ones that are provided by Spinnaker, which is ultimately provided by Google. But there definitely are circumstances where we need an extra tool, be it htop, be it something like Glowroot, which is a Java application profiler where what we’ll do, then, is we’ll go in and we’ll build a custom image using the Spinnaker images as a base and then tack on those additional libraries and put them into the Spinnaker ecosystem and then launch them up just to see what additional information we can get out of there.

    And so, you know, one scenario where that was really useful to us, we ran into a scenario where cloud driver, which is—cloud driver is the main tool that talks back and forth to all of our different cloud infrastructures.

    Ashley: Mm-hmm, mm-hmm.

    Lane: Sometimes calls to cloud driver were taking upwards of, well, two to three minutes to respond. Now, you know, timeouts are set in such a way that if it doesn’t hear anything back in about 30 or 60 seconds, then it just disregards that load and, you know, so that created quite a bit of problems for us.

    By building a custom cloud driver image that had htop in it, we were able to see that the majority of the processes that were running were actually running basically commands to reach out to the Kubernetes clusters and get a list of all of the name spaces that are in the Kubernetes cluster. And, talking it over with some of the Spinnaker developers, what we found is that if you have 15 or so clusters that you’re connecting to Spinnaker, then that’s not so much of a problem to go and query each of those to get the list of name spaces. But as you scale up, on the order of 400 or 500 different clusters the way that we do, then a lot of the delays and a lot of your time can be just essentially Kubectl calls waiting to return to you the lists of name spaces that you need to go and scan.

    So, we implemented a workflow based solution for that, basically, where we let our teams know that when they’re creating their clusters, they should use Terraform to go ahead and create the clusters—I’m sorry, the name spaces that they’ll be using. And then, if they need to use a name space after the fact, then we provide a pipeline that they can use that will dynamically add in an additional name space that their cluster will start to scan. So, that saves us from scanning all the clusters all the time, which was causing quite a bit of performance bottlenecks.

    Ashley: Yeah, I would imagine that would have some overhead, maybe a lot of overhead if you’re doing that frequently. So, it sounds like a way to both reduce overhead, but also to get the information faster.

    Lane: Exactly, exactly—yep, yep.

    Ashley: Mm-hmm. Very cool. So, in your talk, are you going to be doing any demos? I know there’s always the demo Gremlin, or are you gonna be just showing, you know, talking about what some of these techniques are?

    Lane: So, I plan on doing some demos. As far as—

    Ashley: That’s really cool.

    Lane: Yeah. You know, I may do the, what’s the—the Easy Bake Oven a little bit as far as, “Here’s the behavior and here’s what we’ve found in code.” But from what I understand, a lot of that stuff can kinda be dependent on Internet connectivity at the actual site. So, if it’s something where we have good Internet connectivity, then by all means, I plan on walking through a couple of debugging scenarios, as close to what we would do in real time as possible.

    Ashley: Yep. Well, you know, someone somewhere—it’s been a while ago—gave me the great advice of, you know, “Have your live demo and have your disconnected demo ready in the background.” [Laughter] So, you can always at least show something locally if you can’t get on the net. So, if you’re depending upon the network—I’m not sure if you are for your demo, but always a good lesson, right?

    Lane: Sure, exactly.

    Ashley: Great. Are there any other kind of lessons learned, common mistakes, or mistakes that you or others might have made along the way, kinda so there’s hard things you learned by trial and error that you plan on sharing?

    Lane: I mean, well, there are—whew. You know, I’ve been working on Spinnaker for a couple of years. So, there’s definitely been a lot of hard lessons learned, here. But honestly, what I would do is, I would encourage anybody who wants to get into Spinnaker to not just hang around the Slack channel, because the Slack channel does have a tendency to get overrun with, you know, just kinda people posting their stack traces and just saying, “Hey, has anybody ever seen this before?”

    And, you know, I’ve gotten a lot more success by going through the commits, looking at the people who actually authored the code, and then reaching out to them directly with more than, “Hey, can you explain this to me?” but rather, you know, “Hey, I see what you did here and I see what you did here. I’m running into problems with these lines. Do you have a different approach, or is there something that I can be doing differently?”

    And the other thing that I just can’t overstate is the value of being a member of one of the special interest groups.

    Ashley: Hmm, interesting.

    Lane: So—yeah. So, I’m a member of the Kubernetes V2 special interest group that’s lead by Eric Semene and Ethan Rogers from Armory, Eric’s from Google. And it has, you know, it has been just an absolute wealth of information and, you know, honestly, I don’t know if we ever would’ve gotten nearly as far as we had without those two people.

    So, you know, yeah, I would just say that the community is really friendly and we always welcome new members who are ready to learn.

    Ashley: You know, I think both of those are great suggestions, and I really appreciate that you made those. Because, one, things like the Slack channels on projects, open source, those can be a bit intimidating. Sometimes they’re not approachable, because there’s just so much noise so much happening on it and people reaching out for help like you’ve talked about, “Here’s a stack trace I’m trying to figure out.”

    But also, your recommendation of reaching out to the code authors, you know what, it’s—people like to help each other. And, you know, it might seem like, “Hey, the people who wrote this aren’t gonna have time to bother with me”—they love to hear from people that are using their stuff to talk about it and help them out, but also hear about how they’re using it, and of course, they’re always looking for ideas and feedback and stuff like that. It sounds like you’ve had that kind of an experience.

    Lane: Yeah, absolutely. I mean, and you know, the big thing is just, you know, as a coder, it’s easy to tell the people who are coming to you and just kinda wanting you to fix it for them and the people that are coming to you that have really tried to tackle it themselves. And, you know, I can’t speak highly enough about the latter rather than the former. I mean, you know, just give it your best shot and when you get stuck, reach out to somebody and it’s, you know, it can be immensely valuable.

    Ashley: It’s kinda like going to a foreign country. Wherever you are, if you’re American going somewhere else or vice versa, everybody appreciates you trying to speak the native language, and at some point, when they see you struggle enough and how far you can go, they’re glad to help you and, you know, speak in your language.

    So, same kind of thing with helping people with code. If you’re gonna just fob your problem off onto the developer of it—not appreciated so much. But they appreciate that you tried to take it as far as you could. As you mentioned, go look at the code and figure out what’s going on.

    Lane: Yeah.

    Ashley: You’ll get immense respect, you know, even if you aren’t a coder at that level, at that level of software developer, it’ll mean a lot to the developers.

    Lane: Absolutely, absolutely.

    Ashley: Well, hey, I think you’re gonna have a fascinating talk, and I love that, you know, this idea of trying to figure out what’s happening in production and some of the techniques that you’ve come up with and developed and have experienced and the fact that you’re sharing those with others. I’m curious, do you contribute any code to the Spinnaker project in any areas? Are you primarily a practitioner user of it?

    Lane: So, I have. I’ve got a few small RPRs that have been pushed through, but really, the bulk of my commits right now have actually been to the Spinnaker website, the documentation. So, you know, I don’t know Java or Groovy or Kotlin, maybe, as well. Some of the patterns they use are a little bit—I come from a .net world, so they’re a little bit foreign to me.

    But yeah, I’ve definitely written a number of different documentation pages and I found that that’s a great way to kinda get in and I’ve even got some PRs that are coming ups soon that aren’t doc related. So, yeah, hopefully, you’ll see my name more.

    Ashley: Excellent. Well, you know what, documentation is important, too. There’s some folks doing talks at Spinnaker Summit that, you know, kinda ran into that point where, “Hey, there’s documentation for doing it one way, but not under this set of configurations or software.” So, that’s a contribution, too, so congratulations for being a part of the community and for sharing, also, your experience at the Summit.

    So, Chuck, appreciate you being on the podcast today.

    Lane: Oh, thank you. It’s been an absolute privilege. Thank you so much.

    Ashley: Absolutely. Fantastic. I wish you all the best with your talk, I’m sure it’ll be great, and hopefully folks listening to this podcast will draw some more interest and bring folks to listen to you.

    So, I’d like to thank our guest today, Chuck Lane. He’s lead software engineer at Salesforce.com, so you can imagine the environment he’s working in. There’s some super good lessons that Chuck’s bringing to the table. He’s gonna be talking at Spinnaker Summit 2019, which is in San Diego, November 15th through the 19th. His talk is debugging and profiling Spinnaker applications live, and his talk is on Saturday, November 16th at 3:45 p.m.

    I’d also to thank you—you, our listeners—for joining us today. This is Mitch Ashley with devopsy.kinsta.cloud. Have a great day and be careful out there.

    — Mitchell Ashley

  • DevOps Chats: Spinnaker Extensibility, With Armory

    DevOps Chats: Spinnaker Extensibility, With Armory

    Spinnaker Summit 2019 Preview: Most open source software isn’t one size fits all, and Spinnaker is no different. While it captures the most general use cases of software delivery, most organizations find that it lacks some features they need to automate their entire delivery pipeline, such as integrations with internal tooling or compliance systems.

    Out of the box (figuratively), Spinnaker provides Run Job and Webhook stages that teams can use to build custom integrations to help fill any void and enable adoption by a wide range of users.

    Ethan Rogers, staff software engineer with Armory, joins us on DevOps Chat to share a preview of his Spinnaker Summit 2019 talk on Spinnaker extensibility. Ethan shares how Run Job and Webhook stages are used to build custom stages to capture any number of use cases with no code and simple configuration. He also covers situations when it’s necessary to write code.

    Ethan’s talk, “Making Spinnaker Your Own,” is on Saturday, Nov. 16, 3:45 PM, at Spinnaker Summit 2019 in San Diego. Separately, Ethan appears on the “State of the Kubernetes V2 Provider…” panel Friday, Nov. 15 at 2:30 PM.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Ethan Rogers. Ethan is a staff software engineer and also a speaker at Armory. He’s gonna be talking at the upcoming Spinnaker Summit 2019 in San Diego. Now, Ethan’s gonna be talking about making Spinnaker your own, that’s his talk, and then he’s also gonna be on a panel about the state of Kubernetes V2 Provider, plus improved rollout strategies for deploying Kubernetes using Spinnaker.

    So, let’s get started. Ethan, welcome to DevOps Chat.

    Ethan Rogers: Thank you so much for having me. I’m really excited to be here and get to talk about Spinnaker.

    Ashley: Wow. Absolutely. Great topic. I know you’re passionate about it, so fantastic to have you. Would you just introduce yourself? Tell us a little bit about you, what kinda work that you do and a little bit about Armory, the company that you work for?

    Rogers: Yes. So, like Mitch said, my name is Ethan Rogers, I’m a staff software engineer at Armory. I’ve been actually working on the Spinnaker project for about the last three years. I’ve been at Armory for a year and a half, and then a year and a half before that, I was working for my previous company where we were, you know, I was part of a team for delivery and developer experience. And, you know, we were using Kubernetes for this new project that we were working on and looking for a way to deploy and got involved in Spinnaker super early on. So, I’ve kind of been a part of the community ever since then and it’s been a really great experience so far.

    Yeah, so, that is actually kind of how I found Armory. Closer to the end of my time at my last job, I met the founders, DROdio, Ben and Isaac, and they had just—eh, within probably six to nine months started Armory. And we kinda hit it off. I had a really good relationship with the founders and it was kind of an opportunity to go keep working on Spinnaker full time, which is something I was interested in.

    So, Armory, just a little bit about us—we’re bringing Spinnaker to the enterprise. So, what we’re doing is taking open source Spinnaker and, you know, on top of offering stuff like support and training, we also offer layers on top of open source Spinnaker, various different extensions and features that make it more aligned with what the enterprise is looking for today.

    Ashley: Mm-hmm. Very good. Well, that explains, I know there’s yourself, but there’s also some other folks that are speaking at the conference.

    Now, your topic, you gave a nice introduction about how you got involved with Spinnaker—how long ago was that, when you started using it?

    Rogers: Yeah, it was about three years ago, kind of before the—like, right when the first version of the Kubernetes provider landed. I think, at the time, I didn’t realize how early we were. It kind of explains a lot of the PRs I had to submit when I first got involved.

    Ashley: I’ll bet, I’ll bet. Do you write any code, submit any code or are you primarily on the usage end of Spinnaker?

    Rogers: Yeah, so, I’m actually a developer by training. I got my degree in computer science, and transitioned pretty early on in my career into the ops side of things. So, I’m pretty fluent in both. I actually do write a lot of code. These days a lot less, just because of the role I’m in. At Armory, I do a lot more higher level stuff like BitTorrent, but I still try to get a few PRs submitted every week. I lead a couple of initiatives in the Kubernetes provider itself, so I do a little bit of everything.

    Ashley: Well, I won’t ask you which is the dark side, ops or developing—no, I’m just kidding. It’s DevOps, it’s all together now, right?

    Rogers: [Laughter] Yep.

    Ashley: Okay. So, let’s jump into what you’re gonna be talking about. And, of course, we’re not gonna cover it all here, but I know that you’re focusing on kinda how open source, you usually don’t just use it out of the box, if you wanna use that term. I know it’s free and downloaded.

    Rogers: Mm-hmm.

    Ashley: But that you have to do some customizations to kinda get it working in your environment, and we’re talking about Spinnaker here. What’s your experience working with customers? What’s their expectation when they start to use Spinnaker and what do you have to kinda help them get started with?

    Rogers: Yeah. So, the interesting thing about Spinnaker open source is that it’s actually, like, a very small part of what Spinnaker is at Netflix. For anyone listening who doesn’t know, Spinnaker was a project that was born out of Netflix. They built it, it was the next iteration in deployment technology from a project called Asgard.

    Ashley: Mm-hmm.

    Rogers: And one of the driving principles of Spinnaker was making it extendible. And the primary reason for that was because Netflix has a lot of kind of internal workflows and things that they like to—that make their development experience better, that don’t necessarily apply to the broader community.

    And so, when you start using open source Spinnaker kind of out of the box, there are a lot of things that I think most companies want, right? The same reason that Netflix has those, the things that don’t apply to the community but work for them, every other company has something like that.

    Ashley: Mm-hmm.

    Rogers: So, when I was at my previous company, we had this workflow around our Docker images where we would add labels to our Docker images and then we wanted those labels to show up in the Kubernetes manifest where those images were deployed.

    So, that was something that wasn’t available in Spinnaker out of the box, but we were able to use some of the extension methods to actually go grab that information and inject that stuff, which is kind of the basis of the talk I’m giving around Run Job.

    Ashley: Okay, great. Well, let’s explain that, then, because I know you’re gonna be talking about Run Job and about Webhook Stages. What is Run Job, what is Webhook?

    Rogers: Sure, yeah. So, just kind of, at a high level, Run Job and Webhook are ways that you can extend Spinnaker to kind of make it—to do the things that it doesn’t do out of the box that you need it to.

    Ashley: Mm-hmm.

    Rogers: There are more complicated ways to extend Spinnaker that involve actually taking the code and writing Java code, but we found that the majority of users or operators, I should probably say, are not super familiar with Java. So, that’s a pretty—it’s fraught with friction, I would say.

    Ashley: Yeah, it’s a whole other ball of wax, right?

    Rogers: Yeah, exactly. Yeah. So, the Run Job stage and the Webhook stage give kind of a lower friction way of extending Spinnaker. So, the Webhook stage is a stage within Spinnaker that lets you call out to some API. There’s the, you know, I select it as a stage in my pipeline via drop down and I configure it there. There’s also what we call preconfigured Webhooks or custom Webhooks that allow you to actually configure this as part of Spinnaker’s configuration, and then it shows up like it’s a native stage.

    Ashley: Mm-hmm.

    Rogers: So, under the hood, it’s still calling out to APIs. It’s as if you’ve configured it in line, but it shows up and, to the end user, actually looks like it’s part of Spinnaker itself.

    Ashley: Very nice. So, what are some of the most common uses of that to extend integrations into other things? Is it internal third party tools, internal custom software people who built, is there, you know, typically—the typical use cases for this?

    Rogers: I think that we usually end up seeing people integrating more with internal systems than, say, hosted services like GitHub or other—I don’t know, GitHub’s probably the easiest example. New Relic is another one that we see.

    But I think where, in most cases, Spinnaker has integrations for the public stuff out of the box, but obviously, if the tool’s internal to your company, you don’t have that. So, that is usually where we see it. So, I think there was one company that I had talked to that was, they had this internal integration testing suite or load testing suite that they wanted to call out to, and they used the Webhook stage to make that call and kick off those tests.

    Ashley: Mm-hmm. Great, great. Now, I understand you can do a lot of this without writing code, is that correct?

    Rogers: Yeah. Yeah.

    Ashley: So, what does that entail? What kind of a—what’s the configuration process look like to tie, let’s use that example that you gave of someone that wrote their own test framework of some kind that they wanna call out to in a Webhook stage?

    Rogers: Yeah. So, I guess I should probably back up and clarify my answer a little bit.

    Ashley: Okay.

    Rogers: So, at some level, you will need to write code. The question is whether or not you actually have to write that into Spinnaker itself.

    Ashley: Okay.

    Rogers: One of the, I guess we can call it native extensions, which is essentially using Spinnaker as a library and writing Java and doing some spring magic to make it happen.

    Ashley: Mm-hmm.

    Rogers: So, if you were to use a Webhook or a Run Job stage, for that matter, you would have to write a little code. But the benefit of that, if it’s a Webhook the service, whatever service that you’re trying to connect to can be written in whatever language that you want, right? So, if it’s Go, if it’s Python, if it’s Ruby, if it’s—you know, whatever you wanna do, the service is running in the API as the contract that you expect.

    Ashley: Mm-hmm.

    Rogers: And so, with the Webhook stage, all you’re doing is making a call. You know, you could argue that writing a cURL command is writing code, but I think a lot of people are familiar enough with that, that it’s a pretty low overhead.

    Ashley: Mm-hmm.

    Rogers: So, if you can make a call in your terminal, you can make a call from Spinnaker with the Webhook stage.

    The Run Job stage is actually a little more interesting, because what it allows you to do is run a container as if it were a script that you might be running. So, a really common pattern for doing custom logic in CI systems specifically is, I have a container, the container has all my dependencies and it has a script that I need to run. And we just run it like a one shot, you know—once this script finishes, if it fails, the pipeline fails. If it succeeds, the pipeline continues. And then, like, the world is really open, the world is your oyster at that point, because you can write the script in whatever you are most comfortable writing, and then you can run that within Spinnaker.

    One of the more interesting things that we did with the Run Job stage is that you can actually output—you can output these special markers in your logs and Spinnaker will go grab those and inject them back into the pipeline so that you can use them in downstream stages. So, maybe your job is going off and collecting some information for you. You can actually use that in, like, say, your deploy stages or a downstream Run Job stage or Webhook stage as, like, arguments you pass down.

    Ashley: It’s almost used like the log is like an asynchronous message bus, to make it more complicated than it is, but a super simple version of that.

    Rogers: Yeah, yeah.

    Ashley: Some down stage version where you could look at what happened before it and take appropriate action.

    Rogers: Yeah. Yeah, and that’s really—you know, we heard, the motivation kind of behind that feature is, we heard a lot of users, like, you know, they wanna be able to extend Spinnaker, but they don’t wanna write Java code, you know? And containers, for the Run Job stage specifically, containers are obviously super prolific in our industry today. I’ve built my career off of Kubernetes and containers, so.

    Ashley: Mm-hmm.

    Rogers: Containers are kind of like the perfect medium for being able to do this, because you can contain everything that you need—[Laughter] you can contain everything that you need in this container, and then, because Spinnaker can run containers in Kubernetes, it’s kind of like a natural fit, right?

    Ashley: Mm-hmm.

    Rogers: We can kind of take advantage of the fact that it doesn’t matter what your code is, the interface that we need is a container and then the logs, and we can get all of that information out.

    Ashley: I’m curious—this is totally a side question, but did you learn anything in college about containers and that kind of thing in software architectures, or is this something that you really started in after you graduated?

    Rogers: I wish I had learned something in college about this. So, I actually, I graduated college in 2013, and got my start writing PHP and doing a lot more software development. But in the back of my head, I’d always been like—I am very interested in servers.

    Ashley: Mm-hmm.

    Rogers: So, writing the code was interesting to me, but it always seemed like the people with all the power and all the really, people that I looked up to, they were the guys, like, in a shell, deploying software, checking, like, “Is this server overloaded?” It was very much not DevOps, but it was something that really interested me.

    Ashley: Sure, yeah.

    Rogers: And I—you know, I kind of, I spent probably three years as a software developer without having done that. But once I transitioned kind of into this CI/CD space, that really gave me an opportunity to focus more on ops, while doing some development and, you know, we picked up Kubernetes at my last job, and that’s where it all kind of kicked off.

    So, I think we started on Kubernetes 1.4, and that’s really where I learned everything I know about containers and orchestration platforms and stuff like that.

    Ashley: Well, it puts you in a really great position, because you can speak to both sides, both audiences, if you will, you know, developers and operations folks.

    Rogers: Yeah.

    Ashley: And, you know, my experiences before kinda containers and all of this, all the DevOps environment, some of the best developers were engineers who really knew how to write code, but they knew the full stack down to the operating system and the network, right? They kinda knew enough of all of it. Maybe there were really good parts of it, too, besides writing code.

    Rogers: Mm-hmm.

    Ashley: There’s a lot of, I don’t know, power, if you will, in understanding that full stack and how it works.

    Rogers: Yeah. I found that the more you understand about the stack, the better positioned you are to take advantage of it, right, and to actually write better code. You can go back and probably look at the birth of DevOps and see this, but engineers that spend the majority of their time writing code and then throwing it over the wall are more likely to write less performant code, right? Code that is not aware—you know, we wanna write really abstract code, but at some level, you have to be aware of what type of environment you’re running in.

    So, if you’re running on a machine that has 128 cores and hundreds of Gigs of RAM, it doesn’t really matter how inefficient your algorithm is, because it’s gonna run fast. You know, it all depends on scale, but … I’ve found that engineers that understand kind of the cloud and the environments they’re running in and what the actual, you know, where are your network boundaries and what are your actual constraints are writing better code, they’re shipping more frequently, they’re iterating—like I said, they’re iterating more quickly and that all of that is super important.

    I think the other thing that I’ve found is that it’s a growing process, right? I kind of like to look at it as—well, at least in my personal experience, I’ve found that I have a hard time grasping, kind of, the next level of what I’m doing until I’ve fully understood the previous level. And I think that’s really true for a lot of people, right? Until you’ve got a solid grasp of programming, it’s kind of hard for you to imagine what it would be like to understand ops, right? To understand how your program is actually running on servers.

    Ashley: Yeah. I would describe it as kinda context. It helps to know when you understand the environment of what you’re operating in and what you’re doing, what you’re building is going to operate in. And vice versa, right? If you’re in operations or security now, with DevSecOps, understanding now how software is being built. So, it’s the great thing about the accessibility of resources around the cloud and contemporary software architectures and all of that.

    Rogers: Yeah.

    Ashley: Well, we’re running out of time. I wish we had more time to invest in this, but the great thing is that you have your talk coming up, so that’s where you’re really gonna get into more details about that. And Ethan, I thank you for being on the podcast.

    Rogers: Of course, man. Thanks for having me.

    Ashley: Absolutely. Ethan Rogers, staff software engineer and also speaker from Armory, he’s speaking at Spinnaker Summit 2019, which is on November 15th through the 19th in San Diego. Now, he has a topic that we were just talking about, “Making Spinnaker Your Own,” that’s on Saturday, November 16th, 3:45. He’s also on the panel about the state of Kubernetes V2 provider. That is on Friday, the day before—the 15th at 2:30.

    So, thanks to you, all of our listeners, for joining us for this episode. This is Mitch Ashley with devopsy.kinsta.cloud, and you’ve listened to another DevOps Chat podcast. Be careful out there.

    — Mitchell Ashley

  • DevOps Chats: The Future of Farming with Aerobotics

    DevOps Chats: The Future of Farming with Aerobotics

    Spinnaker Summit 2019 Preview: Never underestimate an entrepreneur looking to solve a real-world problem. Aerobotics is bringing drones, imagery, mapping, containers, Kubernetes, Spinnaker, data pipeline processing and machine learning to agriculture. Sounds interesting, right?

    Aerobotics uses aerial drones and sophisticated image processing to help growers manage yield, pest and disease in their farms through artificial intelligence.

    Speaking at the upcoming Spinnaker Summit in San Diego, Aerobotics’ Head of Software Nick Coles, shares with us how his company evolved from making drones to map orchards into a software company using raw image data to create high resolution, georeferenced maps of orchards. Aerobotics technology is comprised of a map engine and tree engine. It’s both a fascinating problem space and exciting application of cloud-native technologies, including Spinnaker.

    This episode of DevOps Chats features a preview of Nick’s talk, “The Future of Farming with Aerobotics.” Nick’s talk is on Sunday, November 17 12:30 PM, at Spinnaker Summit 2019 in San Diego.

     

    Transcript

    Mitch Ashley: Hi everyone. This is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today I’m joined by Nick Coles, head of software at Aerobotics. This is a special edition of DevOps Chat, because we’re talking about talks at the Spinnaker Summit 2019, which is in San Diego this year.

    Our next topic is the future of farming with Aerobotics, aka drones. So you’re gonna be really interested to hear about this. This talk is on Sunday, November 17th at 12:30 p.m. Nick, welcome to DevOps Chat.

    Nick Coles: Hey Mitch. Thanks for having me. Really excited to be on this podcast.

    Ashley: Happy to have you here. Tell us a little bit about yourself. Tell us about what you do at Aerobotics and a little bit about the company.

    Coles: Cool, so I guess just from the very beginning, born and raised in Cape Town, South Africa, as you can probably hear by the slightly strange accent. [Laughs]

    Ashley: Detected. We detected a slight accent there. [Laughter]

    Coles: Yeah. And but yeah, born and raised Cape Town, South Africa. Studied at the University of Cape Town, where I did a electromechanical engineering degree, which was kind of, you know, embedded systems, microcontrollers, processes and stuff. And that’s kind of why I got interested in drones and robotics. And, yeah, just over four years ago joined Aerobotics and sort of–at the time it had just been funded, so there were essentially two people, and I kind of joined in a software engineering role, not really knowing what I was doing.

    Ashley: [Laughs]

    Coles: And yeah, kind of been there ever since heading up the software engineering team. Which obviously a huge component of it has been DevOps and back-end processes. And so that’s kind of why I sort of have been exposed to Spinnaker and, yeah, kind of what led me to this today.

    Ashley: Excellent. So it’s the true startup experience, which means your job is you do whatever needs to get done, and you kind of evolve into your current role, it sounds like.

    Coles: Yeah, pretty much. I mean I joined–back in about four years ago when we joined it’s basically we were a company that was building drones, building the hardware, sort of soldering the motors to the propellers. And essentially drones kind of transformed throughout our journey, and we ended up having to move out of the hardware space and peel into the software space, because companies like DJI came up with these commoditized perfect drones. And it was quite hard to compete with them when you’re, what, five engineers in a garage soldering drones on a piece of [Crosstalk]–

    Ashley: There are a lot of drones available and you can differentiate in the software. So I read from the description of your talk that Aerobotics uses aerial drone imagery to help farmers, growers, manage their yield, pest, disease, that kind of stuff on farms, using some AI. Is that a problem that you started initially with, or did the company evolve to that kind of problem? How did you get started, and the company get started around that area, specifically?

    Coles: So, I mean, the company has actually had quite an interesting sort of journey in that regard. So the idea in the beginning was we are drones. We build drones. We use drones for agriculture. We use drones for security, game counting, mining. We tried to essentially just throw drones at any sort of problem.

    And sort of as we progressed as a company and more and more players sort of entering the field, and we sort of started honing in on our value proposition, and started specifically focusing on tree crops within the agriculture space because it’s such a high value. And so it actually then makes sense for the cost for specific tree crops.

    But originally we weren’t using any AI. We were kind of just trying to use raw sort of computer vision and object detection. The data that we were seeing is so vast, and case-by-case is so unique, that we kept having to change the parameters, slightly tweak things. And so the only sort of long-term solution could’ve been machine learning. And so that’s kind of what we embarked on about two to three years ago and sort of been, yeah, what we’ve been using ever since to try to essentially generalize the imagery we were receiving and sort of try to replicate the human behavior to the best of its ability.

    Ashley: Okay, excellent. So I know your software staff appreciate that about the company in kind of the broadened space. I know your software stack is a combination of, what, Kubernetes, of course you’re using Spinnaker, I think Circle CI for your CI/CD. Tell us a little bit about how you got involved with Spinnaker initially. Was that something that you chose or kind of evolved there kind of by accident? How did you happen down that path?

    Coles: So, I mean, it is–it did happen by complete accident. Traditionally when we were processing all this data we were just spinning up EC2 instances. And so we were essentially just sending SQS messages and spinning up instances. And instances would essentially be able to sort of, I don’t know, kind of retrieve these messages and then figure out what sort of processing to do. You know, it was an absolutely terrible solution for instance _____ a drone with a code. We’d have to SSH into the instance, patch it, figure out what was going on.

    Ashley: Oh boy.

    Coles: At this time also it wasn’t–like, DACA hadn’t really sort of like exploded. And or maybe it had in sort of like San Francisco and everything. But in Cape Town we weren’t really exposed to all these sort of advanced kind of tech solutions. So we had quite an archaic little approach going on, which was working, I guess, with R Scale.

    And then in the beginning of last year we got accepted into Google Launchpad, which was really exciting. Four of us went across to San Francisco, and we basically went on this launchpad program where they kind of helped us all the way from technical to business. And it really was inspiring to kind of go to San Francisco for the first time and sort of see like just the scale of things and just like the intensity of the tech companies there and really what they were doing.

    And we kind of got put in touch with a couple of mentors. And one of them, based on sort of what the problems we were trying to solve were and the problems that we were facing, recommended Spinnaker. And at that time we were kind of on AWS’ stack. And so we kind of then got incentivized to sort of try, set up a system on Google. And you know, we had like Spinnaker set up, Google’s Kubernetes’ engine, and kind of we just took it from there.

    And just it was an interesting journey, ‘cause there was quite a lot of iteration within like Spinnaker itself and in terms of just how the community’s progressed over the last sort of two years, but also just sort of like where we were at in terms of not really understanding how Spinnaker worked, the containers spiraling out of control, realizing we needed to stabilize our Redis pods. Like our Redis pod basically caused so many problems early on.

    But yeah, so that kind of was the journey in terms of how we got involved in Spinnaker. So it was basically just by chance somebody mentioned it, and we kind of were like, wow, this shows the problem we’ve been having.

    Ashley: Like a great solution.

    Coles: And we kind of just ended up, yeah, just going with it.

    Ashley: Fantastic. So from what I understand is the drones that–just giving our audience a little bit of context–the drones that are flying around the orchards, for example, are collecting all of this data. And that goes up into some data processing in the cloud, correct? And it does, I guess, splits into two areas. One is a map engine that creates–takes that raw imagery and creates multilayered geomaps of the orchard. And then the tree–a tree engine, which also then looks at specifically where exactly are the trees. And that’s how you do your processing. Is that a correct description of how this works at a high level?

    Coles: So it is basically quite a good description. So just to maybe give like slightly more detail on like the map engine side, we are–I don’t know if you mentioned–but we are using multispectral cameras attached to the drones. And from that data we can get information that isn’t necessarily visible to the human eye. And that kind of gives us all a health and powerful readings from the plots.

    Ashley: Is that like using infrared kind of things, or what?

    Coles: Exactly. Exactly. Infrared sensors. And so, yeah, so the map engine side is essentially like the big data processing side of things where we’re taking thousands of thousands of images and stitching them into these high resolution maps of individual orchards. I mean, just to give sort of some context on kind of what the data is there, we generally produce about like these five gigabyte high resolution images, which is of a specific orchard. And the kind of process around getting the raw imagery to that five gigabyte sort of image, we use about 64 gig RAM and 16 CPU instances. So that’s kind of just like the scale of what you’re running in. It can take anywhere between two hours to 24 hours to form that stitching process.

    And then moving onto the tree engine, this is sort of where our machine learning models are running in production. And that’s essentially taking these high resolution maps and segmenting the image to get individual trees from that image. And from that we can then calculate additional metrics, such as the height, the health, and the area of the individual trees.

    And one thing that, I guess, we left out because we are still developing it, is we now have a fruit engine, which is essentially now like kind of our new product focus area where we’re going from the drone, which is kind of taking a high level of the orchard, and now we’re going right up to the individual trees. And from those trees we’re starting to detect fruits and sizes from the fruits. And the goal here is to give the farmers an accurate sort of size distribution of the fruit in the orchard so they can plan accordingly when it comes to harvest time.

    Ashley: They’re all–it’s all about yield, of course, and quality of the yield as well. And by the way, I come from–I’m not a farmer, but I grew up in the middle of Nebraska in the U.S., so I’m around a lot of farmers. So I have some idea of what you’re talking about, at least an appreciation for it.

    Well how did–tell us a little bit about why you chose to do this talk at Spinnaker Summit and some of the things that you want to communicate beyond kind of the problem set or the solution that you’ve come up with.

    Coles: Okay, so one of the reasons why I thought it would be quite cool to speak at the summit was, I guess, just based off like chatting to some of the Spinnaker guys on the Slack channels and everything, it feels like we’re using it in quite like a unique sort of way. I mean, we’re using it very much to manage essentially our data pipelines. And I guess we’re almost using it in a similar way to how someone may use something like Apache Airflow or something.

    And I thought that the combination of the uniqueness of how we’re sort of using it, and sort of just to kind of explain how it has completely changed the way we’ve been able to develop and iterate super quickly, I thought it’d be worth kind of sharing that with the community and sort of seeing how other people were using it in comparison. And so I guess, yeah, that was kind of just the main thing, just the type of data that we’re sending through the system is quite different. It’s quite unique, in comparison to the rest of the world.

    You know, we process these large images which come up with a–which have a whole bunch of different problems with them. You know, we have to ensure that our containers are responsible, they’re not using too much RAM, they’re not spiking in terms of, you know, being out of memory because they’re loading an image all at once. And there’s just been a lot of interesting things that we’ve had to sort of develop to kind of handle our case. So I thought it’d be cool to share that with the community and take it from there.

    Ashley: You know, I think it’s super relevant, and not everyone may have an agricultural problem domain, but with a massive quantity of data that we’re collecting through sensors, whether it be drones or IoT and home or IoT and the business or any device, any systems that are collecting all this data, one of the real challenges is how do you process this much information? How do you combine it together? How do you take different sources like your map engine versus the tree engine, look at it, and then do that analysis and produce valuable, interesting results?

    I think a lot of businesses, whether they be startups or enterprises, have those kinds of projects and challenges, or at least they’re thinking about it, if they’re not already doing that. And I bet you’ve got some valuable lessons that you’ve learned over time about why you set up the pipelines the way you did or how to handle some of the production challenges.

    Coles: Yeah, you know, it’s also we had an interesting space, also, because we’ve learned so much over the course of the last two years. Like when you’re doing things in production, you also very quickly start figuring out what it should be. And so there’s even more and more of an end-goal, if that makes sense.

    Ashley: Mm-hmm.

    Coles: So like as a small example, kind of, you know, we’ve set up all of our Spinnaker pipelines, and it’s been amazing. And the devs have been able to iterate really fast. And now we’re starting to see that we want to maybe start introducing some stability to our pipelines. And so to achieve that you want to start setting them up as infrastructure code. And like just starting to build that whole process into the development cycle is obviously quite challenging and takes time. But it’s allowed us to sort of think forward and continually improve on what we’ve been doing.

    So you always feel like you have kind of found the best solution at a point in time. You’re like, wow, this is absolutely incredible, and it’s going so well. And then six months from then, you’re kind of like, wow, it could be so much better. There’s so much more we can still keep doing to improve again. So I guess that’s what’s been quite an awesome part about using Spinnaker. It’s like at every sort of milestones it’s kind of like you can keep iterating and making it better and using it more efficiently.

    Ashley: That’s excellent. When you use a tool like Spinnaker open source to a software like that, that you feel like you’re just getting started sometimes. You thought you knew what you could do with it, and you find out there’s eight or ten different things that it evolves to from where you started.

    Coles: A hundred percent.

    Ashley: Well good. I wish you the best of luck on your talk. You know, I think you may get some folks attracted just to come hear about the drones. I hope you’re gonna have some pictures of maybe some of the orchards and the maps and some of the output that you do, is that correct, just to give folks an idea of kind of how this works?

    Coles: Yeah, we’ll definitely, I think, do a very brief sort of couple minutes on the product and just kind of explaining what we’re trying to do, and for sure. ‘Cause I guess it’s always nice to sort of give context of what Spinnaker is actually producing at the end of the day.

    Ashley: Mm-hmm. Well very good. Well it’s been great talking with you, and I wish you the best with the talk. And hopefully you get a lot of folks coming that are excited to hear about what you’re doing. Is this the first talk that you’ve given at a conference like this, or you’ve done that before?

    Coles: No, this would be the first talk, so I’m quite excited for that.

    Ashley: Really? Excellent. Well you know it’s–there’s lots of ways to give back to an open source community, and of course, writing code is certainly one of ‘em. But most people don’t write code. And I think one of the great ways of giving back is doing exactly what you’re doing, is going and talking, not just in the community and online forums and Slack channels or whatever they are–that, too–but participating in talks like this.

    Because it–I’ll predict you’ll have a lot of folks come up to you after the talk and want to know more or talk about what they’re doing and tell you about their idea or ask your advice on something. So it’s a great way that advances the software and the use of it, because you’re doing things that probably other people never expected or predicted. So congrats. That’s awesome you’re doing it.

    Coles: Thank you. Yeah, I guess that’s one of the things I’m most looking forward to is just sort of meeting a whole bunch of new people and kind of, I guess, like we learn so much, like especially kind of being in South Africa, not kind of in the tech hub of the world, I would guess, is when you go there and you start speaking to some people who are kind of leading the way in terms of the industry, it’s so valuable just to kind of hear what they have to say and what they’re using and how they’re solving problems. Because it’s so relevant and it can translate so nicely back home. And so, yeah, it’s always so much value in just meeting the people and just chatting through things.

    Ashley: Well excellent. Nick, thank you for joining the podcast today.

    Coles: Thanks Mitch. Will you be at the summit in a month’s time?

    Ashley: I don’t think I’m gonna be able to be there. I’ve had several invitations [Laughs]. If I can I will be.

    Coles: [Laughs] Okay.

    Ashley: But I’m like back-to-back for the next month going to conferences. So we’ll see. Hopefully. If I make it I will definitely look you up.

    So I’d like to thank Nick Coles, head of software for Aerobotics, joining us today. Again, he’s speaking at the Spinnaker Summit 2019, the dates are November 15th to the 19th in San Diego. By the way, Nick, you’ll enjoy San Diego, too. It’s a great city like San Francisco is. His topic is the future of farming with Aerobotics, aka drones. And on Sunday November 17th at 12:30 p.m. So please check out his talk.

    And thanks to all you for joining us today on this podcast and listening to Nick and I chat about his talk. This is Mitch Ashley with devopsy.kinsta.cloud, and you’ve listened to another DevOps Chat. Be careful out there.

    — Mitchell Ashley

  • DevOps Chats: Continuous Delivery at Airbnb

    DevOps Chats: Continuous Delivery at Airbnb

    Spinnaker Summit 2019 Preview: Airbnb is rapidly moving from a monolith Ruby on Rails application to a distributed SOA/Kubernetes architecture in Kubernetes. The new architecture uses self-service codified pipelines and easy webhook integrations, scale adoption and collaboration across the company. Even though continuous integration isn’t new to Airbnb, every team now needs to be able to scale CI across 100’s of containerized services in AWS EC2.

    Software Engineer Brian Wolfe co-led the decision to move to Spinnaker and build in more automation. At one year into the project, Airbnb has 40 services in production with many more to follow.

    This episode of DevOps Chats features a preview of Brian’s talk, “Scaling a Migration to Continuous Delivery (Airbnb)”. Brian’s talk is on Saturday, November 16 11:00 am, at Spinnaker Summit 2019 in San Diego.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi everyone. This is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today I’m joined by Brian Wolfe, who is a software engineer at Airbnb. Our topic is a talk that he’s gonna be delivering at Spinnaker Summit 2019 in San Diego. The topic of that talk is scaling of migration to continuous delivery. Continuous delivery is a hot topic right now. That’ll be happening on Saturday, November 16th at 11:00 a.m.

    Brian, welcome to DevOps Chat.

    Brian Wolfe: Thanks so much, Mitch. It’s really an honor to be talking with you today, so.

    Ashley: Well I’m more honored to have you on. I appreciate you taking the time. Tell us a little bit about you. Introduce yourself, tell us what you do and a little bit about–I think we know Airbnb–but tell us about what part in Airbnb that you work in.

    Wolfe: Cool. Yeah, so I assume most people who listen to the podcast know what Airbnb is, but we’re a worldwide community of hosts and guests. We bring people together to have really local experiences. I’m gonna skip the rest of that little spiel.

    Ashley: [Laughs]

    Wolfe: But I’ve been at Airbnb for about three-and-a-half years, and most of that time I’ve been kind of on the operational tooling side, and so a lot of that was observability works, so looking at your metrics and traces and logs, and then performance works, so like figure out how to make Airbnb–monitor Airbnb performance, make it faster and that sort of thing. And so but for the last year I’ve been the tech lead on our continuous delivery team, which is kind of an ambitiously named team. We’re not there yet. But the goal of this team was to really up level how we deliver software at Airbnb.

    And so my experience is mostly looking at how we’re making the transition from a monolith which was written in Ruby on Rails to an SOA over the last two years and kind of continuing on into 2020. And part of that is how do we scale how we deliver our software. It used to be a more core team who understood how to scale our monolith. And now every team kind of needs to know how to deliver that software, how to scale it and how to keep it working.

    Ashley: Now, is the move to an SOA, and so we mean service-oriented architecture, correct? Is that what you’re referring to?

    Wolfe: Yeah.

    Ashley: Is the move to that kind of architecture what prompted you to–or order Airbnb to–invest so heavily in this automated continuous delivery? Or was that kind of happening in parallel? Gives a little bit of idea of context of how this kind of came together.

    Wolfe: Yeah, so it definitely happened because of the move to SOA. And if you look at what our processes were before, it was very consolidated. Most people were applying the same code base. And so you kind of could have practices that worked for that one code base. But as we moved to SOA we now have hundreds of services that are being deployed, and you need to be able to scale those best practices across all of those services. And so that requires that you add these automation pieces so that humans can make fewer mistakes.

    Ashley: That makes a lot of sense. I imagine you have a kind of continuous integration happening at the front end of this, right, leading into your continuous delivery?

    Wolfe: Absolutely. So we’ve had continuous integration for a long time. And so all of our unit tests and that sort of thing and all of our builds are happening in continuous integration. That’s been the case for at least three or four years. And we have fairly mature tooling around that. And we actually did a migration to our new platform there fairly recently that containerizes all of our builds and all of our MCI so it kind of runs in a more uniform manner.

    Ashley: Mm-hmm. Is that moving to Kubernetes? Is that the change you’re referencing or something else?

    Wolfe: So even stuff that’s not running in Kubernetes, so as you can imagine Airbnb is about 10 years old, and so we have a lot of technical history. And so some of that stuff is gonna be running on bare EC2 instances, and some of that is running inside Kubernetes at this point. And so along with the move to SOA we’re also migrating to Kubernetes. And that migration is progressing rapidly. But there’s still gonna be a lot of stuff that has to be built on EC2 and running on EC2. But the builds themselves can happen inside containers.

    Ashley: I know we don’t think of Airbnb as something that’s built 10 years ago and evolved over time, right? We think of it as [Crosstalk] – [Laughs].

    Wolfe: Yeah, yeah. It has a surprising amount of history for such a ____ company, you know?

    Ashley: Well it’s an interesting, you know, it’s an interesting situation that you are involved in, because you’re not talking about a monolith application that was built 30 years ago or 20 years ago. It’s something fairly recent, Ruby on Rails, you know? So it’s still a contemporary programming language, something that everybody is familiar with. But even you have to go through your own kind of architectural and now continuous delivery evolution of that technology.

    Wolfe: Absolutely. I think it’s kind of surprising that even at a company that’s ten years old you have what we consider legacy parts of our stack. I think that happens quickly with a field that evolves as rapidly as ours. And if you look at how continuous delivery has evolved in the last ten years, it’s been a pretty big shift.

    One of the big things we’ve seen is, you know, we had a really good solution for deployment if you look in 2013-2014 time frame with our system and deploy board. But if you look at it now, it looks a lot like a really good CI system, like a really good continuous integration system. And the parts around actually having an organized pipeline that automates the deployment out, we don’t really have that in house yet. And so that’s really why we’ve been adopting Spinnaker as a solution.

    Ashley: Interesting. Well tell us a little bit about were you involved in the choice of bringing in Spinnaker? Was that happening while or before you joined Airbnb?

    Wolfe: Mm-hmm. So the decision for adopting Spinnaker happened last year. And so it was a discussion between myself, my manager and then Jing Jing and Jens, who are on the team. And it was largely about do we want to keep investing in deploy board, which was our existing solution. It was deeply integrated with all of Airbnb stack. Had a lot of stuff in it for how we deploy MonoRail, which is our big Ruby on Rails application.

    Now, do we want to build automation into that, or do we want to bring in something new? And so we did a big research project to look at what are all the available options out there, and what would those give us, and how could we make those work at Airbnb. And at the end of that we decided that we should not invest more in our internal solution and should instead bring in Spinnaker and customize that to kind of encode Airbnb opinions into that via extensions and then use that as our platform for deployment moving forward.

    So we’re still early days in that. We’re about one year in. We have about 40 services onboarded right now. These will be all production services, some of the really critical services out of Airbnb. But considering we have thousands of services at Airbnb, there’s some work to do to kind of make it the standard across.

    Ashley: Mm-hmm. Excellent. Well interesting. Tell us some more about I know in the description of your talk, you talk about that you’re gonna discuss how self-service codified pipelines and easy web hook integrations help you scale adoption and collaboration across the company. Tell us some more about what that is.

    Wolfe: Yeah, so one of the really harder parts about working at a company that’s growing as fast as Airbnb is its really hard to match the needs of everyone. And so you want to make everything self-service if you can so that, you know, I’m creating a standard pipeline that you know I deploy one thing, I do some AB comparison, I roll it out to my next environment, I do another maybe canary analysis rollout, and so on.

    So I can come up with standard components that people can use. But then you have stuff that people have built over time. And so our research team has regression detection that is really specific to them. And they think it provides really high signal if some functionality regression has occurred because you have things like search ranking and the actual layout of the results in the response package, and you need to make sure those things don’t change if you don’t intend them to change.

    And so they built a service to actually do that regression detection, and they want to integrate that with their deployed pipeline. And they want that to just be something that automatically happens every time that they run a deploy. You deploy to staging, you call this service with some arguments, you wait for that regression test to pass, and then if it fails you present some custom UI that says, “Okay, this is what failed. This is why. And this is how you can go learn about it some more.” And you move onto the next stage or you fail the pipeline and figure out what’s going on.

    What we wanted to do is make it really easy for teams to plug in services like this, ‘cause they do provide the most value. The way we’re doing that is by integrating, actually, with our interface definition language that we use internally. And so we actually extend Spinnaker to just provide a little web hook stage that people can reuse and call out to their services. But then they get a nice user interface on top of that. And so they can kind of provide just a base level of functionality within Spinnaker. And it’s kind of a minimal amount of work for us to onboard a new type of stage. Does that kind of make sense?

    Ashley: Yeah. I was just gonna ask you about that. Sometimes it can be a real challenge to introduce such a fundamental tool, like this is fundamental to your software development and delivery process and what kind of things that you can do to minimize the disruption, lower the bar in the area of entry for the teams. And a lot of it depends, too, on how much your organization does software process similarly, or is it very decentralized and everybody kind of does their own thing. It’s such an application if it’s been in a monolith kind of state. I imagine there’s a lot more similarities in how people work at Airbnb. But you’ve gotta still figure out how do you make this easy, right?

    Wolfe: Yeah. So I think we’re lucky in the sense that right now there’s really one deployment tool that people use, and that’s deploy board. And now we’ve introduced a second deployment tool. And our strategy here is actually we make the experience when you’re first onboarding actually look a lot like deploy board.

    And so Spinnaker lets you customize the user interface. And so we actually have a panel within Spinnaker that shows a view that actually looks really similar to the view that you get in our old tool. But then it provides all this power under the hood when you actually start your deploy. And so that’s been pretty valuable for us for onboarding because it doesn’t look that dissimilar, I would say a little rough around the edges, to be honest still.

    It’s, you know, we spent a lot of time optimizing that flow for deploy board. People get a little bit confused sometimes. But just having that initial experience look similar has made onboarding a lot easier. When we first proposed using Spinnaker, people were like, “I can’t figure out what’s going on in the UI. I don’t know what I’m doing here. What are you guys thinking?” But just by providing that similar viewpoint people are like, “Oh, I know what to do. I’ll just click on this button, which looks the same as we saw before, and that should start my deploy doing the right thing.”

    Ashley: You know, never underestimate the value of an easy-to-use user interface or process or whatever.

    Wolfe: Especially just having that familiarity, I think, it’s–Jens on my team has been a continual advocate for really pushing on, like, make this easy to transition from our old tool to our new tool. And I would say that there are some usability gaps that we’ve had to come across. But they’re getting better, I would say.

    Ashley: How far along in the rollout process of this are you? Are you in the still kind of beginning third of it, working with some of the early teams that are adopting? Are you now moving into more broader adoption of it? Where are you kind of in that process, that maturation process?

    Wolfe: Yeah, so we’re kind of in the–we’re calling it a beta stage. So in alpha stage we were really hitting some early adopters, people who were willing to take some risks. Now we’re onboarding kind of the big customers, the ones who are pretty demanding and making sure that this actually works for them. And so what we’re doing is we’re measuring how much do we prevent, and so how many rollbacks are we preventing, how many incidents are we preventing, and making sure that Spinnaker, as the platform we’re currently providing, is actually delivering business value.

    We’re limiting our onboarding throughout 2019 to about 50 teams. And at the end of 2019 we’ll have a really good idea of what works and what doesn’t. And in 2020 we’re going to open up the adoption to the full company. Kind of at the same time we’re going through a lot of scaling exercises to make sure that we will actually be able to scale to the full company.

    Ashley: And give us an idea of the 50 teams, what are usually the size of those teams? What do they range?

    Wolfe: Yeah, so the smallest teams will be probably about six or seven developers deploying a service. The largest team that we have onboarded has about 45 developers. And then we’re aiming for one of our front-end services, which is more monolithic, and so that’ll have several hundred developers working on it. But then there’s kind of an operational expert team that manages what that deployment looks like.

    Ashley: Mm-hmm. Well that’s a pretty good range in different sizes of teams. So I’m sure–and parts of the application–so I assume they bring different challenges with them. How do you measure success from your decision of both going down the path of implementing continuous delivery and Spinnaker? You talked about business value. You also talked about preventing how many rollbacks and incidents that also happened. But how do you demonstrate to the people that said we’re gonna spend this money having you go implement this, and tell them that it was worth it?

    Wolfe: Totally. That is a great question. And so one of the big motivators for us was actually for regression prevention. There are kind of two aspects for CD. One of them is increased productivity for developers. And then another one is having more vigorous processes for rollout. And we were definitely focused on the vigorous processes for rollout and having automated canary analysis by default for every service. And so that’s kind of the route that we’re pursuing, really, in 2019 in terms of proving the business value.

    Ashley: Mm-hmm.

    Wolfe: In 2020 we’re gonna have more metrics and more ability to say something around how productive we have made our engineers, or at least that we haven’t made the story harder. But right now it’s proven a little bit harder to quantify that. So we get people saying, “Yeah, we love it. We love that we can kind of ignore the deploy process now.” But aside from having a CSAT style score where you’re just saying, you know, rate how productive you are on a scale of one to ten, it’s been pretty hard to quantify the productivity gains we’ve seen with Spinnaker.

    Ashley: Well, that is often a challenge, right, for us in the software world in implementing technology, and as you’ve talked about some of the regression testing issues, maybe even incidents in the old process compared to the new, hopefully there are some metrics you can also use from the old methods to the new and to give you some places to start, at least, to benchmark those things.

    Wolfe: Certainly for regression detection we have. We have strong evidence that automating this pipeline–and so what I really think of Spinnaker as is a way to automate your run book. Just the fact that it is automated now has reduced the number of regressions that have gotten out to production. We have very strong evidence of that at this point.

    Ashley: Well and hopefully you can sing some of the praises, the good things of what the teams have learned so far with folks that are adopting it as you take on some of the bigger challenges like the larger teams do.

    Well now, I appreciate you coming on the podcast. It’s been fascinating learning about what you’re gonna talk about. And I wish you the best in your presentation at Spinnaker Summit.

    Wolfe: Thanks so much, Mitch. This has been a pleasure. And I hope to meet you at some point soon.

    Ashley: That would be fantastic. I would enjoy that very much. I’d like to thank my guest today, Brian Wolfe, software engineer at Airbnb for sharing with us his experience with Spinnaker and how they are migrating to a continuous delivery model and scaling that up in the organization. Brian’s gonna be talking at the Spinnaker Summit 2019, which is November 15th through the 19th in San Diego. His talk is on Saturday the 16th at 11:00 a.m. And I believe you’re gonna be doing that with Jens Vanderhaeghe? Is that correct?

    Wolfe: Yes, that is correct.

    Ashley: Awesome. Good to have you both up there. Well thank you everyone for joining us today. I’d like to thank our listeners for taking the time to check out the podcast and hear about this talk. This is Mitch Ashley with devopsy.kinsta.cloud. And you have listened to another DevOps Chat with my guest. Be careful out there.

    — Mitchell Ashley

  • DevOps Chat: From 0 to 1,000 Deploys Per Month with Redbox

    DevOps Chat: From 0 to 1,000 Deploys Per Month with Redbox

    Many organizations want to implement the latest DevOps/SRE practices, but many struggle with transforming the existing processes over to new streamlined processes. How does an organization transform from zero automated deploys to 100s/1000s a month? Most importantly, how do you show the value of Spinnaker to your business as a whole?

    November’s Spinnaker Summit 2019 in San Diego allows us an opportunity to hear and learn from DevOps engineers, developers and practitioners using Spinnaker open source software. Joel Vasallo, manager of Cloud DevOps at Redbox, shares a preview of his talk where he shares his experience about using Spinnaker open source to increase the number of deploys into production. Joel found Spinnaker so useful because it is very extensible, covers AWS well, strong support for Kubernetes, different deployment types and has the automation to create the software delivery platform that fits the need.

    Joel’s talk, Transforming Software Delivery using Spinnaker, is on November 17, 12:30 pm PT at Spinnaker Summit 2019 in San Diego.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Joel Vasallo who is manager of Cloud DevOps at Redbox. We know Redbox is the movies, the kiosk, the online service—great.

    Our topic today is going from zero to thousands of deploys a month. Boy, if you’re not doing a thousand a month already, that sounds like a great place to be. This is actually a preview of a talk Joel is doing at the Spinnaker Summit on “Transforming Software Delivery Using Spinnaker.” It’s 12:30 p.m. Sunday, November 17th in San Diego. Joe, welcome to DevOps Chat.

    Joel Vasallo: Hey, thanks so much for having me, Mitch. I’m glad to be here.

    Ashley: Glad to have you here. Well, tell us a little bit about yourself, what you do at Redbox and also, hopefully people know Redbox, but just a bit about Redbox.

    Vasallo: Yeah, no, my name is Joel Vasallo, I’m the Manager of Cloud DevOps at Redbox involved with software delivery to the cloud. In my spare time, I’m a GDG organizer, so a Google Developers Group here in Chicago. But about Redbox, I mean, you said a lot of people know what Redbox is. We’re the kiosk that’s usually found in your grocery store. So, we do DVDs and Blu-rays, but we also, a little known fact is, we do have an on demand platform, so video on demand, and that’s kind of our new thing that we’re doing right now. And there’s actually a lot of motivation around building fast deployments, and that’s kind of gonna be my drive for the talk.

    Ashley: Awesome. Now, just a question for you for cloud DevOps—does cloud actually include software that’s being delivered or information being delivered out to those kiosks, too, or is this really for the on demand platform in the cloud that you—

    Vasallo: Yeah, there’s some aspects of it as well. I mean, there’s a lot of APIs in use across the company, so there are some that do kind of traverse the cloud in that regard—yeah.

    Ashley: The only thing better than APIs are common APIs, right?

    Vasallo: [Laughter] Exactly.

    Ashley: Reuse.

    Vasallo: Yeah, especially if you have a solid delivery platform—of course.

    Ashley: Absolutely delivery platforms. Okay, so, let’s jump into it. We’re giving a preview of your talk, so there’ll be a lot more content, of course, in your actual talk. Tell us a little bit about, if you’re already achieved this thousands of deploys, you’ve been at this for a while. Tell us a little bit about how you got started on this journey.

    Vasallo: Yeah. You know, to preface a bit, I mean, I’ve been in the Spinnaker community since, like, late 2015, when it was still kind of an early release out of the open—in the open source community. And at a previous role at Gogo, I used to work at Gogo and we implemented Spinnaker basically from day one. And prior to that, we were using some other open source—Netflix open source tooling as well.

    But in that journey, we were going from a traditional monolithic, you know, to a microservices architecture, and the goal was to essentially build a way to deploy to the cloud as fast as possible, right? And just taking and building on those practices, you know, applying all the things, the same—the 12 factor concepts, immutable infrastructure, routing, all that stuff.

    It kind of transformed into some open source tooling that we’ve written in my history, and there’s also some practices when using Spinnaker to make such a big change in an organization, right? I mean, the whole topic of DevOps in an organization, there’s so many great talks out there, right?

    Ashley: Mm-hmm.

    Vasallo: And that’s definitely one of the—I guess, arguably, the hardest bit is, “Well, what does the DevOps mean for you?” Right? That’s the typical phrase you hear when people say, “Well, we’re DevOps.” “Well, what does that mean?” Right?

    Ashley: Well, now, you’re talking, obviously, at the Spinnaker Summit, and it sounds like you’re pretty committed to the product. This isn’t a podcast about that specifically, but you sound pretty dedicated to it. Why Spinnaker?

    Vasallo: You know, it’s—the core fact is, it kind of extends a little bit past just deployments. It has—it’s really, the biggest thing is, it’s so extensible. Right out of the box, you’re just coming out swinging. The fact that Netflix, Google, Microsoft, a lot of big names—Amazon as well, how could I forget?—are contributing and helping build this product up, you have some pretty big names out there, right? And why not lean on the shoulders of giants. They’ve seen, I would argue, probably one of the bigger challenges in terms of software delivery, especially at an in demand space such as, you know, for example, Netflix, right?

    Ashley: Mm-hmm.

    Vasallo: And building on that success, you know, you basically get those batteries included, right? You get some battle tested software, and I think that’s one of the biggest things.

    But in addition to that, I mean, it’s extensible. It covers Amazon very well, it covers Kubernetes, it has support for various deployment types and various customizations as well, some of which are customizations you need to build, and some customizations are actually built into the platform. And the best part, it’s open source, so that’s my favorite thing.

    Ashley: One of the key words I didn’t mention that I know your talk’s gonna be about is automated deploys, right? You’re not doing hundreds of thousands with a lot of manual process. Talk a little bit about the key of automation and you kind of assume a development team is gonna be all about—yeah, let’s automate this and make our lives easier, but maybe it’s not that easy to get from point A to B where you do have that automated. Talk about that part of the journey.

    Vasallo: Yeah, no, I agree with you. And I mean, it’s an unfortunate thing in the landscape right now where a lot of companies are rushing into the DevOps landscape, right, and they just say Jenkins, right—Travis CI, GitLab, whatever. And, you know, you’re essentially flooding tools on the market, right? And it’s very tough.

    But having a lot of tools doesn’t mean you’re DevOps, right? There’s some fundamental business changes that have to occur in order to achieve that delivery that we’re talking about, right?

    Ashley: Mm-hmm, yeah.

    Vasallo: So, with that, it’s kind of taking—how do I call it?—the little infinity loop of DevOps, right? Where that’s great and, you know, the plan, code, build, release—you know, that little cycle, it’s great, but you can’t use that as a guiding principle. There’s not enough detail, right? And then you can have the extreme on the other side, where it’s like the Candyland, I like to call it, where it’s like the Candyland of ITIL process where you’re just essentially passing Go and then passing down a ladder, if conditions. That’s too complex, and that’s hard to explain. So, there has to be something kind of in between.

    And that’s kind of what I’m gonna be focusing on in my talk and really asking the questions of how do you lay out an effective delivery process? What are some steps—what are the steps at your organization to get changes out the door? Because it’s a different answer for every company, right? There’s no one truth out there, unfortunate as it sounds.

    Ashley: Mm-hmm. Well, and everybody’s environment is unique. Maybe some of the problems are the same, but there’s also, you know, what is unique about your organization, whether it’s the business technology or just people, the team you’re working with has strengths in certain areas.

    Vasallo: Yeah, of course. And that’s something that I kind of build on in the talk is about building minimum viable products, right? And the argument is, many minimum viable pipelines, right? Really laying out, end to end, what needs to be done to get software delivery to occur. And the best way to start is really to look at what’s kinda currently happening, right? Lay out, you know, change has to come in, goes through phase two, phase three, phase four, and then ultimately phase five—profit, right?

    Ashley: Mm-hmm.

    Vasallo: Well, what are those phases and how much investment should be done in each spot, right? You can spend your time automating, I don’t know, some aspects of your pipeline, but what’s the return on investment, right? Would it have been better served in, for example, automating quality versus automating security scans, right? What’s gonna be the most immediate value in shaving time and getting you back from, you know, these long, drawn out pipelines to potentially fast and efficient ones, right?

    Ashley: It’s interesting. What was sort of the starting problem that took you down this path? Was the business asking you, “Hey, we need stuff faster, faster, faster”?

    Vasallo: Yeah.

    Ashley: Or was it just your own, “We don’t have resources, so let’s figure out how to automate it”? I mean, what kinda got you going on this?

    Vasallo: Yeah. So, a little bit about myself, I joined Redbox back in 2018 with the mission of building a software delivery pipeline to get software—me and my team to get software into the cloud. And with that, there’s also the transformation aspect of, “Well, what is the cloud? How do we integrate with the cloud?” And a lot of questions kinda came up.

    Fortunately, we took a big investment in doing a cloud native approach, you know, from day one. Which means no lift and shift, which is awesome. It’s arguably the hardest path, [Laughter] as we probably all know, but this means essentially from the ground up, building a software delivery platform, right, that’s integrated from the, all the way down to the development layer and all the way up to the automation layer. So, we’ve been very fortunate to work on something like that.

    That’s kind of where we started, and that was the motivation of building a truly cloud native pipeline, so, yeah, working on that stuff.

    Ashley: Now, you mentioned you’ve developed some tools, too, and with Spinnaker being open source, have you contributed a lot back to the community yourself, or are you more of a consumer of it? You know, everybody plays a different role, and not everybody’s, you know, having an open source project.

    Vasallo: I’ll selfishly say that I’m more of a consumer of it.

    Ashley: That’s okay.

    Vasallo: I would love to contribute more to it. I’ve done a few contributions in my past, but in recent times, I haven’t.

    With that being said, though, my team is empowered to make change and drive changes, so we do have one or two changes in flight, and one of them actually got merged this past week. So, we’re looking to help grow and not just be consumers. We’re hoping to also build and solve the same problems we’re seeing, maybe, for other people as well.

    Ashley: Well, good. That’s great if you get a chance to do that. Keep in mind, you know, exercising it, especially at the volume you’re talking about, that is super valuable, too. That’s what helps improve software, whether it’s open source or paid for. So, that’s another—

    Vasallo: Well, yeah, and I mean, it’s kinda like what you’re saying. If you’re not paying for the software, maybe you can pay for it within support in the sense that Spinnaker’s community is an open community. There’s a Slack channel out there and you can join—anyone’s more than welcome to join, ask questions, and get answers from potentially some high up people. And Netflix, even people in the community in general, right?

    We’re all on the same journey, and I mean, that’s arguably what kinda kept me involved in the Spinnaker community is just seeing such an open and collaborative group of individuals just willing to help each other—special interest groups every week or so meeting and just saying, “Hey, what’s the direction of the project? What else can we do? What are some challenges we’re seeing?” It’s actually been pretty fun.

    Ashley: What are some of the objections you typically see or maybe that you even ran into about automating and doing deploys at that volume?

    Vasallo: Yeah. So, it’s not like we’re doing 1,000 more changes a month, right? It’s arguably, running fast can initially be seen as a reckless approach, right? You know, if you said, “Hey, we’re averaging 10 deployments a month”—maybe that’s a standard organization—“to 1,000,” you’re gonna definitely open up some eyeballs, right? People will be like, “Wait, wait, wait. What do we need that for?”

    Ashley: “What about stability? What about up time? What about”—

    Vasallo: Exactly, exactly. But the argument is, it could be said that smaller changes typically result in less down time, right? If you can break down a purchase path change in some company, right, and maybe saying, “Hey, we’re gonna make a UI change and then we’re gonna do a back end change that’s backwards compatible and we’re gonna do, then, an API tier change to support that change such that everything hooks together,” you then broke up one large change of three major components into three separate small, individual pieces that can be rolled out safely, right? And tested and embedded as well. That’s the most important part. Because you’re only as good as your automation and testing as well.

    Ashley: Now, did you do much shift left in terms of doing more security testing or vulnerability testing earlier in the Dev cycle, too, as part of this?

    Vasallo: Well, so, that’s the beauty of it. I mean, essentially, the pipeline doesn’t, at least for us, doesn’t start at Jenkins or the build time. It starts all the way at the commit. So, we have a lot of tooling, at least at Redbox, that’s built to empower developers. The thing to also note about those 1,000 deployments—those deployments aren’t, you know, some Ops team clicking an OK button. They are developers promoting their own code, and they are empowered to write and deploy code to production when they feel it’s appropriate to release, right?

    But in doing that, we don’t just give them the keys to the city, right? We build a nice, safe, and reliable pipeline—essentially, a paved road, if you will. That’s the kind of term I’ve been hearing it described where, you know, you can ride the paved road and have fun. You can get off road and it’ll be bumpy, but the majority of the time, the paved road will get you to production as fast as possible.

    Ashley: Right. Yeah, that paved road concept comes from the Netflix paved road talk that they gave. I think that was OSCON or something.

    Vasallo: Yeah, there are definitely a lot of great things from the engineers there, a lot of inspirational things kinda came out of it. I mean, again, it’s building on the shoulders of these giants, right? They’ve seen, arguable, large percents of Internet traffic, right? And at that level of success, you know that some of those concepts definitely can be applied elsewhere, right?

    Ashley: Mm-hmm. Well, you know, one of the things that’s maybe a little bit misleading in saying automated deploys is, it isn’t just the deployment process that’s automated, is it? Talk about what all the parts that are automated to be able to deploy at that volume.

    Vasallo: Yeah, no—definitely. I mean, it’s essentially part of the—part of the talk is also gonna, the little secret of it is, it’s not like you can roll in Spinnaker and have an automated deploy and you’ll be good, right? There’s a lot of aspects of a pipeline, and as I was alluding to from a commit layer, right, whether you do your static analysis for security early on, whether you do your unit tests at build time, whether you have an automated way of storing in archiving artifacts, right? You need to know what is making it out to your environment stage prod.

    In addition to that, you need to build that infrastructure automatically, right? So, we have things—you know, you can use things like Terraform and CloudFormation as well, right?

    Ashley: Mm-hmm.

    Vasallo: But your load balancers, your security groups—again, we’re talking within the context of AWS—how do you reliably and repeatedly build your infrastructure in such a way that it can support that rapid deployment, right? You can’t just have an automated pipeline where it’s like, “Okay, hold the lights, let’s go call up somebody and say, ‘Hey, we need a load balancer. Can we get that done?’ ‘Sure, give me five minutes.’” Even five minutes is too slow, right? We need to have an automated way of creating and also getting fast feedback to our developers when these changes work and when they don’t work, right, so they can help triage and figure out what’s going on.

    Ashley: It’s almost like I’ve said before—DevOps is an overnight success 35 years in the making. [Laughter]

    Vasallo: [Laughter] That’s a good one, that’s really good.

    Ashley: Where you kinda get there one step at a time, but it is, and I think that’ll be something interesting to anticipate about your talk is really thinking about it holistically and what that entire process—not that you’re gonna go into every bit of detail in that amount of time, but it is about all of that and connecting the toolchain, the work flow, and the process, people process part of it into something that can be automated and happen at high scale.

    Vasallo: Yeah, that’s definitely something to be said there. I mean, a lot of—it’s generally a soft topic, but it’s arguably one of the more important ones is, there has to be a cultural transformation and understanding if you truly wanna be as successful as some of these giants, right?

    Building the concepts of open culture, right, working together, cross functional teams—there’s no magical fix out there in the world to fix a broken process in a broken culture, right? And that’s the toughest thing. And then that’s something, definitely, we touch on as well. And I alluded to it earlier with the concepts of, for us, that’s empowering our developers. So, building compelling tools for our developers to use every day they come to work.

    Ashley: Well, you mentioned keys to the kingdom for the developers. I can’t imagine you did all this without some form of senior leadership backing. How did you get that and where did you get it from to go down this path?

    Vasallo: Yeah. So, it definitely started very grassroots, right? Say the challenge to get to the cloud, right? That’s an easy challenge and I think a lot of—sorry, that’s an easy statement to say, but what does that mean? And for every organization, it’s different, right? For some, lift and shift is the most appropriate solution if you’re not willing to invest in the cultural changes, right?

    But for me, at least in my experience, it’s always been finding the hero in the regards of not necessarily a hero mentality, but finding that one champion who can help drive that, you know? That team who is under a tight deadline and you’re like, “Hey, you know, you’re a relatively new project. You’ve never really seen—you’ve never really gotten deployed. How about we just take an experiment and see if we can automate your app end to end, right?”

    Ashley: Mm-hmm, mm-hmm.

    Vasallo: The good news is, it’s a pretty low risk, it’s a new feature. The timeline, obviously, is always a concern, right, in terms of meeting business deliverables. But in doing that, working with those individuals, you not only can get fast feedback as to the proposal of the pipeline, you can also build a relationship that essentially is like DevOps and Devs working together to kind of create this delivery solution, if you will.

    Ashley: Hmm. Well, I feel like we just scratched the surface, which is probably a good thing, because it’s the preview of your talk. I appreciate you coming on the podcast today, Joel.

    Vasallo: No—no worries. Hey, thanks so much for having me. Again, I hope to see everyone at the Spinnaker Summit. It’s gonna be a great time, so I hope to see everyone there.

    Ashley: It is. Well, definitely, I’d like to thank Joel Vasallo, Manager of Cloud DevOps at Redbox for joining us today. Again, Joel’s talk at Spinnaker Summit, which is at 12:30 p.m. Sunday, November 17th in San Francisco, it’s titled “Transforming Software Delivery using Spinnaker,” so be sure and check that out if you’re making it. Now you have a reason to go to San Diego in the fall coming on the winter.

    Vasallo: Other than the great tacos, yeah. And everyone, join us on the Spinnaker Slack, we’re happy to help out and answer any questions as well. We’re more than a welcoming community, so hope to see more folks there.

    Ashley: That’s a great point. It’s a very vibrant community. So, we wish you all the best in your talk, and everyone, you’ve listened to another DevOps Chat podcast. I’d like to thank you, our listeners, for joining us today. This is Mitch Ashley with devopsy.kinsta.cloud and you’ve listened to another DevOps Chat. Be careful out there.

    — Mitchell Ashley

  • DevOps Chat: CI/CD Velocity for Large Monolithic Services with Pinterest

    DevOps Chat: CI/CD Velocity for Large Monolithic Services with Pinterest

    Spinnaker Summit 2019 Preview: Software Engineer Rainie Li played an essential role in implementing Spinnaker as part of Pinterest’s CI/CD pipeline. The results moved Pinterest from two scheduled deployments per day to continuous deployments, greater than 15 during business hours.

    This episode of DevOps Chats features a preview of Rainie’s talk, “How we introduced CI/CD for Pinterest’s largest monolith services (API and Web) to improve developer velocity, quality & reliability (Pinterest).” Topics including how Spinnaker was selected, important metrics, Pinterest’s future CI/CD platform Hermez, Canary analysis and lessons learned from the journey are shared.

    Rainie’s talk is on Sunday, November 17th, 1:30 pm PT, at Spinnaker Summit 2019 in San Diego. Joining Rainie on the talk is Jasmine Qin, software engineer with Pinterest.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Rainie Li, who’s a Software Engineer at Pinterest.

    Now, Rainie’s gonna be talking at the Spinnaker Summit 2019 coming up in San Diego, November 15th through the 19th, and her topic she’s presenting on is how Pinterest implemented CI/CD pipeline for large, monolithic, or monolith services, APIs and web, focused around helping them improve developer velocity, quality, or liability, et cetera. So, some real hands on experience that she’s planning on sharing. And she’ll also have a co-presenter, Jasmine Qin, and she’s gonna be joining her, too, at that presentation.

    Well, Rainie, welcome to DevOps Chat.

    Rainie Li: Thank you, Mitch.

    Ashley: Awesome to have you on. Thanks for joining us. Would you just introduce yourself, tell folks a little bit about you, the kind of development work that you do at Pinterest and also maybe a little bit of what you’ve done in your past to become a software engineer?

    Li: Yeah, sure. So, hi, everyone, I’m Rainie Li, I’m working at Pinterest infraorganization and our team is focused on doing continuous delivery platform for all the internal engineers. Before I joined Pinterest, I worked at both Microsoft and Amazon, also as a software engineer, and I found I have a very strong interest in infrasite, and it turns out, continuous delivery is a very important feature for software engineers. So, that’s why I ended up here.

    Ashley: Very interesting. Well, you’ve worked at some very large online services. I know that Pinterest has, what, about 250 million users, and of course, Microsoft and AWS are very large organizations, so you’re kinda used to working on some pretty big infrastructure projects, it sounds like.

    Li: Yeah, I am. [Laughter]

    Ashley: Good, good! Well, great experience. So, tell us a little bit about your talk. I guess maybe start with, you know, you came to Pinterest, maybe about where they were in this process of implementing continuous delivery and where you kinda picked up in the process. Were you at the beginning or they already started?

    Li: Yeah. I kinda joined the one that already started, but I still drive the whole production readiness for Spinnaker at Pinterest. I also drive the design review for Spinnaker as well. So, I kinda joined the one that already picked Spinnaker for Pinterest, but I’m the main person to continue to deliver this project.

    Ashley: Oh, so you kind of picked the project already in progress and then take it forward from there as one of the leads on it?

    Li: Yeah.

    Ashley: Very nice. So, when you said—so, you do a review of Spinnaker. Is that your internal usage and changes in implementation to it, or do you also contribute things back to the open source? Any of that kind of thing?

    Li: Currently, we only reveal how we use it internally and we do some customization for Pinterest only. We haven’t contributed to the upstream yet, but I think long term plan, we would like to contribute to upstream.

    Ashley: Yeah. Well, you know, just using it at the volume that you do, I’m sure, is a way of, a form of contributing back, too. It’s gotta certainly exercise Spinnaker and a lot of other tools about that, too.

    So, tell us about—so, you picked up when they had already gone through the implementation and you’re kinda furthering it, or were they still in the process of implementing CI/CD continuous delivery?

    Li: I picked up when they were still in the process of implementing a CI/CD platform.

    Ashley: Mm-hmm.

    Li: But they kind of already decided to use Spinnaker, but we still have to do Pinterest specific customization and how to make it production ready at Pinterest, that kind of work.

    Ashley: Ah, excellent. So, sounds like you had quite a bit to do with Spinnaker even when  you joined of getting it ready for production. Why don’t you talk about some of the things—I know you’re gonna talk about this in your Spinnaker Summit talk. What are some of the things that you need to do to Spinnaker to get it production ready for that kind of an environment that you’re in?

    Li: So, I think there are several key pieces. The first one is, we have to make sure authentication and authorization are working as expected for Spinnaker, because that’s the most important thing—like, we have to make sure users can go over our OS process before they can—

    Ashley: Your OS, mm-hmm.

    Li: Yeah. The second key piece is monitoring, which includes metrics and alerts. So, basically, we exposed all the Spinnaker service metrics and created a dashboard for those alerts, which we can get paged when there’s some issue that happened if Spinnaker is down or something like that. These are the two most important things for production readiness, I think, for us.

    Ashley: That’s extremely important when you’re automating something like a continuous delivery, continuous integration. It’s automated, but you have to know is it working right?

    Li: Yeah.

    Ashley: Are things happening and are there problems or are you meeting the kind of metrics that you were expecting?

    Li: Yeah.

    Ashley: I know something you mentioned in the description of your talk is what kind of metrics are important to measure. Can you say a little bit about what are some of the metrics that you’ve learned, both while you’re implementing it and now that you’ve had it in production that are really important to watch?

    Li: Yeah, definitely, those metrics can—like, latency and ____ P90 and also, like, those SLA related metrics from all Spinnaker components are very important. I think especially the gate service, which is Spinnaker API Gateway, it’s like the entry point for the rest of Spinnaker components, the metrics for Gateway is extremely important.

    Ashley: That’s kinda the integration component of Spinnaker, right?

    Li: Yeah.

    Ashley: Everything talks to it. It’s sort of that—I don’t know if it’s a central hub, but it’s certainly where all the APIs tie in together.

    Li: Mm-hmm.

    Ashley: I’m sure if you’re having a problem with that, then you’re gonna know [Laughter] there are probably lots of failures happening in the system.

    Li: Yeah.

    Ashley: Do you look at, do you also measure how many builds you’re doing per hour, per day, or how many deliveries into production you’re doing? Is that metrics that you capture through Spinnaker or you do that elsewhere?

    Li: That’s not the metrics that we are capturing in Spinnaker, but we do measure from the pipeline execution history.

    Ashley: Mm-hmm.

    Li: So, currently, we are using Spinnaker to do the continuous delivery for two major services in Pinterest. We finished around 15 deploys per day for these two major systems and each deploy contains roughly 10 commits, and we think this pipeline, it increased our productivity a lot, because previously, we have to do manual deploy.

    Ashley: Mm-hmm.

    Li: Yeah.

    Ashley: Do you also have, then—I know this is outside of Spinnaker, probably, but do you have most or all of your testing automated as well, or are there still some manual steps for testing before it goes into a final deployment?

    Li: You mean testing for the service itself, or testing for Spinnaker service?

    Ashley: Testing more of the software application part of it.

    Li: I see. Yeah, so, currently, we are using a separate stage to run integration test jobs in Jenkins. That’s how we do the testing step. It’s kind of automated already, like, we don’t have to interact with humans to trigger the build or humans to run the test. We don’t need that. It’s just a stage in the Spinnaker pipeline. We run the interpretation test job in Jenkins, yeah.

    Ashley: Mm-hmm, excellent. Okay, good. We have kind of a feel for your workflow, your tool pipelines. It sounds like it’s very automated.

    Li: Mm-hmm.

    Ashley: So, going back to the topic of implementing something at that kind of a scale, both, you’re a very large organization and you’re doing a number of deployments per day. I know there’s some things in Spinnaker—well, you have a platform called Hermes. And is that your own central workflow engine, or is that part of Spinnaker?

    Li: Oh, so, Hermes is our future CI/CD platform. So, we are in the, like, developing stage. Spinnaker will integrate with Hermes and now we will use Spinnaker as a back end workflow engine.

    Ashley: Hmm, okay.

    Li: Yeah.

    Ashley: So, that’s really your—that’s Pinterest’s own central workflow engine and you’re integrating Spinnaker into it as part of that feature?

    Li: Yeah, yeah.

    Ashley: Pipeline—tool pipeline, if you will. Okay.

    Li: Yeah, it’s like a CI/CD platform for Pinterest, and we are going to open source Hermes as well in the future.

    Ashley: Oh! Well, that’s good news. I’ll look forward to that.

    Li: Yeah. [Laughter]

    Ashley: Do you know kinda time frame when that’s gonna be happening, or is that a little farther in the future, don’t know yet?

    Li: I’m not sure the exact timeline, but somewhere early next year or at the end of this year.

    Ashley: Okay. Well, that’s not too far away. That’s great. We’ll look forward and thank you for contributing that as open source. So, talk some more about what are some of the lessons that you learned about going from where you joined Pinterest and where things were with Spinnaker’s implementation, maybe some things you had to do, some of the things that you learned? Here’s a good way to do it, here’s a mistake I made and I fixed it by taking a different approach—what are some of those lessons learned?

    Li: Hmm, I think the lessons for me is, because we deploy Spinnaker services to Kubernetes platform and I never used a Kubernetes platform before I joined Pinterest, so that’s the biggest lesson I learned here, like, how to set up all the Spinnaker components in Kubernetes platform, yeah.

    Ashley: Uh huh, good. Are there—so, learning how to do that was important. Were there any specific lessons that you learned about doing that, or is it just really just kinda learning the how of doing it?

    Li: A specific lesson I think is some special situations where the Kubernetes platform decides to rotate cluster and do some platform testing. On the Spinnaker service side, we have to handle this kind of scenario well instead of bringing down the sides or have some happy customer user experience. That’s a lesson I learned from there.

    Ashley: Okay. I think you also mentioned in your show notes or your description in your talk that you did some canary analysis as part of that as well?

    Li: Mm-hmm.

    Ashley: I assume that was a new thing for you, or have you done that before?

    Li: Yeah, that’s also a new thing, but we have a separate team which is the configuration team which, they are mainly implementing this feature. So, for Pinterest, we support OpenTSDB as a metric, but I think Spinnaker canary analysis can only support Prometheus or Datadog, those metrics. So, I think the ____ team integrated this OpenTSDB into Spinnaker canary analysis component so that we can provide Pinterest ____ on Spinnaker UI.

    Ashley: Ah, interesting.

    Li: Yeah.

    Ashley: Okay, great. What are some other things that you’re planning on talking about at your talk during Spinnaker Summit?

    Li: I think how we use Spinnaker and how we customize Spinnaker at Pinterest and the lessons we have learned when we adopted Spinnaker here. I think these are the main topics I’m going to talk about, yeah.

    Ashley: Mm-hmm. Great. Well, I know you weren’t at Pinterest when they decided to use Spinnaker for this project or for part of the toolset. But based on what you’ve seen and what you’ve learned, do you think Spinnaker accomplished the goals that they had for why they chose it?

    Li: Definitely. Yeah, we like Spinnaker a lot. [Laughter]

    Ashley: What were some of those goals, do you recall? Were there certain, we wanted to get to a certain number of deployments per day, or were there other things that they were trying to achieve that were looking to see how you’re doing?

    Li: I see. So, we don’t have a specific number for the goals, but Spinnaker definitely helped us to use pipeline for deployment. Without Spinnaker, we have to go to each stage, do manual collect deployment one by one, which is not very efficient. And currently, we can use Spinnaker to manage deployments in a consistent and a repeatable way. We really like pipeline, which can provide a sequence of deployment stages. We don’t need too much manual interaction, and—yeah, that’s, I think, the goal we adopted Spinnaker is, we need the pipeline feature and I think it works well for us. Also, the canary analysis report is very useful for us as well.

    Ashley: Yeah, I know a lot of people are very excited about using that.

    Li: Yeah.

    Ashley: And you mentioned that you integrated OpenTSDB as part of your observ—I can’t say it [Laughter]—observability stack, there we go, got that out. Now, I think you also have a deployment system there that Pinterest uses to deploy into AWS, correct?

    Li: Yeah. We have a deployment system which is also open source called Teletron, which we used for four years to deploy to AWS VM. But this deployment system does not provide a good pipeline, so that’s why we integrated with Spinnaker to have the nice pipeline feature, yeah.

    Ashley: Excellent. Well, you’ve sure done a lot of work. What’s kinda next? What’s the next set of projects that you’re thinking about or you’re currently working on now that you’re at this point with deploying Spinnaker into your CI/CD pipeline?

    Li: I think the next project is, our team will implement Hermes at the Pinterest main CI/CD platform. Spinnaker was working at the back end workflow engine, and we were also working on migration, like most of the services from AWS and VM to Kubernetes. Yeah, that’s two major projects that we are going to be working on.

    Ashley: Ah. Well, certainly, the move to Kubernetes, I’m sure that’s a pretty substantial project, a pretty big one.

    Li: Yeah, yeah. [Laughter]

    Ashley: Well, good. Well, thank you so much for joining us on the podcast today. This is very interesting and I think you’ve got a really compelling and interesting talk. I’m looking forward to hearing more about it and have others to get a chance to attend.

    Li: Yeah, thanks for inviting me, Mitch. I’m glad to introduce Pinterest using Spinnaker at the Summit and I’m very happy to talk about more there.

    Ashley: Absolutely. Well, thanks for contributing with your talk. So, you’ve listened to another DevOps Chat podcast. I’d like to thank my guest today, Rainie Li, who is a Software Engineer at Pinterest. Again, her talk during Spinnaker Summit is on Sunday, November 16th at 1:30 p.m. This is according to the agenda currently on the website, you can look there for updates for the Spinnaker Summit site, and her talk, again, is on how we introduced CI/CD for Pinterest’s large monolith services, API and web, to improve developer velocity, quality, and reliability. And as you’ve just heard, there’s a lot of information that Rainie’s gonna be sharing with you.

    So, thanks for joining us, Rainie, and thank you, of course, listeners, for joining us here today. This is Mitch Ashley with devopsy.kinsta.cloud, and you’ve listened to another DevOps Chat podcast. Be careful out there.

    — Mitchell Ashley

  • DevOps Chat: Monitoring Spinnaker on GKE with Miles Matthias

    DevOps Chat: Monitoring Spinnaker on GKE with Miles Matthias

    On a project to move one of Google’s Fortune 500 customers to Google Cloud, Google Kubernetes Engine (GKE) and Spinnaker open source, our DevOps Chat guest ran into the message “hang tight.” No scripts or documentation. Not to be delayed, Miles Matthias, Google Cloud Consultant with Container Heroes, filled the gap and contributed his work back to the Spinnaker community.

    This episode of DevOps Chats features a preview of Mile’s talk, “Monitoring Spinnaker with Prometheus Operator on GKE” that he will be giving on Saturday, November 16th, 3:45 pm PT, at Spinnaker Summit 2019. Miles also talks quite a bit about Canary testing in this episode.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Miles Matthias, he’s a Google Cloud consultant and he’s with Container Heroes—great company name. Our topic today is actually a preview of his talk at the Spinnaker Summit 2019. His topic is gonna be Monitoring Spinnaker with Prometheus Operator in a GKE Environment, Google Kubernetes Engine. His talk is on Saturday, November 16th at 3:45. Miles, welcome to DevOps Chat.

    Matthias: Hey, Mitch. Great to be here. I’m really excited about the Spinnaker Summit.

    Ashley: Well, tell us a little bit about you, what you do and a little bit about Container Heroes.

    Matthias: Yeah, sure. So, like you said, at Crowd Consultant, I’ve done a whole bunch of software development and infrastructure setup and architecture design throughout my career and now I’m helping other companies set up their cloud infrastructure, make migrations, modernize their application development and all sorts of things like that. And I’m a partner at ContainerHeroes.com, and we’re a group of kind of people like me that have been in previous startups, CTOs, who’ve done the whole PC route, we’ve built a bunch of custom applications for clients throughout out career and now we enjoy consulting people that are using the cloud.

    For the past year or so, especially I guess the last nine months or so, one of our big clients, we’ve been helping Google with one of their new customers that was complete on prem to cloud, to Kubernetes containerization migration and they utilized Spinnaker. So, I kinda was brought in as the Spinnaker expert to help them get it up and running, to help customize what they needed in their environment and contributed as much as they could back to the open source, including some stuff that I built for using Prometheus Operator, which we can touch on a bit. And so much so that then the team invited me to give a talk at the conference, so I’m excited about it.

    Ashley: Sounds like a very—very relevant talk, especially sharing your experience from working with this Google customer.

    Matthias: Yeah.

    Ashley: So, tell us a little bit about when you came to this project and first started working with the customer. Had you worked with Spinnaker before, was that a new technology, or were any of these new technologies, or you’ve kinda done all this before?

    Matthias: So, I’ve done a bunch of CI/CD before. Spinnaker is still in its early days, so I have very limited experience with Spinnaker as an application. It’s still—yeah, it’s still very nascent on the development scene. The entire kinda concept of continuous delivery is still very new in the industry, even though it’s been around for a little while.

    And so, CI has been very well developed, everybody knows Jenkins, everybody has ____, a million CI tools, content of continuous delivery and having more thought out, sophisticated delivery strategies that can help you do more automated continuous delivery like canary analysis and different deployment types and automated rollbacks and all these sorts of things are kind of brand new and Spinnaker obviously really helps with a lot of that stuff.

    Yeah, I had some experience, but was really excited to jump in and kinda get into something that’s really taking off in the industry.

    Ashley: Tell us a little bit about the app that you were challenged to work on. I understand this was already, had been developed with Jenkins and Spinnaker before and then you were moving it to GKE, or what was it?

    Matthias: No. So, the client had 50 different job applications, a bunch of different things, and it was all running on prem. So, part of the big effort was to obviously move them onto Google Cloud, but also to help them develop a CI/CD process that allowed them to make a commit in their repo, have artifacts be built in CI, have tests run and then have those passed to Spinnaker and CD and then deployed to GCP.

    They had no experience with Spinnaker and didn’t have any kind of CD solution setup, really, especially for the cloud. Like I said, they were completely new to the cloud, so this is all new stuff for them, so I helped them set it up, install it, configure the options all we wanted, manage it and then introduce more and more advanced features as the project got more and more migrated over to GCP.

    Ashley: Mm-hmm. So, it sounds like maybe a more traditional Java application environment maybe is not even continuous integration yet. Were they down the path with that?

    Matthias: Yeah, they had Jenkins on prem.

    Ashley: They did?

    Matthias: And so we—you know, yeah. So, the architecture was moving everything to Kubernetes and so, you know, one of the GCP projects has GKE cluster just for CI/CD. So, running Jenkins and running Spinnaker on it. So, they had Jenkins on prem, and moved that over to the cloud, hosted it on Kubernetes, and then we installed Spinnaker right alongside it on the same GKE cluster, right?

    Ashley: Mm-hmm.

    Matthias: And then used those tools as the basis of their CI/CD pipelines to deploy to other clusters within the cloud for them.

    Ashley: How about the size of the application? How would you quantify how kind of big or extensive it was?

    Matthias: Like I said, they had a—this was a true microservices organization. So, you know, they have 30 to 50 different applications as microservices, each individually being deployed. As far as traffic, I mean, so the applications, like I said, are microservices, so they’re pretty small in and of themselves individually.

    As far as traffic and resource usage, it’s a very, very large, large client. Like, pushing the boundaries of some of the largest things we’ve seen.

    Ashley: Hmm. Okay.

    Matthias: So, that was really exciting—really cool.

    Ashley: This was a Fortune 500 company, I understand.

    Matthias: Yeah.

    Ashley: We’re not talking about the specific company, but—

    Matthias: Sure, yeah, yeah, yeah.

    Ashley: Well, good. Well, what are the sort of things that you’re planning on talking about, then? I mean, I know this is about the monitoring aspect of it, so you know, are you touching on Prometheus Operator and probably how to configure it or how you decide it and what kinds of groupings to create, what kind of rules and alerts and those kind of things? Are you gonna be talking more fundamental architecture? What are your thoughts?

    Matthias: Kind of a bit on everything, because this kind of—this is kind of some required setup in order to do monitoring and in order to do canary analysis, even, if you’re gonna use Prometheus for canary analysis metrics. And it’s also just fundamental to how you wanna have this setup if you’re running Spinnaker on Kubernetes in general.

    So, a little background on Spinnaker as a project, which I’m sure other people that are familiar with Spinnaker definitely know, you probably know yourself. Spinnaker was originally created by Netflix. Netflix is still very much a VM shop, right? So, they don’t use Kubernetes, and everything runs on individual instances. And so, Spinnaker can be deployed to run on Kubernetes and that support is there, it has support to deploy applications to other Kubernetes clusters.

    But there are still a few things like monitoring, like canary analysis where, in the Kubernetes world, we do things a little differently, and Prometheus is one of those examples, right? In the VM world, you spin up Spinnaker, you spin up one Prometheus instance—and Prometheus obviously is a metric store to collect metrics and then usually is paired with Grafana and ____ and things like that in order to—all CNCF open source projects in order to give you graphs on these metrics.

    And so, there was previously, in the VM world of Spinnaker, running it as Netflix does, a microservice of Spinnaker that does monitoring, that connects to all of the different components of Spinnaker, listens for their metrics and then reports it to Prometheus. Cool. Like, it looks great. They even have some dashboards.

    So, Grafana dashboards, if you’ve ever worked with those you can just upload some JSON files for your dashboard and then you click there and then you’re like, “Hey, look at these dashboards!” You can get real time dashboards based on the Prometheus metrics of Spinnaker. And Spinnaker, all the different components that it’s made up of and the different metric based on the things that it’s doing so that you can see in there when it’s processing your deployment, here are some of the metrics that are coming off, right?

    Ashley: Mm-hmm.

    Matthias: So, all of that support was there. When you want to run Spinnaker on Kubernetes, though, that support was a little…not so much. [Laughter]

    Ashley: Mm-hmm.

    Matthias: And that’s the trouble that we kinda ran into. So, the thing that was there was the concept of, “Hey, we have this monitoring component in Spinnaker,” and if you enable monitoring, that monitoring component will be enabled as a sidecar container to every single pod, every single microservice that Spinnaker is composed of, and it knows how to talk to that microservice and get the metrics from it and then report it somewhere.

    Cool—good. Okay, I can get that. What about the monitoring service then having those metrics that are just collected from all the different components and putting it somewhere? Ideally Prometheus, you can also do Stackdriver or, I believe there might be Datadog support now, but mainly, the two that we see used a lot, Stackdriver and Prometheus.

    Ashley: Mm-hmm.

    Matthias: However, in the Kubernetes world again, especially when you have a bunch of different Kubernetes clusters, one of the tools that you end up using is called Prometheus Operator. And that is essentially a Kubernetes operator that, when you apply to your cluster, will automatically install for you Prometheus Grafana, an alert manager as pods, deployments running on your cluster. It’s a way to provision your infrastructure so that when you deploy a bunch of applications in these clusters, you already have some monitoring infrastructure set up, right? I can deploy and I can basically go to my organization and say, “Give me a new cluster and then I can deploy my application and start admitting Prometheus metrics and there’s already a Prometheus instance on the cluster, so like, I’m good to go.”

    In order to get the Prometheus, the version of Prometheus that the Prometheus operator creates to then go and fetch the metrics from the Spinnaker monitoring component that already existed and already knows how to get those metrics from the Spinnaker different components, you have to kinda connect those two ends, right? And in the Prometheus Operator, the Kubernetes world, basically, you just apply Kubernetes manifest, the CRD called the Service Monitor, and that basically tells your instance of Prometheus on the cluster, “Hey, go fetch the metrics from these places,” right? And so, it’s pretty easy, but there was no setup way in Spinnaker. So, you would read the documentation in Spinnaker and they would have very detailed, very nice—like, if you’re running Spinnaker on VMs and you want to do monitoring and you wanna use Prometheus, just use this fancy and nice, easy setup script, right?

    Ashley: Mm-hmm.

    Matthias: Super nice. Great. And then literally in the documentation, it said, “If you’re running on Kubernetes. Hang tight, support is coming.” [Laughter]

    Ashley: [Laughter] The “hang tight” documentation feature. Wonderful!

    Matthias: Hang tight—yeah, right. So, none of this is extremely complicated. There are a lot of different components and you have to be careful about how you set them up, right? But the very nice setup script that was available for just running plain old Prometheus on VMs was very nice and like I said, it installed a bunch of Grafana dashboards for you and did a bunch of nice stuff for you. There was no such thing for Kubernetes, right?

    Ashley: Mm-hmm, mm-hmm. Kinda starting from scratch. Not completely, but at least to take that next step.

    Matthias: Exactly, exactly, exactly. And so, what I did and what I contributed to the project was creating a setup script for this exactly, saying, “Hey, you’ve got a cluster, you’ve got Spinnaker already installed on there, you’ve got Prometheus Operator installed on there and you’ve got a Prometheus Operator installed on probably a bunch of other clusters, too. Here’s a setup script that connects Prometheus instance to Spinnaker so that Prometheus can go and fetch those metrics about how Spinnaker is running and how Spinnaker is handling your deployments.”

    Ashley: Mm-hmm.

    Matthias: And installs a bunch of these Grafana dashboards that the open source project has already created for us to be able to say, “Here’s a Grafana dashboard about each microservice—Clouddriver, Gate, Orca, all the others, they each have their own dashboard in Grafana that you can go look at and they’re preconfigured and it’s really nice, it’s really neat.” Again, those just had to be kind of converted into the way you apply these in the Prometheus Operator world which, again, is applying your Kubernetes manifest to say, “Hey, it’s actually just a config map with a certain label on it.” “Hey, here’s a dashboard,” and when you apply that, the Prometheus Operator knows how to hook into the Kubernetes master and goes, “Hey, there’s a new config map that is a Grafana dashboard. Let me go fetch it and add it to Grafana for you,” right? That’s the whole purpose of Prometheus Operator.

    So, that’s kind of the thing that I set up was saying, was giving you—so now, and then I PR it to the documentation. So, now, when you go look at the documentation, it says, “Hey, if you’re running Spinnaker and Prometheus Operator on Kubernetes, there’s also a fancy setup script for you, too!” [Laughter]

    Ashley: There you go. Now people will have to start at the same place you did.

    Matthias: Yeah, really.

    Ashley: So, are your planning, on your talk, actually walking through this, the process of the script you built and how to set all this up, get the dashboards to come up, et cetera, or you’ve kinda taken a little different approach?

    Matthias: I don’t want to anger the demo gods, so I’m not sure if I’ll actually be, you know, typing it, running the actual scripts, but I can certainly have something set up that’s like, “Hey, this is the end result. Let’s step through what the script is actually doing to help you understand what components need to be matched up where, and then here’s the end result,” right?

    The other kinda couple things that I’ll probably touch on in the talk are—which is why it’s important. So, all of that is monitoring Spinnaker as an application. So, like, your DevOps or your Release Engineering team, people that are involved in these things who are gonna monitor Spinnaker as an application and are gonna see, “Oh, Orca is overwhelmed and it has a bunch of tasks in its queue that it can’t pop off, so let’s add some more replicas of Orca” or something like that.

    That’s what this script helps you set up is this dashboard to see the performance, to see those metrics, right?

    Ashley: Mm-hmm, mm-hmm.

    Matthias: And other reasons that it’s important are, and the other things I’ll touch on—well, first of all, the one thing I’ll probably touch on is that there’s kind of an effort currently in the works and I hope by the time of the talk, maybe there’s a little more effort built towards it, but there isn’t really a whole lot of guidance in the Spinnaker community about, “Okay, here are the metrics we emit. Here’s what they mean, and if they go outside normal ranges, here’s what you should do to remedy that situation.”

    Ashley: Mm-hmm.

    Matthias: There are no kind of run books, as you might call them, for how to respond to a Spinnaker instance that is under a really extreme load or kinda gone configure sideways or something like that.

    Ashley: Interesting.

    Matthias: Rob Zyner kinda leads the Netflix effort. He’s done a great blog post about some of the metrics and things like that and what each of them mean. But that’s kind of the only resource out there. So, there is kind of this effort in the community to say, “Hey, let’s put together some actual run book and match some of these dashboards that we have in Grafana and some of these metrics and be able to tell people, ‘Hey, if you run into this situation, here are the step 1, 2, 3 to check, and if these are the case, then here are the three options for alleviate that problem or how to address the problem,” right?

    Ashley: Mm-hmm.

    Matthias: So, hopefully, by the talk in November, we might have some progress on that in the community.

    The other thing that I’ll probably talk about on the topic and why the concept of the Prometheus Operator is so important in the Kubernetes world is for canarying.

    Ashley: Say a little bit about what canarying is with your deployment processes.

    Matthias: Yeah, sure. Sure, so, canary testing is—it’s a form of testing that evaluates one of your release candidates. It compares it to—what it does is, it actually runs your candidate, and it runs the version that is running in production, starts a new copy of it, and it runs it side by side. So, then it is—and then it diverts usually a small amount of traffic or just, you know, will have 100 pods behind a service and you’ll add 2 pods so they’ll get a proportional small amount of traffic. And you’ll run it and you’ll see how it performs. You’ll listen to the metrics and you’ll see, “Oh, this is how your candidate that you want to push performed.” And in canarying, in the configurations in Spinnaker, you can say, “These are the metrics I care about when evaluating, when deciding if it performed well and if it performed badly.”

    For instance, let’s say you have a new version of the application that just, like, 500—500 errors out on every request, right? You run a bunch of unit tests, you run a bunch of integration tests and ideally, at that point, at one of those steps or earlier, you would’ve caught that, right? But tests are written by people and sometimes people don’t write them or sometimes some situation that depends on live traffic and live data actually exposes some types of problems, right?

    So, let’s hypothetically pretend you have a new version that comes out and it’s just 500s all over the place. What canarying says, the kind of theory behind it is—let’s actually run it on a small amount of traffic, let’s see what it does. And then canarying would go, “Oh, hey, your release version that you wanna push out? It’s just doing a bunch of 500s, and you told me in your configuration, if you see so many, a spike in 500s, that that’s bad, that we don’t wanna actually then deploy that to all 100 instances running in production,” right?

    That canarying says, “Hey, that’s not a good thing to promote,” right? Like, “I gave it a chance, I ran it on some things, I compared it to what the version that is currently running or our new instance of the version that’s currently running, and it didn’t do well. So, you know what? I’m gonna error out, I’m gonna show you this is how it behaved, you guys go fix it, and then when I come back, I’ll run those things again. And if it performs well, then great, we can release the new version.”

    And that’s kind of one of the main components that you need for continuous delivery. Because you can imagine a world where developers are just committing, committing, committing, committing.

    Ashley: Well, yeah.

    Matthias: They do a bunch of tests, they run some tests, to them it all looks good. And then you just have this other automatic component over here that’s doing canary analysis and that can just test it, it can actually put it into production, give it a little bit of traffic, see how it performs, and if it performs well, then let it go, you know? Promote it, right?

    Ashley: Yeah, that’s extremely useful.

    Matthias: Deploy on a Friday, deploy on a Saturday—who cares, right? It’s actually seeing how it’s running, all the other tests before it’s even gotten to that point have passed, obviously, and if it runs well, then run. And this is what enables organizations like Google, Netflix, and others to deploy thousands of times a day, because it has this automated system to go in and say, “Give it some traffic, see how it runs, and if it runs fine because of the way we configured how we told the system what does it mean to run fine, right? Great—then let it go.”

    Ashley: We’re kinda running up against our time, here. I do have one last question.

    Matthias: Sure.

    Ashley: What kind of folks should come to your talk? Obviously, it seems like developers, people who are working on the CI/CD pipeline, maybe DevOps engineers. Are there other folks you’d recommend, or are those the right folks?

    Matthias: Those are definitely all the right folks. I think if you’re interested in Spinnaker or you’re currently using Spinnaker where you are, want to know about monitoring it as an application and managing it as an application when it experiences a bunch of load and how you want to connect all the dots if you’re running it on Kubernetes and you’re running multiple other Kubernetes clusters that also are using Prometheus Operator, this is a great talk to go listen to.

    Ashley: Again, Miles is a Google Cloud Consultant with ContainerHeroes.com. He’s speaking at the Spinnaker Summit 2019 in San Diego. Now, that conference is the 15th through the 19th of November. His talk, again, is on monitoring Spinnaker with Prometheus Operator on GKE on Saturday the 16th at 3:45. Thank you, everyone, for joining us. You’ve listened to another DevOps Chat. This is Mitch Ashley with devopsy.kinsta.cloud. Be careful out there.

    — Mitchell Ashley