devopsy.kinsta.cloud

Tag: Spinnaker

  • Armory Extends Scope of Spinnaker CD Distribution

    Armory Extends Scope of Spinnaker CD Distribution

    Armory today updated both the self-hosted and managed editions of its distribution of the open source Spinnaker continuous delivery (CD) platform. The updates support concurrent pipelines and provide tighter integration with open source Terraform infrastructure-as-code (IaC) to better enforce policies.

    Previously, the two platforms were known as Armory Enterprise and Armory Enterprise Managed. They are now being rechristened; Armory Continuous Deployment Self-Hosted becomes Armory Continuous Deployment Managed in part to provide more differentiation from a CD cloud service the company rolled out last month that is not based on Spinnaker.

    Adam Frank, vice president of product for Armory, said other capabilities added to the platform include additional testing tools for contributions to the Spinnaker platform as well as enhanced plug-ins and remediation capabilities for Common Vulnerabilities and Exposures (CVEs) that might impact the Spinnaker platform.

    Despite launching a declarative CD-as-a-service platform that is not based on Spinnaker, Armory remains committed to supporting organizations that prefer to continue to rely on the open source platform, said Frank.

    The number of organizations that routinely make use of a dedicated CD platform is still relatively small. While organizations have been employing continuous integration (CI) within a CI/CD platform, making use of the CD element of those platforms has proven challenging. Each platform used to run applications has been unique, so automating delivery has been problematic. CI platforms, in some cases, have been extended using custom scripts to automate application delivery, but those efforts don’t typically scale across an extended enterprise. Many organizations that attempted to extend the open source Jenkins CI/CD platform, for example, are now looking to more loosely couple CI and CD processes to make DevOps workflows less brittle, noted Frank.

    Armory is making a case for a distinct CD platform at a time when more organizations are starting to build microservices-based applications on Kubernetes clusters that expose a standard set of application programming interfaces (APIs) regardless of what infrastructure they are deployed on. In many cases, those organizations now need to manage multiple pipelines in parallel as application environments become more complex rather than relying on a single pipeline, noted Frank.

    There’s no doubt that as the pace of application development continues to accelerate, the need to automate CD will become a more pressing issue. DevOps teams are releasing a larger number of smaller updates to applications and there’s also a need to be able to easily roll back those updates in case an unexpected issue is encountered, said Frank.

    It’s not clear yet how many organizations will be adopting dedicated CD platforms, but as application development continues to evolve many DevOps teams will, at the very least, be revisiting their existing processes. At the same time, there are also many organizations that are just beginning their DevOps journey. Most of those organizations are not committed to one CI/CD approach versus another.

    Regardless of approach, however, the expectation in the era of digital business transformation is organizations will be able to deliver software updates in a timely manner. The only way to achieve that goal, however, will be to have some type of CD capability that reliably operates at the scale required by organizations that need to operate like a true software company.

  • CD Foundation Touts Spinnaker CD Progress

    CD Foundation Touts Spinnaker CD Progress

    At an online Spinnaker Live event today, the Continuous Delivery (CD) Foundation revealed the open source CD platform is gaining significant momentum since being contributed by Netflix late last year.

    There were on average 842 merged pull Spinnaker requests a month in the first quarter of 2020. A total of 1,183 contributors participated in the project in the last year, with 40% of those contributors participating in the first quarter. Since the first quarter, the project has added at least two new contributors a week.

    Rosalind Benoit, chair of the CD Foundation Outreach Committee and director of community at Armory, a provider of an enterprise edition of Spinnaker, said the bulk of the increased contributions to Spinnaker are coming from Amazon Web Services (AWS), which the cloud service provider is already employing to orchestrate software delivery across multiple platforms.

    Benoit said Spinnaker represents an effort to finally address CD issues that have been difficult for most organizations to solve. While continuous integration (CI) is a solved problem, Benoit says CD remains problematic. Each platform code is deployed on within enterprise IT environments is a snowflake; as such, not much progress has been made toward automating the delivery of code across different platforms, said Benoit.

    The goal for Spinnaker is to create a foundation for CD that will achieve groundswell support that was achieved for Kubernetes by the Cloud Native Computing Foundation (CNCF), she noted. Both the CD Foundation and CNCF are arms of the Linux Foundation.

    Last year, the CD Foundation was launched to advance development and adoption of Spinnaker alongside Jenkins, an open source CI/CD platform; Jenkins X, an opinionated open source CI/CD platform for Kubernetes; and Tekton, an open source project and specification for creating CI/CD components.

    While many DevOps teams have attempted to address CD challenges using Jenkins, progress has been uneven at best. Spinnaker was built by Netflix specifically to automate software delivery across multiple clouds using a framework that can be integrated with any CI tool. As such, it serves to provide a clearer line between where CI ends and CD begins.

    Historically, most application code is pushed from a CI tool out to a deployment platform. As Spinnaker and other cloud-native platforms such as Kubernetes continue to evolve, the process of pulling images from CI tools to achieve CD goals will become much more automated.

    It’s not clear just how much depth of support for Spinnaker there is among providers of DevOps platforms. Members of the CD Foundation in addition to Armory include CloudBees, CircleCI, Google, Huawei, IBM, JFrog, Netflix, GitLab, Harness, Puppet, Rancher Labs, Red Hat, SAP and Sysdig. AWS has yet to formally join the CD Foundation. However, few of these vendors have detailed how they intend to incorporate Spinnaker in their respective platforms.

    In the meantime, DevOps teams should be paying close attention to how best CD practices will evolve alongside existing CI processes because it’s clear the next major phase of their continuing DevOps adventure is only now just beginning.

  • DevOps Chats: Demystifying Spinnaker VM Baking, with Salesforce

    DevOps Chats: Demystifying Spinnaker VM Baking, with Salesforce

    Spinnaker Summit 2019 Preview: As you use an open source tool like Spinnaker, the more you develop best practices and learnings. These are useful to share with others internal to the enterprise and other open source users.

    Jing Vergara, principal software engineer at Salesforce, joins DevOps Chats to share a preview of her upcoming talk “Demystifying Spinnaker VM Image Baking and Deployment.” Jing shares with us what to include, and not to include, when baking a VM image, how to avoid configuration drift and how to deploy to multiple cloud providers using Cloud-init files. Jing also shares how to use open source Packer templates with Ansible, Chef and Docker, and how to build your own Packer templates.

    Jing Vergara’s talk is on Sunday, Nov. 17th at 10:45 AM PT. Spinnaker Summit 2019 is San Diego on Nov. 15-17.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Jing Vergara, who is principal software engineer at Salesforce. Now, Jing is going to be talking at the Spinnaker Summit in San Diego in November, and she’s gonna be talking about a really interesting topic, “Demystifying Spinnaker VM Image Baking and Deployment.” Now, don’t think we’ve changed into a baking show. We’re baking VM images, and that’s what Jing’s gonna tell us a little bit about, that whole process—pipelining and creating VMs and baking them. [Laughter] So, her talk is on Sunday, November 17th, at 10:45 a.m.

    Jing, welcome to DevOps Chat.

    Jing Vergara: Thanks, Mitch.

    Ashley: Great to have you.

    Vergara: Thank you for the opportunity to have me talk here.

    Ashley: Well, I’m honored you’re on. Thank you for being here. Tell us about yourself. Tell us a little bit about what you do and maybe what parts of Salesforce product or code that you work on.

    Vergara: Sure. So, I’m part of the continuous deployment team at the company. We offer Spinnaker as a Service for all of the organizations within the company. And I help teams onboard their delivery pipelines into Spinnaker.

    Ashley: Oh, interesting.

    Vergara: Yep, and as part of my job, I also build custom extensions to Spinnaker to support their internal business needs.

    Ashley: So, you’re kind of the Spinnaker service center expert person or group that everybody goes to for help implementing Spinnaker. Is that close?

    Vergara: Yeah—very, very close, yes. So, my role within the team has been changing, but yeah, it has transformed into that one. Another thing that I do for Salesforce is that I represent Salesforce at a couple of the Spinnaker special interest groups for stakes insured so that I could learn more about upcoming features of Spinnaker, so we can leverage those in our roadmap as well so that we know we can offer it to the business.

    Ashley: Mm-hmm.

    Vergara: And then we also contribute back to the ________ and let them know what would and would not work for us as users of Spinnaker.

    Ashley: Well, and I noticed in your bio on the Spinnaker site, you have a Master’s in Computer Science from the University of Illinois at Urbana-Champaign. I used to live not too far, from there, though I didn’t go to that school. [Laughter]

    Vergara: [Laughter]

    Ashley: That’s a very strong C.S. degree, and it sounds like you’re putting it to good use with Spinnaker and DevOps style deployment with these kinda pipeline creation.

    Vergara: Yes, right.

    Ashley: Excellent. Well, tell us a little bit about what—so, we’re talking about your talk. So, describe for us a little bit about what are some of the things that are maybe a little mystical that need demystifying about Spinnaker VM image baking and deployment. What are some of the things people don’t know that you usually have to help them understand with, “Here’s how it works and here’s how we do it”?

    Vergara: Sure. So, for companies that are traditionally in a bare metal world where you don’t use VM images, it’s kind of a definite shift in the mindset, right?

    Ashley: Mm-hmm.

    Vergara: So, the way I usually explain it to them is using the baking metaphor, right? So, when you talk about baking, you think of putting the ingredients together like flour, salt, what have you, right? You form a dough, shape the dough, then you cook it. You start with a base ingredient—like, again, the flour—and you mix it up.

    So, in the VM world, when you bake an image, you start with that base OS image. You add your other ingredients, which means installing your applications or your agents to run on that VM. And then you have your standard or common configurations and possibly some data, right?

    Ashley: Mm-hmm.

    Vergara: You add them all up in the VM, you create a snapshot. So, that snapshot, you can now call it the VM image. So you can save this image into a store, and once you have that image, you can use it for deployment. So, deployment, it’s basically building or provisioning one or multiple VMs out of that same image.

    Ashley: Okay. Now, leading into this, the thing that kinda kicks it off is that, I’m not sure how it works at Salesforce, but is it something like a Jenkins process that then kicks off this creating of the image? How does that work?

    Vergara: Right. So, we have our continuous integration system. So, we have a few and we’re standardizing that at Salesforce as well, right? So, we have those artifacts built by those CI systems. We have our Docker images so they get stored, we put them into the cloud, and then we have triggers. So, we listen for those changes. So, Spinnaker supports a Pub/Sub subscription that triggers the pipeline. So, then, once we have those architects, we can start the baking process.

    Ashley: Great. Excellent. Well, that makes a lot of sense. Think of it as all of the pieces of software, OS on up, that are ingredients, right? Just elements of what’s gonna go into that image.

    Now, I imagine this is something that’s happening frequently if you’re doing a lot of deploys, right? Is this happening every time that you do a deploy or just as part of the integration process, and then it may go through a deployment process, too, or how frequently would you see this running?

    Vergara: Yeah, so it’s for every code change that gets checked in, right? Ideally, you would want that change to go all the way, so for continuous—again, for continuous delivery and deployment, that’s the ideal process. Check in, generate the artifacts, and deploy that artifact, create a baked image out of it, and then go and deploy it.

    Ashley: Okay, great. So, this is—

    Vergara: Right, so, but—yeah, yeah, right. But like, when we say delivery and deployment, of course, you know, there’s release management and change management policies at every company that says, “Okay, this is actually good for production deployment,” right? So, some of them are continuous deployment up to development or test environments, but for production, we still want to have that gate system in there.

    Ashley: Mm-hmm, yeah.

    Vergara: And Spinnaker actually supports it really well, you know? It’s got those manual judgment stages that you can add into the pipelines to get those kinds of approvals.

    Ashley: Yeah. Well, we’ve had some other folks talking about some of those stages like Run Job and Webhook and some of the capabilities that are built in.

    Vergara: Yes.

    Ashley: Easy to extend as well in Spinnaker. So, I think one of the things that you’re planning on talking about in your talk at Spinnaker Summit is maybe some lessons learned or best practices that you’ve figured out from doing this a lot at Salesforce. Any thoughts on that you want to share with us about the kind of best practices you’re gonna get into in your talk?

    Vergara: Sure, yeah. So, some of the best practices that I’d like to share with you is more around the baking and deployment. So, for example, when you do baking and never, ever bake secrets during bake time. [Laughter]

    Ashley: Yeah. No tokens, no keys, no certificates, no—

    Vergara: Exactly. Yes, yes. So, that’s very, very important. So, those kinds of sensitive data, you can actually perform that during the deployment stage. There is a trap stage there during deployment where you can go and, you know, connect to a secret store, like, Vault, HashiCorp’s Vault, right? So, you can pull down those secrets and install it on the resources or the VMs, right? So, that’s the safer way to go about it rather than baking it during bake time.

    And then another thing as well, you know, as much as possible to avoid configuration drift, right? So, if you have any common or standard configuration, be it your application configuration or your OS configuration, right, do it during the baking step, right? So, as much as possible, limit those VM configurations towards them. Again, you can do it during deployment.

    Ashley: Mm-hmm.

    Vergara: And then, for companies that support multiple cloud providers—so, for example, you need to go and deploy to, say, Amazon Cloud or Google Cloud or other clouds that are available and supported by Spinnaker, you can use Cloud-init files so that it’s not tied to one particular provider and you don’t have to repeat your bootstrap steps during that same time. So, you write your script once, your Cloud-init file, and then you can use it in those different cloud providers, so that’s, I think, that’s very, very helpful.

    Ashley: Now, is that capability something that’s built into Spinnaker if you write your script corrects, and I think you said init file for each—

    Vergara: Cloud-init.

    Ashley:—each cloud, Cloud-init.

    Vergara: Yes.

    Ashley: Is that something that, it’s a feature if you know about, you can use it up front and design that into how you’re writing your scripts?

    Vergara: Yeah, so, there’s—there are fields within the Spinnaker deploy stage where you can put that in. So, in, I think in GCEVMs, you can use the user data field in the deploy stage, and then in AWS, there’s the metadata field.

    So, the field kind of differs by provider, but it’s the same concept, right?

    Ashley: Mm-hmm.

    Vergara: So, you put it in there, you plug it in. And then, during the bootstrap process of those VMs, I think the OS has the support Cloud-init, though, so when they choose an OS, make sure it’s got the Cloud-init, and then Cloud-init will pick up those files and then launch the steps or commands from what you have in those Cloud-init scripts.

    Ashley: Mm-hmm, great. What are some of the things that people that are maybe new to Spinnaker or new to creating VM images and doing baking, what are some of the typical struggles someone might have that would be helpful to come to your talk for?

    Vergara: Right, so I experienced the same thing when I started out with Spinnaker. I’m like, “Oh, this is all new to me! How do I go about this?” [Laughter] So, one of the things that I’m gonna share in the Summit is, like, “Hey, you know what? Spinnaker actually uses Packer, it’s a HashiCorp tool that actually performs the baking behind the scenes.

    Now, Spinnaker offers a set of Packer templates, right? So, those are the very basic ones. And it actually varies per provider. So, you can only use one at a time. So, if you’re building for, say, GCP, then use the GCP Packer template. And if you’re using AWS, you have to use a different AWS Packer template.

    Ashley: Mm-hmm.

    Vergara: So, and then, if you have a more advanced use case, you can actually customize and build your own Packer template, so that’s one of the powerful teachers of Spinnaker as well, right? So, if you end up—like, you know, so for in-house, you use Chef or Ansible or Puppet. There are Packer templates that can use those kinds of provisioners, but you have to go and deploy those into Spinnaker.

    So, at the talk, I’m gonna share how you can deploy those custom Packer templates. So, there are a couple of ways on how to do it. So, one is the simpler way, but it’s kind of limited. Those Packer templates are stored like Kubernetes secrets in Spinnaker, so it’s got a limitation in size. So, if you have complex scripts or Packer templates, then you have to deploy it as a sidecar container.

    Again, I’m gonna share those tips at the Summit and give more details around it.

    Ashley: Mm-hmm. Yeah, folks may be familiar with Packer, too, it’s an open source project in and of itself for creating images, machine images.

    Vergara: Yes.

    Ashley: Great. Are there some other struggles that you find folks typically have, or may have when they’re new to this?

    Vergara: What else? So, Spinnaker by itself mostly focuses on releasing software changes, but it has a very flexible pipeline management system. Like, at Salesforce, we actually use it for building resources such as load balancers, DNS records, topics, queues. You can actually create custom job stages to perform those.

    Ashley: Mm-hmm.

    Vergara: So, we found it really awesome, because we can rebuild a full environment carrying down all of those existing resources, running the same set of pipelines that we can deploy together with those infrastructure resource setup. So, we can build it up right away, right, and then it also answers our scaling requirements. Because, you know, large scale enterprises such as Salesforce, you know, we have to build all of these resources and deployments. We’re talking about thousands of servers across different regions and geographical locations.

    Ashley: Mm-hmm. You know, sometimes tools, open source, or even commercial tools that are highly flexible and very powerful, sometimes with that comes complexity, so it helps to have some guidance, some experience mentoring from someone like yourself that’s gone through that process and can help you kinda get through it and understand how to use it, how to get things done, right?

    Vergara: Right. So, Spinnaker has a Slack channel. So, you know, when people like me or others who are new to Spinnaker have any questions, we go and ask in that channel. There’s a few channels there that are specific to certain features. So, for example, if you’re struggling with templatizing pipelines, then you go into the Pipelines as Code channel, right? Or if you have questions about how to use Kubernetes in Spinnaker, then you use the Kubernetes Slack channel.

    That was very, very helpful for us, especially in the beginning when we were trying to figure out how things are working.

    Ashley: Mm-hmm, I’ll bet, I’ll bet. How long have you been using Spinnaker, then, at Salesforce?

    Vergara: We started our journey last year.

    Ashley: Okay. [Cross talk]

    Vergara: Late last year, so almost one year now. [Laughter]

    Ashley: Wow, okay.

    Vergara: Yeah.

    Ashley: Wow. I bet a lot has happened in that one year. I know one of the things that you’re planning on also talking about is some contribution ideas for those that might want to help make Spinnaker better. And those are things that you—ideas you had or lessons you’ve learned implementing Spinnaker there, I assume?

    Vergara: Right. So, some of the contribution ideas that we have is with regards to a new effort called Kill. I’m not sure if you have interviewed the speakers.

    Ashley: I haven’t yet, but I may. [Laughter]

    Vergara: Okay. Yeah, so I’ll let them—you know, I won’t steal their thunder.

    Ashley: Don’t steal their thunder—okay, great. [Laughter]

    Vergara: [Laughter] Right, so, it’s one of those, they’re focused more around the managed delivery, so that’s when. Another contribution idea is centered more around each of the providers. So, again, if you’re building for one provider, it’s a different, totally different stage. So, for example, like GCEVM, right? So, they have their own deploy stage for that. And then AWS, they would have their own deploy stage for that. They’re kind of not, like, at par right now. Like, you know, they focus on certain things.

    I think another contribution there is trying to add more options or features into that, so it can fully support that provider. And so, for example, for GCE, I think one of the features lacking there, if I remember correctly—or they might have fixed it. But it was, say, so if you have a persistent disk and you want this particular persistent disk, then it’s not supported. It always creates a new one. So, it doesn’t give you an option to actually reuse an existing one that you’ve built earlier. So, things like those. I think that would be important.

    Same thing with AWS, right? So, there’s some features for EC2 VMs during deployment that could be added in there as well.

    Ashley: Great. I’m wondering, too, thinking about your talking that’s coming up, who are the kinds of folks that would be, get the most out of it, coming to your—obviously, people that wanna perform this function, but is that usually, is it developers, is it DevOps engineers, is it CI/CD automation specialists? Who are folks that would get the most out of your talk?

    Vergara: It would be around the Spinnaker administrators within their companies. So, if they want to learn how to configure Spinnaker to enable features for baking and deployment, then they would be the ones who would learn the most out of it. But I would say, like, all the DevOps engineers who want to figure out how they actually use Spinnaker for baking and deployment as well. Like, how do they set up their pipelines?

    I’m gonna give a demo on how it actually works from start to finish so that they can have a visual. Sometimes, you need that visual to say, “Oh, yeah! Oh, that’s how it works!” Right?

    Ashley: [Laughter] Mm-hmm. Well, that guidance is very helpful. And I can see, too, what you’re saying about the folks that are obviously administering Spinnaker in that deploy process are ideal to attend, and of course, other DevOps engineers automating and wanna know how this part of it works certainly would be good to go as well.

    Well, I wish you the best of luck. I know you’ll do real well. You obviously know, have learned a lot and know a lot from your experience at Salesforce. So, I wish you the best.

    Vergara: Well, thank you. Thank you, Mitch, again, for the opportunity for having me talk here.

    Ashley: Well, absolutely. Thank you for being on. I wanna thank my guest today, Jing Vergara, who’s princ—I can’t say it [Laughter]—principal software engineer at Salesforce. There, I got it out. Now, Spinnaker Summit 2019 is November 15th through the 19th in San Diego, and Jing’s talk is on Sunday, November 17th, at 10:45. It’s titled, “Demystifying Spinnaker VM Imaging Baking and Deployment.”

    So, thank you for joining us today for this DevOps Chat podcast. This is Mitch Ashley with devopsy.kinsta.cloud. Have a great day, and be careful out there.

    — Mitchell Ashley

  • DevOps Chats: Debugging Spinnaker Apps, With Salesforce

    DevOps Chats: Debugging Spinnaker Apps, With Salesforce

    Spinnaker Summit 2019 Preview: Debugging production issues in any environment can be challenging, and Spinnaker has its production learning curve. Problems aren’t always replicable in a smaller environment, and debug messages can be verbose and confusing to triage what’s happening.

    Our DevOps Chat guest Chuck Lane, Salesforce lead software engineer, is giving a talk on “Debugging and Profiling Spinnaker Applications Live” at the Spinnaker Summit 2019. In Chuck’s talk, you’ll learn skills like remote JVM debugging, custom profiling builds and the magic of figuring out what’s going on with a multithreaded microservice using htop.

    Chuck’s talk is on Saturday, Nov. 16, at 3:40 PM PT. Spinnaker Summit 2019 is Nov. 15-19 in San Diego.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Chuck Lane. He’s a lead software engineer with Salesforce.com. He works on Sales Cloud, which is the Salesforce you and I use, know and love, or our CRM capabilities. Chuck is talking at the Spinnaker Summit 2019 in San Diego, and his talk is about debugging and profiling Spinnaker applications live. And I think live means, like, live while they’re running, in production, and see what’s going on. We’ll explore what that is. This talk is on Saturday, November 16th, at 3:45 p.m. in San Diego. Chuck, welcome to DevOps Chat.

    Chuck Lane: Thank you. It’s nice to be here.

    Ashley: Excellent. Awesome to have you here. Would you start by just introducing yourself? Tell us a little about you and what you do at Salesforce.

    Lane: Yeah. So, my name is Chuck Lane. I am, as you mentioned, a lead software engineer. And basically, my chief responsibility at Salesforce is to help bring Salesforce into using Spinnaker for our public cloud based deployments. So, as we make a transition to try to move away from first party architecture and move over into the public cloud, one of the technologies that we’re using to do that is Spinnaker and I’ve established myself as a subject matter expert in Spinnaker. And so, I kinda help to bridge the gap between the traditional development and deployment cycle and what that looks like using Spinnaker in a containerized world.

    Ashley: Okay, great. You used a term there I don’t know if I’m familiar with. You mentioned something about moving from first party architecture to the public cloud. What’s that mean about the environment you’ve been in that you’re, and what you’re moving to?

    Lane: Yeah. So, basically, what I mean there is that, historically, Salesforce has owned their own data centers, and we’ve deployed to those data centers. We are moving, embracing in a big way public cloud architecture, whether that’s through AWS, whether that’s through GCP, or any of the number, any other number of cloud offerings.

    And so, as opposed to doing what some people would call a lift and shift where you just take your software that’s meant for, to be run on hardware that you own and move it into the cloud, we’re doing a fundamental re-architecture of the software to make use of all of the great things that cloud architecture allows us to take advantage of.

    Ashley: Mm-hmm. Fantastic. That’s interesting to hear a little bit about your own evolution at Salesforce. So, your talk is about using Spinnaker, you know, really about debugging and finding out problems that are happening in production, but sometimes you might struggle with trying to replicate it into a smaller environment to get the same problems working.

    Can you tell us a little bit about what sort of led you down this path to figure out how to do this? Was it a big problem that was happening or something you saw repeated over and over that there wasn’t a good solution to and you figured out how to do that? How did this come about?

    Lane: Yes. So, basically, as you’re taking workloads and migrating them over, I mean, when—you know, when you just start with a small subset of workloads, you know, the path is relatively straightforward. And, you know—hopefully, anyways—and you don’t run into too many issues that can’t kind of easily be solved.

    But the reality is, as you transfer more and more workloads over to the public cloud, that’s—you know, the devil’s in the details, right? So, that’s really where things can pop up where you’re hitting various limits or software isn’t performing in the way that you would expect it to perform. And these are the situations where we—where it can be very beneficial to jump into production software and really, you know, sometimes slap a debugger on it and see exactly what it is that it’s doing that differs from kinda what you expect. You know, and that can help you kinda tailor your workloads in such a way that you can make things run more smoothly.

    Ashley: Mm-hmm. I know sometimes using debuggers, enabling them—that can be helpful, but it also can be too much information, trying to sort through what all is happening, trying to find what should you be looking at. What are some of the challenges you’ve found by turning on debugging or using debuggers?

    Lane: Yeah. Well, so, I mean, there definitely is that problem that you’re saying just as far as too much information that’s coming out. One of the nice things about Spinnaker and the way the services are written is, you have a very fine grained tuning over which libraries inside of Spinnaker you tell it to print out and debug information.

    Ashley: Mm-hmm.

    Lane: So, if we know that there is a hiccup with one of the data binding layers or a hiccup with one of the authorization layers, then we can, through the config files, we can—and Java settings—we can really target that explicit directory, or I’m sorry, that explicit library and say, “Give me all the debug information that you have.”

    Ashley: Mm-hmm.

    Lane: But, you know, failing that, I mean, there are definitely times when it’s been advantageous and the best course of action is just to go down and actually look at the code and see what it’s doing. And, again there, it’s best to do that under—you can’t do it in production, usually, but what we can do is simulate a load that’s similar to what we would see in production and really, really take a look at what’s going on to help us identify those algorithms that might be o event squared instead of a login or something like that.

    Ashley: Mm-hmm. Now, are there some specific techniques—I know in your description of your talk, when you talked about using remote JVM debugging, custom profile builds, even using htop, you know, a UNIX command or a LINUX command to help you figure out what’s happening with multi-threaded microservices. It sounds like a variety of different approaches that you’ve kind uncovered and learned that you can use to figure out what’s happening.

    Lane: Yeah. And so, I mean, in general, we like to run as closely to the open source build as we can, the ones that are provided by Spinnaker, which is ultimately provided by Google. But there definitely are circumstances where we need an extra tool, be it htop, be it something like Glowroot, which is a Java application profiler where what we’ll do, then, is we’ll go in and we’ll build a custom image using the Spinnaker images as a base and then tack on those additional libraries and put them into the Spinnaker ecosystem and then launch them up just to see what additional information we can get out of there.

    And so, you know, one scenario where that was really useful to us, we ran into a scenario where cloud driver, which is—cloud driver is the main tool that talks back and forth to all of our different cloud infrastructures.

    Ashley: Mm-hmm, mm-hmm.

    Lane: Sometimes calls to cloud driver were taking upwards of, well, two to three minutes to respond. Now, you know, timeouts are set in such a way that if it doesn’t hear anything back in about 30 or 60 seconds, then it just disregards that load and, you know, so that created quite a bit of problems for us.

    By building a custom cloud driver image that had htop in it, we were able to see that the majority of the processes that were running were actually running basically commands to reach out to the Kubernetes clusters and get a list of all of the name spaces that are in the Kubernetes cluster. And, talking it over with some of the Spinnaker developers, what we found is that if you have 15 or so clusters that you’re connecting to Spinnaker, then that’s not so much of a problem to go and query each of those to get the list of name spaces. But as you scale up, on the order of 400 or 500 different clusters the way that we do, then a lot of the delays and a lot of your time can be just essentially Kubectl calls waiting to return to you the lists of name spaces that you need to go and scan.

    So, we implemented a workflow based solution for that, basically, where we let our teams know that when they’re creating their clusters, they should use Terraform to go ahead and create the clusters—I’m sorry, the name spaces that they’ll be using. And then, if they need to use a name space after the fact, then we provide a pipeline that they can use that will dynamically add in an additional name space that their cluster will start to scan. So, that saves us from scanning all the clusters all the time, which was causing quite a bit of performance bottlenecks.

    Ashley: Yeah, I would imagine that would have some overhead, maybe a lot of overhead if you’re doing that frequently. So, it sounds like a way to both reduce overhead, but also to get the information faster.

    Lane: Exactly, exactly—yep, yep.

    Ashley: Mm-hmm. Very cool. So, in your talk, are you going to be doing any demos? I know there’s always the demo Gremlin, or are you gonna be just showing, you know, talking about what some of these techniques are?

    Lane: So, I plan on doing some demos. As far as—

    Ashley: That’s really cool.

    Lane: Yeah. You know, I may do the, what’s the—the Easy Bake Oven a little bit as far as, “Here’s the behavior and here’s what we’ve found in code.” But from what I understand, a lot of that stuff can kinda be dependent on Internet connectivity at the actual site. So, if it’s something where we have good Internet connectivity, then by all means, I plan on walking through a couple of debugging scenarios, as close to what we would do in real time as possible.

    Ashley: Yep. Well, you know, someone somewhere—it’s been a while ago—gave me the great advice of, you know, “Have your live demo and have your disconnected demo ready in the background.” [Laughter] So, you can always at least show something locally if you can’t get on the net. So, if you’re depending upon the network—I’m not sure if you are for your demo, but always a good lesson, right?

    Lane: Sure, exactly.

    Ashley: Great. Are there any other kind of lessons learned, common mistakes, or mistakes that you or others might have made along the way, kinda so there’s hard things you learned by trial and error that you plan on sharing?

    Lane: I mean, well, there are—whew. You know, I’ve been working on Spinnaker for a couple of years. So, there’s definitely been a lot of hard lessons learned, here. But honestly, what I would do is, I would encourage anybody who wants to get into Spinnaker to not just hang around the Slack channel, because the Slack channel does have a tendency to get overrun with, you know, just kinda people posting their stack traces and just saying, “Hey, has anybody ever seen this before?”

    And, you know, I’ve gotten a lot more success by going through the commits, looking at the people who actually authored the code, and then reaching out to them directly with more than, “Hey, can you explain this to me?” but rather, you know, “Hey, I see what you did here and I see what you did here. I’m running into problems with these lines. Do you have a different approach, or is there something that I can be doing differently?”

    And the other thing that I just can’t overstate is the value of being a member of one of the special interest groups.

    Ashley: Hmm, interesting.

    Lane: So—yeah. So, I’m a member of the Kubernetes V2 special interest group that’s lead by Eric Semene and Ethan Rogers from Armory, Eric’s from Google. And it has, you know, it has been just an absolute wealth of information and, you know, honestly, I don’t know if we ever would’ve gotten nearly as far as we had without those two people.

    So, you know, yeah, I would just say that the community is really friendly and we always welcome new members who are ready to learn.

    Ashley: You know, I think both of those are great suggestions, and I really appreciate that you made those. Because, one, things like the Slack channels on projects, open source, those can be a bit intimidating. Sometimes they’re not approachable, because there’s just so much noise so much happening on it and people reaching out for help like you’ve talked about, “Here’s a stack trace I’m trying to figure out.”

    But also, your recommendation of reaching out to the code authors, you know what, it’s—people like to help each other. And, you know, it might seem like, “Hey, the people who wrote this aren’t gonna have time to bother with me”—they love to hear from people that are using their stuff to talk about it and help them out, but also hear about how they’re using it, and of course, they’re always looking for ideas and feedback and stuff like that. It sounds like you’ve had that kind of an experience.

    Lane: Yeah, absolutely. I mean, and you know, the big thing is just, you know, as a coder, it’s easy to tell the people who are coming to you and just kinda wanting you to fix it for them and the people that are coming to you that have really tried to tackle it themselves. And, you know, I can’t speak highly enough about the latter rather than the former. I mean, you know, just give it your best shot and when you get stuck, reach out to somebody and it’s, you know, it can be immensely valuable.

    Ashley: It’s kinda like going to a foreign country. Wherever you are, if you’re American going somewhere else or vice versa, everybody appreciates you trying to speak the native language, and at some point, when they see you struggle enough and how far you can go, they’re glad to help you and, you know, speak in your language.

    So, same kind of thing with helping people with code. If you’re gonna just fob your problem off onto the developer of it—not appreciated so much. But they appreciate that you tried to take it as far as you could. As you mentioned, go look at the code and figure out what’s going on.

    Lane: Yeah.

    Ashley: You’ll get immense respect, you know, even if you aren’t a coder at that level, at that level of software developer, it’ll mean a lot to the developers.

    Lane: Absolutely, absolutely.

    Ashley: Well, hey, I think you’re gonna have a fascinating talk, and I love that, you know, this idea of trying to figure out what’s happening in production and some of the techniques that you’ve come up with and developed and have experienced and the fact that you’re sharing those with others. I’m curious, do you contribute any code to the Spinnaker project in any areas? Are you primarily a practitioner user of it?

    Lane: So, I have. I’ve got a few small RPRs that have been pushed through, but really, the bulk of my commits right now have actually been to the Spinnaker website, the documentation. So, you know, I don’t know Java or Groovy or Kotlin, maybe, as well. Some of the patterns they use are a little bit—I come from a .net world, so they’re a little bit foreign to me.

    But yeah, I’ve definitely written a number of different documentation pages and I found that that’s a great way to kinda get in and I’ve even got some PRs that are coming ups soon that aren’t doc related. So, yeah, hopefully, you’ll see my name more.

    Ashley: Excellent. Well, you know what, documentation is important, too. There’s some folks doing talks at Spinnaker Summit that, you know, kinda ran into that point where, “Hey, there’s documentation for doing it one way, but not under this set of configurations or software.” So, that’s a contribution, too, so congratulations for being a part of the community and for sharing, also, your experience at the Summit.

    So, Chuck, appreciate you being on the podcast today.

    Lane: Oh, thank you. It’s been an absolute privilege. Thank you so much.

    Ashley: Absolutely. Fantastic. I wish you all the best with your talk, I’m sure it’ll be great, and hopefully folks listening to this podcast will draw some more interest and bring folks to listen to you.

    So, I’d like to thank our guest today, Chuck Lane. He’s lead software engineer at Salesforce.com, so you can imagine the environment he’s working in. There’s some super good lessons that Chuck’s bringing to the table. He’s gonna be talking at Spinnaker Summit 2019, which is in San Diego, November 15th through the 19th. His talk is debugging and profiling Spinnaker applications live, and his talk is on Saturday, November 16th at 3:45 p.m.

    I’d also to thank you—you, our listeners—for joining us today. This is Mitch Ashley with devopsy.kinsta.cloud. Have a great day and be careful out there.

    — Mitchell Ashley

  • DevOps Chats: Spinnaker Extensibility, With Armory

    DevOps Chats: Spinnaker Extensibility, With Armory

    Spinnaker Summit 2019 Preview: Most open source software isn’t one size fits all, and Spinnaker is no different. While it captures the most general use cases of software delivery, most organizations find that it lacks some features they need to automate their entire delivery pipeline, such as integrations with internal tooling or compliance systems.

    Out of the box (figuratively), Spinnaker provides Run Job and Webhook stages that teams can use to build custom integrations to help fill any void and enable adoption by a wide range of users.

    Ethan Rogers, staff software engineer with Armory, joins us on DevOps Chat to share a preview of his Spinnaker Summit 2019 talk on Spinnaker extensibility. Ethan shares how Run Job and Webhook stages are used to build custom stages to capture any number of use cases with no code and simple configuration. He also covers situations when it’s necessary to write code.

    Ethan’s talk, “Making Spinnaker Your Own,” is on Saturday, Nov. 16, 3:45 PM, at Spinnaker Summit 2019 in San Diego. Separately, Ethan appears on the “State of the Kubernetes V2 Provider…” panel Friday, Nov. 15 at 2:30 PM.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Ethan Rogers. Ethan is a staff software engineer and also a speaker at Armory. He’s gonna be talking at the upcoming Spinnaker Summit 2019 in San Diego. Now, Ethan’s gonna be talking about making Spinnaker your own, that’s his talk, and then he’s also gonna be on a panel about the state of Kubernetes V2 Provider, plus improved rollout strategies for deploying Kubernetes using Spinnaker.

    So, let’s get started. Ethan, welcome to DevOps Chat.

    Ethan Rogers: Thank you so much for having me. I’m really excited to be here and get to talk about Spinnaker.

    Ashley: Wow. Absolutely. Great topic. I know you’re passionate about it, so fantastic to have you. Would you just introduce yourself? Tell us a little bit about you, what kinda work that you do and a little bit about Armory, the company that you work for?

    Rogers: Yes. So, like Mitch said, my name is Ethan Rogers, I’m a staff software engineer at Armory. I’ve been actually working on the Spinnaker project for about the last three years. I’ve been at Armory for a year and a half, and then a year and a half before that, I was working for my previous company where we were, you know, I was part of a team for delivery and developer experience. And, you know, we were using Kubernetes for this new project that we were working on and looking for a way to deploy and got involved in Spinnaker super early on. So, I’ve kind of been a part of the community ever since then and it’s been a really great experience so far.

    Yeah, so, that is actually kind of how I found Armory. Closer to the end of my time at my last job, I met the founders, DROdio, Ben and Isaac, and they had just—eh, within probably six to nine months started Armory. And we kinda hit it off. I had a really good relationship with the founders and it was kind of an opportunity to go keep working on Spinnaker full time, which is something I was interested in.

    So, Armory, just a little bit about us—we’re bringing Spinnaker to the enterprise. So, what we’re doing is taking open source Spinnaker and, you know, on top of offering stuff like support and training, we also offer layers on top of open source Spinnaker, various different extensions and features that make it more aligned with what the enterprise is looking for today.

    Ashley: Mm-hmm. Very good. Well, that explains, I know there’s yourself, but there’s also some other folks that are speaking at the conference.

    Now, your topic, you gave a nice introduction about how you got involved with Spinnaker—how long ago was that, when you started using it?

    Rogers: Yeah, it was about three years ago, kind of before the—like, right when the first version of the Kubernetes provider landed. I think, at the time, I didn’t realize how early we were. It kind of explains a lot of the PRs I had to submit when I first got involved.

    Ashley: I’ll bet, I’ll bet. Do you write any code, submit any code or are you primarily on the usage end of Spinnaker?

    Rogers: Yeah, so, I’m actually a developer by training. I got my degree in computer science, and transitioned pretty early on in my career into the ops side of things. So, I’m pretty fluent in both. I actually do write a lot of code. These days a lot less, just because of the role I’m in. At Armory, I do a lot more higher level stuff like BitTorrent, but I still try to get a few PRs submitted every week. I lead a couple of initiatives in the Kubernetes provider itself, so I do a little bit of everything.

    Ashley: Well, I won’t ask you which is the dark side, ops or developing—no, I’m just kidding. It’s DevOps, it’s all together now, right?

    Rogers: [Laughter] Yep.

    Ashley: Okay. So, let’s jump into what you’re gonna be talking about. And, of course, we’re not gonna cover it all here, but I know that you’re focusing on kinda how open source, you usually don’t just use it out of the box, if you wanna use that term. I know it’s free and downloaded.

    Rogers: Mm-hmm.

    Ashley: But that you have to do some customizations to kinda get it working in your environment, and we’re talking about Spinnaker here. What’s your experience working with customers? What’s their expectation when they start to use Spinnaker and what do you have to kinda help them get started with?

    Rogers: Yeah. So, the interesting thing about Spinnaker open source is that it’s actually, like, a very small part of what Spinnaker is at Netflix. For anyone listening who doesn’t know, Spinnaker was a project that was born out of Netflix. They built it, it was the next iteration in deployment technology from a project called Asgard.

    Ashley: Mm-hmm.

    Rogers: And one of the driving principles of Spinnaker was making it extendible. And the primary reason for that was because Netflix has a lot of kind of internal workflows and things that they like to—that make their development experience better, that don’t necessarily apply to the broader community.

    And so, when you start using open source Spinnaker kind of out of the box, there are a lot of things that I think most companies want, right? The same reason that Netflix has those, the things that don’t apply to the community but work for them, every other company has something like that.

    Ashley: Mm-hmm.

    Rogers: So, when I was at my previous company, we had this workflow around our Docker images where we would add labels to our Docker images and then we wanted those labels to show up in the Kubernetes manifest where those images were deployed.

    So, that was something that wasn’t available in Spinnaker out of the box, but we were able to use some of the extension methods to actually go grab that information and inject that stuff, which is kind of the basis of the talk I’m giving around Run Job.

    Ashley: Okay, great. Well, let’s explain that, then, because I know you’re gonna be talking about Run Job and about Webhook Stages. What is Run Job, what is Webhook?

    Rogers: Sure, yeah. So, just kind of, at a high level, Run Job and Webhook are ways that you can extend Spinnaker to kind of make it—to do the things that it doesn’t do out of the box that you need it to.

    Ashley: Mm-hmm.

    Rogers: There are more complicated ways to extend Spinnaker that involve actually taking the code and writing Java code, but we found that the majority of users or operators, I should probably say, are not super familiar with Java. So, that’s a pretty—it’s fraught with friction, I would say.

    Ashley: Yeah, it’s a whole other ball of wax, right?

    Rogers: Yeah, exactly. Yeah. So, the Run Job stage and the Webhook stage give kind of a lower friction way of extending Spinnaker. So, the Webhook stage is a stage within Spinnaker that lets you call out to some API. There’s the, you know, I select it as a stage in my pipeline via drop down and I configure it there. There’s also what we call preconfigured Webhooks or custom Webhooks that allow you to actually configure this as part of Spinnaker’s configuration, and then it shows up like it’s a native stage.

    Ashley: Mm-hmm.

    Rogers: So, under the hood, it’s still calling out to APIs. It’s as if you’ve configured it in line, but it shows up and, to the end user, actually looks like it’s part of Spinnaker itself.

    Ashley: Very nice. So, what are some of the most common uses of that to extend integrations into other things? Is it internal third party tools, internal custom software people who built, is there, you know, typically—the typical use cases for this?

    Rogers: I think that we usually end up seeing people integrating more with internal systems than, say, hosted services like GitHub or other—I don’t know, GitHub’s probably the easiest example. New Relic is another one that we see.

    But I think where, in most cases, Spinnaker has integrations for the public stuff out of the box, but obviously, if the tool’s internal to your company, you don’t have that. So, that is usually where we see it. So, I think there was one company that I had talked to that was, they had this internal integration testing suite or load testing suite that they wanted to call out to, and they used the Webhook stage to make that call and kick off those tests.

    Ashley: Mm-hmm. Great, great. Now, I understand you can do a lot of this without writing code, is that correct?

    Rogers: Yeah. Yeah.

    Ashley: So, what does that entail? What kind of a—what’s the configuration process look like to tie, let’s use that example that you gave of someone that wrote their own test framework of some kind that they wanna call out to in a Webhook stage?

    Rogers: Yeah. So, I guess I should probably back up and clarify my answer a little bit.

    Ashley: Okay.

    Rogers: So, at some level, you will need to write code. The question is whether or not you actually have to write that into Spinnaker itself.

    Ashley: Okay.

    Rogers: One of the, I guess we can call it native extensions, which is essentially using Spinnaker as a library and writing Java and doing some spring magic to make it happen.

    Ashley: Mm-hmm.

    Rogers: So, if you were to use a Webhook or a Run Job stage, for that matter, you would have to write a little code. But the benefit of that, if it’s a Webhook the service, whatever service that you’re trying to connect to can be written in whatever language that you want, right? So, if it’s Go, if it’s Python, if it’s Ruby, if it’s—you know, whatever you wanna do, the service is running in the API as the contract that you expect.

    Ashley: Mm-hmm.

    Rogers: And so, with the Webhook stage, all you’re doing is making a call. You know, you could argue that writing a cURL command is writing code, but I think a lot of people are familiar enough with that, that it’s a pretty low overhead.

    Ashley: Mm-hmm.

    Rogers: So, if you can make a call in your terminal, you can make a call from Spinnaker with the Webhook stage.

    The Run Job stage is actually a little more interesting, because what it allows you to do is run a container as if it were a script that you might be running. So, a really common pattern for doing custom logic in CI systems specifically is, I have a container, the container has all my dependencies and it has a script that I need to run. And we just run it like a one shot, you know—once this script finishes, if it fails, the pipeline fails. If it succeeds, the pipeline continues. And then, like, the world is really open, the world is your oyster at that point, because you can write the script in whatever you are most comfortable writing, and then you can run that within Spinnaker.

    One of the more interesting things that we did with the Run Job stage is that you can actually output—you can output these special markers in your logs and Spinnaker will go grab those and inject them back into the pipeline so that you can use them in downstream stages. So, maybe your job is going off and collecting some information for you. You can actually use that in, like, say, your deploy stages or a downstream Run Job stage or Webhook stage as, like, arguments you pass down.

    Ashley: It’s almost used like the log is like an asynchronous message bus, to make it more complicated than it is, but a super simple version of that.

    Rogers: Yeah, yeah.

    Ashley: Some down stage version where you could look at what happened before it and take appropriate action.

    Rogers: Yeah. Yeah, and that’s really—you know, we heard, the motivation kind of behind that feature is, we heard a lot of users, like, you know, they wanna be able to extend Spinnaker, but they don’t wanna write Java code, you know? And containers, for the Run Job stage specifically, containers are obviously super prolific in our industry today. I’ve built my career off of Kubernetes and containers, so.

    Ashley: Mm-hmm.

    Rogers: Containers are kind of like the perfect medium for being able to do this, because you can contain everything that you need—[Laughter] you can contain everything that you need in this container, and then, because Spinnaker can run containers in Kubernetes, it’s kind of like a natural fit, right?

    Ashley: Mm-hmm.

    Rogers: We can kind of take advantage of the fact that it doesn’t matter what your code is, the interface that we need is a container and then the logs, and we can get all of that information out.

    Ashley: I’m curious—this is totally a side question, but did you learn anything in college about containers and that kind of thing in software architectures, or is this something that you really started in after you graduated?

    Rogers: I wish I had learned something in college about this. So, I actually, I graduated college in 2013, and got my start writing PHP and doing a lot more software development. But in the back of my head, I’d always been like—I am very interested in servers.

    Ashley: Mm-hmm.

    Rogers: So, writing the code was interesting to me, but it always seemed like the people with all the power and all the really, people that I looked up to, they were the guys, like, in a shell, deploying software, checking, like, “Is this server overloaded?” It was very much not DevOps, but it was something that really interested me.

    Ashley: Sure, yeah.

    Rogers: And I—you know, I kind of, I spent probably three years as a software developer without having done that. But once I transitioned kind of into this CI/CD space, that really gave me an opportunity to focus more on ops, while doing some development and, you know, we picked up Kubernetes at my last job, and that’s where it all kind of kicked off.

    So, I think we started on Kubernetes 1.4, and that’s really where I learned everything I know about containers and orchestration platforms and stuff like that.

    Ashley: Well, it puts you in a really great position, because you can speak to both sides, both audiences, if you will, you know, developers and operations folks.

    Rogers: Yeah.

    Ashley: And, you know, my experiences before kinda containers and all of this, all the DevOps environment, some of the best developers were engineers who really knew how to write code, but they knew the full stack down to the operating system and the network, right? They kinda knew enough of all of it. Maybe there were really good parts of it, too, besides writing code.

    Rogers: Mm-hmm.

    Ashley: There’s a lot of, I don’t know, power, if you will, in understanding that full stack and how it works.

    Rogers: Yeah. I found that the more you understand about the stack, the better positioned you are to take advantage of it, right, and to actually write better code. You can go back and probably look at the birth of DevOps and see this, but engineers that spend the majority of their time writing code and then throwing it over the wall are more likely to write less performant code, right? Code that is not aware—you know, we wanna write really abstract code, but at some level, you have to be aware of what type of environment you’re running in.

    So, if you’re running on a machine that has 128 cores and hundreds of Gigs of RAM, it doesn’t really matter how inefficient your algorithm is, because it’s gonna run fast. You know, it all depends on scale, but … I’ve found that engineers that understand kind of the cloud and the environments they’re running in and what the actual, you know, where are your network boundaries and what are your actual constraints are writing better code, they’re shipping more frequently, they’re iterating—like I said, they’re iterating more quickly and that all of that is super important.

    I think the other thing that I’ve found is that it’s a growing process, right? I kind of like to look at it as—well, at least in my personal experience, I’ve found that I have a hard time grasping, kind of, the next level of what I’m doing until I’ve fully understood the previous level. And I think that’s really true for a lot of people, right? Until you’ve got a solid grasp of programming, it’s kind of hard for you to imagine what it would be like to understand ops, right? To understand how your program is actually running on servers.

    Ashley: Yeah. I would describe it as kinda context. It helps to know when you understand the environment of what you’re operating in and what you’re doing, what you’re building is going to operate in. And vice versa, right? If you’re in operations or security now, with DevSecOps, understanding now how software is being built. So, it’s the great thing about the accessibility of resources around the cloud and contemporary software architectures and all of that.

    Rogers: Yeah.

    Ashley: Well, we’re running out of time. I wish we had more time to invest in this, but the great thing is that you have your talk coming up, so that’s where you’re really gonna get into more details about that. And Ethan, I thank you for being on the podcast.

    Rogers: Of course, man. Thanks for having me.

    Ashley: Absolutely. Ethan Rogers, staff software engineer and also speaker from Armory, he’s speaking at Spinnaker Summit 2019, which is on November 15th through the 19th in San Diego. Now, he has a topic that we were just talking about, “Making Spinnaker Your Own,” that’s on Saturday, November 16th, 3:45. He’s also on the panel about the state of Kubernetes V2 provider. That is on Friday, the day before—the 15th at 2:30.

    So, thanks to you, all of our listeners, for joining us for this episode. This is Mitch Ashley with devopsy.kinsta.cloud, and you’ve listened to another DevOps Chat podcast. Be careful out there.

    — Mitchell Ashley

  • DevOps Chat: From 0 to 1,000 Deploys Per Month with Redbox

    DevOps Chat: From 0 to 1,000 Deploys Per Month with Redbox

    Many organizations want to implement the latest DevOps/SRE practices, but many struggle with transforming the existing processes over to new streamlined processes. How does an organization transform from zero automated deploys to 100s/1000s a month? Most importantly, how do you show the value of Spinnaker to your business as a whole?

    November’s Spinnaker Summit 2019 in San Diego allows us an opportunity to hear and learn from DevOps engineers, developers and practitioners using Spinnaker open source software. Joel Vasallo, manager of Cloud DevOps at Redbox, shares a preview of his talk where he shares his experience about using Spinnaker open source to increase the number of deploys into production. Joel found Spinnaker so useful because it is very extensible, covers AWS well, strong support for Kubernetes, different deployment types and has the automation to create the software delivery platform that fits the need.

    Joel’s talk, Transforming Software Delivery using Spinnaker, is on November 17, 12:30 pm PT at Spinnaker Summit 2019 in San Diego.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Joel Vasallo who is manager of Cloud DevOps at Redbox. We know Redbox is the movies, the kiosk, the online service—great.

    Our topic today is going from zero to thousands of deploys a month. Boy, if you’re not doing a thousand a month already, that sounds like a great place to be. This is actually a preview of a talk Joel is doing at the Spinnaker Summit on “Transforming Software Delivery Using Spinnaker.” It’s 12:30 p.m. Sunday, November 17th in San Diego. Joe, welcome to DevOps Chat.

    Joel Vasallo: Hey, thanks so much for having me, Mitch. I’m glad to be here.

    Ashley: Glad to have you here. Well, tell us a little bit about yourself, what you do at Redbox and also, hopefully people know Redbox, but just a bit about Redbox.

    Vasallo: Yeah, no, my name is Joel Vasallo, I’m the Manager of Cloud DevOps at Redbox involved with software delivery to the cloud. In my spare time, I’m a GDG organizer, so a Google Developers Group here in Chicago. But about Redbox, I mean, you said a lot of people know what Redbox is. We’re the kiosk that’s usually found in your grocery store. So, we do DVDs and Blu-rays, but we also, a little known fact is, we do have an on demand platform, so video on demand, and that’s kind of our new thing that we’re doing right now. And there’s actually a lot of motivation around building fast deployments, and that’s kind of gonna be my drive for the talk.

    Ashley: Awesome. Now, just a question for you for cloud DevOps—does cloud actually include software that’s being delivered or information being delivered out to those kiosks, too, or is this really for the on demand platform in the cloud that you—

    Vasallo: Yeah, there’s some aspects of it as well. I mean, there’s a lot of APIs in use across the company, so there are some that do kind of traverse the cloud in that regard—yeah.

    Ashley: The only thing better than APIs are common APIs, right?

    Vasallo: [Laughter] Exactly.

    Ashley: Reuse.

    Vasallo: Yeah, especially if you have a solid delivery platform—of course.

    Ashley: Absolutely delivery platforms. Okay, so, let’s jump into it. We’re giving a preview of your talk, so there’ll be a lot more content, of course, in your actual talk. Tell us a little bit about, if you’re already achieved this thousands of deploys, you’ve been at this for a while. Tell us a little bit about how you got started on this journey.

    Vasallo: Yeah. You know, to preface a bit, I mean, I’ve been in the Spinnaker community since, like, late 2015, when it was still kind of an early release out of the open—in the open source community. And at a previous role at Gogo, I used to work at Gogo and we implemented Spinnaker basically from day one. And prior to that, we were using some other open source—Netflix open source tooling as well.

    But in that journey, we were going from a traditional monolithic, you know, to a microservices architecture, and the goal was to essentially build a way to deploy to the cloud as fast as possible, right? And just taking and building on those practices, you know, applying all the things, the same—the 12 factor concepts, immutable infrastructure, routing, all that stuff.

    It kind of transformed into some open source tooling that we’ve written in my history, and there’s also some practices when using Spinnaker to make such a big change in an organization, right? I mean, the whole topic of DevOps in an organization, there’s so many great talks out there, right?

    Ashley: Mm-hmm.

    Vasallo: And that’s definitely one of the—I guess, arguably, the hardest bit is, “Well, what does the DevOps mean for you?” Right? That’s the typical phrase you hear when people say, “Well, we’re DevOps.” “Well, what does that mean?” Right?

    Ashley: Well, now, you’re talking, obviously, at the Spinnaker Summit, and it sounds like you’re pretty committed to the product. This isn’t a podcast about that specifically, but you sound pretty dedicated to it. Why Spinnaker?

    Vasallo: You know, it’s—the core fact is, it kind of extends a little bit past just deployments. It has—it’s really, the biggest thing is, it’s so extensible. Right out of the box, you’re just coming out swinging. The fact that Netflix, Google, Microsoft, a lot of big names—Amazon as well, how could I forget?—are contributing and helping build this product up, you have some pretty big names out there, right? And why not lean on the shoulders of giants. They’ve seen, I would argue, probably one of the bigger challenges in terms of software delivery, especially at an in demand space such as, you know, for example, Netflix, right?

    Ashley: Mm-hmm.

    Vasallo: And building on that success, you know, you basically get those batteries included, right? You get some battle tested software, and I think that’s one of the biggest things.

    But in addition to that, I mean, it’s extensible. It covers Amazon very well, it covers Kubernetes, it has support for various deployment types and various customizations as well, some of which are customizations you need to build, and some customizations are actually built into the platform. And the best part, it’s open source, so that’s my favorite thing.

    Ashley: One of the key words I didn’t mention that I know your talk’s gonna be about is automated deploys, right? You’re not doing hundreds of thousands with a lot of manual process. Talk a little bit about the key of automation and you kind of assume a development team is gonna be all about—yeah, let’s automate this and make our lives easier, but maybe it’s not that easy to get from point A to B where you do have that automated. Talk about that part of the journey.

    Vasallo: Yeah, no, I agree with you. And I mean, it’s an unfortunate thing in the landscape right now where a lot of companies are rushing into the DevOps landscape, right, and they just say Jenkins, right—Travis CI, GitLab, whatever. And, you know, you’re essentially flooding tools on the market, right? And it’s very tough.

    But having a lot of tools doesn’t mean you’re DevOps, right? There’s some fundamental business changes that have to occur in order to achieve that delivery that we’re talking about, right?

    Ashley: Mm-hmm, yeah.

    Vasallo: So, with that, it’s kind of taking—how do I call it?—the little infinity loop of DevOps, right? Where that’s great and, you know, the plan, code, build, release—you know, that little cycle, it’s great, but you can’t use that as a guiding principle. There’s not enough detail, right? And then you can have the extreme on the other side, where it’s like the Candyland, I like to call it, where it’s like the Candyland of ITIL process where you’re just essentially passing Go and then passing down a ladder, if conditions. That’s too complex, and that’s hard to explain. So, there has to be something kind of in between.

    And that’s kind of what I’m gonna be focusing on in my talk and really asking the questions of how do you lay out an effective delivery process? What are some steps—what are the steps at your organization to get changes out the door? Because it’s a different answer for every company, right? There’s no one truth out there, unfortunate as it sounds.

    Ashley: Mm-hmm. Well, and everybody’s environment is unique. Maybe some of the problems are the same, but there’s also, you know, what is unique about your organization, whether it’s the business technology or just people, the team you’re working with has strengths in certain areas.

    Vasallo: Yeah, of course. And that’s something that I kind of build on in the talk is about building minimum viable products, right? And the argument is, many minimum viable pipelines, right? Really laying out, end to end, what needs to be done to get software delivery to occur. And the best way to start is really to look at what’s kinda currently happening, right? Lay out, you know, change has to come in, goes through phase two, phase three, phase four, and then ultimately phase five—profit, right?

    Ashley: Mm-hmm.

    Vasallo: Well, what are those phases and how much investment should be done in each spot, right? You can spend your time automating, I don’t know, some aspects of your pipeline, but what’s the return on investment, right? Would it have been better served in, for example, automating quality versus automating security scans, right? What’s gonna be the most immediate value in shaving time and getting you back from, you know, these long, drawn out pipelines to potentially fast and efficient ones, right?

    Ashley: It’s interesting. What was sort of the starting problem that took you down this path? Was the business asking you, “Hey, we need stuff faster, faster, faster”?

    Vasallo: Yeah.

    Ashley: Or was it just your own, “We don’t have resources, so let’s figure out how to automate it”? I mean, what kinda got you going on this?

    Vasallo: Yeah. So, a little bit about myself, I joined Redbox back in 2018 with the mission of building a software delivery pipeline to get software—me and my team to get software into the cloud. And with that, there’s also the transformation aspect of, “Well, what is the cloud? How do we integrate with the cloud?” And a lot of questions kinda came up.

    Fortunately, we took a big investment in doing a cloud native approach, you know, from day one. Which means no lift and shift, which is awesome. It’s arguably the hardest path, [Laughter] as we probably all know, but this means essentially from the ground up, building a software delivery platform, right, that’s integrated from the, all the way down to the development layer and all the way up to the automation layer. So, we’ve been very fortunate to work on something like that.

    That’s kind of where we started, and that was the motivation of building a truly cloud native pipeline, so, yeah, working on that stuff.

    Ashley: Now, you mentioned you’ve developed some tools, too, and with Spinnaker being open source, have you contributed a lot back to the community yourself, or are you more of a consumer of it? You know, everybody plays a different role, and not everybody’s, you know, having an open source project.

    Vasallo: I’ll selfishly say that I’m more of a consumer of it.

    Ashley: That’s okay.

    Vasallo: I would love to contribute more to it. I’ve done a few contributions in my past, but in recent times, I haven’t.

    With that being said, though, my team is empowered to make change and drive changes, so we do have one or two changes in flight, and one of them actually got merged this past week. So, we’re looking to help grow and not just be consumers. We’re hoping to also build and solve the same problems we’re seeing, maybe, for other people as well.

    Ashley: Well, good. That’s great if you get a chance to do that. Keep in mind, you know, exercising it, especially at the volume you’re talking about, that is super valuable, too. That’s what helps improve software, whether it’s open source or paid for. So, that’s another—

    Vasallo: Well, yeah, and I mean, it’s kinda like what you’re saying. If you’re not paying for the software, maybe you can pay for it within support in the sense that Spinnaker’s community is an open community. There’s a Slack channel out there and you can join—anyone’s more than welcome to join, ask questions, and get answers from potentially some high up people. And Netflix, even people in the community in general, right?

    We’re all on the same journey, and I mean, that’s arguably what kinda kept me involved in the Spinnaker community is just seeing such an open and collaborative group of individuals just willing to help each other—special interest groups every week or so meeting and just saying, “Hey, what’s the direction of the project? What else can we do? What are some challenges we’re seeing?” It’s actually been pretty fun.

    Ashley: What are some of the objections you typically see or maybe that you even ran into about automating and doing deploys at that volume?

    Vasallo: Yeah. So, it’s not like we’re doing 1,000 more changes a month, right? It’s arguably, running fast can initially be seen as a reckless approach, right? You know, if you said, “Hey, we’re averaging 10 deployments a month”—maybe that’s a standard organization—“to 1,000,” you’re gonna definitely open up some eyeballs, right? People will be like, “Wait, wait, wait. What do we need that for?”

    Ashley: “What about stability? What about up time? What about”—

    Vasallo: Exactly, exactly. But the argument is, it could be said that smaller changes typically result in less down time, right? If you can break down a purchase path change in some company, right, and maybe saying, “Hey, we’re gonna make a UI change and then we’re gonna do a back end change that’s backwards compatible and we’re gonna do, then, an API tier change to support that change such that everything hooks together,” you then broke up one large change of three major components into three separate small, individual pieces that can be rolled out safely, right? And tested and embedded as well. That’s the most important part. Because you’re only as good as your automation and testing as well.

    Ashley: Now, did you do much shift left in terms of doing more security testing or vulnerability testing earlier in the Dev cycle, too, as part of this?

    Vasallo: Well, so, that’s the beauty of it. I mean, essentially, the pipeline doesn’t, at least for us, doesn’t start at Jenkins or the build time. It starts all the way at the commit. So, we have a lot of tooling, at least at Redbox, that’s built to empower developers. The thing to also note about those 1,000 deployments—those deployments aren’t, you know, some Ops team clicking an OK button. They are developers promoting their own code, and they are empowered to write and deploy code to production when they feel it’s appropriate to release, right?

    But in doing that, we don’t just give them the keys to the city, right? We build a nice, safe, and reliable pipeline—essentially, a paved road, if you will. That’s the kind of term I’ve been hearing it described where, you know, you can ride the paved road and have fun. You can get off road and it’ll be bumpy, but the majority of the time, the paved road will get you to production as fast as possible.

    Ashley: Right. Yeah, that paved road concept comes from the Netflix paved road talk that they gave. I think that was OSCON or something.

    Vasallo: Yeah, there are definitely a lot of great things from the engineers there, a lot of inspirational things kinda came out of it. I mean, again, it’s building on the shoulders of these giants, right? They’ve seen, arguable, large percents of Internet traffic, right? And at that level of success, you know that some of those concepts definitely can be applied elsewhere, right?

    Ashley: Mm-hmm. Well, you know, one of the things that’s maybe a little bit misleading in saying automated deploys is, it isn’t just the deployment process that’s automated, is it? Talk about what all the parts that are automated to be able to deploy at that volume.

    Vasallo: Yeah, no—definitely. I mean, it’s essentially part of the—part of the talk is also gonna, the little secret of it is, it’s not like you can roll in Spinnaker and have an automated deploy and you’ll be good, right? There’s a lot of aspects of a pipeline, and as I was alluding to from a commit layer, right, whether you do your static analysis for security early on, whether you do your unit tests at build time, whether you have an automated way of storing in archiving artifacts, right? You need to know what is making it out to your environment stage prod.

    In addition to that, you need to build that infrastructure automatically, right? So, we have things—you know, you can use things like Terraform and CloudFormation as well, right?

    Ashley: Mm-hmm.

    Vasallo: But your load balancers, your security groups—again, we’re talking within the context of AWS—how do you reliably and repeatedly build your infrastructure in such a way that it can support that rapid deployment, right? You can’t just have an automated pipeline where it’s like, “Okay, hold the lights, let’s go call up somebody and say, ‘Hey, we need a load balancer. Can we get that done?’ ‘Sure, give me five minutes.’” Even five minutes is too slow, right? We need to have an automated way of creating and also getting fast feedback to our developers when these changes work and when they don’t work, right, so they can help triage and figure out what’s going on.

    Ashley: It’s almost like I’ve said before—DevOps is an overnight success 35 years in the making. [Laughter]

    Vasallo: [Laughter] That’s a good one, that’s really good.

    Ashley: Where you kinda get there one step at a time, but it is, and I think that’ll be something interesting to anticipate about your talk is really thinking about it holistically and what that entire process—not that you’re gonna go into every bit of detail in that amount of time, but it is about all of that and connecting the toolchain, the work flow, and the process, people process part of it into something that can be automated and happen at high scale.

    Vasallo: Yeah, that’s definitely something to be said there. I mean, a lot of—it’s generally a soft topic, but it’s arguably one of the more important ones is, there has to be a cultural transformation and understanding if you truly wanna be as successful as some of these giants, right?

    Building the concepts of open culture, right, working together, cross functional teams—there’s no magical fix out there in the world to fix a broken process in a broken culture, right? And that’s the toughest thing. And then that’s something, definitely, we touch on as well. And I alluded to it earlier with the concepts of, for us, that’s empowering our developers. So, building compelling tools for our developers to use every day they come to work.

    Ashley: Well, you mentioned keys to the kingdom for the developers. I can’t imagine you did all this without some form of senior leadership backing. How did you get that and where did you get it from to go down this path?

    Vasallo: Yeah. So, it definitely started very grassroots, right? Say the challenge to get to the cloud, right? That’s an easy challenge and I think a lot of—sorry, that’s an easy statement to say, but what does that mean? And for every organization, it’s different, right? For some, lift and shift is the most appropriate solution if you’re not willing to invest in the cultural changes, right?

    But for me, at least in my experience, it’s always been finding the hero in the regards of not necessarily a hero mentality, but finding that one champion who can help drive that, you know? That team who is under a tight deadline and you’re like, “Hey, you know, you’re a relatively new project. You’ve never really seen—you’ve never really gotten deployed. How about we just take an experiment and see if we can automate your app end to end, right?”

    Ashley: Mm-hmm, mm-hmm.

    Vasallo: The good news is, it’s a pretty low risk, it’s a new feature. The timeline, obviously, is always a concern, right, in terms of meeting business deliverables. But in doing that, working with those individuals, you not only can get fast feedback as to the proposal of the pipeline, you can also build a relationship that essentially is like DevOps and Devs working together to kind of create this delivery solution, if you will.

    Ashley: Hmm. Well, I feel like we just scratched the surface, which is probably a good thing, because it’s the preview of your talk. I appreciate you coming on the podcast today, Joel.

    Vasallo: No—no worries. Hey, thanks so much for having me. Again, I hope to see everyone at the Spinnaker Summit. It’s gonna be a great time, so I hope to see everyone there.

    Ashley: It is. Well, definitely, I’d like to thank Joel Vasallo, Manager of Cloud DevOps at Redbox for joining us today. Again, Joel’s talk at Spinnaker Summit, which is at 12:30 p.m. Sunday, November 17th in San Francisco, it’s titled “Transforming Software Delivery using Spinnaker,” so be sure and check that out if you’re making it. Now you have a reason to go to San Diego in the fall coming on the winter.

    Vasallo: Other than the great tacos, yeah. And everyone, join us on the Spinnaker Slack, we’re happy to help out and answer any questions as well. We’re more than a welcoming community, so hope to see more folks there.

    Ashley: That’s a great point. It’s a very vibrant community. So, we wish you all the best in your talk, and everyone, you’ve listened to another DevOps Chat podcast. I’d like to thank you, our listeners, for joining us today. This is Mitch Ashley with devopsy.kinsta.cloud and you’ve listened to another DevOps Chat. Be careful out there.

    — Mitchell Ashley

  • DevOps Chat: CI/CD Velocity for Large Monolithic Services with Pinterest

    DevOps Chat: CI/CD Velocity for Large Monolithic Services with Pinterest

    Spinnaker Summit 2019 Preview: Software Engineer Rainie Li played an essential role in implementing Spinnaker as part of Pinterest’s CI/CD pipeline. The results moved Pinterest from two scheduled deployments per day to continuous deployments, greater than 15 during business hours.

    This episode of DevOps Chats features a preview of Rainie’s talk, “How we introduced CI/CD for Pinterest’s largest monolith services (API and Web) to improve developer velocity, quality & reliability (Pinterest).” Topics including how Spinnaker was selected, important metrics, Pinterest’s future CI/CD platform Hermez, Canary analysis and lessons learned from the journey are shared.

    Rainie’s talk is on Sunday, November 17th, 1:30 pm PT, at Spinnaker Summit 2019 in San Diego. Joining Rainie on the talk is Jasmine Qin, software engineer with Pinterest.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Rainie Li, who’s a Software Engineer at Pinterest.

    Now, Rainie’s gonna be talking at the Spinnaker Summit 2019 coming up in San Diego, November 15th through the 19th, and her topic she’s presenting on is how Pinterest implemented CI/CD pipeline for large, monolithic, or monolith services, APIs and web, focused around helping them improve developer velocity, quality, or liability, et cetera. So, some real hands on experience that she’s planning on sharing. And she’ll also have a co-presenter, Jasmine Qin, and she’s gonna be joining her, too, at that presentation.

    Well, Rainie, welcome to DevOps Chat.

    Rainie Li: Thank you, Mitch.

    Ashley: Awesome to have you on. Thanks for joining us. Would you just introduce yourself, tell folks a little bit about you, the kind of development work that you do at Pinterest and also maybe a little bit of what you’ve done in your past to become a software engineer?

    Li: Yeah, sure. So, hi, everyone, I’m Rainie Li, I’m working at Pinterest infraorganization and our team is focused on doing continuous delivery platform for all the internal engineers. Before I joined Pinterest, I worked at both Microsoft and Amazon, also as a software engineer, and I found I have a very strong interest in infrasite, and it turns out, continuous delivery is a very important feature for software engineers. So, that’s why I ended up here.

    Ashley: Very interesting. Well, you’ve worked at some very large online services. I know that Pinterest has, what, about 250 million users, and of course, Microsoft and AWS are very large organizations, so you’re kinda used to working on some pretty big infrastructure projects, it sounds like.

    Li: Yeah, I am. [Laughter]

    Ashley: Good, good! Well, great experience. So, tell us a little bit about your talk. I guess maybe start with, you know, you came to Pinterest, maybe about where they were in this process of implementing continuous delivery and where you kinda picked up in the process. Were you at the beginning or they already started?

    Li: Yeah. I kinda joined the one that already started, but I still drive the whole production readiness for Spinnaker at Pinterest. I also drive the design review for Spinnaker as well. So, I kinda joined the one that already picked Spinnaker for Pinterest, but I’m the main person to continue to deliver this project.

    Ashley: Oh, so you kind of picked the project already in progress and then take it forward from there as one of the leads on it?

    Li: Yeah.

    Ashley: Very nice. So, when you said—so, you do a review of Spinnaker. Is that your internal usage and changes in implementation to it, or do you also contribute things back to the open source? Any of that kind of thing?

    Li: Currently, we only reveal how we use it internally and we do some customization for Pinterest only. We haven’t contributed to the upstream yet, but I think long term plan, we would like to contribute to upstream.

    Ashley: Yeah. Well, you know, just using it at the volume that you do, I’m sure, is a way of, a form of contributing back, too. It’s gotta certainly exercise Spinnaker and a lot of other tools about that, too.

    So, tell us about—so, you picked up when they had already gone through the implementation and you’re kinda furthering it, or were they still in the process of implementing CI/CD continuous delivery?

    Li: I picked up when they were still in the process of implementing a CI/CD platform.

    Ashley: Mm-hmm.

    Li: But they kind of already decided to use Spinnaker, but we still have to do Pinterest specific customization and how to make it production ready at Pinterest, that kind of work.

    Ashley: Ah, excellent. So, sounds like you had quite a bit to do with Spinnaker even when  you joined of getting it ready for production. Why don’t you talk about some of the things—I know you’re gonna talk about this in your Spinnaker Summit talk. What are some of the things that you need to do to Spinnaker to get it production ready for that kind of an environment that you’re in?

    Li: So, I think there are several key pieces. The first one is, we have to make sure authentication and authorization are working as expected for Spinnaker, because that’s the most important thing—like, we have to make sure users can go over our OS process before they can—

    Ashley: Your OS, mm-hmm.

    Li: Yeah. The second key piece is monitoring, which includes metrics and alerts. So, basically, we exposed all the Spinnaker service metrics and created a dashboard for those alerts, which we can get paged when there’s some issue that happened if Spinnaker is down or something like that. These are the two most important things for production readiness, I think, for us.

    Ashley: That’s extremely important when you’re automating something like a continuous delivery, continuous integration. It’s automated, but you have to know is it working right?

    Li: Yeah.

    Ashley: Are things happening and are there problems or are you meeting the kind of metrics that you were expecting?

    Li: Yeah.

    Ashley: I know something you mentioned in the description of your talk is what kind of metrics are important to measure. Can you say a little bit about what are some of the metrics that you’ve learned, both while you’re implementing it and now that you’ve had it in production that are really important to watch?

    Li: Yeah, definitely, those metrics can—like, latency and ____ P90 and also, like, those SLA related metrics from all Spinnaker components are very important. I think especially the gate service, which is Spinnaker API Gateway, it’s like the entry point for the rest of Spinnaker components, the metrics for Gateway is extremely important.

    Ashley: That’s kinda the integration component of Spinnaker, right?

    Li: Yeah.

    Ashley: Everything talks to it. It’s sort of that—I don’t know if it’s a central hub, but it’s certainly where all the APIs tie in together.

    Li: Mm-hmm.

    Ashley: I’m sure if you’re having a problem with that, then you’re gonna know [Laughter] there are probably lots of failures happening in the system.

    Li: Yeah.

    Ashley: Do you look at, do you also measure how many builds you’re doing per hour, per day, or how many deliveries into production you’re doing? Is that metrics that you capture through Spinnaker or you do that elsewhere?

    Li: That’s not the metrics that we are capturing in Spinnaker, but we do measure from the pipeline execution history.

    Ashley: Mm-hmm.

    Li: So, currently, we are using Spinnaker to do the continuous delivery for two major services in Pinterest. We finished around 15 deploys per day for these two major systems and each deploy contains roughly 10 commits, and we think this pipeline, it increased our productivity a lot, because previously, we have to do manual deploy.

    Ashley: Mm-hmm.

    Li: Yeah.

    Ashley: Do you also have, then—I know this is outside of Spinnaker, probably, but do you have most or all of your testing automated as well, or are there still some manual steps for testing before it goes into a final deployment?

    Li: You mean testing for the service itself, or testing for Spinnaker service?

    Ashley: Testing more of the software application part of it.

    Li: I see. Yeah, so, currently, we are using a separate stage to run integration test jobs in Jenkins. That’s how we do the testing step. It’s kind of automated already, like, we don’t have to interact with humans to trigger the build or humans to run the test. We don’t need that. It’s just a stage in the Spinnaker pipeline. We run the interpretation test job in Jenkins, yeah.

    Ashley: Mm-hmm, excellent. Okay, good. We have kind of a feel for your workflow, your tool pipelines. It sounds like it’s very automated.

    Li: Mm-hmm.

    Ashley: So, going back to the topic of implementing something at that kind of a scale, both, you’re a very large organization and you’re doing a number of deployments per day. I know there’s some things in Spinnaker—well, you have a platform called Hermes. And is that your own central workflow engine, or is that part of Spinnaker?

    Li: Oh, so, Hermes is our future CI/CD platform. So, we are in the, like, developing stage. Spinnaker will integrate with Hermes and now we will use Spinnaker as a back end workflow engine.

    Ashley: Hmm, okay.

    Li: Yeah.

    Ashley: So, that’s really your—that’s Pinterest’s own central workflow engine and you’re integrating Spinnaker into it as part of that feature?

    Li: Yeah, yeah.

    Ashley: Pipeline—tool pipeline, if you will. Okay.

    Li: Yeah, it’s like a CI/CD platform for Pinterest, and we are going to open source Hermes as well in the future.

    Ashley: Oh! Well, that’s good news. I’ll look forward to that.

    Li: Yeah. [Laughter]

    Ashley: Do you know kinda time frame when that’s gonna be happening, or is that a little farther in the future, don’t know yet?

    Li: I’m not sure the exact timeline, but somewhere early next year or at the end of this year.

    Ashley: Okay. Well, that’s not too far away. That’s great. We’ll look forward and thank you for contributing that as open source. So, talk some more about what are some of the lessons that you learned about going from where you joined Pinterest and where things were with Spinnaker’s implementation, maybe some things you had to do, some of the things that you learned? Here’s a good way to do it, here’s a mistake I made and I fixed it by taking a different approach—what are some of those lessons learned?

    Li: Hmm, I think the lessons for me is, because we deploy Spinnaker services to Kubernetes platform and I never used a Kubernetes platform before I joined Pinterest, so that’s the biggest lesson I learned here, like, how to set up all the Spinnaker components in Kubernetes platform, yeah.

    Ashley: Uh huh, good. Are there—so, learning how to do that was important. Were there any specific lessons that you learned about doing that, or is it just really just kinda learning the how of doing it?

    Li: A specific lesson I think is some special situations where the Kubernetes platform decides to rotate cluster and do some platform testing. On the Spinnaker service side, we have to handle this kind of scenario well instead of bringing down the sides or have some happy customer user experience. That’s a lesson I learned from there.

    Ashley: Okay. I think you also mentioned in your show notes or your description in your talk that you did some canary analysis as part of that as well?

    Li: Mm-hmm.

    Ashley: I assume that was a new thing for you, or have you done that before?

    Li: Yeah, that’s also a new thing, but we have a separate team which is the configuration team which, they are mainly implementing this feature. So, for Pinterest, we support OpenTSDB as a metric, but I think Spinnaker canary analysis can only support Prometheus or Datadog, those metrics. So, I think the ____ team integrated this OpenTSDB into Spinnaker canary analysis component so that we can provide Pinterest ____ on Spinnaker UI.

    Ashley: Ah, interesting.

    Li: Yeah.

    Ashley: Okay, great. What are some other things that you’re planning on talking about at your talk during Spinnaker Summit?

    Li: I think how we use Spinnaker and how we customize Spinnaker at Pinterest and the lessons we have learned when we adopted Spinnaker here. I think these are the main topics I’m going to talk about, yeah.

    Ashley: Mm-hmm. Great. Well, I know you weren’t at Pinterest when they decided to use Spinnaker for this project or for part of the toolset. But based on what you’ve seen and what you’ve learned, do you think Spinnaker accomplished the goals that they had for why they chose it?

    Li: Definitely. Yeah, we like Spinnaker a lot. [Laughter]

    Ashley: What were some of those goals, do you recall? Were there certain, we wanted to get to a certain number of deployments per day, or were there other things that they were trying to achieve that were looking to see how you’re doing?

    Li: I see. So, we don’t have a specific number for the goals, but Spinnaker definitely helped us to use pipeline for deployment. Without Spinnaker, we have to go to each stage, do manual collect deployment one by one, which is not very efficient. And currently, we can use Spinnaker to manage deployments in a consistent and a repeatable way. We really like pipeline, which can provide a sequence of deployment stages. We don’t need too much manual interaction, and—yeah, that’s, I think, the goal we adopted Spinnaker is, we need the pipeline feature and I think it works well for us. Also, the canary analysis report is very useful for us as well.

    Ashley: Yeah, I know a lot of people are very excited about using that.

    Li: Yeah.

    Ashley: And you mentioned that you integrated OpenTSDB as part of your observ—I can’t say it [Laughter]—observability stack, there we go, got that out. Now, I think you also have a deployment system there that Pinterest uses to deploy into AWS, correct?

    Li: Yeah. We have a deployment system which is also open source called Teletron, which we used for four years to deploy to AWS VM. But this deployment system does not provide a good pipeline, so that’s why we integrated with Spinnaker to have the nice pipeline feature, yeah.

    Ashley: Excellent. Well, you’ve sure done a lot of work. What’s kinda next? What’s the next set of projects that you’re thinking about or you’re currently working on now that you’re at this point with deploying Spinnaker into your CI/CD pipeline?

    Li: I think the next project is, our team will implement Hermes at the Pinterest main CI/CD platform. Spinnaker was working at the back end workflow engine, and we were also working on migration, like most of the services from AWS and VM to Kubernetes. Yeah, that’s two major projects that we are going to be working on.

    Ashley: Ah. Well, certainly, the move to Kubernetes, I’m sure that’s a pretty substantial project, a pretty big one.

    Li: Yeah, yeah. [Laughter]

    Ashley: Well, good. Well, thank you so much for joining us on the podcast today. This is very interesting and I think you’ve got a really compelling and interesting talk. I’m looking forward to hearing more about it and have others to get a chance to attend.

    Li: Yeah, thanks for inviting me, Mitch. I’m glad to introduce Pinterest using Spinnaker at the Summit and I’m very happy to talk about more there.

    Ashley: Absolutely. Well, thanks for contributing with your talk. So, you’ve listened to another DevOps Chat podcast. I’d like to thank my guest today, Rainie Li, who is a Software Engineer at Pinterest. Again, her talk during Spinnaker Summit is on Sunday, November 16th at 1:30 p.m. This is according to the agenda currently on the website, you can look there for updates for the Spinnaker Summit site, and her talk, again, is on how we introduced CI/CD for Pinterest’s large monolith services, API and web, to improve developer velocity, quality, and reliability. And as you’ve just heard, there’s a lot of information that Rainie’s gonna be sharing with you.

    So, thanks for joining us, Rainie, and thank you, of course, listeners, for joining us here today. This is Mitch Ashley with devopsy.kinsta.cloud, and you’ve listened to another DevOps Chat podcast. Be careful out there.

    — Mitchell Ashley

  • DevOps Chat: Monitoring Spinnaker on GKE with Miles Matthias

    DevOps Chat: Monitoring Spinnaker on GKE with Miles Matthias

    On a project to move one of Google’s Fortune 500 customers to Google Cloud, Google Kubernetes Engine (GKE) and Spinnaker open source, our DevOps Chat guest ran into the message “hang tight.” No scripts or documentation. Not to be delayed, Miles Matthias, Google Cloud Consultant with Container Heroes, filled the gap and contributed his work back to the Spinnaker community.

    This episode of DevOps Chats features a preview of Mile’s talk, “Monitoring Spinnaker with Prometheus Operator on GKE” that he will be giving on Saturday, November 16th, 3:45 pm PT, at Spinnaker Summit 2019. Miles also talks quite a bit about Canary testing in this episode.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Miles Matthias, he’s a Google Cloud consultant and he’s with Container Heroes—great company name. Our topic today is actually a preview of his talk at the Spinnaker Summit 2019. His topic is gonna be Monitoring Spinnaker with Prometheus Operator in a GKE Environment, Google Kubernetes Engine. His talk is on Saturday, November 16th at 3:45. Miles, welcome to DevOps Chat.

    Matthias: Hey, Mitch. Great to be here. I’m really excited about the Spinnaker Summit.

    Ashley: Well, tell us a little bit about you, what you do and a little bit about Container Heroes.

    Matthias: Yeah, sure. So, like you said, at Crowd Consultant, I’ve done a whole bunch of software development and infrastructure setup and architecture design throughout my career and now I’m helping other companies set up their cloud infrastructure, make migrations, modernize their application development and all sorts of things like that. And I’m a partner at ContainerHeroes.com, and we’re a group of kind of people like me that have been in previous startups, CTOs, who’ve done the whole PC route, we’ve built a bunch of custom applications for clients throughout out career and now we enjoy consulting people that are using the cloud.

    For the past year or so, especially I guess the last nine months or so, one of our big clients, we’ve been helping Google with one of their new customers that was complete on prem to cloud, to Kubernetes containerization migration and they utilized Spinnaker. So, I kinda was brought in as the Spinnaker expert to help them get it up and running, to help customize what they needed in their environment and contributed as much as they could back to the open source, including some stuff that I built for using Prometheus Operator, which we can touch on a bit. And so much so that then the team invited me to give a talk at the conference, so I’m excited about it.

    Ashley: Sounds like a very—very relevant talk, especially sharing your experience from working with this Google customer.

    Matthias: Yeah.

    Ashley: So, tell us a little bit about when you came to this project and first started working with the customer. Had you worked with Spinnaker before, was that a new technology, or were any of these new technologies, or you’ve kinda done all this before?

    Matthias: So, I’ve done a bunch of CI/CD before. Spinnaker is still in its early days, so I have very limited experience with Spinnaker as an application. It’s still—yeah, it’s still very nascent on the development scene. The entire kinda concept of continuous delivery is still very new in the industry, even though it’s been around for a little while.

    And so, CI has been very well developed, everybody knows Jenkins, everybody has ____, a million CI tools, content of continuous delivery and having more thought out, sophisticated delivery strategies that can help you do more automated continuous delivery like canary analysis and different deployment types and automated rollbacks and all these sorts of things are kind of brand new and Spinnaker obviously really helps with a lot of that stuff.

    Yeah, I had some experience, but was really excited to jump in and kinda get into something that’s really taking off in the industry.

    Ashley: Tell us a little bit about the app that you were challenged to work on. I understand this was already, had been developed with Jenkins and Spinnaker before and then you were moving it to GKE, or what was it?

    Matthias: No. So, the client had 50 different job applications, a bunch of different things, and it was all running on prem. So, part of the big effort was to obviously move them onto Google Cloud, but also to help them develop a CI/CD process that allowed them to make a commit in their repo, have artifacts be built in CI, have tests run and then have those passed to Spinnaker and CD and then deployed to GCP.

    They had no experience with Spinnaker and didn’t have any kind of CD solution setup, really, especially for the cloud. Like I said, they were completely new to the cloud, so this is all new stuff for them, so I helped them set it up, install it, configure the options all we wanted, manage it and then introduce more and more advanced features as the project got more and more migrated over to GCP.

    Ashley: Mm-hmm. So, it sounds like maybe a more traditional Java application environment maybe is not even continuous integration yet. Were they down the path with that?

    Matthias: Yeah, they had Jenkins on prem.

    Ashley: They did?

    Matthias: And so we—you know, yeah. So, the architecture was moving everything to Kubernetes and so, you know, one of the GCP projects has GKE cluster just for CI/CD. So, running Jenkins and running Spinnaker on it. So, they had Jenkins on prem, and moved that over to the cloud, hosted it on Kubernetes, and then we installed Spinnaker right alongside it on the same GKE cluster, right?

    Ashley: Mm-hmm.

    Matthias: And then used those tools as the basis of their CI/CD pipelines to deploy to other clusters within the cloud for them.

    Ashley: How about the size of the application? How would you quantify how kind of big or extensive it was?

    Matthias: Like I said, they had a—this was a true microservices organization. So, you know, they have 30 to 50 different applications as microservices, each individually being deployed. As far as traffic, I mean, so the applications, like I said, are microservices, so they’re pretty small in and of themselves individually.

    As far as traffic and resource usage, it’s a very, very large, large client. Like, pushing the boundaries of some of the largest things we’ve seen.

    Ashley: Hmm. Okay.

    Matthias: So, that was really exciting—really cool.

    Ashley: This was a Fortune 500 company, I understand.

    Matthias: Yeah.

    Ashley: We’re not talking about the specific company, but—

    Matthias: Sure, yeah, yeah, yeah.

    Ashley: Well, good. Well, what are the sort of things that you’re planning on talking about, then? I mean, I know this is about the monitoring aspect of it, so you know, are you touching on Prometheus Operator and probably how to configure it or how you decide it and what kinds of groupings to create, what kind of rules and alerts and those kind of things? Are you gonna be talking more fundamental architecture? What are your thoughts?

    Matthias: Kind of a bit on everything, because this kind of—this is kind of some required setup in order to do monitoring and in order to do canary analysis, even, if you’re gonna use Prometheus for canary analysis metrics. And it’s also just fundamental to how you wanna have this setup if you’re running Spinnaker on Kubernetes in general.

    So, a little background on Spinnaker as a project, which I’m sure other people that are familiar with Spinnaker definitely know, you probably know yourself. Spinnaker was originally created by Netflix. Netflix is still very much a VM shop, right? So, they don’t use Kubernetes, and everything runs on individual instances. And so, Spinnaker can be deployed to run on Kubernetes and that support is there, it has support to deploy applications to other Kubernetes clusters.

    But there are still a few things like monitoring, like canary analysis where, in the Kubernetes world, we do things a little differently, and Prometheus is one of those examples, right? In the VM world, you spin up Spinnaker, you spin up one Prometheus instance—and Prometheus obviously is a metric store to collect metrics and then usually is paired with Grafana and ____ and things like that in order to—all CNCF open source projects in order to give you graphs on these metrics.

    And so, there was previously, in the VM world of Spinnaker, running it as Netflix does, a microservice of Spinnaker that does monitoring, that connects to all of the different components of Spinnaker, listens for their metrics and then reports it to Prometheus. Cool. Like, it looks great. They even have some dashboards.

    So, Grafana dashboards, if you’ve ever worked with those you can just upload some JSON files for your dashboard and then you click there and then you’re like, “Hey, look at these dashboards!” You can get real time dashboards based on the Prometheus metrics of Spinnaker. And Spinnaker, all the different components that it’s made up of and the different metric based on the things that it’s doing so that you can see in there when it’s processing your deployment, here are some of the metrics that are coming off, right?

    Ashley: Mm-hmm.

    Matthias: So, all of that support was there. When you want to run Spinnaker on Kubernetes, though, that support was a little…not so much. [Laughter]

    Ashley: Mm-hmm.

    Matthias: And that’s the trouble that we kinda ran into. So, the thing that was there was the concept of, “Hey, we have this monitoring component in Spinnaker,” and if you enable monitoring, that monitoring component will be enabled as a sidecar container to every single pod, every single microservice that Spinnaker is composed of, and it knows how to talk to that microservice and get the metrics from it and then report it somewhere.

    Cool—good. Okay, I can get that. What about the monitoring service then having those metrics that are just collected from all the different components and putting it somewhere? Ideally Prometheus, you can also do Stackdriver or, I believe there might be Datadog support now, but mainly, the two that we see used a lot, Stackdriver and Prometheus.

    Ashley: Mm-hmm.

    Matthias: However, in the Kubernetes world again, especially when you have a bunch of different Kubernetes clusters, one of the tools that you end up using is called Prometheus Operator. And that is essentially a Kubernetes operator that, when you apply to your cluster, will automatically install for you Prometheus Grafana, an alert manager as pods, deployments running on your cluster. It’s a way to provision your infrastructure so that when you deploy a bunch of applications in these clusters, you already have some monitoring infrastructure set up, right? I can deploy and I can basically go to my organization and say, “Give me a new cluster and then I can deploy my application and start admitting Prometheus metrics and there’s already a Prometheus instance on the cluster, so like, I’m good to go.”

    In order to get the Prometheus, the version of Prometheus that the Prometheus operator creates to then go and fetch the metrics from the Spinnaker monitoring component that already existed and already knows how to get those metrics from the Spinnaker different components, you have to kinda connect those two ends, right? And in the Prometheus Operator, the Kubernetes world, basically, you just apply Kubernetes manifest, the CRD called the Service Monitor, and that basically tells your instance of Prometheus on the cluster, “Hey, go fetch the metrics from these places,” right? And so, it’s pretty easy, but there was no setup way in Spinnaker. So, you would read the documentation in Spinnaker and they would have very detailed, very nice—like, if you’re running Spinnaker on VMs and you want to do monitoring and you wanna use Prometheus, just use this fancy and nice, easy setup script, right?

    Ashley: Mm-hmm.

    Matthias: Super nice. Great. And then literally in the documentation, it said, “If you’re running on Kubernetes. Hang tight, support is coming.” [Laughter]

    Ashley: [Laughter] The “hang tight” documentation feature. Wonderful!

    Matthias: Hang tight—yeah, right. So, none of this is extremely complicated. There are a lot of different components and you have to be careful about how you set them up, right? But the very nice setup script that was available for just running plain old Prometheus on VMs was very nice and like I said, it installed a bunch of Grafana dashboards for you and did a bunch of nice stuff for you. There was no such thing for Kubernetes, right?

    Ashley: Mm-hmm, mm-hmm. Kinda starting from scratch. Not completely, but at least to take that next step.

    Matthias: Exactly, exactly, exactly. And so, what I did and what I contributed to the project was creating a setup script for this exactly, saying, “Hey, you’ve got a cluster, you’ve got Spinnaker already installed on there, you’ve got Prometheus Operator installed on there and you’ve got a Prometheus Operator installed on probably a bunch of other clusters, too. Here’s a setup script that connects Prometheus instance to Spinnaker so that Prometheus can go and fetch those metrics about how Spinnaker is running and how Spinnaker is handling your deployments.”

    Ashley: Mm-hmm.

    Matthias: And installs a bunch of these Grafana dashboards that the open source project has already created for us to be able to say, “Here’s a Grafana dashboard about each microservice—Clouddriver, Gate, Orca, all the others, they each have their own dashboard in Grafana that you can go look at and they’re preconfigured and it’s really nice, it’s really neat.” Again, those just had to be kind of converted into the way you apply these in the Prometheus Operator world which, again, is applying your Kubernetes manifest to say, “Hey, it’s actually just a config map with a certain label on it.” “Hey, here’s a dashboard,” and when you apply that, the Prometheus Operator knows how to hook into the Kubernetes master and goes, “Hey, there’s a new config map that is a Grafana dashboard. Let me go fetch it and add it to Grafana for you,” right? That’s the whole purpose of Prometheus Operator.

    So, that’s kind of the thing that I set up was saying, was giving you—so now, and then I PR it to the documentation. So, now, when you go look at the documentation, it says, “Hey, if you’re running Spinnaker and Prometheus Operator on Kubernetes, there’s also a fancy setup script for you, too!” [Laughter]

    Ashley: There you go. Now people will have to start at the same place you did.

    Matthias: Yeah, really.

    Ashley: So, are your planning, on your talk, actually walking through this, the process of the script you built and how to set all this up, get the dashboards to come up, et cetera, or you’ve kinda taken a little different approach?

    Matthias: I don’t want to anger the demo gods, so I’m not sure if I’ll actually be, you know, typing it, running the actual scripts, but I can certainly have something set up that’s like, “Hey, this is the end result. Let’s step through what the script is actually doing to help you understand what components need to be matched up where, and then here’s the end result,” right?

    The other kinda couple things that I’ll probably touch on in the talk are—which is why it’s important. So, all of that is monitoring Spinnaker as an application. So, like, your DevOps or your Release Engineering team, people that are involved in these things who are gonna monitor Spinnaker as an application and are gonna see, “Oh, Orca is overwhelmed and it has a bunch of tasks in its queue that it can’t pop off, so let’s add some more replicas of Orca” or something like that.

    That’s what this script helps you set up is this dashboard to see the performance, to see those metrics, right?

    Ashley: Mm-hmm, mm-hmm.

    Matthias: And other reasons that it’s important are, and the other things I’ll touch on—well, first of all, the one thing I’ll probably touch on is that there’s kind of an effort currently in the works and I hope by the time of the talk, maybe there’s a little more effort built towards it, but there isn’t really a whole lot of guidance in the Spinnaker community about, “Okay, here are the metrics we emit. Here’s what they mean, and if they go outside normal ranges, here’s what you should do to remedy that situation.”

    Ashley: Mm-hmm.

    Matthias: There are no kind of run books, as you might call them, for how to respond to a Spinnaker instance that is under a really extreme load or kinda gone configure sideways or something like that.

    Ashley: Interesting.

    Matthias: Rob Zyner kinda leads the Netflix effort. He’s done a great blog post about some of the metrics and things like that and what each of them mean. But that’s kind of the only resource out there. So, there is kind of this effort in the community to say, “Hey, let’s put together some actual run book and match some of these dashboards that we have in Grafana and some of these metrics and be able to tell people, ‘Hey, if you run into this situation, here are the step 1, 2, 3 to check, and if these are the case, then here are the three options for alleviate that problem or how to address the problem,” right?

    Ashley: Mm-hmm.

    Matthias: So, hopefully, by the talk in November, we might have some progress on that in the community.

    The other thing that I’ll probably talk about on the topic and why the concept of the Prometheus Operator is so important in the Kubernetes world is for canarying.

    Ashley: Say a little bit about what canarying is with your deployment processes.

    Matthias: Yeah, sure. Sure, so, canary testing is—it’s a form of testing that evaluates one of your release candidates. It compares it to—what it does is, it actually runs your candidate, and it runs the version that is running in production, starts a new copy of it, and it runs it side by side. So, then it is—and then it diverts usually a small amount of traffic or just, you know, will have 100 pods behind a service and you’ll add 2 pods so they’ll get a proportional small amount of traffic. And you’ll run it and you’ll see how it performs. You’ll listen to the metrics and you’ll see, “Oh, this is how your candidate that you want to push performed.” And in canarying, in the configurations in Spinnaker, you can say, “These are the metrics I care about when evaluating, when deciding if it performed well and if it performed badly.”

    For instance, let’s say you have a new version of the application that just, like, 500—500 errors out on every request, right? You run a bunch of unit tests, you run a bunch of integration tests and ideally, at that point, at one of those steps or earlier, you would’ve caught that, right? But tests are written by people and sometimes people don’t write them or sometimes some situation that depends on live traffic and live data actually exposes some types of problems, right?

    So, let’s hypothetically pretend you have a new version that comes out and it’s just 500s all over the place. What canarying says, the kind of theory behind it is—let’s actually run it on a small amount of traffic, let’s see what it does. And then canarying would go, “Oh, hey, your release version that you wanna push out? It’s just doing a bunch of 500s, and you told me in your configuration, if you see so many, a spike in 500s, that that’s bad, that we don’t wanna actually then deploy that to all 100 instances running in production,” right?

    That canarying says, “Hey, that’s not a good thing to promote,” right? Like, “I gave it a chance, I ran it on some things, I compared it to what the version that is currently running or our new instance of the version that’s currently running, and it didn’t do well. So, you know what? I’m gonna error out, I’m gonna show you this is how it behaved, you guys go fix it, and then when I come back, I’ll run those things again. And if it performs well, then great, we can release the new version.”

    And that’s kind of one of the main components that you need for continuous delivery. Because you can imagine a world where developers are just committing, committing, committing, committing.

    Ashley: Well, yeah.

    Matthias: They do a bunch of tests, they run some tests, to them it all looks good. And then you just have this other automatic component over here that’s doing canary analysis and that can just test it, it can actually put it into production, give it a little bit of traffic, see how it performs, and if it performs well, then let it go, you know? Promote it, right?

    Ashley: Yeah, that’s extremely useful.

    Matthias: Deploy on a Friday, deploy on a Saturday—who cares, right? It’s actually seeing how it’s running, all the other tests before it’s even gotten to that point have passed, obviously, and if it runs well, then run. And this is what enables organizations like Google, Netflix, and others to deploy thousands of times a day, because it has this automated system to go in and say, “Give it some traffic, see how it runs, and if it runs fine because of the way we configured how we told the system what does it mean to run fine, right? Great—then let it go.”

    Ashley: We’re kinda running up against our time, here. I do have one last question.

    Matthias: Sure.

    Ashley: What kind of folks should come to your talk? Obviously, it seems like developers, people who are working on the CI/CD pipeline, maybe DevOps engineers. Are there other folks you’d recommend, or are those the right folks?

    Matthias: Those are definitely all the right folks. I think if you’re interested in Spinnaker or you’re currently using Spinnaker where you are, want to know about monitoring it as an application and managing it as an application when it experiences a bunch of load and how you want to connect all the dots if you’re running it on Kubernetes and you’re running multiple other Kubernetes clusters that also are using Prometheus Operator, this is a great talk to go listen to.

    Ashley: Again, Miles is a Google Cloud Consultant with ContainerHeroes.com. He’s speaking at the Spinnaker Summit 2019 in San Diego. Now, that conference is the 15th through the 19th of November. His talk, again, is on monitoring Spinnaker with Prometheus Operator on GKE on Saturday the 16th at 3:45. Thank you, everyone, for joining us. You’ve listened to another DevOps Chat. This is Mitch Ashley with devopsy.kinsta.cloud. Be careful out there.

    — Mitchell Ashley

  • Armory Looks to Drive Enterprise Adoption of Spinnaker

    Armory Looks to Drive Enterprise Adoption of Spinnaker

    Armory this week committed to advancing adoption of the open source Spinnaker continuous delivery (CD) platform in enterprise IT environments after revealing it has raised an additional $28 million in funding.

    Company CEO Daniel Odio said Armory will add proprietary tools to a curated distribution of Spinnaker that will make it easier for the average IT organization to deploy and manage Spinnaker. Developed by Netflix, Spinnaker is gaining traction as a CD platform in the wake of becoming one of the four projects being managed under the auspices of the Cloud Delivery Foundation (CDF), an arm of the Linux Foundation.

    Spinnaker is designed to be a CD platform that includes all the management tools required to manage that process. While continuous integration/continuous deployment (CI/CD) platforms such as Jenkins have addressed aspects of CD, most of the daily usage of those platforms has been squarely focused on continuous integration. As part of an effort to meld CI/CD platforms and Spinnaker, the CDF has accepted both Spinnaker and Jenkins as part of the first four projects that will be developed under its auspices. The other two are Jenkins X, a CI/CD platform specifically designed for building microservices-based applications on top of Kubernetes, and Tekton, a set of pipelines also designed specifically for Kubernetes.

    Odio said there are already more than 500 organizations using Spinnaker, and each one is not standardized on a specific CI/CD platform. Armory is part of the CDF but also sees a need for a curated instance of Spinnaker that is optimized to meet the needs of enterprise IT environments, he said.

    In general, Odio said a “flywheel effect” manifests once an organization launches roughly its eighth project using Spinnaker. From that point on, adoption of Spinnaker starts to accelerate to the point where an organization sees the need to standardize on a single curated instance of Spinnaker, he said.

    The bigger issue as far as adoption of Spinnaker and other DevOps tools remains cultural, Odio added. Many organizations still don’t accept the fact that CI/CD platforms are intended to foster a level of accelerated development that makes it more feasible to experiment and innovate. There are, of course, inherent risks to that approach in the sense that many of those projects might fail. Many enterprise IT organizations are still coming to terms with the need to accept higher levels of failure as the price of learning, he noted.

    In the meantime, many organizations that have embraced DevOps now find themselves on the cusp of a major transition as the shift toward cloud-native applications increases. Most CI/CD platforms today were designed to accelerate development of monolithic applications. Now what’s needed is a new generation of CI/CD platforms that are designed to accelerate development of microservices-based applications based on containers. As a result, many organizations will end up employing multiple CI/CD platforms. Armory is betting many of them will try to delineate where CI leaves off and CD begins.

    — Mike Vizard