devopsy.kinsta.cloud

Tag: kubernetes

  • Solving Cloud-Native Challenges in the Rush to the Cloud

    Solving Cloud-Native Challenges in the Rush to the Cloud

    The shift to cloud-native environments away from traditional data center infrastructures continues unabated, but security and complexity challenges remain a struggle for DevOps teams.

    These were two of the main takeaways in a survey and analyst report published by the IT consultancy firm Flexera. 

    The survey, based on the responses of 750 IT professionals, revealed that 92% of the organizations represented in the survey have a multi-cloud strategy, 80% have a hybrid cloud strategy and only a minority (45%) have integrated data between cloud environments.

    The top challenges, according to the survey results, are:

    • security (81%)
    • managing cloud spending (79%)
    • governance, lack of resources/expertise and compliance (75%)
    • comprehending cost implications of software licenses (55%).

    The solution for these challenges largely involves — in addition to implementing the right DevOps culture and practices — adoption of the right tools and platforms, especially for security. However, first-generation cloud-cost management tools “simply are not cutting it” for modern architectures, said Asim Razzaq, CEO of Yotascale, which offers a cloud resource optimization platform.

    “Today’s modern architectures are dynamic, and ownership is transient and not always well-defined. The greatest strength of containers and Kubernetes is the ability to scale on-demand and share cloud resources among multiple applications,” said Razzaq. “Tracking cloud costs in modern architecture requires a strong ability to automate cost attribution — or tag management — for those dynamic resources, and the ability to understand usage at the application level, even for containers that are sharing resource instances.”

    First-generation cloud management tools are, in this way, unable to provide the information DevOps teams require to understand how to automate cost attribution for modern architectures, said Razzaq.

    “[DevOps teams] can’t break down costs into any level of meaningful detail for engineering and finance teams to digest and act upon,” said Razzaq. “Conversely, newer approaches that are focused on Kubernetes cost management lack the ability to understand and break down the costs of non-cloud-native workloads.”

    Indeed, tools are often frustrating to engineering teams because of their static custom reporting, broad and generalized recommendations and no effective workflow for effective collaboration between finance and engineering, Razzaq said.

    “The biggest source of frustration is the inability to drill down and manage costs in a way that matches the way that these teams do business. There may be 10 teams, each with five production applications running in the cloud, for a particular account,” Razzaq said. “Engineers need to see, at the application level, what their resource utilization, costs and cost anomalies are in order to make the best cost/performance decisions for their applications. Without that, they’re flying blind; having to defend their cloud usage without any supporting data.”

    As an example, a misconfigured Kubernetes cluster can “cause a massive spike in costs,” Razzaq explained. “Without detailed cost and utilization reporting at the team and application level, an engineer can spend weeks digging through spreadsheets and logs trying to track down the source of the anomaly. That’s weeks of their time that they aren’t [using to] be productive on new features.”

    cloud-native

    Razzaq said there are a number of solutions DevOps teams can adopt to address these issues, including automated cost attribution and tag management, actionable recommendations and comprehensive observability capabilities.

    Automated cost attribution and tag management “is the foundation of accurate and detailed cloud cost management,” said Razzaq. “Engineers need tools to help them set tag management policies, identify untagged instances and update tags quickly and easily in their cloud environment,” he said. “With this in place, cloud-native workloads can be automatically tagged as they are dynamically created, ensuring that cloud costs are always clearly attributed to the right teams and applications.”

    Actionable recommendations at the cloud-instance level with closed-loop feedback also are an important process “so that engineers can see exactly why the recommendation is being made, and what the cost impact will be of taking that action,” Razzaq said. “The ability to provide feedback on the recommendation and dismiss recommendations that don’t align with business needs ensures that new recommendations are relevant,” Razzaq added.

    Finally, tools that provide a unified view of both cloud-native and traditional workloads that have been shifted to the cloud help to ensure “that engineering and finance have a single source of truth for their cloud costs that enables them to make smarter decisions for their business,” Razzaq said. 

     

  • Survey Shows Mounting DevOps Frustration and Costs

    Survey Shows Mounting DevOps Frustration and Costs

    A survey of 400 developers and DevOps team members in the U.S. and Europe found, on average, 15 hours a week are being spent managing tasks such as debugging pipelines and waiting for tests and builds to complete.

    Conducted by the market research firm Vanson Bourne on behalf of Garden, a provider of a platform for automating application development and deployment tasks in Kubernetes environments, the survey finds more than three-quarters of respondents (76%) said the time spent on these and other manual DevOps tasks is generally being wasted.

    Garden CEO Jon Edvald said, all told, 39% of a 40-hour workweek is typically devoted to these tasks. When the number of developers and DevOps specialists that are employed today are factored into that equation along with their salaries, as much as $61 billion is being wasted on these tasks in the U.S. alone, he noted.

    The top sources of DevOps frustration cited by survey respondents are waiting for pipelines to run (76%), waiting for builds and tests (74%) and setting up, maintaining and debugging pipelines/automation (71%). In fact, only 11% of all respondents are completely happy with their development setups and workflows.

    Those hidden costs may even increase as organizations bring on additional DevOps expertise. A full 83% of respondents said their organizations are planning to hire more staff or increase their use of freelancers/outsourcing into DevOps roles in 2021. Three quarters (75%) said DevOps budgets have increased or will increase compared to 2020.

    In addition, the adoption of more complex platforms, such as Kubernetes, may also conspire to increase costs, noted Edvald. Respondents at organizations not yet using Kubernetes spend an average of 14.3 hours per week on low-level DevOps tasks, as compared to 16.5 hours a week for those who are already using Kubernetes. Nearly two-thirds of respondents (62%) said their organizations are already using Kubernetes, while 31% are currently evaluating it, to varying degrees. A full 95% of respondents using Kubernetes have faced challenges after rolling it out, including training and onboarding (37%) or complexity of setup/configuration (35%).

    Of course, not all the time that might be saved is going to be applied to tasks that increase overall productivity; nearly half of respondents (49%) said if they had it, they would spend more time developing new products and services.

    Specifically, 46% said they would improve speed and delivery of existing products and services, while 44% said they would focus on improving security for existing products and services.

    Edvald said the survey makes it clear there’s still plenty of opportunity to automate repetitive DevOps tasks. Organizations are spending too much time on the care and feeding of DevOps platforms that takes time away from building and deploying applications, noted Edvald.

    Less clear is when organizations will reach some type of DevOps breaking point. Garden is betting that the transition to cloud-native applications based on microservices will eventually force the DevOps automation issue. However, in organizations that only know the DevOps processes they have used for years, change may be a long time in coming.

  • EP 9: The Road to GitOps

    EP 9: The Road to GitOps

    GitOps takes DevOps best practices for development and applies them to infrastructure automation. In short, GitOps delivers a hermetic, immutable deployment. So, is GitOps replacing DevOps?

    Managing infrastructure as code through GitOps can be a game changer for organizations. But, when is a GitOps approach best used or not a fit? In this DevOps Unbound episode, I’m joined by Tracy Ragan of DeployHub, Parag Doshi of Tricentis, Brendan O’Leary of GitLab and Mitch Ashley of Accelerated Strategies Group to discuss these questions, the impact of GitOps on the CI/CD market and where the future of GitOps is heading in 2021.

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript

    Alan Shimel: Hello, everyone. This is Alan Shimel, CEO at MediaOps. And you’re watching another episode of DevOps Unbound. Let me introduce you to our panel for this week’s episode, and we’ll jump right into our subject matter. First of all, we have the newcomer to DevOps Unbound, but he’s not a newcomer to our audience, my good friend Brendan O’Leary, senior DevOps evangelist at GitLab, a governing board member for CNCF. Hey, Brendan. Welcome.

    Brendan O’Leary: Alan, nice to see you, as always, and happy to be here.

    Shimel: Thank you for being here, Brendan. Next up we have our good friend and a semi regular here on many of our video segments at MediaOps, Tracy Ragan. Tracy is CEO of DeployHub, as well as the community-elected board member of the CD Foundation. Hey, Tracy. Welcome.

    Tracy Ragan: Thank you. Thank you so much, Alan, for having me here today. I always love doing these.

    Shimel: Absolutely. We love having you. Our third panel member today comes to us from Tricentis, who’s also the sponsor for DevOps Unbound, and it’s Dr. Parag Doshi. Parag runs engineering, VP engineering at Tricentis. Parag, welcome.

    Parag Doshi: Hello, Alan, everyone. Nice to be here. Thanks for having me.

    Shimel: Thank you. Thanks for being on. And then last but not least, my cohost for DevOps Unbound, Mitch Ashley. Mitchell is the CEO/founder of Accelerated Strategies Group as well as CTO at MediaOps. Hey, Mitchell, thanks for coming on today.

    Mitch Ashley: Absolutely, Alan. And thank you for all of our panelists for joining us. You are all fantastic. We enjoy having you.

    Shimel: You know, just before we jump into it, the continuing evolution of COVID times, we’ve done so much with our backgrounds, [Laughter] on our videos, right? Looking here at Tracy’s snazzy DeployHub background and Mitchell’s background and Parag’s – and Brendan’s old-school, man. Brendan’s giving us – if I’m not mistaken, he used to be down in the basement and now he moved up. I think he displaced one of the kids or something.

    O’Leary: Yeah, I moved upstairs.

    Shimel: Yeah, yeah. And that’s a real background, so good for Brendan.

    Ashley: And he’s got an MV7 microphone there. Nice job there, Brendan, on the mic.

    O’Leary: Oh, thanks. Yeah.

    Shimel: Yeah.

    O’Leary: All, you know, post COVID things to make it work, Alan. I wanted to have the background, the real background to make it work. But that was also a COVID change.

    Shimel: It’s authentic. Yeah. [Laughter] Crazy stuff. Anyway, guys, today’s show is around one of the hottest topics in DevOps – GitOps. Right? And I’ve actually heard from people, you know, that GitOps is the new DevOps or GitOps is somehow replacing DevOps, and I think DevOps has outgrown the – it’s going to be replaced by something.

    It’s transcended to a different level of existence, if you will. But certainly GitOps is on a lot of people’s lips. It’s not only on their lips, it’s in their development plans and how they are delivering software to the cloud, to cloud native, and all over. But is the right move for every situation? Is it a panacea?

    When is GitOps a good thing to use? When is it maybe not a good thing to use? Right? How much does it change sort of the classic CI/CD pipeline that we’ve started to finally wrap our heads around in DevOps? Stuff like that. So I am going to – Brendan, of course, GitOps, GitLab, they both have Git in it.

    So we’re going to make you – I’d like you to start off. Let’s start with some definitions. What’s GitOps mean to you?

    O’Leary: That’s a great question, and I think one that’s still probably [Laughter] unsettled, right? Much like defining DevOps can sometimes be seen as unsettled. GitOps came out of the cloud native community, and so a lot of folks maybe restrict the concepts of GitOps to say it only exists in a Kubernetes world. Because that’s kind of where this approach came from. But we’ve seen folks employ GitOps principles to all kinds of different infrastructure, automation, from old-school VMs to containers to Kubernetes clusters as well.

    So when I think of GitOps, I think it kind of requires three things. One, infrastructure as code. Right? You said that in the beginning. That’s kind of like table stakes to getting in. It’s something we’ve been working on for a long time. But really defining as code in Git – the Git part of GitOps – what our infrastructure’s going to look like, and then keeping that configuration stored in that code.

    And maybe the desired state is stored as code or not, but at least the way we get to that desired state is. Second, I think it’s merge request. So we call it merge request. You’ll hear other folks call them pull requests. But the concept of using the point where someone is proposing a change as the central place where teams collaborate on.

    Is this the change we want to do? Is this going to help solve the problem we’re trying to solve for? Right? Bringing that concept to the world of infrastructure is hugely beneficial, because it’s how our developers have already been working for years and years. And making that merge request, kind of that central source of truth for approvals and security checks and all those other things is really critical.

    And then last, I think it’s – the third thing is CI/CD, right? Of course. So the idea of GitOps is automating that infrastructure, update with a very simple Git workflow. And so you really do have to have embraced and adopted continuous integration and continuous delivery really in order for the whole package of GitOps to make sense for you.

    Shimel: Pretty darn good. Comments, add-ons, thoughts? Parag, Tracy?

    Ragan: I would only add one thing here. If you say what is GitOps, I like to shorten it down. I think what Brendan just defined is all accurate, but it’s operations by pull request. The goal of GitOps, what GitOps delivers is a hermetic, airtight, immutable deployment. And that is in essence what we’re trying to accomplish by using Git as the place where we declare and define what we’re going to push out to or have pulled out to the cluster.

    So it’s the concept of a hermetic deployment, something airtight that no human touched. Don’t you dare touch that and then deploy it out, because of we don’t know what you did. [Laughter] So the whole idea is to create a hermetic process.

    Shimel: I love that. That was concise and right on.

    Doshi: I have a similar viewpoint. I think of GitOps as a combination of infrastructure as code plus event-driven architecture. Right? And in this case we’re talking about merge requests or pull requests or whatever terminology you want to use, depending on the flavor of Git you love. But the way I think of this as a former physicist is entropy is there. It’s too hard to keep track of everything. Humans can’t remember everything.

    We need to be able to describe, as application developers have been doing for the past couple decades, in code the versions of things, the documentation associated with everything. And then the automation that we’ve had in CI/CD, which people think of, oh okay, that’s developers pushing their code out, well there’s no reason why we have to stop at application level. It needs to be the whole thing, because application and infrastructure must stay in sync at all times. So by putting a description of both the application, the infrastructure, and within the surrounding environment, all three of those into Git, you’re bringing the best practices of AppDev into the whole world together so they’re all synchronized.

    Ashley: Yeah. I like that description too, Parag, because in a way that’s breaking down silos that we still have around the code build and deployment process. And it’s almost like if you think back when we started using code repositories and, okay, everybody put their source code in one place, okay, let’s expand that to artifacts and other things. Well, now we’re doing the same thing of let’s abstract at one more level and make the whole thing as cradle to grave, if you will, for getting code out into product. Whether it’s any part of that stack, it’s a GitOps process.

    So to your point, Tracy, about it’s an immutable object, it’s something that we know from beginning to end, it’s all been in one environment and we happen to have manual processes that have just been automated that potentially are weak or could be broken.

    Doshi: I’ll add one more comment on top of Branden’s about – I think it was around – it came out of cloud native computing foundation, but really the way I look at containers is it doesn’t matter that’s a container or VM. We’ve just moved a control plane up to a higher level, but the same principle’s apply. So there’s no reason why you cannot use these same principles in a container type of environment or a non-container environment. The only difference I would say with containers is we can be very surgical, fine grain, down to each individual pot. So there’s a little advantage with cloud native architectures to be very fine grained.

    O’Leary: Yeah, I agree. And I think that’s important for folks to separate the concept of GitOps, the principles that we’ve all talked about. And I agree with the other panelists and how they’ve talked about it. But separate those principles from the technology, right? Because it’s not always true that Kubernetes is going to be the right technology for you to deploy every workload you have in, but still this concept of GitOps.

    And I really liked Tracy’s sealed container – well, overloading the word container there. [Laughter] But that sealed deployment that doesn’t have a lot of human – or doesn’t have any human interaction. That is what gets us to what Mitch was talking about, which is we want to take this isolated, people working in isolation, the kind of cave-like nature that operations typically used to have where you had to have this specialized knowledge that one person knew. Oh, wait. How do we deploy that service?

    Oh, we’ve got to call up Alan. He’s the one that knows how to do that. [Laughter] Right? You want to get rid of that and make it all, what Tracy said, which is like a hermetically-sealed deployment that we know we can reproduce every time.

    Ragan: Yeah. So GitOps was a brilliant way to leverage existing tools and habits to create Chef and Puppet. That is basically what was done. I mean, it has a Puppet kind of master architecture. You have a GitOps operator that’s running on your cluster. It does a pull based on what’s in Git. It very much took the basic concept of Chef and Puppet and leveraged existing tools that people already had, some open source, GitOps operators out there, and they were off and running. And that is the brilliance and beauty of GitOps. I mean, I have a lot of issues with GitOps, and I’m sure we’ll share them, but its brilliance is in that simplicity.

    Shimel: The irony though, Tracy, is that many people will tell you that Kubernetes killed Chef and Puppet, right? And we’ve seen Chef get sold maybe for less than money raises in valuation previously.

    Ragan: No, I’m just talking about the basic concepts.

    Shimel: No, no, I get it. But if we look at the concepts behind what Chef and Puppet did, which was automate that deployment, deployment automation like that, and then with Kubernetes it sort of negated the need for that, but yet we kind of snuck that in the back door. Except we put it in Git to use in Kubernetes. So the irony of that is pretty profound, right?

    Ragan: Well, it’s all about the container.

    Ashley: Right, right.

    Shimel: Yeah. You’re right. Maybe it’s because Chef and Puppet – [Crosstalk]

    Ragan: The containers simplified, yes.

    Shimel: – didn’t figure out how to work in the container. Look, for that matter Jenkins too. Right? We went to Jenkins X because of all of that and everything else.

    Ashley: That’s a really great example, yeah.

    Shimel: But what I’m reminded is of the old Who song, right? “Meet the new boss, same as the old boss.”

    Ragan: Exactly.

    Shimel: And you still have that old boss. With that being said, here’s another thing, and Parag, I’m going to ask you to kick off as I take us a direction here, and that is GitOps, yes, it’s for deployment. What about testing? I said this is all about testing. We’re doing our testing in Git. Is that the – that stops, right? Is that better? Worse? Is that the way it should be, always, sometimes, never?

    Doshi: So I don’t know if I’d call it testing in Git, but I would say now Git becomes a source of truth. So I think what Tracy said is right, that it’s just that same sort of automation of infrastructure but now moved into making Git the centerpiece of where you define this. So the Git part is unchanged, but the testing is now event driven. Right? So because you’ve merged in code, you’ve committed code or gone to do a pull request, that can trigger testing.

    And we call this continuous testing. And I think the evolution of this can even go as far as our test definitions themselves should also be put into Git. That’s the next evolution that we’ll have. I don’t believe it’s really commonplace just yet, but when that happens then, we are trying to describe as much as possible into Git as a repository and use the principles of version control added with the event-driven concepts.

    So when a developer does X or when an operator is in environment Y and something needs to change, that whole Git workflow can invite continuous testing to say do I allow that to merge back in? If you want to have an approval, you could do those kinds of things. So to me, incorporating continuous testing, and the day that we put test definitions into Git as well, we’ve gone quite a bit further of completing this vision and really making DevOps real. Right? Because DevOps isn’t just releasing really fast, deploying really fast. It’s deploying fast with confidence and ease. That’s what we need.

    Shimel: Thoughts on that, panel?

    Ashley: It sure seems like a natural evolution to include the test environment. And as you say, Parag, it’s probably not commonplace yet today. A question for you about that; do you think that there’s incentive with the folks that are taking care of the testing, whether they be the QA team or developers writing tests, things like that? Do you think they have the right incentives to move it into Git or will it be something else that causes it to happen?

    Doshi: I think as long as people have the incentive of delivering value as quickly as possible with the best meantime to recover, the lowest defect leakage rate, as long as we do break down those silos, as you said. If the testing team is, okay, well, I don’t understand what the developers do, just throw it over the wall and then I do my test, that’s not going to work. This is talking about dev testers, the full stack developers. And as long as everybody’s motivation is right, that we’re trying to deliver value faster and faster, that we will not have confidence – that a VP will not say let my applications go out – you know, they’re only going to look at portfolios of apps and say what amount of testing has been done? What amount of security checks have been done?

    That’s another thing. The environment. Okay? That’s the biggest entropy of all. We have the least control over the environment. The same code, the same infrastructure in different environments behaves differently. Until we get that environment description also into Git, coupled with the testing model themselves that says here’s how we test it in that environment or that application or this infrastructure pattern, then DevOps becomes real. Until then, this is all talk.

    O’Leary: Yeah, I agree. I think it’s critical, and it’s something that GitOps, again, this concept of GitOps, I think is exciting to a lot of people that have been trying to make DevOps real for so long, because it enables that in a way. Right? We see kind of the light at the end of the tunnel, I would say. And if that is all brought into Git and all made into something that can deploy independent of any human interaction, well now you have the ability to do continuous testing. You have the ability to stand up an identical environment, the golden goose or [Laughter] golden egg of a, hey, we want an environment that’s exactly like production to really test it and really hit with security tests and know that when we do deploy to production we’re going to have that confidence.

    So if you’re able to continue to iterate on the environment, the infrastructure, just like you’re iterating on the code, it enables you to then say, okay, now we can not only have the code to review and maybe some static code analysis that happens and some testing that happens, and then maybe some human testing that happens, now we’ve also got an entire environment spun up automatically that mirrors production, where we can have a high level of confidence that it’s going to work when it really ships.

    Shimel: So guys, you’re telling me DevOps hasn’t been real quite yet. I feel like I’m living in the Matrix, kind of living in a simulation. Tracy, I sense you’re Agent Smith here, though. You have a different view. Maybe GitOps isn’t the panacea it is or it’s been projected here. What’s your view?

    Ragan: So no hate mail, okay? No hate mail. [Laughter]

    Because I know there’s a lot of enthusiasts out there, and rightfully so. But let me just level set the way I see where we’re headed and where GitOps can’t take us. GitOps is imperative. It’s scripted. And it works really well – so think about how most people use GitOps.

    You have a code repository that you keep all your source code in. You make a change to your source code. You check that in just the same as you always did. You also have an environment repository, and what lives in your environment repository is your Kubernetes deployment YAML file. And that is where you define all of the stuff that we’ve been talking about and that’s what the GitOps operator looks at and pulls across.

    Now if you’re in a Kubernetes environment and you have containerized your application – this works great, it really does – and you don’t have a whole lot of environments – you have three environments dev, test, and prod – because remember, in a Kubernetes YAML file there’s a couple of things that are important to the GitOps operator. One is the tag of the container. So once you register your container image you get a SHA, and that makes it a unique identifier. And you have to update your YAML script with that SHA.

    Now dev, test and prod may have different values, configuration values. So you have to branch it. So you branch dev, test, and prod. So now you have three versions of that YAML file to manage your container deployment. And that’s pretty doable for most organizations today, but it is still imperative. Everything’s being defined by the script for each of those environments.

    Now let’s go and expand it and scale it. Let’s scale it to a microservice environment. Every microservice has its own YAML file because it should be independently deployed. And they should be shared, because that’s the beauty of a service-oriented architecture. They should be shared across teams. So what you do is as an application team, you’re going to make your coding changes in your coding repository and then you’re going to have different YAML files for the different services.

    Now let’s say you’re a store. I use the candy store and hipster store and the clothing store. They all use the card service. Well, they’re all going to put their card service YAML file into their environment repository, and that’s going to get deployed to their own clusters and name spaces. So you have a scaling issue, and it’s because of the imperative nature of GitOps.

    We are starting to move and have more conversations about declarative and events and catalogs, where this stuff is going to be stored and generated at runtime. So for example, instead of having all of these scripts, you’re going to have central places to go to say what’s the key value pair for this environment? I’m going to pull this registry information that has the SHA, I’m going to go get my key value pairs, and that way you don’t have to have so many scripts and you’re not managing – it’s onerous.

    It really can become onerous. We have customers who have 200 microservice and 20 clusters. That’s a lot of scripts. And if you’re an SRE and you want to update your microservice and you want to provide it to everybody, you’re going to have to figure out every single application environment repository that has it in there for every single environment and make those corrections. So it doesn’t scale as well as it should. Now I know that their CNCF has a group, a working group, around GitOps, and they’re starting to think about GitOps version two, and they’re going to have to address these kinds of core problems around an imperative process.

    Shimel: Love it. Brendan?

    O’Leary: Yeah. I mean, I like that concept and I think you’re right, Tracy, that the ability to manage the scale is definitely something you have to be prepared for if you’re thinking about GitOps, because if you just take it at its face value you end up with all those scripts and it could make the problems you have worse rather than fixing some of the problems you hope it would come into and solve. And so I think in a cloud native environment, again, it maybe pairs a little bit more simply, but I think that’s why again it’s great to think about the concept of focusing on that developer-centric approach and the experience that they’re going to have when it comes to shipping something out, to your point of if I want to ship an update to my microservice that impacts everyone, I don’t want to have to go through 78 repositories and update that everywhere.

    And so having GitOps do the infrastructure management in the same version control system as application development is great, but you also have to have a plan in place for how does that scale to the scale that you’re really going to build to? If that the scale of developers, if that’s the scale of services, if that’s the scale of environments. And so I think it’s important to have tooling and an approach in place that accounts for that as well.

    Shimel: Parag, you were going to say something I think before we –

    Doshi: Yeah. Sure. So I love what Tracy described there, and getting into the specifics of the proliferation of YAML files, and especially as we break down microservices. So this is the whole concern with microservices. What you took as monolith and you described as one big black box, now you’ve broken it down. And so now we have the proliferation of all these things. Right?

    So I like things like Helm to help to reduce that and parameterize all these things. But fundamentally with GitOps, the concepts are there. Like I said, infrastructure as code with an event-driven type of approach version control. What I like about CNCF and containerization is the purpose of containers inherently – and Branden can keep me honest on this – is really to make the environment completely separated from your solution, your application, your infrastructure. Right? So today’s industry, the best approach we have is containers or serverless, which is also part of CNCF, so that we have things like Docker files and YAMLs that allow us to reduce the entropy of the environment as we can as much as possible and describe them declaratively in things like GAML files.

    And I do agree with Tracy 100%, that now it becomes, okay, well now I’ve got all these different configs, pulled that out of my code. Even with service mesh technology and CNCF we’re pulling out things like security and rate limiters between every single microservice. All that’s taken out of the logic, the business logic, on everybody doing security different ways. Now it becomes a configuration issue. How do I – if I want to change a configuration, do I need to change 100 different YAML files or config maps? And I think the technology’s evolving.

    Help is just one example. And then I’ll associate that with all the continuous delivery. You all know Spinnaker and a bunch of different technologies in this area, Octopus and so forth, that can help.

    Shimel: Yep. Mitch, did you want to say something? Otherwise, I want to – I’ve got another –

    Ashley: All I can add to the conversation is version control across all of this also gets very complex. [Laughter] I think you all –

    Shimel: But isn’t that what Git and Repositories are supposed to help us with anyway?

    Ashley: It is, but now you’re versioning all of your scripts along with all the environments and all of the variables of what you’re trying to script.

    Ragan: And it’s not relational.

    Ashley: Yeah.

    Ragan: You know, when you have – what’s the blast radius of my microservice? I’m going to push it out because my hipster store needs the new card service. If you’re using an SOA architecture that means that you have one card service and everybody’s using that version and that’s the best way to move forward because you eliminate the sprawl. If you put it all in its own namespace, you have multiple versions of the card service. So now you have a problem with the card service and you have to go update all the namespaces that’s using that card service.

    So you have to have a relational model to be able to track what applications or what namespaces are using what microservices. And unfortunately, Git is not relational. We create relations using tags, using different repositories, but that becomes redundant.

    Ashley: Great example.

    Shimel: Fair enough. So let me –

    Doshi: Tracy, you talked about the structure.

    Shimel: Oh, go ahead, Parag.

    Doshi: I was just going to follow up and ask a question to Tracy. I think what you’re saying there is Git is just a big version control repository. If we can provide some extra structure and relationships around all these artifacts that we’re storing there we might have better control.

    Ragan: And that’s what you call catalog.

    Doshi: Yeah.

    Ragan: Right? So I think we’re starting – when we really look at a service-oriented architecture and we look at – let’s say, for example – let’s push ourselves farther into the future. And Brendan brought this up around service mesh. And I believe this can happen and will happen. I believe service mesh will be enhanced on.

    Right now you can say do a deployment and deploy out to 10% of my users. I think service mesh will start looking at active directory and say deploy out to my developers. So now we have one cluster and we have immutable services out there. We put a new service out there, it gets requests routed to the developers, they do their testing on it, once it’s good it’s rerouted to their testing community.

    Then it’s routed to a certain group of their production community. Which means that we don’t have deployments across multiple clusters. We’re going to start pushing into a one big cluster, which sounds a lot like the mainframe. [Laughter] But we’ve come full circle. And as Alan said, the song, right? Meet the new boss, same as the old boss. [Laughter]

    Shimel: Right. We won’t get fooled again. That is true. There is certainly some truth to that. I wanted to spend a little time talking about the impact of GitOps on the CI/CD market. We’ve written in devopsy.kinsta.cloud several articles, the gist of which was DevOps is the symptom of disease. CI/CD is the cure, if you will. Right? And we’ve built this whole DevOps industry around CI/CD solutions.

    And I mean look, GitLab’s a great example. They had a great Git product to begin with. But it was their CI/CD solution that kind of moved them into that end-to-end DevOps, if you will. Right? GitOps I think has profoundly changed that market because it’s profoundly changed the way we do CI/CD. Well, that and Kubernetes, right?

    Kubernetes I think was the initiator of it. What does it mean for the market? What does it mean for our audience out here? And we have a technical audience, 60% of which are practitioners, 30% are managers or higher, and 10% are just, I don’t know, vendors. But what does it mean for them out there?

    What does it mean? I mean, these CI/CD tools that they’ve gotten used to, that they’ve grown up with a bit over these last years. What does it mean for them?

    O’Leary: Well, Yeah. I think that we love to say CI/CD together. I mean, I love to say CI/CD together, right? That’s now we name our product. But those concerns have always really been separate and never really been completed merged and married. Like it’s great to try and think about them as a single unit – CI slash CD, right?

    But I think the reality is that CI is something that has been around for a really long time, probably 20-plus years if not more, but CD is something that a lot of organizations still struggle with for any number of reasons, whether that’s a technical reason, whether that’s an organizational structure reason, whether that’s a risk aversion reason. The idea of doing continuous deployment is something that really is very dependent on your organizational structure, who your end users are, and the, yes, to some degree like you said, Alan, the tooling that you choose. And what we’ve seen in traditional CI/CD tooling is this concept of applying what we do in CI, which is scripting based on an event, running through a number of scripts.

    That is what then becomes CD. Well, now we’re going to push with scripts out to the environment. And so GitOps, you noticed in my definition I didn’t say that a pull was necessary. I think you can do push-based GitOps if you have infrastructure as code, you have MRs at the point of change control, and you have the ability to do CI/CD. But I think most people when they hear GitOps think about this another new concept, which is this concept of pulling. Right? We’ve heard Tracy talk about that.

    And so having an agent, an active agent inside of your infrastructure that is constantly ensuring that your desired state is the actual state in the world – that’s what Kubernetes brought to us, that’s what – and then that’s what pull-based GitOps brings to us. And so that’s why for us we’ve been doing CD for years, like you said, Alan, and we’re doing mostly push-based. But as our customers’ needs expanded in GitOps we’ve adopted a Kubernetes operator and now we have our own Kubernetes operator, and you’re seeing this more and more in all the tools, that runs inside of the cluster and pulls changes in.

    And that methodology, while – again, credit Tracy – sounds a lot like Puppet, [Laughter] it still is something that was the alure of Kubernetes for a lot of people. Hey, I want to describe my desired state and have something else worry about that desired state always being true regardless of if I push something wrong once. I want this desired state to always be correct. And so I think that’s how we’re going to see a huge change in the tool space. Right?

    To Parag’s point, we’re talking about Spinnaker. That’s something that Netflix has been using for a long time, but no one ever thought that they would be deploying like Netflix. Well, now it’s a huge deal and lots of people are talking about it.

    Shimel: We are.

    O’Leary: And they are, right? And so it’s very interesting.

    Shimel: Mitchell, it sounds like an analysis question.

    Ashley: Well, I was thinking about that very much, and I appreciate Brendan’s perspective on it. Because any time you have a major paradigm change, you have early adopters and then you have a long tail of how that adopts out. And you’re talking about the Netflix. People are now starting – how long have we been talking about Netflix as the example? As long as we’ve had DevOps and continuous deployment.

    And I think we have a bit of the same challenge here of instead of still having layers of our stack and different responsibilities or processes or things that go into creating environments, whether it’s from code or not, when we say it’s all going to be based on a pull and it’s all going to happen in its own environment, if you will, that’s a bit of a big pill to swallow for a lot of organizations, especially in complex applications. I’m not saying it’s not going to happen, but I think it’s going to be slow to adopt. And I’m sorry if I’m going to get a bunch of hate mail.

    It’s a great concept, but I do think it’s going to be more difficult to adopt than even containers were for breaking down applications. Not that it’s not a good idea or the right idea, it’s just instead of software architectures, modern software architectures being something the developers now get their mind around with containers and microservices and fabrics, meshes, now you’re talking about different parts of the organizations going into infrastructure and ops. And that’s all getting compacted together and you’ve got multiple parts of the organization getting behind that paradigm shift. So that’s my view of it.

    Will we see things in the market around GitOps? Of course we will. But I don’t think we’re going to see this tipping point that will happen rapidly. That’s my assessment.

    Shimel: Okay. So it seems like you and Tracy have formed some sort of coalition here, and –

    Ragan: [Laughter]

    Ashley: I always go to Tracy when I need something _____.

    Shimel: Right. Send the hate mail that way.

    Ashley: _____ Brendan and Parag, too. [Laughter]

    Shimel: Right.

    Ragan: I want to say that the CI/CD pipeline can adopt GitOps quite efficiently. Think what Brendan pointed out. When you think about CI/CD you think everything’s a push. Right? And you add a script that you’ve executed that your CI/CD server, your orchestration server, your job scheduler said, “Hey, there’s a step here.” Then that step goes out and pushes an update to production environment.

    How that’s going to – because GitOps is a pull methodology it doesn’t have a promotion methodology. And what really we’re going to see is for folks using GitOps is we’re going to see the CI/CD process do the – create the pull request. And there will be an approval. If you want a pull request to go into production, there’s going to have to be probably a manual step to approve that pull request, and then once it gets there the CI/CD pipeline creates it. Now the challenge will be having the CI/CD pipeline automatically update all of those declarative scripts. [Laughter]

    Because there will have to be a different one based on the environment. But that is totally doable and it doesn’t change that. But what it does do, it says, hey, CI/CD. You need to have a hermetic process. It needs to be airtight. You need to have a single source of truth.

    What that is ultimately will still be called GitOps, even if it’s not Git that is that single source of truth. It’ll be like we reach for a Kleenex, not a tissue. [Laughter]

    Doshi: Let me back up what Tracy just said, and maybe take the view of – I think you said 30% of the listeners are looking at this from a business angle. And your question also sounded like what does this mean as our industry, to our industry? So today we’ve got, I don’t know, 170 different products, if you will, and tools around – and Brendan can probably say, no, it’s now 250 or something [Laughter] – around DevOps and all the new different concepts.

    Shimel: Closer to 700.

    Doshi: Oh, okay. My bad. [Laughter]

    I was a little behind. So we have so many things here, and there’s a reason for that too, right? So we can’t just say, hey, let me take a cloud provider’s full CI/CD DevOps vision and hope that we’re going to impose one way of doing things, because this is not all worked out. The moment you bring the developer mindset to other layers of the computer stack – infrastructure, environment, operations – it becomes so complex. And I think at an industry what it’s going to do is it’s going to actually spur a lot of opportunity, a lot more innovation.

    I don’t know if that means that 750 will go to 1,050, but there’s so much room for different companies that will do things differently – push, pull, I need an approval before production due to a segregation of duties. No, no, no. I want to approve before I even commit the code or at least run the security static analysis, the dynamic analysis, the container image scanning. Everybody will have different ways of doing things, and I think it’s going to spur on lots of opportunities for a lot of companies.

    Shimel: Fair enough. A giant petri dish. Guys, I’m going to need to pull the plug on us here because we’re up against our time limit. But what a great conversation. For our audience, in a couple of weeks – and I don’t have the date in front of me, I apologize – we’re actually going to do an interact roundtable on this subject. Some of our panel members will be back for that.

    And we’re going to allow you to come online and ask questions and comment and join the conversation on this. I think it’s a great conversation. So be on the lookout for that and come check us out. But in the meantime, Tracy, as always, you know, you bring so much to this. Thank you so much. Thank you so, so, much.

    Ragan: Oh, it’s my pleasure. Absolutely my pleasure.

    Shimel: Yes. Brendan, always a pleasure to have you on, my friend.

    O’Leary: Yeah.

    Shimel: Look forward – also, Brendan does a show with us here on TechStrong TV every month. It’s called The Lab. And he discusses – you know, the 22nd of every month is a new GitLab release and Brendan gives us the lowdown on each month’s release. It comes out once a month. It’s called The Lab, and it’s available here on TechStrong TV. So Brendan, thanks so much for that.

    O’Leary: Yeah. Thanks, Alan, for having me. I always learn so much from these panels. So hopefully we all learned something today, but I really enjoy doing them.

    Shimel: Always a pleasure. And you add as much as you learn, Brendan. Thank you. Parag, one question for you. You mentioned earlier that you were a former physicist. Is there such a thing as a former physicist? Isn’t it more like I think, therefore I am?

    Doshi: Yes.

    Shimel: Once a physicist, always a physicist?

    Doshi: It certainly is Hotel California. You’re absolutely right. [Laughter]

    Shimel: Yep. But thank you so much for joining us. Always great to have you here and bringing some great perspective to everything. I appreciate it.

    Doshi: Thank you.

    Shimel: Mitchell, what can I say? We wrapped another one, Mitch. Good stuff.

    Ashley: Yeah, we did. And I’m just so glad that Parag didn’t bring quantum physics into this, because I’d be in trouble.

    Shimel: That could be another show we’ll do.

    Doshi: [Inaudible comment]

    Shimel: Quantum computing. But, no, anyway, that’s going to wrap up this episode of DevOps Unbound. We hope you’ve enjoyed it. This is every other week we do a fresh DevOps Unbound, so be on the lookout. Every month we do the roundtable with the audience, so be on the lookout for that as well. This is Alan Shimel for MediaOps. You’ve – oh, before we go, many thanks to Tricentis for sponsoring DevOps Unbound. We couldn’t do it without them. So thank you all very much. We’ll see you on another episode.

  • Teleport Adds Database Support to Security Gateway

    Teleport Adds Database Support to Security Gateway

    Teleport announced today it has added support for databases to a security gateway, delivered as a cloud service, that is currently used to secure Linux servers and Kubernetes clusters.

    Ev Kontsevoy, CEO, said Teleport intends to extend the reach of the security gateway up the entire stack of IT infrastructure that developers routinely use to remotely access databases. Initially, Teleport 6.0 adds support for open source PostgreSQL and MySQL databases.

    Teleport has gained traction as an access management tool because it provides a single binary that can be easily deployed by developers to encrypt IT platforms and enforce session controls. It automatically discovers all the resources in an IT environment, along with the protocols needed to access them. Teleport also provides a unified audit log, in a JSON format, that enables IT teams to track what resources are being accessed, when and by whom.

    Other features include the ability to authenticate identities via built-in single sign-on (SSO) and multi-factor capabilities, discover and see all database instances, integrate with command line interfaces (CLIs) and a set of workflows for managing privilege escalation.

    Access control is becoming a bigger issue due to a number of factors. The overall IT environment is becoming more complex as the infrastructure stack required to run applications continues to grow; most IT teams continue to work from home to help combat the COVID-19 pandemic. Teleport provides a fine-grained approach to streamlining access management across servers, clusters and databases in a way that eliminates the need to rely on virtual private networks (VPNs). In effect, Teleport enables IT teams to move toward a more identity-centric approach to unifying the management of access, Kontsevoy said.

    That’s critical at a time when the rise of DevSecOps is pushing responsibility for security further left toward developers, Kontsevoy added. While the rate at which that transition occurs will vary widely, Kontsevoy said it’s clear developers are taking more control over access control as part of a broader effort to roll out secure applications, faster.

    It may be a while before DevOps teams are able to aggregate all the access control protocols they currently employ under a single proxy. However, Kontsevoy said, the massive amount of time and effort spent managing access control today will drive DevOps teams to look for ways to automate what is currently a very manual process.

    Less clear is what role cybersecurity teams might play in supervising access management. The days when a separate overlay was needed to manage access may be coming to a close.

    In the meantime, it’s clear VPNs were not designed to address access management, in an era where the bulk of IT teams – and the employees they support – are working from home. More employees may be soon be heading back to the office more frequently, but it’s likely that remote work will remain prevalent. That means legacy approaches to managing remote access that were never designed to scale will becoming increasingly obsolete.

  • How to Know if You Need Multi-Cloud — and Succeed if You Do

    How to Know if You Need Multi-Cloud — and Succeed if You Do

    There are many good reasons for running in multiple clouds, as well as times when multi-cloud environments are unavoidable due to organizational history or legal reasons. However, operating in multiple clouds adds complexity, increases operational overhead and can become expensive. Organizations should only pursue it when they have clear business reasons for doing so.

    In this article, I’m going to cover some common situations when organizations can get value out of multi-cloud as well as strategies for success. Because multi-cloud is hard, the first question organizations should ask themselves is, “Do we really need multi-cloud?” So, the first order of business is to outline when technology leaders are better off avoiding multi-cloud altogether.

    When Multi-Cloud is a Bad Idea

    A single application and its data should never span multiple clouds, if it can be avoided. Here are some common reasons organizations adopt a multi-cloud approach when they would be better off avoiding it.

    Short-term bursting (in hybrid cases). “I’ll just tap into a public cloud if my private cloud suddenly needs extra capacity.”

    Disaster recovery. “Let’s run this application on AWS and Google in case one of them disappears overnight.”

    Staying cloud-agnostic. “By running on multiple clouds, we are no longer locked in.”

    In all the cases above, network latency and egress cost will hurt your application and your business. In the best-case scenario, you will spend a lot of time and effort prematurely optimizing for unlikely scenarios.

    Unless you’re perfectly prepared, please, don’t go there.

    When Multi-Cloud is a Necessary Evil

    There are also plenty of times when multi-cloud is a drag on the engineering organization and a drag on the business — but can’t be avoided. Here are totally legitimate reasons to have a multi-cloud setup that will still require more effort to maintain than operating in a single cloud.

    Mergers and acquisitions. Some organizations can end up operating in public clouds and private clouds as they acquire other companies. The effort required to migrate everything to one cloud provider often isn’t worth it, so everyone continues using the same cloud they used pre-acquisition.

    Legal. Depending on the type of data you gather and the jurisdictions you operate in, you may have to worry about where your cloud provider has regions available. There will be times when you need a data center in multiple jurisdictions; none of the cloud providers are in all of them. In those cases, you don’t have much choice, you need to be multi-cloud.

    If you need to use multiple cloud providers, either for legal reasons or because of mergers, it’s best to focus on making multi-cloud management as painless as possible — while still accepting that it will not be as smooth an experience as running all workloads in a single cloud.

    When Multi-Cloud is a Solid Strategic Move

    There are also some very good reasons to adopt a multi-cloud approach on purpose. For example:

    Facilitating engineering speed. Engineering speed is paramount. Companies who focus on using multi-cloud to become cloud-agnostic and avoid lock-in often sacrifice development velocity, but if the organization instead uses multi-cloud to allow everyone on the team to use the tools and environment they are most comfortable with, it can increase velocity.

    Best-of-breed tooling. The cloud providers’ offerings are not totally identical. Multi-cloud can let organizations use the best tools from each cloud provider; for example, machine learning technology from Google, compute from AWS and business intelligence from Azure.

    Better customer experience. Especially if latency is a big concern, a multi-cloud approach can help organizations get physically closer to users, providing a better customer experience.

    There is a common thread here: When evaluating whether or not you have a legitimate need for multi-cloud, the key question is whether you will be fulfilling a psychological need, a theoretical need (in other words, something that you do not actually need at this moment but might at some future date) or a concrete legal or technical need that you can easily describe. If there are no concrete legal, technical or business reasons for pursuing multi-cloud, don’t. There are costs to pursuing multi-cloud, so organizations that won’t actually reap benefits from it shouldn’t attempt it.

    How to Succeed

    Multi-cloud is hard. Here’s how to increase your chances of success so you can get the business benefits of better tooling or better customer experiences from your multi-cloud deployments.

    Use a cloud management platform. Getting a single pane of glass for all your deployments is critical to managing multiple clouds without pulling your hair out. This abstraction, however, comes at the cost of supporting very deep and cloud provider-specific use cases, for example, using obscure service X.

    Standardize workflows with Terraform and Ansible. You need to have processes that are as repeatable and predictable as possible. Even with standardized workflows, you still have to adjust for each environment, but the more standardization the better.

    Use Kubernetes. Kubernetes lets you abstract the infrastructure layer and move workloads around more easily, even across clouds. The caveat here is that you have to adopt microservices and deal with the increased complexity of managing Kubernetes and its underlying infrastructure.

    Each of the above approaches has its merits and flaws. They are not mutually exclusive, and you should not treat them like they are. Be prepared to mix and match depending on your needs.

    Weigh the Costs and Benefits

    Sometimes companies are not realistic about their technical capacity or what their needs are. They want the ability to do the latest thing they read about without thinking about whether or not that technology or approach will provide business benefits.

    Multi-cloud adds a huge amount of complexity to your deployment. If multi-cloud truly is essential, however, use tools and platforms that hide that complexity from users and can simplify the management process.

  • How to Build Actionable Logging Systems

    How to Build Actionable Logging Systems

    Logging is often considered an automatic process that doesn’t require much thought. We see logs, in general, from any system and take for granted that those logs have the information that’s needed. However, thinking through a logging strategy and architecture is extremely important for troubleshooting and performance efficiency. Regardless of how you log, the most important part of logging, after simply having logs in the first place, is having logs you can actually use. Actionable logs provide enough information, enough detail and enough history to ensure you can use the logs to get something done.

    Let’s examine an example system. This example system is an application, with a microservices architecture, running on a Kubernetes cluster that lives on top of an RHEL box. By default, you get logs from the operating system and Kubernetes. But to get logs for the different microservices, you have to program those logs into your application while building the codebase. So how do you set up useful, actionable logs successfully?

    Three Main Considerations

    Consider the following elements when you think about building actionable logs.

    Enough Information

    Your logs should provide enough information to answer ‘when,’ ‘who’ and ‘where’ for any event. As an example, let’s say you want access logs. If there was an access event for your system, the log line for that access event should tell you when that access occurred; this is known as the timestamp. The log line should also include information about who accessed the system. Often, this detail comes in the form of an IP address or other identifier, and the ‘who’ should include not just human interaction, but also actions by other systems. Always log every access! Finally, the log line should include the location from where the system was accessed, meaning which service or which part of the platform was accessed.

    Enough Details

    Your logs should provide enough details to tell you exactly what happened during any single event. Simply answering the who, when and where doesn’t actually tell you what happened, and a basic answer such as “This pod started,” doesn’t provide enough detail to differentiate a code change triggering a pod restart from a system crash triggering the orchestrator to restart pods.

    Enough History

    Finally, your logs should provide enough history to understand why something happened. Logs, overall, help you understand the context of the action that was taken. Writing log lines to a terminal somewhere without storing them is pointless. A single log line doesn’t provide you with enough context to understand the “why” for any action or event. How could you tell if an event was normal for a system without an easily accessible history?

    All three of these components are necessary to create a plan that will lead to actionable logs—logs you can actually use to understand what happened, why it happened, and how you can act to handle the situation. If you can answer who, what, where, when and why from your logs, you probably have actionable logs, and are well on your way to meeting or exceeding logging best practices.

    Next-Level Considerations

    Once you’ve thought through those higher-level considerations, you can think about the nitty-gritty of your logs themselves. When I think about the practicalities of logging, I think about these main considerations: log levels and data needs, format and structure and security and compliance

    Log Levels and Data Needs

    Using log levels properly ensures that you collect and store the data that you need, and only that data. Logging can get very noisy, very fast in modern environments. Log levels provide the ability to fine-tune your logs to store only what you need; this can help you find information quickly in an emergency, rather than digging through the haystack for the proverbial needle. In addition, someone is more likely to keep logging capabilities turned on when it gets noisy if they have the ability to reduce the noise and fine-tune the information, versus having only an on-off switch.

    Different programming languages use different combinations of log levels, but all of them have general categories of major, minor and debug logs. Major logs, as you might guess, should be on for all environments. Minor logs, generally, are turned on in environments for which you need to be able to understand the data flow without necessarily knowing the minute details. Finally, debug logs are generally turned on in dev or QA environments, and only when you need to know every detail – such as when you’re looking for issues that could turn into problems later on. You will need to ensure the necessary log levels are set in your code, as they aren’t necessarily automatic, but taking this extra step is worthwhile.

    Format and Structure

    You need to think about who or what is the primary audience of your logs. If you anticipate sifting through the logs yourself, you’ll likely not mind a text-based format. However, most modern logging uses machines to parse the logs and make the mounds of data more useful. In that case, you should use structured logging; JSON is the standard for structured logs due to the plethora of tools available for working with the standard.

    Security and Compliance

    Logging for security and compliance could be (and has been!) a full article, on its own. The short version, however, is that logging is a key pillar of a compliant system that is aware of, and actively monitoring, its security. If your stack doesn’t log everything to one or more append-only log files, stored in secure location(s) with a read-only archive stored in a separate location, then you’re at risk of an attacker coming in and modifying or deleting logs so you can’t track their actions across your system.

    Your logging processes shouldn’t store personally identifiable information (PII) in clear text or behind an easily broken cypher, either. It’s better to keep that PII in a secured and hashed database, with a unique identifier (UID) that correlates to the database, if you definitively need to keep that PII in your logs in some form, or strip it out if you don’t need it.

    Working Together

    All of these tips are fine and dandy, until you realize that you cannot get enough buy-in from others to ensure everyone follows the same best practices. It’s generally best to come to an agreement as to logging standards and practices across an organization. That will also help, as anyone working on a different system will start from the same baseline, and not have to understand another team’s standard, if they come in to help in an emergency. Ops teams working across many systems will appreciate the standards, as well. To get everyone to the table in the first place, start by appealing to each of their needs. Ask about people’s problems with logging and listen to responses. You likely will need to address assumptions folks make, misconceptions, complacency and resentment. The key is listening; you won’t get buy-in until everyone feels heard and validated.

    To get everyone on the same page, you should first identify, as an entire organization, the needs you have from your logs based on the components of actionable logs. Ensure everyone has a seat at the table and the ability to discuss and unearth all needs, so no one is tempted to go around the team’s decision. One team’s needs will not match another’s, so you need to ensure that all viewpoints are heard and accounted for. Then, organize related needs and set standards for those needs. Once you’ve agreed to your new standards, work together to update your systems. This will take time, just like it does with resolving any technical debt! Start with smaller systems, or less-critical systems, so you can monitor how the standards are surfacing in real life, and how well everyone’s needs are met. Adjust, if you need to, with everyone’s agreement. Needs and standards will change, just like your systems will, and they need to adapt to fulfill the needs of your teams. If you take the time to get everyone to the table and make sure everyone is on board, you’ll be better off.

    What You Need to Succeed

    Logging may seem like simple magic that ‘just works,’ but there’s a lot to think about before you can have a successful, useful system. You need actionable logs that have enough information, details and history to be useful. You need to think about the data you need for all of your various systems, how you’ll structure that data and how you’ll ensure you stay secure and compliant. Finally, you need to ensure your team is on board with standards so logging can be as useful as possible. If you do all of that, though, you’ll find logging becomes a great way to communicate across teams and organizations to ensure you deliver the best product possible.

  • Grafana Labs Adds On-Premises Observability Platform

    Grafana Labs Adds On-Premises Observability Platform

    Grafana Labs announced today the availability of a unified Grafana Enterprise Stack that enables IT teams to deploy a curated observability platform, based on open source tools such as Prometheus, in their own on-premises IT environment.

    Tom Wilkie, vice president of product for Grafana Labs, said Grafana Enterprise Stack, in addition to providing a version of Prometheus optimized for distributed computing environments, includes curated instances of open source Grafana dashboards, Loki log aggregation tools and the Cortex query engine.

    Grafana Enterprise Stack also includes plugins for visualizing data from other commercial tools in Grafana, such as Splunk, New Relic, MongoDB, ServiceNow, Oracle and Snowflake. In addition, it includes reporting tools and enhanced authentication and security controls.

    Grafana Labs

    Developed under the auspices of the Cloud Native Computing Foundation (CNCF), Prometheus has gained traction as a monitoring platform used primarily in Kubernetes environments. However, Wilkie said it’s only a matter of time before IT organizations expand their use of Prometheus to include a wider range of platforms. The Grafana Enterprise Stack facilitates that transition by making it easier for organizations to deploy a complete observability platform based on open source software, said Wilkie.

    While open source software has a key role to play in enabling observability, most IT organizations today rely on a range of proprietary tools to monitor IT environments. Many of those tools will be replaced, in the months ahead, by observability platforms that provide more context than individual monitoring tools currently do. However, it’s not clear yet if IT teams will prefer open source or proprietary platforms that, in most cases, incorporate open source components.

    As IT environments become more complex, the tools most organizations rely on today don’t surface enough actionable insights. Most IT teams today still waste a fair amount of time navigating amongst multiple tools in an effort to discover the root cause of a problem. It’s not uncommon for IT teams to take days to discover the source of a problem that is then fixed in a few minutes. The expectation is that observability platforms will sharply reduce the amount of time and effort required to investigate, for example, a performance issue.

    Of course, to achieve that goal, applications themselves will need to become more instrumented. A mix of open source and proprietary agent software will make it easier to collect traces, logs and other metrics at a reduced cost. In many cases, IT teams have been reluctant to fully instrument an IT environment, because of both the cost of proprietary agent software and the inherent challenges associated with deploying and managing it across an extended enterprise. However, as the deployment process for agent software becomes more automated, organizations are finding it easier to instrument their applications.

    It may be awhile before IT environments are fully instrumented. However, as it becomes easier to collect traces, logs and metrics, the level of observability that can be achieved across an enterprise will undoubtedly increase. The challenge and the opportunity then becomes employing the greater visibility provided by these platforms to manage the sprawl of applications and platforms across IT more proactively.

  • The State of Digital Innovation, One Year Into the Pandemic

    The State of Digital Innovation, One Year Into the Pandemic

    It’s obvious COVID–19 accelerated digital reliance for many companies. And it’s no surprise that, according to a recent study, 89% of tech leaders believe digital experiences are critical to addressing new COVID–19 challenges. In this race toward 100% digital business, agile innovation is the new elixir of life — without it, a company may face an early retirement.

    Kong just released their 2021 Digital Innovation Benchmark, which summarizes the tech priorities of 400 IT leaders. The findings stress the importance of rapid digital innovation, open source, microservices architectures, and high quality practices for scaling today’s application ecosystems. Here are the key points technology leaders need to know to remain competitive.

    Innovate or Dissipate

    The urgency to innovate is increasing. According to the survey, 84% of respondents predict a business will go under within six years if they lag behind in digital innovation. Fifty one percent of tech leaders believe their business can only survive three years.

    If digital transformation happens too slowly, it can pose an existential threat. But there are many factors that can hinder faster digital innovation; the complexity of using multiple tech stacks, reliance on legacy IT and lack of automation. Not only are there internal woes, but external danger — 62% of respondents worry about failing to keep up with competitors who innovate faster.

    It seems the urgency has increased over the last year. In last year’s report, just 37% reported they’d be out of business in three years if they didn’t keep pace with digital innovation. That figure rose to a whopping 51% in 2021.

    COVID Amplifies Digital Drive

    Virtually no one is anticipating a decreased IT budget in light of COVID–19. On the contrary, long-term digital transformation efforts are well on target at the majority of institutions, and 75% expect to see an increase in IT/developer budgets over the next 12 months.

    Interestingly, when it comes to securing the digital-only economy, European tech may have a head start. Only 40% of European tech companies have increased their security focus, compared to 51% in the U.S. This could be due to previous investment in security and stricter regulation in European countries.

    In terms of cloud adoption, 40% of tech leaders feel COVID–19 has accelerated their move to the cloud. Nearly 40% say the pandemic has encouraged the use of microservices and cloud native architecture.

    Quality Outweighs Speed

    When gauging what matters most to tech leaders, “improving operational efficiency” is still a top priority. However, this year’s report saw a spike in respondents saying application performance and reliability, as well as application security, were moving up the priority list. Though 3 in 5 (62%) respondents said they are “extremely concerned” about competition from speedier rivals, application stability outweighs the collective drive to innovate.

    These trends are likely indicative of a notable rise in cybercrime. As I’ve covered previously, to respond to emerging threats, DevSecOps practices and new security tooling are shifting left within the development cycle, automating things like vulnerability detection. Simultaneously, roles to increase application integrity, like site reliability engineers, are in high demand.

    New Technologies and Open-Source Thrills

    As mentioned earlier, IT budgets are increasing across the board. When adopting new technologies, companies place the highest emphasis on the tech that increases efficiency. A surge in automation follows a low-code movement gaining steam. Companies consider the following key factors when introducing new tech: reducing deployment risk, increasing collaboration, and increasing development. Ninety five percent of US respondents also listed the “need to integrate with existing tools” as necessary — no surprise in today’s API-driven development landscape.

    Ninety one percent of organizations use open-source software in 2021 — this is up from 82.5% in 2020. Across the board, open-source is intrinsically embedded into most software projects. We see open-source especially used in open databases, infrastructure automation, API design and testing, CI/CD tools, Docker containers, Kubernetes and service mesh. Arguably, modern DevOps would not be possible without such open source initiatives.

    Bring it on, Microservices

    If you haven’t heard of microservices yet, we would be worried. An impressive 87% of survey participants believe that microservices-based applications are the future.

    While that sounds substantial, a path toward 100% distributed microservices architecture has not been mapped within most organizations. In 2021, only 33% of companies have transitioned entirely to a distributed architecture, whereas most (53.5%) adopt a hybrid mix of microservice and monolithic styles.

    “This is strong evidence that microservices have quickly become the de-facto approach for modern application architectures that enable digital innovation,” the report said. Though more development is required, a microservices-centric future seems obvious.

    Within organizations that have already adopted a microservices approach, the average number of microservices in production is 102. And those are primarily to increase security. Ironically, the report also found maintaining security to be the top challenge of deploying a microservices system. Other drivers for microservices adoption include integrating new tech, increasing development speed and boosting flexibility.

    Kubernetes Becomes Ubiquitous

    Kubernetes has clearly emerged as the industry standard orchestration platform. Eighty six percentof all respondents report they now use Kubernetes in production, or plan to within 12 months. This is a massive jump from VMWare’s 2020 report, which put Kubernetes use at 48%.

    However, there are challenges to implementing Kubernetes in practice. Companies cite security, complexity, performance and monitoring as top setbacks when introducing Kubernetes. To say Kubernetes has a steep learning curve may be an understatement, as abstraction layers are introduced to rectify a widening knowledge gap.

    Service Mesh Prickly for Early Installs

    Interestingly, API gateway adoption and container usage, according to the survey, are both 38%. Yet, service mesh adoption is only at 20%. This could underscore the fact that service mesh is still an emerging concept — service mesh experts say the tech isn’t ready for widespread adoption. Or, it could reflect that production use cases are rare (primarily for supporting vast microservices deployments).

    Early service mesh adopters see increased service connectivity and discovery as a top benefit of the technology. This seems logical, as service mesh aims to be a common networking framework. Other common benefits include cost reduction, meeting compliances, zero-trust security in networking and traffic observability.

    Naturally, service mesh isn’t without its thorns. Complexity in deployment and ensuring performance at scale top the list of challenges faced when deploying service mesh, slowing its adoption.

    Hybrid Multicloud

    The current status of multicloud deployment sees a mix of on-premises and multiple cloud providers, with 41% of organizations using multicloud deployment strategies. While cloud adoption rates are increasing, 31% of organizations are still running on-premises servers. The saturation of cloud tooling, and the market validity of Azure and Google Cloud Platform to take on AWS, have influenced a multicloud environment. Cloud expertise is now a hot commodity to navigate multicloud modalities — so much so that some companies have introduced a new role, The Other CFO, or Cloud Economist, to balance the cloud checkbook.

    Riding out the Virtual Wave

    The pandemic has accelerated the move toward digital experiences, and in turn, quickened the pace of enterprise technological evolution across the board. As COVID–19 effects continue, there’s no indication the virtual wave will recede anytime soon.

    To summarize the current state of digital innovation:

    • Rising digital tide: The pace to innovate is rising, and those left below the new-digital-normal water line will drown.
    • IT spend is up: Most companies have increased IT budgets moderately to support new digital norms.
    • Needs for adoption: Desires for automation, speed, security and integration rank high for adopting new tech.
    • Microservices on fire: The report describes “accelerated mainstreaming of microservices.” Microservices, and the tech that supports them, are clearly the future.
    • Kubernetes everywhere: As Kubernetes rises to ubiquity, last-mile efforts center on security and developer usability.
    • Bullish on service mesh: Service mesh sees early production uses, which will likely signal more growth in this area.

    There’s nothing unexpected in these findings; instead, they affirm what most of us already know — digital innovation is going full throttle. Check out the entire 2021 Digital Innovation Benchmark here.

  • Gravitational Becomes Teleport to Streamline Access Management

    Gravitational Becomes Teleport to Streamline Access Management

    Gravitational today announced it has changed its name to Teleport and launched the Teleport Unified Access Plane, an open source security gateway that comes in the form of a single binary that acts as a proxy for remotely accessing Linux servers and Kubernetes clusters.

    Teleport Unified Access Plane (UAP) for managing those gateways can be either deployed by an IT team or accessed via a cloud service managed by Teleport.

    Offerings included in Teleport UAP are Teleport Server Access, Teleport Kubernetes Access and Teleport Application Access, which combined enable IT organizations to lock down access to an IT environment.

    Company CEO Ev Kontsevoy said Teleport UAP gives engineers via a single login access to just the IT resources they have permission to access. That approach reduces the myriad credentials that IT organizations currently need to manage across multiple networks.

    Teleport automatically discovers all the resources in an IT environment along with the protocols needed to access them. It provides a unified audit log in a JSON format that enables IT teams to track what resources are being accessed when and by whom.

    Access control is becoming a bigger issue because the overall IT environment is becoming more complex as the infrastructure stack required to run applications grows. Teleport provides a way to streamline access management across servers, clusters and databases.

    That issue is also becoming a bigger challenge with IT personnel now working from home to help combat the spread of the COVID-19 pandemic, added Kontsevoy.

    That fine-grained approach to access will ultimately eliminate the need to rely on virtual private networks (VPNs) at a time when the network perimeter continues to dissolve. In effect, Teleport enables IT teams to move toward a more identity-centric approach to unifying the management of access, he said.

    In general, he noted, Teleport enables organizations to manage security as code rather than having to rely on separate overlays. As such, they can shift responsibility further left toward DevOps teams.

    It may take some time for IT teams to aggregate all the access control protocols they currently employ under a single proxy. However, Kontevoy said, the massive amount of time and effort spent managing access control today will drive organizations that have embraced best DevOps practices to look for ways to automate what today is a very manual process.

    The challenge, of course, is finding the time to achieve that goal. Teleport is betting a single binary in combination with a control plane accessed primarily as a managed service will make it too simple for DevOps teams to ignore. Regardless of approach, however, it’s clear there needs to be a better way to manage access within the sea of servers, databases and applications that now make up the typical IT environment.

  • NGINX Expands Portfolio to Include Service Mesh

    NGINX Expands Portfolio to Include Service Mesh

    NGINX at its online Sprint conference today launched a bevy of offerings including a service mesh as part of an expanded effort to meld network and security operations within the context of a larger DevOps workflow.

    Rob Whiteley, chief marketing officer for the subsidiary of F5 Networks, said NGINX Service Mesh 1.0 provides an easier alternative to deploy a service mesh based on the open source NGINX proxy server software already widely employed in enterprise IT environments. That lighter-weight service mesh is also integrated with the NGNIX ingress controller to enable it to be employed across microservices spanning both legacy virtual machine environments and Kubernetes clusters, he said.

    In general, Whiteley noted, most organizations will not need a service mesh unless they are looking to integrate more than a thousand microservices. If an organization only has hundreds of microservices, an ingress controller should suffice, and anything involving less than a hundred microservices can be handled by proxy software.

    NGINX Service Mesh 1.0 will be offered by NGINIX and F5 Networks will continue to offer a service mesh based on the Istio project for Kubernetes environments that relies on Envoy proxy software, which is not as lightweight as the NGINX platform, noted Whiteley. DevOps teams will have to decide which service mesh to deploy.

    In the meantime, NGINX is also updating its NGINX Controller to add support richer analytics, a declarative application programming interface (API) for continuous integration/continuous delivery (CI/CD) integration, role-based access controls and enhancements to the developer portal for API management.

    NGINX is also driving tighter integration between Controller and NGINX App Protect, a web application firewall that will inherit the API, analytics and self-service capabilities of Controller. NGINIX App Protect is also now being integrated with NGINX Ingress Controller for Kubernetes. The Controller App Security tool that enables those integrations is in private beta now, with general availability expected by end of 2020.

    Finally, NGINX is now leveraging services made available by F5 Networks via software-as-a-service (SaaS) platforms in the form of NGINX Analytics Cloud, which replaces NGINX Amplify. That capability will be available in the first quarter of 2021.

    In general, Whiteley said NGINX is trying to serve the needs of traditional network operation teams that want to provide self-service capabilities to development teams as well as via programmatic interfaces to DevOps teams. In many cases, enterprise IT organizations are employing both approaches to expose network services, noted Whiteley.

    Of course, the issue that ultimately might drive the convergence of NetOps and DevOps is the rise of the service mesh. In addition to integrating microservices, the service mesh provides developers with a consistent set of interfaces for invoking network and security services spanning a dizzying array of servers and appliances. As that transition occurs, IT teams will become much more agile than they are today. The challenge, of course, will be melding network operations and DevOps cultures that today don’t have all that much in common.