— ROELBOB
Category: Blogs
-
State of DevOps Survey – The Ultimate Feedback Loop
If there’s any consistent criticism of the larger DevOps movement, it’s that the talking heads and industry gadflies are driving the conversation, not the people in the trenches.
All the thought leadership panels and “how-to” content (including the written and AV materials on this very site) make for great watercooler discussion, but lack the input of those dev, ops and IT management pros who are actually living the DevOps experience, some critics have said.
Well, fair enough. Talking heads are paid to talk. Thought leaders are chosen to lead the debate at conferences and on webcasts. Bloggers blog. Reporters report. This is simply the way of the world. Certain voices are heard via the channels they frequent. It is, as so many people are fond of saying these days, what it is.
So what about everyone else? What of the people reading this stunning prose and listening to these talks while rolling their eyes based on the connectivity of it all to their real-world experience? How can their voices be heard?
Independent of jumping into the blogsphere or applying to speak at said industry events – which you should, btw – there’s one significant opportunity available to anyone who cares to take the time – participating in industry research.
Over the next few business days, citizens of the DevOps world will have their last chance to contribute to the industry’s most widely read and respected research project, the State of DevOps Report.
If you haven’t done so already, take the time to fill out this year’s survey, and make sure that your voice is heard when the subsequent report is published in June. Sure, it takes a little time, about 15-20 minutes; but that’s because it goes to the trouble of exploring the real-world issues that weigh most heavily on today’s practitioners.
Why is this particular project so important, or even credible? Consider that the minds behind the survey and analysis resulting in the 2016 State of DevOps Report are those authors and talking heads that actually get the most respect.
Backed by DevOps Research and Assessments (DORA), the independent think tank operated by a partnership of Jez Humble, Gene Kim, and Nicole Forsgren – perhaps the most respected voices on the landscape, along with Nigel Kersten, the project is categorically aimed at surfacing the truth about DevOps adoption today – not merely promoting the interests of vendor sponsors.
Like previous iterations of the report, the 2016 edition will focus on topics including IT performance, technical practices, culture, lean management, and organizational performance. This year’s survey also examines several new areas, including the ROI of IT performance, security and compliance, and containers.
To further entice participation, project leaders are also offering free tickets to the DevOps Enterprise Summit, along with forthcoming books from some of the industry’s leading advocates to a lucky few who take the time to get involved.
DevOps success is fundamentally based on the notion of increased collaboration and continuous feedback among the people scattered across the SDLC. Likewise, the legitimacy of this research project is utterly dependent on the input of as many real-world practitioners as possible.
If you’re one of those individuals, and you really care whether or not everyone else involved in the advancement of DevOps understands what this all means to you, do yourself a favor and participate.
Much like an application being built using DevOps practices, this quality of this project will be directly determined by the degree to which anyone with insight is willing to get involved and build toward optimal results.
-
Optimizing an Effective CI/CD Pipeline
Continuous integration and continuous delivery (CI/CD) are often cited as pillars of successful DevOps. To establish and optimize the CI/CD development model and reap the benefits, companies need to build an effective pipeline to automate their build, integration and testing processes. This requires architecting CI/CD with five key ingredients which, when combined effectively, operate as a successful cornerstone of DevOps and provide myriad bottom-line benefits.
Building the Pipeline
Let’s first take a look at the five essential ingredients needed for a healthy CI/CD architecture, following up with a discussion of their benefits to the business:
- Micro-agility in the form of testing and committing code frequently: The development life cycle begins with a request to development, which then takes the request and designs a solution, codes it, tests it and commits it a code repository. From here, the code is tested by QA and then deployed to production. While each step is important, we begin here, as local testing occurs most frequently and bugs found at this stage of the process help avoid the large expense of bugs found in production.To achieve micro-agility, developers must have a clear understanding of where they are expected to commit their code after changes are made and they should be expected to test and update their code frequently—many times a day. This means development testing should be easy to run locally and is a subset of QA. The result is that code pushed by development is by definition QA-ready.
- Automated QA: In my work I see many organizations whose QA team is one of the largest bottlenecks in the process. It’s not for a lack of skill, but too few staff who spend a good portion of their time waiting on the QA test environments they need to do their work. To avoid this situation, the QA process should be fully automated, from QA ready code coming in the door to one-click promotion on the other side. In between the two, QA should have a constant feedback loop with development where communication about—and tracking of—bugs is clear and easy to follow, allowing development to easily repair issues and recommit the code for further testing. Moreover, automated QA in a CI/CD environment should focus on microcycles consisting of small test batches rather than weeklong macro tests. Automation and communication are the overriding imperatives here, exhibited by reports to developers that are communicated immediately and code commits that are fail-rejected automatically.
- Production-like environments: To increase efficiency and effectiveness, development and testing environments should mirror production as closely as possible. They should follow the PRQ rule, as I like to call it—production-like, repeatable and quick. Specifically, it should be fast and easy to set up new environments, to the point where developers can, with the push of a button, deploy new environments for their development and testing needs. Doing so allows developers to fail fast, breaking and re-creating environments without fear as they know they can easily recreate the same environment every time. I often recommend using tools such as Jenkins to help with this process; if developers and IT operations use the same tool, it makes it that much easier to streamline the CI/CD process and achieve the promise of DevOps.
- One-click promotion: As I alluded to earlier, a well-architected CI/CD pipeline provides one-click promotion to production. This ensures changes are seamlessly moved to the overall production environment with the least amount of friction possible between development and operations. To that end, this reduction in friction should occur across all the steps of the process, with efficient code delivery pipelines that are designed to balance security and agility from the outset. Cloud infrastructure coupled with container technology and automation can easily optimize the process across teams and throughout the DevOps journey.
- Fully automated deployments: The last factor of an efficient CI/CD pipeline is full automation of deployment. Any configuration changes made by development and accepted by QA should automatically be applied. Continuous integration in this way helps avoid system downtime and human errors are decreased greatly. Deployments also should be continuously monitored so that IT is the first to learn of potential problems. This helps nip issues in the bud, allowing for faster rollback and return of code to development to fix anything that may not be functioning properly. Catching issues before customers do helps ensure customer satisfaction and strength of reputation.
Benefits of the Pipeline
In addition to these benefits, when combined effectively, these ingredients minimize the time and cost of local test, QA and production processes; reduce the time it takes to identify problems and their causes; and accelerate delivery of production-ready code. We can see this throughout the CI/CD pipeline:
- Developers have greater control over the process and can respond more quickly to market demands by quickly bringing up new environments, testing against them and easily starting over, if need be. Moreover, developers can quickly adjust to new environments—an approach that has proven to decrease failures and resolution time when they do occur.
- Operations benefits as reduced friction in the CI/CD pipeline created by automation decreases repetitive, manual work and with it the opportunity for “fat finger” mistakes and the greater opportunity to introduce inefficiency into the process.
- And, less manual, repetitive work provides everyone in the process with more time for strategic work, which provides direct bottom-line value to the organization.
CI/CD is a pillar of successful DevOps and according to research by CA, organizations that apply DevOps discover an 18 percent reduction in time to market of new applications and a 21 percent increase in delivering new software that they would not otherwise be able to complete. Teams that master an efficient CI/CD pipeline deliver faster, higher-quality applications to market bringing with it a 19 percent increase in revenue. Mixing these ingredients together results in an efficient, streamlined CI/CD pipeline that serves as a fundamental pillar of DevOps success that, when mixed together, delivers direct business benefits.
About the Author/Aater Suleman
Aater Suleman, CEO & co-founder at Flux7, is an industry veteran in performance optimization on servers and distributed systems. He earned his PhD at the University of Texas at Austin, where he also currently teaches computer systems design and architecture. As a faculty member at UT-Austin, Mr. Suleman focuses on optimizing DevOps architectures involving the AWS cloud. Along with the Flux7 consultant team, he works at every step of the dev flow, from creating local developer environments using Vagrant and Docker, to continuous integration and continuous delivery using Jenkins or RunDeck, to configuration management using Chef/Puppet/Ansible, to creating efficient systems for deploying new code from Git using Capistrano/Fabric/bash. His background is as an architect, and he has experience with both hardware and software. This combination brings a unique perspective and broad view to development and operations. -
DevOpsQA NJ Meetup: Jenkins with Gerrit and BDD Automation
DevOpsQA NJ recently held its April Meetup on Automation with Jenkins and BDD, with Kishore Bhatia from CloudBees and Tom Bartolucci from Billtrust as speakers. The venue, food and refreshments were provided by Audible at a state-of-the-art, vast office space in historical city of Newark.
After fun networking over pizza, wine and beer, we kicked off with Tom Bartolucci’s presentation on BDD Automation framework using Codeception. Tom has been in web software development for over 13 years. He has built applications in Java, .NET, PHP, Python and Javascript for clients in publishing and mobile video. Currently, he is a VP of Development at Billtrust – a SaaS-based payment cycle management c
ompany. His team of 13 web developers works on delivering six of the Billtrust SaaS products and uses Codeception, a PHP-based BDD framework to test.According to Tom (and we all agree), the idea is not to find bugs but prevent them. His team at Billtrust chose Codeception because it allows tests to be written in PHP, which is what its development teams use; it also supports Selenium WebDriver, PhantomJS and BrowserStack; and has hooks for various tools and platforms and build-in reports. As a result, it had a low barrier of entry and got Codeception’s team off the ground quickly. Doing cross-browser testing allowed them to find IE bugs, which they previously missed due to testing primarily in Firefox. Codeception allows testers to create BDD-style readable tests, similar to Cucumber spec but written as code. The easy syntax enabled the QA team to write code that they could read and understand. And developers are able to add custom modules because the framework is developed in PHP, which they are familiar with. Codeception has a variety of uses including to write acceptance tests, unit tests and API tests.
Tom also described how Billtrust integrated Codeception with Jenkins to execute tests and publish reports. He then proceeded with a live demo of the framework. At the end of the session there were a lot of good questions on Vagrant integration, parallelization, failed test rerun and differences between Codeception and Cucumber.
Next, Kishore B
hatia from Cloudbees gave a talk on Jenkins pipelines and Gerrit Code reviews. Kishore Bhatia works for CloudBees, building custom frameworks with open source software and helping customers solve engineering problems around continuous delivery and DevOps at scale. Kishore is a Jenkins expert with development experience in FinTech enterprise software and Cloud platforms. Couple of months back, Kishore and Issac gave a presentation on Jenkins Workflow, demonstrating how workflow removes bottleneck of connecting multiple pipelines together.This time, the focus was on a very valid use case: a large codebase, many teams contributing to the code. Teams should not collaborate on emails. They should get notifications and post-commit validations done by the tool that they use. In this case it’s Gerrit—Google open source project, offers its own implementation of Git. You just want the job to magically run behind the scenes doing the code reviews for you. The p
rimary challenges in this scenario are that you want to fail-fast—report CI results as soon as any steps fail and make sure there are no orphaned builds.Kishore then demonstrated the steps involved in implementing this use case in Jenkins including adding pull request validation to the build step, allowing developers to get build status directly in the VCS tool only coming back to Jenkins if something fails. Currently there is a limitation with posting each status to GitHub. Key things that need to be set up—Gerrit infrastructure, Jenkins installation, workflow that Jenkins will refer to. While it might look complex at first it becomes easier once you start using the workflow, with such things as Groovy script helper. Jenkinsfile will define the flow of what should happen in dev, QA, staging and production.
Kishore then gave a great demo of the flow and answered questions from the audience which covered parallel commits and test data management.At the end of the sessions, we did a raffle drawing where three lucky winners got prizes sponsored by Audible, including an Amazon Echo. Looking forward to our next session.
If you would like to learn more about Jenkins Workflow, you can reach Kishore at kbhatia@cloudbees.com. To learn more about exciting opportunities at Audible visit http://about.audible.com/ or reach out to sonatore@audible.com.
-
DevOps Chat: Threat Stack’s Pete Cheslock and Chris Gervais
I recently had a chance to sit down with Pete Cheslock and Chris Gervais of Threat Stack to talk DevOps and Security. Both of these guys are dialed in on both topics so as you can imagine it was a great conversation. Here is the audio file with the transcript underneath.
Pete Cheslock, Senior Director of Ops and Support – As the head of Threat Stack’s operations and support teams, Pete is focused on delivering the highest level of service, reliability, and customer satisfaction to Threat Stacks growing user base. An industry veteran with over 15 years’ experience in DevOps, Pete understands the challenges and and issues faced by security, development and operations professionals everyday and how we can help. Prior to Threat Stack, Pete held senior positions at Dyn and Sonian where he built, managed and developed automation and release engineering teams and projects.
Chris Gervais, VP, Engineering – As Threat Stack’s head of Engineering, Chris is passionate about building, not only a rock solid, high-performance product, but also a team of elite engineers, industry best processes and a culture that attracts the best talent. Prior to Threat Stack, Chris held senior positions at lifeIMAGE, Enservio, Partners Healthcare, Inc., Inflexxion, Inc. and VIS Corporation, where he was responsible for engineering, technical operations, and technology strategy for cloud platforms.Alan: Hi, this is Alan Shimel from DevOps.com for another DevOps chat. Happy to be joined today by two special guests, both from Threat Stack. First is a gentleman pretty well known within the dev ops and security space, Pete Cheslock. Pete, welcome.
Pete: Yeah. Thanks for having me.
Alan: Thank you. And joining Pete and I is, he’s kind of Pete’s partner in crime here, Chris Gervais. Chris, welcome.
Chris: Thanks so much.
Alan: Great. That was our audio check, guys, and we’re good. So Pete and Chris, why don’t – our audience may not be familiar with Threat Stack. Can you give briefly a little background on the company?
Pete: Yeah. Absolutely. So Threat Stack is a company that’s been around for a few years now, and what we’re doing is we have a cloud security platform that is basically designed to work with all kinds of different workloads that are running, whether in the cloud or in your own data center, to analyze for internal/external threat, to analyze for data loss prevention, or – and basically just auditing the kind of, answering the question of who did what when on your system.
So if you’re someone who is on a security team and you need to follow various auditing rules, PCI, or HIPAA, or things like that, you can use Threat Stack to basically comply with those policies. And also if you’re just an operations team and just trying to understand, you’re doing, kind of doing the dev ops. You’re giving people broader access to your systems a you’re trying to understand what they’re doing with that access, trust but verify, you can use Threat Stack to basically determine and answer those questions for you.
Alan: Fantastic. And Pete, is this delivered as a service, or is it – not an appliance any more, but and not even a on prem. But is it a service?
Pete: Yep. It’s a SaaS app. We have a very lightweight agent that runs on the system, and what we do is right now we support Linux systems. We’re capturing events from the Linux kernel, so it’s that source of truth of all activity on a system. And we also do some integration with if you’re on Amazon we do some integration with the cloud trail so that we can alert on not only your system-level events, but also for Amazon users to _____ on cloud trail events, so events that are happening at a higher kind of EPI level.
Alan: Great. And guys, I don’t – I didn’t mention your titles. I apologize. Chris, what’s your role at Threat Stack?
Chris: So I’m the VP of engineering here.
Alan: Okay. Excellent. And Peter?
Pete: So, I run the operations and support teams, the technical operations.
Alan: So we’ve got tech ops and engineering? Excellent. So guys, and we were talking a little bit before we started the recording here about this whole – the evolution, if you will, of security, you know, security tribe merging or at least joining into this larger dev ops tribe, if you will. I don’t know if it’s larger. But the merger of security and dev ops. Chris, I know this is something that you followed now for a few years. What’s kind of your take on it?
Chris: You know, I think it’s, like there’s a couple things here, one of which is, especially for folks who are building in the cloud and trying to take advantage of all of the sort of speed and velocity you can get with software to find everything and all that stuff is you can’t adhere to the model the security is the domain of the few any more, and that it’s a walled-off thing over in the corner, or that it lords over your software development engineering process, roll-out process, infrastructure process.
Like, all that stuff has to get baked in, right? And it has to get baked in at the earliest possible point that it can. And one of the best ways to do this, and it’s a thing we’ve done at Threat Stack is, it’s not just about the tools. It’s about making sure that your entire team is involved with this from the start, and that it’s not just a person who’s got to sit there and be a gateway and a worrier about this, but everybody is focused on it, right?
Just like today, you’ve got, whether you’re a software engineer or infrastructure engineer, you’re an engineer. You’re sitting together, and you’re solving problems together of how do I get my stuff out? What kind of systems do we need? What do my work loads look like? You’ve got to take that same approach with security and make it just part of that flow. And so in some ways, like if you think about the democratization of this stuff, it’s really about – and this is one of the things behind Threat Stack is make this inclusive, not exclusive, because you can solve problems better that way.
Alan: Amen. Pete, you and I first met, I think it was shortly before I even launched DevOps.com. And so I know you’ve dealt with this and have been dealing with this issue for years. My impression in speaking to a lot of people is that the developer and even ops community seems much more open, if you will, to welcoming the security folks into the meeting, into the tribe, and giving them the seat at the table. Where too often I’m afraid to say, because I come from the security side of the house, it’s the security guy who’s digging in his heels and saying, “No. We can’t have that. We need to maintain control. We need to maintain independence. We need to have our own kind of thing.” But the good news is I see that kind of hard headedness softening and I do see us, the security folks coming in. What’s your impression?
Pete: Yes. I remember. We were actually doing a – it was like a kind of roundtable dev ops discussion, and what I thought was most interesting is a lot of the _____ were all tilting more towards the security side. And yeah, that was a couple years ago at this point. Yeah. So I definitely agree. The conversation that I see in attending a lot of the different kind of web operations and dev ops related conferences is much more on the side of operators and developers who have essentially kind of figured out how to work well together and deliver value for businesses are realizing that security is much more important than it ever was.
The kind of joke is it was always important. It’s just a lot of people just didn’t really care about it. But now it’s affecting so much of what we’re doing and more and more issues and vulnerabilities and open-source software are kind of coming to light that I think more operations and dev teams are realizing that this is something that we really need to deal with now versus having some sort of company-ending event. And so I think what’s happening is that especially smaller start-up companies, even companies that get up into a few hundred engineers don’t even have a dedicated security team, so they kind of have to do it themselves.
In more legacy teams, I see it as well where they want to, more legacy companies, they want to build _____ they bring in security engineers for that organization. And you find a lot of the same thing where, like you were saying is, they want to slow things down and they want the control. The problem is is that that model doesn’t work any more. And we’ve seen it. It’s the concept of shadow IT, right? The security team up there constantly saying, “Nope. Can’t use Amazon,” because of whatever reason. It doesn’t stop anything. The company will end up having ten Amazon accounts under ten corporate credit cards.
Chris: Yeah. That’s such a good point, too, because I think one of the things, like the fundamental shifts you have to make is, right, a lot of that old security thinking was rooted in prevention, right? And so if you really think that you can use that model in sort of the modern era, God bless ya. But, man, that’s going to be tough. Versus if you can now focus on detection and involvement, you’re going to be – and then, right, you’re dev opsing the things because you want to go fast. You can now put changes into effect super fast by connecting your information sources to the things that allow you to take, allow you to make change quickly. That’s how you get – I think you really make some progress. And that’s where we see a lot of folks fall down when they think they can take the sort of legacy enterprise IT approach and just bring that to the cloud, including the type of people that are involved.
Alan: Absolutely. So, guys, let me bring this back to Threat Stack, if I can for a moment. Your customers, you think they come from the dev and sys admin or ops space? Or are the security people your customer, as well?
Chris: It’s both, and as our company’s grown and as our customer base had grown and we go up market more, we’ll start having more and more folks who have info sec and other titles like that in discussions with us. Where we kind of started was with the dev ops community, right? We’re a very dev ops friendly tool in terms of getting our agent deployed. And like Pete did an awesome job of building cookbooks and templates that people could use to get Threat Stack running really quickly so that it wasn’t a pain in the ass to get started. It was really, really simple. But that is going to sort of start to change over time, but what we hope is that we’ll continue to see the folks who are dev opsing and infrastructuring involved in that conversation and it doesn’t just turn into the domain of just the security people.
Pete: Yeah. In a lot of cases, too, what happens is you have a CTO or a head of engineering for some company, and maybe they’re running 200, 300, 400 servers on Amazon. Not trivial number of systems, but still not kind of enterprise scale of thousands of systems. But they’ll have their head of engineering come to them and say, “We’re about to close this deal. It’s with this company. They are going to require us to get our PCI compliance, or HIPAA compliance,” or some sort of audit _____. And so as an operations person, I’ve been in this scenario before at a previous company. I was running an operations team and we were about to sell it to the US government, which required us to go through a thisba audit, which was very intensive audit. And this was years ago, and we ended up having to build a lot of the tools. And I always kind of joke here. It’s like I love working at Threat Stack in operations because I get to use Threat Stack on my platform that I run. It’s awesome that way.
Chris: I have to go build tools like I used to.
Alan: Yeah.
Alan: Exactly.
Chris: Yeah. And to that point, it’s actually one of the reasons why I joined Threat Stack was that in previous roles where I delivered – my teams delivered – stuff into health care and other regulated industries and something like Threat Stack didn’t exist, so I had to stitch together stuff painfully from other types of tools to answer like all these security requirements being dumped on my desk from larger enterprises who weren’t used to dealing with sass companies. And thrown into the world of print security. And having a tool for me like Threat Stack, and frankly getting the opportunity to help build it out, was such a huge opportunity and attractive for me to come here because you could just instantly see the problem that this was going to solve.
Alan: Absolutely. So, guys, we’re running low on time because these things always go quickly – too quickly. But I’m doing a webinar next month with a couple of companies called New Frontiers in Cloud Security. I’m going to ask each of you, what do you see as sort of the biggest new threat, if you will, on the horizon for cloud security?
Pete: Yeah. That’s a really good question. I think what we’re finding is that as people move to the cloud, there was a long time a lot of thud around the cloud is not secure. You can’t move to the cloud. It’s not secure. And in reality, the cloud is just as or can be more secure than your existing environment if you use kind of the right tools to manage and monitor the systems, especially for Amazon users. They have a whole suite of tools that can really assist you to secure your system well.
What I regret it’s, and especially as someone who runs an operations team for a security company, we do everything we can to lock down the front door, to make sure that it is extremely hard to get onto systems to do things. Like we have a lot of checks and balances in order to do so. But honestly the biggest fear I have is people coming in from the back door, essentially, and trying to get in through some sort of malware on their systems, or through some sort of fundamental breakage in an open-source code where like Heartley had happen last year.
Alan: Yeah.
Pete: So that’s where I’m glad to have tools here via Threat Stack and other tools that we have that I can continually monitor my systems for these anomalies. Because it’s going to be nearly impossible to know of what the next zero day kind of critical bug is. But if I can monitor for anomalous activity, that can give me a heads up in advance.
Chris: Right.
Alan: Perfect. Chris, how about you?
Chris: So, I think one of the biggest threats, actually, is, and just adding to what Pete said, is people thinking they can solve cloud security like they solve on-prem security. And I can just worry about things at the ingress and egress point of my network. And I used this appliance thing at my last place that had, you know, it was co-load or my own data center, and I just want to replicate that model in the cloud, right?
Which, I just think is – you have to look at the work load and the host as that’s the, right, that’s the battle ground, and if you’re thinking you can solve this at a network layer, it’s going to be really tough because that doesn’t tell you what’s happening on your systems, which is, again, these things are coming and going, right? You’ve got probably more cattle than pets. You’re able to build up and burn down really quickly.
Are you able to really identify what was anomalous on those work loads over time? And if you can and if you think you can just solve it like you solved stuff five, ten years ago, it’s just like a rude surprise. To me, that’s one of the biggest threats out there.
Alan: I don’t disagree, guys. Hey, I’d love to sit and chat all day, and maybe we will in the future. We’ll get together at some dev ops days or something. But I’m going to need to wrap up here. We’re kind of at the end of our time limit. So Pete Cheslock, Chris Gervais, Threat Stack. Thank you so much for appearing in this segment of DevOps chat, and we’re looking forward to hearing more about Threat Stack and dev ops and security working together to make the cloud better for everyone, and continued success.
-
DevOps Without Scale, Part 2
In Part 1 of this series, I covered why it’s important to be aware of your product’s health and how you can do so. I covered the user experience and technical types of metrics and how to get started with collecting them. In this part of the series I will walk you through how to use all these metrics to maximize the user experience and product uptime.
Metrics, Metrics Everywhere
There are no silver bullets here and you have to be aware continuously of your product’s health and how it fares against user expectations. It surely takes time and effort, but it’s an essential part of building a good product. Imagine if chair designers didn’t care for how people use their creations; those chairs would not be comfortable to sit in. Similarly, understanding user needs is a paramount part for any software engineer.
Once you’ve established the basic understanding of your application patterns, you are not done. Every release is an opportunity to introduce more latency or failures and the baseline is ever-shifting. It is easy to significantly affect the latency and as easy to break an edge case and drive user failures up by a few percent with any release. The only way to ensure high-quality user experience is to be aware of your product’s health and your users’ expectations at all times.
This is the case where less is more and users prefer fewer features that work and are responsive. Try it out and see what your users say!
User Experience Monitoring
The first part of a good user experience is low error rates. For web products, these include all http error responses as well as all application errors returned to the user. For APIs and GUIs, this includes application failures (and http errors for REST APIs). One recurring comment I hear is that users submit invalid input, which invariably causes error responses and so it should not count. It is certainly useful to be able to differentiate hard failures from bad user input, but the line is more blurred than you think. For example, a new release might change input expectations and previously valid requests can start erroring out. Is that bad user input or did you just break your product?
In either case, if the failure numbers are high, users won’t be happy and you need to deal with this problem. The solution here can be non-technical: Maybe you need better documentation or user training, or maybe the API or UX design is cumbersome and drives misunderstanding and misuse. Whatever the case is, it must be addressed. Users are as likely to walk away from a cumbersome product as from a product that doesn’t work.
As with everything else, I suggest that you identify a few high-level and most important parts of the product and keep track of those metrics. If you do notice an issue, then you have the more detailed metrics on hand for a deep dive.
The other part of good user experience is performance, and the approach here is no different. Typically you need to track the latency of the same user actions as you’re tracking for failures. Performance tracking is where you really need to understand what’s acceptable and what’s not for your users and that depends on the domain and your users’ expectations. Two seconds to open a reservation on a public travel site seems like a long time, but two seconds to pull up a loan record for an internal user at a financial company is not bad.
An important fact to remember is that you will always have some failures in the system and that’s normal. No matter how great your product is, how clear its documentation, and how in tune you are with your users, there always will be some failures caused by bad user interactions. At some point, once you feel good about your product overall, you will have to assume that the given error rate is your baseline. Then you need to pay attention to variation over time and revisit critically every once in a while.
Predictive Monitoring
In addition to user experience metrics, there are key technical metrics that are good to watch, as they can point to upcoming problems. Some of the common ones are CPU and memory utilization, thread and database pool sizes, VM heap size and garbage collection times. Just like with user metrics, you need to know the operational signature of your application. But in addition, you need to understand how changes in these metrics impact your users, if at all.
A growing thread pool might mean that response times are slowing down or it could simply mean that more users are accessing your service. A growing email queue size means that emails aren’t going out as fast, but is it bad? Only knowing your domain and users will give you the answer. If the CPU is hovering at 80 percent everything is fine—for now, but you need to scale up quickly before a tiny addition of traffic kicks your whole app over.
When tracking hardware metrics everyone pays close attention to overutilization because that’s what causes outages. But what about underutilization? Performance fluctuates up and down and at times you can find yourself underutilizing resources. Paying attention to this is a good way to stay frugal and keep costs in check.
Alerting
How do you actually know when something breaks? You can’t stare at the graphs all day long. Once you get a feel for the normal characteristics of your system, you’re ready to create automated monitors and alerts. One of the most important things about automated alerting is that it always must be trustworthy. The moment you and your team start getting false alerts, everyone will learn to disregard some of them and the the value of the system will plummet. Thus I recommend you start with auto-alerting only on several most critical metrics. Run the whole system for a few weeks and slowly add more metrics over time.
Another aspect to be aware of when setting up automated alerting is metric variations over time, the most common one being day vs. night. Nobody wants to get paged because traffic is too low at 3 a.m. At the same time, if your metrics skyrocket or drop suddenly, it can be a cause for concern. Are you being DDoSed or scraped? Is there a network failure that’s preventing your service from being reached?
Tooling to the rescue, DataDog has built-in capability to create alarms, called Monitors. They provide a few different ways to ensure no false positives. For example, instead of setting absolute thresholds, you can create a change alert that fires only on deltas. And adding a new monitor only takes a few minutes.
Finally, someone needs to get paged out when an alert goes off. DataDog can do email notifications but nobody checks that at night and even during core hours it takes time to notice a new email. For critical issues like yours, someone needs to get notified immediately.
Services like PagerDuty and VictorOps have made it trivial to do this. As expected, DataDog integrates with both of them natively. Both services support on-call rotation scheduling with an escalation policy, calling out to phones, text messaging, emails, and more.
Production Troubleshooting
So you’ve been alarmed and need to quickly figure out what’s going on and what to do. Your next stop is typically logs, but looking through log files one server at a time is a sure way to waste quite a bit of time.
Log management tools and services have matured over time and there are good alternatives to Splunk. Loggly and LogEntries cover more than just the basics and provide indexing and search capabilities, graphing of log occurrence, understanding of JSON, and out-of-the-box integration with just about all logging frameworks out there. At their core they provide a quick way to look through all logs without having to go to individual servers. Moreover, you can monitor various aspects of logging à la DataDog and alarm (via VictorOps or PagerDuty) when metrics are off.
The above approach already solves most needs, but I’d like to touch on a more advanced problem, especially for a microservice architecture. When multiple services are involved, finding the failure at the entry point is easy but figuring out which back-end service failed in the call chain while processing the user request can be quite difficult.
An architectural pattern that I call log stitching solves this issue. With log stitching you can pull up all logs across your entire architecture for a given user request. This allows you to reconstruct the complete call chain and quickly find the root cause of failures. Conceptually this technique is straightforward: You generate a unique request ID at the entry point, pass it down the chain for every call, and log it. Now you can pull up all logs with that request ID by searching for it. In practice it can get complicated because you need to: a) ensure consistency across all calls, b) account for all entry points including batch jobs, and c) preserve the request ID internally in multithreaded services.
Log stitching is not something I recommend from the get-go; however, it’s definitely something to consider as your system matures. If you are going down the microservices path, this really becomes a must since tracing call chains get progressively more difficult as the number of services grows.
Putting it All Together
There is not much development work in putting it all together, but nevertheless you will need to put in some time. I’d like to emphasize how important it is to work in iterations. Set up one to three alarms, connect your DataDog to VictorOps or PageDuty, study them for a couple weeks and then move on. Keep in mind that you will not get it right at first; your thresholds will be off and you will misinterpret or misunderstand system behavior.
Operational pattern analysis is not trivial, but don’t spend too much time analyzing the hell out of everything up front. Configure some alarms, get them out there, learn and adjust as you go. When you find an alarm that went off erroneously, take some time to understand why. Specifically, you need to focus on why you thought it was the correct alarm setup vs. why you now know it is not. You will learn by leaps and bounds if you take this systematic approach.
As I mentioned above, configuring Loggly or LogEntries is super easy and you should do that right away. Log stitching, however, is an involved endeavor that requires planning. Due to its nature it only becomes useful when adopted by all, or most, of your systems. Thus, you should only start it when you feel the need and when you have the buy-in from the rest of your development organization.
What’s Next?
Part 3 will cover the topics of continuous integration and continuous delivery, breaking them down into multiple stages of complexity so you can pick and choose what makes the most sense for you from the cost/benefit point of view.
-
The Cost of Not Building with Security in Mind
The unfortunate reality for today’s organizations is the fact that a security breach is bound to happen. Major breaches are happening with alarming frequency and fill the news headlines almost daily. And behind many of these major breach stories is a software vulnerability that has been exploited. There is a silver lining, however. Addressing security at the development stage can help organizations “get upstream” of certain issues, which goes a long way toward alleviating potential breaches.
Now there are always two sides to every coin, and this is where companies need to weigh the costs of not taking a proactive approach to making security a key part of their software development process. Is the upfront time and effort worth it when it comes to developing software and ensuring bugs and vulnerabilities are addressed before it is put into the wild? The simple answer is, yes! We are already seeing this in certain industries such as financial services, which learned the hard way that it is a critical component of software development—mitigating risks and significantly lowering the financial burden associated with a major breach.
In fact, security should have been more central to software development a long time ago. As software, and more broadly technology, continues to evolve and permeate our lives from more angles, it is crucial that security be a consideration—a “use case” as developers call it—from the earliest conceptual stages of a project. The costs of not doing so are simply too painful to imagine otherwise.
Shifting to Security-First
To build more secure software, organizations typically must implement a software security initiative. You can break it down into four steps:
- Assess your current software assurance posture
- Define the strategy your organization should take to ensure a viable posture
- Formulate a road map of how to achieve that posture
- Implement your strategy
The above steps are simple and when you think of the alternative to not doing this, adding security as a pivotal part of the software development process seems like a no-brainer. To better understand what I mean, you need only look at the cost the average breach has on an organization. According to the 10th annual “Cost of Data Breach Study” independently conducted by Ponemon Institute, the average consolidated total cost of a data breach in 2015 was $3.8 million, an increase of 23 percent from 2013. The study also reported that the cost incurred for each lost or stolen record containing sensitive and confidential information increased 6 percent, to a consolidated average of $154 from $145. Depending on how much data has been compromised, the financial fallout can be staggering, if not crippling, to an organization.
Conversely, the cost to repair a single vulnerability in an application during the design phase is less than $500, according to the IEEE Computer Society. The fact that the financial impact of not addressing security during the development process can quickly become cost prohibitive, is a clear indication that secure software development practices can no longer be overlooked. Breaches are now a boardroom issue and require a seat at the development table.
As we can see from the stats above, the economic fallout for organizations is obvious, but with the Internet of Things coming to a house near you, this issue is suddenly becoming more personal and painful for the average person. Whether an attacker takes over your thermostat from Russia or finds a way to monitor your home via your baby camera, vulnerabilities are making it easier for malicious actors to corrupt corporate, government and now individual networks like never before.
We are well beyond the “it will never happen to me” phase. Breach stories that used to happen to so called “other” people are more likely to happen to you if software defects continue to be ignored and are not fixed when they should be. No matter how you crunch the numbers, if security isn’t addressed at the software development level, the result will always end in a costly lesson learned.
About the Author/John Dickson
John B. Dickson is an internationally recognized security leader, entrepreneur and Principal at Denim Group, Ltd. He has nearly 20 years hands-on experience in intrusion detection, network security and application security in the commercial, public and military sectors.Dickson is a popular speaker on security at industry venues including the RSA Security Conference, the SANS Institute, the Open Web Application Security Project (OWASP) and at other international conferences. He is a sought-after expert and regularly contributes to Dark Reading and other publications. A Distinguished Fellow of the International Systems Security Association, he has been a Certified Information Systems Security Professional (CISSP) since 1998.
As a Denim Group Principal, he helps executives and Chief Security Officers (CSOs) of Fortune 500 companies and government organizations launch and expand their critical application security initiatives.
Twitter: http://twitter.com/johnbdickson
Linkedin: https://www.linkedin.com/in/john-b-dickson-cissp-41a149 -
To Be Continuous Podcast: Open Source Economics
Welcome to To Be Continuous, a show about continuous delivery and software development, hosted by Heavybit members Paul Biggar, founder of CircleCI and Edith Harbaugh, CEO of LaunchDarkly.
In this episode of To Be Continuous, Edith and Paul are joined by Sean Byrnes, CEO of Outlier.ai, and Nadia Eghbal. The group discusses open-source economics and examines several potential evolutions of adding money to the open-source equation.
[soundcloud url=”https://api.soundcloud.com/tracks/257873002?secret_token=s-fHPo1″ params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]
Edith: So, Nadia what do you like best about open source?
Nadia: I like that open source is where I feel I spend time with a lot of creative people. It’s sort of like creative talent at its best. And it feeds directly back into the rest of startups and technology and everything else that I love.
But it seems like everything starts with open source. Which makes it really magical.
Edith: So Sean, what do you like best about open source?
Sean: It’s the community. I enjoy learning from other people. I enjoy being able to find people I would never connect with that I have interests in common with. And a community that helps, because I consider open source to extend beyond just writing lines of code to sharing ideas, and sharing concepts and sharing solutions to problems, and while I might not always use those same solutions, I’m always inspired for new ways to solve other problems.
Paul: So that’s super interesting, because nobody said that they like how it supports their Silicon Valley startup through sharecropping. Or … nobody said, I like how I get paid to work.
Edith: Well, Paul, that comes later in the episode, but this would be a great time for our guests to introduce themselves.
Nadia: I’m Nadia Eghbal. I’ve been writing recently and thinking a lot about how to support open source infrastructure.
Originally, I have background in nonprofits and impact investing and working with foundations but most recently I was a founder of a startup and then I went into venture capital. And have sort of seen the system from all different sides and would love to marry together all those different interests and topics in which seems to happen in open source.
Sean: Hey, I’m Sean Byrnes. I’ve been writing software most of my life. I’ve also been a founder of companies and an investor. I would not consider myself an expert on open source and I don’t wanna claim that pedestal but I have, am, the maintainer of some open source libraries which seem to be pretty popular and I have contributed to others and I think that I’ve been lucky enough to know a lot of open source developers in my life which is, by no means, a majority of them but enough that I feel part of the community, at least.
Paul: So, one of the topics that we were discussing just before the show started, is this idea of how open source developers get paid. Or, if open source developers should get paid.
Nadia, sounds like you’ve been talking about this for a while or thinking about this for a while. Can you kick us off?
Nadia: Yeah, actually, have a new thought on this topic from Michael Rogers whose, who runs Node.js Foundation. And he said recently, I think, on the last post I had written that everybody gets paid for open source. And I think that’s a really great way to start thinking about it vs. saying that nobody gets paid. Because in the end we’re contributing time to open source, right and in the end, somebody is paying you to have the time to work on open source.
Whether it’s your employer and then you can work on open source full-time at your job. Or you’re full-time employed and that gives you the bandwidth to be able to work on open source on the weekends or whatever.
So when I think about paying people to work on open source, I think people sort of like, bristle at the idea because they think that money is not involved in open source at all. And what I really liked about Michael’s point is that, in the end, money is actually really closely tied to open source and we’re just not talking about it.
Sean: I get it. It’s interesting because my point of view is that it’s actually more nuanced, I agree with you 100 percent but I think that it’s very nuanced and that a lot of people do get paid for open source, it’s not even that we aspire to wanting them to get paid, right. ‘Cause there’s lots of kinds of open source. It’s funny ’cause on Medium, Nadia and I had an exchange.
I kind of quantified it into the four categories of open source the way I see them. Which is by no means the universal standard but at least, how I see the world there, there are places where open source is part of your, a corporate product. It’s part of a whole.
Like, LaunchDarkly. Edith’s company, has open source SDKs. It’s not because they’re doing anything grand, it reduced their sales friction if their customers can see the code before they integrate it into their app, right. So that’s one kind of open source.
A second is what I call corporate open source, where there’s an open source project but there’s a company that sells services and products and stuff around it, right. So Confluence does that for Kafka. And Databricks does that for Spark. And MySQL was, I think, one of the first companies to really make a good model out of it. Red Hat, you could argue, had mixed success but that’s another kind where there’s a company whose corporate goal and sales is towards this project.
There’s the third kind which I call, kinda like corporate marketing, which is you take a project out of a company and you roll it out like LinkedIn did with Kafka, where they’re actually, the company’s maintaining it they just make it available. So Google, the Angular and Facebook work react. And again there, I think, that it’s. You could argue why companies do this, some of ’em do it for recruiting, some of ’em do it for marketing. Some of ’em do it for because they’re very nice people.
I think that the fourth category is where it’s a little bit weirder, where it’s like, what I call the community where somebody started a project. Maybe it’s an individual, maybe a few people, to solve a problem. And a lot of the most important open source products today started that way. Even Linux started that way, right. And it grew, and it got bigger, and eventually, then it became unclear because other people were using that project to build other companies and build other products, and make more money.
And I think in that fourth category is where I think it’s less, because the first three categories is usually pretty clear who’s getting paid to do what and where the money’s coming from, but in the fourth category it gets very muddied and very quickly. Because if I start something, and the metaphor I like to use is, the open source library that I wrote was something called Likely.js which is a recommend ender system for no JS where you can, like Amazon product recommendations. And I know that there are people in the world that are using that as part of companies. And those companies are obviously making money because it to recommend products, right. And so, the question is, you know, so in that world, do they owe me anything for creating that which is underlying their products? And then, in turn, what do I owe the projects that I build Likely on top of? And then where does the flow of economics in that world go?
Paul: So I have an interesting view on that. Years ago, I got involved in this project called PHC, which is a compiler for PHP. And there was a couple of guys who had start this and I joined a year later. And when they started it, they made it GPL. When I joined, I argued, we should change from being GPL to BSD because if someone comes along and uses that technology to build something that’s cool, I would rather that happen than not now happen and it might be that if was GPL that it wouldn’t happen. And so that, essentially, in sort of the kinda related to Likely.js you decided by the license.
Whether or not you were going to make it publicly available or if you didn’t want people to build business on it, you could’ve made it GPL or AGPL or something along those lines and you could’ve restricted it in any way that you wanted and you chose not to. And there’s something implicit in that, that communicates to people who are building their companies on it, etc., as to what your intent was.
Sean: No, that’s true. Although there’s a nuance in there which is that, even the GPL suffers from this, what Richard Solomon still complains about today which is, you could still use Likely.js on a server.
Paul: Right, right, right. Well, so AGPL would work around that.
Sean: Sure, and there’s other licenses that I could choose, but the majority of licenses out there today but even then, let me argue counter side, not because I agree with it just because, hey, let’s argue because—why not! You know, as an individual developer, I don’t have the time resources to go pursue people. I’m not gonna go sue somebody. Really? No, come on, like it’s not gonna happen. So, I think, you’re right only in the act of releasing it as open source on GitHub. I was making a decision that it’s possible for people to use this in ways that may, either A, I don’t intend, they could’ve used on Silk Road to recommend drugs for all I know, seriously, right.
Or, to make money on it in ways that I am not gonna benefit from. And I made that decision knowing full well the impact of that. And I would actually argue most developers do that. The question is whether or not there’s remorse later on when it turns out, wait, I really wish I hadn’t made that decision, I can go back in time.
Paul: Well, I think this is a common thing that people create a thing and then they say, well, if this is popular, I could sort of keep it or I could, there’s value here. I could keep it to myself or I could keep the value to myself. Or try to profit from it in some way. But they make, what is essentially, a marketing decision. I’m just gonna release this, I’m gonna see what happens to it. I’m gonna see if it spreads and then if it spreads, I can then make a decision as to, as to what, how possibly I can make money out of this. Or I can get value out of this. But at that point, obviously, the cat is out of the bag in terms of people actually using the software.
Nadia: Yeah, I mean, I think it’s actually should start from the principle, like you said, of it’s really great when people, you give something away and then people can use for whatever they want. Whether it’s for a hobby project. Whether it’s for illegal drugs or whatever, you know. Businesses, whatever, that’s like, kinda one of the coolest things about open source is that you can use it for whatever you want and that’s kinda what you signed up for right.
If you didn’t wanna do that, then don’t license it that way. I don’t know, I wouldn’t frame, at least the problems that I’m seeing and that I’m interested in, is not really about seller’s remorse, I guess of, “Oh, I made this thing and now it’s really popular and how can I make money off of it.”
But I think it’s more really about investing back into the infrastructure that we’re all using and that requires some sort of maintenance and upkeep. And that’s why I like the term infrastructure and I’m trying to use it in that sense of things that don’t have a business model or business attached to it. Like you, literally, could not charge for Python. It’s just not gonna happen. Or the packaging associated with it or whatever, but still like millions of people are using it. It’s touched billions of people. And you literally have like one person working on a library or a project, that doesn’t seem, that seems like it’s in everyone’s best interest to make sure that it works properly.
Edith: Well I think Nadia made a really good point that a lot of the fun of creation is the creation. And the hard work is all the maintenance and the support and the documentation after. So like, Sean, you basically got the fun.
Sean: Oh, totally. Well I mean, if you even think back to the original “The Cathedral and the Bazaar” essay from years ago, that was one of the fundamental topics of, you know Eric was inheriting, I think it was called Pop Mail from somebody else.
He had gotten tired of maintaining it and so he wanted somebody to take it over and he found Eric and Eric took it over. I mean, that’s, you know, I think, yes, as engineers we like creating and we like creating things in our free time and we don’t like maintaining them as much and maintenance feels more like work.
And so, I agree with Nadia’s point that if everybody is driving economic value from it somebody has to be willing to step forward to support it. I think the problem is that I also, it’s funny that you have a background in nonprofits because I also have worked for nonprofits and I see the similar lament in nonprofit companies.
Which is that—not nonprofit companies, nonprofit organizations, where the person who starts it usually really wants to solve this problem, they’re usually really good at that. They’re not good at the fundraising
And in this case, the nonprofits, it’s like raising grant money or getting donations. But they have a good mission and they have a good organization to go do it. And they have this lament which is why can they not get more support to solve the problem they see in the way that they wanna solve it.
And I see the same around maintainers of open source projects and I think the challenge is going to be that I wonder if it is even possible for the general economy to support them, to do what they wanna do, the way they wanna do it. Or if they are gonna have to start doing it the way the general—
Paul: Right, because they have other options. They could start a consulting company. Get hired by a category two company or something like that. To build around it and support themselves that way and then their job doesn’t, then their job isn’t what they wanted their job to be which is making cool software. Their job is, you know, doing maintenance contracts for a big soul-less enterprise.
Edith: You say that like it’s, I mean, soul-less enterprises, I mean …
Paul: I’m a big fan of soul-less enterprises. But I’m just looking to sort of put the—
Sean: They have a lot of money. We have to sell them.
Paul: And if any soul-less enterprise would like to buy CircleCI they should contact me for enterprise pricing.
Sean: Just mention the code OSS for 10 percent more.
Edith: And by the way, thank you for wearing your LaunchDarkly t-shirt today, Paul.
Paul: Always. So, Nadia and Sean had this conversation on Twitter a week ago which led to them being on the show. And one of the Tweets that got involved was from a Jacobian who’s one of the senior Python kind of people. And what he said is, OSS developers deserve to be paid. SV, Silicon Valley, has made massive profits by exploiting free labor. This needs to end.
And I think that there is this, this feeling in open source by, at least by a lot of people who end up on the maintenance side of things that, there is a sort of a sharecropping going on here. And that’s something needs to happen about that.
Edith: I think Nadia actually put this extremely well in the essay; like cool, I wish I had written that essay, it was brilliant.
Nadia: Aww.
Edith: But just like, what’s really fed this is the explosion of GitHub and it’s really easy now to start an open source project.
Nadia: Yeah, I feel like right now is just a time when we’re so used to thinking of open source in a certain way and things have kind of gone along as they have for so long that we haven’t necessarily noticed or taken a step back that the past five years look completely different than the past 30 years. And stuff is just not the same anymore. And in part, I think it should be framed as a really good thing because it means that open source kind of won, in a way.
Companies want things to be open source. They wanna learn how to open source their own projects. They wanna understand the whole thing. Everybody is defaulting to open source and that’s a really positive thing and the question is more of, not like, oh, we’re exploiting people and that really sucks but how can we help support that underlying layer that has helped make everything so great. It obviously needs some sort of reinvestment back into it.
Edith: Yeah, I mean, one of the most fascinating thing about “The Cathedral and the Bazaar,” which was written 30 years ago, is that they referred to Linus Torvalds as this charming guy. Yeah, I know, and you’re like wow, here’s this charming, personable guy who like persuaded people through sheer force of will to work on his project.
Sean: Yeah.
Edith: I mean, to go back to Sean’s example about a nonprofit and everybody in the studio right now is like, whoa, that’s not …
Nadia: Not at all.
Edith: By anybody’s perception now, and I don’t know whether that’s just the grind down of having to keep maintaining a project or …
Sean: First, I’d like to thank, I don’t know if his name is actually Jacobian, whatever it is, thank him for his service and work on Python ’cause I think, no matter what we agree on here, we can all agree that the people doing this are doing a service for the general community.
Nadia: Definitely.
For more on Continuous Delivery and to hear other Heavybit podcasts, copy this link into your favorite podcast player. We’ll have show-specific RSS and iTunes subscription options soon. You can also follow To Be Continuous on Twitter at @continuouscast.
