devopsy.kinsta.cloud

Tag: production

Continuous Integration vs. Delivery vs. Deployment: What’s the Difference?

“Continuous” is one of the most popular words in the DevOps lexicon. What does it really mean, and what is the difference between continuous delivery, continuous deployment and continuous integration? Keep reading for some perspective.

Like many technological concepts, continuous integration, continuous delivery and continuous deployment are terms that DevOps teams tend to use without defining them precisely. For that reason, it’s worth taking a closer look at what each of these terms actually means.

Continuous Integration

Continuous integration can really mean one of two things, depending on the context.

In the narrower sense, continuous integration refers to the work done by a continuous integration server, such as Jenkins or Bamboo. A continuous integration server automatically tests code written by individual developers to make sure it can be merged into the main code base. By automating this process, continuous integration servers help developers to write and test small chunks of code on an ongoing basis, which minimizes the risk that a code change could cause serious problems and force developers to revert their application to an earlier state.

In a broader sense, continuous integration means the constant integration of changes to an application at all stages of the delivery chain. This includes but is not limited to the automated integration testing and code merging performed by an integration server. In this broader definition, continuous integration can include the integration of new design concepts to the delivery pipeline, for example. It also can involve other types of software testing beyond just automated integration tests.

Continuous Delivery

In most definitions, continuous delivery refers to the process of delivering software updates to users on a nearly constant basis. Continuous delivery is made possible by continuous integration and other optimizations at earlier stages of the development process. But the definition of continuous delivery gets a little cloudy when you start comparing it to continuous deployment.

Continuous Deployment

What’s the difference between continuous delivery and continuous deployment? For some DevOps teams, nothing. People tend to use these terms interchangeably. In fact, they are usually simply abbreviated as “CD,” leaving it up to the listener to decide what the speaker is actually talking about when he hears the term during a conversation about DevOps.

Some developers draw a distinction between continuous delivery and continuous deployment, however. For example, Mirco Hering writes that continuous delivery requires that the DevOps team manually release updates to users. In contrast, the continuous deployment pipeline is fully automated; users get updates as soon as they are written and tested, with no manual intervention by developers.

This distinction is a good way to think about the difference between the closely related concepts of continuous delivery and continuous deployment. It’s a distinction that will become increasingly important as technologies such as Docker make it easier to automate application deployment by allowing DevOps teams to drop new container images into production.

What Does Continuous Mean, Anyway?

It’s worth mentioning, too, that the word “continuous” is usually something of an exaggeration when used by DevOps folks. It implies that software changes roll down the pipeline in a totally constant, never-stopping fashion.

That is an exaggeration. Integration, delivery and deployment are almost never completely continuous. In practice, a “continuously” integrated application is likely to be rebuilt and delivered something like every 24 hours, not every single time a code change reaches the end of the pipe.

That rhythm is still a lot faster, and closer to continuous, than the “waterfall” delivery pipelines that were common before the introduction of DevOps. But it’s still important to understand that, even with all of the automation tools available today, software development is rarely completely continuous.

— Chris Tozzi

November 18, 2016
7 Best Practices for Continuous Delivery Success

Continuous delivery refers to the development of software in short and ongoing build-test-release cycles along the deployment pipeline. Repeated testing of processes and scripts before deployment to production means most errors are discovered early on and have minimal impact. Plus, finding and fixing such errors is much easier with fewer changes per release—software is released faster and more frequently with fewer deployment problems due to a heavy focus on visibility, instant feedback and incremental changes.

To accomplish this, companies must have a straightforward and repeatable deployment process. Most reliable, of course, is an automated deployment pipeline that can feed into any environment.

We have identified seven continuous delivery best practices that are supported by a set of complementary task-specific tools, in what’s known as the “DevOps toolchain,” to automate the process from end to end and minimize human error.

Version-Control Everything

The native code deployment process generally includes a built-in safety net, preventing out-of-process, locally generated artifacts from entering the production environment. Every change a developer makes must be documented in the source control repository or it is not included in the build process. If a developer copies an artifact generated locally to a test environment, the next deployment will override this out-of-process change.

This insistence on having a single source of truth—and a reliable one—creates a stable foundation for all development processes. However, the processes are only as strong as their weakest link. That’s why the primary golden rule for effective continuous delivery is to version-control everything. What works well for code also will preserve the integrity of configuration, scripts, databases, website html and even documentation.

Build Binaries Once

Not every organization’s build process is the same or uses the same tools, but every build process in a single organization must be completely consistent. Whether the build is a single file deployed to an automated test environment or a complex build with several different possible deployment versions, each build version should happen exactly the same way and result in unique binary artifacts.

This one-time-only compiling eliminates the risk of untracked differences due to various deployment environments, third-party libraries or different compilation contexts or configurations that will result in an unstable or unpredictable release. Save the compilation phase output (the binaries) to a binary repository, from which the deployment process can retrieve the relevant artifacts.

Deploy the Same Way Every Time

An inconsistent deployment process can become a source of configuration drift across environments, especially with the rapid releases common in a continuous delivery system. As a result, valuable time and effort are wasted in identifying and addressing difficulties arising from the application working in one environment and not another.

For a reliable process in all environments, then, the same set of steps must be repeated from start to finish. This ensures the same results in lower environments, with more frequent deployments (such as integration, QA, etc.), as in higher environments (pre-production and production), with fewer deployments.

Smoke Test

Smoke testing deployments is a very rapid way to make sure that the most crucial functions of a program work and will pass basic diagnostics. This non-exhaustive software testing (of all elements, such as services, database, messaging bus, external services, etc.) does not provide the same fine-grain comprehensiveness as full test suites; however, it can be run frequently and quickly, often in a matter minutes (rather than hours or days). This allows for a much quicker turnaround on what can become very time-consuming and basic issues.

The faster the feedback loop works (especially with automated test tools such as Telerik, QTP, TestComplete), the higher the final product’s quality will be and the quicker it will reach its release-ready state.

Deploy into a Production-like Environment

Each new deployment (using, for example, IBM uDeploy, CA Release Automation, XebiaLabs, Automic) should be made into an environment that mimics as closely as possible the actual final production environment. This includes infrastructure, operating system, databases, patches, network topology, firewalls and configuration. By validating software changes in this type of detailed pre-production environment, mismatches and last-minute surprises can be effectively eliminated and applications can be released safely to production at any time.

Instant Propagation

Continuous delivery by definition must be an ongoing process of building, testing and releasing. Every check-in to a source control repository should trigger compiling (if needed) and packaging by a build server. This, in turn, should automatically initiate certain predefined testing until the software can either be marked as releasable or returned to development.

Continuous integration tools (e.g., Jenkins / CloudBees, Bamboo, TeamCity) ensure that cascade of actions along the delivery pipeline is consistent, automatic and instant. Thus, development teams merge developed code and get feedback from automated test tools many times a day, for a more efficient deployment and update process.

Include the Database

No list of continuous delivery best practices would be complete without advising that the database be managed using the same basic protocols that ensure secure and reliable source code, task, configuration, build and deployment management. However, many times this is neglected because the database is unlike other software components or compiled code that are easily copied from development to testing to production. Yet, the database is actually a repository of the most valued and irreplaceable asset—our data—and preserving it accurately is imperative to continuous delivery.

Although the database poses several unique challenges, specialized database tools such as enforced database source control (for all environments), a database build automation tool and database release and verification processes can ensure a stable resource in your DevOps chain.

Considering these seven continuous delivery best practices will allow you to create a deployment pipeline with increased productivity, faster time to market, reduced risk and increased quality.

The key to ensuring a quick, robust and accurately repeatable continuous delivery is, as we have seen, automation, deployment consistency, comprehensive testing and database version control. With these fail-safes in place, pushing the “release to production” button can be done much more often and with much more confidence.

About the Author / Yaniv Yehuda

Yaniv Yehuda is the co-founder and CTO of DBmaestro, an enterprise software development company focusing on database development and deployment technologies. Yaniv is a DevOps expert who spent the last couple of years raising awareness about the challenges surrounding database development and deployment, and how to support database continuous delivery. Connect with him on LinkedIn and Twitter.

November 9, 2016
Delivering DevOps’ Business Value

DevOps is an interesting space to be operating in right now. More than just culture or a set of tools and processes, DevOps delivers speed and agility that can be truly meaningful for the business. Unfortunately, for most large enterprises, the path to the benefits isn’t always well-known and isn’t assured.

Over the last five to 10 years, a number of DevOps tools and solutions have emerged: Puppet, Chef, Jenkins, Git, SaltStack, Elasticsearch … and the list keeps growing. Originally, these tools were sought by developers and system engineers for their ease of use and low cost of ownership: You could download it, use it the same day and solve your specific problem. The issue for enterprises, however, is that nature abhors a vacuum; the problem one developer addressed with Chef is being fixed with Puppet by another developer in a different business unit. Group A is using Jenkins, Group B is using Chef Delivery. In short, after five to 10 years of technology adoption by developers and system engineers, the tooling landscape in the DevOps space is packed to the rafters with competing technologies, all of which are solving similar problems with overlapping features.

So why should this matter to you at all? Without careful planning, consideration and execution, your business will derive little value from this DevOps exercise.

Here are a few common pitfalls we see far too often:

Too Many DevOps Tools Doing Too Few Things

Building Rube Goldberg devices looks neat on paper, but those mousetraps fail very often in real life. The more tooling and solution handoffs you have, the greater your likelihood of failure: the less likely you are to be able to scale horizontally or vertically, the more toolsets you need to maintain, and the more skills you need on your team. Convoluted messes do not yield speed and agility. If it feels wrong, it probably is. If it is overwrought, look to remove complexity from the landscape.

Reintroducing Risk and OpEx Spent on Menial Tasks

As an organization you’re onboarding risk, hidden costs and a lot of the things you just spent the last 10 years automating away: tribal knowledge, lack of governance, lack of compliance, lack of third-party auditability and reporting. Many enterprises spend millions of dollars making it so their admins and developers could spend time on more valuable workstreams, just to throw them back into the salt mines of making continuous integration (CI), continuous delivery (CD) and infrastructure-as-code “work” over umpteen toolsets. The goal isn’t for DevOps to “work”; the goals need to tie to some tangible (i.e. measurable) business benefits. Anything can be made to work, but far fewer IT programs deliver value. Be the people who deliver real value.

Taxation Without Representation

All of the stakeholders need a seat at the table. Whether that means forming a SWAT team from operations (all infrastructure domains and security), development, QA or a permanent DevOps matrix organization, all vested interests need a say in how the process and tooling develops for all impacted domains (which is just about everything) across all environments (i.e Dev, Test, QA, Production). Far too often we see one set of stakeholders acting as the service provider for another group of stakeholders playing the role of customer. This is a solution for and by Development and Operations.

Congratulations, you’re having a baby. Let’s make sure it’s not ugly, okay?

About the Author / Jesse Langhoff

Jesse Langhoff is co-founder and CEO of Raindance, a cloud and automation consultancy serving enterprises nationwide. With over 15 years of experience in leveraging technology to address business challenges, Jesse works with customers to determine the optimal path to meeting their goals through the use of automation and solutions provided by the leading vendors in the space. Areas of expertise include strategy and assessments, private/hybrid cloud services, process automation and orchestration, DevOps and operations as a service. Jesse holds an MBA from University of St. Thomas – Opus College of Business.

August 3, 2016
DevOps by Osmosis — Is That Even Possible?

“We don’t do DevOps. We can’t, as our development plans are agreed so far in advance and our priority from a production point of view is to keep our systems available 24/7.”

I heard this comment from a pal of mine as we had a quiet beer over the weekend, but I’m sure this comment, or a variation on it, has been uttered many times in many places. Here’s the irony: I had met my pal, who works as a developer for a major financial services company, after he had had the week from hell. His group tried—and succeeded—to fix a major system issue that had come out of the blue. He had certainly earned the beer that week.

So, I asked him: “How did you get the problem sorted in the end?”

“Well, we had to form a Tiger Team,” he replied. “Pulled together some of the DBA team, the key developers and the Ops guys, worked throughout the night and managed to keep the systems up and running the next day, while we used that time to address the root cause.”

“How did you get around the Governance guys?” I asked.

“Well, we brought one of them into the team for a few hours, and she was able to get us past some of the key bottlenecks,” he said.

“And how about QA?” I asked.

“Oh, they knew this was a major issue and so we cut out a lot of the QA and just went ahead,” he replied. “It was a bit risky, but we had all the right guys in the room to fix anything that cropped up.”

“So, you manged to get a major change through in a couple days, kept your systems up and running, kept your customers happy, and earned that beer,” I said.

“Yep, that’s about it, alright,” he said.

“So, that’s pretty much what DevOps is all about—minus the emergency element,” I pointed out. “Seems to me like you guys are already on the DevOps road.”

It was about then that the penny dropped with my pal. He suddenly realized that yes, they could do changes quickly, yes, they could do them in an orderly manner and yes, they could keep their systems up and running while all that happened. But, as quickly as the penny dropped and the realization dawned on him, the light went out again.

“There’s still no way we can do DevOps in our place,” he said. “There’s too many hurdles to overcome, too many guys used to doing things the existing way, which to be fair, has worked well for us. To be honest, there’s too many guys who would feel under threat if we suddenly changed how we do things.”

And that, in a nutshell, sums up the major challenge facing people promoting a DevOps approach. I have spent many years in the IT industry; I go so far back that I wrote my first code using punch cards (look them up) so I’ve seen change come and go, and change has been the one constant. I think many people in the IT industry have become too focused, whether it’s on the latest tools in their area of expertise, or on the newest piece of kit that has been bought, or indeed on the latest problem the business has put in front of them. The industry has become mature—if we compare the IT world to a field such as medicine, you pay passing attention—if you even notice at all—to things outside your specialty field.

Now that’s right and proper. I don’t want an orthopedic surgeon trying to fix the problem with my eye, and I don’t want a DBA to write me some code. But when something comes along that affects the whole industry, I want them to pay attention and not ignore it.

Daysha ran a lunchtime seminar last week, and we titled it “BizDevOps – DevOps isn’t compulsory but neither is survival.” We wanted to use this seminar to engage business people who, let’s face it, are the people we support and who need to understand how IT can make things better. Somewhat disappointingly, while we attracted a great crowd, they were all IT people. No one from the business attended. Now maybe we advertised this event in the wrong places, or maybe we held it in the wrong place, but here’s the thing: We attracted IT people from major companies, and none of those IT guys said to their business people, “You need to hear about this, come along for lunch.” I hope that after our seminar they went back to their business people and said just that, and maybe the next lunch will have the business people in the majority.

When PRINCE came out into the project management world, it was not bought into by a majority of project managers. The same happened with ITIL. And the same will happen with DevOps. It is fair to say that PRINCE and ITIL are pretty much standard on a worldwide basis. I am old enough to remember the time I first heard of them, and the time I did my first course and passed my first tests, but nowadays, they’re part of the furniture—taken for granted, and rightly so.

I think people are a little bit afraid of DevOps, and I can see why this is the case. Think about it for a second: Someone is now saying that the way we have done things for years has to change. That we can do things better, faster, with higher quality, but we just can’t do the massive 12-month long project in the same way. We might need some new tools, primarily on the Ops side. We might need some new skills—again, primarily on the Ops side. But, by the way, we’re not going to do away with the Ops guys (so don’t expect to make major budget savings there). But here’s the thing: Our end customers will notice the difference, and so will not be tempted by the new kid on the block—who, as it happens, is probably using a DevOps approach to develop his application, which is threatening our business.

Change is scary, it’s not easy, and making a major change while you’re still having to churn out the new business requirements, and keep the current systems running … now that’s really scary. But, like my pal whom I shared the beer with last weekend, maybe you’re not as far away from that change as you think you are, maybe it’s not as scary as you think it is, maybe it will affect people positively, maybe you just have to do it.

About the Author/John Brophy

John Brophy is currently Head of Professional Services at Daysha Consulting in Ireland, where he leads a team of 25 project managers and business analysts working on projects in blue-chip clients. He is a 35-year veteran of the IT industry, starting out as a database systems programmer, moving onto leading a team of systems programmers looking after what were then called midrange systems. He then became a senior project manager with a global IT company and followed that up by running his own IT infrastructure support company for 6 years. His background is Ops – and he is proud of that!

May 3, 2016
Supporting Production Applications the DevOps Way
Software applications are more complex than ever. If your application is slow or down for even five minutes customers will blow up on Twitter for the whole world to see. To prevent this, developers and operations have to work together in lockstep when deploying and supporting applications. Preventing and fixing application problems quickly is critical to keep your outage from becoming a trending topic on Twitter.

The DevOps Movement Begins

For software developers, ensuring early success of your new application starts with keeping IT operations in the loop. Traditionally a lot of software development teams have worked for months on new projects and then thrown them at the IT operations team at the last minute when it came time to deploy them. If you want a good example of this, look no further than the big healthcare.gov debacle. The DevOps movement started originally to try and solve that problem. It fosters a culture of keeping development and IT operations working together from the beginning to the end of the project so software deployments go more smoothly.

Long-term success of your application depends on proper maintenance and monitoring of your applications as you continue to roll out new features. Thanks to agile development, that is usually a weekly or monthly cycle at most companies. A constant stream of new features is awesome, but with that comes some risk in introducing new bugs, performance problems, or instability. This constant stream of change coupled with the overall complexity of today’s applications makes them difficult to support

Application Support: “It Takes a Village”

Collaboration between developers and IT operations is needed to keep your applications running and your customers happy. If you only have a couple senior developers who handle all the production support issues they will have little time to deal with more important issues like architecting new features.

The key to supporting production applications is removing the bottlenecks and getting more people involved in application support. Operations needs to make sure that the development team has visibility of basic troubleshooting information like errors, logs, and key metrics while ensuring security.. Without this information it is hard for developers to fix basic bugs and improve application performance. They must contact other developers or system administrators for help which creates a huge bottleneck. You could say “It takes a village.”

Granting access to production data raises both security concerns and fear of unauthorized changes to the servers without going through proper release processes. However, developers don’t really need administrator level access to servers (jumping between dozens of servers is not very efficient anyways), they just need visibility to the aggregated information. This will allow them to find the source of the problem fast, while not creating any security risks or letting developers create new issues while fixing others.

DevOps Should Include Production Support

The goal of DevOps is to create collaboration and improve the working relationships between development and operations. The DevOps movement initially started focusing on software deployment and continuous delivery. However, support of production applications is the second and equally important component in the application lifecycle that is sometimes overlooked. To solve production problems quickly, developers and operations can not be pointing fingers at each other. They need to work together as a team.

To help support production applications, developer need access to lots of data:
- Visibility and notifications around all application errors
- Access to centralized aggregated log files for viewing and searching
- Basic server utilization trends and stats. (e.g. CPU, memory, etc)
- Recording and alerting of key application metrics (KPIs)
- Tracking of application web page load times
- Ability to access the application database and run test queries
Most companies utilize several different tools to track all of this data. This makes it difficult to correlate and identify issues. It is also very hard to train all team members on how to effectively use all the available tools, not to mention the economic inefficiency. Companies should look for solutions that combine these features into one dashboard so that solving issues can be done quickly and with minimal training.

Companies should look for tools that gives developers access to all the data they need to monitor and troubleshoot production applications, without giving them administrator type access to all the servers. One application dashboard that will show them where the app is deployed, the current health, performance stats, recent errors, full logs, and key metrics. This is what we try to do here in Stackify – to make it easy for companies to embrace DevOps when it comes to supporting production applications. Some of our customers have shared with us how DevOps affected their organization. We would love to hear how DevOps has affected your company.

About the Author/Matt Watson

Matt Watson founded Stackify in 2012 and as CEO provides the vision and leadership for the direction of the company. Prior to founding Stackify, he was the founder and CTO of VinSolutions. VinSolutions’ innovative product offering led to rapid company growth and was even featured on the Inc. 500 (#447) in 2010. Matt is an entrepreneur at heart and excels at product and software development.

Feature photo by JD Hancock
November 6, 2014
Closing the Deltas Between Development and Production

Who hasn’t heard the classic statements “It works on my machine!” or, “Man that would have been easier with real data!”? In either case, had the development happened in production, problems would have been avoided! Fortunately, in a DevOps world, the walls between production and non-production get broken down making the reality of developing and testing in production more tangible. In no particular order, here are some of the things I’ve done to breakdown those walls.

Develop on and deploy from a single code line.

Many times we find ourselves in complex processes around deployments and production bug fixes. For example, the code is branched for release candidates and deployments are done from the branches. Branching before release by itself isn’t complicated, however, when there is a production bug, engineers find themselves repeating or merging the fix into at least two, if not several, different places. Repeat that a few times over and, uh oh, now there is a problem because there are many moving parts and questions about what exactly is in production. Unfortunately, the differences are most likely unintentional.

Instead, simplify the process and have everyone work off the same code line and have that same code line deployed directly to production. Now there is only one code base, therefore no question about what code is running in production. Production isn’t static, so change will still happen after deployment. Notation is required indicating what version was deployed, so you can still reproduce production.

Let testing happen on the production database.

Ok, even I’m not that crazy, but let developers use a copy of the production database. There are two very important aspects to this. First, refresh it often. I suggest daily. I have had very good luck using EBS snapshots to get a copy of the database on a different server, but if your database is small, good old SQL dumps also will work. Second, make it safe. Put some safe guards in place to make sure a change doesn’t accidently get made to the real production database. Keep your ACLs tight, change the passwords and practice proper database user management. Lastly, a bonus outcome is that you are constantly testing your database backups and recovery procedures.

Minimize the steps between development and production.

Each step, especially the manual ones, between the developer’s machine and production is another point where unexpected changes can be introduced. It sounds obvious, but the gates code must pass through before entering production must be purposeful and prove their usefulness. For example, if code must go through a regression, but no regressions are being found, then that step is not useful and just increases the probability of unintended change. I have been a part of a process where code had to be deployed to “staging” where it was then hooked up to the live production database, but was only to be exercised in a read-only manner. After doing that for many, many releases and very rarely finding defects before production, we decided to just get better at deploying to production and eliminated the deployment to staging. In the end, we got faster and with no impacts to quality. Less can be more.

Depending on your situation, database size, regulatory compliance, etc. you might to be able to do all of them, but breaking down those walls will yield better code, fewer bugs, happier engineers and fewer sleepless nights to name a few. So, grab that wrecking ball and break down a few walls.

March 19, 2014