devopsy.kinsta.cloud

Tag: DevOps in the Cloud

Best Practices for DevOps in the Cloud
DevOps is commonly defined as a methodology that combines software development with IT operations. The goal is to shorten a system’s development life cycle while delivering features, fixes and updates frequently in close alignment with business objectives. Putting DevOps into practice is an ongoing experiment for many organizations.

Developers often struggle with continuous integration (CI) or continuous delivery (CD) performance, testing delays and other bottlenecks. In fact, one survey reported that only 34% of projects finish on time and only 42% come in on budget. Security and development teams must collaborate to keep up with the rapid development life cycle without sacrificing security with every update.

Enterprise SaaS apps are not distributed like traditional software—they are continuously running services. So, CD is all about updating the service with a set of changes. In public clouds such as AWS and Heroku, this is done through pipeline stages (e.g. dev, test, staging and production). In platforms such as Salesforce, stages become sandbox environments, ephemeral scratch orgs and production instances.

How can development teams shorten the development life cycle without sacrificing the integrity of the features and updates? In this article you will learn key DevOps practices to achieve predictable, reliable release management.

Advanced Planning Ensures All Boxes Are Checked

Agile planning is now de rigueur for enterprise development. It is best used if:
- The product itself will most benefit from flexibility.
- Regular face-to-face interactions and collaborations are needed between team members.
- The end product requires regular updating, monitoring and continuous delivery to the client throughout the life cycle.
For CD, ensure your user stories are sized correctly and that each story rolls up to an Epic that represents a standalone feature. This enables your team to release bite-sized pieces of functionality that are consumable by users and traceable back to your release plan. Also, define acceptance criteria for use in the verify stage so you know when something is done-done.

Manage the Source Code, Especially When Working in a Shared Environment

There are a few practices of app creation that impact CD. First, source code is not the only element of an enterprise solution. Schema, user interface, access control rights and static resources such as icons and images are all part of the creation process and should be managed just as diligently as source code. They should all be checked in to a version control system (VCS) as a single source of truth.

Second, ensure that developers are making changes in a segregated environment. In some low code platforms, such as Salesforce.com, this becomes a challenge, but even in a database, multiple database administrations (DBAs) can step on each other’s changes when working in a shared environment.

Functional Testing

Automated quality processes are difficult to implement, but essential Selenium is a great tool for testing functionality and should be part of your DevOps practice. But there are several verifications that should be made before you get to functional testing. Static code analysis tools, such as PMD, are essential to ensure your development team is coding against a single style.

Define a team style and enforce its use. Unit test coverage is also essential—incorporate them with a coverage of at least 75%. Finally, after these automated tests pass, implement a manual peer review. Old hands are able to spot opportunities for performance improvement where automated tools may not.

Changes Should be Released in a Defined Pipeline with Strict Quality Gates

Packaging for CD is about capturing changes that can be incrementally released into the next stage or environment. Development teams should dedicate a branch for each stage and bundle incremental changes into a promotional branch to leverage the merge conflict detection in your VCS.

Changes should be released in a defined pipeline with strict quality gates that ensure the verification steps and enforce the order of release stages. Developers should not be able to bypass key stages throughout the development cycle—for example, moving from development to production.

Focus on making smaller changes, which result in reliable, predictable releases. Finally, be sure to automate promotions. This enables your team to validate the promotion scripts in integration, test and staging before releasing to production, which decreases the likelihood of an unhappy surprise.

Automate Configuration, Especially for End User Access Rights

Make testing for proper configuration part of your verification process. Platforms such as Salesforce have powerful profiles and permission sets that enable user access to only the right information. Improper configuration can expose personally identifiable information (PII), credit card numbers or medical health data.

Monitor for More Than Just Service Availability and Performance

Most SaaS companies monitor service availability and performance. But excellent companies also monitor data access to detect improper configuration changes and monitor end user feature usage, which gives product managers visibility to their usage.

— David Brooks
May 2, 2019
DevOps’ Contribution to UK’s Digital Sector

The UK’s G-Cloud is a national initiative that helps the public sector find and use cloud-based services. This is basically a DevOps initiative that combines cultural tools, practices and philosophies to help increase the ability of an organization to provide services and applications at high speeds, which helps them provide better services to their customers. Its high-security protocols, improved collaboration, reliability, rapid delivery and high speeds makes it better than traditional management processes.

This technology seeks to breach the gap between private companies and the public sector—in particular, the government—by offering a wide range of digital services. So far, the adoption of DevOps in the UK’s public sector has benefited the UK’s digital sector in numerous ways—for instance, DevOps has enabled the government of the UK to ensure the digital-first experience for the public.

DevOps enables marketers to establish clear marketing-related analytics. Using its cloud-based technology, marketers get a chance to influence how they can use available data in decision-making and be competitive. Besides, its interdepartmental communication platform offers a solution to companies to change how they operate to meet customer needs. This can easily be done by integrating customer feedback in quality improvement. The use of DevOps in making IT infrastructure, application operations and application development work as one reduced failure rates by more than 50 percent, reduced time to market by 50 percent to 75 percent and increased capacity creation by 25 percent to 30 percent.

In the engineering digital sector, DevOps has played a crucial role in changing the focus of from software development life cycles (SDLCs) and software to engineering and services. The adoption of DevOps in different sectors—particularly engineering and governments—has shaped its demand and supply over the last decade.

Before the invention of DevOps as a cloud service, the government was not digitally structured to handle the demands of the population of today. DevOps bridges the gap and offer digital-first services to the public. Since its implementation, DevOps has made operations that were once a dream become a reality for the public. The most benefiting organizations are public organizations; they procure digital cloud services, then move on to use the services to enhance the provision of services to the public through a faster delivery framework that enhances the competitive advantage of the involved organizations.

The growing use of digital technology in the UK has led to an astronomic type of transformation that is bound to positively affect the infrastructure of the nation moving forward. It is undeniable that more companies using DevOps is spurring a revolution of digital transformation that is bound to change the way services are provided to the public. The public sector in the UK has now a more valuable competitive advantage in that it is able to provide its services in a digital-first kind of experience. DevOps has therefore not only made services available to the digital population of the UK but has enabled access to every single individual in the nation.

Conclusion

The transformation brought about by DevOps will also increase the quality of services by ensuring the continuation of cloud services that provide faster service delivered. However, one unique aspect of the digital transformation related to the adoption of DevOps is that the rate at which IT skills will grow in the public sector will be faster than previously experienced in the public sector. This is bound to be the inevitable future of the use of DevOps in the digital sector. The growth of IT skills to match the use of the advanced technology will enable better use of the cloud services, creating the potential for continued growth in the future.

— Subramonian

May 4, 2018
Legacy On-Premises Apps Got You Down? Cloud DevOps to the Rescue!

With all the buzz around DevOps we tend to assume that everyone is doing it. But the reality is that most companies are not. That’s because organizations using legacy, on-premises applications think that it can’t work for them, and, according to IDC, the vast majority of enterprise apps still reside on-premises. Even in 2018, less than 28 percent of the worldwide enterprise applications market is expected to be software-as-a-service (SaaS)-based. But even if your organization is weighed down by legacy apps in your data center, that doesn’t mean you can’t do DevOps in the cloud.

DevOps Requires Cattle

A good reference point for DevOps is the “cattle versus pets” analogy. That is to say, you should treat your servers as cattle, not pets: They all should get fed and treated equally so that all your test labs will be the same. The problem is, when it comes to test labs for on-premises apps, it seems as though everyone has pets, while only the new breed of SaaS apps are treated like cattle. This is because the labs are very complex and have a way of accumulating their own unique settings, prompting conversations such as:

“What is this 5000-line XML?!?”

“Oh, that was configured by Adam in 1999 and we copy it verbatim ever since.”

“Who’s Adam?”

“Oh, he hasn’t worked here for five years.”

Good DevOps practices require an organization to have multiple identical labs, available on-demand. You don’t get that with pet labs. Therein lies the problem.

The Dreaded Configuration Drift

Pet labs also lead to some other bad phenomena, such as configuration drift. Configuration drift results from the fact that it’s really hard to maintain even one lab correctly, so deploying and managing two or three or 20 labs becomes a daunting and very error-prone task. Since it’s easier, people often will simply copy images of other labs. But over time, those labs keep getting manually configured to account for various version needs and bug fixes. Hence, the “drift.”

One cure for drift is starting from scratch every time. But for those “pet” labs (like the one Adam, who no longer works here, configured himself and was the last person alive who knew how and why), starting from scratch every time is intimidating and impractical, so they just keep drifting along. That means even though your test labs are supposed to be exact duplicates, they aren’t.

‘It Works for Me’ and ‘Cannot Reproduce’ Syndromes

Inconsistencies between labs also lead to some common syndromes in bug tracking systems known as “works for me” and “cannot reproduce.”

Here’s how these play out: The QA team has its own set of labs (hopefully they aren’t individual snowflakes), as does each dev team, and there are customer environments—another type of beast. Now all hell breaks loose—some bugs appear only in QA and not in the customer or dev environments, while others are only at the customer level. Miraculously, everything always works in the developer lab because those labs haven’t been reinstalled since the era of vacuum tubes and they contain every registry hack, debug symbol and kernel patch known to mankind.

The “cannot reproduce” syndrome creates a lot of wasted time. When the QA person finds a bug, they talk to a developer, but developers are not in a bug-fixing sprint, and say they will get to it in a couple of weeks. So the labs move on. Meanwhile, two weeks later, a developer finally puts in a few hours but doesn’t find it and closes out the bug with “doesn’t reproduce.” Then the QA puts in another three hours trying to find it again.

The Serious Issue of Lab Scarcity

Last, but not least, physical in-house labs cause lab scarcity. Many companies find it too expensive and not cost-effective to build and maintain the number of labs they would need to enable all teams to work in parallel. With a limited number of labs, developers have to wait for others to finish their tasks before they can claim the lab for their own work. From my own experience, we had developers that wouldn’t leave the lab to take lunch breaks because they didn’t want to risk losing their place in the queue; they actually lost weight from not eating lunch all week (maybe a good thing for some?).

Lab scarcity is literally a DevOps killer, not to mention the consequence of wasted developer time. (One can argue that wait time for lab spent on Reddit isn’t truly wasted, but that’s another story.) The point is, you want a lab for every developer, for every tester and for every CI run. Disney World-level wait lines are not a DevOps enabler.

DevOps in the Cloud

The ultimate goal is to create identical, reproducible test environments for every test, tester and run that are elastic, burstable and present short feedback loop times. This is what brings you to cloud.

Using the cloud to replicate and share identical labs that can easily be started over and saved at a specific state is the way to avoid these problems. But not all public cloud providers are created equal, especially when it comes to re-creating complex, on-premises environments in the cloud.

Commodity cloud providers AWS, Google and Azure present one option. However, they have little to no support for uploading your images, no complex L2/L3 networking support, little to no nested virtualization and no promiscuous mode adapter. After all, they are built for “cloud-native” apps.

Another option is to build your own data center. But this is costly in terms of hardware (which repeats every two to four years) and fairly complicated to do properly, and involves lots of overhead for creating and maintaining them. Eventually, you’ll probably realize you’re not an infrastructure-as-a-service (IaaS) company and that you prefer to focus on your core business.

The best option is to look for something as convenient and agile as commodity cloud, but which is specialized for your on-premises challenges.

For many cases, you’ll find that specialized virtual IT lab providers let you replicate production server images in the cloud without modification, and also save templates of production labs regardless of complexity or number of nodes. These providers should enable you to quickly spin up as many identical environments as needed and then easily connect tools and scripts to automate processes across systems.

When it comes to debugging, make sure you can easily capture snapshots of issues to get back to the same spot later. Then look for automation capabilities that save time and control costs, such as environment policies, auto-suspend, Zapier integration, organizational structure modelling and more.

A Sample Use Case

Smaller startups often run into challenges because they want to run fast with very big systems. Take, for example, the case of a financial trading company that started out by investing in production instances and tweaking them manually. The system comprised more than a half-dozen subsystems, each with its own programming languages, technology stacks and deployment methodologies. Developers working on one subsystem had virtually no knowledge about the others, except for how the interfaces behaved. The setup worked in production (most of the time …), but then quality issues mounted.

The company first invested in a waterfall-like methodology with long QA cycles. But much of the QA happened in production because there was no complete staging system. And when you are a financial trading organization dealing with people’s money, doing QA on your production environment becomes costly in multiple ways.

The lack of a safety net in the form of a complete staging environment also slowed the deployment of new features. Even if a change was working nicely internally within the subsystem where it was made, it still would have to be orchestrated with the larger system—sort of like a composer who changes a note for one instrument then has to call all the players to hear it with the entire symphony.

The company eventually solved its issues by uploading complete images of its existing production machines to the cloud, then divided them into subsystems with different blueprints. Once the company established a “golden blueprint” of each subsystem, it became very easy to test by mixing and matching servers from different blueprints. To continue the composer analogy, the company could now change the notes in one subsystem and see how it sounds together by playing them together with recordings of all the other players.

The company ended up achieving a dramatic decline in the number of production issues, and significantly increased the speed at which it could develop new features (no doubt one runs faster on the high wire when a safety net exists).

Special Bonus for Docker Users

Snapshots also come in particularly handy when using Docker. In a Docker-based environment, current practice is to run CI tests on the actual Jenkins slaves in an ephemeral way, which disappears between consecutive runs. This means you only have logs of the end results. But what happens when there is a failure and you want to debug? The bug may appear to be non-deterministic when it is actually deterministic because something has changed between tests. And even if the bug is deterministic, you still have to re-create it. Imagine if instead of running tests on ephemeral Jenkins slaves, you spin up an ephemeral cloud-based Docker machine and, in case of test failure, you set your CI to take a snapshot. You can then go back to your snapshot later and employ traditional debugging techniques, saving precious developer time.

DevOps for All

Doing DevOps in the cloud can make all of this possible. No matter how old and complicated your architecture may have become, you can still get into the game. Just remember that cattle are easier to herd than cats.

— Muly Gottlieb

January 9, 2018
DevOps: The (Absolutely Critical) Cloud Enabler
A couple weeks ago, I gave a talk titled, “DevOps: The Cloud Enabler” at a conference. As I was putting together some requisite materials previewing my session, I made a few mistakes. (Yes, I’ll readily admit it.) I used the words “should” and “can” and “consider.” Instead, the words “must” or “not optional” or “operationally imperative” make better sense.

Companies are moving to the cloud (or at least endeavoring to); that’s a fact of IT life. If you’re not doing it, your competitors are lapping you. That may not be what you want to hear, but hey, reality bites. The cloud is simply the most scalable, flexible, efficient way to run a business these days. Cloud was just a buzzword for the longest time, but companies in all industries are actually putting it to work now. If your CIO isn’t at least in the midst of forming an application migration plan to the cloud, it might be time to light a fire under him or her; otherwise, start reading the writing on the wall and look for a new gig.

Easier said than done, though, right? I’ve heard many a CIO say, “We’re going to have all our apps in the cloud by 2018!” No, you’re not. You don’t know how. You don’t have the tools and road map to get there. Let me enlighten you with some of those stronger, more urgent words I mentioned earlier. In your move to the cloud, DevOps MUST be the conduit. Period. Stop. End of argument. Game, set, match. Not optional; it’s imperative.

Here’s why: One of the most fundamental problems that’s part and parcel of a move to reliance on the cloud is that IT orgs want every scrum team to have its own environment, complemented by an individual database instance. Eventually, that leads to creating a distinct database instance for every single developer. You probably can see where this is headed.

I’ve used this comparison time and time again, but cloud and database instances become like the wire hangers in your closet you accumulate every time you pick up clothes from the dry cleaner. They multiply over time and, all of a sudden, you seemingly have a million on your hands, with no idea where they came from and no good way to get rid of them. To compound things, once the proliferation begins, it’s hard to stop. Shut down a database instance (as with an application instance), and you lose the underlying data. No can do.

That reality spells disaster for the database administrators (DBAs) being flooded with thousands of development SQL scripts and database changes across multiple app release cycles across multiple environments. For DBAs to attempt to keep pace with this rise in change requests, organizations all but have to go through DevOps’ economies of scale. DBAs need reprieve, and DevOps can be just that. David Linthicum put the solution thusly: “Stop separating the development and operations skills; they are tightly coupled and need to be peers. You’ll find this is the hardest part of the process of moving to devops and the cloud.”

Spot on. To be more prescriptive:
- Developers need to understand what DBAs are dealing with as part of the cloud migration and know how their work rolls downhill.
- Automation needs to reign supreme, removing manual work.
- Developers need to have the ability to be involved directly in cloud ops and enact 95 percent of changes themselves.
- DBAs need to be freed to oversee the 5 percent of changes that truly need their attention and management skills.
- The flow from cloud application updates to cloud DB changes needs to be seamless.
In bringing development and DBAs into sync, DevOps then becomes the vital remedy, the critical cloud enabler.

TL;DR: If you have DevOps alignment, processes, responsibilities and tooling properly implemented in advance, then when you decide to move to the cloud, you can simply flip the switch from on-premise to the cloud. One click and you won’t miss a beat. Seriously, it’s that easy. Sound appealing? I know.

If you truly want to get to the cloud, you must have a sound DevOps strategy that encompasses application development, web and the database. Without that in place, swim at your own risk. This is the year that we’ll see much more of the true impact of DevOps. And that’s coming just in the nick of time, because without DevOps, CIOs and their IT orgs would be in for some nasty surprises in their all-important moves to the cloud.
July 25, 2016
DevOps in a Hybrid Model
As enterprises have begun to demystify the value proposition around cloud adoption, several best practices, optimization techniques and delivery models have been described in the technical community around cloud enablement and large scale adoption. While the debate between choosing private cloud vs. public cloud model has been longstanding, enterprises have been looking to mitigate the level of software disruption caused by such large transformation. A Hybrid model is thus a very appealing prospect since enterprises can still technically control the security policies and governance models, while also taking advantage of capacity bursting and other benefits of the cloud.

As with most technologies, adopting a hybrid cloud solution doesn’t buy you a free lunch either. There are several complications that are being discovered as enterprises mature into this model. One of the biggest challenges is to keep the software development lifecycle in sync across multiple target environments that are the basic backbone of a hybrid cloud model. Thus, having a robust DevOps solution is one of the key necessities for making a hybrid model work.

The Business Case for a Hybrid Cloud Model

In today’s complex business world, enterprises either by choice, or by circumstances, are choosing a hybrid model of IT solution. Enterprises are best served by implementing a “best of breed” solution, which means a combination of several domain specific SaaS (Software as a Service) and PaaS (Platform as a Service) solutions. This also eliminates any risks of a “vendor-lock in”.

So by design, a hybrid model will have a combination of the following characteristics:
- Multiple platforms within an enterprise architecture
- Both traditional as well as virtual platforms
- Domain specific SaaS/PaaS solutions, typically in a multi-vendor scenario
- Lack of standardization across cloud platforms (hypervisors, hardware types supported, so on and so forth)
Now, the challenge is to make IT delivery work seamlessly around these characteristics. This is only possible by a high degree of automation in the delivery process, operations process as well as the business planning processes. Those are the very elements a DevOps platform attempts to solve.

The Challenges of Cloud Adoption

Let’s take a quick look at some of the core existing IT capabilities built into a large enterprise.
- Security model
One of the biggest challenges towards cloud adoption is the notion of extended or revamping existing enterprise security model for cloud (and as a consequence, into DevOps solutions). How can one make sure the same security policies for data classifications and user management are applied especially on a public cloud model? Will existing audit processes suffice in a cloud model? Can existing security policies be leveraged as-is or with minor modifications and still be able to take advantage of a cloud model? These challenges will drive enterprises into a hybrid solution.
- Support and Governance models
Many enterprises have built up a history and matured best practices around IT governance and support structure (think ITIL). The cloud model in many ways disrupts those practices. One of the ways to mitigate the disruption is to adopt some form of a hybrid model so some best practices and core support capabilities can still be leveraged while taking advantage of the cloud.

One of the emerging business models for enterprises is to avail a more “managed services” model so they can focus more on their core business capabilities rather than on IT. In a managed services model, one or more service providers will be responsible for portions of the overall cloud solution while the enterprise would retain responsibilities over centralized capabilities like security. So the “hybrid” nature is not just limited to physical environments, but also around support models.

Implementing a DevOps strategy could help mitigate a lot of these issues.

Hybrid DevOps Scenario

A typical SDLC (Software Development Life cycle) DevOps looks like the following.

Figure 1 – A Typical DevOps Pipeline

The aforementioned actors in figure 1 may or may not actually be “people” since in a true DevOps world, those actors are replaced with some degree of automation. But it sets the context on what types of roles & responsibilities are expected in a typical pipeline.

Now, imagine this pipeline that is multi-folds, across several application or components. This is the real nightmare of deployment and release management issues that one has to account for since in a hybrid model. Each of these applications/components may technically reside in a totally different environment developed and managed by totally separate groups with different service level agreements.

A centralized DevOps platform, standardized across these application environment boundaries (for the most part) to provide a single pane of control over code, release and deployment management is what is required for a successful adoption of hybrid solutions.

Here is an indicative centralized architecture to solve the deployment and release management issue described above:

Figure 2 – DevOps Platform Architecture Example

The components listed in figure 2 is purely indicative and shouldn’t be misconstrued as any product marketing or messaging. The main challenge will be to standardize components to the extent possible (this is easier said than done) and stitch together a meaningful platform that works within an enterprise (based on existing tooling & processes, skills, funding etc.). Bear in mind I’m merely touching the surface in this article with regards to a DevOps platform, and limiting this to a tooling discussion. Bigger challenges do exist in the form of process and organizational transformation, which are also necessary to make a DevOps implementation successful. I’ll pen those topics in a separate article 🙂

Summary

DevOps for Hybrid cloud implementations will soon become the de facto model for process transformation in large enterprises. There is simply no other way to go about such large transformation in a purist way. There is a wide verity of use cases for a hybrid model, and most enterprise situations would fall under one or more of those categories. A robust DevOps platform is absolutely critical to making a Hybrid cloud adoption successful.

About the Author/ Sunil Joshi

Sunil is an Executive Architect with IBM. He is currently the Application Development & Innovation (AD&I) DevOps offering owner for Global Business Services. He is IBM Senior Certified and Open group certified Distinguished Architect. His specialization are in the areas of hybrid cloud solutions, platform as a service and DevOps strategy. He’s created complex hybrid solutions on cloud and DevOps for large enterprises. Sunil has authored several IBM Redbooks, blogs and also a co-author of the IBM Cloud Computing Reference Architecture (CCRA).

Sunil is passionate about international music, multi-cultural cuisine, active sports and mentoring school & college students on career path and technology.

Contact Sunil on
July 31, 2015