devopsy.kinsta.cloud

Author: Rajat Bhargava

Tips for Modern IT Admins’ Career to the Cloud

Fifteen years ago IT professionals experienced a drastically different day-to-day on the job. While the main mission has always remained constant—to take charge of all internal infrastructure—the infrastructure itself has moved off-premises over time.

The complexity rested in networking for IT professionals. Routers, switches, firewalls and other gear needed to be managed so the network would function. Employee devices also required attention from IT admins, but these devices were typically all Microsoft Windows machines. This allowed IT to standardize its tools and platform.

What Has Changed for IT Admins?

Time has drastically altered the lives of IT admins and, after 15 years of evolution, organizations have moved away from large, ungainly on-premises networks and data centers. However, the list of endpoints for infrastructure has grown rather complex, specifically as a result of the implementation of different platforms and operating systems.

The challenges faced daily and the knowledge needed to be successful has exponentially grown. Below we will take a deeper look at three of the larger changes that IT professionals must ponder as they elevate their careers in the burgeoning cloud environment.

Rethinking Networking

The time when it was important to know how to configure a router or switch is slipping away. In the modern IT world, this task has been adopted by cloud infrastructure providers such as AWS and GCP, as well as the telecom companies that deliver your internet.

The job for IT admins has leveled up, as now it is vital to know how to manage and configure the higher levels of infrastructure. The ability to route traffic through security groups and automate the configuration of systems is a necessity. Writing code to automate these tools is now the norm for an IT admin.

Regulating Employee Wants

In the past, IT could dictate the tools that employees had to use, but that ship has sailed. Employees have begun to adopt their own tools to further maximize their efficiency, which forces IT to adapt and get ahead of the game.

It is crucial to show employees both flexibility and confidence when leveraging new technologies. As a result, employees will work with you and not go out and create shadow IT projects.

Seeking out and securing cross-platform tools that allow you to manage the various endpoints and services your team is leveraging is of the utmost importance. These tools do exist; however, they typically are not offered by traditional vendors such as Microsoft, HP and Oracle.

Security is the No. 1 Concern

Not long ago, security for IT admins was simply firewalls and anti-virus solutions. Fifteen years later the world of IT security is far more intricate.

To be successful in the IT world, an admin needs to be an expert at infosec so they can understand and defend against the massive world of cybercriminals. Identity theft is rampant and hackers are always trying to locate vulnerable systems that have not been patched. It is because of this that IT admins need to be constantly on their toes and protecting their organization.

IT is the Core of an Organization

Without IT, business simply wouldn’t get done. Sales would not be able to email contracts, products would go undelivered, customers would go unbilled and other core activities would fail.

The IT admin is a vital part of every organization, which is why they are in high demand today. Mastering new technologies and understanding the constantly evolving approaches to IT can be the difference between an organization falling flat on its face or leading an industry.

— Rajat Bhargava

February 4, 2020
SSH Key Management and Your Organization

It always starts out the same way: a few machines, a few users and a few SSH keys to manage. You manually copy their keys (and your own) to the appropriate machines. You might even say, “Screw it,” and share one key for root access. Admit it. It’s OK. Well, no, it’s not OK, but we all do it at some point, even though we know better. Like pouring grease down the drain: We know we shouldn’t do it, but sometimes it’s just so much more expedient.

Anyway, you’re lucky enough that your company starts to scale and things start to get more complicated. More users, more servers, tons more keys. Like having too many kids at a birthday party: You start spending all your time making sure they have ice cream, they don’t fall off the trampoline, they don’t get into fights. You’re spending way too much time managing your users and their keys, so you realize you need some sort of system to keep everything coordinated. Often this starts off as an Excel spreadsheet tracking who has what key on what machine (another thing we all do but don’t like to admit to our friends). That’s a start, but that’s just one of many things that you need for SSH key management, and the less manual the process, the better.

Centralized Store of Identities

First, let’s admit that Excel spreadsheets really aren’t sufficient to track who can access what on your servers. Without centralized key management, you’re likely to have temporary accounts that are forgotten and hang around forever. You’ve also got the problem of rogue users: One of the users you authorized users goes around you to add someone else and you as the administrator have no idea. You want to have one source of identity for these machines—one place to add, one place to delete, one place to audit.

Provisioning New Users

If I’m adding a user a month, sure, I can go and add peoples’ keys to each machine manually. Of course, this presupposes that you’ve got the public key from the user and have it to move around. Which presupposes they’re savvy enough to manage making keys themselves. A key management system should make this process simpler for you, reading from your identity store and putting them onto the appropriate machines.

Deprovision Departing Users

If you have someone leaving on good terms, it’s just good practice to remove their keys from your servers. It keeps things clean and secure. If they’re leaving on not-so-good terms, though, you better know where the keys are on every machine and be able to remove them quickly. This also reiterates the need for keeping separate keys for all of your users: If they have the private key for root, you’re going to have to not only get that key off the machines but also hand out the new key to everyone that is supposed to have access to the shared account.

SSH Key Rotation

Keeping the same key around makes you more vulnerable to attack if an old key is compromised. The PCI DSS standard says that keys must be rotated at least annually. Making this as easy as possible for the user is important, and you must have someplace to record when each key was last rotated. For a lot of organizations, this is probably that Excel spreadsheet somewhere that’s dusted off right before an audit. Doing this right—rotating all your keys across all your machines on a regular schedule—calls out for an automated process.

SSH Key Robustness

Let’s say everyone in your organization is using 1024-bit keys and suddenly word comes down that you need 2048-bit keys instead. Having a key management system is going to help coordinate the update. Otherwise, you’re going to be looking at every key on every server (notice a pattern here?) to see which need updating.

Getting Rid of Defunct SSH Keys

This isn’t a one-time thing; keys (and users) have a tendency to be provisioned “temporarily” and then forgotten and left on a machine. A good key management system will find and remove these keys—or at least notify you of their existence.

Allow Users to Self-Service

Although we don’t want to allow users to willy-nilly add accounts for themselves and each other, we also don’t want to have to create and handle their private/public key pairs and go into every machine and make changes whenever they lose a key or want to update their keys. Ideally, our key management system would allow for (secure) user control of their keys, allowing resets and updates.

While most of these things are possible to do by hand, it’s just not easy and it will end up taking way too much time. Maybe you can script it—and in fact, many companies do. But many companies used to write their own string libraries, too, and run their own mail servers and manage their own payroll. It’s not the best use of your time.

— Rajat Bhargava

February 3, 2020
Setting Up Google Authenticator on Multiple Servers

Google Authenticator is pretty great. It allows me as an administrator to set up and configure multi-factor authentication into my UNIX boxes without having to spend money on a tool like YubiKey or RSA tokens. It’s easy to set up on any type of phone—no specialized hardware or dongles needed. It’s also pretty cool in that you don’t have to have network access from the server to the outside world. Since Google Authenticator is time-based, it doesn’t need to send an SMS or do a callout to a centralized server to get the current valid token.

What is a bit painful, though, is needing to have a different Google Authenticator token for every one of my servers. The standard setup would have you run the google-authenticator command on each and have as many tokens as you have servers. Obviously, this quickly becomes unwieldy and untenable.

Instead, I want to have one token for multiple servers. Here’s how I installed and configured Google Authenticator on each system.

First Machine

On my first machine I’m going to install Google Authenticator and create a secret key—the exact flow I’d use normally.

1: Install Google Authenticator. This is pretty well-documented; examples can be seen at untrusted connection and How-To Geek. I won’t walk through each step since this part varies from OS to OS; the rest of the steps are identical, however.

2: Restart the ssh service

$ sudo restart ssh

3: Run the google-authenticator command to generate a secret key for your account, which you’ll store in your phone. This information will be stored in a configuration file that we’re going to get into later. I don’t need to comment that you actually have to enter the secret key into your phone, do I?

4: Give it a spin. From another shell go ahead and try it out.

5: Let’s take a look at the configuration file. We’re going to copy these contents to our other machines that we want to have the same secret key.

Installing Google Authenticator on Additional Machines

For all other machines I’m going to install Google Authenticator as normal, but I’m going to use the secret key from the first machine. This will let me log in to each of them using that same secret key that I stored from the first machine.

1: Install Google Authenticator. Again, other places describe this in detail. We’re going to install the program but not do the creation of any secret keys

2: Create the configuration file and add the content that we got from the other machine.

3: Set permissions for the configuration file.

4: Restart the ssh service.

5: Test the login.

Voila! A shared secret key across my servers! Happy day.

— Rajat Bhargava

January 31, 2020
2014 – The year of identity security breaches

There’s no doubt that 2014 will be remembered in the IT industry as a year of major security breaches.

The year started with virtually every US resident worrying whether they’d been affected by the Target breach, and ended with the massive security breach of Sony and their film “The Interview.” And was, all along the way, peppered with other notable breaches.

The interesting theme, with all of the major security breaches, was that they were almost all related to compromised credentials. As we know, having the right privileged user management is one of the top ways to protect your network, you don’t want to share same hacker-limelight that Sony did.

But just for posterity, here’s a review of some of the top security breaches, and how they happened:

Target

What is considered the “biggest retail hack in history,” shook nearly every US resident in early 2014 as credit cards from 1,797 stores were compromised around Thanksgiving 2013. The hackers plotted against the company while Target was installing new security software. The repercussions were severe. Target reportedly spent $61 million on legal damages instigated by the breach, and profits fell by 46% in the holiday shopping season.

Suggested read: http://www.businessweek.com/articles/2014-03-13/target-missed-alarms-in-epic-hack-of-credit-card-data

Sony

Although the investigation into what occurred is still on-going, early accounts point to compromised user credentials (or even perhaps an inside actor providing credentials) being leveraged to access, gather and exfiltrate data. We know, of course, that the attackers gained a tremendous amount of data including unreleased films, personal emails, HR data, and other confidential documents. Bummer for Sony. The WSJ estimates the breach cost them $100 million.

Suggested read: http://techcrunch.com/2014/12/16/hack-sony-twice-shame-on-sony/

Celebrity iCloud Photo Hacking

While this particular breach was extra-sensationalized because of all of the celebrities involved, the Celebrity iCloud hack boiled down to plain old identity theft. Hackers identified email address, passwords, and the like and started gaining access to the iCloud data. Once in, they were able to copy all of the files. When celebrities realized they couldn’t login to their iCloud accounts, they naturally reset their passwords and were able to continue. Most of them didn’t even know they had been compromised.

Suggested read: http://www.cnn.com/2014/09/02/showbiz/hacked-nude-photos-five-things/

eBay

In late May, eBay notified its customers: it had been compromised. Credentials for employees with access to their critical systems were compromised, is appears, several months prior to their detection. Financial information, which was coyly stored separately, was not hacked. Lucky for eBay.

Suggested read: http://www.forbes.com/sites/gordonkelly/2014/05/21/ebay-suffers-massive-security-breach-all-users-must-their-change-passwords/

Home Depot

Over 56 million credit cards and 53 email addresses were stolen after a third party vendor’s employee’s credentials were stolen. This allowed hackers to gain access to the Home Depot network and make off with the confidential data. This breach was reminiscent of the Target breach in which another third party vendor (HVAC provider) was compromised and that subsequently led to Target’s compromise.

Suggested read: http://www.usatoday.com/story/money/business/2014/11/06/home-depot-hackers-stolen-data/18613167/

As you can see, the common thread to these security breaches is compromised credentials. Hackers realize the power of admin credentials in leveraging open systems. And, because many companies do an inadequate job of protecting their systems—despite patching, hardening, and firewalls—user admin access becomes an easy hacker target.

How can we stop this seemingly simple vector of breaches? Protect your credentials with good user management. Just patching isn’t going to get us there. At Jumpcloud we spend a lot of time looking at this, and ensure that security and user credentials are watched with close vigilance and best-practice security to keep hackers out. Drop me a comment if you have any other thoughts on this topic.

January 13, 2015
How to talk to your CFO about DevOps
How to talk to your CFO about DevOps

As DevOps practitioners, we understand the value of DevOps: It helps keep our infrastructures automated, allows us to push out code regularly, and reduces the frequency of breaks. But, does our CFO understand its value as well as we do? Probably not. The question is: How do you explain the benefits of DevOps to your CFO in order to justify the cost?

The quickest way to a CFO’s heart is through his wallet. So, the key here is that as DevOps practitioners we need to talk in the CFO’s language, not our own. We need to speak in dollars and cents rather than bits and bytes. This is not dissimilar to many other instances when IT departments have to work with C-level teams that are business, if not technology focused. But sometimes this is incredibly hard for IT folks to do. It’s just not how we are wired, and similarly, CFOs aren’t wired to think like IT folks. While every CFO and organization is different, there are some common things they all care about.

For example, most CFOs love to count things, especially money. Where is it going? Why? What is the benefit to the company? What’s the ROI? You know typical bean counter type of questions that make a lot of technical people squirm.

Your CFO can be one of two types and you can have a lot to do with how your CFO views DevOps. The two types of CFO are:
1. A CFO that understands the why behind DevOps and is a valuable ally to an IT organization.
2. A CFO that doesn’t understand the benefits and will stifle even the best, most productive programs.
So the question is how do you wind up with CFO number 1? Well, part of it is that you need to educate your CFO in the value of DevOps. This is exactly a situation described in “The Phoenix Project” by Gene Kim et al, but it bears repeating here. The other part is that you need to talk to your CFO about a few key topics that will be central to winning them over.

Let’s dig in.
- Generating cash (and by consequence revenue) – the lifeblood of any company, cash, is a top priority for CFOs. A strong DevOps program can tie directly to generating cash for an organization. Here are some examples of how DevOps can help in this area:
o   New products come out to market quicker and more tightly aligned to customer needs. Translation: revenue comes quicker from new products because they are more on the mark.

o   Products are more stable which means that there are fewer customer issues and less downtime. If the organization is subject to an SLA, there should be less money going back to customers. If they aren’t subject to an SLA, with a more stable product, customers are less likely to churn.

o   Back-office systems such as billing and provisioning will have better uptime and reliability. Fewer errors means that there is less churn and more opportunities to sell happy customers more products and services.
- Reducing expenses – the other half of the equation for a CFO is how to keep costs down. DevOps can be a huge win in this area. A fundamental tenet of DevOps is automation. By automating rote tasks, your team can accomplish more. They don’t need to hire more people and/or they can take on more projects. Either way, it is a significant opportunity for IT organizations to do more with less by having resources better aligned on the most critical problems. IT organization should walk CFOs through how investments in automation can pay off.
- Growth – for CFOs, it’s not just about managing the revenue and the costs, it’s also about how the company is going to grow. How can DevOps help a business grow? Growth comes from the increased sales of existing products/services and/or the introduction of new solutions. Building a better product is core to the DevOps mission. In my opinion, the definition of DevOps is actually the increased probability that you will find product/market fit. This means, if you can quickly iterate to find what customers need, you will sell more of it. By building DevOps into your organization, you will also find what customers need so you can introduce new solutions as well.
- Long-term stability – CFOs tend to feel more comfortable with long-term stability solutions. Part of the mission of DevOps is to increase stability of the product and infrastructure. By turning around quick iterations, product solutions should continually be more reliable as well. If the product performs well, customers rely on it. The result? Loyal customer bases and steadier revenue streams. Another aspect of long-term stability is predictability of resources and expenses. DevOps can drive that predictability by closely tying demand for solutions to the expenses. A DevOps program can deliver on a regular cadence of product which can translate into increased visibility and stability for organizations.
The conversation with a CFO about why an organization should invest in DevOps is crucial. If IT leaders can have that conversation using a CFO’s key concerns, then the outcome is very likely to be more positive. Our suggestion is incorporate the four key concerns CFOs have – cash, expenses, growth, and stability – into the conversation.
October 14, 2014
Is Ops the quarterback of the DevOps team?

Without stepping too deeply into the charged debate on whether or not developers are at the pinnacle of the IT food chain, I’ve recently observed that many of the DevOps leaders I’ve been speaking with are from operations. For me it begs the question – are the ops folks quarterbacking the DevOps process at most companies? In other words let me put it bluntly, are the Ops folks leading the way forward in DevOps?

Many DevOps thought leaders have taken great pains to describe and articulate that development, ops, qa, security and many other groups need to work closely together to make DevOps a success inside of a company.

Some have focused in on the leadership role that developers have taken (and the associated burden that goes along with that role). For larger companies developers are often not the leads when it comes to DevOps. Developers are no doubt a key part of the equation and integral to practicing DevOps within the organization, but the Ops people seem to be calling the plays and leading the teams. This has led me to the question of who really leads the DevOps process within a company?

Why Ops?

As with any key initiative within a company, there needs to be a focal point, a champion or leader to advance it. DevOps is no different. So, then, how do companies organize themselves to become and execute a DevOps methodology? Recently, we’ve spoken with three senior IT leaders – all in ops – from companies that are beyond the initial startup stage, but aren’t yet behemoth enterprises either. All three ops leaders either had the title and role of VP of Operations or VP of DevOps (without trying to get into the “is DevOps a job title controversy). Their job is to deliver code to production as fast and safely as possible. This meant that they are directing, helping even conniving their counterparts in engineering to deliver smaller batches of code faster and on a regular basis. They then push this code out to production. In essence they were playing the role of the quarterback of the DevOps process.

Just as the quarterback directs the players, calling the play in the huddle and directing the offense, these Ops folks are quarterbacking the process of DevOps.

As such, these senior leaders were building the overall product delivery pipeline – from ideation all the way through deployment and feedback. While the ops team may not be involved in each part of the pipeline, these leaders were integral in building out the product delivery pipeline. For most organizations that product delivery pipeline is starting with sales and marketing – what needs are the business trying to solve in this particular iteration of the product or service. From there, the various teams are understanding the implications for their particular piece of the pie. Engineering is understanding how to build the features and functions. QA is understanding how to test the functionality as well as automate the QA process. Security is making sure that the functionality doesn’t create vulnerabilities or security holes. And, ops is thinking through how to push all of these features out to the customer. As in any DevOps organization, there are a lot of moving parts with many pieces needing cross functional coordination. And, therein lies the question – we all may practice DevOps and should all practice it; but, who leads the effort?

Clearly other senior leaders could take the lead on this activity. There is frankly no reason why the VP of Product or VP of Engineering could take the lead, but it has been an interesting observation that I have been seeing more and more ops leaders taking the reigns of DevOps at mid-sized organizations. We would really appreciate hearing who you have seen taking the leadership role in DevOps – in your organization or elsewhere. Let us know by commenting below!

June 4, 2014
DevOps and Enterprises, it’s a culture thing

Today author, DevOps evangelist and devopsy.kinsta.cloud Advisory Board member Gene Kim published an article in the CIO Journal of the Wall Street Journal. The article was a response to an earlier article by Rachel Shannon-Solomon in which the author claimed that DevOps was not likely to be successful in the enterprise at least not at the present time. Rajat Bhargava, CEO of JumpCloud and another Advisory Board member of devopsy.kinsta.cloud wrote the below in support of and amplifying Gene Kim’s thoughts in the WSJ article.

Recently an article was published in the WSJ CIO Journal that DevOps likely won’t work for enterprises – at least not now. The author goes on to state that there are structural impediments to enterprises adopting DevOps including siloed organizations, lack of platform technology, cultural issues, and lack of ROI. I completely agree with the author that enterprises face an uphill battle on the implementation of DevOps. She is right that they have different structural challenges than startups or SMBs.

DevOps bring companies close to customers

However, I have to flat out disagree with her conclusions and many of her data points that are extrapolated into the conclusion that DevOps won’t work in the enterprise. Let’s start at the beginning where the author states that DevOps is “…a software development method that stresses communication and collaboration between developers and IT professionals (QA and ops)”. I’m going to respectfully disagree with that definition. DevOps as a name is a misnomer. DevOps is about collaboration and communication, but not just between developers and ops in the enterprise. This is about communicating across the whole company: sales, marketing, finance, security, and any other relevant area. DevOps is about increasing company performance through better IT execution. The hypothesis is that by more closely aligning what cus with what gets built in a more timely fashion, organizations will sell more products and services. That’s not a small company or SMB issue, that’s an every company issue.

The author continues by saying “…the value of implementing devops beginning in the early days is clear—achieving radically higher productivity at lower costs resulting in more reliable systems.” I have to disagree here as well. DevOps is not about higher productivity at lower costs resulting in reliable systems. That may be a byproduct of DevOps, but the goal of DevOps is to more closely build what customers want in a shorter timeframe. DevOps is about cycling a product or service quicker to have it match more closely what the need is in the market. From there, if that holds, the organization will sell more products and services. DevOps organizations hope that it will be cheaper and more reliable but it very well may not be right away. In fact, DevOps may be more expensive if you factor in the time and energy required by the organization to run much faster product release cycles.

DevOps: a cultural shift

Unfortunately with those fundamental tenets at issue, the rest of the article becomes harder to agree with. The author states that there aren’t any real legacy vendors for DevOps which is a matter of perspective. DevOps isn’t about the vendors or tools ultimately. It is about culture. Every legacy vendor out there is going to adapt their tools to work with DevOps methodologies. Some will succeed. Some won’t. “Shiny” new startups will also compete for the business, but in the end, it’s not about the vendors or tools for enterprises. It’s going to be about cultural change.

The author is right that silos will make a difference in the pace of adoption. Enterprises are big, complex animals. Cultural change at these organizations is next to impossible. Add in departmental and divisional factions and it’s even harder. But cultural change has happened before and it will again. Will it be easy? No. Will it be fast? No. Will it be free? No. But it will happen. Or the company will fall behind, because their competitor will be able to move faster, innovate quicker, and introduce new products and services.

The next challenges described focus in on selling to enterprises and the lack of hard dollar return on investment. The first focuses in on vendors trying to sell to enterprises. The author states that enterprises are predisposed to buy. She’s correct that it’s hard to sell to enterprises, but many startups and larger companies have solved that problem successfully before. Historically, that hasn’t been an impediment to adoption of technologies. Does it take longer? Yes, it does, but if the technology is powerful and enterprises believe that it will help them, they will look for solutions to those problems. The lack of ROI is an interesting observation. As with many things in IT, ROI is elusive and it likely will be with DevOps early on as well. Over time, though, DevOps should be a clearly measurable activity. Product cycle time, defects, and ultimately revenue and profit will be measurable. Whether DevOps is contributing to the bottom line or not will be evident and clear – just maybe not in the short-term.

Fundamentally, DevOps is a different animal at a startup than at an enterprise. This is a topic that we have studied before. The WSJ perspective is an interesting one and can be taken in the right light as a cautionary tale. Enterprises do face different challenges when it comes to DevOps – or any cultural change for that matter. DevOps at an enterprise is a cultural challenge first and foremost. Those enterprises that embrace it as such and focus their efforts on that area first will likely have more success than those who treat it more as a tool issue. There are countless successful enterprises leveraging DevOps. Gene Kim’s response makes that clear. Gene, has been studying these “unicorns” and “horses” and so his response is based on the enterprise DevOps adoptions, successes and failures he has observed. I look forward to more data coming out soon that will shed more light on DevOps in the enterprise.

May 22, 2014
DevOps adoption – Enterprises (part 2 of 2)

A large enterprise has institutionalized an incredible number of processes. Their fundamental approach to IT has been built over years if not decades of experience. Their leaders may have grown up with these methodologies and approaches.

In this article I wanted to continue the thread I started in my last article about how different organizations adopt DevOps differently. The focus of this article will be on how enterprises adopt DevOps. I will also contrast it with how startups adopt DevOps, which was the focus of the first part of this article.

To recap the first part of the article, we discussed how startups are adopting DevOps. We stated that adoption occurs organically because of the nature of startups. Because they only start with a few people and they have little to no process, DevOps starts through the addition of tools. The founding developers or CTO will find tools to solve the practical problem of automating different aspects of their development, QA, or operations functions.

With so few resources, automation is critical. As a startup grows they add more people to the team and process becomes more critical. So, they extend Agile into other areas of their product development process. They become more intentional with developing a DevOps oriented team.

As the company continues to grow, DevOps is embedded into the culture. Hiring decisions are made based on whether a candidate understands how to be a part of a team practicing DevOps. The rest of the company starts to beat to the drum of rapid iteration, cross-functional integration, and increased automation. DevOps adoption in startups follows the tools, process, and then culture sequence.

Enterprises follow a different path, and so usually take a contrasting trail to DevOps adoption for a variety of reasons. A large enterprise has institutionalized an incredible number of processes. Their fundamental approach to IT has been built over years if not decades of experience. Their leaders may have grown up with these methodologies and approaches. Further, their entire business is beating to a certain IT drumbeat. Changing the beat throws everybody in the organization off.

There are also other major issues to DevOps adoption within an enterprise. Institutional resistance and inertia is a significant one. Change is always difficult to people. Some groups handle change better than others, but in any organization – especially a large enterprise – change needs to be orchestrated and worked effectively. Change in large organizations takes time and perhaps nothing takes longer than bringing people on board with a new approach. In addition to the personnel aspect, the organization has built strong, existing processes. Those have been over many years. Changing complex processes doesn’t happen overnight. Those processes have been built with internal and external customers in mind. Those customers expect their “solutions” delivered in a specific manner and timeframe. Accelerating that though DevOps may be great, but not if they aren’t on board and can accept the increased pace.

DevOps impacts the entire system of an organization. The system has been tuned to a certain speed and level of quality. Increasing the front end of the process will not necessarily increase overall throughput unless the rest of the organization can increase their speed in concert. In a startup, that system is generally small and contained. Most of the individuals, systems, and processes are contained and when the pace increases, the rest of the organization can react quickly or better yet the entire organization has been built with those expectations in place. Not so for a large enterprise.

The startup approach of adopting DevOps through adopting tools, changing processes, and embedding into the culture, will be easily rejected by a large enterprise. Even if the IT organization can deliver much faster and does, the rest of the organization won’t know what to do with the completely different approach. Perhaps the rest of the organization will even continue operating at their pace while IT operates at a different one. While not catastrophic, it is not helpful either. As larger organizations think about the best way to adopt DevOps understanding this perspective aids in adoption. Enterprises have a much harder uphill climb when it comes to adoption, however, the rewards can also be monumental.

An enterprise’s journey into DevOps – if it is to be successful across the entire organization – starts at the cultural level. And, as we all know, cultural change in a large organization is perhaps the hardest step. Clearly the positive here is that the organization adopting DevOps will have an opportunity to perform far more effectively. Cultural change starts with the key stakeholders at the senior level creating a commitment to adopt this new methodology. Then that commitment needs to be evangelized through the organization. A top down edict doesn’t work just as a grassroots movement without senior level support doesn’t. It should be reasonably straight forward to bring those two groups together to build a shared commitment to moving to DevOps (or not). Note that large organizations can adopt and test DevOps in one of their divisions or smaller groups to understand how different the approach will be from their previous business methodology.

Once a shared vision is reached, some of the detail work can begin. As discussed earlier, a different pace of product development needs to be matched by a different pace of distribution, sales, and marketing too. Updating and changing processes is the step that makes this happen. How will development processes change? And, how do those changes impact existing customers both internal and external? Can the same approach to quality be taken or do new processes in development require new processes in QA? Security? Operations? All of these downstream impacts will need to be matched, reviewed, and changed. Changing processes that are upwards of decades old is also fraught with risk. Many of those steps exist for a reason. Altering them may have consequences that aren’t known to all. This is yet another reason why DevOps adoption doesn’t happen overnight at large enterprises.

Finally, as processes are updated and the cultural shift is largely underway, the IT organization can turn its attention to finding better tools. These tools need to match the processes built and, of course, the cultural style of the organization. This is why large organizations would be better served by deeply understanding their newfound processes before diving into tool selection or even purchases. Only after they understand their needs is it worthwhile to embark on deep tool selection.

As we said in the last article as well, we’ve shown this to be a serial process, and, of course, it is never so cut and dry, especially, in a large organization. There will be overlap, there will be things that happen out of sequence, there even may be areas that jump far ahead, but largely, the adoption approach takes on the character of culture first, process second, and tools third. DevOps can be transformational for a large organization with some even going so far as to say that a DevOps culture is a predictor of stock performance. Understanding how DevOps takes shape at a large organization will help those in other large organizations as well as the vendors and consultants supporting their efforts.

April 21, 2014
DevOps Adoption – Startups

This is part 1 of a 2 part article

DevOps is somewhat unique in that it has applications to both large enterprises and to startups. The way DevOps is adopted in startups is incredibly different than how it happens in large organizations. The profile of how DevOps is adopted in each of these cases is the focus for this two part article series.

In this first installment, we will study how DevOps is adopted within startups and in the second, how enterprises adopt DevOps. There are lessons for each in the way that DevOps is adopted. Organizations can leverage this information to enable better, faster, and easier adoption of DevOps.

Much of the DevOps way is rooted in startup culture. From methodologies such as Agile, lean principles, and just in time manufacturing, startups adopted DevOps as a means to compete, innovate, and move faster. DevOps is inherently a quicker, better way to get product to market – a perfect fit for startups. They could adjust quickly, iterate again and again to get to the point where customers en masse were interested in their technology.

In order to understand how DevOps is adopted in startups, we need to look at how startups evolve. In the high-tech world, a startup gets going when a founder or group of founders has an epiphany for a new product or service. Generally one of the founders will start to develop the solution by writing some code. The startup may add a couple of developers but it is usually just a few folks cranking out the first version of the product. There is often a business or product focused function where requirements and feedback are gathered from potential users or customers.

In this first phase, the development team often adopts Agile as a way to quickly iterate on the development process. These short-term sprints build the initial functionality of the product. While initially, there isn’t really much to release to potential customers, the core developers are starting to build their development and release cadence. The short sprint help to keep the functionality requirements small and the team on-track to deliver a small piece of functionality in a short amount of time. This foundation also ends up being an important part of the future development process. And, just as importantly becomes the kernel of DevOps.

As a small team building the initial releases of the product, the culture and environment of the startup is generally fast, highly communicative with little to no process. Often the development team is sitting side-by-side cranking out code quickly. No specs, just a list of what needs to be done often stored in Pivotal Tracker or some similar tool. Sometimes mockups and screens designed to describe the functionality. There isn’t a rigorous process in the choice of technology platforms – language, database, etc. are chosen often based on what the core team knows or thinks is the right solution rather than on research. This small, agile environment is critical to understand when thinking about how DevOps is adopted.

There is little formal process at most startups. Things are evolving and moving so quickly that there just isn’t time. The goal, after all, is to get a product into market and start selling. While startups are high adopters of Agile as a development methodology, there is generally little rigor in its implementation. There doesn’t need to be because the team is so small and trying to move quickly. Adding in product management, sales, and marketing functions are often ad hoc. There isn’t a well-developed end-to-end product release process. The movement from development to operations is also less than well defined. Often, the developers building the product are the operations people deploying and managing the product or service. Process gives way to the practicality of just getting things done.

The first phases of DevOps adoption start to occur at this early stage in a company’s life. Developers as they are building their product often encounter some basic challenges and begin to solve these issues. How they solve them often dictates whether they take a DevOps path or not. One of the first things that developers do is build an environment to test their software. Often, they will find a cloud provider and start spinning up a few instances to host their application. They also may need a database server to round out their testing. They might manually do this a few times, but as they test more and potentially on different platforms they will start to look for some tools to automate this process. Of course, at this stage, there are likely zero operations people, so the tasks fall on the developers. And, developers wanting to spend more time coding pick-up some tools to do this work.

The tools might include infrastructure providers like AWS, Rackspace, or SoftLayer to configuration automation tools such as Chef or Puppet. There may be a variety of other toolsets chosen as well including project management tracking, libraries, language frameworks, and others. The goal with this first wave of tools is just to save time. That could be by not having to code basic functions, the setup of test environments, or manually tracking tasks. As the organization launches and brings their product or service to market, they may adopt some management and monitoring tools to help them gain visibility into their product or service. Generally at this early stage the decisions are focused on driving short-term problem solving. A developer needs a test environment quickly and repeatedly. A solution to that need is quickly found and implemented. The underlying paradigm is to move fast, solve problems, and get to market.

While tools are hardly a complete DevOps methodology, they are the catalyst for the next step in a startup’s journey to adopt DevOps. Many of the tools chosen in this first period have APIs, can create automation, and are highly configurable. As a startup keeps moving forward and increasing their users, the flexibility of these tools and the various ways that the developers utilize them starts to drive some conversation around process. A few hiccups in the production environment with some bugs, outages, or failures, and the technical leaders begin to start to say the ‘p’ word. Process. While there may be some Agile components already embedded, the growth of the startup starts to create a desire to add in some control, accountability, and guard rails around the product development process. Many organizations will also start to loop in the business side so that there is a more comprehensive process around releasing a product to market. Agile now starts to extend up and down the organization in a more rigorous way. DevOps is starting to take hold and become more fully formed.

The early stage DevOps lifecycle wraps up with it becoming a part of the culture. The organization now shifts into a mode where this methodology becomes a fundamental underpinning of the business. Values such as agility, intense customer focus, rapid iteration and others are a part of the fabric of the company and encompass the entire organization. Hiring decisions are made based on whether the candidates match the philosophy and culture of the company. DevOps is viewed as more than just a process that the company follows, but a competitive advantage in fulfilling its mission to its stakeholders. At this point, DevOps is firmly embedded into the organization and becomes its way of life.

Startups are unique in their approach to adoption of technology and methodologies because of their size, focus, and evolution. DevOps adoption, as we have seen out in the wild, seems to go through some natural stages. While the stages may overlap, it generally follows a tools, process, and then culture sequence. It could be said that some startups just start with the culture of DevOps, and that can be true for some, but many that we see are often so small at that stage that the culture is largely undefined. IT leaders who understand this sequence can become more intentional about the way that DevOps is adopted, perhaps accelerating it or even shaping it earlier in the company’s lifecycle. A little bit from now, our hypothesis is that startups won’t even start without DevOps embedded into their plan.

In part 2 of this series, we will examine the adoption of DevOps at larger companies.

March 27, 2014
Is DevOps a Title?

So, I’ve wanted to write on this topic for a while as I think it deserves a little attention! I’ve heard numerous times that you shouldn’t have DevOps in your title or that job reqs shouldn’t be “DevOps Engineers”. This came up again at the DevOps State of the Union event that we hosted in Boston recently. There were definitely some very vocal folks saying that it just didn’t make sense to look for a DevOps engineer or hire a DevOps engineer. These folks make the following points:

–          DevOps is a methodology not a job description – we don’t call our developers Agile engineers, they are just developers. They happen to follow the Agile methodology but they their job description isn’t to do Agile, it’s to create awesome products.

–          DevOps should permeate an organization – having a DevOps group sort of missed the point of DevOps. DevOps should be embedded into the fabric of the company, not an adjunct to the development process. If it isn’t embedded how can you actually practice DevOps?

–          DevOps does not equal tools – asking for a DevOps engineer with 5 years of experience in Chef or Puppet is sort of an oxymoron. Very few people have long-term experience with new, up-and-coming toolsets. So asking for that doesn’t make sense and again, it’s a methodology, not a toolset.

It was an interesting discussion and I’m curious to hear what other people think about this perspective. We see a number of job descriptions and cards that say DevOps engineer on them. I can completely understand why people would do that, but I also understand the criticism of that approach as well.

Before I sign-off, I’ll weigh in with my thoughts and hopefully we can add some discussion in the comments. As a proponent of DevOps, I’d like to see as many people adopt DevOps as possible. Strategically for businesses, I think it will make a significant difference in their operations. They will move faster, understand customers better, and be more efficient with the resources that they have. If it takes assigning a group or a title to bring DevOps into an organization, I think that they should do it. It’s a way to get started and work at embedding DevOps into their culture. I fully agree with the statements that DevOps isn’t a title or a tool, but I guess I’m practical about it too – some organizations will take different views on how to bring it into their company and have it take hold. In the end, I’m pro-DevOps, so if that’s how you need to make it happen, go for it! My guess is that over time, you’ll end up building DevOps into the fabric of the company and you won’t necessarily think of it as a job description or title, but just as a part of everybody’s job.

What are your thoughts – is DevOps a title? If so, why? If not, why not? I’m looking forward to hearing your thoughts in the comments!

March 20, 2014