devopsy.kinsta.cloud

Category: Features

DevOps and Enterprises, it’s a culture thing

Today author, DevOps evangelist and devopsy.kinsta.cloud Advisory Board member Gene Kim published an article in the CIO Journal of the Wall Street Journal. The article was a response to an earlier article by Rachel Shannon-Solomon in which the author claimed that DevOps was not likely to be successful in the enterprise at least not at the present time. Rajat Bhargava, CEO of JumpCloud and another Advisory Board member of devopsy.kinsta.cloud wrote the below in support of and amplifying Gene Kim’s thoughts in the WSJ article.

Recently an article was published in the WSJ CIO Journal that DevOps likely won’t work for enterprises – at least not now. The author goes on to state that there are structural impediments to enterprises adopting DevOps including siloed organizations, lack of platform technology, cultural issues, and lack of ROI. I completely agree with the author that enterprises face an uphill battle on the implementation of DevOps. She is right that they have different structural challenges than startups or SMBs.

DevOps bring companies close to customers

However, I have to flat out disagree with her conclusions and many of her data points that are extrapolated into the conclusion that DevOps won’t work in the enterprise. Let’s start at the beginning where the author states that DevOps is “…a software development method that stresses communication and collaboration between developers and IT professionals (QA and ops)”. I’m going to respectfully disagree with that definition. DevOps as a name is a misnomer. DevOps is about collaboration and communication, but not just between developers and ops in the enterprise. This is about communicating across the whole company: sales, marketing, finance, security, and any other relevant area. DevOps is about increasing company performance through better IT execution. The hypothesis is that by more closely aligning what cus with what gets built in a more timely fashion, organizations will sell more products and services. That’s not a small company or SMB issue, that’s an every company issue.

The author continues by saying “…the value of implementing devops beginning in the early days is clear—achieving radically higher productivity at lower costs resulting in more reliable systems.” I have to disagree here as well. DevOps is not about higher productivity at lower costs resulting in reliable systems. That may be a byproduct of DevOps, but the goal of DevOps is to more closely build what customers want in a shorter timeframe. DevOps is about cycling a product or service quicker to have it match more closely what the need is in the market. From there, if that holds, the organization will sell more products and services. DevOps organizations hope that it will be cheaper and more reliable but it very well may not be right away. In fact, DevOps may be more expensive if you factor in the time and energy required by the organization to run much faster product release cycles.

DevOps: a cultural shift

Unfortunately with those fundamental tenets at issue, the rest of the article becomes harder to agree with. The author states that there aren’t any real legacy vendors for DevOps which is a matter of perspective. DevOps isn’t about the vendors or tools ultimately. It is about culture. Every legacy vendor out there is going to adapt their tools to work with DevOps methodologies. Some will succeed. Some won’t. “Shiny” new startups will also compete for the business, but in the end, it’s not about the vendors or tools for enterprises. It’s going to be about cultural change.

The author is right that silos will make a difference in the pace of adoption. Enterprises are big, complex animals. Cultural change at these organizations is next to impossible. Add in departmental and divisional factions and it’s even harder. But cultural change has happened before and it will again. Will it be easy? No. Will it be fast? No. Will it be free? No. But it will happen. Or the company will fall behind, because their competitor will be able to move faster, innovate quicker, and introduce new products and services.

The next challenges described focus in on selling to enterprises and the lack of hard dollar return on investment. The first focuses in on vendors trying to sell to enterprises. The author states that enterprises are predisposed to buy. She’s correct that it’s hard to sell to enterprises, but many startups and larger companies have solved that problem successfully before. Historically, that hasn’t been an impediment to adoption of technologies. Does it take longer? Yes, it does, but if the technology is powerful and enterprises believe that it will help them, they will look for solutions to those problems. The lack of ROI is an interesting observation. As with many things in IT, ROI is elusive and it likely will be with DevOps early on as well. Over time, though, DevOps should be a clearly measurable activity. Product cycle time, defects, and ultimately revenue and profit will be measurable. Whether DevOps is contributing to the bottom line or not will be evident and clear – just maybe not in the short-term.

Fundamentally, DevOps is a different animal at a startup than at an enterprise. This is a topic that we have studied before. The WSJ perspective is an interesting one and can be taken in the right light as a cautionary tale. Enterprises do face different challenges when it comes to DevOps – or any cultural change for that matter. DevOps at an enterprise is a cultural challenge first and foremost. Those enterprises that embrace it as such and focus their efforts on that area first will likely have more success than those who treat it more as a tool issue. There are countless successful enterprises leveraging DevOps. Gene Kim’s response makes that clear. Gene, has been studying these “unicorns” and “horses” and so his response is based on the enterprise DevOps adoptions, successes and failures he has observed. I look forward to more data coming out soon that will shed more light on DevOps in the enterprise.

May 22, 2014
Simplify and expedite server management

Deploying, managing, and maintaining servers can be tedious. Deploying, managing, and maintaining thousands, or tens of thousands of server instances is overwhelming—bordering on impossible—without the right tools. Thankfully, the right tools exist.

The Pain of Doing Things the Hard Way

I know I am dating myself, but when I was an IT admin for a dot.com startup, I had to deploy servers the old-fashioned way. UPS would show up with a box from Dell. I’d unpack it, and mount it on the rack in the datacenter. Then, I’d begin the process of configuring the operating system, and installing the application software. Each server was essentially a full-day project, and we seemed to constantly be adding new servers, so my full-time role was basically “racking and stacking” for a while.

Adam Jacob was experiencing similar obstacles when he developed Chef. Working as an IT infrastructure consultant, Jacob was frustrated by the configuration tools that were available, and the lack of a simple, repeatable process. Rather than starting from scratch and reinventing the wheel for each client, he developed Chef to enable IT admins to create modular, reusable “recipes” that automate the process.

Server management doesn’t end with deployment and initial configuration, though. You also have to maintain the servers, and resolve issues that might occur. When an issue arises, it could mean hours of troubleshooting to identify the root cause, and another few hours to work out and implement a resolution.

I used to get into philosophical debates with management and other IT personnel because it was my opinion that we should just skip the troubleshooting and rebuild the server from scratch. My theory was that it might be nice to know what caused the problem, but from an operational perspective I knew I could rebuild the server and restore normal operations in a couple hours. Troubleshooting, on the other hand, was an unknown—both in terms of whether or not it would even be successful, as well as how long it would take.

The Beauty of Doing Things the Easy Way

Tools that automate configuration and deployment of server instances automatically solve both of these problems. They expedite deployment and configuration, and simplify management and maintenance—freeing IT admins and IT personnel to focus on more important, less mundane tasks.

Tools like Chef help IT manage complexity. It is great from a DevOps, automation perspective to have a tool that enables you to quickly deploy and configure servers, but the real value comes from the ability to maintain a stable, dependable network with minimal effort.

Chef nodes “phone home” periodically to ensure the configuration matches the recipe it has been assigned from the Chef server. Any deviations are automatically overwritten—so any changes that may have been made locally on the node are erased, and the server remains configured as it should be.

In order to implement lasting configuration updates, the changes have to be made to the recipe at the Chef server level. The bonus is that once the recipe is updated, all Chef nodes working from that recipe will be automatically updated.

When an issue occurs on a server instance, it should ostensibly be corrected automatically when the node communicates with the Chef server and overwrites the configuration. However, even if that doesn’t happen for some reason, the use of DevOps tools like Chef still makes the task of troubleshooting and/or resolving server issues much more efficient.

With the push of a button, you can just deploy a new server—properly configured and ready to go. You could choose to kill the server experiencing issues and just replace it with the new one, or you could launch a new server to take over the roles and responsibilities of the affected server while you take the time to troubleshoot if you really need to understand what the root cause of the problem was.

Embrace simplicity

Apparently, I was an IT admin at the wrong time, and I got out of that career path right when things started to get interesting. Necessity being the mother of invention and all that, Adam Jacob created a platform with Chef that reduces the complexity and tedium of deploying and managing servers.

Chef isn’t alone. Tools like this form the foundation of DevOps because a culture that combines development and operations, and iterates new releases on a rapid scale needs to have this sort of automated IT infrastructure to keep up.

May 21, 2014
DevOps Best Practices: Break Down Silos, Avoid Drift and Optimize for Flow
In organizations trying to achieve agility with stability and quality, silos (of teams, tools, and processes) create major stumbling blocks to managing configuration drift between pre-production and production systems. Breaking down those silos, improving visibility into configuration drift and facilitating collaboration for improved flow are the keys to overcoming this challenge.

The challenge of staying in sync when both sides are moving

Production environments aren’t static. Whether you are applying patches for improved security, optimizing settings for availability and performance, or making changes for a technology refresh, production systems are continuously evolving. As new releases are being developed, tested, and staged, pre-production environments are continuously evolving too. Ideally you want to develop with production in mind, but that gets harder and harder when both sides are changing and different teams are responsible for different environments and live in their “own worlds.” (e.g. the common split between QA teams, release teams and production operations).

Even within pre-production, staying in sync is a significant challenge. For example, because applications ride on middleware and middleware settings are unique to each application, keeping systems in sync from QA to staging can be a real challenge.

QA and staging environments are regularly reconfigured to support multiple releases that are in process at the same time. The costly way, often employed to minimize changes, is to “protect” QA environments for a certain application or release leading to low utilization and bottlenecks. The rapid proliferation of mobile applications to front end legacy systems magnifies the challenge by adding another dependency layer.

Disconnected teams, information, and perspectives (those pesky silos)

Why are these challenges so hard to overcome if we know the issues? Many IT teams operate in their “own worlds” with tailor-made but disconnected tools and information resources. Silos have evolved because they permit the specialization required to deal with the immense, growing complexity IT deals with everyday, but they have now become the biggest impediment to agility and quality.

For example, the operations team has access to systems management alerts that most others don’t, engineers have access to build systems and scripts that others don’t, developers have access to functional requirements and stories that others don’t, and change approvals are stored in a system of record that only certain people (often managers) have access to – just to name a few silos.

Coordinating work and sharing information across these silos of teams often equates to lots of meetings, conference calls, and email threads. This is not the most efficient way to get work done and it’s not particularly effective in sharing information to keep environments in sync.

“Didn’t you see my email from yesterday with the attachment listing all the middleware settings required for today’s test?” “Weren’t you in the meeting three weeks ago when I said a Unix server was probably going to be needed in addition to a Windows machine?” “I told so and so in an instant message on Friday to run this script before he went out on PTO, but he must have forgotten. Who else has root permission to run it while he’s out?”

Sadly, these kinds of statements are the norm, not the exception, in most IT shops.

The reality is there is inefficient coordination to manage all the steps and handoffs between application owners, development, QA, release and production functions. Instead, I recommend thinking about the software development and deployment lifecycle as an “uber change” and “uber collaboration” process that must be viewed and managed from end to end; from business requirements, to development, to QA to staging to production. All of the different silos of teams must have a way to work together in a unified way with access to critical information from upstream and downstream teams to ensure a smooth lifecycle end to end.

Automation ≠ DevOps

The heart of DevOps is collaboration and managing change, yet the conversation often seems to start with automation. The fallacy here is the belief that we can eliminate the need to collaborate and manage change by removing people from the process and relying on the machine. This out of balance perspective between the machine and people creates more problems than it solves.

Automation tools, while important and necessary, are not all that’s needed to truly solve these synchronization and deployment issues. In addition, to have environments only changed by automation tools and never by people during the firefighting of incidents is unrealistic. You need your people armed with intelligence and tribal knowledge to drive your automations, but all too often we rely on individuals with disconnected perspectives and limited awareness so things get missed or overlooked.

Automation does what it’s supposed to do, but it makes a critical assumption; what’s being changed is configured how it’s supposed to be configured. If drift has occurred because critical settings were modified to remediate an incident or compliance issue, without the automation engineer aware of the change, the automated deployment will result in you hearing a loud “boom!”

As one popular DevOps Twitter handle has said, “To make a configuration error is human – to do it across a thousand servers is DevOps.” While that may seem like an extreme statement designed for comedic effect, it isn’t extreme to say that thousands of deployments go south every day because systems aren’t configured the way we expect them to be given that the complex software stacks that multi-tiered applications now depend on are touched in so many ways. Automation alone doesn’t solve the DevOps challenge.

Negative consequences to business

Of course, the issues I’m raising here aren’t just IT issues; they are real business issues. Time to market is typically the number one impacted area. For example, if a two-week development sprint is used to get a business critical need addressed, but then it takes weeks and weeks to get the sprint through the quality cycle and into production due to resource constraints and configuration issues, what have we achieved?

Stability and quality are the next big area of impact. When change is rapidly introduced through new releases and unexpected drift occurs in environments, performance and availability impacts occur. It is one thing for IT to be perceived as slow and unresponsive to the business, but it is quite another when they are perceived as slow and quality is poor resulting in business impact.

Just when IT thinks it can’t get any worse, the business asks a presumably simple question, “What happened?” IT’s typical response, “Let me get back to you on that.” And then begins a scramble of hours, days or weeks as IT’s best people desperately try to figure out what went wrong. This is obviously not the best confidence builder for the business.

Compliance is typically the other casualty as there is often very little awareness of operational policies on the development side of the house so deployment packages and instructions aren’t sensitive to compliance issues. Building and testing an application without full awareness of policy governance across the whole, multi-tier stack is bound to introduce delays and errors when attempting to deploy in production.

Visibility, configuration transparency and collaboration; a better way!
1. Understand drift for the entire application service. Rather than everyone having their own disjointed perspectives on databases, app middleware, network or firewall, and server configurations, provide team members with an understanding from the business service (or application) perspective with direct visibility into the differences between how things should be configured and how they are actually configured for the entire multi-tier application stack. This enables you to create smarter automations and initiate change processes to bring configurations back in line when necessary rather than waiting for a failed deployment to occur, identifying the root cause, and correcting it while the business waits.
2. Optimize for flow not individual processes. As stated in the previous example, making one process faster (e.g. development) without considering its impact on QA, staging and production will generate new constraints downstream. Instead, view DevOps as one continuous process from goals and requirements all the way through to deployment into production and define success for the team in those terms. When you optimize for flow instead of silos, you will be amazed at how teams share more and collaborate better.
The benefits of breaking down silos through better visibility, understanding drift and collaboration are significant:
- faster time to market;
- quicker ability to react to competitive pressures;
- deployment with fewer errors;
- better compliance with policies; and
- greater productivity for everyone.
To sum it up, if you break down silos, identify and resolve drift, and manage for flow (while avoiding islands of automation, information, and people) you can unlock agility, maintain stability and ensure the quality your business demands.
May 16, 2014
How to turn executives into DevOps champions

IT practitioners must bridge the gap of understanding about DevOps with their managers and executives if they really want continuous delivery to take root in their organizations. Next week at Camp DevOps @ Gluecon, Andi Mann, vice president in the office of the CTO for CA Technologies, will propose ways to do that in his talk, Building Top-Down Support for DevOps.

“You can only push so far from below until you need to start getting support from above,” Mann says, explaining that the practitioner audience that tends to dominate events like Gluecon are running into a ceiling with DevOps adoption because the organization-wide changes necessary require top-level support from managers and executives to carry them out. He believes passionate advocates of DevOps need to do a better job garnering that top-level support.

From practitioners and even middle managers, that will require some mind shifts and improved ways of approaching people further up the food chain. He gave devopsy.kinsta.cloud a quick preview of some of the suggestions for practitioners he’ll lay out in the talk he’s developed to help them start improving their DevOps executive pitches.

Understand management’s goals

Mann: “To start with, they need to understand their manager’s goals and even their manager’s manager’s goals. Employees often don’t think about that, they have tunnel vision and only consider their own goals or sometimes their team’s goals. Practitioners often fall into the trap of looking at something and saying ‘That’s obviously good for the business’ without really knowing if that’s the case or not.”

Think about politics

“If you’re looking to go into this major cultural change which DevOps must affect, then it needs to be in line with the organization’s goals. When you think about accelerating delivery of new capabilities, the question might be whether the rest of the organization is ready for that. So if you’re creating a new feature in a front office application, but you’ve got to train 15,000 front office staff in that application, is that something you want to be doing. You might think it’s a great thing, but your manager may say, ‘Hang on, this is just causing work for other people.’ You’ve got to understand where the boundaries are politically within an organization and figure out how to break those down.”

Bring fact-based results projections to the table

“How will you make your boss a hero? That’s glib, but you need to think of it in those terms, particularly when it comes to ROI and the business case for DevOps. Start by proving other people’s successes first to your managers and then show how your success will be achieved second. There’s survey data out there, there’s gartner data, and there’s data from vendors on what the outcome of DevOps are. It’s not enough to say to a manager, ‘I’ve got this idea, it’s a philosophy and amovement and a cultural change and it’s going to make our company better.’ Not many managers are going to spend a lot of time listening to that proposal. You need to come in with the ROI and how DevOps addresses their business challenges.”

Understand and communicate costs

Mann: “Not enough practitioners actually put together business cases for their managers to take to their executives to support the change in culture—and a big part of that is budget. Think about what training you need, whether there’s going to be additional hires to fill out skill sets. There’s cost of software—the DevOps tool chain is rapidly maturing and not all of it is free. Even the open source stuff costs money. If you want to get the most out of it you’re going to be looking at support contracts, services and training. So there’s cost involved and that is an important part of making that business case.”

As Mann will explain in his talk, these considerations need to be the building blocks of a complete and polished proposal. Bringing executives that “finished work” will more likely sway them than a few stray conversations made up of hints and gut feelings about DevOps benefits.

“Go to your boss with a finished idea. Explain ‘here’s what we need to do, this is the spend, this is the result we’re going to get in terms of dollar returns and savings, speed to market and so on,’” he says. “And then explain, ‘Here’s a project plan for how we’re going to do it,’ preferably with a paced, layered approach proposed rather than a big bang.”

May 15, 2014
DevOps: Getting Past Audit

New business processes (and technologies) too often throw auditors into a tailspin. It’s no one’s fault, really: the decision making on what IT and process controls that need to be in place moves slow – and it is the tail to the rapidly moving technology dog. In fact, while there is no way a standard or regulatory body can predict technological change from year to year, there are ways to help ease the relationship between auditors and organizations embracing DevOps and continuous deployment practices.

“DevOps audit is an interesting problem,” says David Mortman, distinguished engineer at Dell (formerly Entratius). “Audit is something you just can’t avoid, especially in any industry that’s regulatory controlled. And auditors are a lot like security people, in my experience, which is to say that they have a lot of responsibility and they are justifiably concerned about things they don’t understand,” Mortman says.

And concern their certainly is when it comes to any organization that must deal with acronyms like HIPAA, PCI DSS, SSAE 16, SOX, FedRAMP, among others means few things will raise auditors’ eyebrows more than hearing that developers are performing deploys on their own code.

That’s why it’s important to know how to manage these audits, and continuously improve the audit process.

Rapid technological change, glacial control placement
One of the first things to note is that these standards and management programs move slow. “If you are familiar with ISO 27001, which is a security management program, they just released a new version at the end of September [2013]. From 2005 to now, the standard hadn’t changed,” said James J. DeLuccia, a senior manager in the advisory services practice of Ernst & Young LLP in his Flowcon presentation, Successfully Establishing and Representing DevOps in an Audit. But technology certainly has changed in that time: increased use of virtualization, the rise of cloud computing, ubiquitous mobile, as well as changes in development and operational processes such as continuous deployment, continuous integration, and DevOps.

One of the biggest issues that come up with disciplines such as DevOps and continuous deployment is separation, or segregation, of duties – that development, testing and operations should be separated duties. Not only do many government and industry mandates require it, so does many enterprise security policies. That’s why DevOps, when not explained or approached properly, can create bug-eyed expressions in unfamiliar auditors.

The goal of separation of duties comes down, essentially, to proper change management. If the developer who wrote the code can deploy the code into production, and they are given too much access (such as root) or shared access, there’s a lost control check, and potentially no log of the event. But it’s a concern that is overstated, experts say.

“In a proper DevOps environment, the person deploying the code doesn’t need to have root access. There’s no reason why, when properly setup, developers can’t logon, push the deploy button, and create an auditable log that says at 2:32 on Friday the 18th, David deployed a version of the code. And you have it down to the hundredth of second who did what,” says Mortman.

As DeLuccia explained in his talk, when auditors are asking to see change control logs they are really trying to see that these controls are in place. “They’re really asking for help to understand this linkage,” he said. Depending on the policy and requirements, the auditor needs to see reports that demonstrate, in a statistically relevant way, that everything in IT is working the way it needs to and is said to be working. “Because remember, when we’re putting together these reports, we’re trying to say, everything is working the way you say it’s working,” he said.

Auditing DevOps: An education process
When done right, Mortman says DevOps and continuous deployments create more accurate logging and change management. “You have a log of every change that’s ever happened and it’s all generated automatically. You don’t have to worry about someone changing something. When setup properly, they don’t have the access to change anything other than the very limited roles of what that application they can deploy to,” Mortman says. “So in a lot of ways, in fact you’re actually maintaining better control over your environment.

The challenge there, as DeLuccia explained, is that audit programs are built a year in advance. “When they come to see you, they’re already have it planned out based off last year’s risk. That’s already too late [to keep up with rapidly changing technology and processes]. You really want to plan this a year in advance,” he said. The goal, he explained is to become part of the process and show how the procedures they are asking the IT team to execute against are in line with reality of the processes and the technology on the ground.

“Much like any other industry, you have an education problem,” says Mortman. “Security people run into this all the time. The business will say, “Hey, we want to do this new thing, and the security people reply, “Whoa, hold on a second. You can’t do that. That’s not secure. Or, we can’t audit that.” So a lot of the challenge with DevOps is an educational challenge. And you need to educate auditors that the control processes they are looking for are in place and that there are adequate logs that can be audited and evaluated,” says Mortman.

May 14, 2014
DevOps and faster feedback: fewer problems, better features (part 2)

As the experts established in part one of this series , the tightening of feedback loops through DevOps gives developers a better chance to learn how well their coding choices perform in the context of production environments. This makes for higher quality code in the long-run and constantly improved skillsets among development teams.

But fast feedback loops don’t just improve the line-by-line coding work the development team puts out. It can also push through a huge boost in the value of the features they’re producing for end users.

“Feedback loops are important not just for technical feedback or feedback about things not working,” says Aater Suleman, CEO of Flux7 Labs and a professor in computer systems design and architecture at the University of Texas. “There’s also the feedback around what customers are looking form in terms of particular features and whether or not those features are doing well in the market or not.”

The tighter the feedback loop is about feature sets—and the more quickly the team can release responses to that business-related feedback–the faster the pivot time to account for market conditions. The quicker assessment of how well features are doing gives development teams and the business units they answer to a more accurate barometric reading to decide where to direct programming time. As a result, it isn’t just the quality of work that goes up, but also the business value.

“With a DevOps flow that allows developers to start seeing feedback from end customers right away, it helps them be more agile on a business level so they can be more strategic about where to spend their actual time,” Suleman says, explaining that this exposes developers to a big-picture view of their work. “You may have the most productive development environment in the world, but if a feature developers have worked on for two months is not going to be liked by the end users, it doesn’t matter how productive the developers were during that time period.”

Not only that, but it eliminates the hamster-wheel of playing catch-up with competitors’ features.
“I know of many companies which still hold to quarterly release cycles. Did you competitor just release an awesome new feature? Have fun waiting three months to catch up,” says Curtis Poe, a consultant and Agile development coach for All Around The World. ” In today’s competitive world and fast-paced business environments, do you really want to playing catch up all the time? Switch to continuous delivery and never worry about a lengthy release process blocking you from offering what your competitors offer.”
Poe says that the best DevOps teams can get the most differentiation from the crowd by understanding that they’re not just monitoring technology and code, they’re monitoring people. Namely, end users.

“Do you sell shoes online and you have a typical conversion rate of 3.2 percent? What happens if you make a release and the conversion rate drops to 2.5 percent?” he says. “Assuming that nothing technological is going wrong , monitoring business value, when appropriate, can instantly tell your designers, ‘Hey, moving that button from the left to the right side impacted our sales.’”

Additionally, that kind of business monitoring makes it more easy to understand the dependencies between technical and business functional aspects of software updates. Then the team can more easily explain to business leaders what technical obstacles stand in the way of functional improvements, and how to overcome them.

“You always want to deliver the best experience for your customers, and you want to iterate on both the product and the non-functional aspects as often and quickly as possible,” says Arnold Goldberg, vice president of customer engineering for PayPal. “Integrating new features might be the wrong thing to do if, for example, there are challenges with performance, stability or availability.”

May 12, 2014
Automating code helps businesses adapt to customer demand quickly

This is the first article in a series of pieces on digital innovation, accelerating change and the rise of the coded business from innovators in the Chef (click here) community.

How does the need for speed and scale affect businesses today?

Let’s start with scale. As a technology company grows, its infrastructure grows – more machines (or machine images), more networking, more data storage needs, more code, more developers, and more users. The only way to scale with growth is with automation. And, as more and more of the world’s interactions happen online (be it commercial transactions, email, social networking, video streaming, etc.), that scale of infrastructure is breaking records all the time. With these new levels of scale come ever-increasing automation requirements.

On the speed side, first-to-market has always been an important part of business – but what that means has evolved. It’s no longer simply about producing that new cool item before the other guys. Now it also means you must be able to iterate quickly, adjust quickly for changing user tastes, and adapt quickly to the ever-increasing pace of new technologies. To be able to do this effectively, instead of just developing fast, you need to be able to deploy quickly, react quickly, troubleshoot quickly, and update quickly. For many companies, this is no longer icing on the cake, but instead, the structure underpinning their business – in the same way that having an e-commerce site was 10 years ago.

Speed is becoming ever more critical for business in the digital age. With the rise of social media, a lot of customer feedback is public or semi-public, allowing people to unite behind specific complaints or desires and provide more clear and unified messages to companies. This helps companies get more feedback of higher quality and – hopefully – to react to it more quickly. This is also dramatically accelerating the cycle of getting features/products/releases to users, and, perhaps more significantly, prompting users to provide more useful feedback faster. As I mentioned, the scale and speed of business operations in every industry are growing daily, and, because of that, automation has gone from being beneficial to critical. In the technology world, this automation comes in the form of code.

Pushing your code faster and faster means you have to be able to update and deploy infrastructure faster, which in turn means infrastructure/systems teams must also be able to deploy, react, troubleshoot and fix faster.

But code, in and of itself, isn’t necessarily sufficient to meet the challenge of escalating speed and scale. How quickly you deliver that code is what’s important. Continuous Delivery is one technique used to address this need.

Continuous Delivery (CD)

People talk a lot about CD and it’s important to remember that this phrase refers to a methodology that has pros and cons.

For most technology companies, especially those whose product is primarily Web-based, CD is often the best choice to enable speed of delivery. The main benefit is that the closer to “continuous” your release-cycle gets, the easier it is to have an iterative process. Faster feedback from users means you can more rapidly deliver updates based on that feedback. As you approach CD, you approach near-real-time feedback, which allows you to hone your product or feature far faster than if it were sitting in long dev/qa cycles. Another benefit is best described with a comparison. In troubleshooting, a key rule is make one change at a time, so you know which change fixed the problem – or made it worse. CD is an extension of this idea, although it comes earlier in the lifecycle. If you are pushing/delivering a codebase more often, there are less changes per push, and so breakages are easier to track down – and it’s easier to push fixes.

A common misconception is that an organization either achieves CD or it does not. CD is a spectrum with frequent releases on one end and less frequent on the other. A CD solution for a bank may mean something very different than a CD solution for a gaming Web site. The idea of the CD methodology is continuing to push toward faster deployments, in order to reap the benefits that come from doing so.

In the end, no matter what part of the CD spectrum a company aims to achieve, they must be willing to adapt and change.

From an internal IT perspective, this means realizing that this isn’t the old world, where businesses lock users into small boxes to make things simpler. We’re in a world where everyone has the power of the Internet in his or her pocket and there’s enough information out there for anyone to break out of his or her box. Businesses need to provide engineers, designers, assistants, VPs, etc. the flexibility and freedom to work and think in ways they do best, so they can take the business to the next level.

From an external (production) IT perspective, companies must be able to quickly adapt to new growth and new technology. It took the Internet a (relatively) long time to start adapting to the small displays on mobile devices, and it adapted only slightly faster to tablets. But new form-factors are just the beginning. Broadband is becoming ubiquitous in first-world countries, and companies are just starting to rely on that fact. Yet there are still many users the Web isn’t taking into account very well. Mobile is becoming the default interface for a variety of locales and ages, and many countries have limited bandwidth and/or high latency. Further, it seems likely that we will see drastically different user input mechanisms, as well as display mechanisms, in the near future.

Technologists need to be agile and readily adapt to technology and infrastructure changes, as well as speed and approach changes, such as CD, which will be required for the next generation of user and technology demands.

About the author

Phil Dibowitz has been working in systems engineering for 12 years and is currently a production engineer at Facebook. Initially, he worked on the traffic infrastructure team, automating load balancer configuration management, as well as designing and building the production IPv6 infrastructure. He now leads the team responsible for rebuilding the configuration management system from the ground up. Prior to Facebook, he worked at Google, where he managed the large Gmail environment, and at Ticketmaster, where he co-authored and open sourced a configuration management tool called Spine. He also contributes to, and maintains, various open source projects and has spoken at conferences and LUG’s on a variety of topics from Path MTU Discovery to X509.

May 8, 2014
Automation, Operations, DevOps – What We Thought Was Missing

Stackstorm is a new company focused on automation, operations and DevOps. It’s co-founder and CEO Evan Powell has been keeping a running journal as the company made ready to emerge from stealth. Over the next few days we will share Powell’s thoughts and experiences as a new DevOps company comes to market.

Over the next few posts I’ll be sharing my thoughts on why StackStorm – a new company focused on DevOps based automation – was still needed. My first comment is that the users will decide how we overlap or complement existing automation solutions. So far they are confirming our belief – that we are strongly complementary to trail blazers like Chef, Puppet and of course CF engine and more recently Salt. Our approach is to focus on operations automation that wires together the data center and that – in the case of systems like Facebook’s FBAR and our own beta level software – learns over time. With that as some context, over the next five days I’ll unpack our key drivers for us starting StackStorm – starting with something I think all readers of devopsy.kinsta.cloud should agree with:

DevOps and the shift to the cloud is much bigger than generally assumed. Over the last few years, my co-founder Dmitri and I have met with thousands of enterprises, service providers, government agencies, integrators and sundry analysts and experts. And I’ve been speaking about the inevitability of the shift to Software Defined Data Centers for many years while helping to create the software defined storage market as founding CEO of Nexenta Systems. More recently, as StackStormers, Dmitri and I have met with approximately 100 DevOps practitioners with OpenStack projects either ongoing or kicking off. The shift to DevOps – sometimes lumped in with the shift to Software Defined Data Centers and also the shift to the cloud – is the biggest shift in IT we’ve seen in our careers. The top DevOps practitioners are 10-100x more productive than your legacy siloed IT operations department. Software is “eating everything” AND the approach to building and operating software has improved radically. Nearly every company and government in the world must either leverage this step function improvement in productivity of software development and operations or face disruption. It might be worth discussing the 10-100x productivity improvement:

a. Is it real? Organizations like PayPal and eBay have published their own data showing at least a 7x improvement in development velocity. And of course the DevOps surveys by folks like Puppet Labs have shown even higher improvements in the speed and quality of development. And, again, our own experience indicates it is very real. One operator tells us that they went from building and deploying 1 new service a year before DevOps to 54 in the 12 months after they got up to speed with a DevOps approach. Revolutionary.

b. Is 10x a lot? Software getting built and operated 10x better is changing the world. A comparison that Dmitri made in his talk at SCALE12x this past February was to the tractor. As I think any casual student of the 20th century knows, the tractor changed the world by enabling urbanization and more. According to most economic historians, the tractor was about 4x more productive as measured by crops produced vs. input in human and machine energy. IT fairly suddenly figuring out how to develop and operate software 10-100x more productively compares favorably with the most important improvements in human history.

So – GREAT. The world is changing. What do we and other DevOps solution providers have to do with this? Ask yourself – why don’t we ALL just jump on the DevOps bandwagon and start building and operating software in this new way? I’ll take on that question tomorrow as the second driver that led us to start StackStorm.

Feedback welcome. Comment below please.

May 7, 2014
DevOps and faster feedback: fewer problems, better features (part 1)

As organizations begin to institute DevOps patterns and continuous delivery into their IT shops, one of the most dramatic long-term returns on their investment is that it bolsters the class and consistency of work produced by existing development teams.

Organizations that evolve to DevOps don’t necessarily need to hire rock star programmers to boost code and feature quality. That’s because, in the long-run, continuous delivery actually fosters an environment that makes it possible for the developers already in place to learn more efficiently and better improve the day-to-day quality of their work. The secret ingredient to this? Faster feedback loops.

“Shorter time to feedback is the key difference between DevOps and the traditional waterfall approach,” says Aater Suleman, CEO of Flux7 Labs and a professor in computer systems design and architecture at the University of Texas. “The main challenge with waterfall was the fact that there was a very long loop for feedback, which removed the agility from the picture. It could be a month or longer before developers would actually get feedback about what they’d done.”

The longer the cycle, the more code to sift through to find problems and the longer it takes to fix issues, says Curtis Poe, a consultant and Agile development coach for All Around The World. Shorter feedback cycles make it easier to find issues quickly and attend to them as soon as possible.

“Having continuous delivery means that typically very little code needs to be reviewed for failure and as a developer, if I get a bug report for code I wrote in the last hour, I’m going to be far more efficient in fixing in than for a bug report for code I wrote last month,” Poe says.

This is true not just for out-and-out failure, but also tweaking for improved performance.

“When frequent deployments are coupled with well-instrumented monitoring metrics on all aspects of your application, small nuances in performance can be detected and teased out for relatively small amounts of code change,” says Jeff Behl, chief network architect at Logic Monitor. “Developers want quick feedback to know changing this particular small part of code can affect this aspect of the application. The sooner you learn how to optimize the application code, the better.”

Even more important, fast feedback loops are not only critical for fixing past problems, but making sure they don’t happen again in the future. The shorter the time between when a developer writes code and receives information about how well it did out in the real world, the easier it is for them to learn from those lessons and apply them to future work.

With longer release cycles, be they quarterly or monthly, there’s often a lot of energy expended trying to find the person who created an offending line of code that broke something, says BLANK. The top triage person will go through the ringer to figure out who broke the code, until they might pinpoint something like, this person made this code commit six months ago that broke the application.

“Does anyone remember what they were doing six weeks ago? Of course not. But if you were writing code at ten o’ clock and I came by at one o’ clock and told you exactly what was broken and I sent you a zip file that had everything that you needed to do to reproduce that on your desktop do you think you’re going t o be a better programmer?,” Gary Gruver, vice president of quality engineering release and IT operations at Macy’s said last fall at FlowCon, explaining that every developer wants to do a good job and is typically trying to do the best job that he or she can. “If you can’t provide them that rapid feedback , real-time in an efficiency manner, how do you expect people to get better?”

In the scenario above, the process of developing the offending code is long gone in a developer’s memory, which means it takes more work for the coder to review their work, remember the project and then mentally link that process to the infrastructure dependencies that were affected by it. With shorter cycles, devs spend less time remembering what they did and more cognitive work on understanding dependencies and the context in which the code operated. Long-term, that’s going to make for better code from the get-go later on.

“Developers who can understand how their applications work under additional loads, abnormal conditions, and degraded circumstances, will end up designing better applications and systems, and in the end play a more strategic role at the company,” says Arnold Goldberg, vice president of customer engineering for PayPal.

According to Goldberg, context in development is critical and the faster, richer feedback provided by DevOps allows developers to better understand how application improvements work in production context. And that is huge for informing best practices.

“Much of it is specific to the fabric of the existing organic infrastructure that a company has, so it needs to be a pragmatic application of it in the context of the company’s current situation,” he says.

Faster feedback loops aren’t just about making technical coding improvements, either. These more rapid assessment cycles can also greatly improve features and business value offered by developers. We’ll explain more in part two of this series.

May 6, 2014
DevOps and PaaS: ‘Give me a platform. Let’s rock, let’s rock, today’
Give me a platform. Let’s rock, let’s rock, today.

– Dewey Finn, School of Rock

The term Platform as a Service (PaaS) has become overloaded to define any ‘platform’ that provides ‘services’ to its users. The term PaaS is formally defined by NIST as:

“The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages, libraries, services, and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly configuration settings for the application-hosting environment.”

– The NIST Definition of Cloud Computing, NIST, US Department of Commerce, Sept. 2011

Like most terms in the IT industry (or to really generalize, the human communication medium known as language), PaaS is overloaded, overused and misunderstood. A quick web search or even a visit to the Wikipedia page on PaaS proves my point.

I prefer to look at such overloaded terms in context. In this post I am going to look at PaaS thru the lens of DevOps (no surprise to a reader of this website). I will explore what it means to an organization looking to adopt DevOps and leverage the services offered by utilizing a PaaS platform. I will explore and contrast a ‘build your own’ in-house platform as a Service or a subscription based commercial PaaS offered by a vendor. I will use PaaS offerings from IBM to illustrate my points. (Disclosure – IBM pays my mortgage).

Lets start.

Defining PaaS:

The most critical section of NISTs definition of PaaS, that differentiates PaaS from an ‘Infrastructure as a Service’ (IaaS) consumption model for cloud is:

“… The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly configuration settings for the application-hosting environment.”

This picture better illustrates the differences.

As you can see in this picture, what distinguishes the two cloud adoption models – IaaS and PaaS – is how much of the stack is managed by the user (or client) vs how much is managed by the Cloud Platform provider. What is important to highlight is two things:
1. Every capability in the stack is available as a managed service – this would be a shared, multi-tenant, service available to users, with the underlying implementation abstracted from the user
2. The Client/user only has to be concerned about managing their own applications, data and user access, leveraging the services available on the platform, which manages the rest
Platforms as a Service may be either public (IBM BlueMix, CloudFoundry from Pivotal, Google App Engine, etc) or private (hosted CloudFoundry; self built and hosted platform). Organizations may also build their own hosted and managed leveraging multiple technologies, self hosted on a private cloud. For a private instance of course, the organization hosting it in their own data center would be responsible for managing all the services hosted on the platform. Building a PaaS on an IBM PureApplication System, would be a good example.

DevOps and PaaS:

When one looks at DevOps on PaaS one has to look at the services that need to be hosted on the platform in order to implement a DevOps services platform. If one looks at a DevOps pipeline and the core components that make up a delivery pipeline up, each of the components need to be available as a service, in order to provide a complete ‘DevOps on PaaS’ solution. Namely:
- Development tools as a service (and/or an ability to connect the services on the platform to a desktop IDE)
- Build as a service
- Deploy as a service
- Test as a Service
- Monitoring as a Service
These would be what one refers to ‘DevOps Services’.

In IBM’s new PaaS offering – Codename: BlueMix (in open beta), these DevOps are services are an inherent part of the platform. There services on Bluemix are:
- Git hosting as a Service
- Web based IDE
- Agile planning & tracking, team collaboration as a Service (Jazz.net)
- Mobile Quality Assurance (MQA) as a Service
- Continuous Integration as a Service (Jenkins as a Service)
- Deployment Automation as a Service (powered by UrbanCode Deploy)
- Performance Monitoring as a Service
These capabilities working in concert, provide a Continuous Delivery Pipeline on IBM Codename: BlueMix.

DevOps as a Service (DaaS?)

So, to conclude. The value proposition for adopting a PaaS platform is self-evident. If you are an organization that delivers software, and is looking to adopt DevOps, a PaaS offering that includes DevOps services allows you to adopt DevOps at a very low cost of entry. You do not need to craft a Delivery Pipeline and implement the whole Continuous Delivery toolchain. Integrations, hosting, servicing – not your problem. Pay-as-you-go, and allow for scale. It is however, not a viable option for all organizations, as it is a drastic shift in corporate culture and potentially policies. Especially if you look at a public PaaS. If you are a born-on-the-web organization, or an enterprise with born-on-the-web applications, you should take a closer look. Check out Codename: BlueMix at www.bluemix.net.
May 1, 2014