devopsy.kinsta.cloud

Tag: automation

Does DevOps drive agility, or does agility result in DevOps?

DevOps is a hot topic. Hell, I’m writing this on a site dedicated specifically to DevOps and you’re reading it, so it seems safe to assume we agree on the general premise that DevOps is an important trend for both business and IT. I question however, giving all of the credit for cost cutting, or for more agile and effective organizations to DevOps.

A recent survey commissioned by CA Technologies titled DevOps: The Worst-Kept Secret to Winning in the Application Economy makes the claim that 88 percent of IT professionals surveyed have already implemented DevOps or plan to within the next five years. It also notes that companies that have already adopted DevOps have experienced 14 to 21 percent improvement in quality and performance of applications and/or faster time-to-market. Clearly DevOps is virtually ubiquitous, and companies that embrace DevOps are all but guaranteed to be more effective and efficient.

Well, no. Not really. I’m a huge fan of DevOps, and I think there is a very good chance that organizations that adopt DevOps are likely to experience tangible, measurable benefits. However, one does not necessarily lead directly to the other and, in fact, it may be the other way around. Perhaps organizations that are doing the right things to yield improvements that result in more efficient and effective operations are also the organizations that are likely to embrace DevOps. I call this “Does DevOps drive agility, or does agility result in DevOps?”

For starters, we still can’t even narrow down a concise explanation or definition of DevOps we can all agree on. So when we say that an organization’s success is the result of DevOps, what do we mean? Is it because the company broke down barriers between teams and enabled workers to collaborate together more efficiently? Is it because the company moved its infrastructure to the cloud and automated deployment of virtual servers and applications? Or is it because the company embraced specific tools or platforms like Chef, or Docker?

There is an argument to be made that much of what comprises the strategic side of DevOps is just a natural, cyclical occurrence in business. There’s always some new management philosophy or business model du jour with a catchy buzzword, and right now it’s DevOps. Part of the reason that DevOps remains such an ethereal concept is that most of the organizations employing DevOps don’t realize they’re using DevOps, and they don’t call it that. Those organizations are just responding to the realities of competing today in the best way they can. That effort seems to lead invariably to embracing philosophies and strategies that have been collectively dubbed part of the DevOps culture.

The organizations that started it all didn’t sit down and say “OK. We should start by adopting DevOps tools and strategies.” Companies like Netflix didn’t “embrace” DevOps—they created it from scratch, organically, and developed the tools and strategies that form the foundation of the DevOps movement. Everything that follows is an attempt to mimic the success of those organizations.

If it’s a question of which came first, the chicken or the egg, it seems clear to me that DevOps is the outcome of a culture shift more than the engine or catalyst of the culture shift. To be fair, DevOps has matured, and is evolving into a culture all its own, and the tools and philosophies that make up DevOps are more clearly defined as time goes on. A company looking at it today can explore successful tools like Chef and Docker, and study examples of other organizations that have achieved success through DevOps principles.

We still need to be careful, though, when breaking down statistics like those in the CA survey. Companies that are more agile, and deliver better quality products more quickly may also have adopted DevOps tools and strategies, but it doesn’t necessarily imply a direct cause and effect between the two. Perhaps as DevOps matures and the definition of the strategies and tools that comprise it becomes more consistent it will be easier to state clearly whether or not an organization has “adopted DevOps” and apply metrics that directly measure the effects of DevOps unique from the metrics the company would achieve anyway with or without DevOps.

January 22, 2015
DevOps needs infrastructure multi-tenancy

DevOps will, under whatever moniker you want to give it – NetOps, DevOps for Networks, operationalization – continue to make inroads into the network infrastructure because ultimately it’s part of the application deployment lifecycle. While DevOps folks call it “application delivery” the reality is that an application is not ready to deliver to a user (internal or external) until all its requisite services have been provisioned and configured.

Yes, infrastructure and network devices are increasingly API-enabled and supportive of a variety of tools and frameworks most often associated with DevOps – Puppet, Chef, OpenStack, VMware – and those more commonly associated with just the network – Arista, Cisco ACI, and OpenDaylight.

But it takes more than APIs. APIs do not inherently bestow upon devices the ability to support multi-tenancy. That is, the isolation of services unique to a team (or application) required to effectively deploy application-supporting infrastructure services within what is traditionally a shared environment.

According to ThoughtWorks Technology Radar (jan 2015) partitioning infrastructure along team bounds is becoming an imperative to removing roadblocks in the application deployment lifecycle:

Many of our customers have made DevOps a reality in their organization with delivery teams that build, deploy, and support their own applications and services. Unfortunately, a regular roadblock on that journey is allowing teams to have superuser privileges in production environments. In most organizations, the production environment is shared, and therefore risky to provide access widely. It is effective when we can partition infrastructure along team bounds,so that those teams can have safe isolated access to do their work, without risking impact to other systems. Where cloud environments are used, this is much easier to implement, aligning account structures to team boundaries. [emphasis mine]

This has always been an issue where shared services are relied upon. A change to support one application (or team) can be disruptive or cause issues for another application or team. Thus, most infrastructure and network teams staunchly refuse to let anyone in the application or operations side of the house play with their toys.

They are the network ninjas; guarding their configurations with their very lives.

It’s this kind of conflict that needs resolution if we – as a unified IT organization – reach goals of improving the time to market and frequency of deployment. It doesn’t matter if the app team can deliver an app in 3 weeks if it takes 3 months to deliver it to the customer because of overloaded network and infrastructure staff.

There are two good solutions to this problem*.

1. The governance over more application-affine infrastructure services – load balancing, caching, proxies, etc… – is necessarily moving closer to the application and under the control of app and operations teams. This is supported by concepts like Network Service Virtualization, proposed in Lippis Report 217 as well as the increasing adoption of open source proxy-based (virtual and software) services. This model assumes continued shared network and infrastructure services as appropriate while migrating per-application services to the more agile, operational architecture.

2. The introduction of multi-tenancy in infrastructure architecture provides a similar approach while still maintaining shared resources (hardware). A single, high capacity system is able to host multiple, virtual instances of its services, each dedicated to either teams or applications, as is the organization’s need. This approach spreads the cost of the hardware resources across the entire organization while allowing a more per-application approach to provisioning, configuration and ultimately, cost. Isolation addresses the concerns of shared resources and configurations that can cause consternation among network and infrastructure teams because the underlying shared system is managed by them while role-based control over application services is offered to teams or designated responsible individuals.

Both approaches are valid and in many cases both architectural solutions will be used to resolve the issues caused by traditionally shared infrastructure.

The key takeaway here is that there are answers to the need for partitioned infrastructure that are supportive of DevOps-driven workflows and processes and can certainly provide a path toward ensuring a more seamless, time-sensitive end-to-end application deployment process.

* Oh, I’m sure there’s more, but right now these appear to be the best two options.

January 22, 2015
Why DevOps is like fitness or religion
What is DevOps?

I noted in a blog post last summer that limiting DevOps to a single definition is challenging. The Wikipedia entry for DevOps contains a variety of possible explanations that seem to disagree in some ways:
- It’s a “software development method that stresses communication, collaboration and integration between software developers and information technology (IT) operations professionals.”
- The goal of DevOps is “to maximize the predictability, efficiency, security and maintainability of operational processes. This objective is very often supported by automation.”
- It “targets product delivery, quality testing, feature development and maintenance releases in order to improve reliability and security and faster development and deployment cycles.”
- It “aids in software application release management for a company by standardizing development environments.”
All of these are true, and yet none of them is necessarily THE definition of DevOps.

A recent ZDNet post titled Why there will never be one do-it-all DevOps tool proclaims, “There is really no such thing as a specific DevOps “tool”—and don’t buy into the “DevOps” labeling many vendors are attaching to their solutions. Successful DevOps depends upon a multitude of tools and methodologies that can help organizations better align their software development and release cycles.”

The question, “What is DevOps?” isn’t like asking, “What’s the temperature today in Phoenix, AZ?” or, “What’s the fastest land animal?” Those questions have precise answers that are based on measurable, irrefutable metrics. The answer is the answer and it isn’t really open to debate.

“What is DevOps?” is more like asking, “What is the best fitness program?” or, “Which religious belief system should I subscribe to?” There is a vast array of answers and passionate beliefs on all sides. But, there really isn’t so much a “right answer” as a “right answer for you”.

When it comes to fitness, there seems to be as many diet and exercise programs as there are people trying to get fit. Books, magazines, and infomercials are filled with extraordinary examples that “prove” that a given fitness program or philosophy works…at least for the subject of the story. There are some elements that are universal truths like exercise is better than being sedentary, and making healthy eating choices is better than eating an entire cheesecake for dinner. Almost everything else is subjective. There are too many variables involved that make people, and how they respond to different diet and exercise programs, different.

DevOps is also like religion. Most people don’t choose their faith. They’re raised from birth to follow in the footsteps of their parents, and their religion is more or less predetermined. Religion is also personal. While some will insist that there is only one true faith and that theirs is the only real religion, most people recognize that religious faith is very subjective and it’s a function of finding the belief system that makes sense to you.

DevOps is like fitness and religion in that what it means to an organization, and how it is first implemented, is influenced by the existing culture and infrastructure—the IT the company was “raised on”. DevOps is also like fitness and religion because there is no “one-size-fits-all” answer to the question “What is DevOps?”

It isn’t a defined process—it’s more of a concept or philosophy. It isn’t a product or service that can be packaged and sold—it’s more a collection of the products and services that enable you to embrace the concept or philosophy. To some extent, DevOps is whatever works for your organization to automate routine tasks, reduce redundant effort, streamline operations, and remove unnecessary obstacles so people can get things done as efficiently as possible.
January 21, 2015
Webinar: How to Achieve Continuous Delivery Without Downtime
Today’s businesses demand quick access to new technology, which means they need shorter release cycles and centralized control of both your on and off premise resources. Integrating cloud resources into your traditional datacenter management can prove challenging and time consuming at best. What if you could increase your organization’s ability to innovate utilizing business automation to bridge the gap to the cloud?

This webinar will be looking at how you can:
- Automate your deployment of code updates with the correct configuration
- Reduce errors by automating manual steps and scripts
- Achieve quicker deployment times to meet business demands
- Integrate with Jenkins build management
Join us on February 18th as Tom Flitter, Director of Applications and Integrations at TASC, discusses how TASC solved long release cycles, extended service downtimes per release, and highly manual processes for releasing their main application across all environments.

RECORDING

SLIDES

[seoslides embed_id=”b0034e485c27″ script_src=”https://devopsy.kinsta.cloud/embed-script/automate-your-world-tasc-v1-0-pdf3531/4773/” overview_src=”https://devopsy.kinsta.cloud/slides/automate-your-world-tasc-v1-0-pdf3531/” site_src=”https://devopsy.kinsta.cloud” site_title=”devopsy.kinsta.cloud” title=”automate-your-world-tasc-v1-0-pdf3531″ /]

Date: Wednesday, February 18th, 2015

Time: 2pm eastern standard time
```
REGISTRATION LINK:(CLOSED) 
```
About Speakers:

Tom Flitter is currently the Director of Applications and Intergrations at TASC (Total Administrative Services Corporation). He is an information technology leader with substantial experiene in application development, program management, project management and team development. He has years of experience in leading complex, strategic, cross-functional, and high ROI technology initiatives. He holds an MBA for Business Administration Management and has been working at TASC for 5 years. You can connect with Tom on LinkedIn: Tom Flitter

Ralf Paschen is currently the Business Automation Executive at Automic. Over the past years he has held various positions in the software industry. Coming from a technical datacenter role, he moved into presales, sales to product management, and product marketing in several software companies. He was responsible for the complete SAP portfolio at Automic and is now building the solutions marketing department to give customers and partners a clear understanding about the innovation value of the unique One Automation Platform. You can connect with Ralf on LinkedIn: Ralf Paschen or Twitter: @RalfPaschen.
January 14, 2015
Awkward questions for those getting started with automation
This blog asking awkward questions for folks jumping on the microservices bandwagon was enlightening from the perspective that many of the same questions are applicable to operations as they begin to tackle operationalizing all the things in an application lifecycle.

For example: “What is your deployable unit?”

Great question, isn’t it? What is your deployable unit of automation? Is it per service? Per device? Per cluster of devices? Per container?

Wait, but there’s more from this great list of awkward questions:

Is it acceptable for another team to take your code and spin up another instance of your microservice?

Can team A use team B’s microservice or are they only used within rather than between teams?

Do you have consumer contacts for your microservices or is it the consumer’s responsibility to keep up with the changes to your API?

Is each microservice a snowflake or are there common conventions?

How are these conventions enforced?

How are these conventions documented?

What’s involved in supporting these conventions?

Are there common libraries that help with supporting these conventions?

All these questions are applicable to devops, in particular, as you go about building out automated scripts, systems and orchestrating processes that operationalize the data center and make it a lean, green deployment machine.

You are going to have some kind of code, whether it’s a script or a recipe or a template of some kind, and questions regarding the usage of that code is important. Can others use it? Change it? Claim it as their own? Are they useful in a general, “here’s a basic load balancing provisioning and configuration script” or are they specific, as in “this is a piece of code that provisions and configures the networking required to ensure all 18 tiers of this very specific application can communicate”?

And as you’re developing those scripts (that code) are you documenting it? Are there <gasp> comments? Do you use a standard convention for code formatting (cause let me tell you, where you place your brackets is a matter of great technical religious significance)?

Most of those jumping into automation head first aren’t thinking of the long term ramifications of the choices they are making right now. Proof of concepts turn into pilots that turn into systems upon which all of IT relies. Rarely are you given the opportunity to go back and put any kind of standard conventions or practices into place. That’s what builds technical debt in software and, no matter what you think you’re building with those scripts, you’re building software.

Adopting the technical and methodological practices of agile development is a good thing, but let’s not ignore that along the way from waterfall to agile developers have learned a lot of lessons the hard way. That means we should, as practitioners of software-related activities within operations (and that means any one of the four operations groups in IT), pay attention to those lessons and try to avoid the pitfalls that developers have already encountered and, one hopes, overcome.

To do that, don’t just ask the awkward questions; answer them.
January 7, 2015
Infrastructure Automation and API Taxes

That automation is one of the critical components of DevOps is apparent in numerous surveys and studies that focus on the benefits organizations have realized from implementing the approach across operational groups. Automation itself starts with (or should start with) a comprehensive API. That means infrastructure – network and application focused – needs to expose via an API the myriad configuration options it requires as well as optional features it offers.

It is through these APIs, after all, that integration between devices and systems and across operational groups occurs. The much vaunted SDN or cloud “ecosystem” is really a euphemism for pre-packaged and validated integration. Integration that occurs via an API.

These same APIs are available to everyone, of course, and lay a foundation to operationalize app deployments by automating the various provisioning and configuration tasks required to move an app from dev to production.

APIs allow for fine-grained automation and orchestration, with each and every knob, button and slider option available to be flipped, switched or turned. The more features and functions possible, the more buttons and knobs and sliders there are to manipulate, figuratively speaking.

Each and every one of those calls to do so needs to be carefully managed. Exceptions happen, errors occur, and any script or tool ought to carefully validate that each and every one of those calls has been successful. Which is going to take a lot of code. Code that itself will introduce the possibility of errors and require maintenance and reviews and all that goes along with good development practices. Code that will also impact the time required to execute. Given that most APIs are RESTful and require invocation via standard HTTP exchanges, each and every API call incurs time to send, time to execute, and time to receive a response. The overhead is basically an API tax; for every API call you invoke, you’re paying with time… and more time.

Martin Fowler touches on this in his post “Microservices and the First Law of Distributed Objects“:

The consequence of this difference is that your guidelines for APIs are different. In process calls can be fine-grained, if you want 100 product prices and availabilities, you can happily make 100 calls to your product price function and another 100 for the availabilities. But if that function is a remote call, you’re usually better off to batch all that into a single call that asks for all 100 prices and availabilities in one go.

Thus for sufficiently complex systems another option is needed: templates. Templates do not replace APIs, but rather provide an additional and far more compact method of accomplishing the same tasks automatically, but without the overhead imposed by highly granular sets of API calls. Templates encapsulate a configuration; codifying the settings for a variety of knobs, buttons and sliders into a single configuration artifact. In a fine-grained API-base integration, each of those settings must be maintained somewhere external to the automation script or system. Each API call will include one (or maybe two) of those variables as read from some file or system that maintains the application-specific set of data. This introduces many potential points of failure. Whether file or database, variables must be retrieved before being associated with an API call and then transmitted to the receiving device. Errors may occur on read, on format, or on transmission.

In contrast, if all the variables exist as a single, validated configuration template artifact, it can then be sent, as a single entity, via an API call to the device or service in question. The number of potential points of failure is effectively reduced to read/retrieve and on transmission. This greatly simplifies the process as well as reduces the time it takes to actually execute the configuration of the target device or service.

In addition to simplifying integration by reducing the provisioning and configuration tasks from potentially hundreds of API calls down to a small set of calls, this has the benefit of fitting much easier into an infrastructure as code strategy, where common configurations are treated as artifacts and stored in repositories much like its application counterparts.

There is an inflection point at which it becomes advantageous to use a template-based approach over individual API calls. That point occurs before the scripts driving the API-based automation become as complex as an application and thus bring with it associated issues like error rates, time spent troubleshooting and maintainability.

A template-based approach simplifies upstream workloads and integration as well as discrete automation scripts by reducing the set of variables required to just a few – usually those dependent on network-specific values such as IP addresses and network segmentation membership (tags or ids representing VLAN, VXLAN, NVGRE, etc…).

This further ensures a consistency (and in turn results in greater stability of the underlying infrastructure) in app deployments.

APIs are a good thing. They’re a key enabler of software-defined architectures like SDDC, cloud and SDN. But API-enabling infrastructure doesn’t necessarily mean only on a checkbox and radio-button basis. That can be valuable but it can also lead to integration efforts that are just as complex (or more so) than their manual counterparts. A template or policy-based (application-driven) approach coupled with an API through which to deliver and execute such constructs results in a much cleaner, more consistent and stable means of integrating provisioning processes into the greater software-defined architecture.

November 20, 2014
Automating Away the Regulatory Compliance Myth

At the recent DevOps Enterprise Summit, we had a chance to sit down with Justin Arbuckle and discuss his concept of “compliance as code” and why it’s so important that security and compliance tests be built into DevOps practices at large enterprises. Recently, Arbuckle was chief architect at GE Capital where he introduced and sponsored DevOps at that organization – so he certainly knows about what it takes to “do DevOps” in large, highly regulated environments.

Currently, Arbuckle is the vice president of EMEA at CHEF and also chief enterprise architect there. If you watch his talk from the DevOps Enterprise Summit, Hunting the DevOps Whale, you’ll see that he is passionate about the transformative potential of DevOps in large enterprises.

At CHEF, Arbuckle is in helping large enterprises understand how to “de-risk” their migration to cloud with automation. We caught up with him at the DevOps Enterprise Summit to ask why that is so important. Here’s what he had to say.

devopsy.kinsta.cloud: When it comes to DevOps and regulatory and security concerns, many people say, “Well, just automate those tests.” What does that mean, to you? How do you “automate in” security and compliance?

Arbuckle: I think, in terms of the specific issue of regulatory compliance, a lot of what we think of as the being compliant process today is a complete myth. There is just so much detail that needs to be checked that the organization is always in a position where they have to trade off against, “It’s good enough, we’re ready to go” versus, “We’re not going to go anywhere until we’ve literally crossed every T and dotted every I.”

The reality is that a lot of the regulations, Dodd-Frank is a good example, are getting to a level of detail where they’re beginning to talk about process. They’re beginning to talk about how you design product. They’re getting very deep indeed.

So the organization that has to govern that, can’t do it based on just this idea of oversight at particular points in the life cycle. Because if you’re going to do it properly you literally slow your velocity to zero, and so you’re making a trade-off between compliance and competitiveness.

The way to break that cycle is to, and this is a big paradigm shift for regulators, a big paradigm shift for compliance people inside organizations, you have to start being able to express those compliance requirements in a form that’s codable. Using something like Chef for instance what we’re able to do is we’re able to specify a particular policy in a way that is automatable. In a way that we know is always going to be implemented every single time.

devopsy.kinsta.cloud: Would that include input from security people, compliance teams, design teams, system architects?

Arbuckle: Absolutely. Honestly, in my experience I think the number of organizations that can count fully detailed, fully implementable — and that’s the key word — security policy by their infrastructure people is countable on one hand.

Generally speaking, there are many obvious guidelines that are standards-based. But as the security organization tries to keep up with security threats, they’re obviously learning as they go. In this way policy always lags. And the only way for the organization to catch it is through this process of documentation, policy, and checks. And through it all they know that the standard is nonsense because it’s out of date by definition. So they have to create a point-in-time review. And these point in time reviews bring velocity to a halt.

devopsy.kinsta.cloud: Yes. Especially when something is wrong. They hit a red button and everything stops coming off the assembly line.

Arbuckle: That’s right. The approach that we have to get to is the idea of policy as code. But you are not going to be able to, on the first go, be able to codify everything. Very often with automation, based on my experience, is that the difficulty at large enterprises has actually less to do with the tools and more to do with knowing what you want to automate. Knowing what you actually want to put in infrastructure as code.

It’s the same thing with security teams. Ask them to go through the most important things that need to be tested in automation, and getting them to go through that process. It’s often difficult because they are not development people typically. They’re not infrastructure people, And they’re not operations people.

They’ve kind of created their own “securocracy” if you like.

What security teams have to do is understand their work is subject to source control as well. Their work is subject to variation and evolution over time and we should write tests to ensure that the following policies are being applied consistently.

DevOps: And I assume these tests are designed over time, as one goes along? For example, if there’s a new malware or regulatory beast that you haven’t seen before, you can design a test and then implement that as part of the life cycle?

Arbuckle: That’s exactly right. And the great thing about tests is they are additive. You’ve got a test that tests for the range of ports that are open on the operating system and then you add another test, maybe testing for the types of applications that are installed on the box. They are two separate tests and they are additive. So as you continue, you build up your library of tests.

And for most organizations going through the adoption of test driven development, as an example, they have to do it that way. A large application that is beginning to develop in an agile way is going to have a lot of tests. If you take a typical large core banking application, for instance, it will probably have something in the region of about 5,000 use cases. That’s thousands and thousands of tests. Let’s just agree that in no short period of time are you going to write those tests.

devopsy.kinsta.cloud: That’s an enormous catalog of tests. How do you keep them up to date and fresh? I imagine as security changes, regulatory compliance changes, you may have to tweak these tests as you go along. How do enterprises keep up?

Arbuckle: Like any requirement you have stakeholders that are responsible for those requirements. In just the same way, if you had for instance a mobile application and you have a product owner for that mobile application and that product owner is prioritizing feature sets you should also have a product owner for compliance.

And the product owner should be detailing what the set of compliance requirements are for the product.

Essentially compliance is embodied in the tests that are run against the appropriate applications. And so to manage automation and compliance as code, you need to have a product owner for compliance. And compliance as code, or automation, is the way it has to be in order for compliance to actually be true and consistent, as opposed to a myth.

November 4, 2014
How to Enhance DevOps by Supporting the Work of the Build-Release Engineer

Support Through Freedom

As we’ll see, supporting the work of the build-release engineer enhances DevOps by adding consistency, easing minds, and freeing DevOps teams to do more. To support the work of the release engineer, the enterprise must set them free as well, by letting them automate everything, treat manual interventions as process failures, and configure the target deployment machine and its monitoring context.

Enable the Release Engineer to Automate Everything & Treat Manual Interventions as Process Failures

One of the best ways to support the work of the build-release engineer is to encourage them to become actively hostile to manual interventions, says Robert Fischer, VP of Engineering, Webonise Lab www.webonise.com. Release engineers should shoulder the responsibility for putting manual interventions out of reach.

An enterprise’s internal DevOps community should empower the release engineer to automate as many steps as possible, which may require the enterprise to give them programmatic access to environment credentials, says Fischer. “For the internal DevOps community, the reward for permitting this risk is that they will have a cleaner environment that everyone understands better. This will make it easier for them to respond correctly in stressful situations,” says Fischer.

“So, let’s say you have some Linux VMs spinning up as part of the release. The release engineers could be tempted to use some kind of global SSH key for accessing these VMs, which they would install nicely on their laptops or workstations,” starts Fischer. Unfortunately, this will make it easy to add undesirable and unnecessary manual steps such as “‘Log in and set this IPTables value’ or ‘Log in and manually determine this hostname’”, says Fischer.

And while it would be easier to enable and apply these changes in the short run, it would leave a monkey wrench under the hood of automation and discombobulate the efforts of DevOps true believers in the long run. And it would all start simply by enabling the global SSH key and making these manual steps possible.

These kinds of manual processes also crop up in branches in the release logic: ‘Login to the server and see if packages X, Y, or Z are up-to-date in the image. IF they aren’t, THEN…’ All of that needs to be automated. “If the release engineers make it hard to implement manual steps, then they are more likely to A) notice and B) avoid these manual dependencies,” clarifies Fischer.

Instead of making manual steps possible, the release engineer should create active blocks for manual intervention, such as putting virtual machines within a virtual private cloud or making them only accessible using the automation tools’ SSH keys, says Fischer. This will actively impede manual interventions like those first cited herein and force the release engineer to think. “The result is that you are more likely to build the appropriate automation steps and error handling instead of creating a dependency on manual intervention,” says Fischer.

Empower the Release Engineer to Configure the Target Deployment Machine and It’s Monitoring Context

Another way to undergird the work of the release engineer is to open up the target deployment machine and monitoring context configurations to them. To enable the release engineer to configure the target deployment machine, make tools like Docker www.docker.com available. To enable them to configure the monitoring context, automate the configuration of the virtual machine platform (e.g.: AWS http://aws.amazon.com security groups, load balancers, etc.). “This allows for tuning to be a part of the release planning,” says Fischer.

The enterprise DevOps folks can still make manual adjustments if they must, but they should resist the impulse to use that option, except when handling anomalies. “Leaders in enterprise DevOps approaches can audit these deployment instructions for sanity and compliance, of course,” says Fischer.

As is the case with automating otherwise manual processes, the enterprise DevOps team should have a lot less to worry about once they set the release engineer free to configure these things. “This should give them more time to tackle the hard and interesting problems instead of performing repetitious and error-prone manual interventions by letting the computer handle the brain-dead computational and rote part of DevOps, which is what computers are good at,” says Fischer.

October 30, 2014
Scripts or Machine Images, What is the Best Way to bootstrap
Between the advent of handy tools, like Chef and Puppet, and virtual machine infrastructures, like VMWare and AWS, I feel like there has been a great debate about how to bootstrap machines. It seems like there are two extremes. On one side is heavy reliance on machine images (AMIs, in AWS speak) or scripting (recipes, in Chef speak) on the other with a bunch of grey in the middle. Naturally, there are pros and cons with each so let’s explore them.

Machine Images

The great thing about machine images are you get from zero to a fully functional machine in the time it takes to create the virtual machine. This can be an extremely valuable thing when you need to spin up or down quickly. However, they become dated with the rate at which your configuration changes.

Scripts

With the all scripts approach you are immune to the out of date issue encountered with machine images. The big drawback is that it can take a long time to spin up a new machine off of scripts, especially when you have to compile one or many components. In my opinion, this approach is most valuable when your system configuration changes quickly and there is some tolerance with regard to spin latency.

Either approach is suitable for things like auto-scaling and unlocks the elasticity of the cloud, so which is the best? To help you decide on which approach, or blend there of, best suits your situation. These may seem obvious, but I think you must weigh three major trade-offs or benefits afforded by scripting or imaging.
- Time to being service ready
How much time can you afford to spend from the time you start bootstrapping a new machine until handling live traffic? To meet our on-demand scale requirements, at Clip, we have a target of 10 minutes from when the virtual machine is requested to handling live traffic.
- Resilience to change
How dynamic is your environment? How often are new services deployed, dependencies added, or code deployed? At Clip, we add new services on a fairly regular basis and our systems change frequently from both the code and dependencies stand point.
- Cross platform support
How many platforms do you need to support? Platform, in this case, could be interpreted as operating systems, cloud providers and/or virtual or physical hardware. To date we have been able to minimize any cross platform issues.

Since we value flexibility and have the luxury of an acceptable spin up time of 10 minutes, we have chosen to lean heavily on Chef for our bootstrap process and only use basic machine images. However, as the product and business requirements evolve, the time to being service ready is gaining more emphasis, so our process will also change. At other companies, spin up time is so important, their CI system builds a new machine image for each version of the product that is deployed, so they can scale as quickly as possible. Regardless of the technology or process used, it is most important to satisfy business needs in terms of service ready time, resilience to change and cross platform support and continue to adapt as the requirements change.
October 28, 2014
Cloud-Based Documentation at the Speed of DevOps

Welcome back to this (semi-)regular column where I attempt to answer questions about DevOps, especially in larger enterprises. This time I address a vexing issue facing many large enterprises – how to make sure downstream processes move as fast as development and operations:

Q. In my enterprise we are getting faster at releasing code, but we are still slow to update and translate downstream product documentation. How can documentation and technical writing be optimized as part of a DevOps approach?

As I have said before, you can ask me any question you want (in the comments below, via e-mail, or on Twitter), but that doesn’t mean I know all the answers. If (when) I don’t know, I have promised to try to find someone who does – and this is one of those times.

This question stumped me, so I asked around to find some insight into this issue. As it turns out, my company (a large enterprise, building out our DevOps capabilities, developing and operating new software every day, in many different languages, across about 40 countries worldwide) is working with what we think is an answer to this question. So I asked my colleague Jim Turcotte to explain what we are doing to address this problem. Here is what he said:

The problem – serial downstream processing

I know exactly the problem that this writer is facing! We have been wrestling with this ourselves. The problem is that with current industry practice, a team of technical writers shadow developers to document the latest product functionality. By the time product is delivered, it’s a mad dash to publish the PDF/HTML. Then, to make matters worse for international customers [our software users], it may be days or weeks before localized [translated] versions of content follow. It’s a slow, tedious, serial process that ends up being a rush to the content finish line every time.

To make matters worse, both documentation and code are typically “thrown over the wall” for Support to discover and address bugs. Documentation bugs are often handled by Support engineers writing dozens of knowledge docs. The end result is that skilled Support engineers are often diverted from addressing customer issues, to updating documentation. We then expect users of our software – our customers – to search through the product documentation and sometimes hundreds of Support knowledge documents to find answers to their questions—not exactly an optimal self-service experience!

Introducing a cloud-based collaborative approach

Something we have started doing to address this concern is by applying what we call ‘DocOps.’ It is a continuous content supply chain that connects all the stakeholders, and which is agile and collaborative. In this way, our approach mirrors many of the characteristics and culture of DevOps.

Instead of waiting until code is complete before starting on documentation, multiple writers “curate” content throughout the product lifecycle. Instead of producing static, standalone knowledge documents in PDF (or even HTML), we use cloud-based documentation, wikis, forums and chat groups. Instead of a single one-time effort, original content topics are matured continuously in a collaborative effort. Instead of channeling documentation through a single technical writer, we allow developers, operations, support, and even technical field staff [pre-sales] to curate documentation.

On-demand translation is performed as needed using local cloud-based translation services. Highly automated workflows determine which vendors will be used for each product and language. The platform senses the customer’s language preference and automatically pulls up the content in the language they prefer.

Then there’s online help. With the approach, we provide contextual help by linking content directly to product UIs. This allows customers to access additional sources of information like education courses and discussion threads all from the same source. Customers no longer have to log into multiple systems or sites to find the information they need to be successful.

Benefits of the new approach for end-users – and for us

For end-users, collaborative curation ensures documentation is always crisp and relevant, reducing the age-old problem of large quantities of duplicated knowledge/content. Customers who need help using a product are always able to find and use the latest curated content – in their language, relevant to their experience. Customers can also now access not only the developer’s original instructions, but also knowledge from other product experts, colleagues and peers.

For us, we can use its analytics to provide real-time operational feedback, so technical writers can make updates on the fly to further improve the customer experience. The new approach also enables us to close the feedback loop, connecting end user questions with engagement and lead generation tools such as Marketo, marking the first time product documentation can also be leveraged as a serious sales tool.

The old school Content Supply Chain cannot support the business agility that DevOps requires to be successful. The new documentation approach solves that. It was conceived and built around the need for an improved customer experience with product information. Do you really want to maximize your DevOps platform? Support the content with the new approach for the ultimate agile experience.

Jim Turcotte is a Senior Vice President with CA Technologies and the Business Unit Executive for its Information Services team. For more information about DocOps, follow him on LinkedIn Pulse.

October 26, 2014