devopsy.kinsta.cloud

Tag: ops

Automic Helps DevOps Transformation Sparkle
Working closely with Automic, an international wholesaler and retailer of jewelery transformed a department of project managers, developers and operations responsible for all of the customer-facing websites into a DevOps organization. As a result, project delivery was completed with 90 percent fewer delays, and first customer-facing releases of projects could be delivered in less than half the time. The usability of its websites increased significantly, resulting in more customer traffic, longer average time spent on the sites and increased revenue.

Haphazard Development

Prior to the DevOps implementation, new projects were delivered by “waterfall” project management means. The cyclical resource planning process was based on the project portfolio and business priorities. Project managers then planned project milestones working closely with business stakeholders, business analysts and application developers. Analysts created specifications and developers were assigned to work on packages and tasks to implement them.

Different web technologies were used with varying levels of dev and ops expertise in each technology. The Operations team members usually fulfilled requests made during the course of a project as part of their overall daily tasks—for example, to provision a new environment, deploy a package or change the environment configuration.

This approach had a major disadvantage. Delays in one project impacted the overall resource planning and could have a cascading effect on other projects. Delays in projects could be for multiple reasons, such as waiting for business stakeholder impact, frequently changing requirements, unexpected sick leaves of developers or poor estimates. Other frequent causes included delays in operations tasks such as a deployment or provisioning of a new environment. These delays were caused by unaligned planning as well as a mismatch in priorities between projects and operations. Besides delays, some developers were underutilized, which also was exacerbated because they were specialized in a technology or component.

DevOps Transformation Solution

Working closely with the customer, Automic restructured dev and ops by:
1. Switching the development from a project to a product mode (cross-project development teams based on capabilities rather than assignment of individual developers to projects).
2. Moving project managers from development micromanagement to focus on major milestones and deliverables to the business.
3. Enabling some project managers as product owners (project management remained necessary).
4. Providing developers with agile methodologies through initial training plus ongoing coaching (six to 12 months).
5. Providing ops with an agile methodology.
6. Organizing both dev and ops into multiskilled teams focused on individual products rather than technologies.
7. Aligning ops into delivery teams, initially part-time and later full-time.
8. Giving delivery teams the end-to-end responsibility for their products, including their operations.
9. Standardizing the technology stacks within teams (but not across them) over time.
10. Delivering projects into production (or alpha, beta) in slices (releases) instead of just once to get early feedback and improve.
Results: Fewer Delays, Better Usability

By standardizing, this customer has transformed its dev and ops teams, creating an agile, scalable environment that can drive the business forward. The company achieved the following specific results:
- Project delivery are now completed with 90 percent fewer delays. First customer-facing releases of projects now are usually delivered in less than half time.
- The usability of websites has increased significantly, resulting in more customer traffic, customers remaining longer on the sites and increased revenue opportunities.
About the Author/Michael Schmidt

Michael Schmidt heads DevOps Practice at Automic, a management and technology advisory service established to help Automic customers to prepare for and transform their organizations toward DevOps. Michael spent his carrier as consultant, entrepreneur and in several product-related leadership positions. He has a passion for building lean IT organizations. Michael holds a joint MBA from New York University, London School of Economics and HEC Paris and a Master of Computing Science.
January 5, 2017
DevOps Chat: Lucas Carlson, VP Strategy, Automic, Can Ops Thrive In DevOps?

In this DevOps Chat our guest is Lucas Carlson, newly appointed VP of Strategy at Automic. Of course, Lucas has already had a distinguished career. Here is his bio from Linkedin:

Lucas Carlson is an author, entrepreneur and executive. He also has a background of 20 years of programming to back it all up.

He’s written scalable software that has been downloaded millions of times and used by millions of users. As a keynote speaker in the technology arena and innovation leader in the cloud space, Lucas helps companies discover opportunities for growth.

Lucas wrote a bestselling book on Ruby for O’Reilly called The Ruby Cookbook and has created over a dozen open source libraries, some of which have been downloaded almost a million times.

Lucas was formerly the CEO and founder of AppFog, a massively popular DevOps startup acquired by CenturyLink in 2013. AppFog signed up more than 100,000 developers, raised nearly $10M in venture funding and has become an industry leading Platform-as-a-Service.

Before following his passion and writing books, Lucas served as the Chief Innovation Officer for CenturyLink, in charge of building next-generation Docker and Linux Container products for a $20B company.

Lucas also writes for major tech and entrepreneur publications including Inc, Business Insider, InfoWorld and InformationWeek.

I spoke with Lucas about why he joined Automic. Our discussion proved to be a provocative one, in which we explored Lucas’s view that DevOps is not a great thing for Ops. Ops needs its “own DevOps” that is more geared to its challenges and world view. Of course, many thought that DevOps is just that. Before you comment, have a listen to our conversation to see what Lucas said. I am interested in your thoughts and comments on this.

Alan Shimel: Hi, everyone. Alan Shimel, editor of devopsy.kinsta.cloud here for another DevOps Chat. Very happy to be joined in this DevOps Chat by VP of Strategy at Automic Lucas Carlson. Lucas, welcome.

Lucas Carlson: Thank you so much. Glad to be here.

Shimel: Thank you. Lucas, I know you’ve been at Automic now, what, about two or three months?

Carlson: Yep.

Shimel: But I’ve known Lucas Carlson for years before Automic and I don’t know if our audience is familiar with you, but you’ve had quite a storied career already at a relatively young age.

Carlson: Thank you. Yeah.

Shimel: Why don’t you give our listeners a little bit of the Lucas Carlson story.

Carlson: Sure. Well, I started out as a programmer: for 20 years I wrote code. I started code when I was in elementary school and ever since I’ve been writing code. And I wrote two programming books during my career. I’ve mainly focused on the open-source side of things so Ruby, PHP, Python worlds.

And then I started a platform-as-a-service startup about seven years ago and I used Cloud Foundry as the foundation for my startup and it was called AppFog. We signed up 100,000 developers, it was a lot of fun. It was a great time.

We were acquired by a telco and it was really interesting to spend a few years inside of a telco and watching them attempting a digital transformation—trying to move their applications into the cloud was a very interesting story.

And after that I quit that job, I moved to Paris for a while—it was a life dream of ours—moved my family, and after Paris I looked around for what to do next and this opportunity came at Automic, and I thought it was a very interesting opportunity for reasons we might get into.

Shimel: Great. Great. And of course, you know, interesting with your background and accomplishments Lucas that Atomic, which is an automation company, is where you wound up and as VP of Strategy there.

So as someone who was a chief strategy officer at the last company I co-founded, Lucas, I can appreciate the title of VP of Strategy. But for our listeners who are saying, “Well, that sounds like just another corporate title,” give them a little insight. What’s your role as VP of Strategy at Atomic?

Carlson: Yeah. Well, my role is to take a company that’s been around for almost 30 years that’s had a product that is battle tested, been around the block, really works, works at scale, has thousands upon thousands of enterprise scale customers, has a lot of pedigree and DNA, and yet is a company nobody’s heard of.

That’s actually why I kind of joined. I thought it was a really interesting challenge to take a company that nobody’s heard of yet has had a product that is—has stood the test of time, is very, very robust, very strong, and see if there wasn’t a way to kind of help that company get recognized for the work that they do, which is actually really great work.

And so I joined the company with that kind of concept in mind. And it’s been very interesting because fundamentally when I came into the company they called themselves a DevOps company and I come from a background—I’ve been doing DevOps for a long time. I grew up around Portland, I hung out with Luke Kaines who founded Puppet and he was one of my mentors when I was building AppFog and—so I knew DevOps inside and out. I used DevOps to build my own startup.

And so I came in and I looked at what Automic had and Automic has a very interesting, strong product, but it’s not DevOps, because DevOps is predominantly—you know, having grown up around it I know you know DevOps is not some mystical union of developers and operations into this perfect combination that can do both and is just as adept at writing Ruby as they are debugging MySQL.

You know, that’s kind of a story—a fairytale that people like to tell, but it’s not the truth. The truth is, DevOps was created by developers for developers to get around operations. They were sick and tired of waiting for operations to get around to provisioning their SQL server so they wrote some code that did it for them.

They were tired of kind of the delays and lags so they wrote code that got around it. It wasn’t really meant to create this mystical union it was really meant to bypass operations.

So, I guess, how does that apply to Automic? So, Automic came in saying they were DevOps—they weren’t a tool for developers and yet I thought they have this long track record. They’re doing something right, but what is it?

And I spent a lot of time thinking about that before I joined and when I joined I became convinced that there is a huge part of the DevOps story that is getting left out: it’s getting forgotten, it is getting sidelined and ignored that doesn’t deserve to be ignored.

I think that DevOps isn’t enough on its own. Having run a platforms and service company myself, having put platform as a service into major name-brand banks, having seen people trying to adopt these DevOps technologies—what I’m seeing over and over again is that one of the biggest struggles about DevOps is a myth that I’m trying to shatter. And the myth is that DevOps doesn’t need operations: that DevOps outsources operations, that operations isn’t necessary for DevOps.

In fact, I think the absolute opposite is true and I don’t think ops is becoming part of dev. I think that ops deserves its own practices equivalent to what DevOps brought to developers. Equivalent and yet different, because I think that fundamentally operations was a different world of developers: always will, always has.

To developers, failing fast is important and breaking stuff is totally acceptable. And going iteratively is great. So if you write some code, it doesn’t work you can fix it really quickly because you failed fast and it makes it better. And that kind of iterative approach is something that works really, really well for developers but it does not translate for operations. Operations cannot fail fast. If you fail in operations and your database goes down in production, that could be millions upon millions of dollars lost for the business.

So it’s not that operations doesn’t want speed—of course they want speed, it’s just that speed is not the primary. For developers, speed is primary; for operations, reliability, robustness, scalability trumps speed. It’s not that they don’t want speed, it’s that they have a different core value set—a different world view—and instead of trying to mush these things together, instead of making them one idealized person that’s never going to exist, what I say is, let’s raise up the ops side of DevOps, bring it to the same level that developers have been raised. When developers adopted DevOps practices, it raised them up to a new level as a power-up—it was a mushroom in Mario. And what we need is to give the ops side a mushroom.

We need ops to level up and be able to play at the same level as developers, but with tools built for operations, not built for developers.

The tools out there for developers in DevOps are not the right tools for operations in DevOps and that’s what –

Shimel: Hold on, I’m going to stop here because I’ve got to jump in—I’m sorry—I’ve been biting my tongue.

So this is actually a pretty—I don’t want to say radical, but a very different kind of view of DevOps and what goes on between dev and ops than I think we hear from most people.

And it actually reminds me I had a conversation—actually a DevOps Chat—with Dave West, who is the CEO of Scrum.org. And you know, obviously, Scrum and part of Agile is really aimed at developers. And though he saw sort of a potential bridge from dev to ops he had a similar view of yours that, you know, we have tools that are written for developers that developers are using that are letting them get more involved in ops but how much is ops really being involved in this DevOps, if you will.

But yet, you know, let’s look—like configuration management tools.

Carlson: Yep.

Shimel: Do you think those are dev tools not ops tools?

Carlson: Yep, I absolutely do.

Shimel: Absolutely. And –

Carlson: I could go into that more but I think it’s self-evident. I think that developers use those tools vastly more than operations. I think that the reasons are obvious. The tools were built for developers by developers.

Operations looks at the world through a different lens; they use a different way of approaching the same problems and they kind of—the way that configuration management works embraces a development mindset.

Shimel: Okay, fair enough. Let me ask—so how does this, Lucas, play into Automic and their future toolset? Will we –

Carlson: Well, I joined Automic specifically because I saw this as the biggest problem in DevOps today and that Automic has solutions on the ops side of DevOps. Automic’s solutions are the level-up for ops teams. They don’t say—like one of my biggest pet peeves right now is this rip-and-replace mentality. You see it … with Mesosphere, with Docker containers, with Cloud Foundry, with OpenShift. All the big vendors out, there they all have the same promise: “Take everything you’ve done, throw it away, import all of your applications into our platform that will do it all for you and it will solve all your problems.”

Well, I’m sorry, but that’s just not going to happen. Those platforms are generally underdeveloped, undertested at scale, they are a brand new way of managing applications that is untested. They don’t embrace the vast majority of enterprise scale applications architectures. They basically force you to rip and replace. And rip and replace is not just a thing for the code it’s a thing for all the IT processes that have been known to work for the last 10, 15, 20 years.

So you’re trading a potential promise for a solution to all your problems, but the risks are exponential and vastly untested. So this whole concept of rip and replace I think is very, very dangerous in our industry today and very prevalent.

And I joined Automic because their approach is to embrace what already exists and make it better. Instead of saying, “We’re going to have a new platform for you to import all your apps to,” they have a way to describe the topology of your existing infrastructure and built on top of that and automate that instead of throwing it out and replacing with something new.

And so, I think to me, it really levels up the ops side of DevOps—it makes operations a hero, the co-hero of the DevOps story. It doesn’t deny operations their vital role.

Shimel: Excellent. Excellent. You know what? Lucas, I’d love to see this at a Speak—you’ve got to submit to Speak about this somewhere.

Carlson: Yeah, for sure.

Shimel: Or, maybe we can do a webinar and get into it because honestly, our DevOps chat format being just 15 minutes doesn’t give us enough time to really delve in here and I’d love to get a counterview of these things too and let’s really have a discussion around it. I think I’m going to take that as my job. I’m going to go find someone and we’ll go back and forth on this.

But unfortunately we are just about out of time—we’re at 15 minutes already. As I said, it goes quickly.

Carlson: No problem.

Shimel: This has been a very interesting discussion. I would call it truly an ignite or a fire starter in that you’ve opened the door here and we’re not finished with this one Lucas. We’re going to come back to it.

Carlson: Sounds good.

Shimel: But for now we’re going to call it a wrap. I hope if you’re listening to this you found the conversation interesting. Of course on devopsy.kinsta.cloud you’ll also see a written transcript of the conversation so you can hold Lucas to his words here and it’s going to be in writing.

And if you have any questions, comments, feel free to also put them on devopsy.kinsta.cloud we’ll get them out to Lucas, and as I said we’re going to try to continue this discussion at a later date.

But for now Lucas Carlson, VP of Strategy Automic. Thank you so much for really kind of blowing the doors up today on this and we look forward to seeing what the follow-up’s going to be. And success at the new position.

Carlson: Thank you so much. I’m very happy to be here.

Shimel: Thank you. Yeah, no, it’s a great company. We’ve worked with Automic for some time and I’ve met a lot of great people there.

Anyway, that’s a wrap for this edition of DevOps Chat. Lucas Carlson from Automic was our guest and this is Alan Shimel for devopsy.kinsta.cloud. We’ll see you at the next DevOps Chat. Thanks everyone, have a great day.

— Alan Shimel

November 4, 2016
First We Fixed Dev, But Let’s Not Forget About Ops

Since its beginning in 2008, DevOps has gone from an underground rebellion by developers to a mainstream movement with representatives in every industry and at every scale. You can see the change just by looking at how people attending DevOps events dress: the black T-shirts are still there, but these days there are lots of suits and ties as well. Another hallmark of this success is companies trying to fit DevOps into their existing conceptual frameworks—hence, the plethora of “ITIL and DevOps” papers out there.

Despite all this success, though, the DevOps movement is still young. Of course, things move fast in IT, but we aren’t talking about a technology or a technique that can be adopted rapidly and spread very widely. Committing to DevOps is a major cultural shift for organizations, and that takes time. Some organizations have been doing DevOps for five years and have fully re-engineered their processes around this new approach, but most are still only at the beginning of their DevOps journey. As William Gibson said, “The future is here; it’s just not evenly distributed.”

First Things First: Speed Up Releases

The early phase of DevOps adoption focuses on activities around releasing code to production, because that was the bottleneck that the DevOps movement was originally created to break. While it still may be too soon to declare victory, it’s safe to say the improvement in the release process is enormous. Today, any new project that does not propose to implement DevOps principles would have to have a very good reason and explanation.

Notice that qualification about new projects. It’s easy to adopt DevOps methodologies and expectations in a greenfield situation; however, it is much harder in a brownfield project, where previous cultural norms have been expressed in tooling and processes that are hard to change in-flight. The good thing is that over time, the old way of doing things will die off naturally, as code bases get re-engineered, ported and replaced. Eventually, every project will be a DevOps project.

Are We Running So Fast That Others Can’t Keep Up?

DevOps, however, is not just about release and deployment. It also covers operation of what has been deployed. It’s right there in the name, after all! Coming from an operations background, though, there can be a feeling that DevOps is not for “people like me.” Some simplistic interpretations of DevOps focus on the sometimes adversarial relationship between Dev and Ops, forgetting that good Ops people are just as incentivized as Devs to get Devs more involved in Ops.

One of the difficulties that arise when engineers try to “do the DevOps” without involving Ops is that Ops are not ready to join in. When Dev are doing non-agile releases on an 18-month cycle, Ops have a certain set of assumptions about what a release represents for them. Doing these sorts of monolithic releases is a major pain for Ops for a while, but there will be plenty of time to get ready for it, and other required activities can be scheduled around that release.

One of those important dependencies is preparing the monitoring and management of the code that is being deployed. What needs to be instrumented, where do those feeds need to go, what needs to be in place to make sense of those feeds when they arrive, who needs to jump when one of the feeds goes red, and so on. There is a lot going on in the NOC that Devs are not always aware of.

If Devs suddenly show up with a more Agile cadence of a release every couple of weeks, Ops are simply not going to be able to adapt their existing tools and process on the fly to that new cadence. When you have months of lead time, it’s feasible to write and modify extensive sets of rules and filters to make sense of monitoring data, filter out the cruft, and route the remainder correctly. When your lead time on a release is measured in days or hours—not so much.

Devs should remember that “infrastructure as code” also includes debugging that code. Adopting release automation, containerization or whatever will be the next cool tool will not solve the problem on its own. Bridget Kromhout has a wonderful articulation of this mechanism:

Containers will not fix your broken culture. Microservices won’t prevent your two-pizza teams from needing to have conversations with one another over that pizza. No amount of industrial-strength job scheduling makes your organization immune to Conway’s Law, which states that “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.

Leaving aside the communications barriers that all too often still exist between Dev and Ops, one of the constraints that Ops teams have to deal with is that Ops is segmented into many different teams, each of which deals only with one area or one technology. These days, though, problems tend to have symptoms that cross these domain boundaries, and so operations teams have to communicate with each other.

The historical ways of doing this are not very efficient, relying on ticket ping-pong, escalations, war-rooms and good old blame-storming. What do you think happens if you suddenly accelerate a system like that? The answer is that it just torques itself into scrap.

Completing the DevOps Culture Shift

The good news is that others have been where you are now, and there are both tools and techniques that can help you operate what you have developed. In common with the rest of the DevOps movement, it’s as much a cultural shift as a technological one. Ops teams need to start thinking of operations in a new way.

One aspect of that change is the ongoing shift to immutable architecture, also known as puppies vs. cattle. Another is to enable efficient communication across support domains. Today, like in the parable of the blind men and the elephant, support teams often get hold of different bits of the problem, misidentifying them because they only have partial information, and therefore cannot truly resolve the problem. Nobody really has a complete view of the whole elephant, except perhaps for some savant who can be reached only after many escalations—and really has better things he or she should be doing.

If Ops teams can work side by side with Dev, on the other hand, there is a virtuous cycle that becomes possible, with efficient operations removing even more friction from the development process. Communications are the key, whether that means machine-to-machine feeds, human-to-human conversation or machine-to-human interaction. Organizations that get this right reap enormous rewards in both Dev and Ops.

Dominic Wellington serves as Moogsoft’s Chief Evangelist

March 23, 2016
What If We Had Git for DevOps?

I used to hate Git. Compared to my previous source control systems (SVN, SourceForge and floppies), it was cryptic, opaque and hard to learn. It was comfortable to create a set of files and folder, then snapshot them as a unit. It was simple to understand and improved my individual productivity.

But that snapshot created a huge tangled mess when I tried to collaborate with other developers.

Once I got over the learning curve, Git revolutionized how I wrote code. Small changes made a lot more sense with frequent integrations. I could easily review the impacts of collaborators’ changes. Even more critically, I could keep my work around while I tooled their incremental fixes and improvements. Development—especially open source development—is a team sport and requires tools that facilitate collaboration at the most fundamental level.

But it was not just learning Git; I also had to adapt how I structured projects and collaborated.

In that context, DevOps automation has been missing the equivalent of Git. We need composability!

Can a Git Model Work in DevOps?

We’ve been building great configuration automation tools (Chef, Puppet, Ansible and Saltstack) over the last few years and now there’s a new wave of cloud templates (AWS CloudFormation, OpenStack Heat, Terraform) entering the fray. These tools are like my original source control systems: they create a vertically integrated stack of configuration that works for a specific case and platform. There are ways to merge them into new stacks, but that ends up feeling like creating a whole new stacks.

Why does merging scripts create new stacks? Changes in other components often break each other.

When you reuse someone’s script, their changes can quickly break your carefully built automation. Instead of collaborating, you’re just forking their work. It feels like we’re running around data centers with automation on floppies. When we fix something in one module, it’s very hard to push that back into the community—we have not built clear, functional boundaries between modules that isolate changes in a way that reduces the risk of using other authors’ scripts and makes it easier to contribute improvements.

Isolating changes does not mean we don’t break integration! It just makes it easier to collaborate upstream.

The answer in our experience is to think of ops scripts like Git: Even inside of a larger set, we need to make the parts independent and decomposable. How do we translate Git into ops terms? Repos are like full systems—directories map to components, files to roles. Like Git, when we make incremental changes to any item, the whole system is impacted. However, we can still isolate that change down to a single line so we can manage collaborative work. This works when the impacts of changes are isolated (like a text file) and fails when changes are part of a tightly integrated unit (like an image file).

We already have git with ops scripts. How is this different?

We are suggesting an even more composable approach. Each ops role should be designed as a functional role with clear inputs and minimal side effects. To achieve that isolation, we’ve been building an orchestration platform, Digital Rebar, to feed the right inputs into existing opts scripts. That allows collaboration within a module without unexpected external side effects. It also allows reuse of modules because changes are well isolated.

When changes are contained, incremental and managed, the whole system is more tolerant to multiple authors. In that way, we may build system automation that’s more Git-like.

February 19, 2016
What happened to the “Ops” in “DevOps”?
The wave of digital transformation that has been enabled by cloud computing has provided both new opportunities and fresh challenges. On one hand, it is now possible for application developers to provide innovative new functionality to end-users almost instantaneously. One the other hand, this agility has created an environment in which users expect constant improvements and demand ever-higher levels of service. Indeed, the need to increase agility and quality simultaneously is largely responsible for the growing popularity of DevOps practices.

When I ask customers who say they are “doing DevOps”, the conversation always seems to revolve around speed. Iterate faster. Turn out more innovations and new features – multiple times a week. Pull in microservices from this cloud provider over here, program in polyglot languages over there, virtualize and automate everything on no-name infrastructure, and do it all as quickly as possible. Leverage continuous builds and immediately push to production in a perpetual beta model. Test only at code-time, or even not at all and instead let the end-users test it for you. Forget about run-time instrumentation and monitoring — just get the feature out the door, wait for social media to report any problems, and in the meantime, move on to coding the next feature.

It all sounds fun and productive, in the same way that dot-com-era rapid application development was fun and productive. And it certainly can reduce the cycle time on new features and provide opportunities to introduce new innovations. But that scenario as described isn’t truly DevOps, and it is built around a dangerous assumption about unlimited end-user tolerance for failures.

Take a break from coding the next feature, stop, and read the first paragraph again. If you look at it, the only actual “Ops” in it is the automation used to enable the development process and push to production. In other words, it’s all “Dev” – as if all important activities stop the moment an actual user touches the application. But does that make any sense? After all, aren’t we building all of these cool new features for end-users? And isn’t the reason that we want end-users to consume these features that the applications support a business goal of some kind? And for end-users to consume these features, isn’t it important that our applications actually work and provide high levels of service in production scenarios?

Here are a few tips to ensure there is some “Ops” in your DevOps efforts.
1. Stop denying there is a problem. IT Operations is “so ten years ago,” many say, because the infinite elasticity of cloud and built-in exception-handling of modern programming environments renders all of that old-school NOC stuff as antiquated as the pager those folks used to wear. But that assumption is patently incorrect – just look at November 2015’s “Black Friday” shopping season, which was a mess for many online retailers, as noted in the Wall Street Journal, Business Insider, CBC, Express and the Independent. IT Operations is more important now, not less. DevOps without Ops is not sufficient.
1. Eliminate operational information silos so you eliminate finger-pointing. Collaboration is a central tenet of DevOps and yet most people troubleshoot through inefficient “war rooms” full of individual component owners who are feverishly looking through their individual consoles to try and ensure the problem isn’t with “their” stuff. Many of those well-intentioned component owners bang their own heads against the table in frustration about how hard it is to even access the information they need in the first place, not to mention the fun job of trying to manually correlate timestamps and hostnames between their and others’ individual operational data stores. There’s no reason to perpetuate the silos of yesteryear – instead, put your operational data in one place, and let today’s machine-learning-powered management tools do the heavy lifting for you. You will certainly reduce finger-pointing, troubleshoot faster, and you may be able to eliminate the “war room” entirely.
1. Monitor what (really) matters – your actual end-users. The concept of real end-user monitoring (monitoring the actual activity inside of browsers, mobile devices, etc.) has been around for a while, yet remains an under-utilized capability in most DevOps organizations – perhaps because it used to be technically hard to implement. But that was in the past – today’s next-generation application performance monitoring tools make end-user instrumentation simple to deploy. It’s important to see real user activity not only because end-users are the whole point of the exercise, but because server-side monitoring alone doesn’t tell you what’s actually going on. As late as June 2014, in an Oracle Applications User Group/Unisphere survey entitled “Performance Under Pressure: The State of Enterprise Web App. Quality and Availability,” 85% of customers reported that usually learn about application problems from end-user complaints – a staggering number which is actually up from 79% in 2009. There is simply no excuse for such a reactive view of monitoring.
1. It’s in the logs! Logs are everywhere, but most organizations don’t use them because they are overwhelmed – logs contain too much data, they are too widely dispersed and too hard for humans to decipher. However, logs represent one of the best ways for development and operations to collaborate together, since the logging that is enabled by application and platform developers only becomes full of information in production. Next-generation management clouds that are designed to ingest big data at enterprise-scale can cope with today’s log data volume and velocity – which can be on the order of terabytes per day. Once the information is collected, your unified management regime automatically takes you to the few individual rows in the few individual logs that correspond to the exact problem you’re looking for (at the moment you’re troubleshooting), this mountain of data can become a treasure trove of information. Similarly, if machine learning algorithms can discover patterns and anomalies automatically, your logs can become ongoing sources of high-value IT and business insights. The data is there, you just need to use it.
1. Planning is an everyday activity. DevOps organizations are necessarily often focused “in the moment” – but future business results depend on accurate advance planning. In most organizations, planning takes place in artificial time windows (calendar year, budget year, off-season, etc.) with incomplete information and a whole lot of guessing – and is often performed by analysts with no direct connection to the operational context. Instead of viewing planning as a once-a-year exercise in an ivory tower, regularly leverage analytical capabilities against your unified store of operational information to answer a variety of forward-looking questions. Your DBA wants to know if she has enough database capacity for next year? Analyze at the individual instance level. Your marketing people are worried that you can’t support their upcoming launch? Analyze at the application level. Your CIO wants to know where people are spending their troubleshooting time and the corresponding health of your entire infrastructure? Analyze across your entire estate. If you’ve followed my advice in steps 1 through 4 above, you have all the data you need already available. Now it’s time to use it.
But hang on a minute, you say. Let’s not forget that we are doing DevOps for speed – and all this management, while clearly useful, could take forever to implement and slow everything down in the process. A few years ago, you might have been right. But technology marches on and today’s cloud-based management regimes can be leveraged instantaneously, with no changes to your infrastructure, and start providing the kind of value you need within minutes. With cloud-based IT Operations Management, your DevOps efforts can truly combine development and operations best practices while still delivering the speed you need.

No more excuses. Put the “Ops” back in DevOps.

About the Author/Dan Koloski

Dan Koloski is a software industry expert with broad experience as both a technologist working on the IT side and as a management executive on the vendor side. Dan is a Senior Director of Product Management for Enterprise Manager – Oracle’s integrated IT operations and Hybrid Cloud Management product line. Dan holds a B.A. from Yale University and an M.B.A. from Harvard Business School.
December 17, 2015
How IT Operations Analytics is Delivering on the Promise of DevOps

DevOps is one of the most significant movements in IT of the past several years, and the same trends that are making DevOps so important – speed, automation, and iteration – are also opening up new opportunities for IT infrastructure management. DevOps, at heart, is a matter of adopting new philosophies of working as an IT team, while letting go of legacy processes and technologies that no longer deliver an adequate level of efficiency. DevOps is all about doing things in a more efficient, smart and cost-effective way by adopting the latest innovations, working across siloes to combine the strengths of different people on the team, and bringing different functions together to make a more responsive and effective DevOps organization.

In the same way, today’s IT operations analytics solutions are becoming more friendly to this spirit of collaboration, as infrastructure is becoming more complex and heterogeneous. DevOps teams have to work together in new ways, with different processes and different parts of the organization interacting in unique ways that might not have been expected just a few years ago. Similarly, today’s IT operations analytics world is evolving to run IT more smartly by taking advantage of new opportunities to manage the stack more efficiently.

Ultimately, the role of DevOps and the role of IT operations analytics are about developing competitive advantage. If your organization can use DevOps principles to build packages or release code faster or more accurately, you’re going to be ahead of the game compared to your competitors. If your organization can harness the power of the latest IT operations analytics innovations, you will similarly have a competitive edge because you will have a higher degree of control and visibility into what’s happening within your infrastructure. IT operations analytics can serve as an ideal complement to DevOps – because it helps to reinforce and protect the competitive advantage gains that are realized by DevOps.

Here’s what I mean by that statement: DevOps does a great job of combining speed, automation, and iteration. But as a practical issue, creating competitive advantage is more than new features, the apps you deliver must be reliable and high performance or customer expectations will not be met. IT operations analytics serves as a complement to the speed and agility of DevOps by identifying new innovation opportunities and providing “wow!” moments for the customer. With better IT operations analytics, you can exert better control over your IT infrastructure – and better control leads to bigger innovation opportunities, because you can keep the control in-house without having to rely on an outsourcer for identifying your innovation opportunities.

There are several market segments where the link between IT, the customer experience, and revenue are quite obvious. Optimizing online game play using real-time analytics is a great example, but the trend is much broader than that. With so much of the customer experience determined by IT systems, we are seeing more and more large customers moving to: (a) Continuous Integration and Continuous Delivery to accelerate delivery and (b) IT operations analytics as the way to ensure service delivery. IT operations analytics helps create the stable foundation for the high-speed and dynamic innovations of CI/CD.

With DevOps opening up an unprecedented level of speed, it’s important to have the right tools in place to process significant volumes of data. This is another area where IT operations analytics serves as an ideal complement to DevOps – because today’s IT operations analytics tools are powered by Big Data and machine learning, making it possible to achieve new levels of scalability. Unlike just a few years ago when getting a Big Data system up and running required significantly more resources, companies can get a lot more Big Data bang for the buck today than was possible even 3 years ago.

Now, companies are focusing on the analytic applications on top of Big Data, which is how businesses will find the optimization and innovation opportunities. Companies like Netflix are leading the charge in how Big Data can be used, not only to ensure service levels but also to create great customer experiences. While customer data may be the “crown jewel” of a corporation, IT operational data is the “diamond in the rough” that companies will be mining for the next big wins. We are just barely scratching the surface of what is possible with Big Data analytics of IT operational data – by drawing upon a fuller picture of what’s happening within the stack (various applications, servers, shared services, etc.), today’s IT operations analytics tools are going to help IT managers identify new insights and innovation opportunities that would not have been possible in the pre-DevOps era.

Another lesson of today’s reality of DevOps and IT operations analytics is that the divide that used to exist between customers and IT is almost gone. The future of IT is represented by the Internet of Things (IoT), where every system and behavior is monitored and analyzed in real-time, triggering operational processes from turning off a light to delighting a customer. Everyday objects, from household appliances to cars, are going to get exponentially “smarter,” and our IT processes need to evolve along with them to continually analyze, evaluate and respond to the ever-shifting ocean of data that is silently speaking to us. While DevOps and CI/CD define the trend to more rapid innovation, the collection and analysis of the reams of data generated by systems and customers will be the key to finding and maintaining competitive advantage.

This is where IT operations analytics will have an ever-more-important role as a complement to DevOps – with Big Data analytics and machine learning tools becoming more responsive and integrated into the life of overall IT operations, our IT admins are going to have more powerful insights at their fingertips each day. The convergence of IT operations analytics and DevOps will be a virtuous cycle of enhanced speed, smarter automation, and better control of IT operations for more reliable service and more delightful customer experiences.

About the Author/Omer Trajman

Omer Trajman, Co-Founder and CEO of Rocana. Omer’s fifteen years of front-line business and technical experience guide his vision for Rocana. Leading teams responsible for some of today’s largest modern database management system deployments, Omer has worked with customers and partners to identify where Big Data technology solutions address business needs. Omer’s experience in Big Data includes responsibilities as Vice President of Technology Solutions at Cloudera, focusing on Cloudera’s technology strategy and communication. Prior to this role, Omer served as Vice President of Customer Solutions at Cloudera, which included responsibility for Cloudera University, Cloudera’s Architectural Services and Cloudera’s Partner Engineering team. Before joining Cloudera, Omer was responsible for the Cloud Computing, Hadoop and Virtualization initiatives at Vertica. Omer received his Bachelor’s degree in Computer Engineering from Tufts University and was a visiting scholar at Oxford University reading in Computation and Electrical Engineering with a focus on large scale distributed systems.

August 19, 2015
How to “hear” OpenStack silent failures earlier and remediate faster
Troubleshooting “silent failures” in an OpenStack cloud environment has been a tough problem to solve for many production users. eBay’s chief engineer of cloud, Subbu Allamaraju, whose team has been operating PayPal on an OpenStack-based private cloud since March 2015, has publicly defined one of his ongoing challenges as: “Too much noise, too many alerts, and still working on how to detect troubles before customers find out.”

For enterprises and service providers in various stages of rolling out OpenStack, tackling silent failures invariably starts with two questions:
- Why and where do silent failures occur?
- Can my Ops team receive push notifications of warnings earlier and proactively remediate them before silent failures materialize?
Silent failures happen because OpenStack Ops teams often don’t know which service domains to tune into for early-stage warnings. Even if they do, the warnings get drowned out by noise. For instance, your metrics and logs for the compute domain show a successful VM creation. Yet, a tenant is still not able to use her requested VM instance – a silent failure – while the warnings actually show up in metrics and logs outside compute domain, such as DHCP agent failure and identity database query timeout.

To catch a situation like this, OpenStack Ops teams need a real-time, data-driven approach that automatically detects early warnings in DHCP agent and identity service, then correlates them with VM creation, and finally push-notifies only the relevant domain experts to collaboratively remediate “tenant can’t use requested VM.” All noise has been removed and related warning signals have been clustered and contextualized together – automatically.

And Operators shouldn’t have to rely on writing rules to model silent failures. Writing rules will be too time consuming and are unable to cover all failure scenarios. There are too many service domains – at least 13 of them (e.g. compute, identity, dashboard and storage) – plus their service daemons and virtual switch tunnels that stitch them all together.

Using real-time, data driven management to resolve OpenStack silent failures has been a recent success for OpenStack. For example, a large OpenStack production shop was able to proactively detect OpenStack cell failures by receiving early notifications from outside the cell domain – CEPH and JBOD I/O failures.

By becoming data-driven and ingesting all events and status messages across the entire OpenStack into real-time, machine learning algorithms, OpenStack Ops team automatically receive cleaned and contextualized alerts, then use modern social collaboration technologies to rapidly troubleshoot issues, capturing tribal knowledge for the future. The new mantra for these teams should be “clean, contextualize and collaborate,” the 3 pillars of real-time situation management.

But real-time situation management doesn’t just stop here. It scales across the entire ecosystem that lives in and around OpenStack. Data sources include the ever-pervasive RabbitMQ message bus across all services, metrics and logs across the diverse tenant operating systems, and metrics and logs across all technology and vendors. In essence, monitoring the entire technology and vendor independent reference architecture can be rolled up and simplified as “situation management” vs. reacting to many seemingly unrelated “domain silo alerts.”

Real-time situation management can also sit above multiple OpenStack instances for load sharing and DR purposes, and the interactions with public cloud. For instance, Amazon CloudWatch can be quickly incorporated as a data feed, giving OpenStack Ops teams a “manager of managers” view across OpenStack hybrid clouds.

Real-time situation management essentially provides Ops team with a single pane of glass to see and resolve failures earlier, with amplified signals and suppressed noise. Doing so without relying on extensive rules offers much needed flexibility for Ops team to keep up with so many “unknown unknowns,” as the OpenStack community continues to evolve its entire reference architecture and ecosystem.

About the Author/Feng Meng

Feng Meng is a Senior Director, Product Marketing at Moogsoft. His expertise lies in IT service assurance, with 15-year technical marketing background in Application Performance Management (Bytemobile, acquired by Citrix), Network Performance Management (Cisco), Infrastructure Monitoring (VSS Monitoring), and Server Management (HP). Twitter: @fengmengmoog @moogsoft Linkedin: https://www.linkedin.com/in/fengmeng1970
July 10, 2015
Nine Common Ops Mistakes (and How to Prevent Them)
Constant change is a reality for any growing, dynamic organization. Being receptive to change helps us quickly innovate and learn, yet at the same time, poorly-managed change can create instability and downtime. Ask any engineer what the number one cause of downtime in their systems is, and they’ll say change – software changes, network changes, configuration changes. While it would be nice to avoid this instability by working on completely static systems, to cope with the needs of modern technology businesses, operations teams must learn to better manage change by preventing common ops mistakes before they compromise systems.

Arup Chakrabarti, operations engineering manager and my colleague at PagerDuty, stopped by Heavybit Industries, a community workspace for developer-focused entrepreneurs, to discuss the biggest mistakes an operations team can make and how to head them off:

1. Getting It Wrong in Infrastructure Setup
- Creating Accounts
  A lot of people use personal accounts when setting up enterprise infrastructure deployments. Instead, create new accounts using corporate addresses to enforce consistency.
  Be careful storing passwords. Keeping them in your Git repository could require you to wipe out your entire Git history at a later date. It’s better to save passwords within configuration management so they can be plugged in as needed.
- Selecting Tools
  Another important consideration with new deployments is selecting your tools wisely. Leverage PaaS tools as long as possible so you can focus on acquiring customers instead of building infrastructure. And don’t be afraid to employ “boring” products like Java. Well-established, tried-and-true tech can let you do some really cool stuff.
2. Poorly Designed Test Environments
- Keep Test and Production Separate
  Don’t mingle your test and production environments. Be sure to set up test environments with different hosting and provider accounts than what you use in production. At the same time, make sure your test environments resemble production infrastructure as best as possible.
- Virtual Machines
  Are you performing local development? There’s no way around it: applications will run differently on local machines and in production. To simulate a production environment as closely as possible, create VMs with a tool like Vagrant.
3. Incorrect Configuration Management
- Infrastructure-as-code
  Essentially, infrastructure-as-code is the process of building infrastructure in such a way that it can be spun up or down quickly and consistently. Server configurations are going to pose problems regardless of where your infrastructure is running, so you have to be prepared to restore your servers in as little time as possible. Ansible and Chef are two tools that makes infrastructure-as-code deployment super-simple for ops teams.Whatever tool you use, as a rule of thumb, it’s best to limit the number of automation software tools you’re using. Each one is a source of truth in your infrastructure, which means it’s also a point of failure.
4. Deploying the Wrong Way
- Consistency matters
  Every piece of code must be deployed in as similar a fashion as possible. Standardizing deployment practices takes time and effort, but eliminating variability and the potential for human error is essential for success. And that applies to rollbacks, too— anyone with deploy rights should also be able to easily roll back any code they ship.
- Orchestrate your efforts
  Powerful automation software can certainly help enforce consistency, but automation tools are only appropriate for big deployments. When you’re getting started, Arup suggests running development using Git and employing an orchestration tool, such as Capistrano for Rails, Celery for Python or Ansible and Salt for both orchestration and configuration management.
5. Not Handling Incidents Correctly
- Have a process in place
  Creating and documenting an incident management process is absolutely necessary, even if the process isn’t perfect. Be sure to review the incident-management document on an ongoing basis, too. Incident response should be defined relative to the severity of the incident— a minor latency blip and a total downtime event have different response workflows and expectations.
- Put everyone on-call
  It’s becoming less and less common for companies to have dedicated on-call teams – instead, everyone who touches production code is expected to be reachable in the event of downtime. This requires a platform that can notify different people in different ways. What really matters is getting a hold of the right people at the right time.
6. Neglecting Monitoring and Alerting
- Start anywhere
  The specific tool you use for monitoring is less important than just putting something in place. At PagerDuty, we use StatsD in concert with Datadog; open-source tools like Nagios can be just as effective. For application performance management, you might also look at a tool like New Relic or AppDynamics. There are hundreds of great monitoring tools out there, but the important thing is to get something in place and make sure you’re alerting on the exceptions that matter to you.
- Check externally
  If uptime is important to you (and let’s face it, it always is), make sure you’re also running some sort of external check on your service or site, with a tool like NodePing or Ghost Inspector.
7. Failing to Maintain Backups
- Systematizing backups and restores
  Just like monitoring and alerting, backing up your data is non-negotiable. Scheduling regular backups to S3 is an industry-standard practice today, and always have one more backup method than you think you need.And backups are useless if you can’t restore from them! At least once a month, try restoring your production dataset into a test environment to confirm that your backups are working as designed.
8. Ignoring High Availability Principles
- “Multiple” is the key.
  Having multiple servers at every layer, multiple stateless app servers and multiple load balancers is a no-brainer. Only with multiple failover options can you truly say you’ve optimized for HA.
- Datastore design matters, too.
  With multimaster data clusters like Cassandra, individual nodes can be taken out with absolutely no customer-facing impact. Clustered datastores are ideal in fast-moving deployment environments for this reason.
9. Falling Into Common Security Traps
- Relying solely on SSH
  Use gateway boxes instead of SSH on your database servers and load balancers. You can run proxies through these gateways and lock traffic down if you suspect an incursion.
- Not configuring individual user accounts
  When an employee leaves your organization, it’s nice to be able to revoke his or her access expediently. But there are other reasons to set people up with user accounts to your various tools. Someone’s laptop may get lost. An individual might need his password reset. It’s a lot easier to revoke or reset one user password than a master account password.
- Failing to activate encryption in dev
  Making encryption a part of the development cycle helps you catch security-related bugs early in development. Plus, forcing devs to think constantly about security is simply a good practice.
These are just a few of the mistakes that operations teams can potentially make when managing change. There are millions more out there that you’re likely to run across, but these nine mistakes tend to be the most commonly seen, even at larger companies like Amazon and Netflix. By putting these best practices in place, you’ll help set your team up to effectively manage change.
April 23, 2015
Five essential Ops strategies that help make DevOps the real deal
With its roots in Agile and Lean thinking it’s not surprising that DevOps is often looked at through a development lens. After all, it’s proving absolutely essential to support accelerated delivery times; providing a much needed jolt of velocity to protracted development cycles. But of course with the increased need for speed there’s always the danger that quality can be compromised – so much so that faster time-to-value can very quickly become faster-time-to-failure.

In what seems like a byzantine age of enterprise computing, IT operations tended to call the shots when it came to delivering quality. Sure, a transactional system might suffer a glitch or two, but since constituents were largely internal and customer’s generally one step removed from the application and business process, the impact could often be controlled. Ah, so much for the good old days.

But as we know this model hasn’t just changed, it’s been completely disrupted.

Quality is now a different headed beast that bites hard in many places. It bites when customers engage directly with your brand during discrete mobile moments – like when they compare your goods in the shopping aisle with a mobile app, or assess car insurance based on your ability to provide usage-based policies from their in-vehicle software system. Yes, speed is important in the application economy, but it’s also a high quality customer experience that shapes the winners from losers.

The fabulous thing about DevOps is the fundamental premise that teams can collaborate in real-time to establish, maintain and continuously improve the quality of software systems and applications. For operations this doesn’t mean waiting for applications to reach production stage before enacting performance management processes. Rather, it involves instantiating best practices throughout the development cycle – that way teams resolve quality issues much earlier in development (where they are cheaper to fix), while also preventing a cumulative buildup of crippling technical debt.

So with application quality and customer experience being essential for DevOps success, what are some of the ops related strategies we can employ to support better business outcomes?

I see five essential, ‘rubber meets the road’ strategies ops can use to help DevOps become the real deal:
1. Focus collectively on the user experience – too often organizations become detached from business reality by looking inside-out from a silo’d technology and diagnostic perspective. DevOps practitioners put themselves in the shoes of the customers by constructing, visualizing and managing end-to-end user-centric service views that traverse functional boundaries. Additionally, user-experience management will involve gaining deeper insights into real-time performance of externally facing applications and mobile services, bringing to the forefront analytics to guide decision making and drive continuous improvement.
2. Give Developers a clear view of how code works in the real world – operational intelligence shouldn’t wait until applications are moved into production. Rather, it should be instantiated across every stage of the lifecycle – whether that’s during regression and unit testing, or pre-production. Many teams recognize this by for example activating monitoring during release automation; giving developers critical insights into both functional and non-functional code behaviors. Others go further, taking user-experience to another level by exposing key behavioral and usage-based intelligence – especially for newer mobile apps, APIs and digital services.
3. Create a single source of the truth using a common framework – to be fully effective, developers and operations should reference a single version of the truth when it comes to problem triage and resolution. Traditionally this has proved difficult with teams having discrete tools providing diagnostic insights into one aspect of technology. This however falls short due to the complex interdependencies across technologies and results in finger pointing and blames games when problems can’t easily be pigeon-holed. To address this, operations teams use a common unified framework that quickly discovers and exposes infrastructure and application configurations, and importantly the relationships and interdependencies that exist between them. Now, root-cause can be quickly established from either an application or infrastructure perspective, but always prioritized according to business and customer impact.
4. Define common goals across teams and use shared metrics – traditionally development and operations teams have been incentivized according to separate goals that are often diametrically opposed (e.g. speed of producing code –v- resilience and availability). By using shared customer and business-centric metrics, developers and operations build a ‘common language’ for evaluating progress and understanding the impact of change and continuous delivery. My advice here is quality over quantity; building out metrics that are actionable by all stakeholders. For example, this could include lead and cycle times to pinpoint operational bottlenecks and waste during deployments, or transaction response monitoring in pre-production to pinpoint code deficiencies and avoid technical debt associated with provisioning unnecessary capacity.
5. Converge around a common understanding to transcend organizational boundaries – a collection of diagnostic based monitoring and alerting provides little more than vanity metrics and feel-good green lights. Sure, they can alert operations to when something is broken, but they do little to drive the more essential element of DevOps success – feedback. Therefore common understanding involves operations automatically correlating performance information end-to-end so as to expose far more beneficial insights that developers can use to improve the code base.
At the recent Gartner Data Center, Infrastructure and Operations Management Conference in Las Vegas in December, I presented a session – Unified Infrastructure Management is Critical to Enhancing the Ops in DevOps – that further described these strategies together with supporting use-cases. It was certainly a dynamic and lively session, highlighting the viability of DevOps. As organizations look to increase speed to market of new applications while both lowering cost and increasing quality, DevOps will continue to gain traction, and I can’t wait till everyone truly understands its transformative power.

About the Author

Aruna Ravichandran is vice president of DevOps Solution Marketing and Management at CA Technologies, responsible for cross-portfolio solution marketing for DevOps across all product lines. Before joining CA Technologies, Aruna held several management positions in product marketing and engineering at Juniper Networks and HP. Aruna earned an MBA and master’s degree in Computer Engineering from Santa Clara University and holds a bachelor of science degree in computer science from Bangalore Institute of Technology. Ravichandran is on twitter @aruna13 and on linked in
January 9, 2015
Removing the Wall Between Dev and Ops
Welcome to the new devopsy.kinsta.cloud Q&A series on Enterprise DevOps. In the first post in this series, I answered five key concerns for adopting a DevOps approach in a large enterprise. Now, I want to address a more specific question from an admin at a large enterprise about the silos and barriers between dev and ops …

Q. How do you work with customers to remove the imaginary wall between dev and ops to create that DevOps environment?

This is perhaps the core of a DevOps transformation – dev and ops teams working closely together toward common goals, rather than in isolated silos. I have seen several methods have substantive impacts on these entrenched silos, so I will take you through some of the more effective approaches.

Familiarization

In many organizations, dev and ops barely know each other – a huge barrier to the empathy at the centre of devops. Breaking this wall down may be as simple as getting both departments together for some social activities. Some successful approaches I have seen include:
- Sponsor a one-off or regular shared non-work team activity, like paintball, a pool tournament, team picnic, LAN game, or cooking classes. Bring both teams together in a non-workplace environment to work together. Remember to mix up the teams – you do not want to reinforce any us vs them mentality;
- Schedule sessions for team members to present to each other in short sessions (think lightning talks like Ignite or TedX) beyond technology to any areas of interest – like ‘My Sports Team,’ ‘Steampunk in the 21^st Century,’ ‘Restoring Train Carriages,’ or ‘Strategies for TF2 Domination.’ Share common interests to create empathy for each other as people, not just functionaries;
- Initiate informal knowledge sharing, with regular ‘lunch-and-learn’ or ‘brown bag’ sessions, where team members can explain what their role is, why they do it, how they do it. Offer a ‘Day in the Life…’ presentation, a Q&A session, or a whiteboard over pizza and sodas and your technical staff will come out like vultures, ravenous for … knowledge!
Co-Location

Telecommuting has many benefits, but standup meetings are hard to achieve across five time zones, ten locations and six languages. People communicate better in person, so physically co-locate teams as we at CA and others have done, to enable them to connect and talk in person, not just via e-mail or IM.

Geo Alignment

In multi-national enterprises central developers often push code to global operations, reinforcing the silo mentality. Align local devs and ops with local business users instead to facilitate customer-centric agile development, and enable smaller, closer, more agile teams with more immediate shared goals.

Secondment

Consider seconding team members into the other team e.g. for one day a week, one week a month, or one month in six. When ops learn firsthand the basics of their dev counterparts, and vice versa, they will better internalize how to work together, and the impact of each role on the overall process.

Restructuring

Some enterprises have combined dev and ops into one team. While often seen as an ‘anti-pattern,’ for some organizations this is working well. Others have adopted a flexible org structure with hierarchical management for administration (reviews, goals, training, etc.), but loose teaming for functional duties.

Shared Technologies

Beyond cultural and attitudinal shift, technology can also help. A shared toolchain for the end-to-end service delivery lifecycle can bring teams together too. Common tools for planning, testing, release, monitoring, or triage will help break down barriers with a common terminology and shared experience.

Other Approaches?

While there are no silver bullets, these are some ways I have seen enterprises break down these silos. But perhaps you have seen or been a part of other approaches? If so, other readers and I would love to know more, so please take the time to leave a note in the comments below, or drop me a line directly.

And remember, if you have questions about Enterprise DevOps, let me know and I will try to answer them for you. If I can’t answer them myself, I will find someone who can! Please leave your questions in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.
August 4, 2014