devopsy.kinsta.cloud

Category: Blogs

DevOps the movement, or the function?

All the ISVs and Enterprises are having the same question, who/what is DevOps. We get in a virtual room and talk about it, but we are often all scratching our heads at the same time. We are not confused from a technology or definition standpoint. There is so much chatter about this thing, it’s just hard to find the person or group that is doing it.

My response to when I’m asked about DevOps is always the same. Are you are talking about DevOps the movement, or the function? The movement is what we are all hot and bothered about, but the function is where it really comes together and problems are solved. And even if you hate the movement, I guarantee your dev and operations team are inching in that direction, and some portion of the work already being done is in the DevOps spirit. (more…)

August 12, 2014
The DevOps Diet (Getting Lean)
IT has gotten fat – fat silos, fat processes, fat procedures. Adopting DevOps is a great opportunity to look at some of our unhealthy habits and identify where we can eliminate unproductive waste from our process diets. DevOps will ultimately teach IT how to exercise more with less effort, trim the fat and get lean.

According to lean principles, the seven main areas of waste are:
- Defects – variations from requirements that result in interruptions and re-work
- Overproduction – delivering something more or before it is required
- Inventory – carrying excess raw materials, work in progress (WIP) or finished goods
- Over-processing – doing more work than is required
- Motion – moving people or equipment more than is required
- Transportation – moving products from one location to another
- Waiting – doing nothing or moving slowly while waiting on a previous step
What are the greatest sources of waste in your IT organization? Are defects being regularly passed downstream? Are ITSM processes overproducing through complexity and bureaucracy? Are Agile teams dealing with excessive work in progress and less finished product? Are unresolved bottlenecks resulting in frequent delays while waiting for something to get done or someone to be available? Could automation reduce “motion sickness” and make transportation nonstop? These questions will hopefully facilitate dialog between Dev and Ops about analyzing and streamlining existing workflows.

While waste is fattening, it is not necessarily ineffective. Projects and other tasks are still getting done and therefore some areas of waste may be difficult to recognize. To become leaner, start simple by identifying and eliminating one unhealthy habit from your IT diet. Engage stakeholders from Dev, Ops, customers and suppliers (if appropriate). Discuss ways to improve the work with healthier habits. Agree on what to do next or what not to do anymore.

Every diet should be supported by recipes for success. The upcoming DevOps Cookbook by Gene Kim, et al. will provide more tangible guidance and recipes for healthy IT habits. More on that in future blogs.

Most of all, identifying and eliminating waste through Lean DevOps is a great opportunity for collaboration. After all, it’s always better to diet with a friend.
August 11, 2014
Enabling Collaboration Across Diverse Cultures

Welcome back to this series, ‘Enterprise DevOps Q&A.’ In my last post, I talked about breaking down the barriers between dev and ops. Now, I want to address another, very typical question from a global enterprise about managing cultural differences in a globally dispersed enterprise.

Q. Being part of a large org, our biggest issue seems to be a global culture clash. How can we address the issues of geographic location and culture that are a big obstacle for us?

Large multi-national enterprises, like CA Technologies too, often have major data centers in multiple geographies in North America, EMEA, Asia and Latin America. Despite so much data center consolidation and cloud migration, cross-geo cultural issues remain a major issue.

For example, in Israel the holy day is Saturday, while in the US and Europe it is Sunday, so it can be hard to lock down a release window. In Japan it is common to work long, late, and on the weekends, while by contrast some French labor agreements specifically limit the workday, so ‘crunch time’ may not have the same meaning in both locales. In the USA, staff may only get ten days annual vacation, and not even take all of it, while much of Europe takes all of July or August off, so consistent teaming can be a challenge.

To help you address these geo-cultural challenges, here are some approaches I have seen my customers adopt including:

A local approach

A local approach, such as co-locating devs and ops to service business users in a common geographic or cultural area allows teams to adopt DevOps practices within cultural norms. This may mean imposing a difficult structural change, but it can also be very effective. See my last Q&A for more on this approach.

A ‘core team’ approach

Some businesses have assigned one or two team members from each cultural area to represent their teams in a common virtual or ‘core’ DevOps transformation team. Representatives on the core team can translate (sometimes literally) for their ‘home’ teams, and bring unique culture back to the core team. This can raise process/hierarchy barriers, but when done well can also help to break them down.

A vertical approach

I have seen a handful of global businesses implement DevOps vertically, with devs and ops teams, often geo-located, acting as service providers to individual business units, rather than the whole company. This is best suited for organizations that are already organized by business unit, but it ensures teams have common focus on specific and shared business problems.

A ‘known good process’ approach

Speed the flow of software delivery by focusing on refining and documenting specific processes all teams can rely on. Each geographical or cultural team can define and share their own ‘known good process’ (cf. the Visible Ops handbooks), and share the interfaces to those processes, so they can continue to optimize their own activity, but work seamlessly with other teams at handoff points.

A process automation approach

Even better is if each team can encode their processes in automation to make it possible for anyone, anywhere, using any language, to execute those processes remotely. Even small improvements matter, so even task automation can help get closer to a cross-cultural, ‘enterprise-wide,’ DevOps approach.

A ‘pace-layered’ approach

Different business services have differing levels of acceptable risk and tolerance to change. Mobile app dev, for example, is more tolerant of risk than, say, accounts receivable. Rather than changing the world (literally!), instead build a DevOps approach into a specific service delivery team, even in a single geography. This ‘pace-layered’ approach is working very well for some of the world’s largest enterprises.

A tactical approach

Similarly, you do not need to fix all your service delivery problems at once. In a global enterprise, pick one big winner at a time instead. Identify constraints in the test process, a roadblock in QA, a provisioning delay, a problematic release process, a monitoring gap, or a broken feedback loop, and connect cross-cultural staffs from similar functional teams to solve specific shared issues.

A technology-assisted approach

Regardless of the approach, technology can assist. Global tools with multiple language support can enable common experience and terminology. Telecommuting can give regional staff a virtual presence in team meetings. At CA Technologies, we are trialing remote-controlled robotics to give staff a physical avatar in meetings from anywhere across the globe.

Other approaches?

As with most of DevOps, there is no definitive prescription, and I am sure there are many other approaches that can work. If you have ideas or experience to share, please post in the comments below, and let us all know how you have or could address these cultural and geographic barriers.

And remember, if you have questions about Enterprise DevOps, let me know and I will try to answer them for you. If I can’t answer them myself, I will find someone who can! Please leave your questions in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.

August 11, 2014
A DevOps Approach to Delivering Frequent, High-Performing Releases
We live in an application-driven world. Applications power businesses, provide connectivity, and help fulfill goals and customer needs. In a competitive world, needs evolve quickly as our users demand more features — all the while expecting around-the-clock availability and reliability, whether they use smartphones, tablets, or computers.

Keeping Up With User Demands

As IT folk — irrespective of whether we are in development or operations — we need to deliver new apps and features frequently to satisfy our users’ needs or else we risk losing them to competitors or alternatives. What do we do to stay ahead? We release updates faster by:
- Organizing around small teams and delivering more agile releases
- Automating where possible (test and release processes)
- Deploying updates into production when available
However, the unfortunate truth is that frequent changes often cause errors and instability in production. We need to mitigate risk by taking a close look at our processes and practices before we hit the gas.

Preparing for fast and frequent application releases

Here are two issues (amongst many others!) that hold us back from delivering frequent updates, while ensuring reliability:
- Tedious release processes: Developers have a multitude of collaboration tools, Kanban boards, build tools, and agile practices, and can build features quickly. But once it comes to releasing these changes through QA, staging, and production, the process is more manual, ad hoc, and slow.
- Unreliability in production: Code that runs as expected in development sometimes falls short in production. Production environments are subject to limitations imposed by third-party services, regional ISPs, firewalls, and so on. They are often quite different from development and QA environments.
Collaborate on a singular goal

Development and operations teams come from different schools of thought. Devs push for change and ops drive for stability. While they both aspire to offer superior software that adds value to users, their goals are diametrically opposed! What we need is a culture where development and operations teams draw from each other’s experiences, perspectives, concerns, and continuously improve processes to build and deliver resilient applications rapidly, into the hands of their customers. This requires discipline, a sense of mutual respect, focus on the big picture, and a strong commitment toward delivering on business goals.

Streamline and automate releases

Developers enjoy streamlined processes with their collaboration and build tools to produce releases effectively. However, the operations folk do not share the same good fortune. Quite often, they get artifacts, scripts, and resources from different members of the team, and run them in sequence in order to deploy and test the application. This lack of coordination and automation greatly slows down the release process. Here’s how you can begin to address this issue:
- Critique your processes: Where do the dependencies lie? Are there any redundancies that can be consolidated? Where are the bottlenecks, what’s causing them, and how expensive are they? Start with small opportunities that can bring big improvements.
- Automate: Once you have optimized release processes, look to automating them. Time and effort saved from repetitive tasks can be put toward innovating and adding value.
Once released, monitor for quality

By automating steps within the release pipeline, you set yourself up for increasing release velocity while ensuring optimal quality. However, no amount of preparation can guarantee expected results in production. Elements out of your control — like network traffic, third-party services, local ISPs, or browsers — can potentially impact user experience or disrupt service. Many of these outages have been very public. What we need is a strong feedback loop with every release that measures metrics around performance, exceptions, and, most importantly, end-user experience. Watch out for subtle changes in performance Subtle changes can alter your application’s end-user experience, costing you customer acquisition and brand loyalty. It is death by a million pinpricks. Slow applications, JavaScript errors, and 404s are very effective in turning users away. Here is where application performance monitoring tools — like Riverbed SteelCentral™ AppInternals — help diagnose problems that degrade application performance and behavior. They monitor end-user experience from your users’ devices, trace transactions through the application tiers and infrastructure, and diagnose performance problems down to application code. These insights gives developers the visibility they need to quickly fix problems that occur in production without going through the painful process of recreating and analyzing them.

Monitor end-to-end application performance continuously

Monitor your application for performance and behavior in production. Make this an ongoing activity during peak and off-peak hours alike. A keen insight into your application and infrastructure health can reveal snags that would otherwise go unnoticed. You really want to look for symptoms that compromise end-user experience, such as:
- How are my servers and infrastructure performing at current capacity?
- How are users in one location faring compared to others?
- What parts of my application — internal or third party — impact performance?
Diagnose problems with performance monitoring tools

Diagnose where and why these symptoms occur. Are they because of poor code, network conditions, server capacity, or third-party services? What is the context in which these issues occur? Which code segment, SQL query, or app component is at fault? You will be able to package up everything the development team needs to fix the problem and prevent it from recurring. Many of these tools, like AppInternals, allow developers can view this information from within IDEs like Eclipse and Visual Studio.

In conclusion, aim for continuous improvement

Particularly in the case of rapid release cycles, constant monitoring is a necessity. Identify symptoms in runtime — in QA and production — and quickly fix them using diagnostic data captured by your tools. This will keep the backlog low, your IT team productive and focused delivering new features, and your customers happy.
August 8, 2014
DevOps with purpose: It’s about your applications

This is Part 1 of a series on DevOps with a purpose by Matthew Selheimer, Chief Technical Evangelist and SVP of Marketing at Houston-based ITinvolve.

Unless you’ve been living under the proverbial rock these days you have no doubt heard about DevOps and the groundswell building around it. Today DevOps is being defined, promoted, touted and discussed as the next game changer in IT. Rather than argue about the best way to define DevOps in a general sense, this article will focus on something hopefully more useful – how do you determine what DevOps means for your organization?

A common goal cited for DevOps is to enable faster release and deployment cycles by taking advantage of agile development methodologies; improved collaboration between business stakeholders, application development and operations teams; and automation tools. Beyond these elements, DevOps also requires a cultural acceptance of the need to focus on the flow of work across teams vs. simply optimizing individual teams and their specific units of work.

By contrast, defining DevOps with a purpose means ensuring how you do DevOps is grounded in the realities of your specific organization. There is no “one-size-fits-all” DevOps, no matter what pundits, consultants, and vendors might tell you. Start by actually taking a step back. Ensure you are clearly taking into account your industry, your applications, your culture, and your people when developing your DevOps strategy; then apply DevOps principles against that foundation. Because every organization’s purpose will be different, your way of doing DevOps will not be the same as anyone else, even among your industry peers.

There are, however, some general guidelines you can start with and then apply them to your specific situation. For example, I recommend analyzing your business goals and application portfolio using the pace-layering method, which focuses on three categories of applications – systems of record, systems of differentiation, and systems of innovation.

Systems of record are usually quite stable with infrequent updates. This is often due to regulatory requirements and internal policies for theyse systems, and they tend to have a high degree of consistency within an industry and even across industries. Systems of record are often good candidates for waterfall development methodologies and longer release timelines. Your general ledger or payroll systems are good examples of systems of record.

Systems of differentiation are those applications that are likely common to your industry but where you have an opportunity to differentiate your company from the competition based on functionality and the pace with which you update them. For example, if you are a financial services firm that provides retirement investment programs, you might make changes to your customer portal and how clients select new investment options on a quarterly or monthly basis in order to enhance customer experience and remain competitive and differentiate. If you are an airline, this might be your crew scheduling application, which ensures you have the right flight crews available when needed and can improve your on-time departure percentages to differentiate from the competition. In healthcare, your systems of differentiation might be the applications that health professionals use to store patient information and correlate test results to aid in diagnosis and treatment. In manufacturing it may be your supply chain management, process control, or shop floor applications.

Systems of innovation are those truly groundbreaking applications that help you create new markets and revenue streams. In financial services, that might be a new kind of application that gives day traders a market advantage. For an airline that might be a whole new way of providing inflight entertainment, such as when Wifi and inflight entertainment first became available to passengers. In healthcare, it might be a new customer portal that promotes wellness services and reduces office co-pays by uploading fitness tracker data. For a retailer, it might be applications that support a new type of customer loyalty program, such as when Amazon introduced their Amazon Prime service a few years ago.

Both systems of differentiation and systems of innovation are excellent candidates for agile development and DevOps principles. This is because they require high degrees of collaboration between business stakeholders, developers, and operations personnel to ensure the applications are in line with requirements, developed and tested quickly, and then made available in the market to drive differentiation and innovation. They are excellent candidates for the continuous delivery concept in DevOps where small releases are deployed quickly (and can be rolled back just as quickly if there are issues). In this manner, a large number of small deployments moves the competitive needle faster and provides greater value to customers over traditional waterfall development and disconnected development and operations practices.

Defining DevOps with a purpose starts with an understanding of your industry and your applications. In the next article in this series, we’ll turn to the topic of organizational culture and its impact on DevOps transformation.

About the Author

Matthew Selheimer is Chief Technical Evangelist and SVP of Marketing at Houston-based ITinvolve. Reach him at matthew@itinvolve.com

August 7, 2014
Now that you can consolidate, do not implode

The development world is changing in a big way. We all know it. And it usually happens first with the early stage startups, who have the luxury of adopting new approaches from day one. One of those major shifts is consolidation of team members, where automation is taking over previous roles. In the modern development shop for example you are not just a front-end developer, you are a data scientist, a tester, a orchestration master, and a release manager. There are a lot of upsides to this consolidation, but a lot of downsides as well.

(more…)

August 7, 2014
New CEO wants to take MongoDB where NoSQL has gone before

MongoDB, the leading NoSQL database company that has raised about $220m in venture funding so far, announced a change at the helm today. Max Schireson who has been with the company for 4 years, first as President and then about two years ago taking over as CEO from founder and chairman Dwight Merriman, is stepping up to be Vice Chairman of the Board. Taking his place as CEO is industry veteran, Dev Ittycheria (pronounced Dave).

Ittycheria was the founder and CEO of BladeLogic which had a successful IPO and then was acquired by BMC. Ittycheria then served in an executive role there before moving over into the VC side of things. He was involved with Greylock Partners and OpenView Venture Partners. Many industry insiders thought that he would not want to come back to an operations position, but the opportunity with MongoDB was too hard to pass up. The next step for the company is probably an IPO which would say investors start cashing out on the hundreds of millions of dollars they have invested.

Schireson posted on his own personal blog that he was “stepping out and up” for several reasons, not least of which is the fact that he has 3 young children and being away from home so much was painful. Max’s family is based in Silicon Valley where is wife is a physician and Stanford professor. MongoDB is headquartered in NYC. So the bi-coastal routine plus regular CEO duties had Max logging airlines miles at a 300k annual clip. That is a tough number to live with for an extended period of time.

I spoke with Max today about the news. I have been spoken with Max several times over the years about MongoDB and knew how instrumental he was in the companies success. But Max realizes that MongoDB is about to move into another phase. While the growth has been phenomenal, a different skill set is now required for this next jump to warp speed.

This same lesson was taught to me many years ago by a man named Len Fassler. You may never have heard of Len, but Len was one of the smartest men I ever met in business. He had done multiple roll ups in several industries. My friend Brad Feld counts Len as a mentor. Len actually bought Brad’s first company, Feld Technologies. Anyway, Len used to tell me that anyone can get lucky and catch lightning in a bottle, starting a company and building it up to a $4 to $6 million dollar run rate. But it takes a whole different skill set to build and manage a team that can take a company to 25 to 35 million dollars in revenue. To make the next leap to lets say $100m in revenue takes a whole different set of skills. To go beyond that yet another set of skills involved. It is rare that one person has the spectrum of skills to take a company from near start up all the way to billions of dollars.

I told Max today when I spoke to him, bravo to him for recognizing this fact without making MongoDB and its investors pay the price. He has done a tremendous job making MongoDB a near household name in the tech world. Now it is up to Dev Ittycheria to take the captains chair and steer this star ship to its next destination.

Opportunities to lead a company with the trajectory of MongoDB are rare indeed. Schireson believes that the future is even rosier though. He thinks MongoDB may be just hitting a place where revenue starts to hockey stick up. The foundation is laid for good things to continue.

It will not be easy, it never is. MongoDB still has to overcome some growing and scalability issues. There are other NoSQL and other databases gunning for them. But I would rather be playing their hand of cards then any of their competitors frankly.

In the meantime congratulations to Max for a job well done at MongoDB. Another thing I learned from Len Fassler was that building any company into a success is not easy and anyone who does so should be congratulated. So Max be proud of all you have accomplished here.

Now lets see where Captain Dev can take MongoDB on the next phase of its 5 year mission to boldly go where NoSQL has gone before!

August 6, 2014
Change your thinking about Change Management

Change Management, let’s face it, it’s usually a checklist item and a CYA tool. But in the world of DevOps where change is part of the culture and processes. Change Management needs to be more than a cost center, and really a way to improve what you are doing. And there is no excuse not to.

(more…)

August 5, 2014
Enterprises, DevOps and Economies of Scale

In infrastructure there are two economies of scale: operational and capital. DevOps should attend to the former, especially in the enterprise.

Soon after cloud burst (sorry, pun not intended) onto the scene we all learned about economy of scale because that, primarily, is what cloud computing’s business model is based on. At great volume, standardization can be used to achieve greater output and, thus, can cost less.

But there’s a second economy of scale at work with cloud; the operational economy of scale. If cloud providers scaled operations in the same way that enterprises traditionally scale operations, well, instance prices would be much, much higher.

What cloud providers recognized is that in order to achieve the economy of scale required to make cloud a success they had to address both the operational and capital economies of scale.

According to research, surveys and studies, operational overhead consumes anywhere from 63% to 80% of IT budgets. That’s just maintenance, management and general “keeping the lights on” kind of things. And each time a new application is pushed into production, that number gets bigger because there are more of everything – from storage to compute to network – to maintain.

To wit, Computer Economics “IT Spending and Staffing Benchmark 2014/2015” indicates that 67% of organizations were increasing their operational expenditures versus only 20% that planned on a decrease. Because IT budgets are generally determined as a percentage of revenue, that means operational budgets are chewing up the business, slowly but surely, and leaving little room for becoming an enabler of business rather than an application assembly line.

The holy grail, as its been described, for IT is to move budget from operational overhead to innovation. To do that means somehow magically reducing the costs associated with operations. Adding more people doesn’t help; that just adds more overhead. The goal is to do more with less; to increase the ratio of systems and services and applications per operator that can be managed. That’s operational economy of scale at work.

To do that, enterprises need to look to DevOps and its leverage of automation tools to enable leaner, more efficient operational processes. That means not just automating, but enabling a self-service model in which common provisioning and deployment processes can be orchestrated at the behest of the app or business owner. It means applying the same “gold image” standard used for provisioning and managing virtual machines to infrastructure; leveraging templates to gather the requisite input from app or business owners and automatically provisioning systems.

Limited variations on an infrastructure deployment template will enable customization where necessary, but otherwise provide standardized (stable) provisioning and deployment processes that can be self-service enabled. By empowering developers, app and line of business owners to provision application required infrastructure services, operations can use technology to scale to meet demand while lowering costs.

The goal of DevOps in the enterprise is not to enable 10 application deployments a day. It’s to achieve operational economies of scale and a more consistent, predictable and repeatable operational environment.

August 5, 2014
Removing the Wall Between Dev and Ops
Welcome to the new devopsy.kinsta.cloud Q&A series on Enterprise DevOps. In the first post in this series, I answered five key concerns for adopting a DevOps approach in a large enterprise. Now, I want to address a more specific question from an admin at a large enterprise about the silos and barriers between dev and ops …

Q. How do you work with customers to remove the imaginary wall between dev and ops to create that DevOps environment?

This is perhaps the core of a DevOps transformation – dev and ops teams working closely together toward common goals, rather than in isolated silos. I have seen several methods have substantive impacts on these entrenched silos, so I will take you through some of the more effective approaches.

Familiarization

In many organizations, dev and ops barely know each other – a huge barrier to the empathy at the centre of devops. Breaking this wall down may be as simple as getting both departments together for some social activities. Some successful approaches I have seen include:
- Sponsor a one-off or regular shared non-work team activity, like paintball, a pool tournament, team picnic, LAN game, or cooking classes. Bring both teams together in a non-workplace environment to work together. Remember to mix up the teams – you do not want to reinforce any us vs them mentality;
- Schedule sessions for team members to present to each other in short sessions (think lightning talks like Ignite or TedX) beyond technology to any areas of interest – like ‘My Sports Team,’ ‘Steampunk in the 21^st Century,’ ‘Restoring Train Carriages,’ or ‘Strategies for TF2 Domination.’ Share common interests to create empathy for each other as people, not just functionaries;
- Initiate informal knowledge sharing, with regular ‘lunch-and-learn’ or ‘brown bag’ sessions, where team members can explain what their role is, why they do it, how they do it. Offer a ‘Day in the Life…’ presentation, a Q&A session, or a whiteboard over pizza and sodas and your technical staff will come out like vultures, ravenous for … knowledge!
Co-Location

Telecommuting has many benefits, but standup meetings are hard to achieve across five time zones, ten locations and six languages. People communicate better in person, so physically co-locate teams as we at CA and others have done, to enable them to connect and talk in person, not just via e-mail or IM.

Geo Alignment

In multi-national enterprises central developers often push code to global operations, reinforcing the silo mentality. Align local devs and ops with local business users instead to facilitate customer-centric agile development, and enable smaller, closer, more agile teams with more immediate shared goals.

Secondment

Consider seconding team members into the other team e.g. for one day a week, one week a month, or one month in six. When ops learn firsthand the basics of their dev counterparts, and vice versa, they will better internalize how to work together, and the impact of each role on the overall process.

Restructuring

Some enterprises have combined dev and ops into one team. While often seen as an ‘anti-pattern,’ for some organizations this is working well. Others have adopted a flexible org structure with hierarchical management for administration (reviews, goals, training, etc.), but loose teaming for functional duties.

Shared Technologies

Beyond cultural and attitudinal shift, technology can also help. A shared toolchain for the end-to-end service delivery lifecycle can bring teams together too. Common tools for planning, testing, release, monitoring, or triage will help break down barriers with a common terminology and shared experience.

Other Approaches?

While there are no silver bullets, these are some ways I have seen enterprises break down these silos. But perhaps you have seen or been a part of other approaches? If so, other readers and I would love to know more, so please take the time to leave a note in the comments below, or drop me a line directly.

And remember, if you have questions about Enterprise DevOps, let me know and I will try to answer them for you. If I can’t answer them myself, I will find someone who can! Please leave your questions in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.
August 4, 2014