devopsy.kinsta.cloud

Category: Features

DevOps & Continuous Change

A remark by a colleague while waiting for the coffee machine to complete its cycle started my train of thought. “Should we have multiple minor releases or just do a few major ones in a year?”

In large organizations, due to many factors, the turnaround time for a single successful release is quite extensive; but if the conceptual change which we are talking about bringing whilst using the #DEVOPS methodology would not only reduce these long cycles but deliver quicker and quality software.

I have already discussed in my earlier article “DEVOPS: Getting our organizations aligned” on the traditional approach and how the combination of this two (DEV & OPS) has been proven beneficial in various organizations and how many are still hesitant in embarking on that path.

Continuous Change which encompasses both Continuous Integration and Delivery is paramount to achieving this objective. To attain the objective of delivering rapid and consistent value to the stakeholders regularly, we have to have Continuous Change happening.

I am sure we have already moved out from the Mainframe era to a more Distributed systems state. But unfortunately we have yet to let go of that culture where we pile up components on a single quasi Distributed Application, instead of creating smaller independent, yet interdependent systems.

So, back to the same question; shall we go with a few Dinosaur sized, labored releases or make multiple short and swifter runs? In my opinion, we should strive towards the leaner model. Easier said than done you say! Agree. If I have to, then I would go about it this way:

In an environment having a suite of Distributed Applications, start with Analyzing the components which make up the Major release, List out the changes (fixes and new features) which are planned, asses functional and technological impact, Identify dependencies, upstream/downstream connectivity’s, architectural deficiencies, list out the identified subset of components or group of components by functionality, which can go as an independent release.

I am resisting the urge to say ‘standalone’ as with the current set of complex intervened collection of applications, it would be an incorrect statement. It would rather be a subset of components which can be independently upgraded. Now the big task is to review and decide if this subset or group would add functional value to the overall suite of applications without the rest of components going in.

Once identified, the success of this release working as expected would be in the extensive testing (Unit/ Regression/Performance) of these components, and as we gain confidence in this split and deploy procedure, we can start with, prioritizing and scheduling critical must have features or bug fixes which also should be backward compatible with other components lagging behind. We can even come up with scenarios where these components are not just getting updated frequently, but would also have multiple independent tracks. Of course it goes without saying the testing and signoff process is strictly followed albeit in a shorter window.

Deployments to production have always been a source of heartburn to the OPS teams, we have often seen finger pointing and firefights to contain change fails. In order to maintain stability and maximize availability, stricter controls and reduced deployment windows are put in place. In promoting frequent and smaller component level changes, OPS will also be in control of what’s going in, and smaller component level releases can be rolled back quickly within the Green-Zone window if the change is not working as expected. Though I would categorize this rollback, as a process fail than a deployment fail, and should be reviewed very seriously with right mitigation plan and lessons learnt retrospection, Continuous learning after every successful or failed sprint ensures competent and confident releases.

Another major factor impacting multiple runs of Production releases is the long Green-Zone windows, by moving to the quicker and leaner deployments cycles, these requests can be substantially reduced, and can eventually aim to reach a stage where changes are deployed online without extensive downtime.

Similarly, frequent maintenance activities when there are no planned deployments, effectively has the same risk on environment unavailability. Analyzing and moving to a state where these activities are conducted online without a Green-Zone would alleviate the maintenance downtimes and ensure seamless availability, adding stakeholder value.

I have not touched on the Continuous Integration or the Automation of the deployment process for obvious reasons, we can’t achieve the objective of Continuous Change without having the Integration process in place where incremental changes to code are build into a package swiftly and seamlessly, even a sanity or basic testing might be integrated to ensure the quality of the builds. Same goes for automating the deployments, develop & integrate tools required for rapid deployments. We have seen instances where automating a single activity or revisiting a process has created substantial time savings.

While it’s easy to propose changes to the Release Process or advocate rapid deployment cycles, we have to ensure adequate checks and balances are in place, compliance adherence and audit trails are created.

Now to the most important measure of this article, Cost Savings!

Proponents of the Dinosaur Releases would argue that the common activities which entail a single release like the builds, Testing, Release Management, OPS availability or reduced GZ would cut costs and multiple runs to PROD would logically increase costs.

But it’s definitely the other way around, we wouldn’t need to feed the Dino anymore, it will be a cost worthy rapid delivery with optimized resources, a leaner process where DEV and OPS are working together to ensure quality value added deliverables are made available to business faster.. This cultural change and collaborative effort also creates new vistas of growth and provides a perfect platform to excel. The cost benefit analysis would eventually be in favor of Continuous Delivery.

Opportunity cost is another variable which has not been considered; an innovation or a new feature available to customers quickly with reduced time to market cycle will provide the much needed edge in these competitive times. I recall some conversations where a few must have’s were pushed out due to various reasons, frustrating the business.

DEVOPS was never meant to replace the traditional approach to software development; rather it is an efficient use of the existing resources within a collaborative model to rapidly deliver quality software products. A major cultural shift in our approach is required and to reach that goal we need to embrace the shades of grays, many a times over 50 (sorry, couldn’t resist) when it’s not truly black or white.

Thank you for the time.

Visit my blog http://askhurram.com/

Read On, Live On.

February 24, 2015
Five Key Components of Application Release Automation (ARA)

What are the fundamental building blocks of successful ARA?

Application Release Automation, or ARA, is the consistent, repeatable and auditable process of packaging and deploying an application or update of an application from development, across various environments, and ultimately to production.

Successful ARA eliminates the need to build and maintain custom scripts for application deployments, while simultaneously reducing configuration errors and downtime. By providing a model-based approach to performing critical automation tasks, it effectively increases the speed to market associated with agile development and gives stakeholders the ability to coordinate and automate releases between multiple groups and people.

In this article, we’re going to take a look at the five fundamental aspects that make up application release automation and talk about how each one serves a critical role in ARA’s success.

Packaging

The first tenet of ARA is packaging. Packaging is just what it sounds like – taking a bunch of parts and pieces and bundling them up together in a manageable and functional way. So why is this important and how does it fit within the context of Application Release Automation?

Consider the following example: You and your company decide to switch offices, from your current building to one just down the street. You hire a couple of movers, get them in, and tell them you want to take all the stuff in your office and move it to the new location. Easy enough, right? So the movers start grabbing chairs, desks, computers – everything – and carrying them down the street. They’re throwing things out the door, through the windows, everywhere! Next thing you know, a few chairs are broken. A laptop is missing. Your iPhone is gone! The movers say, “Hey, look, those chairs were already broken. I didn’t see any laptop on the table…” In this scenario, there is no bill of materials – just a bunch of confusion and some missing and broken parts. That is where packaging comes in.

Packaging creates the bill of materials for your deployment. It is your way of knowing exactly what you need to move from point A to point B. You take all the pieces of your release and wrap them up neatly so they are ready to ship. It’s the method by which you can assure that everything that was supposed to make the move, in fact, did. Packaging allows you to confidently say that what you had at the source location is what ended up at the target location – and it is critical to a successful ARA process.

Dependencies

Dependencies are another critical cog in the ARA process. When you are working on your release, you need to understand what other applications or components are involved for its success. For example, your web application uses a database hosted on a different server. If the data in that database is changed, or columns are created or deleted, how will that impact your application’s functionality? This is a dependency – the way your application functions is dependent on the data in another location.

In the case where you are pushing out an update to an existing application, you need to also take a look at how other things might have relied on the old application. Are you doing away with a feature or function that is going to have an impact on something else? This highlights how you need to not only examine what your application relies on to function, but also the mirrored function – what outside processes and/or applications might depend on your existing app.

As you can imagine, the process of charting dependencies can grow exponentially. If application A depends on databases B and C, and C depends on D, E and F and then F… you get the idea. It can get very big, very quickly. This is why it is helpful to chart dependencies as you move, so you have a dependency map that is up-to-date and adaptable. You know where the dependencies are, how and what your updates or releases will affect them, and what you need to keep an eye on post-deployment. If you start charting dependencies at the beginning of your development, it will make ARA much easier down the road.

Promotion

Promotion in the context of Application Release Automation is the process of moving the application or update from one stakeholder to the next so that the release is as close to perfect as possible before it is deployed.

This is a crucial step in the ARA process because it creates an audit trail that assures that the right people along the way gave this deployment the “OK” to move forward. Getting the input from multiple parties is imperative to a deployment’s success because each party – from the developers to release engineers to the Quality Assurance (QA) team – has a different perspective. For example, developers are looking at an application and saying, “How can I make this the best application possible with the most robust set of features?” While someone on the QA Team says, “How can I break this?” They are running your application through tests, wringing out inefficiencies and errors along the way.

By passing (or promoting) your release through the chain of stakeholders, you can find the holes in your release before the customer does.

Deployment

Deployment is the act of launching your application and from point A to point B. Using the word “launch” is no mistake – it’s just like the commander of a space operation pushing the launch key. It’s pressing one (or maybe a few) buttons and releasing your application or update, configuring its operating settings, and letting it out into the wild. Deployment is making sure the release makes it to its destination safely. At its core, this is the fruit of the labors of ARA – letting your deployment free to do what it was designed to do! The core objective here is that ARA moves this from a scripted, expert level process to a more model-driven perspective (where less skilled resources can understand and perform the deployments…). So when applications, components, and dependencies are modeled within an ARA solution, companies now have the means to deploy their critical applications without having to rely on the superhuman resources who designed and built the application.

Compliance

Another key aspect of Application Release Automation is compliance. This is literally, “Are you following the rules of the game?” However, the rules of compliance vary wildly within different environments. At a small startup, with a team of three people making a web app – there are probably not too many rules to follow. The compliance necessary for a game launching birds at disgruntled pigs is minimal.

At the other end of the spectrum are things like banking and government applications. Here, there are laundry lists of rules and regulations that need to be strictly obeyed or you may find your application being tossed – or find yourself in a CIA interrogation room (ok, we haven’t heard about that actually happening…but you can imagine!). With compliance, it can be helpful to start by asking yourself, “What do I really need here?” Once you figure out the requirements that are necessary for your specific application, you can map them out and discover what tools you can use to solve the problem.

This is an essential key to ARA because if you have four out of the five steps above, but miss on compliance, all your work was a waste because it’s not going to play nice in its environment. Adhering to compliance – and documenting that you did – is crucial to successful ARA.

With a strong understanding of these five key building blocks, you will have a solid base to start exploring the wonderful world of Application Release Automation. Go forth…and automate!

About the Author/ Wesley Pullen

Wesley Pullen is the general manager and vice president of deployment solutions at Electric Cloud. Pullen leads the Release and Deployment business at Electric Cloud and is responsible for go-to-market strategy to aid in the company’s growth and expansion efforts around DevOps, Continuous Delivery and Deployment. He brings more than 18 years of experience in software development methodologies and design standards, applying this experience to both commercial and private sectors. Linked in: https://www.linkedin.com/in/wgpullen Twitter: @WesleyGPullen

February 23, 2015
eHarmony finds its perfect match with PagerDuty and DevOps

Did you know that 4% of all marriages in the US are the result of people “finding their perfect match” on the eHarmony service? Keeping the systems that are making all of those matches up and doing it 24/7/365 is no easy task. Especially at peak load times like Valentine’s Day for instance. The pioneer in making online relationships relies on many DevOps tools including PagerDuty to minimize downtime and keeps its relationship science systems humming along.

I had a chance to speak with Hav Mustamandy, Director of System Operations at eHarmony. Since Hav joined the team there about 7 months ago he has been busy modernizing the eHarmony network using a lot of tools that you would expect. They are a big Chef shop for example. PagerDuty is another tool they lean on a lot. More than tools though Hav has been instilling a sense of team between his Dev, Ops and QA teams. In a short period of time, Hav has made tremendous progress in making the eHarmony IT business more responsive, reliable and faster.

Frankly, why shouldn’t eHarmony act more like a Unicorn anyway? After all they are a web based business like many of the companies that we think of as being so successful. But before Hav came it was not necessarily so.

Upon arriving he had to evaluate everything and everyone and start doing things differently. One of those things was the way they use PagerDuty. More than just an update to the old pager, Hav realized that PagerDuty could help him automate their incident resolution processes. Additionally using PagerDuty’s advanced analytics reporting, Hav was able to pinpoint problem areas in both systems and people. With the problem systems he was able to upgrade or change them. With people he realized where re-training was required.

As a result Hav has seen some amazing results. The uptime of eHarmony’s systems even during peak load times has improved significantly. Technical issues are identified and dealt with before they rise to critical levels.

By leveraging APIs that PagerDuty makes available Hav has integrated several of his other systems into PagerDuty so he can gain insight into their performance as well. To him this is the future. Event management with PagerDuty at the core is what he is aiming for.

For me PagerDuty and ChatOps applications biggest advantage is the ability to move inter-team communications into hyper-speed. If you are relying on email for communication it is probably not fast enough to keep up with your business.

We have been using Slack here at devopsy.kinsta.cloud for ChatOps and I can tell you that our communication now moves so much quicker than it did when we communicated strictly through email. However, the discussion I have had with my team is if we are just chatting, what is the advantage of Slack to just google messenger or something like that?

I think the answer lies in integration. Using APIs we should see more than just chatting going on with ChatOps. Systems will communicate with us in real time as well. This is so much more than chatting on a messenger client.

The name of the game whether it is the dating game, the online media game or whatever game you play is doing it faster, better and with less downtime. For eHarmony it looks like they have “found their perfect match”.

February 20, 2015
Inside the mind of a true DevOps evangelist

IBM InterConnect is imminent. Next week more than 20,000 people will descend on Las Vegas and take over the north end of the strip for IBM’s massive event. One of the primary topics attendees will be focusing on is DevOps. IBM’s Rosalind Radcliffe—a respected evangelist for DevOps—will be there to share her passion and knowledge as well.

Rosalind Radcliffe is a Distinguished Engineer, in Rational currently responsible for the driving the System z and Power Systems aspects for the Jazz foundation and Jazz foundation based products—or so her LinkedIn profile indicates. Really she’s just a very smart person who loves to help customers solve problems.

At InterConnect Radcliffe will be talking about DevOps. She is hosting a number of workshops for DevOps for enterprise systems. She will also be engaging in panel discussions with clients to discuss things like how an organization can transform its current environment and the part that DevOps plays.

I had an opportunity to speak with Rosalind recently, and gain some insight into her perspective on DevOps and what it means for IBM enterprise customers. Two themes seemed to come up repeatedly: breaking down silos, and finding a common ground.

Radcliffe stressed that top-down support from executive management is crucial to a smooth transition. “At a business level [executives] understand DevOps is necessary. They know the value and they want to do it,” she explained. The challenge is how to adapt and make that transformation without disrupting business.

Where do you begin? If you’re Rosalind Radcliffe you start by finding common ground. All the way from the business perspective to deployment and across all platforms it’s important for all parties to see the same point of view and work toward a common goal.

That’s not always easy to do when you’re dealing with different teams using different tools and platforms, and with separate political agendas. The first step is breaking down silos. Rosalind told me, “Everyone says that, but that’s really what has to happen.”

She explained that customers don’t have to throw out the organizations structure or retool the teams themselves necessarily. It’s about redefining how the teams interact and work together, not about reorganizing the teams.

Radcliffe is passionate about DevOps, but her real drive is helping customers achieve their goals. DevOps is hot now, but it’s just an evolution of the tools and processes businesses use to find a competitive advantage. DevOps may pass, but the need to streamline processes and accelerate delivery will live on.

“A few years from now we won’t be using the word DevOps,” said Radcliffe. She told me that DevOps is just the Agile principle taken to the full enterprise. It provides a more comprehensive end-to-end perspective that enables enterprise-scale transformation.

The reality, according to Radcliffe, is that customers need to come up with an enterprise strategy, but it helps to start small. She works with customers to find those initial baby steps that help them gain momentum that eventually allows them to make the bigger transformation.

Vegas isn’t really her scene, but she’ll be there next week for IBM InterConnect. She can do without the flashing and dinging of the slot machines or the Vegas shows. But, she’s driven to engage with and help enterprise customers, so she’ll brave the busy Vegas environment to share her passion for DevOps.

February 18, 2015
The DevOps Institute heralds the dawn of enterprise class DevOps training and certification

I am very proud to announce that today marks the launch of the DevOps Institute. Myself and the devopsy.kinsta.cloud team are partners in this joint venture with some very successful, experienced IT training professionals, Jayne Groll and Lisa Schwartz. The DevOps Institute (DOI) seeks to codify best practices in DevOps into repeatable training, education and certifications. We are launching with two classes, DevOps Foundation and Certified Agile Service Manager being offered. Several other courses including role and scheme based certifications and classes will be announced shortly. The press release is also available at the end of this article.

I know there is a segment of the DevOps community who view this as an apocalyptic event. While I obviously don’t agree with that view, I respect their opinion. As I have said all along the DevOps community is big and getting bigger. There is room for a wide spectrum of opinion. If you don’t think that DevOps needs formal education, training and certification, so be it. I and many folks I have spoken to believe that as DevOps continues to cross the chasm into mainstream enterprise IT, a formal training and certification program will be required. Whether it is from the DOI or some other entity is the only question.

My co-founders and I believe we are in the best position to bring quality, best practices based DevOps training to market. We have assembled a world-class Board of Regents (BOR) who will help guide the course of the DOI, plan and approve all of our courses and ensure we are training based on best practices as they evolve. You can see and read about the BOR here. We have leading lights of the DevOps community, the IT training world and general IT business leaders as members. Looking at some of the names on this list will show you the kind of people we are involved with and should speak volumes as to what our commitment to this will be.

We welcome those of you who would like to join us in this mission. Our mandate is to be inclusive. We invite all providers who are offering or planning to offer DevOps training to join with us. We will examine your curriculum, provide you our syllabus to develop your own courses, certify your courses and even allow you to license and teach our courses. You can find out more about this at the DOI website, DevOpsInstitute.com

If you are interested in DevOps training and would like to be involved with the Institute as a board member or in some other capacity, contact us.

The launch of the DevOps Institute represents the culmination of over 9 months of work and research. However if we are to succeed this is just the end of the beginning and the start of a great new ad-venture!

The DevOps Institute Brings Enterprise Class DevOps Training To Enterprise IT Market

THE RAPID ADOPTION OF DEVOPS NECESSITATES CERTIFIED TRAINING AND EDUCATION FOR ENTERPRISE IT MARKET

Boca Raton, Fl February, 18, 2015– The DevOps Institute (DOI) launched today with the mission of serving as the premier source for aligning industry standard quality DevOps training and examination services for enterprise IT. The Institute is led by a Board of Regents who will oversee DOI’s offerings in an effort to codify and promote DevOps’ best practices and standards that enable enterprise IT to deliver more value faster to their customers.

The initial Board of Regents includes Gene Kim, Lori Macvittie, Sanjeev Sharma, Helen Beal, Rohit Antao, Gary Gruver, Lou Hunnebeck, Matthew Selheimer, Donna Knapp and JP Morgenthal. They join co-founders Jayne Groll, Lisa Schwartz and Alan Shimel in leading the DevOps Institute to fulfill its mission.

“The DevOps movement has matured rapidly over the last few years. We believe that the time is right to have established best practices training and certification”, said Lisa Schwartz, CEO of DevOps Institute. “DevOps training is a logical step in DevOps adoption by the enterprise IT market. The DevOps Institute’s mission is to codify emerging best practices in DevOps and translate them into the highest quality education and certified training courses. Working with leaders of the community, DOI will continue to develop and offer the finest in DevOps and associated programs and certifications.”, continued Schwartz.

“DevOps is transforming how technology work is done, for Development, Test and IT Operations. I support any effort that helps elevate the state of the practice, and bring consistency and quality to the training offered,” said Gene Kim, DevOps researcher and co-author of The Phoenix Project and the upcoming DevOps Cookbook.

The DevOps Institute’s initial courses including certification are DevOps Foundation and Certified Agile Service Manager (CASM). Additional certification courses will be announced in the coming weeks including role based schemes and certifications. DOI will offer courses directly and through a network of Registered Education Providers. Courses will be available both in person and virtually utilizing state-of-art virtual classrooms. First courses will begin in early March. More information can be found at DOI’s website: http://www.devopsinstitute.com

About DevOps Institute
The DevOps Institute is bringing enterprise class DevOps training and certification to the IT market. Working with thought leaders from the DevOps and IT Service Management communities, as well as the IT training market, the DevOps Institute is the standard in quality, enterprise grade DevOps education.

February 18, 2015
Comparing DevOps to traditional IT: Eight key differences
You can make the argument that in the Software Delivery Life Cycle (SDLC) value is created in two places. First, when the software is created (development) and second, when the software is delivered to the customer and feedback received. The rest is transportation. The faster one can move code from the hands of the developer to the customer and receive feedback, the more value can be realized by both. Don’t get me wrong, there is benefit to various “transportation” activities (testing, QA, staging etc.). But those are not value creating activities, they are risk mitigating processes.

A DevOps organization understands this value equation. They focus on the processes that add value, organize themselves around those actions and do the best to minimize the risk. Traditional IT gets it too – at least conceptually. However they treat all stages in the SDLC with equal importance or worse emphasize the wrong stage(s).

In general, when comparing DevOps organizations to traditional IT, DevOps will have policies and practices that differ from those of traditional IT along the following eight key dimensions,

Planning & Organization
1. Batch Sizes: Go from Big to Mirco
Traditional IT has a bias for going big and for good reason. First, most development shops grew out of the waterfall method which by its inherent nature takes a lot of time. Second, since releases are costly and disruptive, operations allows developers only a few windows a year to release software. And third, its simply not sexy – doing big is how you get promoted. As a result, development organizations maximize productivity, by planning big projects, that involve a lot of code, bundled (hastily at times) into a release, and jammed into production.

A DevOps organization takes the opposite point of view and believe that small is beautiful. They understand that large batch sizes are inherently complex, risky (since there are so many moving parts), and hard to coordinate. Small batches sizes, on the other hand, are simple, easy to understand and test rigorously, and less risky. If things go wrong the impact is minimal and it is much easier and faster to fix. In other words, by going small organizations are able to perform more frequent releases and become more responsive to the customer.
1. Organization: From Skill Centric Silos to Dedicated Cells
Traditional IT is organized around skill centric silos. For the most part silos work, they band like skills together, drive greater utilization, and benefit from economies of scale. However, where these “cost optimized” silos break down is at the hand offs. In a typical IT environment, a new feature has to go through at least 3 – 4 silos before the customer gets it. It is not uncommon for an idea / code to spend 80% of its time waiting or ping-ponging back and forth between silos.

A DevOps organization also operates in a silos but at a different cross section. Here teams are arranged in “cells”, consisting of dedicated cross functional teams, and focused on only one application. By creating this self-sufficient cell that consists of developers, testers, business analysts, and operators, an idea can move from one stage to another without hand offs, promotes cross-training / understanding (key to collaboration), focuses the team on the end goal, and encourages the “shift left” thinking. As for utilization and scale benefits, the rules for response states that for every ¼ reduction in cycle time, productivity will improve 2x, and operating costs by 20%.
1. Scheduling: Centralize to Decentralize & Continuous
Efficient scheduling is at the heart of a Traditional IT organization. Since resources are pooled, projects are usually clamoring for access to SMEs and / or infrastructure. To get around this, enterprises have invested in sophisticated scheduling planning systems. And while these systems are quite sensitive, they are inherently inaccurate, take up to much time to manage, and in some cases become the very bottleneck that they are trying to alleviate.

In a DevOps organization scheduling is pushed to the local cell level. The combination of smaller batch sizes, dedicated teams, and automated processes makes scheduling more simpler to operate. How? First, forecasting is limited to the very near future (2 – 3 weeks) where the teams have better insight. Second, there is no fighting for people’s time since it is a dedicated team. Third, there is no waiting for infrastructure since it has already been defined and automatically provisioned. And fourth, it eliminates time consuming loop backs to management for escalations and decisions.

Performance & Culture
1. Release: Turn a High Risk Event to a Non Event
In a Traditional IT organization, releasing software into production is a high risk proposition. It is fraught with issues, escalations, and constant fire-fighting. The process is tightly managed, governed from the highest levels, and requires participation from all parts of the organization. It is not uncommon for IT to set up war rooms staffed 24×7 for weeks before and after the release.

DevOps organizations, on the other hand, make the release of software into a non-event much as possible. They reduce risk by daily integrating code into the trunk, automating testing, ensuring all environments are in sync, reducing batch sizes (as mentioned above) etc. In other words, they only promote code from one stage to another after they are confident that it will work in production. Thus eliminating all the hoopla about release windows and they are able to move new functionality into production at a much faster clip.
1. Information: Focus is on Dissemination vs. Actionable
Both types of organizations generate and share a ton of data. The difference lies in how the teams uses data. In Traditional IT, information is generated by specialists (e.g. operations team), bundled together with other data into a massive report, goes through management approval, is then shared with other managers, who then send it out their specialist (testers, developers etc.). In most cases the report goes unread, simply because there is too much data, not timely enough, and / or not the right data. In other words, information is shared but poorly consumed, and rarely used to take any actions.

Within a DevOps organization, it’s the team cell that gather and creates the data. Since the data is to be consumed locally they only collect the data (also automated) that the team deems is necessary. And since the data is processed within the team, it eliminates the time lag of creating lengthy reports, manager approvals, and queue time (sitting in some ones email box). As a result, the team is quickly able to read and react to the data – resulting in faster feedback time.
1. Culture: Do Not Fail vs. Fail Early
Traditional IT is fundamentally a risk averse organization. A CIO’s first priority is to do no harm to the business. It is the reason why IT invests in so much red tape, processes, approvals etc. All focused on preventing failure. And yet despite all these investments, IT has a terrible track record – 30% of new projects are delivered late, 50% of all new enhancements are rolled back due to quality issues, and 40% of the delay is caused by infrastructure issues.

A DevOps organization is risk averse too but they also understand that failure is inevitable. So instead of trying to eliminate failure they prefer to choose when and how they fail. They prefer to fail small, fail early, and recover fast. And they have built their structure and process around it. Again the building blocks we have referred to in this article – from test driven development, daily integration, done mean deployable, small batch sizes, cell structure, automation etc. all reinforce this mindset.

Measure
1. Metric: Cost & Capacity to Cost, Capacity, & Flow
Silos work for Traditional IT because they are directly tied to its measurement model – cost and capacity. IT is measured on how much can it get done (capacity) for the least amount of money (cost). It is no wonder that cutting cost, while trying to keep capacity constant, has been all the rage in IT over the last two decades. It’s one of the reasons why outsourcing became so popular.

And while this model works great for stable legacy applications, the newer “system of engagement” applications require an additional metric – the element of time. DevOps organization understand this paradigm shift and have added “flow” as an additional metric. Flow forces an organization to take a look at its end to end cycle time, identify areas of waste, calculate true productive time, quantify quality, and focus on activities that add the most value.
1. Definition of Done: “I did my job” vs. “it’s ready to deploy”
Another source of measurement difference is in how the two types of organizations define done. In Traditional IT, done is defined as the specialist doing “just” their part and handing it off. In essence, their done is more focused on meeting the hand off deadline vs. making sure what is done is deployable. This varying definition leads to some bad habits such as sub-par work, quality loss, finger pointing, and an overall sub-optimized process.

To get way from the finger pointing mentality is another reason why a DevOps organization prefers to create a dedicated cross functional team. With every member of the various parts of IT represented in this “cell” structure and all of them being held accountable for one and only one thing, they all end up having just one definition of done – bring quality software to market.

If you would like to learn more about this topic, please join me at IBM InterConnect 2015. I will be hosting several workshops on DevOps.

About the Author/Mustafa Kapadia

Mustafa Kapadia is the Service Line Leader for IBM’s DevOps practice, a business advisory practice focused on helping large enterprises transform their software & application delivery. He has over 17+ years of experience in the tech. space, both as a service provider and a management consultant.

Prior to joining IBM, Mustafa was a management consultant with Deloitte’s Strategy & Operations practice. He lives in San Francisco Bay Area, is an avid blogger (when time permitting), a speaker, and adviser to start ups.

Connect with Mustafa LinkedIn| Twitter
February 17, 2015
GitHub, Bug bounties and DevOps

A year after starting up its bug bounty program GitHub is showing how complementary bug bounties can be to DevOps practices. Essentially crowdsourcing vulnerability hunting by systematically paying independent researchers prizes for flaws they find, bug bounties are in a sense an extension of the DevOps ethos. They offer a means of continuously delivering application security assurance.

“The advantage of having a community that is just working 24/7 worldwide is it does tend to be a more continuous approach to vulnerability hunting as opposed to the traditional method of hiring out a consultancy or third party to spot check at some point in time,” explains Shawn Davenport, vice president of security for GitHub.

And according to GitHub’s security leaders, their program has become one of the most effective and cost-conscious tools they have for extending internal appsec resources. With approximately four application security experts on staff, GitHub has a more than respectable team in place for a company of its size, with an employee total of just over 250. But with so many deploys a day, the team needs air support to ensure they’re thoroughly inspecting the code base.

“We ship software 100 times a day, we update the site constantly and the security team isn’t looking at every one of those changes,” says Ben Toews, application security engineer at GitHub and the mastermind behind its bug bounty program. “The way we fit security into our software development life cycle is that when someone is getting ready to ship a feature, they’ll reach out to the security team and ask for a review if they think it’s necessary and we’ll get a review in that point in time, but then a lot of smaller changes don’t necessarily get our attention. And then there’s a lot of legacy code we’ve never looked at.”

Toews says the bug bounty fills the holes a number of ways. He estimates that for any given feature on the site, there’s probably been several hundred researchers who have tested them, “so the odds of someone looking at new code pretty quickly after a change is pretty high.” In its first year, GitHub’s bug bounty program fielded nearly 2,000 reports from bug hunters, 869 of which warranted further review.

As GitHub rounds the corner into its second year of offering bug bounties, it wants to increase those odds of good coverage with some tweaks to its program. In addition to doubling its max bounty prize to $10,000, it is also experimenting with giving bounty hunters preview access to new features before they ship.

“We think by doing that we can get even better coverage on features before potentially putting the community at risk,” says Toews, who admits it does complicate rapid release schedules. “We’ve been trying recently to be more methodical about how we ship larger features. If we’re developing a new feature, you will have that feature hidden from the general population and you can develop that and ship it many times a day and iterate on it quickly, but then we do have a little bit of a process around making that feature published and exposing it to the rest of the world.”

Of course, the efficacy of a bug bounty program depends on organizations finding a good way to marry bounty vulnerability remediation with the deployment process. It’s not good enough to find the vulnerability—you also need to be able to fix it quickly. Fortunately, the continuous delivery model makes this easier than in traditional organizations, says Toews, who reports that independent researchers are often “astounded” that his team can sometimes have problems fixed within hours of receiving a report, considering that they’re used to other organizations taking months to remediate problems.

Toews says the secret to success is rapid triage of vulnerabilities. If it is simple, he and his team of application security engineers will fix it on the spot. If it is more complex, they’ll loop in the development team. This is where it is important to have buy-in from developers and senior leadership about bug bounties and application security in general, he says.

“Our developers care a lot about security and they know that our leadership cares a lot about security, so if there’s a problem, everyone is going to jump in and try to fix it,” he says.

February 16, 2015
The benefits of micro-services

The adoption of a micro-services architecture has two main advantages, according to ThoughtWorks technologist Sam Newman.

One is that micro-services increase the autonomy of individual development teams within an organization, as ideas can be implemented and deployed without having to coordinate with a wider IT delivery function.

This allows new or changed features to be delivered more quickly. Organizations that understand they make money through technology – Netflix is one example Newman cited – know they need to get new features out the door as quickly as possible.

The other main advantage is that micro-services aids the adoption of new technologies, although this is arguably a subset of the autonomy argument.

Since a micro-service is by definition a small, self-contained function, it provides a low-risk opportunity to try a new technology. That is definitely not the case where monolithic systems are concerned, Newman said, as greater care is required when all the eggs are in one basket. But where a less-critical element is concerned, there are fewer barriers to experimenting with a new programming language or database, for example.

So which organizations are setting the pace? “Netflix is a very good example,” said Newman, observing that the driver for its adoption of micro-services was to ship new features quickly, rather than handling scaling issues.

Locally, REA (the company behind realestate.com.au and related real-estate sites) “is probably a couple of years ahead” in the use of micro-services, he said, noting that REA has been a ThoughtWorks client for many years, and has also embraced DevOps and automation. Its technology architecture is aligned with its organizational architecture, so REA can readily absorb companies that it acquires and therefore grow more easily. Organizational issues are key to understanding the significance of micro-services, he said.

REA approached micro-services progressively. In the first three months it made sure the foundations were in place and put two micro-services into production. But in the second trimester it was able to deploy around a dozen more, and in 18 months more than 60 were live.

Other micro-services adopters mentioned by Newman included MYOB (see DevOps in Action: MYOB) and PwC (which is progressively migrating its system for handling expatriates’ tax returns and related issues from a traditional enterprise system to a finer-grained, more decomposed architecture using micro-services, to reflect the way some parts of the system are common to all geographies while others are country-specific).

There is a very real risk that building micro-services will lead to duplication, but that is not necessarily a bad thing, Newman said. Many SOA (service-oriented architecture) implementations have been hamstrung by efforts to optimize services for reuse slowing the pace of change, so organizations may choose to accept a degree of duplication in order to speed delivery.

An Australian bank is currently exploring this issue, realizing that it previously made a mistake by emphasizing reuse over time to market, he said. “It has to be a balance… [but] reuse in itself shouldn’t be the goal.”

That said, it can be beneficial to eliminate at least some of this duplication over time in order to improve the maintainability of the systems. Otherwise the result can be “quite a chaotic environment.”

“It’s a governance thing,” Newman explained – organizations need to detect any duplicate micro-services, and then make a deliberate decision to either continue as they are or to converge them. This is healthier than simply forbidding divergence and duplication as that can negatively affect time to market.

Allowing duplication – even if only in the first instance – also mitigates questions of ownership, allowing the team developing a micro-service to proceed without the need for ‘sign off’ from other parties that may be potentially affected.

In any case, developing a micro-service typically takes around two weeks, so there is no great loss if duplicates are retired. It is a different situation when monolithic systems are involved, as they are very hard to retire in an SOA environment – an enterprise application may be grossly underused, but it can’t be removed as long as other programs are consuming its services.

The idea behind SOA was good, he said, but it became dominated by software vendors rather than by consideration of what works well for their customers.

The micro-service architecture can be thought of as ‘SOA done well,’ Newman suggested, especially as it provides a mechanism for surfacing capabilities in multiple different ways, such as (in the case of a retailer) via web sites, native and web apps for mobile devices, and in-store software. When different capabilities are to be combined in different ways, working with smaller pieces is easier than larger ones.

Newman’s book Building Micro-services is published by O’Reilly Media. It is now available as an ebook and will appear in print shortly.

February 12, 2015
Five Great Books on DevOps
While the DevOps movement began in earnest in 2009, there are still a lot of questions about what DevOps means, how it’s done, and what the purported benefits of DevOps are. Big questions, for sure. And there are no quick answers. In fact, to thoroughly answer each would require numerous books.

Which brings us to the question, what are the best books that define DevOps, describe how to operationalize it in the real world, and show what beneficial outcomes one could expect?

Here are my choices of five great books on DevOps, in no particular order:

Book: The Phoenix Project

Authors: Gene Kim, Kevin Behr, George Spafford

This one is a fictional story, written in engaging novel style, that tells the story of how to apply DevOps to unleash IT’s potential. The story is about Bill, the IT manager at the fictional company Parts Unlimited. Not an exciting business name, but the adventure starts in earnest when on a drive to work, Bill gets the call from the CEO that sets the story in motion.

The book’s cover description tells it best:

The company’s new IT initiative, code named Phoenix Project, is critical to the future of Parts Unlimited, but the project is massively over budget and very late. The CEO wants Bill to report directly to him and fix the mess in ninety days or else Bill’s entire department will be outsourced.

With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.

As one Amazon reviewer put it: “This “parable” of how applying DevOps to pull out of an IT death spiral of delays and defects is an enjoyable read backed with great references and support materials for understanding the principles underlying DevOps. It’s not enough to get you all the way there, but it will get you inspired to start down the path.”

Book: Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

Authors: Jez Humble and David Farley

This seminal read is all about putting into practice rapidly and consistently delivering new highly-functional software to your users. From the back cover: “Through automation of the build, deployment, and testing process, and improved collaboration between developers, testers, and operations, delivery teams can get changes released in a matter of hours—sometimes even minutes-no matter what the size of a project or the complexity of its code base.”

The book covers automated infrastructure management and data migration and the use of virtualization, and automating creating, integrating, testing, and deploying software, as well as improvement collaboration within the teams.

As one Amazon reviewer put it, “This is an absolute must read for anyone in software or application development. It is amazing how well the authors capture the real pain points and provide all the information you need to getting to a more effective IT practice.”

Book: The Visible Ops Handbook

Authors: Kevin Behr, Gene Kim and George Spafford, IT Process Institute

This is the first book I ever read on IT process improvement, and it was an eye-opener for me. This book came out about 4 years before the term DevOps was coined, but it nailed the essence of what high-performance IT is all about. And it’s an essential read today:

Visible Ops codifies how these [High performance enterprise] organizations achieved their transformation from good to great, showing how interested organizations can replicate the key processes of these high-performing organizations in just four steps:
1. Stabilize Patient, Modify First Response – Almost 80% of outages are self-inflicted. The first step is to control risky changes and reduce MTTR by addressing how changes are managed and how problems are resolved.
1. Catch and Release, Find Fragile Artifacts – Often, infrastructure exists that cannot be repeatedly replicated. In this step, we inventory assets, configurations and services, to identify those with the lowest change success rates, highest MTTR and highest business downtime costs.
1. Establish Repeatable Build Library – The highest return on investment is implementing effective release management processes. This step creates repeatable builds for the most critical assets and services, to make it “cheaper to rebuild than to repair.”
1. Enable Continuous Improvement – The previous steps have progressively built a closed-loop between the Release, Control and Resolution processes. This step implements metrics to allow continuous improvement of all of these process areas, to best ensure that business objectives are met.
As this Amazon reviewer this had to say, this fall, on the book: “This is the first and foremost required reading on my team! If you are an IT professional, you must read this. It is a concise, powerful expression of how to reinvigorate your operation and become a top performing IT organization.”

Book: DevOps for Developers

Author: Michael Hüttermann

This book is for software developers and engineers that aim to help improve the delivery of software by using lean practices to get apps and software into production.

From its description:

DevOps for Developers is for motivated software engineers, particularly programmers, testers, QA, system admins, database admins, both beginners and experts, who want to improve their software delivery process. It’s the perfect choice for engineers who want to go the next step by integrating their approaches for development and delivery of software. This book is for engineers who want to shape their processes and decide on and integrate open source tools and seek for guidance how to integrate standard tools in advanced real world use cases.

And as an Amazon reviewer put it: “Developers tend to produce updates in short cycles whilst admins prefer a stable system. And if something bad happens, they often start finger-pointing to blame the other side. One solution is to tear down the silos of development and operation. In other words, set up integrated teams.

This is the core message which Michael Hüttermann would like to stress out. Agile methods and tools might be used for development as well as for deployment and operation. In his book, Michael explains how to do and which (free) tools to use.”

To be fair, the book has taken dings about not being a programming book, and repeating some of its concepts. But the value in this book is its real-life examples, and explanations as to how different roles on teams in Dev and Ops can work together more collaboratively. It also dives into common pitfalls organizations hit in their journeys of operationalizing DevOps.

Book: The Goal: A Process of Ongoing Improvement

Authors: Eliyahu M. Goldratt, Jeff Cox Author, David Whitford (Other Contributor)

The Goal is the second fictional book in this list. Written more than 30 years ago, it could have been written yesterday. When discussing concepts and systems, many times story is the way to go because the motivations and thought processes of different roles can be discussed in depth and elegantly so. The Goal succeeds in this, and is a must read for anyone who seeks to help be a catalyst of improvement in their organization.

From the book description:

One of Eli Goldratt’s convictions was that the goal of an individual or an organization should not be defined in absolute terms. A good definition of a goal is one that sets us on a path of ongoing improvement.

Pursuing such a goal necessitates more than one breakthrough. In fact it requires many. To be in a position to identify these breakthroughs we should have a deep understanding of the underlying rules of our environment. Twenty-five years after writing The Goal, Dr. Goldratt wrote Standing on the Shoulders of Giants. In this article he provided the underlying rules of operations.

This article appears at the end of this book.

Here’s what an Amazon reviewer had to say: “I have often heard myself complaining that companies over complicate the simplest of activities. Measurements do not truly measure the end result and are often held in a high regard by executives. These measurements cause further loss and mask the true issues. This book was fascinating, very “simple” in a sense which lends to making it even more fascinating. I learned a lot about the production aspect of my “department” which would often not be looked as a production department in the literal sense of the word. Thought provoking and inspirational, every leader can benefit from the story told. A must read.”

I would love to hear from you, what are the most important books on DevOps that you’ve read?
February 11, 2015
11 DevOps start-ups funded for a busy 2015

Slides

[seoslides embed_id=”4b8ee43e173e” script_src=”https://devopsy.kinsta.cloud/embed-script/11-devops-start-ups-funded-busy-2015/title/” overview_src=”https://devopsy.kinsta.cloud/slides/11-devops-start-ups-funded-busy-2015/” site_src=”https://devopsy.kinsta.cloud” site_title=”devopsy.kinsta.cloud” title=”11 DevOps start-ups funded for a busy 2015″ /]
Advance slides with arrow buttons or touchscreen

February 9, 2015