devopsy.kinsta.cloud

Tag: continuous delivery

DevOps and Organizational Debt

I recently published a blog on the Perficient Digital Transformation blog entitled “Is Cloud & DevOps Factored Into Your Digital Transformation?” In that blog I discuss the concept of organizational debt and its impact on your DevOps adoption. Here’s what I had to say in that blog:

If technical debt addresses too little investment in upgrading, poor vendor selection, bad platform selection, etc. then, many businesses have also built up organizational debt. Organizational debt could be over regulation, too many low-value policies, to much middle management attempting to hold onto their area of control, and too much governance. If these are not revisited often to ensure that they are continually adding value for the cost of reducing throughput, then that organizational debt will continue to increase backlog.

Interestingly, there’s a lot written about technical debt, but very little has been written about organizational debt, which can have a significantly deeper impact on your ability to continuously deliver than your technical debt.

In working with clients on their DevOps initiatives, we map the current value stream for a given solution area. What becomes apparent from these value stream maps is that controls and policies have been derived within organizational silos with no attention given to how that will impact IT’s overall ability to be agile and deliver more quickly for the business. Moreover, the owners of these controls and policies seem reticent to remove these bottlenecks even after being shown how it impacts business agility.

Here’s an anecdote from one client when asked why they aren’t using snapshots to speed preparation for QA testing. The suggestion was made in response to the client stating it can take upwards of a week to prepare the QA environment for a single round of testing. Their current approach serializes all testing and believes they need to perform a complete regression test due to the resource contention issues. Hence, they cannot test small changes easily and they cannot test often due to these limitations.

“The main issue is that we don’t control the application or database environment, so we need to prepare a QA release and then have those groups perform their duties in preparation of the QA environment for testing.”

Upon hearing this my mind summoned up past experiences working conferences at major venues, such as the Jacob Javitz Center in NY, where you can’t plug in an extension cord or run an Ethernet cable yourself, but you need to use the Javitz Center’s electrician because of union rules.

Is this what IT has become a set of unofficial unions that hold the business in a death grip? This is clearly a case of organizational debt that needs to be addressed.

Of note, I’m not admonishing the various groups for instituting their controls and policies. I’m sure they have emerged due out of need due to past grievances, failures or as part of IT management practices. Regardless, they are now acting as a bottleneck in implementing proven practices for delivering fewer features more often, which have been proven to increase trust and facilitate better prioritization of work.

The result of this organizational debt is lead times have grown to four months or longer to release functionality that is directly related to key business initiatives by the business for increasing revenue.

In his book, “Kanban,” author David Anderson discusses his recipe for addressing bottlenecks and constraints with existing teams. One of his steps is ‘attack sources of variability to improve predictability’. However, this is his last step because, as he notes, “some types of variability require behavioral changes in order to reduce them. Asking people to change behavior is difficult!” And, so it goes, many businesses never address this key constraint because the hurdles of organizational debt have become so burdensome to overcome. It requires a high degree of trust and political capital along with executive sponsorship. As shown by the anecdote earlier, even if the constraint is directly impeding other business priorities it is not easy to overcome.

I believe DevOps practices can help organizations significantly reduce their technical debt. I have seen and read about organizations that have been able to rapidly reduce their lead times while still refactoring and removing backlog. However, the goal of DevOps is to improve the entire flow, not just development, and a good majority of the release aspects of applications will require organizational debt to be addressed to enable improvement across the entire continuous delivery pipeline.

February 6, 2015
DevOps needs infrastructure multi-tenancy

DevOps will, under whatever moniker you want to give it – NetOps, DevOps for Networks, operationalization – continue to make inroads into the network infrastructure because ultimately it’s part of the application deployment lifecycle. While DevOps folks call it “application delivery” the reality is that an application is not ready to deliver to a user (internal or external) until all its requisite services have been provisioned and configured.

Yes, infrastructure and network devices are increasingly API-enabled and supportive of a variety of tools and frameworks most often associated with DevOps – Puppet, Chef, OpenStack, VMware – and those more commonly associated with just the network – Arista, Cisco ACI, and OpenDaylight.

But it takes more than APIs. APIs do not inherently bestow upon devices the ability to support multi-tenancy. That is, the isolation of services unique to a team (or application) required to effectively deploy application-supporting infrastructure services within what is traditionally a shared environment.

According to ThoughtWorks Technology Radar (jan 2015) partitioning infrastructure along team bounds is becoming an imperative to removing roadblocks in the application deployment lifecycle:

Many of our customers have made DevOps a reality in their organization with delivery teams that build, deploy, and support their own applications and services. Unfortunately, a regular roadblock on that journey is allowing teams to have superuser privileges in production environments. In most organizations, the production environment is shared, and therefore risky to provide access widely. It is effective when we can partition infrastructure along team bounds,so that those teams can have safe isolated access to do their work, without risking impact to other systems. Where cloud environments are used, this is much easier to implement, aligning account structures to team boundaries. [emphasis mine]

This has always been an issue where shared services are relied upon. A change to support one application (or team) can be disruptive or cause issues for another application or team. Thus, most infrastructure and network teams staunchly refuse to let anyone in the application or operations side of the house play with their toys.

They are the network ninjas; guarding their configurations with their very lives.

It’s this kind of conflict that needs resolution if we – as a unified IT organization – reach goals of improving the time to market and frequency of deployment. It doesn’t matter if the app team can deliver an app in 3 weeks if it takes 3 months to deliver it to the customer because of overloaded network and infrastructure staff.

There are two good solutions to this problem*.

1. The governance over more application-affine infrastructure services – load balancing, caching, proxies, etc… – is necessarily moving closer to the application and under the control of app and operations teams. This is supported by concepts like Network Service Virtualization, proposed in Lippis Report 217 as well as the increasing adoption of open source proxy-based (virtual and software) services. This model assumes continued shared network and infrastructure services as appropriate while migrating per-application services to the more agile, operational architecture.

2. The introduction of multi-tenancy in infrastructure architecture provides a similar approach while still maintaining shared resources (hardware). A single, high capacity system is able to host multiple, virtual instances of its services, each dedicated to either teams or applications, as is the organization’s need. This approach spreads the cost of the hardware resources across the entire organization while allowing a more per-application approach to provisioning, configuration and ultimately, cost. Isolation addresses the concerns of shared resources and configurations that can cause consternation among network and infrastructure teams because the underlying shared system is managed by them while role-based control over application services is offered to teams or designated responsible individuals.

Both approaches are valid and in many cases both architectural solutions will be used to resolve the issues caused by traditionally shared infrastructure.

The key takeaway here is that there are answers to the need for partitioned infrastructure that are supportive of DevOps-driven workflows and processes and can certainly provide a path toward ensuring a more seamless, time-sensitive end-to-end application deployment process.

* Oh, I’m sure there’s more, but right now these appear to be the best two options.

January 22, 2015
Change Management & Continuous Delivery: Clash or Harmony?
Introducing changes into running software systems is risky. There’s always a chance of failure, and that can have a negative impact on your business. It’s important to mitigate that risk, but there’s more than one way to do so.

Change Management is all about ensuring that changes to existing systems and software in production are done in a controlled way. Change managers review the changes that are to be implemented, assess their impact, and perform due diligence, before giving or withholding their approval.

Continuous Delivery (CD) is about improving the process of delivering software and increasing efficiency. It aims to cut down the time between ideation and realization. It’s about reducing the scope for error and increasing the speed to market to generate a competitive advantage.

Is there an inevitable clash between CM and CD? Are they really compatible? At heart, the CD pipeline is intended to achieve the same thing as CM by enabling the smooth, error-free roll-out of changes, but today they are usually coming from very different angles.

If CM is to have a future in the CD revolution, then it needs to change. Before we discuss how that might be achieved, let’s take a look at the current state of play

Is Change Management delivering?

Ask yourself what impact change management is having on your organization. If change management was working perfectly then there wouldn’t be any production problems, and yet issues still crop up. There are a few things to consider:
1. How many entries in the CM checklist could be automated? Is the change manager doing anything that really requires human input? Fact collection and verification could be a part of your CD pipeline.
2. How much value do check boxes deliver in the first place? What is change management really bringing to the table?
3. Is the process fit for purpose? Does it actually prevent problems, or is the change manager really just a fall guy, there to sign off on the release and take the blame if something goes wrong?
Clashing with Continuous Delivery

CM is intimately linked with the idea and practice of big complicated transformations, but with CD we’re not talking about enormous, disruptive changes. CD is about changing small parts of a system and doing it often. The risk to reward ratio is different when you think and work iteratively. Smaller change sets are less likely to cause problems and easier to rollback when they do.

Does the process really require human input? Heavyweight validation will slow it down and speed is one of the key benefits Continuous Delivery offers. Streamlining and automating the checklist process makes perfect sense in many business contexts. The aims of CM and CD are in sync, but CD is better placed to deliver.

A possible advantage of traditional CM is the presence of a human eye, offering intuition that a machine or a piece of software may not be capable of. By moving beyond the checklists and reimagining the CM role we can retain this advantage and push CD forward.

From Gatekeepers to Masters of Continuous Delivery

If the bulk of CM is really an administrative exercise in checking boxes then it could be an unnecessary overhead, but there is a way to retain that human oversight for organizations wary of full automation. Why not ask the former gatekeeper to own the CD process? For a CD pipeline to be truly effective someone has to guide the process, engage with all the teams involved and understand exactly why and how the process generates business value.

A change manager, paired with a build or release engineer to handle the automation, could work to help organizations move towards CD. They are uniquely well placed to identify dependencies, map processes, and help construct the CD pipeline with minimal disruption. They can help sketch out what the pipeline should verify, what data and dependencies need to be taken into account, what automated tests are required, and how it fits with other components to automate that release procedure. They also have experience mediating between business and IT.

The traditional CM approach of committee meetings and spreadsheet checklists is dated, and for many organizations it no longer makes sense. You can find much greater business value in focusing on an efficient rollback procedure that kicks in immediately and automatically when changes fail.

Take a look at your CM and ask yourself – is it really adding anything to the process, or is it just another bottleneck? It may be time for a change.

About the Author/Andrew Phillips

Andrew Phillips heads up product management at XebiaLabs. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.
January 21, 2015
Webinar: How to Achieve Continuous Delivery Without Downtime
Today’s businesses demand quick access to new technology, which means they need shorter release cycles and centralized control of both your on and off premise resources. Integrating cloud resources into your traditional datacenter management can prove challenging and time consuming at best. What if you could increase your organization’s ability to innovate utilizing business automation to bridge the gap to the cloud?

This webinar will be looking at how you can:
- Automate your deployment of code updates with the correct configuration
- Reduce errors by automating manual steps and scripts
- Achieve quicker deployment times to meet business demands
- Integrate with Jenkins build management
Join us on February 18th as Tom Flitter, Director of Applications and Integrations at TASC, discusses how TASC solved long release cycles, extended service downtimes per release, and highly manual processes for releasing their main application across all environments.

RECORDING

SLIDES

[seoslides embed_id=”b0034e485c27″ script_src=”https://devopsy.kinsta.cloud/embed-script/automate-your-world-tasc-v1-0-pdf3531/4773/” overview_src=”https://devopsy.kinsta.cloud/slides/automate-your-world-tasc-v1-0-pdf3531/” site_src=”https://devopsy.kinsta.cloud” site_title=”devopsy.kinsta.cloud” title=”automate-your-world-tasc-v1-0-pdf3531″ /]

Date: Wednesday, February 18th, 2015

Time: 2pm eastern standard time
```
REGISTRATION LINK:(CLOSED) 
```
About Speakers:

Tom Flitter is currently the Director of Applications and Intergrations at TASC (Total Administrative Services Corporation). He is an information technology leader with substantial experiene in application development, program management, project management and team development. He has years of experience in leading complex, strategic, cross-functional, and high ROI technology initiatives. He holds an MBA for Business Administration Management and has been working at TASC for 5 years. You can connect with Tom on LinkedIn: Tom Flitter

Ralf Paschen is currently the Business Automation Executive at Automic. Over the past years he has held various positions in the software industry. Coming from a technical datacenter role, he moved into presales, sales to product management, and product marketing in several software companies. He was responsible for the complete SAP portfolio at Automic and is now building the solutions marketing department to give customers and partners a clear understanding about the innovation value of the unique One Automation Platform. You can connect with Ralf on LinkedIn: Ralf Paschen or Twitter: @RalfPaschen.
January 14, 2015
Continuous Delivery snapshot: a universe of fragmented tools
Technology has ushered in an era of real-time information, immediate feedback and instantaneous response. As a result, there is tremendous pressure on development teams to deliver a continuous flow of updates and fixes to software applications. In support of this objective, development philosophies and methodologies abound: agile, lean, scrum, kanban and extreme programming, just to name a few. Development teams are adopting and implementing these new methodologies at a breakneck pace in an attempt to bring efficiency and speed to the development process.

This increased velocity has placed added pressure on software delivery teams to build, validate and deploy newly developed code faster and more reliably. Over time, the DevOps movement has emerged, injecting robust automation into the delivery process while spawning a diverse collection of hundreds of single-purpose delivery automation tools. As a result, DevOps has improved delivery speed and quality. Even still, continuous development comes with its challenges. Namely, the lack of a real-time “big picture” view, of the end-to-end software delivery process.

New Challenges for CD Teams: The Unified Pipeline

While highly diversified delivery toolchains can be effective and have increased speed and agility, significant challenges continue to plague the software delivery process as a whole.

Real-time visibility and visualization – Typically, the delivery process begins as soon as newly developed code is considered “dev-complete.” Often, the sequence of steps and stages new code progresses through on the way to potential release is referred to as the delivery pipeline. In a moderately sized organization, delivery pipelines are comprised of dozens of people performing hundreds of tasks. Without a centralized “delivery portal,” team members have little or no insight into the overall upstream or downstream process flow. While delivery pipelines benefit from diverse automation, they are still a “black box,” largely absent of communication, collaboration and audit/tracking capabilities.

Data fragmentation – Important delivery data is generated at virtually every step along the delivery pipeline. Disconnected toolchains generate siloed, disaggregated and uncorrelated data that is trapped within a dozen or more automation tools. As a result, team members often lack access to a single comprehensive audit trail document that fully describes the functional and technical contents of a specific release along with any relevant data generated during each step of the delivery pipeline. At a macro level, self-organizing delivery teams do not have the real- time and historical analytic data required to quickly identify and react to exceptions or make performance optimizations across pipelines and/or teams.

Is dynamic pipeline orchestration the answer?

While the build and integration stages of a delivery pipeline are largely static and predictable, pipeline flow can become much more dynamic once manual testing begins. Defects can be discovered at any point along the delivery pipeline and judgment calls often determine a dynamic process flow. Often tools designed to automate the build/integrate phase of the pipeline are stretched beyond their original intent when it comes to pipeline orchestration. While these tools automate a critical portion of the delivery pipeline, they are far too static to support a real-world pipeline flow through the manual steps that occur in the later stages of a delivery pipeline. As a result, pipeline orchestration is largely a manual process beyond integration and into the quality assurance phases of delivery.

What’s Missing?

When it comes to alleviating the challenges associated with delivery pipelines, what will it take to ease the burden on continuous development teams?
- Detailed Inventory of Each Bundle-of-Change – As multiple revisions get merged together in downstream pipelines to form a release candidate, a clear understanding of exactly what “changes” have been incorporated within any particular change bundle (release candidate) can become crucial.
- Current Deployed Environment – At any given time, any revision or release candidate can be deployed to one or more environments or locations. The ability to quickly determine which change bundles have been deployed to which environment(s) is a fundamental need of virtually every stakeholder along the delivery process.
- Broken code lines – Release candidates that contain known material defects are usually considered terminal and/or not deployable. Identifying a release candidate as broken can dramatically reduce the amount of time invested into a terminal release candidate. Subsequent testing within environments that contain a broken release often leads to wasted effort.
- Detailed release documentation that fully describes the functional and technical contents of a specific release-in-transit along with all relevant data generated during each stage of delivery.
- Reporting that describes stalled releases awaiting manual intervention with corresponding wait times.
- Work queues that outline manual tasks or pending decisions/approvals assigned to a specific individual or group.
- Travel time details on the average time it takes a release to traverse a pipeline from start to finish.
- Reporting that describes the “base” release (starting version) and all subsequent change that has flowed thru any given development or test environment.
What is the Answer?

Without a technology that can oversee the entire development process and fill these gaps, teams often try to leverage Continuous Integration (CI) tools. While these tools greatly reduce the effort required to automate the build/integrate phase of the continuous delivery pipeline, this stretches them beyond their practical limits.

However, a new type of pipeline orchestration technology that further streamlines and advances the effectiveness of the CD process is emerging and can effectively alleviate the challenges associated with fragmented and siloed delivery pipelines. This technology, often called a “pipeline orchestration framework,” expedites movement and raises awareness of a software “release candidate” as it travels through one or more delivery pipelines.

So how do they work? Pipeline Orchestration Frameworks:
- Leverage existing tools and processes – Because Pipeline Orchestration Frameworks sit on top of existing delivery automation tools, there is no need to reconfigure current process flow or abandon existing tooling. Pipeline Optimization Frameworks simply enhance and automate what is already in place.
- Provide audit trail – Comprehensive audit trail documentation of each part of the process delivers highly detailed and valuable information regarding each new potential release. Having this audit trail ensures higher production quality results, increases cross-team awareness and reduces the time and effort required to perform root-cause analysis when defects are found (pre/post-production).
- Manage deployments – Efficiently managing the cost and friction associated with dynamic development and testing environments over time can dramatically reduce the cost of delivery. Pipeline Orchestration Frameworks are capable of automatically provisioning and/or refreshing fully formed application environments as often as needed while keeping track of environmental drift or change over time.
- Workflow automation – One of the goals of continuous delivery is to automate as much as possible along the delivery pipeline. However there are often steps that can’t (or shouldn’t) be automated, such as code reviews, exploratory testing, and manual decisions to automatically move a release into late stage test environments or even production. Pipeline Orchestration Frameworks manage and orchestrate these manual workflows as efficiently as possible, ensuring all delivery team members are aware of items in their “work queue,” and identifying stalled releases immediately.
What will 2015 Hold?

The purpose of continuous delivery is to support development teams’ need to release reliable code faster. Therefore, the delivery team must keep up with an ever-increasing development velocity. Managing the delivery process not only requires managing each of the different phases of delivery, but also orchestrating the delivery process as a whole. That, in turn, requires visibility into every part of the end-to-end process, and a 10,000-foot view of the delivery pipeline. Pipeline Orchestration Frameworks offer CD teams a powerful new tool to unlock the full potential for successful continuous development.

About the Author

Dennis Ehle is the CEO and founder of cloud sidekick, which provides an automated, self-service integration and deployment framework for enterprises seeking continuous delivery. The cloud sidekick Velocity Pipeline Optimization Platform enables agile development teams to push code through multi-stage pipelines, automatically testing in virtual environments and ultimately publishing reliable code faster. For more information, visit www.cloudsidekick.com.
January 14, 2015
Breaking Builds II – The Butler did it
As editor-in-chief here at devopsy.kinsta.cloud and writing for several other publications I see more than my share of contests, promotions and other marketing campaigns (not to mention pitches from PR agencies), but I came across one this week that I really wanted to share with you. My friends at CloudBees, the Jenkins company are once again running their Breaking Builds promotion. In case you don’t know about it, it combines two very different characters. The Butler from Jenkins and Walter White from Breaking Bad. As Sacha Labourey, CEO of CloudBees says in his blog post on the contest morphing these two together is “probably one of the best examples of cognitive dissonance you can get.”

The marketing and engineering teams at CloudBees have come up with a series of banners with graphics of the Butler as Breaking Bad characters, especially Water White. They then have come up with captions for the images that are spoofs of lines from Breaking Bad and also have a Continuous Delivery angle to them. You can see what they have come up with here. Your mission if you accept it is to take the graphics and come up with a better caption than the CloudBees team. You then tweet the image with your caption and you’re entered.

There are prizes for several categories of captions. The categories are
- Most Original Caption for any of the 10 #BreakingBuilds images + your caption
- Most Humorous Caption for any of the 10 #BreakingBuilds images + your caption
- Most ReTweeted Twitter post* for any of the 10 #BreakingBuilds images + your caption *Dare we call this CI (Continuous Interaction)? 😉
- Most Favorited Twitter post for any of the 10 #BreakingBuilds images + your caption
- We may also create additional categories during the contest if we see an emerging theme!
The categories, prizes and full rules are also available here.

Personally my favorites of the images and captions done by CloudBees are these two:

So if you are a Jenkins and/or a Breaking Bad fan why not join in the fun. Give it a try. We will also feature some of the winning banners here on devopsy.kinsta.cloud
December 18, 2014
Continuous Delivery and Continuous Deployment
A question comes to mind in what exactly do we need to do to move forward with Continuous Delivery and Continuous deployment, How should we start or rather where we should start from. Many times teams may not have achieved the continuous integration properly, maybe due to lack of proper tools, workflow or historical process.

What we need to do before even thinking of CI/CD, what are our prerequisites:
- Source code
- Builds System
- Deployments
- QA Automation
We need to understand release != deployment

Lets assume with the help of right source control tool, build system and QA, automation team has achieved the continuous integration which means developers integrate code into a shared repository several times a day and each check-in can be verified by an automated build, allowing teams to detect problems early.

The next step: From CI to Continuous Delivery deployment

What exactly is Continuous Delivery, Its attitude to make software safe and to go fast
Lets begin with basics, How much time does it take to deploy one line change or a small bug fix in to the production? Exactly, our goal should be to achieve this as fast as we can.

Why We Need Continuous Delivery?
- Yes, its Speed which is needed in current fast paced software industry.
- Stop waiting around on each other to provide go ahead and open the gate.
- Quality – It raises the overall quality of your application – No surprises.
- Capacity and Clarity – Get every team on one tool – Eliminate manual error
How are we going to do this…..

Key Areas:-
- Speed is essential because there is an opportunity and cost associated with not delivering Software
- Frequent Releases, How frequent is frequent? Why do developers need to wait after writing code?
- Every build can be potential release candidate.
Bottom Line

Remove Manual Work/Steps.
Move towards complete automation

Delivery Flow

Here is the Delivery Flow.

Ingredients:-

Configuration Management
continuous integration/ Automated Build
Automated Deployment
Automated Testing

Obviously no single team in a company alone can achieve CD without the help of other teams. It needs collaboration and team work to implement it, Lets take a look at what do we need from various teams

Dev Team
- Collaboration – Integration is no longer an event
- Check in regularly
- Make small frequent releases because Big Change -> Big risk –> Big effort and Small Change –> Small risk –> Small effort
QA team
- Should have complete automation or whatever max is possible. More manual testing leads to more delay.
- Keep the Build and Test Process Short
- Break the QA automation into small pieces so that they can run simultaneously and do the QA frequently.
- Publish the report
Environment
- Should have complete Environment to test everything.
- Flexible and scalable so that multiple env can be spawn any time and can be extended.
- Managed as prod or at least similar to prod
This increases Confidence in build’s production readiness reducing release risk.

Main blocker to achieve CD
1. manual work
2. Lengthy Process
3. ticket system
4. approval
Now, How do we deal with blockers

Do not add gates, add only checks. Adding gates to prevent issues from creeping into prod is not so smart. There are many more issues hiding, you just don’t know them

Approval!!! Why do we need CAB?

Dev Sign off
QA sign off
Ops and Dependency sign off
Infra sign off

Dev Sign off -> Use proper project mgmt tool to track everything that goes in to software with adequate workflow. Get everyone together at the beginning so that these will not be surprises at the last moment. Keep meeting across the team during entire development. This way you don’t need any approval since you have already taken care of all necessary elements during the entire development process.

QA Sign off -> Publish consolidated QA report with each and every build which should be visible to everyone. You should have proper definition of “Go Ahead” instead of just random checks, Nobody has to ask QA for a thumbs up for the go. • Minimum criteria for Q has to be set.

Ops, Dependency and Infra Sign off -> Include them at the beginning of the project. Have capacity and change planning during requirement gathering, This gives enough time for ops and infra to get ready for the release.

Role of SCM/ Release Mgmt

SCM/ Release Mgmt provide the workflow, tools and necessary filler to enable all teams to interact and automate their part.
The picture below shows How SCM and Release Management help each team.

Source code support for Dev by defining branching policy, Build Setup Support for Dev team to provide continuous build with predefined configuration and in managed environment which includes static code analysis like unit test, code coverage, checkstyle etc..

Enabling QA Automation and reporting via build tools (Like Jenkins) to support QA and Dev to run their automation and aggregate report on central location. which include QA automation/regression, functional code coverage etc..
Infrastructure for QA Support like managing integrating environment, Deployment Automation etc.

Ops, Dev and QA CAB Automation and release planning support.

It’s clearly visible that CI and CD is not a one man show. Planning, collaboration and teamwork are essential parts of the execution.

Tools can be chosen by the org/team whichever are suitable for them, but the main part is automation and integration of the tool with each other to make flow smooth and free.
December 11, 2014
Does Automating Application Release Mean the Same that it Used to?
Over the last ten years, we have seen a shift towards automation in how applications have been released. Automation has been encouraged first by agile’s embrace of continuous integration and delivery, and now the broader move to DevOps. As deployment automation left the sphere of the individual development team, another trend has emerged. The understanding of what it means to deploy an application in an automated fashion has steadily expanded. For developers in the early days of continuous delivery, production concerns were usually secondary. They would take a single build, and automate its deployment to test environments. Some teams would succeed in expanding this approach to production with a full-fledged delivery pipeline, but most would stumble. Ignoring production concerns in the lower environments resulted in their work being dismissed as unworkable in production environments. However, the success of agile development meant an increased demand for change in production. At the same time, the trend towards cloud services expanded the number of targets that a given deployment would impact. Production support teams had to automate, and in a way that would also work for the rapid rate of change in development and test. The build pipeline model was close, but didn’t meet the Ops needs of incremental change (rather than full builds), and orchestrating changes across multiple builds and tiers.

A 2012 survey of over 500 technologists by UrbanCode found that only 12% of teams scoped production releases to a single build. With the business demanding faster change, and DevOps ascending, we have seen automation mature in the production environments rapidly in the past several years. This includes the rise of Application Deployment/Release Automation as a tool category at the start of this decade. These ADA tools are defined by automation scoped at a business application – they coordinate across multiple tiers and components. So they deploy not just the new version of a web service, but also five other web services the backend databases, etc. With DevOps better linking development and operations, and an increased willingness to automate in production, we saw the first big shift in the scope of automation. From deploying a build, to deploying the build and related builds, content and database changes. The web app was still the most common scenario. Today we are seeing the next major expansion underway, and it’s hitting on several fronts at once.
1. The first front is that organizations whose releases span multiple business applications are adding release centric orchestration to speed releases of coupled applications.
2. The next front is a move beyond the web application itself. Mobile development tied to backend services is getting pulled in to the definition of the “business application” as are systems of record including the mainframe.
3. The third front is a shift down the stack. The environment and infrastructure are seen increasingly not as a place the application runs, but as part of the application itself. As that shift happens, we see a push to make a change to the CPU count in a VM go through the same types of automation and governance that updates to the behavior of a web service go through. Change is change.
4. Finally, we are seeing the normalization of Platform as a Service offering like Bluemix. That results in increased acceptance of the technology, but also the desire to coordinate change to services delivered through a PaaS platform and those delivered through other cloud or traditional data-center platforms.
Broadly, IT is delivering value to the business by delivering services. Changes to those services tend to be scoped to applications that are complex. The trend is to treat everything in that application that is software defined the same through the same channels. That lets system level testing validate the whole system, and for the whole system to be promoted forward together. The above discussion not only informs about the change taking place but also prompts at taking the step towards the change. The move to DevOps is bringing about change rapidly. On November 20, some of the industry experts and leaders got together to discuss the effects and steps to adopt the change effectively, at the IBM DevOps Symposium in the New York area (Jersey City). Understand how to adopt DevOps successfully by listening to distinguished industry leaders and experts. Join us for the replay of the webcast. Be a part of the change

About the Author:

Eric Minick is a Technical Evangelist. He joined IBM through its acquisition of UrbanCode. He has spent the last ten years helping organizations – large and small – adopt continuous integration, delivery and now –DevOps. He is a frequent speaker on the topic, and co-authored Application Release and Deployment For Dummies. Prior to consulting, Eric was a developer, test automation engineer and support engineer, and has contributed to multiple generations of UrbanCode products.
December 9, 2014
DevOps: As critical for the Systems of Record as for the Systems of Engagement

There is a lot of talk about DevOps, and how it improves the process of delivering business value. Much of the discussion of DevOps has been focused on web based companies such as Facebook, or the mobile and distributed development teams. But DevOps principles apply just as well to the traditional Systems of Record, z/OS applications.   Many of these applications have been around for a long time and have very mature development and deployment processes. These processes include the requirements for separation of duties and audit tracking. Over the years these processes have developed into lengthy procedures, to be comprehensive, the net effect of this has been to reduce the number and frequency of the deployments to reduce risk. However, there is a better way.

In the recent blog posting by Ashok Reddy he talked about the 8 critical DevOps Practices: innovate, deliver, repeat. All of these are as applicable to z/OS applications as any other, for this blog I want to focus on just a few of the practices, Loosely Coupled Architecture, Automate Testing using APIs, Transparency and Minimize hand-offs, Maximize flow.

DevOps is about bringing the entire organization together, including Development and Operations, to more efficiently deliver the business value. The practices of “Transparency” and “Minimize hand-off, Maximize flow” go to the core concepts of the people aspects of DevOps i.e., integrating the organization.   By breaking down the barriers between the organizations, people can work together to accomplish the business goals more quickly. This does not necessarily mean actually changing the organizational structure, but it does mean having the organizational structures work together as cross functional teams.

One example would be the monitoring and optimization practices. These are generally done by Operations after the applications are deployed into production. However, if instead Operations and Development worked together in the early part of development cycle to monitor and optimize the application using the same capabilities that will be used in production any issues can be addressed much earlier in the cycle. This also helps both teams learn what is required to monitor and manage the application long before it is ever deployed to production. In many z/OS environments today, the same infrastructure is already available in the test environments, but it is seldom used.

A second example is to have Operations define the deployment automation such that it can be done by Development in early stages and turned over in later stages. This ensures continued testing of the deployment capability and a validation that by the time the deployment is done in production; the deployment process itself will be well tested. Through this integration of Development and Operations you remove the hand off that currently happens only at the final stage just before production deployment.

The practice of loosely coupled architecture is key when building multi-platform applications. Having clear interface definitions between the different parts of the application is essential of a loosely coupled architecture. It is easier to allow the capability to build independently, tested at the interface level and then brought together when the parts are ready. By defining clear interfaces between application components and properly handling forward and backward compatibility, there is less of a requirement for coordinated simultaneous deployments. This allows teams to be less dependent on other parts of the application during the development process, and allows for easier automated testing.

The practice of Automate testing is critical to any DevOps transformation. It provides the capability to improve the speed of delivery while reducing risk. Moving to agile development practices while still requiring people to perform manual tests, will not allow the quick iteration on the capability. Even for existing applications you can generate interface tests to begin to build up automated regression test cases. Automated tests can be generated based on interface definitions such as a JSON schema or a COBOL copybook for new development. The increased focus on testing the interfaces early helps reduce the integration problems generally found late in the development cycle.

These are just a few of the key practices of DevOps but hopefully they have started to demonstrate how DevOps is as applicable to the Systems of Record on z/OS as it is to Systems of Engagement.   To learn more read the best practices for a DevOps approach with IBM System z whitepaper or watch the replay of the November 20, IBM DevOps Symposium webcast.

About the Author:

Rosalind Radcliffe is a Distinguished Engineer within the IBM Rational organization. She is Chief Architect for CLM and DevOps. She is responsible for driving the DevOps for multi-platform architecture. This includes System z and Power system. In addition she is responsible for the architecture for the Collaborative Management capability for Enterprise solutions. This includes UrbanCode Deploy and Rational Team Concert’s support for standard Mainframe development activities.

She is a member of the IBM Academy of Technology and a Master Inventor. Prior to Rational, she was in Tivoli responsible for the SOA Management Strategy for IBM.

December 2, 2014
Choose Digital Chooses Jenkins For Continuous Deployment

Challenge: Shrink Time To Innovate

Choose Digital offers a white label media store solution (think iTunes) for enterprises that want to reward brand loyalty with opportunities to use miles, points, or cash to buy digital products. As a competitive media company, Choose Digital was eager to test and adopt innovations to its software platform as fast as its developers could produce them. The media firm was ready for a model that lifts the burden of maintaining tools in-house.

“We were looking for a way to move from a straight compute environment to a DevOps approach that would enable us to push lots of code and new builds out daily while quickly iterating a lot of new ideas for improvements to our digital marketplace,” says Mario Cruz, Co-Founder & CTO, Choose Digital. A managed service could also relieve the headaches of supporting all the development applications internally. Choose Digital chose CloudBees’ Jenkins , a Java-based pipeline and orchestration tool for continuous deployment, which was also available as a hosted service, to meet those needs.

Solution: Simplification

Once Choose Digital made its decision, CloudBees made it simple to get up and running. “We opened our account with CloudBees and were productive with our Dev and Stage environments in two days using a fully functional Jenkins trial implementation,” says Cruz. Choose Digital moved production into the cloud 30-days later.

Jenkins enabled Choose Digital to find and fix bugs and try new things fairly quickly, without waiting for monolithic releases. “Jenkins is how we innovate, test our code, and enable the coder to immediately fix what he breaks,” says Cruz. Jenkins provides continuous integration services for software development and enables Cruz’s team to run build tests and functional tests.

“Jenkins gives us a quick snapshot so we can find broken code right away, not five weeks from now,” says Cruz. Enterprises that are still using older approaches don’t typically have time to fix small bugs, and certainly can’t fix them right away.

Choose Digital uses Jenkins in the cloud (DEV@cloud), which CloudBees fully manages and hosts on AWS EC2 http://aws.amazon.com/ec2/. Cruz doesn’t worry about maintaining the environment or whether his media platform has enough space or capacity. “CloudBees manages all the updates and issues with Jenkins CI. They support all the plugins for Jenkins, even if it is not their own plugin,” says Cruz.

As a managed service, Jenkins simply works from machine to machine. “My developers can focus on the product. They don’t have to worry about whether they have the right software build, the right apps on their machines, or the right versions of those apps,” says Cruz.

“As a result, development is up to 5 times faster in our continuous deployment environment than it would be if we had set up our own hardware or used an IaaS solution. Things that could take months or years we can now do in days,” says Cruz.

A Bump In The Road

As Cruz and his team moved to Jenkins, the biggest challenge was communication. “The toolset is not the challenge, it’s having everyone thinking with and getting used to the new DevOps mindset,” says Cruz; “while we were fully functional in two days, it took two weeks to get used to the new mindset.”

Today, Cruz’s software team communicates across stakeholders and team members in a DevOps approach that leverages Slack https://slack.com/ (IRC) and HipChat https://www.hipchat.com/ plugins for Jenkins. “DevOps is more a culture than it is tools; DevOps is all about communication. We communicate to our team, to the QA department, and to product managers,” says Cruz.

Current State: Everyone Knows, So We Can Fix Everything

Jenkins brings transparency so that what only some could see—such as what a developer has broken—becomes everyone’s knowledge. “Things get flushed out pretty quickly. By using Jenkins CI, you see the hidden things that no one knows about quickly,” says Cruz.

Cruz has some realistic advice for his peers: The sooner you get to an environment and mindset like Choose Digital uses, says Cruz, the sooner you will see the benefits and flush out bigger problems. “It’s not perfect the first time around but it gets better and better on every build. You get rid of the technical debt you didn’t know you had,” says Cruz.

November 25, 2014