devopsy.kinsta.cloud

Author: contributor

Nationwide banks on DevOps to drive towards Continuous Delivery

The Financial Services industry is fast moving, highly regulated and customer-centric. The general perception in our industry is that any change in processes implies risk and delay, neither of which we can afford. But at the same time, we are faced by the mammoth task of needing to continually introduce new products and services to stay competitive and cater to the growing needs of our customers. This is unthinkable without a fast software delivery lifecycle, as our business is highly dependent on technology. The best way to accelerate software delivery in a complex, fast moving environment like ours is to adopt a DevOps approach.

As a speaker at a number of events, I have discussed the ongoing transformational journey at Nationwide and shared our approach with other enterprises who want to understand how they can follow suit. I recently had the opportunity to co-present with Hayden Lindsey from IBM at the DevOps Enterprise Summit (Oct 21 – 23) hosted by Gene Kim. You can watch a replay of our joint session Patterns for Enterprise Success: The DevOps Journey at Nationwide below.

From humble beginnings as a small auto insurer serving the farming community, Nationwide has grown into one of the world’s largest insurance and financial services companies. Like many businesses today, we saw that we needed a better way to respond to changing market requirements and regulations, to increase speed to market, and to support new channels for customer interaction. Mobile, cloud and big data technologies were further raising the pressure on us to facilitate rapid digital innovation. What we required was not just a solution, but a change in culture (the way we look at things), processes (the way we do things) and tools (the technology we use). DevOps gives us the capability to achieve all of this—and more.

The immediate challenge was to achieve these changes by leveraging our existing heterogeneous tool suite. We partnered with IBM DevOps, which has played a significant role in our success. We were already working towards agile transformation. With IBM’s support, we were able to use the DevOps approach to application development and delivery across our distributed and mainframe environments. While a DevOps approach as supported by IBM enables organizations to start anywhere based on their specific needs, we at Nationwide decided to address all four adoption paths across the software delivery lifecycle: Steer, Develop/Test, Deploy and Operate.

DevOps enabled our IT department to perform continuous integration of its code and continuous deployment into its development environment several times a day. Teams can also perform acceptance testing of customer requirements within the same iteration as development. IT managers can show their internal customers, in near-real time, what developers are producing. This capability became a game-changer as we became more agile as a business and more responsive to customers. We achieved a 50% improvement in code quality, a 70% reduction in system downtime and were able to move 58% of our teams into the top quartile in key productivity measures. I believe that, the more agile we can be from an IT perspective, the more agile the business can be in terms of going from the development of an idea to the deployment of a customer-facing product.?

The next step in our DevOps journey is to build on the gains generated by the adoption of Agile and Lean practices across the entire SDLC. This is where technologies such as TaskTop and UrbanCode come to the fore by providing the integrated continuous flow and visibility needed for a continuous delivery infrastructure. By creating this delivery highway, we are giving Business Areas the ability to deliver as fast as they need to based on their own determination of cost, risk and value. Time-consuming manual processes based on out-of-date information are replaced with automated deployment policies that use real-time information synched from systems of record. The combination of dashboard and dials gives the business the visibility and control to drive at a much faster pace than is currently possible. Lack of trust in development outcomes has historically been an impediment to speed. Our new automated approach is giving us the confidence to move faster to serve the business without compromising the quality of our deliveries.

IBM was a great partner for us here as they share our support of the open standards approach, which gives us the ability to work with a heterogeneous set of tools, across the full lifecycle, allowing us to determine holistically the best tools, the best architecture and the best approach to meet our goals of high quality, high productivity and faster delivery for our customers.

By balancing speed, cost, quality and risk, DevOps is helping us increase our capacity to innovate. That in turn helps us get to market faster and be more competitive. I mentioned during DevOps Enterprise Summit that the phrase “time is the new currency” resonated with our goals of optimizing the path from “concept to customer value”. We need to ensure we don’t let it slip through our fingers. We have long been dependent on manual processes, but in this new digital age they are only going to slow us down. This is the time to be agile, competitive and pragmatic. It’s time for all of us to break the age-old shackles and embrace the opportunity for change that DevOps represents.

About the Author

Carmen DeArdo is the Director of Application Development Tools and Technologies at Nationwide Insurance. Carmen is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across Mobile, Distributed and Mainframe and other technologies. This includes recommendations and implementation of technologies used across the development life cycle (e.g. IBM Rational tool suite, open source technologies).

November 12, 2014
3 Prerequisites before Jumping on the DevOps Bandwagon

Top Three Prerequisites before Jumping on the DevOps Bandwagon

The DevOps frenzy is truly upon us and organizations are under increasing pressure to jump on the bandwagon in order to deliver software faster and faster. However, Agile delivery and DevOps is not a plug-and-play proposition – it requires time, effort and persistence. Also most enterprises, unlike Silicon Valley startups often held out as shining examples, don’t always have the luxury of dealing with newly designed software and freshly minted teams. Implementing DevOps practices is a significant change and often an uncomfortable process for many enterprises. Let me share what I believe are the top pre-requisites that must be in place to ensure success.

Organizational Commitment

This may seem obvious, but ensuring alignment and setting the right expectations on the goals and path to DevOps nirvana is a first and important step. Organizational structures, support at a senior level, a clear but flexible roadmap and adoption of common objectives are key concepts to drive organization commitment to this transformation. Let me elaborate on these a bit.

Overcoming cultural differences between Dev and Ops organizations is essential – and often enterprises create a new “DevOps” organization with personnel drawn from both organizations chartered with instrumenting the handoff from Dev through to production rollout. This new centralized org, however, also runs the risk of becoming yet another silo. Ideally a DevOps group should operate as a center of excellence managing the tools required across the lifecycle and guiding/overseeing the right methodology within the Dev and Ops teams. Similar to the way Agile development practices have driven testing and quality assurance methods deeper into the dev cycle, DevOps methodology demands that configuration and deployment requirements in a true production environment be considered and tested as part of the design and development cycle. On the Ops front, this new methodology demands a shift from managing risk by limiting change towards enabling change while reducing risk by automating repeatable steps for deploying small packets of changes often.

While DevOps initiatives are typically sponsored by high ranking IT executives, it is essential that senior and mid-level leadership in the Dev and Ops organizations completely support and prioritize the changes that, while benefiting the overall goal of shipping software faster and better, may in the short term seem to impact narrow departmental goals. Continued visibility and focus from management, and joint ownership by Dev and Ops leaders goes a long a way in ensuring the teams stay aligned.

A clear and well-articulated vision and compelling reason for change communicated clearly through the organization is essential for execution. This must be accompanied by a phased roadmap that enables short-term wins and clear milestones. Besides benefiting the business, the aim is to make the jobs of Ops and Dev personnel easier. Transparent and frequent communication on progress against the roadmap and its impact is key.

In order to keep the momentum avoid getting bogged down in a particular step in the roadmap, especially early in the process. Most enterprises present target rich environments and a flexible roadmap focused on ensuring demonstrable rapid progress will ensure continuing focus and commitment.

DevOps relies on a sense of shared responsibility and mutual respect between Dev, Ops and the business. Organizations can encourage and engender this by instituting joint ownership for a common set of objectives that encompass both ‘change velocity’ metrics and ‘quality of service’ metrics.

Automation with discipline

DevOps relies on the “automate everything” mantra. However, its important to realize that automation does not simply mean speeding things up because machines are faster than people. Automation is a means to codify repeatable practices and steps to ensure speed with reliability.

Automating poor practices can actually be counter-productive and hence it is first necessary that the various steps and procedures in the delivery process be rationalized, optimized and made consistent.

The key to improving velocity is to minimize variability in the process of deployment through the path to production. It’s therefore important to automate the complete configuration and deployment process in the lower environments (dev, test) first, and then clearly track and minimize changes to these in higher environments (stage, production).

In addition to speeding things up, automation can also serve as a powerful way to ensure adherence to process, and done right can substantially help compliance and audits by ensuring change is made quickly yet consistently and is inherently documented without overhead. It is also important that automation of both the rollout as well as verification go hand in hand.

Tools and infrastructure

Easy to adopt tools in a frictionless infrastructure is a critical component of DevOps. Rather than locking in a toolset up front, its important to have to have flexibility as the organization evolves its practices and the tools that enable them. Open source tools are well suited to this. They incorporate new innovation faster, provide the ability to tailor them to meet specific needs, and developers can leverage the community that typically exists around these for best practices and knowhow.

It is also a good idea to optimize and fine tune the tool chain for the project or program that is the initial focus. Once a degree of repeatable success has been established, and the team has acquired proficiency, the tool chain can be scaled out to more broadly. It’s always harder to adopt a new tool chain, especially when coupled with a change in behavior and approach, if the scope of it is too wide initially. As discussed earlier, establishing a DevOps center of excellence team that owns and evolves the tool chain and its usage is often a good practice for enterprises.

Summary

As more enterprises rely on software to drive revenue, the pressure is on to innovate and deliver value faster. While this is indeed exciting, it requires a significant undertaking in rolling out cultural, organizational and methodology changes needed to enable DevOps practices. Ensuring clear alignment and goals for all stakeholders, a common sense of purpose and an achievable yet ambitious roadmap are key to securing organizational commitment for this endeavor. Approaching automation in a complete way and enabling the adoption of the right tool chain can set enterprises on the right path to realize benefits of small, frequent deployments without compromising quality of service and compliance.

About the Author

Kartik Raghavan is the Senior Vice President of World-wide Engineering, Operations and Support at CollabNet where he brings deep experience in ideating, defining, developing and launching products through On Premise and SaaS channels.

November 12, 2014
Product Management’s Role in a DevOps World
When we started Datical, we knew that in order to be successful we needed to practice what we were preaching: DevOps. As the VP of Products and product owner of Datical DB, our flagship Database Lifecycle Management tool, I recognize that Product Management is well positioned to facilitate DevOps adoption across the entire organization. It’s our job to ensure that the solutions we deliver to the market are aligned with our customers needs, valuable, and appropriately represented. To achieve this end, my team has embraced the Three Ways and established practices that map them.

The First Way: Systems Thinking

“Systems Thinking” in the product management group goes beyond application development and delivery. Our PM team must also consider how we improve, market, sell, and support our solutions to fully realize our company’s potential. To gain the deep understanding of all aspects of our Product Lifecycle, we’ve agreed upon the following guiding principles:
- Every Sales Call is a Product Call – My team and I spend a lot of time shadowing calls in all stages of the sales cycle. By attending as many calls as we can we learn what first motivates our market to seek a solution in our space. We get insight into what features resonate most and what areas of our solution are uninteresting to them. We hear how our Sales team is positioning the product and we understand the challenges they face when presenting our solutions to interested parties.
- We Are a Required Member of our Scrum Teams – The PM & Engineering organizations are highly engaged with one another. Product & Engineering management meet 2-3 times a week to discuss sprint progress, new development stories, and long term product roadmap. Product Managers attend daily development stand ups to answer questions and eliminate roadblocks in the current sprint based on the needs of internal and external customers. We balance technical debt, infrastructure requirements and feature development during sprint planning to ensure high-quality product releases and a reliable delivery chain.
- We are Marketing’s Lyricist – Our deep understanding of our customers, our market, and our products allow us to arm our marketing group with the content and messaging they need to be heard over the chatter. We influence marketing plans with what we know is working in sales and what we know is coming out of engineering.
- Everyone is Customer Support – We take reactive and proactive participation in customer support seriously. PM responds directly to customer support requests and we constantly reach out to customers we haven’t heard from to make sure they are getting value from the product and that their needs are being met.
The Second Way: Amplifying Feedback Loops

The Second Way, Amplifying Feedback Loops, is where the product group provides the greatest value to the company. By adhering to our guiding principles of Systems Thinking detailed above, the product group has a good handle on all aspects of our business. We know how we sell, we know what we need to do to increase our value, we know the engineering effort involved in delivering new features, we know where our customers are thriving and where they are struggling, and we know what we’re telling the market.

The physical manifestation of the Second Way in the product group is our Product Roadmap. Too often, roadmaps become little more than flashy marketing tools that promise a lot, but often fail to deliver on the hype. We’ve found that customers respond more favorably to a smaller roadmap that is accurate than they do to a road map that is full of wild, cleverly named features whose delivery is uncertain at best.

At Datical, we take what we’ve learned from participation in sales and support activities to prioritize our development efforts on real customer needs. We apply what we’ve learned from engineering about the effort involved to deliver on our customer prioritized feature list, balance those estimations with what’s necessary to maintain a smooth operating environment for development, and provide realistic delivery goals for product features. Once we have dates for new feature delivery, marketing can plan their strategies ahead of time and sales can speak with confidence and certainty when discussing our future with prospects and customers. When every person in your organization can speak confidently about your customers, your product, and your market, the deck is stacked in your favor.

The Third Way: Embracing Experimentation

Product Managers that know their customer’s problems are crucial to reaping the most reward from Embracing Experimentation. A great example of this is a feature we are releasing in our 3.0 version of Datical DB this Friday. We call it the Rules Engine, and it is the direct result of Datical’s culture of encouraging experimentation.

Datical DB has a feature called Forecast that allows you to simulate your proposed database updates in memory without actually affecting the database instance. This takes the uncertainty out of the deployment process by identifying errors prior to execution. Customers really liked the feature, but there were always questions about the extent of Forecast’s capability. Typically they would ask about standards and best practices for their specific organization and whether we could modify the Forecast feature to accommodate their specific use cases. If we were to commit to modifying Forecast to suit everyone’s needs, it’s all we would have time to do. We recommended that the customers manually review the changes in those cases before promotion to downstream environments, something they were already doing with piece meal SQL scripts.

While discussing the frequency and consistency of these feature requests with our VP of Engineering, we had an idea. What if customers didn’t have to rely on us to modify Forecast for their specific needs? If we could deliver on this, we would drastically reduce the time and money companies were spending on reviewing database change prior to approval. We decided to turn our best minds on the idea with a very sparse set of loosely defined requirements.

Our first attempt failed. We reviewed the results of our research and prototype attempts, refined the story, and tried again. Our second attempt showed promise. We refined it further, built on what we learned, and had a viable solution in sight.

We began talking to our customers about it. I actually heard a grown DBA giggle with excitement! He had DBAs on his team spending 70% of their time manually reviewing SQL scripts from development to make sure they were in compliance with their standards.

After collecting customer feedback, we combined it with the results of our technical experimentation, and are very thrilled to be bringing this feature to market at last. By encouraging experimentation and not bowing to initial failures, we’re delivering a great differentiating feature to the market.

About the Author

Pete Pickerill is Vice President of Products and Co-founder of Datical. Pete is a software industry veteran who has built his career in Austin’s technology sector. Prior to co-founding Datical, he was employee number one at Phurnace Software and helped lead the company to a high profile acquisition by BMC Software, Inc. In addition to managing a development team and product schedule at BMC, Pete worked directly with large financial services institutions to provide the features they needed to achieve greater efficiency and cost savings in their application deployment processes.
November 6, 2014
Supporting Production Applications the DevOps Way
Software applications are more complex than ever. If your application is slow or down for even five minutes customers will blow up on Twitter for the whole world to see. To prevent this, developers and operations have to work together in lockstep when deploying and supporting applications. Preventing and fixing application problems quickly is critical to keep your outage from becoming a trending topic on Twitter.

The DevOps Movement Begins

For software developers, ensuring early success of your new application starts with keeping IT operations in the loop. Traditionally a lot of software development teams have worked for months on new projects and then thrown them at the IT operations team at the last minute when it came time to deploy them. If you want a good example of this, look no further than the big healthcare.gov debacle. The DevOps movement started originally to try and solve that problem. It fosters a culture of keeping development and IT operations working together from the beginning to the end of the project so software deployments go more smoothly.

Long-term success of your application depends on proper maintenance and monitoring of your applications as you continue to roll out new features. Thanks to agile development, that is usually a weekly or monthly cycle at most companies. A constant stream of new features is awesome, but with that comes some risk in introducing new bugs, performance problems, or instability. This constant stream of change coupled with the overall complexity of today’s applications makes them difficult to support

Application Support: “It Takes a Village”

Collaboration between developers and IT operations is needed to keep your applications running and your customers happy. If you only have a couple senior developers who handle all the production support issues they will have little time to deal with more important issues like architecting new features.

The key to supporting production applications is removing the bottlenecks and getting more people involved in application support. Operations needs to make sure that the development team has visibility of basic troubleshooting information like errors, logs, and key metrics while ensuring security.. Without this information it is hard for developers to fix basic bugs and improve application performance. They must contact other developers or system administrators for help which creates a huge bottleneck. You could say “It takes a village.”

Granting access to production data raises both security concerns and fear of unauthorized changes to the servers without going through proper release processes. However, developers don’t really need administrator level access to servers (jumping between dozens of servers is not very efficient anyways), they just need visibility to the aggregated information. This will allow them to find the source of the problem fast, while not creating any security risks or letting developers create new issues while fixing others.

DevOps Should Include Production Support

The goal of DevOps is to create collaboration and improve the working relationships between development and operations. The DevOps movement initially started focusing on software deployment and continuous delivery. However, support of production applications is the second and equally important component in the application lifecycle that is sometimes overlooked. To solve production problems quickly, developers and operations can not be pointing fingers at each other. They need to work together as a team.

To help support production applications, developer need access to lots of data:
- Visibility and notifications around all application errors
- Access to centralized aggregated log files for viewing and searching
- Basic server utilization trends and stats. (e.g. CPU, memory, etc)
- Recording and alerting of key application metrics (KPIs)
- Tracking of application web page load times
- Ability to access the application database and run test queries
Most companies utilize several different tools to track all of this data. This makes it difficult to correlate and identify issues. It is also very hard to train all team members on how to effectively use all the available tools, not to mention the economic inefficiency. Companies should look for solutions that combine these features into one dashboard so that solving issues can be done quickly and with minimal training.

Companies should look for tools that gives developers access to all the data they need to monitor and troubleshoot production applications, without giving them administrator type access to all the servers. One application dashboard that will show them where the app is deployed, the current health, performance stats, recent errors, full logs, and key metrics. This is what we try to do here in Stackify – to make it easy for companies to embrace DevOps when it comes to supporting production applications. Some of our customers have shared with us how DevOps affected their organization. We would love to hear how DevOps has affected your company.

About the Author/Matt Watson

Matt Watson founded Stackify in 2012 and as CEO provides the vision and leadership for the direction of the company. Prior to founding Stackify, he was the founder and CTO of VinSolutions. VinSolutions’ innovative product offering led to rapid company growth and was even featured on the Inc. 500 (#447) in 2010. Matt is an entrepreneur at heart and excels at product and software development.

Feature photo by JD Hancock
November 6, 2014
Improve your Orchestration to Deliver More Features to Customers, Faster

The big business challenge facing all companies today is how to deliver more features to their customers, faster. The expectations of today’s end-users are increasingly shaped by the experience of web and mobile applications that ‘just keep doing more’ automatically or at the click of a button.

With competitors able to release new features within days or even hours, companies can no longer afford unpredictable, lengthy and inefficient release processes that barely support one update every couple of months.

Unsurprisingly, companies are accelerating and streamlining their delivery pipelines to stay “ahead of the game” and succeed in today’s competitive marketplace.

The Situation Now — Inefficient Orchestration of Processes and People

The current reality in most companies is that orchestrating all the steps needed to get a completed feature into production is a time-consuming and effort-intensive process.

Even if the first part of the delivery pipeline, such as deployment to test or QA, is already automated, the “last mile” to production is generally a complex, manual effort. Typically, the release manager spends long meetings with all the teams participating in the release. Then, they draw up a spreadsheet or document that defines the release plan which is subsequently modified numerous times following comments from the planning team.

When the time comes to execute the release, the technical teams will use a whole bunch of tools to actually carry out the work – different tools from those that are used to plan and track the release.

Since there is generally no direct link between the planning tools and the tools used to execute the release, the release manager must constantly keep chasing the technical teams to find out how things are going.

This lack of integration and shared insight between the “technical track” and the “management track” ensures that releases in enterprise environments are typically highly reactive and stressful events. Release managers have little or no warning of potential conflicts, delays or bottlenecks within and especially across releases, and hence little ability to act early to prevent surprises.

One level up, service delivery managers and business owners are unable to compare and analyze the planning and evolution of releases across the entire service portfolio to identify the areas of the process that need to be improved in order to accelerate and streamline feature delivery.

Streamline and Accelerate: a Roadmap for Enterprise Releases

The first step towards achieving an accelerated, streamlined release process is to provide integrated insight into the technical and management tracks – a single, shared pane of glass for the entire release team. This pane of glass enables all manual and automated activities to be tracked and executed.

Once the release manager is freed from the dual chores of catching up and fire-fighting, he/she is better able to identify and prevent conflicts and bottlenecks in advance. In addition, the entire team benefits from a full audit trail that tracks the evolution of releases, visualizes the degree of deviation from the original plan, and quickly highlights unpredictable applications or projects.

Any technology you consider to assist your push for pipeline orchestration should promote process and data analysis, and provide comprehensive metrics and reporting — allowing you to track the effectiveness of your efforts.

DevOps and Continuous Delivery Growing Popular

The technical track of releases rarely proceeds flawlessly in most enterprises. Many tasks, especially in application deployment, environment provisioning, and configuration are still manual or semi-scripted — consequently slow and error-prone. Those tasks are immediate candidates for improvement. Savvy companies seek improvements via DevOps and/or Continuous Delivery (CD) initiatives.

Those initiatives usually involve introducing Continuous Integration and Application Release Automation, a measure of test automation, and on-demand provisioning and configuration of virtualized environments. The goal then is to integrate these to create an “end-to-end” pipeline that can orchestrate the appropriate build, environment provisioning, deployment and testing steps.

These steps can be linked using something as simple as chained jobs or builds in your Continuous Integration tool, or you can make use of one of a number of dedicated automation orchestrators.

If you are starting a greenfield project or working in a small, highly flexible organization, end-to-end automation all the way to production is often possible. In most enterprise environments, though, releases still require a mixture of manual and automated tasks to be coordinated across multiple teams, as well as the ability to deviate from the original plan in a controlled, auditable manner.

Since a streamlined, highly automated release process remains your end goal, it is critical that your orchestration tooling supports fully automated CD-style pipelines and DevOps-style automated platforms. Technology focused solely on the management track – planning and analysis – or on the automation track – no ability to handle manual tasks or deviate from the plan – will not enable you to reach that goal.

To start accelerating and streamlining your release process, ensure that all steps – manual and automated – are agreed upon by the entire team, and that everyone works from a single, updated release plan.

Next, ensure the technology you choose supports this collaborative planning and visibility and, critically, can unify the management and technical tracks of your release by supporting both assignment and tracking of manual tasks as well as automated execution.

Use the analysis capabilities of your technology to identify unpredictable and slow elements of your release process and initiate improvement projects to accelerate and streamline these. Track the progress and effectiveness of these projects using metrics provided by the tool.

About the Author/Andrew Phillips

Andrew Phillips heads up product management at XebiaLabs. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.

November 4, 2014
CI and CD Across the Enterprise with Jenkins – CloudBees
We are happy to announce the next devopsy.kinsta.cloud Webinar, in partnership with CloudBees :

CI and CD Across the Enterprise with Jenkins

Delivering value to the business faster thanks to Continuous Delivery and DevOps is the new mantra of IT organizations. In this webinar, CloudBees will discuss how Jenkins, the most popular open source Continuous Integration tool, allows DevOps teams to implement Continuous Delivery.

You will learn how to:
- Orchestrate Continuous Delivery pipelines with the new workflow feature,
- Scale Jenkins horizontally in your organization using Jenkins Operations Center by CloudBees,
- Implement end to end traceability with Jenkins and Puppet and Chef.
Date: November 19th

Time: 1:00pm eastern, 10:00am pacific
```
REGISTRATION LINK: (Limited registration, so don't wait!) 
```
About the Speaker:

Cyrille Le Clerc is Director of Product Management at CloudBees with more than 15 years experience in Java. Prior to CloudBees, Cyrille was Chief Technology Officer at Xebia. As an early adopter of the “you build it you run it” model, he put it in place for a number of high volume web sites. He naturally embraced DevOps culture. He is also very active in the Java community, creator of the open source project “embedded jmxtrans” and a speaker at various conferences.
November 3, 2014
DevOps with Purpose: It’s about your tools!
This is Part Three in a four-part series. In Part One, I focused on the critical first step of defining DevOps with a purpose by thinking about DevOps in the context of your organization’s applications. In Part Two, I provided four tips to fostering a DevOps culture in your organization.

By now you’ve hopefully noticed the emphasis on “your” in this series, because, at the end of the day adopting DevOps is about your business, your applications, and your culture. In this third part of the series, I’m going to discuss your tools.

In IT, we’re indundated with tools. Developers have their favorite tools and sys admins do too, so do the project office, the service support group, and the QA team. There are IT tools purchased by our organization years ago that we are using and others we aren’t, tools we’ve recently started using, and others we are considering at any given point in time. There are also general-purpose tools the company wants us to use for things like document sharing, instant messaging, and so on. It’s got to the point where a lot of IT organizations say, “we have two of everything like Noah’s ark”, yet they still want more tools.

Part of the reason is that, as IT practitioners, we like gadgets and tools. We like the fact that they can help us do things and can amplify our own abilities, but we also know they can fragment information and can blindside us when someone uses a tool to do something that others weren’t aware of.

Do we really need more tools to do DevOps?

DevOps has a close association with the Agile Software Development movement, and one of the core tenets of the Agile Manifesto is to value “Individuals and Interactions over processes and tools”. Nevertheless, it’s hard to find a DevOps talk or article that doesn’t discuss the need for tools like git, docker, jenkins, puppet, chef, and so on. The reason for this is actually pretty straight-forward: continuous integration and continuous delivery are best done through automation so we can follow a repeatable method and roll things back quickly if there are issues.

At a more basic level, however, we need to acknowledge that there are really two types of tools:
- Tools that help us amplify our abilities as individuals (e.g. I can drive a nail into a board with a hammer much more effectively than with my hand or a rock because of the strength of the hammer’s material and the principle of leverage)
- Tools that help us coordinate work across many humans, thereby, amplifying our individual abilities (e.g. I can build a house much faster and with better quality by leveraging different experts’ skills and using a shared set of blueprints for building construction, plumbing, electrical, heating/cooling, and so on)
The same distinction holds true for software development, deployment, and operations. One individual can theoretically gather requirements, build the software, test the software, deploy the software, and support the software while simultaneously project manage their personal time. Most computer science students have done this in fact at one time or another, but when we are talking about enterprise applications that support core business functions it clearly doesn’t make sense because of the amount of effort required at each step and the specialized skills necessary for a high level of competency in each area.

What this means is that we absolutely need tools that amplify our abilities, or that of our individual teams, i.e. the hammers; but we also need to invest in tools that help us coordinate work across teams, i.e the shared sets of blueprints.

Tools That Amplify Individual / Team Abilities

A lot of the tools attention in the DevOps community has been focused on: source code management systems like git or bitbucket; requirements planning tools like jira or rally; build tools like Jenkins; automation tools like puppet, chef, and ansible; and project management tools like MS project or AtTask. These tools help us amplify work in each of these respective areas just like hammers, saws, drills, and screwdrivers perform specific functions when building a house and often follow a natural sequence (first you hammer boards in place to create a wall, then hammer on the sheet rock, then saw or drill holes in the sheetrock, screw in electrical outlets and fixtures, and so on.)

Just like building a house has a natural sequence, so does the sequence of tools from code to deployment and it’s often referred to as the DevOps “tool chain”. This brief article isn’t intended to cover each area of the typical DevOps tool chain (I could have also added bug tracking, automated testing, and other categories too), the point is you need to take time to define what the DevOps tool chain should be for your organization and to do so intentionally with a purpose, taking into account the tool investments your organization already has, the needs and skillsets of your organization, and your own practical budget realities.

Do you have experience in using and self-supporting open source tools or do you have a preference for commercially provided tools?, Are you comfortable having multiple requirements tools in different teams or business units, or do you want a single standard? Only you can answer these questions for your organization. There is no prescriptive formula for these types of DevOps tools, although I’ve mentioned a number of commonly used ones.

Tools that Amplify Work Across Teams

The second area you should focus on are tools to help coordinate work and amplify our abilities across teams, i.e. bridging the inherent gaps in the DevOps tool chain, a subject that has received less attention in the DevOps community to-date. This makes sense because it’s human nature to think first about our own function and team. However, this silo thinking is one of the core problems DevOps was developed to address; the long-time focus on tools for specific IT “work stations” has actually entrenched the cultural silos of development, QA, operations, and project office in IT. For example, most IT organizations have two totally separate systems for tracking issues – in operations, it’s the service desk application and for code issues in development it’s the bug tracking application – and there is little or no relationship between them. In fact, you are lucky if an incident in the service desk can be tied to a bug tracking number.

If we are adopting DevOps in order to optimize the flow of new software releases to support business agility, then we need to look at how we will coordinate work and avoid information distortion and loss when different teams are using different, disconnected tools. In addition to your tool strategy for optimizing individual DevOps workstations, therefore, you also need to have a strategy for the tools that are going to help you effectively manage flow across those workstations.

Most DevOps transformation efforts start small with a core team of perhaps a dozen or so members, and their often located in one physical location. In this scenario, you might be able to make do with regular face-to-face meetings and general-purpose communication tools like email, instant messaging, and a SharePoint site or Wiki with MS Office documents. The scope of your cross-team collaboration tools aligns with the scope of your DevOps effort and you might be able to rely on people interactions to unify the tool chain.

But as you scale your DevOps efforts to larger, distributed teams across time zones and multiple business units and applications, your cross-team tools approach should scale as well. The point-to-point nature of instant messaging, ignored reply-all emails, and SharePoint ‘document dumps’ aren’t effective in coordinating the efforts of dozens or hundreds of developers, testers, admins, and project managers. Information gets lost, information gets overlooked, and the gaps in your tool chain expand. A feature misses a commit and doesn’t get into the build and isn’t deployed. An operational requirement is missed so the test environment isn’t configured properly and the release gets pushed. Three weeks are spent solving a performance issue through code when it could have been addressed faster via hardware. A legacy application dependency in production is overlooked and the new mobile app doesn’t work in production as it did in test.

There is an emerging class of purpose-built IT collaboration tools, such as ITinvolve, that enable the creation of cross-team workspaces where information is proactively shared as IT teams and tools get work done. This helps large, distributed cross-functional teams collaborate more effectively with each other in-context to raise questions, provide answers, and incorporate what’s happening up and down the DevOps tool chain in their individual work.

For example, the developer will have better visibility into when the next build is going to occur and can ensure the feature is committed in time or can reach out to the build engineer in the workspace to request a delay in order to ensure the feature makes it in. The operations engineer can have earlier and better visibility into operational requirements so he can ensure the test environment is configured properly and avoid unnecessary delays. The legacy application dependency in production can be reproduced in test to ensure the application works properly once moved to production. And so on. By eliminating the handoff gaps and information loss across the DevOps tool chain, you can reduce risk of communication issues, solve problems more holistically rather than individually, and better achieve the goal of improving business agility.

In the final part of this series, I will bring together the themes of the first three articles to demonstrate how you can chart a DevOps plan based on your Applications, your Culture, and your Tools to A.C.T. with greater agility.

About the Author

Matthew Selheimer is Chief Technical Evangelist and SVP of Marketing at Houston-based ITinvolve. Reach him at matthew@itinvolve.com
October 28, 2014
Eight critical DevOps practices: innovate, deliver, repeat
DevOps has been garnering a lot of interest for some time. Discussions and debates on forums like devopsy.kinsta.cloud are improving our collective understanding of the right approach and now we are seeing concrete proof points of fast-movers applying DevOps practices to disrupt entire industries with rapid-fire, digital-innovation based on Cloud, Big Data and Mobile technologies. I have personally been part of this transformation at IBM for several years and would like to reflect on what we have learned, and also share with you results of a recent study from Forrester Research.

DevOps helps organizations address a critical business challenge – that of continuously capturing and responding to feedback (both internal and external) so we can rapidly and repeatedly turn innovative ideas into highly relevant, and desirable, products and services. This involves not only development and operations, but all aspects of the organization, as it entails the collaboration across the entire organization including business and supply chain. Compared to single-direction, traditional models, DevOps is bi-directional and is based on not just pushing something out to the customer but also on getting their feedback to make improvements.

In a recent webcast, I discussed eight DevOps practices that speed up software delivery with Kurt Bittner, Principal Analyst at Forrester and a leading expert on agile and iterative software development. This webcast highlighted a recent Forrester survey on DevOps practices, cycle times and where enterprises are in terms of the maturity of DevOps practices adoption.
You can watch the webcast replay here to know more – ibm.biz/Forrester-talk

Their survey of 600 IT professionals with application development responsibilities found that about a third of teams consistently deliver at cadences of one to three weeks and that the fastest teams generated higher business satisfaction than slower teams, an indication they don’t sacrifice quality to deliver fast. The study reveals how these teams did it: by adopting the following eight DevOps practices:
1. Minimum Viable Product – Deliver the smallest viable product or application and then improve it in subsequent rapid releases, based on feedback, and delivering value incrementally.
2. Dedicated Teams – Form dedicated, cross-functional teams, avoid over-specialization and ensure they are not interrupted by side projects.
3. Loosely Coupled Architecture – Use loose architectural coupling within and between applications to reduce complexity and enable delivery in small increments.
4. Minimize Hand-offs, Maximize Flow – Eliminate unnecessary steps, delays and friction between steps to increase the flow of work.
5. Deliver in Small Batches – Deliver in smaller batches helps to expose the most uncertain issues first and enables feedback on the most valuable usage models earlier.
6. Transparency – Transparency into progress helps everybody to see where you are at any point in time and also visibility into risk without decreasing productivity.
7. Eliminate Overhead –Eliminate the overhead of periodic and manual status reports and status meetings which absorbs valuable time and distracts focus from delivering value.
8. Automate Testing using APIs – Shift away from heavy manual testing to improve delivery speed, quality, and testing accuracy, and dramatically reduce cost.
The study refutes the myth that releasing fast requires you to sacrifice quality and accept higher risk. Adopting these eight DevOps practices, especially automated testing and delivering in small batches, does exactly the opposite: small incremental releases are more thoroughly tested, more stable and lower risk.

The best part about DevOps adoption is the flexibility to choose which practices to adopt first, based on each organization’s priorities, and then to incrementally adopt the rest of the practices as needed to optimize the software delivery lifecycle and accelerate innovation.

To learn more about the survey and the eight DevOps practices, download the Forrester study along with the approach that you can follow to adopt these as per your organization’s current state of delivery cycles.

About the Author

Ashok Reddy is responsible for Product Management & Strategy, product Design/user experience and Business Development for Rational Software Division of IBM Software Group and for the IBM DevOps portfolio. He is based in Raleigh, NC, USA. Ashok’s role involves all aspects of global product delivery including setting product strategy and for product line management, establishing product roadmaps and partnerships for IBM DevOps portfolio.
October 22, 2014
Task Organization for Enterprise DevOps

In 2007 I served as a captain in the U.S. Army during a 15-month tour in support of the “Surge” in Iraq. I deployed with the 4^th Stryker Brigade Combat Team to Baqubah, where our unit’s mission was, literally, to neutralize Al Qaeda in Iraq and unseat them from their position of consolidated power, in order to deny them the ability to influence events in the capital of Baghdad. You wear many hats during a combat deployment, but my primary roles were that of lead planner and battalion Fire Support Officer (FSO).

As a planner I formulated the maneuver strategy and led the battle staff’s planning efforts in over a dozen battalion-sized operations, and over 30 company operations. Once the planning was complete, I would switch hats and go out on mission as FSO, responsible for integrating all manner of indirect fires to support the ground commander’s fight. This came mostly in the form of calling in airstrikes on targets ranging from suspected deep-buried improvised explosive devices (IEDs) to high-level meetings of Al Qaeda leadership.

I have since transitioned into the private sector and the world of software, and am constantly amazed by how many shared concepts exist between the military and business, particularly in the areas of strategy, project management, and operations. Having said that, there is one concept I found to be extremely powerful in planning military operations which seems to have no civilian counterpart, and may be useful in the discussion on implementing DevOps in the enterprise. It is the concept of “Task Organization.”

Army doctrine defines task organization as “the process of allocating available assets to subordinate commanders and establishing their command and support relationships.”

I know, I know. Right now you’re sitting in rapt jubilation over reading that statement, thinking to yourself, “What the heck is this goober talking about, and what does any of it have to do with software?!?” Stick with me for a minute – I’ll get you there.

Military operations are mission-oriented, and by their very nature extremely dynamic and fluid situations. Task organization is a construct to assist commanders in choosing the right “tools” to accomplish the mission. Back in the homeland, units are homogeneous to facilitate training and administration. But when deployed, units become heterogeneous as commanders mix and match capabilities, forcing collaboration among disparate skill sets in order to achieve a common goal – in military parlance, this is known as the “combined arms fight.” This is how task organization relates to DevOps.

To illustrate, consider the following real-world example. In 2007 my unit’s area of operations (AO) was nestled right in the Fertile Crescent, dominated by irrigated farmland with villages scattered throughout, and a handful of urban centers. Many of those villages had never seen Coalition Forces before, but it was my commander’s intent to visit each, link up with the leadership, assess the security situation, and provide any help that we could. The gambit was intended to co-opt and partner with the local populace to root out terrorist cells.

However, at the time Al Qaeda was very adept at placing IEDs on highly trafficked routes to disrupt our freedom of maneuver, making it extremely risky for us to conduct vehicle movement. Like Waterfall, it was difficult for us to go anywhere due to the extra time, effort, and resources required to travel safely. We simply would not have been able to reach every village in our AO using this method.

To combat this deadly tactic, we first took a look at our task organization and asked ourselves, “What combination of capabilities will give us the most advantage and the least risk in achieving our stated goals?” Actually, we used a lot more swear words when speaking to each other, but I’m paraphrasing to keep this thing PG. After our analysis, we decided the best course of action was to take to the skies, by requesting and task organizing a unit of Blackhawks under our headquarters. We reverted to old-school air assault tactics, delivering units by helicopter – and completely circumvented the deadly IEDs Al Qaeda had laid out for us. Goal achieved – this method worked wonderfully, and the locals were so appreciative of receiving some support that they partnered with us in the fight against Al Qaeda.

Incidentally, it was during this time that I invented a new method for bomb disposal. Our dismounted patrols would travel with an Explosive Ordnance Disposal team (again, task org), but the EOD guys could only hump so much C-4 in their rucks, and there were more IEDs scattered throughout than they could dispose of. Having acquired a lot of experience dropping JDAMs, I knew how deadly accurate they were, and suggested that when they found a bomb buried in the road, I could detonate it by dropping another bomb on top of it. This method worked so well that I started to joke that I’d figure out how to open a can of soda with a JDAM by the end of the tour – again using a lot of swear words. But I digress…

That’s what DevOps is about – not cussing, but collaboration. By discussing our problems with units who had completely different capabilities, we were able to find new and innovative solutions to our problems together. We were able to turn a perceived weakness into a strength, and gained additional speed and agility by using air assault tactics.

Imagine for a minute that the Blackhawk unit in this example is like IT operations – they were responsible for providing the infrastructure and capability to deliver our “application” to the correct “market segment” in a timely manner, so that the “business” could achieve its desired objectives. The dismounted patrol is like the development organization, translating “business requirements” into tactics on the ground. By collaborating together, we were able to refine our process over multiple iterations so that we could Continuously Deliver combat power anywhere on the battlefield, at any time and place of our choosing. This also enabled a sustainable advantage over our “competitors,” allowing us to beat them to market time and again.

Gene Kim offers two DevOps patterns which dovetail nicely with the concept of task organization – embedding development in operations, and embedding operations in development. By task organizing an individual or team from operations into development, operations becomes aware of proposed changes before they come down the pipe, allowing them to better prepare to support those changes. By task organizing a section from development into operations, developers become more aware of the effects of their changes on downstream environments, and can adjust their code or provide more information to make the lives of operations easier.

For those exploring DevOps in the enterprise, I’d suggest an exercise in task organization before you begin implementation. What capabilities and skill sets do you have available to you? What can you request from “higher?” How can you mix and match these capabilities to achieve your stated objectives with the most expedience and the least amount of waste? After you’ve mixed capabilities, who will report to whom, and how will they be supported?

And don’t forget to use a lot of swear words.

About the Author

Rex Morrow is the Marketing Director at Datical, an enterprise software company whose solution, Datical DB, manages and simplifies database schema change management in support of complex, high velocity application releases across any environment, for any database. Prior to Datical, he co-founded Texas Venture Labs, a startup accelerator at the University of Texas, and received his MBA from the McCombs School of Business. Before graduate school, Rex served as a Company Commander in the U.S. Army in the rank of Captain, and was awarded two bronze stars during combat deployments in Iraq. Website: http://www.datical.com/ Twitter: @Datical Linkedin: https://www.linkedin.com/company/datical

October 10, 2014
ShellShock vulnerability: Impact, analysis and protection

This week Stephane Chazelas, an Akamai security researcher, discovered an interesting bug in the Unix Bash (Bourne Again Shell) shell – known as “Shellshock” or “Bash Bug”. This vulnerability affects a wide range of operating systems (Unix, Linux, OS X), routers, and other technologies, making many of the systems we use on a daily basis vulnerable to attacks. A system vulnerability in the GNU Bourne Again Shell (Bash), the command-line shell used in many Linux and Unix operating systems, could leave systems running those operating systems open to attack. The bug utilizes the Bash process variables that are passed by the operating system or an application. If Bash is configured as the default system shell, it can be used to attack devices via web requests, secure shell, telnet sessions, or other programs that use Bash to execute scripts. Patches for this vulnerability are now available for all the major Linux distribution vendors.

THE IMPACT
The reason this vulnerability raises many concerns is the breadth in threat vectors available to attackers. For example, the simple task of loading a website gives attackers the opportunity to exploit the vulnerability if the server handling your website request uses bash commands to access the information you requested. So if you still don’t think we use command line anymore, think again.

In fact, there are some computer programs that use Bash to execute commands on other computers. For example, how does a website know if you are accessing the site via mobile device or computer web browser? Based on the response your device returns ,the server will modify the content delivery to be compatible with the corresponding platform. From a technical perspective, your browser sends an agent to tell the web server that you are using either a mobile device or computer browser. The web server then uses that information to set the agent string as a variable before using Bash to execute code that generates the web page.

Attackers are making use of this vulnerability as we speak. Several researchers are reporting that honeypot infrastructures are being actively attacked by a group exploiting the Bash vulnerability. The vulnerability is allowing the installation of a zero day bash injected ELF malware that is currently being used. A system administrator using the @yinettesys Twitter account published a GitHub post reporting that threat actors exploited the Bash Bug flaw to launch kernel exploits on a computer system. The malicious code then performs a Web GET request from a user agent to begin a brute force dictionary attack to harvest credentials. The malicious code for this attack has been analyzed by virustotal and at this time there is 0/55 detection by antivirus.

PROTECTING YOURSELF
There are several detection methods that can be used to mitigate this risk. Since the Bash Bug affects versions 1.14 through 4.3 of the Bash shell, upgrading to the latest version will mitigate the risk.

From a web application standpoint, the bug can be exploited by code that passes through the Bash interpreter. CGI’s (Common Gateway Interface) and CGI scripts are the most affected, but anything that is passed to the Bash interpreter can be exploited. Command execution can be achieved through HTTP Headers and POST/GET parameters on vulnerable systems. So you could use a web application firewall to monitor the vulnerability in the header. Additionally, a signature will be added to the POST/GET field. This signature will monitor for attempts to bypass the detection signature via multiple whitespace using ( ){ command and ( ) { command and ( ) {.

Use an IDS/IPS to detect for any type of network communication; IDS/IPS will notify you when a connection is established and remote commands are executed.

We will continue to monitor the situation and post any new information we uncover to the Alert Logic blog.

SEE IF YOU ARE VULNERABLE

To see if you are vulnerable to this exploit run the commands below, the response will tell you if you are vulnerable

env x='() { :;}; echo vulnerable’ bash -c “echo this is a test”

If the system is vulnerable, the output will be:

vulnerable this is a test

An unaffected (or patched) system will output:

bash: warning: x: ignoring function definition attempt bash: error importing function definition for `x’ this is a test

CREDITS
https://lists.debian.org/debian-security-announce/2014/msg00220.html

http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html#.VCRpVitdXKN

http://arstechnica.com/security/2014/09/bug-in-bash-shell-creates-big-security-hole-on-anything-with-nix-in-it/

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-6271

——————————————————

Here are signatures for the IDS for detection that have been provided by Alert Logic.

These rules have been created with resource consumption (No PCRE) of everyday average hardware to have more comprehensive coverage of use.

We have to use “any” here because lots of services open web servers on non-standard http ports such as cups and devices any, because local attacks would be common ground for this (cups, avahi, etc.) and http_header likely only uses a whitelist we will miss unique header values. This can also be bypassed due to the “within” modifer, but we got to try to narrow this down for resource issues.

External net would be the outside Internet IP address. Home Net would be any communication from internal IP space.

alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: HTTP Server
Headers Bash Environment Escape RCE”; flow:to_server,established;
content:”() {“; http_header; fast_pattern; content:”}|3b|”; within:20;
http_header; tag:session,5,packets; classtype:attempted-admin;
sid:1006236; rev:1; )

Uri and method are theoretical for now, but check just in case another server does this; any input poorly parsed to an internal scripting function like Popen could trigger:

alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: HTTP Server
URI Bash Environment Escape RCE”; flow:to_server,established;
uricontent:”() {“; fast_pattern; uricontent:”}|3b|”; within:10;
classtype:attempted-admin; tag:session,5,packets; sid:1006237; rev:1;
)
() { should never be in a method, so we don’t need additional checks here:

alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: HTTP Server
Method Bash Environment Escape RCE”; flow:to_server,established;
content:”() {“; http_method; classtype:attempted-admin;
tag:session,5,packets; sid:1006238; rev:1; )

POST values might get parsed by the end application into environment variables (parsers, interpreters, formats, etc.) making one just to be safe, this might false, though

alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: HTTP Server
POST Bash Environment Escape RCE”; flow:to_server,established;
content:”() {“; http_client_body; fast_pattern; content:”}|3b|”;
within:20; http_client_body; tag:session,5,packets;
classtype:attempted-admin; sid:1006239; rev:1; )

Now the big bad one. An uploaded file could have metadata parsed and stuffed into an env variable (cups is notorious for this.) The optimization for this is going to be hard, but we have to do it so we don’t parse the max file parse byte:

alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: File
Transfer Parsing Bash Environment Escape RCE”;
flow:to_server,established; file_data; content:”() {“; within:100;
fast_pattern; content:”}|3b|”; within:20; tag:session,5,packets;
classtype:attempted-admin; sid:1006240; rev:1; )

Alright, now cookie values for base64 && unserialize/json_decode values that could be used. This could likely false due to these characters used often in these compressed data formats. This attempt is totally random because we don’t know the size and start of random people’s base64 encoded values + so rule might not even work, but we need to try disabled unless we need it due to the unknown and random chance of it working customer specific values where the cookie is known can easily be replaced in this rule:

#alert tcp $EXTERNAL_NET [1024:] -> $HOME_NET any (msg:”AL: HTTP
Cookie value Bash Environment Escape RCE”; flow:to_server,established;
content:”=”; http_cookie; base64_decode:relative; base64_data;
content:”() {“; tag:session,5,packets; classtype:attempted-admin;
sid:1006241; rev:1; )

DHCP sets user hooks

alert udp $HOME_NET [67,68] -> $HOME_NET any (msg:”AL: DHCP Bash
Environment Escape RCE”; flow:from_server,established; content:”() {“;
fast_pattern; content:”}|3b|”; within:20; tag:session,3,packets;
classtype:attempted-admin; sid:1006242; rev:1; )

This article was also posted to the Alert Logic blog

About the Author

Stephen Coty joined Alert Logic in 2012 as the director of threat research and author of the Cloud Security Report, and has a wealth of experience in the security space. In his role at Alert Logic, Stephen is currently in the process of building a premier threat research team that is innovating the industry by taking a more proactive versus reactive approach to security through solid research, reverse engineering of malware, and constant searches for stolen data, all of which can be leveraged as actionable data for remediation. Stephen has over 15 years of experience in systems engineering and security consulting, in industries ranging from healthcare to finance and government. Most recently, he held a senior position at Rackspace Hosting as the Manager of Cyber Security, where he served as the architect of the Security Operations Center and Threat Research Team. He is also a member of ISSA, Infragard and the HTCIA.

September 27, 2014