Category: Blogs

  • Webinar:  How Nationwide Insurance is Adopting DevOps for Variable Speed IT Systems

    Webinar: How Nationwide Insurance is Adopting DevOps for Variable Speed IT Systems

    To drive towards faster innovation, and also ensure the stability of the core business, the need arises for what is being termed as “two-speed or variable speed IT.” While innovative Systems of Engagement demand speed, agility and experimentation, legacy Systems of Record need governance and predictability. Two-speed IT enables adoption of DevOps capabilities by adapting them to the “speed” of delivery. It requires two-speed business planning and processes and even two-speed organizational structure and budgeting. Join Carmen DeArdo, Director of Application Development at Nationwide Insurance, Sanjeev Sharma, lead DevOps Technical Evangelist at IBM, and Lee Reid, executive IT specialist as they show how IBM’s solution is helping Nationwide adopt DevOps practices for variable speed IT systems.

    This webinar is the first in the Variable Speed DevOps – A Deep Dive series.

    Event recorded Tuesday May 5th, 2015

    SLIDES

    [seoslides embed_id=”be17733a1254″ script_src=”https://devopsy.kinsta.cloud/embed-script/webinar-how-nationwide-insurance-is-adopting-devops-for-variable-speed-it-systems/6643/” overview_src=”https://devopsy.kinsta.cloud/slides/webinar-how-nationwide-insurance-is-adopting-devops-for-variable-speed-it-systems/” site_src=”https://devopsy.kinsta.cloud” site_title=”devopsy.kinsta.cloud” title=”Webinar: How Nationwide Insurance is Adopting DevOps for Variable-Speed IT Systems” /]

    VIDEO

     

    Your Host:

    alan_new_head_shot Alan Shimel, Editor-in-Chief devopsy.kinsta.cloud, An often-cited personality in the security and technology community and a sought-after speaker at industry and government events, Alan has helped build several successful technology companies by combining a strong business background with a deep knowledge of technology.


     

    About Panelists:

     

     Carmen-Deangelo-300x225Carmen DeArdo, Nationwide – Director – Tools and Technology Leader

    Carmen DeArdo, Director – Tools and Technology Leader, Nationwide Insurance, An ardent advocate of DevOps and a leading speaker in the DevOps and technology related events, Carmen is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across Mobile, Distributed and Mainframe and other technologies.  

     

    Sanjeev Sharsanjeev-sharmama, IBM Worldwide Lead – DevOps Technical
    Sales and Adoption, IBM Cloud Unit

    Sanjeev Sharma, IBM Worldwide Lead – DevOps Technical Sales and Adoption, IBM Cloud Unit, is a 20-year veteran of the software industry with expertise in DevOps, Mobile Development and UX, Lean and Agile Transformation, Application Lifecycle Management and Software Supply Chains. He is a DevOps Thought Leader at IBM and speaks regularly at conferences. He has written several papers and is the author of the DevOps for Dummies book.

     

    leeLee Reid, IBM – Senior IT Specialist

    Lee Reid, Senior IT Specialist, IBM, is a senior solution architect with focus on practical adoption of software engineering tools, software delivery solutions, application life cycle management, and DevOps. He has over 25 years of experience in software engineering, design, programming, testing, development, tools & methodology, team leadership, and strategic planning.

  • Reducing Incident Resolution Time

    Reducing Incident Resolution Time

    From raw incident count to response time, there are several key metrics that top Operations teams track to measure and improve their performance. One of the most popular metrics teams track for is mean time to resolution (MTTR). It’s the time between failure and recovery from failure, and is directly linked to your uptime. While MTTR may be the gold standard when it comes to operational readiness, it’s important for teams to look at the bigger picture to effectively decrease incident resolution time.

    Putting MTTR into perspective
    Your overall downtime is a function of the number of outages as well as the length of each. Dan Slimmon does a great job discussing these two factors and how you may want to think about prioritizing them. Depending on your situation, it may be more important to minimize noisy alerts that resolve quickly (meaning your MTTR may actually increase when you do this). But if you’ve identified MTTR as an area for improvement, here are some strategies that may help.

    Working faster won’t solve the problem
    It would be nice if we could fix outages faster simply by working faster, but we all know this isn’t true. To make sustainable, measurable improvements to your MTTR, you need to do a deep investigation into what happens during an outage. True – there will always be variability in your resolution time due to the complexity of incidents. But taking a look at your processes is a good place to start – often the key to shaving minutes lies in how your people and systems work together.

    Check out your RESPONSE time
    Some call it MTTA (mean time to acknowledge) or MTTR (same acronym, but meaning mean time to respond), but the clock starts ticking as soon as an incident is triggered, and with adjustments to your notification processes, you may be able to achieve some quick wins.

    If your response time is on the longer side, you may want to look at how the team is being alerted. Do alerts reliably reach the right person? If the first person notified does not respond, can the alerts automatically be escalated, and how much time do you really need to wait before moving on? Setting the right expectations and goals around response time can help ensure that all team members are responding to their alerts as quickly as possible.

    Establish a process for outages
    An outage is a stressful time, and it’s not when you want to be figuring out how you respond to incidents. Establish a process (even if it’s not perfect at first) so everyone knows what to do. When you’re designing your process for responding to incidents, make sure you have the following elements in place:

    1. Establish a communication protocol – if the incident is something more than one person needs to work on, make sure everyone understands where they need to be. Conference calls or Google Hangouts are a good idea, or a room in HipChat or Slack.
    2. Establish a leader – this is the person who will be directing the work of the team in resolving the outage. They will be taking notes and giving orders. If the rest of the team disagrees, the leader can be voted out, but another leader should be established immediately.
    3. Take great notes – about everything that’s happening during the outage. These notes will be a helpful reference when you look back during the post mortem. At PagerDuty, some of our call leaders like using a paper notebook beside their laptop as a visual reminder that they should be recording everything.
    4. Practice makes perfect – if you’re not having frequent outages practice your incident response plan monthly to make sure the team is well-versed. Also, don’t forget to train new-hires on the process.

    To learn more, check out this talk about incident management from Blake Gentry, a former lead software engineer at Heroku.

    Find and fix the problem
    Finding out what is  actually going wrong is often the lion’s share of your resolution time. It’s critical to have instrumentation and analytics for each of your services, and make sure that information helps you identify what’s going wrong. For problems that are somewhat common and well understood, you may be able to implement automated fixes.

  • Continuous Documentation

    Continuous Documentation

    Is documentation dead in DevOps? Yes and no. It is transformed, it is automatic, and it is actionable.

    (more…)

  • Making DevOps Work with External Contractors

    Making DevOps Work with External Contractors

    On a recent ‘virtual panel’ session, many questions came in real-time from almost a thousand attendees around the globe. I’ll try to cover the most interesting questions in this column over coming weeks, but one that came through many times is very typical of the discussions I have with my customers and others, especially those from larger enterprises:

    Q. How can we implement a DevOps approach with external contractors, consultants, vendors and service providers?

    Many of my customers regularly use external programming shops and third-party consultants, from across the street and across the globe. A majority also use third-party service providers, especially cloud and hosting services, to help delivery the on-demand infrastructure that fuels DevOps.

    The good news is that you can definitely incorporate these third-party vendors into your DevOps approach, even in a large enterprise, starting with a few simple steps.

    Establish Open Communication

    Establishing open communication lines is critical to DevOps, and no less so with external contractors and service providers. This open dialog should start at the planning stages, working together to establish common goals and expectations, including timelines, work methods, pace of delivery, and generally ‘how things work.’

    I recommend connecting face-to-face if at all possible, even if you can only do it once or twice, whether you meet at their place or they come to you. There is something special about meeting in person that brings to life many important DevOps concepts – like empathy, sharing, communication and cultural understanding.

    For large and globally diverse teams, in-person meetings can be cost- and time-prohibitive, but you can also achieve similar outcomes through virtual conferencing. Here I recommend immersive ‘tele-presence’ style systems, rather than the more common ‘cheap-and-cheerful’ web conferencing tools. Grainy headshots taken with low-res webcams, displayed seven-to-a-row in tiny windows, simply will not create the deep personal connections the same way as high fidelity communications.

    Maintain Ongoing Collaboration

    However, inexpensive web-based technologies are ideal for maintaining ongoing collaboration, once you have established those initial lines of communication. Use web–based video-conferencing to bring external consultants into your teams’ regular communications, from planning sessions and standup meetings to post-mortems and ad hoc chats.

    Also consider integrating external teams into your internal messaging systems, providing them with corporate e-mail addresses, including them in mailing groups and directories, connecting them to your chat and IM applications, and providing access to collaboration tools like shared workspaces and shared storage. None of this requires them to be on-site, or even in-country, but will help immensely to make them part of a collaborative, integrated team.

    Share Access to Internal Technologies

    Speaking of technologies, I have also seen multi-national and other cross-silo teams using common tools to share SDLC activities, status reports, system views, operational insights, and even vocabularies. Sharing access to common tools between contractors and employees alike – for example to share goals and assignments during planning, manage and report on project or backlog status, check code in and out, perform QA and integration tests, integrate new code into existing repos, release code updates into QA or prod, monitor and triage operational performance, or provide feedback from one system or team to another – definitely helps to connect and integrate external teams with internal teams.

    Share Processes with Automation Tools

    Automation tools in particular will help to integrate third-party contractors into established DevOps practices and processes. Shared automation allows you to make your ‘known-good’ processes a repeatable and reliable shared resource to all your teams, internal and external. Providing shared access to automated processes for check-out/check-in, code integration, test data management, provisioning and configuration, code release, and more helps external teams to adopt the same approaches, and implement change in the same ways, as internal teams – while also providing essential audit and control.

    Use Virtualization to Span the Globe

    Virtualization technologies especially will help to abstract internal environments for external workers – across the street or across the globe. Server and OS virtualization (and related approaches like containerization and cloud computing) allow easy exchange of common platform and build environments; network virtualization allows remote teams to code for abstracted internal networks; service virtualization allows multiple teams to develop and test in parallel even without shared access to internal systems; virtual storage allows teams to easily swap in or out common shared datasets; exposing functions or microservices as virtual APIs allows remote specialists to work independently yet still leverage others’ work; and of course virtual private networks allow secure access to internal systems for remote users anywhere.

    Find Your Own Approaches

    Fundamentally, there is no reason DevOps will not work with external contractors, consultants, vendors and service providers. Communication, collaboration and integration can certainly face new barriers at the enterprise’s edge, but these are not insurmountable. Ultimately you must find approaches that work for your enterprise – and I would love to hear about them in the comments below or on Twitter – but if you establish mutual goals, ongoing communications, common processes, and shared tools, you will be well on the way.

  • 8 Signs of Docker Ecosystem Empire-Building

    8 Signs of Docker Ecosystem Empire-Building

    Docker’s meteoric rise on the strength of the continuous delivery movement’s thirst for containerization has been nothing short of remarkable. There’s perhaps no bigger sign of Docker’s uptake than in the build-up of its third-party ecosystem. The past year has seen partnerships galore from Docker and the biggest IT industry players,  plus notable support of the Docker ecosystem amongst key service providers. Here’s our take on some of the cross-industry relationships that will make the Docker ecosystem stronger.

    1. Google

    Google has had a big hand in helping to foster a collaborative Docker community, notably with the open source release of its Kubernetes container cluster manager. The project has gained tons of cross-industry support from IBM, Microsoft, Red Hat and more.  Google is further strengthening its ties in collaboration with Mirantis Partners with the reveal last month that Goog would work on a project with the OpenStack firm to  simplify the installation and management of Kubernetes applications on OpenStack.

    2. VMware

    Late last summer, VMware announced that it was working alongside Google, Pivotal and Docker to make it easier for enterprises to scale container-based applications on VMware infrastructure. As part of the deal, VMware agreed to collaborate on open source projects like libswarm, libcontainer and libchan, and to work with Docker to improve interoperability between Docker Hub, vCloud Air, vCenter Server and vCloud Automation Center.

    3. RedHat

    Just this month, Red Hat brought to fruition its plans to support Docker and other containerization technologies with the launch of its Red Hat Enterprise  Linux Atomic  Host operating system. Based on work done in the Project Atomic open source project, the OS is meant to offer a reliable platform for Docker, Kubernetes, Flannel and others for the enterprise market.

    4. Amazon

    For the past year Amazon has been bending over backwards to help customers more effectively leverage Docker on AWS infrastructure. Most notable was its big splash last November with the launch of its EC2 Container service (ECS), designed to simplify the management of Docker containers across a managed cluster of EC2 instances. It continues to add to that service—last month it provided updated enhancements to ECS that makes it able to use images stored in private docker repositories and to mount volumes in containers in ECS. 

    5. Rackspace

    Rackspace’s long-running relationship with Docker starts first as a user, but extended notably beyond that in June last year when it was working to contribute to the Libswarm project and had created a backend plug-in for Libswarm that would enable its users to launch Rackspace Cloud Servers directly from the Docker CLI. 

    6. IBM

    After a year full of big wins for the Docker ecosystem in 2014, it ended with a bang in November when IBM joined forces with the containerization upstart to integrate it into a range of applications for the enterprise market. As a part of the partnership, IBM launched IBM Containers, a Docker-based service delivered as a part of its Bluemix open cloud platform for application development. 

    7. Microsoft

    Microsoft has been incrementally upping the ante with its Docker relationship, starting in June 2014 when it added support for Docker  Microsoft added support for Docker in Microsoft Azure on Linux VMs. By October it had publicly committed to support Docker on future Windows Server releases, by adding resources to implement in-house tweaks to Windows Server and to support the open-source development of the Docker Engine for Windows Server. And in January Microsoft showed its commitment to the partnership with the release an Ubuntu image fully integrated with Docker and ready to deploy from the Microsoft Azure marketplace. 

    8. Puppet Labs               

    The most recent among Docker’s ecosystem wins, Puppet Labs this week pushed forward a new version of Puppet Enterprise  that includes a special module specifically for managing the Docker Platform and launching containers in Puppet-managed infrastructure.  The idea is to help users side-step configuration issues with the Docker daemon running containers.

  • A Puzzle – Gradle or Maven

    A Puzzle – Gradle or Maven

    Build Tool

    Gradle vs Maven (or Both)

    The world of build tools evolved from simple compilation (gcc) to make files, then evolution of ant(+ivy) and Maven, now the latest build tool is Gradle.

    It’s always a puzzle which build tool to choose – Maven, Gradle or both. Sometime it’s first time project setup or it could be migration from existing tool, but it’s always a debate to choose one or the other.

    There have been lot of talks and discussions on Gradle, Maven (and/or Gradle vs maven). There is buzz around that Android/Google adopted Gradle as its standard build tool.

    Here, I would like to explore the possibilities of adopting Gradle or Maven (Or both), Intentionally i am keeping ant+ivy out of this talk.

    Lets take a look at the details on why Android may have adopted Gradle

    Key points for Google to go ahead with Gradle for their android development are:

    • Create multiple APKs for your app with different features using the same project.(create several variants of an application)
    • Reuse code and resources.
    • Customize, configure, and extend the build process.
    • Customized integration with their own IDE Android Studio.
    • Single build tools to support multiple languages.

    They had some special requirements for their diverse eco system, they worked with gradleware development team to include customized feature for the android application. They even have customized IDE for android developers build around Gradle.

    What we may be looking from build tools

    • How easy is the learning curve?
    • How fast are different builds with each tool?
    • How complex is it to create and maintain the build script?
    • How many plugins exist and how simple is it to customize your own plugins?
    • How good is the community and documentation for each tool?
    • How well does each tool integrate with developer tools? (IDE, App Server, CI server).

    Lets take a look at Maven and Gradle build tools on basis of above points and give them score as per our use case.

    Here is the Tabular comparison for each point between Maven and Gradle.

    Points Maven Gradle Score
    How easy is the initial learning curve 1) Maven is XML based tool, XML is very commonly used/known.2) If existing project are using maven then developers are comfortable with the system. 1) Gradle is DSL based system and need to learn explicitly.2) Developers need to learn new system from beginning if they are not familier Maven – 8/10Gradle – 5/10
    How fast are different builds with each tool Taken a report from zeroturnaround, they have done detail analysis of the speed of builds with both built tools and found maven and gradle are close enough on build timings. Maven – 8/10Gradle – 9/10
    How complex is it to create and maintain the build script? Maven build scripts are xml based, which has predefined structure and only one way to write.So it makes it more standard and less flexible. Gradle has its own DSL which is introduced by Gradleitself and tightly connected to Gradle internals But its flexible and simple and short Maven – 5/10Gradle – 7/10
    How many plugins exist and how simple is it to customize your own plugins? Maven is called “plugin execution framework” Hundreds of plugins exist for Maven and you can create your own plugin is simple. Gradle’s architecture is also plugin-based, It’s easy to write plugins But availability of plugin in community may not be ease. Maven – 8/10Gradle – 6/10
    How good is the community and documentation for each tool? Maven is in the market for very long time, Documentation is good, Lots of resources and help available in open community and forum. Gradle is very recent. It is open source but still under control of gradleware. They have option for commercial support. Maven – 9/10Gradle – 8/10
    How well does each tool integrate with developer tools? (IDE, App Server, CI server). With many years of background Maven has full support to almost each tool and every category (IDE, App Server, CI). Lacking in the App Server and CI ServerCategories, mainly due to newness. Maven – 9/10Gradle – 5/10

     

    (Note – I haven’t given any tool 10/10 because there is always room for improvement).

    My scoring is based on following fact.

    • One is already using maven for all their projects.

    Total Score

    Maven – 47/60

    Gradle – 40/60

    One can decide the score on the basis of their user case and requirement.

    Choices

    • Gradle

    You may not be using other features which are used by Google for android development as those features may or may not match with your needs, however Gradle is essentially emerging as a popular tool and you may definitely want to have a look at it.


    • Maven

    Maven is widely used and has robust defined workflow around it. Developers are very familiar with the system. However this may not be a viable option as android development needs to use Gradle.


    • Maven and Gradle (both)

    This would be the one option for a company with a wide range of development options. It may give developers flexibility to adopt tools as per their choice with a given workflow.

    Reference #1

    References #2

  • Test Results Analysis at the Speed of DevOps

    Test Results Analysis at the Speed of DevOps

    Fast continuous testing is counterproductive if the analysis of test results does not keep up with the speed of testing. In my prior blog Continuous Testing – Accelerated! I discussed the importance and practices for speeding up continuous testing. Unless test analysis is as fast as the tests, all that will happen is an accumulated debt of test results to be analyzed resulting in, at best, no net gain in time savings and, at worse, valuable results going overlooked and mass confusion that actually slows down the overall CI/CT cycles. Typically many development team members are looking for results, so the time savings for speeding up the availability and the total work associated with results analysis is huge for every minute of results analysis time saved on each CI/CT cycle. When aggregated over multiple code branches a few minutes saved per test result can be man-years of time saved over a year!

    So what can be done to ensure continuous test results analysis keeps pace with accelerated continuous testing? Below are some suggestions in a checklist format that have proven useful in high performance DevOps deployments.

    1. Determine clear responsibilities and SLAs for test results analysis:When the organization sets specific responsibilities and goals for results analysis the entire DevOps process will be accelerated.
      • Assign first responders and 2nd line escalation team members with definitive workflows.
      • Define a strategy for responding to test failures such as pre-determined actions if test failure rates exceed certain thresholds for each category of test. Examples of actions could include “continue with CI”, “revert changes”, or “stop and fix faults” depending on failure priorities.
      • Train the team on best practices test design and peer-review the scripts to ensure they define clear verdicts.
    2. Design test scripts for efficient test results analysis: When the test scripts are designed for efficient results analysis the process is accelerated. Here are some examples of good practices:
      • Each script must have a clear “Pass”, “Fail” or “Inconclusive” verdict, and a description of the verdict.
      • Each verdict should be reported together with pre-defined tags that can be used to categorize the verdicts.
      • Assign a priority code to each verdict. The highest priority verdicts should be analyzed first.
      • If a script has multiple verdicts, the highest priority verdicts should be reported first if possible.

    3. Choose test tools that are designed for fast test results analysis: Look for the following features when you are evaluating test tools. Not only will this save you a lot of headaches in the long term, it will give you the flexibility to design tests that integrate with other DevOps tools.
      • Restful APIs that do not require wait states between commands and responses.
      • Support a framework that enforces test results standards that ensure each test, when completed, reports a clear verdict including “Pass”, Fail”, or “Inconclusive” , a description of the verdict, event logs from both the test system and system-under-test POV, and tags that identify the build campaign, and functional attributes which can be used for cause analysis.
      • All timers can be sync’d to a common clock across the entire test environment and the verdicts record time at a precision that is appropriate for the fastest events relevant to the system under test.
      • Caching features that enable results data to be processed for one test while the tool proceeds to execute other tests.
      • Results data mapping and processing utilities integrated within the tool without requiring slow speed scripting to parse results.

    4. Choose and configure test analysis tools to process results as they are available:In a best practices large scale DevOps environment test cases are running in parallel across many machines simultaneously and there may be multiple code branches and versions being tested at the same time. According to best practices powerful test result analysis tools collect and aggregate test results data from all of the test machines, filter the results in real time for processing according to functional tags, analyze the aggregated data to determine probable causes and report the analyzed results together with probable cause information in priority order to the most relevant first responders, in real time “results-so-far” cumulative format.
      • Choose powerful servers for results aggregation and results processing, with ample processor speed, I/O, large and fast memory and data storage. Consider separate servers for aggregation and analysis to all result collection and processing threads to be run in parallel without interfering with each other.
      • Configure the system with fast high priority channels to collect all results data to a common results repository.
      • The results repository should be big enough to save all the results of many past test campaigns, such as a month of results data, because best practices test analysis tools will mine the past data to determine failure trends.
      • Results are collected across all test agents, processed and reported in real time as the verdicts are reported.
      • Results are reported in categories by build, tag and priority, and probable cause, giving highest priority in the reports to code areas or features that exceed pre-set failure thresholds
      • Probable causes are automatically determined from statistical trend analysis which weight the most recent build results heaviest and also use static code defect density information to point to code modules that most likely caused the failures.

    5. Configure the workflow for optimal test results analysis: When optimized, test analysis can prevent DevOps cycle interruptions significantly. The optimal workflow will use just-in-time concepts such as the following:
      • Only send results to the appropriate first responders when failure thresholds are exceeded.
      • Automatically escalate failures to the appropriate 2nd line escalation team members when the SLA time for first responders expires.
      • Analyze the highest priority verdicts first.
      • Prune or refactors tests that have a low failure history.
      • Define threshold levels that will cause an automatic revert of a change but make sure the team gets all the results for analysis and probable causes so they can fix the problem off-line before re-applying the corrected change.

    The above is a partial list of suggestions for continuous results analysis that have been proven to yield good results to complement accelerated continuous testing and DevOps.

    At Spirent we think testing has a bright future in DevOps. You can read more about our views at
    Spirent.com/solutions/devops

    What do you think of these suggestions and do you have others that should be mentioned?

  • Chef  best practices

    Chef best practices

    Chef Best practices 

    This article is the II part covering Chef. Please refer to Automation, Provisioning & Configuration Management (CHEF) for an introduction.

    As a matter of fact, Chef is truly an astounding tool. It keeps the cooking metaphor actively going.

    Following is the list of best practices that I would like to cover in the article:

    • Project creation and “readme driven development”
    • A sane build system with Rake
    • Continuous integration with Travis
    • TDD with Chefspec
    • Workflow with Spork
    • Verification with Minitest

    Getting Started:

    1). First and prime important step is to create a new Project. In this section everything should be source controlled. Here in the example a Github repo has been created for all the codes and for the verification or to see step to step process of the development as well as mistakes, they can be easily viewed from the commit history. Follow all the steps and do same for your project:

    git init
    

    Second objective is to perform a Source Code Management. By running the following command you can make use of git flow to manage your features by selecting all the default settings and if you aren’t using it, then its time you should start working on it.

    git flow init

    Using Readme Driven Development is top best practice that can be started by README.md file. This practice is really important as it helps you to focus your mind and cause you think like a user.

    Here Rake is being used for an already build system. Following is the list of tasks that are to be considered:

    • Rake check: This is responsible for checking all other dependencies.
    • Rake build: The main aim of rake build is to manage and run the test suites.
    • Rake start: It acts as an envelope for vagrant up.

    2). Another best practice is to Set up a Ruby version for the project. This is done by creating two version files: .rvmrc and .rbenv.

    After Ruby files have been created, next step is to create a Gemfile. The main aim of creating this file is to set up all the project dependencies.

    Below is the example with the commands that will help you in creating a Gemfile:

    source "http://rubygems.org"

    ruby ‘1.9.3’

    gem ‘rake’, ‘10.0.4’

    After creation, installation of these dependencies is the essential key step which is possible with below mentioned instruction:

    bundle install

    Creating a new Rakefile is also necessary. It contains a placeholder task per those that have already been defined in our Readme file.

    By making use of Rake for setting up a build system helps in automating some common tasks.

    3). Continuous Integration with Travis is another elite practice.

    But before we proceed further, it’s important to create a link between the repository and the Travis. It is an essential step in the cases where users have already created a repository for the code as part of the series.

    After link is made, you can configure Travis by creating a new Travis file i.e; .travis.yml in the root of the project directory.

    Configuring Travis helps in eliminating development gem dependencies so as to pace things up and tests on the CI server. Furthermore, it also tests the master as well as the development branches.

    Updating your Gemfile is also important in order to ensure that any future development related Gems are correctly excluded. Add this to your Gemfile:

    group :development do

    end

    After Travis is configured, you can update your README.md by adding the Travis notifications to indicate the status of the build.

    4). Another best practice is to perform TDD using Chefspec to “spec” out the cookbook.

    First step is to add chefspec to your Gemfile by updating the file and by all means also do bundle update.

    After updating is done you can Automate Chefspec Cookbook Creation. As per the Chefspec documentation, you are capable of creating a cookbook and on the other hand use another command to create the spec directory.

    According to user or your varying needs you may even wish to create a new_cookbook.

    For example: To build up a new rake task: new_cookbook below are the instructions to be followed

    desc "Creates a new cookbook."
     task :new_cookbook, :name do |t, args|
     sh "bundle exec knife cookbook create #{args.name}"
     sh "bundle exec knife cookbook create_specs #{args.name}"
     end

    As soon as you generate new cookbook you can also create your sample Cookbook

    rake new_cookbook motd

    At this point you have a new blank cookbook and can run the basic tests with rspec cookbooks. You can even add a rake task and upgrade the build tasks for the usage. At last you can add the spec for the motd cookbook in the motd/spec/default_spec.rb file by creating a Failing Spec. By executing the rake, build should be able to show “this now fails”. Thereafter you are required to implement the cookbook.

    Make your Spec Pass and the templates/default/motd.tail.erb file:

    Finally test driven cookbook is made and as Travis file is already referencing the build tasks, when we push this code up, CI will run this test and verify your work.

    5).Working with Spork is another practice in Chef. This tool helps in collaborative cookbook development. It also helps single developer to control and place things in order when it comes to bumping versions.

    So to test this out, initially we will install knife-spork and will do a small edit to your motd cookbook to use its features. First step is to always update your Gemfile by adding the spork gem to the Gemfile. This is not required by the CI server and is put under the development block:

    Following example exemplify the instruction:

    group :development do
     gem "knife-spork", "~> 1.0.17"
     end

    After updating, next step is to configure your Spork by creating a config/spork-config.yml file to put in the default configuration. Though there is a little issue of having lots of configurations in various places and is consolidated over a period of time.

    Configure your Spork using following instructions:

    mkdir config
     touch spork-config.yml

    Running the rake build task shows that our test has failed and can be set right just by updating the template. The moment you complete your updates and the test pass, we should bump the version number of the cookbook by bundle exec knife spork bump motd patch. And finally you will make use of the upload feature so as to upload it to the chef server “freezing” it at the same time.

    Another great feature of spork is the spork promote command that allows you to constrain a specific version of a cookbook to a specific environment. A use case could be:

    • Cookbook 2.0 is being verified on your development environment.
    • Cookbook 1.9 is in production environment.
    • When things looks good can be rolled out to production.

    6). Last but not the least is to set up the mini test suite to verify your cookbooks on a real server. This is done by placing your files in the right place and adding default to your existing cookbook:

    mkdir -p cookbooks/motd/files/default/tests/minitest
     touch cookbooks/motd/files/default/tests/minitest/default_test.rb

    There is also another option available with you in order to avoid the above mentioned action. You can simply modify the new_cookbook rake task to make this part of the cookbook buildout. Last stage comes with deploying the Cookbook and testing it. This helps you in building up a cookbooks that actually works when used on the servers. They also act as a safety net as they run whenever the server is provisioned.

    Conclusion:

    Above mentioned are the best practices of Chef that will help you in building cookbooks in a true, test-driven way. Implementing all these will help in saving you from all key headaches by keeping your custom cookbook changes in new cookbooks and without problems be able to create targeted pull requests to share back with the community.

    I expect you’ll be smart enough to take away some good practices here.

    Happy Chefing!

  • Business Value of Log Analysis

    Business Value of Log Analysis

    The modern software delivery pipeline breaks the mold of traditional process and tool acquisition, but doesn’t warrant any less consideration. Measuring the impact of log analysis for your ops and dev teams (and the organization overall) is more of an exercise in measuring process improvement—not hard ROI.

    The soft ROI components of log analysis are:

    1. Wasting Less Time: IT Operations wastes dozens of hours each week hunting down system events that relate to a problem. Log analysis not only eliminates the lost time, but also this reactive approach to addressing problems. Identifying system issues is an arduous task, especially when it’s not clear if the problem is on the application layer or the infrastructure. If the issue is a combination of the two, a delayed release is the likely outcome. With log analysis, you’ll discover issues either before or as they happen and avoid both lost time and unnecessary delays!
    2. Responding and Resolving Issues More Quickly: Developers working with IT provided analytics respond to and resolve bugs more quickly. This helps improve the stability of production and frees up developer time to focus on adding new functionalities that will ultimately increase the value of your application.
    3. Reducing Customer Churn: Modern customers demand stable applications with regular updates. Without this positive user experience, they’ll move on and find a different application to meet their needs. Frequent outages and poor product quality result in churn even in previously entrenched industries. For instance in financial services, a few bottom-up constructed DevOps teams are blowing traditional organizations out of the water. Some of the older institutions have stepped up their game but only by thinking ahead and using better metrics.
    4. Accelerating Releases: You cannot do agile if you are always playing catch up with production. Often organizations delay six or more planned releases a year due to unexpected issues. When a publicized release is delayed, customers are particularly unhappy and prone to jumping ship. In agile, with the expectation of regular bi-weekly or fast sprints, delays are a major let down. If your goal is continuous integration or delivery, delayed releases are not acceptable.
    5. Optimizing Production Infrastructure Costs: Growing infrastructure quickly is easy, but how fast is too fast? Because of its low visibility, bad infrastructure can control your organization without you even realizing. Using log analysis, you can attack server sprawl head-on, track workloads assigned to specific machines and identify underutilized or dead assets. You can also use that data to restructure your on premise or cloud infrastructure spend!
    6. Exploratory Infrastructure: Log analysis builds confidence that server logging is happening automatically, freeing IT resources and providing them the time to consider how and where your system can be improved. From identifying infrastructure configurations that will enhance the environment to auto-scaling initiatives or simply engaging in more productive activities than justifying outages, your IT department will have the time to focus on higher-level thinking and tasks.
    7. Improving Communication: With better reporting, comes better communication to both the team and executives. Presentations are built more quickly, analysis showing what went well and poorly is generated easily and action items for improvement are implemented in more detail and without any additional effort. Consistent presentations means standard dashboards and greater efficiency across the board for the team.

    Forget the ROI

    You may have noticed the absence of any ROI calculation in our benefits analysis. This is intentional, because in the end, the calculations don’t matter. The point of implementing log analysis is to replace your existing procurement model with a system that your team already knows is better. Unfortunately, those who continue to use old ROI calculations to evaluate these tools will miss these benefits.

    Cloud-based log analysis sometimes seems too easy, but it is the future of the modern software delivery pipeline. The processes and tools that drive log analysis are simplified and seamless, adding value to your culture, team and processes that cannot be measured in dollars.

  • Financial Sector Leading the Way with CD for the database

    Financial Sector Leading the Way with CD for the database

    The financial industry, especially banks, are still using a programming language that is over 50 years old. However, this doesn’t necessarily mean they are out of date, rather these giant institutions have found in some cases that their ancient systems just work. They are stable and secure software programs. But databases are not like static source code; they are always moving and changing. With the most important client information sitting in the database, it’s no surprise that banks and other financial institutions lead all other industries in adoption of Continuous Delivery methodologies for the database.

    A recent survey we conducted revealed that the banking and finance industry DBAs far outpace non-financial companies in terms of database Continuous Delivery adoption. Of the 55% of respondents not in the finance space practicing Continuous Delivery, only 49% said they practiced CD on their databases. When we asked banking and finance DBAs and managers, we found that over 70% utilize database automation for software and 64% of those also practiced CD for the database.

    These institutions are required to be agile in their database methodologies due to having such a large number of applications to deal with on a daily basis. Moreover, database errors that cause downtime are incredibly costly and financial institutions cannot afford to have any. Aside from the peace of mind that your database can run itself and adapt on the fly, proper database automation and version control can save financial institutions 80%-95% of time it would normally take to manually process changes and there is clear ROI when implementing Continuous Delivery.

    The economic effects of unplanned outages are prohibitive. According to a report by Gartner research, unplanned outages, on average, cost $42K per hour. When it comes to financial services companies, research shows the cost are much higher at about $110K per hour, the bigger the footprint that a company has, the more money it will lose when it fails.

    These statistics are one of the main reasons that we’re starting to see mainstream adoption of Agile and DevOps practices in finance companies. DevOps favors automating as many processes as possible and automation removes the potential for human error. By automating our processes, we allow machines to handle the redundant tasks freeing up the time and resources of skilled database professionals for more valuable and cost effective responsibilities.

    DevOps for Database, automation, control and enforced change management, provide confidence in the integrity and quality of the database code and virtually eliminates the potential that harmful database changes will find their way into production.

    About the Author/ Yaniv Yehuda

    Yaniv Yehuda, CTO, DBmaestro (http://www.dbmaestro.com/)

    Yaniv is the Co-Founder and CTO of DBmaestro, the leading provider of DevOps for Database solutions which enable control of database development and deployment. Yaniv is also the Co-Founder and the head of development for the Extreme group, a leading IT services solutions provider group.