Category: Doin’ DevOps

  • Tips to change culture (at least in your team).

    Tips to change culture (at least in your team).

    DevOps is all about automation, enhancing the Application Lifecycle Management and culture change. In the last few years there have been tons of conferences and articles about the benefits of a cultural change on a project, but I have yet to read a paper about how to actually change culture.

    How do you manage to get people to change the way they think and act towards each other and professionally? This is a huge challenge that most DevOps leads will have to overcome. Here are some of my ideas.

    Create cohesion.

    The first thing to keep in mind when trying to implement DevOps (or on any other kind of project for that matter) is that people who don’t have fun at work will not perform as good as they could. How to make people happy to go to work is up to you, but I find that simple activities such as team exercises and/or after-work beer work best.

    Whenever a new member comes to a team, try to get everyone together to have fun and talk. Take them out to a game of laser tag, get some sumo suits and make people try wrestling amongst other sports. Try to identify a common interest for a sport and have everyone play together. Remember, this is not a competition, do not force people to participate if they don’t feel like it.  There are neither losers nor winners, so try to balance the teams as well as you can.

    Another way to get people to know each other and ease potential tension is to take people out. Take your team out to discover a new restaurant or pub and discuss the place together, what they would have changed in the decoration, which Belgium beer is missing from the menu, why on earth do they serve foie gras on a scallop… The point once again is to try to discover others’ interests and behavior outside of work. This specific activity however needs some rules; try to limit the alcohol intake and by all means – do not talk about religions or politics. While those two topics can be fascinating with your friends, some people might get offended, even if they say they don’t mind.

    Back on the project, the use of Scrum is highly recommended. Even though the benefits are debatable, I find that shy people tend to open up and contribute more during those sessions than they normally would. On top of that, Scrum meetings include everyone on the ladder, from bottom to top. Only one rule here: no matter what, remember to actually do your daily stand-up, if you can’t do it personally, get one of you team members to conduct it and check afterwards that is has been done. Delegating is not a sign of weakness; it just shows trust in your team.

    Create incentive.

    People make mistakes, always. There are, however, ways to get to get people to make fewer of them. Pointing fingers and blaming people is definitely not a solution. Making people pay for their mistakes is more efficient. Say each team member has to pay 50 cents (adapt to your own country and project) for every mistake (new bugs, fail in commit, forgot to write a comment in SVN/Git at commit…) they make, and things might start to change. The point here is not to rob your employees, so you will have to find a creative way to use that money. You could for example gather that money at the end of the month to buy candies for everyone. But then that’s just punishing people, which does not make it fun, so you have to come up with a way to compensate for that too. If the management doubles every penny in game, then everyone is involved and there starts to be some interest. But then again, you might get extra bugs because everyone loves candies, so you could triple the sum of gathered money when there are fewer problems than the previous month.

    At that point of reading, you might think it’s actually costly and doesn’t benefit anyone except dentists. Well you’re wrong. In the worst case scenario we’re talking about tens of dollars a month, and given that you will actually get some accurate metrics about this, this is a no-brainer.

    If candies are not popular in your project (is that possible?), you can also get the good old “employee of the month” award to whomever created the least problems and reward him/her with cinema/theater tickets, a dinner for two… Please note that on high stress periods, dinners for two, even though a lot more expensive than regular rewards, are a lot more appreciated as they might represent a good opportunity to ease work related tension in a couple.

    Develop people.

    Don’t see your colleagues, especially the young ones as just resources, rather think of them as potential wonders. Everyone aspires to shine and it’s all up to you to make it happen. Do not ever take someone else’s ideas and get the glory. That would be idiotic, mean and potentially a good way to get your colleagues to quit or resent you.  Instead, go on, try to encourage people to speak up, make them give technical presentations, make them speak and give them a positive feedback (even on failures, there are always good ways to explain how one can improve).

    Once again, Scrum can help you with that. Identify the elements in your team who need some help and pair them with some others that are more comfortable for some small sessions of peer-programming, peer-presenting, peer-testing… once a week. It can cost some time in the beginning but the benefits are potentially huge in a long term perspective. There is nothing better than a team that has learned to work together and where the first reflex when someone ask for help is to actually help, not to ignore.

    Demonstrate values.

    One of the things that really is important to remember, is where you come from. No matter your position today, you didn’t start at the top and you’ve had to learn all the values that you try to get your team to demonstrate today. So show them every day, what it means to create DevOps; a potentially highly stressful job, especially in the beginning, where tension between people should not exist. Smile as much as you can, take time to help others even if you that means staying later at night, be proactive once again, try to identify potential bombs and defuse them before they are seen by others. All in all, be nice to others, treat them as you would like them to treat you.

    But what if you are faced with a pigheaded team member? Someone who is convinced they are right, no matter how many people disagree? Well once again, do not ignore them; that will only create frustration. Instead, get some documentation or an expert and talk openly about the topic without being aggressive. In the end, the team member should be respectful of your efforts and have won some valuable knowledge in the process.

    The biggest piece of advice I have is this: Do not ever be arrogant, do not ever be pompous. Help others, always, and do it nicely and in a humble way. The leaders we like are not the ones that talk too much but the ones that are the nicest to work with.

    Changing a culture on a project is a tough thing, but it can be done with a few tricks and a lot of time spent helping others. Luckily, smiles and good atmosphere have a tendency to spread fast and the tight links between Dev and Ops can be strengthened quite fast with some goodwill.

  • Continuous Delivery and Continuous Deployment

    Continuous Delivery and Continuous Deployment

    A question comes to mind in what exactly do we need to do to move forward with Continuous Delivery and Continuous deployment, How should we start or rather where we should start from. Many times teams may not have achieved the continuous integration properly, maybe due to lack of proper tools, workflow or historical process.

    What we need to do before even thinking of CI/CD, what are our prerequisites:

    • Source code
    • Builds System
    • Deployments
    • QA Automation

    We need to understand release != deployment

    Lets assume with the help of right source control tool, build system and QA, automation team has achieved the continuous integration which means developers integrate code into a shared repository several times a day and each check-in can be verified by an automated build, allowing teams to detect problems early.

    The next step: From CI to Continuous Delivery deployment

    What exactly is Continuous Delivery, Its attitude to make software safe and to go fast
    Lets begin with basics, How much time does it take to deploy one line change or a small bug fix in to the production? Exactly, our goal should be to achieve this as fast as we can.

    Why We Need Continuous Delivery?

    • Yes, its Speed which is needed in current fast paced software industry.
    • Stop waiting around on each other to provide go ahead and open the gate.
    • Quality – It raises the overall quality of your application – No surprises.
    • Capacity and Clarity – Get every team on one tool – Eliminate manual error

    How are we going to do this…..

    Key Areas:-

    • Speed is essential because there is an opportunity and cost associated with not delivering Software
    • Frequent Releases, How frequent is frequent? Why do developers need to wait after writing code?
    • Every build can be potential release candidate.

    Bottom Line

    Remove Manual Work/Steps.
    Move towards complete automation

    Delivery Flow

    Here is the Delivery Flow.

    Ingredients:-

    Configuration Management
    continuous integration/ Automated Build
    Automated Deployment
    Automated Testing

    Obviously no single team in a company alone can achieve CD without the help of other teams. It needs collaboration and team work to implement it, Lets take a look at what do we need from various teams

    Dev Team

    • Collaboration – Integration is no longer an event
    • Check in regularly
    • Make small frequent releases because Big Change -> Big risk –> Big effort and Small Change –> Small risk –> Small effort

    QA team

    • Should have complete automation or whatever max is possible. More manual testing leads to more delay.
    • Keep the Build and Test Process Short
    • Break the QA automation into small pieces so that they can run simultaneously and do the QA frequently.
    • Publish the report

    Environment

    • Should have complete Environment to test everything.
    • Flexible and scalable so that multiple env can be spawn any time and can be extended.
    • Managed as prod or at least similar to prod

    This increases Confidence in build’s production readiness reducing release risk.

    Main blocker to achieve CD

    1. manual work
    2. Lengthy Process
    3. ticket system
    4. approval

    Now, How do we deal with blockers

    Do not add gates, add only checks. Adding gates to prevent issues from creeping into prod is not so smart. There are many more issues hiding, you just don’t know them

    Approval!!! Why do we need CAB?

    Dev Sign off
    QA sign off
    Ops and Dependency sign off
    Infra sign off

    Dev Sign off -> Use proper project mgmt tool to track everything that goes in to software with adequate workflow. Get everyone together at the beginning so that these will not be surprises at the last moment. Keep meeting across the team during entire development. This way you don’t need any approval since you have already taken care of all necessary elements during the entire development process.

    QA Sign off -> Publish consolidated QA report with each and every build which should be visible to everyone. You should have proper definition of “Go Ahead” instead of just random checks, Nobody has to ask QA for a thumbs up for the go. • Minimum criteria for Q has to be set.

    Ops, Dependency and Infra Sign off -> Include them at the beginning of the project. Have capacity and change planning during requirement gathering, This gives enough time for ops and infra to get ready for the release.

    Role of SCM/ Release Mgmt

    SCM/ Release Mgmt provide the workflow, tools and necessary filler to enable all teams to interact and automate their part.
    The picture below shows How SCM and Release Management help each team.

    Source code support for Dev by defining branching policy, Build Setup Support for Dev team to provide continuous build with predefined configuration and in managed environment which includes static code analysis like unit test, code coverage, checkstyle etc..

    Enabling QA Automation and reporting via build tools (Like Jenkins) to support QA and Dev to run their automation and aggregate report on central location. which include QA automation/regression, functional code coverage etc..
    Infrastructure for QA Support like managing integrating environment, Deployment Automation etc.

    Ops, Dev and QA CAB Automation and release planning support.

    It’s clearly visible that CI and CD is not a one man show. Planning, collaboration and teamwork are essential parts of the execution.

    Tools can be chosen by the org/team whichever are suitable for them, but the main part is automation and integration of the tool with each other to make flow smooth and free.

  • Service Virtualization – Window into “Advanced” DevOps

    Service Virtualization – Window into “Advanced” DevOps

    The new style of IT is pushing DevOps teams to reinvent software building, validation and delivering processes with an aim to shrink development cycle time and reducing defects in production. But in constrained testing environments, DevOps teams face a range of challenges in following the continuous integration and delivery approach for large multi-tier applications. Because release cycles are compressed application testing coverage, and what you test is constrained. And on top of that, consumers demand for an always-on, easily enhanced and mobile experience that comes with cloud-sourced and cloud-delivered composite mobile app offerings.

    (more…)

  • 10 Top Tips for DevOps Cultural Change

    10 Top Tips for DevOps Cultural Change

    DevOps is commonly recognised to be primarily about creating a culture of effective and seamless collaboration but this is often the hardest piece to enact when it involves changing an existing culture (as opposed to creating a culture from scratch in a startup type environment and having the luxury of hiring individuals with certain mindsets and defining working practices from day one). So how can you change culture? Here are our top ten tips:

    1. Learn to Trust

    It sounds a little cheesy I know, but at the core of what DevOps is trying to fix is the conflict that has come about from years, decades even, of development and operations teams being segregated from one another and not being enabled to communicate, let alone collaborate, effectively. This situation has been exarcebated by the pressures to innovate and push rapid change battling with the need to stabilise fragile systems – and created what James Turnbull described as a ‘toxic’ working environment.

    So the very first thing that needs to happen is a unified agreement and understanding that we are all on the same team.

    DevOpstastic Tip ~=>
    Take some time to review your individual and team goals and ensure none are in conflict. Then create a single goal that brings everyone together. Get everyone to agree it. Fire anyone that can’t. Kind of joking – but kind of not.

    A change in culture will demand changes in the way that you do your daily work – habits will need to be broken.

    DevOpstastic Tip ~=>
    Shift left – arrange your teams around projects not linear tasks and involve operations staff who will be responsible for your application in production much earlier in the process. Being involved sooner will help them plan for when the code is ready and feel they also have some skin in the game.

    2. Understand Motivations

    Conflict often occurs when people don’t understand each other – I’m sure we’ve heard the phrase: “They have a personality clash” countless times over the years. Humans and our brains, personalities, backgrounds and characters come in a whole heap of flavours and this is a good thing since, as my mother always says: “It would be boring if we were all the same.” However, sometimes at work, under pressure, it’s difficult to take the time to understand why someone is reacting or thinking differently to you and the immediate response is to fight or denounce it; not just unpleasant but downright unhealthy.

    So if we accept we are all on the same team, how can we all get to know each other better and grease the wheels of collaboration? I know I’ve been subjected to countless team-bonding exercises that I’ve found quite excruciating over the years so I am applauding the new wave of neuroscience that allows us to be a little more exact about what’s going on in our heads than just try and decide whether I am a green or red personality type.

    Our particular favourite at the moment is SCARF from The Brain at Work – the premise is that the same reward circuit in our brains that we use for survival and defence is also used for our motivations at work. So if I am motivated by Autonomy (the ‘A’ in SCARF) and someone at work attacks my sense of autonomy in the workplace (by, for example, micro-managing me) I will have the same neuro-response as if they made a direct attack on my life – including the release of the related neurochemicals, in this case, cortisol, the stress hormone. Similarly, if something happens to reinforce my sense of autonomy (for instance, an event I created has an excellent turn out and feedback) I feel great and get a nice rush of dopamine.

    DevOpstastic Tip ~=>
    Have a SCARF workshop with your team(s) and ask everyone to complete this self-assessment first, share the results and discuss. We don’t always agree with the results of this assessment tool but finds that only adds fuel to the fire of debate.

    My key takeaway here is that we should try to understand ourselves first, and then those around us (a bit more ‘fluffy’, but this is about mindfulness). Having this level of insight into ourselves and our colleagues means that when conflict occurs we have a better chance of taking a breath and understanding what’s really going on, remembering we share that common goal and working closer together, collaborating to reach it.

    We’ve more recently also starting working with unconscious bias – we’d love to hear from you if you have any experiences in this area.

    DevOpstastic Tip ~=>
    If you want to know if you have unconscious bias you can test yourself using Harvard’s online Project Implicit testing tool.

    3. Eliminate Blame

    Let’s start by watching one of my favourite DevOps videos – here you are:

    [youtube http://www.youtube.com/watch?v=2PjVuTzA2lk?wmode=transparent]

    If we’ve taken on board tips one and two and are now in a ‘circle of trust’ we should be able to avoid this kind of finger-pointing blame game. There is no place for blame in a DevOps culture – it is a pointless waste of emotion. We need to understand what went wrong in a retrospective but the most important thing is how to make sure it doesn’t happen again.

    DevOpstastic Tip ~=>
    Try running a blame free post mortem/retrospective next time you have an outage. Here are some retrospective plans you might want to road-test.

    4. Embrace Smart Failure

    In order for businesses to thrive, survive, even, in today’s uber-competitive markets, innovating at speed is an imperative. When we are assessing organisations’ DevOps cultural maturity, one of the features we measure is the organisations’ capability to ‘dance around the edge of failure’. Innovating at hitherto unseen velocity introduces a whole host of risk that must be mitigated. Systems need to be created that can warn and pre-empt failure and capabilities need to be implemented to do things like instantly redeploy a last known good state or working version of an application in the event of a failure. In no way are we suggesting that failure should be celebrated.

    In addition to creating an safe environment, where failures are quickly and easily fixed, you can also prepare for failure by spending some time prioritising features and architecting applications in a composable manner – for example, your business transactions are probably more important than your social feeds. At a recent workshop we were told a story of how the French railways online ticketing application was down for three days due to a problem with its analytics – if the components weren’t so tightly coupled, the users wouldn’t have suffered as the reservations system could have been operating standalone.

    DevOpstastic Tip ~=>
    Study Netflix’s Chaos Monkey approach and identify a way to test failures safely in your infrastructure.

    5. Focus on Bottlenecks and Flow

    If you’ve read The Phoenix Project you’ll know all about The Theory of Constraints – our favourite summary of the concept is that “Any improvement not made at the constraint is an illusion”. Work hard as a team to identify the real constraints – and watch out for that common pitfall as described via the Brent character in the book that those people who often look like heroes are, in fact, your bottleneck. Try to understand the real value to your business of each improvement and prioritise them accordingly.

    DevOpstastic Tip ~=>
    Try playing ‘Featureban’ – exercise #7 in our DevOps LiftOff Workshop

    Don’t be afraid to rethink your process or find a new pair of eyes to see the things you can’t because you are too close. Check out this video to see the improvements motor racing have made to their process over the last fifty odd years – check out the resource utilization, perfect process and automation in 2013:

    [youtube http://www.youtube.com/watch?v=RRy_73ivcms?wmode=transparent]

    6. Eliminate Unplanned Work

    Another major topic from The Phoenix Project and a universal complaint we hear from organisations when we start working with them: “How do I find the time to save time?”. People know their processes can be improved and there’s a good chance automation could exploit advantages even further but if the culture is to keep doing things that are breaking and broken it’s tough to put aside the hours outside of the firefighting to make DevOps happen.

    It’s nigh on impossible to tell a business they can have innovation or support but not both but it’s sensible to get a handle on the projects running and the resources available and the volume of unplanned work coming through the pipe. Some lower priority projects may need to go on hold while the DevOps project has higher priority – this will probably need to be mandated from management and have a solid case behind it that all subsequent projects will be accelerated as a result once the foundation has been laid.

    DevOpstastic Tip ~=>
    Read The Phoenix Project – what on earth are you waiting for?!

    7. Be Continuous

    There a lot of these out there – we’ve seen continuously:

    • integrate code
    • test
    • integrate
    • deploy
    • deliver
    • fund

    The idea is to always have software in a shippable state and have a DevOps toolchain capable of deploying on demand – have your software development lifecycle like a manufacturing pipeline.

    Our favourite here though is ‘continuously fund’ – as Forrester say:

    “Organizations that deliver at rapid release cycles create a stable basis for funding a continuous stream of innovation. They use customer feedback and business priorities to continuously prioritize the capabilities that will result in the greatest business value.”

    DevOpstastic Tip ~=>
    Try setting a pre-approved innovation budget and short, iterative approval cycles rather than annual, large project driven budgets.

    8. Form Dedicated, Cross-Functional Teams

    Lots of people ask us about creating DevOps teams, something we think it mostly counterproductive as it has a tendency to create another silo. However, there are challenges here as it’s impossible to make everyone a master of everything, particularly in the case of more complex technologies. Simpler technologies can have the advantage of bringing people closer to business.

    We recently held a workshop with Jonny Wooldridge, the CTO at The Cambridge Satchel Company where we discussed the DevOps team he built and managed during his time at Marks and Spencer. Jonny’s team acted more like a ‘tiger’ team, where key resources could be drafted into project teams as needed to evangelise the DevOps way and help cross-skill through the project lifecycle.

    DevOpstastic Tip ~=>
    When you are considering bringing on a new technology, consider the time to mastery – the shorter this is, the better chance your team has for cross skilling for it.

    The key is to get everyone thinking in a DevOps manner and cross-skilling and obtaining mastery as time and resources allow – but arrange your resources around projects, rather than their core skillset.

    9. Love Transparency

    The adage says that knowledge is power – but this isn’t always the case. Sometimes you are better off sharing your knowledge – not only may you gain respect and credibility, you may find that the team around you are able to then work faster to achieve your shared goal – and encourage others to share their knowledge with you too. We can’t possibly all know everything but if we keep things to ourselves, we are often stymying the performance of our team and therefore ultimately ourselves.

    DevOpstastic Tip ~=>
    Write a blog post about an area of your work you are passionate and enthusiastic about. Share it with your team. Share it with the world. Repeat.

    Knowledge hoggers can often be bottlenecks – for instance, we often see cases where an individual has had responsibility for a deployment process, become bored of repeating it manually and written scripts to speed it up – and more scripts and more until they have a ‘script-monster’. Now they are the only person that can possibly understand this part of the process and when anything goes wrong they’re the only one that can fix it – what a hero! If a person is motivated by Certainty (back to SCARF) they’ll certainly enjoy the sense of security and indispensability it will give them – but it’s not so helpful for the rest of the organisation as now they are a serious bottleneck.

    Transparency and visibility are features of DevOps cultures and, going back to tip 2, Understanding Motivations, those of us motivated by fairness will feel happier in a culture of openess and equality. Secrets can be very counterproductive and alienating.

    10. Build Autonomy, Mastery and Purpose

    Changing your culture will probably mean you need to revisit the way you target and reward the people in your business and review the goals you have set, ensuring they are not in conflict and there is a clear, single goal everyone can focus on. The psychology of rewards may surprise you – have a look at this Ted Talk from Dan Pink.

    You need to be able to create a working environment where people feel they have autonomy, mastery and purpose – these are the ultimate rewards. Take a look at this blog for an exploration of DevOps goal setting and rewards in more detail.

    DevOpstastic Tip ~=>Have you heard of The Illusion of Control? If not, spend a little time researching it and imagining how it could affect your outputs at work.

    What’s your top tip for making DevOps cultural change happen?

  • DevOps: What It is and How to Cook It

    DevOps: What It is and How to Cook It

    What is DevOps?

    DevOps is a funny concept.I have spent the last few months trying to find an accurate explanation of what it is, reading books, articles and participating in discussions which led me to the following personal definition: DevOps is a set of processes which aims to enhance the Application Lifecycle Management (ALM) of an Agile project by creating tighter links between the different actors (Developers, Operations and testers) through the use of automation and communication. Wow!

    Why DevOps?

    The traditional development cycle of an Agile project usually leads to two to four major releases a year which means the delivery of a lot of new functionalities, tons of bugs fixes and therefore a lot of testing pre and post production. Such releases can prevent the good visibility of specific problems in the application itself; create a huge stress on the people involved as it is not conceivable to miss a deadline with two releases a year, and create an ever-extending list of requirements for the next release.

    While many important projects succeed in this approach, many major companies such as Facebook, Amazon or Flickr have chosen to perform shorter cycles of over ten deployments to production a day. That process enables them to have a very specific scope per release and with the help of automation almost everywhere, especially in testing, creates higher quality deliveries.

    How to DevOps?

    One important thing to remember here is that you cannot call Merlin and ask him to send you a “DevOps person” to solve all your problems, including your personal ones.

    In order to achieve a successful transition from a two-releases-a-year cycle to a ten-releases-a-day one, it’s going to take time, a lot and skill, a lot. The team in charge of the changes will have to study the way you work, analyze the current process, identify potential problems and then make a list of recommendations

    Those recommendations usually include breaking the different walls between people. One big goal of DevOps is to make people realize that whatever their role, their objective is common with the rest of the project: to deliver quality and to do so efficiently. Imagine for one second that a bug is found in pre-production (it happens). Usually the sysadmin will perform an analysis within its domain and if he cannot find the cause or identifies it as a “not my fault”, will forward the bug to the development team who as usual will answer “it worked in dev., must be environment related”. The pointing fingers game will take some hours maybe days before someone finally jumps in and finds the problem.

    Now imagine the same bug handled by people on a project where DevOps is used correctly. When the bug is identified by testers, developers and operations are contacted together and retrace all the steps. Is it due to an error in the specifications? Can we reproduce it in other environments? What changes in the code triggered that bug and so on and so on. By having people working together instead of one after another, you create ownership of the product and a strong team feeling. If everyone is concerned when a problem arises and if everyone understands the application from the conception to its application, problem solving becomes a breeze and an actual good way to create natural collaboration between people of different groups.

    The other main changes that come with DevOps are related to automation. Maybe easier to implement than mentality changes, setting up automated releases, testing, deployments requires expertise and a lot of testing on the way but can lead to great time sparing.

    DevOps in Real Life.

    One my favorite thing in life is food. Eating it, making, talking about it… Let’s use food as a simple example of how to use DevOps.

    Imagine yourself in a starred restaurant kitchen; there is the executive chef, a sous-chef, a saucier and a vegetable chef.

    The chef has envisioned a dish and asked his new staff to do their parts based on what he explained to them. Ideally this would go well but: the sauce is too acidic for the vegetables and the plating is terrible with the shapes and colors making the dish look sad. Everyone blames each other and the utensils for not being adapted.

    Add a zest of DevOps to the recipe and try again.

    Now the vegetable and saucier chefs work together and make sure that what they produce respects the wishes of the chef while working perfectly with the protein produced by the sous. It doesn’t happen instantly but they all try the overall dish together and after many iterations, manage to produce an experience that is visually astonishing on a plate and which flavors combine divinely in a way that is neither too heavy nor too light. To be able to cook and plate correctly, they had to buy more adapted utensils that everyone in the staff could use if needed.

    One month goes by and the head chef asks them once again to repeat the operation on a new dish.

    This time the whole staff knows how to work together and while everyone is working on his own specialty, they all work together from the beginning, sharing their experience with the utensils and understand which way to go, tasting their creation as they progress, leading to another perfect dish done in a much shorter time.

    DevOps is not magic.

    Many things can go wrong in a project and DevOps might be one possible solution to solve some specific problems but it is important to remember that implementing new processes, changing mentalities and using new tools can take a very long time and be really expensive.

    You have another definition? You don’t agree With me? Feel like adding something? The comments are here for that, please let me know what you think, sharing is caring!

  • DevOps:  As critical for the Systems of Record as for the Systems of Engagement

    DevOps: As critical for the Systems of Record as for the Systems of Engagement

    There is a lot of talk about DevOps, and how it improves the process of delivering business value. Much of the discussion of DevOps has been focused on web based companies such as Facebook, or the mobile and distributed development teams. But DevOps principles apply just as well to the traditional Systems of Record, z/OS applications.   Many of these applications have been around for a long time and have very mature development and deployment processes. These processes include the requirements for separation of duties and audit tracking. Over the years these processes have developed into lengthy procedures, to be comprehensive, the net effect of this has been to reduce the number and frequency of the deployments to reduce risk. However, there is a better way.

    In the recent blog posting by Ashok Reddy he talked about the 8 critical DevOps Practices: innovate, deliver, repeat. All of these are as applicable to z/OS applications as any other, for this blog I want to focus on just a few of the practices, Loosely Coupled Architecture, Automate Testing using APIs, Transparency and Minimize hand-offs, Maximize flow.

    DevOps is about bringing the entire organization together, including Development and Operations, to more efficiently deliver the business value. The practices of “Transparency” and “Minimize hand-off, Maximize flow” go to the core concepts of the people aspects of DevOps i.e., integrating the organization.   By breaking down the barriers between the organizations, people can work together to accomplish the business goals more quickly. This does not necessarily mean actually changing the organizational structure, but it does mean having the organizational structures work together as cross functional teams.

    One example would be the monitoring and optimization practices. These are generally done by Operations after the applications are deployed into production. However, if instead Operations and Development worked together in the early part of development cycle to monitor and optimize the application using the same capabilities that will be used in production any issues can be addressed much earlier in the cycle. This also helps both teams learn what is required to monitor and manage the application long before it is ever deployed to production. In many z/OS environments today, the same infrastructure is already available in the test environments, but it is seldom used.

    A second example is to have Operations define the deployment automation such that it can be done by Development in early stages and turned over in later stages. This ensures continued testing of the deployment capability and a validation that by the time the deployment is done in production; the deployment process itself will be well tested. Through this integration of Development and Operations you remove the hand off that currently happens only at the final stage just before production deployment.

    The practice of loosely coupled architecture is key when building multi-platform applications. Having clear interface definitions between the different parts of the application is essential of a loosely coupled architecture. It is easier to allow the capability to build independently, tested at the interface level and then brought together when the parts are ready. By defining clear interfaces between application components and properly handling forward and backward compatibility, there is less of a requirement for coordinated simultaneous deployments. This allows teams to be less dependent on other parts of the application during the development process, and allows for easier automated testing.

    The practice of Automate testing is critical to any DevOps transformation. It provides the capability to improve the speed of delivery while reducing risk. Moving to agile development practices while still requiring people to perform manual tests, will not allow the quick iteration on the capability. Even for existing applications you can generate interface tests to begin to build up automated regression test cases. Automated tests can be generated based on interface definitions such as a JSON schema or a COBOL copybook for new development. The increased focus on testing the interfaces early helps reduce the integration problems generally found late in the development cycle.

    These are just a few of the key practices of DevOps but hopefully they have started to demonstrate how DevOps is as applicable to the Systems of Record on z/OS as it is to Systems of Engagement.   To learn more read the best practices for a DevOps approach with IBM System z whitepaper or watch the replay of the November 20, IBM DevOps Symposium webcast.

     

    About the Author:

    RosalindRosalind Radcliffe is a Distinguished Engineer within the IBM Rational organization. She is Chief Architect for CLM and DevOps.  She is responsible for driving the DevOps for multi-platform architecture.  This includes System z and Power system.  In addition she is responsible for the architecture for the Collaborative Management capability for Enterprise solutions. This includes UrbanCode Deploy and Rational Team Concert’s support for standard Mainframe development activities.

    She is a member of the IBM Academy of Technology and a Master Inventor.  Prior to Rational, she was in Tivoli responsible for the SOA Management Strategy for IBM.

  • SQL vs. NoSQL – Unsettled Debate & DevOps Pitfalls

    SQL vs. NoSQL – Unsettled Debate & DevOps Pitfalls

    Big Data, a relatively new phenomenon driving the industry hype-cycle is a mainstay in the database business. IT companies are convinced over the endless value proposition of the operational insight that comes with leveraging Big Data ( or Big Small Data ), so much so that it’s easy to overlook its impact on traditional database technologies and practices.

    (more…)

  • AWS Best Practices – Enterprise Perspective

    AWS Best Practices – Enterprise Perspective

    Amazon AWS has changed the IaaS game for startups and growth companies. There are several practices we must acknowledge the importance of during the adoption and implementation of AWS services. Readers are more than welcome to comment, suggest modifications and even add their own little practices they follow during roll outs and implementations. As clients start piloting cloud initiatives it is best to avoid common pitfalls. The below best practices are a good first step in that effort. Below we have compiled a top ten list to be expanded in the future of best practices we have learned during our aws deployments. We look forward to expanding this list in the future.

    1.   Choose your VPC infrastructure carefully

    a. As you move into the AWS environment often the first thing firms want to do is create a VPC, the question is what type of architecture is appropriate for you?  The answer to this question is driven by the intended use case.  Often internet based organizations will opt to go with a VPC containing Public and Private Subnets. Opting for this type of VPC ensures the services that don’t require direct access to the internet are in a segregated subnet while locating public services access to the public subnet.

    b.   Existing enterprise firms intending to burst into the cloud often don’t intend to utilize public facing services and may opt instead for private only subnet and create a secure tunnel between their enterprise and the public cloud.

    c.   The architecture with the greatest flexibility tends to be a public/private subnet ensuring the public subnet is available if necessary and can remain unpopulated if not needed.

    2.   CIDR Block Selection is driven by two realities: VPC address space is fixed and can’t be changed after it is created.

    a.Ensure your VPC address space doesn’t overlap with your corporate space, for internet based firms this is less of an issue, however for enterprise firms this can prevent the need to rebuild the environment after a significant investment in time and resources

    b. While AWS allows CIDR blocks to be as large as /16 don’t waste space, having several very large subnets reduces one flexibility as needs arise to support multi-AZ services, or other eventual requirements.  On the other hand don’t overly constrain your environment.

    3.   Environment Isolation is an important consideration, the same isolation that’s present in the physical environment should be present in the cloud environment.  The cloud give firms more flexibility in implementation of this isolation, in an effort to control cost, development, integration/test and production environments can be registered under separate accounts ensuring accurate billing and cost controls.

    4.   Security: Use security groups to isolate services and management rules

    a. Separate public traffic from private using subnets

    b. Create a SSH gateway as an entry point for SSH communication

    c. Create a VPN tunnel for management access

    5.   VPC/Enterprise Integration

    When integrating AWS into an existing enterprise environment planning will ensure minimal rework.  Often the key consideration is assigning IP ranges to AWS that do not overlap with the corporate space.  In this way routing between the corporate LAN and AWS will be seamless.

    6. IAM: The primary AWS account should be treated like the traditional root user in Linux systems.

    a. Create a separate account specifically for administration, with the minimum permissions necessary to perform the task at hand.

    b. Create separate accounts for each user, or service to ensure audit traceablity

    c. Permissions should be assigned to groups and users to the group, this will minimize duplication of effort

    d. All users should utilize strong passwords, in this case refer to your corporate password policy.  Cloud passwords should be as secure as the corporate infrastructure.

    e. Don’t share credentials this will wreck audit traceability, and corporate policy, use roles to assign permissions.

    f.  Rotate credentials on a regular basis, the same way corporate credentials are rotated.

    7.   Disaster Recovery :Remember Disaster Recovery is designed to get backup after a failure.  Traditional enterprise environments are often limited by the question do we need to building a duplicate system for recovery or can we through vendor SLAs perform a data recovery.  In terms of the cloud enterprise customers are able to (at a lower cost) duplicate their environments across availability zones, regions and vendors to create an exceptionally resilient service offering.

    a. Do create multiple availability zones.

    b. Do use an automated CM tool to maintain your configurations

    c. Do create snapshot of your volumes.

    d. Do run drills to ensure everything operates as it was architected.

    e. Do run drill more than once.

    f. Do ensure everyone relevant is aware of the procedure

    g. Do architect you environment with sufficient redundancy, to minimize the need recovery efforts.

    8.   Security Groups: Security Groups like traditional firewalls should be set with the least permission necessary to accomplish the mission.

    a. Decide on a security group methodology; by creating groups to manage access to services ie allow access to ports 80 and 443 for nodes that will be a web server.

    b. Create a ssh gateway/ Bastion node and group:

    i. Only allow external SSH access to the SSH Gateway

    ii.Only allow local SSH access from within a security group, or from the gateway node, this minimized an attackers ability to traverse the networks between zones.

    c. Define an enterprise naming convention and stick to it.

    d.Utilize the ability to assign multiple security groups to a single asset (up to 5 groups per asset).

    9.   Naming Conventions: AWS assets should be named in such a way they can be readily identified, a suggested standardization around the purpose of the Instance, its environment, its region, and its sequence number.  An example of this might be haddop_dn_prod_usw1_001, this represents a production Hadoop data node, in the US-West 1 region.

    a. When naming an AMI its a good practice to include the creation date as part of its name, and a complete description. This will minimize confusion when selecting ami for new instances.

    b. When naming security groups and key pairs, continue to use the convention purpose, environment, region, function.  For example db_prod_usw1_key or ws_prod_usw1_sg

    10. Elastic Load Balancing: Use multiple availability zones to balance traffic in the event of an environment failure (this can still happen within the cloud)

    a. Use Route53 to balance traffic between regions, this is not supported by ELB

    b. Use ELBs for more than just web traffic, most any protocol can be supported by the ELB, ensure your service can support it

    c. ELB timeout after 40 seconds, ensure your application touches the socket before then to prevent the session from timing out.

    These are my top 10.  Let me know what you think I missed.

  • You Asked for Quality, You Got Alienated QA

    You Asked for Quality, You Got Alienated QA

    The role of Quality Assurance (QA) is creating a bit of a buzz these days, because even as teams and budgets are shrinking, the demand for better quality is increasing. The chilling repercussions of service disruptions caused by software flaws warrants interest even at the executive level: “Nobody’s madder than me about the fact that the website isn’t working as well as it should,” said President Obama when HealthCare.gov went down last year due to software defects.

    (more…)

  • 3 Prerequisites before Jumping on the DevOps Bandwagon

    3 Prerequisites before Jumping on the DevOps Bandwagon

    Top Three Prerequisites before Jumping on the DevOps Bandwagon

    The DevOps frenzy is truly upon us and organizations are under increasing pressure to jump on the bandwagon in order to deliver software faster and faster. However, Agile delivery and DevOps is not a plug-and-play proposition – it requires time, effort and persistence. Also most enterprises, unlike Silicon Valley startups often held out as shining examples, don’t always have the luxury of dealing with newly designed software and freshly minted teams. Implementing DevOps practices is a significant change and often an uncomfortable process for many enterprises. Let me share what I believe are the top pre-requisites that must be in place to ensure success.

    Organizational Commitment

    This may seem obvious, but ensuring alignment and setting the right expectations on the goals and path to DevOps nirvana is a first and important step. Organizational structures, support at a senior level, a clear but flexible roadmap and adoption of common objectives are key concepts to drive organization commitment to this transformation. Let me elaborate on these a bit.

    Overcoming cultural differences between Dev and Ops organizations is essential – and often enterprises create a new “DevOps” organization with personnel drawn from both organizations chartered with instrumenting the handoff from Dev through to production rollout. This new centralized org, however, also runs the risk of becoming yet another silo. Ideally a DevOps group should operate as a center of excellence managing the tools required across the lifecycle and guiding/overseeing the right methodology within the Dev and Ops teams. Similar to the way Agile development practices have driven testing and quality assurance methods deeper into the dev cycle, DevOps methodology demands that configuration and deployment requirements in a true production environment be considered and tested as part of the design and development cycle. On the Ops front, this new methodology demands a shift from managing risk by limiting change towards enabling change while reducing risk by automating repeatable steps for deploying small packets of changes often.

    While DevOps initiatives are typically sponsored by high ranking IT executives, it is essential that senior and mid-level leadership in the Dev and Ops organizations completely support and prioritize the changes that, while benefiting the overall goal of shipping software faster and better, may in the short term seem to impact narrow departmental goals. Continued visibility and focus from management, and joint ownership by Dev and Ops leaders goes a long a way in ensuring the teams stay aligned.

    A clear and well-articulated vision and compelling reason for change communicated clearly through the organization is essential for execution. This must be accompanied by a phased roadmap that enables short-term wins and clear milestones. Besides benefiting the business, the aim is to make the jobs of Ops and Dev personnel easier. Transparent and frequent communication on progress against the roadmap and its impact is key.

    In order to keep the momentum avoid getting bogged down in a particular step in the roadmap, especially early in the process. Most enterprises present target rich environments and a flexible roadmap focused on ensuring demonstrable rapid progress will ensure continuing focus and commitment.

    DevOps relies on a sense of shared responsibility and mutual respect between Dev, Ops and the business. Organizations can encourage and engender this by instituting joint ownership for a common set of objectives that encompass both ‘change velocity’ metrics and ‘quality of service’ metrics.

    Automation with discipline

    DevOps relies on the “automate everything” mantra. However, its important to realize that automation does not simply mean speeding things up because machines are faster than people. Automation is a means to codify repeatable practices and steps to ensure speed with reliability.

    Automating poor practices can actually be counter-productive and hence it is first necessary that the various steps and procedures in the delivery process be rationalized, optimized and made consistent.

    The key to improving velocity is to minimize variability in the process of deployment through the path to production. It’s therefore important to automate the complete configuration and deployment process in the lower environments (dev, test) first, and then clearly track and minimize changes to these in higher environments (stage, production).

    In addition to speeding things up, automation can also serve as a powerful way to ensure adherence to process, and done right can substantially help compliance and audits by ensuring change is made quickly yet consistently and is inherently documented without overhead. It is also important that automation of both the rollout as well as verification go hand in hand.

    Tools and infrastructure

    Easy to adopt tools in a frictionless infrastructure is a critical component of DevOps. Rather than locking in a toolset up front, its important to have to have flexibility as the organization evolves its practices and the tools that enable them. Open source tools are well suited to this. They incorporate new innovation faster, provide the ability to tailor them to meet specific needs, and developers can leverage the community that typically exists around these for best practices and knowhow.

    It is also a good idea to optimize and fine tune the tool chain for the project or program that is the initial focus. Once a degree of repeatable success has been established, and the team has acquired proficiency, the tool chain can be scaled out to more broadly. It’s always harder to adopt a new tool chain, especially when coupled with a change in behavior and approach, if the scope of it is too wide initially. As discussed earlier, establishing a DevOps center of excellence team that owns and evolves the tool chain and its usage is often a good practice for enterprises.

    Summary

    As more enterprises rely on software to drive revenue, the pressure is on to innovate and deliver value faster. While this is indeed exciting, it requires a significant undertaking in rolling out cultural, organizational and methodology changes needed to enable DevOps practices. Ensuring clear alignment and goals for all stakeholders, a common sense of purpose and an achievable yet ambitious roadmap are key to securing organizational commitment for this endeavor. Approaching automation in a complete way and enabling the adoption of the right tool chain can set enterprises on the right path to realize benefits of small, frequent deployments without compromising quality of service and compliance.


     

    About the Author

    KartikRaghavan

    Kartik Raghavan is the Senior Vice President of World-wide Engineering, Operations and Support at CollabNet where he brings deep experience in ideating, defining, developing and launching products through On Premise and SaaS channels.