Category: DevOps Toolbox

  • Building an agile data tier improves devops efficiences

    Building an agile data tier improves devops efficiences

    Meet the modern day agile data tier – improving DevOp efficiencies
    DevOps teams are constantly in the high-pressure environment of reacting to real-time market and industry changes and launching new features and functionality to maintain competitive positioning. Accelerating development is simply not possible if scalability is an issue. Scaling databases has proven to be especially challenging. Software rollout cycles can easily get delayed because of the complexity of coding for database scale out. At the same time, a lack of understanding and visibility of app behavior can also result in slow apps getting launched. Without intense technical development, databases can be the function hindering your apps – but they don’t have to be.

    DevOps: Resist the urge to do-it-yourself
    As pressures mount to deliver more and more applications, DevOps teams can benefit from creating an agile data tier that seamlessly addresses the most technical, yet burdensome, administrative functions of databases and lets software do the hard work instead. It is now possible for DevOps teams to offload database scalability and failover without increasing app complexity and enable a zero downtime experience along the way.

    Database load balancing software enables this agile data tier. It acts as an abstraction layer between apps and databases, getting developers out of the business of programming for the database and letting them instead spend their time on more strategic projects. Having an agile data tier profoundly reduces development time, taking dev cycles from months to weeks.

    The last thing any developer wants is for an app to become more complex and burdensome – both are big risks when dealing with databases. But there is more to database load balancing then other kinds of load balancing. Replication-aware load balancing, read/write split and caching functionality are all crucial for database scale out. But designing those functions directly into the app is time consuming and repetitive. Designing for high availability just adds to the complications.

    With database load balancing the idea is to take this burden off of the app – and therefore off the developer – resulting in faster rollout, better scalability, and the ability to refocus time and attention to features and functionality that contribute to the business. What developer woke up this morning craving to spend a good chunk of their time on programming for database scale out?

    Making applications fundamentally less complex and improving their availability, database abstraction solutions that provide database load balancing such as ScaleArc, the company I founded deliver on the promise of scale and always-on app availability. Such software makes application design far simpler, leaving more time to develop and build applications that drive business advantage.

    About the Author/Varun Singh

    Varun-SinghAs Founder and Chief Technology Officer, Varun Singh leads ScaleArc’s product development and technology strategy. He also served as ScaleArc’s CEO from March 2009 to September 2013 and was responsible for impressive company growth and customer acquisition of top brands including Tesla Motors and Zuililly with limited if non existant marketing or sales budgets. Varun is a technologist with broad experience in several fiends, from running Linux-based BBS sytems in the pre-Internet days to writing about technology and open source software at PC Quest, CHIP, Digit and Network Computing. He was responsible for creating India’s top online technology brands and his team kept CNN India running as the only uninterrupted news network during the Mumbai terrorist attacks.

  • Enough monitoring. Act!

    Enough monitoring. Act!

    Or How I learned to stop worrying and love the auto-remediation

    These days StackStorm – and other approaches to auto-remediation and event driven automation more generally – are getting broadly adopted.

    Everyone from Facebook through Netflix and WebEx (yeah users!) are not just better “managing” their 2am pages (although that’s cool too PagerDuty) – they are exterminating them.  

    And as they adopt StackStorm for remediation inevitably a discussion emerges about monitoring vs. fixing.  Why?  

    At some level it is because of the excellent work that has been done recently in response to monitoring sucks.  Thanks Sensu, thanks SignalFX, NewRelic, DataDog, AppDynamic, Splunk, the entire ELK stack, and the other >150 monitoring solutions I’ve left out.  

    But at another level it is because of OODA tells us that all that observing and orienting is only as good as the decisions and actions you take.   

    OODA

    The thing is, even though folks gradually are coming to realize that good event-driven automation INCLUDES the D in it – and does not purely rely on the smarts of monitoring since with event driven automation it is possible to run a routine that validates events before acting upon them, most people still miss the fundamental insight of ODDA.  And it is an insight that is crucial especially if you, like those users above and many folks in the world of active cyber-defense, are going hand to hand with smart actors and otherwise are in a world in which what goes wrong is changing.  

    Here is that perhaps non-intuitive insight that, again, I’ve learned from folks that live the OODA against tricky actors (human and the entropy of scale and creeping complexity) everyday:  

    Make your false positives and the creation and control of your actions cheap and you can move fast to fix, way before a root cause analysis (RCA) would typically have completed.  

    Some of the tips that even the wikipedia on ODDA repeats for actual war fighters utilizing the ODDA is “embrace the chaos” – don’t wait to know what is happening before your act because the pace of your actions is itself a key determinate in whether you triumph or not.  

    One way to think of it is that you have a budget from O through A and the total of those steps is your Mean Time To Recover (MTTR).  

    If you spend all of your MTTR budget on O and O then your D and A will be too little, too late.

    Returning to the world of infrastructure and DevOps – here are some drivers pushing us all to focus more on the D and the A and to obsess less over the O and the other O:

    • Docker and containers and cloud itself.  Now compute can be ephemeral (sometimes).  Same deal with lots else that can be software defined.  Even storage! (hi Nexenta ;))
    • Imperative stuff exposed via APIs and declarative systems.  Yes, this is a StackStorm plug. U R surprise?  Check it – when Netflix knows that there is a happy API there that will do the care and feeding of their Cassandra reliably even when AWS is doing an AWS entropy dance, they can obsess slightly less over RCA and more over making sure StackStorm has the resources it needs to fix and fix and fix and fix again before any of us start noticing Netflix acting strange.  (You want to see how Netflix auto-remediates their Cassandra environment?  Hands on tutorial with code samples here. )
    • Infrastructure as code.  Without infrastructure as code the cost of changing the integrations and automations that live inside of event driven automation can be astronomical.  In the world of legacy run book, needing to tweak an automation to respond to RIAK burps instead of Cassandra burps is a bunch of perfectly performed mouse clicks (intentional irony) OR a nice set of proprietary code.  In our world – all the glue, the rules, and the workflows are themselves just code.  
    • And – sorry to extend the point but this is crucial – the code is YAML or maybe just whatever scripts you have lying around or maybe the configuration management system you have already deployed.  A system like StackStorm can start with the bunch of scripts you already have – those that, along with a lot of human activity comprise your D and A – and ingest them. (yum, yum, Java, Python, Bash and Ruby)   And then those scripts, those configuration management systems, all of it plus your monitoring and more are there for you to tie into a system that automates the entire OODA loop.  And then the good stuff happens because you have incredibly fast MTTR (and even proactive MTTR, preventing the outage before it occurs).  
    • Workflow.  With workflow you can see what is happening.  Especially if exposed via FLOW (another pitch, I know).  But not just FLOW and not just StackStorm.  The return of workflows is real and widespread.  Read a good overview of many of the projects published this Spring on devopsy.kinsta.cloud here.  
    • Feedback loop  Maybe goes without saying – but once you automate your entire OODA and have it as code – you can iterate better on it.  And that’s especially true if you take every single transaction from O through A and log it.  How could you do it? Don’t worry, works out of the box with StackStorm.  Or just set up a unique execution ID on every event and pass that through to each interpretation of the event via your rules engine and then keep that ID as you get the multiple workers in your highly resilient workflow system to start executing and then use that ID as well when doing post-hoc reporting; and don’t forget the APIs, GUIs and CLI as well.  Or just grab StackStorm – it is open source after all.

    In conclusion – OODA is a great frame of reference.   However we in DevOps seem to be overly obsessed with the O and the O whereas those that first articulated OODA – those folks using OODA in combat – focus at least as much on D and A.  

    To me this not only implies do the D and A faster and better (via automation) – it also implies that we may not need yet another monitoring project.  The world can stop reinventing monitoring for now, we have 157 projects at least.

    Instead – let’s turn to figuring out how to lower the cost of false positives and the cost of automating ever changing environments – so we can iterate faster and faster – staying farther ahead of the entropy monster and the real monsters trying to steal and corrupt what we build.  

    If you like this sort of discussion and want to learn how Facebook, WebEx, LinkedIn, and coming up in a few weeks – Netflix is doing it – you may want to join the Event Driven Automation Meet-up which I help organize with my friend Brian Sherwin, LinkedIn SRE and author of the remediation system Nurse.  Netflix has been kind enough to agree to host the next one at their new offices nearby to their old offices in the South Bay here in the Bay Area.  Even if you don’t think you can make it, we try to share content via this meet-up and whenever possible we also stream the sessions.  

    Also, at the risk of being “too vendory”, StackStorm is the leading open source project and company dedicated to event driven automation and specifically auto-remediation – we stormers and the broader StackStorm community are here to work with you, to support you, to get you good code and to pick your brain and to ask for your PRs, feedback, integrations and automations (aka packs).  So grab some code, join our Slack based community, and let’s get going.  

  • To Be Continuous: Shipping Velocity and Team Structure

    To Be Continuous: Shipping Velocity and Team Structure

    In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk about shipping velocity and it’s affect on team structure.

    Edith Harbaugh, CEO LaunchDarkly

    Edith is CEO and cofounder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

     

    Paul Biggar, CEO CircleCIPaul is the CEO and founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

     


    [soundcloud url=”https://api.soundcloud.com/tracks/226489524?secret_token=s-Xkrrb” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]


    Paul: Okay, so one of the things we wanted to talk about this week was the idea of shipping velocity and how it’s affected by team structure, sort of miscellaneous factors that aren’t part of continuous delivery.

    Edith: Yeah, I mean, can you explain a little bit more about what you mean by velocity first?

    Paul: Sure, so velocity is a sort of nebulous term that they use a lot in product management and that sort of thing to indicate how quickly, or the sort of through-push with which engineers and product teams are able to put through code.

    One thing that isn’t particularly clear, and there’s this amazing blog post by a guy called Phil Calcado who worked at SoundCloud, and he talked about the idea that even though that they had a very high velocity, they weren’t able to ship features very, very quickly, so often it would take 60 days to ship a feature even though no one got blocked.

    The idea of shipping quickly, is a combination of velocity and latency, and it depends on your kind of business goals, whether you need to ship correctly or you just need to keep the through-put up.

    Edith: Well, do you think part of the issue with SoundCloud might have been that they were trying to do too large features? I mean, that’s somebody’s first reaction when they hear it takes 60 days.

    Paul: So in the blog post what he said is first someone would spec out something, and then they’d put onto the next person’s queue, and that would be maybe a designer who’d drop wire frames, and then the wire frames would end up on an engineer’s queue to implement it, and in between someone doing work and someone else doing work, there would be, sometimes, you know, 10 or 11 days of just waiting for it to get to the front of that person’s backlog.

    So it wasn’t so much, you know, are you biting off more than you can chew, it’s just the process wasn’t particularly set up to be able to ship things very quickly.

    When he got in there, the first thing he did is he said, “Oh, you’re not doing continuous delivery,” or “You aren’t doing continuous deployment,” so he turned on continuous deployment and got that set up, and it was a 66-day process before he finished, and when he got continuous deployment set up it was a 65-day process.

    Edith: So all that work to save a single day.

    Paul: Right. Well, it was more about, the discussion was about how it wasn’t really about, continuous delivery. It wasn’t really about the time that people were spending on certain things, the vast majority of it was the time that was spent waiting on it.

    Edith: Yeah, that takes me back a lot to the whole idea of just in time manufacturing and lean.

    The whole idea of lean was to reduce waste, and one area of waste is time.

    Like if you have people on a factory line who are waiting for the next part, or you know, stuff queuing up, then you’re being very wasteful.

    Paul: I think this is how people, I think that that’s the thing that the backlog is intended to fix, that you’re never waiting for things because there’s always something in the backlog that needs to get done.

    But what isn’t considered as part of that is something is sitting in the backlog, for five days, 10 days, two weeks or whatever waiting for someone to address it, then the speed with which you can get a feature out increases dramatically, or decreases dramatically, I guess.

    Edith: Well, that’s the whole idea behind the Toyota factory line is that they, the just in time delivery meant that there was always a part right then, not before and not after.

    Paul: Right, okay. But right when they’re ready. But you couldn’t say, “I need a car now.” I think it’s not really optimized around the idea of how quickly something goes from one end to the other.

    It’s more optimized around how quick, or how much, how little waste that you can have in the system itself, which means, in this case, how much developer, engineer, or designer engineer and project manager time is being spent doing nothing.

    And I think you can certainly reduce that time to very, very low without necessarily optimizing the speed that which it takes to assemble a car, which is something that you need, really, to get to market quickly, and that sort of thing.

    Edith: Yeah, and it’s a trade off. You know, I went to a talk that the former CTO of Yammer gave, Adam Pisoni, and he talked about how on projects they always had an extra engineer. Like if they thought a project would take four engineers, they would actually put five on it.

    Paul: Right, right.

    Edith: For precisely that reason, that they never wanted anybody to be sitting around blocked. So they would rather take the cost of an extra engineer, just so there was always somebody to do the next task.

    Paul: Right. And so there’s an interesting thing about how teams are structured in that. In the Yammer example, they have teams, and teams do a project, or do a particular feature, and I presume that they’re cross-functional, there’s a couple of designers, couple of engineers, whatever.

    A different way to structure it is to have teams that are aligned on a particular, by what they do, and you have the engineers who are sitting at the front of the front-end queue, or who work on the front-end rather than work on a particular project which has components of front-end and back-end.

    And you end up with a very different structure, or it’s obviously a very different structure, but you end up with a different ability to ship, depending on how you’re, depending on what you’re trying to do, and what’s the most important thing for your organization, also how your organization is staffed.

    So the backlog works very well if you’re short-staffed in any particular area, and so you spend a lot of time prioritizing so that the people, maybe you’re short on design time, so that the design time is spent really, really well, but it doesn’t necessarily allow you to turn around a feature from start to finish and get it out to your customers incredibly quickly.

    Edith: Yeah, and that’s a decision that different organizations make. Some organizations say, “We’d rather have 100% utilization, even if it means our features take longer.”

    Paul: Right. My feeling is that most organizations don’t actually make that decision consciously.

    Edith: It’s implicit.

    Paul: Right. Oh. So, what do you mean by that?

    Edith: Oh, well so I worked at a lot of big enterprises, and my saying was always,

    The lack of a decision sometimes is a decision.

    Like, we’re not going hire an extra designer to unblock a lot of people, so therefore our feature’s are always just gonna be gated by that.

    Paul: So when you say it’s implicit, you mean that, I guess you kinda mean the same thing as I do, that people aren’t really thinking about the advantages or the disadvantages of the structure of particular teams.

    Edith: Yeah, that people are just kinda, they say, “Oh, everybody looks busy,  and why does a feature take 65 days?”

    Paul: So one of the structures that I really liked, there was a talk at Heavybit, and I think it’s on the Heavybit website by Peter van Hardenberg from Heroku.

    Edith: Oh, he’s so great.

    Paul: So in the talk he talked about that at Heroku they have product teams that are around specific areas. And the product teams have two engineers and a PM, and maybe a designer and that’s kind of roughly it.

    Teams have like two to five people, and they’re focused on a particular area, so they do the prioritization within their areas, they do the support and the ops within their specific features, and then they’re able to trade-off, or they’re very, very close to the problem.

    They have then project managers who are working to protect them from other areas, or bleeding into other areas, but also working to solve their particular problems, rather than necessarily one backlog for the entire, maybe the entire platform or the entire web, or something along those lines.

    Edith: So basically, little small start-ups within a bigger company.

    Paul: Yes, and I think actually that’s how they structured it at the start. I remember them saying that there was a, they would actually have presentations to the board, and that each of the teams was structured, the project manager was like a CEO, and they actually like gave presentations to the board, and I think that didn’t necessarily last very long, but I think that was the initial idea behind it.

    Edith: Yeah, we did something very similar at TripIt. We had a, you know, we would have a new project we wanted to do. We were trying to do TripIt offers, and we had a project team just for that, and that’s what they did.

    And the idea was that they could focus. I’ve also heard of people who do stuff completely different. Like I talked to a guy who ran an agile shop and every project had to be a week long, and nobody had any ownership whatsoever.

    So he set it at, and not even like ownership in the way we would think of it. He said our front-end people were back-end, people were mobile, also.

    Thank you for listening, the entire Heavybit podcast can be found here.

  • Building a compelling business case for continuous delivery

    Building a compelling business case for continuous delivery

    Companies are striving to deliver better customer experiences by delivering higher quality software, faster. In order to do so, many organizations have come to realize that the traditional approach to IT management is insufficient. Development and Operations teams can no longer work in departmental siloes; they must work collaboratively towards shared goals if they are to serve the needs of the business effectively. The business too must learn to communicate feedback and work with Development and Operations teams to close the loop on what features their customers most need. With methodologies like Continuous Delivery and DevOps, service design and delivery teams work together to continuously improve services based on their performance, enabling new services to be brought to market much faster. However, switching to a DevOps model is not like flicking on a light switch; it takes time and careful strategic planning.

    How do you identify opportunities for improvement in your value delivery stream? Here are some places to start:

    • Infrastructure-as-Code – where there is standard hardware, virtual machine configurations, version controlled configurations and auditable, repeatable automated code changes.

    • Automated Testing – going from labor-intensive manual testing to fast, scalable and repeatable testing

    • To use manufacturing parlance, maximize throughput – take the typical worker utilization scenario where everyone is busy all the time. There is also more work in progress (and more waiting and waste) as opposed to maximizing throughput where people may be less busy but all are focused on the same goal and more useful work is getting done.

    The logic is easy to follow when you consider that shorter development cycles result in faster feedback loops, which in turn reduce waste and cost. But in the real world, IT budgets remain conservative and any new expenditure, whether it be on tools, equipment or a new approach to development, requires business justification. Successful business cases for Continuous Delivery improvements span Development, Operations, and the Business.

    Tune in Tuesday, October 6th at 1pm to hear from Kurt Bittner, Principal Analyst at Forrester and Andrew Philips of XebiaLabs for a live webinar that will help you understand how to create a successful business case for Continuous Delivery, the potential return of investment, how to measure the benefits and how to track these benefits over time. The webinar will highlight:

    • How methodologies like Continuous Delivery and DevOps can reduce the cost of compliance and the cost associated with security risks
    • How to estimate the value of these improvements that reduce cycle time by removing bottlenecks and barriers to delivery
    • How to estimate the value Continuous Delivery creates by growing or accelerating revenues
    • Real life examples of the benefits organizations have achieved through Continuous Delivery

     

    About the Author/Heather Moses, Vice President of Marketing

    HeatherMosesHeather heads marketing for XebiaLabs. In this role, Heather drives global marketing strategy and operations for the Company. As part of the senior leadership team, Heather is responsible for market direction, brand evolution and market outreach best practices.

  • Re-thinking DevOps tools for Continuous Deployment success

    Re-thinking DevOps tools for Continuous Deployment success

    Gartner predicts that 25% of Global 2000 companies will be actively implementing DevOps methodologies in their organizations by 2016. As DevOps goes mainstream, it may be time to re-think some of the implementation techniques that have worked in small companies and in non-production uses. The common assumptions that are often applied to make a continuous integration process work are generally not valid in production real-world datacenters. Common assumptions include:

    • Dedicated resources are available for separate dev and test environments
    • Pools of virtual machines are sufficient for executing regression and QA tests
    • Testing environments mimic production environments
    • Tool chaining is sufficient to implement DevOps workflows

    Let’s consider each of the above assumptions and how they impact success in real production environments.

    Dedicated resources for separate dev and test environments

    In any large organization, it is prohibitively expensive to maintain separate and dedicated infrastructure for development and test environments. In our experience, any organization with more than five or so development teams will need to have a better way to manage and share development, test and QA infrastructure. This means that they will not be able to just trigger regression tests directly from build completions. They will need to first ensure that they have access to the resources needed for test execution. We believe that DevOps tools need to also incorporate reservation and scheduling capability for shared and constrained physical and virtual infrastructure.

    Pools of virtual machines for sufficient for executing tests

    DevOps tools are mainly focused on automating the process of developing, testing and running applications in homogeneous pools of virtual or physical servers. In these environments, it is possible for the DevOps tool to depend on the virtualization environment, PaaS tool, or OS for the server pool (such as CoreOS) to provide execution and deployment. Unfortunately, delegating the scheduling (or lack thereof) to the virtualization or OS tool does not work in large production environments and is certainly not going to provide enough control for regression testing that needs to mimic real world datacenter infrastructure. This can become a severe limitation of DevOps tools. We believe that a more complete DevOps toolset must incorporate workflow orchestration that allows the developers to have control over how applications are executed onto the target virtual environment in each step of the DevOps process.

    Testing environments mimic production environments

    Global 2000 datacenters are not homogeneous. They are typically hybrid infrastructures where applications are deployed across a combination of traditional, virtual and public cloud computing. In addition, production network topologies are typically complex and require configuration to support applications. Ignoring this complexity in development and testing will result in applications that fail to transition to production successfully. Most continuous integration tools are not capable of controlling deployment into this more complex environment. This explains why most DevOps projects today stop at “continuous integration” and never get all the way to “continuous deployment” into production. DevOps tools need to have the capability of controlling not just the deployment of applications, but the configuration of the hybrid infrastructure on which those applications will run – and this needs to result in test infrastructure configurations that really do mimic production infrastructure.

    Tool chaining is sufficient to implement DevOps workflows

    The prevalent solution to a continuous deployment process is to chain DevOps tools together end to end from development to deployment. This creates problems because each tool must have a point to point integration with the tools on either side of it in the process flow. By the time an IT shop strings them all together, there is often no complete solution. Eventually they will find that the next tool in the chain has only one (or zero) options because it doesn’t have the specific integrations needed for all of the tools already selected.

    More critically though, simply chaining tools together ignores the three key problems described above. A true end-to-end solution should not just chain development, test, QA and deployment tools together, but should also incorporate:

    • Reservation and scheduling of resources prior to initiating the next step in the DevOps process;
    • Workflow orchestration capability that controls placement of applications onto an appropriate mix of virtual or physical servers; and
    • Automation capability for configuring the target infrastructure and network topology before running applications.

    We believe that we will see customers augmenting their existing DevOps tools as they move into full continuous deployment, with additional software that can provide each of the above capabilities.

    About the Author/Joan Wrabetz

    Joan's image croppedJoan Wrabetz is the CTO for QualiSystems. Most recently, she was theVP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.

  • Adding users to Google Apps quickly

    Adding users to Google Apps quickly

    With over 5 million organizations using Google Apps (yes it’s that many), the market is demanding enterprise-class solutions for administration and management of the Google productivity suite. Yeah sure you can have a decent word processor with with collaboration features, even a spreadsheet and presentation program. But if you want to play with the big boys you need management and administration built around it.  Only then can Google Apps really rival MS Office for the mainstream market.

    Beyond that though, Google Apps is only part of the story. There is the whole “Google Infrastructure”.  One of Google’s aims seems to be to bring Google like capability to IT teams the world over. This is one of the aims with Kubernetes, as well as CoreOS Tectonic (see our article in ContainerJournal on this).

    So what is holding you back?  There are 5 areas that you need to focus on according to a recent ebook by JumpCloud:

    1. SECURITY AND COMPLIANCE
    2. SSO
    3. DEVICES AND SERVERS
    4. WIFI
    5. DIRECTORY SERVICES

    The directory services can be added to the mix via integration with JumpCloud’s Directory-as-a-Service (DaaS). This gives Google Apps an equivalent to Microsoft’s AD except in the cloud and available as a service. Think about it, you don’t just set up a user to use Google Apps, you want them to use your entire infrastructure where appropriate. Also you don’t want to be setting up users one at a time in an enterprise setting. Both of these use cases are handled by a directory-as-a-service solution like JumpClouds’. 

    Two keys to effective directory management using Google infrastructure are authentication protocols and device management. Directory-as-a-Service can deal with both of these issues if done correctly. Once you have bridged this gap, it opens up a wide range of uses and interactions with the rest of your Google infrastructure, other infrastructure, your users and your devices.

    The book by JumpCloud featuring devopsy.kinsta.cloud’s own Ericka Chickowski does a great job of explaining this along with how to overcome some of the security issues that using Google infrastructure can present. You can download the book below.

    jumpcloud ebook
    There is also a good infographic with some metrics around this topic here.

     

  • How to prepare for a successful data migration

    How to prepare for a successful data migration

    It can be tough to find useful, concrete advice to assist you with an enterprise-level data migration project. There’s an awful lot that can go wrong. Common suggestions about finding the right people, minimizing risk, or employing an automated system that does everything for you, offer limited help.

    What you really need is a step-by-step checklist that you can work through.

    The success or failure of a data migration depends largely on how thoroughly and carefully you prepare for it. If we analyze large migration projects, we invariably find that the successes share many common characteristics. If you’re planning a data migration, these are the questions that you really need to answer:

    What type of data is being migrated?

    Not all data is created equal. There are various different types of data, so you need to be clear about exactly what you want to move.

    • NAS-based – Where network shares are migrated from one system to another.
    • LUN-based – Where the disks (LUNs) are migrated from one place to another, at block level.
    • VM-based – Where the entire virtual hosts are migrated from one hypervisor to another.
    • Application-specific – Where one set of databases or objects, is migrated from one host to another.

    What type of migration is it?

    Where your data is going, can make a big difference to your overall plan. Is this a local migration, within the same data center, a remote migration, shifting data from one center to another, or is it a cloud migration, where you might be moving data from one cloud provider to another?

    What is the current storage environment?

    You need a complete overview of your current storage environment before you move a single file. If you’re using SAN, then think about what hosts, in terms of operating systems and applications, are involved. What are the FC switches and RAIDs? Do you have a clear picture of all the inter-dependencies within the SAN? A deep understanding of the environment and its complexity, gives you a better chance of avoiding disaster.

    How much data is being migrated?

    An accurate estimate of the volume of data that needs to be moved is absolutely vital. You need to account for all the hosts, LUNs, and storage systems. It’s not enough to say how many terabytes of data need to be migrated, you’ll need a picture of how that breaks down before you can determine the time and effort required.

    How much time do you have?

    There’s always a deadline for any project. Your timeframe may ultimately be dictated by the volume of data and the migration rate, but you can only determine that rate with a complete understanding of the data, application, and environment. Can you migrate in a divide-and-conquer manner? The more you know, the easier it is to control operational complexity and reduce costs.

    How active is the data and how much downtime is possible?

    In order to determine whether the migration can be completed on time, or even at all, you need to know how active the data is. How much of an impact on production will be tolerated? Some downtime is inevitable, but it has to be worked out and agreed upon ahead of time. Creating the right strategy in advance, with all the key stakeholders, will dramatically boost your chances of success, and reduce stress levels on the day.

    What’s the right tool for the job?

    This is really going to depend on all of your answers so far. The right way to go about selecting is to draw up a list of evaluation criteria. Think about everything you need and create a list that you can cross-reference against the candidates.

    As an example, for a SAN migration, you might want something that offers the following:

    • A seamless installation with no disruption to the production environment.
    • A clear display for the automatically discovered SAN configuration, complete with identities, status, and I/O statistics for all hosts and LUNs to be migrated (as well as those not to be touched), so the migration operation can be organized correctly and accurately with high confidence.
    • The ability to migrate during non-production hours, or automatically yield to production traffic, with adjustable intensity/throttling.
    • A centralized reporting system that delivers the statistics you need.
    • Data integrity checking of intermediate images before cutover, so issues are flagged early before they can cause problems.
    • Secure wiping of the data disks after the data is successfully migrated out.

    When you’ve drawn up a list, you can apply it, and see how different options compare.

    Who are the right people for the job?

    A successful data migration is going to require some level of expertise. Ideally, you want experienced personnel to handle a process like this, and you may need to look externally to find them. If they can give you the answers you need to address the questions we’ve posed here in this article, then you know they’re qualified for the job.

    About the Author/Wai T. Lam

    CTOWaiLamCirrus

    Wai T. Lam is co-founder and CTO of Cirrus Data Solutions, a developer of Data Migration Server and Data Caching Server for storage area networks (SANs). He was previously CTO and VP of Engineering at FalconStor, a company he co-founded in 2000. There, he was the chief architect, holding 18 of 21 company patents. Wai received the prestigious China national “Top 1000 Technological Leaders” award in 2013.

  • Open Source software license and security management with WhiteSource

    Open Source software license and security management with WhiteSource

    With the growing speed and availability of open source components, it becomes easy to add features and integration of software with other components which makes software development easier. But there are a few points to be remembered while using any open source component.

    • Security vulnerabilities
    • Licensing risks of open source component.
    • Outdated open source components.

    The above are a very important part of the the software development life cycle to eliminate any discrepancies related to security or legal issues.

    WhiteSource provides one fine platform to solve such issues without putting in much effort by developers, They can concentrate on core development instead of spending time on finding these issues which can be easily handled by WhiteSource.

    WhiteSource is an open source management solution which does

    • Open source licensing and compliance management
    • Open Source security vulnerabilities alerts and management
    • Executive dashboards, policy enforcement, and reporting

    WhiteSource checks your software and generates open source inventory report, including detail from open source. The WhiteSource gathers information on open source components and they keep the inventory updated.

    WhiteSource covers almost all commonly used languages and provide detail report, This tool can track all your open source components used knowing or unknowingly within your software.

    It sends alerts for any potential issue observed in open source component used by software. It does check for outdated/expired component, It provides detail on security issues found in any opensource component.

    WhiteSource does not store (keep track of) any software component which is not open source, this means it’s safe and your code will not be touched.

    CI tool Integration (Jenkins)

    The best part of the tool is that it has plugin available for the widely used CI tool – Jenkins.

    WhiteSource Jenkins plugin is the best and easy way to integrate with WhiteSource to run the checks during build and integration phase in an automatic way.

    Its very easy to use, not even just for Maven project also for free style projects.

    Easy to configure and use within project. Only token needs to be added and plugin automatically takes care of everything.

    Multiple options are available at job level to define the project and modules to be included.

    Logs are pretty descriptive when plugin start processing in Jenkins job.

    WhiteSource basic principle talks about

    Managing the component vs governing the component.

    Developers can leave the managing part up to WhiteSource and only concentrate on the issue of governance at the component level in software.

  • Continuous test monitoring DevOps ”health-beat”

    Continuous test monitoring DevOps ”health-beat”

    In my prior blog Test Results Analysis at the Speed of DevOps I discussed ways that test analysis features and practices can be optimized to ensure that test results for each CI/CT cycle keep pace with accelerated continuous testing. Beyond optimizing continuous test analysis for each cycle, longer term continuous test monitoring is necessary to assure overall delivery confidence for each release and longer term product performance trend in a positive direction. Individual test results or a fixed number of test campaigns can only tell you so much. Some of the more difficult problems to detect and diagnose are those that only occur occasionally or under “corner case conditions”.

    Unless the results of testing and test analysis are continuously monitored and results aggregated over multiple test and release cycles then there is no way to build sufficient sustainable confidence in the long term health of the DevOps pipeline or products produced by the pipeline. The combination of continuous test monitoring tools with continuous testing and analysis tools can provide a longer term strategic view of test results that is necessary to collect, aggregate and organize test results data to gain confidence in the product for each release and its evolution over multiple releases. Below are some suggestions in a checklist format that have proven useful for continuous test monitoring for sustainable high performance DevOps environments.

    1. Determine continuous test monitoring priorities: Some examples of problems that continuous test monitoring can help with include intermittent failures caused by marginal designs, marginal test designs, environmental condition changes not detected by individual tests, memory leaks, varying starting conditions, interactions with other systems, system topologies, and performance degradation within the margin of a test but accumulate over time. The best practice for continuous monitoring indicates that the problems of most concern to a specific product or DevOps environment will be monitored.
    2. Regression test product areas even though there were no expected changes: One of the practices in high performance DevOps environments is to skip tests for areas that are “known” not to have changed during individual CI/CT cycles. Unintended consequences of indirect changes may impact performance so a regression suite should audit these areas occasionally just to be sure. Typical examples that are often caught by this are features important audit functions, system backwards compatibility features and upgrade/downgrade administration features.

    3. Select continuous test monitoring tools that collect and report trends: Tools that can correlate and report test results across multiple dimensions such as test cases, product features, system performance categories, build versions, releases, and functional tags can find intermittent bugs or problem trends. Thresholds and email alerts and dashboards that highlight short term results views from long term results views are especially valuable.

    4. Use continuous test monitoring results to diagnose and resolve problems: Intermittent failures and problems that only become apparent when test results are presented as trends over a long term trend are easier to diagnose when a large data set is accumulated and filtered by most probable cause tags. Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.

    The above is a partial list of suggestions for continuous test monitoring that have been proven to support sustainable DevOps. At Spirent we think testing has a bright future in DevOps. You can hear more about our views on this topic by reviewing our joint webinar conducted together with test management product provider Zephyr at: Overcoming-DevOps Challenges

    What do you think of these suggestions and do you have others that should be mentioned?

  • Tool Development – In-House or Ready made

    Tool Development – In-House or Ready made

    Tool development and adoption

    The current pace of software development cannot manage its speed without the automation.

    Automation is the most important element in order to achieve agility in delivery of the product. Every part of the software development life cycle needs tools to perform their action automatically. These phases could be coding, code review, build & deployment, environment management, resource assignment, notification and many others.

    There could be 2 options, which a team can opt for:

    • Use a third party tool
      • One can select the tools, which are available in market, either open-sourced or free. You can even buy enterprise solution if needed.
    • Build an in-house tool
      • Tool can be built or developed in-house for the consumption as well.

    Lets take look different aspects of choosing options

    • Features

    Each organization and team may have different features for different use cases. Some time it makes more sense to build your own tool just for your need rather adaption of a third-party one for which most of the features are not going to be used.

    • Time for on-boarding

    Time is definitely a big factor here, if one thinks of developing in-house tool and use it then this will definitely take time. If, need of tool is not urgent or critical, in house development option could be considered.

    Third-party tool can be immediately on boarded and there won’t be much delay other then some logistics reason.

    • Effort and Resource

    If the tool is complex, its not easy to build, one will need resources and lots of time to build it and then next challenge is implementation.

    Above effort can be reduced by adaption the tools which is available in the market. There are chances that other users are aware of the tools and training time could be reduced.

    • Cost

    Lets discuss one more important part, which is the cost. Yes, money is also one of the biggest factors on deciding what to opt for. There may be good options to choose from free and open source tools. But if enterprise tools selected, then cost may incur.

    You will need a dedicated team and resources to build the tool with in organization.

    Its hard to decide if in-house tools will be costly or enterprise. Decision has be to taken by considering long term investment and return.

    • Support

    Enterprise tools usually come with a proper support. This is really important factor if tool is very critical. Open source tools may not have enough support, as it is community driven effort. In-house tools may need resource to provide support and training in the team/organization.  Even bug fix needs to be taken care under support category.

    • Adaption/Expertise

    In-House tools are highly customize as per use case and will need proper training for adaption with in organization. In case of thirdparty tool, as I mentioned earlier there are chances that other users are aware of the tools and training time could be reduced.

    So, Adaption of tool is not an easy task. Decision has to be taken with a very strategic thought process.

    If tool is complex then it is definitely not a part time task.

    One more important thing to consider is if the team is trying to build in-house tool, which is going to be major business for somebody else then it has to be developed with proper planning and priorities, otherwise there are high chances of failure.

    Please Note- I am not considering or referring to any tooling like script or some simple UI dashboard.