devopsy.kinsta.cloud

Author: contributor

Analytics from a Developer’s Point-of-View

As a developer, I have found that most systems analytics tools can be both an important asset and an unnecessary expense. Their usefulness, in my experience, is ultimately determined by the size and stage of the company. If the company is at a low funding stage, the cost of some of the more advanced analytics tools just isn’t practical. If they have the funding, but not the traffic, the benefits of analytics are completely negated by the lack of data to… you know… analyze.

The same goes for the team size. Analytics tools can be incredibly useful to any size team, however after a certain level of growth a large Development team will likely split into two separate departments: Development and Operations. At this level, these tools tend to be delegated to Operations, leaving Development free to focus on the whole developing part of the job.

So, your development and operations departments are still on one team (it’s likely both jobs are shared by everyone on the team), and your company has the right amount of funding and traffic to justify the cost and need for a systems analytics tool. But how do you choose from all of the available tools out there?

This is a question that can bring any developer in a leadership role to their knees because the effects can last for years. I’ve always found that tools that can be isolated apart from the core application itself are always the best choice in this situation because they often take a language agnostic approach and, in case you don’t like them, they can be hot-swapped for another service with minimal effort.

Being able to easily undo a poor decision shouldn’t be the only measuring stick you use to make your choice. The type of data you can get, and how accessible it is, are both far more important than being able to hit the reset button. As a developer, when dealing with third party applications, there are three things that I look for: usability, accessibility, and visibility.

Usability

We all know about usability. It’s a popular buzzword that is thrown around a lot in the startup world but, if you ask me, it is doubly important when dealing with full-stack tools built for developers. Full-stack tools are often more in-depth than standard consumer products. Rather than only requiring good UI/UX on the application frontend, a good full-stack platform requires a lot of foresight in the backend as well. There is nothing that turns me off more than API-as-an-afterthought. Systems analytics tools almost always fall under the umbrella of “full-stack tools” because they are designed for developers and non-developers alike.

While the application itself should be easy to navigate and use, it is just as important for the API to be clearly documented and logically architected. The more time I have to spend dealing with integrating something that isn’t directly related to the product, the more likely I am going to cut my losses and pick something that was designed with my needs in mind.

Accessibility

While a systems analytics platform may be managed by just one person initially, as the team grows, it becomes exceedingly impractical to keep the knowledge isolated. The one thing that will cause my to ditch a tool immediately is a lack of multiple account management. While there are some great business tools for password management, I would prefer that everybody on my team has their own login. This is a major security concern for me. If we let somebody go (or they quit), I would rather delete their account than have to change a password and disseminate it to the rest of the team.

Visibility

Having easily viewable data should be a no-brainer for an analytics platform, but you’d be surprised how often I come across tools that do a very poor job at actually doing what they’re supposed to. Showing me pretty graphs and tables isn’t enough, though. I want control of the data. Let me import, export, manipulate, search, and filter it. If the time it takes to get raw information is longer than the time it takes to actually read and understand it, there is something wrong.
Ultimately, the most useful systems analytics tools I’ve ever worked with do an amazing job at getting out of the way. They don’t try to predict how I want to work and they don’t limit my ability to get what I need, they simply give me the tools to do things my way. While this probably clashes with the common million-features developer mindset, in practice, simple can often be significantly more powerful.

About the Author/Zachary Flower

Zachary Flower (@zachflower) is a freelance web developer, writer, and polymath. He has an eye for simplicity and usability, and strives to build products with both the end user, and business goals, in mind. From building projects for the NSA to features for Buffer, Zach has always taken a strong stand against needlessly reinventing the wheel, often advocating for using well established third-party and open source services and solutions to improve the efficiency and reliability of a development project.

October 15, 2015
Building an agile data tier improves devops efficiences

Meet the modern day agile data tier – improving DevOp efficiencies
DevOps teams are constantly in the high-pressure environment of reacting to real-time market and industry changes and launching new features and functionality to maintain competitive positioning. Accelerating development is simply not possible if scalability is an issue. Scaling databases has proven to be especially challenging. Software rollout cycles can easily get delayed because of the complexity of coding for database scale out. At the same time, a lack of understanding and visibility of app behavior can also result in slow apps getting launched. Without intense technical development, databases can be the function hindering your apps – but they don’t have to be.

DevOps: Resist the urge to do-it-yourself
As pressures mount to deliver more and more applications, DevOps teams can benefit from creating an agile data tier that seamlessly addresses the most technical, yet burdensome, administrative functions of databases and lets software do the hard work instead. It is now possible for DevOps teams to offload database scalability and failover without increasing app complexity and enable a zero downtime experience along the way.

Database load balancing software enables this agile data tier. It acts as an abstraction layer between apps and databases, getting developers out of the business of programming for the database and letting them instead spend their time on more strategic projects. Having an agile data tier profoundly reduces development time, taking dev cycles from months to weeks.

The last thing any developer wants is for an app to become more complex and burdensome – both are big risks when dealing with databases. But there is more to database load balancing then other kinds of load balancing. Replication-aware load balancing, read/write split and caching functionality are all crucial for database scale out. But designing those functions directly into the app is time consuming and repetitive. Designing for high availability just adds to the complications.

With database load balancing the idea is to take this burden off of the app – and therefore off the developer – resulting in faster rollout, better scalability, and the ability to refocus time and attention to features and functionality that contribute to the business. What developer woke up this morning craving to spend a good chunk of their time on programming for database scale out?

Making applications fundamentally less complex and improving their availability, database abstraction solutions that provide database load balancing such as ScaleArc, the company I founded deliver on the promise of scale and always-on app availability. Such software makes application design far simpler, leaving more time to develop and build applications that drive business advantage.

About the Author/Varun Singh

As Founder and Chief Technology Officer, Varun Singh leads ScaleArc’s product development and technology strategy. He also served as ScaleArc’s CEO from March 2009 to September 2013 and was responsible for impressive company growth and customer acquisition of top brands including Tesla Motors and Zuililly with limited if non existant marketing or sales budgets. Varun is a technologist with broad experience in several fiends, from running Linux-based BBS sytems in the pre-Internet days to writing about technology and open source software at PC Quest, CHIP, Digit and Network Computing. He was responsible for creating India’s top online technology brands and his team kept CNN India running as the only uninterrupted news network during the Mumbai terrorist attacks.

October 15, 2015
How Fast Is Fast Enough?
One of the primary goal of DevOps is the ability of an organization to bring an idea to market faster. But how fast is fast enough? Do they need to improve time to market by 10%, 20%, 50%..90%? Do all applications need to ultimately be released within hours? In order to compete do we all have to be the Etsy or the Netflix’s of the world?

Unfortunately there is no simple answer. Optimal application delivery speed (aka time to market) varies drastically across industries, markets, organizations, and even within an organization’s own application portfolio. Remember “two speed IT”.

In other words there is no “one fit for all” when it comes to speed. Yet defining your optimal speed – goal – is critical to your DevOps success. Without it, its almost impossible to provide a vision, measure progress, and fundamentally change the way you delivery software. For example, IT is very good at measuring costs. In fact they measure costs from every way to Sunday. As a result, cost takeout becomes the fundamental value metric for service delivery – “how low can you go to delivery the same service”. Similarly, if speed and agility is the goal then we must start with measuring our current velocity and then setting (realistic) goals for the future state.

The Definition of Speed

Before we go about trying to determine an applications optimal speed it is important that we first agree on the definition of speed. We like to think that speed is a fairly simple concept, but its amazing how mis understood it is.

Many organizations new to DevOps simply equate speed to release cycles. They believe that if they increase the number of cycles then they are improving their time to market. Unfortunately it does not work that way. Improvement in release frequency is a natural outcome of going faster (as we will see in the example below) but not the driver of speed.

True cycle time is measured from end to end – when the idea enters the development funnel to the time it is released to the end user. In other words it’s the Work In Progress (WIP) time for that particular piece of code and any improvement in speed is the reduction of this WIP time.

For example, a healthcare client recently shared with us that it takes them just 20 days to develop a medium size enhancement. For them, their cycle time was 20 days per enhancement. But on further inspection we found that it takes 20 days to determine requirements, 10 days for setting up environments (while using cloud), another 15 days for testing, 10 days for moving code from one environment to another, and a whopping 15 days for approvals. When we added all that up, we discovered that their true time to market (speed) for that enhancement was just over 90 days.

Once we were able to define speed in tsurehis context, it opened up an entire new world of dialogue within the organization. Now instead of optimizing 20 days, they have another 70 days of activities to pick on. Even if they were to shave just a mere 1/3 of the 70 days (cue automation with some simple processes engineering) they would be able to improve their time to market to 65 days. And as an added bonus, the team instead of just delivering 4 enhancements per year (90 days x 4) they can now deliver 5 enhancements (65 days x 5) much faster.

By defining speed in such broad terms, it not only gives IT teams a glimpse at the big picture but also visibility into areas of wastes, opportunities for true end to end improvement, and a platform for collaboration.

Now that we have our definition right, lets get on to answering the million dollar question,

How fast should I be (what is my optimal speed)?

The simple answer is fast enough to delight the customer and beat the competition. But to determine at what point you are able to achieve this goal is a bit more complicated.

Most organizations try to answer this question by looking inward. They perform a detailed value stream mapping exercise, determine cycle time for each activity, identify bottlenecks, and then triangulate how fast they can go if they eliminate some of the key bottleneck.

Unfortunately, speed is not a function of how fast can you go (IT view) but a function of potential benefits (business / market view). More precisely, your optimal speed is directly depended on the additional benefits the business will enjoy (additional revenue, higher customer acquisition, better customer retention etc.) as a result of greater velocity.

To help understand this relationship between benefit and speed better, ask yourself the following three questions,
1. If we were to reduce our cycle time (from idea to production) by 10%, 20%, 30%, 40% etc., what is the incremental benefit to the business
2. Can we quantify this benefit?
3. How sensitive is this benefit to speed? Are they inversely proportionate, is there a benefit / speed plateau?
By doing so, there is a good chance that most applications will fall within one of the three scenarios.
1. Benefit and Speed Are Directly Proportional
For some businesses, speed is king. The faster you can deliver a product the greater the incremental benefit. Consumer facing apps (Facebook, Linked In etc.) focused on revenue generation / customer acquisition work well in this scenario. For these applications, there is essentially no speed limit. The rule of thumb is step on the gas and get out of my way.

Optimal Speed: No speed limit, step on the gas
1. Benefit Improves But Only To A Certain Point
Most SaaS applications (SFDC, Workday etc.) fall within this category. At a certain point the incremental benefit diminishes as speed increases. In other words, users like the regular updates and upgrades but to a certain point. Change too much too fast and you end up delivering little additional value. In this case, there is an optimal speed and any investment in speed after that is a waste of money.

Optimal Speed: Limit your speed to the point where benefits plateau
1. Benefit and Speed Are Adversely / Not Related
This is the opposite of A, where speed has an adverse / no impact on benefit. Here users prize stability and status quo over anything else. Changes are welcomed but need to be few and far between. Many on-prem legacy applications fall into this category. Imagine if your ERP provider published a new update (mini upgrade) every day. It would wreak havoc to the organization. If this happened to your general ledger system, you would never be able to close your books. If you find yourself in this category it is best to go pick a different application.

Optimal Speed: None, stability is prized over changes

Two Caveats

The graphs are a great representation of the various options but there are two caveats that every organization should be aware of, cost and competition.

Cost: The speed limit makes little sense if the cost of getting to that speed is greater than the incremental benefit potential. After all, like any other IT project ROI is key in deciding if DevOps is worth investing in.

Competition: Ultimately…competition holds the trump card. No matter what your charts say and / or your cost model shows, if the competition is going faster…you have no choice but to match that speed or better yet, beat that speed. Which is why we urge many of our clients to invest into DevOps early, set the benchmark, and build a competitive advantage.

Happy speeding!!

About the Author/Mustafa Kapadia

Mustafa Kapadia is Service Line Leader for IBM’s DevOps practice, a business advisory practice focused on helping large enterprises transform their software & application delivery. Through DevOps, organizations can get software to market faster, with better quality, less defects, reduced risk and shortened lead times.

Prior to joining IBM, Mustafa was a management consultant with Deloitte’s Strategy & Operations practice. He lives in San Francisco Bay Area, is an avid blogger (when time permitting), a speaker, and and advisor to start ups.

You can follow Mustafa on LinkedIn: https://www.linkedin.com/in/kapadiamustafa and Twitter: https://twitter.com/mkapadiatweets
October 14, 2015
Ten things you need to know before starting a career in DevOps

So you’re considering a shift to DevOps, Forbes recently reported that highly skilled DevOps specialists with just a high school degree earn a median salary of $106,734. A recent study of IT salaries revealed that 46% of companies plan on hiring DevOps in the coming year.

The DevOps space is red hot but here’s the skills you need, and what you should expect to encounter, in a career in DevOps.

What is DevOps?

DevOps is a culture shift or a movement that encourages great communication and collaboration (aka teamwork) to foster building better-quality software more quickly with more reliability.

It is also an automation process that allows quick, safe and high quality software development and release while keeping all the stakeholders in the loop.

What DevOps is not

DevOps is not a person, a role, or a title. You are not a DevOps engineer, even though you may call yourself one. DevOps is not about using a specific set of tools and it most definitely is not about renaming your Ops team the DevOps team.

Understand continuous delivery

Continuous delivery is an automated process for building software that expedites its release. It allows you to deploy a feature or upgrade rapidly, and with minimum manual intervention. The goal is more frequent, and far less risky small changes, to get dramatically faster and more focused feedback.

The ability to communicate is a must

Sharing and empathy is central in DevOps as it is in life. Knowledge not shared with the team is useless and will just create a bottleneck. Empathy for everyone on the team and their tasks is also crucial. You must have empathy for the challenges that each role brings to create a harmonies workplace.

Bring a wide business understanding

A DevOps specialist must be able to see the big picture in terms of processes, needs, and constraints. You have to be very familiar with all sides; Development, Operations, Sales, Product, etc. and you must know how they all work together to achieve the organizations goals.

Be prepared for late nights

DevOps means becoming ever more agile and responding to a client’s needs in real time. Critical events often occur in large and complicated systems. Hence, don’t be shocked if you get a 2 am call in the case of a crises.

Questions you can expect to be asked in an interview

According to Mike Baukes, if the interviewer wants to make sure you understand the framework you’ll be working with, they may ask “how does HTTP work? How does a web page appear in a browser?” Baukes advises that you may get a question like “How would you prepare for a migration from one platform to another?”, “Tell me about the worst-run/best-run outage you’ve been a part of. What made it bad/well-run?” or “What is the purpose of a post-mortem meeting?” and “Do you know how to learn from mistakes?”

Programs you need to know

Learn tools like Jenkins, Ansible, Chef and Puppe. It is well worth doing it on your own time and with your own dime, hands-on experience with actual tools and projects and an understanding of best practices can be more valuable than a college degree.

What you need to know about operations

If you’re coming from the developer side, obsessively get involved in everything your team does involving operations — deployment, scale, etc. If your team doesn’t do any of that for any number of reason, go over to the operations team and sit in on a few deployments.

Understand the databases role

Your database holds your companies most valuable asset, you must understand how the database and DBAs fit into the DevOps process so that the database does not become the weak link holding back the entire application development.

Ultimately DevOps is knowledge, concepts and best practices. It gives an edge to every developer or operations who practices it over those who don’t. It might not be something to do as a full time job, but any developer with DevOps experience will find themselves at the focal point in any organization striving to improve its processes, and hence position themselves to climb the ladder quicker.

About the Author/Yaniv Yehuda

Yaniv Yehuda is the co-founder and CTO of DBmaestro, an enterprise software development company.

October 7, 2015
Building a compelling business case for continuous delivery

Companies are striving to deliver better customer experiences by delivering higher quality software, faster. In order to do so, many organizations have come to realize that the traditional approach to IT management is insufficient. Development and Operations teams can no longer work in departmental siloes; they must work collaboratively towards shared goals if they are to serve the needs of the business effectively. The business too must learn to communicate feedback and work with Development and Operations teams to close the loop on what features their customers most need. With methodologies like Continuous Delivery and DevOps, service design and delivery teams work together to continuously improve services based on their performance, enabling new services to be brought to market much faster. However, switching to a DevOps model is not like flicking on a light switch; it takes time and careful strategic planning.

How do you identify opportunities for improvement in your value delivery stream? Here are some places to start:

• Infrastructure-as-Code – where there is standard hardware, virtual machine configurations, version controlled configurations and auditable, repeatable automated code changes.

• Automated Testing – going from labor-intensive manual testing to fast, scalable and repeatable testing

• To use manufacturing parlance, maximize throughput – take the typical worker utilization scenario where everyone is busy all the time. There is also more work in progress (and more waiting and waste) as opposed to maximizing throughput where people may be less busy but all are focused on the same goal and more useful work is getting done.

The logic is easy to follow when you consider that shorter development cycles result in faster feedback loops, which in turn reduce waste and cost. But in the real world, IT budgets remain conservative and any new expenditure, whether it be on tools, equipment or a new approach to development, requires business justification. Successful business cases for Continuous Delivery improvements span Development, Operations, and the Business.

Tune in Tuesday, October 6th at 1pm to hear from Kurt Bittner, Principal Analyst at Forrester and Andrew Philips of XebiaLabs for a live webinar that will help you understand how to create a successful business case for Continuous Delivery, the potential return of investment, how to measure the benefits and how to track these benefits over time. The webinar will highlight:

• How methodologies like Continuous Delivery and DevOps can reduce the cost of compliance and the cost associated with security risks
• How to estimate the value of these improvements that reduce cycle time by removing bottlenecks and barriers to delivery
• How to estimate the value Continuous Delivery creates by growing or accelerating revenues
• Real life examples of the benefits organizations have achieved through Continuous Delivery

About the Author/Heather Moses, Vice President of Marketing

Heather heads marketing for XebiaLabs. In this role, Heather drives global marketing strategy and operations for the Company. As part of the senior leadership team, Heather is responsible for market direction, brand evolution and market outreach best practices.

October 5, 2015
Developing your organisation’s collective consciousness
Many organizations across the world are taking a long, hard look at DevOps and wondering how they can start to take advantage of its benefits.

The 2014 State of DevOps report has some pretty compelling facts:
- Companies using DevOps are twice as likely to exceed profitability, market share, and productivity goals
- They will have 50 percent higher market capitalization and growth than rival businesses
DevOps might just be the key to organizational survival in the modern business environment.

Sounds good!

So, where do we start?

Organizations that want to start getting the benefits of DevOps need to recognize their implementation for what it is – organizational change. No, not hard-core bureaucratic change management with millions of forms to fill in, but an initiative that needs to win hearts and minds across the organizational.

Remember – DevOps isn’t just an IT thing, and it isn’t just a business thing. It’s about collaboration, communication and integration right across our organizational.

To be successful with any organizational change, there are a few simple things we need. We need a goal, we need a plan, and we need some enthusiasm. If we can’t get our people enthused and bring them with us, our goal and our plan will not have the momentum they need.

One way to build enthusiasm and a common starting point is through education. It’s really important that every stakeholder understands the goal, the plan, and feels they have the tools, skills and level of empowerment to contribute.

Getting staff the training they need can help build their capabilities and create a common understanding across the organization – a collective consciousness. DevOps is a simple concept, but some of the terminology that comes with it won’t be familiar to everyone. When you start to talk about continuous delivery and continuous deployment, you want to be sure that everyone is on the same page!

DevOps implementations can learn from the way organizations have adopted other frameworks in the past. Common mistakes include a statement of management intent with no plan behind it – saying you’re doing DevOps doesn’t mean you are. Making sure your staff are all speaking a common language can really help to deliver success.

The DevOps Foundation course from the DevOps Institute is a great way to start building that collective consciousness and get your staff involved in your DevOps initiative.

The syllabus has been developed by leaders in the DevOps community and looks at where DevOps has come from, and covers topics including:
- DevOps Culture
- Organizational Considerations
- DevOps Practices (the three ways)
- DevOps and IT Service Management Processes
- Automation Practices and Tool Categories
Once your staff are trained, you can start to have a collective conversation about where you can make improvements in your organization, building on a common foundation.

To get the most from your training, make sure staff know why they are attending the classroom course or taking the e-learning, and then have a plan to build on what they’ve learnt when they are finished.

Read more about DevOps courses and find training providers here: http://devopsinstitute.com/doi-reps/

About the Author/Claire Agutter

Claire Agutter is the lead trainer for ITSM Zone, global e-learning providers for IT management professionals. Claire has more than 15 years of IT management experience and loves anything that helps the business and IT work together to create value.
ITSM Zone are the DevOps Institute’s exclusive e-learning partner.
October 2, 2015
Re-thinking DevOps tools for Continuous Deployment success
Gartner predicts that 25% of Global 2000 companies will be actively implementing DevOps methodologies in their organizations by 2016. As DevOps goes mainstream, it may be time to re-think some of the implementation techniques that have worked in small companies and in non-production uses. The common assumptions that are often applied to make a continuous integration process work are generally not valid in production real-world datacenters. Common assumptions include:
- Dedicated resources are available for separate dev and test environments
- Pools of virtual machines are sufficient for executing regression and QA tests
- Testing environments mimic production environments
- Tool chaining is sufficient to implement DevOps workflows
Let’s consider each of the above assumptions and how they impact success in real production environments.

Dedicated resources for separate dev and test environments

In any large organization, it is prohibitively expensive to maintain separate and dedicated infrastructure for development and test environments. In our experience, any organization with more than five or so development teams will need to have a better way to manage and share development, test and QA infrastructure. This means that they will not be able to just trigger regression tests directly from build completions. They will need to first ensure that they have access to the resources needed for test execution. We believe that DevOps tools need to also incorporate reservation and scheduling capability for shared and constrained physical and virtual infrastructure.

Pools of virtual machines for sufficient for executing tests

DevOps tools are mainly focused on automating the process of developing, testing and running applications in homogeneous pools of virtual or physical servers. In these environments, it is possible for the DevOps tool to depend on the virtualization environment, PaaS tool, or OS for the server pool (such as CoreOS) to provide execution and deployment. Unfortunately, delegating the scheduling (or lack thereof) to the virtualization or OS tool does not work in large production environments and is certainly not going to provide enough control for regression testing that needs to mimic real world datacenter infrastructure. This can become a severe limitation of DevOps tools. We believe that a more complete DevOps toolset must incorporate workflow orchestration that allows the developers to have control over how applications are executed onto the target virtual environment in each step of the DevOps process.

Testing environments mimic production environments

Global 2000 datacenters are not homogeneous. They are typically hybrid infrastructures where applications are deployed across a combination of traditional, virtual and public cloud computing. In addition, production network topologies are typically complex and require configuration to support applications. Ignoring this complexity in development and testing will result in applications that fail to transition to production successfully. Most continuous integration tools are not capable of controlling deployment into this more complex environment. This explains why most DevOps projects today stop at “continuous integration” and never get all the way to “continuous deployment” into production. DevOps tools need to have the capability of controlling not just the deployment of applications, but the configuration of the hybrid infrastructure on which those applications will run – and this needs to result in test infrastructure configurations that really do mimic production infrastructure.

Tool chaining is sufficient to implement DevOps workflows

The prevalent solution to a continuous deployment process is to chain DevOps tools together end to end from development to deployment. This creates problems because each tool must have a point to point integration with the tools on either side of it in the process flow. By the time an IT shop strings them all together, there is often no complete solution. Eventually they will find that the next tool in the chain has only one (or zero) options because it doesn’t have the specific integrations needed for all of the tools already selected.

More critically though, simply chaining tools together ignores the three key problems described above. A true end-to-end solution should not just chain development, test, QA and deployment tools together, but should also incorporate:
- Reservation and scheduling of resources prior to initiating the next step in the DevOps process;
- Workflow orchestration capability that controls placement of applications onto an appropriate mix of virtual or physical servers; and
- Automation capability for configuring the target infrastructure and network topology before running applications.
We believe that we will see customers augmenting their existing DevOps tools as they move into full continuous deployment, with additional software that can provide each of the above capabilities.

About the Author/Joan Wrabetz

Joan Wrabetz is the CTO for QualiSystems. Most recently, she was theVP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.
October 2, 2015
How to prepare for a successful data migration
It can be tough to find useful, concrete advice to assist you with an enterprise-level data migration project. There’s an awful lot that can go wrong. Common suggestions about finding the right people, minimizing risk, or employing an automated system that does everything for you, offer limited help.

What you really need is a step-by-step checklist that you can work through.

The success or failure of a data migration depends largely on how thoroughly and carefully you prepare for it. If we analyze large migration projects, we invariably find that the successes share many common characteristics. If you’re planning a data migration, these are the questions that you really need to answer:

What type of data is being migrated?

Not all data is created equal. There are various different types of data, so you need to be clear about exactly what you want to move.
- NAS-based – Where network shares are migrated from one system to another.
- LUN-based – Where the disks (LUNs) are migrated from one place to another, at block level.
- VM-based – Where the entire virtual hosts are migrated from one hypervisor to another.
- Application-specific – Where one set of databases or objects, is migrated from one host to another.
What type of migration is it?

Where your data is going, can make a big difference to your overall plan. Is this a local migration, within the same data center, a remote migration, shifting data from one center to another, or is it a cloud migration, where you might be moving data from one cloud provider to another?

What is the current storage environment?

You need a complete overview of your current storage environment before you move a single file. If you’re using SAN, then think about what hosts, in terms of operating systems and applications, are involved. What are the FC switches and RAIDs? Do you have a clear picture of all the inter-dependencies within the SAN? A deep understanding of the environment and its complexity, gives you a better chance of avoiding disaster.

How much data is being migrated?

An accurate estimate of the volume of data that needs to be moved is absolutely vital. You need to account for all the hosts, LUNs, and storage systems. It’s not enough to say how many terabytes of data need to be migrated, you’ll need a picture of how that breaks down before you can determine the time and effort required.

How much time do you have?

There’s always a deadline for any project. Your timeframe may ultimately be dictated by the volume of data and the migration rate, but you can only determine that rate with a complete understanding of the data, application, and environment. Can you migrate in a divide-and-conquer manner? The more you know, the easier it is to control operational complexity and reduce costs.

How active is the data and how much downtime is possible?

In order to determine whether the migration can be completed on time, or even at all, you need to know how active the data is. How much of an impact on production will be tolerated? Some downtime is inevitable, but it has to be worked out and agreed upon ahead of time. Creating the right strategy in advance, with all the key stakeholders, will dramatically boost your chances of success, and reduce stress levels on the day.

What’s the right tool for the job?

This is really going to depend on all of your answers so far. The right way to go about selecting is to draw up a list of evaluation criteria. Think about everything you need and create a list that you can cross-reference against the candidates.

As an example, for a SAN migration, you might want something that offers the following:
- A seamless installation with no disruption to the production environment.
- A clear display for the automatically discovered SAN configuration, complete with identities, status, and I/O statistics for all hosts and LUNs to be migrated (as well as those not to be touched), so the migration operation can be organized correctly and accurately with high confidence.
- The ability to migrate during non-production hours, or automatically yield to production traffic, with adjustable intensity/throttling.
- A centralized reporting system that delivers the statistics you need.
- Data integrity checking of intermediate images before cutover, so issues are flagged early before they can cause problems.
- Secure wiping of the data disks after the data is successfully migrated out.
When you’ve drawn up a list, you can apply it, and see how different options compare.

Who are the right people for the job?

A successful data migration is going to require some level of expertise. Ideally, you want experienced personnel to handle a process like this, and you may need to look externally to find them. If they can give you the answers you need to address the questions we’ve posed here in this article, then you know they’re qualified for the job.

About the Author/Wai T. Lam

Wai T. Lam is co-founder and CTO of Cirrus Data Solutions, a developer of Data Migration Server and Data Caching Server for storage area networks (SANs). He was previously CTO and VP of Engineering at FalconStor, a company he co-founded in 2000. There, he was the chief architect, holding 18 of 21 company patents. Wai received the prestigious China national “Top 1000 Technological Leaders” award in 2013.
September 29, 2015
Webinar: Scaling Jenkins with Docker and Kubernetes

Docker is revolutionizing the way people think about applications and deployments. It provides a simple way to run and distribute Linux containers for a variety of use cases, from lightweight virtual machines to complex distributed micro-services architectures. Kubernetes is an open source project to manage a cluster of Linux containers as a single system, managing and running Docker containers across multiple Docker hosts, offering co-location of containers, service discovery and replication control. It was started by Google and now it is supported by Microsoft, RedHat, IBM and Docker Inc amongst others. Jenkins Continuous Integration environment can be dynamically scaled by using the Kubernetes and Docker plugins, using containers to run slaves and jobs, and also isolate job execution.

Recording

Slides

[seoslides embed_id=”8ade4e25d29a” script_src=”https://devopsy.kinsta.cloud/embed-script/cloudbees-scaling-jenkins/10902/” overview_src=”https://devopsy.kinsta.cloud/slides/cloudbees-scaling-jenkins/” site_src=”https://devopsy.kinsta.cloud” site_title=”devopsy.kinsta.cloud” title=”CloudBees, Scaling Jenkins” /]

Your Host

Alan Shimel, Editor-in-Chief devopsy.kinsta.cloud, An often-cited personality in the security and technology community and a sought-after speaker at industry and government events, Alan has helped build several successful technology companies by combining a strong business background with a deep knowledge of technology.

About the Panelists

Carlos Sanchez Gonzalez,Senior Software Engineer, CloudBees

Carlos Sanchez specializes in the automation and quality of software development, and QA and operations processes, from build tools to Continuous Delivery following the DevOps philosophy. He has spoken on the subject at several conferences around the world. Involved in Open Source for over ten years, he is a member of the Apache Software Foundation amongst other open source groups, contributing to several projects, such as Apache Maven, Fog, or Puppet. He works at CloudBees scaling the Jenkins platform.

September 25, 2015
DevOpsSec – Creating the Full Triangle

Introduction

As a discipline, DevOps emphasizes uniting development and IT operations teams through modernized culture, integrated tooling and processes in order to increase the frequency, quality and business alignment of software roll-outs. But while developers and IT Ops teams need to be in lockstep, security cannot be an afterthought. Rather, DevOps teams need to adapt to include security as an integral part of the agile triangle.

The challenge facing DevOps teams today however is that incorporating security into their day-to-day work is not always easy or intuitive. Security often runs one step behind or out of sync with lean DevOps teams. In addition, the besieged security professional or team often faces a never-ending barrage of breaches, designed by a large, amorphous and highly motivated hacker ecosystem and resulting in a continually escalating threat environment. In an effort to save time, agile developers may leverage existing code in rapid iterations and quickly deploy them to production through continuous delivery pipelines, without considering existing architectural defects or knowing its defect profile and potential for security vulnerabilities. This can expose organizations to huge risks if left unassessed and mitigated.

Fortunately, “security at the speed of agile” is entirely possible, allowing organizations to bring new software products and services to the market, while staying in control. Here are some best practices for better aligning security with DevOps objectives, evolving beyond just DevOps to DevOpsSec:

Shifting security “left” in the DevOps chain

Like functional quality testing (making sure an application works as it’s supposed to), security testing needs to happen in the earliest possible stages of the development process. Traditionally, the application lifecycle has followed a sequential model – ideation, development, testing and finally, production. Functional testing used to occur only during the testing phase. But now, developers are taking on more responsibility, which helps prevent defective code from getting “baked in” further down the application lifecycle, where it becomes much more time-consuming and costly to undo. However, security is also an extremely important part of the quality equation, and if a security defect isn’t detected early on, a very unpleasant surprise is likely in store as the application gets closer to production. Building in and continuously integrating static code analysis and dynamic security testing into the Agile application lifecycle is a key success factor to delivering a high quality, secure application experience in production.

Automating security testing, for both developers and testers

In agile environments, DevOps teams are under constant time pressures. As noted above, developers are expected to develop more and more code, all while assuming greater responsibility for functional quality. Testers, too, are under increased pressure as more frequent application roll-outs mean more testing, in less time. For these reasons, if security testing is going to take place earlier on in the application lifecycle, basic security tests need to be automated. Since security threats are constantly evolving, this kind of automation will not completely rule out all security flaws in production. But it will allow developers and testers to spend more time on their core functions – namely development and functional testing – while lowering the attack surface, once an application does reach production.

Leveraging Big Data analytics

Data generated from DevOps processes and operations, particularly related to automated security tests, often hold a wealth of insights which can be used to step up security validation efforts throughout the application development lifecycle. For example, what code types show the most vulnerabilities and therefore may need more rigorous testing? Which groups of developers are discovering the most security holes in automated tests – such that their development processes may need to be refined — and which groups may not be testing enough? How long are basic security tests taking on average, and how does this need to be factored into roll-out timelines? This type of data can help DevOps teams prioritize and optimize their security testing, and better reconcile security testing requirements with velocity and resources to hit anticipated delivery dates. Also, while using Big Data analytics in production will always be useful (particularly when it comes to detecting new security threats, and determining new automated testing needs), applying Big Data analytics earlier on, to DevOps pre-production data, can prevent more security flaws from making it into production in the first place.

Fostering a shared vision and objectives

To date, real progress has been made uniting developers and IT operations teams together in DevOps organizations. Traditional roles and responsibilities within DevOps teams continue to evolve and coalesce, and that’s a good thing. Developers are contributing to testing and taking more responsibility for overall quality. IT ops teams are thinking like developers, analyzing and feeding production data back to developers and testers so they can fine-tune and modify their offerings. Data flows left-to-right, and right-to-left are becoming more automatic and continuous.

But increasingly, security professionals need to be included in this mix. Developers need to think with security considerations in mind – for instance, will a new feature they want to add to an application bring increased security risk, and if so, how can this risk be minimized? Conversely, security professionals need to think like IT ops teams; for example, making sure a new mobile app doesn’t potentially create a hole for hackers to penetrate back-end systems, and if a hack were to happen, how to contain it and minimize the impact. Only by capturing the right data, communicating and committing to working together can everyone in the DevOpsSec chain make the best, most well-rounded decisions that factor in all concerns, with business success being the top priority.

Conclusion

Often, DevOps teams may view security teams as potential logjams preventing them from doing what they want to do – deliver working software quickly to delight users and create competitive edge. But if security in the software supply chain is not addressed, the results may do the exact opposite, frustrating users, freezing business processes or worse, causing massive financial liability and loss of brand goodwill.

Security is a key ingredient of application quality. By addressing security earlier in the development process; increasing security testing automation; leveraging Big Data analytics and incorporating security in the DevOps mission, organizations can achieve true DevOpsSec and deliver applications that are high-quality, through and through.

About the Author/ Kelly Emo

Kelly Emo, Director, Lifecycle and Quality Product Marketing, HP, Kelly has been involved in all things IT for few decades, including engineering, product management, and product marketing across application development, SOA and middleware, enterprise architecture, IT operations, quality and testing. She now focuses on application lifecycle management, quality, agile development and DevOps. She has worked in other organizations at HP, BEA Software, and Jamcracker Inc .Kelly has a BS in Computer Science from Cal Poly, San Luis Obispo and an MBA from Santa Clara University.

September 23, 2015