devopsy.kinsta.cloud

Tag: migration

  • Migrating to GitHub

    Migrating to GitHub

    In recent months, Newt Global Consulting migrated multiple source code repositories to GitHub. The repositories are anywhere from 10MB to few Gigabytes of source code. This blog is to share our experiences in migrating source code from SVN and AccuRev to GitHub. Most of the material below should be valid for any Git repositories.

    We used 100 percent open-source tools that are freely available, and we developed the process that fits our needs. Thank you very much for the authors of these tools; without them, we couldn’t have done it. Long live open-source community and may the day never come when machines code!

    This article is mainly focused on executing migration; best practices or branching strategies for GitHub are not in the scope.

    Before diving into migration, here are some of the motivators we observed that are steering organizations away from traditional source code repositories:

    • Managers and architects like the distributed and offline feature of GitHub.
    • Enforces discipline on developer’s part (less excuses of dependency—you have local copy of everything, intelligent merging of GitHub, etc.).
    • Technical-savvy groups already are using GitHub (Git Bridges).
    • Developers hate the change in the beginning but they are quick learners and like to adapt with rest of the industry.
    • Integration with other DevOps toolkits is a motivator (ex: Webhooks).
    • Naturally fits with open-source software development (easily extend and contribute).
    • Enterprise-level groups are pushing the change (playing catchup with individual development groups and the rest of the industry).
    • File size limitation of GitHub is forcing the use of binary repositories and security teams like it, as they can now monitor the large open-source libraries spread across the org.

    SVN to GitHub Migration

    There is plenty of material on the web about the difference between SVN and GitHub. I suggest that you go through them (if you have to). Here are few links that we found interesting:

    Prerequisites

    • Get read-only user access for SVN repository.
    • Make sure “gitsvn” utility is available in local environment.
    • Download svn-migration-scripts.jar from https://bitbucket.org/atlassian/svn-migrationscripts/
      downloads
    • Refer to the following URL for more documentation on the above script: https://www.atlassian.com/git/tutorials/migrating-prepare/
    • Decide on GitHub organization name, team name and members who will be part of the team.
    • Set up the organization.
    • Set up the team and assign team members.
    • Create a repository to push the code from your work area.

    Migration

    • Extract the users information from SVN java -jar svn-migration-scripts.jar authors <SVN Repo URL> > authors.txt
    • Clone the SVN Repository git svn clone –stdlayout –authors-file=authors.txt <SVN Repo URL> <GitHub Repo
      Name>
    • Create connectivity to remote repository git remote add origin <GitHub Repo URL>
    • Push the code from local to remote repo git push –u origin master
    • Converting remote braches to local repo java -Dfile.encoding=utf-8 -jar svn-migration-scripts.jar clean-git –force
    • Push all the changes to remote repo git push –all

    Noteworthy

    • We ran the scripts on Linux (case-sensitive file system); OS X has additional steps to follow (refer: https://www.atlassian.com/git/tutorials/migrating-prepare/)
    • If SVN instances have local user IDs defined (not SSO), the authors file created is not of use. But you need this file to successfully to run the scripts.
    • Incremental migration is possible, but we scheduled weekly jobs for full migration during the transition phase.
    • Full migration takes time; try to avoid network delays. For us, our 300MB repo took around four to six hours.
    • In some cases, we had to clone the SVN repo to our local file system due to access issues.

    AccuRev to GitHub Migration

    AccuRev is a centralized version control system developed by Borland (now Micro Focus). It uses client-server architecture and preparatory technology, and so it’s a bit tricky to extract code for full migration. We used the ac2git utility developed by Navico—the tool does the heavy lifting but took few iterations and bit of tweaking to fit our need. Most of the difficulty was in understanding how AccRev works and the relative mapping to GitHub.

    Prerequisites

    • Install python 3.4, Git-Bash version 2.7.4 and AccuRev
    • Make sure the paths to the AccuRev and git executables are correct for your machine, and git default configuration has been set
    • Clone the ac2git repo from the https://github.com/NavicoOS/ac2git
    • Run python ac2git.py –help to see all the options (strongly recommend you do this)
    • Run python ac2git.py –example-config to get an example configuration
    • Strictly follow the steps outlined in the “How to Use” section; otherwise, you will be forced after few failed attempts

    Migration

    • Make an example config file:
      • python ac2git.py –example-config
    • Modify the generated file ac2git.config.example.xml, (there are plenty of notes in the file and it is time to run –help option if you have not done it as advised in prerequisites)
    • Rename the ac2git.config.example.xml file as ac2git.config.xml
    • Modify the configuration file and add the following information:
      • Set accurev username & password
      • Name the depot. Map each depot to single Git repository. Run the script for each depot separately
      • Running the script for multiple depots to single folder overrides all depot streams into same folder. Scripts fail when given multiple folders, so always run the script for one depot at a time
      • Create an empty folder and provide the complete path in the config xml file. The folder must exist and preferably should be empty
      • There is no concept of having same folder name as stream name. There just needs to be an empty folder where all the contents of stream will store
      • Start & end transactions which correspond to what you would enter in the accurev hist command as the<time-spec> (the keyword highest or the keyword now)
      • If the start-transaction and the end-transaction are time-spec, the script will fetch the data and history only within this time period. For example start-transaction = “2013-02-07 13:41:17” and endtransaction=
        “2014-02-07 13:41:17”
      • Use “highest” keyword instead of using “now” in end-transaction
        • “now” will fetch data until the date and if there was some history deleted from workspace or not promoted from other streams than “now” keyword will not work
        • “highest” keyword always look for the latest and highest commit history (preferred option)
      • User mapping from Accurev to GitHub. Hint: Run accurev show -fi users to see a list of all the users
      • Choose the preferred method for converting the streams
        • Recommend ”deep-hist” method for sparse streams (transactions that have changed the stream contents are far apart)
        • Recommend “diff” method for regular streams (when in doubt, just use “deep-hist”.)
      • Run the script
        • python ac2git.py
      • If you encounter any trouble, run the script with the –help flag for more options.

    Noteworthy

    • The stream must always have some history. The script did not work for us when no history was available.
    • If there are any duplicate or missing usernames, make sure to use the corresponding parameter. We ended up changing the script to handle duplicate users (this is not needed if you use the right parameters).
    • In GetMissingUsers(config) method of ac2git.py code, comment last two lines
      • # if not found:
      • # missingList.append(user)
    • Migration takes time, try to avoid network delays. For us, a 300MB repo took around six to eight hours.
    • We have not tried incremental migration—we scheduled a weekly full migration and cut over all the users over a long weekend.

    Conclusion

    Overall, the experience was very rewarding, not only in assembling the right toolkit but process-wise as well. Every organization has its own methodology when comes to source code management and how it promotes code (branching strategy). We found small-to-medium-size enterprises have much more mature process than some of the large enterprises, though they are catching up fast.

    • Don’t expect large enterprises have centrally managed source code repos (SVN hosted in some sharable desktop under manager’s desk).
    • Difficult to get developers to think distributed mindset (why should I clone the entire repo?).
    • Working closely with Enterprise GitHub helped in reassuring clients.
    • Most of developers like to use IDE plugins than command line or web client.
    • Prepare to answer backup, HA and DR-related questions.
    • Enterprise GitHub comes as virtual appliance not application—prepare yourself to deal with infrastructure groups on deploying VMs into production.
    • Some application groups preferred lift and shift mode than migration; don’t get shocked when they say “don’t care about history” (typically these are small dev groups).
    • Integrated Slack with GitHub—this was hit not only with developers but also leadership.
    • Extended Hygeia dashboard to provide team level analytics for leadership view.

    About the Author / Sridhar Peddinti

    sridharpeddintiSridhar Peddinti is vice president, DevOps and Cloud practice at Newt Global Consulting. He is a 20-year technology and business leader with global experience in executing large scale projects across multiple industry verticals. He champions the consulting space helping customers with complex problems and providing them with DevOps and Cloud strategy. He has extensive hands on experience in process improvements using combination of tools kits and migrating number of legacy systems to Cloud and related technologies. Connect with him on LinkedIn and Twitter.

  • Legacy IT Challenges Continuous Delivery

    Legacy IT Challenges Continuous Delivery

    Legacy information technology infrastructure has a strong effect on the potential success of DevOps. In environments where different teams have always used the same servers, change can be difficult due to inter-team coordination and limited server transparency and application inventories, says David Blank-Edelman, a 30-year systems administration/DevOps/SRE veteran and currently Technical Evangelist at Apcera.

    “Unraveling the Gordian knot to bring DevOps principles and practices into that environment will be quite a challenge,” he says.

    In such a legacy environment, the enterprise walks a topological tightrope stretched between one or more development machine clusters shared by multiple teams with no real isolation between teams and projects, Blank-Edelman notes. “In most cases, there is a second parallel set of machines for production and, if the business is very fortunate, for test/QA as well. There is seldom anything, however, to prevent a production workload from talking to a database server in development by mistake, which can lead to all sorts of fun.”

    How to Move to CD

    One solution is a policy system approach, in which the organization provides for a policy system to migrate a legacy information technology environment to a platform that uses continuous delivery (CD), Blank-Edelman says.

    To create the policy system, he notes, the organization will need to address questions such as:

    1. Are the workloads allocated/using only the resources that they should?
    2. What is inside the workload? What is its provenance? Is it acceptable and in good order from security, coding and business requirements perspectives?
    3. Where is the workload allowed to run? In which cloud? In which environment? In which country? Near what other containers?
    4. Can the workload communicate with only the things it should? If I move it to another location, is this still true?

    “The process of developing a policy system and then enacting it can lead an organization on a journey that brings their existing infrastructure to a much better place. If that platform can also automatically enforce the decisions that organization made during this question and answer discovery process, you can be certain that the answers to these questions stay answered,” says Blank-Edelman.

    Other approaches to moving to continuous delivery include tool-first approaches in which the organization adopts one to several CI/CD/workflow tools and then makes an effort to implement those, possibly by shoehorning existing processes and runbooks into that tool or toolset.

    A better tactic is the people-first or culture-first approach that focuses on how the development and operations personnel will work together to create a product vs. a project, says Blank-Edelman. While tools still have a role to play in the people-first approach, they are not the priority; rather, organizational change comes first. Once the people have made the commitment and the change, everything else will follow.

    Challenges During the CD Move

    It’s not the existing information technology infrastructure that presents moderate challenges to CD that is the issue; it’s the infrastructure that poses either very many challenges or very few that is the hurdle. “If you start with a more challenging situation, this will incentivize people to work very hard to reduce the associated pain points in their life,” Blank-Edelman says.

    On the other hand, he notes, when an organization begins a move to CD with an infrastructure that is already in pretty good shape, you won’t often hear its people say that they should stop working on the product and work on CD all the time instead. People will still claim they have plenty to do already.

    Continuous Deployment Can’t Exist Without CD

    Continuous delivery starts with a code base and takes logical steps through that code base to increase confidence that it is ready for production, using a process that is as smooth, automated and loosely coupled as possible, he says. With confidence in that pipeline comes confidence in moving the result of that process into production automatically, which is continuous deployment, he adds.

  • Avoiding Database Migration Pitfalls with Continuous Software Delivery

    Avoiding Database Migration Pitfalls with Continuous Software Delivery

    When people talk about continuous software delivery, they often focus on the delivery mechanism itself, such as automation tooling or branching strategies. What is frequently overlooked is the simple truth that continuous software delivery is worthless without continuous service availability. In other words, if you’re going to frequently update your service, your customers shouldn’t frequently experience disruptions.

    Despite all good intentions, the cause of many unintentionally-disruptive deployments are database migrations. This makes sense, as all other components of a service infrastructure are fairly easy to swap out or to roll back. Software directory symlinks can be easily changed, EC2 instances can be easily started, DNS entries can be easily updated, but performing a breaking change on a multi-million row table in a database that’s under heavy load? Not so easy.

    Successful Migrations

    There are three necessary requirements for a database migration to be considered a success:

    1. The delivery of the service was not disrupted or degraded
    2. The database migration must be repeatable (re-entrant)
    3. The software version can be rolled back to the previous release if needed

    The first seems obvious, but can be surprisingly difficult to achieve. The chief reason for this lies in the vagaries of how different database technologies, and even versions of the same technology, handle things like table locking during an ALTER statement. The other reason is a hidden complexity within your application code. For example, a JOIN statement that causes a frequent query to be blocked by a seemingly unrelated table change.

    The second criteria means that if a migration fails for some reason, it cannot leave the data in such a state that the migration cannot be run again – essentially, running the same migration twice should not result in a negative outcome.

    The third criteria absolutely precludes ever doing a “one way” release where rollback is made impossible due to data destruction. It’s okay to have a release that rules out rolling back several releases; without that, the database would become a giant pile of discarded data. That said, it’s never a good idea to paint yourself into a corner with a release that you cannot possibly revert.

    Avoiding Service Disruption

    The simplest way to avoid service disruptions is to never perform a database migration. But, that just isn’t feasible, so how do we avoid stepping on a landmine during a migration?

    First and foremost, implement a policy of strict database migration reviews. This doesn’t need to be burdensome or time consuming. It just means that someone who is best equipped to understand the impact to the database technology actually gets eyes-on the migration before it is run. This is a role often served by your database administrator, chief architect or most senior developer.

    Another good idea is to set up a hook in your code versioning repository, or a test that’s part of your build process, that examines database migrations and looks for certain keywords – and the lack of other keywords. For example, it might “fail the build” if it finds an ALTER TABLE statement that is not accompanied by the word CONCURRENT.

    If you’re dealing with a lot of legacy code, or a monolithic application with significant complexity, then there are two very important safeguards you can put in place: load testing and SQL statement logging.

    Load testing means having a development environment setup where you can apply a database migration to a database that is under load, i.e., experiencing a level of usage that roughly approximates your production database load and which contains an amount of data similar to production. Then, while your database migration is running, you can run an application smoke test or similar to see if production-like load combined with the migration results in any unexpected slowness or errors.

    Remember, all database migrations are fast when they are the only thing running on your database, and faster yet when the testing database is a tiny fraction of the size of the real thing. Going into production feeling confident that a migration took only ten milliseconds to complete on a half-empty database with zero active connections is a fool’s errand.

    When you have a poorly-understood or monolithic application, it can be a very enlightening experience to enable SQL statement logging on your production database server. Remember that MyISAM table that never seems to change and against which it should be safe to run a giant blocking migration? SQL statement logging might reveal that every single page load executes a long-forgotten JOIN statement that scans that entire MyISAM table. Or that the user login process looks in that table for reasons nobody can remember. This is also an area where using a solid application performance tool like New Relic can make a very big difference.

    Repeatable Database Migrations

    This one sounds simple but can actually be a bit tricky to implement. Consider the following scenario, you have a table that contains three columns: id (integer), username (string) and password (string).

    As it currently stands, the passwords are stored in plaintext in the database, and your application does a simple string comparison between what the user enters and the results of a SELECT statement. The next version of your application is going to hash the password entered by the user, and compare the hash against a hash returned from the database by SELECT.

    There are two possible ways of getting from where you are now, to where you want to be – one is dangerous, and the other is safe.

    The dangerous way would be to create a database migration that iterates through the password column and replaces each plaintext value with a hash of that value. Basically a SELECT; md5(); UPDATE; loop. This is dangerous for a number of reasons:

    If the migration stops before completing, the database will be left in an unusable state where neither the old nor new code can make use of the entire table.

    1. If the migration fails to complete for any reason, it cannot be run again without corrupting all the previously-migrated values.
    2. If the new code has to be reverted to the old code for any reason, the table is now in a state that cannot be used by the old code.

    The safe, and repeatable way of performing this migration actually involves three releases:

    Version 2.0 migration adds a new column (pass_hash) and migrates password to it, and new application code must INSERT and UPDATE to both

    1. Version 2.1 application code must now use only the pass_hash column, and ignore the password column.
    2. Version 2.2 migration removes the password column, and new application code continues to use only the pass_hash column.

    You can rollback from the first release (2.0), as all the original, unencrypted password data still exists in the database, and the previous code release won’t care about the new column (pass_hash) being present. You can also run that first migration multiple times – just have it remove the pass_hash column if it exists prior to creating it.

    Next, you deploy code (2.1) that makes use of the new column only. You might be tempted to bundle the database cleanup migration with it, but don’t – if you need to rollback for any reason, you won’t be able to if you’ve removed the password column. It’s always a good idea to delay breaking changes, like a column deletion or dropping a table, by at least one release.

    And finally, run a migration (2.2) that drops the password column, if it exists. That latter condition is so you can run the overall migration more than once if needed. If the password column exists, remove it; otherwise, merrily continue with the rest of the release. Note that at this point in time, the live application code is ignoring the password column entirely. This is important, since the current application code will be using the database while your migration is running. In other words, any migration must be safe to run under both the current and the about-to-be-deployed code version.

    In case you hadn’t already guessed, this approach to migrations does take a bit more conscious thought and effort than the old, dangerous way. You’ll need to keep track of “deprecated” bits of your database for future removal. And you’ll need to occasionally write code to intelligently update two columns or tables in lockstep, and then remember to remove that code in the future. That said, it’s definitely worth it in the long run when you’re able to release frequently with zero downtime.

    Release Rollback

    Nothing produces longer outages and greater negative customer impact than being unable to rollback or revert a release. If you’ve accidentally deployed a major bug into production, you want to quickly revert to the previous version that didn’t have that bug, and then reproduce and fix the bug at your leisure in your development environment. Being unable to reproduce the bug in your development environment is not a valid reason to fail to rollback a release, and it’s an entirely separate discussion apart from migrations.

    The essence of rolling back is speed. It must be a fast and reliable process. The number one way to ensure it is anything but fast and reliable is to have a breaking or one-way database migration in the release you wish to roll back. Thus, anytime you are creating a new database migration, ask yourself two simple questions:

    1. What would happen if I ran this migration with the current application version?
    2. What would happen if I ran this migration twice?

    If it would change the database in a way that would cause the current application version to explode, then it needs fixing. And if it would change the database in such a way that it can’t be run again, then it needs fixing.

    The End Results?

    In a nutshell, this approach to migrations means that each migration will be very small and very simple. Expect to get from state A in your database to state B via many safe steps, instead of one giant, dangerous step. Your customers, and your application uptime, will thank you!

    About the Author/Nate Lindstrom, VP solutions engineering

    highres_250316251Nate Lindstrom is the VP of solutions engineering for NS1, an intelligent DNS and traffic management platform with data driven architecture and a unique Filter Chain routing engine. NS1 is purpose-built for the most demanding, mission-critical applications on the Internet. NS1’s comprehensive platform technology leverages infrastructure, application, and network data to make intelligent routing decisions in real time, ensuring optimal application performance and reliability.