How to migrate 9 million documents to a new SharePoint platform

Published on 05 October 2018 by Andries. Reading time: 4 minutes.

Migrating 9 million documents to a new SharePoint platform is a big migration. In this blog you read how you can execute a large migration. This blog is a case study which of a migration project which was completed in 2018 for a large international firm. 

Scenario outline

A large international firm with over 5.000 employees in the Netherlands used SharePoint 2010 to support document collaboration for their customer engagements. Based on the administration in their ERP system, the solution would create and configure customer- and case sites (for each project) using standard site templates and metadata definitions.
Site creation was performed automatically when a new customer or case was created in the ERP system. Within the same approach permissions were automatically granted for specific sites whenever employees were assigned to certain customers and cases.

As this solution provided the primary location for storing and sharing documents, the utilized storage had grown to 14.5 Terabytes. As the existing software versions were end-of-life, the platform required an update to SharePoint 2013 to be (Cloud) ready for future versions of SharePoint.

 

Main challenges

Business continuity and scale were the primary challenges to perform the migration to the new platform. As this platform was used daily by all employees, any significant downtime would have an impact on customer facing activities. Furthermore, the initial analysis of the data showed a large amount of URL length exceptions which would cause a problem during the migration.

These were caused by end-users creating in-depth folder structures and/or storing files with large names. Any file with a Url over 260 characters would be blocked from the migration due to its length.

Approximately 40 sites were labelled as large sites, as each of these sites would contain more than 15 GB (and some over 100 GB) in terms of documents and versions.

Finally, due to the type of services that were provided to customers, a full proof of successful migrations was a key requirement.

 

Solution approach

As Sharegate had already been used successfully in previous migration projects for this organization, there was a preference of using it again.

ETTU was involved in this project to design and execute the migration approach together with the internal customer team. Due to the large number of sites and documents, defining the migration strategy required extensive testing to result in a successful outcome and optimal performance.

 

Ultimately, the migration strategy consisted of 3 stages:

migration strategy

 

To support this approach, four instances of Sharegate were installed on 2 separate servers with 2 different accounts on each server. This provided a stable setup with 4 parallel sessions (Sharegate Insane Mode), targeting separate web applications for each division. The Sharegate servers would target separate SharePoint webservers in both the source and destination environments, which were not hit by end-users. The average throughput was approximately 15 GB/hour.

As Sharegate comes with a rich PowerShell module, a script library was created to support all copy, validation and reporting tasks required to perform a migration. The script logic would connect to each site in the source and destination using an input (“to-do”) list with site URL’s and iterate through each library to copy the content to the destination. Completion of each migration job would result in the addition of a row in a migration result report.

This would contain all relevant information for the business to approve the migration, including the source/destination site pairs and migration result in terms of success, warning and errors.

This report would also include an item count measured in both source and destination showing potential difference between the two. The migration team would use this information to focus on resolving issues with items missing in destination.

Working towards the cut-over migration stage, the team collaborated intensively with the business representatives to shorten the URL’s of the file/folder combination exceeding the threshold blocking the migration. Alternatively, Sharegate does provide capabilities to map folder structures towards metadata. You can learn more about this feature in this video

 

Result & lessons learned

With the final cut-over, the full migration of over 52.000 sites and 9 million documents was completed successfully with minimal disruption of the platform availability.


The scale and characteristics of this migration resulted in some valuable lessons learned:

  • Large scale migrations require extensive analysis and testing (especially with large data sets) to define the optimal migration strategy. Choosing the right tooling and extending this to automate as much as possible, is essential to ensure continuity of copying content;
  • Be aware of how a collaboration platform can reflect organizational changes, which may result in additional effort to ensure that “hidden” data is also copied over;
  • Large scale migrations are not only about copying large amounts of data, but also about the impact on the underlying infrastructure. Something that needs to be considered to avoid capacity issues or data loss during the migration;
  • Invest in stakeholders as these know the business best and can support in resolving issues or inform the user groups;
  • Due to the amount of data, large scale migrations require a process approach. Many activities are recurring in nature and require continuity, which should be provided in terms of availability within the team and keeping up-to-date on progress and issues;
  • Sharegate provided us with a cost-effective migration solution with the flexibility to create a fully automated approach for this scale and to fulfill the reporting requirements (e.g. source and destination urls, stating the amount of items migrated, migration outcome, comparison of source and destination item count).

Tags: SharePoint, DMS

Andries

Published by Andries on 05 October 2018

Since 2001 I have been working in various roles in advising companies on digital (collaborating) and sharing information based on Microsoft (cloud) technologies and products. I gained a lot of experience in the field of platform optimization as well as implementation and (large-scale) migration projects. In addition to a strong technical basis, I focus on Business & IT Alignment and in particular the modern ways of IT companies and making continuous change manageable to achieve business benefits. In recent years I am increasingly working strategically in the role as an architect. Inspiration and innovation are central to what I do, and preferably together with the customer and from modern workplace concepts.

 

Comments? Share them below

Learn more about ETTU

And the way we work