![\"By](\"http:\/\/blogs.nd.edu\/devops\/files\/2016\/06\/Weldon_Spring_Site_containment_stairway_1.jpg\")
<\/a>Over 7 million files!<\/p><\/div>\n
<\/a>7 million documents zipped into one large file.<\/p><\/div>\n
Since OnBase is a Windows-centric platform, the need to retain Windows file permissions as part of the data transfer was of prime concern. \u00a0Fundamentally, there were two ways to approach this: trickling the data out using multiple robocopy threads, or doing a bulk data transfer.<\/p>\n
A word on CIFS in AWS<\/h1>\n
A brief aside on providing CIFS<\/a> storage in AWS. \u00a0A straightforward way to get CIFS is\u00a0to use\u00a0Windows Server 2012 and Distributed File System<\/a>\u00a0from Microsoft.<\/p>\n CIFS in AWS<\/p><\/div>\n You could also use a product from a company such\u00a0as Avere<\/a> or Panzura<\/a> to present CIFS. \u00a0These products are storage caching devices that use RAM, EBS<\/a>, and S3<\/a> in a tiered fashion, serving as a\u00a0translation layer between S3 object storage and CIFS. \u00a0Our current configuration makes use of Panzura, striped EBS volumes, and S3.<\/p>\n Robocopy versus bulk transfer<\/p><\/div>\n The initial goal was to get the data from its current CIFS system to a CIFS environment in AWS with all relevant metadata in place. \u00a0We evaluated a number of options, including:<\/p>\n Using the robocopy approach would take weeks to transfer all of the data. \u00a0We immediately started trickling the data out using robocopy. \u00a0That said, the\u00a0team was interested in seeing if it was possible to compress that data transfer time to fit within an outage weekend.<\/p>\n 1 PB of capacity…overkill for 2 TB<\/p><\/div>\n So we tested Snowball. \u00a0The first try\u00a0didn’t go so well. \u00a0The Snowball was dead on arrival, and we had to get a second one shipped to us. \u00a0Ultimately, it worked, but it was overkill for the volume of data we needed to move.<\/p>\n We broke down the transfer process into three basic steps:<\/p>\n The transmit was the easy part. \u00a0We have a 10 GB network connection, and were able to use multipart upload to pump data to S3 at 6 Gpbs, transmitting\u00a0a 200 GB test file\u00a0in under an hour.<\/p>\n 10 GB network connection + S3 multipart upload == speedy transfer<\/p><\/div>\n While discussing the different packaging\/rehydration options, we talked a bit more about how OnBase actually works. \u00a0It turns out, it manages files a bit like a rotating log file. \u00a0That is, it writes to a directory, then switches to a new directory and starts writing to that. \u00a0After files files are written, the old directory essentially becomes read only.<\/p>\n That took the time pressure off the bulk of our data. \u00a0We could robocopy out the bulk of our data at our leisure. \u00a0On cutover weekend, we will migrate\u00a0the active directory\/directories.<\/p>\n Problem solved.<\/p>\n <\/p>\n Stay tuned for adventures in load balancing!<\/p>\n","protected":false},"excerpt":{"rendered":" Over 7 million files? As we were pondering moving OnBase, one of the first considerations was\u00a0how to move over five years’ worth of documents from our local data center to our primary data center in AWS. \u00a0The challenge wasn’t massive … Continue reading So, let’s move this data<\/h1>\n
\n
Snowball<\/h2>\n
Zip, transmit, unzip<\/h2>\n
\n
Reality Check<\/h1>\n
What did we learn?<\/h1>\n
\n