{"id":526,"date":"2016-06-21T15:45:13","date_gmt":"2016-06-21T15:45:13","guid":{"rendered":"http:\/\/blogs.nd.edu\/devops\/?p=526"},"modified":"2016-06-21T15:45:13","modified_gmt":"2016-06-21T15:45:13","slug":"onbase-to-aws-database-migration","status":"publish","type":"post","link":"https:\/\/sites.nd.edu\/devops\/2016\/06\/21\/onbase-to-aws-database-migration\/","title":{"rendered":"OnBase to AWS: Database Migration"},"content":{"rendered":"

As part of migrating OnBase to AWS, we also had to migrate the database itself. \u00a0This proved to be a much less complicated task than moving all of the document files, somewhat of an anti-climax.<\/span><\/p>\n

A Bit of History<\/strong><\/p>\n

When we first implemented OnBase, we chose to use Oracle as the database engine. In order to improve operational stability, we switched database platforms to Microsoft SQL Server in 2014. \u00a0Our primary reason for the switch was to align us with Hyland\u2019s main line of development. \u00a0We felt good about getting that work done prior to our move to AWS. <\/span><\/p>\n

It’s important to understand what purpose the OnBase database serves. \u00a0It contains the metadata and user annotations, while the underlying documents reside on a file system. \u00a0As such, the database doesn’t take up much space. \u00a0A compressed backup of our database is only 16 GB. <\/span><\/p>\n

Transfer the Data<\/strong><\/h1>\n

Transferring a 16 GB file to AWS is a relatively straightforward task. \u00a0The most important feature you to be aware of is multi-part upload.\u00a0\u00a0This feature takes a single large file and splits it into multiple parts.\u00a0\u00a0These parts are then uploaded in parallel to S3.<\/span><\/p>\n

Initially, we looked into using CrossFTP<\/a>. \u00a0CrossFTP is a third party, GUI tool which allows you to specify parameters to the S3 copy command, including thread count and chunk size. \u00a0That said, CrossFTP simply takes what you can do natively using the AWS Command Line Interface<\/a> (CLI) and gives you a graphical front end. The graphical interface abstracts the messiness of a command line and is quite approachable for a novice. \u00a0However, that was not necessary in this case. \u00a0We just used AWS CLI to transfer the data from here to S3<\/a>. \u00a0It couldn’t be much more straightforward, as the syntax is:<\/span><\/p>\n

aws s3 cp <fileName> s3:\/\/<bucketName>\/<fileName><\/pre>\n
In order to get the performance to an acceptable level, we made the following configuration adjustments in the ~\/.aws\/config file:<\/span><\/p>\n
[profile xferonbase]\r\naws_access_key_id=<serviceId>\r\naws_secret_access_key=<serviceKey>\r\ns3 =\r\n  max_concurrent_requests = 40\r\n  multipart_threshold = 512MB\r\n  multipart_chunksize = 512MB<\/pre>\n
Appropriately configured, transferring our 16 GB database backup took just under\u00a017 minutes. \u00a0Once the file was in S3, we simply transferred it to our SQL Server database instance and performed a restore. \u00a0Simple and easy. \u00a0Highly advisable if you have the luxury of bringing your system entirely down when migrating.<\/span><\/p>\n
A look to the future<\/strong><\/h1>\n
In fact, the simplicity of this backup\/transfer\/restore approach is influencing how we think about migrating our ERP. \u00a0We use Banner running on Oracle. \u00a0While our database size is a bit over 1 TB,\u00a0the compressed backup is 175 GB. \u00a0Conservatively, that’s a 3 hour segment of time to get the file to S3. \u00a0Once the data is in S3, we’re using an S3 VPC endpoint for private connectivity and speed.<\/span><\/p>\n
Thinking about migration weekend and the need to build in sleep time, the backup, validate, copy to S3, copy to EC2, restore process is something\u00a0easy to script and test. \u00a0And a great way to build in sleep time! \u00a0We’ve all done enough migrations to know that people don’t make good decisions when they are exhausted.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
As part of migrating OnBase to AWS, we also had to migrate the database itself. \u00a0This proved to be a much less complicated task than moving all of the document files, somewhat of an anti-climax. A Bit of History When … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1551,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[65088,65095],"tags":[],"class_list":["post-526","post","type-post","status-publish","format-standard","hentry","category-aws","category-cloud-infrastructure"],"_links":{"self":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts\/526","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/users\/1551"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/comments?post=526"}],"version-history":[{"count":3,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts\/526\/revisions"}],"predecessor-version":[{"id":530,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts\/526\/revisions\/530"}],"wp:attachment":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/media?parent=526"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/categories?post=526"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/tags?post=526"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}