{"id":521,"date":"2016-06-20T15:21:59","date_gmt":"2016-06-20T15:21:59","guid":{"rendered":"http:\/\/blogs.nd.edu\/devops\/?p=521"},"modified":"2016-06-20T15:21:59","modified_gmt":"2016-06-20T15:21:59","slug":"onbase-to-aws-adventures-in-load-balancing","status":"publish","type":"post","link":"https:\/\/sites.nd.edu\/devops\/2016\/06\/20\/onbase-to-aws-adventures-in-load-balancing\/","title":{"rendered":"OnBase to AWS: Adventures in Load Balancing"},"content":{"rendered":"<h1><b>Good to have Goals<\/b><\/h1>\n<p><span style=\"font-weight: 400\">One architectural goal we had in mind as we designed our OnBase implementation in AWS was to break out the single, collapsed web and application tier into distinct web and application tiers.<\/span><\/p>\n<p><span style=\"font-weight: 400\">Here\u2019s a simplification of what the design of our on-premises implementation looked like:<\/span><\/p>\n<p><a href=\"http:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-2.jpeg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-523\" src=\"http:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-2-300x238.jpeg\" alt=\"SBN Existing - 2\" width=\"300\" height=\"238\" srcset=\"https:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-2-300x238.jpeg 300w, https:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-2-768x609.jpeg 768w, https:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-2-1024x812.jpeg 1024w, https:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-2-378x300.jpeg 378w, https:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-2.jpeg 1160w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><br \/>\n<span style=\"font-weight: 400\">Due to licensing considerations, we wanted to retain two, right-sized web servers behind an Elastic Load Balancer (ELB). \u00a0We then wanted to have that web tier interact with a load balanced application tier. \u00a0Since licensing at the application tier is very favorable, we knew we could horizontally scale as necessary to meet our production workload.<\/span><\/p>\n<h1><b>To ELB, or not to ELB, that is the question<\/b><\/h1>\n<p><span style=\"font-weight: 400\">Our going-in assumption was that we would use a pair of ELBs. \u00a0It was remarkably quick to configure. \u00a0In under an hour, we had built out an ELB to front the web servers, and an ELB to front the application servers. \u00a0It was so easy to set up. \u00a0Too easy, as it turned out.<\/span><\/p>\n<p><span style=\"font-weight: 400\">During functional testing, we observed very strange session collisions. \u00a0We worked with our partners, looked at configuration settings, and did quite a bit of good, old-fashioned shovel work. \u00a0The big sticking point turned out to be in how we have OnBase authentication implemented. \u00a0Currently, we are using Windows Challenge\/Response (<\/span><a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/windows\/desktop\/aa378749(v=vs.85).aspx\"><span style=\"font-weight: 400\">NTLM<\/span><\/a><span style=\"font-weight: 400\"> to be specific, dating back to the days of NT LAN Manager) to authenticate users. \u00a0The problem is, NTLM authentication does not work across an HTTP proxy because it needs a point-to-point connection between the user\u2019s browser and server. \u00a0See <\/span><a href=\"https:\/\/commscentral.net\/tech\/?post=52\"><span style=\"font-weight: 400\">https:\/\/commscentral.net\/tech\/?post=52<\/span><\/a><span style=\"font-weight: 400\"> for an explanation of NTLM via an ELB.<\/span><\/p>\n<p><span style=\"font-weight: 400\">When an IIS server gets a request, it sends a 401 response code (auth required) and keeps the HTTP connection alive. \u00a0An HTTP proxy closes the connection after getting a 401. \u00a0So in order to proxy traffic successfully, the proxy needs to be configured to proxy TCP, not HTTP. \u00a0NTLM authenticates the TCP connection.<\/span><\/p>\n<h1><b>Enter HAProxy<\/b><\/h1>\n<p><span style=\"font-weight: 400\">In order to get past this obstacle, we ended up standing up an EC2 instance with HAProxy on it, and configuring it to balance TCP and stick sessions to backends. \u00a0Properly configured, we were cooking with gas. \u00a0<\/span><\/p>\n<h1><b>A word on scheduling<\/b><\/h1>\n<p><span style=\"font-weight: 400\">For our production account, we make extensive use of reserved instances. \u00a0On a monthly basis, 85% of our average workload is covered by a reservation. \u00a0For OnBase, we use t2.large running Windows for our application servers.<\/span><\/p>\n<p><span style=\"font-weight: 400\">Currently, a one-year reservation costs $782. \u00a0If you were to run the same instance at the on demand rate of $0.134 per hour, the cost for a year would be $1174. \u00a0For an instance which needs to be on 24&#215;7, reservations are great. \u00a0For a t2.large, we end up saving just over 33%.<\/span><\/p>\n<p><span style=\"font-weight: 400\">Due to application load characteristics and the desire to span availability zones, we ended up with four application servers. \u00a0However, from a workload perspective, we only need all four during business hours: 7 am to 7 pm. \u00a0So, 52 weeks per year, 5 days per week, 12 hours per day, $0.134 per hour. \u00a0That comes out to just over $418, which is a savings of over 64%. \u00a0If you can schedule, by all means, do it!<\/span><\/p>\n<h1><b>Simplified final design<\/b><\/h1>\n<p><span style=\"font-weight: 400\">So, where did we end up? \u00a0Consider the following simplified diagram:<\/span><\/p>\n<p><a href=\"http:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-1.jpeg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-522\" src=\"http:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-1-300x247.jpeg\" alt=\"SBN Existing - 1\" width=\"300\" height=\"247\" srcset=\"https:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-1-300x247.jpeg 300w, https:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-1-768x633.jpeg 768w, https:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-1-1024x844.jpeg 1024w, https:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-1-364x300.jpeg 364w, https:\/\/sites.nd.edu\/devops\/files\/2016\/06\/SBN-Existing-1.jpeg 1591w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400\">The web servers are in separate availability zones, as are the application servers. \u00a0There are still single points of failure in the design, but we our confident in our ability to recover within a timeframe acceptable to our customers.<\/span><\/p>\n<h1><b>So, what did we learn?<\/b><\/h1>\n<p><span style=\"font-weight: 400\">We learned quite a bit here, including:<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">NTLM is not our friend. \u00a0Replacing it with SAML\/CAS this summer will allow us to jettison the HAProxy instance and replace it with an ELB.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Scheduling is important. \u00a0Reserved Instances optimize your spend, but they don\u2019t optimize your usage. \u00a0You\u2019re leaving a lot of money on the table if you\u2019re not actively involved in scheduling your instances, which you can only do if you have a deep understanding of your system\u2019s usage profile.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Working in AWS is a lot of fun. \u00a0If you can imagine it, it\u2019s really easy to build, prototype, and shift as appropriate.<\/span><\/li>\n<\/ol>\n<p>Coming soon, migrating the OnBase database.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Good to have Goals One architectural goal we had in mind as we designed our OnBase implementation in AWS was to break out the single, collapsed web and application tier into distinct web and application tiers. Here\u2019s a simplification of &hellip; <a href=\"https:\/\/sites.nd.edu\/devops\/2016\/06\/20\/onbase-to-aws-adventures-in-load-balancing\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1551,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[65088,65095],"tags":[],"class_list":["post-521","post","type-post","status-publish","format-standard","hentry","category-aws","category-cloud-infrastructure"],"_links":{"self":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts\/521","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/users\/1551"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/comments?post=521"}],"version-history":[{"count":3,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts\/521\/revisions"}],"predecessor-version":[{"id":528,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts\/521\/revisions\/528"}],"wp:attachment":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/media?parent=521"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/categories?post=521"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/tags?post=521"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}