Thursday, April 11, 2013

Using Fiddler to bypass a load balancer on your terms

So, you're working on this amazing web application, and you've made your first deployment to your integration environment that's reasonably close to your production setup - you even got management to spring for the two servers + load balancer setup like it will be running on in production. Great! And your users start running through tests, and things look great. At least, they do until the user sits idle for a while, then some weird things start happening. Connections get dropped that shouldn't or a user gets a weird error. But if they hit refresh, things are working just fine.

After sifting through logs and sprinkling in some debugging code over a few releases, you begin to suspect that there's something weird happening when the user is idle for a while and the load balancer sends their next request to the other server. The problem is, reproducing the issue is hard. You have to start using the app, leave it alone for a while (whatever length of time sticky sessions is configured for), and then try using it again. And it *may* break (since you may get lucky and get sent back to the same server). If it doesn't, you try again. And again… Or maybe you got sophisticated and added a host header to your local machine to point the domain name directly to one web server or the other. But that has a problem too - you have to clear your cache or restart your browser to get it to pick up the change in the hosts file. Which may very well break the setup of the issue.

If only there was a better way to skip the load balancer in a way that left you in control of how it happened, and allowed you to switch back and forth at will. As it turns out, one of the web developer’s best friends, Fiddler2 (http://www.fiddler2.com/fiddler2/), has something that can help with that. Custom scripting plus the "x-overrideHost" session property.

While most web developers know Fiddler for it's value in letting you see what's really flowing between a web browser and a web server (including seeing inside of HTTPS traffic), as it's name implies, Fiddler also lets you manipulate the requests as they flow past. You can even write scripts that do it automatically for you. And that's exactly what we're going to do.

First, we need to open the rules file:

clip_image001

(There's a custom editor you can install from http://www.fiddler2.com/fiddler/fse.asp which gives you some extra goodies like highlighting and some details of the object model, but it's not strictly required – Notepad or any other text editor will work just fine too)

Now, there's a lot of stuff you can do here (and http://www.fiddler2.com/fiddler/dev/scriptsamples.asp has tons of examples), but let's stay focused here.  First we want to add a configuration option to let us control which server we're hitting, and the method we'll use to wire in the column to display this info (add this anywhere just inside of the Handlers class):

// Override server for www.rightpoint.com
RulesString("Override server for www.rightpoint.com", true)
RulesStringValue(0, "web1", "web1.rightpoint.com")
RulesStringValue(1, "web2", "web2.rightpoint.com")
RulesStringValue(2, "web3", "web3.rightpoint.com")
RulesStringValue(3, "web4", "web4.rightpoint.com")
public static var sRP: String = null;
static function getOverrideHost(oSession: Session){ return oSession["x-overrideHost"]; }

Next, we need to add a chunk of code to the static function OnBeforeRequest (beginning or end works fine):

if (oSession.HostnameIs("www.rightpoint.com") && sRP!= null){
        oSession.bypassGateway = true; // Prevent this request from going through an upstream proxy
        oSession["x-overrideHost"] = sRP; // DNS name or IP address of target server
        oSession["ui-color"] = "green";
}

Finally, we need to add this line to the static function Main (again, beginning or end is fine):

FiddlerObject.UI.lvSessions.AddBoundColumn("Override Host", 60, getOverrideHost);

Save your changes, and you'll notice a new item appear in the 'rules' menu:

clip_image002

With the value set to 'disabled', make a request to the site, and see that we don't override the target host:

clip_image003

And with a simple change of the override setting:

clip_image004

Our next request is routed to a specific web server, even if that request is to load some images the page requested because we expanded the slider in the top navigation:

clip_image005

And a quick check of the headers for one of those requests indicates that the Host field in the HTTP request remains exactly as we'd expect:

clip_image006

And change it to web2 and we see:

clip_image007

As long as there's nothing special** involved, we can skip the load balancer, point our client directly to a specific web server, and potentially isolate any issues our application has when switching servers by swapping at a time and to a server that we choose.

Is this technique really useful? How often do you need to skip a load balancer to dig into what's breaking? Well, you're right - it's not a terribly common thing to need to do, but when you need to, you need to. The initial impetus for this was debugging an issue a client was having in their SharePoint environment where the built-in content types were corrupted on certain servers in the farm, but didn't leave any mark in ULS to let us identify which box it was happening on. With this script, I could flip between the servers at will from my laptop, and quickly use the UI to identify the server whose configuration was off so it could be removed from the farm and fixed.

** What are some of the 'special' things that might break this? Off the top of my head….

  • The load balancer does SSL offloading (and hence the web server is expecting only HTTP requests). You may be able to get Fiddler to handle the HTTPS->HTTP conversion for you, but I'm not sure how to achieve that. If you app is ok with it, you may also be able to just use HTTP URLs, letting fiddler send them directly to the application servers, and have everything work.
  • The web servers are configured to only accept requests from the load balancer.
  • The web application relies on special headers added by the load balancer (ie. X-Forwarded-For). First, I'd say you shouldn't *rely* on those things, but just use them when they're present. However, having Fiddler add the necessary custom headers isn't hard - see http://www.fiddler2.com/fiddler/dev/scriptsamples.asp for some examples, and add the necessary lines in the if block we added to OnBeforeRequest.