The Trainline Website vs Advert Blockers

We use a popular UK based website at work for booking train tickets – thetrainline.com. Recently we noticed that users were unable to perform advanced searches on the website, instead being told “Oops, something went wrong”. Not the most helpful of error messages! We could see a whole host of embedded advert/tracking/analytics websites being blocked at our proxy when users were performing the search, so I went through each one one-at-a-time to find out which was the one causing an issue. The search worked just fine off the corporate LAN, so it must have been one of our corporate security tools causing the problem. This is how it was done.

Using Chrome, browse to the site. Open the Developer tools (F12, or Menu -> More tools -> Developer tools) and look at the Network tab. Sort by the Status column to see which are failing to load:

How you do the next step depends on your network layout. As our issue was users on a corporate network behind a proxy, I tested from a computer which was not on the corporate LAN and therefore not behind the proxy. You could do it using tools your your local firewall/gateway, proxy or ZAProxy for testing. Or you could do it simply by editing your hosts file. I added each of the failed hosts to my hosts file pointing to 127.0.0.1. This replicated the fault on my computer off the corporate network. Then I removed them one at a time until the search worked. The culprit was delivery.g.switchadhub.com – adding that to our proxy whitelist fixed advanced searches on thetrainline.com – our proxy was blocking it under the category “Web Ads” as any advert blocker would.

Update: thetrainline.com don’t seem to be embedding javascript from delivery.g.switchadhub.com anymore, so this is no longer an issue. However I’m leaving this page here for future reference.

Plex, pfSense, OpenDNS and DNS Rebinding

I’m starting to use pfSense a lot more at home now, making use of the advanced security features rather than it just being a router. I got quite frustrated last night when my girlfriend and I sat down to watch a film on Plex to discover that my Xbox One (Plex Client) could not see the Plex Server that was sat right next to it, on the same network. Further investigation proved that neither could my smart TV. I could access the server remotely via the Plex web app, so it wasn’t a port forwarding issue – I had already allowed 32400/tcp through pfSense to the Plex server anyway.

It turns out it was some security protection against DNS Rebinding. DNS Rebinding attacks are where someone directs you to an address which resolves to an internal IP. Loads of appliances and broadband routers use this functionality to present you with captive portals, and so on. I had seen a lot of people having similar trouble with Plex behind pfSense, so I followed the instructions. In fact, Plex themselves even provide you with instructions on their How To Use Secure Connections support page.

Actually, you should follow the DD-WRT dnsmasq instructions if you use the “DNS Forwarder” on pfSense as this is dnsmasq. The config line goes in Services -> DNS Forwarder in the Custom Options section, like this:

So, I had configured pfSense as Plex support advised, but I was still unable to connect to the server. Eventually I read up on how the plex.direct domain name is used. When you log in to your Plex account you’re granted an authentication token which looks like any other guid (some long random hex string). The Plex service creates a wildcard HTTPS certificate for *.guid.plex.direct which is unique to your account. It then tries to access your Plex server at https://ipaddress.guid.plexdirect:32400/ substituting the dots in the IP address for dashes. So such a URL could look like

https://192-168-0-1.a6b0cf6c7cx949da9845f9ab816c70ad.plex.direct:32400/

I saw this URL pop up in the Developer Tools -> Network section of my browser when using the Plex web app at https://app.plex.tv/web/app with some security warning or other. When I browsed to that address, I got an OpenDNS page saying that it had blocked some malware!

That’s right folks, I’m also using OpenDNS and it also has some DNS Rebind protection built in! It can be completely disabled via Settings -> Security and unchecking Suspicious Responses. However to retain the layer of security OpenDNS was providing I created a “Never block” entry under Settings -> Web Content Filtering for plex.direct which seems to have done the trick.

The 2 lessons here are

  1. When employing a multi-layered approach to security and creating an exception make sure you create the exception at every layer
  2. Never assume that you know how network protocols work because someone will always find some obscure way of using totally standard stuff that makes no sense.