The Trainline Website vs Advert Blockers

We use a popular UK based website at work for booking train tickets – thetrainline.com. Recently we noticed that users were unable to perform advanced searches on the website, instead being told “Oops, something went wrong”. Not the most helpful of error messages! We could see a whole host of embedded advert/tracking/analytics websites being blocked at our proxy when users were performing the search, so I went through each one one-at-a-time to find out which was the one causing an issue. The search worked just fine off the corporate LAN, so it must have been one of our corporate security tools causing the problem. This is how it was done.

Using Chrome, browse to the site. Open the Developer tools (F12, or Menu -> More tools -> Developer tools) and look at the Network tab. Sort by the Status column to see which are failing to load:

How you do the next step depends on your network layout. As our issue was users on a corporate network behind a proxy, I tested from a computer which was not on the corporate LAN and therefore not behind the proxy. You could do it using tools your your local firewall/gateway, proxy or ZAProxy for testing. Or you could do it simply by editing your hosts file. I added each of the failed hosts to my hosts file pointing to 127.0.0.1. This replicated the fault on my computer off the corporate network. Then I removed them one at a time until the search worked. The culprit was delivery.g.switchadhub.com – adding that to our proxy whitelist fixed advanced searches on thetrainline.com – our proxy was blocking it under the category “Web Ads” as any advert blocker would.

Update: thetrainline.com don’t seem to be embedding javascript from delivery.g.switchadhub.com anymore, so this is no longer an issue. However I’m leaving this page here for future reference.

Plex, pfSense, OpenDNS and DNS Rebinding

I’m starting to use pfSense a lot more at home now, making use of the advanced security features rather than it just being a router. I got quite frustrated last night when my girlfriend and I sat down to watch a film on Plex to discover that my Xbox One (Plex Client) could not see the Plex Server that was sat right next to it, on the same network. Further investigation proved that neither could my smart TV. I could access the server remotely via the Plex web app, so it wasn’t a port forwarding issue – I had already allowed 32400/tcp through pfSense to the Plex server anyway.

It turns out it was some security protection against DNS Rebinding. DNS Rebinding attacks are where someone directs you to an address which resolves to an internal IP. Loads of appliances and broadband routers use this functionality to present you with captive portals, and so on. I had seen a lot of people having similar trouble with Plex behind pfSense, so I followed the instructions. In fact, Plex themselves even provide you with instructions on their How To Use Secure Connections support page.

Actually, you should follow the DD-WRT dnsmasq instructions if you use the “DNS Forwarder” on pfSense as this is dnsmasq. The config line goes in Services -> DNS Forwarder in the Custom Options section, like this:

So, I had configured pfSense as Plex support advised, but I was still unable to connect to the server. Eventually I read up on how the plex.direct domain name is used. When you log in to your Plex account you’re granted an authentication token which looks like any other guid (some long random hex string). The Plex service creates a wildcard HTTPS certificate for *.guid.plex.direct which is unique to your account. It then tries to access your Plex server at https://ipaddress.guid.plexdirect:32400/ substituting the dots in the IP address for dashes. So such a URL could look like

https://192-168-0-1.a6b0cf6c7cx949da9845f9ab816c70ad.plex.direct:32400/

I saw this URL pop up in the Developer Tools -> Network section of my browser when using the Plex web app at https://app.plex.tv/web/app with some security warning or other. When I browsed to that address, I got an OpenDNS page saying that it had blocked some malware!

That’s right folks, I’m also using OpenDNS and it also has some DNS Rebind protection built in! It can be completely disabled via Settings -> Security and unchecking Suspicious Responses. However to retain the layer of security OpenDNS was providing I created a “Never block” entry under Settings -> Web Content Filtering for plex.direct which seems to have done the trick.

The 2 lessons here are

  1. When employing a multi-layered approach to security and creating an exception make sure you create the exception at every layer
  2. Never assume that you know how network protocols work because someone will always find some obscure way of using totally standard stuff that makes no sense.

Modifying SquidGuard in pfSense for email notifications

I recently set up a pfSense box as a trial for transparent proxying in a school. The driver behind this was to try and meet the UK’s Prevent legislation in a cash-strapped organisation. I’m a huge advocate of pfSense, I think it’s an excellent product with a staggering feature set for free. Alternative UTM products could have run the school into thousands of pounds of hardware, support costs and subscription charges, not to mention the cost of a consultant to come in and set it up.

pfSense can use the open source Squid proxy with the SquidGuard filtering add-on as a transparent proxy. SquidGuard can make use of freely available online lists to categorise websites and control access to those categories. However one thing it doesn’t do is notify you when someone hits a blocked category. You can view the logs in realtime and the SquidGuard logs are separated off from the main Squid ones, so you can only see policy contraventions (as well as it rewriting search URLs to enforce SafeSearch, for example) but you have to be looking. A requirement from the school’s IT administrator was that they wanted an email the second someone tried to access something forbidden.

Now given that pfSense is FreeBSD under the hood that should be easy right, just call sendmail, job done. However it’s a very stripped down FreeBSD installation designed for absolutely minimal hardware requirements, so it doesn’t have sendmail. No sendmail, no postfix, no mail command. Email notifications are handled internally via PHP. There is a PHP script in place of mail (mail.php) which you can pipe text to. However the SquidGuard block page is PHP already, and it includes a tonne of pfSense functions out of the box, so why not just use the built in notification handler. Adding the following line to /usr/local/www/sgerror.php worked lovely:

send_smtp_message($cl['a'] . " tried to browse to " . $cl['u'], "SquidGuard Blacklist Hit");

Run CentOS 7 or RHEL 7 in PV mode with Citrix XenServer

Due to relying on crappy hardware in my lab I had a need to run some VMs for a new project (more on that later) on a box which did not support hardware VT. Most hypervisors won’t touch anything more than 32bit single core VMs without hardware VT – fortunately the project requirements specified the use of Citrix XenServer which is a little bit more flexible in that area. In XenServer Windows VMs have to be run in HVM mode (hardware virtualisation) but Linux VMs can be run in PV (software virtualisation) under certain conditions. PV uses bootloaders like pygrub to boot the Linux kernel rather than emulating a complete BIOS etc.

CentOS/RHEL 7 installation media has a very unusual structure, the image is partitioned like a hard drive rather than the traditional ISO9660 file layout. The OS also standardises on LVM and XFS to move away from more archaic structures which hold technology back. None of these things work nicely with Xen’s PV abilities, so Citrix recommend that you run CentOS/RHEL 7 VMs in HVM. No good for me though!

There are a number of hurdles to overcome if you want to run CentOS 7 in PV, but it’s absolutely possible. Step 1 is getting the installer to boot. The installation media is partitioned as above, and the bootloader passes parameters to the kernel to tell it where to find the rest of the installer (on the second partition of the media). If you use PV mode’s pygrub to boot the installation kernel it doesn’t know about any of those parameters so the installation will fail. In fact, you won’t even get to the installation wizard! Fortunately you can just lift the parameters out of the grub.cfg file in the ISO and paste them into XenServer:

utf8 nogpt noipv6 inst.stage2=hd:LABEL=CentOS\x207\x20x86_64

Really it’s the “inst.stage2” parameter which is the key here. OK great, now our installer’s kernel can find the 2nd stage of the installation media. However we still need to get it to stop using XFS and use something that pygrub can understand, even if it’s only for the /boot partition. So when the installer starts, just go in to the hard drive layout settings and custom partitioning… wait, it’s not there in the text mode installer! From RedHat’s own documentation:

If you install Red Hat Enterprise Linux in text mode, you can only use the default partitioning schemes described in this section. You cannot add or remove partitions or file systems beyond those that the installation program automatically adds or removes.

Alright, so we have to use the graphical installer. You can trigger this by adding the graphical keyword to the kernel parameters when you create the VM. It won’t work though, as the Xen PV emulated console isn’t capable of displaying graphics. My solution to this problem was CentOS’s VNC installation feature, whereby it will start an X server with the graphical installer running in a VNC session. You need to add more kernel parameters to specify the IP address settings, like this:

ip=<ipaddress>::<defaultgateway>:<netmask>:<hostname>:<nicid>:none nameserver=<dnsserver>

Yes there are 2 colons between the IP address and the default gateway, that’s not a typo. All of these IP settings are to be used by the VM, so set them accordingly. nicid is the OS assigned device name of the network card, usually eth0. Some example settings:

ip=192.168.0.1::192.168.0.254:255.255.255.0:controller:eth0:none nameserver=192.168.1.254

So my complete set of kernel parameters was:

graphical utf8 nogpt noipv6 inst.stage2=hd:LABEL=CentOS\x207\x20x86_64 ip=192.168.0.1::192.168.0.254:255.255.255.0:controller:eth0:none nameserver=192.168.1.254

Follow these steps to set up the VM:

  1. Create your VM using the CentOS 6 (64-bit) template in XenServer – remember it needs 3GB of RAM or the installer will get upset!
  2. At the bootloader parameters, you will need to enter the kernel parameters discussed above, obviously setting the IP address information suitably for your environment
  3. Once the installer has booted you can grab VNCViewer and point it at the IP address specified in the kernel parameters. You’ll need to add :1 on the end to specify which VNC service to connect to
  4. When you’re connected with VNC, you should see the normal CentOS/RHEL graphical installer. The changes you need to make are all in the hard drive partitioning settings. I changed /boot and / to ext4 and used traditional MBR partitioning rather than LVM. You may be able to use LVM and only change /boot to ext4, I didn’t have time to try various different configurations.
  5. Complete the installation and hey presto, x64 CentOS 7 running in PV mode in XenServer!

Hello world!

Just set the site up, because I’m about to do some interesting anti-malware stuff and I want to document it somewhere. Thought about calling it YAWPITS (yet-another-wordpress-IT-site) or some other crappy acronym but settled on the Griff theme because a) it’s my name and b) there are a tonne of free Griffin logos around that people have designed as tattoos.

Anyway stay tuned!