Repurposing Old Hardware as a NAS

I’ve been helping out a charity recently who have a lot of old kit and no money to replace it. Open source software can really help organisations out like this, and let’s face it everyone wants to save some money where they can. These days there are loads of off-the-shelf distributions designed to turn commodity PC hardware into dedicated appliances, and you don’t need a degree in the Linux kernel to administer appliances like that. If you pick a well supported mature package the learning curve can be shallow and the gains huge. You’ll know if you’ve read this blog that I’m a big fan of using pfSense to turn old PC hardware into feature-rich firewall appliances. There must be something similar for storage, right?

We were talking about the charity’s storage strategy and it was established that a NAS would be beneficial to complement their aging server hardware. As they’ve got no money, they can’t just go out and buy a Synology or QNAP box. “Oh, maybe you can do something with that old box over there, we don’t use it for anything.” The box in question is an “Equiinet ServerPilot“. It was an appliance that they got as part of a contract with a local ISP to function as a squid HTTP proxy, SMTP gateway etc. That ISP has since gone bust, and Equiinet no longer support the device. When I opened it up it was clear to see why!

  • Intel Pentium Dual-Core E5200 @ 2.5GHz
  • Gigabyte GA-G31M-ES2L – 2 RAM slots, 1 PCIe x16, 1 PCIe x1, 2 PCI. 4 onboard SATA ports

  • 2 x 512MB sticks of RAM
  • 1 x 500GB SATA drive in a removable bay
  • Antec Aria Micro-ATX case

It was obvious to turn this into a NAS the base hardware would need supplementing. There’s no point in setting up a NAS with 500GB of storage and no fault tolerance. It seemed like a good starting point, there’s plenty of onboard SATA ports, the CPU is not too old. It was definitely going to need more RAM and more drives, though. I was keen to leverage some new functionality available in a few opensource filesystems these days like SSD caching – I have an HP Gen8 server which does this natively and it makes a massive difference. There’s also a conundrum around where to put the OS. A commercial NAS box would store the OS in flash, and commercials servers tend to have an internal USB port or SD card slot for operating system storage in these scenarios. You don’t really want the OS to be sat on the redundant disks as it reduces the flexibility of the drive array. You don’t really want to waste one of your 4 onboard SATA ports to a dedicated OS storage drive, and someone can easily knock out or snap off a protruding USB stick with the OS on (and this board only has USB2). I decided I’d need to get hold of the following:

  • 4GB of RAM. 2x2GB sticks of DDR2 is the most that this motherboard can support
  • 4x 1TB drives. 2TB of RAID10 will be sufficient to address current and future storage requirements, resiliency and performance.
  • ~100GB SSD for caching purposes and a method of connecting it
  • Dual-port PCIe server NIC
  • Some OS storage device

This hardware spec was tweaked a little bit once I decided what software solution I was going to use. On that note…

It turns out there are loads of options. The obvious leader in this area is FreeNAS. It’s based on FreeBSD like pfSense and really all it’s doing is providing a friendly user interface to FreeBSD’s ZFS filesystem. ZFS has all the features you’d expect from a modern filesystem like resilience, online expansion, snapshotting and advanced caching to increase performance. It’s a complex beast but FreeNAS does a great job of making it user friendly. FreeNAS is a good product and quite mature. It has other features too like hypervisor management so that you can turn your NAS into an all-in-one network appliance providing a multitude of services to your users. There is one massive caveat to all of this though – ZFS loves RAM. The more RAM you can give it, the better it performs – a good rule of thumb is 1GB of RAM per 1TB of storage. This completely rules it out of our scenario where we’re trying to repurpose old hardware. A lot of the older devices that I support are either 32bit or the motherboards only support maybe 4GB or 8GB of RAM. The RAM that ZFS needs is dedicated to ZFS, it can’t be shared with other processes. So to run a 4TB NAS using ZFS you really need at least 6GB of RAM.

A competitor to FreeNAS is Rockstor. You can consider Rockstor to be a Linux-based equivalent of FreeNAS, although it’s not quite as mature in its development and does not benefit from the same size user base. It’s Linux under the hood rather than FreeBSD (in fact it’s CentOS 7) which is advantageous to me as I’ve been administering CentOS servers for 10+ years. It uses BTRFS (developed by Oracle) rather than FreeNAS’s ZFS (developed by Sun). It has a plugin architecture based around docker. BTRFS in turn is not quite as mature as ZFS, it will only reliably work at certain RAID levels for example. It has been integrated into the Linux kernel since 2009. One really notable difference between the two is that ZFS natively supports tiered levels of caching using system RAM, SSD etc. BTRFS does not – it will rely on the Linux kernel’s ability to use free RAM as an IO cache but that’s about it. Fortunately there is some functionality in the Linux kernel since version 3.10 called bcache which does just this. Rockstor takes the standard CentOS kernel and chucks it away in favour of kernel-ml from the epel repository to add support for these advanced features.

While I was browsing the Rockstor website I had a quick flick through their shop and found this PCIe card.

I had no idea these existed, but it seemed to resolve my issues around OS storage, and an additional SATA port for my caching SSD! It’s even bootable.

  • PCIe mSATA & SATA card – £9.80
  • Samsung mSATA 32GB – £22.99

So now I have OS storage and a 5th SATA port for £32.79. Not bad! Next up I needed some hard drives. I have a few contacts in the business of stripping servers and liquidating old stock, and managed to find someone selling 1TB NetApp SATA drives. These turned out to be standard Hitachi drives with a plastic caddy around them. They’d never been used and were £15 each with a 1yr warranty!

  • 4 x NetAPP X302A 1TB SATA drives – £60

That puts me at a running total of £92.79. I still needed an SSD, some way of mounting it (the Antec Aria case only has 4 hard drive bays which would be taken up by the 4 1TB drives) and some RAM. I decided that the case had quite compromised airflow but Equiinet had managed to squeeze a fan in the front by removing a PCB with USB and audio jacks on. There was definitely space for another one! Back to eBay we go…

  • Samsung 840 EVO SSD for caching – £34.99
  • PCI Backplane Adapter for Mounting 2.5 SSD/HDDs Into Vacant PCI Slot (yes that is exactly what it sounds like) – this allows me to mount the SSD above an unused PCI slot – £4.65
  • 80mm fan – £5.56
  • Fan power splitter – £1.95
  • Short SATA cable to go between mSATA card and SSD mounted over PCI slot – £2.99
  • Screws to mount the fan – £1.80

I ordered some cheap Chinese RAM too, but the board didn’t like it and would only ever see 50% of it at a time, so I had to go to Crucial for the RAM to make sure I’d get stuff guaranteed to work:

  • Crucial 2 x 2GB DDR2-800 DIMMS – £49.19

My overall total for the 2TB NAS build from the old ServerPilot appliance is £193.92. You can’t buy a QNAP or Synology (or even Netgear) 4 drive NAS box for that, even without drives. The ones that do SSD caching are even more expensive! I’m cheating a little there as I donated an old PCIe Intel Pro/1000PT dual port adaptor I had lying around – they go for about £15 on eBay. Either way you’re looking at around £200 to repurpose the box, which I’m pretty please with. Next up, installation.

I’m not gonna talk about how to install Rockstor as there’s loads of great guides online. It’s essentially a case of writing the ISO to USB using rufus or dd and booting it. I did this with the SATA drives disconnected to ensure the OS and bootloader/MBR ended up on the right device. One thing to note is that the filesystem for the OS HAS to be BTRFS but the installer will let you choose other formats. If you choose LVM, ext3 etc the product will install and then tell you to reinstall as soon as you try and log in to it.

Installation instructions: http://rockstor.com/docs/quickstart.html

I then connected up all my drives and rebooted the device. These steps were followed to enable bcache to use the SSD as a cache for the mechanical drives: https://forum.rockstor.com/t/bcache-developers-notes/2762

The OS will see the mechanical drives, the SSD, and the virtual bcache devices created for each mechanical drive. The GUI is clever enough to figure out which devices are bcache backing devices and it won’t let you do anything with them. I created a RAID10 BTRFS pool using lzo compression – I don’t have that much CPU so lightweight compression is the order of the day. I also had to specify the nossd mount option. BTRFS sees the bcache devices as SSDs when they really aren’t. I found that manually telling it that they were not true SSD devices increased performance.

I also created some scheduled tasks to snapshot the share twice a day (I have found through trial and error that 1200 and 1900 are good times to snapshot) and to scrub the RAIDset once a week. Samba understands BTRFS snapshots and can pass them through to the Windows Previous Versions/Shadow Copy interface, so your users can retrieve old versions of their files from the OS native snapshots right from within Windows! The charity is an Active Directory user so Samba will be integrated into that when I get it on site.

The dual port NIC was configured as a bonded interface using LACP – the device will be connected to 2 switches in a stack which support LACP bonds.

And that’s it! Obviously RAID and snapshotting are no substitute for backups, another old server will be repurposed once this NAS box is in use to fulfil the backup requirement. I’m able to saturate a gigabit ethernet interface copying either to or from the NAS box, retrieve mistakenly deleted files from the snapshots with a few clicks from Windows, and I have terabytes of fault tolerant storage on the charity’s network for under £200. The device is monitored using SNMP by an open source NMS which will notify of any impending errors, Rockstor will notify of drive health from SMART status by email and storage can be expanded on the fly by swapping the drives out for larger ones. I’m not yet sure how to calculate usage on the bcache SSD to determine the perfect ratio between SSD and HDD but I’m sure there will be one!

Plus the dashboard is COOL

The Trainline Website vs Advert Blockers

We use a popular UK based website at work for booking train tickets – thetrainline.com. Recently we noticed that users were unable to perform advanced searches on the website, instead being told “Oops, something went wrong”. Not the most helpful of error messages! We could see a whole host of embedded advert/tracking/analytics websites being blocked at our proxy when users were performing the search, so I went through each one one-at-a-time to find out which was the one causing an issue. The search worked just fine off the corporate LAN, so it must have been one of our corporate security tools causing the problem. This is how it was done.

Using Chrome, browse to the site. Open the Developer tools (F12, or Menu -> More tools -> Developer tools) and look at the Network tab. Sort by the Status column to see which are failing to load:

How you do the next step depends on your network layout. As our issue was users on a corporate network behind a proxy, I tested from a computer which was not on the corporate LAN and therefore not behind the proxy. You could do it using tools your your local firewall/gateway, proxy or ZAProxy for testing. Or you could do it simply by editing your hosts file. I added each of the failed hosts to my hosts file pointing to 127.0.0.1. This replicated the fault on my computer off the corporate network. Then I removed them one at a time until the search worked. The culprit was delivery.g.switchadhub.com – adding that to our proxy whitelist fixed advanced searches on thetrainline.com – our proxy was blocking it under the category “Web Ads” as any advert blocker would.

Update: thetrainline.com don’t seem to be embedding javascript from delivery.g.switchadhub.com anymore, so this is no longer an issue. However I’m leaving this page here for future reference.

Plex, pfSense, OpenDNS and DNS Rebinding

I’m starting to use pfSense a lot more at home now, making use of the advanced security features rather than it just being a router. I got quite frustrated last night when my girlfriend and I sat down to watch a film on Plex to discover that my Xbox One (Plex Client) could not see the Plex Server that was sat right next to it, on the same network. Further investigation proved that neither could my smart TV. I could access the server remotely via the Plex web app, so it wasn’t a port forwarding issue – I had already allowed 32400/tcp through pfSense to the Plex server anyway.

It turns out it was some security protection against DNS Rebinding. DNS Rebinding attacks are where someone directs you to an address which resolves to an internal IP. Loads of appliances and broadband routers use this functionality to present you with captive portals, and so on. I had seen a lot of people having similar trouble with Plex behind pfSense, so I followed the instructions. In fact, Plex themselves even provide you with instructions on their How To Use Secure Connections support page.

Actually, you should follow the DD-WRT dnsmasq instructions if you use the “DNS Forwarder” on pfSense as this is dnsmasq. The config line goes in Services -> DNS Forwarder in the Custom Options section, like this:

So, I had configured pfSense as Plex support advised, but I was still unable to connect to the server. Eventually I read up on how the plex.direct domain name is used. When you log in to your Plex account you’re granted an authentication token which looks like any other guid (some long random hex string). The Plex service creates a wildcard HTTPS certificate for *.guid.plex.direct which is unique to your account. It then tries to access your Plex server at https://ipaddress.guid.plexdirect:32400/ substituting the dots in the IP address for dashes. So such a URL could look like

https://192-168-0-1.a6b0cf6c7cx949da9845f9ab816c70ad.plex.direct:32400/

I saw this URL pop up in the Developer Tools -> Network section of my browser when using the Plex web app at https://app.plex.tv/web/app with some security warning or other. When I browsed to that address, I got an OpenDNS page saying that it had blocked some malware!

That’s right folks, I’m also using OpenDNS and it also has some DNS Rebind protection built in! It can be completely disabled via Settings -> Security and unchecking Suspicious Responses. However to retain the layer of security OpenDNS was providing I created a “Never block” entry under Settings -> Web Content Filtering for plex.direct which seems to have done the trick.

The 2 lessons here are

  1. When employing a multi-layered approach to security and creating an exception make sure you create the exception at every layer
  2. Never assume that you know how network protocols work because someone will always find some obscure way of using totally standard stuff that makes no sense.

Modifying SquidGuard in pfSense for email notifications

I recently set up a pfSense box as a trial for transparent proxying in a school. The driver behind this was to try and meet the UK’s Prevent legislation in a cash-strapped organisation. I’m a huge advocate of pfSense, I think it’s an excellent product with a staggering feature set for free. Alternative UTM products could have run the school into thousands of pounds of hardware, support costs and subscription charges, not to mention the cost of a consultant to come in and set it up.

pfSense can use the open source Squid proxy with the SquidGuard filtering add-on as a transparent proxy. SquidGuard can make use of freely available online lists to categorise websites and control access to those categories. However one thing it doesn’t do is notify you when someone hits a blocked category. You can view the logs in realtime and the SquidGuard logs are separated off from the main Squid ones, so you can only see policy contraventions (as well as it rewriting search URLs to enforce SafeSearch, for example) but you have to be looking. A requirement from the school’s IT administrator was that they wanted an email the second someone tried to access something forbidden.

Now given that pfSense is FreeBSD under the hood that should be easy right, just call sendmail, job done. However it’s a very stripped down FreeBSD installation designed for absolutely minimal hardware requirements, so it doesn’t have sendmail. No sendmail, no postfix, no mail command. Email notifications are handled internally via PHP. There is a PHP script in place of mail (mail.php) which you can pipe text to. However the SquidGuard block page is PHP already, and it includes a tonne of pfSense functions out of the box, so why not just use the built in notification handler. Adding the following line to /usr/local/www/sgerror.php worked lovely:

send_smtp_message($cl['a'] . " tried to browse to " . $cl['u'], "SquidGuard Blacklist Hit");

Run CentOS 7 or RHEL 7 in PV mode with Citrix XenServer

Due to relying on crappy hardware in my lab I had a need to run some VMs for a new project (more on that later) on a box which did not support hardware VT. Most hypervisors won’t touch anything more than 32bit single core VMs without hardware VT – fortunately the project requirements specified the use of Citrix XenServer which is a little bit more flexible in that area. In XenServer Windows VMs have to be run in HVM mode (hardware virtualisation) but Linux VMs can be run in PV (software virtualisation) under certain conditions. PV uses bootloaders like pygrub to boot the Linux kernel rather than emulating a complete BIOS etc.

CentOS/RHEL 7 installation media has a very unusual structure, the image is partitioned like a hard drive rather than the traditional ISO9660 file layout. The OS also standardises on LVM and XFS to move away from more archaic structures which hold technology back. None of these things work nicely with Xen’s PV abilities, so Citrix recommend that you run CentOS/RHEL 7 VMs in HVM. No good for me though!

There are a number of hurdles to overcome if you want to run CentOS 7 in PV, but it’s absolutely possible. Step 1 is getting the installer to boot. The installation media is partitioned as above, and the bootloader passes parameters to the kernel to tell it where to find the rest of the installer (on the second partition of the media). If you use PV mode’s pygrub to boot the installation kernel it doesn’t know about any of those parameters so the installation will fail. In fact, you won’t even get to the installation wizard! Fortunately you can just lift the parameters out of the grub.cfg file in the ISO and paste them into XenServer:

utf8 nogpt noipv6 inst.stage2=hd:LABEL=CentOS\x207\x20x86_64

Really it’s the “inst.stage2” parameter which is the key here. OK great, now our installer’s kernel can find the 2nd stage of the installation media. However we still need to get it to stop using XFS and use something that pygrub can understand, even if it’s only for the /boot partition. So when the installer starts, just go in to the hard drive layout settings and custom partitioning… wait, it’s not there in the text mode installer! From RedHat’s own documentation:

If you install Red Hat Enterprise Linux in text mode, you can only use the default partitioning schemes described in this section. You cannot add or remove partitions or file systems beyond those that the installation program automatically adds or removes.

Alright, so we have to use the graphical installer. You can trigger this by adding the graphical keyword to the kernel parameters when you create the VM. It won’t work though, as the Xen PV emulated console isn’t capable of displaying graphics. My solution to this problem was CentOS’s VNC installation feature, whereby it will start an X server with the graphical installer running in a VNC session. You need to add more kernel parameters to specify the IP address settings, like this:

ip=<ipaddress>::<defaultgateway>:<netmask>:<hostname>:<nicid>:none nameserver=<dnsserver>

Yes there are 2 colons between the IP address and the default gateway, that’s not a typo. All of these IP settings are to be used by the VM, so set them accordingly. nicid is the OS assigned device name of the network card, usually eth0. Some example settings:

ip=192.168.0.1::192.168.0.254:255.255.255.0:controller:eth0:none nameserver=192.168.1.254

So my complete set of kernel parameters was:

graphical utf8 nogpt noipv6 inst.stage2=hd:LABEL=CentOS\x207\x20x86_64 ip=192.168.0.1::192.168.0.254:255.255.255.0:controller:eth0:none nameserver=192.168.1.254

Follow these steps to set up the VM:

  1. Create your VM using the CentOS 6 (64-bit) template in XenServer – remember it needs 3GB of RAM or the installer will get upset!
  2. At the bootloader parameters, you will need to enter the kernel parameters discussed above, obviously setting the IP address information suitably for your environment
  3. Once the installer has booted you can grab VNCViewer and point it at the IP address specified in the kernel parameters. You’ll need to add :1 on the end to specify which VNC service to connect to
  4. When you’re connected with VNC, you should see the normal CentOS/RHEL graphical installer. The changes you need to make are all in the hard drive partitioning settings. I changed /boot and / to ext4 and used traditional MBR partitioning rather than LVM. You may be able to use LVM and only change /boot to ext4, I didn’t have time to try various different configurations.
  5. Complete the installation and hey presto, x64 CentOS 7 running in PV mode in XenServer!

Hello world!

Just set the site up, because I’m about to do some interesting anti-malware stuff and I want to document it somewhere. Thought about calling it YAWPITS (yet-another-wordpress-IT-site) or some other crappy acronym but settled on the Griff theme because a) it’s my name and b) there are a tonne of free Griffin logos around that people have designed as tattoos.

Anyway stay tuned!