FUSE and encrypted file systems

Now that I've reinstalled my colo server, I've got plenty of space (for now). Time to do something useful with it. Because my server is placed in a shared rack (and because it's an interesting exercise) I'd like to encrypt the backups I will be making on that machine.

One way to do this would be to simply encrypt the files before placing them on the server. Actually, encryption on the server itself would probably be fine to, since files will be transferred via SSH anyway. However, if I were to do this, metadata (such as file and directory names) would not be encrypted.

Another option would be encrypting the entire file system. There are several ways of doing this, but in general the encrypted file system would be placed in a file or on a raw device using some kind of loopback driver. That would mean that the data itself would need to be backed up the block level, either using tools like dumpfs or using snapshots (ZFS, LVM, etc). Since I would like to be able to use regular tools like tar to create backups, I don't consider encrypted file systems ideal for what I'd like to do with it. Since backup up a backup would not be that useful, this would only be an option for that is not stored anywhere else.

Since I don't expect performance to be a real issue, a userland file system would probably be fine. A big advantage would be that developing and testing something myself would not require root privileges and considerably lowers the chance of screwing up the server itself. FUSE looks like it might be a good choice.

FUSE was developed on Linux (requires some kernel support) and as far as I've seen there is support for running it on FreeBSD, NetBSD, OpenBSD and Mac OS X. There is also a OpenSolaris project for FUSE.

What I'm thinking about now is a FUSE file system, in which:

• file contents would be encrypted (probably AES) with the algorithm running in CBC mode
• files will optionally be compressed
• file and directory names would be stored as fixed-size identifier (not a hash of the actual name)
• the underlying file system could be of any type supported by the kernel
• the underlying storage would contain an encrypted index file which would contain the per-file encryption key, data to translate file/directory identifiers to their original name and a checksum (CRC32 or MD5 should be good enough, as it would only be used to quickly check for changes) of the original file before compression/encryption

This is somewhat similar to the way in which Apple stores music on an iPod. They move meta data to a proprietary 'database' to frustrate attempts at copying the music from an iPod to a computer. Perhaps the data is only obfuscated, but it might be encrypted with a key that's also stored in the firmware. Either way, I am sure there is software out there which has no problem extracting music from an iPod with the meta data intact.

Intel PATA/SATA compatibility mode

I've reinstalled my colocated server (Tyan Transport GS12) last night with CentOS 5.0.

I figured I could just run the machine with the case open and a DVD drive connected to install CentOS. Unfortunately, the are only two power connectors (molex plugs) for the disks, so there was no way to connect the DVD drive at the same time as both disks. I worked around that by only connecting the first disk, but I had to change some BIOS settings (specifically, I had to set the Intel SATA controller to 'compatible mode'). I could then boot from the DVD I had brought with me.

After copying a bootable image (images/diskboot.img) to the disk, I booted from the first disk after disconnecting the DVD drive and reconnecting the disk. CentOS booted just fine, but complained about the partition table. It wanted to format the drive, which was fine with me. A side effect of this is that it screws up the bootable disk image, so you cannot use it a second time and have to restore it from the OS media.

After installing the OS, which went terribly slow, I checked that the server was running all required services and that the software RAID1 devices were resynching. Then I went home.

After some sleep I logged in on my server again, only to find it absurdly slow. 20% CPU load during disk transfers and a maximum throughput of about 3.2 MB/sec. After some googling it turns out I forgot to turn that compatibility mode off again. This causes the regular IDE driver to probe and configure the disk before the SATA (ata_piix in this case) and because both drivers are messing with the same disks, neither can enable DMA. This explains why I couldn't enable DMA with hdparm.

Some more googling led me to the combined_mode kernel parameter. Supposedly the default value is both, but you can also set it to ide or libsata to allow one of these drivers to set DMA mode. I did several tests, but I could not get this to work. In the end I used the following options to prevent the IDE driver from probing the disks: hda=noprobe hdc=noprobe

Now things are back to normal. Transfers of up to 60 MB/sec and negligible CPU load.

This upside of this mistake is that I found this page. I was thinking of getting another server and turning the old one into a database server with two Western Digital Raptor disks. Unfortunately, it seems only the Linux AHCI driver has NCQ support and my GS12 has a ICH5R chipset, which isn't supported by the new AHCI driver.

Btw. I have two older 74 GB Raptor disks - the ones with 8 MB cache. I've tried to use them in my GS12, but when I do the machines becomes rather unstable. This may have been caused by CentOS 4.2 in some way. Other reasons may include too much load on the PSU or incompatibilities between the ICH5R chipset and the (older) Raptor disks.

nVidia Tesla press release

nVidia announced it's range of Tesla computing products today.

There's the C870, which is a card with specifications similar to a Quadro FX 5600 - it does seem to be quite a bit cheaper than the Quadro FX cards though.

If one of those cards is not enough, there's the D870 which houses two GPU cores in a desktop enclosure and the S870 which houses four GPU cores in a rackmountable 1U chassis. Both the D870 and S870 are connected to an interface card which you need to place in a free PCIe (x8 or x16) slot in your server or workstation. With the S870 you have the choice of connecting to it using either one (four GPUs per connection) or two interface cards (two GPUs per connection).

Using a systemboard (such as this Gigabyte board) which has four PCIe x8/x16 slots, you could conceivably connect 16 GPU cores to a single server. I would assume that both the D870 and S870 have a similar setup to the Quadro Plex, which has been around for a bit longer. You can actually get it with the Quadro FX 5600 cards, which would make it very similar (but probably more expensive and flexible, since it can actually generate a video signal) than the D870.

Luckely, you don't need these products to try things out for yourself. Unlike AMDs CTM software, nVidia's CUDA software is available as a free download. You can find it here. It does require a Gefore 8 card, but one of those can be had for under $200 - a 8800GTS (320 MB) for example.

Here is a fairly high-level article about CUDA. You may also find this nVidia presentation on gpgpu.org interesting.

Maybe I can finally do something interesting with that 8800 I have lying around :-)

Shuttle SN26P & nVidia 8800 GTS

Very recently I was able to get a hold of a new Shuttle SN26P for just under a third of what it initially cost. Ofcourse, it's been on the market for more than two years and the only reason I got this price was the fact that it was the last one that shop had on offer. Never mind.

I plugged in an Athlon X2 3800, two strips of Kingston HyperX memory (2 x 1024 MB), a Western Digital Raptor disk, a NEC DVD burner and - to finish it off - a XFX 8800 GTS (320 MB) card I'd also bought. I intended this to be a small games machine.

The system has a 350 watt power supply and after removing a small plastic clip used to bundle some cables the card fit perfectly.

I did some testing with memtest86+ to see how far I could overclock the CPU. The CPU could easily be overclocked to 2.4 GHz and beyond without breaking a sweat, but the memory was being a bit troublesome.

I installed Windows XP, heaps of drivers and then I was ready to install the video drivers. Right after the installer tried to initialize the video card, the screen went blank and after about 30 seconds the machine rebooted.

After a lot of testing, minimizing the power draw on the PSU (underclocking the CPU, using a less power-hungry harddrive, etc.) there was no way I could get the thing to run anything more than text or 2D.

After reading reviews I had concluded that the machine should be able to run a single 8800 GTS, since it was appearently able to run one or two 7800- or 7900-class cards. I was wrong.

However, in another system with a fanless 300 watt Silverstone ST30F it worked without any problems. I guess not all power supplies are created equal. Appearently there is a 400 watt replacement PSU (part number PS-400W) but it costs almost as much as I paid for the whole system and I have no guarantee it will work.

Oh well, you win some and you loose some. For the price it is a really nice system. Not the most quiet system I have, but it looks nice, it is cleverly designed and the system is easy to work with (upgrades, etc).

Network appliances

A group of colleagues all bought a Soekris a while back. Some are using it to experiment with CARP (possibly inspired by the rather graphic EuroBSDCon 2005 presentation by Ryan McBride), others run M0N0wall/pfSense or fairly standard Linux distribution to shape their internet connections. At least one colleague connected USB speakers and uses it as a fancy alarmclock.

Personally, I find the the different units available from RouterBoard more interesting, as these have more CPU horsepower and are (IMHO) more expandable. I don't have any experience with the, supposedly, carrier-grade RouterOS from MikroTik, but I know I can get Linux on such a board to do what I need.

Unfortunately, sometimes you need something faster. You could go for a 1U server with a PCI-X or PCIe slot and plug in a dual- or quad-port Intel Ether Express Pro 1000 card. On the other hand, both iGoLogic and Arbor (not to be confused with Arbor Networks) have some pretty nice network appliances.

iGoLogic sells two models: the i9043N is based on a low-power Celeron M/Pentium M processor and has four 10/100 Mbps ports, while the i9011N is powered by a Pentium4 (it is not clear to me whether a dual-core Pentium D would be supported) and has four gigabit ports. The smaller unit should make a nice firewall/VPN solution for smaller networks - more or less the market segment being targeted by, for instance, the smaller Juniper (Netscreen) or Sonicwall units. The gigabit unit might fit nicely into a Linux Virtual Server (LVS) or CARP setup.

Arbor sells a few more different units. Their line-up starts with the VIA Eden-based MBX-1610 which has three 10/100 Mbps ports up to the Pentium D-based MBX-1736 which has six gigabit ports.

Depending on the price, products from both companies might be interesting.

Update: I just came across the FabiaTech FX5620 unit sold by LinITX and several units sold by Nexcom. The FX5620 is has a 1 GHz VIA C3 processor, five 10/100 Mbps and one gigabit port - unfortunately it's using Realtek chips. Nexcom has units ranging from the DNA 730 (with it's 667 MHz XScale, three 10/100 Mbps ports and an 8-port switch) to the NSA 2189 which can have two quad-core Xeons, up to 24 gigabit ports and has redundant power supplies and disks.