(November 2013)

Building a tiny ARM-based server

About a year ago, a no longer functional NAS device made by Western Digital came into my hands - a MyBook World Edition. This is a rather old, low-powered, ARM-based file server... and even though it is considered obsolete by today's standards, I knew the moment I got it that I would very much enjoy hacking it to do my bidding :‑)

I am a programmer by vocation, but I don't really believe in the separation of programming and administration - in my humble opinion, there are a lot of advantages in practising both.

Disassembling and soldering

First, I had to break the thing down, so I could build it up to what I wanted. Following in the tradition of many engineers before me, I slaved over it for half an hour and succeeded in disassembling it - only to realize afterwards that other kind souls had documented the process. Oh well - figuring things out on your own is the best part about all this :‑)

I proceeded to remove the two drives in the machine, and after attaching them to my external USB/SATA enclosure, run a quick SMART check on them: smartctl -a /dev/sdX showed that both drives had bad sectors (Reallocated_Event_Count, Current_Pending_Sector were both non-zero - that's not the kind of drive you can trust)... I therefore cannibalized an old 160GB external USB disk, and attached it to the empty enclosure.

Since this is an embedded platform, with no VGA or serial plug on it, I needed to find a way to monitor its boot process. After a bit of Googling, it turned out that the board in fact had soldering pads for traditional RS-232 serial interfaces. The "specs" provided by the almighty Internet were simple:

• Pin 1 = +3.3V
• Pin 2 = GND
• Pin 3 = Rx
• Pin 4 = Tx

...and since we are way past the age of motherboards with RS-232 interfaces, I ordered a TTL/RS-232 cable from an online electronics store (translation: a cable that I can attach to the board on one end, and plug the other end into a normal PC USB port - where the serial interface will be accessible by any serial port program).

The board's serial interface, after soldering 3 pins and hooking 3 cables (red arrow)

Two days later, the package arrived - so I soldered the 3 pins (2,3 and 4) and hooked them up to the USB/serial cable^[1].

On my main machine (an Atom 330 based ArchLinux), I attached the USB cable, fired up a serial interface program (minicom -D /dev/ttyUSB0 -b 115200), and then powered up the little board...

The game begins :‑) The board is sitting on the now naked drive enclosure.

Nothing happened.

Hmm...

U-booting from no BIOS whatsoever

I was expecting this to show some kind of BIOS, but maybe this tiny board had no such luxury... maybe it read everything it needed from the attached drives?

I mounted the old drives up to my main PC's USB/SATA enclosure, and sure enough, I saw clear signs of a Linux-based machine: the software RAID driver in my ArchLinux detected raid devices (cat /proc/mdstat showed information about the RAID structure). Apparently, MyBook had the two drives in a RAID formation - which I proceeded to successfully mount. There were 4 partitions in each of the two drives, clearly in a RAID1 mirror formation - with the 4th and final partition being the one with the "File Server" storage area.

I proceeded to copy the first three partitions (including the partition table) to my 160GB drive (via dd). I then used fdisk to fix the size of the 4th partition to be the remaining space of my drive.

Attaching the 160GB drive and booting again, minicom logged tremendous improvement:

Welcome to minicom 2.6.2

OPTIONS: I18n 
Compiled on Mar  5 2013, 16:40:55.
Port /dev/ttyUSB0, 12:22:35

Press CTRL-A Z for help on special keys

�NASOx_0800
Mon Aug  5 21:45:27 EEST 2013


U-Boot 1.1.2 (Jan 21 2008 - 08:50:09)

U-Boot code: 48D00000 -> 48D17648  BSS: -> 48D1B2B8
RAM Configuration:
Bank #0: 48000000 32 MB
In:    serial
Out:   serial
Err:   serial
Initialising disks
No FIS received from device 1
Detecting SATA busses:
Bus 0: Found first device OK
  Device 0: Model: TOSHIBA MK1652GSX  Firm: LV010J Ser#: 29GGF8WNS
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 152627.8 MB = 149.0 GB (312581808 x 512)
  Device 1: not available

IDE read: device 0 block # 63, count 1 ... 1 blocks read: OK

...and proceeded to boot a normal Linux kernel.

Success!

Installing Debian (and some thoughts on the Unix way)

Having access to the Western Digital provided Linux of the board was a very good start. I then succeeded in using the GPL sources from Western Digital to built my own kernel and busybox-based mini-distro. I wanted more than just a toy, though ; and since I use Debian at work, I followed Mario's consolidated version of instructions ('Quick install steps'), and in 15min, installed the debootstrap-ed main parts of Debian on my 160GB drive.

<rant>

When possible, I always prefer doing things from the console - not only does it improve my knowledge of the OS I work with, it also allows me to do things over serial lines or SSH connections. In the case of this little ARM box, I applied the same knowledge that I am using for normal machines: editing /etc/network/interfaces, /etc/resolv.conf, etc.

Sadly, this is a skill that is fading away, with people becoming increasingly dependent on "wizards"... and facing the inevitable side-effect: clicking on buttons while being unaware of what is going on behind the scenes, ends up with systems that keep running slower and slower, and can only eventually be "rescued" with... clean reinstalls. In contrast, under UNIXes (Linux, the BSDs, etc) your applications' settings are simple files and folders under /etc and $HOME... Personally, I have always kept them under revision control (e.g my main dot files and vim configuration), happily migrating them over the last 15 years across the many machines I've worked with. Backing them up and restoring them, e.g. when I need to work on a brand-new machine at work, is simply a matter of checking out files and folders from my repositories...

And that alone, from what I can see in my dealings with some of my colleagues, is something they would kill for. And if they ever realized what apt-get does, and has been doing for decades...

But I digress - people reading this blog probably already know all this.

</rant>

I quickly setup the network interface, and got up to a working...

# apt-get update && apt-get upgrade

At that point, I knew the hard part was over - I now had a Debian/ARM distribution, which I could configure to do whatever I wanted.

Power? Not so much :‑)

Spec-wise...

# cat /proc/cpuinfo
Processor   : ARM926EJ-Sid(wb) rev 5 (v5l)
BogoMIPS    : 99.73
Features    : swp half thumb fastmult edsp java 
CPU implementer : 0x41
CPU architecture: 5TEJ
CPU variant : 0x0
CPU part    : 0x926
CPU revision    : 5
Cache type  : write-back
Cache clean : cp15 c7 ops
Cache lockdown  : format C
Cache format    : Harvard
I size      : 32768
I assoc     : 4
I line length   : 32
I sets      : 256
D size      : 32768
D assoc     : 4
D line length   : 32
D sets      : 256

Hardware    : Oxsemi NAS
Revision    : 0000
Serial      : 0000000000000000

...the machine definitely doesn't win any contest: it's very low powered, but that can also be seen as an advantage: nothing wrong with an always on "server" that consumes 0.7 watt when idling! With the 160GB drive added in the mix, I get up to 3-3.5W... not bad at all.

DNS

I am behind a DSL line, so my IP address is constantly changing. I needed to be able to get to my machine no matter where I am, so I opened up a (free) account on DynDNS, and set it up.

Since the board only has 32MB RAM, I chose against using the DynDNS Perl script, and instead used a native C client, inadyn:

# apt-get install gcc
...
# wget https://.../inadyn.v1.96.2.zip
# unzip inadyn.v1.96.2.zip
# cd inadyn
# make
...
# cp bin/linux/inadyn /usr/local/bin

I then configured it to run at startup, via cron:

# cat /var/spool/cron/crontabs/root 
...
@reboot /usr/local/bin/inadyn

...and set up my DynDNS credentials:

# cat /etc/inadyn.conf 
--username USER --password PASSWORD --alias UBER-SECRET.dyndns.org
    --update_period_sec 300 --background

That's it - once every 5 minutes (5 x 60 = 300), the tiny server communicates it's current IP address to DynDNS:

# host UBER-SECRET.dyndns.org
UBER-SECRET.dyndns.org has address AA.BB.CC.DD

Update, a year later: DynDNS removed its free plan, so I registered to two more free DNS providers, EntryDNS and DuckDNS. They both work via 'magic' web requests, so I simply set up two cron jobs (running every 5 min) that invoke curl, passing in the string that identifies my machine.

So, now that the ARM box had an Internet 'name', it was time to setup... "stuff" :‑)

Lighttpd

First, I installed a web server - and was now able to export everything that I wanted, to friends and family. As long as they had a browser, they could connect to my little ARM.

Initially, I went with NGINX - but since the board only has 32MB of RAM, I switched to Lighttpd, which is designed to have a smaller memory footprint.

Not that the tiny ARM server and my 700 Kbit upstream DSL speed would ever survive a Slashdoting, of course :‑) Still, there are other uses: I rsync-ed the photo folder of my (jailbroken, company-provided) iPhone, and published it in a password-protected folder...

# rsync -av mobile@iphone:/private/var/mobile/Media/DCIM/100APPLE/ \
            /var/www/site/Media/

My pictures therefore became accessible from anywhere in the world, just by browsing to my mini-server (and using the folder password). Nifty!

I added this rsync to a cronjob, so my iPhone's photos are rsync-ed automatically every night, while I sleep and the iPhone is charging.

Exim, mutt

Privacy issues not withstanding, it's nice to be able to have a mail presence that doesn't depend on others. apt-get install exim4, and my friends can now e-mail me at ttsiod@UBER-SECRET.dyndns.org. I read the mail over SSH, via 'mutt' - which runs fine even in this tiny little CPU.

<rant2>

In a world populated by self-respecting human beings, that would be the end of it ; unfortunately, even though GMail accounts accepted mail sent from me with no problems, others (e.g. Yahoo) considered me a spammer, since I was sending mail from the dark pits of hell (i.e. an IP belonging to a DSL line). How can they be sure that I am not a zombie Windows machine, infected with malware and serving The Spam Lords?

Neither SPF nor DomainKeys convinced them - so I switched my outgoing direction to use smarthost - and therefore route outgoing mail via my ISP.

(Sigh)

</rant2>

Firewalls and sslh

There are places where hitting my SSH server (e.g. to use it as a SOCKS proxy via ssh -D ...) is impossible, because there are firewalls in place that only allow HTTP traffic.

Initially, I tried exposing the server over HTTPS's port (443), but that was not enough. I ended up using sslh, which cleverly sits between a port and a number of daemons. In my case, since it speaks enough of the SSH and HTTPS protocols, it can determine when an incoming connection is hunting for SSH responses, and when for HTTPS responses - and tunnel the request to the proper local daemons (sshd, nginx). The firewall therefore sees me as a legitimate HTTPS site (which I am, thanks to Lighttpd) and lets ssh -p 443 ... pass.

Samba

I can download anything I want with my little server. rtorrent is a text-based torrent client that works fine, but since I am not exactly a mainstream guy, the things that actually interest me are usually found elsewhere. Mostly, I prefer gathering up interesting URLs and setting up scripts that use wget or youtube-dl to download them inside screen-ed sessions. This way, when I get back from work, I mount the tiny server's download folder from my jailbroken Android tablet (over Samba - i.e. with the usual mount -t cifs ...), and enjoy watching without network 'hiccups'.

In case you're wondering, I am currently watching Drew Neil's amazing Vimcasts series, and egghead.io's videos on AngularJS. Scrapping is relatively easy - I scrap enough HTML to find the video URLs, and then feed them to a wget with a rate limit - so that I don't overload the kind people that share these treasures.

From Lenny to Wheezy (Oct 2014)

The Debian rootfs from Mario was made back in 2007, in the days of Lenny. I wanted to move past that, to today's Debian Wheezy. Starting with a chroot test in a debootstrap-ed Wheezy, I realized this would not be easy:

# # From inside the little ARM machine running Debian Lenny
$ sudo debootstrap --arch armel wheezy /wheezy \
     http://ftp.whateverCountry.debian.org/debian
$ sudo -i
# mount -t proc none /wheezy/proc
# mount -t sysfs none /wheezy/sys
# mount -t devpts none /wheezy/dev/pts
# chroot /wheezy
Fatal: Kernel too old

The fatal message comes from the GLIBC of wheezy, informing that GLIBC was not compiled to work with this old kernel (2.6.17). So, can we recompile a newer kernel for this SoC?

No - not easily. Even though WD provides the sources of the custom 2.6.17 kernel it used, it never updated them to later versions, and they were never merged with the mainline kernel. The oxnas SoC is doomed to die a peaceful death...

But not before its time :‑)

Because even though I can't spend the time to port all the device drivers in the custom WD 2.6.17 to newer kernels, I can try something else: I can compile the Wheezy's GLIBC to support older kernels - including this one!

I asked for help on StackOverflow and UnixStackexchange with this, but eventually I figured it out on my own and posted my own answer. I basically used crosstool-ng to cross-compile GLIBC on my Core i5 and generated the ARM-ie version of GLIBC that could run on my 2.6.17.

After all that, I can probably proclaim my WD MyBook World Edition to be the only one on the planet running Debian Wheezy. Maybe it should now be called WD MyBook Galaxy Edition :‑)

If anyone else is interested, I can share a tarball of my updated libc files.

Conclusion - UNIX glory

If you think about it, the end result is rather amazing - and done completely over ARM processors, not Intel ones:

• I pay, energy-wise, about 3-3.5W worth of energy for an always on server that lives in my house.
• The server is always accessible even though I have a dynamic IP (via free DNS providers).
• The Lighttpd web server allows me to publish "stuff" (including a mirror of this blog).
• Since iPhones also run UNIX, I rsync my phone's photos daily to the little server, and therefore my photos are always available over any browser. Update: I also ended up using lazygal to create a nice arrangement of my photos in galleries with previews and navigation controls.
• The Exim mail server allows me to receive mail over (E)SMTP, store it in my house, and read it via SSH/mutt from wherever I am.
• When I need to, I can ssh to the tiny server and use Wake-On-LAN on my main desktop, to easily access any of the work I've done in my coding life. Or my music collection :‑)
• I can run long running web downloads (e.g. wget/youtube-dl) from within screen-ed sessions, and share them over Samba to my house-bound devices (e.g. watch movies from my jailbroken Android tablet)
• Thanks to sslh, I can bypass several firewall limitations that exist in various places, and access my tiny server.
• Thanks to crosstool-ng, I was able to cross-compile GLIBC, and bring the machine up-to-date with today's Debian packages. It is a server, after all - it needs to get security updates (unattended-upgrades).

In plain words, UNIX power - in the tiny server, in my phone, in my tablet. All 3 of them, running on ARM processors. To be honest, I didn't expect that ; 15 years ago I was sure that Intel and Microsoft had cornered the galaxy... but somehow, Linux managed to change all that.

I still have to jailbreak them and/or hack them to do my bidding, of course - the world is still not perfect.

But that's why this is fun :‑)

[1] Many thanks go to my good friend John Kydonakis, for his advice and help in everything electronics-related.