Yesterday, I bought a Toppy PVR when my Tivo died. The details of what it does are irrelevant (although quite fun). The important thing is that it’s very hackable – so there are lots of extensions and access programs available. While the Windows ones are typically in binary form, the Linux ones aren’t. The Toppy gives access via a USB port, and programs either access that directly or use FTP to transfer files to it via an intermediate server which basically converts FTP requests into whatever protocol the USB connection uses.
Now, I have a Linkstation with the OpenLink firmware installed on it – a hard disk running a very cut down Linux on a fairly pitiful processor. I had a few bits and bobs on it already, notably TwonkyMedia and Subversion. While TwonkyMedia was a binary installation, Subversion was built from scratch, which took a little bit of doing, mostly because the configure script required sort, which wasn’t provided in the tools for the Linkstation. Doesn’t sound too bad – you just need to download the right package to build and install sort, right? Guess what the configure script for sort requires? Yup – sort. Fortunately a friend helped me out, battling with makefiles until we had a verison of sort which worked well enough to rebuild it properly.
All of this is partly background, but partly the whole point of this blog – build annoyances.
Anyway, back to the Toppy. Over the course of the last 24 hours or so, I’ve fetched/built the following packages on the Linkstation:
- puppy – a “direct connection” client to fetch/store files
- ftpd-topfield – an FTP/USB proxying server
- toppy-web – a web application allowing (limited) remote control of the Toppy
- lighttpd – a light-weight web server to host toppy-web
- php – PHP, required as a fastcgi module for lighttpd to run toppy-web
- libxml2 – an XML library required for some PHP features
- byacc – Berkeley’s lex/yacc implementation; libxml2 needs yacc
- ncftp – a Linux CLI FTP client
(Apologies if any of the dependencies in here are wrong. It was getting pretty late by the time I’d got a php-enabled webserver…)
The build/install procedure of all of these varied immensely, and the impression I gained of the quality of the software reflects this. For those of you who don’t do Linux builds regularly, the normal procedure is to run ./configure, then make, then make install. Sounds simple, right? Well…
puppy was straightforward, although it didn’t have a make install – it just built a binary. Still, it was simple enough, and worked first time.
Running it without specifying any arguments gave a useful help message, and it was easy to get to grips with.
ftpd-topfield wasn’t bad either. This time there was a make install, and even a make test. On the other hand, running it without any arguments just returned back to the console with no hint of what was going on. Using --help produced a reasonable help message, but it’s still not clear to me what it does when you don’t specify -D for standalone mode. It could be for running from inetd, but I don’t know. Anyway, it worked pretty well, and I don’t remember any build problems, so it can’t have been that bad. I had to write my own rc.init script, but a simple one is sufficing for the moment.
toppy-web was where the problems began. Now, it doesn’t need to be built itself, but it requires a PHP-capable web server. It’s not quite “onto the next item” though, because this is the one thing I still haven’t got running – due to the configuration aspect. It comes with a bunch of sample configuration files, but you have to hunt around the web for documentation, which still seems to be flaky at best. Now, I can entirely sympathise with the developers, but the point of this blog post is the comparison of build/configure procedures. I’ll get it going at some point, I’m sure (given the effort I’ve put into the rest) but I’m not sure I’ve got the energy just yet. This is a web application – why can’t it be configured with a web page?
lighttpd – the web server itself. This wasn’t too bad, if I remember rightly. The docs have fairly good descriptions of how to configure it appropriately, including what’s required from a php build. Which brings us to…
php – oh dear. It’s never a good sign when the configure script complains about syntax errors to start with. After googling these errors and finding that other people who had received them were told that basically they could be ignored, I let the script continue. After quite a few minutes, it decided that I needed libxml2. At first, I tried to disable this – at which point (well, after starting the script again and letting it run for a few minutes) it complained that without libxml2 it wouldn’t be able to have any DOM support. I don’t know whether toppy-web requires DOM support, but it seemed like an important thing to be without. So, I decided to download libxml2 and build it. In retrospect, I should probably have looked through the toppy-web source to see whether I really needed it…
libxml2 fairly quickly announced that it needed yacc. That’s not particularly unreasonable, and I was slightly surprised not to have it. However, it was yet another step. Fortunately, byacc built and installed easily enough to not deserve its own paragraph here. Hurrah – finally I could configure and build libxml2. You wouldn’t expect that to take too long, would you? XML isn’t that hard. Hmm. It wasn’t actually difficult but I felt very sorry for the Linkstation afterwards. Most of the C files had to be compiled twice for some reason (I didn’t look at the depths of why – something to do with how the libraries are deployed, I believe) and some of them are large. Very large. xmlschemas.c is over 28,000 lines long, and the (fortunately auto-generated) testapi.c is over 50,000 lines long. C compilers are slow beasts at the best of times, and this is on a box which was really only meant to run samba. It took ages. Not only that, but the first time I had to link php with it, it failed. No idea whose fault that was (mine, php or libxml2 but it was frustrating.) Anyway, back to php…
With libxml2 built, I set php configuring and building – with the three features enabled which I knew I had to have (thanks to the lighttpd docs) and which all sound like they should be enabled by default. It got there in the end, so by the time I went to bed (very late) I had a working php-enabled web server.
Tonight, I decided to find a command-line FTP client for Linux. After finding a very useful Linux FTP client comparison page I decided to plump for ncftp. After the previous night’s frustrations, I was somewhat nervous. Fortunately, my fears were unwarranted. More than that – the authors of ncftp deserve awards.
The readme file is suitably undaunting. The configure script runs reasonably quickly. So far, so unremarkable… but the make file. Oooh… instead of showing you the exact command it’s running at any time (which in some cases for the previous packages was over a screenful for a single link, and frequently 5 or 6 lines per compile), it just tells you what it’s doing (e.g. Compiling c_utime.c... and has a colour-coded [OK] (in the same way that most Linux distributions do for starting and stopping services) for each build step. Green means a clean build, yellow means there were warnings (which are printed). At the end of the build, it shows you the files it’s built. At the end of make install it shows you the files you’ve installed. The difference in professionalism and the impression you’re left with is marked. I’ve no idea whether the ncftp guys wrote the makefile themselves or whether it’s a framework which will is generally available – but I’ve never seen anything as clean when it comes to make. Indeed, it leaves me wish that Ant builds were logged as cleanly (when they’re successful).
So, the moral of this post (which is rather longer than I’d anticipated)? Builds matter. Configuration matters. Documentation matters. There’s more to a build being good than just it working first time: giving feedback and a warm fuzzy feeling are important too. Making it look simple even if there are complex things going on behind the scenes makes life feel smoother. (I’m sure the ncftp makefile has options for seeing the commands themselves if you really need to.) I’ve understood the importance of a good build system before, but usually in terms of a developer having all the options they need. In the open source world, particularly on Linux where in many cases the end user will have to build the package in order to run it on their custom devices, the build is just as much a part of “ease of use” as the product itself – and if a user falls at the first hurdle, they’ll never see your pretty UI.