Just upgraded my ubuntu dapper to the latest kernel 2.6.15-26 (via the auto updater thingy) and it's fixed the suspend and hibernate on my IBM X40. Hurrah!

Pretty much all the 'work' applications I've built in the last couple of years can be split into 2 parts:

1) CRUD: The database, and a UI for manipulating the data in it 2) LOGIC: The actual application functionality. (i.e. the reason for the app to exist)

These days I'm finding that the amount of actual logic/functionality I put into each app is pretty small, and so part (1) becomes a correspondingly big piece of each application. This is especially true now that we're getting used to gluing small applications together with HTTP rather than building big pieces. This appears to be a global trend - especially with web2.0 mashup buzz and lots of apps that focus on doing a single thing well.

The interesting thing is that the CRUD part is usually pretty generic. One UI fronting a bunch of crud operations on a database isn't that much different to another. Thus it makes sense to factor this out and implement the two parts as seperate physical deployments that share the same database.

Breaking the two parts out means you can choose different technologies for each one. This works really well with web applications because you can stitch the seperate deployments together into a cohesive looking site using links and a shared CSS. In my most recent app I used Django for the CRUD UI, and implemented the application logic (which didn't need much of a UI - it was an opaque service) using tomcat. In fact the actual app functionality was implemented as a single servlet.

So why Django for the CRUD UI? Well mainly because it dynamically creates a pretty sophisticated admin UI for you for free.

Aside: A few years ago I wrote a small webapp that dynamically generated a UI to edit relational database tables. Sortof similar to phpmyadmin, but it used FK information to work out relationships between entities and gave you hyperlinks and drop-down lists. This was fine, except that RDBs don't provide particularly intuitive mechanisms for dealing with complex relationships (ownership, many-many etc..), so you can only get so far using one.

Django goes further than this by moving the model definition from the database into a configuration file (well, actually a python source file but you can treat it as a config file). This allows you to augument the basic entities with relationship metadata. Django can then autogenerate tables from the config (or you can wire the model up to existing tables). However, the icing on the cake is that the config also enables you to give django UI hints for how the dynamically-generated admin GUI should look. I think this is a really powerful idea and gives Django a edge over Rails when building simple CRUD apps.

So this leads me to think that there's a killer business application here just waiting to be written: Decent dynamic RDB CRUD GUIs without code.

(yes, I know dabbledb does something like this, but I think relational DBs are key here. Both because they're well understood by 'business' developers and because they're an excellent technology for integrating data with the 'functionality' bit of an application.)

Here's some basic ideas:

• UI for creating tables and metadata. Metadata stored in the RDB with the tables.
• Auto generated sophisticated admin gui, a-la Django.
• Security model that enables access control at a data level. (e.g. think unix file permissioning: if I add a row, only people in appropriate group can edit it)

It might make sense to actually implement this with django.

(N.B. I'm not intending to actually write this app - I've got my spare-time hands full with large-scale data aggregation. Hopefully somebody else will do it. LazyWeb?)

I've been reading 'The Little Schemer' (a new edition of 'The Little LISPer') and 'The Seasoned Schemer' recently, recommended to me by JP (who (of course!) has a signed copy of the original). This is classic text, with the first edition of 'lisper' appearing in the 70s. The striking thing about these books isn't so much the content but the approach: Rather than being structured like a normal CS textbook with a descriptive chapter and then questions at the end, the whole book is a set of questions and answers from the beginning, all framed in chatty prose.

On each page the questions are on the left and the answers on the right, so you always see the answer to each question posed immediately. Each question/answer adds just enough information for you to be able to do the next (well, most of the time). I was surprised by how well this approached worked for me.

In the forward, the authors point out that the goal of the 'little schemer' is to teach you to think recursively. Proceeding through the book raised an important insight to me that I hadn't properly groked before: the distinction between having understood something and having learnt it. When reading CS textbooks I have a tendency to read the chapter and then only do a couple of the questions before getting bored and skipping to the next. This is how I proceeded through SICP for example: As soon as I understood the material, I wanted to move on.

The problem with this approach is that my brain hasn't done enough repetition to form the higher-order concepts/patterns of the material learnt, and this makes the material much more heavy going later on.

Funnily enough, the distinction has always been obvious to me when learning music (practicing my french horn), because the difference between understanding and doing in music (as in sport) is a wide chasm. For some reason I'd never really come to terms with this when learning computer science.

Anyway, the little schemer rattles through example after example, Q/A after Q/A. Having the answers next to the questions means you can proceed at impressive speed because you don't need to formulate and complete the whole answer each time - you can just pick out the patterns, the structure and artifacts that you expect to see in the answer, and then check them.

The forward to the book recommends that you don't read it all in one sitting - in fact it recommends at least 3 sittings. I suspect this is because you need rest time for your brain to construct the appropriate structures to allow you to 'think' recursively rather than merely understanding recursion. This of course allows you to proceed even faster.

Finally, don't be put off by the childishness of the artwork or prose: this shit is hardcore!. Lambda expressions are there from the beginning. Higher order functions and Currying feature early on and the book blasts through the Y-Combinator at breakneck pace towards the end. Personally I had to resort to wikipedia and various tutorials to supplement learning the Y-combinator stuff. The last chapter quickly builds a scheme interpreter capable of evaluating most of the expressions in the book.

I'm now rattling through 'The Seasoned Schemer'. I'm a quarter of the way through and have just hit continuations...

Have been pondering Malcolm and JP's posts re: Paul Graham's "What business can learn from opensource"

Having thought about it it's pretty obvious that the employer/employee thing doesn't scale very efficiently. The whole structure is top down and fundamentally limits innovators at the bottom (i.e. most of the workforce) both in their motivations and in what they're able to observe and do.

E.g. Sure, I can optimize my IT day-job to oblivion, but what ensures that the innovation translates into effect on the company bottom line? More to the point, why should I worry about the company bottom line? - my financial reward is based more on internal relationships and yearly appraisal processes than carefully managing the bottom line. AFAICS the feedback loop is too loose and slow to be anywhere near optimal.

So my question is: could investor/founder work better on this scale? If everybody's salary was scrapped and then came in to work the next day to attempt to earn money by offering services to other people/teams on a real-money basis, what would happen to the company? What structures would result? Would all the tight-feedback value get eaten up in the contractual/accountancy noise?

And on a practical note, could this be incubated on a small scale within a large company?

(caveat: am totally out of my intellectual depth)

I was surprised to find that lisp had a more powerful way of dealing with errors than exceptions (which I had previously assumed were state-of-the-art in program error handling). Lisp conditions are similar to exceptions, except that the condition handler is able, if desired, to resume execution to the erroring function by invoking one of any named 'restart points' declared by that function.

To understand why this is important, the Seibel book illustrates the case of a line parser function. This function is given a file to parse from and returns a stream of parsed tokens. The error occurs when the parser hits a line that it can't parse. At that point, the program has a number of options, including:

(1) Cancel parsing the file all together (2) Ignore the line and continue (maybe logging an error) (3) Fix up the input in some way or emit a marker in the parsed token stream.

The problem with the exception model is that an exception handler in a calling function is only in the position to do (1). By the time it receives the exception the stack has been unrolled and its associated state lost. This means that if the program wants to do any of the other possibilites the exception handling implementation code needs to be in the parsing function itself, which strongly couples its behaviour with the rest of the program.

In lisp the condition handler further up the stack can do any of the 3 options, including e.g. logging the error and then invoking a 'skip-log-entry' restart, or maybe the 'use-value' restart passing it a valid value to put in the token scheme.

I've been working through the Seibel book and I'm quite keen to draw comparisons with other languages (especially python+java), since that's what first attracted me to learn more in the first place. I thought it would be better to blog as I went along (while I'm motivated) rather than when I'd got to the end (when I might not be bothered). The obvious caveat being that I'm only half way through the book so I may have mis-understood something.

Anyway, here's my take on lisp macros:

The lisp language lacks syntactic sugar - you've basically got brackets and spaces to create structure and that's it. The lack of operators to define structure means you end up with a load of verbose text even if you're doing basic stuff.

E.g. Consider accessing an attribute of an object: Java has '.' hardcoded into the language grammar to mean 'attribute of object', whereas lisp has to make 'slot-value' function calls all over the shop:

In Java:

if (account.balance < account.minimum_balance){
    account.balance -= account.balance * 0.01;
}

In Lisp:

(when (< (slot-value account 'balance) (slot-value account 'minimum-balance))
    (decf (slot-value account 'balance) (* (slot-value account 'balance) .01)))

To combat this verbosity, lisp has macros.

Lisp macros are vaguely similar to C macros in that they allow the programmer to rewrite the code before it gets compiled/run. Unlike C macros, which operate on text and perform limited operations, lisp macros operate on parsed s-expressions and can use the full power of the lisp language to do the transformation (including utilising other functions and macros). Since lisp programs are themselves list datastructures, translating some new invented construct into vanilla lisp s-expressions is pretty straightforward.

To illustrate the power, consider the verbose OO lisp code in the above example. In order to allieviate the 'slot-value ...' redundancy somebody has implemented a 'with-slots' construct which factors out the 'slot-value' calls and allows you to use the variables names directly. Now the code looks like:

(with-slots (balance minimum-balance) account
   (when (< balance minimum-balance)
      (decf balance (* balance .01))))

In this example the args and body of the construct are passed to the with-slots macro which then generates the vanilla lisp code in the original example.

From my limited knowledge of lisp, it looks like the entire OO system is implemented as a bunch of macros - including class declarations etc.. This is quite impressive when you consider that the language itself was invented before object orientation was conceived.

This ability to add new constructs to the language appears to run deep in lisp, with proponents calling it the 'programmable programming language'. The litmus test for me will be whether this style translates into better abstractions and overall productivity in my software development.

[1] which is part of the default library

Found this really handy yesterday. It's a c binary that wraps your script and checks a lockfile before running it.

e.g. to run 'myscript' if it's not currently running:

lockrun --lockfile=/tmp/mylockfile -- myscript

The cool thing is that it uses flock to implement the lockfile, so the lock is automatically free'd when the process dies. (hence the reason for being a C binary rather than a shell script).

Lockrun is ultra handy if you need to run a command every minute, but don't want them backing up if something causes one to take longer than a minute. Also I guess you could use it to keep processes up - i.e. lockrun exits if the process is already running, but if not it starts a new one.

Found this gem of a book while browsing del.icio.us/popular/. I not sure if compsci books are getting easier to read or if I'm just getting better at reading them. Either way, this one looks fantastic.

N.B. I used to think that making books free for download would reduce the income you'd get from them, but since then I've bought a number of books that I wouldn't have done if I hadn't been able to browse them online first. (the last one was Practical Common Lisp, which is thoroughly excellent btw)

This post hits the nail on the head. I've been blathering on about some of this stuff at work* for a while: Don't tie your protocol to your programming language, protocol is the most important thing in a distributed system, make data as simple/flexible/loosely-coupled as possible, don't constrain yourself with schemas. Now I've got something to link to.

N.B. I don't think this is just applicable to protocols - also data storage. Serialized objects and O-R mapping tools can seriously louse up your data if you're not careful.

• usually to people that aren't interested

I started programming seriously when I was at school. I became seduced by the idea of writing games and ended up learning 6502 and then ARM assembler. I sort of skipped the typically-british BBC BASIC introduction because I was impatient and had read in my Dad's computer magazines that real games didn't get written in basic. Assembler was the real game-programmers language.

A couple of years later I got an x86 PC and learnt 386 assembler language with all the dirty hardware-hacking tricks that it entailed. My mum characterized the process of me programming as hours of typing occasionally punctuated by a flurry of excitement and a small graphic moving across the screen. (moving very smoothly across the screen I'd always point out).

At university I learnt C and C++, and over the first two year's transitioned slowly from one to the other. The seductive thing about C was that it would interface easily with assembler, and so I started using it as a prototying language. Eventually I wasn't writing any assembler any more.

A similar thing happened a couple of years after I left university. I started using python, mainly because I could use it as a prototyping language, and it supported all the nice OO stuff I'd got used to in C++. Eventually I stopped writing stuff in C/C++.

N.B. During this time I used Java a lot at work (after all, I did end up working for BEA/Weblogic), but I never really fell in love with this language: It wasn't as fast as C/C++, and wasn't as productive to program as python. I estimated that I could turn stuff around in python about 3-4 times faster than I could in Java.

Anyway, I've been writing my stuff almost exclusively in python for the last 6 years or so and am starting to look for the next leap. All the previous moves have been ones that involve a large productivity boost. Assembler->C->python. The main reason I didn't properly switch to Ruby is that (despite the nice block syntax) I didn't seem to get a large productivity gain. The amount of code you write appears to be pretty comparible between python and ruby - certainly not the less-than-half you get between python/ruby and java.

So now I'm left with lisp. The 1950's language that all the others are slowly emulating. Most of the features that lisp introduced have now been integrated into the mainstream dynamic languages - dynamic typing/strong typing, garbage collection, conditionals/loops, first class functions, closures. Even continuations are now getting a look-in in python and ruby. However, lispers keep going on about this macros thing - writing code to write code. Now that sound's like something worth investigating...