importing statements at speed
Sep 22nd, 2004 by Phil Dawes
Spent the evening working on bulk assert speed for my ifp smushing store. Managed to get a 120000 statement import down from 410 seconds to 109 seconds - that’s over 1000 triples per second. Pretty good considering the rdflib rdf/xml parsing takes 59 seconds on its own - I wonder how fast this would be with raptor…
The trick was to lock the database, generate all data into files (allocating ids in memory), then bulk import using ‘LOAD DATA INFILE’.
Actually this has lead me to wonder whether there’s an rdf store profiling toolkit I should be using to test this - the performance comparison documents I’ve read use proprietary data, which means I can’t use them to compare my store. There was a post a while back with some example (copyright-free) large data files, but I can’t find it now. If anybody remembers and knows where they are, please leave a comment.
Hi Phil! Nice work. Re data, you might be thinking of http://rdfdata.org/
Re imports, I’d wondered about doing this sort of think chunked into, say, 10000 triple blocks, for large imports. But yes, having a benchmarking framework would take out some of the guesswork…
That’s it - many thanks.
Have been thinking about chunked imports too - makes sense with the bulk-import approach since most of the time is spent in the parsing/preparing rather than the actual dumping to db. Chunking it would enable parallelization of the time consuming preparation stage. Would work best with something like 3store, which stores md5 hashes for URIs in its main triples table - no centralization required to agree IDs for URIs.
Unfortunately (from this perspective), my store uses generated IDs for URIs to enable a 1:many logical-resource -> URI mapping for smushing. Would probably need to import in chunks and then reconcile IDs in a subsequent sweep.
Wow, those figures are looking good.
If you’re looking at big data it may be worth keeping one eye on developments in RFC3229 and feeds as a possible means to sync’ing big stores, see:
http://bobwyman.pubsub.com/