Comments on: Indexing structured data (again) http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/ Mostly programming with a few bits of other stuff Wed, 07 Jan 2009 03:24:07 +0000 http://wordpress.org/?v=2.1.1 By: Nick Johnson http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-30525 Nick Johnson Sun, 29 Apr 2007 21:05:33 +0000 http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-30525 I'd consider using an n-ary structure for your indexes instead of just a sorted list: Sort your index, then take every nth item (sized so the number of items returned just fits in a single disk block), and create a node from them. Recurse for each subdivision thus created. This will give you _much_ better locality of reference when you access the index, so for an infrequently used index, you won't have to read nearly as much into memory. I’d consider using an n-ary structure for your indexes instead of just a sorted list: Sort your index, then take every nth item (sized so the number of items returned just fits in a single disk block), and create a node from them. Recurse for each subdivision thus created. This will give you _much_ better locality of reference when you access the index, so for an infrequently used index, you won’t have to read nearly as much into memory.

]]> By: carmen http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-30726 carmen Mon, 30 Apr 2007 17:46:45 +0000 http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-30726 are there any stores that don't support named-graphs/quads/contexts and instead focus on providing niceties for a single 'shared context' as you put it. my biggest problem has been stuff like - how do you grab all the children of http://mysite/cars without either bypassing the RDF lib and doing "select * from resources where id like 'http://mysite.com/cars%'" since doing the same in SPARQL is just waaaay to slow, and doing a triple like: is pretty insane. don't get me started on count/group_by etc.. are there any stores that don’t support named-graphs/quads/contexts and instead focus on providing niceties for a single ’shared context’ as you put it.

my biggest problem has been stuff like - how do you grab all the children of http://mysite/cars without either bypassing the RDF lib and doing “select * from resources where id like ‘http://mysite.com/cars%’” since doing the same in SPARQL is just waaaay to slow, and doing a triple like:

is pretty insane. don’t get me started on count/group_by etc..

]]> By: Phil Dawes’ Stuff » Blog Archive » Some ideas for static triple indexing http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-30728 Phil Dawes’ Stuff » Blog Archive » Some ideas for static triple indexing Mon, 30 Apr 2007 20:54:31 +0000 http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-30728 [...] wrote a bit about representing structured data in the last post. Here’s some ideas for how I plan to index the [...] […] wrote a bit about representing structured data in the last post. Here’s some ideas for how I plan to index the […]

]]> By: Semantic Web Blog http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-44428 Semantic Web Blog Mon, 20 Aug 2007 17:48:04 +0000 http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-44428 [...] Indexing structured data (again) Now that I’m up and running and starting to get productive on Gambit-C, I’ve turned my attention back to indexing structured data. I’ve modified the tagtriples idea a bit to reflect my experience on importing data in other formats. I still think the most effective approach is not to try and … [...] […] Indexing structured data (again) Now that I’m up and running and starting to get productive on Gambit-C, I’ve turned my attention back to indexing structured data. I’ve modified the tagtriples idea a bit to reflect my experience on importing data in other formats. I still think the most effective approach is not to try and … […]

]]> By: Jonathan http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-56361 Jonathan Tue, 20 Nov 2007 17:46:12 +0000 http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-56361 check out www.coppereye.com already doing this... check out www.coppereye.com

already doing this…

]]> By: Laura Taylor http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-74618 Laura Taylor Mon, 10 Mar 2008 14:52:37 +0000 http://phildawes.net/blog/2007/04/29/indexing-structured-data-again/#comment-74618 ... or SAND Technology. Their DNA Access product not only indexes every value, it also deduplicates, compacts and compresses - in some instances the resulting footprint is less than 2% of original data (so 100 Terabytes reduced to 2 Terabytes - but still queryable with SQL). … or SAND Technology. Their DNA Access product not only indexes every value, it also deduplicates, compacts and compresses - in some instances the resulting footprint is less than 2% of original data (so 100 Terabytes reduced to 2 Terabytes - but still queryable with SQL).

]]>