Comments on: Indexes, Hashes & Compression http://phildawes.net/blog/2007/07/26/indexes-hashes-compression/ Mostly programming with a few bits of other stuff Tue, 06 Jan 2009 19:22:20 +0000 http://wordpress.org/?v=2.1.1 By: pigalle http://phildawes.net/blog/2007/07/26/indexes-hashes-compression/#comment-41758 pigalle Thu, 26 Jul 2007 21:39:18 +0000 http://phildawes.net/blog/2007/07/26/indexes-hashes-compression/#comment-41758 re: optimal storage / read efficiency - have you tried reiser4? it does a wonderful job of not wasting disk space. a 'du -k' inside a dir used roughly the same amount of total space as a n3 serialization of the same data. said ~30 mb of data took up 230 mb on ext3. and, about one in every 5 triples is a blog post / news story text where theres a 5K chunk of text - the difference would be even more absurd if not for that. also read back is much faster than your numbers would suggest - its nowhere near 10 ms per call. what kind of drive are you using a 423 mb thing you found in a discared PC on the street? as for 'in memory' - the kernel disk cache is a great for 'in memory' - especially in the concurrency department - 10 mongrels can all benefit from it w/o a seperate memcached.. as for indexing - i havent thought about it much yet - my query engine takes about 0.1 seconds for a basic 'fetch the content, title, author, date, abstract of ___ resources sorted by ascending date'.. hopefully that can be shaved down once i learn some stuff, and your previous post is my jumping off point - thanks! oh ya. wheres your source? mines http://whats-your.name/yard re: optimal storage / read efficiency - have you tried reiser4? it does a wonderful job of not wasting disk space. a ‘du -k’ inside a dir used roughly the same amount of total space as a n3 serialization of the same data. said ~30 mb of data took up 230 mb on ext3. and, about one in every 5 triples is a blog post / news story text where theres a 5K chunk of text - the difference would be even more absurd if not for that. also read back is much faster than your numbers would suggest - its nowhere near 10 ms per call. what kind of drive are you using a 423 mb thing you found in a discared PC on the street?

as for ‘in memory’ - the kernel disk cache is a great for ‘in memory’ - especially in the concurrency department - 10 mongrels can all benefit from it w/o a seperate memcached..

as for indexing - i havent thought about it much yet - my query engine takes about 0.1 seconds for a basic ‘fetch the content, title, author, date, abstract of ___ resources sorted by ascending date’.. hopefully that can be shaved down once i learn some stuff, and your previous post is my jumping off point - thanks!

oh ya. wheres your source? mines http://whats-your.name/yard

]]> By: Seth Ladd http://phildawes.net/blog/2007/07/26/indexes-hashes-compression/#comment-41765 Seth Ladd Fri, 27 Jul 2007 01:02:51 +0000 http://phildawes.net/blog/2007/07/26/indexes-hashes-compression/#comment-41765 I've had good experience storing my data in columns. If I sort the data in each column, I'll get very good compression. I’ve had good experience storing my data in columns. If I sort the data in each column, I’ll get very good compression.

]]> By: Phil Dawes http://phildawes.net/blog/2007/07/26/indexes-hashes-compression/#comment-41786 Phil Dawes Fri, 27 Jul 2007 09:27:34 +0000 http://phildawes.net/blog/2007/07/26/indexes-hashes-compression/#comment-41786 @pigalle: thanks for the comments - I'll take a look at reiser4. The ~10ms latency is for a disk seek not a read. What sort of timings are you getting? @Seth: Cool - I'm planning on doing the same thing (have you read the research papers for cstore?). @pigalle: thanks for the comments - I’ll take a look at reiser4.
The ~10ms latency is for a disk seek not a read. What sort of timings are you getting?

@Seth: Cool - I’m planning on doing the same thing (have you read the research papers for cstore?).

]]>