Comments on: Suffix array performance problems http://phildawes.net/blog/2005/01/01/suffix-array-performance-problems/ Mostly programming with a few bits of other stuff Fri, 21 Nov 2008 01:23:52 +0000 http://wordpress.org/?v=2.1.1 By: Phil Dawes http://phildawes.net/blog/2005/01/01/suffix-array-performance-problems/#comment-252 Phil Dawes Sat, 01 Jan 2005 12:37:18 +0000 http://phildawes.net/blog/2005/01/01/suffix-array-performance-problems/#comment-252 Cool - think I've cracked it. The solution was to remove the original suffix and links tables, and replace them with a single suffixes table that just maps char(4) suffixes to literals. This reduces the accuracy of the suffix match (meaning that substring searches > 4 chars need to join and filter with the nodes table (containing the literals). The width of the suffix field could be increased if this becomes a bottleneck, trading off more space for speed. Have got the query down to 30ms on my laptop (down from 82.61 seconds on the original scheme). Other queries are looking good as well (mostly in the 10-30ms mark). Should fly on the 10GB 8 proc box at work :-) The reason this is so much faster is that the mysql query optimiser gets a better view of the number of literals each substring will match, and so correctly identifies the best match order. Also, by indexing the new suffix table both ways (suffix-literal and literal-suffix), the query analyser can chose to impose other constraints in between substring filtering (e.g. reducing the literal matches to those that are rdfs:labels). Now I just need to update the veudastore code to use the new scheme... Cool - think I’ve cracked it. The solution was to remove the original suffix and links tables, and replace them with a single suffixes table that just maps char(4) suffixes to literals.
This reduces the accuracy of the suffix match (meaning that substring searches > 4 chars need to join and filter with the nodes table (containing the literals). The width of the suffix field could be increased if this becomes a bottleneck, trading off more space for speed.

Have got the query down to 30ms on my laptop (down from 82.61 seconds on the original scheme). Other queries are looking good as well (mostly in the 10-30ms mark). Should fly on the 10GB 8 proc box at work :-)

The reason this is so much faster is that the mysql query optimiser gets a better view of the number of literals each substring will match, and so correctly identifies the best match order. Also, by indexing the new suffix table both ways (suffix-literal and literal-suffix), the query analyser can chose to impose other constraints in between substring filtering
(e.g. reducing the literal matches to those that are rdfs:labels).

Now I just need to update the veudastore code to use the new scheme…

]]> By: Phil Dawes http://phildawes.net/blog/2005/01/01/suffix-array-performance-problems/#comment-256 Phil Dawes Sat, 01 Jan 2005 21:32:59 +0000 http://phildawes.net/blog/2005/01/01/suffix-array-performance-problems/#comment-256 Have implemented the changes in veudastore - released <a href="http://www.sf.net/projects/veudas">version 0.3</a>. To use this with veudas, replace the cgi-bin/veudas-0.6/veudastore directory with this release Have implemented the changes in veudastore - released version 0.3.
To use this with veudas, replace the cgi-bin/veudas-0.6/veudastore directory with this release

]]> By: Ed http://phildawes.net/blog/2005/01/01/suffix-array-performance-problems/#comment-567 Ed Thu, 10 Feb 2005 21:38:12 +0000 http://phildawes.net/blog/2005/01/01/suffix-array-performance-problems/#comment-567 Hi ! Looking for a suffix index that would tell me what programs open files with what suffix. Such that it would tell me what program created on what programs would open files say with .jpg suffix. Probably totally in the wrong place, but could you help? Ed. Hi !

Looking for a suffix index that would tell me what programs open files with what suffix. Such that it would tell me what program created on what programs would open files say with .jpg suffix.

Probably totally in the wrong place, but could you help?

Ed.

]]> By: HoangThanh http://phildawes.net/blog/2005/01/01/suffix-array-performance-problems/#comment-11744 HoangThanh Sat, 14 Oct 2006 01:58:36 +0000 http://phildawes.net/blog/2005/01/01/suffix-array-performance-problems/#comment-11744 i'm HMT i’m HMT

]]>