Comments for codejanitor

Comment on Levenshtein Distance as a MySQL Stored Function by Jerome S.

Jerome S. — Thu, 25 Feb 2010 10:38:14 +0000

Thanks a lot for this algorithm implementation.

After considering all the sources available on the web, I decided to use your implementation, and I have adapted it according to http://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance#The_algorithm so that I have a MySQL UDF for Damerau-Levenshtein.
Basically, you just have to add 3 lines between the 2 last lines of the main loop :
IF c > c_temp THEN SET c = c_temp; END IF;
## Add here
SET cv0 = CONCAT(cv0, UNHEX(HEX(c))), j = j + 1;

Here are the lines to be added :
IF (i>1 AND j>1 AND s1_char = SUBSTRING(s2, j-1, 1) AND SUBSTRING(s1, i-1, 1) = SUBSTRING(s2, j, 1) THEN
SET c_temp = CONV(HEX(SUBSTRING(cv1, j+1, 1)), 16, 10);
END IF
IF c > c_temp THEN SET c = c_temp; END IF;

Please let me know if you think it is incorrect, but in my hands it seem to work.
Have fun!

Comment on Levenshtein Distance as a MySQL Stored Function by Meer

Meer — Fri, 29 Jan 2010 18:35:29 +0000

mmmm i give up. is there any way i can post without getting messed up by this blog’s html parser?

Comment on Levenshtein Distance as a MySQL Stored Function by Meer

Meer — Fri, 29 Jan 2010 18:31:13 +0000

mmm, again, the different operator doesn’t work. changed it to !=

final code:

CREATE PROC EDIT_DISTANCE

@string1 nvarchar(3999),
@string2 nvarchar(3999),
@edit_distance int output

BEGIN

IF @string1 = @string2
BEGIN

SET @edit_distance = 0
RETURN

END

DECLARE @string1_len int,
@string2_len int,
@i int,
@j int,
@string1_char nchar,
@eval int,
@next_set varbinary(8000),
@current_set varbinary(8000)

SELECT @string1_len = LEN(@string1),
@string2_len = LEN(@string2),
@current_set = cast(0 as binary(2)),
@j = 1,
@i = 1,
@edit_distance = 0

IF @string1_len = 0

BEGIN

SET @edit_distance = @string2_len
RETURN

END

IF @string2_len = 0

BEGIN

SET @edit_distance = @string1_len
RETURN

END

IF Len(@string1) < Len(@string2) AND CharIndex(@string1, @string2) != 0

BEGIN

SET @edit_distance = Len(@string2) - Len(@string1)
RETURN

END

IF Len(@string2) @string2_len

SELECT @current_set = @current_set + CAST(@j AS binary(2)),
@j = @j + 1

WHILE @i !> @string1_len

BEGIN

SELECT @string1_char = SUBSTRING(@string1, @i, 1),
@edit_distance = @i,
@next_set = CAST(@i AS binary(2)),
@j = 1

WHILE @j !> @string2_len

BEGIN

SET @edit_distance = @edit_distance + 1

SET @eval = CAST(SUBSTRING(@current_set, 2 * @j - 1, 2) AS int) +
CASE
WHEN @string1_char = SUBSTRING(@string2, @j, 1)
THEN 0
ELSE
1
END

IF @edit_distance > @eval
SET @edit_distance = @eval

SET @eval = CAST(SUBSTRING(@current_set, 2 * @j + 1, 2) AS int)+1

IF @edit_distance > @eval
SET @edit_distance = @eval

SELECT @next_set = @next_set + CAST(@edit_distance AS binary(2)), @j = @j + 1

END

SELECT @current_set = @next_set, @i = @i + 1

END

END
GO

Comment on Levenshtein Distance as a MySQL Stored Function by Meer

Meer — Fri, 29 Jan 2010 18:24:10 +0000

looks like minor_equal operators get messed while posting. changed them to !>

–here is the code

CREATE PROC EDIT_DISTANCE

@string1 nvarchar(3999),
@string2 nvarchar(3999),
@edit_distance int output

BEGIN

IF @string1 = @string2
BEGIN

SET @edit_distance = 0
RETURN

END

DECLARE @string1_len int,
@string2_len int,
@i int,
@j int,
@string1_char nchar,
@eval int,
@next_set varbinary(8000),
@current_set varbinary(8000)

SELECT @string1_len = LEN(@string1),
@string2_len = LEN(@string2),
@current_set = cast(0 as binary(2)),
@j = 1,
@i = 1,
@edit_distance = 0

IF @string1_len = 0

BEGIN

SET @edit_distance = @string2_len
RETURN

END

IF @string2_len = 0

BEGIN

SET @edit_distance = @string1_len
RETURN

END

IF Len(@string1) < Len(@string2) AND CharIndex(@string1, @string2) 0

BEGIN

SET @edit_distance = Len(@string2) - Len(@string1)
RETURN

END

IF Len(@string2) < Len(@string1) AND CharIndex(@string2, @string1) 0

BEGIN

SET @edit_distance = Len(@string1) - Len(@string2)
RETURN

END

WHILE @j !> @string2_len

SELECT @current_set = @current_set + CAST(@j AS binary(2)),
@j = @j + 1

WHILE @i !> @string1_len

BEGIN

SELECT @string1_char = SUBSTRING(@string1, @i, 1),
@edit_distance = @i,
@next_set = CAST(@i AS binary(2)),
@j = 1

WHILE @j !> @string2_len

BEGIN

SET @edit_distance = @edit_distance + 1

SET @eval = CAST(SUBSTRING(@current_set, 2 * @j - 1, 2) AS int) +
CASE
WHEN @string1_char = SUBSTRING(@string2, @j, 1)
THEN 0
ELSE
1
END

IF @edit_distance > @eval
SET @edit_distance = @eval

SET @eval = CAST(SUBSTRING(@current_set, 2 * @j + 1, 2) AS int)+1

IF @edit_distance > @eval
SET @edit_distance = @eval

SELECT @next_set = @next_set + CAST(@edit_distance AS binary(2)), @j = @j + 1

END

SELECT @current_set = @next_set, @i = @i + 1

END

END
GO

Comment on Levenshtein Distance as a MySQL Stored Function by Meer

Meer — Fri, 29 Jan 2010 18:20:26 +0000

This is my version:

[code]
CREATE PROC EDIT_DISTANCE

@string1 nvarchar(3999),
@string2 nvarchar(3999),
@edit_distance int output

BEGIN

IF @string1 = @string2
BEGIN

SET @edit_distance = 0
RETURN

END

DECLARE @string1_len int,
@string2_len int,
@i int,
@j int,
@string1_char nchar,
@eval int,
@next_set varbinary(8000),
@current_set varbinary(8000)

SELECT @string1_len = LEN(@string1),
@string2_len = LEN(@string2),
@current_set = cast(0 as binary(2)),
@j = 1,
@i = 1,
@edit_distance = 0

IF @string1_len = 0

BEGIN

SET @edit_distance = @string2_len
RETURN

END

IF @string2_len = 0

BEGIN

SET @edit_distance = @string1_len
RETURN

END

IF Len(@string1) < Len(@string2) AND CharIndex(@string1, @string2) 0

BEGIN

SET @edit_distance = Len(@string2) - Len(@string1)
RETURN

END

IF Len(@string2) < Len(@string1) AND CharIndex(@string2, @string1) 0

BEGIN

SET @edit_distance = Len(@string1) - Len(@string2)
RETURN

END

WHILE @j <= @string2_len

SELECT @current_set = @current_set + CAST(@j AS binary(2)),
@j = @j + 1

WHILE @i <= @string1_len

BEGIN

SELECT @string1_char = SUBSTRING(@string1, @i, 1),
@edit_distance = @i,
@next_set = CAST(@i AS binary(2)),
@j = 1

WHILE @j @eval
SET @edit_distance = @eval

SET @eval = CAST(SUBSTRING(@current_set, 2 * @j + 1, 2) AS int)+1

IF @edit_distance > @eval
SET @edit_distance = @eval

SELECT @next_set = @next_set + CAST(@edit_distance AS binary(2)), @j = @j + 1

END

SELECT @current_set = @next_set, @i = @i + 1

END

END
GO
[/code]

Comment on Levenshtein Distance as a MySQL Stored Function by Henry Days

Henry Days — Fri, 04 Sep 2009 01:02:57 +0000

Good contribution!
Thanks!

Comment on Levenshtein Distance as a MySQL Stored Function by Bitte um Feedback zu Buchverwaltungsideen - php.de

Bitte um Feedback zu Buchverwaltungsideen - php.de — Wed, 29 Jul 2009 18:01:41 +0000

[...] MySQL implementiert soweit ich weiß. Daher könnte vielleicht das hier für dich interessant sein: codejanitor Levenshtein Distance as a MySQL Stored Function Die Datenbank normalisiert könnte nun so aussehen: author id | name book id | title publisher [...]

Comment on Levenshtein Distance as a MySQL Stored Function by Sean

Sean — Fri, 10 Jul 2009 05:54:05 +0000

@Aaron ‘took 1 seconds’. That’s not bad for 70,000 rows. You should avoid doing table scans if you’re planning to do this sort of thing online. Your SQL is a bit odd - you seem to be allowing for an edit distance of 10 in a 5 character string. What I do with one of my projects is search the table on string length first, to limit the number of times the levenshtein function is invoked. I use a version of Levenshtein that returns early if an edit distance limit is met, so it doesn’t check whole strings when they’re already too distant from each other. Also the data is already LOWER()ed in the database, that would save you 70,000 LOWER()s in your application, each time you check a zip code.

I’d be interested to know if the hacks get you an appreciable speedup - the damlev UDFs (and some Java) versions are on my blog - linked above.

Comment on Levenshtein Distance as a MySQL Stored Function by Aaron D. Campbell

Aaron D. Campbell — Sun, 28 Jun 2009 14:50:08 +0000

I have a zip code database for the US. When I do a query like this it took almost 30 seconds:

SELECT `zipcode` , `city` , `statecode` , (
( 5 * levenshtein(
‘85326′, LOWER( `zipcode` ) )
)
) AS ld
FROM `zipcode`
HAVING `ld` <50
ORDER BY `ld` , `city`
LIMIT 50

The same query after compiling and installing the UDF from http://joshdrew.com/ took .1 seconds. I don’t know if it’s related to the all number issue mentioned above or if it’s because it’s an 8.7M table with 70k+ rows, but it’s definitely too slow for me to use.

Comment on Levenshtein Distance as a MySQL Stored Function by Morgan

Morgan — Thu, 18 Jun 2009 06:41:41 +0000

Good work. I add this part to check those null values. Thanks

DECLARE cv0, cv1 VARBINARY(256);
SET cv1 = 0×00, j = 1, i = 1, c = 0;
IF s1 IS NULL THEN SET s1_len =0;
ELSE SET s1_len = CHAR_LENGTH(s1);
END IF;
IF s2 IS NULL THEN SET s2_len =0;
ELSE SET s2_len = CHAR_LENGTH(s2);
END IF;