How to implement insert ignore unique rows only?

[d47081]

My crawler grabs links from each page and before insert, do lot of search queries to make sure that new link will be unique in database. Also it's spam query.log file.

I think it could be optimized, thoughts, instead of search request for each link, do something like:

INSERT IGNORE INTO index VALUES ...

And make CRC32 of URL as the document ID - just to simply ignore and not insert on 'primary key' already exist.

it is possible with manticoresearch-php?

[sanikolaev]

Manticore doesn't support INSERT IGNORE, but it does support REPLACE - https://github.com/manticoresoftware/manticoresearch-php/blob/master/docs/usage.md#replace

If you want INSERT IGNORE, you can ignore an error due to a duplicate document in your app.

[d47081]

Thanks.

I have last idea to implement this with try/catch construction on insert with duplicated id.