Improve CJK tokenizer support

Issues Dgraph
,
1

Moved from GitHub dgraph/2801

Posted by srfrog:

The current CJK tokenizer in v1.0.10 is the one included in Bleve. It has limited support and can yield extra tokens when that aren’t needed. We need to use a CJK package/library specifically designed for CJK support that can handle these languages better.

For example, the term “first name” or “名字” is tokenized as “名”, “字”. But in this form, “字” is “name” and “字” is “word”, so we have lost “first” as a token. So a fulltext/term lookup for “字” won’t return the expected results. The expected term should have “名字”.

Some CJK packages considered are:

Refers #1421

2

ls84 commented :

Yes, this issue cause unexpected search result. it makes anyofterm(text, "名字") intoanyofterm(text, "名 字"), but hese two characters ”名“ and ”字“ together should be seem as a word.

This makes search very unperdicatble for chinese language, you can only use allofterm() for now, but it still treat “名字” as two separate words, which returns many extra search results.