Martin Civáňandmcivan 0ee7580672 Add Slovak & Czech language dataset support (#19013) ...

### Summary

Slovak and Czech datasets are effectively BM25-blind today. The tokenizer only keeps ASCII letter runs whole, so every accented word is fragmented before indexing (`škola` -> `š kola`) and the keyword leg of hybrid search never matches. Neither language has a Snowball stemmer either, so the analyzer inherits whatever stemmer the previously processed dataset left behind.


Co-authored-by: mcivan <martin.civan@nases.gov.sk>

2026-10-15 10:24:09 +08:00

..

darts

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

opencc

fix: add buffer-length check in text.c (CWE-120) (#20803)

2026-10-12 19:13:59 +08:00

re2

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

stemmer

fix: release reused stemmer environments (#19935)

2026-09-20 18:46:32 +08:00

util

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

analyzer.h

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

CMakeLists.txt

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

dart_trie.h

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

darts_trie.cpp

Go: format code (#16813)

2026-07-10 20:30:00 +08:00

main.cpp

Go: format code (#16813)

2026-07-10 20:30:00 +08:00

Makefile

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

pcre2.h

Go: format code (#16813)

2026-07-10 20:30:00 +08:00

pcre2posix.h

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

rag_analyzer_c_api_debug.cpp

Port PR14140 and PR16881 to GO (#17102)

2026-07-21 17:46:23 +08:00

rag_analyzer_c_api.cpp

Port PR14140 and PR16881 to GO (#17102)

2026-07-21 17:46:23 +08:00

rag_analyzer_c_api.h

Port PR14140 and PR16881 to GO (#17102)

2026-07-21 17:46:23 +08:00

rag_analyzer_c_test.cpp

Go: format code (#16813)

2026-07-10 20:30:00 +08:00

rag_analyzer.cpp

Add Slovak & Czech language dataset support (#19013)

2026-10-15 10:24:09 +08:00

rag_analyzer.h

Add Slovak & Czech language dataset support (#19013)

2026-10-15 10:24:09 +08:00

string_utils.h

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

term.cpp

Go: format code (#16813)

2026-07-10 20:30:00 +08:00

term.h

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

thinc_ner.cpp

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

thinc_ner.h

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

thinc_parser.cpp

Go: format code (#16813)

2026-07-10 20:30:00 +08:00

thinc_parser.h

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

tokenizer.cpp

Go: format code (#16813)

2026-07-10 20:30:00 +08:00

tokenizer.h

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00

wordnet_lemmatizer.cpp

Go: format code (#16813)

2026-07-10 20:30:00 +08:00

wordnet_lemmatizer.h

Go: refactor (#16602)

2026-07-03 17:00:43 +08:00