Normalizer Work

Before every word check for violation, APVE normalizes it with TextNormalizer class to avoid filter bypasses like b-a-d-w-o-r-d or mрA3b.

TextNormalizer clears message from every Unicode, diacritical marks, minecraft colors (like &a), symbols, numbers. It translates homoglyphs and leet speak to their latinic identical (Or to their similar). It also removes repeated letters that appear consecutively. (e.g. baaaaaad wooooord —> bad word, insignificant fuuck —> insignificant fuck)

Additionally, it transliterates all non-Latin characters into the Latin alphabet. If a character lacks a direct Latin equivalent, the closest matching letter combination is used.

All words to dictionaries should be writted only in translit with latinic alphabet, without any symbol, number, space or uppercase letter.

Below has been showed the human readable TextNormalizer normalizing table:

    Transliteration

    а —> a       б —> b       в —> v       г —> g
    д —> d       е —> e       ё —> e       ж —> zh
    з —> z       и —> i       й —> y       к —> k
    л —> l       м —> m       н —> n       о —> o
    п —> p       р —> r       с —> s       т —> t
    у —> u       ф —> f       х —> h       ц —> c
    ч —> ch      ш —> sh      щ —> sch     ъ —> empty
    ы —> y       ь —> b       э —> e       ю —> yu
    я —> ya      

    Diacritical Transliteration
    α —> a       β —> b       γ —> g
    δ —> d       ε —> e       η —> n       θ —> th
    ι —> i       κ —> k       λ —> l       μ —> m
    ν —> v       ο —> o       π —> p       ρ —> r
    σ —> s       τ —> t       υ —> u       χ —> x
    ψ —> ps      ω —> w       ø —> o       ð —> d
    þ —> th      æ —> ae      œ —> oe      ŋ —> n
    ß —> s       ł —> l       đ —> d       ħ —> h
    ĸ —> k       ı —> i       

    Leet Speak convertion
    0 —> o       1 —> i
    3 —> z       4 —> ch      5 —> s       6 —> b
    8 —> b       @ —> a       $ —> s       ! —> i
    | —> i       + —> t       € —> e       £ —> l
    ¥ —> y       

    Symbol cleaning
    § —> empty   * —> empty   _ —> empty
    - —> empty   . —> empty   , —> empty   / —> empty
    \ —> empty   ~ —> empty   ` —> empty   ' —> empty
    " —> empty   = —> empty   ^ —> empty   # —> empty
    & —> empty   ( —> empty   ) —> empty   [ —> empty
    ] —> empty   { —> empty   } —> empty

    Patterns

    (?i)[&§]([0-9a-fk-or]|#[0-9a-f]{6}|x([&§][0-9a-f]){6})") Clears every minecraft legacy color 

    [\\p{Cf}\\p{Cc}\\u200B-\\u200D\\uFEFF\\u00AD] Clears every invisible chars

    ("\\p{M}") Clears combining marks (Diacritical)