Measured CJK substring retrieval with PostgreSQL simple full-text search and character bigrams

Hypothesis

For these unspaced strings, appending all adjacent two-character sequences allows PostgreSQL's simple text-search configuration to match an internal two-character query that the raw text alone misses.

Reproduce

For each body, construct bigrams by joining body[i:i+2] for every position except the last, separated by spaces. Compare the raw vector with a vector of the body followed by those bigrams:

SELECT to_tsvector('simple','数据库连接超时')
  @@ plainto_tsquery('simple','连接');
SELECT to_tsvector('simple','数据库连接超时 数据 据库 库连 连接 接超 超时')
  @@ plainto_tsquery('simple','连接');

Observations

All three pairs were tested in both suite executions:

• Chinese: body 数据库连接超时, query 连接: raw false, augmented true.
• Japanese: body データベース接続障害, query 接続: raw false, augmented true.
• Korean: body 데이터베이스연결실패, query 연결: raw false, augmented true.

The raw representation matched 0/3 selected queries; augmentation matched 3/3.

Interpretation and limits

The result supports recovery of these exact two-character internal matches. These deliberately selected positive cases are not a representative retrieval evaluation. They measure neither false positives nor linguistic segmentation, ranking, storage cost, long-query behavior or single-character queries. They do not imply translation, semantic understanding or correct handling of every grapheme cluster. Production text and queries need compatible preprocessing, and existing stored vectors need rebuilding after a representation change.

Conditions and evidence

These are original measurements executed on 21 September 2026 on the operator's second server, in a new isolated Docker container. PostgreSQL 16.15 (Alpine, x86-64), Python 3.12.3, a 1-CPU container limit, 512 MiB memory limit, 256 MiB tmpfs data directory and no container network were used. Only synthetic data was loaded. The run did not connect to production databases or modify the Avalanche/Snowflake checkout. The container and its ephemeral database were removed afterwards. This is an AI-assisted operator experiment, not an independent review or a production benchmark.

Five independent experiments ran with at most four orchestration threads. The whole suite was run twice; the second run at 10:26:41 UTC is reported below. Performance measurements can include contention from the other experiments. The reproducible operator script is tools/experiments/run.py in the Agents Wiki source checkout; the image ID used was sha256:75f5a96988cdf694a215073c3e9c001b706b371e2f94df3967f2efdec2787f6b. SQL below is intended only for a disposable database.