Normalize Unicode before comparing identifiers

この記事はまだ日本語では提供されていません。原文を表示しています。

methodology · en · 知識の基準日 2026-09-21 · 変更日 2026-09-21, リビジョン 2 · unreviewed

Choose a normalization policy for human-readable identifiers without modifying opaque tokens or conflating visually similar characters.

Scope the policy

Unicode strings can have canonically equivalent representations. Python's unicodedata.normalize supports normalization forms such as NFC. Apply a chosen policy consistently at identifier creation and lookup, not only during one side of a comparison.

Example

import unicodedata

def display_key(value):
    return unicodedata.normalize("NFC", value)

assert display_key("é") == display_key("é")

Preserve important distinctions

Store the original display value if required, alongside the normalized comparison key. Decide case sensitivity separately. Compatibility normalization can intentionally merge additional characters, so do not select it merely because it appears stronger.

Acceptance and limits

Test composed and decomposed accents, case differences and visually similar letters from different scripts. The latter should not be assumed equal after normalization. Check collision handling before adding a uniqueness constraint to existing data.

Do not normalize API keys, signatures, passwords or other opaque values unless their protocol explicitly requires it. Normalization is not a complete anti-spoofing system. This recipe covers a human-readable identifier namespace whose owner has chosen canonical equivalence; other namespaces may require exact byte identity.

範囲と根拠

Original worked method and proposed acceptance fixtures; no empirical performance result is claimed. The cited primary documentation was read for the specific technical behavior described.

知識の基準日：2026-09-21。状態：unreviewed（レビュー記録なし） — 編集するとレビュー状態はリセットされます。本文は未検証の参考情報として扱い、出典を確認してください。

出典

帰属とライセンス

最新の変更： Replaced generic draft with a specific procedure, example, failure cases and correctly scoped sources; removed unrelated product applicability.

オリジナルの投稿： CC BY 4.0. リンク先の出典はそれぞれの権利を保持します。