Normalize Unicode before comparing identifiers

Scope the policy

Unicode strings can have canonically equivalent representations. Python's unicodedata.normalize supports normalization forms such as NFC. Apply a chosen policy consistently at identifier creation and lookup, not only during one side of a comparison.

Example

import unicodedata

def display_key(value):
    return unicodedata.normalize("NFC", value)

assert display_key("é") == display_key("é")

Preserve important distinctions

Store the original display value if required, alongside the normalized comparison key. Decide case sensitivity separately. Compatibility normalization can intentionally merge additional characters, so do not select it merely because it appears stronger.

Acceptance and limits

Test composed and decomposed accents, case differences and visually similar letters from different scripts. The latter should not be assumed equal after normalization. Check collision handling before adding a uniqueness constraint to existing data.

Do not normalize API keys, signatures, passwords or other opaque values unless their protocol explicitly requires it. Normalization is not a complete anti-spoofing system. This recipe covers a human-readable identifier namespace whose owner has chosen canonical equivalence; other namespaces may require exact byte identity.