Unicode – Software Online Review

Unicode Normalization Explained: NFC vs NFD and Why Text Comparison Bugs Happen

Posted on November 5, 2026 by Filip Keser

AT A GLANCE

Form	What it does	Typical use
NFC (Normalization Form C)	Canonical decomposition, then canonical composition — combines base letters with marks into single codepoints where possible	Default for storage, URLs, and most text on the web
NFD (Normalization Form D)	Canonical decomposition only — splits precomposed characters into base + combining marks	Text analysis, diacritic-insensitive search, some macOS filesystem APIs
NFKC (Normalization Form KC)	Compatibility decomposition, then canonical composition — also folds stylistic/compatibility variants (e.g. ligatures, roman numerals) into plain equivalents	Search, security checks on identifiers, loose matching
NFKD (Normalization Form KD)	Compatibility decomposition only, no recomposition	Aggressive text normalization before further processing

Short Verdict

Unicode allows more than one sequence of codepoints to render as the same visible string. “café” typed on one system can be a single precomposed character (U+00E9, LATIN SMALL LETTER E WITH ACUTE) and on another a base letter plus a separate combining accent (U+0065 U+0301). Both look identical. Neither is “more correct.” But a naive == comparison, a database unique index, or a cache key built from the raw string will treat them as different values. The fix, per the Unicode Standard itself, is to normalize both sides to the same form — usually NFC — before comparing, storing, hashing, or indexing. This is not an edge case for CJK or emoji only; it hits any text with accents, umlauts, or combining marks, which means most non-English (and plenty of English) text that passes through more than one system.

What Unicode Normalization Actually Is

Unicode Standard Annex #15 (UAX #15), the Unicode Normalization Forms specification, defines the problem as canonical equivalence: “a fundamental equivalency between characters or sequences of characters” that represent the same abstract character and are supposed to look and behave identically when rendered correctly. A weaker relationship, compatibility equivalence, covers characters that are related in meaning but may look or behave differently — the roman numeral “Ⅰ” (U+2160) and the letter “I” (U+0049), for instance, which share meaning but not necessarily rendering.

Normalization resolves this by giving every equivalent sequence a single, well-defined representative form. UAX #15 defines four:

NFD — Canonical Decomposition: break every precomposed character into its base character plus combining marks.
NFC — Canonical Decomposition followed by Canonical Composition: decompose, then recombine every base-plus-mark sequence back into a single precomposed character wherever one exists.
NFKD — Compatibility Decomposition: like NFD, but also expands compatibility characters (ligatures, fullwidth forms, superscripts, roman numerals) into their plain equivalents.
NFKC — Compatibility Decomposition followed by Canonical Composition: NFKC and NFC differ only in that decomposition step.

The mechanics behind composition and decomposition matter for correctness, not just trivia. Every character carries a Canonical_Combining_Class; characters with class zero (“starters”) anchor a run, and non-zero-class combining marks following a starter get sorted by that class under the Canonical Ordering Algorithm — so the same accent marks typed in a different order still normalize identically. The Canonical Composition Algorithm then scans left to right and merges any eligible starter+mark pair into its precomposed codepoint, except for a documented set of characters that UAX #15 excludes from composition entirely (so they never get recombined, by design).

Two properties from the spec matter in practice. First, normalization is stable across Unicode versions: a string normalized under one version of Unicode stays normalized under every later version, which is why it’s safe to normalize once and store the result long-term. Second, and easy to miss, none of the four forms is closed under string concatenation — joining two independently-normalized strings (e.g. gluing a normalized prefix to a normalized suffix at a boundary) can produce a string that is no longer normalized, so systems that build text by concatenation need to re-normalize after assembly, not just at the edges.

Where This Breaks Real Systems

The classic failure mode: a user registers an account with a name typed on a Mac, which may produce decomposed sequences from certain input methods; a second system (a database, a search index, an API from a different vendor) receives the same visual string but precomposed, because its OS or browser normalized on input. "café" == "café" returns false even though a human reading both sees the same word. This shows up as: duplicate user accounts that look identical, failed login/username lookups, broken full-text search, HMAC or hash mismatches on supposedly-identical payloads, and inconsistent sort order. It is especially common at the boundary between macOS (whose HFS+/APFS filesystem APIs have historically normalized filenames toward a decomposed form) and systems on Linux or Windows that leave filenames exactly as given, byte for byte.

NFKC/NFKD introduce a second, security-relevant failure mode in the opposite direction: because they fold visually or semantically similar characters together, they can be used to defeat naive blocklists (a filter that blocks the ASCII word “admin” might not catch a fullwidth or ligature variant until it’s been NFKC-normalized) — but the same folding is exactly why UAX #15 warns compatibility forms “must not be blindly applied to arbitrary text”: they are lossy with respect to formatting and can change meaning (e.g. collapsing a trademark symbol into plain letters) in contexts where that distinction matters, such as user-facing display or legal text.

A Minimal, Correct Example

Python’s standard library implements all four forms via unicodedata.normalize(form, text), with form one of "NFC", "NFD", "NFKC", "NFKD":

import unicodedata

composed   = "caf\u00e9"          # NFC: single codepoint U+00E9 (é)
decomposed = "cafe\u0301"         # NFD: "e" + combining acute U+0301

print(composed == decomposed)
# False -- same visible text, different codepoint sequence

print(unicodedata.normalize("NFC", decomposed) == composed)
# True -- normalize both sides before comparing

print(unicodedata.is_normalized("NFC", composed))
# True -- cheap check before doing the full normalization pass

The Python docs flag the same root issue directly: “Even if two unicode strings are normalized and look the same to a human reader, if one has combining characters and the other doesn’t, they may not compare equal.” The fix is always the same shape — normalize to one form at the boundary (on input, before storage, before comparison) rather than trying to special-case every possible equivalent sequence.

Trade-offs: NFC vs. NFD vs. NFKC/NFKD

NFC pros: most compact representation; what most web content, URLs, and the W3C’s own guidance for HTML/XML use by default; round-trips cleanly through most legacy systems that expect precomposed accented letters.
NFC cons: not every combining sequence has a precomposed codepoint, so some marks never recompose — NFC does not guarantee a mark-free string.
NFD pros: useful when you need to isolate or strip diacritics (e.g. building an accent-insensitive search index by decomposing, then dropping combining-mark codepoints).
NFD cons: larger byte size; many older libraries and protocols assume precomposed input and mishandle bare combining marks.
NFKC/NFKD pros: good for loose matching, search, and some identifier-spoofing defenses, since it collapses compatibility variants that would otherwise bypass exact-match filters.
NFKC/NFKD cons: lossy and one-directional — you cannot recover the original formatting distinctions, so it is unsafe for anything that must preserve exact user input (legal names, file content, cryptographic payloads).

Who Should Use Which Form

Use NFC as the default for anything you store, compare, hash, index, or put in a URL — it’s the form most interoperable systems already expect. Use NFD only when you specifically need to manipulate base characters and marks separately, such as building a diacritic-folding search feature, or when working directly with macOS filesystem APIs that normalize toward it. Use NFKC/NFKD only for non-reversible operations like fuzzy search, spam/abuse filtering, or identifier collision checks — never for data you need to preserve exactly, and never as a silent step before storage, since you’d be discarding information users typed on purpose.

Bottom Line

Most “duplicate account” and “search can’t find text that’s clearly there” bugs involving accented or non-Latin text trace back to one root cause: two systems chose different, equally valid, codepoint sequences for the same visible string, and nobody normalized before comparing. Pick NFC as your storage and comparison default, normalize at every system boundary (not just once at the start), and reach for NFKC/NFKD deliberately and only for lossy, non-reversible matching — never as an invisible default.

FAQ

Is NFC “more correct” than NFD? No. Both represent the exact same abstract text under Unicode’s canonical equivalence; NFC is simply the more common interoperability default because it’s more compact and matches what most legacy encodings and the web already expect.

Does normalizing fix emoji or CJK comparison bugs too? It fixes any text built from combining sequences or compatibility variants that have multiple valid representations — this includes many CJK compatibility ideographs and some emoji ZWJ sequences, though the exact equivalence rules differ by category and UAX #15 should be checked for edge cases.

Can I normalize once and never worry about it again? Normalization is stable across Unicode versions, so a string normalized today stays normalized later — but concatenating two normalized strings is not guaranteed to produce a normalized result, so re-normalize after any operation that joins or edits text, not just at ingestion.

Is NFKC safe for security filtering (e.g. blocking reserved usernames)? It helps catch some lookalike-character bypasses, but it is lossy by design and should be paired with other checks (such as Unicode confusables detection), not treated as a complete defense on its own.

Sources

Engineering

Unicode for Developers: Code Points, Grapheme Clusters, and Normalization Without the Bugs

Posted on October 28, 2026October 28, 2026 by Filip Keser

Most string bugs that reach production come from one wrong assumption: that one “character” equals one unit of storage. It does not. A single letter can be one code point or two, a single emoji can be eight UTF-16 units, and two strings that look identical can fail an equality check. This deep dive explains the three layers you need to keep straight (code units, code points, and grapheme clusters), how normalization fixes the equality problem, and which built-in tools handle all of it correctly.

At a Glance

Concept	What it means
Code unit	The storage unit of an encoding; JavaScript’s `length` counts UTF-16 code units
Code point	A single Unicode value such as U+00F1; may take one or two UTF-16 units
Grapheme cluster	What a user perceives as one character; may span several code points
Normalization	Converting text to a standard form (NFC, NFD, NFKC, NFKD) so equivalent strings compare equal
Governing specs	UAX #29 (text segmentation) and UAX #15 (normalization forms)
Built-in tools (JavaScript)	`String.prototype.normalize()` and `Intl.Segmenter`

Short Verdict

Store and transmit text as-is, normalize to NFC before comparing or indexing it, and use grapheme segmentation, not length or code point counts, whenever the result is shown to a person: truncation, cursor movement, character limits, and “reverse this string.” Use compatibility normalization (NFKC/NFKD) only for search and matching, never to rewrite what the user typed.

Layer 1 and 2: Code Units vs Code Points

Unicode assigns every character a code point, written like U+00F1. Encodings then store those code points as code units: UTF-8 uses 8-bit units, UTF-16 uses 16-bit units. JavaScript strings are UTF-16, so length counts 16-bit units, not characters. Code points above U+FFFF, which include most emoji, need two UTF-16 units each (a surrogate pair). Splitting a string in the middle of a surrogate pair produces broken text, which is why naive slice() truncation can corrupt emoji.

Layer 3: Grapheme Clusters

Unicode Standard Annex #29, the specification for text segmentation (its current revision covers Unicode 18.0.0), puts the core problem plainly: a single code point is often, but not always, the same as a basic unit of a writing system. What users see as one character is approximated by a grapheme cluster. A letter followed by a combining accent is one cluster made of two code points. A Hangul syllable can be one precomposed code point or a sequence of conjoining jamo. And per UAX #29, each emoji sequence in the recommended (RGI) emoji set is a single grapheme cluster, even when it is built from several emoji joined by zero-width joiners.

UAX #29 defines two kinds of clusters. Legacy grapheme clusters are a base character plus continuing marks. Extended grapheme clusters, the default, also absorb spacing combining marks such as many Indic vowel signs, and the annex recommends them over legacy clusters except where the legacy variant is specifically required. Two useful properties: cluster boundaries are the same whether text is in NFC or NFD, and clusters do not necessarily match what the font draws, since a ligature like “fi” may render as one glyph while still being two clusters.

In JavaScript, Intl.Segmenter implements this segmentation and is marked Baseline widely available on MDN, supported across browsers since April 2024. A family emoji built from three emoji and two zero-width joiners shows the gap between storage and perception:

const family = "\u{1F468}\u200D\u{1F469}\u200D\u{1F467}"; // man ZWJ woman ZWJ girl
console.log(family.length);            // 8 UTF-16 code units
console.log([...family].length);       // 5 code points

const seg = new Intl.Segmenter("en", { granularity: "grapheme" });
console.log([...seg.segment(family)].length); // 1 user-perceived character

The same API supports "word" and "sentence" granularity. That matters because, as UAX #29 notes, word boundaries are not restricted to whitespace and punctuation, and languages such as Chinese, Japanese, and Thai do not separate words with spaces at all, so split(" ") fails on them.

Normalization: Making Equal Strings Equal

The letter “ñ” can be stored as the single code point U+00F1 or as “n” (U+006E) followed by a combining tilde (U+0303). The two render identically, but a byte-for-byte comparison says they differ. Unicode calls them canonically equivalent: they represent the same abstract character and should look and behave the same. Normalization picks one standard representation so comparisons work.

Form	What it does	Use it for
NFC	Canonical decomposition, then canonical composition (composed form)	The default for storage keys, comparison, and most text processing
NFD	Canonical decomposition (decomposed form)	Processing that wants base letters and marks separated, such as accent stripping
NFKC	Compatibility decomposition, then canonical composition	Search and matching, where “ﬀ” should match “ff”
NFKD	Compatibility decomposition	Search indexing that also wants marks separated

const a = "\u00F1";        // precomposed
const b = "\u006E\u0303";  // n + combining tilde
console.log(a === b);                                   // false
console.log(a.normalize("NFC") === b.normalize("NFC")); // true

const lig = "\uFB00";      // the "ff" ligature
console.log(lig.normalize("NFKC"));                     // "ff"

Compatibility equivalence is weaker than canonical equivalence: all canonically equivalent sequences are also compatible, but not the other way round. Compatible sequences should be treated alike in some applications but not necessarily all. MDN’s guidance captures the trade-off well: compatibility normalization suits search, because a user can find the ligature by typing “f”, but may not suit display, because it changes how the text looks. That is why you should normalize a copy for matching rather than overwrite what the user entered.

Common Bugs and Their Fixes

Character limits that reject valid input. A 20-character username limit enforced with length counts emoji as several characters. Count grapheme clusters for user-facing limits, and enforce a separate byte limit for storage.
Truncation that breaks emoji or accents. Cut at grapheme boundaries, not at a fixed code unit index.
Duplicate accounts or failed lookups. “Amélie” typed on two keyboards can produce two different code point sequences. Normalize to NFC before using text as a key.
String reversal that mangles text. Reverse the array of grapheme segments, not the code units.
Word counts that fail in some languages. Use word segmentation instead of splitting on spaces.

Bottom Line

Keep three layers in mind: code units for storage limits, code points for encoding logic, and grapheme clusters for anything a person sees. Normalize to NFC before comparing, reserve NFKC/NFKD for search, and lean on the platform: normalize() and Intl.Segmenter already implement the Unicode rules so you do not have to.

Frequently Asked Questions

Why is an emoji’s length more than 1 in JavaScript? length counts UTF-16 code units; emoji above U+FFFF take two units each, and joined sequences add zero-width joiners.
Which normalization form should I store? NFC is the usual choice for storage and comparison; it is also the default for normalize().
When should I use NFKC? For search and matching, not for display or for rewriting user input.
How do I count characters the way users see them? Use grapheme segmentation, such as Intl.Segmenter with granularity: "grapheme".