Category: Engineering

Most string bugs that reach production come from one wrong assumption: that one “character” equals one unit of storage. It does not. A single letter can be one code point or two, a single emoji can be eight UTF-16 units, and two strings that look identical can fail an equality check. This deep dive explains the three layers you need to keep straight (code units, code points, and grapheme clusters), how normalization fixes the equality problem, and which built-in tools handle all of it correctly.

At a Glance

Short Verdict

Store and transmit text as-is, normalize to NFC before comparing or indexing it, and use grapheme segmentation, not length or code point counts, whenever the result is shown to a person: truncation, cursor movement, character limits, and “reverse this string.” Use compatibility normalization (NFKC/NFKD) only for search and matching, never to rewrite what the user typed.

Layer 1 and 2: Code Units vs Code Points

Unicode assigns every character a code point, written like U+00F1. Encodings then store those code points as code units: UTF-8 uses 8-bit units, UTF-16 uses 16-bit units. JavaScript strings are UTF-16, so length counts 16-bit units, not characters. Code points above U+FFFF, which include most emoji, need two UTF-16 units each (a surrogate pair). Splitting a string in the middle of a surrogate pair produces broken text, which is why naive slice() truncation can corrupt emoji.

Layer 3: Grapheme Clusters

Unicode Standard Annex #29, the specification for text segmentation (its current revision covers Unicode 18.0.0), puts the core problem plainly: a single code point is often, but not always, the same as a basic unit of a writing system. What users see as one character is approximated by a grapheme cluster. A letter followed by a combining accent is one cluster made of two code points. A Hangul syllable can be one precomposed code point or a sequence of conjoining jamo. And per UAX #29, each emoji sequence in the recommended (RGI) emoji set is a single grapheme cluster, even when it is built from several emoji joined by zero-width joiners.

UAX #29 defines two kinds of clusters. Legacy grapheme clusters are a base character plus continuing marks. Extended grapheme clusters, the default, also absorb spacing combining marks such as many Indic vowel signs, and the annex recommends them over legacy clusters except where the legacy variant is specifically required. Two useful properties: cluster boundaries are the same whether text is in NFC or NFD, and clusters do not necessarily match what the font draws, since a ligature like “fi” may render as one glyph while still being two clusters.

In JavaScript, Intl.Segmenter implements this segmentation and is marked Baseline widely available on MDN, supported across browsers since April 2024. A family emoji built from three emoji and two zero-width joiners shows the gap between storage and perception:

const family = "\u{1F468}\u200D\u{1F469}\u200D\u{1F467}"; // man ZWJ woman ZWJ girl
console.log(family.length);            // 8 UTF-16 code units
console.log([...family].length);       // 5 code points

const seg = new Intl.Segmenter("en", { granularity: "grapheme" });
console.log([...seg.segment(family)].length); // 1 user-perceived character

The same API supports "word" and "sentence" granularity. That matters because, as UAX #29 notes, word boundaries are not restricted to whitespace and punctuation, and languages such as Chinese, Japanese, and Thai do not separate words with spaces at all, so split(" ") fails on them.

Normalization: Making Equal Strings Equal

The letter “ñ” can be stored as the single code point U+00F1 or as “n” (U+006E) followed by a combining tilde (U+0303). The two render identically, but a byte-for-byte comparison says they differ. Unicode calls them canonically equivalent: they represent the same abstract character and should look and behave the same. Normalization picks one standard representation so comparisons work.

const a = "\u00F1";        // precomposed
const b = "\u006E\u0303";  // n + combining tilde
console.log(a === b);                                   // false
console.log(a.normalize("NFC") === b.normalize("NFC")); // true

const lig = "\uFB00";      // the "ff" ligature
console.log(lig.normalize("NFKC"));                     // "ff"

Compatibility equivalence is weaker than canonical equivalence: all canonically equivalent sequences are also compatible, but not the other way round. Compatible sequences should be treated alike in some applications but not necessarily all. MDN’s guidance captures the trade-off well: compatibility normalization suits search, because a user can find the ligature by typing “f”, but may not suit display, because it changes how the text looks. That is why you should normalize a copy for matching rather than overwrite what the user entered.

Common Bugs and Their Fixes

• Character limits that reject valid input. A 20-character username limit enforced with length counts emoji as several characters. Count grapheme clusters for user-facing limits, and enforce a separate byte limit for storage.
• Truncation that breaks emoji or accents. Cut at grapheme boundaries, not at a fixed code unit index.
• Duplicate accounts or failed lookups. “Amélie” typed on two keyboards can produce two different code point sequences. Normalize to NFC before using text as a key.
• String reversal that mangles text. Reverse the array of grapheme segments, not the code units.
• Word counts that fail in some languages. Use word segmentation instead of splitting on spaces.

Bottom Line

Keep three layers in mind: code units for storage limits, code points for encoding logic, and grapheme clusters for anything a person sees. Normalize to NFC before comparing, reserve NFKC/NFKD for search, and lean on the platform: normalize() and Intl.Segmenter already implement the Unicode rules so you do not have to.

Frequently Asked Questions

• Why is an emoji’s length more than 1 in JavaScript? length counts UTF-16 code units; emoji above U+FFFF take two units each, and joined sequences add zero-width joiners.
• Which normalization form should I store? NFC is the usual choice for storage and comparison; it is also the default for normalize().
• When should I use NFKC? For search and matching, not for display or for rewriting user input.
• How do I count characters the way users see them? Use grapheme segmentation, such as Intl.Segmenter with granularity: "grapheme".

Sources

• Unicode — UAX #29: Unicode Text Segmentation
• Unicode — UAX #15: Unicode Normalization Forms
• MDN — String.prototype.normalize()
• MDN — Intl.Segmenter