Unicode Support and Text Processing

<h3 id="overview" class="position-relative d-flex align-items-center group"> Overview <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="overview" aria-haspopup="dialog" aria-label="Share link: Overview"> Share link </button> </h3><div id="headingShareModal" class="heading-share-modal" role="dialog" aria-modal="true" aria-labelledby="headingShareTitle" hidden> <div class="hsm-dialog" role="document"> <div class="hsm-header"> <h2 id="headingShareTitle" class="h6 mb-0 fw-bold">Share this section</h2> <button type="button" class="hsm-close" aria-label="Close"> </button> </div> <div class="hsm-body"> <label for="headingShareInput" class="form-label small text-muted mb-1 text-uppercase fw-bold" style="font-size: 0.7rem; letter-spacing: 0.5px;">Permalink</label> <div class="input-group mb-4 hsm-url-group"> <input id="headingShareInput" type="text" class="form-control font-monospace" readonly aria-readonly="true" style="font-size: 0.85rem;" /> <button class="btn btn-primary hsm-copy" type="button" aria-label="Copy" title="Copy"> </button> </div> <div class="small fw-bold mb-2 text-muted text-uppercase" style="font-size: 0.7rem; letter-spacing: 0.5px;">Share via</div> <div class="hsm-share-grid"> <a id="share-twitter" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Twitter </a> <a id="share-linkedin" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> LinkedIn </a> <a id="share-facebook" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Facebook </a> </div> </div> </div> </div> <style> .heading-share-modal { position: fixed; inset: 0; display: flex; justify-content: center; align-items: center; background: rgba(0, 0, 0, 0.6); z-index: 1050; padding: 1rem; backdrop-filter: blur(4px); -webkit-backdrop-filter: blur(4px); } .heading-share-modal[hidden] { display: none !important; } .hsm-dialog { max-width: 420px; width: 100%; background: var(--bs-body-bg, #fff); color: var(--bs-body-color, #212529); border: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); border-radius: 1rem; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); overflow: hidden; animation: hsm-fade-in 0.2s ease-out; } @keyframes hsm-fade-in { from { opacity: 0; transform: scale(0.95); } to { opacity: 1; transform: scale(1); } } [data-bs-theme="dark"] .hsm-dialog { background: #1e293b; border-color: rgba(255,255,255,0.1); color: #f8f9fa; } .hsm-header { display: flex; justify-content: space-between; align-items: center; padding: 1rem 1.5rem; border-bottom: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); background: rgba(0,0,0,0.02); } [data-bs-theme="dark"] .hsm-header { background: rgba(255,255,255,0.02); border-color: rgba(255,255,255,0.1); } .hsm-close { background: transparent; border: none; color: inherit; opacity: 0.5; padding: 0.25rem 0.5rem; border-radius: 0.25rem; font-size: 1.2rem; line-height: 1; transition: opacity 0.2s; } .hsm-close:hover { opacity: 1; } .hsm-body { padding: 1.5rem; } .hsm-url-group { display: flex !important; align-items: stretch; } .hsm-url-group .form-control { flex: 1; min-width: 0; margin: 0; background: var(--bs-secondary-bg, #f8f9fa); border-color: var(--bs-border-color, #dee2e6); border-top-right-radius: 0; border-bottom-right-radius: 0; height: 42px; } .hsm-url-group .btn { flex: 0 0 auto; margin: 0; margin-left: -1px; border-top-left-radius: 0; border-bottom-left-radius: 0; height: 42px; display: flex; align-items: center; justify-content: center; padding: 0 1.25rem; z-index: 2; } [data-bs-theme="dark"] .hsm-url-group .form-control { background: #0f172a; border-color: #334155; color: #e2e8f0; } .hsm-share-grid { display: flex; flex-direction: column; gap: 0.5rem; } .hsm-share-grid .btn { display: flex; align-items: center; justify-content: center; font-size: 0.9rem; padding: 0.6rem; border-color: var(--bs-border-color); width: 100%; } [data-bs-theme="dark"] .hsm-share-grid .btn { color: #e2e8f0; border-color: #475569; } [data-bs-theme="dark"] .hsm-share-grid .btn:hover { background: #334155; border-color: #cbd5e1; } </style> <script> (function(){ const modal = document.getElementById('headingShareModal'); if(!modal) return; const input = modal.querySelector('#headingShareInput'); const copyBtn = modal.querySelector('.hsm-copy'); const twitter = modal.querySelector('#share-twitter'); const linkedin = modal.querySelector('#share-linkedin'); const facebook = modal.querySelector('#share-facebook'); const closeBtn = modal.querySelector('.hsm-close'); let lastFocus=null; let trapBound=false; function buildUrl(id){ return window.location.origin + window.location.pathname + '#' + id; } function isOpen(){ return !modal.hasAttribute('hidden'); } function hydrate(id){ const url=buildUrl(id); input.value=url; const enc=encodeURIComponent(url); const text=encodeURIComponent(document.title); if(twitter) twitter.href=`https://twitter.com/intent/tweet?url=${enc}&text=${text}`; if(linkedin) linkedin.href=`https://www.linkedin.com/sharing/share-offsite/?url=${enc}`; if(facebook) facebook.href=`https://www.facebook.com/sharer/sharer.php?u=${enc}`; } function openModal(id){ lastFocus=document.activeElement; hydrate(id); if(!isOpen()){ modal.removeAttribute('hidden'); } requestAnimationFrame(()=>{ input.focus(); }); trapFocus(); } function closeModal(){ if(!isOpen()) return; modal.setAttribute('hidden',''); if(lastFocus && typeof lastFocus.focus==='function') lastFocus.focus(); } function copyCurrent(){ try{ navigator.clipboard.writeText(input.value).then(()=>feedback(true),()=>fallback()); } catch(e){ fallback(); } } function fallback(){ input.select(); try{ document.execCommand('copy'); feedback(true);}catch(e){ feedback(false);} } function feedback(ok){ if(!copyBtn) return; const icon=copyBtn.querySelector('i'); if(!icon) return; const prev=copyBtn.getAttribute('data-prev')||icon.className; if(!copyBtn.getAttribute('data-prev')) copyBtn.setAttribute('data-prev',prev); icon.className= ok ? 'fa-duotone fa-clipboard-check':'fa-duotone fa-circle-exclamation'; setTimeout(()=>{ icon.className=prev; },1800); } function handleShareClick(e){ e.preventDefault(); const btn=e.currentTarget; const id=btn.getAttribute('data-share-target'); if(id) openModal(id); } function bindShareButtons(){ document.querySelectorAll('.h-share').forEach(btn=>{ if(!btn.dataset.hShareBound){ btn.addEventListener('click', handleShareClick); btn.dataset.hShareBound='1'; } }); } bindShareButtons(); if(document.readyState==='loading'){ document.addEventListener('DOMContentLoaded', bindShareButtons); } else { requestAnimationFrame(bindShareButtons); } document.addEventListener('click', function(e){ const shareBtn=e.target.closest && e.target.closest('.h-share'); if(shareBtn && !shareBtn.dataset.hShareBound){ handleShareClick.call(shareBtn, e); } }, true); document.addEventListener('click', e=>{ if(e.target===modal) closeModal(); if(e.target.closest && e.target.closest('.hsm-close')){ e.preventDefault(); closeModal(); } if(copyBtn && (e.target===copyBtn || (e.target.closest && e.target.closest('.hsm-copy')))) { e.preventDefault(); copyCurrent(); } }); document.addEventListener('keydown', e=>{ if(e.key==='Escape' && isOpen()) closeModal(); }); function trapFocus(){ if(trapBound) return; trapBound=true; modal.addEventListener('keydown', f=>{ if(f.key==='Tab' && isOpen()){ const focusable=[...modal.querySelectorAll('a[href],button,input,textarea,select,[tabindex]:not([tabindex="-1"])')].filter(el=>!el.hasAttribute('disabled')); if(!focusable.length) return; const first=focusable[0]; const last=focusable[focusable.length-1]; if(f.shiftKey && document.activeElement===first){ f.preventDefault(); last.focus(); } else if(!f.shiftKey && document.activeElement===last){ f.preventDefault(); first.focus(); } } }); } if(closeBtn) closeBtn.addEventListener('click', e=>{ e.preventDefault(); closeModal(); }); })(); </script>Geode provides comprehensive Unicode support for international text processing with NFC normalization, case folding, UTF encoding conversions, and robust error handling. The implementation focuses on correctness, performance, and practical use cases for graph database operations. <h4 id="features" class="position-relative d-flex align-items-center group"> Features <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="features" aria-haspopup="dialog" aria-label="Share link: Features"> Share link </button> </h4>Core Capabilities: <ul> <li>NFC Normalization: Canonical composition for consistent text representation</li> <li>Case Folding: Locale-independent case-insensitive comparisons</li> <li>UTF-8 ↔ UTF-16: Bidirectional encoding conversion</li> <li>WTF-8 Lossy Decoding: Graceful handling of ill-formed UTF-8 sequences</li> <li>Full-Text Search Integration: Normalized text indexing and search</li> <li>Constraint Normalization: Consistent unique key enforcement</li> </ul> <h3 id="unicode-fundamentals" class="position-relative d-flex align-items-center group"> Unicode Fundamentals <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="unicode-fundamentals" aria-haspopup="dialog" aria-label="Share link: Unicode Fundamentals"> Share link </button> </h3> <h4 id="what-is-unicode" class="position-relative d-flex align-items-center group"> What is Unicode? <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="what-is-unicode" aria-haspopup="dialog" aria-label="Share link: What is Unicode?"> Share link </button> </h4>Unicode is a universal character encoding standard that provides a unique number (code point) for every character across all writing systems, modern and historic. Key Concepts: <ul> <li>Code Point: Numeric value representing a character (U+0041 = ‘A’)</li> <li>UTF-8: Variable-length encoding (1-4 bytes per character)</li> <li>UTF-16: Variable-length encoding (2 or 4 bytes per character)</li> <li>Normalization: Converting text to canonical form</li> <li>Case Folding: Locale-independent case conversion for comparison</li> </ul> <h4 id="why-normalization-matters" class="position-relative d-flex align-items-center group"> Why Normalization Matters <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="why-normalization-matters" aria-haspopup="dialog" aria-label="Share link: Why Normalization Matters"> Share link </button> </h4>Problem: Multiple representations of “same” text <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-plaintext" data-lang="plaintext">"café" can be encoded as: 1. U+0063 U+0061 U+0066 U+00E9 (precomposed é) 2. U+0063 U+0061 U+0066 U+0065 U+0301 (e + combining acute) Without normalization: These don't match in string comparison! With NFC normalization: Both become the same canonical form </code></pre></div>Impact on Graph Databases: <ul> <li>Unique constraints may fail incorrectly</li> <li>Full-text search misses matches</li> <li>Case-insensitive comparisons fail</li> <li>Index lookups miss data</li> </ul> <h3 id="nfc-normalization" class="position-relative d-flex align-items-center group"> NFC Normalization <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="nfc-normalization" aria-haspopup="dialog" aria-label="Share link: NFC Normalization"> Share link </button> </h3> <h4 id="overview-1" class="position-relative d-flex align-items-center group"> Overview <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="overview-1" aria-haspopup="dialog" aria-label="Share link: Overview"> Share link </button> </h4>NFC (Normalization Form C) is a canonical composition that converts decomposed characters to their precomposed equivalents where possible. <h4 id="api" class="position-relative d-flex align-items-center group"> API <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="api" aria-haspopup="dialog" aria-label="Share link: API"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">pub fn nfcNormalize(alloc: std.mem.Allocator, s: []const u8) ![]u8 </code></pre></div>Parameters: <ul> <li><code>alloc</code>: Memory allocator</li> <li><code>s</code>: UTF-8 encoded input string</li> </ul> Returns: <ul> <li>Normalized UTF-8 string (caller owns memory)</li> <li><code>error.InvalidUtf8</code> if input is not valid UTF-8</li> <li><code>error.OutOfMemory</code> if allocation fails</li> </ul> <h4 id="examples" class="position-relative d-flex align-items-center group"> Examples <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="examples" aria-haspopup="dialog" aria-label="Share link: Examples"> Share link </button> </h4>Basic Normalization: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">const unicode = @import("geode").unicode; const allocator = std.heap.page_allocator; // Decomposed → Composed const input = "café"; // e + combining acute (U+0065 U+0301) const normalized = try unicode.nfcNormalize(allocator, input); defer allocator.free(normalized); // Result: "café" with precomposed é (U+00E9) </code></pre></div>Full-Text Search: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// Normalize before indexing const text = "Zürich naïve résumé"; const normalized = try unicode.nfcNormalize(allocator, text); // Index normalized form for consistent search try fulltext_index.add(normalized); </code></pre></div>Unique Constraints: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// Normalize before uniqueness check const email = "user@exämple.com"; const normalized_email = try unicode.nfcNormalize(allocator, email); // Check normalized form prevents duplicate variations if (isEmailTaken(normalized_email)) { return error.EmailAlreadyExists; } </code></pre></div> <h4 id="use-cases" class="position-relative d-flex align-items-center group"> Use Cases <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="use-cases" aria-haspopup="dialog" aria-label="Share link: Use Cases"> Share link </button> </h4><ol> <li> Full-Text Search: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Automatic normalization in search MATCH (doc:Document) WHERE doc.text CONTAINS 'café' RETURN doc.title -- Matches both composed and decomposed forms </code></pre></div></li> <li> Unique Key Enforcement: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Create constraint with normalization CREATE CONSTRAINT unique_email ON User (email) -- "user@exämple.com" and "user@exa\u0308mple.com" treated as same </code></pre></div></li> <li> ORDER BY Determinism: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Consistent ordering regardless of composition MATCH (p:Person) RETURN p.name ORDER BY p.name </code></pre></div></li> </ol> <h3 id="case-folding" class="position-relative d-flex align-items-center group"> Case Folding <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="case-folding" aria-haspopup="dialog" aria-label="Share link: Case Folding"> Share link </button> </h3> <h4 id="overview-2" class="position-relative d-flex align-items-center group"> Overview <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="overview-2" aria-haspopup="dialog" aria-label="Share link: Overview"> Share link </button> </h4>Case Folding provides locale-independent case-insensitive string comparison. Unlike simple lowercase conversion, case folding handles special cases like German ß → ss. <h4 id="api-1" class="position-relative d-flex align-items-center group"> API <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="api-1" aria-haspopup="dialog" aria-label="Share link: API"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">pub fn foldCase(alloc: std.mem.Allocator, s: []const u8) ![]u8 </code></pre></div>Parameters: <ul> <li><code>alloc</code>: Memory allocator</li> <li><code>s</code>: UTF-8 encoded input string</li> </ul> Returns: <ul> <li>Case-folded UTF-8 string (caller owns memory)</li> <li><code>error.InvalidUtf8</code> if input is not valid UTF-8</li> </ul> <h4 id="mappings" class="position-relative d-flex align-items-center group"> Mappings <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="mappings" aria-haspopup="dialog" aria-label="Share link: Mappings"> Share link </button> </h4>Supported Case Foldings: <ul> <li>German ß: ß → ss (U+00DF → U+0073 U+0073)</li> <li>Greek Σ: Σ → σ (uppercase sigma → lowercase sigma)</li> <li>Basic Latin: A-Z → a-z (U+0041-U+005A → U+0061-U+007A)</li> <li>Latin Accents: À → à, É → é, etc.</li> </ul> <h4 id="examples-1" class="position-relative d-flex align-items-center group"> Examples <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="examples-1" aria-haspopup="dialog" aria-label="Share link: Examples"> Share link </button> </h4>Basic Folding: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">const folded = try unicode.foldCase(allocator, "Straße"); // Result: "strasse" (ß → ss) const greek = try unicode.foldCase(allocator, "ΣΕΛΛΑΣ"); // Result: "σελλασ" (Σ → σ) </code></pre></div>Case-Insensitive Search: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// Fold both search term and indexed text const query = try unicode.foldCase(allocator, "CAFÉ"); const document = try unicode.foldCase(allocator, "café"); // Now: query == document (case-insensitive match) </code></pre></div>Username Comparison: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// Case-insensitive username lookup const input = try unicode.foldCase(allocator, "Müller"); const stored = try unicode.foldCase(allocator, "MÜLLER"); if (std.mem.eql(u8, input, stored)) { // Usernames match (case-insensitive) } </code></pre></div> <h4 id="use-cases-1" class="position-relative d-flex align-items-center group"> Use Cases <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="use-cases-1" aria-haspopup="dialog" aria-label="Share link: Use Cases"> Share link </button> </h4><ol> <li> Full-Text Search: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Case-insensitive search (automatic) CREATE INDEX doc_text_idx ON Document (text) USING fulltext MATCH (doc:Document) WHERE doc.text CONTAINS 'Straße' RETURN doc.title -- Matches: "straße", "Straße", "STRASSE", etc. </code></pre></div></li> <li> Case-Insensitive Constraints: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Username uniqueness (case-insensitive) CREATE CONSTRAINT unique_username ON User (username) -- "Alice", "alice", "ALICE" all rejected as duplicates </code></pre></div></li> <li> Comparisons: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Case-insensitive WHERE clause MATCH (u:User) WHERE lower(u.name) = lower('MÜLLER') RETURN u.email </code></pre></div></li> </ol> <h3 id="utf-8--utf-16-conversion" class="position-relative d-flex align-items-center group"> UTF-8 ↔ UTF-16 Conversion <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="utf-8--utf-16-conversion" aria-haspopup="dialog" aria-label="Share link: UTF-8 ↔ UTF-16 Conversion"> Share link </button> </h3> <h4 id="utf-8-to-utf-16" class="position-relative d-flex align-items-center group"> UTF-8 to UTF-16 <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="utf-8-to-utf-16" aria-haspopup="dialog" aria-label="Share link: UTF-8 to UTF-16"> Share link </button> </h4>API: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">pub fn utf8ToUtf16Alloc(ally: std.mem.Allocator, s: []const u8) ![]u16 </code></pre></div>Parameters: <ul> <li><code>ally</code>: Memory allocator</li> <li><code>s</code>: UTF-8 encoded string</li> </ul> Returns: <ul> <li>UTF-16LE encoded string as <code>[]u16</code> (caller owns)</li> <li><code>error.InvalidUtf8</code> if input is invalid</li> </ul> Use Cases: <ul> <li>Windows API interoperability</li> <li>Java/C# string compatibility</li> <li>UTF-16 based protocols</li> </ul> Example: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">const utf8_str = "Hello 世界"; const utf16 = try unicode.utf8ToUtf16Alloc(allocator, utf8_str); defer allocator.free(utf16); // utf16 now contains UTF-16LE code units </code></pre></div> <h4 id="utf-16-to-utf-8" class="position-relative d-flex align-items-center group"> UTF-16 to UTF-8 <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="utf-16-to-utf-8" aria-haspopup="dialog" aria-label="Share link: UTF-16 to UTF-8"> Share link </button> </h4>API: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">pub fn utf16ToUtf8Alloc(ally: std.mem.Allocator, units: []const u16) ![]u8 </code></pre></div>Parameters: <ul> <li><code>ally</code>: Memory allocator</li> <li><code>units</code>: UTF-16LE code units</li> </ul> Returns: <ul> <li>UTF-8 encoded string (caller owns)</li> <li>Allocation errors only (UTF-16 validation handled by stdlib)</li> </ul> Use Cases: <ul> <li>Processing Windows filenames</li> <li>Importing data from UTF-16 sources</li> <li>Interoperability with .NET applications</li> </ul> Example: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">const utf16_data: []const u16 = getWindowsString(); const utf8 = try unicode.utf16ToUtf8Alloc(allocator, utf16_data); defer allocator.free(utf8); // utf8 now contains UTF-8 encoded string </code></pre></div> <h4 id="round-trip-safety" class="position-relative d-flex align-items-center group"> Round-Trip Safety <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="round-trip-safety" aria-haspopup="dialog" aria-label="Share link: Round-Trip Safety"> Share link </button> </h4>Guaranteed Properties: <ul> <li>UTF-8 → UTF-16 → UTF-8 preserves original (if valid UTF-8)</li> <li>UTF-16 → UTF-8 → UTF-16 preserves original (if valid UTF-16)</li> <li>Surrogates handled correctly</li> </ul> Test: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// Round-trip test const original = "Hello 世界 🚀"; const utf16 = try unicode.utf8ToUtf16Alloc(allocator, original); const back = try unicode.utf16ToUtf8Alloc(allocator, utf16); assert(std.mem.eql(u8, original, back)); // true </code></pre></div> <h3 id="wtf-8-lossy-decoding" class="position-relative d-flex align-items-center group"> WTF-8 Lossy Decoding <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="wtf-8-lossy-decoding" aria-haspopup="dialog" aria-label="Share link: WTF-8 Lossy Decoding"> Share link </button> </h3> <h4 id="overview-3" class="position-relative d-flex align-items-center group"> Overview <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="overview-3" aria-haspopup="dialog" aria-label="Share link: Overview"> Share link </button> </h4>WTF-8 (Wobbly Transformation Format) is a superset of UTF-8 that allows unpaired surrogates. Lossy decoding converts potentially ill-formed sequences to valid UTF-8 by replacing invalid sequences with U+FFFD (replacement character). <h4 id="api-2" class="position-relative d-flex align-items-center group"> API <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="api-2" aria-haspopup="dialog" aria-label="Share link: API"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">pub fn wtf8LossyToUtf8Alloc(ally: std.mem.Allocator, bytes: []const u8) ![]u8 </code></pre></div>Parameters: <ul> <li><code>ally</code>: Memory allocator</li> <li><code>bytes</code>: Potentially ill-formed UTF-8/WTF-8 data</li> </ul> Returns: <ul> <li>Valid UTF-8 string with replacements (caller owns)</li> <li>Only allocation errors (never fails on invalid input)</li> </ul> <h4 id="use-cases-2" class="position-relative d-flex align-items-center group"> Use Cases <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="use-cases-2" aria-haspopup="dialog" aria-label="Share link: Use Cases"> Share link </button> </h4>Robust File Processing: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// Read file with unknown encoding const file_data = try readFile("unknown.txt"); const valid_utf8 = try unicode.wtf8LossyToUtf8Alloc(allocator, file_data); // valid_utf8 is guaranteed valid UTF-8 </code></pre></div>User Input Sanitization: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// Handle potentially malformed user input const user_input = getFormData(); const sanitized = try unicode.wtf8LossyToUtf8Alloc(allocator, user_input); // Safe to store and display </code></pre></div>Legacy Data Migration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// Import data from legacy system with encoding issues const legacy_data = importFromLegacyDB(); const clean_data = try unicode.wtf8LossyToUtf8Alloc(allocator, legacy_data); // Store clean data in Geode </code></pre></div> <h4 id="replacement-character" class="position-relative d-flex align-items-center group"> Replacement Character <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="replacement-character" aria-haspopup="dialog" aria-label="Share link: Replacement Character"> Share link </button> </h4>U+FFFD: � (Replacement Character) Replacement Rules: <ul> <li>Invalid UTF-8 sequences → U+FFFD</li> <li>Unpaired surrogates → U+FFFD</li> <li>Overlong encodings → U+FFFD</li> <li>Out-of-range code points → U+FFFD</li> </ul> Example: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">const invalid = &[_]u8{0xFF, 0xFE, 0x41}; // Invalid bytes + 'A' const fixed = try unicode.wtf8LossyToUtf8Alloc(allocator, invalid); // Result: "��A" (two replacements + valid 'A') </code></pre></div> <h3 id="integration-with-geode-features" class="position-relative d-flex align-items-center group"> Integration with Geode Features <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="integration-with-geode-features" aria-haspopup="dialog" aria-label="Share link: Integration with Geode Features"> Share link </button> </h3> <h4 id="full-text-search" class="position-relative d-flex align-items-center group"> Full-Text Search <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="full-text-search" aria-haspopup="dialog" aria-label="Share link: Full-Text Search"> Share link </button> </h4>Automatic Normalization: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Index creation normalizes text CREATE INDEX article_idx ON Article (content) USING fulltext -- Automatic NFC normalization + case folding INSERT (a:Article {content: "Zürich café résumé"}) -- Search works with any variation MATCH (a:Article) WHERE a.content CONTAINS 'zurich cafe resume' RETURN a -- Finds the article (normalized + folded) </code></pre></div>Implementation: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// FTS analyzer pipeline fn analyzeText(text: []const u8) ![]Token { // 1. NFC normalization const normalized = try unicode.nfcNormalize(allocator, text); // 2. Case folding const folded = try unicode.foldCase(allocator, normalized); // 3. Tokenization return tokenize(folded); } </code></pre></div> <h4 id="unique-constraints" class="position-relative d-flex align-items-center group"> Unique Constraints <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="unique-constraints" aria-haspopup="dialog" aria-label="Share link: Unique Constraints"> Share link </button> </h4>Normalized Keys: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">CREATE CONSTRAINT unique_email ON User (email) -- These are treated as duplicates: INSERT (u1:User {email: "user@exämple.com"}) -- Composed INSERT (u2:User {email: "user@exa\u0308mple.com"}) -- Decomposed -- Second INSERT fails: duplicate key (after normalization) </code></pre></div>Implementation: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">fn checkUniqueConstraint(key: []const u8) !void { const normalized = try unicode.nfcNormalize(allocator, key); if (exists(normalized)) { return error.UniqueConstraintViolation; } } </code></pre></div> <h4 id="order-by-determinism" class="position-relative d-flex align-items-center group"> ORDER BY Determinism <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="order-by-determinism" aria-haspopup="dialog" aria-label="Share link: ORDER BY Determinism"> Share link </button> </h4>Consistent Ordering: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">MATCH (p:Person) RETURN p.name ORDER BY p.name -- Results ordered by normalized form: -- "Müller" (composed) -- "Müller" (decomposed) -- Same position (normalized) -- "Schulz" </code></pre></div> <h4 id="expression-comparison" class="position-relative d-flex align-items-center group"> Expression Comparison <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="expression-comparison" aria-haspopup="dialog" aria-label="Share link: Expression Comparison"> Share link </button> </h4>NFC Comparison: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">MATCH (u:User) WHERE u.name = 'Müller' RETURN u.email -- Matches both: -- name = "Müller" (U+00FC composed) -- name = "Mu\u0308ller" (U+0075 U+0308 decomposed) </code></pre></div> <h3 id="limitations--future-work" class="position-relative d-flex align-items-center group"> Limitations &amp; Future Work <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="limitations--future-work" aria-haspopup="dialog" aria-label="Share link: Limitations &amp; Future Work"> Share link </button> </h3> <h4 id="current-scope" class="position-relative d-flex align-items-center group"> Current Scope <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="current-scope" aria-haspopup="dialog" aria-label="Share link: Current Scope"> Share link </button> </h4>Implemented: <ul> <li>✅ NFC normalization (subset)</li> <li>✅ Case folding (targeted mappings)</li> <li>✅ UTF-8 ↔ UTF-16 conversion</li> <li>✅ WTF-8 lossy decoding</li> </ul> Not Yet Implemented: <ul> <li>❌ Full canonical combining class ordering</li> <li>❌ Complete composition table</li> <li>❌ Locale-sensitive collation</li> <li>❌ Grapheme cluster segmentation</li> <li>❌ NFKC/NFKD normalization</li> <li>❌ UTF-32 operations</li> </ul> <h4 id="subset-coverage" class="position-relative d-flex align-items-center group"> Subset Coverage <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="subset-coverage" aria-haspopup="dialog" aria-label="Share link: Subset Coverage"> Share link </button> </h4>Normalization: <ul> <li>Selected decompositions only</li> <li>Common Latin, Greek, Cyrillic characters</li> <li>Full table generation via <code>zig build unicode-gen</code></li> </ul> Case Folding: <ul> <li>Sharp S (ß → ss)</li> <li>Greek sigma variants (Σ/ς/σ)</li> <li>Basic Latin (A-Z → a-z)</li> <li>Common accented characters</li> </ul> <h4 id="future-enhancements" class="position-relative d-flex align-items-center group"> Future Enhancements <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="future-enhancements" aria-haspopup="dialog" aria-label="Share link: Future Enhancements"> Share link </button> </h4>Planned Features: <ul> <li>Full canonical combining class support</li> <li>Complete composition tables</li> <li>Locale-sensitive collation (ICU integration)</li> <li>Grapheme cluster boundary detection</li> <li>NFKC compatibility normalization</li> <li>Advanced text segmentation</li> </ul> <h3 id="performance" class="position-relative d-flex align-items-center group"> Performance <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="performance" aria-haspopup="dialog" aria-label="Share link: Performance"> Share link </button> </h3> <h4 id="benchmarks" class="position-relative d-flex align-items-center group"> Benchmarks <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="benchmarks" aria-haspopup="dialog" aria-label="Share link: Benchmarks"> Share link </button> </h4>NFC Normalization: <ul> <li>ASCII-only text: <100ns (fast path)</li> <li>Optimized for common Latin characters</li> <li>Handles complex scripts</li> <li>Memory overhead varies by input</li> </ul> Case Folding: <ul> <li>Fast path for ASCII-only strings</li> <li>Handles German ß expansion</li> <li>Correct Greek sigma handling</li> </ul> UTF-8 ↔ UTF-16: <ul> <li>Efficient ASCII conversion</li> <li>Full multi-byte support</li> <li>Surrogate pair handling for rare characters</li> </ul> WTF-8 Lossy: <ul> <li>Valid UTF-8: <100ns overhead</li> <li>Invalid sequences: ~500ns per replacement</li> <li>Memory: 1.1x input size</li> </ul> <h4 id="optimization-tips" class="position-relative d-flex align-items-center group"> Optimization Tips <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="optimization-tips" aria-haspopup="dialog" aria-label="Share link: Optimization Tips"> Share link </button> </h4>Reuse Allocations: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// Use arena allocator for batch processing var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator); defer arena.deinit(); const allocator = arena.allocator(); for (texts) |text| { const normalized = try unicode.nfcNormalize(allocator, text); // Process normalized text // Memory freed in bulk at arena.deinit() } </code></pre></div>Cache Results: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// Cache normalized strings var cache = std.StringHashMap([]const u8).init(allocator); fn getNormalized(text: []const u8) ![]const u8 { if (cache.get(text)) |cached| { return cached; } const normalized = try unicode.nfcNormalize(allocator, text); try cache.put(text, normalized); return normalized; } </code></pre></div> <h3 id="testing" class="position-relative d-flex align-items-center group"> Testing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="testing" aria-haspopup="dialog" aria-label="Share link: Testing"> Share link </button> </h3> <h4 id="test-coverage" class="position-relative d-flex align-items-center group"> Test Coverage <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="test-coverage" aria-haspopup="dialog" aria-label="Share link: Test Coverage"> Share link </button> </h4>Unit Tests: <code>tests/canary_unicode_utf16.zig</code> <ul> <li><code>TestCANARY_REQ_GQL_UNICODE_012_Utf8ToUtf16RoundTrip</code></li> <li><code>TestCANARY_REQ_GQL_UNICODE_013_Utf16ToUtf8RoundTrip</code></li> <li><code>TestCANARY_REQ_GQL_UNICODE_014_Wtf8Lossy</code></li> </ul> Test Cases: <ul> <li>Round-trip conversions</li> <li>Invalid sequence handling</li> <li>Surrogate pair processing</li> <li>Emoji support (🚀, 😀, etc.)</li> <li>CJK characters (Chinese, Japanese, Korean)</li> <li>Combining characters</li> </ul> <h4 id="example-tests" class="position-relative d-flex align-items-center group"> Example Tests <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="example-tests" aria-haspopup="dialog" aria-label="Share link: Example Tests"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">test "UTF-8 to UTF-16 round-trip" { const original = "Hello 世界 🚀"; const utf16 = try unicode.utf8ToUtf16Alloc(allocator, original); defer allocator.free(utf16); const back = try unicode.utf16ToUtf8Alloc(allocator, utf16); defer allocator.free(back); try std.testing.expectEqualSlices(u8, original, back); } test "WTF-8 lossy decoding" { const invalid = &[_]u8{0xFF, 0xFE}; const valid = try unicode.wtf8LossyToUtf8Alloc(allocator, invalid); defer allocator.free(valid); // Should contain replacement characters try std.testing.expect(valid.len > 0); } </code></pre></div> <h3 id="best-practices" class="position-relative d-flex align-items-center group"> Best Practices <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="best-practices" aria-haspopup="dialog" aria-label="Share link: Best Practices"> Share link </button> </h3> <h4 id="always-normalize-user-input" class="position-relative d-flex align-items-center group"> Always Normalize User Input <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="always-normalize-user-input" aria-haspopup="dialog" aria-label="Share link: Always Normalize User Input"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// ✅ Good: Normalize before storing const user_input = getFormData(); const normalized = try unicode.nfcNormalize(allocator, user_input); try storeInDatabase(normalized); // ❌ Bad: Store raw input const user_input = getFormData(); try storeInDatabase(user_input); // May cause duplicate key issues </code></pre></div> <h4 id="use-case-folding-for-search" class="position-relative d-flex align-items-center group"> Use Case Folding for Search <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="use-case-folding-for-search" aria-haspopup="dialog" aria-label="Share link: Use Case Folding for Search"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// ✅ Good: Case-insensitive search const query = try unicode.foldCase(allocator, user_query); const results = try searchFullText(query); // ❌ Bad: Case-sensitive (misses matches) const results = try searchFullText(user_query); </code></pre></div> <h4 id="handle-errors-gracefully" class="position-relative d-flex align-items-center group"> Handle Errors Gracefully <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="handle-errors-gracefully" aria-haspopup="dialog" aria-label="Share link: Handle Errors Gracefully"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// ✅ Good: Fallback on error const normalized = unicode.nfcNormalize(allocator, text) catch |err| { // Log error, use lossy decoding as fallback std.log.warn("Normalization failed: {}, using lossy", .{err}); const safe = try unicode.wtf8LossyToUtf8Alloc(allocator, text); return safe; }; // ❌ Bad: Crash on invalid input const normalized = try unicode.nfcNormalize(allocator, text); </code></pre></div> <h4 id="free-allocated-memory" class="position-relative d-flex align-items-center group"> Free Allocated Memory <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="free-allocated-memory" aria-haspopup="dialog" aria-label="Share link: Free Allocated Memory"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// ✅ Good: Free allocated strings const normalized = try unicode.nfcNormalize(allocator, text); defer allocator.free(normalized); // Use normalized... // ❌ Bad: Memory leak const normalized = try unicode.nfcNormalize(allocator, text); // Forgot to free! </code></pre></div> <h3 id="references" class="position-relative d-flex align-items-center group"> References <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="references" aria-haspopup="dialog" aria-label="Share link: References"> Share link </button> </h3> <h4 id="standards" class="position-relative d-flex align-items-center group"> Standards <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="standards" aria-haspopup="dialog" aria-label="Share link: Standards"> Share link </button> </h4><ul> <li> Unicode Standard 15.0: Character encoding and normalization <ul> <li><a href="https://unicode.org/versions/Unicode15.0.0/" aria-label="https://unicode.org/versions/Unicode15.0.0/ – opens in new window" target="_blank" rel="noopener noreferrer" >https://unicode.org/versions/Unicode15.0.0/ ↗ </a> </li> </ul> </li> <li> UAX #15: Unicode Normalization Forms <ul> <li><a href="https://unicode.org/reports/tr15/" aria-label="https://unicode.org/reports/tr15/ – opens in new window" target="_blank" rel="noopener noreferrer" >https://unicode.org/reports/tr15/ ↗ </a> </li> </ul> </li> <li> UAX #21: Case Mappings <ul> <li><a href="https://unicode.org/reports/tr21/" aria-label="https://unicode.org/reports/tr21/ – opens in new window" target="_blank" rel="noopener noreferrer" >https://unicode.org/reports/tr21/ ↗ </a> </li> </ul> </li> </ul> <h4 id="implementation" class="position-relative d-flex align-items-center group"> Implementation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="implementation" aria-haspopup="dialog" aria-label="Share link: Implementation"> Share link </button> </h4><ul> <li>Zig Stdlib: <code>std.unicode</code> module <ul> <li>UTF-8/UTF-16 conversion functions</li> <li>Validation and error handling</li> </ul> </li> </ul> <h4 id="code-location" class="position-relative d-flex align-items-center group"> Code Location <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="code-location" aria-haspopup="dialog" aria-label="Share link: Code Location"> Share link </button> </h4><ul> <li>Implementation: <code>src/unicode/geode_unicode.zig</code></li> <li>Tests: <code>tests/canary_unicode_utf16.zig</code></li> <li>Documentation: <code>docs/UNICODE.md</code></li> </ul> <h3 id="next-steps" class="position-relative d-flex align-items-center group"> Next Steps <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="next-steps" aria-haspopup="dialog" aria-label="Share link: Next Steps"> Share link </button> </h3>For New Users: <ul> <li><a href="/docs/data-types/_index/" >Data Types</a> - String type fundamentals</li> <li><a href="/docs/query/full-text-search/" >Full-Text Search</a> - Search with Unicode</li> <li><a href="/docs/gql/guide/" >GQL Guide</a> - Query language basics</li> </ul> For Developers: <ul> <li><a href="/docs/reference/type-conversion/" >Type Conversion</a> - String conversions</li> <li><a href="/docs/reference/api-reference-complete/" >API Reference</a> - Complete function list</li> <li><a href="/docs/guides/testing-strategies/" >Testing</a> - Text processing tests</li> </ul> For Advanced Users: <ul> <li><a href="/docs/query/performance-tuning/" >Performance Tuning</a> - Text query optimization</li> <li><a href="/docs/query/indexing-and-optimization/" >Indexing</a> - Full-text index configuration</li> </ul> <hr> Document Version: 1.0 Last Updated: January 24, 2026 Status: Production Ready Unicode Version: 15.0 (subset) CANARY: REQ-GQL-UNICODE-004 through REQ-GQL-UNICODE-014 (TESTED)