Data Quality Management

<h2 id="data-quality-management" class="position-relative d-flex align-items-center group"> Data Quality Management <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="data-quality-management" aria-haspopup="dialog" aria-label="Share link: Data Quality Management"> Share link </button> </h2><div id="headingShareModal" class="heading-share-modal" role="dialog" aria-modal="true" aria-labelledby="headingShareTitle" hidden> <div class="hsm-dialog" role="document"> <div class="hsm-header"> <h2 id="headingShareTitle" class="h6 mb-0 fw-bold">Share this section</h2> <button type="button" class="hsm-close" aria-label="Close"> </button> </div> <div class="hsm-body"> <label for="headingShareInput" class="form-label small text-muted mb-1 text-uppercase fw-bold" style="font-size: 0.7rem; letter-spacing: 0.5px;">Permalink</label> <div class="input-group mb-4 hsm-url-group"> <input id="headingShareInput" type="text" class="form-control font-monospace" readonly aria-readonly="true" style="font-size: 0.85rem;" /> <button class="btn btn-primary hsm-copy" type="button" aria-label="Copy" title="Copy"> </button> </div> <div class="small fw-bold mb-2 text-muted text-uppercase" style="font-size: 0.7rem; letter-spacing: 0.5px;">Share via</div> <div class="hsm-share-grid"> <a id="share-twitter" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Twitter </a> <a id="share-linkedin" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> LinkedIn </a> <a id="share-facebook" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Facebook </a> </div> </div> </div> </div> <style> .heading-share-modal { position: fixed; inset: 0; display: flex; justify-content: center; align-items: center; background: rgba(0, 0, 0, 0.6); z-index: 1050; padding: 1rem; backdrop-filter: blur(4px); -webkit-backdrop-filter: blur(4px); } .heading-share-modal[hidden] { display: none !important; } .hsm-dialog { max-width: 420px; width: 100%; background: var(--bs-body-bg, #fff); color: var(--bs-body-color, #212529); border: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); border-radius: 1rem; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); overflow: hidden; animation: hsm-fade-in 0.2s ease-out; } @keyframes hsm-fade-in { from { opacity: 0; transform: scale(0.95); } to { opacity: 1; transform: scale(1); } } [data-bs-theme="dark"] .hsm-dialog { background: #1e293b; border-color: rgba(255,255,255,0.1); color: #f8f9fa; } .hsm-header { display: flex; justify-content: space-between; align-items: center; padding: 1rem 1.5rem; border-bottom: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); background: rgba(0,0,0,0.02); } [data-bs-theme="dark"] .hsm-header { background: rgba(255,255,255,0.02); border-color: rgba(255,255,255,0.1); } .hsm-close { background: transparent; border: none; color: inherit; opacity: 0.5; padding: 0.25rem 0.5rem; border-radius: 0.25rem; font-size: 1.2rem; line-height: 1; transition: opacity 0.2s; } .hsm-close:hover { opacity: 1; } .hsm-body { padding: 1.5rem; } .hsm-url-group { display: flex !important; align-items: stretch; } .hsm-url-group .form-control { flex: 1; min-width: 0; margin: 0; background: var(--bs-secondary-bg, #f8f9fa); border-color: var(--bs-border-color, #dee2e6); border-top-right-radius: 0; border-bottom-right-radius: 0; height: 42px; } .hsm-url-group .btn { flex: 0 0 auto; margin: 0; margin-left: -1px; border-top-left-radius: 0; border-bottom-left-radius: 0; height: 42px; display: flex; align-items: center; justify-content: center; padding: 0 1.25rem; z-index: 2; } [data-bs-theme="dark"] .hsm-url-group .form-control { background: #0f172a; border-color: #334155; color: #e2e8f0; } .hsm-share-grid { display: flex; flex-direction: column; gap: 0.5rem; } .hsm-share-grid .btn { display: flex; align-items: center; justify-content: center; font-size: 0.9rem; padding: 0.6rem; border-color: var(--bs-border-color); width: 100%; } [data-bs-theme="dark"] .hsm-share-grid .btn { color: #e2e8f0; border-color: #475569; } [data-bs-theme="dark"] .hsm-share-grid .btn:hover { background: #334155; border-color: #cbd5e1; } </style> <script> (function(){ const modal = document.getElementById('headingShareModal'); if(!modal) return; const input = modal.querySelector('#headingShareInput'); const copyBtn = modal.querySelector('.hsm-copy'); const twitter = modal.querySelector('#share-twitter'); const linkedin = modal.querySelector('#share-linkedin'); const facebook = modal.querySelector('#share-facebook'); const closeBtn = modal.querySelector('.hsm-close'); let lastFocus=null; let trapBound=false; function buildUrl(id){ return window.location.origin + window.location.pathname + '#' + id; } function isOpen(){ return !modal.hasAttribute('hidden'); } function hydrate(id){ const url=buildUrl(id); input.value=url; const enc=encodeURIComponent(url); const text=encodeURIComponent(document.title); if(twitter) twitter.href=`https://twitter.com/intent/tweet?url=${enc}&text=${text}`; if(linkedin) linkedin.href=`https://www.linkedin.com/sharing/share-offsite/?url=${enc}`; if(facebook) facebook.href=`https://www.facebook.com/sharer/sharer.php?u=${enc}`; } function openModal(id){ lastFocus=document.activeElement; hydrate(id); if(!isOpen()){ modal.removeAttribute('hidden'); } requestAnimationFrame(()=>{ input.focus(); }); trapFocus(); } function closeModal(){ if(!isOpen()) return; modal.setAttribute('hidden',''); if(lastFocus && typeof lastFocus.focus==='function') lastFocus.focus(); } function copyCurrent(){ try{ navigator.clipboard.writeText(input.value).then(()=>feedback(true),()=>fallback()); } catch(e){ fallback(); } } function fallback(){ input.select(); try{ document.execCommand('copy'); feedback(true);}catch(e){ feedback(false);} } function feedback(ok){ if(!copyBtn) return; const icon=copyBtn.querySelector('i'); if(!icon) return; const prev=copyBtn.getAttribute('data-prev')||icon.className; if(!copyBtn.getAttribute('data-prev')) copyBtn.setAttribute('data-prev',prev); icon.className= ok ? 'fa-duotone fa-clipboard-check':'fa-duotone fa-circle-exclamation'; setTimeout(()=>{ icon.className=prev; },1800); } function handleShareClick(e){ e.preventDefault(); const btn=e.currentTarget; const id=btn.getAttribute('data-share-target'); if(id) openModal(id); } function bindShareButtons(){ document.querySelectorAll('.h-share').forEach(btn=>{ if(!btn.dataset.hShareBound){ btn.addEventListener('click', handleShareClick); btn.dataset.hShareBound='1'; } }); } bindShareButtons(); if(document.readyState==='loading'){ document.addEventListener('DOMContentLoaded', bindShareButtons); } else { requestAnimationFrame(bindShareButtons); } document.addEventListener('click', function(e){ const shareBtn=e.target.closest && e.target.closest('.h-share'); if(shareBtn && !shareBtn.dataset.hShareBound){ handleShareClick.call(shareBtn, e); } }, true); document.addEventListener('click', e=>{ if(e.target===modal) closeModal(); if(e.target.closest && e.target.closest('.hsm-close')){ e.preventDefault(); closeModal(); } if(copyBtn && (e.target===copyBtn || (e.target.closest && e.target.closest('.hsm-copy')))) { e.preventDefault(); copyCurrent(); } }); document.addEventListener('keydown', e=>{ if(e.key==='Escape' && isOpen()) closeModal(); }); function trapFocus(){ if(trapBound) return; trapBound=true; modal.addEventListener('keydown', f=>{ if(f.key==='Tab' && isOpen()){ const focusable=[...modal.querySelectorAll('a[href],button,input,textarea,select,[tabindex]:not([tabindex="-1"])')].filter(el=>!el.hasAttribute('disabled')); if(!focusable.length) return; const first=focusable[0]; const last=focusable[focusable.length-1]; if(f.shiftKey && document.activeElement===first){ f.preventDefault(); last.focus(); } else if(!f.shiftKey && document.activeElement===last){ f.preventDefault(); first.focus(); } } }); } if(closeBtn) closeBtn.addEventListener('click', e=>{ e.preventDefault(); closeModal(); }); })(); </script>Data quality management in Geode encompasses the processes, techniques, and tools that ensure your graph database contains accurate, complete, consistent, and timely data. High-quality data is essential for reliable analytics, trustworthy decision-making, and operational efficiency. Poor data quality leads to incorrect insights, failed transactions, and loss of user trust. <h3 id="the-six-dimensions-of-data-quality" class="position-relative d-flex align-items-center group"> The Six Dimensions of Data Quality <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="the-six-dimensions-of-data-quality" aria-haspopup="dialog" aria-label="Share link: The Six Dimensions of Data Quality"> Share link </button> </h3>Understanding these six fundamental dimensions helps frame your quality management strategy: <h4 id="1-accuracy" class="position-relative d-flex align-items-center group"> 1. Accuracy <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="1-accuracy" aria-haspopup="dialog" aria-label="Share link: 1. Accuracy"> Share link </button> </h4>Accuracy measures whether data correctly represents the real-world entities and relationships it describes. Inaccurate data leads to wrong conclusions and faulty business decisions. Examples: <ul> <li>Person’s age matches their birth date</li> <li>Product prices reflect current market rates</li> <li>Addresses correspond to real locations</li> </ul> Validation: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Verify age matches birth date MATCH (p:Person) WHERE p.age IS NOT NULL AND p.date_of_birth IS NOT NULL AND p.age != EXTRACT(YEAR FROM AGE(CURRENT_DATE, p.date_of_birth)) RETURN p.id, p.age, p.date_of_birth, 'age mismatch' AS issue; </code></pre></div> <h4 id="2-completeness" class="position-relative d-flex align-items-center group"> 2. Completeness <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="2-completeness" aria-haspopup="dialog" aria-label="Share link: 2. Completeness"> Share link </button> </h4>Completeness measures whether all required data is present. Missing critical data prevents proper graph traversals and analytics. Examples: <ul> <li>All customers have email addresses</li> <li>Products have prices and SKUs</li> <li>Employees have hire dates</li> </ul> Validation: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Find incomplete records MATCH (p:Person) WHERE p.email IS NULL OR p.name IS NULL OR p.created_at IS NULL RETURN p.id, 'missing required fields' AS issue; -- Completeness percentage MATCH (p:Person) RETURN COUNT(*) AS total, COUNT(p.phone) AS with_phone, ROUND(COUNT(p.phone) * 100.0 / COUNT(*), 2) AS phone_completeness_pct; </code></pre></div> <h4 id="3-consistency" class="position-relative d-flex align-items-center group"> 3. Consistency <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="3-consistency" aria-haspopup="dialog" aria-label="Share link: 3. Consistency"> Share link </button> </h4>Consistency ensures data is uniform across the system and doesn’t contradict itself. Inconsistent data creates confusion and unreliable queries. Examples: <ul> <li>Same email format across all records</li> <li>Consistent date formats</li> <li>Matching foreign key references</li> </ul> Validation: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Check for date inconsistencies MATCH (o:Order) WHERE o.shipped_date < o.created_date OR o.delivered_date < o.shipped_date RETURN o.id, o.created_date, o.shipped_date, o.delivered_date, 'date sequence error' AS issue; -- Find orphaned relationships MATCH (e)-[r:WORKS_FOR]->(c:Company) WHERE NOT EXISTS((e:Employee)) RETURN e, r, c, 'orphaned relationship' AS issue; </code></pre></div> <h4 id="4-timeliness" class="position-relative d-flex align-items-center group"> 4. Timeliness <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="4-timeliness" aria-haspopup="dialog" aria-label="Share link: 4. Timeliness"> Share link </button> </h4>Timeliness measures whether data is up-to-date and available when needed. Stale data leads to decisions based on outdated information. Examples: <ul> <li>Inventory levels updated in real-time</li> <li>User profiles reflect recent changes</li> <li>Analytics dashboards show current data</li> </ul> Validation: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Find stale records (not updated in 30 days) MATCH (p:Product) WHERE p.updated_at < (NOW() - INTERVAL '30 days') RETURN p.sku, p.name, p.updated_at, 'stale data' AS issue; </code></pre></div> <h4 id="5-validity" class="position-relative d-flex align-items-center group"> 5. Validity <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="5-validity" aria-haspopup="dialog" aria-label="Share link: 5. Validity"> Share link </button> </h4>Validity ensures data conforms to defined formats, ranges, and business rules. Invalid data fails to meet schema constraints. Examples: <ul> <li>Email addresses match email format</li> <li>Ages within reasonable range (0-150)</li> <li>Dates are valid calendar dates</li> </ul> Validation: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Find invalid formats MATCH (p:Person) WHERE p.email IS NOT NULL AND NOT (p.email ~ '^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$') RETURN p.id, p.email, 'invalid email format' AS issue; -- Find out-of-range values MATCH (p:Person) WHERE p.age < 0 OR p.age > 150 RETURN p.id, p.age, 'age out of range' AS issue; </code></pre></div> <h4 id="6-uniqueness" class="position-relative d-flex align-items-center group"> 6. Uniqueness <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="6-uniqueness" aria-haspopup="dialog" aria-label="Share link: 6. Uniqueness"> Share link </button> </h4>Uniqueness ensures no duplicate records exist that should be unique. Duplicates skew analytics and create confusion. Examples: <ul> <li>No duplicate email addresses</li> <li>Unique product SKUs</li> <li>Single canonical record per entity</li> </ul> Validation: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Find duplicate emails MATCH (p:Person) WITH p.email AS email, COLLECT(p) AS persons WHERE SIZE(persons) > 1 RETURN email, SIZE(persons) AS duplicate_count, 'duplicate email' AS issue; </code></pre></div> <h3 id="data-profiling" class="position-relative d-flex align-items-center group"> Data Profiling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="data-profiling" aria-haspopup="dialog" aria-label="Share link: Data Profiling"> Share link </button> </h3>Data profiling analyzes your database to understand patterns, distributions, and anomalies. Regular profiling helps identify quality issues before they impact operations. <h4 id="statistical-profiling" class="position-relative d-flex align-items-center group"> Statistical Profiling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="statistical-profiling" aria-haspopup="dialog" aria-label="Share link: Statistical Profiling"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Comprehensive data profile for Person nodes MATCH (p:Person) RETURN COUNT(*) AS total_persons, COUNT(p.email) AS emails_present, COUNT(p.phone) AS phones_present, COUNT(DISTINCT p.country) AS distinct_countries, MIN(p.age) AS min_age, MAX(p.age) AS max_age, AVG(p.age) AS avg_age, PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY p.age) AS median_age; </code></pre></div> <h4 id="distribution-analysis" class="position-relative d-flex align-items-center group"> Distribution Analysis <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="distribution-analysis" aria-haspopup="dialog" aria-label="Share link: Distribution Analysis"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"># Python - analyze age distribution from geode_client import Client import matplotlib.pyplot as plt async def profile_age_distribution(client: Client): result, _ = await client.query(""" MATCH (p:Person) WHERE p.age IS NOT NULL RETURN CASE WHEN p.age < 18 THEN 'Under 18' WHEN p.age < 30 THEN '18-29' WHEN p.age < 50 THEN '30-49' WHEN p.age < 65 THEN '50-64' ELSE '65+' END AS age_range, COUNT(*) AS count ORDER BY age_range """) age_ranges = [row['age_range'] for row in result.bindings] counts = [row['count'] for row in result.bindings] # Visualize distribution plt.bar(age_ranges, counts) plt.title('Age Distribution') plt.xlabel('Age Range') plt.ylabel('Count') plt.savefig('age_distribution.png') # Calculate percentages total = sum(counts) for age_range, count in zip(age_ranges, counts): pct = (count / total) * 100 print(f"{age_range}: {count} ({pct:.1f}%)") </code></pre></div> <h4 id="outlier-detection" class="position-relative d-flex align-items-center group"> Outlier Detection <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="outlier-detection" aria-haspopup="dialog" aria-label="Share link: Outlier Detection"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Find statistical outliers (values beyond 3 standard deviations) WITH stats AS ( MATCH (p:Product) RETURN AVG(p.price) AS mean_price, STDDEV(p.price) AS stddev_price ) MATCH (p:Product), (s:stats) WHERE p.price < (s.mean_price - 3 * s.stddev_price) OR p.price > (s.mean_price + 3 * s.stddev_price) RETURN p.sku, p.name, p.price, s.mean_price, 'price outlier' AS issue; </code></pre></div> <h3 id="data-cleansing-strategies" class="position-relative d-flex align-items-center group"> Data Cleansing Strategies <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="data-cleansing-strategies" aria-haspopup="dialog" aria-label="Share link: Data Cleansing Strategies"> Share link </button> </h3>Data cleansing removes or corrects invalid, duplicate, or inconsistent data. <h4 id="deduplication" class="position-relative d-flex align-items-center group"> Deduplication <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="deduplication" aria-haspopup="dialog" aria-label="Share link: Deduplication"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Find and merge duplicate person records by email MATCH (p1:Person), (p2:Person) WHERE p1.email = p2.email AND p1.id < p2.id -- Prevent self-matches and duplicate pairs WITH p1, p2 MATCH (p2)-[r]->(other) CREATE (p1)-[r2:SAME_TYPE(r)]->(other) SET r2 = properties(r) DELETE r WITH p1, p2 DELETE p2; </code></pre></div>Python script for safe deduplication: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"># Python - deduplicate with manual review async def find_duplicates(client: Client): """Find potential duplicate persons based on email.""" result, _ = await tx.query(""" MATCH (p:Person) WITH p.email AS email, COLLECT(p) AS persons WHERE SIZE(persons) > 1 RETURN email, [person IN persons | { id: person.id, name: person.name, created_at: person.created_at }] AS duplicates """) return result.bindings async def merge_persons(client: Client, keep_id: str, merge_id: str): """Merge two person records, keeping the first.""" async with client.connection() as tx: await tx.begin() # Transfer relationships await tx.execute(""" MATCH (keep:Person {id: $keep_id}) MATCH (merge:Person {id: $merge_id}) MATCH (merge)-[r]->(other) WHERE NOT EXISTS((keep)-[:SAME_TYPE(r)]->(other)) CREATE (keep)-[r2:SAME_TYPE(r)]->(other) SET r2 = properties(r) DELETE r """, {"keep_id": keep_id, "merge_id": merge_id}) # Delete duplicate await tx.execute( "MATCH (p:Person {id: $merge_id}) DELETE p", {"merge_id": merge_id} ) await tx.commit() </code></pre></div> <h4 id="data-standardization" class="position-relative d-flex align-items-center group"> Data Standardization <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="data-standardization" aria-haspopup="dialog" aria-label="Share link: Data Standardization"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Standardize email addresses (lowercase, trimmed) MATCH (p:Person) WHERE p.email IS NOT NULL SET p.email = LOWER(TRIM(p.email)); -- Standardize phone numbers (remove formatting) MATCH (p:Person) WHERE p.phone IS NOT NULL SET p.phone = REGEXP_REPLACE(p.phone, '[^0-9+]', '', 'g'); -- Standardize country codes MATCH (p:Person) WHERE p.country IS NOT NULL SET p.country = UPPER(TRIM(p.country)) WHERE LENGTH(p.country) = 2; -- ISO 3166-1 alpha-2 </code></pre></div> <h4 id="fixing-null-vs-empty-string" class="position-relative d-flex align-items-center group"> Fixing NULL vs Empty String <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="fixing-null-vs-empty-string" aria-haspopup="dialog" aria-label="Share link: Fixing NULL vs Empty String"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Convert empty strings to NULL for consistency MATCH (p:Person) WHERE p.middle_name = '' OR p.phone = '' OR p.bio = '' SET p.middle_name = CASE WHEN p.middle_name = '' THEN NULL ELSE p.middle_name END, p.phone = CASE WHEN p.phone = '' THEN NULL ELSE p.phone END, p.bio = CASE WHEN p.bio = '' THEN NULL ELSE p.bio END; </code></pre></div> <h3 id="quality-monitoring-and-metrics" class="position-relative d-flex align-items-center group"> Quality Monitoring and Metrics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="quality-monitoring-and-metrics" aria-haspopup="dialog" aria-label="Share link: Quality Monitoring and Metrics"> Share link </button> </h3>Continuous monitoring ensures data quality doesn’t degrade over time. <h4 id="quality-dashboard" class="position-relative d-flex align-items-center group"> Quality Dashboard <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="quality-dashboard" aria-haspopup="dialog" aria-label="Share link: Quality Dashboard"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Comprehensive data quality metrics WITH person_metrics AS ( MATCH (p:Person) RETURN COUNT(*) AS total, COUNT(p.email) AS has_email, COUNT(p.phone) AS has_phone, SUM(CASE WHEN p.age < 0 OR p.age > 150 THEN 1 ELSE 0 END) AS invalid_age, SUM(CASE WHEN p.email !~ '^[^@]+@[^@]+\.[^@]+$' THEN 1 ELSE 0 END) AS invalid_email ) SELECT 'Completeness: Email' AS metric, ROUND(100.0 * has_email / total, 2) AS score FROM person_metrics UNION ALL SELECT 'Completeness: Phone', ROUND(100.0 * has_phone / total, 2) FROM person_metrics UNION ALL SELECT 'Validity: Age', ROUND(100.0 * (total - invalid_age) / total, 2) FROM person_metrics UNION ALL SELECT 'Validity: Email', ROUND(100.0 * (has_email - invalid_email) / has_email, 2) FROM person_metrics; </code></pre></div> <h4 id="automated-quality-checks" class="position-relative d-flex align-items-center group"> Automated Quality Checks <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="automated-quality-checks" aria-haspopup="dialog" aria-label="Share link: Automated Quality Checks"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"># Python - automated quality monitoring from dataclasses import dataclass from typing import List import asyncio @dataclass class QualityCheck: name: str query: str threshold: float # Acceptable quality score (0-100) critical: bool = False class QualityMonitor: def __init__(self, client: Client): self.client = client self.checks = [ QualityCheck( "Email Completeness", "MATCH (p:Person) RETURN COUNT(p.email) * 100.0 / COUNT(*) AS score", threshold=95.0, critical=True ), QualityCheck( "Valid Email Format", """MATCH (p:Person) WHERE p.email IS NOT NULL RETURN SUM(CASE WHEN p.email ~ '^[^@]+@[^@]+\.[^@]+$' THEN 1 ELSE 0 END) * 100.0 / COUNT(*) AS score""", threshold=99.0, critical=True ), QualityCheck( "Age Validity", """MATCH (p:Person) WHERE p.age IS NOT NULL RETURN SUM(CASE WHEN p.age >= 0 AND p.age <= 150 THEN 1 ELSE 0 END) * 100.0 / COUNT(*) AS score""", threshold=99.5 ), ] async def run_check(self, check: QualityCheck) -> dict: result = await self.conn.execute(check.query) score = result.bindings[0]['score'] passed = score >= check.threshold return { "name": check.name, "score": score, "threshold": check.threshold, "passed": passed, "critical": check.critical } async def run_all_checks(self) -> dict: results = await asyncio.gather(*[ self.run_check(check) for check in self.checks ]) failed_critical = [r for r in results if not r['passed'] and r['critical']] return { "timestamp": datetime.now().isoformat(), "checks": results, "overall_passed": len(failed_critical) == 0, "failed_critical_count": len(failed_critical) } # Usage async def main(): client = Client(host="localhost", port=3141) async with client.connection() as conn: monitor = QualityMonitor(client) report = await monitor.run_all_checks() print(f"Quality Report - {report['timestamp']}") print(f"Overall Status: {'PASS' if report['overall_passed'] else 'FAIL'}") for check in report['checks']: status = '✓' if check['passed'] else '✗' critical = ' [CRITICAL]' if check['critical'] else '' print(f"{status} {check['name']}: {check['score']:.2f}% " f"(threshold: {check['threshold']}%){critical}") </code></pre></div> <h3 id="data-governance" class="position-relative d-flex align-items-center group"> Data Governance <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="data-governance" aria-haspopup="dialog" aria-label="Share link: Data Governance"> Share link </button> </h3>Establish policies and procedures for maintaining data quality. <h4 id="quality-rules-documentation" class="position-relative d-flex align-items-center group"> Quality Rules Documentation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="quality-rules-documentation" aria-haspopup="dialog" aria-label="Share link: Quality Rules Documentation"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"># Define quality rules as code QUALITY_RULES = { "person_email": { "description": "Every person must have a valid email address", "check": "p.email IS NOT NULL AND p.email ~ '^[^@]+@[^@]+\\.[^@]+$'", "severity": "critical", "owner": "[email protected]" }, "order_total": { "description": "Order total must match sum of line items", "check": "o.total = (MATCH (o)-[:HAS_ITEM]->(i) RETURN SUM(i.price * i.quantity))", "severity": "high", "owner": "[email protected]" }, } </code></pre></div> <h4 id="audit-trail" class="position-relative d-flex align-items-center group"> Audit Trail <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="audit-trail" aria-haspopup="dialog" aria-label="Share link: Audit Trail"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Track data quality issues over time CREATE NODE TYPE DataQualityIssue ( id STRING DEFAULT gen_random_uuid(), issue_type STRING NOT NULL, severity STRING CHECK (severity IN ('low', 'medium', 'high', 'critical')), detected_at TIMESTAMP NOT NULL DEFAULT NOW(), resolved_at TIMESTAMP, entity_type STRING NOT NULL, entity_id STRING NOT NULL, description TEXT NOT NULL, resolution TEXT ); -- Log quality issue INSERT (issue:DataQualityIssue { issue_type: 'invalid_email', severity: 'high', entity_type: 'Person', entity_id: 'person-123', description: 'Email format validation failed' }); </code></pre></div> <h3 id="best-practices" class="position-relative d-flex align-items-center group"> Best Practices <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="best-practices" aria-haspopup="dialog" aria-label="Share link: Best Practices"> Share link </button> </h3><ol> <li>Define Quality Standards Early: Establish quality metrics during schema design</li> <li>Automate Quality Checks: Run validation on every data import and update</li> <li>Profile Regularly: Schedule weekly or daily data profiling jobs</li> <li>Track Metrics Over Time: Monitor quality trends to detect degradation</li> <li>Document Quality Rules: Maintain a catalog of what constitutes “quality”</li> <li>Assign Ownership: Each data domain should have a quality owner</li> <li>Prevent at Source: Validate data at input rather than fixing later</li> <li>Clean Incrementally: Don’t wait for major cleansing projects</li> <li>Version Quality Rules: Track changes to quality definitions</li> <li>Alert on Critical Issues: Notify stakeholders when quality drops below thresholds</li> </ol> <h3 id="common-quality-anti-patterns" class="position-relative d-flex align-items-center group"> Common Quality Anti-Patterns <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="common-quality-anti-patterns" aria-haspopup="dialog" aria-label="Share link: Common Quality Anti-Patterns"> Share link </button> </h3>Accepting “good enough” data: Quality degrades exponentially over time Manual cleansing: Doesn’t scale, introduces human error Ignoring outliers: Outliers often indicate deeper quality issues No monitoring: Quality issues go unnoticed until crisis Fixing symptoms not causes: Cleanse source data, not just database <h3 id="related-topics" class="position-relative d-flex align-items-center group"> Related Topics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="related-topics" aria-haspopup="dialog" aria-label="Share link: Related Topics"> Share link </button> </h3><ul> <li><a href="/tags/validation" >Validation</a> - Data validation techniques</li> <li><a href="/tags/constraints" >Constraints</a> - Schema constraint enforcement</li> <li><a href="/tags/etl" >ETL</a> - Data integration quality</li> <li><a href="/tags/monitoring" >Monitoring</a> - System monitoring</li> <li><a href="/tags/governance/" >Data Governance</a> - Governance policies</li> <li><a href="/tags/testing" >Testing</a> - Quality testing strategies</li> </ul>

Popular

Related Articles