Multi-Datacenter Deployment and Replication

<h2 id="multi-datacenter-deployment-and-replication" class="position-relative d-flex align-items-center group"> Multi-Datacenter Deployment and Replication <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="multi-datacenter-deployment-and-replication" aria-haspopup="dialog" aria-label="Share link: Multi-Datacenter Deployment and Replication"> Share link </button> </h2><div id="headingShareModal" class="heading-share-modal" role="dialog" aria-modal="true" aria-labelledby="headingShareTitle" hidden> <div class="hsm-dialog" role="document"> <div class="hsm-header"> <h2 id="headingShareTitle" class="h6 mb-0 fw-bold">Share this section</h2> <button type="button" class="hsm-close" aria-label="Close"> </button> </div> <div class="hsm-body"> <label for="headingShareInput" class="form-label small text-muted mb-1 text-uppercase fw-bold" style="font-size: 0.7rem; letter-spacing: 0.5px;">Permalink</label> <div class="input-group mb-4 hsm-url-group"> <input id="headingShareInput" type="text" class="form-control font-monospace" readonly aria-readonly="true" style="font-size: 0.85rem;" /> <button class="btn btn-primary hsm-copy" type="button" aria-label="Copy" title="Copy"> </button> </div> <div class="small fw-bold mb-2 text-muted text-uppercase" style="font-size: 0.7rem; letter-spacing: 0.5px;">Share via</div> <div class="hsm-share-grid"> <a id="share-twitter" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Twitter </a> <a id="share-linkedin" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> LinkedIn </a> <a id="share-facebook" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Facebook </a> </div> </div> </div> </div> <style> .heading-share-modal { position: fixed; inset: 0; display: flex; justify-content: center; align-items: center; background: rgba(0, 0, 0, 0.6); z-index: 1050; padding: 1rem; backdrop-filter: blur(4px); -webkit-backdrop-filter: blur(4px); } .heading-share-modal[hidden] { display: none !important; } .hsm-dialog { max-width: 420px; width: 100%; background: var(--bs-body-bg, #fff); color: var(--bs-body-color, #212529); border: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); border-radius: 1rem; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); overflow: hidden; animation: hsm-fade-in 0.2s ease-out; } @keyframes hsm-fade-in { from { opacity: 0; transform: scale(0.95); } to { opacity: 1; transform: scale(1); } } [data-bs-theme="dark"] .hsm-dialog { background: #1e293b; border-color: rgba(255,255,255,0.1); color: #f8f9fa; } .hsm-header { display: flex; justify-content: space-between; align-items: center; padding: 1rem 1.5rem; border-bottom: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); background: rgba(0,0,0,0.02); } [data-bs-theme="dark"] .hsm-header { background: rgba(255,255,255,0.02); border-color: rgba(255,255,255,0.1); } .hsm-close { background: transparent; border: none; color: inherit; opacity: 0.5; padding: 0.25rem 0.5rem; border-radius: 0.25rem; font-size: 1.2rem; line-height: 1; transition: opacity 0.2s; } .hsm-close:hover { opacity: 1; } .hsm-body { padding: 1.5rem; } .hsm-url-group { display: flex !important; align-items: stretch; } .hsm-url-group .form-control { flex: 1; min-width: 0; margin: 0; background: var(--bs-secondary-bg, #f8f9fa); border-color: var(--bs-border-color, #dee2e6); border-top-right-radius: 0; border-bottom-right-radius: 0; height: 42px; } .hsm-url-group .btn { flex: 0 0 auto; margin: 0; margin-left: -1px; border-top-left-radius: 0; border-bottom-left-radius: 0; height: 42px; display: flex; align-items: center; justify-content: center; padding: 0 1.25rem; z-index: 2; } [data-bs-theme="dark"] .hsm-url-group .form-control { background: #0f172a; border-color: #334155; color: #e2e8f0; } .hsm-share-grid { display: flex; flex-direction: column; gap: 0.5rem; } .hsm-share-grid .btn { display: flex; align-items: center; justify-content: center; font-size: 0.9rem; padding: 0.6rem; border-color: var(--bs-border-color); width: 100%; } [data-bs-theme="dark"] .hsm-share-grid .btn { color: #e2e8f0; border-color: #475569; } [data-bs-theme="dark"] .hsm-share-grid .btn:hover { background: #334155; border-color: #cbd5e1; } </style> <script> (function(){ const modal = document.getElementById('headingShareModal'); if(!modal) return; const input = modal.querySelector('#headingShareInput'); const copyBtn = modal.querySelector('.hsm-copy'); const twitter = modal.querySelector('#share-twitter'); const linkedin = modal.querySelector('#share-linkedin'); const facebook = modal.querySelector('#share-facebook'); const closeBtn = modal.querySelector('.hsm-close'); let lastFocus=null; let trapBound=false; function buildUrl(id){ return window.location.origin + window.location.pathname + '#' + id; } function isOpen(){ return !modal.hasAttribute('hidden'); } function hydrate(id){ const url=buildUrl(id); input.value=url; const enc=encodeURIComponent(url); const text=encodeURIComponent(document.title); if(twitter) twitter.href=`https://twitter.com/intent/tweet?url=${enc}&text=${text}`; if(linkedin) linkedin.href=`https://www.linkedin.com/sharing/share-offsite/?url=${enc}`; if(facebook) facebook.href=`https://www.facebook.com/sharer/sharer.php?u=${enc}`; } function openModal(id){ lastFocus=document.activeElement; hydrate(id); if(!isOpen()){ modal.removeAttribute('hidden'); } requestAnimationFrame(()=>{ input.focus(); }); trapFocus(); } function closeModal(){ if(!isOpen()) return; modal.setAttribute('hidden',''); if(lastFocus && typeof lastFocus.focus==='function') lastFocus.focus(); } function copyCurrent(){ try{ navigator.clipboard.writeText(input.value).then(()=>feedback(true),()=>fallback()); } catch(e){ fallback(); } } function fallback(){ input.select(); try{ document.execCommand('copy'); feedback(true);}catch(e){ feedback(false);} } function feedback(ok){ if(!copyBtn) return; const icon=copyBtn.querySelector('i'); if(!icon) return; const prev=copyBtn.getAttribute('data-prev')||icon.className; if(!copyBtn.getAttribute('data-prev')) copyBtn.setAttribute('data-prev',prev); icon.className= ok ? 'fa-duotone fa-clipboard-check':'fa-duotone fa-circle-exclamation'; setTimeout(()=>{ icon.className=prev; },1800); } function handleShareClick(e){ e.preventDefault(); const btn=e.currentTarget; const id=btn.getAttribute('data-share-target'); if(id) openModal(id); } function bindShareButtons(){ document.querySelectorAll('.h-share').forEach(btn=>{ if(!btn.dataset.hShareBound){ btn.addEventListener('click', handleShareClick); btn.dataset.hShareBound='1'; } }); } bindShareButtons(); if(document.readyState==='loading'){ document.addEventListener('DOMContentLoaded', bindShareButtons); } else { requestAnimationFrame(bindShareButtons); } document.addEventListener('click', function(e){ const shareBtn=e.target.closest && e.target.closest('.h-share'); if(shareBtn && !shareBtn.dataset.hShareBound){ handleShareClick.call(shareBtn, e); } }, true); document.addEventListener('click', e=>{ if(e.target===modal) closeModal(); if(e.target.closest && e.target.closest('.hsm-close')){ e.preventDefault(); closeModal(); } if(copyBtn && (e.target===copyBtn || (e.target.closest && e.target.closest('.hsm-copy')))) { e.preventDefault(); copyCurrent(); } }); document.addEventListener('keydown', e=>{ if(e.key==='Escape' && isOpen()) closeModal(); }); function trapFocus(){ if(trapBound) return; trapBound=true; modal.addEventListener('keydown', f=>{ if(f.key==='Tab' && isOpen()){ const focusable=[...modal.querySelectorAll('a[href],button,input,textarea,select,[tabindex]:not([tabindex="-1"])')].filter(el=>!el.hasAttribute('disabled')); if(!focusable.length) return; const first=focusable[0]; const last=focusable[focusable.length-1]; if(f.shiftKey && document.activeElement===first){ f.preventDefault(); last.focus(); } else if(!f.shiftKey && document.activeElement===last){ f.preventDefault(); first.focus(); } } }); } if(closeBtn) closeBtn.addEventListener('click', e=>{ e.preventDefault(); closeModal(); }); })(); </script>Deploy Geode across multiple datacenters for high availability, disaster recovery, and global data distribution. <h3 id="overview" class="position-relative d-flex align-items-center group"> Overview <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="overview" aria-haspopup="dialog" aria-label="Share link: Overview"> Share link </button> </h3>Geode supports several multi-datacenter deployment patterns: <ol> <li>Federation - Distributed query coordination across shards</li> <li>CDC Replication - Change Data Capture for asynchronous replication</li> <li>Active-Active - Multiple writeable datacenters</li> <li>Active-Passive - Primary datacenter with standby replicas</li> <li>Disaster Recovery - Backup datacenter for failover</li> </ol> <h4 id="key-features" class="position-relative d-flex align-items-center group"> Key Features <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="key-features" aria-haspopup="dialog" aria-label="Share link: Key Features"> Share link </button> </h4><ul> <li>High Availability (HA): Automatic failover and redundancy</li> <li>Geographic Distribution: Deploy close to users for low latency</li> <li>Disaster Recovery: Survive datacenter failures</li> <li>Horizontal Scaling: Distribute workload across regions</li> <li>Consistency Models: Eventual, quorum, or strong consistency</li> </ul> <h3 id="architecture-patterns" class="position-relative d-flex align-items-center group"> Architecture Patterns <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="architecture-patterns" aria-haspopup="dialog" aria-label="Share link: Architecture Patterns"> Share link </button> </h3> <h4 id="pattern-1-federation-query-distribution" class="position-relative d-flex align-items-center group"> Pattern 1: Federation (Query Distribution) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="pattern-1-federation-query-distribution" aria-haspopup="dialog" aria-label="Share link: Pattern 1: Federation (Query Distribution)"> Share link </button> </h4>Use Case: Horizontal scaling within or across datacenters Federation distributes queries across multiple shards without data replication. Each shard contains a subset of data. <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">┌─────────────────────────────────────────────────┐ │ Query Coordinator │ │ (Distributed Query Engine) │ └──────┬──────────┬──────────┬──────────┬─────────┘ │ │ │ │ ┌──▼──┐ ┌──▼──┐ ┌──▼──┐ ┌──▼──┐ │Shard│ │Shard│ │Shard│ │Shard│ │ 1 │ │ 2 │ │ 3 │ │ 4 │ └─────┘ └─────┘ └─────┘ └─────┘ DC-East DC-East DC-West DC-West </code></pre></div>Characteristics: <ul> <li>Data partitioned across shards</li> <li>Queries distributed and results merged</li> <li>No replication (each record exists once)</li> <li>Best for read-heavy workloads</li> </ul> Configuration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">federation: enabled: true coordinator: true shards: - id: 'shard1-east' endpoint: 'shard1.us-east.internal:3141' weight: 1.0 datacenter: 'us-east-1' - id: 'shard2-east' endpoint: 'shard2.us-east.internal:3141' weight: 1.0 datacenter: 'us-east-1' - id: 'shard3-west' endpoint: 'shard3.us-west.internal:3141' weight: 1.0 datacenter: 'us-west-2' - id: 'shard4-west' endpoint: 'shard4.us-west.internal:3141' weight: 1.0 datacenter: 'us-west-2' query: timeout: '30s' max_concurrent: 100 retry_attempts: 3 retry_delay: '1s' </code></pre></div> <h4 id="pattern-2-active-passive-disaster-recovery" class="position-relative d-flex align-items-center group"> Pattern 2: Active-Passive (Disaster Recovery) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="pattern-2-active-passive-disaster-recovery" aria-haspopup="dialog" aria-label="Share link: Pattern 2: Active-Passive (Disaster Recovery)"> Share link </button> </h4>Use Case: Production + disaster recovery standby Primary datacenter handles all traffic. Secondary datacenter replicates via CDC for failover. <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">┌──────────────────────┐ ┌──────────────────────┐ │ Primary DC │ │ Secondary DC │ │ (Active) │ │ (Passive/Standby) │ │ │ │ │ │ ┌────────────┐ │ CDC │ ┌────────────┐ │ │ │ Geode ├──────┼─────────►│ │ Geode │ │ │ │ Primary │ │ Stream │ │ Replica │ │ │ └────────────┘ │ │ └────────────┘ │ │ │ │ │ │ Writes + Reads │ │ Reads Only │ └──────────────────────┘ └──────────────────────┘ us-east-1 us-west-2 </code></pre></div>Characteristics: <ul> <li>One writable primary datacenter</li> <li>Asynchronous replication to standby</li> <li>Manual or automatic failover</li> <li>Best for disaster recovery</li> </ul> Configuration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Primary datacenter (us-east-1) cdc: enabled: true sinks: - type: 'kafka' config: brokers: 'kafka.us-east.internal:9092' topic: 'geode-changes' compression: 'zstd' acks: 'all' retention_period_ms: 604800000 # 7 days batch_size: 1000 flush_interval_ms: 1000 # Secondary datacenter (us-west-2) - CDC consumer cdc_consumer: enabled: true source: type: 'kafka' brokers: 'kafka.us-east.internal:9092' topic: 'geode-changes' group_id: 'geode-replica-us-west' apply_mode: 'async' batch_size: 1000 conflict_resolution: 'source_wins' </code></pre></div> <h4 id="pattern-3-active-active-multi-master" class="position-relative d-flex align-items-center group"> Pattern 3: Active-Active (Multi-Master) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="pattern-3-active-active-multi-master" aria-haspopup="dialog" aria-label="Share link: Pattern 3: Active-Active (Multi-Master)"> Share link </button> </h4>Use Case: Multiple writable datacenters for global distribution Both datacenters accept writes. Bidirectional CDC keeps them synchronized. <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">┌──────────────────────┐ ┌──────────────────────┐ │ Datacenter 1 │◄────────►│ Datacenter 2 │ │ (Active) │ CDC │ (Active) │ │ │ Sync │ │ │ ┌────────────┐ │ │ ┌────────────┐ │ │ │ Geode │◄─────┼─────────►│ │ Geode │ │ │ │ Primary │ │ │ │ Primary │ │ │ └────────────┘ │ │ └────────────┘ │ │ │ │ │ │ Writes + Reads │ │ Writes + Reads │ └──────────────────────┘ └──────────────────────┘ us-east-1 eu-west-1 </code></pre></div>Characteristics: <ul> <li>Both datacenters accept writes</li> <li>Bidirectional asynchronous replication</li> <li>Conflict resolution required</li> <li>Best for global low-latency writes</li> </ul> Configuration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Datacenter 1 (us-east-1) cdc: enabled: true sinks: - type: 'kafka' config: brokers: 'kafka-global.internal:9092' topic: 'geode-changes-dc1' cdc_consumer: enabled: true source: type: 'kafka' brokers: 'kafka-global.internal:9092' topic: 'geode-changes-dc2' # Consume from DC2 conflict_resolution: 'last_write_wins' # or 'custom' # Datacenter 2 (eu-west-1) cdc: enabled: true sinks: - type: 'kafka' config: brokers: 'kafka-global.internal:9092' topic: 'geode-changes-dc2' cdc_consumer: enabled: true source: type: 'kafka' brokers: 'kafka-global.internal:9092' topic: 'geode-changes-dc1' # Consume from DC1 conflict_resolution: 'last_write_wins' </code></pre></div> <h4 id="pattern-4-hybrid-federation--replication" class="position-relative d-flex align-items-center group"> Pattern 4: Hybrid (Federation + Replication) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="pattern-4-hybrid-federation--replication" aria-haspopup="dialog" aria-label="Share link: Pattern 4: Hybrid (Federation &#43; Replication)"> Share link </button> </h4>Use Case: Regional sharding with local replication Combine federation for query distribution with replication for high availability. <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">Region: US-EAST Region: US-WEST ┌──────────────────┐ ┌──────────────────┐ │ Coordinator │◄─────────────►│ Coordinator │ │ │ Federation │ │ │ ┌────┐ ┌────┐ │ │ ┌────┐ ┌────┐ │ │ │Shd1│ │Rep1│ │ │ │Shd2│ │Rep2│ │ │ └────┘ └────┘ │ │ └────┘ └────┘ │ │ ▲ ▲ │ │ ▲ ▲ │ │ └───CDC─┘ │ │ └───CDC─┘ │ └──────────────────┘ └──────────────────┘ </code></pre></div>Characteristics: <ul> <li>Data partitioned by region (sharding)</li> <li>Local replication for HA within region</li> <li>Cross-region federation for global queries</li> <li>Best for global scale + high availability</li> </ul> <h3 id="deployment-guide" class="position-relative d-flex align-items-center group"> Deployment Guide <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="deployment-guide" aria-haspopup="dialog" aria-label="Share link: Deployment Guide"> Share link </button> </h3> <h4 id="prerequisites" class="position-relative d-flex align-items-center group"> Prerequisites <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="prerequisites" aria-haspopup="dialog" aria-label="Share link: Prerequisites"> Share link </button> </h4><ul> <li>Network: Cross-datacenter connectivity (VPN or dedicated links)</li> <li>Latency: <100ms between datacenters (recommended)</li> <li>Bandwidth: 100 Mbps+ for CDC replication</li> <li>Storage: Sufficient capacity for replication lag</li> <li>Monitoring: Cross-datacenter monitoring setup</li> </ul> <h4 id="step-1-network-setup" class="position-relative d-flex align-items-center group"> Step 1: Network Setup <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="step-1-network-setup" aria-haspopup="dialog" aria-label="Share link: Step 1: Network Setup"> Share link </button> </h4> <h5 id="configure-cross-datacenter-connectivity" class="position-relative d-flex align-items-center group"> Configure Cross-Datacenter Connectivity <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="configure-cross-datacenter-connectivity" aria-haspopup="dialog" aria-label="Share link: Configure Cross-Datacenter Connectivity"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Example: WireGuard VPN between datacenters # On DC1 (us-east-1) wg genkey | tee dc1-private.key | wg pubkey > dc1-public.key # /etc/wireguard/wg0.conf [Interface] Address = 10.0.1.1/24 PrivateKey = <dc1-private-key> ListenPort = 51820 [Peer] PublicKey = <dc2-public-key> Endpoint = dc2-public-ip:51820 AllowedIPs = 10.0.2.0/24 PersistentKeepalive = 25 # Start WireGuard wg-quick up wg0 </code></pre></div> <h5 id="verify-connectivity" class="position-relative d-flex align-items-center group"> Verify Connectivity <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="verify-connectivity" aria-haspopup="dialog" aria-label="Share link: Verify Connectivity"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># From DC1, ping DC2 ping -c 5 10.0.2.1 # Test latency ping -c 100 10.0.2.1 | tail -1 # Test bandwidth iperf3 -s # On DC2 iperf3 -c 10.0.2.1 -t 30 # On DC1 </code></pre></div> <h4 id="step-2-deploy-federation-query-distribution" class="position-relative d-flex align-items-center group"> Step 2: Deploy Federation (Query Distribution) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="step-2-deploy-federation-query-distribution" aria-haspopup="dialog" aria-label="Share link: Step 2: Deploy Federation (Query Distribution)"> Share link </button> </h4> <h5 id="deploy-coordinator-node" class="position-relative d-flex align-items-center group"> Deploy Coordinator Node <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="deploy-coordinator-node" aria-haspopup="dialog" aria-label="Share link: Deploy Coordinator Node"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># On coordinator node (DC1) cat > /etc/geode/geode.yaml <<EOF server: listen: '0.0.0.0:3141' data_dir: '/var/lib/geode/coordinator' federation: enabled: true coordinator: true shards: - id: 'shard1-dc1' endpoint: 'geode-shard1.dc1.internal:3141' weight: 1.0 datacenter: 'us-east-1' - id: 'shard2-dc1' endpoint: 'geode-shard2.dc1.internal:3141' weight: 1.0 datacenter: 'us-east-1' - id: 'shard1-dc2' endpoint: 'geode-shard1.dc2.internal:3141' weight: 1.0 datacenter: 'us-west-2' - id: 'shard2-dc2' endpoint: 'geode-shard2.dc2.internal:3141' weight: 1.0 datacenter: 'us-west-2' query: timeout: '30s' max_concurrent: 100 retry_attempts: 3 logging: level: 'info' format: 'json' EOF # Start coordinator geode serve --config /etc/geode/geode.yaml </code></pre></div> <h5 id="deploy-shard-nodes" class="position-relative d-flex align-items-center group"> Deploy Shard Nodes <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="deploy-shard-nodes" aria-haspopup="dialog" aria-label="Share link: Deploy Shard Nodes"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># On each shard node cat > /etc/geode/geode.yaml <<EOF server: listen: '0.0.0.0:3141' data_dir: '/var/lib/geode/shard1' federation: enabled: true coordinator: false # This is a shard, not coordinator storage: page_cache_size: '8GB' wal_sync_interval: '100ms' logging: level: 'info' format: 'json' EOF # Start shard geode serve --config /etc/geode/geode.yaml </code></pre></div> <h5 id="verify-federation" class="position-relative d-flex align-items-center group"> Verify Federation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="verify-federation" aria-haspopup="dialog" aria-label="Share link: Verify Federation"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Connect to coordinator geode shell --server geode-coordinator.dc1.internal:3141 -- Run distributed query MATCH (n:Person) WHERE n.age > 30 RETURN count(n); -- Check query distribution EXPLAIN MATCH (n:Person) WHERE n.age > 30 RETURN count(n); -- Should show shards involved in query </code></pre></div> <h4 id="step-3-deploy-cdc-replication" class="position-relative d-flex align-items-center group"> Step 3: Deploy CDC Replication <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="step-3-deploy-cdc-replication" aria-haspopup="dialog" aria-label="Share link: Step 3: Deploy CDC Replication"> Share link </button> </h4> <h5 id="setup-kafka-for-cdc-stream" class="position-relative d-flex align-items-center group"> Setup Kafka for CDC Stream <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="setup-kafka-for-cdc-stream" aria-haspopup="dialog" aria-label="Share link: Setup Kafka for CDC Stream"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Deploy Kafka cluster (cross-datacenter accessible) # kafka-dc1.internal, kafka-dc2.internal # Create CDC topic kafka-topics --create \ --bootstrap-server kafka-dc1.internal:9092 \ --topic geode-cdc-events \ --partitions 12 \ --replication-factor 3 \ --config retention.ms=604800000 # 7 days </code></pre></div> <h5 id="configure-primary-for-cdc" class="position-relative d-flex align-items-center group"> Configure Primary for CDC <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="configure-primary-for-cdc" aria-haspopup="dialog" aria-label="Share link: Configure Primary for CDC"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Primary datacenter (us-east-1) # /etc/geode/geode.yaml cdc: enabled: true sinks: - type: 'kafka' config: brokers: 'kafka-dc1.internal:9092,kafka-dc2.internal:9092' topic: 'geode-cdc-events' compression: 'zstd' batch_size: 16384 linger_ms: 100 acks: 'all' retention_period_ms: 604800000 # 7 days batch_size: 1000 flush_interval_ms: 1000 # Adaptive batching for high throughput adaptive_flush: true min_batch_size: 256 max_batch_size: 32000 # Backpressure management adaptive_backpressure: true backpressure_high_pct: 0.85 backpressure_low_pct: 0.30 # Filtering (optional) include_before_image: true include_metadata: true </code></pre></div> <h5 id="configure-replica-for-cdc-consumption" class="position-relative d-flex align-items-center group"> Configure Replica for CDC Consumption <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="configure-replica-for-cdc-consumption" aria-haspopup="dialog" aria-label="Share link: Configure Replica for CDC Consumption"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Replica datacenter (us-west-2) # /etc/geode/geode.yaml cdc_consumer: enabled: true source: type: 'kafka' brokers: 'kafka-dc1.internal:9092,kafka-dc2.internal:9092' topic: 'geode-cdc-events' group_id: 'geode-replica-dc2' auto_offset_reset: 'earliest' # or 'latest' for new data only apply_mode: 'async' batch_size: 1000 workers: 4 # Parallel consumers # Conflict resolution conflict_resolution: 'source_wins' # or 'timestamp', 'custom' # Resume on failure checkpoint_interval_ms: 5000 </code></pre></div> <h5 id="verify-cdc-replication" class="position-relative d-flex align-items-center group"> Verify CDC Replication <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="verify-cdc-replication" aria-haspopup="dialog" aria-label="Share link: Verify CDC Replication"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># On primary geode shell --server geode-primary.dc1.internal:3141 </code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Create test data CREATE GRAPH TestReplication; USE TestReplication; CREATE (:Person {name: "Alice", timestamp: timestamp()}); </code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Wait 5-10 seconds for replication # On replica geode shell --server geode-replica.dc2.internal:3141 </code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">USE TestReplication; MATCH (p:Person {name: "Alice"}) RETURN p.name, p.timestamp; -- Should return Alice with timestamp </code></pre></div> <h4 id="step-4-configure-high-availability" class="position-relative d-flex align-items-center group"> Step 4: Configure High Availability <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="step-4-configure-high-availability" aria-haspopup="dialog" aria-label="Share link: Step 4: Configure High Availability"> Share link </button> </h4> <h5 id="enable-cluster-heartbeats" class="position-relative d-flex align-items-center group"> Enable Cluster Heartbeats <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="enable-cluster-heartbeats" aria-haspopup="dialog" aria-label="Share link: Enable Cluster Heartbeats"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># On all nodes cluster: heartbeat_interval: '5s' election_timeout: '30s' replication_factor: 3 # Number of replicas # Consistency settings consistency_level: 'quorum' # or 'eventual', 'strong' read_preference: 'primary_preferred' # or 'primary', 'secondary', 'nearest' </code></pre></div> <h5 id="deploy-load-balancer" class="position-relative d-flex align-items-center group"> Deploy Load Balancer <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="deploy-load-balancer" aria-haspopup="dialog" aria-label="Share link: Deploy Load Balancer"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"># HAProxy configuration for Geode # /etc/haproxy/haproxy.cfg global daemon maxconn 10000 defaults mode tcp timeout connect 5000ms timeout client 50000ms timeout server 50000ms balance roundrobin frontend geode_frontend bind *:3141 default_backend geode_coordinators backend geode_coordinators # Health checks option tcp-check # Coordinators in DC1 server coord1-dc1 geode-coord1.dc1.internal:3141 check inter 2000 rise 2 fall 3 server coord2-dc1 geode-coord2.dc1.internal:3141 check inter 2000 rise 2 fall 3 # Coordinators in DC2 (backup) server coord1-dc2 geode-coord1.dc2.internal:3141 check inter 2000 rise 2 fall 3 backup </code></pre></div> <h5 id="test-failover" class="position-relative d-flex align-items-center group"> Test Failover <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="test-failover" aria-haspopup="dialog" aria-label="Share link: Test Failover"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Simulate primary coordinator failure ssh geode-coord1.dc1.internal "systemctl stop geode" # Verify failover (should connect to coord2) geode shell --server geode-lb.internal:3141 </code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">RETURN 1 AS health_check; -- Should succeed via backup coordinator </code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Restore primary ssh geode-coord1.dc1.internal "systemctl start geode" </code></pre></div> <h3 id="disaster-recovery-procedures" class="position-relative d-flex align-items-center group"> Disaster Recovery Procedures <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="disaster-recovery-procedures" aria-haspopup="dialog" aria-label="Share link: Disaster Recovery Procedures"> Share link </button> </h3> <h4 id="scenario-1-primary-datacenter-failure" class="position-relative d-flex align-items-center group"> Scenario 1: Primary Datacenter Failure <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="scenario-1-primary-datacenter-failure" aria-haspopup="dialog" aria-label="Share link: Scenario 1: Primary Datacenter Failure"> Share link </button> </h4> <h5 id="automatic-failover-active-passive" class="position-relative d-flex align-items-center group"> Automatic Failover (Active-Passive) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="automatic-failover-active-passive" aria-haspopup="dialog" aria-label="Share link: Automatic Failover (Active-Passive)"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># 1. Detect primary failure curl -f https://geode-primary.dc1.internal:3141/health || echo "Primary down" # 2. Promote replica to primary ssh geode-replica.dc2.internal # Stop CDC consumer systemctl stop geode-cdc-consumer # Reconfigure as primary cat > /etc/geode/geode.yaml <<EOF server: listen: '0.0.0.0:3141' # Enable CDC producer (was consumer) cdc: enabled: true sinks: - type: 'kafka' config: brokers: 'kafka-dc2.internal:9092' topic: 'geode-cdc-events' EOF # Restart as primary systemctl restart geode # 3. Update DNS/load balancer # Point geode.example.com -> geode-replica.dc2.internal # 4. Verify new primary geode shell --server geode-replica.dc2.internal:3141 </code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">CREATE (:FailoverTest {timestamp: timestamp()}); -- Should succeed </code></pre></div> <h5 id="restore-primary-datacenter" class="position-relative d-flex align-items-center group"> Restore Primary Datacenter <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="restore-primary-datacenter" aria-haspopup="dialog" aria-label="Share link: Restore Primary Datacenter"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># 1. Restore primary infrastructure # Network, power, hardware, etc. # 2. Restore from backup geode restore \ --source s3://geode-backups/latest \ --data-dir /var/lib/geode # 3. Catch up with CDC # Reconfigure as replica (reverse roles) cat > /etc/geode/geode.yaml <<EOF cdc_consumer: enabled: true source: type: 'kafka' topic: 'geode-cdc-events' auto_offset_reset: 'latest' # Only new changes EOF systemctl start geode # 4. Verify replication lag geode-admin replication-status # Wait for lag: 0 # 5. Failback (reverse roles again) # Promote DC1 to primary, demote DC2 to replica </code></pre></div> <h4 id="scenario-2-network-partition-split-brain" class="position-relative d-flex align-items-center group"> Scenario 2: Network Partition (Split Brain) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="scenario-2-network-partition-split-brain" aria-haspopup="dialog" aria-label="Share link: Scenario 2: Network Partition (Split Brain)"> Share link </button> </h4> <h5 id="detection" class="position-relative d-flex align-items-center group"> Detection <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="detection" aria-haspopup="dialog" aria-label="Share link: Detection"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Monitor cluster health geode-admin cluster-status # Output if partitioned: # DC1: 2 nodes connected # DC2: 2 nodes connected # ERROR: Split brain detected (quorum lost) </code></pre></div> <h5 id="resolution" class="position-relative d-flex align-items-center group"> Resolution <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="resolution" aria-haspopup="dialog" aria-label="Share link: Resolution"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># 1. Identify which partition has quorum # DC1: 2 nodes, DC2: 2 nodes (tie) # 2. Manual intervention: choose primary # Shut down DC2 nodes temporarily ssh geode-coord1.dc2.internal "systemctl stop geode" ssh geode-coord2.dc2.internal "systemctl stop geode" # 3. DC1 forms quorum # Verify writes work geode shell --server geode-coord1.dc1.internal:3141 </code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">CREATE (:SplitBrainTest {resolved: true}); </code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># 4. Restore DC2 as replicas # Restore data from DC1 geode-admin sync-replica \ --source geode-coord1.dc1.internal:3141 \ --target geode-coord1.dc2.internal:3141 # Start DC2 nodes systemctl start geode # 5. Re-enable replication systemctl start geode-cdc-consumer </code></pre></div> <h3 id="monitoring-and-observability" class="position-relative d-flex align-items-center group"> Monitoring and Observability <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="monitoring-and-observability" aria-haspopup="dialog" aria-label="Share link: Monitoring and Observability"> Share link </button> </h3> <h4 id="key-metrics" class="position-relative d-flex align-items-center group"> Key Metrics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="key-metrics" aria-haspopup="dialog" aria-label="Share link: Key Metrics"> Share link </button> </h4> <h5 id="replication-lag" class="position-relative d-flex align-items-center group"> Replication Lag <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="replication-lag" aria-haspopup="dialog" aria-label="Share link: Replication Lag"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"># Prometheus query geode_cdc_replication_lag_seconds{datacenter="dc2"} # Alert if lag > 60 seconds geode_cdc_replication_lag_seconds > 60 </code></pre></div> <h5 id="cdc-throughput" class="position-relative d-flex align-items-center group"> CDC Throughput <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="cdc-throughput" aria-haspopup="dialog" aria-label="Share link: CDC Throughput"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"># Events per second rate(geode_cdc_events_total[5m]) # Bytes per second rate(geode_cdc_bytes_total[5m]) </code></pre></div> <h5 id="federation-query-performance" class="position-relative d-flex align-items-center group"> Federation Query Performance <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="federation-query-performance" aria-haspopup="dialog" aria-label="Share link: Federation Query Performance"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"># Cross-datacenter query latency histogram_quantile(0.99, geode_federation_query_duration_seconds) # Shard availability geode_federation_shard_available{shard="shard1-dc2"} </code></pre></div> <h4 id="monitoring-dashboard" class="position-relative d-flex align-items-center group"> Monitoring Dashboard <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="monitoring-dashboard" aria-haspopup="dialog" aria-label="Share link: Monitoring Dashboard"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Import Grafana dashboard curl -X POST \ -H "Content-Type: application/json" \ -d @multi-dc-dashboard.json \ http://grafana.internal:3000/api/dashboards/db </code></pre></div>Dashboard Panels: <ul> <li>Replication lag per datacenter</li> <li>CDC event throughput</li> <li>Cross-datacenter network latency</li> <li>Shard health status</li> <li>Conflict resolution rate</li> <li>Query distribution across shards</li> </ul> <h4 id="alerting-rules" class="position-relative d-flex align-items-center group"> Alerting Rules <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="alerting-rules" aria-haspopup="dialog" aria-label="Share link: Alerting Rules"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Prometheus alerts groups: - name: multi_datacenter rules: - alert: HighReplicationLag expr: geode_cdc_replication_lag_seconds > 60 for: 5m annotations: summary: "CDC replication lag exceeds 60s" - alert: DatacenterUnavailable expr: up{job="geode", datacenter="dc2"} == 0 for: 2m annotations: summary: "Datacenter DC2 is unavailable" - alert: SplitBrainDetected expr: | count(geode_cluster_quorum{status="active"}) > 1 annotations: summary: "Multiple quorums detected (split brain)" </code></pre></div> <h3 id="conflict-resolution" class="position-relative d-flex align-items-center group"> Conflict Resolution <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="conflict-resolution" aria-haspopup="dialog" aria-label="Share link: Conflict Resolution"> Share link </button> </h3> <h4 id="conflict-types" class="position-relative d-flex align-items-center group"> Conflict Types <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="conflict-types" aria-haspopup="dialog" aria-label="Share link: Conflict Types"> Share link </button> </h4><ol> <li>Write-Write Conflicts: Same entity modified in both datacenters</li> <li>Delete-Write Conflicts: Entity deleted in DC1, modified in DC2</li> <li>Schema Conflicts: Schema changes in both datacenters</li> </ol> <h4 id="resolution-strategies" class="position-relative d-flex align-items-center group"> Resolution Strategies <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="resolution-strategies" aria-haspopup="dialog" aria-label="Share link: Resolution Strategies"> Share link </button> </h4> <h5 id="last-write-wins-timestamp-based" class="position-relative d-flex align-items-center group"> Last-Write-Wins (Timestamp-Based) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="last-write-wins-timestamp-based" aria-haspopup="dialog" aria-label="Share link: Last-Write-Wins (Timestamp-Based)"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">cdc_consumer: conflict_resolution: 'last_write_wins' timestamp_field: '_last_modified' </code></pre></div>Pros: Simple, deterministic Cons: May lose updates <h5 id="source-wins-primary-takes-precedence" class="position-relative d-flex align-items-center group"> Source-Wins (Primary Takes Precedence) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="source-wins-primary-takes-precedence" aria-haspopup="dialog" aria-label="Share link: Source-Wins (Primary Takes Precedence)"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">cdc_consumer: conflict_resolution: 'source_wins' </code></pre></div>Pros: Consistent with primary Cons: Replica writes lost <h5 id="custom-resolution" class="position-relative d-flex align-items-center group"> Custom Resolution <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="custom-resolution" aria-haspopup="dialog" aria-label="Share link: Custom Resolution"> Share link </button> </h5><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-zig" data-lang="zig">// Custom conflict handler pub fn resolveConflict( local: ChangeEvent, remote: ChangeEvent, ) !ResolvedEvent { // Business logic if (local.entity_type == .node) { // Merge properties return try mergeNodeProperties(local, remote); } // Default: last-write-wins return if (local.timestamp > remote.timestamp) local else remote; } </code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">cdc_consumer: conflict_resolution: 'custom' conflict_handler: '/etc/geode/conflict_handler.so' </code></pre></div> <h3 id="performance-tuning" class="position-relative d-flex align-items-center group"> Performance Tuning <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="performance-tuning" aria-haspopup="dialog" aria-label="Share link: Performance Tuning"> Share link </button> </h3> <h4 id="network-optimization" class="position-relative d-flex align-items-center group"> Network Optimization <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="network-optimization" aria-haspopup="dialog" aria-label="Share link: Network Optimization"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Increase batch sizes for cross-datacenter CDC cdc: batch_size: 32000 # Up from 1000 flush_interval_ms: 5000 # Buffer longer # Enable compression sinks: - type: 'kafka' config: compression: 'zstd' # Best compression ratio </code></pre></div> <h4 id="replication-throughput" class="position-relative d-flex align-items-center group"> Replication Throughput <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="replication-throughput" aria-haspopup="dialog" aria-label="Share link: Replication Throughput"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Parallel CDC consumers cdc_consumer: workers: 8 # Match number of cores batch_size: 5000 prefetch_count: 50 # Kafka prefetch </code></pre></div> <h4 id="query-routing" class="position-relative d-flex align-items-center group"> Query Routing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="query-routing" aria-haspopup="dialog" aria-label="Share link: Query Routing"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Prefer local shards for reads federation: query: locality_preference: true # Route to nearest datacenter shard_affinity: true # Sticky routing for related queries </code></pre></div> <h3 id="best-practices" class="position-relative d-flex align-items-center group"> Best Practices <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="best-practices" aria-haspopup="dialog" aria-label="Share link: Best Practices"> Share link </button> </h3> <h4 id="dos" class="position-relative d-flex align-items-center group"> Do&rsquo;s <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="dos" aria-haspopup="dialog" aria-label="Share link: Dos"> Share link </button> </h4>✅ Monitor replication lag - Alert if lag exceeds threshold ✅ Test failover regularly - Quarterly DR drills ✅ Use compression - Reduce cross-datacenter bandwidth ✅ Implement idempotency - Handle duplicate events ✅ Plan for split-brain - Clear resolution procedures ✅ Document runbooks - Failover and recovery steps ✅ Version CDC schema - Handle schema evolution <h4 id="donts" class="position-relative d-flex align-items-center group"> Don&rsquo;ts <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="donts" aria-haspopup="dialog" aria-label="Share link: Donts"> Share link </button> </h4>❌ Don’t ignore conflicts - Silent data loss ❌ Don’t skip backups - CDC isn’t backup ❌ Don’t overload network - Saturating cross-DC links ❌ Don’t assume synchronous - CDC is async ❌ Don’t forget monitoring - Blind to issues ❌ Don’t hard-code IPs - Use service discovery ❌ Don’t skip testing - Fail during real disaster <h3 id="troubleshooting" class="position-relative d-flex align-items-center group"> Troubleshooting <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="troubleshooting" aria-haspopup="dialog" aria-label="Share link: Troubleshooting"> Share link </button> </h3> <h4 id="replication-lag-increasing" class="position-relative d-flex align-items-center group"> Replication Lag Increasing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="replication-lag-increasing" aria-haspopup="dialog" aria-label="Share link: Replication Lag Increasing"> Share link </button> </h4>Symptoms: <code>geode_cdc_replication_lag_seconds</code> growing Causes: <ol> <li>Network congestion</li> <li>Kafka throughput insufficient</li> <li>Consumer too slow</li> </ol> Solutions: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Check network bandwidth iperf3 -c kafka.dc2.internal # Increase Kafka partitions kafka-topics --alter --partitions 24 --topic geode-cdc-events # Add CDC consumer workers # In geode.yaml cdc_consumer: workers: 16 # Increase parallelism </code></pre></div> <h4 id="split-brain-scenario" class="position-relative d-flex align-items-center group"> Split Brain Scenario <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="split-brain-scenario" aria-haspopup="dialog" aria-label="Share link: Split Brain Scenario"> Share link </button> </h4>Symptoms: Multiple nodes claim to be primary Solutions: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># 1. Identify nodes in each partition geode-admin cluster-status --all # 2. Choose primary partition (larger or DC1) # 3. Shut down minority partition for node in dc2-nodes; do ssh $node "systemctl stop geode" done # 4. Resync minority from primary geode-admin resync-cluster \ --source dc1 \ --targets dc2-nodes </code></pre></div> <h4 id="cdc-events-lost" class="position-relative d-flex align-items-center group"> CDC Events Lost <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="cdc-events-lost" aria-haspopup="dialog" aria-label="Share link: CDC Events Lost"> Share link </button> </h4>Symptoms: Missing data in replica Causes: <ol> <li>Kafka retention expired</li> <li>Consumer offset reset</li> <li>Network partition during write</li> </ol> Solutions: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Check Kafka retention kafka-topics --describe --topic geode-cdc-events # Restore from backup + replay CDC geode restore --source s3://backups/last-full geode-admin replay-cdc \ --from-timestamp "2024-01-20T00:00:00Z" \ --to-timestamp "now" </code></pre></div> <h3 id="next-steps" class="position-relative d-flex align-items-center group"> Next Steps <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="next-steps" aria-haspopup="dialog" aria-label="Share link: Next Steps"> Share link </button> </h3><ul> <li><a href="/docs/guides/backup-automation/" >Backup Automation</a> - Automated backup strategies</li> <li><a href="/docs/configuration/server-configuration/" >Server Configuration</a> - Federation config details</li> <li><a href="/docs/architecture/distributed-architecture/" >Distributed Architecture</a> - Architecture deep dive</li> <li><a href="/docs/ops/observability/" >Monitoring</a> - Comprehensive monitoring setup</li> <li><a href="/docs/performance/" >Performance Tuning</a> - Optimize multi-DC performance</li> </ul> <h3 id="reference" class="position-relative d-flex align-items-center group"> Reference <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="reference" aria-haspopup="dialog" aria-label="Share link: Reference"> Share link </button> </h3> <h4 id="configuration-files" class="position-relative d-flex align-items-center group"> Configuration Files <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="configuration-files" aria-haspopup="dialog" aria-label="Share link: Configuration Files"> Share link </button> </h4><ul> <li><a href="https://geodedb.com/docs/guides/multi-datacenter/#step-2-deploy-federation-query-distribution" >Federation Config</a> </li> <li><a href="https://geodedb.com/docs/guides/multi-datacenter/#step-3-deploy-cdc-replication" >CDC Config</a> </li> <li><a href="https://geodedb.com/docs/guides/multi-datacenter/#step-4-configure-high-availability" >HA Config</a> </li> </ul> <h4 id="cli-commands" class="position-relative d-flex align-items-center group"> CLI Commands <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="cli-commands" aria-haspopup="dialog" aria-label="Share link: CLI Commands"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Cluster management geode-admin cluster-status geode-admin shard-add --id shard3 --endpoint host:3141 geode-admin shard-remove --id shard1 # CDC management geode-admin cdc-status geode-admin cdc-reset-offset --topic geode-cdc-events geode-admin replay-cdc --from-timestamp <ts> # Replication management geode-admin replication-status geode-admin resync-replica --source <primary> --target <replica> geode-admin promote-replica --node <node-id> </code></pre></div><hr> License: Apache License 2.0 Copyright: 2024-2025 CodePros Last Updated: January 2026