High Availability Guide | Geode Database

<h2 id="high-availability-guide" class="position-relative d-flex align-items-center group"> High Availability Guide <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="high-availability-guide" aria-haspopup="dialog" aria-label="Share link: High Availability Guide"> Share link </button> </h2><div id="headingShareModal" class="heading-share-modal" role="dialog" aria-modal="true" aria-labelledby="headingShareTitle" hidden> <div class="hsm-dialog" role="document"> <div class="hsm-header"> <h2 id="headingShareTitle" class="h6 mb-0 fw-bold">Share this section</h2> <button type="button" class="hsm-close" aria-label="Close"> </button> </div> <div class="hsm-body"> <label for="headingShareInput" class="form-label small text-muted mb-1 text-uppercase fw-bold" style="font-size: 0.7rem; letter-spacing: 0.5px;">Permalink</label> <div class="input-group mb-4 hsm-url-group"> <input id="headingShareInput" type="text" class="form-control font-monospace" readonly aria-readonly="true" style="font-size: 0.85rem;" /> <button class="btn btn-primary hsm-copy" type="button" aria-label="Copy" title="Copy"> </button> </div> <div class="small fw-bold mb-2 text-muted text-uppercase" style="font-size: 0.7rem; letter-spacing: 0.5px;">Share via</div> <div class="hsm-share-grid"> <a id="share-twitter" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Twitter </a> <a id="share-linkedin" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> LinkedIn </a> <a id="share-facebook" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Facebook </a> </div> </div> </div> </div> <style> .heading-share-modal { position: fixed; inset: 0; display: flex; justify-content: center; align-items: center; background: rgba(0, 0, 0, 0.6); z-index: 1050; padding: 1rem; backdrop-filter: blur(4px); -webkit-backdrop-filter: blur(4px); } .heading-share-modal[hidden] { display: none !important; } .hsm-dialog { max-width: 420px; width: 100%; background: var(--bs-body-bg, #fff); color: var(--bs-body-color, #212529); border: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); border-radius: 1rem; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); overflow: hidden; animation: hsm-fade-in 0.2s ease-out; } @keyframes hsm-fade-in { from { opacity: 0; transform: scale(0.95); } to { opacity: 1; transform: scale(1); } } [data-bs-theme="dark"] .hsm-dialog { background: #1e293b; border-color: rgba(255,255,255,0.1); color: #f8f9fa; } .hsm-header { display: flex; justify-content: space-between; align-items: center; padding: 1rem 1.5rem; border-bottom: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); background: rgba(0,0,0,0.02); } [data-bs-theme="dark"] .hsm-header { background: rgba(255,255,255,0.02); border-color: rgba(255,255,255,0.1); } .hsm-close { background: transparent; border: none; color: inherit; opacity: 0.5; padding: 0.25rem 0.5rem; border-radius: 0.25rem; font-size: 1.2rem; line-height: 1; transition: opacity 0.2s; } .hsm-close:hover { opacity: 1; } .hsm-body { padding: 1.5rem; } .hsm-url-group { display: flex !important; align-items: stretch; } .hsm-url-group .form-control { flex: 1; min-width: 0; margin: 0; background: var(--bs-secondary-bg, #f8f9fa); border-color: var(--bs-border-color, #dee2e6); border-top-right-radius: 0; border-bottom-right-radius: 0; height: 42px; } .hsm-url-group .btn { flex: 0 0 auto; margin: 0; margin-left: -1px; border-top-left-radius: 0; border-bottom-left-radius: 0; height: 42px; display: flex; align-items: center; justify-content: center; padding: 0 1.25rem; z-index: 2; } [data-bs-theme="dark"] .hsm-url-group .form-control { background: #0f172a; border-color: #334155; color: #e2e8f0; } .hsm-share-grid { display: flex; flex-direction: column; gap: 0.5rem; } .hsm-share-grid .btn { display: flex; align-items: center; justify-content: center; font-size: 0.9rem; padding: 0.6rem; border-color: var(--bs-border-color); width: 100%; } [data-bs-theme="dark"] .hsm-share-grid .btn { color: #e2e8f0; border-color: #475569; } [data-bs-theme="dark"] .hsm-share-grid .btn:hover { background: #334155; border-color: #cbd5e1; } </style> <script> (function(){ const modal = document.getElementById('headingShareModal'); if(!modal) return; const input = modal.querySelector('#headingShareInput'); const copyBtn = modal.querySelector('.hsm-copy'); const twitter = modal.querySelector('#share-twitter'); const linkedin = modal.querySelector('#share-linkedin'); const facebook = modal.querySelector('#share-facebook'); const closeBtn = modal.querySelector('.hsm-close'); let lastFocus=null; let trapBound=false; function buildUrl(id){ return window.location.origin + window.location.pathname + '#' + id; } function isOpen(){ return !modal.hasAttribute('hidden'); } function hydrate(id){ const url=buildUrl(id); input.value=url; const enc=encodeURIComponent(url); const text=encodeURIComponent(document.title); if(twitter) twitter.href=`https://twitter.com/intent/tweet?url=${enc}&text=${text}`; if(linkedin) linkedin.href=`https://www.linkedin.com/sharing/share-offsite/?url=${enc}`; if(facebook) facebook.href=`https://www.facebook.com/sharer/sharer.php?u=${enc}`; } function openModal(id){ lastFocus=document.activeElement; hydrate(id); if(!isOpen()){ modal.removeAttribute('hidden'); } requestAnimationFrame(()=>{ input.focus(); }); trapFocus(); } function closeModal(){ if(!isOpen()) return; modal.setAttribute('hidden',''); if(lastFocus && typeof lastFocus.focus==='function') lastFocus.focus(); } function copyCurrent(){ try{ navigator.clipboard.writeText(input.value).then(()=>feedback(true),()=>fallback()); } catch(e){ fallback(); } } function fallback(){ input.select(); try{ document.execCommand('copy'); feedback(true);}catch(e){ feedback(false);} } function feedback(ok){ if(!copyBtn) return; const icon=copyBtn.querySelector('i'); if(!icon) return; const prev=copyBtn.getAttribute('data-prev')||icon.className; if(!copyBtn.getAttribute('data-prev')) copyBtn.setAttribute('data-prev',prev); icon.className= ok ? 'fa-duotone fa-clipboard-check':'fa-duotone fa-circle-exclamation'; setTimeout(()=>{ icon.className=prev; },1800); } function handleShareClick(e){ e.preventDefault(); const btn=e.currentTarget; const id=btn.getAttribute('data-share-target'); if(id) openModal(id); } function bindShareButtons(){ document.querySelectorAll('.h-share').forEach(btn=>{ if(!btn.dataset.hShareBound){ btn.addEventListener('click', handleShareClick); btn.dataset.hShareBound='1'; } }); } bindShareButtons(); if(document.readyState==='loading'){ document.addEventListener('DOMContentLoaded', bindShareButtons); } else { requestAnimationFrame(bindShareButtons); } document.addEventListener('click', function(e){ const shareBtn=e.target.closest && e.target.closest('.h-share'); if(shareBtn && !shareBtn.dataset.hShareBound){ handleShareClick.call(shareBtn, e); } }, true); document.addEventListener('click', e=>{ if(e.target===modal) closeModal(); if(e.target.closest && e.target.closest('.hsm-close')){ e.preventDefault(); closeModal(); } if(copyBtn && (e.target===copyBtn || (e.target.closest && e.target.closest('.hsm-copy')))) { e.preventDefault(); copyCurrent(); } }); document.addEventListener('keydown', e=>{ if(e.key==='Escape' && isOpen()) closeModal(); }); function trapFocus(){ if(trapBound) return; trapBound=true; modal.addEventListener('keydown', f=>{ if(f.key==='Tab' && isOpen()){ const focusable=[...modal.querySelectorAll('a[href],button,input,textarea,select,[tabindex]:not([tabindex="-1"])')].filter(el=>!el.hasAttribute('disabled')); if(!focusable.length) return; const first=focusable[0]; const last=focusable[focusable.length-1]; if(f.shiftKey && document.activeElement===first){ f.preventDefault(); last.focus(); } else if(!f.shiftKey && document.activeElement===last){ f.preventDefault(); first.focus(); } } }); } if(closeBtn) closeBtn.addEventListener('click', e=>{ e.preventDefault(); closeModal(); }); })(); </script>This guide covers configuring Geode for high availability (HA), including replication, automatic failover, load balancing, and disaster recovery strategies. <h3 id="ha-architecture-overview" class="position-relative d-flex align-items-center group"> HA Architecture Overview <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="ha-architecture-overview" aria-haspopup="dialog" aria-label="Share link: HA Architecture Overview"> Share link </button> </h3> <h4 id="architecture-patterns" class="position-relative d-flex align-items-center group"> Architecture Patterns <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="architecture-patterns" aria-haspopup="dialog" aria-label="Share link: Architecture Patterns"> Share link </button> </h4>Geode supports multiple HA deployment patterns: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">┌─────────────────────────────────────────────────────────────┐ │ Single Region HA │ │ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Primary │─────▶│ Replica │─────▶│ Replica │ │ │ │ (RW) │ │ (RO) │ │ (RO) │ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────┐ │ │ │ Load Balancer │ │ │ └─────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────┐ │ Multi-Region HA │ │ │ │ Region A Region B Region C │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Primary │◀────────▶│ Replica │◀────────▶│ Replica │ │ │ │ (RW) │ sync │ (RO) │ sync │ (RO) │ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Clients │ │ Clients │ │ Clients │ │ │ └─────────┘ └─────────┘ └─────────┘ │ └─────────────────────────────────────────────────────────────┘ </code></pre></div> <h4 id="ha-modes" class="position-relative d-flex align-items-center group"> HA Modes <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="ha-modes" aria-haspopup="dialog" aria-label="Share link: HA Modes"> Share link </button> </h4><table> <thead> <tr> <th>Mode</th> <th>Description</th> <th>Use Case</th> </tr> </thead> <tbody> <tr> <td>Single Primary</td> <td>One read-write primary, multiple read-only replicas</td> <td>Most production deployments</td> </tr> <tr> <td>Multi-Primary</td> <td>Multiple read-write nodes with conflict resolution</td> <td>Global write availability</td> </tr> <tr> <td>Active-Passive</td> <td>Hot standby for failover</td> <td>Simpler HA requirements</td> </tr> </tbody> </table> <h4 id="consistency-levels" class="position-relative d-flex align-items-center group"> Consistency Levels <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="consistency-levels" aria-haspopup="dialog" aria-label="Share link: Consistency Levels"> Share link </button> </h4><table> <thead> <tr> <th>Level</th> <th>Description</th> <th>Latency</th> <th>Durability</th> </tr> </thead> <tbody> <tr> <td>Strong</td> <td>All replicas acknowledge</td> <td>Higher</td> <td>Highest</td> </tr> <tr> <td>Quorum</td> <td>Majority acknowledges</td> <td>Medium</td> <td>High</td> </tr> <tr> <td>Eventual</td> <td>Primary acknowledges</td> <td>Lowest</td> <td>Medium</td> </tr> </tbody> </table> <h3 id="replication-setup" class="position-relative d-flex align-items-center group"> Replication Setup <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="replication-setup" aria-haspopup="dialog" aria-label="Share link: Replication Setup"> Share link </button> </h3> <h4 id="cluster-configuration" class="position-relative d-flex align-items-center group"> Cluster Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="cluster-configuration" aria-haspopup="dialog" aria-label="Share link: Cluster Configuration"> Share link </button> </h4>Create <code>/etc/geode/cluster.yaml</code>: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">cluster: name: "geode-production" # Node identity node: id: "node-1" # Unique per node address: "192.168.1.10:3141" zone: "us-east-1a" # Cluster membership discovery: method: "static" # static, dns, kubernetes seeds: - "192.168.1.10:3141" - "192.168.1.11:3141" - "192.168.1.12:3141" # Replication settings replication: enabled: true factor: 3 # Number of copies consistency: "quorum" # strong, quorum, eventual # Leader election election: timeout: 10s heartbeat: 1s </code></pre></div> <h4 id="primary-node-configuration" class="position-relative d-flex align-items-center group"> Primary Node Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="primary-node-configuration" aria-haspopup="dialog" aria-label="Share link: Primary Node Configuration"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># /etc/geode/geode.yaml on primary server: listen: "0.0.0.0:3141" role: "primary" cluster: name: "geode-production" node: id: "primary-1" address: "192.168.1.10:3141" replication: mode: "sync" # sync or async # Sync replication settings sync: min_replicas: 2 # Minimum replicas for commit timeout: 5s # Async replication settings async: batch_size: 1000 flush_interval: 100ms max_lag: 10s </code></pre></div> <h4 id="replica-node-configuration" class="position-relative d-flex align-items-center group"> Replica Node Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="replica-node-configuration" aria-haspopup="dialog" aria-label="Share link: Replica Node Configuration"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># /etc/geode/geode.yaml on replica server: listen: "0.0.0.0:3141" role: "replica" cluster: name: "geode-production" node: id: "replica-1" address: "192.168.1.11:3141" replication: primary: address: "192.168.1.10:3141" # Replica behavior read_only: true catch_up: enabled: true batch_size: 10000 </code></pre></div> <h4 id="starting-a-cluster" class="position-relative d-flex align-items-center group"> Starting a Cluster <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="starting-a-cluster" aria-haspopup="dialog" aria-label="Share link: Starting a Cluster"> Share link </button> </h4>Node 1 (Initial Primary): <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">geode serve --config /etc/geode/geode.yaml --cluster-init </code></pre></div>Nodes 2 & 3 (Join as Replicas): <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">geode serve --config /etc/geode/geode.yaml --join 192.168.1.10:3141 </code></pre></div> <h4 id="verifying-cluster-status" class="position-relative d-flex align-items-center group"> Verifying Cluster Status <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="verifying-cluster-status" aria-haspopup="dialog" aria-label="Share link: Verifying Cluster Status"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Check cluster membership geode cluster status # Output: # Cluster: geode-production # State: healthy # # Nodes: # ┌──────────┬─────────────────┬─────────┬────────┬──────────┐ # │ ID │ Address │ Role │ State │ Lag │ # ├──────────┼─────────────────┼─────────┼────────┼──────────┤ # │ primary-1│ 192.168.1.10 │ primary │ online │ - │ # │ replica-1│ 192.168.1.11 │ replica │ online │ 0ms │ # │ replica-2│ 192.168.1.12 │ replica │ online │ 2ms │ # └──────────┴─────────────────┴─────────┴────────┴──────────┘ </code></pre></div> <h4 id="replication-monitoring" class="position-relative d-flex align-items-center group"> Replication Monitoring <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="replication-monitoring" aria-haspopup="dialog" aria-label="Share link: Replication Monitoring"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Check replication lag geode cluster lag # Check replication health geode cluster health --verbose # View replication stream geode cluster stream --follow </code></pre></div> <h3 id="failover-configuration" class="position-relative d-flex align-items-center group"> Failover Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="failover-configuration" aria-haspopup="dialog" aria-label="Share link: Failover Configuration"> Share link </button> </h3> <h4 id="automatic-failover" class="position-relative d-flex align-items-center group"> Automatic Failover <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="automatic-failover" aria-haspopup="dialog" aria-label="Share link: Automatic Failover"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">failover: enabled: true # Detection settings detection: heartbeat_interval: 1s failure_threshold: 3 timeout: 5s # Election settings election: algorithm: "raft" quorum: 2 # (n/2) + 1 # Promotion settings promotion: auto: true priority_zones: - "us-east-1a" - "us-east-1b" - "us-east-1c" # Recovery settings recovery: rejoin_as: "replica" catch_up_timeout: 300s </code></pre></div> <h4 id="failover-priority" class="position-relative d-flex align-items-center group"> Failover Priority <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="failover-priority" aria-haspopup="dialog" aria-label="Share link: Failover Priority"> Share link </button> </h4>Configure node priority for leader election: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">cluster: node: id: "node-1" priority: 100 # Higher = more likely to be primary # Exclude from primary election # priority: 0 </code></pre></div> <h4 id="manual-failover" class="position-relative d-flex align-items-center group"> Manual Failover <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="manual-failover" aria-haspopup="dialog" aria-label="Share link: Manual Failover"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Promote a specific replica to primary geode cluster failover --promote replica-1 # Demote current primary geode cluster demote primary-1 # Force failover (emergency) geode cluster failover --force </code></pre></div> <h4 id="failover-events" class="position-relative d-flex align-items-center group"> Failover Events <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="failover-events" aria-haspopup="dialog" aria-label="Share link: Failover Events"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># View failover history geode cluster events --type failover # Output: # ┌─────────────────────┬──────────┬──────────┬───────────────────────┐ # │ Timestamp │ Old │ New │ Reason │ # ├─────────────────────┼──────────┼──────────┼───────────────────────┤ # │ 2026-01-28 10:23:45 │ primary-1│ replica-1│ node_failure │ # │ 2026-01-28 10:24:30 │ - │ primary-1│ node_recovered │ # │ 2026-01-28 14:00:00 │ primary-1│ replica-2│ manual_failover │ # └─────────────────────┴──────────┴──────────┴───────────────────────┘ </code></pre></div> <h4 id="client-failover-handling" class="position-relative d-flex align-items-center group"> Client Failover Handling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="client-failover-handling" aria-haspopup="dialog" aria-label="Share link: Client Failover Handling"> Share link </button> </h4>Go Client: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go">import ( "database/sql" "time" _ "geodedb.com/geode" ) func main() { // Configure with multiple endpoints db, err := sql.Open("geode", "quic://primary:3141,replica1:3141,replica2:3141") if err != nil { log.Fatal(err) } // Configure connection pool for HA db.SetMaxOpenConns(25) db.SetMaxIdleConns(5) db.SetConnMaxLifetime(5 * time.Minute) db.SetConnMaxIdleTime(1 * time.Minute) } </code></pre></div>Python Client: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">from geode_client import Client, LoadBalancer # Configure with multiple endpoints client = Client( endpoints=[ "primary.geode.local:3141", "replica1.geode.local:3141", "replica2.geode.local:3141", ], load_balancer=LoadBalancer.ROUND_ROBIN, failover=True, retry_attempts=3, retry_delay=1.0, ) async with client.connection() as conn: # Automatically retries on connection failure result = await conn.query("MATCH (n) RETURN count(n)") </code></pre></div>Rust Client: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust">use geode_client::{Client, LoadBalancing, RetryPolicy}; #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let client = Client::builder() .endpoints(vec![ "primary.geode.local:3141", "replica1.geode.local:3141", "replica2.geode.local:3141", ]) .load_balancing(LoadBalancing::RoundRobin) .retry_policy(RetryPolicy::exponential(3, Duration::from_secs(1))) .build()?; let conn = client.connect().await?; // ... Ok(()) } </code></pre></div> <h3 id="load-balancing" class="position-relative d-flex align-items-center group"> Load Balancing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="load-balancing" aria-haspopup="dialog" aria-label="Share link: Load Balancing"> Share link </button> </h3> <h4 id="client-side-load-balancing" class="position-relative d-flex align-items-center group"> Client-Side Load Balancing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="client-side-load-balancing" aria-haspopup="dialog" aria-label="Share link: Client-Side Load Balancing"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Client configuration client: load_balancing: strategy: "round_robin" # round_robin, least_connections, random # Health checking health_check: enabled: true interval: 5s timeout: 2s # Routing preferences routing: # Route reads to replicas read_preference: "replica" # Route writes to primary write_preference: "primary" </code></pre></div> <h4 id="haproxy-configuration" class="position-relative d-flex align-items-center group"> HAProxy Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="haproxy-configuration" aria-haspopup="dialog" aria-label="Share link: HAProxy Configuration"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"># /etc/haproxy/haproxy.cfg global maxconn 10000 log stdout format raw local0 defaults mode tcp timeout connect 5s timeout client 60s timeout server 60s # Health checking option tcp-check frontend geode_frontend bind *:3141 default_backend geode_primary # Route based on connection flags (if supported) # use_backend geode_replicas if { src -f /etc/haproxy/read_clients.txt } backend geode_primary balance first # Primary node server primary 192.168.1.10:3141 check # Fallback to replicas if primary fails server replica1 192.168.1.11:3141 check backup server replica2 192.168.1.12:3141 check backup backend geode_replicas balance roundrobin # All nodes can handle reads server primary 192.168.1.10:3141 check server replica1 192.168.1.11:3141 check server replica2 192.168.1.12:3141 check </code></pre></div> <h4 id="nginx-configuration-udp-load-balancing" class="position-relative d-flex align-items-center group"> NGINX Configuration (UDP Load Balancing) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="nginx-configuration-udp-load-balancing" aria-haspopup="dialog" aria-label="Share link: NGINX Configuration (UDP Load Balancing)"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"># /etc/nginx/nginx.conf stream { upstream geode_cluster { # Health checks zone geode_cluster 64k; # Servers server 192.168.1.10:3141 weight=5; server 192.168.1.11:3141 weight=1; server 192.168.1.12:3141 weight=1; # Load balancing method least_conn; } server { listen 3141 udp; proxy_pass geode_cluster; proxy_timeout 60s; proxy_responses 1; # Enable proxy protocol for client IP preservation # proxy_protocol on; } } </code></pre></div> <h4 id="kubernetes-service-load-balancing" class="position-relative d-flex align-items-center group"> Kubernetes Service Load Balancing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="kubernetes-service-load-balancing" aria-haspopup="dialog" aria-label="Share link: Kubernetes Service Load Balancing"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">apiVersion: v1 kind: Service metadata: name: geode-lb namespace: geode annotations: # AWS NLB service.beta.kubernetes.io/aws-load-balancer-type: nlb service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true" spec: type: LoadBalancer externalTrafficPolicy: Local ports: - name: quic port: 3141 protocol: UDP targetPort: 3141 selector: app: geode </code></pre></div> <h3 id="connection-pooling" class="position-relative d-flex align-items-center group"> Connection Pooling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="connection-pooling" aria-haspopup="dialog" aria-label="Share link: Connection Pooling"> Share link </button> </h3> <h4 id="server-side-connection-pooling" class="position-relative d-flex align-items-center group"> Server-Side Connection Pooling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="server-side-connection-pooling" aria-haspopup="dialog" aria-label="Share link: Server-Side Connection Pooling"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">server: connections: max: 10000 per_client: 100 # Connection lifecycle idle_timeout: 300s max_lifetime: 3600s # Queue settings queue_size: 1000 queue_timeout: 30s </code></pre></div> <h4 id="client-side-connection-pooling" class="position-relative d-flex align-items-center group"> Client-Side Connection Pooling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="client-side-connection-pooling" aria-haspopup="dialog" aria-label="Share link: Client-Side Connection Pooling"> Share link </button> </h4>Go: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go">db, _ := sql.Open("geode", "quic://localhost:3141") // Pool configuration db.SetMaxOpenConns(100) // Max connections db.SetMaxIdleConns(25) // Idle connections db.SetConnMaxLifetime(time.Hour) // Max connection age db.SetConnMaxIdleTime(10 * time.Minute) // Max idle time </code></pre></div>Python: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">from geode_client import Client, ConnectionPool pool = ConnectionPool( endpoints=["localhost:3141"], min_size=5, max_size=50, max_idle_time=300, max_lifetime=3600, ) client = Client(pool=pool) # Get connection from pool async with client.connection() as conn: await conn.query("MATCH (n) RETURN n LIMIT 10") # Connection returned to pool </code></pre></div>Rust: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust">use geode_client::{Client, PoolConfig}; let config = PoolConfig { min_connections: 5, max_connections: 50, connection_timeout: Duration::from_secs(30), idle_timeout: Duration::from_secs(300), max_lifetime: Duration::from_secs(3600), }; let client = Client::with_pool("localhost:3141", config).await?; </code></pre></div> <h4 id="pgbouncer-style-pooling" class="position-relative d-flex align-items-center group"> PgBouncer-Style Pooling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="pgbouncer-style-pooling" aria-haspopup="dialog" aria-label="Share link: PgBouncer-Style Pooling"> Share link </button> </h4>For very high connection counts, use external connection pooling: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># /etc/geode-pooler/config.yaml listen: address: "0.0.0.0:3142" upstream: address: "geode:3141" pool: mode: "transaction" # session, transaction, statement size: 100 reserve: 20 # Per-user limits max_client_connections: 1000 default_pool_size: 20 min_pool_size: 5 # Connection handling server_idle_timeout: 600s server_lifetime: 3600s client_idle_timeout: 0 # No timeout # Query handling query_timeout: 120s query_wait_timeout: 30s </code></pre></div> <h3 id="disaster-recovery" class="position-relative d-flex align-items-center group"> Disaster Recovery <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="disaster-recovery" aria-haspopup="dialog" aria-label="Share link: Disaster Recovery"> Share link </button> </h3> <h4 id="backup-strategy" class="position-relative d-flex align-items-center group"> Backup Strategy <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="backup-strategy" aria-haspopup="dialog" aria-label="Share link: Backup Strategy"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">backup: enabled: true # Full backup schedule full: schedule: "0 2 * * 0" # Weekly Sunday 2 AM retention: 4 # Keep 4 full backups # Incremental backup schedule incremental: schedule: "0 2 * * 1-6" # Daily except Sunday retention: 7 # Keep 7 days # Storage location storage: type: "s3" bucket: "geode-backups" prefix: "production/" region: "us-east-1" # Encryption encryption: enabled: true key_file: "/etc/geode/backup-key" </code></pre></div> <h4 id="point-in-time-recovery" class="position-relative d-flex align-items-center group"> Point-in-Time Recovery <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="point-in-time-recovery" aria-haspopup="dialog" aria-label="Share link: Point-in-Time Recovery"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">wal: enabled: true # WAL archiving archive: enabled: true command: "s3 cp %p s3://geode-backups/wal/%f" timeout: 60s # Retention retention: min_segments: 100 max_size: 10GB </code></pre></div> <h4 id="recovery-procedures" class="position-relative d-flex align-items-center group"> Recovery Procedures <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="recovery-procedures" aria-haspopup="dialog" aria-label="Share link: Recovery Procedures"> Share link </button> </h4>Full Recovery: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Stop Geode sudo systemctl stop geode # Restore from backup geode restore \ --source s3://geode-backups/production/full-2026-01-28.backup \ --target /var/lib/geode # Start Geode sudo systemctl start geode </code></pre></div>Point-in-Time Recovery: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Restore to specific point in time geode restore \ --source s3://geode-backups/production/full-2026-01-28.backup \ --wal-source s3://geode-backups/wal/ \ --target-time "2026-01-28 14:30:00" \ --target /var/lib/geode </code></pre></div> <h4 id="cross-region-replication" class="position-relative d-flex align-items-center group"> Cross-Region Replication <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="cross-region-replication" aria-haspopup="dialog" aria-label="Share link: Cross-Region Replication"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Primary region (us-east-1) replication: cross_region: enabled: true mode: "async" targets: - name: "us-west-2" address: "geode-replica.us-west-2.example.com:3141" priority: 1 - name: "eu-west-1" address: "geode-replica.eu-west-1.example.com:3141" priority: 2 # Async settings batch_size: 5000 flush_interval: 1s max_lag: 60s </code></pre></div> <h4 id="disaster-recovery-runbook" class="position-relative d-flex align-items-center group"> Disaster Recovery Runbook <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="disaster-recovery-runbook" aria-haspopup="dialog" aria-label="Share link: Disaster Recovery Runbook"> Share link </button> </h4><ol> <li> Detection: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Check primary region health geode cluster status --region us-east-1 # Check cross-region replication lag geode cluster lag --cross-region </code></pre></div></li> <li> Assessment: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Determine data loss window geode cluster last-transaction --region us-west-2 </code></pre></div></li> <li> Failover: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Promote DR region to primary geode cluster promote --region us-west-2 # Update DNS aws route53 change-resource-record-sets \ --hosted-zone-id Z123456 \ --change-batch file://dns-failover.json </code></pre></div></li> <li> Verification: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Verify new primary geode cluster status # Test connectivity geode ping geode.example.com:3141 </code></pre></div></li> <li> Recovery: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># When original region recovers, sync data geode cluster sync --from us-west-2 --to us-east-1 # Failback (optional) geode cluster failback --to us-east-1 </code></pre></div></li> </ol> <h3 id="geographic-distribution" class="position-relative d-flex align-items-center group"> Geographic Distribution <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="geographic-distribution" aria-haspopup="dialog" aria-label="Share link: Geographic Distribution"> Share link </button> </h3> <h4 id="multi-region-architecture" class="position-relative d-flex align-items-center group"> Multi-Region Architecture <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="multi-region-architecture" aria-haspopup="dialog" aria-label="Share link: Multi-Region Architecture"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">┌─────────────────────────────────────────────────────────────┐ │ Global Load Balancer │ │ (Route53, CloudFlare, etc.) │ └─────────────────────────────────────────────────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ US-EAST-1 │ │ EU-WEST-1 │ │ AP-SOUTH-1 │ │ │ │ │ │ │ │ ┌───────────┐ │ │ ┌───────────┐ │ │ ┌───────────┐ │ │ │ Primary │◀─┼──┼─▶│ Replica │◀─┼──┼─▶│ Replica │ │ │ └───────────┘ │ │ └───────────┘ │ │ └───────────┘ │ │ │ │ │ │ │ │ │ │ │ ▼ │ │ ▼ │ │ ▼ │ │ ┌───────────┐ │ │ ┌───────────┐ │ │ ┌───────────┐ │ │ │ Replica │ │ │ │ Replica │ │ │ │ Replica │ │ │ └───────────┘ │ │ └───────────┘ │ │ └───────────┘ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ </code></pre></div> <h4 id="region-configuration" class="position-relative d-flex align-items-center group"> Region Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="region-configuration" aria-haspopup="dialog" aria-label="Share link: Region Configuration"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Primary region (us-east-1) cluster: name: "geode-global" region: "us-east-1" node: id: "us-east-1-primary" role: "primary" regions: - name: "us-east-1" is_primary: true nodes: - "192.168.1.10:3141" - "192.168.1.11:3141" - name: "eu-west-1" is_primary: false nodes: - "10.0.1.10:3141" - "10.0.1.11:3141" - name: "ap-south-1" is_primary: false nodes: - "172.16.1.10:3141" - "172.16.1.11:3141" </code></pre></div> <h4 id="read-routing" class="position-relative d-flex align-items-center group"> Read Routing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="read-routing" aria-haspopup="dialog" aria-label="Share link: Read Routing"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">routing: # Route reads to nearest region read: strategy: "nearest" fallback: "primary" # Route writes to primary write: strategy: "primary" # Latency-based routing latency: measurement_interval: 30s threshold_ms: 50 </code></pre></div> <h4 id="conflict-resolution-multi-primary" class="position-relative d-flex align-items-center group"> Conflict Resolution (Multi-Primary) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="conflict-resolution-multi-primary" aria-haspopup="dialog" aria-label="Share link: Conflict Resolution (Multi-Primary)"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">multi_primary: enabled: true conflict_resolution: strategy: "last_write_wins" # last_write_wins, merge, custom # Custom resolution function # custom_handler: "conflict_handler.wasm" # Vector clock for causality causality: enabled: true clock_type: "vector" </code></pre></div> <h3 id="monitoring-ha-clusters" class="position-relative d-flex align-items-center group"> Monitoring HA Clusters <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="monitoring-ha-clusters" aria-haspopup="dialog" aria-label="Share link: Monitoring HA Clusters"> Share link </button> </h3> <h4 id="key-metrics" class="position-relative d-flex align-items-center group"> Key Metrics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="key-metrics" aria-haspopup="dialog" aria-label="Share link: Key Metrics"> Share link </button> </h4><table> <thead> <tr> <th>Metric</th> <th>Description</th> <th>Alert Threshold</th> </tr> </thead> <tbody> <tr> <td><code>geode_cluster_size</code></td> <td>Number of nodes</td> <td>< 3</td> </tr> <tr> <td><code>geode_replication_lag_ms</code></td> <td>Replication lag</td> <td>> 1000ms</td> </tr> <tr> <td><code>geode_leader_changes</code></td> <td>Leader elections</td> <td>> 2/hour</td> </tr> <tr> <td><code>geode_split_brain_detected</code></td> <td>Split brain events</td> <td>> 0</td> </tr> <tr> <td><code>geode_quorum_lost</code></td> <td>Quorum lost events</td> <td>> 0</td> </tr> </tbody> </table> <h4 id="prometheus-metrics" class="position-relative d-flex align-items-center group"> Prometheus Metrics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="prometheus-metrics" aria-haspopup="dialog" aria-label="Share link: Prometheus Metrics"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># prometheus.yml scrape_configs: - job_name: 'geode-cluster' static_configs: - targets: - 'node1:9090' - 'node2:9090' - 'node3:9090' relabel_configs: - source_labels: [__address__] target_label: node </code></pre></div> <h4 id="grafana-dashboard-panels" class="position-relative d-flex align-items-center group"> Grafana Dashboard Panels <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="grafana-dashboard-panels" aria-haspopup="dialog" aria-label="Share link: Grafana Dashboard Panels"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json">{ "panels": [ { "title": "Cluster Health", "type": "stat", "targets": [ { "expr": "sum(geode_cluster_node_healthy)", "legendFormat": "Healthy Nodes" } ] }, { "title": "Replication Lag", "type": "timeseries", "targets": [ { "expr": "geode_replication_lag_ms", "legendFormat": "{{node}}" } ] }, { "title": "Leader Elections", "type": "timeseries", "targets": [ { "expr": "rate(geode_leader_elections_total[5m])", "legendFormat": "Elections/min" } ] } ] } </code></pre></div> <h4 id="alerting-rules" class="position-relative d-flex align-items-center group"> Alerting Rules <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="alerting-rules" aria-haspopup="dialog" aria-label="Share link: Alerting Rules"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Prometheus alerting rules groups: - name: geode-ha rules: - alert: GeodeClusterDegraded expr: sum(geode_cluster_node_healthy) < 3 for: 5m labels: severity: critical annotations: summary: "Geode cluster has fewer than 3 healthy nodes" - alert: GeodeReplicationLagHigh expr: geode_replication_lag_ms > 5000 for: 2m labels: severity: warning annotations: summary: "Replication lag exceeds 5 seconds" - alert: GeodeLeaderFlapping expr: increase(geode_leader_elections_total[10m]) > 5 labels: severity: warning annotations: summary: "Frequent leader elections detected" - alert: GeodeSplitBrain expr: geode_split_brain_detected > 0 labels: severity: critical annotations: summary: "Split brain condition detected" </code></pre></div> <h4 id="health-check-script" class="position-relative d-flex align-items-center group"> Health Check Script <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="health-check-script" aria-haspopup="dialog" aria-label="Share link: Health Check Script"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">#!/bin/bash # /usr/local/bin/check-geode-cluster set -e # Check cluster size CLUSTER_SIZE=$(geode cluster status --format json | jq '.nodes | length') if [ "$CLUSTER_SIZE" -lt 3 ]; then echo "CRITICAL: Cluster size is $CLUSTER_SIZE (expected >= 3)" exit 2 fi # Check replication lag MAX_LAG=$(geode cluster lag --format json | jq '[.nodes[].lag_ms] | max') if [ "$MAX_LAG" -gt 5000 ]; then echo "WARNING: Max replication lag is ${MAX_LAG}ms" exit 1 fi # Check for split brain PRIMARIES=$(geode cluster status --format json | jq '[.nodes[] | select(.role=="primary")] | length') if [ "$PRIMARIES" -gt 1 ]; then echo "CRITICAL: Multiple primaries detected (split brain)" exit 2 fi echo "OK: Cluster healthy with $CLUSTER_SIZE nodes, max lag ${MAX_LAG}ms" exit 0 </code></pre></div> <h3 id="testing-ha" class="position-relative d-flex align-items-center group"> Testing HA <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="testing-ha" aria-haspopup="dialog" aria-label="Share link: Testing HA"> Share link </button> </h3> <h4 id="chaos-engineering" class="position-relative d-flex align-items-center group"> Chaos Engineering <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="chaos-engineering" aria-haspopup="dialog" aria-label="Share link: Chaos Engineering"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Kill primary node geode cluster kill-node primary-1 # Network partition simulation iptables -A INPUT -s 192.168.1.11 -j DROP # Slow network tc qdisc add dev eth0 root netem delay 500ms # Disk I/O pressure stress-ng --io 4 --timeout 60s </code></pre></div> <h4 id="failover-testing" class="position-relative d-flex align-items-center group"> Failover Testing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="failover-testing" aria-haspopup="dialog" aria-label="Share link: Failover Testing"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Automated failover test geode cluster test failover --duration 5m # Output: # Failover Test Results # ===================== # Scenarios tested: 5 # Passed: 5 # Failed: 0 # # Details: # - Primary failure: PASSED (failover in 3.2s) # - Network partition: PASSED (failover in 4.1s) # - Disk full: PASSED (failover in 2.8s) # - Memory pressure: PASSED (no failover needed) # - Graceful shutdown: PASSED (failover in 1.5s) </code></pre></div> <h4 id="load-testing-during-failover" class="position-relative d-flex align-items-center group"> Load Testing During Failover <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="load-testing-during-failover" aria-haspopup="dialog" aria-label="Share link: Load Testing During Failover"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Start load test geode bench --rate 10000 --duration 10m & # Trigger failover mid-test sleep 300 geode cluster failover --promote replica-1 # Observe error rate and latency impact </code></pre></div> <h3 id="best-practices" class="position-relative d-flex align-items-center group"> Best Practices <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="best-practices" aria-haspopup="dialog" aria-label="Share link: Best Practices"> Share link </button> </h3> <h4 id="dos" class="position-relative d-flex align-items-center group"> Do&rsquo;s <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="dos" aria-haspopup="dialog" aria-label="Share link: Dos"> Share link </button> </h4><ol> <li>Use odd number of nodes (3, 5, 7) for quorum</li> <li>Spread across availability zones</li> <li>Monitor replication lag continuously</li> <li>Test failover regularly</li> <li>Automate recovery procedures</li> <li>Keep backups in multiple regions</li> </ol> <h4 id="donts" class="position-relative d-flex align-items-center group"> Don&rsquo;ts <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="donts" aria-haspopup="dialog" aria-label="Share link: Donts"> Share link </button> </h4><ol> <li>Don’t use 2-node clusters (no quorum on failure)</li> <li>Don’t ignore replication lag alerts</li> <li>Don’t skip failover testing</li> <li>Don’t use synchronous replication across regions (too slow)</li> <li>Don’t rely solely on automatic failover (test manual too)</li> </ol> <h3 id="next-steps" class="position-relative d-flex align-items-center group"> Next Steps <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="next-steps" aria-haspopup="dialog" aria-label="Share link: Next Steps"> Share link </button> </h3><ul> <li><a href="/guides/production-deployment/" >Production Deployment</a> - Deploy to production</li> <li><a href="/guides/monitoring/" >Monitoring Guide</a> - Set up monitoring and alerting</li> <li><a href="/guides/backup-restore/" >Backup and Restore</a> - Protect your data</li> <li><a href="/docs/performance/" >Performance Tuning</a> - Optimize cluster performance</li> </ul> <hr> Questions? Contact us at <a href="mailto:[email protected]" >[email protected]</a> or visit our <a href="https://forum.geodedb.com" aria-label="community forum – opens in new window" target="_blank" rel="noopener noreferrer" >community forum ↗ </a> .

Related Guides

Disaster Recovery

Backup and Restore Guide