High Availability

<h2 id="high-availability" class="position-relative d-flex align-items-center group"> High Availability <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="high-availability" aria-haspopup="dialog" aria-label="Share link: High Availability"> Share link </button> </h2><div id="headingShareModal" class="heading-share-modal" role="dialog" aria-modal="true" aria-labelledby="headingShareTitle" hidden> <div class="hsm-dialog" role="document"> <div class="hsm-header"> <h2 id="headingShareTitle" class="h6 mb-0 fw-bold">Share this section</h2> <button type="button" class="hsm-close" aria-label="Close"> </button> </div> <div class="hsm-body"> <label for="headingShareInput" class="form-label small text-muted mb-1 text-uppercase fw-bold" style="font-size: 0.7rem; letter-spacing: 0.5px;">Permalink</label> <div class="input-group mb-4 hsm-url-group"> <input id="headingShareInput" type="text" class="form-control font-monospace" readonly aria-readonly="true" style="font-size: 0.85rem;" /> <button class="btn btn-primary hsm-copy" type="button" aria-label="Copy" title="Copy"> </button> </div> <div class="small fw-bold mb-2 text-muted text-uppercase" style="font-size: 0.7rem; letter-spacing: 0.5px;">Share via</div> <div class="hsm-share-grid"> <a id="share-twitter" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Twitter </a> <a id="share-linkedin" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> LinkedIn </a> <a id="share-facebook" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Facebook </a> </div> </div> </div> </div> <style> .heading-share-modal { position: fixed; inset: 0; display: flex; justify-content: center; align-items: center; background: rgba(0, 0, 0, 0.6); z-index: 1050; padding: 1rem; backdrop-filter: blur(4px); -webkit-backdrop-filter: blur(4px); } .heading-share-modal[hidden] { display: none !important; } .hsm-dialog { max-width: 420px; width: 100%; background: var(--bs-body-bg, #fff); color: var(--bs-body-color, #212529); border: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); border-radius: 1rem; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); overflow: hidden; animation: hsm-fade-in 0.2s ease-out; } @keyframes hsm-fade-in { from { opacity: 0; transform: scale(0.95); } to { opacity: 1; transform: scale(1); } } [data-bs-theme="dark"] .hsm-dialog { background: #1e293b; border-color: rgba(255,255,255,0.1); color: #f8f9fa; } .hsm-header { display: flex; justify-content: space-between; align-items: center; padding: 1rem 1.5rem; border-bottom: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); background: rgba(0,0,0,0.02); } [data-bs-theme="dark"] .hsm-header { background: rgba(255,255,255,0.02); border-color: rgba(255,255,255,0.1); } .hsm-close { background: transparent; border: none; color: inherit; opacity: 0.5; padding: 0.25rem 0.5rem; border-radius: 0.25rem; font-size: 1.2rem; line-height: 1; transition: opacity 0.2s; } .hsm-close:hover { opacity: 1; } .hsm-body { padding: 1.5rem; } .hsm-url-group { display: flex !important; align-items: stretch; } .hsm-url-group .form-control { flex: 1; min-width: 0; margin: 0; background: var(--bs-secondary-bg, #f8f9fa); border-color: var(--bs-border-color, #dee2e6); border-top-right-radius: 0; border-bottom-right-radius: 0; height: 42px; } .hsm-url-group .btn { flex: 0 0 auto; margin: 0; margin-left: -1px; border-top-left-radius: 0; border-bottom-left-radius: 0; height: 42px; display: flex; align-items: center; justify-content: center; padding: 0 1.25rem; z-index: 2; } [data-bs-theme="dark"] .hsm-url-group .form-control { background: #0f172a; border-color: #334155; color: #e2e8f0; } .hsm-share-grid { display: flex; flex-direction: column; gap: 0.5rem; } .hsm-share-grid .btn { display: flex; align-items: center; justify-content: center; font-size: 0.9rem; padding: 0.6rem; border-color: var(--bs-border-color); width: 100%; } [data-bs-theme="dark"] .hsm-share-grid .btn { color: #e2e8f0; border-color: #475569; } [data-bs-theme="dark"] .hsm-share-grid .btn:hover { background: #334155; border-color: #cbd5e1; } </style> <script> (function(){ const modal = document.getElementById('headingShareModal'); if(!modal) return; const input = modal.querySelector('#headingShareInput'); const copyBtn = modal.querySelector('.hsm-copy'); const twitter = modal.querySelector('#share-twitter'); const linkedin = modal.querySelector('#share-linkedin'); const facebook = modal.querySelector('#share-facebook'); const closeBtn = modal.querySelector('.hsm-close'); let lastFocus=null; let trapBound=false; function buildUrl(id){ return window.location.origin + window.location.pathname + '#' + id; } function isOpen(){ return !modal.hasAttribute('hidden'); } function hydrate(id){ const url=buildUrl(id); input.value=url; const enc=encodeURIComponent(url); const text=encodeURIComponent(document.title); if(twitter) twitter.href=`https://twitter.com/intent/tweet?url=${enc}&text=${text}`; if(linkedin) linkedin.href=`https://www.linkedin.com/sharing/share-offsite/?url=${enc}`; if(facebook) facebook.href=`https://www.facebook.com/sharer/sharer.php?u=${enc}`; } function openModal(id){ lastFocus=document.activeElement; hydrate(id); if(!isOpen()){ modal.removeAttribute('hidden'); } requestAnimationFrame(()=>{ input.focus(); }); trapFocus(); } function closeModal(){ if(!isOpen()) return; modal.setAttribute('hidden',''); if(lastFocus && typeof lastFocus.focus==='function') lastFocus.focus(); } function copyCurrent(){ try{ navigator.clipboard.writeText(input.value).then(()=>feedback(true),()=>fallback()); } catch(e){ fallback(); } } function fallback(){ input.select(); try{ document.execCommand('copy'); feedback(true);}catch(e){ feedback(false);} } function feedback(ok){ if(!copyBtn) return; const icon=copyBtn.querySelector('i'); if(!icon) return; const prev=copyBtn.getAttribute('data-prev')||icon.className; if(!copyBtn.getAttribute('data-prev')) copyBtn.setAttribute('data-prev',prev); icon.className= ok ? 'fa-duotone fa-clipboard-check':'fa-duotone fa-circle-exclamation'; setTimeout(()=>{ icon.className=prev; },1800); } function handleShareClick(e){ e.preventDefault(); const btn=e.currentTarget; const id=btn.getAttribute('data-share-target'); if(id) openModal(id); } function bindShareButtons(){ document.querySelectorAll('.h-share').forEach(btn=>{ if(!btn.dataset.hShareBound){ btn.addEventListener('click', handleShareClick); btn.dataset.hShareBound='1'; } }); } bindShareButtons(); if(document.readyState==='loading'){ document.addEventListener('DOMContentLoaded', bindShareButtons); } else { requestAnimationFrame(bindShareButtons); } document.addEventListener('click', function(e){ const shareBtn=e.target.closest && e.target.closest('.h-share'); if(shareBtn && !shareBtn.dataset.hShareBound){ handleShareClick.call(shareBtn, e); } }, true); document.addEventListener('click', e=>{ if(e.target===modal) closeModal(); if(e.target.closest && e.target.closest('.hsm-close')){ e.preventDefault(); closeModal(); } if(copyBtn && (e.target===copyBtn || (e.target.closest && e.target.closest('.hsm-copy')))) { e.preventDefault(); copyCurrent(); } }); document.addEventListener('keydown', e=>{ if(e.key==='Escape' && isOpen()) closeModal(); }); function trapFocus(){ if(trapBound) return; trapBound=true; modal.addEventListener('keydown', f=>{ if(f.key==='Tab' && isOpen()){ const focusable=[...modal.querySelectorAll('a[href],button,input,textarea,select,[tabindex]:not([tabindex="-1"])')].filter(el=>!el.hasAttribute('disabled')); if(!focusable.length) return; const first=focusable[0]; const last=focusable[focusable.length-1]; if(f.shiftKey && document.activeElement===first){ f.preventDefault(); last.focus(); } else if(!f.shiftKey && document.activeElement===last){ f.preventDefault(); first.focus(); } } }); } if(closeBtn) closeBtn.addEventListener('click', e=>{ e.preventDefault(); closeModal(); }); })(); </script>Geode implements comprehensive high availability features to ensure continuous operation even in the face of hardware failures, network partitions, or maintenance activities. The system achieves 99.99% uptime through automatic failover, multi-node replication, and zero-downtime deployments. <h3 id="architecture-overview" class="position-relative d-flex align-items-center group"> Architecture Overview <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="architecture-overview" aria-haspopup="dialog" aria-label="Share link: Architecture Overview"> Share link </button> </h3>Geode’s high availability architecture includes: <ul> <li>Multi-Node Replication: Synchronous and asynchronous replication modes</li> <li>Automatic Failover: Sub-second detection and promotion of standby nodes</li> <li>Quorum-Based Consensus: Raft consensus for cluster coordination</li> <li>Split-Brain Prevention: Network partition detection and resolution</li> <li>Load Balancing: Intelligent query routing across read replicas</li> <li>Zero-Downtime Upgrades: Rolling updates without service interruption</li> </ul> <h3 id="cluster-configuration" class="position-relative d-flex align-items-center group"> Cluster Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="cluster-configuration" aria-haspopup="dialog" aria-label="Share link: Cluster Configuration"> Share link </button> </h3> <h4 id="three-node-cluster-production-minimum" class="position-relative d-flex align-items-center group"> Three-Node Cluster (Production Minimum) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="three-node-cluster-production-minimum" aria-haspopup="dialog" aria-label="Share link: Three-Node Cluster (Production Minimum)"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Primary node node1: role: primary listen: 0.0.0.0:3141 cluster_peers: - node2:3141 - node3:3141 replication_mode: synchronous quorum_size: 2 # Standby nodes node2: role: standby listen: 0.0.0.0:3141 cluster_peers: - node1:3141 - node3:3141 replication_mode: synchronous node3: role: standby listen: 0.0.0.0:3141 cluster_peers: - node1:3141 - node2:3141 replication_mode: synchronous </code></pre></div> <h4 id="five-node-cluster-high-availability" class="position-relative d-flex align-items-center group"> Five-Node Cluster (High Availability) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="five-node-cluster-high-availability" aria-haspopup="dialog" aria-label="Share link: Five-Node Cluster (High Availability)"> Share link </button> </h4>For maximum availability across multiple failure domains: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">cluster: nodes: 5 quorum_size: 3 replication_factor: 3 sync_replicas: 2 async_replicas: 2 availability_zones: - us-east-1a - us-east-1b - us-east-1c </code></pre></div> <h3 id="replication-modes" class="position-relative d-flex align-items-center group"> Replication Modes <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="replication-modes" aria-haspopup="dialog" aria-label="Share link: Replication Modes"> Share link </button> </h3> <h4 id="synchronous-replication" class="position-relative d-flex align-items-center group"> Synchronous Replication <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="synchronous-replication" aria-haspopup="dialog" aria-label="Share link: Synchronous Replication"> Share link </button> </h4>Ensures zero data loss but higher latency: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Enable synchronous replication SET CLUSTER REPLICATION MODE synchronous; SET CLUSTER QUORUM SIZE 2; -- All writes wait for quorum acknowledgment CREATE (n:CriticalData {value: $value}) -- Returns only after 2 nodes confirm </code></pre></div>Use Cases: Financial transactions, critical data, regulatory compliance Latency Impact: +2-5ms per write operation <h4 id="asynchronous-replication" class="position-relative d-flex align-items-center group"> Asynchronous Replication <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="asynchronous-replication" aria-haspopup="dialog" aria-label="Share link: Asynchronous Replication"> Share link </button> </h4>Lower latency with eventual consistency: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">SET CLUSTER REPLICATION MODE asynchronous; -- Writes return immediately after primary acknowledges CREATE (n:LogEntry {timestamp: datetime()}) -- Replicas sync in background </code></pre></div>Use Cases: Log data, analytics, high-throughput workloads Latency Impact: No additional latency <h4 id="semi-synchronous-replication" class="position-relative d-flex align-items-center group"> Semi-Synchronous Replication <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="semi-synchronous-replication" aria-haspopup="dialog" aria-label="Share link: Semi-Synchronous Replication"> Share link </button> </h4>Balance between durability and performance: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">SET CLUSTER REPLICATION MODE semi_synchronous; SET CLUSTER SYNC REPLICAS 1; -- Primary + 1 replica must acknowledge -- Additional replicas async CREATE (n:UserData {id: $id}) </code></pre></div> <h3 id="automatic-failover" class="position-relative d-flex align-items-center group"> Automatic Failover <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="automatic-failover" aria-haspopup="dialog" aria-label="Share link: Automatic Failover"> Share link </button> </h3> <h4 id="detection" class="position-relative d-flex align-items-center group"> Detection <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="detection" aria-haspopup="dialog" aria-label="Share link: Detection"> Share link </button> </h4>Geode detects primary failure through: <ul> <li>Heartbeat Monitoring: 100ms heartbeat interval</li> <li>Health Checks: Active query execution validation</li> <li>Network Partition Detection: Quorum-based split-brain prevention</li> </ul> <h4 id="promotion-process" class="position-relative d-flex align-items-center group"> Promotion Process <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="promotion-process" aria-haspopup="dialog" aria-label="Share link: Promotion Process"> Share link </button> </h4>When primary fails: <ol> <li>Detection (100-500ms): Standby nodes detect missing heartbeats</li> <li>Election (100-300ms): Raft consensus elects new primary</li> <li>Promotion (50-100ms): New primary begins accepting writes</li> <li>Notification (immediate): Clients redirected to new primary</li> </ol> Total Failover Time: < 1 second <h4 id="client-behavior" class="position-relative d-flex align-items-center group"> Client Behavior <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="client-behavior" aria-haspopup="dialog" aria-label="Share link: Client Behavior"> Share link </button> </h4>Clients automatically retry with new primary: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">import asyncio from geode_client import Client, GeodeConnectionError async def query_with_failover(hosts, query): """Try hosts in order until a primary responds.""" for host in hosts: try: client = Client(host=host, port=3141) async with client.connection() as conn: result, _ = await conn.query(query) return result.rows except GeodeConnectionError: await asyncio.sleep(0.2) continue raise GeodeConnectionError("All primaries unavailable") </code></pre></div> <h3 id="load-balancing" class="position-relative d-flex align-items-center group"> Load Balancing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="load-balancing" aria-haspopup="dialog" aria-label="Share link: Load Balancing"> Share link </button> </h3> <h4 id="read-write-splitting" class="position-relative d-flex align-items-center group"> Read-Write Splitting <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="read-write-splitting" aria-haspopup="dialog" aria-label="Share link: Read-Write Splitting"> Share link </button> </h4>Route reads to replicas for horizontal scaling: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">from geode_client import Client primary = Client(host="node1", port=3141) replica = Client(host="node2", port=3141) # Write to primary async with primary.connection() as conn: await conn.execute("CREATE (n:Node {id: $id})", {"id": 1}) # Read from replica async with replica.connection() as conn: result, _ = await conn.query("MATCH (n:Node) RETURN COUNT(n) AS total") count = result.rows[0]["total"] if result.rows else 0 </code></pre></div> <h4 id="load-balancing-strategies" class="position-relative d-flex align-items-center group"> Load Balancing Strategies <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="load-balancing-strategies" aria-haspopup="dialog" aria-label="Share link: Load Balancing Strategies"> Share link </button> </h4>Round Robin: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">SET SESSION READ PREFERENCE round_robin; </code></pre></div>Least Connections: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">SET SESSION READ PREFERENCE least_connections; </code></pre></div>Geographic Proximity: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">SET SESSION READ PREFERENCE nearest; </code></pre></div> <h3 id="zero-downtime-deployments" class="position-relative d-flex align-items-center group"> Zero-Downtime Deployments <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="zero-downtime-deployments" aria-haspopup="dialog" aria-label="Share link: Zero-Downtime Deployments"> Share link </button> </h3> <h4 id="rolling-updates" class="position-relative d-flex align-items-center group"> Rolling Updates <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="rolling-updates" aria-haspopup="dialog" aria-label="Share link: Rolling Updates"> Share link </button> </h4>Update cluster nodes one at a time: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Update node 3 (standby) kubectl set image statefulset/geode geode=geodedb/geode:0.1.4 --index=2 # Wait for health check kubectl wait --for=condition=ready pod/geode-2 # Update node 2 (standby) kubectl set image statefulset/geode geode=geodedb/geode:0.1.4 --index=1 kubectl wait --for=condition=ready pod/geode-1 # Trigger primary failover to updated node geode cluster failover --target=node2 # Update node 1 (now standby) kubectl set image statefulset/geode geode=geodedb/geode:0.1.4 --index=0 kubectl wait --for=condition=ready pod/geode-0 </code></pre></div> <h4 id="schema-migrations" class="position-relative d-flex align-items-center group"> Schema Migrations <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="schema-migrations" aria-haspopup="dialog" aria-label="Share link: Schema Migrations"> Share link </button> </h4>Online schema changes without downtime: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Add property with online migration ALTER GRAPH ADD PROPERTY User.verified BOOLEAN DEFAULT false WITH MIGRATION STRATEGY online; -- Rebuild index without locking CREATE INDEX ON :User(email) WITH MODE online; </code></pre></div> <h3 id="monitoring-high-availability" class="position-relative d-flex align-items-center group"> Monitoring High Availability <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="monitoring-high-availability" aria-haspopup="dialog" aria-label="Share link: Monitoring High Availability"> Share link </button> </h3> <h4 id="health-checks" class="position-relative d-flex align-items-center group"> Health Checks <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="health-checks" aria-haspopup="dialog" aria-label="Share link: Health Checks"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Check cluster status curl https://geode.example.com/api/v1/cluster/status { "cluster_id": "cluster_abc123", "nodes": [ { "id": "node1", "role": "primary", "status": "healthy", "replication_lag": 0 }, { "id": "node2", "role": "standby", "status": "healthy", "replication_lag": 5 }, { "id": "node3", "role": "standby", "status": "healthy", "replication_lag": 3 } ], "quorum_status": "healthy" } </code></pre></div> <h4 id="metrics" class="position-relative d-flex align-items-center group"> Metrics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="metrics" aria-haspopup="dialog" aria-label="Share link: Metrics"> Share link </button> </h4>Monitor critical HA metrics: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"># Replication lag geode_replication_lag_seconds{node="node2"} # Failover events rate(geode_failover_total[1h]) # Node health geode_node_health{node="node1"} # Quorum status geode_quorum_healthy </code></pre></div> <h3 id="disaster-recovery" class="position-relative d-flex align-items-center group"> Disaster Recovery <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="disaster-recovery" aria-haspopup="dialog" aria-label="Share link: Disaster Recovery"> Share link </button> </h3> <h4 id="backup-strategy" class="position-relative d-flex align-items-center group"> Backup Strategy <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="backup-strategy" aria-haspopup="dialog" aria-label="Share link: Backup Strategy"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Continuous archiving to S3 geode backup configure \ --mode continuous \ --destination s3://backups/geode \ --retention 30d # Point-in-time recovery geode restore \ --from s3://backups/geode \ --timestamp "2025-01-24T15:30:00Z" </code></pre></div> <h4 id="geographic-distribution" class="position-relative d-flex align-items-center group"> Geographic Distribution <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="geographic-distribution" aria-haspopup="dialog" aria-label="Share link: Geographic Distribution"> Share link </button> </h4>Deploy across multiple regions: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">regions: us-east-1: nodes: 3 role: primary_cluster us-west-2: nodes: 3 role: disaster_recovery replication_mode: async </code></pre></div> <h3 id="best-practices" class="position-relative d-flex align-items-center group"> Best Practices <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="best-practices" aria-haspopup="dialog" aria-label="Share link: Best Practices"> Share link </button> </h3><ol> <li>Minimum 3 Nodes: Always deploy at least 3 nodes for quorum</li> <li>Odd Number of Nodes: Use 3, 5, or 7 nodes to prevent split votes</li> <li>Geographic Distribution: Spread nodes across availability zones</li> <li>Monitor Replication Lag: Alert on lag > 100ms for sync replicas</li> <li>Test Failover: Regularly test failover procedures</li> <li>Backup Validation: Test restore procedures monthly</li> <li>Capacity Planning: Maintain 30% headroom for failover scenarios</li> </ol> <h3 id="related-topics" class="position-relative d-flex align-items-center group"> Related Topics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="related-topics" aria-haspopup="dialog" aria-label="Share link: Related Topics"> Share link </button> </h3><ul> <li><a href="/tags/clustering/" >Clustering</a> - Cluster setup and management</li> <li><a href="/tags/replication/" >Replication</a> - Data replication strategies</li> <li><a href="/tags/monitoring/" >Monitoring</a> - Observability and metrics</li> </ul> <h3 id="further-reading" class="position-relative d-flex align-items-center group"> Further Reading <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="further-reading" aria-haspopup="dialog" aria-label="Share link: Further Reading"> Share link </button> </h3><ul> <li><a href="/docs/architecture/distributed-architecture/" >Distributed Architecture</a> - Distributed systems design</li> <li><a href="/docs/deployment/deployment-patterns/" >Deployment Patterns</a> - Production deployment strategies</li> </ul> <h3 id="advanced-high-availability-patterns" class="position-relative d-flex align-items-center group"> Advanced High Availability Patterns <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="advanced-high-availability-patterns" aria-haspopup="dialog" aria-label="Share link: Advanced High Availability Patterns"> Share link </button> </h3> <h4 id="multi-region-deployments" class="position-relative d-flex align-items-center group"> Multi-Region Deployments <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="multi-region-deployments" aria-haspopup="dialog" aria-label="Share link: Multi-Region Deployments"> Share link </button> </h4>Deploy across geographic regions for global availability: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">global_deployment: regions: us-east-1: nodes: 3 role: primary write_priority: 1 eu-west-1: nodes: 3 role: secondary write_priority: 2 replication_lag_max: 100ms ap-southeast-1: nodes: 2 role: read_replica replication_mode: async routing: strategy: geographic_proximity failover_policy: automatic health_check_interval: 5s </code></pre></div> <h4 id="active-active-clusters" class="position-relative d-flex align-items-center group"> Active-Active Clusters <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="active-active-clusters" aria-haspopup="dialog" aria-label="Share link: Active-Active Clusters"> Share link </button> </h4>Multi-master configuration for write scalability: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Configure active-active cluster geode cluster configure --topology active-active \ --nodes node1:3141,node2:3141,node3:3141 \ --conflict-resolution last-write-wins \ --quorum-size 2 # Enable cross-region writes geode cluster set-region us-east-1 --writable true geode cluster set-region eu-west-1 --writable true # Monitor cross-region replication lag geode cluster monitor --metric replication_lag </code></pre></div> <h4 id="read-replica-scaling" class="position-relative d-flex align-items-center group"> Read Replica Scaling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="read-replica-scaling" aria-haspopup="dialog" aria-label="Share link: Read Replica Scaling"> Share link </button> </h4>Scale reads horizontally with read replicas: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">from geode_client import Client, ReplicaSelector # Configure connection pool with replicas client = Client( primary="primary.geode.local:3141", replicas=[ "replica1.geode.local:3141", "replica2.geode.local:3141", "replica3.geode.local:3141" ], replica_selector=ReplicaSelector.ROUND_ROBIN, max_replication_lag=100 # ms ) # Writes always go to primary await client.execute(""" CREATE (u:User {id: $id, name: $name}) """, {"id": 123, "name": "Alice"}) # Reads load-balanced across replicas result, _ = await client.query(""" MATCH (u:User) RETURN COUNT(u) """) </code></pre></div> <h3 id="fault-tolerance-strategies" class="position-relative d-flex align-items-center group"> Fault Tolerance Strategies <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="fault-tolerance-strategies" aria-haspopup="dialog" aria-label="Share link: Fault Tolerance Strategies"> Share link </button> </h3> <h4 id="graceful-degradation" class="position-relative d-flex align-items-center group"> Graceful Degradation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="graceful-degradation" aria-haspopup="dialog" aria-label="Share link: Graceful Degradation"> Share link </button> </h4>Continue operating with reduced functionality during failures: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">async def fetch_user_with_degradation(user_id): try: # Try primary data source result, _ = await client.query(""" MATCH (u:User {id: $id}) RETURN u """, {"id": user_id}) return result.rows[0] if result.rows else None except PrimaryUnavailable: try: # Fall back to cache return await cache.get(f"user:{user_id}") except CacheMiss: # Minimal degraded response return {"id": user_id, "name": "User", "degraded": True} </code></pre></div> <h4 id="circuit-breaker-pattern" class="position-relative d-flex align-items-center group"> Circuit Breaker Pattern <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="circuit-breaker-pattern" aria-haspopup="dialog" aria-label="Share link: Circuit Breaker Pattern"> Share link </button> </h4>Protect system from cascading failures: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">from circuitbreaker import CircuitBreaker, CircuitBreakerError @CircuitBreaker( failure_threshold=5, recovery_timeout=30, expected_exception=DatabaseError ) async def query_with_circuit_breaker(query, params): """Execute query with circuit breaker protection""" result, _ = await client.query(query, params) return result # Usage try: result = await query_with_circuit_breaker( "MATCH (u:User) RETURN u", {} ) except CircuitBreakerError: # Circuit open - use cached data or degraded mode result = await fallback_data_source() </code></pre></div> <h4 id="bulkhead-isolation" class="position-relative d-flex align-items-center group"> Bulkhead Isolation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="bulkhead-isolation" aria-haspopup="dialog" aria-label="Share link: Bulkhead Isolation"> Share link </button> </h4>Isolate components to prevent total failure: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">import asyncio from asyncio import Semaphore class BulkheadPool: def __init__(self, max_concurrent=100): self.read_semaphore = Semaphore(max_concurrent // 2) self.write_semaphore = Semaphore(max_concurrent // 2) async def execute_read(self, query): async with self.read_semaphore: result, _ = await client.query(query) return result async def execute_write(self, query): async with self.write_semaphore: await client.execute(query) # Reads and writes isolated - write failures don't block reads pool = BulkheadPool() read_result = await pool.execute_read("MATCH (n) RETURN n LIMIT 10") </code></pre></div> <h3 id="monitoring-and-alerting" class="position-relative d-flex align-items-center group"> Monitoring and Alerting <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="monitoring-and-alerting" aria-haspopup="dialog" aria-label="Share link: Monitoring and Alerting"> Share link </button> </h3> <h4 id="health-check-implementation" class="position-relative d-flex align-items-center group"> Health Check Implementation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="health-check-implementation" aria-haspopup="dialog" aria-label="Share link: Health Check Implementation"> Share link </button> </h4>Comprehensive health monitoring: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">from dataclasses import dataclass from enum import Enum class HealthStatus(Enum): HEALTHY = "healthy" DEGRADED = "degraded" UNHEALTHY = "unhealthy" @dataclass class HealthCheck: status: HealthStatus checks: dict timestamp: datetime async def comprehensive_health_check(client): """Perform comprehensive health check""" checks = {} # Database connectivity try: await client.execute("MATCH (n) RETURN n LIMIT 1") checks['database'] = {'status': 'pass', 'latency_ms': 5} except Exception as e: checks['database'] = {'status': 'fail', 'error': str(e)} # Replication lag try: lag, _ = await client.query(""" SELECT MAX(replication_lag_ms) as max_lag FROM system.replication_status """) lag_ms = lag.rows[0]['max_lag'] if lag.rows else 0 checks['replication'] = { 'status': 'pass' if lag_ms < 100 else 'warn', 'lag_ms': lag_ms } except Exception as e: checks['replication'] = {'status': 'fail', 'error': str(e)} # Disk space disk_usage = await get_disk_usage() checks['disk'] = { 'status': 'pass' if disk_usage < 80 else 'warn', 'usage_percent': disk_usage } # Overall status if any(c['status'] == 'fail' for c in checks.values()): status = HealthStatus.UNHEALTHY elif any(c['status'] == 'warn' for c in checks.values()): status = HealthStatus.DEGRADED else: status = HealthStatus.HEALTHY return HealthCheck(status, checks, datetime.now()) </code></pre></div> <h4 id="prometheus-metrics" class="position-relative d-flex align-items-center group"> Prometheus Metrics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="prometheus-metrics" aria-haspopup="dialog" aria-label="Share link: Prometheus Metrics"> Share link </button> </h4>Export HA metrics for monitoring: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">from prometheus_client import Gauge, Counter, Histogram # Replication metrics replication_lag = Gauge('geode_replication_lag_seconds', 'Replication lag', ['node']) failover_count = Counter('geode_failover_total', 'Total failovers') node_health = Gauge('geode_node_health', 'Node health status', ['node']) # Update metrics async def update_ha_metrics(client): # Replication lag replicas, _ = await client.query(""" SELECT node_id, replication_lag_ms FROM system.replication_status """) for replica in replicas.rows: replication_lag.labels(node=replica['node_id']).set( replica['replication_lag_ms'] / 1000.0 ) # Node health nodes, _ = await client.query("SELECT * FROM system.cluster_nodes") for node in nodes.rows: node_health.labels(node=node['node_id']).set( 1 if node['status'] == 'healthy' else 0 ) </code></pre></div> <h3 id="disaster-recovery-procedures" class="position-relative d-flex align-items-center group"> Disaster Recovery Procedures <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="disaster-recovery-procedures" aria-haspopup="dialog" aria-label="Share link: Disaster Recovery Procedures"> Share link </button> </h3> <h4 id="automated-failover-testing" class="position-relative d-flex align-items-center group"> Automated Failover Testing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="automated-failover-testing" aria-haspopup="dialog" aria-label="Share link: Automated Failover Testing"> Share link </button> </h4>Chaos engineering for HA validation: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">#!/bin/bash # failover-test.sh - Automated failover testing echo "=== Starting Failover Test ===" # 1. Baseline health check echo "Checking baseline health..." geode cluster health # 2. Simulate primary failure echo "Simulating primary node failure..." PRIMARY_NODE=$(geode cluster primary) geode cluster kill-node $PRIMARY_NODE # 3. Wait for failover echo "Waiting for automatic failover..." timeout=30 while [ $timeout -gt 0 ]; do NEW_PRIMARY=$(geode cluster primary) if [ "$NEW_PRIMARY" != "$PRIMARY_NODE" ]; then echo "Failover completed to $NEW_PRIMARY" break fi sleep 1 ((timeout--)) done # 4. Verify cluster health echo "Verifying post-failover health..." geode cluster health # 5. Check data consistency echo "Checking data consistency..." geode cluster verify-data # 6. Restore original primary echo "Restoring original primary..." geode cluster start-node $PRIMARY_NODE geode cluster wait-sync $PRIMARY_NODE echo "=== Failover Test Complete ===" </code></pre></div> <h4 id="backup-verification" class="position-relative d-flex align-items-center group"> Backup Verification <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="backup-verification" aria-haspopup="dialog" aria-label="Share link: Backup Verification"> Share link </button> </h4>Ensure backups are restorable: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">async def verify_backup(backup_path): """Verify backup integrity and restorability""" # 1. Check backup files exist backup_files = list_backup_files(backup_path) assert len(backup_files) > 0, "No backup files found" # 2. Verify checksums for file in backup_files: expected_checksum = read_checksum_file(f"{file}.sha256") actual_checksum = compute_sha256(file) assert expected_checksum == actual_checksum, f"Checksum mismatch: {file}" # 3. Test restore to temporary location temp_db = create_temp_database() await restore_backup(backup_path, temp_db) # 4. Verify data integrity client = Client(temp_db) async with client.connection() as client: # Check node count result, _ = await client.query("MATCH (n) RETURN COUNT(n) as count") assert result.rows[0]['count'] > 0, "No nodes in restored database" # Check sample data user, _ = await client.query("MATCH (u:User) RETURN u LIMIT 1") assert len(user) > 0, "No users in restored database" # 5. Cleanup cleanup_temp_database(temp_db) print(f"Backup verification successful: {backup_path}") </code></pre></div> <h3 id="performance-under-high-availability" class="position-relative d-flex align-items-center group"> Performance Under High Availability <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="performance-under-high-availability" aria-haspopup="dialog" aria-label="Share link: Performance Under High Availability"> Share link </button> </h3> <h4 id="load-balancing-algorithms" class="position-relative d-flex align-items-center group"> Load Balancing Algorithms <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="load-balancing-algorithms" aria-haspopup="dialog" aria-label="Share link: Load Balancing Algorithms"> Share link </button> </h4>Intelligent client-side load balancing: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">class SmartLoadBalancer: def __init__(self, replicas): self.replicas = replicas self.stats = {r: {'latency': 0, 'errors': 0} for r in replicas} def select_replica(self): """Select replica based on latency and error rate""" scores = {} for replica, stats in self.stats.items(): # Lower latency and fewer errors = higher score latency_score = 1.0 / (stats['latency'] + 1) error_penalty = 0.5 ** stats['errors'] scores[replica] = latency_score * error_penalty return max(scores, key=scores.get) async def execute_on_best_replica(self, query): replica = self.select_replica() start = time.time() try: result = await self.execute_on(replica, query) latency = time.time() - start # Update stats with exponential moving average self.stats[replica]['latency'] = ( 0.7 * self.stats[replica]['latency'] + 0.3 * latency ) self.stats[replica]['errors'] = max(0, self.stats[replica]['errors'] - 1) return result except Exception as e: self.stats[replica]['errors'] += 1 raise </code></pre></div> <h4 id="connection-pooling-for-ha" class="position-relative d-flex align-items-center group"> Connection Pooling for HA <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="connection-pooling-for-ha" aria-haspopup="dialog" aria-label="Share link: Connection Pooling for HA"> Share link </button> </h4>Maintain healthy connection pools: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">from geode_client import ConnectionPool pool = ConnectionPool( nodes=[ "node1:3141", "node2:3141", "node3:3141" ], min_connections=5, max_connections=50, connection_timeout=5.0, idle_timeout=300.0, health_check_interval=30.0, max_retries=3, retry_backoff=lambda attempt: 2 ** attempt ) # Automatic connection management async with pool.acquire() as conn: result, _ = await conn.query("MATCH (u:User) RETURN u") </code></pre></div> <h3 id="best-practices-summary" class="position-relative d-flex align-items-center group"> Best Practices Summary <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="best-practices-summary" aria-haspopup="dialog" aria-label="Share link: Best Practices Summary"> Share link </button> </h3><ol> <li>Deploy Minimum 3 Nodes: Enables quorum-based decisions</li> <li>Use Odd Node Counts: Prevents split-brain scenarios (3, 5, 7)</li> <li>Geographic Distribution: Spread across availability zones</li> <li>Monitor Replication Lag: Alert on lag > 100ms for sync replicas</li> <li>Test Failover Regularly: Monthly chaos engineering exercises</li> <li>Validate Backups: Weekly restore tests</li> <li>Capacity Planning: Maintain 30% headroom for failover</li> <li>Document Procedures: Runbooks for all failure scenarios</li> <li>Automate Everything: No manual steps in critical path</li> <li>Measure and Improve: Track MTTR and MTBF metrics</li> </ol> Browse the tagged content below to discover comprehensive high availability documentation, deployment patterns, and production best practices for Geode.

Popular

Related Articles