Tag: Scaling

Documentation tagged with scaling in the Geode graph database. This comprehensive collection covers scaling strategies, capacity planning, replication patterns, sharding techniques, and best practices for building highly scalable graph database deployments that handle enterprise workloads. <h3 id="overview" class="position-relative d-flex align-items-center group"> Overview <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="overview" aria-haspopup="dialog" aria-label="Share link: Overview"> Share link </button> </h3><div id="headingShareModal" class="heading-share-modal" role="dialog" aria-modal="true" aria-labelledby="headingShareTitle" hidden> <div class="hsm-dialog" role="document"> <div class="hsm-header"> <h2 id="headingShareTitle" class="h6 mb-0 fw-bold">Share this section</h2> <button type="button" class="hsm-close" aria-label="Close"> </button> </div> <div class="hsm-body"> <label for="headingShareInput" class="form-label small text-muted mb-1 text-uppercase fw-bold" style="font-size: 0.7rem; letter-spacing: 0.5px;">Permalink</label> <div class="input-group mb-4 hsm-url-group"> <input id="headingShareInput" type="text" class="form-control font-monospace" readonly aria-readonly="true" style="font-size: 0.85rem;" /> <button class="btn btn-primary hsm-copy" type="button" aria-label="Copy" title="Copy"> </button> </div> <div class="small fw-bold mb-2 text-muted text-uppercase" style="font-size: 0.7rem; letter-spacing: 0.5px;">Share via</div> <div class="hsm-share-grid"> <a id="share-twitter" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Twitter </a> <a id="share-linkedin" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> LinkedIn </a> <a id="share-facebook" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Facebook </a> </div> </div> </div> </div> <style> .heading-share-modal { position: fixed; inset: 0; display: flex; justify-content: center; align-items: center; background: rgba(0, 0, 0, 0.6); z-index: 1050; padding: 1rem; backdrop-filter: blur(4px); -webkit-backdrop-filter: blur(4px); } .heading-share-modal[hidden] { display: none !important; } .hsm-dialog { max-width: 420px; width: 100%; background: var(--bs-body-bg, #fff); color: var(--bs-body-color, #212529); border: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); border-radius: 1rem; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); overflow: hidden; animation: hsm-fade-in 0.2s ease-out; } @keyframes hsm-fade-in { from { opacity: 0; transform: scale(0.95); } to { opacity: 1; transform: scale(1); } } [data-bs-theme="dark"] .hsm-dialog { background: #1e293b; border-color: rgba(255,255,255,0.1); color: #f8f9fa; } .hsm-header { display: flex; justify-content: space-between; align-items: center; padding: 1rem 1.5rem; border-bottom: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); background: rgba(0,0,0,0.02); } [data-bs-theme="dark"] .hsm-header { background: rgba(255,255,255,0.02); border-color: rgba(255,255,255,0.1); } .hsm-close { background: transparent; border: none; color: inherit; opacity: 0.5; padding: 0.25rem 0.5rem; border-radius: 0.25rem; font-size: 1.2rem; line-height: 1; transition: opacity 0.2s; } .hsm-close:hover { opacity: 1; } .hsm-body { padding: 1.5rem; } .hsm-url-group { display: flex !important; align-items: stretch; } .hsm-url-group .form-control { flex: 1; min-width: 0; margin: 0; background: var(--bs-secondary-bg, #f8f9fa); border-color: var(--bs-border-color, #dee2e6); border-top-right-radius: 0; border-bottom-right-radius: 0; height: 42px; } .hsm-url-group .btn { flex: 0 0 auto; margin: 0; margin-left: -1px; border-top-left-radius: 0; border-bottom-left-radius: 0; height: 42px; display: flex; align-items: center; justify-content: center; padding: 0 1.25rem; z-index: 2; } [data-bs-theme="dark"] .hsm-url-group .form-control { background: #0f172a; border-color: #334155; color: #e2e8f0; } .hsm-share-grid { display: flex; flex-direction: column; gap: 0.5rem; } .hsm-share-grid .btn { display: flex; align-items: center; justify-content: center; font-size: 0.9rem; padding: 0.6rem; border-color: var(--bs-border-color); width: 100%; } [data-bs-theme="dark"] .hsm-share-grid .btn { color: #e2e8f0; border-color: #475569; } [data-bs-theme="dark"] .hsm-share-grid .btn:hover { background: #334155; border-color: #cbd5e1; } </style> <script> (function(){ const modal = document.getElementById('headingShareModal'); if(!modal) return; const input = modal.querySelector('#headingShareInput'); const copyBtn = modal.querySelector('.hsm-copy'); const twitter = modal.querySelector('#share-twitter'); const linkedin = modal.querySelector('#share-linkedin'); const facebook = modal.querySelector('#share-facebook'); const closeBtn = modal.querySelector('.hsm-close'); let lastFocus=null; let trapBound=false; function buildUrl(id){ return window.location.origin + window.location.pathname + '#' + id; } function isOpen(){ return !modal.hasAttribute('hidden'); } function hydrate(id){ const url=buildUrl(id); input.value=url; const enc=encodeURIComponent(url); const text=encodeURIComponent(document.title); if(twitter) twitter.href=`https://twitter.com/intent/tweet?url=${enc}&text=${text}`; if(linkedin) linkedin.href=`https://www.linkedin.com/sharing/share-offsite/?url=${enc}`; if(facebook) facebook.href=`https://www.facebook.com/sharer/sharer.php?u=${enc}`; } function openModal(id){ lastFocus=document.activeElement; hydrate(id); if(!isOpen()){ modal.removeAttribute('hidden'); } requestAnimationFrame(()=>{ input.focus(); }); trapFocus(); } function closeModal(){ if(!isOpen()) return; modal.setAttribute('hidden',''); if(lastFocus && typeof lastFocus.focus==='function') lastFocus.focus(); } function copyCurrent(){ try{ navigator.clipboard.writeText(input.value).then(()=>feedback(true),()=>fallback()); } catch(e){ fallback(); } } function fallback(){ input.select(); try{ document.execCommand('copy'); feedback(true);}catch(e){ feedback(false);} } function feedback(ok){ if(!copyBtn) return; const icon=copyBtn.querySelector('i'); if(!icon) return; const prev=copyBtn.getAttribute('data-prev')||icon.className; if(!copyBtn.getAttribute('data-prev')) copyBtn.setAttribute('data-prev',prev); icon.className= ok ? 'fa-duotone fa-clipboard-check':'fa-duotone fa-circle-exclamation'; setTimeout(()=>{ icon.className=prev; },1800); } function handleShareClick(e){ e.preventDefault(); const btn=e.currentTarget; const id=btn.getAttribute('data-share-target'); if(id) openModal(id); } function bindShareButtons(){ document.querySelectorAll('.h-share').forEach(btn=>{ if(!btn.dataset.hShareBound){ btn.addEventListener('click', handleShareClick); btn.dataset.hShareBound='1'; } }); } bindShareButtons(); if(document.readyState==='loading'){ document.addEventListener('DOMContentLoaded', bindShareButtons); } else { requestAnimationFrame(bindShareButtons); } document.addEventListener('click', function(e){ const shareBtn=e.target.closest && e.target.closest('.h-share'); if(shareBtn && !shareBtn.dataset.hShareBound){ handleShareClick.call(shareBtn, e); } }, true); document.addEventListener('click', e=>{ if(e.target===modal) closeModal(); if(e.target.closest && e.target.closest('.hsm-close')){ e.preventDefault(); closeModal(); } if(copyBtn && (e.target===copyBtn || (e.target.closest && e.target.closest('.hsm-copy')))) { e.preventDefault(); copyCurrent(); } }); document.addEventListener('keydown', e=>{ if(e.key==='Escape' && isOpen()) closeModal(); }); function trapFocus(){ if(trapBound) return; trapBound=true; modal.addEventListener('keydown', f=>{ if(f.key==='Tab' && isOpen()){ const focusable=[...modal.querySelectorAll('a[href],button,input,textarea,select,[tabindex]:not([tabindex="-1"])')].filter(el=>!el.hasAttribute('disabled')); if(!focusable.length) return; const first=focusable[0]; const last=focusable[focusable.length-1]; if(f.shiftKey && document.activeElement===first){ f.preventDefault(); last.focus(); } else if(!f.shiftKey && document.activeElement===last){ f.preventDefault(); first.focus(); } } }); } if(closeBtn) closeBtn.addEventListener('click', e=>{ e.preventDefault(); closeModal(); }); })(); </script>Scaling is essential for growing graph applications that need to handle increasing data volumes, higher query throughput, or greater concurrent user loads. Geode provides multiple scaling strategies to meet different performance and availability requirements, from simple vertical scaling to complex distributed architectures. Scaling dimensions: <ul> <li>Data Volume: Growing graph size (nodes and edges)</li> <li>Query Throughput: Increasing queries per second</li> <li>Concurrent Users: More simultaneous connections</li> <li>Geographic Distribution: Multi-region deployments</li> <li>High Availability: Redundancy and fault tolerance</li> </ul> <h3 id="scaling-strategies" class="position-relative d-flex align-items-center group"> Scaling Strategies <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="scaling-strategies" aria-haspopup="dialog" aria-label="Share link: Scaling Strategies"> Share link </button> </h3> <h4 id="vertical-scaling" class="position-relative d-flex align-items-center group"> Vertical Scaling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="vertical-scaling" aria-haspopup="dialog" aria-label="Share link: Vertical Scaling"> Share link </button> </h4>Scale up by adding resources to a single server: When to Use: <ul> <li>Data fits on a single machine (size depends on hardware and indexing)</li> <li>Simple operational requirements</li> <li>Cost-effective for small to medium workloads</li> <li>No application changes needed</li> </ul> Implementation: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Before: Small instance # - 4 vCPUs # - 16 GB RAM # - 500 GB SSD # Performance: workload dependent (benchmark-specific) # After: Large instance # - 32 vCPUs # - 256 GB RAM # - 2 TB NVMe SSD # Performance: workload dependent (benchmark-specific) </code></pre></div>Configuration for Vertical Scaling: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># geode.yaml - Optimized for large single server server: worker_threads: 32 # Match vCPU count max_connections: 2000 performance: index_cache_size: 128GB # 50% of RAM query_cache_size: 50000 worker_pool_size: 64 storage: buffer_pool_size: 64GB checkpoint_interval: 15m </code></pre></div>Benefits: <ul> <li>Simple architecture</li> <li>No data partitioning complexity</li> <li>ACID guarantees preserved</li> <li>Lower operational overhead</li> </ul> Limitations: <ul> <li>Single point of failure</li> <li>Limited by hardware maximum</li> <li>Expensive at large scale</li> <li>Downtime for upgrades</li> </ul> <h4 id="horizontal-scaling-read-replicas" class="position-relative d-flex align-items-center group"> Horizontal Scaling (Read Replicas) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="horizontal-scaling-read-replicas" aria-haspopup="dialog" aria-label="Share link: Horizontal Scaling (Read Replicas)"> Share link </button> </h4>Add read-only replicas to distribute query load: Architecture: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">┌─────────────┐ │ Primary │ ← Writes │ (Master) │ └──────┬──────┘ │ │ Replication │ ┌───┴────┬────────┬────────┐ │ │ │ │ ┌──▼──┐ ┌──▼──┐ ┌──▼──┐ ┌──▼──┐ │Rep 1│ │Rep 2│ │Rep 3│ │Rep 4│ ← Reads └─────┘ └─────┘ └─────┘ └─────┘ </code></pre></div>Setup: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Start primary node geode serve \ --listen 0.0.0.0:3141 \ --data-dir /data/primary \ --role primary # Start read replicas geode serve \ --listen 0.0.0.0:3141 \ --data-dir /data/replica1 \ --role replica \ --primary-host primary.example.com:3141 geode serve \ --listen 0.0.0.0:3141 \ --data-dir /data/replica2 \ --role replica \ --primary-host primary.example.com:3141 </code></pre></div>Application Configuration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go">// Go client with read replicas import "geodedb.com/geode" // Configure connection pool with read replicas config := &geode.Config{ Primary: "primary.example.com:3141", Replicas: []string{ "replica1.example.com:3141", "replica2.example.com:3141", "replica3.example.com:3141", }, ReadPreference: geode.ReadPreferenceReplica, } db, err := geode.Connect(config) // Writes go to primary db.Exec("CREATE (u:User {name: 'Alice'})") // Reads load-balanced across replicas db.Query("MATCH (u:User) RETURN u") </code></pre></div>Benefits: <ul> <li>Increase read throughput linearly</li> <li>High availability (replicas can failover)</li> <li>Geographic distribution</li> <li>Read-heavy workloads scale well</li> </ul> Limitations: <ul> <li>Write throughput limited by primary</li> <li>Replication lag (eventual consistency)</li> <li>Storage duplication</li> <li>Complexity increases with replica count</li> </ul> <h4 id="horizontal-scaling-sharding" class="position-relative d-flex align-items-center group"> Horizontal Scaling (Sharding) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="horizontal-scaling-sharding" aria-haspopup="dialog" aria-label="Share link: Horizontal Scaling (Sharding)"> Share link </button> </h4>Partition data across multiple nodes: Sharding Strategies: 1. Label-Based Sharding: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Partition by node type sharding: strategy: label shards: - name: shard1 labels: [User, Profile] servers: [shard1.example.com:3141] - name: shard2 labels: [Product, Category] servers: [shard2.example.com:3141] - name: shard3 labels: [Order, Payment] servers: [shard3.example.com:3141] </code></pre></div>2. Hash-Based Sharding: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Partition by ID hash sharding: strategy: hash shards: 8 key_property: id servers: - shard1.example.com:3141 - shard2.example.com:3141 - shard3.example.com:3141 - shard4.example.com:3141 - shard5.example.com:3141 - shard6.example.com:3141 - shard7.example.com:3141 - shard8.example.com:3141 </code></pre></div>3. Range-Based Sharding: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Partition by property ranges sharding: strategy: range key_property: created_at shards: - name: historical range: [null, 2023-01-01] servers: [historical.example.com:3141] - name: recent range: [2023-01-01, 2025-01-01] servers: [recent.example.com:3141] - name: current range: [2025-01-01, null] servers: [current.example.com:3141] </code></pre></div>Cross-Shard Queries: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">// Query spanning multiple shards // Coordinator routes to all shards and merges results MATCH (u:User)-[:PURCHASED]->(p:Product) WHERE u.region = 'West' RETURN u.name, p.name ORDER BY u.created_at DESC LIMIT 100; // Execution: // 1. Parse query at coordinator // 2. Send sub-queries to relevant shards // 3. Merge results at coordinator // 4. Apply ORDER BY and LIMIT // 5. Return to client </code></pre></div>Benefits: <ul> <li>Scale beyond single machine limits</li> <li>Distribute both reads and writes</li> <li>Handle massive datasets (billions of nodes)</li> <li>Geographic data locality</li> </ul> Limitations: <ul> <li>Complex operationally</li> <li>Cross-shard queries slower</li> <li>Rebalancing data is expensive</li> <li>Transaction guarantees limited across shards</li> </ul> <h3 id="replication-patterns" class="position-relative d-flex align-items-center group"> Replication Patterns <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="replication-patterns" aria-haspopup="dialog" aria-label="Share link: Replication Patterns"> Share link </button> </h3> <h4 id="synchronous-replication" class="position-relative d-flex align-items-center group"> Synchronous Replication <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="synchronous-replication" aria-haspopup="dialog" aria-label="Share link: Synchronous Replication"> Share link </button> </h4>Writes replicated before acknowledging to client: Configuration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">replication: mode: synchronous min_replicas: 2 # Wait for 2 replicas timeout: 5s # Max wait time # Guarantees: # - No data loss on primary failure # - Consistent reads from any replica # - Higher write latency (+5-20ms) </code></pre></div>Use Cases: <ul> <li>Financial transactions</li> <li>Critical business data</li> <li>Compliance requirements</li> <li>Zero data loss tolerance</li> </ul> <h4 id="asynchronous-replication" class="position-relative d-flex align-items-center group"> Asynchronous Replication <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="asynchronous-replication" aria-haspopup="dialog" aria-label="Share link: Asynchronous Replication"> Share link </button> </h4>Writes acknowledged immediately, replicated in background: Configuration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">replication: mode: asynchronous replication_lag_target: 100ms catch_up_priority: high # Characteristics: # - Low write latency # - Possible replication lag # - Rare data loss on failure # - Higher throughput </code></pre></div>Use Cases: <ul> <li>Read-heavy workloads</li> <li>Social networks</li> <li>Content management</li> <li>Analytics</li> </ul> <h4 id="multi-master-replication" class="position-relative d-flex align-items-center group"> Multi-Master Replication <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="multi-master-replication" aria-haspopup="dialog" aria-label="Share link: Multi-Master Replication"> Share link </button> </h4>Multiple writable nodes with conflict resolution: Configuration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">replication: mode: multi-master conflict_resolution: last-write-wins topology: mesh # Features: # - Write to any node # - Automatic conflict resolution # - Geographic distribution # - Complex edge cases </code></pre></div> <h3 id="load-balancing" class="position-relative d-flex align-items-center group"> Load Balancing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="load-balancing" aria-haspopup="dialog" aria-label="Share link: Load Balancing"> Share link </button> </h3> <h4 id="query-router" class="position-relative d-flex align-items-center group"> Query Router <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="query-router" aria-haspopup="dialog" aria-label="Share link: Query Router"> Share link </button> </h4>Distribute queries across servers: HAProxy Configuration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"># /etc/haproxy/haproxy.cfg frontend geode_frontend bind *:3141 mode tcp default_backend geode_replicas backend geode_replicas mode tcp balance leastconn option tcp-check # Health check tcp-check connect tcp-check send-binary 48454c4c4f0a # HELLO tcp-check expect binary 4f4b0a # OK # Server pool server replica1 replica1.example.com:3141 check server replica2 replica2.example.com:3141 check server replica3 replica3.example.com:3141 check server replica4 replica4.example.com:3141 check </code></pre></div>NGINX Stream Configuration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"># /etc/nginx/nginx.conf stream { upstream geode_cluster { least_conn; server replica1.example.com:3141 max_fails=3 fail_timeout=30s; server replica2.example.com:3141 max_fails=3 fail_timeout=30s; server replica3.example.com:3141 max_fails=3 fail_timeout=30s; } server { listen 3141; proxy_pass geode_cluster; proxy_connect_timeout 5s; } } </code></pre></div> <h4 id="client-side-load-balancing" class="position-relative d-flex align-items-center group"> Client-Side Load Balancing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="client-side-load-balancing" aria-haspopup="dialog" aria-label="Share link: Client-Side Load Balancing"> Share link </button> </h4>Application manages distribution: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go">// Go client with built-in load balancing config := &geode.Config{ Servers: []string{ "replica1.example.com:3141", "replica2.example.com:3141", "replica3.example.com:3141", }, LoadBalancer: geode.RoundRobin, HealthCheckInterval: 30 * time.Second, } client, err := geode.Connect(config) // Automatically distributes queries </code></pre></div> <h3 id="capacity-planning" class="position-relative d-flex align-items-center group"> Capacity Planning <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="capacity-planning" aria-haspopup="dialog" aria-label="Share link: Capacity Planning"> Share link </button> </h3> <h4 id="growth-modeling" class="position-relative d-flex align-items-center group"> Growth Modeling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="growth-modeling" aria-haspopup="dialog" aria-label="Share link: Growth Modeling"> Share link </button> </h4>Project resource needs (numbers below are illustrative; calibrate with your benchmarks): <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"># capacity_planner.py import math def estimate_resources( nodes_count, edges_count, avg_node_properties=10, avg_edge_properties=3, qps_target=10000, replication_factor=3 ): # Storage calculation node_storage_mb = (nodes_count * avg_node_properties * 50) / 1_000_000 edge_storage_mb = (edges_count * avg_edge_properties * 40) / 1_000_000 index_storage_mb = (node_storage_mb + edge_storage_mb) * 0.3 total_storage_gb = ( (node_storage_mb + edge_storage_mb + index_storage_mb) * replication_factor / 1000 ) # Memory calculation (working set = 20% of data + index cache) working_set_gb = total_storage_gb * 0.2 index_cache_gb = index_storage_mb / 1000 required_memory_gb = working_set_gb + index_cache_gb + 16 # OS overhead # CPU calculation (example heuristic: 1 core per 500 QPS) required_cores = math.ceil(qps_target / 500) # Server count (assuming 32-core, 256GB servers) servers_needed = max( math.ceil(total_storage_gb / (2000 / replication_factor)), # Storage math.ceil(required_memory_gb / 200), # Memory math.ceil(required_cores / 28) # CPU (leave headroom) ) return { "total_storage_gb": total_storage_gb, "required_memory_gb": required_memory_gb, "required_cores": required_cores, "servers_needed": servers_needed, } # Example: 100M nodes, 500M edges, 10k QPS resources = estimate_resources( nodes_count=100_000_000, edges_count=500_000_000, qps_target=10_000 ) print(f"Storage: {resources['total_storage_gb']:.0f} GB") print(f"Memory: {resources['required_memory_gb']:.0f} GB") print(f"Cores: {resources['required_cores']}") print(f"Servers: {resources['servers_needed']}") </code></pre></div> <h4 id="monitoring-for-scale" class="position-relative d-flex align-items-center group"> Monitoring for Scale <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="monitoring-for-scale" aria-haspopup="dialog" aria-label="Share link: Monitoring for Scale"> Share link </button> </h4>Track scaling indicators: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># Prometheus alerts for scaling triggers groups: - name: scaling_alerts rules: # CPU saturation - alert: HighCPUUsage expr: avg(cpu_usage_percent) > 70 for: 15m annotations: summary: "Consider adding CPU capacity" # Memory pressure - alert: HighMemoryUsage expr: memory_usage_percent > 80 for: 10m annotations: summary: "Add more RAM or replicas" # Storage growth - alert: StorageGrowth expr: rate(storage_bytes_used[7d]) > 0 annotations: summary: "Storage growing, plan expansion" # Query latency - alert: HighQueryLatency expr: p99_query_duration_ms > 500 for: 15m annotations: summary: "Query performance degrading" # Connection saturation - alert: ConnectionPoolExhaustion expr: (active_connections / max_connections) > 0.8 for: 5m annotations: summary: "Near connection limit" </code></pre></div> <h3 id="high-availability" class="position-relative d-flex align-items-center group"> High Availability <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="high-availability" aria-haspopup="dialog" aria-label="Share link: High Availability"> Share link </button> </h3> <h4 id="failover-configuration" class="position-relative d-flex align-items-center group"> Failover Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="failover-configuration" aria-haspopup="dialog" aria-label="Share link: Failover Configuration"> Share link </button> </h4>Automatic primary failover: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># geode-ha.yaml high_availability: enabled: true election_timeout: 10s heartbeat_interval: 2s min_quorum: 2 cluster: nodes: - host: node1.example.com port: 3141 priority: 100 # Preferred primary - host: node2.example.com port: 3141 priority: 90 - host: node3.example.com port: 3141 priority: 80 # Automatic failover on primary failure # - Quorum elects new primary (highest priority) # - Clients redirect to new primary # - Old primary rejoins as replica </code></pre></div> <h4 id="health-checks" class="position-relative d-flex align-items-center group"> Health Checks <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="health-checks" aria-haspopup="dialog" aria-label="Share link: Health Checks"> Share link </button> </h4>Monitor node health: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Kubernetes readiness probe livenessProbe: exec: command: - geode - health - --check - readiness initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 readinessProbe: exec: command: - geode - health - --check - liveness initialDelaySeconds: 10 periodSeconds: 5 timeoutSeconds: 3 failureThreshold: 2 </code></pre></div> <h3 id="best-practices" class="position-relative d-flex align-items-center group"> Best Practices <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="best-practices" aria-haspopup="dialog" aria-label="Share link: Best Practices"> Share link </button> </h3> <h4 id="start-simple-scale-gradually" class="position-relative d-flex align-items-center group"> Start Simple, Scale Gradually <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="start-simple-scale-gradually" aria-haspopup="dialog" aria-label="Share link: Start Simple, Scale Gradually"> Share link </button> </h4>Phase 1: Single Server (initial deployments): <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">deployment: single-server resources: cpu: 8 cores memory: 64 GB storage: 500 GB SSD performance: workload-dependent (benchmark-specific) </code></pre></div>Phase 2: Vertical Scaling (single-node scale-up): <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">deployment: single-server resources: cpu: 32 cores memory: 256 GB storage: 2 TB NVMe performance: workload-dependent (benchmark-specific) </code></pre></div>Phase 3: Read Replicas (read-heavy workloads): <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">deployment: primary + replicas topology: primary: 1 replicas: 3-5 resources_per_server: cpu: 16 cores memory: 128 GB storage: 2 TB NVMe performance: workload-dependent (benchmark-specific) </code></pre></div>Phase 4: Sharding (distributed datasets): <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">deployment: sharded-cluster topology: shards: 4-8 replicas_per_shard: 3 resources_per_server: cpu: 32 cores memory: 256 GB storage: 4 TB NVMe performance: workload-dependent (benchmark-specific) </code></pre></div> <h4 id="data-locality" class="position-relative d-flex align-items-center group"> Data Locality <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="data-locality" aria-haspopup="dialog" aria-label="Share link: Data Locality"> Share link </button> </h4>Keep related data together: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Good: Related entities on same shard -- User and their posts on same shard (by user_id hash) -- Query runs on single shard: MATCH (u:User {user_id: '12345'})-[:POSTS]->(p:Post) RETURN p ORDER BY p.created_at DESC LIMIT 20; -- Bad: User and posts on different shards -- Requires cross-shard query (slower) </code></pre></div> <h4 id="monitor-replication-lag" class="position-relative d-flex align-items-center group"> Monitor Replication Lag <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="monitor-replication-lag" aria-haspopup="dialog" aria-label="Share link: Monitor Replication Lag"> Share link </button> </h4>Track replica freshness: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Check replication status SELECT replica_name, last_update_time, lag_seconds, lag_bytes FROM system.replication_status ORDER BY lag_seconds DESC; -- Alert on excessive lag -- If lag_seconds > 60, investigate </code></pre></div> <h4 id="connection-pooling" class="position-relative d-flex align-items-center group"> Connection Pooling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="connection-pooling" aria-haspopup="dialog" aria-label="Share link: Connection Pooling"> Share link </button> </h4>Right-size connection pools: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"># Connection pool sizing formula max_connections_per_client = ( server_max_connections / number_of_application_instances ) # Example: # - Server max: 1000 connections # - Application instances: 20 # - Per-client max: 50 connections client = Client( host="geode.example.com", port=3141, max_connections=50, min_connections=5 ) </code></pre></div> <h3 id="troubleshooting" class="position-relative d-flex align-items-center group"> Troubleshooting <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="troubleshooting" aria-haspopup="dialog" aria-label="Share link: Troubleshooting"> Share link </button> </h3> <h4 id="hot-spots" class="position-relative d-flex align-items-center group"> Hot Spots <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="hot-spots" aria-haspopup="dialog" aria-label="Share link: Hot Spots"> Share link </button> </h4>Issue: One shard overloaded while others idle. Diagnosis: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Check shard load distribution SELECT shard_name, query_count, avg_cpu_percent, storage_gb FROM system.shard_stats ORDER BY query_count DESC; </code></pre></div>Solutions: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># 1. Rebalance sharding key sharding: key_property: user_id # High cardinality # Not: region # Low cardinality causes hot spots # 2. Split hot shard # If shard_3 is overloaded, split into shard_3a and shard_3b # 3. Add caching layer # Cache hot queries at application level </code></pre></div> <h4 id="replication-lag" class="position-relative d-flex align-items-center group"> Replication Lag <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="replication-lag" aria-haspopup="dialog" aria-label="Share link: Replication Lag"> Share link </button> </h4>Issue: Replicas falling behind primary. Solutions: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># 1. Increase replication threads replication: parallel_workers: 8 # Default: 4 # 2. Increase network bandwidth # Use 10GbE between primary and replicas # 3. Reduce write load on primary # - Batch writes # - Use async replication for non-critical data </code></pre></div> <h3 id="integration-with-geode" class="position-relative d-flex align-items-center group"> Integration with Geode <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="integration-with-geode" aria-haspopup="dialog" aria-label="Share link: Integration with Geode"> Share link </button> </h3>Scaling integrates with Geode features: <ul> <li>Monitoring: Track scaling metrics via Prometheus</li> <li>Transactions: ACID within shard, eventual across shards</li> <li>Security: RLS policies enforced on all nodes</li> <li>Backup: Coordinated backups across cluster</li> <li>Upgrades: Rolling upgrades with zero downtime</li> </ul> <h3 id="related-topics" class="position-relative d-flex align-items-center group"> Related Topics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="related-topics" aria-haspopup="dialog" aria-label="Share link: Related Topics"> Share link </button> </h3><ul> <li>High Availability: Failover and redundancy strategies</li> <li>Replication: Data replication patterns</li> <li>Performance: Query performance optimization</li> <li>Monitoring: Production monitoring and alerting</li> <li>Deployment: Production deployment patterns</li> <li>Cloud: Cloud-specific scaling strategies</li> </ul> <hr> Browse the tagged content below to discover comprehensive guides, tutorials, and best practices for scaling Geode. Learn how to build highly scalable graph database deployments that handle enterprise workloads with high availability and optimal performance.

Popular

Related Articles

Performance and Scalability