Performance Benchmarking

<h2 id="performance-benchmarking-in-geode" class="position-relative d-flex align-items-center group"> Performance Benchmarking in Geode <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="performance-benchmarking-in-geode" aria-haspopup="dialog" aria-label="Share link: Performance Benchmarking in Geode"> Share link </button> </h2><div id="headingShareModal" class="heading-share-modal" role="dialog" aria-modal="true" aria-labelledby="headingShareTitle" hidden> <div class="hsm-dialog" role="document"> <div class="hsm-header"> <h2 id="headingShareTitle" class="h6 mb-0 fw-bold">Share this section</h2> <button type="button" class="hsm-close" aria-label="Close"> </button> </div> <div class="hsm-body"> <label for="headingShareInput" class="form-label small text-muted mb-1 text-uppercase fw-bold" style="font-size: 0.7rem; letter-spacing: 0.5px;">Permalink</label> <div class="input-group mb-4 hsm-url-group"> <input id="headingShareInput" type="text" class="form-control font-monospace" readonly aria-readonly="true" style="font-size: 0.85rem;" /> <button class="btn btn-primary hsm-copy" type="button" aria-label="Copy" title="Copy"> </button> </div> <div class="small fw-bold mb-2 text-muted text-uppercase" style="font-size: 0.7rem; letter-spacing: 0.5px;">Share via</div> <div class="hsm-share-grid"> <a id="share-twitter" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Twitter </a> <a id="share-linkedin" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> LinkedIn </a> <a id="share-facebook" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Facebook </a> </div> </div> </div> </div> <style> .heading-share-modal { position: fixed; inset: 0; display: flex; justify-content: center; align-items: center; background: rgba(0, 0, 0, 0.6); z-index: 1050; padding: 1rem; backdrop-filter: blur(4px); -webkit-backdrop-filter: blur(4px); } .heading-share-modal[hidden] { display: none !important; } .hsm-dialog { max-width: 420px; width: 100%; background: var(--bs-body-bg, #fff); color: var(--bs-body-color, #212529); border: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); border-radius: 1rem; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); overflow: hidden; animation: hsm-fade-in 0.2s ease-out; } @keyframes hsm-fade-in { from { opacity: 0; transform: scale(0.95); } to { opacity: 1; transform: scale(1); } } [data-bs-theme="dark"] .hsm-dialog { background: #1e293b; border-color: rgba(255,255,255,0.1); color: #f8f9fa; } .hsm-header { display: flex; justify-content: space-between; align-items: center; padding: 1rem 1.5rem; border-bottom: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); background: rgba(0,0,0,0.02); } [data-bs-theme="dark"] .hsm-header { background: rgba(255,255,255,0.02); border-color: rgba(255,255,255,0.1); } .hsm-close { background: transparent; border: none; color: inherit; opacity: 0.5; padding: 0.25rem 0.5rem; border-radius: 0.25rem; font-size: 1.2rem; line-height: 1; transition: opacity 0.2s; } .hsm-close:hover { opacity: 1; } .hsm-body { padding: 1.5rem; } .hsm-url-group { display: flex !important; align-items: stretch; } .hsm-url-group .form-control { flex: 1; min-width: 0; margin: 0; background: var(--bs-secondary-bg, #f8f9fa); border-color: var(--bs-border-color, #dee2e6); border-top-right-radius: 0; border-bottom-right-radius: 0; height: 42px; } .hsm-url-group .btn { flex: 0 0 auto; margin: 0; margin-left: -1px; border-top-left-radius: 0; border-bottom-left-radius: 0; height: 42px; display: flex; align-items: center; justify-content: center; padding: 0 1.25rem; z-index: 2; } [data-bs-theme="dark"] .hsm-url-group .form-control { background: #0f172a; border-color: #334155; color: #e2e8f0; } .hsm-share-grid { display: flex; flex-direction: column; gap: 0.5rem; } .hsm-share-grid .btn { display: flex; align-items: center; justify-content: center; font-size: 0.9rem; padding: 0.6rem; border-color: var(--bs-border-color); width: 100%; } [data-bs-theme="dark"] .hsm-share-grid .btn { color: #e2e8f0; border-color: #475569; } [data-bs-theme="dark"] .hsm-share-grid .btn:hover { background: #334155; border-color: #cbd5e1; } </style> <script> (function(){ const modal = document.getElementById('headingShareModal'); if(!modal) return; const input = modal.querySelector('#headingShareInput'); const copyBtn = modal.querySelector('.hsm-copy'); const twitter = modal.querySelector('#share-twitter'); const linkedin = modal.querySelector('#share-linkedin'); const facebook = modal.querySelector('#share-facebook'); const closeBtn = modal.querySelector('.hsm-close'); let lastFocus=null; let trapBound=false; function buildUrl(id){ return window.location.origin + window.location.pathname + '#' + id; } function isOpen(){ return !modal.hasAttribute('hidden'); } function hydrate(id){ const url=buildUrl(id); input.value=url; const enc=encodeURIComponent(url); const text=encodeURIComponent(document.title); if(twitter) twitter.href=`https://twitter.com/intent/tweet?url=${enc}&text=${text}`; if(linkedin) linkedin.href=`https://www.linkedin.com/sharing/share-offsite/?url=${enc}`; if(facebook) facebook.href=`https://www.facebook.com/sharer/sharer.php?u=${enc}`; } function openModal(id){ lastFocus=document.activeElement; hydrate(id); if(!isOpen()){ modal.removeAttribute('hidden'); } requestAnimationFrame(()=>{ input.focus(); }); trapFocus(); } function closeModal(){ if(!isOpen()) return; modal.setAttribute('hidden',''); if(lastFocus && typeof lastFocus.focus==='function') lastFocus.focus(); } function copyCurrent(){ try{ navigator.clipboard.writeText(input.value).then(()=>feedback(true),()=>fallback()); } catch(e){ fallback(); } } function fallback(){ input.select(); try{ document.execCommand('copy'); feedback(true);}catch(e){ feedback(false);} } function feedback(ok){ if(!copyBtn) return; const icon=copyBtn.querySelector('i'); if(!icon) return; const prev=copyBtn.getAttribute('data-prev')||icon.className; if(!copyBtn.getAttribute('data-prev')) copyBtn.setAttribute('data-prev',prev); icon.className= ok ? 'fa-duotone fa-clipboard-check':'fa-duotone fa-circle-exclamation'; setTimeout(()=>{ icon.className=prev; },1800); } function handleShareClick(e){ e.preventDefault(); const btn=e.currentTarget; const id=btn.getAttribute('data-share-target'); if(id) openModal(id); } function bindShareButtons(){ document.querySelectorAll('.h-share').forEach(btn=>{ if(!btn.dataset.hShareBound){ btn.addEventListener('click', handleShareClick); btn.dataset.hShareBound='1'; } }); } bindShareButtons(); if(document.readyState==='loading'){ document.addEventListener('DOMContentLoaded', bindShareButtons); } else { requestAnimationFrame(bindShareButtons); } document.addEventListener('click', function(e){ const shareBtn=e.target.closest && e.target.closest('.h-share'); if(shareBtn && !shareBtn.dataset.hShareBound){ handleShareClick.call(shareBtn, e); } }, true); document.addEventListener('click', e=>{ if(e.target===modal) closeModal(); if(e.target.closest && e.target.closest('.hsm-close')){ e.preventDefault(); closeModal(); } if(copyBtn && (e.target===copyBtn || (e.target.closest && e.target.closest('.hsm-copy')))) { e.preventDefault(); copyCurrent(); } }); document.addEventListener('keydown', e=>{ if(e.key==='Escape' && isOpen()) closeModal(); }); function trapFocus(){ if(trapBound) return; trapBound=true; modal.addEventListener('keydown', f=>{ if(f.key==='Tab' && isOpen()){ const focusable=[...modal.querySelectorAll('a[href],button,input,textarea,select,[tabindex]:not([tabindex="-1"])')].filter(el=>!el.hasAttribute('disabled')); if(!focusable.length) return; const first=focusable[0]; const last=focusable[focusable.length-1]; if(f.shiftKey && document.activeElement===first){ f.preventDefault(); last.focus(); } else if(!f.shiftKey && document.activeElement===last){ f.preventDefault(); first.focus(); } } }); } if(closeBtn) closeBtn.addEventListener('click', e=>{ e.preventDefault(); closeModal(); }); })(); </script>Benchmarking provides objective measurements of database performance, enabling informed decisions about configuration, capacity planning, and optimization. Effective benchmarking reveals performance characteristics under various workloads and helps identify bottlenecks before they impact production systems. This guide covers benchmarking methodologies, tools, metrics, and best practices for measuring and comparing Geode performance. <h3 id="benchmarking-fundamentals" class="position-relative d-flex align-items-center group"> Benchmarking Fundamentals <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="benchmarking-fundamentals" aria-haspopup="dialog" aria-label="Share link: Benchmarking Fundamentals"> Share link </button> </h3> <h4 id="why-benchmark" class="position-relative d-flex align-items-center group"> Why Benchmark? <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="why-benchmark" aria-haspopup="dialog" aria-label="Share link: Why Benchmark?"> Share link </button> </h4>Benchmarking serves multiple purposes: Capacity Planning: Determine hardware requirements for expected workloads Configuration Tuning: Measure impact of configuration changes Regression Detection: Ensure updates don’t degrade performance Comparison: Evaluate Geode against alternatives or previous versions Optimization Validation: Verify that optimizations achieve intended improvements <h4 id="key-performance-metrics" class="position-relative d-flex align-items-center group"> Key Performance Metrics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="key-performance-metrics" aria-haspopup="dialog" aria-label="Share link: Key Performance Metrics"> Share link </button> </h4>Throughput: Operations per second the system can sustain <ul> <li>Queries per second (QPS)</li> <li>Transactions per second (TPS)</li> <li>Nodes/edges created per second</li> </ul> Latency: Time to complete individual operations <ul> <li>Average latency</li> <li>Percentiles (p50, p95, p99, p99.9)</li> <li>Maximum latency</li> </ul> Scalability: How performance changes with load <ul> <li>Vertical scaling (larger machines)</li> <li>Horizontal scaling (more nodes)</li> <li>Data volume scaling</li> </ul> Resource Utilization: System resource consumption <ul> <li>CPU usage</li> <li>Memory usage</li> <li>Disk I/O</li> <li>Network I/O</li> </ul> <h4 id="benchmarking-principles" class="position-relative d-flex align-items-center group"> Benchmarking Principles <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="benchmarking-principles" aria-haspopup="dialog" aria-label="Share link: Benchmarking Principles"> Share link </button> </h4>Reproducibility: Results should be consistent across runs Isolation: Control for external factors Realistic Workloads: Test patterns that match production Statistical Rigor: Collect sufficient samples, report variance Warm-up: Allow system to reach steady state before measuring <h3 id="geode-benchmark-suite" class="position-relative d-flex align-items-center group"> Geode Benchmark Suite <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="geode-benchmark-suite" aria-haspopup="dialog" aria-label="Share link: Geode Benchmark Suite"> Share link </button> </h3> <h4 id="built-in-benchmark-tool" class="position-relative d-flex align-items-center group"> Built-in Benchmark Tool <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="built-in-benchmark-tool" aria-haspopup="dialog" aria-label="Share link: Built-in Benchmark Tool"> Share link </button> </h4>Geode includes a benchmarking tool for common scenarios: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Run standard benchmark suite ./geode benchmark run \ --workload standard \ --duration 300 \ --connections 100 \ --output /benchmarks/results/ # Available workloads ./geode benchmark list-workloads # Workloads: # - standard: Mixed read/write operations # - read-heavy: 95% reads, 5% writes # - write-heavy: 20% reads, 80% writes # - traversal: Graph traversal patterns # - analytics: Aggregation and analysis queries # - import: Bulk data loading </code></pre></div> <h4 id="benchmark-configuration" class="position-relative d-flex align-items-center group"> Benchmark Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="benchmark-configuration" aria-haspopup="dialog" aria-label="Share link: Benchmark Configuration"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"># benchmark.toml [benchmark] name = "production-baseline" description = "Baseline performance measurement" [benchmark.workload] type = "standard" read_ratio = 0.8 write_ratio = 0.2 [benchmark.load] # Concurrent connections connections = 100 # Queries per connection per second rate_per_connection = 100 # Total duration duration_seconds = 300 [benchmark.data] # Initial dataset size initial_nodes = 1000000 initial_edges = 5000000 # Node/edge properties avg_properties_per_node = 5 avg_properties_per_edge = 2 [benchmark.queries] # Query distribution simple_lookup_percent = 40 index_scan_percent = 30 traversal_1hop_percent = 15 traversal_3hop_percent = 10 aggregation_percent = 5 [benchmark.output] format = "json" directory = "/benchmarks/results" include_histograms = true </code></pre></div> <h4 id="running-benchmarks" class="position-relative d-flex align-items-center group"> Running Benchmarks <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="running-benchmarks" aria-haspopup="dialog" aria-label="Share link: Running Benchmarks"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Run with configuration file ./geode benchmark run --config benchmark.toml # Run specific workload ./geode benchmark run \ --workload read-heavy \ --duration 600 \ --connections 200 \ --data-size 10000000 # Incremental load test ./geode benchmark run \ --workload standard \ --ramp-up 60 \ --connections-start 10 \ --connections-end 500 \ --duration 600 </code></pre></div> <h3 id="load-testing-patterns" class="position-relative d-flex align-items-center group"> Load Testing Patterns <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="load-testing-patterns" aria-haspopup="dialog" aria-label="Share link: Load Testing Patterns"> Share link </button> </h3> <h4 id="point-query-performance" class="position-relative d-flex align-items-center group"> Point Query Performance <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="point-query-performance" aria-haspopup="dialog" aria-label="Share link: Point Query Performance"> Share link </button> </h4>Measure single-record lookup performance: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">-- Benchmark query: lookup by ID MATCH (u:User {id: $id}) RETURN u.name, u.email, u.created_at; </code></pre></div>Benchmark Script: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">import asyncio import time import statistics from geode_client import Client async def benchmark_point_queries( host: str, num_queries: int, concurrency: int ): """Benchmark point query performance""" client = Client(host=host, port=3141) latencies = [] errors = 0 # Generate random user IDs user_ids = [f"user-{i}" for i in range(1, num_queries + 1)] semaphore = asyncio.Semaphore(concurrency) async def run_query(user_id): nonlocal errors async with semaphore: start = time.perf_counter() try: async with client.connection() as conn: await conn.query( "MATCH (u:User {id: $id}) RETURN u", {"id": user_id} ) latencies.append((time.perf_counter() - start) * 1000) except Exception: errors += 1 # Run queries start_time = time.perf_counter() await asyncio.gather(*[run_query(uid) for uid in user_ids]) total_time = time.perf_counter() - start_time # Report results print(f"Total queries: {num_queries}") print(f"Errors: {errors}") print(f"Duration: {total_time:.2f}s") print(f"Throughput: {num_queries/total_time:.0f} QPS") print(f"Latency avg: {statistics.mean(latencies):.2f}ms") print(f"Latency p50: {statistics.median(latencies):.2f}ms") print(f"Latency p95: {sorted(latencies)[int(len(latencies)*0.95)]:.2f}ms") print(f"Latency p99: {sorted(latencies)[int(len(latencies)*0.99)]:.2f}ms") asyncio.run(benchmark_point_queries("localhost", 100000, 100)) </code></pre></div> <h4 id="traversal-performance" class="position-relative d-flex align-items-center group"> Traversal Performance <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="traversal-performance" aria-haspopup="dialog" aria-label="Share link: Traversal Performance"> Share link </button> </h4>Measure graph traversal efficiency: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">async def benchmark_traversals( host: str, num_queries: int, max_depth: int ): """Benchmark traversal performance at various depths""" client = Client(host=host, port=3141) for depth in range(1, max_depth + 1): query = f""" MATCH (u:User {{id: $id}})-[:FOLLOWS*1..{depth}]->(friend) RETURN DISTINCT friend.id, friend.name LIMIT 100 """ latencies = [] for _ in range(num_queries): start = time.perf_counter() async with client.connection() as conn: result, _ = await conn.query(query, {"id": "user-1"}) latencies.append((time.perf_counter() - start) * 1000) print(f"Depth {depth}: avg={statistics.mean(latencies):.2f}ms, " f"p95={sorted(latencies)[int(len(latencies)*0.95)]:.2f}ms") </code></pre></div> <h4 id="write-performance" class="position-relative d-flex align-items-center group"> Write Performance <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="write-performance" aria-haspopup="dialog" aria-label="Share link: Write Performance"> Share link </button> </h4>Measure insert and update throughput: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">async def benchmark_writes( host: str, num_operations: int, batch_size: int ): """Benchmark write performance""" client = Client(host=host, port=3141) # Single inserts single_latencies = [] for i in range(num_operations): start = time.perf_counter() async with client.connection() as conn: await conn.execute(""" CREATE (u:User { id: $id, name: $name, email: $email, created_at: datetime() }) """, { "id": f"bench-user-{i}", "name": f"User {i}", "email": f"user{i}@example.com" }) single_latencies.append((time.perf_counter() - start) * 1000) print(f"Single inserts: {num_operations/sum(single_latencies)*1000:.0f} ops/sec") # Batch inserts batch_latencies = [] for batch_start in range(0, num_operations, batch_size): users = [ {"id": f"batch-user-{i}", "name": f"User {i}", "email": f"user{i}@example.com"} for i in range(batch_start, min(batch_start + batch_size, num_operations)) ] start = time.perf_counter() async with client.connection() as conn: await conn.execute(""" UNWIND $users AS u CREATE (n:User { id: u.id, name: u.name, email: u.email, created_at: datetime() }) """, {"users": users}) batch_latencies.append((time.perf_counter() - start) * 1000) print(f"Batch inserts ({batch_size}): " f"{num_operations/sum(batch_latencies)*1000:.0f} ops/sec") </code></pre></div> <h4 id="mixed-workload" class="position-relative d-flex align-items-center group"> Mixed Workload <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="mixed-workload" aria-haspopup="dialog" aria-label="Share link: Mixed Workload"> Share link </button> </h4>Simulate realistic production workloads: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">import random async def benchmark_mixed_workload( host: str, duration_seconds: int, read_ratio: float = 0.8 ): """Benchmark mixed read/write workload""" client = Client(host=host, port=3141) operations = { "read": 0, "write": 0, "read_latency": [], "write_latency": [] } start_time = time.time() while time.time() - start_time < duration_seconds: if random.random() < read_ratio: # Read operation user_id = f"user-{random.randint(1, 100000)}" start = time.perf_counter() async with client.connection() as conn: await conn.query( "MATCH (u:User {id: $id}) RETURN u", {"id": user_id} ) operations["read"] += 1 operations["read_latency"].append((time.perf_counter() - start) * 1000) else: # Write operation start = time.perf_counter() async with client.connection() as conn: await conn.execute( "MATCH (u:User {id: $id}) SET u.updated_at = datetime()", {"id": f"user-{random.randint(1, 100000)}"} ) operations["write"] += 1 operations["write_latency"].append((time.perf_counter() - start) * 1000) elapsed = time.time() - start_time print(f"Duration: {elapsed:.0f}s") print(f"Read throughput: {operations['read']/elapsed:.0f} QPS") print(f"Write throughput: {operations['write']/elapsed:.0f} TPS") print(f"Read latency avg: {statistics.mean(operations['read_latency']):.2f}ms") print(f"Write latency avg: {statistics.mean(operations['write_latency']):.2f}ms") </code></pre></div> <h3 id="stress-testing" class="position-relative d-flex align-items-center group"> Stress Testing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="stress-testing" aria-haspopup="dialog" aria-label="Share link: Stress Testing"> Share link </button> </h3> <h4 id="finding-breaking-points" class="position-relative d-flex align-items-center group"> Finding Breaking Points <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="finding-breaking-points" aria-haspopup="dialog" aria-label="Share link: Finding Breaking Points"> Share link </button> </h4>Gradually increase load to find system limits: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Incremental load test ./geode benchmark stress \ --start-rate 100 \ --end-rate 10000 \ --step 100 \ --step-duration 30 \ --workload standard </code></pre></div>Custom Stress Test: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">async def stress_test(host: str, max_rate: int, step: int): """Find maximum sustainable throughput""" results = [] for rate in range(step, max_rate + 1, step): print(f"Testing rate: {rate} QPS") latencies = [] errors = 0 target_interval = 1.0 / rate start_time = time.time() last_query = start_time while time.time() - start_time < 30: # 30 second test per rate # Rate limiting now = time.time() if now - last_query < target_interval: await asyncio.sleep(target_interval - (now - last_query)) last_query = time.time() # Execute query query_start = time.perf_counter() try: async with client.connection() as conn: await conn.query("MATCH (u:User {id: $id}) RETURN u", {"id": "user-1"}) latencies.append((time.perf_counter() - query_start) * 1000) except Exception: errors += 1 actual_rate = len(latencies) / 30 p99_latency = sorted(latencies)[int(len(latencies) * 0.99)] if latencies else 0 results.append({ "target_rate": rate, "actual_rate": actual_rate, "p99_latency": p99_latency, "error_rate": errors / (len(latencies) + errors) if latencies else 1 }) # Stop if system is saturated if p99_latency > 100 or errors > len(latencies) * 0.01: print(f"System saturated at {rate} QPS") break return results </code></pre></div> <h4 id="sustained-load-testing" class="position-relative d-flex align-items-center group"> Sustained Load Testing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="sustained-load-testing" aria-haspopup="dialog" aria-label="Share link: Sustained Load Testing"> Share link </button> </h4>Test stability under continuous load: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># 24-hour stability test ./geode benchmark run \ --workload standard \ --duration 86400 \ --connections 100 \ --rate 5000 \ --report-interval 60 </code></pre></div> <h4 id="failure-injection" class="position-relative d-flex align-items-center group"> Failure Injection <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="failure-injection" aria-haspopup="dialog" aria-label="Share link: Failure Injection"> Share link </button> </h4>Test performance under failure conditions: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">async def benchmark_with_failures(host: str, duration: int): """Benchmark while injecting failures""" import subprocess async def inject_failure(): """Periodically restart a node""" while True: await asyncio.sleep(60) # Every minute print("Injecting failure: restarting node2") subprocess.run(["docker", "restart", "geode-node2"]) # Run benchmark and failure injection concurrently await asyncio.gather( benchmark_mixed_workload(host, duration), inject_failure() ) </code></pre></div> <h3 id="comparison-benchmarking" class="position-relative d-flex align-items-center group"> Comparison Benchmarking <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="comparison-benchmarking" aria-haspopup="dialog" aria-label="Share link: Comparison Benchmarking"> Share link </button> </h3> <h4 id="comparing-configurations" class="position-relative d-flex align-items-center group"> Comparing Configurations <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="comparing-configurations" aria-haspopup="dialog" aria-label="Share link: Comparing Configurations"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">async def compare_configurations(configs: list): """Compare performance across different configurations""" results = {} for config_name, config_file in configs: print(f"Testing configuration: {config_name}") # Apply configuration subprocess.run(["./geode", "admin", "reload-config", config_file]) await asyncio.sleep(10) # Wait for config to apply # Run benchmark result = await benchmark_mixed_workload("localhost", 300) results[config_name] = result # Report comparison print("\n=== Configuration Comparison ===") print(f"{'Config':<20} {'Read QPS':<12} {'Write TPS':<12} {'P99 Latency':<12}") for name, result in results.items(): print(f"{name:<20} {result['read_qps']:<12.0f} " f"{result['write_tps']:<12.0f} {result['p99_latency']:<12.2f}") </code></pre></div> <h4 id="version-comparison" class="position-relative d-flex align-items-center group"> Version Comparison <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="version-comparison" aria-haspopup="dialog" aria-label="Share link: Version Comparison"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Benchmark current version ./geode benchmark run --workload standard --output results-v0.18.json # Switch to previous version ./deploy-version.sh v0.17 # Benchmark previous version ./geode benchmark run --workload standard --output results-v0.17.json # Compare results ./geode benchmark compare results-v0.17.json results-v0.18.json </code></pre></div> <h3 id="benchmark-reporting" class="position-relative d-flex align-items-center group"> Benchmark Reporting <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="benchmark-reporting" aria-haspopup="dialog" aria-label="Share link: Benchmark Reporting"> Share link </button> </h3> <h4 id="json-output" class="position-relative d-flex align-items-center group"> JSON Output <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="json-output" aria-haspopup="dialog" aria-label="Share link: JSON Output"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json">{ "benchmark_name": "production-baseline", "timestamp": "2026-01-28T10:30:00Z", "duration_seconds": 300, "configuration": { "connections": 100, "workload": "standard", "data_size": 1000000 }, "results": { "throughput": { "total_operations": 1547832, "operations_per_second": 5159.44, "reads_per_second": 4127.55, "writes_per_second": 1031.89 }, "latency": { "read": { "avg_ms": 2.34, "p50_ms": 1.89, "p95_ms": 4.56, "p99_ms": 8.12, "p999_ms": 23.45, "max_ms": 156.78 }, "write": { "avg_ms": 5.67, "p50_ms": 4.23, "p95_ms": 12.34, "p99_ms": 24.56, "p999_ms": 67.89, "max_ms": 234.56 } }, "errors": { "total": 23, "rate": 0.000015, "by_type": { "timeout": 15, "connection_refused": 8 } }, "resources": { "cpu_avg_percent": 67.5, "memory_avg_mb": 4523, "disk_iops_avg": 12456, "network_mbps_avg": 234.5 } } } </code></pre></div> <h4 id="visualization" class="position-relative d-flex align-items-center group"> Visualization <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="visualization" aria-haspopup="dialog" aria-label="Share link: Visualization"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">import matplotlib.pyplot as plt import json def plot_benchmark_results(results_file: str): """Generate benchmark visualization""" with open(results_file) as f: data = json.load(f) # Latency distribution fig, axes = plt.subplots(2, 2, figsize=(12, 10)) # Throughput over time ax1 = axes[0, 0] ax1.plot(data['time_series']['throughput']) ax1.set_title('Throughput Over Time') ax1.set_xlabel('Time (seconds)') ax1.set_ylabel('Operations/second') # Latency percentiles ax2 = axes[0, 1] percentiles = ['p50', 'p95', 'p99', 'p999'] values = [data['results']['latency']['read'][f'{p}_ms'] for p in percentiles] ax2.bar(percentiles, values) ax2.set_title('Read Latency Percentiles') ax2.set_ylabel('Latency (ms)') # Resource utilization ax3 = axes[1, 0] ax3.plot(data['time_series']['cpu'], label='CPU') ax3.plot(data['time_series']['memory'], label='Memory') ax3.set_title('Resource Utilization') ax3.legend() # Error rate ax4 = axes[1, 1] ax4.plot(data['time_series']['error_rate']) ax4.set_title('Error Rate Over Time') ax4.set_xlabel('Time (seconds)') ax4.set_ylabel('Error Rate') plt.tight_layout() plt.savefig('benchmark_results.png') plt.show() </code></pre></div> <h3 id="monitoring-during-benchmarks" class="position-relative d-flex align-items-center group"> Monitoring During Benchmarks <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="monitoring-during-benchmarks" aria-haspopup="dialog" aria-label="Share link: Monitoring During Benchmarks"> Share link </button> </h3> <h4 id="metrics-collection" class="position-relative d-flex align-items-center group"> Metrics Collection <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="metrics-collection" aria-haspopup="dialog" aria-label="Share link: Metrics Collection"> Share link </button> </h4>Collect comprehensive metrics during benchmarks: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Start Prometheus metrics collection curl http://localhost:3141/metrics > metrics_before.txt # Run benchmark ./geode benchmark run --workload standard --duration 300 # Collect metrics after curl http://localhost:3141/metrics > metrics_after.txt </code></pre></div> <h4 id="real-time-monitoring" class="position-relative d-flex align-items-center group"> Real-Time Monitoring <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="real-time-monitoring" aria-haspopup="dialog" aria-label="Share link: Real-Time Monitoring"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">import prometheus_client async def monitor_benchmark(benchmark_coro, metrics_port=8000): """Monitor benchmark with Prometheus metrics""" # Start metrics server prometheus_client.start_http_server(metrics_port) # Custom metrics ops_counter = prometheus_client.Counter( 'benchmark_operations_total', 'Total benchmark operations', ['operation_type'] ) latency_histogram = prometheus_client.Histogram( 'benchmark_latency_seconds', 'Operation latency', ['operation_type'], buckets=[.001, .005, .01, .025, .05, .1, .25, .5, 1.0, 2.5, 5.0] ) # Run benchmark with metrics async def instrumented_query(query_func, op_type): with latency_histogram.labels(op_type).time(): result = await query_func() ops_counter.labels(op_type).inc() return result await benchmark_coro(instrumented_query) </code></pre></div> <h3 id="best-practices" class="position-relative d-flex align-items-center group"> Best Practices <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="best-practices" aria-haspopup="dialog" aria-label="Share link: Best Practices"> Share link </button> </h3> <h4 id="benchmark-environment" class="position-relative d-flex align-items-center group"> Benchmark Environment <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="benchmark-environment" aria-haspopup="dialog" aria-label="Share link: Benchmark Environment"> Share link </button> </h4><ol> <li>Isolate the system: Dedicated hardware, no competing workloads</li> <li>Control external factors: Network latency, disk state</li> <li>Use production-like data: Size, distribution, and patterns</li> <li>Document environment: Hardware, OS, configuration</li> </ol> <h4 id="methodology" class="position-relative d-flex align-items-center group"> Methodology <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="methodology" aria-haspopup="dialog" aria-label="Share link: Methodology"> Share link </button> </h4><ol> <li>Warm up the system: Run queries before measuring</li> <li>Use sufficient duration: Minimum 5 minutes, preferably longer</li> <li>Multiple runs: Average results across 3-5 runs</li> <li>Report variance: Include standard deviation</li> <li>Test at multiple load levels: Find saturation point</li> </ol> <h4 id="reporting" class="position-relative d-flex align-items-center group"> Reporting <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="reporting" aria-haspopup="dialog" aria-label="Share link: Reporting"> Share link </button> </h4><ol> <li>Include all percentiles: p50, p95, p99, p99.9, max</li> <li>Report throughput and latency: Both matter</li> <li>Document configuration: Complete reproducibility</li> <li>Show resource utilization: CPU, memory, I/O</li> <li>Include error rates: Completeness matters</li> </ol> <h4 id="common-pitfalls" class="position-relative d-flex align-items-center group"> Common Pitfalls <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="common-pitfalls" aria-haspopup="dialog" aria-label="Share link: Common Pitfalls"> Share link </button> </h4><ol> <li>Insufficient warm-up: Cold caches skew results</li> <li>Too short duration: May miss periodic behaviors</li> <li>Client-side bottlenecks: Ensure client can generate load</li> <li>Network effects: Latency may be network, not database</li> <li>Ignoring errors: High throughput with errors is meaningless</li> </ol> <h3 id="benchmark-checklist" class="position-relative d-flex align-items-center group"> Benchmark Checklist <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="benchmark-checklist" aria-haspopup="dialog" aria-label="Share link: Benchmark Checklist"> Share link </button> </h3>Before running benchmarks: <ul> <li><input disabled="" type="checkbox"> Environment isolated and documented</li> <li><input disabled="" type="checkbox"> Data loaded and representative</li> <li><input disabled="" type="checkbox"> Configuration documented</li> <li><input disabled="" type="checkbox"> Monitoring in place</li> <li><input disabled="" type="checkbox"> Warm-up period defined</li> </ul> During benchmarks: <ul> <li><input disabled="" type="checkbox"> Monitor resource utilization</li> <li><input disabled="" type="checkbox"> Check for errors</li> <li><input disabled="" type="checkbox"> Verify steady-state reached</li> <li><input disabled="" type="checkbox"> Collect all metrics</li> </ul> After benchmarks: <ul> <li><input disabled="" type="checkbox"> Verify data integrity</li> <li><input disabled="" type="checkbox"> Analyze results</li> <li><input disabled="" type="checkbox"> Document findings</li> <li><input disabled="" type="checkbox"> Archive raw data</li> </ul> <h3 id="related-topics" class="position-relative d-flex align-items-center group"> Related Topics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="related-topics" aria-haspopup="dialog" aria-label="Share link: Related Topics"> Share link </button> </h3><ul> <li><a href="/tags/performance/" >Performance</a> - Performance optimization</li> <li><a href="/tags/profiling/" >Profiling</a> - Query profiling</li> <li><a href="/tags/monitoring/" >Monitoring</a> - System monitoring</li> <li><a href="/tags/query-optimization/" >Query Optimization</a> - Query tuning</li> <li><a href="/tags/configuration/" >Configuration</a> - Server configuration</li> <li><a href="/tags/scaling/" >Scaling</a> - Scaling strategies</li> </ul> <h3 id="further-reading" class="position-relative d-flex align-items-center group"> Further Reading <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="further-reading" aria-haspopup="dialog" aria-label="Share link: Further Reading"> Share link </button> </h3><ul> <li>Benchmarking Methodology Guide</li> <li>Load Testing Best Practices</li> <li>Performance Regression Testing</li> <li>Capacity Planning with Benchmarks</li> <li>Benchmark Result Analysis</li> <li>Industry Standard Benchmarks (LDBC, etc.)</li> </ul>

Popular

Related Articles

Benchmarking