System Observability

Observability is the ability to understand a system’s internal state by examining its outputs. Unlike traditional monitoring that focuses on predefined metrics, observability enables you to ask arbitrary questions about your system’s behavior, making it essential for debugging complex distributed systems and understanding emergent behaviors. Geode implements comprehensive observability through the three pillars: metrics for quantitative measurements, logs for detailed event records, and traces for request flow visualization. This multi-dimensional approach provides complete visibility into query execution, transaction behavior, resource utilization, and system health. This guide covers observability architecture, implementation patterns, debugging strategies, and best practices for maintaining observable Geode deployments. <h3 id="the-three-pillars-of-observability" class="position-relative d-flex align-items-center group"> The Three Pillars of Observability <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="the-three-pillars-of-observability" aria-haspopup="dialog" aria-label="Share link: The Three Pillars of Observability"> Share link </button> </h3><div id="headingShareModal" class="heading-share-modal" role="dialog" aria-modal="true" aria-labelledby="headingShareTitle" hidden> <div class="hsm-dialog" role="document"> <div class="hsm-header"> <h2 id="headingShareTitle" class="h6 mb-0 fw-bold">Share this section</h2> <button type="button" class="hsm-close" aria-label="Close"> </button> </div> <div class="hsm-body"> <label for="headingShareInput" class="form-label small text-muted mb-1 text-uppercase fw-bold" style="font-size: 0.7rem; letter-spacing: 0.5px;">Permalink</label> <div class="input-group mb-4 hsm-url-group"> <input id="headingShareInput" type="text" class="form-control font-monospace" readonly aria-readonly="true" style="font-size: 0.85rem;" /> <button class="btn btn-primary hsm-copy" type="button" aria-label="Copy" title="Copy"> </button> </div> <div class="small fw-bold mb-2 text-muted text-uppercase" style="font-size: 0.7rem; letter-spacing: 0.5px;">Share via</div> <div class="hsm-share-grid"> <a id="share-twitter" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Twitter </a> <a id="share-linkedin" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> LinkedIn </a> <a id="share-facebook" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Facebook </a> </div> </div> </div> </div> <style> .heading-share-modal { position: fixed; inset: 0; display: flex; justify-content: center; align-items: center; background: rgba(0, 0, 0, 0.6); z-index: 1050; padding: 1rem; backdrop-filter: blur(4px); -webkit-backdrop-filter: blur(4px); } .heading-share-modal[hidden] { display: none !important; } .hsm-dialog { max-width: 420px; width: 100%; background: var(--bs-body-bg, #fff); color: var(--bs-body-color, #212529); border: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); border-radius: 1rem; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); overflow: hidden; animation: hsm-fade-in 0.2s ease-out; } @keyframes hsm-fade-in { from { opacity: 0; transform: scale(0.95); } to { opacity: 1; transform: scale(1); } } [data-bs-theme="dark"] .hsm-dialog { background: #1e293b; border-color: rgba(255,255,255,0.1); color: #f8f9fa; } .hsm-header { display: flex; justify-content: space-between; align-items: center; padding: 1rem 1.5rem; border-bottom: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); background: rgba(0,0,0,0.02); } [data-bs-theme="dark"] .hsm-header { background: rgba(255,255,255,0.02); border-color: rgba(255,255,255,0.1); } .hsm-close { background: transparent; border: none; color: inherit; opacity: 0.5; padding: 0.25rem 0.5rem; border-radius: 0.25rem; font-size: 1.2rem; line-height: 1; transition: opacity 0.2s; } .hsm-close:hover { opacity: 1; } .hsm-body { padding: 1.5rem; } .hsm-url-group { display: flex !important; align-items: stretch; } .hsm-url-group .form-control { flex: 1; min-width: 0; margin: 0; background: var(--bs-secondary-bg, #f8f9fa); border-color: var(--bs-border-color, #dee2e6); border-top-right-radius: 0; border-bottom-right-radius: 0; height: 42px; } .hsm-url-group .btn { flex: 0 0 auto; margin: 0; margin-left: -1px; border-top-left-radius: 0; border-bottom-left-radius: 0; height: 42px; display: flex; align-items: center; justify-content: center; padding: 0 1.25rem; z-index: 2; } [data-bs-theme="dark"] .hsm-url-group .form-control { background: #0f172a; border-color: #334155; color: #e2e8f0; } .hsm-share-grid { display: flex; flex-direction: column; gap: 0.5rem; } .hsm-share-grid .btn { display: flex; align-items: center; justify-content: center; font-size: 0.9rem; padding: 0.6rem; border-color: var(--bs-border-color); width: 100%; } [data-bs-theme="dark"] .hsm-share-grid .btn { color: #e2e8f0; border-color: #475569; } [data-bs-theme="dark"] .hsm-share-grid .btn:hover { background: #334155; border-color: #cbd5e1; } </style> <script> (function(){ const modal = document.getElementById('headingShareModal'); if(!modal) return; const input = modal.querySelector('#headingShareInput'); const copyBtn = modal.querySelector('.hsm-copy'); const twitter = modal.querySelector('#share-twitter'); const linkedin = modal.querySelector('#share-linkedin'); const facebook = modal.querySelector('#share-facebook'); const closeBtn = modal.querySelector('.hsm-close'); let lastFocus=null; let trapBound=false; function buildUrl(id){ return window.location.origin + window.location.pathname + '#' + id; } function isOpen(){ return !modal.hasAttribute('hidden'); } function hydrate(id){ const url=buildUrl(id); input.value=url; const enc=encodeURIComponent(url); const text=encodeURIComponent(document.title); if(twitter) twitter.href=`https://twitter.com/intent/tweet?url=${enc}&text=${text}`; if(linkedin) linkedin.href=`https://www.linkedin.com/sharing/share-offsite/?url=${enc}`; if(facebook) facebook.href=`https://www.facebook.com/sharer/sharer.php?u=${enc}`; } function openModal(id){ lastFocus=document.activeElement; hydrate(id); if(!isOpen()){ modal.removeAttribute('hidden'); } requestAnimationFrame(()=>{ input.focus(); }); trapFocus(); } function closeModal(){ if(!isOpen()) return; modal.setAttribute('hidden',''); if(lastFocus && typeof lastFocus.focus==='function') lastFocus.focus(); } function copyCurrent(){ try{ navigator.clipboard.writeText(input.value).then(()=>feedback(true),()=>fallback()); } catch(e){ fallback(); } } function fallback(){ input.select(); try{ document.execCommand('copy'); feedback(true);}catch(e){ feedback(false);} } function feedback(ok){ if(!copyBtn) return; const icon=copyBtn.querySelector('i'); if(!icon) return; const prev=copyBtn.getAttribute('data-prev')||icon.className; if(!copyBtn.getAttribute('data-prev')) copyBtn.setAttribute('data-prev',prev); icon.className= ok ? 'fa-duotone fa-clipboard-check':'fa-duotone fa-circle-exclamation'; setTimeout(()=>{ icon.className=prev; },1800); } function handleShareClick(e){ e.preventDefault(); const btn=e.currentTarget; const id=btn.getAttribute('data-share-target'); if(id) openModal(id); } function bindShareButtons(){ document.querySelectorAll('.h-share').forEach(btn=>{ if(!btn.dataset.hShareBound){ btn.addEventListener('click', handleShareClick); btn.dataset.hShareBound='1'; } }); } bindShareButtons(); if(document.readyState==='loading'){ document.addEventListener('DOMContentLoaded', bindShareButtons); } else { requestAnimationFrame(bindShareButtons); } document.addEventListener('click', function(e){ const shareBtn=e.target.closest && e.target.closest('.h-share'); if(shareBtn && !shareBtn.dataset.hShareBound){ handleShareClick.call(shareBtn, e); } }, true); document.addEventListener('click', e=>{ if(e.target===modal) closeModal(); if(e.target.closest && e.target.closest('.hsm-close')){ e.preventDefault(); closeModal(); } if(copyBtn && (e.target===copyBtn || (e.target.closest && e.target.closest('.hsm-copy')))) { e.preventDefault(); copyCurrent(); } }); document.addEventListener('keydown', e=>{ if(e.key==='Escape' && isOpen()) closeModal(); }); function trapFocus(){ if(trapBound) return; trapBound=true; modal.addEventListener('keydown', f=>{ if(f.key==='Tab' && isOpen()){ const focusable=[...modal.querySelectorAll('a[href],button,input,textarea,select,[tabindex]:not([tabindex="-1"])')].filter(el=>!el.hasAttribute('disabled')); if(!focusable.length) return; const first=focusable[0]; const last=focusable[focusable.length-1]; if(f.shiftKey && document.activeElement===first){ f.preventDefault(); last.focus(); } else if(!f.shiftKey && document.activeElement===last){ f.preventDefault(); first.focus(); } } }); } if(closeBtn) closeBtn.addEventListener('click', e=>{ e.preventDefault(); closeModal(); }); })(); </script> <h4 id="metrics-what-is-happening" class="position-relative d-flex align-items-center group"> Metrics: What is Happening <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="metrics-what-is-happening" aria-haspopup="dialog" aria-label="Share link: Metrics: What is Happening"> Share link </button> </h4>Metrics provide aggregated, time-series data about system behavior: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"># Query throughput over time rate(geode_queries_total[5m]) # p95 query latency trend histogram_quantile(0.95, rate(geode_query_duration_seconds_bucket[5m])) # Memory usage pattern geode_memory_used_bytes </code></pre></div>Use Cases: <ul> <li>Trend analysis and anomaly detection</li> <li>Performance baseline establishment</li> <li>Capacity planning</li> <li>SLO monitoring</li> <li>Alerting on threshold violations</li> </ul> <h4 id="logs-why-is-it-happening" class="position-relative d-flex align-items-center group"> Logs: Why is it Happening <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="logs-why-is-it-happening" aria-haspopup="dialog" aria-label="Share link: Logs: Why is it Happening"> Share link </button> </h4>Logs capture discrete events with contextual details: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json">{ "timestamp": "2026-01-24T10:15:30.123Z", "level": "ERROR", "message": "Query execution failed", "query_id": "q-12847", "user": "analyst", "error": "Index out of bounds", "query_text": "MATCH (n:User) WHERE n.id > $limit RETURN n", "stack_trace": "...", "duration_ms": 234.5 } </code></pre></div>Use Cases: <ul> <li>Root cause analysis</li> <li>Error investigation</li> <li>Audit trails</li> <li>Security forensics</li> <li>Understanding specific execution paths</li> </ul> <h4 id="traces-how-is-it-flowing" class="position-relative d-flex align-items-center group"> Traces: How is it Flowing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="traces-how-is-it-flowing" aria-haspopup="dialog" aria-label="Share link: Traces: How is it Flowing"> Share link </button> </h4>Traces show request flows through distributed systems: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">Trace ID: abc123-def456-ghi789 Span: http_handler [250ms] └─ Span: authenticate_user [15ms] └─ Span: execute_gql_query [220ms] ├─ Span: parse_query [5ms] ├─ Span: optimize_plan [10ms] ├─ Span: execute_plan [200ms] │ ├─ Span: index_lookup [30ms] │ ├─ Span: expand_relationships [150ms] │ └─ Span: aggregate_results [20ms] └─ Span: serialize_response [5ms] </code></pre></div>Use Cases: <ul> <li>Performance bottleneck identification</li> <li>Understanding service dependencies</li> <li>Latency attribution</li> <li>Distributed debugging</li> <li>Optimization targeting</li> </ul> <h3 id="structured-logging" class="position-relative d-flex align-items-center group"> Structured Logging <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="structured-logging" aria-haspopup="dialog" aria-label="Share link: Structured Logging"> Share link </button> </h3>Geode uses structured JSON logging for machine-readable, queryable log data: <h4 id="log-configuration" class="position-relative d-flex align-items-center group"> Log Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="log-configuration" aria-haspopup="dialog" aria-label="Share link: Log Configuration"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"># geode.toml [logging] level = "INFO" # DEBUG, INFO, WARN, ERROR format = "json" # json or text output = "stdout" # stdout, stderr, file file = "/var/log/geode/geode.log" rotate_size = "100MB" rotate_count = 10 include_caller = true # Include file:line information </code></pre></div> <h4 id="log-levels" class="position-relative d-flex align-items-center group"> Log Levels <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="log-levels" aria-haspopup="dialog" aria-label="Share link: Log Levels"> Share link </button> </h4>DEBUG: Detailed diagnostic information <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json">{ "level": "DEBUG", "message": "Query plan generated", "query_id": "q-12847", "plan_type": "indexed_lookup", "estimated_rows": 1250, "index_used": "User.email" } </code></pre></div>INFO: General operational events <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json">{ "level": "INFO", "message": "Query executed successfully", "query_id": "q-12847", "duration_ms": 45.3, "rows_returned": 1250 } </code></pre></div>WARN: Warning conditions <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json">{ "level": "WARN", "message": "Slow query detected", "query_id": "q-12847", "duration_ms": 1234.5, "threshold_ms": 1000, "query_text": "MATCH (n) RETURN n" } </code></pre></div>ERROR: Error events requiring attention <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json">{ "level": "ERROR", "message": "Transaction conflict", "transaction_id": "tx-456", "conflict_type": "write_write", "retry_count": 3, "error_code": "40001" } </code></pre></div> <h4 id="contextual-logging" class="position-relative d-flex align-items-center group"> Contextual Logging <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="contextual-logging" aria-haspopup="dialog" aria-label="Share link: Contextual Logging"> Share link </button> </h4>Add context to log entries for correlation: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"># Python client with logging context import logging import geode_client logger = logging.getLogger(__name__) async def process_user_query(user_id, query): # Add context to all subsequent logs with logger.contextualize(user_id=user_id, request_id=generate_id()): logger.info("Processing user query", query_type="recommendation") try: result, _ = await client.query(query) logger.info( "Query completed", rows_returned=len(result.rows), duration_ms=result.duration ) return result except Exception as e: logger.error( "Query failed", error=str(e), query_text=query ) raise </code></pre></div> <h4 id="log-aggregation" class="position-relative d-flex align-items-center group"> Log Aggregation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="log-aggregation" aria-haspopup="dialog" aria-label="Share link: Log Aggregation"> Share link </button> </h4>Centralize logs with popular aggregation tools: Elasticsearch Integration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/geode/*.log json.keys_under_root: true json.add_error_key: true output.elasticsearch: hosts: ["elasticsearch:9200"] index: "geode-logs-%{+yyyy.MM.dd}" </code></pre></div>Loki Integration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># promtail.yml clients: - url: http://loki:3100/loki/api/v1/push scrape_configs: - job_name: geode static_configs: - targets: - localhost labels: job: geode __path__: /var/log/geode/*.log pipeline_stages: - json: expressions: level: level timestamp: timestamp message: message - labels: level: </code></pre></div> <h3 id="distributed-tracing" class="position-relative d-flex align-items-center group"> Distributed Tracing <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="distributed-tracing" aria-haspopup="dialog" aria-label="Share link: Distributed Tracing"> Share link </button> </h3> <h4 id="opentelemetry-integration" class="position-relative d-flex align-items-center group"> OpenTelemetry Integration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="opentelemetry-integration" aria-haspopup="dialog" aria-label="Share link: OpenTelemetry Integration"> Share link </button> </h4>Geode supports OpenTelemetry for standardized distributed tracing: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"># geode.toml [tracing] enabled = true exporter = "otlp" endpoint = "http://localhost:4317" sample_rate = 0.1 # Sample 10% of traces service_name = "geode" environment = "production" # Trace specific operations trace_queries = true trace_transactions = true trace_index_operations = true </code></pre></div> <h4 id="trace-instrumentation" class="position-relative d-flex align-items-center group"> Trace Instrumentation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="trace-instrumentation" aria-haspopup="dialog" aria-label="Share link: Trace Instrumentation"> Share link </button> </h4>Automatic instrumentation captures spans for: Query Execution: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">Span: execute_gql_query Attributes: - query.text: "MATCH (u:User) WHERE u.age > 25 RETURN u" - query.id: "q-12847" - query.status: "success" - query.rows: 1250 Duration: 145ms Child Spans: - parse_query (5ms) - optimize_plan (10ms) - execute_plan (125ms) - serialize_response (5ms) </code></pre></div>Transaction Lifecycle: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">Span: transaction Attributes: - tx.id: "tx-456" - tx.isolation_level: "SERIALIZABLE" - tx.status: "committed" Duration: 2340ms Child Spans: - begin (2ms) - execute_query_1 (145ms) - execute_query_2 (234ms) - commit (15ms) </code></pre></div>Index Operations: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">Span: index_build Attributes: - index.name: "User.email" - index.type: "btree" - index.rows: 1000000 Duration: 45000ms </code></pre></div> <h4 id="custom-spans" class="position-relative d-flex align-items-center group"> Custom Spans <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="custom-spans" aria-haspopup="dialog" aria-label="Share link: Custom Spans"> Share link </button> </h4>Add application-specific tracing: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">from opentelemetry import trace tracer = trace.get_tracer(__name__) async def recommend_products(user_id): with tracer.start_as_current_span("recommend_products") as span: span.set_attribute("user_id", user_id) # Fetch user preferences with tracer.start_as_current_span("fetch_preferences"): prefs, _ = await client.query( "MATCH (u:User {id: $id})-[:LIKES]->(p:Product) RETURN p", {"id": user_id} ) # Generate recommendations with tracer.start_as_current_span("generate_recommendations") as rec_span: rec_span.set_attribute("input_products", len(prefs)) recommendations = await compute_recommendations(prefs) rec_span.set_attribute("recommendations_count", len(recommendations)) return recommendations </code></pre></div> <h4 id="trace-sampling" class="position-relative d-flex align-items-center group"> Trace Sampling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="trace-sampling" aria-haspopup="dialog" aria-label="Share link: Trace Sampling"> Share link </button> </h4>Control trace volume with intelligent sampling: Tail-Based Sampling: Sample based on trace characteristics <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml">[tracing.sampling] strategy = "tail_based" # Always sample errors sample_on_error = true # Always sample slow requests slow_threshold_ms = 1000 sample_slow = true # Sample 10% of normal requests default_rate = 0.1 </code></pre></div>Probabilistic Sampling: Random sampling percentage <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml">[tracing.sampling] strategy = "probabilistic" rate = 0.05 # Sample 5% of traces </code></pre></div> <h3 id="correlation-across-pillars" class="position-relative d-flex align-items-center group"> Correlation Across Pillars <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="correlation-across-pillars" aria-haspopup="dialog" aria-label="Share link: Correlation Across Pillars"> Share link </button> </h3>Link metrics, logs, and traces for comprehensive debugging: <h4 id="request-id-propagation" class="position-relative d-flex align-items-center group"> Request ID Propagation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="request-id-propagation" aria-haspopup="dialog" aria-label="Share link: Request ID Propagation"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">import uuid # Generate request ID request_id = str(uuid.uuid4()) # Include in query metadata result, _ = await client.query( query, params, metadata={"request_id": request_id} ) # Request ID appears in: # - Trace (span attribute) # - Logs (log field) # - Metrics (optional label for custom metrics) </code></pre></div> <h4 id="trace-id-in-logs" class="position-relative d-flex align-items-center group"> Trace ID in Logs <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="trace-id-in-logs" aria-haspopup="dialog" aria-label="Share link: Trace ID in Logs"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json">{ "timestamp": "2026-01-24T10:15:30.123Z", "level": "ERROR", "message": "Query execution failed", "trace_id": "abc123def456ghi789", "span_id": "xyz789", "query_id": "q-12847", "error": "Index out of bounds" } </code></pre></div>Query logs in trace UI or trace from logs: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Find logs for specific trace jq 'select(.trace_id == "abc123def456ghi789")' /var/log/geode/geode.log # Find trace from log entry curl "http://jaeger:16686/api/traces/abc123def456ghi789" </code></pre></div> <h3 id="observability-driven-debugging" class="position-relative d-flex align-items-center group"> Observability-Driven Debugging <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="observability-driven-debugging" aria-haspopup="dialog" aria-label="Share link: Observability-Driven Debugging"> Share link </button> </h3> <h4 id="performance-regression-investigation" class="position-relative d-flex align-items-center group"> Performance Regression Investigation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="performance-regression-investigation" aria-haspopup="dialog" aria-label="Share link: Performance Regression Investigation"> Share link </button> </h4><ol> <li>Detect anomaly in metrics:</li> </ol> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"># p95 latency spiked histogram_quantile(0.95, rate(geode_query_duration_seconds_bucket[5m])) > 0.5 </code></pre></div><ol start="2"> <li>Identify affected queries from logs:</li> </ol> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">jq 'select(.duration_ms > 500) | {query_id, query_text, duration_ms}' \ /var/log/geode/geode.log </code></pre></div><ol start="3"> <li> Analyze traces for bottlenecks: Look for spans with unexpectedly high duration in trace viewer. </li> <li> Profile slow query: </li> </ol> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-gql" data-lang="gql">PROFILE MATCH (u:User)-[:FOLLOWS]->(other) RETURN u.name, count(other); </code></pre></div> <h4 id="error-rate-spike-investigation" class="position-relative d-flex align-items-center group"> Error Rate Spike Investigation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="error-rate-spike-investigation" aria-haspopup="dialog" aria-label="Share link: Error Rate Spike Investigation"> Share link </button> </h4><ol> <li>Detect in metrics:</li> </ol> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql">rate(geode_queries_total{status="error"}[5m]) > 10 </code></pre></div><ol start="2"> <li>Analyze error logs:</li> </ol> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">jq 'select(.level == "ERROR") | {timestamp, message, error, query_text}' \ /var/log/geode/geode.log | tail -100 </code></pre></div><ol start="3"> <li>Group errors by type:</li> </ol> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">jq -r 'select(.level == "ERROR") | .error' /var/log/geode/geode.log | \ sort | uniq -c | sort -rn </code></pre></div><ol start="4"> <li>Examine failed traces: Filter traces by error status to see failure patterns.</li> </ol> <h3 id="observability-best-practices" class="position-relative d-flex align-items-center group"> Observability Best Practices <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="observability-best-practices" aria-haspopup="dialog" aria-label="Share link: Observability Best Practices"> Share link </button> </h3>High Cardinality Awareness: Avoid unbounded label/field values (user IDs, session IDs) that explode storage requirements. Consistent Naming: Use consistent naming conventions across metrics, logs, and traces for easy correlation. Contextual Enrichment: Include relevant context (user, query type, client) in all observability signals. Sampling Strategy: Sample traces appropriately to balance coverage and overhead (1-10% for high-volume systems). Retention Policies: Define appropriate retention periods for each pillar based on use case (metrics: 30d, logs: 7d, traces: 3d). Alert on Symptoms, Not Causes: Alert on user-impacting symptoms (high latency, errors) rather than internal metrics (CPU, memory). Documentation: Maintain runbooks linking alerts to investigation procedures using observability tools. Cost Management: Monitor observability pipeline costs and optimize sampling, retention, and cardinality. <h3 id="related-topics" class="position-relative d-flex align-items-center group"> Related Topics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="related-topics" aria-haspopup="dialog" aria-label="Share link: Related Topics"> Share link </button> </h3><ul> <li><a href="/tags/monitoring/" >System Monitoring</a> - Monitoring strategies</li> <li><a href="/tags/metrics/" >Performance Metrics</a> - Metrics collection</li> <li><a href="/tags/logging/" >Application Logging</a> - Logging best practices</li> <li><a href="/tags/tracing/" >Distributed Tracing</a> - Tracing implementation</li> <li><a href="/tags/prometheus/" >Prometheus Integration</a> - Metrics collection</li> <li><a href="/tags/performance/" >Performance Tuning</a> - Optimization techniques</li> <li><a href="/tags/troubleshooting/" >Troubleshooting</a> - Debugging guide</li> </ul> <h3 id="further-reading" class="position-relative d-flex align-items-center group"> Further Reading <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="further-reading" aria-haspopup="dialog" aria-label="Share link: Further Reading"> Share link </button> </h3><ul> <li>Observability Engineering Handbook</li> <li>OpenTelemetry Integration Guide</li> <li>Distributed Tracing Patterns</li> <li>Log Analysis Best Practices</li> <li>Production Debugging Strategies</li> </ul>

Popular

Related Articles

Monitoring

Monitoring and Telemetry

Statistics and Metrics Reference

Advanced Telemetry and Monitoring Guide

Monitoring Guide