Distributed Tracing

Distributed tracing provides end-to-end visibility into request flows through complex systems, enabling you to understand how queries propagate through Geode’s execution pipeline and identify performance bottlenecks with precision. By capturing detailed timing information for each operation, tracing reveals exactly where time is spent during query execution. Geode implements distributed tracing using OpenTelemetry, the industry-standard observability framework. Traces capture spans for query parsing, optimization, execution, index lookups, relationship traversals, and result serialization, providing comprehensive visibility into database operations. This guide covers trace architecture, OpenTelemetry integration, instrumentation patterns, sampling strategies, and trace-based performance optimization. <h3 id="distributed-tracing-concepts" class="position-relative d-flex align-items-center group"> Distributed Tracing Concepts <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="distributed-tracing-concepts" aria-haspopup="dialog" aria-label="Share link: Distributed Tracing Concepts"> Share link </button> </h3><div id="headingShareModal" class="heading-share-modal" role="dialog" aria-modal="true" aria-labelledby="headingShareTitle" hidden> <div class="hsm-dialog" role="document"> <div class="hsm-header"> <h2 id="headingShareTitle" class="h6 mb-0 fw-bold">Share this section</h2> <button type="button" class="hsm-close" aria-label="Close"> </button> </div> <div class="hsm-body"> <label for="headingShareInput" class="form-label small text-muted mb-1 text-uppercase fw-bold" style="font-size: 0.7rem; letter-spacing: 0.5px;">Permalink</label> <div class="input-group mb-4 hsm-url-group"> <input id="headingShareInput" type="text" class="form-control font-monospace" readonly aria-readonly="true" style="font-size: 0.85rem;" /> <button class="btn btn-primary hsm-copy" type="button" aria-label="Copy" title="Copy"> </button> </div> <div class="small fw-bold mb-2 text-muted text-uppercase" style="font-size: 0.7rem; letter-spacing: 0.5px;">Share via</div> <div class="hsm-share-grid"> <a id="share-twitter" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Twitter </a> <a id="share-linkedin" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> LinkedIn </a> <a id="share-facebook" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Facebook </a> </div> </div> </div> </div> <style> .heading-share-modal { position: fixed; inset: 0; display: flex; justify-content: center; align-items: center; background: rgba(0, 0, 0, 0.6); z-index: 1050; padding: 1rem; backdrop-filter: blur(4px); -webkit-backdrop-filter: blur(4px); } .heading-share-modal[hidden] { display: none !important; } .hsm-dialog { max-width: 420px; width: 100%; background: var(--bs-body-bg, #fff); color: var(--bs-body-color, #212529); border: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); border-radius: 1rem; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); overflow: hidden; animation: hsm-fade-in 0.2s ease-out; } @keyframes hsm-fade-in { from { opacity: 0; transform: scale(0.95); } to { opacity: 1; transform: scale(1); } } [data-bs-theme="dark"] .hsm-dialog { background: #1e293b; border-color: rgba(255,255,255,0.1); color: #f8f9fa; } .hsm-header { display: flex; justify-content: space-between; align-items: center; padding: 1rem 1.5rem; border-bottom: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); background: rgba(0,0,0,0.02); } [data-bs-theme="dark"] .hsm-header { background: rgba(255,255,255,0.02); border-color: rgba(255,255,255,0.1); } .hsm-close { background: transparent; border: none; color: inherit; opacity: 0.5; padding: 0.25rem 0.5rem; border-radius: 0.25rem; font-size: 1.2rem; line-height: 1; transition: opacity 0.2s; } .hsm-close:hover { opacity: 1; } .hsm-body { padding: 1.5rem; } .hsm-url-group { display: flex !important; align-items: stretch; } .hsm-url-group .form-control { flex: 1; min-width: 0; margin: 0; background: var(--bs-secondary-bg, #f8f9fa); border-color: var(--bs-border-color, #dee2e6); border-top-right-radius: 0; border-bottom-right-radius: 0; height: 42px; } .hsm-url-group .btn { flex: 0 0 auto; margin: 0; margin-left: -1px; border-top-left-radius: 0; border-bottom-left-radius: 0; height: 42px; display: flex; align-items: center; justify-content: center; padding: 0 1.25rem; z-index: 2; } [data-bs-theme="dark"] .hsm-url-group .form-control { background: #0f172a; border-color: #334155; color: #e2e8f0; } .hsm-share-grid { display: flex; flex-direction: column; gap: 0.5rem; } .hsm-share-grid .btn { display: flex; align-items: center; justify-content: center; font-size: 0.9rem; padding: 0.6rem; border-color: var(--bs-border-color); width: 100%; } [data-bs-theme="dark"] .hsm-share-grid .btn { color: #e2e8f0; border-color: #475569; } [data-bs-theme="dark"] .hsm-share-grid .btn:hover { background: #334155; border-color: #cbd5e1; } </style> <script> (function(){ const modal = document.getElementById('headingShareModal'); if(!modal) return; const input = modal.querySelector('#headingShareInput'); const copyBtn = modal.querySelector('.hsm-copy'); const twitter = modal.querySelector('#share-twitter'); const linkedin = modal.querySelector('#share-linkedin'); const facebook = modal.querySelector('#share-facebook'); const closeBtn = modal.querySelector('.hsm-close'); let lastFocus=null; let trapBound=false; function buildUrl(id){ return window.location.origin + window.location.pathname + '#' + id; } function isOpen(){ return !modal.hasAttribute('hidden'); } function hydrate(id){ const url=buildUrl(id); input.value=url; const enc=encodeURIComponent(url); const text=encodeURIComponent(document.title); if(twitter) twitter.href=`https://twitter.com/intent/tweet?url=${enc}&text=${text}`; if(linkedin) linkedin.href=`https://www.linkedin.com/sharing/share-offsite/?url=${enc}`; if(facebook) facebook.href=`https://www.facebook.com/sharer/sharer.php?u=${enc}`; } function openModal(id){ lastFocus=document.activeElement; hydrate(id); if(!isOpen()){ modal.removeAttribute('hidden'); } requestAnimationFrame(()=>{ input.focus(); }); trapFocus(); } function closeModal(){ if(!isOpen()) return; modal.setAttribute('hidden',''); if(lastFocus && typeof lastFocus.focus==='function') lastFocus.focus(); } function copyCurrent(){ try{ navigator.clipboard.writeText(input.value).then(()=>feedback(true),()=>fallback()); } catch(e){ fallback(); } } function fallback(){ input.select(); try{ document.execCommand('copy'); feedback(true);}catch(e){ feedback(false);} } function feedback(ok){ if(!copyBtn) return; const icon=copyBtn.querySelector('i'); if(!icon) return; const prev=copyBtn.getAttribute('data-prev')||icon.className; if(!copyBtn.getAttribute('data-prev')) copyBtn.setAttribute('data-prev',prev); icon.className= ok ? 'fa-duotone fa-clipboard-check':'fa-duotone fa-circle-exclamation'; setTimeout(()=>{ icon.className=prev; },1800); } function handleShareClick(e){ e.preventDefault(); const btn=e.currentTarget; const id=btn.getAttribute('data-share-target'); if(id) openModal(id); } function bindShareButtons(){ document.querySelectorAll('.h-share').forEach(btn=>{ if(!btn.dataset.hShareBound){ btn.addEventListener('click', handleShareClick); btn.dataset.hShareBound='1'; } }); } bindShareButtons(); if(document.readyState==='loading'){ document.addEventListener('DOMContentLoaded', bindShareButtons); } else { requestAnimationFrame(bindShareButtons); } document.addEventListener('click', function(e){ const shareBtn=e.target.closest && e.target.closest('.h-share'); if(shareBtn && !shareBtn.dataset.hShareBound){ handleShareClick.call(shareBtn, e); } }, true); document.addEventListener('click', e=>{ if(e.target===modal) closeModal(); if(e.target.closest && e.target.closest('.hsm-close')){ e.preventDefault(); closeModal(); } if(copyBtn && (e.target===copyBtn || (e.target.closest && e.target.closest('.hsm-copy')))) { e.preventDefault(); copyCurrent(); } }); document.addEventListener('keydown', e=>{ if(e.key==='Escape' && isOpen()) closeModal(); }); function trapFocus(){ if(trapBound) return; trapBound=true; modal.addEventListener('keydown', f=>{ if(f.key==='Tab' && isOpen()){ const focusable=[...modal.querySelectorAll('a[href],button,input,textarea,select,[tabindex]:not([tabindex="-1"])')].filter(el=>!el.hasAttribute('disabled')); if(!focusable.length) return; const first=focusable[0]; const last=focusable[focusable.length-1]; if(f.shiftKey && document.activeElement===first){ f.preventDefault(); last.focus(); } else if(!f.shiftKey && document.activeElement===last){ f.preventDefault(); first.focus(); } } }); } if(closeBtn) closeBtn.addEventListener('click', e=>{ e.preventDefault(); closeModal(); }); })(); </script> <h4 id="traces-and-spans" class="position-relative d-flex align-items-center group"> Traces and Spans <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="traces-and-spans" aria-haspopup="dialog" aria-label="Share link: Traces and Spans"> Share link </button> </h4>A trace represents a complete request flow through the system, composed of multiple spans: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">Trace ID: abc123-def456-ghi789 (total: 250ms) │ ├─ Span: http_request (250ms) │ │ │ ├─ Span: authenticate_user (15ms) │ │ │ ├─ Span: execute_gql_query (220ms) │ │ │ │ │ ├─ Span: parse_query (5ms) │ │ │ │ │ ├─ Span: optimize_plan (10ms) │ │ │ │ │ ├─ Span: execute_plan (200ms) │ │ │ │ │ │ │ ├─ Span: index_lookup (30ms) │ │ │ │ │ │ │ ├─ Span: expand_relationships (150ms) │ │ │ │ │ │ │ └─ Span: aggregate_results (20ms) │ │ │ │ │ └─ Span: serialize_response (5ms) │ │ │ └─ Span: cache_update (15ms) </code></pre></div>Each span captures: <ul> <li>Start time and duration</li> <li>Operation name (e.g., “execute_gql_query”)</li> <li>Attributes (key-value metadata)</li> <li>Events (timestamped log entries within span)</li> <li>Status (OK, ERROR)</li> <li>Parent-child relationships</li> </ul> <h4 id="trace-context-propagation" class="position-relative d-flex align-items-center group"> Trace Context Propagation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="trace-context-propagation" aria-haspopup="dialog" aria-label="Share link: Trace Context Propagation"> Share link </button> </h4>Trace context flows across service boundaries via HTTP headers or metadata: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">traceparent: 00-abc123def456ghi789-xyz789012345-01 tracestate: geode=query_id:q-12847,user:analyst </code></pre></div> <h3 id="opentelemetry-integration" class="position-relative d-flex align-items-center group"> OpenTelemetry Integration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="opentelemetry-integration" aria-haspopup="dialog" aria-label="Share link: OpenTelemetry Integration"> Share link </button> </h3> <h4 id="configuration" class="position-relative d-flex align-items-center group"> Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="configuration" aria-haspopup="dialog" aria-label="Share link: Configuration"> Share link </button> </h4>Enable tracing in Geode configuration: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"># geode.toml [tracing] # Enable distributed tracing enabled = true # Exporter type: otlp, jaeger, zipkin exporter = "otlp" # OTLP endpoint (gRPC or HTTP) endpoint = "http://localhost:4317" # Service name in traces service_name = "geode" # Environment label environment = "production" # Sample rate (0.0 to 1.0) sample_rate = 0.1 # Trace specific operations trace_queries = true trace_transactions = true trace_index_operations = true trace_storage_operations = false # High volume trace_network_operations = false # Very high volume </code></pre></div> <h4 id="otlp-exporter" class="position-relative d-flex align-items-center group"> OTLP Exporter <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="otlp-exporter" aria-haspopup="dialog" aria-label="Share link: OTLP Exporter"> Share link </button> </h4>Send traces to OpenTelemetry Collector: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml">[tracing] exporter = "otlp" endpoint = "http://otel-collector:4317" # Optional authentication [tracing.otlp] headers = { "Authorization" = "Bearer ${OTEL_TOKEN}" } compression = "gzip" timeout_seconds = 10 </code></pre></div> <h4 id="jaeger-exporter" class="position-relative d-flex align-items-center group"> Jaeger Exporter <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="jaeger-exporter" aria-haspopup="dialog" aria-label="Share link: Jaeger Exporter"> Share link </button> </h4>Send traces directly to Jaeger: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml">[tracing] exporter = "jaeger" endpoint = "http://jaeger:14250" [tracing.jaeger] agent_endpoint = "jaeger:6831" max_packet_size = 65000 </code></pre></div> <h4 id="zipkin-exporter" class="position-relative d-flex align-items-center group"> Zipkin Exporter <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="zipkin-exporter" aria-haspopup="dialog" aria-label="Share link: Zipkin Exporter"> Share link </button> </h4>Send traces to Zipkin: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml">[tracing] exporter = "zipkin" endpoint = "http://zipkin:9411/api/v2/spans" </code></pre></div> <h3 id="automatic-instrumentation" class="position-relative d-flex align-items-center group"> Automatic Instrumentation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="automatic-instrumentation" aria-haspopup="dialog" aria-label="Share link: Automatic Instrumentation"> Share link </button> </h3>Geode automatically instruments key operations: <h4 id="query-execution-spans" class="position-relative d-flex align-items-center group"> Query Execution Spans <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="query-execution-spans" aria-haspopup="dialog" aria-label="Share link: Query Execution Spans"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">Span: execute_gql_query trace_id: abc123def456ghi789 span_id: xyz789012345 start_time: 2026-01-24T10:15:30.000Z duration: 145ms status: OK Attributes: db.system: geode db.operation: query db.statement: "MATCH (u:User) WHERE u.age > 25 RETURN u" db.query_id: q-12847 db.user: [email protected] db.client_type: python db.rows_returned: 1250 db.cache_hit: false Events: - timestamp: +5ms, name: "parsing_completed" - timestamp: +15ms, name: "optimization_completed" - timestamp: +140ms, name: "execution_completed" </code></pre></div>Child spans for query pipeline stages: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">├─ Span: parse_query (5ms) │ Attributes: │ query.length: 42 │ query.tokens: 8 ├─ Span: optimize_plan (10ms) │ Attributes: │ plan.type: indexed_lookup │ plan.estimated_cost: 125.4 │ plan.index_used: User.age ├─ Span: execute_plan (125ms) │ Attributes: │ execution.rows_scanned: 5234 │ execution.rows_filtered: 3984 │ execution.rows_returned: 1250 └─ Span: serialize_response (5ms) Attributes: serialization.format: json serialization.bytes: 125840 </code></pre></div> <h4 id="transaction-spans" class="position-relative d-flex align-items-center group"> Transaction Spans <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="transaction-spans" aria-haspopup="dialog" aria-label="Share link: Transaction Spans"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">Span: transaction trace_id: def456ghi789abc123 span_id: abc123xyz789 start_time: 2026-01-24T10:16:00.000Z duration: 2340ms status: OK Attributes: db.system: geode db.operation: transaction db.transaction_id: tx-456 db.isolation_level: SERIALIZABLE db.queries_count: 5 db.rows_modified: 125 Child Spans: ├─ begin (2ms) ├─ execute_query_1 (145ms) ├─ execute_query_2 (234ms) ├─ execute_query_3 (87ms) ├─ execute_query_4 (156ms) ├─ execute_query_5 (98ms) └─ commit (15ms) Attributes: commit.wal_position: 00000001000000CD commit.fsync_duration_ms: 8 </code></pre></div> <h4 id="index-operation-spans" class="position-relative d-flex align-items-center group"> Index Operation Spans <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="index-operation-spans" aria-haspopup="dialog" aria-label="Share link: Index Operation Spans"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">Span: index_build trace_id: ghi789abc123def456 span_id: def456abc123 start_time: 2026-01-24T10:17:00.000Z duration: 45000ms status: OK Attributes: db.system: geode db.operation: index_build db.index_name: User.email db.index_type: btree db.table: User db.column: email db.rows_indexed: 1000000 db.index_size_bytes: 67108864 Events: - timestamp: +100ms, name: "scan_started" - timestamp: +30000ms, name: "sort_completed" - timestamp: +44000ms, name: "index_written" - timestamp: +45000ms, name: "index_activated" </code></pre></div> <h3 id="custom-instrumentation" class="position-relative d-flex align-items-center group"> Custom Instrumentation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="custom-instrumentation" aria-haspopup="dialog" aria-label="Share link: Custom Instrumentation"> Share link </button> </h3>Add application-specific spans for business logic: <h4 id="python-client" class="position-relative d-flex align-items-center group"> Python Client <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="python-client" aria-haspopup="dialog" aria-label="Share link: Python Client"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">from opentelemetry import trace from opentelemetry.trace import Status, StatusCode import geode_client # Get tracer tracer = trace.get_tracer(__name__) async def recommend_products(user_id): # Create parent span for entire recommendation flow with tracer.start_as_current_span("recommend_products") as span: span.set_attribute("user_id", user_id) span.set_attribute("algorithm", "collaborative_filtering") try: # Fetch user preferences (creates child span automatically) with tracer.start_as_current_span("fetch_user_preferences"): client = geode_client.open_database("localhost:3141") preferences, _ = await client.query( "MATCH (u:User {id: $id})-[:LIKES]->(p:Product) RETURN p", {"id": user_id} ) span.set_attribute("preferences_count", len(preferences)) # Find similar users with tracer.start_as_current_span("find_similar_users") as sim_span: similar_users, _ = await client.query(""" MATCH (u1:User {id: $id})-[:LIKES]->(p:Product)<-[:LIKES]-(u2:User) WHERE u1 <> u2 RETURN u2, count(p) as common_likes ORDER BY common_likes DESC LIMIT 10 """, {"id": user_id}) sim_span.set_attribute("similar_users_found", len(similar_users)) # Generate recommendations with tracer.start_as_current_span("generate_recommendations") as rec_span: recommendations = await compute_recommendations( preferences, similar_users ) rec_span.set_attribute("recommendations_count", len(recommendations)) # Add event to span rec_span.add_event( "recommendations_generated", attributes={"algorithm_version": "2.1"} ) # Mark span as successful span.set_status(Status(StatusCode.OK)) return recommendations except Exception as e: # Record error in span span.record_exception(e) span.set_status(Status(StatusCode.ERROR, str(e))) raise </code></pre></div> <h4 id="go-client" class="position-relative d-flex align-items-center group"> Go Client <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="go-client" aria-haspopup="dialog" aria-label="Share link: Go Client"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go">package main import ( "context" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/trace" "geodedb.com/geode" ) var tracer = otel.Tracer("myapp") func recommendProducts(ctx context.Context, userID int64) ([]Product, error) { ctx, span := tracer.Start(ctx, "recommend_products") defer span.End() span.SetAttributes( attribute.Int64("user_id", userID), attribute.String("algorithm", "collaborative_filtering"), ) // Fetch preferences ctx, prefSpan := tracer.Start(ctx, "fetch_user_preferences") preferences, err := db.Query(ctx, ` MATCH (u:User {id: $id})-[:LIKES]->(p:Product) RETURN p `, geode.Params{"id": userID}) if err != nil { span.RecordError(err) span.SetStatus(codes.Error, err.Error()) return nil, err } prefSpan.SetAttributes(attribute.Int("preferences_count", len(preferences))) prefSpan.End() // Generate recommendations ctx, recSpan := tracer.Start(ctx, "generate_recommendations") recommendations := computeRecommendations(preferences) recSpan.SetAttributes(attribute.Int("recommendations_count", len(recommendations))) recSpan.End() span.SetStatus(codes.Ok, "Success") return recommendations, nil } </code></pre></div> <h4 id="rust-client" class="position-relative d-flex align-items-center group"> Rust Client <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="rust-client" aria-haspopup="dialog" aria-label="Share link: Rust Client"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust">use opentelemetry::trace::{Tracer, Span, Status}; use opentelemetry::KeyValue; async fn recommend_products(user_id: i64) -> Result<Vec<Product>> { let tracer = opentelemetry::global::tracer("myapp"); let mut span = tracer.start("recommend_products"); span.set_attribute(KeyValue::new("user_id", user_id)); span.set_attribute(KeyValue::new("algorithm", "collaborative_filtering")); let result = async { // Fetch preferences let mut pref_span = tracer.start("fetch_user_preferences"); let preferences = client.execute( "MATCH (u:User {id: $id})-[:LIKES]->(p:Product) RETURN p", params!("id" => user_id) ).await?; pref_span.set_attribute(KeyValue::new("preferences_count", preferences.len() as i64)); pref_span.end(); // Generate recommendations let mut rec_span = tracer.start("generate_recommendations"); let recommendations = compute_recommendations(&preferences); rec_span.set_attribute(KeyValue::new("recommendations_count", recommendations.len() as i64)); rec_span.end(); Ok(recommendations) }.await; match result { Ok(recs) => { span.set_status(Status::Ok); span.end(); Ok(recs) } Err(e) => { span.record_error(&e); span.set_status(Status::error(e.to_string())); span.end(); Err(e) } } } </code></pre></div> <h3 id="sampling-strategies" class="position-relative d-flex align-items-center group"> Sampling Strategies <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="sampling-strategies" aria-haspopup="dialog" aria-label="Share link: Sampling Strategies"> Share link </button> </h3>Control trace volume with intelligent sampling: <h4 id="probabilistic-sampling" class="position-relative d-flex align-items-center group"> Probabilistic Sampling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="probabilistic-sampling" aria-haspopup="dialog" aria-label="Share link: Probabilistic Sampling"> Share link </button> </h4>Sample a fixed percentage of traces: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml">[tracing.sampling] strategy = "probabilistic" rate = 0.1 # Sample 10% of all traces </code></pre></div> <h4 id="tail-based-sampling" class="position-relative d-flex align-items-center group"> Tail-Based Sampling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="tail-based-sampling" aria-haspopup="dialog" aria-label="Share link: Tail-Based Sampling"> Share link </button> </h4>Sample based on trace characteristics after completion: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml">[tracing.sampling] strategy = "tail_based" # Always sample errors sample_on_error = true # Always sample slow requests slow_threshold_ms = 1000 sample_slow = true # Sample transactions sample_transactions = true # Default rate for normal requests default_rate = 0.05 # 5% </code></pre></div> <h4 id="adaptive-sampling" class="position-relative d-flex align-items-center group"> Adaptive Sampling <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="adaptive-sampling" aria-haspopup="dialog" aria-label="Share link: Adaptive Sampling"> Share link </button> </h4>Dynamically adjust sample rate based on load: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml">[tracing.sampling] strategy = "adaptive" # Target traces per second target_tps = 100 # Adjust sample rate every N seconds adjust_interval_seconds = 60 # Minimum sample rate min_rate = 0.01 # 1% # Maximum sample rate max_rate = 1.0 # 100% </code></pre></div> <h4 id="custom-sampling-rules" class="position-relative d-flex align-items-center group"> Custom Sampling Rules <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="custom-sampling-rules" aria-haspopup="dialog" aria-label="Share link: Custom Sampling Rules"> Share link </button> </h4>Define rules for specific scenarios: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml">[tracing.sampling.rules] # Always sample specific users [[tracing.sampling.rules.always]] user = "[email protected]" # Always sample specific operations [[tracing.sampling.rules.always]] operation = "CREATE_GRAPH" # Never sample health checks [[tracing.sampling.rules.never]] query_text = "MATCH (n) RETURN count(n)" </code></pre></div> <h3 id="trace-analysis" class="position-relative d-flex align-items-center group"> Trace Analysis <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="trace-analysis" aria-haspopup="dialog" aria-label="Share link: Trace Analysis"> Share link </button> </h3> <h4 id="jaeger-ui" class="position-relative d-flex align-items-center group"> Jaeger UI <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="jaeger-ui" aria-haspopup="dialog" aria-label="Share link: Jaeger UI"> Share link </button> </h4>Query and visualize traces in Jaeger: Find slow queries: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">service: geode operation: execute_gql_query min_duration: 1s </code></pre></div>Find errors: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">service: geode tags: error=true </code></pre></div>Find by user: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">service: geode tags: [email protected] </code></pre></div> <h4 id="programmatic-analysis" class="position-relative d-flex align-items-center group"> Programmatic Analysis <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="programmatic-analysis" aria-haspopup="dialog" aria-label="Share link: Programmatic Analysis"> Share link </button> </h4>Query trace data via Jaeger API: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">import requests # Find traces response = requests.get( "http://jaeger:16686/api/traces", params={ "service": "geode", "operation": "execute_gql_query", "minDuration": "1s", "limit": 100 } ) traces = response.json() # Analyze trace durations for trace in traces["data"]: root_span = trace["spans"][0] print(f"Trace {trace['traceID']}: {root_span['duration']/1000}ms") # Find slowest child span slowest = max(trace["spans"], key=lambda s: s["duration"]) print(f" Slowest operation: {slowest['operationName']} ({slowest['duration']/1000}ms)") </code></pre></div> <h3 id="performance-optimization-with-traces" class="position-relative d-flex align-items-center group"> Performance Optimization with Traces <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="performance-optimization-with-traces" aria-haspopup="dialog" aria-label="Share link: Performance Optimization with Traces"> Share link </button> </h3> <h4 id="identify-bottlenecks" class="position-relative d-flex align-items-center group"> Identify Bottlenecks <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="identify-bottlenecks" aria-haspopup="dialog" aria-label="Share link: Identify Bottlenecks"> Share link </button> </h4>Analyze span durations to find optimization opportunities: <ol> <li>Sort spans by duration to find slowest operations</li> <li>Compare similar traces to find anomalies</li> <li>Examine span attributes for context (index usage, row counts)</li> <li>Look for sequential operations that could be parallelized</li> </ol> <h4 id="example-analysis" class="position-relative d-flex align-items-center group"> Example Analysis <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="example-analysis" aria-haspopup="dialog" aria-label="Share link: Example Analysis"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback">Trace: User recommendation flow (total: 2500ms) ├─ fetch_user_preferences (50ms) ✓ Fast ├─ find_similar_users (2200ms) ⚠️ BOTTLENECK │ └─ expand_relationships (2150ms) ⚠️ Missing index? └─ generate_recommendations (250ms) ✓ Acceptable Action: Add index on LIKES relationship Expected improvement: 2200ms → 150ms </code></pre></div> <h3 id="best-practices" class="position-relative d-flex align-items-center group"> Best Practices <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="best-practices" aria-haspopup="dialog" aria-label="Share link: Best Practices"> Share link </button> </h3>Strategic Sampling: Use tail-based sampling to capture all errors and slow requests while sampling normal traffic. Meaningful Span Names: Use descriptive operation names (e.g., “recommend_products”, not “function_A”). Rich Attributes: Include relevant context in span attributes for filtering and analysis. Avoid High Cardinality: Don’t use unbounded values (user IDs, query text) as span names. Propagate Context: Ensure trace context flows through all service boundaries. Monitor Overhead: Tracing should consume <2% of CPU; adjust sampling if higher. Correlate with Logs: Include trace IDs in logs for cross-pillar correlation. Set Retention Policies: Retain traces for 3-7 days (longer than logs, shorter than metrics). <h3 id="related-topics" class="position-relative d-flex align-items-center group"> Related Topics <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="related-topics" aria-haspopup="dialog" aria-label="Share link: Related Topics"> Share link </button> </h3><ul> <li><a href="/tags/observability/" >System Observability</a> - Observability pillars</li> <li><a href="/tags/logging/" >Application Logging</a> - Structured logging</li> <li><a href="/tags/metrics/" >Performance Metrics</a> - Metrics collection</li> <li><a href="/tags/monitoring/" >System Monitoring</a> - Monitoring strategies</li> <li><a href="/tags/performance/" >Performance Tuning</a> - Optimization techniques</li> <li><a href="/tags/profiling/" >Profiling</a> - Query profiling</li> </ul> <h3 id="further-reading" class="position-relative d-flex align-items-center group"> Further Reading <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="further-reading" aria-haspopup="dialog" aria-label="Share link: Further Reading"> Share link </button> </h3><ul> <li>OpenTelemetry Documentation</li> <li>Distributed Tracing Best Practices</li> <li>Jaeger Deployment Guide</li> <li>Trace-Based Debugging Patterns</li> <li>Performance Optimization with Traces</li> </ul>

Popular

Related Articles