Database Recovery | Geode Database

<h2 id="database-recovery" class="position-relative d-flex align-items-center group"> Database Recovery <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="database-recovery" aria-haspopup="dialog" aria-label="Share link: Database Recovery"> Share link </button> </h2><div id="headingShareModal" class="heading-share-modal" role="dialog" aria-modal="true" aria-labelledby="headingShareTitle" hidden> <div class="hsm-dialog" role="document"> <div class="hsm-header"> <h2 id="headingShareTitle" class="h6 mb-0 fw-bold">Share this section</h2> <button type="button" class="hsm-close" aria-label="Close"> </button> </div> <div class="hsm-body"> <label for="headingShareInput" class="form-label small text-muted mb-1 text-uppercase fw-bold" style="font-size: 0.7rem; letter-spacing: 0.5px;">Permalink</label> <div class="input-group mb-4 hsm-url-group"> <input id="headingShareInput" type="text" class="form-control font-monospace" readonly aria-readonly="true" style="font-size: 0.85rem;" /> <button class="btn btn-primary hsm-copy" type="button" aria-label="Copy" title="Copy"> </button> </div> <div class="small fw-bold mb-2 text-muted text-uppercase" style="font-size: 0.7rem; letter-spacing: 0.5px;">Share via</div> <div class="hsm-share-grid"> <a id="share-twitter" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Twitter </a> <a id="share-linkedin" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> LinkedIn </a> <a id="share-facebook" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> Facebook </a> </div> </div> </div> </div> <style> .heading-share-modal { position: fixed; inset: 0; display: flex; justify-content: center; align-items: center; background: rgba(0, 0, 0, 0.6); z-index: 1050; padding: 1rem; backdrop-filter: blur(4px); -webkit-backdrop-filter: blur(4px); } .heading-share-modal[hidden] { display: none !important; } .hsm-dialog { max-width: 420px; width: 100%; background: var(--bs-body-bg, #fff); color: var(--bs-body-color, #212529); border: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); border-radius: 1rem; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); overflow: hidden; animation: hsm-fade-in 0.2s ease-out; } @keyframes hsm-fade-in { from { opacity: 0; transform: scale(0.95); } to { opacity: 1; transform: scale(1); } } [data-bs-theme="dark"] .hsm-dialog { background: #1e293b; border-color: rgba(255,255,255,0.1); color: #f8f9fa; } .hsm-header { display: flex; justify-content: space-between; align-items: center; padding: 1rem 1.5rem; border-bottom: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); background: rgba(0,0,0,0.02); } [data-bs-theme="dark"] .hsm-header { background: rgba(255,255,255,0.02); border-color: rgba(255,255,255,0.1); } .hsm-close { background: transparent; border: none; color: inherit; opacity: 0.5; padding: 0.25rem 0.5rem; border-radius: 0.25rem; font-size: 1.2rem; line-height: 1; transition: opacity 0.2s; } .hsm-close:hover { opacity: 1; } .hsm-body { padding: 1.5rem; } .hsm-url-group { display: flex !important; align-items: stretch; } .hsm-url-group .form-control { flex: 1; min-width: 0; margin: 0; background: var(--bs-secondary-bg, #f8f9fa); border-color: var(--bs-border-color, #dee2e6); border-top-right-radius: 0; border-bottom-right-radius: 0; height: 42px; } .hsm-url-group .btn { flex: 0 0 auto; margin: 0; margin-left: -1px; border-top-left-radius: 0; border-bottom-left-radius: 0; height: 42px; display: flex; align-items: center; justify-content: center; padding: 0 1.25rem; z-index: 2; } [data-bs-theme="dark"] .hsm-url-group .form-control { background: #0f172a; border-color: #334155; color: #e2e8f0; } .hsm-share-grid { display: flex; flex-direction: column; gap: 0.5rem; } .hsm-share-grid .btn { display: flex; align-items: center; justify-content: center; font-size: 0.9rem; padding: 0.6rem; border-color: var(--bs-border-color); width: 100%; } [data-bs-theme="dark"] .hsm-share-grid .btn { color: #e2e8f0; border-color: #475569; } [data-bs-theme="dark"] .hsm-share-grid .btn:hover { background: #334155; border-color: #cbd5e1; } </style> <script> (function(){ const modal = document.getElementById('headingShareModal'); if(!modal) return; const input = modal.querySelector('#headingShareInput'); const copyBtn = modal.querySelector('.hsm-copy'); const twitter = modal.querySelector('#share-twitter'); const linkedin = modal.querySelector('#share-linkedin'); const facebook = modal.querySelector('#share-facebook'); const closeBtn = modal.querySelector('.hsm-close'); let lastFocus=null; let trapBound=false; function buildUrl(id){ return window.location.origin + window.location.pathname + '#' + id; } function isOpen(){ return !modal.hasAttribute('hidden'); } function hydrate(id){ const url=buildUrl(id); input.value=url; const enc=encodeURIComponent(url); const text=encodeURIComponent(document.title); if(twitter) twitter.href=`https://twitter.com/intent/tweet?url=${enc}&text=${text}`; if(linkedin) linkedin.href=`https://www.linkedin.com/sharing/share-offsite/?url=${enc}`; if(facebook) facebook.href=`https://www.facebook.com/sharer/sharer.php?u=${enc}`; } function openModal(id){ lastFocus=document.activeElement; hydrate(id); if(!isOpen()){ modal.removeAttribute('hidden'); } requestAnimationFrame(()=>{ input.focus(); }); trapFocus(); } function closeModal(){ if(!isOpen()) return; modal.setAttribute('hidden',''); if(lastFocus && typeof lastFocus.focus==='function') lastFocus.focus(); } function copyCurrent(){ try{ navigator.clipboard.writeText(input.value).then(()=>feedback(true),()=>fallback()); } catch(e){ fallback(); } } function fallback(){ input.select(); try{ document.execCommand('copy'); feedback(true);}catch(e){ feedback(false);} } function feedback(ok){ if(!copyBtn) return; const icon=copyBtn.querySelector('i'); if(!icon) return; const prev=copyBtn.getAttribute('data-prev')||icon.className; if(!copyBtn.getAttribute('data-prev')) copyBtn.setAttribute('data-prev',prev); icon.className= ok ? 'fa-duotone fa-clipboard-check':'fa-duotone fa-circle-exclamation'; setTimeout(()=>{ icon.className=prev; },1800); } function handleShareClick(e){ e.preventDefault(); const btn=e.currentTarget; const id=btn.getAttribute('data-share-target'); if(id) openModal(id); } function bindShareButtons(){ document.querySelectorAll('.h-share').forEach(btn=>{ if(!btn.dataset.hShareBound){ btn.addEventListener('click', handleShareClick); btn.dataset.hShareBound='1'; } }); } bindShareButtons(); if(document.readyState==='loading'){ document.addEventListener('DOMContentLoaded', bindShareButtons); } else { requestAnimationFrame(bindShareButtons); } document.addEventListener('click', function(e){ const shareBtn=e.target.closest && e.target.closest('.h-share'); if(shareBtn && !shareBtn.dataset.hShareBound){ handleShareClick.call(shareBtn, e); } }, true); document.addEventListener('click', e=>{ if(e.target===modal) closeModal(); if(e.target.closest && e.target.closest('.hsm-close')){ e.preventDefault(); closeModal(); } if(copyBtn && (e.target===copyBtn || (e.target.closest && e.target.closest('.hsm-copy')))) { e.preventDefault(); copyCurrent(); } }); document.addEventListener('keydown', e=>{ if(e.key==='Escape' && isOpen()) closeModal(); }); function trapFocus(){ if(trapBound) return; trapBound=true; modal.addEventListener('keydown', f=>{ if(f.key==='Tab' && isOpen()){ const focusable=[...modal.querySelectorAll('a[href],button,input,textarea,select,[tabindex]:not([tabindex="-1"])')].filter(el=>!el.hasAttribute('disabled')); if(!focusable.length) return; const first=focusable[0]; const last=focusable[focusable.length-1]; if(f.shiftKey && document.activeElement===first){ f.preventDefault(); last.focus(); } else if(!f.shiftKey && document.activeElement===last){ f.preventDefault(); first.focus(); } } }); } if(closeBtn) closeBtn.addEventListener('click', e=>{ e.preventDefault(); closeModal(); }); })(); </script>This guide covers all aspects of database recovery for Geode, from automatic crash recovery to manual disaster recovery procedures. Understanding these concepts ensures minimal data loss and rapid service restoration. <h3 id="overview" class="position-relative d-flex align-items-center group"> Overview <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="overview" aria-haspopup="dialog" aria-label="Share link: Overview"> Share link </button> </h3>Geode provides multiple recovery mechanisms to handle various failure scenarios: <table> <thead> <tr> <th>Recovery Type</th> <th>Use Case</th> <th>RTO</th> <th>RPO</th> </tr> </thead> <tbody> <tr> <td>Automatic Crash Recovery</td> <td>Server restart after crash</td> <td>Seconds</td> <td>Zero</td> </tr> <tr> <td>WAL Replay</td> <td>Incomplete transactions</td> <td>Seconds</td> <td>Zero</td> </tr> <tr> <td>Point-in-Time Recovery</td> <td>Restore to specific timestamp</td> <td>Minutes</td> <td>Configurable</td> </tr> <tr> <td>Full Restore</td> <td>Complete database restoration</td> <td>Minutes</td> <td>Last backup</td> </tr> <tr> <td>Disaster Recovery</td> <td>Site failure, data corruption</td> <td>Minutes-Hours</td> <td>Last backup</td> </tr> </tbody> </table> Key Terms: <ul> <li>RTO (Recovery Time Objective): Maximum acceptable downtime</li> <li>RPO (Recovery Point Objective): Maximum acceptable data loss</li> </ul> <h3 id="automatic-crash-recovery" class="position-relative d-flex align-items-center group"> Automatic Crash Recovery <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="automatic-crash-recovery" aria-haspopup="dialog" aria-label="Share link: Automatic Crash Recovery"> Share link </button> </h3> <h4 id="how-it-works" class="position-relative d-flex align-items-center group"> How It Works <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="how-it-works" aria-haspopup="dialog" aria-label="Share link: How It Works"> Share link </button> </h4>Geode uses Write-Ahead Logging (WAL) to ensure durability. When the server restarts after a crash: <ol> <li>WAL Analysis: Scans WAL to identify incomplete transactions</li> <li>Redo Phase: Replays committed transactions not yet flushed to disk</li> <li>Undo Phase: Rolls back uncommitted transactions</li> <li>Checkpoint: Creates recovery checkpoint</li> </ol> <h4 id="crash-recovery-process" class="position-relative d-flex align-items-center group"> Crash Recovery Process <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="crash-recovery-process" aria-haspopup="dialog" aria-label="Share link: Crash Recovery Process"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-plaintext" data-lang="plaintext">Server Crash │ ▼ Server Restart │ ▼ ┌─────────────────────────────────────┐ │ 1. Load last checkpoint │ │ 2. Scan WAL from checkpoint LSN │ │ 3. Build transaction table │ │ 4. Redo committed transactions │ │ 5. Undo uncommitted transactions │ │ 6. Create new checkpoint │ └─────────────────────────────────────┘ │ ▼ Server Ready </code></pre></div> <h4 id="verifying-crash-recovery" class="position-relative d-flex align-items-center group"> Verifying Crash Recovery <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="verifying-crash-recovery" aria-haspopup="dialog" aria-label="Share link: Verifying Crash Recovery"> Share link </button> </h4>After server restart, verify recovery completed: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Check server logs for recovery messages geode logs --tail 100 | grep -i recovery # Expected output: # [INFO] Starting crash recovery from checkpoint LSN: 12345678 # [INFO] WAL replay: 156 transactions to redo # [INFO] WAL replay: 3 transactions to undo # [INFO] Crash recovery completed in 1.2s # [INFO] Server ready on port 3141 # Verify database integrity geode verify --data-dir /var/lib/geode/data # Run health check query geode query "RETURN 1 AS health_check" </code></pre></div> <h4 id="recovery-configuration" class="position-relative d-flex align-items-center group"> Recovery Configuration <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="recovery-configuration" aria-haspopup="dialog" aria-label="Share link: Recovery Configuration"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># geode.yaml recovery: # WAL configuration wal: enabled: true sync_mode: fsync # Options: none, fsync, fdatasync buffer_size: 16MB segment_size: 64MB # Checkpoint configuration checkpoint: interval: 5m # Checkpoint frequency min_wal_size: 256MB # Minimum WAL before checkpoint # Recovery limits max_recovery_time: 10m # Abort if recovery exceeds parallel_redo: true # Parallel transaction replay redo_threads: 4 # Number of redo threads </code></pre></div> <h3 id="point-in-time-recovery-pitr" class="position-relative d-flex align-items-center group"> Point-in-Time Recovery (PITR) <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="point-in-time-recovery-pitr" aria-haspopup="dialog" aria-label="Share link: Point-in-Time Recovery (PITR)"> Share link </button> </h3>PITR allows restoring the database to any point in time within the retention period. <h4 id="prerequisites" class="position-relative d-flex align-items-center group"> Prerequisites <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="prerequisites" aria-haspopup="dialog" aria-label="Share link: Prerequisites"> Share link </button> </h4><ul> <li>WAL archiving enabled</li> <li>Periodic full backups</li> <li>Archived WAL segments</li> </ul> <h4 id="enable-wal-archiving" class="position-relative d-flex align-items-center group"> Enable WAL Archiving <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="enable-wal-archiving" aria-haspopup="dialog" aria-label="Share link: Enable WAL Archiving"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># geode.yaml backup: wal_archiving: enabled: true archive_path: /var/lib/geode/wal_archive # Or use S3 s3: enabled: true bucket: geode-wal-archive prefix: production/wal retention_hours: 168 # 7 days </code></pre></div> <h4 id="pitr-restore-procedure" class="position-relative d-flex align-items-center group"> PITR Restore Procedure <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="pitr-restore-procedure" aria-haspopup="dialog" aria-label="Share link: PITR Restore Procedure"> Share link </button> </h4>Step 1: Stop the Server <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">sudo systemctl stop geode </code></pre></div>Step 2: Identify Target Time Determine the point in time to restore to: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># List available recovery points geode backup --dest s3://geode-backups/production --list-recovery-points # Output: # Recovery Points Available: # ───────────────────────────────────────────────────────── # Base Backup ID Timestamp Earliest PITR # 1738012345 2026-01-23 02:00:00 2026-01-23 02:00:00 # WAL Segments: 2026-01-23 02:00:00 to 2026-01-28 10:30:00 # ───────────────────────────────────────────────────────── </code></pre></div>Step 3: Restore Base Backup <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Create fresh data directory sudo mv /var/lib/geode/data /var/lib/geode/data.old # Restore base backup geode restore \ --source s3://geode-backups/production \ --backup-id 1738012345 \ --target /var/lib/geode/data </code></pre></div>Step 4: Apply WAL to Target Time <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">geode recover \ --data-dir /var/lib/geode/data \ --wal-archive s3://geode-wal-archive/production/wal \ --target-time "2026-01-28 10:30:00" \ --timezone UTC # Output: # Starting point-in-time recovery # Base backup: 1738012345 (2026-01-23 02:00:00) # Target time: 2026-01-28 10:30:00 UTC # Applying WAL segments: # - segment-0001.wal (2026-01-23 02:05:00 - 2026-01-24 02:00:00) # - segment-0002.wal (2026-01-24 02:00:00 - 2026-01-25 02:00:00) # - segment-0003.wal (2026-01-25 02:00:00 - 2026-01-26 02:00:00) # - segment-0004.wal (2026-01-26 02:00:00 - 2026-01-27 02:00:00) # - segment-0005.wal (2026-01-27 02:00:00 - 2026-01-28 02:00:00) # - segment-0006.wal (2026-01-28 02:00:00 - 2026-01-28 11:00:00) # Stopping at: 2026-01-28 10:30:00 UTC # Recovery complete # Transactions applied: 12,456 # Final LSN: 0x1A2B3C4D </code></pre></div>Step 5: Verify and Start <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Verify data integrity geode verify --data-dir /var/lib/geode/data # Start server sudo systemctl start geode # Verify recovery geode query "MATCH (n) RETURN count(n) AS node_count" </code></pre></div> <h4 id="pitr-to-named-recovery-point" class="position-relative d-flex align-items-center group"> PITR to Named Recovery Point <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="pitr-to-named-recovery-point" aria-haspopup="dialog" aria-label="Share link: PITR to Named Recovery Point"> Share link </button> </h4>If you created named recovery points: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Recover to named point geode recover \ --data-dir /var/lib/geode/data \ --wal-archive s3://geode-wal-archive/production/wal \ --target-name "pre-migration-backup" </code></pre></div> <h3 id="full-database-restore" class="position-relative d-flex align-items-center group"> Full Database Restore <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="full-database-restore" aria-haspopup="dialog" aria-label="Share link: Full Database Restore"> Share link </button> </h3>For complete database restoration from backup. <h4 id="standard-restore" class="position-relative d-flex align-items-center group"> Standard Restore <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="standard-restore" aria-haspopup="dialog" aria-label="Share link: Standard Restore"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Stop server sudo systemctl stop geode # Backup current data (safety) sudo mv /var/lib/geode/data /var/lib/geode/data.backup-$(date +%Y%m%d) # List available backups geode backup --dest s3://geode-backups/production --list # Restore latest backup LATEST_BACKUP=$(geode backup --dest s3://geode-backups/production --list --latest --quiet) geode restore \ --source s3://geode-backups/production \ --backup-id $LATEST_BACKUP \ --target /var/lib/geode/data # Verify integrity geode verify --data-dir /var/lib/geode/data # Start server sudo systemctl start geode </code></pre></div> <h4 id="restore-with-incremental-backups" class="position-relative d-flex align-items-center group"> Restore with Incremental Backups <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="restore-with-incremental-backups" aria-haspopup="dialog" aria-label="Share link: Restore with Incremental Backups"> Share link </button> </h4>When using incremental backup strategy: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Restore full backup chain geode restore \ --source s3://geode-backups/production \ --backup-id 1738012345 \ # Base full backup --apply-incrementals \ # Apply all incremental backups --target /var/lib/geode/data # Output: # Restoring base backup 1738012345... # Applying incremental 1738098745 (2026-01-24)... # Applying incremental 1738185145 (2026-01-25)... # Applying incremental 1738271545 (2026-01-26)... # Applying incremental 1738357945 (2026-01-27)... # Restore complete </code></pre></div> <h4 id="restore-to-different-location" class="position-relative d-flex align-items-center group"> Restore to Different Location <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="restore-to-different-location" aria-haspopup="dialog" aria-label="Share link: Restore to Different Location"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Restore to alternate directory geode restore \ --source s3://geode-backups/production \ --backup-id 1738012345 \ --target /var/lib/geode/data-restored # Start with alternate data directory geode serve \ --data-dir /var/lib/geode/data-restored \ --listen 0.0.0.0:3142 </code></pre></div> <h3 id="disaster-recovery" class="position-relative d-flex align-items-center group"> Disaster Recovery <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="disaster-recovery" aria-haspopup="dialog" aria-label="Share link: Disaster Recovery"> Share link </button> </h3> <h4 id="complete-site-failure" class="position-relative d-flex align-items-center group"> Complete Site Failure <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="complete-site-failure" aria-haspopup="dialog" aria-label="Share link: Complete Site Failure"> Share link </button> </h4>When the primary site is unavailable: Step 1: Provision New Infrastructure <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Deploy new Geode server # (Use your provisioning tool: Terraform, Ansible, etc.) # Install Geode curl -fsSL https://geodedb.com/install.sh | sudo bash </code></pre></div>Step 2: Configure Cloud Storage Access <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">export AWS_ACCESS_KEY_ID='your-access-key' export AWS_SECRET_ACCESS_KEY='your-secret-key' export AWS_REGION='us-west-2' </code></pre></div>Step 3: Restore from Off-Site Backup <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># List available backups geode backup --dest s3://geode-backups/production --list # Restore latest backup geode restore \ --source s3://geode-backups/production \ --backup-id latest \ --target /var/lib/geode/data # Apply WAL if PITR is needed geode recover \ --data-dir /var/lib/geode/data \ --wal-archive s3://geode-wal-archive/production/wal \ --target-time "latest" </code></pre></div>Step 4: Verify and Start <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Verify data geode verify --data-dir /var/lib/geode/data # Start server sudo systemctl start geode # Run comprehensive health check geode query "MATCH (n) RETURN labels(n) AS label, count(n) AS count" </code></pre></div> <h4 id="data-corruption-recovery" class="position-relative d-flex align-items-center group"> Data Corruption Recovery <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="data-corruption-recovery" aria-haspopup="dialog" aria-label="Share link: Data Corruption Recovery"> Share link </button> </h4>If data files are corrupted: Step 1: Identify Corruption <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Run integrity check geode verify --data-dir /var/lib/geode/data --full # Output: # Checking nodes.db... ERROR: Checksum mismatch at block 12345 # Checking edges.db... OK # Checking indexes/... OK # Verification FAILED: 1 corrupted file(s) </code></pre></div>Step 2: Attempt Repair <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Try automatic repair (minor corruption) geode repair --data-dir /var/lib/geode/data --auto # Output: # Repairing nodes.db... # Block 12345: Reconstructing from WAL # Block 12345: Repair successful # Repair complete </code></pre></div>Step 3: If Repair Fails, Restore from Backup <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Stop server sudo systemctl stop geode # Restore from last known good backup geode restore \ --source s3://geode-backups/production \ --backup-id last-verified \ --target /var/lib/geode/data # Start server sudo systemctl start geode </code></pre></div> <h4 id="accidental-data-deletion-recovery" class="position-relative d-flex align-items-center group"> Accidental Data Deletion Recovery <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="accidental-data-deletion-recovery" aria-haspopup="dialog" aria-label="Share link: Accidental Data Deletion Recovery"> Share link </button> </h4>If data was accidentally deleted: Option 1: Point-in-Time Recovery (Preferred) <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Restore to time before deletion geode recover \ --data-dir /var/lib/geode/data-recovery \ --wal-archive s3://geode-wal-archive/production/wal \ --target-time "2026-01-28 09:00:00" # Before deletion at 10:00 # Extract deleted data from recovered instance geode query "MATCH (n:DeletedLabel) RETURN n" \ --server localhost:3142 \ --format json > deleted_data.json # Re-import to production geode import --file deleted_data.json --server localhost:3141 </code></pre></div>Option 2: Partial Restore <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Export specific data from backup geode backup-extract \ --source s3://geode-backups/production \ --backup-id 1738012345 \ --query "MATCH (n:ImportantLabel) RETURN n" \ --output recovered_data.json # Import to production geode import --file recovered_data.json </code></pre></div> <h3 id="recovery-verification" class="position-relative d-flex align-items-center group"> Recovery Verification <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="recovery-verification" aria-haspopup="dialog" aria-label="Share link: Recovery Verification"> Share link </button> </h3> <h4 id="post-recovery-checks" class="position-relative d-flex align-items-center group"> Post-Recovery Checks <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="post-recovery-checks" aria-haspopup="dialog" aria-label="Share link: Post-Recovery Checks"> Share link </button> </h4>After any recovery, run these verification steps: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">#!/bin/bash # /usr/local/bin/geode-recovery-verify.sh echo "=== Geode Recovery Verification ===" # 1. Check server is running echo "1. Checking server status..." systemctl is-active geode || { echo "FAILED: Server not running"; exit 1; } # 2. Verify data integrity echo "2. Verifying data integrity..." geode verify --data-dir /var/lib/geode/data --quick || { echo "FAILED: Integrity check failed"; exit 1; } # 3. Check node count echo "3. Checking node count..." NODE_COUNT=$(geode query "MATCH (n) RETURN count(n)" --format json | jq -r '.result.rows[0][0]') echo " Node count: $NODE_COUNT" # 4. Check relationship count echo "4. Checking relationship count..." REL_COUNT=$(geode query "MATCH ()-[r]->() RETURN count(r)" --format json | jq -r '.result.rows[0][0]') echo " Relationship count: $REL_COUNT" # 5. Test transactions echo "5. Testing transactions..." geode query "BEGIN; CREATE (:RecoveryTest {ts: datetime()}); ROLLBACK" || { echo "FAILED: Transaction test failed"; exit 1; } # 6. Check replication (if cluster) echo "6. Checking cluster status..." geode cluster status 2>/dev/null || echo " Single-node mode (no cluster)" echo "" echo "=== Recovery Verification PASSED ===" </code></pre></div> <h4 id="data-comparison" class="position-relative d-flex align-items-center group"> Data Comparison <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="data-comparison" aria-haspopup="dialog" aria-label="Share link: Data Comparison"> Share link </button> </h4>Compare recovered data with expected state: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"># Export node counts by label geode query " MATCH (n) RETURN labels(n) AS label, count(n) AS count ORDER BY label " --format json > current_counts.json # Compare with expected (from before incident) diff expected_counts.json current_counts.json </code></pre></div> <h3 id="recovery-runbook" class="position-relative d-flex align-items-center group"> Recovery Runbook <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="recovery-runbook" aria-haspopup="dialog" aria-label="Share link: Recovery Runbook"> Share link </button> </h3> <h4 id="quick-reference-recovery-decision-tree" class="position-relative d-flex align-items-center group"> Quick Reference: Recovery Decision Tree <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="quick-reference-recovery-decision-tree" aria-haspopup="dialog" aria-label="Share link: Quick Reference: Recovery Decision Tree"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-plaintext" data-lang="plaintext">Server Down? │ ├── Server crash → Restart server (auto recovery) │ └── Data issue? │ ├── Minor corruption → Run geode repair │ ├── Major corruption → Restore from backup │ └── Accidental deletion? │ ├── Need specific time → PITR │ └── Need latest data → Restore + WAL replay </code></pre></div> <h4 id="emergency-recovery-checklist" class="position-relative d-flex align-items-center group"> Emergency Recovery Checklist <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="emergency-recovery-checklist" aria-haspopup="dialog" aria-label="Share link: Emergency Recovery Checklist"> Share link </button> </h4>Immediate Actions: <ul> <li><input disabled="" type="checkbox"> Assess the situation (type of failure)</li> <li><input disabled="" type="checkbox"> Check server logs: <code>geode logs --tail 500</code></li> <li><input disabled="" type="checkbox"> Preserve evidence: <code>cp -r /var/lib/geode/data /var/lib/geode/data.incident-$(date +%s)</code></li> <li><input disabled="" type="checkbox"> Notify stakeholders</li> </ul> Recovery Actions: <ul> <li><input disabled="" type="checkbox"> Determine recovery strategy (auto, PITR, full restore)</li> <li><input disabled="" type="checkbox"> Execute recovery procedure</li> <li><input disabled="" type="checkbox"> Run verification checks</li> <li><input disabled="" type="checkbox"> Monitor server stability</li> </ul> Post-Recovery: <ul> <li><input disabled="" type="checkbox"> Document incident timeline</li> <li><input disabled="" type="checkbox"> Identify root cause</li> <li><input disabled="" type="checkbox"> Update runbooks if needed</li> <li><input disabled="" type="checkbox"> Schedule post-incident review</li> </ul> <h3 id="best-practices" class="position-relative d-flex align-items-center group"> Best Practices <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="best-practices" aria-haspopup="dialog" aria-label="Share link: Best Practices"> Share link </button> </h3> <h4 id="prevention" class="position-relative d-flex align-items-center group"> Prevention <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="prevention" aria-haspopup="dialog" aria-label="Share link: Prevention"> Share link </button> </h4><ol> <li>Regular Backups: Daily incremental, weekly full</li> <li>WAL Archiving: Continuous to enable PITR</li> <li>Monitoring: Alert on backup failures, disk space</li> <li>Testing: Monthly recovery drills</li> </ol> <h4 id="configuration-recommendations" class="position-relative d-flex align-items-center group"> Configuration Recommendations <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="configuration-recommendations" aria-haspopup="dialog" aria-label="Share link: Configuration Recommendations"> Share link </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"># geode.yaml - Production recovery settings recovery: wal: enabled: true sync_mode: fsync segment_size: 64MB checkpoint: interval: 5m backup: wal_archiving: enabled: true retention_hours: 168 # 7 days schedule: full_backup: '0 2 * * 0' # Weekly Sunday incremental_backup: '0 2 * * 1-6' # Daily retention_days: 90 </code></pre></div> <h4 id="recovery-time-targets" class="position-relative d-flex align-items-center group"> Recovery Time Targets <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="recovery-time-targets" aria-haspopup="dialog" aria-label="Share link: Recovery Time Targets"> Share link </button> </h4><table> <thead> <tr> <th>Scenario</th> <th>Target RTO</th> <th>Target RPO</th> </tr> </thead> <tbody> <tr> <td>Server crash</td> <td>< 1 minute</td> <td>0 (no data loss)</td> </tr> <tr> <td>Minor corruption</td> <td>< 5 minutes</td> <td>0</td> </tr> <tr> <td>Full restore</td> <td>< 15 minutes</td> <td>Last backup</td> </tr> <tr> <td>PITR</td> <td>< 30 minutes</td> <td>Target timestamp</td> </tr> <tr> <td>Disaster recovery</td> <td>< 1 hour</td> <td>Last WAL archive</td> </tr> </tbody> </table> <h3 id="related-documentation" class="position-relative d-flex align-items-center group"> Related Documentation <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="related-documentation" aria-haspopup="dialog" aria-label="Share link: Related Documentation"> Share link </button> </h3><ul> <li><a href="/docs/guides/backup-automation/" >Backup Automation</a> - Automated backup configuration</li> <li><a href="/docs/configuration/server-configuration/" >Server Configuration</a> - Server settings</li> <li><a href="/docs/guides/troubleshooting/" >Troubleshooting</a> - Common issues and solutions</li> <li><a href="/docs/deployment/deployment-patterns/" >Deployment Patterns</a> - High availability setup</li> <li><a href="/docs/architecture/distributed-architecture/" >Distributed Architecture</a> - Cluster recovery</li> </ul> <h3 id="summary" class="position-relative d-flex align-items-center group"> Summary <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="summary" aria-haspopup="dialog" aria-label="Share link: Summary"> Share link </button> </h3>Geode provides robust recovery mechanisms: <ul> <li>Automatic crash recovery with WAL ensures zero data loss for committed transactions</li> <li>Point-in-time recovery enables restoration to any moment within retention period</li> <li>Full backup restore provides complete database reconstruction</li> <li>Disaster recovery procedures handle site-wide failures</li> </ul> Key Takeaways: <ul> <li>Enable WAL archiving for PITR capability</li> <li>Test recovery procedures regularly (monthly)</li> <li>Document and practice recovery runbooks</li> <li>Monitor backup health continuously</li> </ul> <hr> Last Updated: January 28, 2026 Geode Version: v0.2.18+ Status: Production Ready