<!-- CANARY: REQ=REQ-DOCS-001; FEATURE="Docs"; ASPECT=Documentation; STATUS=TESTED; OWNER=docs; UPDATED=2026-01-15 --> <p>Implementing robust automated backup strategies is critical for production Geode deployments. This guide covers automated backup configuration, scheduling, testing, and disaster recovery procedures.</p> <h3 id="overview" class="position-relative d-flex align-items-center group"> <span>Overview</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="overview" aria-haspopup="dialog" aria-label="Share link: Overview"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h3><div id="headingShareModal" class="heading-share-modal" role="dialog" aria-modal="true" aria-labelledby="headingShareTitle" hidden> <div class="hsm-dialog" role="document"> <div class="hsm-header"> <h2 id="headingShareTitle" class="h6 mb-0 fw-bold">Share this section</h2> <button type="button" class="hsm-close" aria-label="Close"> <i class="fa-solid fa-xmark"></i> </button> </div> <div class="hsm-body"> <label for="headingShareInput" class="form-label small text-muted mb-1 text-uppercase fw-bold" style="font-size: 0.7rem; letter-spacing: 0.5px;">Permalink</label> <div class="input-group mb-4 hsm-url-group"> <input id="headingShareInput" type="text" class="form-control font-monospace" readonly aria-readonly="true" style="font-size: 0.85rem;" /> <button class="btn btn-primary hsm-copy" type="button" aria-label="Copy" title="Copy"> <i class="fa-duotone fa-clipboard" aria-hidden="true"></i> </button> </div> <div class="small fw-bold mb-2 text-muted text-uppercase" style="font-size: 0.7rem; letter-spacing: 0.5px;">Share via</div> <div class="hsm-share-grid"> <a id="share-twitter" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> <i class="fa-brands fa-twitter me-2"></i>Twitter </a> <a id="share-linkedin" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> <i class="fa-brands fa-linkedin me-2"></i>LinkedIn </a> <a id="share-facebook" class="btn btn-outline-secondary w-100" target="_blank" rel="noopener noreferrer"> <i class="fa-brands fa-facebook me-2"></i>Facebook </a> </div> </div> </div> </div> <style> .heading-share-modal { position: fixed; inset: 0; display: flex; justify-content: center; align-items: center; background: rgba(0, 0, 0, 0.6); z-index: 1050; padding: 1rem; backdrop-filter: blur(4px); -webkit-backdrop-filter: blur(4px); } .heading-share-modal[hidden] { display: none !important; } .hsm-dialog { max-width: 420px; width: 100%; background: var(--bs-body-bg, #fff); color: var(--bs-body-color, #212529); border: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); border-radius: 1rem; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); overflow: hidden; animation: hsm-fade-in 0.2s ease-out; } @keyframes hsm-fade-in { from { opacity: 0; transform: scale(0.95); } to { opacity: 1; transform: scale(1); } } [data-bs-theme="dark"] .hsm-dialog { background: #1e293b; border-color: rgba(255,255,255,0.1); color: #f8f9fa; } .hsm-header { display: flex; justify-content: space-between; align-items: center; padding: 1rem 1.5rem; border-bottom: 1px solid var(--bs-border-color, rgba(0,0,0,0.1)); background: rgba(0,0,0,0.02); } [data-bs-theme="dark"] .hsm-header { background: rgba(255,255,255,0.02); border-color: rgba(255,255,255,0.1); } .hsm-close { background: transparent; border: none; color: inherit; opacity: 0.5; padding: 0.25rem 0.5rem; border-radius: 0.25rem; font-size: 1.2rem; line-height: 1; transition: opacity 0.2s; } .hsm-close:hover { opacity: 1; } .hsm-body { padding: 1.5rem; } .hsm-url-group { display: flex !important; align-items: stretch; } .hsm-url-group .form-control { flex: 1; min-width: 0; margin: 0; background: var(--bs-secondary-bg, #f8f9fa); border-color: var(--bs-border-color, #dee2e6); border-top-right-radius: 0; border-bottom-right-radius: 0; height: 42px; } .hsm-url-group .btn { flex: 0 0 auto; margin: 0; margin-left: -1px; border-top-left-radius: 0; border-bottom-left-radius: 0; height: 42px; display: flex; align-items: center; justify-content: center; padding: 0 1.25rem; z-index: 2; } [data-bs-theme="dark"] .hsm-url-group .form-control { background: #0f172a; border-color: #334155; color: #e2e8f0; } .hsm-share-grid { display: flex; flex-direction: column; gap: 0.5rem; } .hsm-share-grid .btn { display: flex; align-items: center; justify-content: center; font-size: 0.9rem; padding: 0.6rem; border-color: var(--bs-border-color); width: 100%; } [data-bs-theme="dark"] .hsm-share-grid .btn { color: #e2e8f0; border-color: #475569; } [data-bs-theme="dark"] .hsm-share-grid .btn:hover { background: #334155; border-color: #cbd5e1; } </style> <script> (function(){ const modal = document.getElementById('headingShareModal'); if(!modal) return; const input = modal.querySelector('#headingShareInput'); const copyBtn = modal.querySelector('.hsm-copy'); const twitter = modal.querySelector('#share-twitter'); const linkedin = modal.querySelector('#share-linkedin'); const facebook = modal.querySelector('#share-facebook'); const closeBtn = modal.querySelector('.hsm-close'); let lastFocus=null; let trapBound=false; function buildUrl(id){ return window.location.origin + window.location.pathname + '#' + id; } function isOpen(){ return !modal.hasAttribute('hidden'); } function hydrate(id){ const url=buildUrl(id); input.value=url; const enc=encodeURIComponent(url); const text=encodeURIComponent(document.title); if(twitter) twitter.href=`https://twitter.com/intent/tweet?url=${enc}&text=${text}`; if(linkedin) linkedin.href=`https://www.linkedin.com/sharing/share-offsite/?url=${enc}`; if(facebook) facebook.href=`https://www.facebook.com/sharer/sharer.php?u=${enc}`; } function openModal(id){ lastFocus=document.activeElement; hydrate(id); if(!isOpen()){ modal.removeAttribute('hidden'); } requestAnimationFrame(()=>{ input.focus(); }); trapFocus(); } function closeModal(){ if(!isOpen()) return; modal.setAttribute('hidden',''); if(lastFocus && typeof lastFocus.focus==='function') lastFocus.focus(); } function copyCurrent(){ try{ navigator.clipboard.writeText(input.value).then(()=>feedback(true),()=>fallback()); } catch(e){ fallback(); } } function fallback(){ input.select(); try{ document.execCommand('copy'); feedback(true);}catch(e){ feedback(false);} } function feedback(ok){ if(!copyBtn) return; const icon=copyBtn.querySelector('i'); if(!icon) return; const prev=copyBtn.getAttribute('data-prev')||icon.className; if(!copyBtn.getAttribute('data-prev')) copyBtn.setAttribute('data-prev',prev); icon.className= ok ? 'fa-duotone fa-clipboard-check':'fa-duotone fa-circle-exclamation'; setTimeout(()=>{ icon.className=prev; },1800); } function handleShareClick(e){ e.preventDefault(); const btn=e.currentTarget; const id=btn.getAttribute('data-share-target'); if(id) openModal(id); } function bindShareButtons(){ document.querySelectorAll('.h-share').forEach(btn=>{ if(!btn.dataset.hShareBound){ btn.addEventListener('click', handleShareClick); btn.dataset.hShareBound='1'; } }); } bindShareButtons(); if(document.readyState==='loading'){ document.addEventListener('DOMContentLoaded', bindShareButtons); } else { requestAnimationFrame(bindShareButtons); } document.addEventListener('click', function(e){ const shareBtn=e.target.closest && e.target.closest('.h-share'); if(shareBtn && !shareBtn.dataset.hShareBound){ handleShareClick.call(shareBtn, e); } }, true); document.addEventListener('click', e=>{ if(e.target===modal) closeModal(); if(e.target.closest && e.target.closest('.hsm-close')){ e.preventDefault(); closeModal(); } if(copyBtn && (e.target===copyBtn || (e.target.closest && e.target.closest('.hsm-copy')))) { e.preventDefault(); copyCurrent(); } }); document.addEventListener('keydown', e=>{ if(e.key==='Escape' && isOpen()) closeModal(); }); function trapFocus(){ if(trapBound) return; trapBound=true; modal.addEventListener('keydown', f=>{ if(f.key==='Tab' && isOpen()){ const focusable=[...modal.querySelectorAll('a[href],button,input,textarea,select,[tabindex]:not([tabindex="-1"])')].filter(el=>!el.hasAttribute('disabled')); if(!focusable.length) return; const first=focusable[0]; const last=focusable[focusable.length-1]; if(f.shiftKey && document.activeElement===first){ f.preventDefault(); last.focus(); } else if(!f.shiftKey && document.activeElement===last){ f.preventDefault(); first.focus(); } } }); } if(closeBtn) closeBtn.addEventListener('click', e=>{ e.preventDefault(); closeModal(); }); })(); </script><p>Geode supports multiple backup strategies:</p> <ul> <li><strong>Full Backups</strong>: Complete database snapshot</li> <li><strong>Incremental Backups</strong>: Changes since last full backup</li> <li><strong>Point-in-Time Recovery (PITR)</strong>: Restore to specific timestamp</li> <li><strong>Cloud Storage</strong>: S3-compatible object storage integration</li> <li><strong>Automated Scheduling</strong>: Cron-based backup automation</li> </ul> <p><strong>Recovery Time Objective (RTO)</strong>: &lt; 5 minutes for full restore <strong>Recovery Point Objective (RPO)</strong>: &lt; 15 minutes with incremental backups</p> <h3 id="s3-cloud-backup-configuration" class="position-relative d-flex align-items-center group"> <span>S3 Cloud Backup Configuration</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="s3-cloud-backup-configuration" aria-haspopup="dialog" aria-label="Share link: S3 Cloud Backup Configuration"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h3> <h4 id="prerequisites" class="position-relative d-flex align-items-center group"> <span>Prerequisites</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="prerequisites" aria-haspopup="dialog" aria-label="Share link: Prerequisites"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><p><strong>S3-Compatible Storage</strong>:</p> <ul> <li>Amazon S3</li> <li>Digital Ocean Spaces</li> <li>MinIO</li> <li>Wasabi</li> <li>Backblaze B2</li> </ul> <p><strong>Required Credentials</strong>:</p> <ul> <li>Access Key ID</li> <li>Secret Access Key</li> <li>Region (for AWS)</li> <li>Endpoint URL (for non-AWS providers)</li> </ul> <h4 id="environment-setup" class="position-relative d-flex align-items-center group"> <span>Environment Setup</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="environment-setup" aria-haspopup="dialog" aria-label="Share link: Environment Setup"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># AWS S3</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_ACCESS_KEY_ID</span><span class="o">=</span><span class="s1">&#39;AKIAIOSFODNN7EXAMPLE&#39;</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_SECRET_ACCESS_KEY</span><span class="o">=</span><span class="s1">&#39;wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY&#39;</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_REGION</span><span class="o">=</span><span class="s1">&#39;us-east-1&#39;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Digital Ocean Spaces</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_ACCESS_KEY_ID</span><span class="o">=</span><span class="s1">&#39;DO_SPACES_KEY&#39;</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_SECRET_ACCESS_KEY</span><span class="o">=</span><span class="s1">&#39;DO_SPACES_SECRET&#39;</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_ENDPOINT_URL</span><span class="o">=</span><span class="s1">&#39;https://nyc3.digitaloceanspaces.com&#39;</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_REGION</span><span class="o">=</span><span class="s1">&#39;us-east-1&#39;</span> <span class="c1"># Required but not used by DO</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># MinIO</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_ACCESS_KEY_ID</span><span class="o">=</span><span class="s1">&#39;minio_access_key&#39;</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_SECRET_ACCESS_KEY</span><span class="o">=</span><span class="s1">&#39;minio_secret_key&#39;</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_ENDPOINT_URL</span><span class="o">=</span><span class="s1">&#39;https://minio.example.com:9000&#39;</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_REGION</span><span class="o">=</span><span class="s1">&#39;us-east-1&#39;</span> </span></span></code></pre></div> <h4 id="server-configuration" class="position-relative d-flex align-items-center group"> <span>Server Configuration</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="server-configuration" aria-haspopup="dialog" aria-label="Share link: Server Configuration"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># geode.yaml</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">backup</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">s3</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">bucket</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;geode-production-backups&#39;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">prefix: &#39;prod&#39; # Optional</span><span class="p">:</span><span class="w"> </span><span class="l">organize by environment</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">region</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;us-east-1&#39;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">endpoint</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;&#39;</span><span class="w"> </span><span class="c"># Leave empty for AWS S3</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">access_key_id</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;${AWS_ACCESS_KEY_ID}&#39;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">secret_access_key</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;${AWS_SECRET_ACCESS_KEY}&#39;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">compression</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="c"># gzip compression</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">encryption</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="c"># Server-side encryption</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">retention_days</span><span class="p">:</span><span class="w"> </span><span class="m">90</span><span class="w"> </span><span class="c"># Auto-delete old backups</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">schedule</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">full_backup</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0 2 * * 0&#39;</span><span class="w"> </span><span class="c"># Weekly on Sunday at 2 AM</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">incremental_backup</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0 2 * * 1-6&#39;</span><span class="w"> </span><span class="c"># Daily at 2 AM except Sunday</span><span class="w"> </span></span></span></code></pre></div> <h3 id="manual-backup-operations" class="position-relative d-flex align-items-center group"> <span>Manual Backup Operations</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="manual-backup-operations" aria-haspopup="dialog" aria-label="Share link: Manual Backup Operations"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h3> <h4 id="full-backup" class="position-relative d-flex align-items-center group"> <span>Full Backup</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="full-backup" aria-haspopup="dialog" aria-label="Share link: Full Backup"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Create full backup</span> </span></span><span class="line"><span class="cl">geode backup <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --dest s3://geode-backups/production <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --mode full <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --compression gzip </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Output:</span> </span></span><span class="line"><span class="cl"><span class="c1"># Backup started: backup-20260123-020000</span> </span></span><span class="line"><span class="cl"><span class="c1"># Backup ID: 1738012345</span> </span></span><span class="line"><span class="cl"><span class="c1"># Compressing data...</span> </span></span><span class="line"><span class="cl"><span class="c1"># Uploading to S3: s3://geode-backups/production/backup-1738012345.tar.gz</span> </span></span><span class="line"><span class="cl"><span class="c1"># Backup completed successfully</span> </span></span><span class="line"><span class="cl"><span class="c1"># Size: 2.3 GB (compressed from 5.1 GB)</span> </span></span><span class="line"><span class="cl"><span class="c1"># Duration: 45s</span> </span></span></code></pre></div> <h4 id="incremental-backup" class="position-relative d-flex align-items-center group"> <span>Incremental Backup</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="incremental-backup" aria-haspopup="dialog" aria-label="Share link: Incremental Backup"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Create incremental backup (delta since last full)</span> </span></span><span class="line"><span class="cl">geode backup <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --dest s3://geode-backups/production <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --mode incremental <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --parent <span class="m">1738012345</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Output:</span> </span></span><span class="line"><span class="cl"><span class="c1"># Incremental backup started</span> </span></span><span class="line"><span class="cl"><span class="c1"># Parent backup: 1738012345 (2026-01-23 02:00:00)</span> </span></span><span class="line"><span class="cl"><span class="c1"># Backup ID: 1738098745</span> </span></span><span class="line"><span class="cl"><span class="c1"># Changes: 156 MB</span> </span></span><span class="line"><span class="cl"><span class="c1"># Duration: 8s</span> </span></span></code></pre></div> <h4 id="list-backups" class="position-relative d-flex align-items-center group"> <span>List Backups</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="list-backups" aria-haspopup="dialog" aria-label="Share link: List Backups"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># List all backups</span> </span></span><span class="line"><span class="cl">geode backup <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --dest s3://geode-backups/production <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --list </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Output:</span> </span></span><span class="line"><span class="cl"><span class="c1"># Backup ID Type Size Timestamp Status</span> </span></span><span class="line"><span class="cl"><span class="c1"># 1738012345 full 2.3 GB 2026-01-23 02:00:00 complete</span> </span></span><span class="line"><span class="cl"><span class="c1"># 1738098745 incremental 156 MB 2026-01-24 02:00:00 complete</span> </span></span><span class="line"><span class="cl"><span class="c1"># 1738185145 incremental 89 MB 2026-01-25 02:00:00 complete</span> </span></span><span class="line"><span class="cl"><span class="c1"># 1738271545 incremental 134 MB 2026-01-26 02:00:00 complete</span> </span></span></code></pre></div> <h4 id="verify-backup-integrity" class="position-relative d-flex align-items-center group"> <span>Verify Backup Integrity</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="verify-backup-integrity" aria-haspopup="dialog" aria-label="Share link: Verify Backup Integrity"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Verify backup without restoring</span> </span></span><span class="line"><span class="cl">geode backup <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --dest s3://geode-backups/production <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --verify <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --backup-id <span class="m">1738012345</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Output:</span> </span></span><span class="line"><span class="cl"><span class="c1"># Verifying backup 1738012345...</span> </span></span><span class="line"><span class="cl"><span class="c1"># Downloading metadata...</span> </span></span><span class="line"><span class="cl"><span class="c1"># Checking file integrity (SHA256)...</span> </span></span><span class="line"><span class="cl"><span class="c1"># ✓ data/nodes.db: OK</span> </span></span><span class="line"><span class="cl"><span class="c1"># ✓ data/edges.db: OK</span> </span></span><span class="line"><span class="cl"><span class="c1"># ✓ data/indexes/: OK</span> </span></span><span class="line"><span class="cl"><span class="c1"># ✓ wal/: OK</span> </span></span><span class="line"><span class="cl"><span class="c1"># Backup integrity: VALID</span> </span></span></code></pre></div> <h3 id="automated-backup-scripts" class="position-relative d-flex align-items-center group"> <span>Automated Backup Scripts</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="automated-backup-scripts" aria-haspopup="dialog" aria-label="Share link: Automated Backup Scripts"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h3> <h4 id="backup-script" class="position-relative d-flex align-items-center group"> <span>Backup Script</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="backup-script" aria-haspopup="dialog" aria-label="Share link: Backup Script"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash </span></span></span><span class="line"><span class="cl"><span class="cp"></span><span class="c1"># /usr/local/bin/geode-backup.sh</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">set</span> -euo pipefail </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Configuration</span> </span></span><span class="line"><span class="cl"><span class="nv">BUCKET</span><span class="o">=</span><span class="s2">&#34;s3://geode-production-backups&#34;</span> </span></span><span class="line"><span class="cl"><span class="nv">RETENTION_DAYS</span><span class="o">=</span><span class="m">90</span> </span></span><span class="line"><span class="cl"><span class="nv">LOG_FILE</span><span class="o">=</span><span class="s2">&#34;/var/log/geode/backup.log&#34;</span> </span></span><span class="line"><span class="cl"><span class="nv">ALERT_EMAIL</span><span class="o">=</span><span class="s2">&#34;[email protected]&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Logging function</span> </span></span><span class="line"><span class="cl">log<span class="o">()</span> <span class="o">{</span> </span></span><span class="line"><span class="cl"> <span class="nb">echo</span> <span class="s2">&#34;[</span><span class="k">$(</span>date +<span class="s1">&#39;%Y-%m-%d %H:%M:%S&#39;</span><span class="k">)</span><span class="s2">] </span><span class="nv">$*</span><span class="s2">&#34;</span> <span class="p">|</span> tee -a <span class="s2">&#34;</span><span class="nv">$LOG_FILE</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"><span class="o">}</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Error handler</span> </span></span><span class="line"><span class="cl">handle_error<span class="o">()</span> <span class="o">{</span> </span></span><span class="line"><span class="cl"> log <span class="s2">&#34;ERROR: Backup failed at line </span><span class="nv">$1</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"> <span class="nb">echo</span> <span class="s2">&#34;Geode backup failed. Check </span><span class="nv">$LOG_FILE</span><span class="s2"> for details.&#34;</span> <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> mail -s <span class="s2">&#34;Geode Backup Failure&#34;</span> <span class="s2">&#34;</span><span class="nv">$ALERT_EMAIL</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"> <span class="nb">exit</span> <span class="m">1</span> </span></span><span class="line"><span class="cl"><span class="o">}</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">trap</span> <span class="s1">&#39;handle_error $LINENO&#39;</span> ERR </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Starting backup&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Determine backup type (full on Sunday, incremental otherwise)</span> </span></span><span class="line"><span class="cl"><span class="nv">DOW</span><span class="o">=</span><span class="k">$(</span>date +%u<span class="k">)</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="o">[</span> <span class="s2">&#34;</span><span class="nv">$DOW</span><span class="s2">&#34;</span> -eq <span class="m">7</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> </span></span><span class="line"><span class="cl"> log <span class="s2">&#34;Performing full backup (Sunday)&#34;</span> </span></span><span class="line"><span class="cl"> <span class="nv">BACKUP_ID</span><span class="o">=</span><span class="k">$(</span>geode backup <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --dest <span class="s2">&#34;</span><span class="nv">$BUCKET</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --mode full <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --compression gzip <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> grep <span class="s2">&#34;Backup ID&#34;</span> <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> awk <span class="s1">&#39;{print $3}&#39;</span><span class="k">)</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> <span class="nb">echo</span> <span class="s2">&#34;</span><span class="nv">$BACKUP_ID</span><span class="s2">&#34;</span> &gt; /var/lib/geode/last-full-backup </span></span><span class="line"><span class="cl"> log <span class="s2">&#34;Full backup completed: </span><span class="nv">$BACKUP_ID</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"><span class="k">else</span> </span></span><span class="line"><span class="cl"> log <span class="s2">&#34;Performing incremental backup&#34;</span> </span></span><span class="line"><span class="cl"> <span class="nv">PARENT</span><span class="o">=</span><span class="k">$(</span>cat /var/lib/geode/last-full-backup<span class="k">)</span> </span></span><span class="line"><span class="cl"> <span class="nv">BACKUP_ID</span><span class="o">=</span><span class="k">$(</span>geode backup <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --dest <span class="s2">&#34;</span><span class="nv">$BUCKET</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --mode incremental <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --parent <span class="s2">&#34;</span><span class="nv">$PARENT</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --compression gzip <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> grep <span class="s2">&#34;Backup ID&#34;</span> <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> awk <span class="s1">&#39;{print $3}&#39;</span><span class="k">)</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> log <span class="s2">&#34;Incremental backup completed: </span><span class="nv">$BACKUP_ID</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"><span class="k">fi</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Verify backup integrity</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Verifying backup integrity&#34;</span> </span></span><span class="line"><span class="cl">geode backup <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --dest <span class="s2">&#34;</span><span class="nv">$BUCKET</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --verify <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --backup-id <span class="s2">&#34;</span><span class="nv">$BACKUP_ID</span><span class="s2">&#34;</span> &gt;&gt; <span class="s2">&#34;</span><span class="nv">$LOG_FILE</span><span class="s2">&#34;</span> 2&gt;<span class="p">&amp;</span><span class="m">1</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Prune old backups</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Pruning backups older than </span><span class="nv">$RETENTION_DAYS</span><span class="s2"> days&#34;</span> </span></span><span class="line"><span class="cl">geode backup <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --dest <span class="s2">&#34;</span><span class="nv">$BUCKET</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --prune <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --older-than-days <span class="s2">&#34;</span><span class="nv">$RETENTION_DAYS</span><span class="s2">&#34;</span> &gt;&gt; <span class="s2">&#34;</span><span class="nv">$LOG_FILE</span><span class="s2">&#34;</span> 2&gt;<span class="p">&amp;</span><span class="m">1</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Backup completed successfully&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Send success notification</span> </span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Backup completed successfully. ID: </span><span class="nv">$BACKUP_ID</span><span class="s2">&#34;</span> <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> mail -s <span class="s2">&#34;Geode Backup Success&#34;</span> <span class="s2">&#34;</span><span class="nv">$ALERT_EMAIL</span><span class="s2">&#34;</span> </span></span></code></pre></div> <h4 id="crontab-configuration" class="position-relative d-flex align-items-center group"> <span>Crontab Configuration</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="crontab-configuration" aria-haspopup="dialog" aria-label="Share link: Crontab Configuration"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Install backup script</span> </span></span><span class="line"><span class="cl">sudo cp geode-backup.sh /usr/local/bin/ </span></span><span class="line"><span class="cl">sudo chmod +x /usr/local/bin/geode-backup.sh </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Add to crontab</span> </span></span><span class="line"><span class="cl">sudo crontab -e </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Daily backups at 2 AM</span> </span></span><span class="line"><span class="cl"><span class="m">0</span> <span class="m">2</span> * * * /usr/local/bin/geode-backup.sh &gt;&gt; /var/log/geode/backup-cron.log 2&gt;<span class="p">&amp;</span><span class="m">1</span> </span></span></code></pre></div> <h4 id="systemd-timer-alternative-to-cron" class="position-relative d-flex align-items-center group"> <span>Systemd Timer (Alternative to Cron)</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="systemd-timer-alternative-to-cron" aria-haspopup="dialog" aria-label="Share link: Systemd Timer (Alternative to Cron)"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># /etc/systemd/system/geode-backup.service</span> </span></span><span class="line"><span class="cl"><span class="k">[Unit]</span> </span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">Geode Automated Backup</span> </span></span><span class="line"><span class="cl"><span class="na">Wants</span><span class="o">=</span><span class="s">geode-backup.timer</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Service]</span> </span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">oneshot</span> </span></span><span class="line"><span class="cl"><span class="na">User</span><span class="o">=</span><span class="s">geode</span> </span></span><span class="line"><span class="cl"><span class="na">Group</span><span class="o">=</span><span class="s">geode</span> </span></span><span class="line"><span class="cl"><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/geode-backup.sh</span> </span></span><span class="line"><span class="cl"><span class="na">StandardOutput</span><span class="o">=</span><span class="s">append:/var/log/geode/backup.log</span> </span></span><span class="line"><span class="cl"><span class="na">StandardError</span><span class="o">=</span><span class="s">append:/var/log/geode/backup.log</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Install]</span> </span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span> </span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># /etc/systemd/system/geode-backup.timer</span> </span></span><span class="line"><span class="cl"><span class="k">[Unit]</span> </span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">Geode Backup Timer</span> </span></span><span class="line"><span class="cl"><span class="na">Requires</span><span class="o">=</span><span class="s">geode-backup.service</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Timer]</span> </span></span><span class="line"><span class="cl"><span class="c1"># Daily at 2 AM</span> </span></span><span class="line"><span class="cl"><span class="na">OnCalendar</span><span class="o">=</span><span class="s">daily</span> </span></span><span class="line"><span class="cl"><span class="na">OnCalendar</span><span class="o">=</span><span class="s">*-*-* 02:00:00</span> </span></span><span class="line"><span class="cl"><span class="na">Persistent</span><span class="o">=</span><span class="s">true</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Install]</span> </span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">timers.target</span> </span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Enable timer</span> </span></span><span class="line"><span class="cl">sudo systemctl daemon-reload </span></span><span class="line"><span class="cl">sudo systemctl <span class="nb">enable</span> geode-backup.timer </span></span><span class="line"><span class="cl">sudo systemctl start geode-backup.timer </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Check status</span> </span></span><span class="line"><span class="cl">sudo systemctl status geode-backup.timer </span></span><span class="line"><span class="cl">sudo systemctl list-timers geode-backup.timer </span></span></code></pre></div> <h3 id="restore-procedures" class="position-relative d-flex align-items-center group"> <span>Restore Procedures</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="restore-procedures" aria-haspopup="dialog" aria-label="Share link: Restore Procedures"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h3> <h4 id="full-restore" class="position-relative d-flex align-items-center group"> <span>Full Restore</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="full-restore" aria-haspopup="dialog" aria-label="Share link: Full Restore"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Stop Geode server</span> </span></span><span class="line"><span class="cl">sudo systemctl stop geode </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Backup current data (safety)</span> </span></span><span class="line"><span class="cl">sudo mv /var/lib/geode/data /var/lib/geode/data.backup-<span class="k">$(</span>date +%Y%m%d<span class="k">)</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Restore from backup</span> </span></span><span class="line"><span class="cl">geode restore <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --source s3://geode-backups/production <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --backup-id <span class="m">1738012345</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --target /var/lib/geode/data </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Verify data integrity</span> </span></span><span class="line"><span class="cl">geode verify --data-dir /var/lib/geode/data </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Start server</span> </span></span><span class="line"><span class="cl">sudo systemctl start geode </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Verify server health</span> </span></span><span class="line"><span class="cl">geode query <span class="s2">&#34;RETURN 1 AS health_check&#34;</span> </span></span></code></pre></div> <h4 id="point-in-time-recovery-pitr" class="position-relative d-flex align-items-center group"> <span>Point-in-Time Recovery (PITR)</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="point-in-time-recovery-pitr" aria-haspopup="dialog" aria-label="Share link: Point-in-Time Recovery (PITR)"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Restore to specific timestamp</span> </span></span><span class="line"><span class="cl">geode restore <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --source s3://geode-backups/production <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --backup-id <span class="m">1738012345</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --target /var/lib/geode/data <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --pitr-timestamp <span class="s2">&#34;2026-01-23 10:30:00&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Process:</span> </span></span><span class="line"><span class="cl"><span class="c1"># 1. Restore base backup (full backup 1738012345)</span> </span></span><span class="line"><span class="cl"><span class="c1"># 2. Apply WAL segments up to specified timestamp</span> </span></span><span class="line"><span class="cl"><span class="c1"># 3. Stop at exact recovery point</span> </span></span><span class="line"><span class="cl"><span class="c1">#</span> </span></span><span class="line"><span class="cl"><span class="c1"># Output:</span> </span></span><span class="line"><span class="cl"><span class="c1"># Restoring base backup...</span> </span></span><span class="line"><span class="cl"><span class="c1"># Applying WAL segments...</span> </span></span><span class="line"><span class="cl"><span class="c1"># - wal/segment-001.log (2026-01-23 02:05:00 - 03:00:00)</span> </span></span><span class="line"><span class="cl"><span class="c1"># - wal/segment-002.log (2026-01-23 03:00:00 - 04:00:00)</span> </span></span><span class="line"><span class="cl"><span class="c1"># ...</span> </span></span><span class="line"><span class="cl"><span class="c1"># - wal/segment-009.log (2026-01-23 10:00:00 - 10:35:12)</span> </span></span><span class="line"><span class="cl"><span class="c1"># Stopping at 2026-01-23 10:30:00</span> </span></span><span class="line"><span class="cl"><span class="c1"># Recovery complete</span> </span></span></code></pre></div> <h4 id="automated-restore-script" class="position-relative d-flex align-items-center group"> <span>Automated Restore Script</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="automated-restore-script" aria-haspopup="dialog" aria-label="Share link: Automated Restore Script"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash </span></span></span><span class="line"><span class="cl"><span class="cp"></span><span class="c1"># /usr/local/bin/geode-restore.sh</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">set</span> -euo pipefail </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nv">BUCKET</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$1</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"><span class="nv">BACKUP_ID</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$2</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"><span class="nv">TARGET</span><span class="o">=</span><span class="s2">&#34;</span><span class="si">${</span><span class="nv">3</span><span class="k">:-</span><span class="p">/var/lib/geode/data</span><span class="si">}</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"><span class="nv">PITR_TIMESTAMP</span><span class="o">=</span><span class="s2">&#34;</span><span class="si">${</span><span class="nv">4</span><span class="k">:-</span><span class="si">}</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Stopping Geode server...&#34;</span> </span></span><span class="line"><span class="cl">sudo systemctl stop geode </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Creating safety backup of current data...&#34;</span> </span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="o">[</span> -d <span class="s2">&#34;</span><span class="nv">$TARGET</span><span class="s2">&#34;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> </span></span><span class="line"><span class="cl"> sudo mv <span class="s2">&#34;</span><span class="nv">$TARGET</span><span class="s2">&#34;</span> <span class="s2">&#34;</span><span class="si">${</span><span class="nv">TARGET</span><span class="si">}</span><span class="s2">.before-restore-</span><span class="k">$(</span>date +%Y%m%d-%H%M%S<span class="k">)</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"><span class="k">fi</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Restoring from backup </span><span class="nv">$BACKUP_ID</span><span class="s2">...&#34;</span> </span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="o">[</span> -n <span class="s2">&#34;</span><span class="nv">$PITR_TIMESTAMP</span><span class="s2">&#34;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> </span></span><span class="line"><span class="cl"> geode restore <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --source <span class="s2">&#34;</span><span class="nv">$BUCKET</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --backup-id <span class="s2">&#34;</span><span class="nv">$BACKUP_ID</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --target <span class="s2">&#34;</span><span class="nv">$TARGET</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --pitr-timestamp <span class="s2">&#34;</span><span class="nv">$PITR_TIMESTAMP</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"><span class="k">else</span> </span></span><span class="line"><span class="cl"> geode restore <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --source <span class="s2">&#34;</span><span class="nv">$BUCKET</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --backup-id <span class="s2">&#34;</span><span class="nv">$BACKUP_ID</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --target <span class="s2">&#34;</span><span class="nv">$TARGET</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"><span class="k">fi</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Verifying data integrity...&#34;</span> </span></span><span class="line"><span class="cl">geode verify --data-dir <span class="s2">&#34;</span><span class="nv">$TARGET</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Starting Geode server...&#34;</span> </span></span><span class="line"><span class="cl">sudo systemctl start geode </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Wait for server to start</span> </span></span><span class="line"><span class="cl">sleep <span class="m">5</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Verifying server health...&#34;</span> </span></span><span class="line"><span class="cl">geode query <span class="s2">&#34;RETURN 1 AS health_check&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Restore completed successfully&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Usage:</span> </span></span><span class="line"><span class="cl"><span class="c1"># ./geode-restore.sh s3://geode-backups/production 1738012345</span> </span></span><span class="line"><span class="cl"><span class="c1"># ./geode-restore.sh s3://geode-backups/production 1738012345 /var/lib/geode/data &#34;2026-01-23 10:30:00&#34;</span> </span></span></code></pre></div> <h3 id="disaster-recovery-testing" class="position-relative d-flex align-items-center group"> <span>Disaster Recovery Testing</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="disaster-recovery-testing" aria-haspopup="dialog" aria-label="Share link: Disaster Recovery Testing"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h3> <h4 id="monthly-dr-test" class="position-relative d-flex align-items-center group"> <span>Monthly DR Test</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="monthly-dr-test" aria-haspopup="dialog" aria-label="Share link: Monthly DR Test"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash </span></span></span><span class="line"><span class="cl"><span class="cp"></span><span class="c1"># /usr/local/bin/geode-dr-test.sh</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">set</span> -euo pipefail </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nv">BUCKET</span><span class="o">=</span><span class="s2">&#34;s3://geode-backups/production&#34;</span> </span></span><span class="line"><span class="cl"><span class="nv">TEST_DIR</span><span class="o">=</span><span class="s2">&#34;/tmp/geode-dr-test-</span><span class="k">$(</span>date +%Y%m%d<span class="k">)</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"><span class="nv">REPORT_FILE</span><span class="o">=</span><span class="s2">&#34;/var/log/geode/dr-test-</span><span class="k">$(</span>date +%Y%m%d<span class="k">)</span><span class="s2">.log&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">log<span class="o">()</span> <span class="o">{</span> </span></span><span class="line"><span class="cl"> <span class="nb">echo</span> <span class="s2">&#34;[</span><span class="k">$(</span>date +<span class="s1">&#39;%Y-%m-%d %H:%M:%S&#39;</span><span class="k">)</span><span class="s2">] </span><span class="nv">$*</span><span class="s2">&#34;</span> <span class="p">|</span> tee -a <span class="s2">&#34;</span><span class="nv">$REPORT_FILE</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"><span class="o">}</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;=== Disaster Recovery Test Started ===&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Get latest backup</span> </span></span><span class="line"><span class="cl"><span class="nv">LATEST_BACKUP</span><span class="o">=</span><span class="k">$(</span>geode backup --dest <span class="s2">&#34;</span><span class="nv">$BUCKET</span><span class="s2">&#34;</span> --list <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> grep <span class="s2">&#34;full&#34;</span> <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> head -1 <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> awk <span class="s1">&#39;{print $1}&#39;</span><span class="k">)</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Testing restore of backup: </span><span class="nv">$LATEST_BACKUP</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Create test directory</span> </span></span><span class="line"><span class="cl">mkdir -p <span class="s2">&#34;</span><span class="nv">$TEST_DIR</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Restore to test directory</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Restoring backup...&#34;</span> </span></span><span class="line"><span class="cl"><span class="nv">START_TIME</span><span class="o">=</span><span class="k">$(</span>date +%s<span class="k">)</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">geode restore <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --source <span class="s2">&#34;</span><span class="nv">$BUCKET</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --backup-id <span class="s2">&#34;</span><span class="nv">$LATEST_BACKUP</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --target <span class="s2">&#34;</span><span class="nv">$TEST_DIR</span><span class="s2">&#34;</span> &gt;&gt; <span class="s2">&#34;</span><span class="nv">$REPORT_FILE</span><span class="s2">&#34;</span> 2&gt;<span class="p">&amp;</span><span class="m">1</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nv">END_TIME</span><span class="o">=</span><span class="k">$(</span>date +%s<span class="k">)</span> </span></span><span class="line"><span class="cl"><span class="nv">RESTORE_DURATION</span><span class="o">=</span><span class="k">$((</span>END_TIME <span class="o">-</span> START_TIME<span class="k">))</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Restore completed in </span><span class="si">${</span><span class="nv">RESTORE_DURATION</span><span class="si">}</span><span class="s2">s (RTO target: 300s)&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Verify data integrity</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Verifying data integrity...&#34;</span> </span></span><span class="line"><span class="cl">geode verify --data-dir <span class="s2">&#34;</span><span class="nv">$TEST_DIR</span><span class="s2">&#34;</span> &gt;&gt; <span class="s2">&#34;</span><span class="nv">$REPORT_FILE</span><span class="s2">&#34;</span> 2&gt;<span class="p">&amp;</span><span class="m">1</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Start test server</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Starting test server...&#34;</span> </span></span><span class="line"><span class="cl">geode serve <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --data-dir <span class="s2">&#34;</span><span class="nv">$TEST_DIR</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --listen 127.0.0.1:3142 <span class="p">&amp;</span> </span></span><span class="line"><span class="cl"><span class="nv">SERVER_PID</span><span class="o">=</span><span class="nv">$!</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">sleep <span class="m">5</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Run validation queries</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Running validation queries...&#34;</span> </span></span><span class="line"><span class="cl"><span class="nv">QUERY_COUNT</span><span class="o">=</span><span class="k">$(</span>geode query <span class="s2">&#34;MATCH (n) RETURN count(n)&#34;</span> --server 127.0.0.1:3142 <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> jq -r <span class="s1">&#39;.result.rows[0].count&#39;</span><span class="k">)</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Node count: </span><span class="nv">$QUERY_COUNT</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Stop test server</span> </span></span><span class="line"><span class="cl"><span class="nb">kill</span> <span class="nv">$SERVER_PID</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Cleanup</span> </span></span><span class="line"><span class="cl">rm -rf <span class="s2">&#34;</span><span class="nv">$TEST_DIR</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Generate report</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;=== Disaster Recovery Test Summary ===&#34;</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Backup ID: </span><span class="nv">$LATEST_BACKUP</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Restore Duration: </span><span class="si">${</span><span class="nv">RESTORE_DURATION</span><span class="si">}</span><span class="s2">s (RTO: 300s)&#34;</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;RTO Status: </span><span class="k">$(</span><span class="o">[</span> <span class="nv">$RESTORE_DURATION</span> -lt <span class="m">300</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="nb">echo</span> <span class="s1">&#39;PASS&#39;</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s1">&#39;FAIL&#39;</span><span class="k">)</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Data Integrity: VERIFIED&#34;</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Node Count: </span><span class="nv">$QUERY_COUNT</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl">log <span class="s2">&#34;Test Status: SUCCESS&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Schedule monthly</span> </span></span><span class="line"><span class="cl"><span class="c1"># 0 3 1 * * /usr/local/bin/geode-dr-test.sh</span> </span></span></code></pre></div> <h3 id="monitoring-and-alerting" class="position-relative d-flex align-items-center group"> <span>Monitoring and Alerting</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="monitoring-and-alerting" aria-haspopup="dialog" aria-label="Share link: Monitoring and Alerting"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h3> <h4 id="backup-monitoring-script" class="position-relative d-flex align-items-center group"> <span>Backup Monitoring Script</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="backup-monitoring-script" aria-haspopup="dialog" aria-label="Share link: Backup Monitoring Script"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash </span></span></span><span class="line"><span class="cl"><span class="cp"></span><span class="c1"># /usr/local/bin/geode-backup-monitor.sh</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nv">BUCKET</span><span class="o">=</span><span class="s2">&#34;s3://geode-backups/production&#34;</span> </span></span><span class="line"><span class="cl"><span class="nv">ALERT_EMAIL</span><span class="o">=</span><span class="s2">&#34;[email protected]&#34;</span> </span></span><span class="line"><span class="cl"><span class="nv">MAX_AGE_HOURS</span><span class="o">=</span><span class="m">26</span> <span class="c1"># Alert if no backup in 26 hours</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Get latest backup timestamp</span> </span></span><span class="line"><span class="cl"><span class="nv">LATEST</span><span class="o">=</span><span class="k">$(</span>geode backup --dest <span class="s2">&#34;</span><span class="nv">$BUCKET</span><span class="s2">&#34;</span> --list <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> head -2 <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> tail -1 <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> awk <span class="s1">&#39;{print $4&#34; &#34;$5}&#39;</span><span class="k">)</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nv">LATEST_EPOCH</span><span class="o">=</span><span class="k">$(</span>date -d <span class="s2">&#34;</span><span class="nv">$LATEST</span><span class="s2">&#34;</span> +%s<span class="k">)</span> </span></span><span class="line"><span class="cl"><span class="nv">NOW_EPOCH</span><span class="o">=</span><span class="k">$(</span>date +%s<span class="k">)</span> </span></span><span class="line"><span class="cl"><span class="nv">AGE_HOURS</span><span class="o">=</span><span class="k">$((</span> <span class="o">(</span>NOW_EPOCH <span class="o">-</span> LATEST_EPOCH<span class="o">)</span> <span class="o">/</span> <span class="m">3600</span> <span class="k">))</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="o">[</span> <span class="nv">$AGE_HOURS</span> -gt <span class="nv">$MAX_AGE_HOURS</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> </span></span><span class="line"><span class="cl"> <span class="nb">echo</span> <span class="s2">&#34;WARNING: Last backup is </span><span class="si">${</span><span class="nv">AGE_HOURS</span><span class="si">}</span><span class="s2"> hours old (max: </span><span class="si">${</span><span class="nv">MAX_AGE_HOURS</span><span class="si">}</span><span class="s2">)&#34;</span> <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> mail -s <span class="s2">&#34;Geode Backup Alert&#34;</span> <span class="s2">&#34;</span><span class="nv">$ALERT_EMAIL</span><span class="s2">&#34;</span> </span></span><span class="line"><span class="cl"> <span class="nb">exit</span> <span class="m">1</span> </span></span><span class="line"><span class="cl"><span class="k">fi</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;OK: Last backup is </span><span class="si">${</span><span class="nv">AGE_HOURS</span><span class="si">}</span><span class="s2"> hours old&#34;</span> </span></span><span class="line"><span class="cl"><span class="nb">exit</span> <span class="m">0</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Run hourly</span> </span></span><span class="line"><span class="cl"><span class="c1"># 0 * * * * /usr/local/bin/geode-backup-monitor.sh</span> </span></span></code></pre></div> <h4 id="prometheus-metrics" class="position-relative d-flex align-items-center group"> <span>Prometheus Metrics</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="prometheus-metrics" aria-haspopup="dialog" aria-label="Share link: Prometheus Metrics"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># prometheus.yml</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">scrape_configs</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;geode-backups&#39;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">static_configs</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">targets</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;localhost:9090&#39;</span><span class="p">]</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">metrics_path</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;/metrics&#39;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="c"># Exposed metrics:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="c"># geode_backup_last_success_timestamp</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="c"># geode_backup_duration_seconds</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="c"># geode_backup_size_bytes</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="c"># geode_backup_age_hours</span><span class="w"> </span></span></span></code></pre></div> <h4 id="alert-rules" class="position-relative d-flex align-items-center group"> <span>Alert Rules</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="alert-rules" aria-haspopup="dialog" aria-label="Share link: Alert Rules"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># alerts.yml</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">groups</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">geode_backups</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">rules</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BackupTooOld</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">geode_backup_age_hours &gt; 26</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">1h</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">labels</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">annotations</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Geode backup is too old&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Last backup is {{ $value }} hours old (max: 26)&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BackupFailed</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">increase(geode_backup_failures_total[1h]) &gt; 0</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">labels</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">annotations</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Geode backup failed&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ $value }} backup failures in last hour&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BackupSlowDuration</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">geode_backup_duration_seconds &gt; 600</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">labels</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">annotations</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Geode backup duration exceeded threshold&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Backup took {{ $value }}s (max: 600s)&#34;</span><span class="w"> </span></span></span></code></pre></div> <h3 id="best-practices" class="position-relative d-flex align-items-center group"> <span>Best Practices</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="best-practices" aria-haspopup="dialog" aria-label="Share link: Best Practices"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h3> <h4 id="backup-strategy" class="position-relative d-flex align-items-center group"> <span>Backup Strategy</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="backup-strategy" aria-haspopup="dialog" aria-label="Share link: Backup Strategy"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><p><strong>3-2-1 Rule</strong>:</p> <ul> <li><strong>3</strong> copies of data (production + 2 backups)</li> <li><strong>2</strong> different storage types (local + cloud)</li> <li><strong>1</strong> offsite copy (different region/provider)</li> </ul> <p><strong>Backup Schedule</strong>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">Full backup: Weekly (Sunday 2 AM) </span></span><span class="line"><span class="cl">Incremental backups: Daily (Monday-Saturday 2 AM) </span></span><span class="line"><span class="cl">WAL archival: Continuous (every 5 minutes) </span></span><span class="line"><span class="cl">Retention: 90 days </span></span></code></pre></div> <h4 id="testing-requirements" class="position-relative d-flex align-items-center group"> <span>Testing Requirements</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="testing-requirements" aria-haspopup="dialog" aria-label="Share link: Testing Requirements"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><p><strong>Monthly DR Tests</strong>:</p> <ul> <li>Full restore to test environment</li> <li>Verify data integrity</li> <li>Measure RTO (should be &lt; 5 minutes)</li> <li>Test PITR accuracy</li> </ul> <p><strong>Quarterly Full DR Drills</strong>:</p> <ul> <li>Complete failover simulation</li> <li>Document recovery procedures</li> <li>Update runbooks</li> <li>Train operations team</li> </ul> <h4 id="security" class="position-relative d-flex align-items-center group"> <span>Security</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="security" aria-haspopup="dialog" aria-label="Share link: Security"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><p><strong>Encryption</strong>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">backup</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">s3</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">encryption</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="c"># Server-side encryption (AES-256)</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">sse_kms_key_id</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;arn:aws:kms:us-east-1:123456789:key/...&#39;</span><span class="w"> </span><span class="c"># Optional KMS</span><span class="w"> </span></span></span></code></pre></div><p><strong>Access Control</strong>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="nt">&#34;Version&#34;</span><span class="p">:</span> <span class="s2">&#34;2012-10-17&#34;</span><span class="p">,</span> </span></span><span class="line"><span class="cl"> <span class="nt">&#34;Statement&#34;</span><span class="p">:</span> <span class="p">[</span> </span></span><span class="line"><span class="cl"> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="nt">&#34;Effect&#34;</span><span class="p">:</span> <span class="s2">&#34;Allow&#34;</span><span class="p">,</span> </span></span><span class="line"><span class="cl"> <span class="nt">&#34;Principal&#34;</span><span class="p">:</span> <span class="p">{</span><span class="nt">&#34;AWS&#34;</span><span class="p">:</span> <span class="s2">&#34;arn:aws:iam::123456789:user/geode-backup&#34;</span><span class="p">},</span> </span></span><span class="line"><span class="cl"> <span class="nt">&#34;Action&#34;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&#34;s3:PutObject&#34;</span><span class="p">,</span> <span class="s2">&#34;s3:GetObject&#34;</span><span class="p">,</span> <span class="s2">&#34;s3:ListBucket&#34;</span><span class="p">],</span> </span></span><span class="line"><span class="cl"> <span class="nt">&#34;Resource&#34;</span><span class="p">:</span> <span class="p">[</span> </span></span><span class="line"><span class="cl"> <span class="s2">&#34;arn:aws:s3:::geode-backups/*&#34;</span><span class="p">,</span> </span></span><span class="line"><span class="cl"> <span class="s2">&#34;arn:aws:s3:::geode-backups&#34;</span> </span></span><span class="line"><span class="cl"> <span class="p">]</span> </span></span><span class="line"><span class="cl"> <span class="p">}</span> </span></span><span class="line"><span class="cl"> <span class="p">]</span> </span></span><span class="line"><span class="cl"><span class="p">}</span> </span></span></code></pre></div> <h4 id="performance-optimization" class="position-relative d-flex align-items-center group"> <span>Performance Optimization</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="performance-optimization" aria-haspopup="dialog" aria-label="Share link: Performance Optimization"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h4><p><strong>Compression</strong>:</p> <ul> <li>Use gzip for general workloads (good compression, fast)</li> <li>Use zstd for better compression (requires more CPU)</li> <li>Disable for pre-compressed data</li> </ul> <p><strong>Parallel Uploads</strong>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Configure parallel uploads</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_MAX_CONCURRENT_REQUESTS</span><span class="o">=</span><span class="m">10</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_MAX_BANDWIDTH</span><span class="o">=</span>100MB/s </span></span></code></pre></div><p><strong>Incremental Strategy</strong>:</p> <ul> <li>Reduces backup time by 80-90%</li> <li>Lower network bandwidth usage</li> <li>Faster recovery for recent data</li> </ul> <h3 id="related-documentation" class="position-relative d-flex align-items-center group"> <span>Related Documentation</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="related-documentation" aria-haspopup="dialog" aria-label="Share link: Related Documentation"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h3><ul> <li><strong><a href="/docs/guides/migration-guide/" >Migration Guide</a> </strong> - Backup and restore procedures</li> <li><strong><a href="/docs/configuration/server-configuration/" >Server Configuration</a> </strong> - Backup configuration options</li> <li><strong><a href="/docs/guides/troubleshooting/" >Troubleshooting Guide</a> </strong> - Backup troubleshooting</li> <li><strong><a href="/docs/recovery/" >Disaster Recovery</a> </strong> - Complete DR procedures</li> </ul> <h3 id="summary" class="position-relative d-flex align-items-center group"> <span>Summary</span> <button type="button" class="h-share btn btn-link p-0 text-decoration-none link-secondary opacity-50 hover-opacity-100 transition-all ms-1" data-share-target="summary" aria-haspopup="dialog" aria-label="Share link: Summary"> <i class="fa-sharp-duotone fa-solid fa-share-nodes" aria-hidden="true" style="font-size: 0.8em;"></i> <span class="visually-hidden">Share link</span> </button> </h3><p>Automated backup implementation checklist:</p> <p>✅ <strong>Configure S3 credentials</strong> - Set up cloud storage access ✅ <strong>Enable automated backups</strong> - Schedule full + incremental backups ✅ <strong>Test restore procedures</strong> - Verify backups work (monthly) ✅ <strong>Implement monitoring</strong> - Alert on backup failures ✅ <strong>Document procedures</strong> - Maintain DR runbooks ✅ <strong>Practice DR drills</strong> - Regular failover testing</p> <p><strong>Key Metrics</strong>:</p> <ul> <li>RTO: &lt; 5 minutes (full restore)</li> <li>RPO: &lt; 15 minutes (with incremental backups)</li> <li>Backup frequency: Daily</li> <li>Retention: 90 days</li> <li>Test frequency: Monthly</li> </ul> <p>Automated backups ensure business continuity and data protection for production Geode deployments.</p>