vaultspec-rag

Storage and maintenance

Copied from vaultspec-rag's own documentation, and read against the pages written here at commit 04610a5c. Nothing in it needed answering differently.

Before you copy these

The command blocks below are written with the uv run prefix, and one full invocation in the prose is written bare. Subcommands named in prose without the program - server start and the like - carry no prefix either way and are not counted here. If you installed as a tool or as a binary, drop it and run vaultspec-rag bare; if you run it without installing, it becomes uvx --from "vaultspec-rag[gpu]" vaultspec-rag. Choose an install route sets out the four routes and which prefix each one needs.

vaultspec-rag keeps semantic search indexes so agents and operators can search your projects by meaning instead of by exact keyword. In server mode, the default, one background service per machine owns a single managed vector store, and every project you index - each one a root - lives inside that one store. This guide covers how to inspect what that store holds, how the service reclaims dead data on its own, how to restore an index from an archive, how to prune space by hand, and how to watch maintenance as it runs.

Everything here needs the background service running. Start it with uv run vaultspec-rag server start. If you haven’t set the service up yet, work through getting started first, then run the background service for the full startup and configuration walkthrough.

Maintenance operates on the shared managed store. Migration also accesses a project’s local-only store.

Examples use the uv run prefix, which runs the command inside a project environment. If you installed vaultspec-rag as a standalone tool, drop the prefix and call vaultspec-rag directly; see the installation guide for lane selection.

Vocabulary

Term	Meaning
root	A project directory that has been indexed.
collection prefix	r + 12 hex characters + _, derived from a one-way hash of the resolved root path. It cannot be reversed to the path.
namespace	All collections in the store sharing one root’s prefix (typically a vault collection and a codebase collection).
live	The namespace’s recorded root directory exists.
orphaned	The recorded root is gone, and its drive or share is reachable - a true deletion.
unknown	The store holds the namespace, but no root can be attributed to it. Never touched automatically.
unverifiable	The root’s volume or network share is offline, so existence cannot be checked. Never touched automatically.
debris	A half-written collection directory the server cannot load, left behind by a crash. Removable only with prune --debris, and never archived.
temp_rooted	A survey flag marking a namespace whose root lives under an operating-system temp directory. Subject to the extra idle clock.
dangling data	Namespaces whose source root no longer exists. They occupy disk but can never serve a useful search.
grace window	The continuous time a root must stay orphaned before automatic reclamation may act. The clock survives restarts and resets when the root reappears.
maintenance cycle	One scheduled pass of the service’s storage maintenance: classify, advance grace clocks, reclaim, sweep archives, report.
snapshot archive	A recoverable copy of a data-bearing namespace’s collections, written immediately before automatic reclamation drops them.

Why disk usage grows

Creating a namespace preallocates a large block of storage immediately, before a single document is indexed. Every root you have ever indexed - including throwaway ones like test directories, temporary worktrees, and scratch checkouts - keeps costing that space until it is reclaimed. One development machine accumulated 79 dead namespaces totalling 167.9 GiB, all holding zero documents.

That preallocation is why disk usage tracks the number of roots you have indexed far more closely than the amount of code and documents in them. It also means there are two different ways to get space back: removing namespaces you no longer need, and shrinking the ones you are keeping. Both have their own sections, Reclaim space manually and Shrinking collections you keep, and the service does both on its own.

The store lives at ~/.vaultspec-rag/qdrant-server/storage by default; VAULTSPEC_RAG_QDRANT_STORAGE_DIR relocates it. Any location works, including deeply nested ones - on Windows the service hands the storage engine extended-length paths, so the classic 260-character path limit does not constrain where the store lives.

Inspect what is stored

List every namespace with its classification, document count, footprint, and attributed root:

Command

uv run vaultspec-rag server storage survey

Captured output

34 namespaces  (orphaned=10 unknown=0 unverifiable=0 live=24)  33.4 GiB on disk  [19 temp-rooted]
  orphaned r02c5d80096c3_         2 pts  318.6 MiB  C:\Users\me\AppData\Local\Temp\.tmpMum3wV  [temp]
  live     r45b56789f389_     12322 pts  1020.3 MiB  C:\projects\my-project

Two of the thirty-four rows are shown, and the namespace prefixes and root paths are stand-ins; the classifications, counts, and footprints are as the tool reported them, and the two progress lines it prints while it works are cut. Each row reads left to right as the classification, the namespace prefix, the document count, the on-disk footprint, and the attributed root path; a namespace no root can be attributed to shows (unattributable) in the final column. A row whose root lives under an operating-system temp directory carries a trailing [temp], and the summary line counts those separately. --orphaned and --unknown narrow the list to those states. With a running daemon the survey is answered by the service itself, so the CLI, the MCP tools, and HTTP consumers all see one classification; without a daemon the CLI reads the store directly.

A running daemon answers from a cached survey snapshot rather than re-measuring every namespace per call, so the survey stays fast (sub-second) no matter how many namespaces the store holds. The snapshot is computed shortly after startup and refreshed by every maintenance cycle; the HTTP response carries computed_at (when the underlying survey ran) and source (cache or fresh) so a consumer can see exactly how old the data is. The CLI’s --json carries the namespaces and the counts rather than those two fields, so a script that needs the age of the answer should read it over HTTP. Survey data is therefore eventually-consistent, up to one maintenance interval behind. When you need up-to-the-second truth - for example immediately after indexing or deleting a namespace - pass --fresh (HTTP: ?fresh=true), which recomputes the survey and reseeds the cache.

To look up a single root - which namespace and collection prefix belong to it - pass --root:

Command

uv run vaultspec-rag server storage survey --root C:\projects\my-project

The output leads with Queried root: <resolved path>  prefix: r..._, and the same lookup is available as queried_root in --json output. This works even for a root that has never been indexed: the service computes the authoritative prefix, so an external consumer never has to reimplement the hash.

The full flag and exit-code table is in the CLI reference.

Automatic reclamation

The running service reclaims confirmed-dangling namespaces on its own: a maintenance cycle runs every 60 minutes by default, with the first cycle one interval after startup. Set VAULTSPEC_RAG_STORAGE_AUTOPRUNE=0 to disable it.

A namespace is only ever reclaimed automatically when all of the following hold:

• it is attributed to a known root,

• that root is classified orphaned - gone, on a reachable volume,

• and it has been orphaned continuously for its full grace window. The clock is persisted, so restarting the service does not reset it; the root reappearing resets it to zero instantly.

Reclamation is tiered. A namespace holding zero documents is dropped after 24 hours of continuous orphan-hood. A namespace holding data waits 7 days, then each of its collections is written to a snapshot archive, and the namespace is dropped only if every snapshot succeeded - a failed archive always cancels the drop.

One class waits less. A namespace whose root lived under the OS temp directory was a throwaway sandbox, and an orphaned one is a sandbox that was torn down, so it is reclaimed after 24 hours whatever it holds. That shortens the waiting period and nothing else: a point-bearing temp-rooted namespace is still archived first and still not dropped if the archive fails. An absent root on an unreachable volume classifies unverifiable rather than orphaned and so never draws this window.

The cycle never touches unknown namespaces, unverifiable namespaces (an unplugged drive looks exactly like a deleted root, so it is never treated as one), or - with one exception, temp-rooted namespaces - anything live. At most 16 namespaces are reclaimed per cycle; the remainder waits for the next one.

The exception is temp-rooted namespaces. A harness temp directory that still exists classifies live and would otherwise survive every prune forever, which is exactly how leaked harness namespaces once filled a disk. A namespace whose root lives under an OS temp directory therefore runs on an additional clock: every successful index run stamps a persisted last_indexed time, and once that stamp is older than the ephemeral idle TTL (72 hours by default) the namespace is treated as dangling even though its root exists. The same tiers then apply - empty ones drop, data-bearing ones are archived first - under the same per-cycle cap, with ordinary orphans taking priority. An actively re-indexed temp root keeps refreshing its stamp and is never touched; set VAULTSPEC_RAG_STORAGE_AUTOPRUNE_EPHEMERAL_IDLE_HOURS=0 to disable the tier.

The interval, all three grace windows, the ephemeral idle TTL, the per-cycle cap, and the archive bounds are tunable - see the storage maintenance knobs.

Shrinking collections you keep

Reclamation removes namespaces you no longer need. Geometry reconcile is the other half: it shrinks the preallocation of namespaces you are keeping.

A collection’s on-disk cost is dominated by a fixed floor rather than by its contents - the storage engine preallocates a set of memory-mapped pages per segment, and it sizes the segment count from the host’s CPU count unless told otherwise. On a 24-core machine that meant eight segments and roughly 1.2 GiB for a collection holding zero documents. Newer versions cap this at creation, but a collection carries the geometry it was created with for its whole life, so upgrading does not shrink anything that already exists.

The service converges them for you. Each maintenance cycle reconciles up to four drifted collections onto the bounded geometry, so a backend full of oversized collections shrinks over the following few hours without any action. Measured across collection sizes from empty to 20,000 documents, this reclaims 63-84% of each collection’s footprint.

Reconcile is not destructive. It changes a setting and lets the storage engine merge segments in the background: no document is moved or deleted, and the collection stays searchable throughout. It is also idempotent - a collection already at the target is skipped, so once your backend has converged the stage does nothing.

To converge everything at once instead of waiting for the cycles, preview and then apply:

Command

uv run vaultspec-rag server storage reconcile --dry-run

After reviewing the preview, apply the changes:

Command

uv run vaultspec-rag server storage reconcile --yes

Captured output

Reconciled 6 collections (23.4 GiB reclaimed); 0 still converging.
  reconciled       r45b56789f389_vault_docs         8->1 segments, 1.0 GiB freed

Merging is a background operation, so the command waits for each collection to settle before reporting what it saved. That wait matters: while the engine restructures a collection, both its segment count and its on-disk size briefly rise above where they started before falling well below. A figure read mid-merge would report a reclamation in progress as growth, so vaultspec-rag only ever reports a size it has watched stop changing - and it waits for the merge to actually begin before it starts watching, because a merge still queued behind a busy engine looks exactly as motionless as a finished one.

Pass --no-wait to issue the changes and return immediately; the collections still converge on their own, but that run reports no reclaimed bytes, because at that moment there is no honest number to give. --limit bounds how many collections one run touches.

A merge needs room to inflate before it shrinks, so a collection is skipped with insufficient_headroom when the volume is too full to absorb it safely. If you hit that, prune first and reconcile afterwards.

Set VAULTSPEC_RAG_STORAGE_RECONCILE=0 to disable the automatic stage; the per-cycle cap and the convergence budget are tunable alongside it.

One residue is left behind deliberately. The write-ahead log size is fixed when a collection is created and cannot be changed in place, so a reconciled collection keeps a 32 MiB log where a freshly created one takes 16 MiB. That is a small fraction of a floor measured in gigabytes, and removing it would mean recreating the collection - a far riskier operation for far less space.

Archives and how to restore them

Snapshot archives land in ~/.vaultspec-rag/qdrant-server/archive/<prefix>/, one subdirectory per reclaimed namespace and one .snapshot file per collection. Each maintenance cycle deletes archives older than 30 days, then evicts oldest-first if the archive directory exceeds 64 GiB; both bounds are configurable. The setting for the second is named VAULTSPEC_RAG_STORAGE_AUTOPRUNE_ARCHIVE_MAX_GB, and the number you give it is multiplied by 1024 three times, so it is read as GiB despite the name.

Reindex first. The index is derived data, so if the root still exists, or came back from your own backup, indexing it rebuilds everything.

To restore the archived index directly, use the storage command. Point it at the archive directory holding the snapshot manifest, and name the root the restored namespace is keyed to:

Command

uv run vaultspec-rag server storage restore ~/.vaultspec-rag/qdrant-server/archive/r0123456789ab --root /path/to/project --dry-run

After checking the destination collections, restore them:

Command

uv run vaultspec-rag server storage restore ~/.vaultspec-rag/qdrant-server/archive/r0123456789ab --root /path/to/project --yes

The destination root must hold no collections. There is no override, so restore into a root you have already pruned or one you have not indexed. Preview with --dry-run to see the destination collections before anything is written.

Reclaim space manually

Manual pruning deletes; it does not archive. Automatic reclamation snapshots a data-bearing namespace before dropping it. prune and delete do not: they remove the namespace outright, with no grace window, because the operator running the command is the confirmation. If you want a copy, take one before you run these.

Work in this order:

1. Survey the store to see what is there.

2. Prune the orphaned namespaces.

3. Delete a single namespace only when prune will not remove it.

Preview first, then apply:

Command

uv run vaultspec-rag server storage prune --dry-run

After checking which namespaces will be deleted, apply the prune:

Command

uv run vaultspec-rag server storage prune --yes

Captured output

Reclaimed 79 orphaned namespaces (167.9 GiB); 0 unknown left untouched.

Prune targets only orphaned namespaces; unknown and unverifiable are never touched. To remove one specific namespace, name its prefix:

Command

uv run vaultspec-rag server storage delete r0123456789ab_ --yes

delete refuses a prefix the manifest cannot attribute to a root unless you pass --allow-unknown.

A crash can leave a half-written collection directory whose config file never landed. The server cannot load it, skips it at startup, and it sits on disk indefinitely. The survey lists such directories with status debris.

prune --debris is an unrecoverable filesystem delete. Qdrant cannot snapshot or drop a collection it never loaded, so there is no archive and no restore path. Debris removal is never automatic: it has no manifest attribution, so it stays behind the explicit flag as well as the prune confirmation.

Command

uv run vaultspec-rag server storage prune --debris --dry-run

After checking the listed directories, delete them:

Command

uv run vaultspec-rag server storage prune --debris --yes

You can also address a namespace by its root path instead of its prefix - the sanctioned teardown for test harnesses and consumers that register throwaway roots against the resident service:

Command

uv run vaultspec-rag server storage delete --root C:\Temp\my-throwaway-root --yes --json

The path is resolved and hashed exactly as indexing does, so it removes precisely the namespace that root’s indexing created. Deletion is idempotent: an already-absent namespace reports already_absent and exits 0, so a teardown hook can run unconditionally. A harness that deletes its temp roots instead also works - the scheduled reclamation removes the leftover namespaces once their grace window passes.

Harnesses must isolate or tear down

A test, demo, or acceptance harness that indexes a throwaway root against the resident service mints a real namespace in the shared backend - often gigabytes once fully indexed - and as long as the temp directory still exists it classifies live and survives every prune. Left alone, leaked harness namespaces can exhaust the disk (the issue-242 incident: 36 temp-rooted namespaces, ~74 GiB). Every harness must do one of these, in order of preference:

1. Isolate: point VAULTSPEC_RAG_STATUS_DIR and VAULTSPEC_RAG_QDRANT_STORAGE_DIR at harness-owned temp paths so its indexing never reaches the shared backend at all.

2. Tear down: run vaultspec-rag server storage delete --root <dir> --yes unconditionally in the harness’s cleanup (idempotent, exits 0 when already absent).

3. Delete the root and wait: removing the temp directory itself lets the scheduled reclamation collect the namespace after its grace window.

server storage survey marks suspect entries: any namespace whose root lives under an OS temp directory is flagged temp_rooted in --json and [temp] in the human table, so leaked harness indexes are visible before they become a disk incident.

Flags and exit codes are in the CLI reference.

Migrate a root between backends

Copy one project’s index between local-only storage and managed Qdrant. The source collections are retained.

1. Keep managed Qdrant reachable. Close processes holding the local store, and prevent indexing, watchers, and maintenance from writing to either participating index. Migration does not pause writers.

2. In the migration shell, set VAULTSPEC_RAG_QDRANT_SERVER to 1 and VAULTSPEC_RAG_LOCAL_ONLY to 0 for either direction. These settings configure the invoking CLI, not the running service.

3. Replace the sample path with your project directory and preview the copy to managed Qdrant. For the other direction, use --to local in both commands.

   Command

   uv run vaultspec-rag server storage migrate "C:\code\my-project" --to server --dry-run
   

   Preview opens the local store but copies nothing. Inspect the results before proceeding.

4. Apply the copy:

   Command

   uv run vaultspec-rag server storage migrate "C:\code\my-project" --to server --yes
   

5. Inspect every collection’s result. Existing destinations are skipped, including partial copies left by failed runs; rerunning does not repair them. See the result and exit semantics. For unresolved failures, open an issue.

6. Once the required collections have copied successfully, change the backend and run a search. Keep the source collections until the destination is verified.

Observe maintenance

Every cycle is a job: uv run vaultspec-rag server jobs lists it as a storage maintenance cycle with a result summary like removed=2 failed=0 pending=5 reclaimed_bytes=4508876800. Every cycle also writes one structured service.maintenance log line with the same counts plus archive activity and free disk, and logs an explicit disk_low warning when the store’s volume drops under 10 GiB free - only a few namespaces of headroom, so treat the warning as a prompt to prune or add capacity.

The token-gated /metrics route exports the rollup in Prometheus text format. All names carry the vaultspec_rag_ prefix:

Metric	Type	Meaning
maintenance_cycles_total	counter	Cycles run since service start
maintenance_reclaims_total	counter	Namespaces reclaimed since service start
maintenance_disk_free_bytes	gauge	Free disk on the store’s volume at the last cycle
maintenance_dangling_bytes	gauge	Total footprint of currently orphaned namespaces
maintenance_pending_grace	gauge	Orphans still inside their grace window
maintenance_orphaned_namespaces	gauge	Orphaned namespace count at the last cycle
maintenance_last_reclaimed_bytes	gauge	Bytes reclaimed by the last cycle
store_total_bytes	gauge	Whole-backend on-disk footprint (all statuses)
store_namespaces	gauge	Total namespace count at the last cycle
maintenance_reconciled_total	counter	Collections shrunk onto the bounded geometry
maintenance_reconciled_bytes_total	counter	Bytes reclaimed by geometry reconcile
store_drifted_collections	gauge	Collections not yet converged onto the bounded geometry

store_drifted_collections counts both collections still carrying oversized geometry and collections whose setting is already correct but whose merge is still running - so it reaching zero genuinely means the backend has finished converging, not merely that the settings have been written. Note that maintenance_reconciled_bytes_total credits only merges a cycle watched to completion; a merge that outlives its convergence budget still finishes, but its bytes go uncounted.

The GET /storage/survey route carries a whole-backend rollup as a totals object: total bytes, namespace count, total collection count, a per-status byte breakdown, the ephemeral backlog - the footprint still held by orphaned temp-rooted namespaces, the population next eligible for reclamation - and the count of namespaces whose point count could not be fully read - so a pile of live-but-leaked namespaces is visible even though it never counts as dangling, and an unread count is never mistaken for a verified zero. The CLI --json survey emits the per-namespace list (namespaces, returned, total, queried_root) without that totals rollup.

Tuning the schedule, grace windows, cap, and archive bounds is covered by the storage maintenance knobs.

Where to go next

• CLI reference - full flag and exit-code tables for server storage survey, server storage prune, server storage delete, and server stop.

• Configuration - every tunable, including the storage maintenance knobs.

• Storage backends - the server-first backend model and the managed Qdrant server.

• Run the background service - service lifecycle, status, jobs, and logs.

• Support: open an issue at github.com/nevenincs/vaultspec-rag/issues.