Self-healing agent VMs: living on an ephemeral root filesystem

Some agent platforms rebuild a VM's root filesystem from a base image while
$HOME persists. This package describes a recovery pattern for any agent
stack on a rebuild-prone VM: identify the persistence boundary, keep
byte-exact restore caches in persistent storage, and use a sentinel
watchdog to detect and repair missing files.

The failure mode

An incomplete rebuild — a package step failing partway, or a runtime dropped
from the image — can return a machine with pieces missing, possibly
different pieces each time. The resulting failures can affect standard
libraries, runtime binaries, timezone data, or project tooling.

For a VM whose root filesystem is rebuilt and whose home is persistent:

Location	Survives a rebuild?
Home directory ($HOME)	Verify that it is persistent storage mounted over the image
/usr, /usr/local, /etc (system packages, runtimes)	Rebuilt from the base image
/tmp	No guarantee; it can also be reaped between restarts
System-wide installations	Only if included in the image or replayed successfully

For example, ModuleNotFoundError: No module named 'json' can indicate a
missing Python standard library file. Check the runtime and filesystem
before diagnosing application code. A dedicated checker can detect this
damage before a scheduled workload encounters it.

The pattern

Worked example: the scripts and examples watch a Python + Node stack.
Adapt the sentinels, restore caches, and verification checks to the local
stack; the recovery pattern applies to other runtimes and agent systems.

1. Keep restore caches in the persistent tree

For everything your stack depends on that lives outside $HOME, keep a
pristine copy inside $HOME and restore from there — never from the
network at heal time.

Cache the same build the system runs, not a close cousin:

• Python stdlib (Debian/Ubuntu): the distro package itself. Extract it
  without installing: dpkg-deb -x libpython3.12-stdlib_*.deb ~/stack-restore/ubuntu/.
  Verify whether /var/cache/apt/archives/ survives on your platform; if it
  does not, download the matching .deb into $HOME while healthy.
• Upstream standalone Python builds are an incomplete stdlib source.
  Some builds link most C extension modules statically into t