Housekeeping

A run is keyed by the workflow’s name, so running the same file again reuses its directory rather than starting a new one. What builds up over time is names: every workflow you have ever run keeps a directory, holding that run’s output and whatever its steps wrote.

$ dagweave runs
  Succeeded  pipeline                 just now
$ dagweave delete pipeline
dagweave: pipeline is gone from this machine.
dagweave: its step output went with it, and nothing else had a copy.

dagweave runs lists what is on this machine. dagweave delete takes a name from that list, and --all takes every run at once. Neither asks first, and neither can be undone: dagweave logs is the only place a step’s output exists at all, so deleting a run really does remove it.

Running the same file again also starts from nothing: everything the previous run wrote is removed before the new one starts, the same guarantee a fresh pod would give you. That is also why dagweave logs only ever shows the latest run of a given workflow, never an earlier one.

Clearing the hidden cluster

dagweave prune is the other command that removes something, and it leaves your runs alone. It takes the hidden cluster and the disk under it, not your run directories. dagweave prune NAME takes a cluster by name, which is the only way to reach one that is not the default hidden cluster; naming one this machine does not have is an error, not a quiet success.

The hidden cluster is left running between workflows on purpose: the next cluster-tier run finds it already up and pays no boot, which is most of what makes that tier usable at all. Measured on one machine, an idle cluster held about a gigabyte and a half of memory and a tenth of a core. dagweave prune is what gets that back, and starting a new cluster after pruning costs you roughly a minute again.