Run timeline
When a run took longer than it should, the timeline tells you which step held it up and why.
Open a run from the Runs screen, or click Logs on the live run panel. The Steps list is the timeline: one bar per step, all on one clock from the run’s start to its finish. Each row shows the phase, the bar, how long the step took and how long it waited for its pod. You don’t need to turn anything on.
Where the rest of the time went
Section titled “Where the rest of the time went”A run takes longer than its steps. Argo needs time to start, to move from one step to the next, and to wrap up. A thin strip along the top shows those gaps, and faint bands behind every step show them too:
- Starting, before the first step.
- Between steps, when no step was running.
- Finishing, after the last step.
A finished run adds them up after its duration, for example
Took 1m25s · 57s running steps · 24s between steps · 4s finishing. With Argo’s
defaults, expect up to about ten seconds between two steps in a row.
Pod wait and work time
Section titled “Pod wait and work time”Argo’s step duration includes waiting for a pod, so 20 minutes waiting for a GPU
node looks the same as 20 minutes of work. The timeline splits each bar where the
main container started. The shaded start of the bar is the pod wait, and the
Pod wait column gives it, for example 3m12s, or 0s if the container
started straight away.
Open the row to split the wait into:
- Creating the pod
- Finding a node. A long one means the cluster had no room.
- Starting containers: pulling images, and Argo’s
initcontainer loading input artifacts.
If the pod was never scheduled, the whole bar is pod wait.
Steps that ran no pod of their own read No pod: step groups, retry parents,
suspend steps and cached results. Dagweave keeps these times after Kubernetes
deletes the pod.
Sort by run order or duration
Section titled “Sort by run order or duration”With more than one bar, sort by:
- Run order, when each step started.
- Duration, longest first. On a wide DAG, that’s the quickest way to find what held the run up.
Bars stay at the same place on the clock either way. You can also filter by step name.
If the longest step is mostly pod wait, faster code won’t help. Look at what it’s waiting for: a node with enough CPU, memory or GPUs, or one that matches its selector and tolerations.
What gets a row
Section titled “What gets a row”Everything Argo reports except the workflow itself:
- Each retry attempt. A step that passed on its third try has three rows.
- Each item of a fan-out (
withItems,withParam,withSequence). A thousand items give a thousand rows, ungrouped. - Steps in a nested workflow, in the same flat list.
- Argo’s group nodes, like the parent of a set of retries. They read
No pod.
A step with no start time sits below the rest with no bar and reads no timing reported.
Open a step
Section titled “Open a step”Some things show on a closed row: a lock or node wait, and a failed step’s exit code and Argo’s message. Click the row to add:
- requested, limited and peak CPU and memory, and a new size for the step when Dagweave has one (see Right-size a step)
- a View trace link, if tracing is set up
- the step’s logs
A failed run opens with the failing step already expanded. See Runs and logs for where logs come from and how many lines you get.
Right-size a step
Section titled “Right-size a step”An open step shows its requested, limited and peak CPU and memory, for example
mem 1Gi req/2Gi lim/140Mi peak. The peak is the largest of every sample the
connector took while the step ran, about one every 30 seconds. A flag marks a
step that’s out of line:
near limit: the peak reached 90% of a limit. Past a memory limit the step is killed. Past a CPU limit it’s throttled.over-provisioned: the peak stayed under 30% of the request. The cluster holds the rest of the request for a step that never uses it.
Why the peak
Section titled “Why the peak”One sample says little. A step that peaks at 900Mi while it loads its data and ends at 100Mi would look over-provisioned from its last sample, and a request cut to match would get it killed for running out of memory on the next run. So Dagweave sizes from the peak and nothing else.
Connectors older than 0.14.0 send only the last sample, which Dagweave shows as
last seen. That’s still enough to say a step is near its limit, because the
peak was at least that high. It isn’t enough to call a step over-provisioned or
to recommend a size. Upgrade the connector to get both.
The recommendation
Section titled “The recommendation”Once a workflow has three or more successful runs that report peaks, a finished step gets a recommendation for its template, from the last five of those runs:
- Memory request: the peak plus 30% headroom, rounded up to a multiple of 16Mi, and at least 32Mi.
- CPU request: the peak plus 20%, rounded up to 10m, and at least 10m.
- A step near its limit: the limit raised to the peak plus 50%, rounded the same way.
A lower request is only suggested when it’s at most 70% of the current one, since a smaller cut isn’t worth a change. When several steps run the same template, such as the items of a fan-out, the largest peak counts. Lowering a request leaves the limit alone, and raising a limit leaves the request alone. Test runs don’t count.
The step shows a button such as Set memory request to 192Mi, with the basis
under it: Peak 140Mi over the last 5 runs, plus 30% headroom.
What one click does
Section titled “What one click does”That depends on where the template lives.
- A workflow saved in Dagweave gets a new version with the new value, and the next run uses it. If the workflow is open on a canvas, that canvas’s next save is refused and tells you the workflow changed.
- A workflow linked to Git gets a pull request on its file. Nothing changes in Dagweave or on the cluster until it’s merged.
- A step that calls a WorkflowTemplate changes that WorkflowTemplate instead, saved or through a pull request by the same rules. Every workflow that calls it gets the new value. If Dagweave has no copy of the WorkflowTemplate, you get YAML to paste instead.
- An adopted run, one Dagweave didn’t start, has no button. You get a YAML snippet naming the template and its resources, to paste wherever that template is defined. Adopted runs count as the same workflow when Argo named them from the same prefix, on the same cluster and namespace.
If you can’t edit the workflow, the button is greyed out and says why. Each
change is in the audit log as workflow.rightsize. Over the
API the same change is
POST /api/v1/workflows/{id}/resources, and over MCP it’s
dagweave_set_step_resources.
Steps that are running
Section titled “Steps that are running”A running step’s bar grows every second, and the clock runs to now. Time since the last step ended counts as Between steps until the run finishes.
If a bar keeps growing and nothing’s happening, see stuck runs.
Break a step down further
Section titled “Break a step down further”If the connector receives Argo’s OpenTelemetry spans, the working part of each bar splits into pod creation, artifact loading, artifact saving and log saving. What’s left is usually your own command. See Tracing.
To follow a trace into your own Jaeger or Tempo, use View trace on a step or Trace in the run’s header. Those links need:
- the workflow controller exporting spans, with
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTin its environment pointing at your collector traceBackendandtraceBackendURLunderargoConnection.extraSpecin the connector’s Helm values, plustraceBackendDatasourcefor Tempo
If either is missing, the run view says which.
Argo’s init-less pod layout
Section titled “Argo’s init-less pod layout”Dagweave only reads the main container: its resources, start time and logs.
Argo Workflows 4.1 has a beta init-less layout, off by default, where a
supervisor container runs beside main and does the jobs of init and wait.
A cluster operator turns it on with initlessPod.enabled: true in the workflow
controller ConfigMap. It needs Kubernetes image volumes: the ImageVolume
feature gate on 1.33 to 1.35, on by default from 1.36.
It moves input artifact loading. On the default layout, main starts after
loading, so loading counts as pod wait. On the init-less layout, main is
already running, so loading counts as work. Don’t compare pod wait across the two.