Skip to content

Stuck runs

When a run or step sits at Pending, open the run. Dagweave names two waits for you: a step behind a lock, and a pod the scheduler can’t place. For anything else, the step’s row shows Argo’s own message, and this page says where to look.

◷ No step has been reported yet, 6m12s so far.

The run was sent but nothing has come back. After two minutes the line names the cluster the run went to and links to Clusters. Work through these in order:

  1. Is the connector online? Follow the link, or open the settings menu and choose Clusters. If the row says offline or pending, the run is waiting on the connector. See the connector.
  2. Read the connector’s logs with Logs on the same row (admins and owners only). If it refused the dispatch, for example because no ArgoConnection serves the namespace, the logs say so.
  3. Did Argo pick up the Workflow? kubectl -n <namespace> get workflows shows it if the connector submitted it. If the workflow controller isn’t running or isn’t watching that namespace, it sits there untouched. The Argo UI shows why.

If another workflow holds a step’s mutex or semaphore, the row names the lock, the wait and the holder:

◷ Waiting on mutex deploy-lock for 12m14s, held by deploy-prod-x7k2q

Semaphores look the same. A lock in another namespace is prefixed with it. Dagweave names one holder even when a semaphore has several, and shows no queue position because Argo doesn’t record one.

Dagweave’s own run queue is a different wait, and it tells you how many of your team’s runs are ahead. A run that waits in both shows both, in order.

Usually the lock is doing its job and you can leave it. Act when the holder has already finished without releasing it. Open the holding Workflow on the cluster, and if it’s really gone, delete it to free the lock.

If steps take turns more than they need to, give the semaphore more slots or use a narrower lock. Set a step’s lock under Step lock in the inspector, and the workflow’s under Concurrency.

If the scheduler can’t place the pod, the row says:

◷ No node can take this pod for 12m14s. The cluster may be out of CPU or memory, or a node selector or toleration may not match any node

Below it is the scheduler’s own message, such as 0/12 nodes are available: 12 Insufficient nvidia.com/gpu. It names what ran out. Then:

  • If the step asks for more CPU, memory or GPUs than any node has, lower the request.
  • If its nodeSelector, tolerations or affinity match no node, fix them.
  • If the cluster is scaling up, wait. The step starts once a node is free.

A pod that can’t be scheduled waits forever by default. Set pendingTimeout in the step’s Scheduling & Retries section and Argo fails the step after that long. It needs Argo Workflows 4.1, and the inspector warns you if the cluster’s Argo is too old.

Dagweave doesn’t name this one. A pod stuck on ImagePullBackOff or ErrImagePull shows as a Pending step with whatever message Argo copied from the pod.

A slow pull on a new node clears by itself. A backoff doesn’t. Usually the tag is wrong, the image was never pushed, or the namespace has no credentials for a private registry. Check the tag first.

A limit is often holding it back, and Dagweave doesn’t name these:

  • Parallelism. A workflow with parallelism set runs that many pods at once. 500 items at 10 go through ten at a time. It’s under Parallelism (pods) in the workflow’s Concurrency settings.
  • Controller limits. The workflow controller can cap workflows per namespace. Nothing in your workflow changes that, so ask whoever runs the controller.
  • Approvals. An approval step is an Argo suspend. Dagweave has no resume button, so use argo resume or the Argo UI.

If every run stopped updating at once, check the connector first. It shows as offline in Clusters after 300 seconds without a heartbeat, and teams with connector-offline alerts get one message per outage.

Runs keep going on the cluster meanwhile. When the connector reconnects, the run view catches up. Admins and owners can read its logs from its row in Clusters.

Limit Where to set it What it caps
Pending timeout (pendingTimeout) The step’s Scheduling & Retries section How long a step’s pod can stay Pending. Use it for scheduling and image problems.
Timeout (timeout) The step’s Scheduling & Retries section The step’s total time, counted from when Argo creates it, so time spent Pending counts too. Use it for work that hangs.
Workflow timeout (activeDeadlineSeconds) The workflow’s Lifecycle settings How long the whole run can take, whichever step is slow.

They’re all standard Argo fields, so they stay in the YAML you export.