Skip to content

Test with real data

Fill a test run’s parameters from a real record instead of typing sample JSON. Pick a row from a Postgres table or an object in an S3 bucket, and Dagweave copies its fields into the sample.

You need the connector and a Production or Business plan. Test runs don’t count toward runs a month, but they do count toward a limit on runs at once if your team has one.

  1. Open Test with sample data and expand Advanced: use a sample record.
  2. Switch to the From a connected source tab.
  3. Choose a source. Click Load more to see the next page of records.
  4. Select a record and click Use this item.

The record’s fields land in the Paste a sample tab as JSON. Fields whose names match an input parameter fill that parameter. Edit the sample if you want, then run the test.

Source Record fields
Postgres The columns listed in preview_columns in the source’s Secret
S3 The object’s key, size, last_modified, content_type and etag

Dagweave never reads an S3 object’s contents. Only sources whose connector is online show up.

You do this once per source. You need access to the connector’s namespace, and you need to be on the team that owns the connector.

It’s off by default:

Terminal window
helm upgrade connector oci://ghcr.io/dagweave/charts/connector \
--namespace connector-system --reset-then-reuse-values \
--set features.sources=true

This adds a Role that can only list Secrets, but it can list every Secret in the connector’s namespace, since RBAC can’t filter by label. The connector reads only the ones with a source label.

Open the gear menu, Clusters, and click Add data source under Data sources. Give it a name, pick Postgres table or S3 bucket, and pick the cluster whose connector reads it. The test panel’s Add a data source link opens the same form.

The new row shows the label its Secret needs, sources.connector.dagweave.io/id=<source id>, with a Copy label button and the kubectl label command to run. Remove takes the source out of Dagweave and leaves the Secret in your cluster.

To script it, send POST /api/v1/sources with an API token that has clusters:write, or have an assistant call dagweave_create_source over MCP. Both take the connector’s ID, a name, and a type of postgres or s3, and answer with the same label:

{ "connectorId": "<connector id>", "name": "orders", "type": "postgres" }

In the connector’s namespace, create a Secret with any name, holding the connection details and a read-only credential. Add the label from step 2.

For Postgres:

Terminal window
kubectl create secret generic orders-db -n connector-system \
--from-literal=host=db.internal \
--from-literal=database=app \
--from-literal=username=dagweave_source \
--from-literal=password='...' \
--from-literal=table=orders \
--from-literal=id_column=id \
--from-literal=preview_columns=id,customer,total
kubectl label secret orders-db -n connector-system \
sources.connector.dagweave.io/id=<source id>
Key Notes
host, database, username, password Connection details
port Defaults to 5432
sslmode Defaults to require
table One table, optionally schema.table
id_column The column a record is fetched by. It must also be in preview_columns
preview_columns Comma-separated. These are the only columns Dagweave reads

For S3, bucket is required. Optional keys are prefix, region, endpoint (for S3-compatible stores; leave empty for AWS), access_key_id, secret_access_key, session_token, insecure and addressing_style.

Scope the credential to the one table or bucket. For Postgres, SELECT on that table. For S3, s3:ListBucket and s3:GetObject on that bucket. The metadata read is a HEAD request, which needs s3:GetObject.

Dagweave stores the source’s ID, name, type and connector, nothing more. The credential and the Secret’s name never leave your cluster.

The connector only reads:

  • Postgres queries run in read-only transactions and select only the ID and preview columns. Nobody writes SQL, and the record ID is a bound parameter.
  • S3 reads list objects and read metadata. Keys outside the configured prefix are rejected.

Dagweave doesn’t store the records you browse. A test run saves only the parameter values, as with a pasted sample. Fields that don’t match a parameter aren’t sent.

The audit log records every list and fetch with who, the source, the connector and the time. Fetches also record the row’s ID or the S3 object key.