Factories > Guides

Testing and tuning your factory

Evaluate your factory's output and cost with small test changes, then tune one setting at a time before expanding its workload.

Test your factory with small changes before expanding its workload. Evaluate the output and cost, adjust one setting, and repeat the test.

The example adds a “Copy link” button. Adapt it to your repository. The approval and review checks are part of this test, not automatic factory behavior.

Prerequisites

A factory connected to your repository, with spec, implement, and review agents available to its foreman.
A runner with your project’s toolchain and access to the repository’s documented checks.
Permission to edit the factory.

Test with a small change

Choose changes from your own backlog.

Choose two or three changes, each small enough for one pull request. Use agreed requirements or reproducible bugs; resolve major product questions before testing.
Record the starting commit, current behavior, requested behavior, and acceptance criteria for the first change.

For example, if your app has a share dialog with a URL field, request a “Copy link” button. Require it to copy the displayed URL, show confirmation for two seconds, and work with the keyboard. Ask the spec to resolve copy failures and accessibility behavior before implementation.
In the Warp Factories web app, click New Chat and choose your factory below the prompt input. Send the test prompt, adapted to your change.

Check for a draft pull request containing the spec and a request for your approval. Implementation must wait.
Review the committed spec. Ask for revisions or edit its branch if needed.
When the spec is ready, explicitly approve the current version in the same conversation. Check that implementation reuses the spec pull request, runs the requested checks, and returns review findings and any questions for you. Do not merge test output.

Evaluate the run

Read the runs

In the web app sidebar, click Runs. Related agent runs group under their parent.
Check that the spec preceded implementation and that implementation waited for your approval. Other activity, such as answering a question, is not itself an approval failure.
Open the runs to inspect their timelines, cost, and sessions. Check which files the spec agent read, whether implementation ran the reported checks, and whether review examined the diff.

Check the outputs

Spec

Describes behavior, scope, assumptions, and decisions needing your approval.
Gives an executable check for each acceptance criterion.
Scales to the change rather than prescribing unnecessary implementation detail.

For the button example, “copies the URL” needs a test of the exact copied value. “Works with the keyboard” needs evidence of keyboard activation, not only a screenshot.

Pull request

Uses the approved spec’s branch and pull request.
Meets the acceptance criteria without unrelated changes.
Lists actual verification commands and results, including visual evidence for UI behavior.

Review

Findings identify the problem, location, severity, and correction.
Defects are distinguished from product decisions requiring your judgment.

For example, a missing assertion for a required fallback is a test gap. Choosing behavior the spec never defined needs a product decision, not an automatic correction.

Handoff

The foreman returns the pull request, verification evidence, review summary, and remaining questions.
The pull request remains unmerged.

Check cost and quality

Record each agent run’s cost alongside its timeline. Compare the same stage across trials.
On the factory’s Dashboard page, compare Cost per PR and Most expensive PRs across runs. These are comparison metrics, not invoice totals. See factory cost metrics.
Record gaps against your team’s standards for scope, tests, and description quality.

A Scorer classifies completed runs against criteria you write, such as whether the agent ran tests before opening the pull request. Try it on a run from its run page and compare its reasoning with your own review before using sampled scoring.

Tune one setting

Choose one adjustment based on the results. Test prompt and model changes separately.

Observation	Adjustment to test	Agent
Implementation made an unresolved product decision	Require the spec to surface that decision for approval	Spec
The spec is too detailed	Shorten its template for small changes	Spec
Implementation costs too much	Compare another supported model on the same task	Implement
Review reports irrelevant style findings	Narrow the blocking criteria without removing correctness checks	Review
Review misses defects	Add the missed failure mode to the instructions	Review
An agent skips a required check	Name the repository’s exact check command	The agent that skipped it

In your factory’s Agents page, open the relevant agent. For a Warp-managed factory, edit its Prompt or “Model” setting and click Save. For a GitHub-backed factory, edit that agent’s existing agents/<name>/agent.md or its attached skill in the definition repository.
Apply a definition-file change by saving Factory definition for a Warp-managed factory, or merging the definition pull request into its production branch for a GitHub-backed factory. Confirm validation passes. See where the definition lives.

Change one agent’s model, not agentDefaults.model, which also affects agents inheriting that default. For model comparisons, use a benchmark before changing production; a completed run’s page can supply a benchmark task.
Start a fresh trial from New Chat using the same requirements and recorded starting commit. Ask for a new test branch and pull request, with no commits from the earlier trial. If you track tests in issues, use a separate test issue rather than continuing the old work item.
Review the new spec and approve it separately. Compare outputs and the same stage’s cost with the baseline.

Closing a pull request or reapplying a trigger label is not a reset of an existing work item. Check the new branch’s starting commit and diff before treating trials as comparable.

Repeat, then widen the scope

Repeat a test with unchanged settings before attributing a difference to tuning. Two or three small changes are a starting sample, not proof of reliability. Expand only when the results consistently include:

A spec you can approve without edits.
An implementation meeting the acceptance criteria.
Review findings that match the checks you would make yourself.

Expand one dimension at a time: task size, task type, or intake source. A reproducible bug, review-only request, and documentation change may need different agents; see factory use cases.

As volume grows, use sampled Scorers and the practical improvement loop, while keeping human review.

Start a test with a prompt

Replace the uppercase placeholders with your repository, starting commit, change, and checks.
Send the prompt to your foreman through New Chat or a coding agent connected through Factory MCP.

Run a test change in OWNER/REPO from BASE_COMMIT.
Requested change: CHANGE_DESCRIPTION.
Acceptance criteria: ACCEPTANCE_CRITERIA.
Documented checks: CHECK_COMMANDS.

Use a new test branch and pull request, without changes from earlier trials.
Have the spec agent write a short spec in a draft pull request. Ask me about
unresolved behavior, name a check for each acceptance criterion, and stop
for my explicit approval. Do not start implementation before I approve
this spec, even if a previous test was approved.

After approval, implement on the spec's branch and pull request, run the
documented checks, and run one review pass. Return the pull request,
verification results, and unresolved questions. Never merge. Report a
missing tool or permission rather than claiming an unrun check passed.

Troubleshooting

Implementation starts before approval. The test depends on the foreman’s instructions. Stop the run, check its prompt and any overlapping automations, and restore the approval instruction before retrying. See one action starts two runs.

The new trial continues earlier work. A follow-up retains context. Start through New Chat, remove references to the earlier work item, and verify the requested baseline and new branch before approving.

Results vary with unchanged settings. Repeat the test before tuning. For model comparisons, use a benchmark with repetitions and review individual trials.

A run costs more than expected. Inspect cost by stage and look for repeated validation or revision rounds. Test one adjustment rather than assuming the model caused the increase.

Previous stage: Configure your factory - Revisit the complementary setup options
Next stage: Connect work intake through the tools your team uses:
All stages: Factory guides

Configuring factory agents - Change one agent’s instructions or model
Measure and improve a factory - Move from manual checks to sampled evaluation
Factory benchmarks - Compare configurations on fixed tasks