Compare prompts, models, and product variants on real outcomes. $0.03 per 1,000 remote evaluations, per 64 KiB of response. The base URL is https://api.motor.dev/prompt_flags, and the OpenAPI 3.1 specification describes each endpoint.
Examples
Compare two support prompts
Compare two versions of a support-answer prompt. Keep immutable templates named support-v1 and support-v2 in your application: v1 answers from supplied help articles; v2 also asks for article citations and a clarifying question when the articles do not answer the request. Hold the model, retrieval inputs, and generation settings constant so the experiment measures the prompt change.
Each conversation gets one prompt version. The experiment measures whether it’s resolved within an hour of its first prompt use. Set MOTOR_KEY to a secret key. Responses are trimmed; replace IDs, ETags, dates, and receipts with real values. Save each write’s idempotency key and reuse it on retries. Use an environment without required approvals for setup; protected environments require change sets.
Create the prompt flag
curl -i "https://api.motor.dev/prompt_flags/projects" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H "Content-Type: application/json" \
-d '{"key": "support", "name": "Support prompts"}'
HTTP 201
{
"id": "ffp_...",
"key": "support"
}
curl -i "https://api.motor.dev/prompt_flags/environments" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H "Content-Type: application/json" \
-d '{"project_id": "ffp_...", "key": "staging", "name": "Staging"}'
HTTP 201
ETag: "ff:ffe_...:1"
{
"id": "ffe_...",
"revision": "1"
}
curl -i "https://api.motor.dev/prompt_flags/environments/ffe_.../flags" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H 'If-Match: "ff:ffe_...:1"' \
-H "Content-Type: application/json" \
-d '{"key": "support_prompt", "name": "Support prompt", "type": "string", "purpose": "experiment", "enabled": true, "variations": [{"key": "control", "value": "support-v1"}, {"key": "candidate", "value": "support-v2"}], "off_variation": "control", "fallthrough": {"variation": "control"}}'
HTTP 201
ETag: "ff:ffe_...:2"
{
"id": "ffl_...",
"key": "support_prompt",
"revision": "2"
}
Your backend loads the template version returned by the flag. Use the control template when the flag is disabled or evaluation fails.
Define the outcome and experiment
curl -i "https://api.motor.dev/prompt_flags/environments/ffe_.../metrics" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H "Content-Type: application/json" \
-d '{"key": "conversation_resolved", "name": "Conversation resolved", "kind": "binary", "aggregation": "any", "direction": "increase"}'
HTTP 201
{
"id": "ffm_...",
"key": "conversation_resolved",
"version": "1"
}
Set the start and end dates for accepting new conversations. Half use support-v1 (control), and half use support-v2 (candidate). The API expresses this split as two weights of 50,000. Meeting the minimum sample size and duration doesn’t guarantee a conclusive result.
curl -i "https://api.motor.dev/prompt_flags/environments/ffe_.../experiments" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H "Content-Type: application/json" \
-d '{"key": "support_prompt_test", "name": "Support prompt test", "flag_key": "support_prompt", "control": "control", "weights": [{"variation": "control", "weight": 50000}, {"variation": "candidate", "weight": 50000}], "bucket_by": "targeting_key", "seed": "support-prompt-v2", "metrics": ["ffm_..."], "primary_metric_id": "ffm_...", "enrollment": {"start_at": "2026-11-01T00:00:00Z", "end_at": "2026-12-01T00:00:00Z", "max_subjects": 10000}, "attribution_seconds": 3600, "analysis": {"mode": "sequential", "alpha": 0.05, "minimum_subjects": 100, "minimum_duration_seconds": 3600}}'
HTTP 201
ETag: "ffr:ffx_...:1"
{
"id": "ffx_...",
"status": "draft",
"version": "1"
}
curl -i "https://api.motor.dev/prompt_flags/environments/ffe_..." \
-H "Authorization: Bearer $MOTOR_KEY"
HTTP 200
ETag: "ff:ffe_...:2"
{
"id": "ffe_...",
"revision": "2"
}
curl -i -X PATCH "https://api.motor.dev/prompt_flags/environments/ffe_.../flags/ffl_..." \
-H "Authorization: Bearer $MOTOR_KEY" \
-H 'If-Match: "ff:ffe_...:2"' \
-H "Content-Type: application/json" \
-d '{"fallthrough": {"experiment": "support_prompt_test", "fallback_variation": "control"}}'
HTTP 200
{
"id": "ffl_...",
"revision": "3"
}
curl -i -X POST "https://api.motor.dev/prompt_flags/environments/ffe_.../experiments/ffx_.../start" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H 'If-Match: "ffr:ffx_...:1"' \
-H "Content-Type: application/json" -d '{}'
HTTP 200
{
"id": "ffx_...",
"status": "running",
"version": "2"
}
Assign and use a prompt
Evaluate before the first answer in a conversation. Reuse the same targeting_key throughout that conversation. Set record_exposure to false because a successful evaluation does not mean the model call happened. Evaluation requests do not accept an Idempotency-Key.
curl -i "https://api.motor.dev/prompt_flags/evaluations" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Content-Type: application/json" \
-d '{"environment_id": "ffe_...", "context": {"targeting_key": "conversation-42"}, "record_exposure": false, "flags": [{"key": "support_prompt", "type": "string", "default": "support-v1"}]}'
HTTP 200
{
"results": [
{
"key": "support_prompt",
"value": "support-v2",
"variation": "candidate",
"reason": "SPLIT",
"error_code": null,
"experiment": {
"assignment_receipt": "motor_fa_..."
}
}
]
}
Load the returned template and call your model with it. Keep that template and receipt with the conversation. If evaluation fails, use support-v1 locally. Without a valid assignment receipt, don’t report the conversation as part of the experiment.
Send an exposure event when you use the prompt. Send a metric event when the conversation is resolved (1) or ends unresolved (0). Use distinct lowercase UUIDs, retain them across retries, and replace the timestamps with actual event times. Report each event within 24 hours of when it happened. A resolution counts only if it happens within one hour of the conversation’s first prompt use.
curl -i "https://api.motor.dev/prompt_flags/events" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H "Content-Type: application/json" \
-d '{"events": [{"id": "319d506a-f282-4c7b-a956-3fdd4b55d650", "environment_id": "ffe_...", "occurred_at": "2026-11-01T12:00:00Z", "type": "exposure", "assignment_receipt": "motor_fa_..."}, {"id": "805786c1-ab10-4a9b-80bf-e7ae53b8499b", "environment_id": "ffe_...", "occurred_at": "2026-11-01T12:10:00Z", "type": "metric", "metric_key": "conversation_resolved", "value": 1, "assignment_receipt": "motor_fa_..."}]}'
HTTP 200
{
"accepted": 2,
"duplicates": 0,
"unattributed": 0
}
Read the results
curl -i "https://api.motor.dev/prompt_flags/environments/ffe_.../experiments/ffx_.../results" \
-H "Authorization: Bearer $MOTOR_KEY"
HTTP 200
{
"experiment_id": "ffx_...",
"status": "insufficient_data",
"reasons": [
"minimum_subjects",
"minimum_duration"
]
}
Collect outcomes for both prompt versions. Motor waits 25 hours after a conversation’s first prompt use before including it in the results: one hour for resolution, then 24 hours for late reports. Check status and reasons in the response. If status is insufficient_data, keep collecting outcomes. The comparisons array reports improvement, harm, or inconclusive for each metric. Check any allocation_mismatch warning before using the results.
Switch to the new prompt
Use this example if the experiment reports an improvement for support-v2 and your review of its answers supports switching to it. Otherwise, keep support-v1 or continue the experiment. The calls below propose and apply a configuration change so future evaluations return support-v2. Creating the proposal alone doesn’t switch the prompt.
curl -i "https://api.motor.dev/prompt_flags/environments/ffe_.../experiments/ffx_..." \
-H "Authorization: Bearer $MOTOR_KEY"
HTTP 200
ETag: "ffr:ffx_...:2"
{
"id": "ffx_...",
"status": "running",
"version": "2"
}
curl -i "https://api.motor.dev/prompt_flags/environments/ffe_.../experiments/ffx_.../conclude" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H 'If-Match: "ffr:ffx_...:2"' \
-H "Content-Type: application/json" \
-d '{"variation": "candidate", "comment": "Reviewed resolution outcomes and sampled answers; select support-v2"}'
HTTP 201
{
"id": "ffc_...",
"status": "draft",
"source": "experiment",
"version": "1"
}
Read the proposed change and save its ETag. The example’s environment doesn’t require reviewers, so submitting the change approves it automatically. If your environment requires approvals, submission returns pending; wait for the required reviewers to approve it before applying it.
curl -i "https://api.motor.dev/prompt_flags/environments/ffe_.../changes/ffc_..." \
-H "Authorization: Bearer $MOTOR_KEY"
HTTP 200
ETag: "ffr:ffc_...:1"
{
"id": "ffc_...",
"status": "draft",
"version": "1"
}
curl -i -X POST "https://api.motor.dev/prompt_flags/environments/ffe_.../changes/ffc_.../submit" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H 'If-Match: "ffr:ffc_...:1"' \
-H "Content-Type: application/json" -d '{}'
HTTP 200
{
"id": "ffc_...",
"status": "approved",
"version": "2"
}
curl -i "https://api.motor.dev/prompt_flags/environments/ffe_.../changes/ffc_..." \
-H "Authorization: Bearer $MOTOR_KEY"
HTTP 200
ETag: "ffr:ffc_...:2"
{
"id": "ffc_...",
"status": "approved",
"version": "2"
}
curl -i -X POST "https://api.motor.dev/prompt_flags/environments/ffe_.../changes/ffc_.../apply" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H 'If-Match: "ffr:ffc_...:2"' \
-H "Content-Type: application/json" -d '{}'
HTTP 200
{
"id": "ffc_...",
"status": "applied",
"applied_revision": "5"
}
After you apply the change, evaluations that previously used this experiment return the selected prompt version. An environment revision conflict means someone changed the configuration: inspect the new state and prepare a fresh proposal. Before archiving the experiment, save its results and check that your application no longer depends on it.
Switch back to the original prompt
If sampled answers reveal a regression, an environment admin can disable the prompt flag without waiting for approvals. Because off_variation is control, future evaluations return support-v1. Decide how your application handles conversations already using support-v2; this call cannot cancel model calls or rewrite cached choices.
curl -i "https://api.motor.dev/prompt_flags/environments/ffe_.../flags/ffl_.../disable" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H "Content-Type: application/json" \
-d '{"comment": "Return new conversations to support-v1 while reviewing answer quality"}'
HTTP 200
{
"id": "ffl_...",
"key": "support_prompt",
"enabled": false,
"revision": "6"
}
curl -i "https://api.motor.dev/prompt_flags/evaluations" \
-H "Authorization: Bearer $MOTOR_KEY" \
-H "Content-Type: application/json" \
-d '{"environment_id": "ffe_...", "context": {"targeting_key": "conversation-99"}, "flags": [{"key": "support_prompt", "type": "string", "default": "support-v1"}]}'
HTTP 200
{
"results": [
{
"key": "support_prompt",
"value": "support-v1",
"variation": "control",
"reason": "DISABLED",
"error_code": null
}
]
}
Stop counting baseline fallback traffic as exposure to the candidate. Investigate the regression, create a new immutable prompt version, and use a new experiment for the next comparison.