API stability and the OpenAPI snapshots
The v1 API is a contract. Two checked-in OpenAPI documents pin it, and a smoke test fails the build when a route that is part of the contract changes shape.
| File | What it is | Generated by |
|---|---|---|
docs/api/openapi-v1.stable.json |
The core API: every route a build with modules/ deleted serves, with components. |
python -m backend.scripts.dump_openapi --profile core --full … |
docs/api/openapi-v1.full.json |
The full API: the core routes plus the ones the modules registration mounts. | python -m backend.scripts.dump_openapi --profile full --full … |
frontend/src/tests/fixtures/openapi.json |
Paths and methods only, for the dashboard's URL-literal guard (frontend/src/tests/services/url-literals.test.ts). A full dump: the guard subtracts openapi.module-paths.json from it for a core run, so a core dump here would make it subtract them twice. |
python -m backend.scripts.dump_openapi --profile full … |
make openapi regenerates all three. --profile core hides the modules
package from the import system before the application is imported, so the core
document can be produced from a full checkout; it is byte-identical to the
document a tree with modules/ deleted produces (scripts/core_build.sh
checks that tree the long way round). --profile full requires the modules
registration to load and fails otherwise.
The rule
backend/tests/smoke/test_openapi_snapshot.py regenerates the live documents
in a fresh interpreter and compares them with the snapshots:
- A path/method in the snapshot that is stable must still exist, and its operation object — parameters, request body, responses, and every component schema those reference, transitively — must be identical. Any difference fails the smoke suite.
- A path/method marked beta may change or disappear: the test warns instead.
- A path/method that exists but is not in the snapshot yet is a warning too:
run
make openapi, commit the result, and decide whether it is beta. openapi-v1.full.jsonmust be a superset ofopenapi-v1.stable.json, and the module-only paths must be exactly the list infrontend/src/tests/fixtures/openapi.module-paths.json, so the three fixtures cannot drift apart. That fixture is maintained by hand.
Every route is stable by default. Stability is a promise about the shape of the route — its URL, parameters, request and response schemas — not about its behaviour.
Marking a route beta
Stability is declared on the route, in the OpenAPI document itself, with the
x-stability extension, so the snapshot carries it and the test can read it
without a side table:
@router.post(
"/experiments/{experiment_id}/something-new",
response_model=SomethingNewResponse,
openapi_extra={"x-stability": "beta"},
)
def something_new(...):
...
openapi_extra is merged into the operation object by FastAPI, so the
snapshot shows "x-stability": "beta" on that method. Only the value beta
means anything; leave the key off a stable route. Promoting a route is deleting
the line and regenerating the snapshots.
Use beta for a route whose request or response is still being shaped by real use — a new analysis endpoint, a first cut of a bulk operation — and take it off once a released SDK or the dashboard depends on it.
x-stability and analysis_status are different promises
x-stability is about the shape of a route. An analysis response's
analysis_status field (ga or beta, with an analysis_notice exactly when
it is beta) is about its numbers: beta there means part of the analysis
is not yet computed as its fields describe, and the notice says which part and
links the issue. The values come from one table,
backend/app/core/analysis_status.py. A route whose numbers are beta is also
marked x-stability: beta (the interaction-pair route, #219), because fixing the
numbers will change its response. The CUPED route's numbers are ga (#217), and
the route stays x-stability: beta while its response is still being shaped
(mean metrics, #439);
/interactions/scan stays stable, and its items keep their shape with the
interaction, novelty and SUTVA sub-results null: the scan reports overlap only,
and novelty and SUTVA are not offered.
GET /api/v1/holdout/{holdout_id}/results is beta in both senses (#445): its
numbers carry analysis_status: "beta" and the route is x-stability: beta.
POST /api/v1/results/{id}/post-stratification is beta as well (#443): it
answers 501 until it computes its estimate from an experiment's own
assignments and events, rather than report numbers that do not.
Changing a stable route
Do not. Add a new route (mark it beta while it settles) or bump the API
version. If a change to a stable route is unavoidable — a security fix that
has to remove a field, say — regenerate the snapshots with make openapi,
and say in the pull request which stable operations changed and why; the
smoke test's failure message lists them by method and path, with the first
differing keys.
Stable operations changed this way:
| Operations | Changed by | What changed |
|---|---|---|
POST /api/v1/feature-flags/, GET /api/v1/feature-flags/, GET/PUT /api/v1/feature-flags/{flag_id}, POST /api/v1/feature-flags/{flag_id}/activate, POST /api/v1/feature-flags/{flag_id}/deactivate |
#94 | The request schemas refuse unknown fields and explicit nulls, a new flag is off unless is_active: true is sent, and default_value accepts only false. Every one of these operations answers with one documented flag schema, FeatureFlagRead, which names targeting targeting_rules and no longer carries rules, variants, metrics or last_evaluated. See Creating Feature Flags. |
POST /api/v1/safety/feature-flags/{feature_flag_id}/config, POST /api/v1/safety/feature-flags/{feature_flag_id}/rollback |
#629 | The config request's rollback_percentage and the rollback's percentage query parameter accept 0–100; any other value answers 422. A value stored before the change still reads back unchanged, and the safety monitor clamps it to 0–100 when it rolls the flag back. See Safety Monitoring. |
POST /api/v1/segments, PUT /api/v1/segments/{segment_id}, POST /api/v1/segments/{segment_id}/preview, POST /api/v1/segments/{segment_id}/evaluate, GET/DELETE /api/v1/segments/{segment_id}, GET /api/v1/segments/{segment_id}/experiments, POST /api/v1/segments/bulk-evaluate |
#440 | Segment rules must be in the targeting rule format that flags and experiments use ({"logical_operator", "groups": [{"conditions": [...]}]}, operators such as equals), with at least one group and at most 20 groups, 50 conditions, 10 regex conditions and 1,000 list values; the old {"operator", "conditions"} format and anything else answer 422 on create, update and preview. Evaluate answers 409 for a segment stored in the old format, and bulk-evaluate answers false for it; stored segments are left as they are. {segment_id} is a UUID, and other text answers 422. See Upgrading: segment rules. |
POST /api/v1/segments, GET /api/v1/segments, GET/PUT /api/v1/segments/{segment_id}, POST /api/v1/segments/{segment_id}/evaluate, POST /api/v1/segments/bulk-evaluate, POST /api/v1/segments/{segment_id}/preview |
#440 (id lists) | A segment has a kind, rules (the default) or id_list, set on create and never changed. rules is optional on create: required for a rules segment and refused for an id list (422), so a create that leaves kind out still needs rules as before. Every segment in a response carries kind; an id list's rules is null, and GET /api/v1/segments/{segment_id} fills member_count for it. A PUT that sends kind, or rules for an id list, answers 422. Evaluate and bulk-evaluate answer an id list on user_context.user_id. Preview takes the same body and answers 422 when it has no rules. Segments stored before the upgrade are rules segments. The member routes POST /api/v1/segments/{segment_id}/members and /members/remove are new and beta. See Segments made from a list of user IDs. |
DELETE /api/v1/segments/{segment_id}, PUT /api/v1/segments/{segment_id}, POST /api/v1/feature-flags/, PUT /api/v1/feature-flags/{flag_id}, POST /api/v1/experiments/, PUT /api/v1/experiments/{experiment_id}, POST /api/v1/experiments/{experiment_id}/clone |
#440 (segment targeting) | Targeting rules accept in_segment and not_in_segment (attribute segment, the value a segment id), at most 10 segments per ruleset. Flag and experiment create and update answer 422 at ["body", "targeting_rules"] for a segment that is unknown, inactive or archived, or whose rules are not valid, and for a segment condition in a native default_rule; clone answers 409 when the rules it would copy name such a segment. Archiving a segment, or a PUT that sets its status to inactive or archived, answers 409 with {"detail": {"code": "segment_in_use", "message", "feature_flags", "experiments"}} while a flag that is not archived or a draft, active or paused experiment uses it. See Targeting Rules. |
POST /api/v1/tracking/track, POST /api/v1/tracking/batch |
#217 | experiment_key and feature_flag_key are both optional. An event with neither (missing, null or "") is stored with experiment_id, feature_flag_id and variant_id null, where it used to answer 422 (and one such entry failed a whole batch with 422). Such an event is history and counts in no experiment's results; tag outcome events. A key that is given and not found still answers 404, or is reported in the batch's errors; the detail names only the keys that were sent. The OpenAPI document does not change. See Track Event. |
GET /api/v1/interactions/scan |
#853 | When reading the active experiments or their assignments fails, the scan answers 500 with a fixed message and the request ID. It used to answer 200 as if nothing were running: total_active_experiments: 0, or the pairs whose assignments could not be read left out. A scan whose reads succeed answers as before. The OpenAPI document does not change. |
GET /api/v1/results/{experiment_id}, GET /api/v1/experiments/{experiment_id}/results, GET /api/v1/export/experiments, GET /api/v1/export/reports/experiments/{experiment_id} |
#880 | When the sample-ratio check fails (srm is present and srm.warning is true), summary.recommendation is INCONCLUSIVE, whatever it would otherwise be, CONTINUE_TESTING included, and recommendation_reason names the mismatch and its p-value; the export's recommendation column says the same. has_winner, winning_variant_id, the Bayesian decision and the sequential results are unchanged. When srm is null (a bandit, fewer than two variants, no assignments, or a check that could not run) the recommendation is unchanged. The OpenAPI document does not change. |
DELETE /api/v1/feature-flags/{flag_id} |
#855 | Deleting a flag keeps the events tagged with it: they stay, with feature_flag_id null and every other field unchanged, an experiment tag included. They used to be deleted with the flag. Everything else the delete removes is unchanged, and the response is still 204. The OpenAPI document changes only in the operation's description. |
GET /api/v1/holdout/all, POST /api/v1/holdout, PUT /api/v1/holdout/{holdout_id}, DELETE /api/v1/mutual-exclusion-groups/{group_id}, PUT /api/v1/mutual-exclusion-groups/{group_id} |
#904 | These operations accept the roles their documentation states. Listing, creating and updating holdouts and archiving a group with DELETE need ADMIN or a superuser; a DEVELOPER now answers 403 on them. Changing a group's status with PUT (archiving or unarchiving) needs ADMIN too, as archiving does; a DEVELOPER may still change a group's name, description and traffic allocation, and may send its current status back. ADMIN and superusers are unchanged. The OpenAPI document changes only in the description of PUT /api/v1/mutual-exclusion-groups/{group_id}. See Mutual Exclusion Groups & Global Holdout. |
GET /api/v1/results/{experiment_id}/sequential |
#854 | The confidence sequence and can_stop use the Agresti-Caffo variance of the difference in rates instead of the plug-in one, so the interval keeps its coverage when the arms are split unequally. Some experiments that showed can_stop: true and recommended_action: "stop_for_effect" (mostly with one small, low-rate arm) now show false and "continue", a few at equal splits stop slightly later, and a very few, mostly with rates near 50%, stop slightly earlier; an arm in which every user has the same outcome now gets a finite interval instead of [-1, 1]. lambda_ratio is capped at the largest finite double (about 1.8e308), so an overwhelming difference reports a finite number and can stop. The response's fields are unchanged and analysis_status stays beta. The engine version becomes 1.4.0; the OpenAPI document changes only in the engine_version default of the responses that carry it (bandit, Bayesian, variance reduction). See Sequential Testing. |
POST /api/v1/tracking/assign, POST /api/v1/tracking/assign/batch |
#822 | Experiment targeting treats a missing attribute as flags do. A new user whose context lacks an attribute is no longer refused outright (assigned: false, reason: "targeting"): they fail only the conditions on that attribute, and is_null on it passes. So a user who matches one OR branch is enrolled even when they lack an attribute used only in another branch, and a user without an attribute now matches a NOT group on it (a user with no country matches NOT (country equals US), but still not country not_equals US). AND still needs every condition to hold. Users already assigned keep their assignment. The OpenAPI document does not change. See Targeting Rules. |
GET /api/v1/results/{experiment_id}, GET /api/v1/experiments/{experiment_id}/results |
#922 | When the experiment has sequential testing on, sequential_testing carries the analysis GET /api/v1/results/{experiment_id}/sequential serves at the experiment's stored alpha (the primary metric, the control against the first treatment); it used to be null always. It stays null when sequential testing is off, and is null when the analysis could not be computed (the API logs a warning); the rest of the response is unaffected. It is cached with the results, so an answer cached before the upgrade keeps null until it expires (5 minutes for a running experiment, 24 hours otherwise) or POST /api/v1/results/{experiment_id}/invalidate-cache clears it. For the same reason a change to sequential_testing_enabled or sequential_testing_config, which is accepted on a running experiment, shows on these routes when the cached answer expires or is cleared, not immediately. The OpenAPI document does not change. See Sequential Testing. |
GET /api/v1/results/{experiment_id}/sequential, GET /api/v1/results/{experiment_id}, GET /api/v1/experiments/{experiment_id}/results |
#929 | The sequential analysis compares the control against the treatment created first (created_at, then id), on every call. It used to take the treatment the database returned first, which on an experiment with three or more variants could change between two calls with no change to the data, so the analysis could switch arms. Two-variant experiments are unchanged, and so are the numbers for a given pair of arms. The results routes' cache is not keyed on the arm, so an answer cached before the upgrade is served until it expires (5 minutes for a running experiment, 24 hours otherwise) or POST /api/v1/results/{experiment_id}/invalidate-cache clears it. The OpenAPI document does not change. See Sequential Testing. |
GET /api/v1/experiments/, POST /api/v1/experiments/, GET/PUT /api/v1/experiments/{experiment_id}, POST /api/v1/experiments/{experiment_id}/start, POST /api/v1/experiments/{experiment_id}/pause, PUT /api/v1/experiments/{experiment_id}/schedule, POST /api/v1/experiments/{experiment_id}/complete, POST /api/v1/experiments/{experiment_id}/archive, POST /api/v1/experiments/{experiment_id}/clone |
#935 | An experiment whose creator's account was removed still reads and lists, with owner_id: null; every one of these operations answers with the same ExperimentResponse, whose owner_id is now nullable and no longer required. An experiment whose creator still exists answers as before, with the creator's id. The dashboard shows such an experiment's owner as "No owner". The OpenAPI document changes only in ExperimentResponse: owner_id becomes nullable and leaves the schema's required list. |
GET /api/v1/experiments/{experiment_id} |
#921 | The response carries owner_name, the name of the user who created the experiment: their full name, else their username when it contains no @, else null. It is never their email, and it is null for an experiment with no owner (owner_id: null). It is looked up on every read and is not cached, so a change of name shows on the next read. Every other field is unchanged, and the list and the other experiment routes do not carry it. The dashboard shows it as the owner of an experiment someone else created. The OpenAPI document changes only in this operation's response schema, which becomes ExperimentDetailResponse (ExperimentResponse plus owner_name). |
DELETE /api/v1/users/{user_id}, DELETE /api/v1/admin/users/{user_id} |
#946 | Removing a user's account keeps the segments and rollout schedules it created, with no owner: the segments are still listed and read, and each rollout schedule still reads with all of its stages and owner_id: null. Its API keys are still removed with it, and the response is still 204. The OpenAPI document does not change. |
GET /api/v1/integrations, POST /api/v1/integrations, GET/PUT /api/v1/integrations/{integration_type} |
#968 | These operations no longer return the values of stored secrets: encrypted_config carries only base_url, email, project_key, instance_url, client_id, repo_owner and repo_name, and every other stored key is named in the new stored_secrets list. PUT merges encrypted_config key by key: a key sent with a value replaces it, a key sent as null removes it, and a key not sent is kept (it used to replace the whole object). The OpenAPI document changes in IntegrationConfigResponse (adds stored_secrets) and in the PUT description. See What a response shows. |
POST/GET /api/v1/safety/feature-flags/{feature_flag_id}/config, GET /api/v1/safety/feature-flags/{feature_flag_id}/check |
#954 | The config request refuses null for enabled, metrics and rollback_percentage with 422 ("type": "null_not_allowed" at ["body", <field>]) and writes nothing. A null used to answer 500, and a null metrics was stored first; the exception was a create that also sent metrics, where a null enabled or rollback_percentage was taken as its default and answered 200, and that create now answers 422 as well. A field left out still keeps its stored value, and on create takes its default: a create that left metrics out used to store a null and answer 500, and now stores {}. A configuration stored with a null metrics reads back with metrics: {}, and the check and the safety monitor treat it as no thresholds; the reads used to answer 500 and the monitor skipped the flag. The OpenAPI document changes only in FeatureFlagSafetyConfigUpdate: its three fields are no longer nullable, and its description. See Safety Monitoring. |
POST /api/v1/experiments/{experiment_id}/start, POST /api/v1/experiments/{experiment_id}/complete |
#974 | Starting a draft before its scheduled start_date sets its start_date to now; it used to keep the scheduled date, still ahead, and completing the experiment then answered 500. A scheduled end_date is kept, and a draft whose scheduled start has passed, a resume from paused and an experiment the scheduler starts keep their start_date as before. Completing an experiment whose start_date is still after now (one an earlier version started early) answers 200 and sets that start_date to one microsecond before the new end_date; it used to answer 500. The OpenAPI document changes only in the two operations' descriptions. |
POST /api/v1/power/sample-size, POST /api/v1/power/mde, POST /api/v1/power/runtime |
#1003 | n_variants accepts at most 100, and sample_size_per_variant, required_sample_size and daily_traffic at most 1,000,000,000,000 (10^12); a larger value answers 422 at ["body", <field>] with "type": "less_than_equal". Before, most of these fields answered 500 for a value too large to convert to a float (above about 1.8e308) and 200 for smaller ones; sample_size_per_variant on /mde answered 200 at any size, and a very large n_variants on /sample-size could answer a 422 about the effect size. Any of these values above the new bound now answers 422 at the field. A sample size whose calculation still leaves floating-point range (a mean or ratio metric with a very large or infinite baseline_std, or a baseline so small that the squared difference underflows) answers 422 with a fixed message; it used to answer 500. Every other request answers as before. The OpenAPI document changes only in the maximum of those seven request fields. See Statistical Power Analysis. |
POST /api/v1/power/plan |
#1011 | The advice is always the built-in planning advice and generated_by is always "template", whatever the API's environment; it no longer calls Claude when ANTHROPIC_API_KEY is set. The request and response are unchanged. The OpenAPI document changes only in the operation's summary and description, in the descriptions of PlanRequest, PlanResponse and its generated_by field, and in the description of the Power Calculator tag. See Statistical Power Analysis. |
POST /api/v1/tracking/events |
#400 | An experiment_id, feature_flag_id or variant_id that is a UUID naming no stored experiment, feature flag or variant answers 404, and nothing is stored; it used to answer 500 with "Could not store the event". The detail names each such field, never the id that was sent: {"detail": "No experiment has that experiment_id."}, with one sentence per field when more than one names nothing. Only existence is checked, as the database did: an event whose ids all name stored rows is stored and answered as before, whoever owns them. An id that is not a UUID still answers 422, before any lookup. The OpenAPI document changes only in this operation's description and its new 404 response. See Track Event by Ids. |
POST /api/v1/ai/design, POST /api/v1/ai/interpret/{experiment_id} |
#1043 | Each operation allows 10 requests a minute per client address, where it had the default of 300, and above that answers 429 Too Many Requests with Retry-After: 60; every experiment id draws on the interpretation's one count. With ANTHROPIC_API_KEY set, each call to Claude gives up when Claude has not answered in 30 seconds, and is tried at most twice (it used to give up only after 600 seconds and be tried three times); when it fails, the operation answers with its template-based suggestion or interpretation, as it already did when Claude was unavailable. Without the key only the limit is new. The request and response are the same, and so is the OpenAPI document. See AI Endpoints and the MCP Manifest. |
GET /api/v1/workspaces/ |
#1008 | Each workspace in the list carries member_count, the number of its members, as GET /api/v1/workspaces/{workspace_id} already did; every other field is unchanged. The dashboard's Workspaces page shows it, where it used to show "0 members" for every workspace. The route is served by the full profile only. The OpenAPI document changes only in this operation's description and its response schema, which becomes a list of WorkspaceWithStatsResponse (WorkspaceResponse plus member_count). See Workspaces. |
POST /api/v1/tracking/errors, POST /api/v1/tracking/errors/batch, GET /api/v1/safety/feature-flags/{feature_flag_id}/check |
0.29.0 | Error reports are stored and answered as before. Only those sent with an API key that carries sdk:ruleset, whose owner can change feature flags, count toward a flag's error rate and automatic rollback, as evaluations that fail on the server do. A report sent with any other key is stored with request_data.source "client", and the check's details carries uncounted_error_count, the number of such reports in its window; error_count and the error rate count the rest. Reports stored before the upgrade do not count. The OpenAPI document changes only in the two report operations' descriptions. See Which errors count. |
GET/POST /api/v1/feature-flags/evaluate/{flag_key}, GET /api/v1/feature-flags/user/{user_id}, POST /api/v1/tracking/assign, POST /api/v1/tracking/assign/batch |
#1178 | A targeting context nested more than 10 levels deep or larger than 8,192 bytes answers 422, where it used to be accepted. Every JSON object and array is a level, the context itself the first, and the size is the context as compact JSON in UTF-8. Depth is checked first; the detail is Targeting context nested more than 10 levels deep or Targeting context larger than 8192 bytes, and nothing is evaluated, assigned or recorded for the request. On the two GET routes the context parameter is held to 8,192 bytes as sent, before it is parsed, and a parameter nested too deeply to parse answers the same 422. The batch checks every user's context before it assigns the first user: one over a limit refuses the whole request, its detail prefixed with the user's position (users[3]:), and nobody is assigned. A POST body nested too deeply to parse still answers 400. Every other request answers as before; the contexts the SDKs send are a few hundred bytes. The OpenAPI document does not change. See Targeting Context Limits. |
POST /api/v1/segments/{segment_id}/evaluate, POST /api/v1/segments/bulk-evaluate |
#1183 | A user_context nested more than 10 levels deep or larger than 8,192 bytes answers 422, where it used to be accepted. The limits and the detail are the flag evaluation routes': every JSON object and array is a level, the context itself the first, the size is the context as compact JSON in UTF-8, depth is checked first, and the detail is Targeting context nested more than 10 levels deep or Targeting context larger than 8192 bytes. No segment is read or evaluated for the request. Bulk-evaluate checks its one user_context once, whatever the number of segments. A body nested too deeply to parse still answers 400. Every other request answers as before, an id list's user_context.user_id included. The OpenAPI document does not change. See Targeting Context Limits. |
GET /api/v1/metrics/summary |
#1206 | error_rate counts only the errors the safety check counts: an error report stored with request_data.source "client" is left out, where every error_logs row in the period used to count, so the rate can be lower than before for the same data. The new uncounted_error_count is the number of such reports in the period. The period (start_date to end_date, by default the 30 days before now), the denominator (the aggregated flag evaluations) and the unit (a percentage) are unchanged, and still differ from the check's. The OpenAPI document changes only in MetricsSummary, which adds the required uncounted_error_count and describes error_rate, and in the operation's description. See Which errors count. |
POST /api/v1/llm-experiments/{experiment_id}/judge, GET /api/v1/llm-experiments/{experiment_id}/results |
#1192 | A judge call that fails, or a reply without a JSON score from 0 to 1, stores no score: the evaluation's auto_eval_score becomes null, replacing any earlier score, and its entry in the judge's response has "score": null and error "call_failed" or "no_score". It used to store 0.5 for a failed call or a missing score, and otherwise a number taken from the reply's text or clamped into 0 to 1. The results leave unscored evaluations out of the auto-eval statistics and count them in each variant's n_auto_eval_unscored and the experiment's total_auto_eval_unscored, evaluations never judged included. p_value and effect_size are null when neither variant's values vary; they used to be 1.0 and 0.0. Scores stored before the upgrade are unchanged; run the judge again to re-score them. The OpenAPI document changes only in VariantStats and LLMExperimentResults, which add the two required counts, and in the judge operation's description. See LLM-as-Judge Scoring. |
POST /api/v1/experiments/, PUT /api/v1/experiments/{experiment_id}, GET /api/v1/experiments/{experiment_id}/split-url/preview |
#1194 | A split URL variant's traffic_allocation must be from 0 to 100, where only the sum was checked: -50 and 150 were accepted because they add up to 100. A value outside the range answers 422 at ["body", "split_url_config", "variants", <i>, "traffic_allocation"] on a create or an update, and nothing is written. A PUT that does not send split_url_config is unaffected and leaves a stored one as it was, but one that sends back a stored configuration with an allocation outside the range answers 422 until it is corrected. The preview of such a stored configuration answers 409 with the detail split_url_config is not valid; update it with PUT /api/v1/experiments/{id}, where it answered 200 with the stored allocations, as they were. Every other request answers as before. The OpenAPI document changes only in SplitUrlVariant, which gains minimum 0 and maximum 100 on traffic_allocation, and in the descriptions of SplitUrlConfig and VarianceReductionConfig: covariate_metric_id is described as optional, as it is, because without it each metric's own event before the user's assignment is the history. See Server-Side Split URL Testing API. |
POST /api/v1/feature-flags/, PUT /api/v1/feature-flags/{flag_id} |
#828 | Dashboard-shaped targeting_rules (logical_operator and groups) with more than 1,000 groups and conditions in all answer 422, where they used to be accepted. Each group and each condition counts as one, and the number is the one native rules already had for rules, groups and conditions. The refusal is at ["body", "targeting_rules"], its msg is Value error, targeting rules: at most 1000 groups and conditions are allowed, and nothing is written. A PUT that does not send targeting_rules is unaffected, but one that sends back stored rules over the limit answers 422 until they are reduced. Stored rules over the limit are still read and evaluated as before, and python -m backend.scripts.check_targeting_rules lists them. Experiment targeting rules are unchanged. The OpenAPI document does not change. See Creating Feature Flags. |
GET /api/v1/results/{experiment_id}/sample-size |
#1257 | With no mde sent, the plan is for the effect saved with the experiment, its primary metric's expected_effect, when that is above 0 and below 1; otherwise for 0.05 as before. The response carries mde_source: request, experiment or default. An unknown experiment answers 404 whatever the other parameters, where one with a baseline_conversion_rate that mde raised to 100% or more answered 422. The OpenAPI document changes in the mde parameter (it has no default), the new property and the descriptions. See The planned effect. |
POST /api/v1/experiments/, PUT /api/v1/experiments/{experiment_id}, POST /api/v1/experiments/{experiment_id}/clone |
#1257 | A primary metric's expected_effect is read: it is the relative effect the sample-size plan is for (0.10 means 5% → 5.5%). Values are accepted and stored as before, and one that is not above 0 and below 1 is not used. An experiment stored before the change with an expected_effect on its primary metric now plans from it. The OpenAPI document changes only in the field's description. |
POST /api/v1/wizard/drafts/{draft_id}/submit |
#1257 | The experiment it creates carries the draft's mde as its primary metric's expected_effect, which the results page plans the sample for; the mde used to be dropped, leaving expected_effect null. The guardrail metrics carry none, and a draft with no mde creates the experiment as before. The OpenAPI document does not change. See Guided experiment builder. |
Deprecated operations
A deprecated operation keeps its URL, shape and behaviour. It carries
"deprecated": true in the snapshots (deprecated=True on the route
decorator), its summary ends in "(deprecated)", and its description says why
it is deprecated. Marking a stable operation deprecated, or rewording its
description, changes its operation object, so it is a change to a stable route
under the rule above: regenerate with make openapi and name the operation in
the pull request. Removing a deprecated operation later is a breaking change.
The API sends no Deprecation or Sunset response header; the OpenAPI
document and this page are where a deprecation is declared.
Removed operations
A removed operation answers 404 and is absent from both snapshots. Removing
one is a breaking change.
| Operation | Deprecated by | Removed in | Use instead |
|---|---|---|---|
GET /api/v1/edge/bootstrap |
#226 | the release after 0.17.0 (#737) | GET /api/v1/sdk/ruleset (beta; an API key with the sdk:ruleset scope) for server-side local evaluation, or GET /api/v1/feature-flags/evaluate/{flag_key} for one flag; for experiments, POST /api/v1/tracking/assign |
GET /api/v1/openfeature/flags |
#241 | the release after 0.17.0 (#737) | GET /api/v1/feature-flags/evaluate/{flag_key}, which both OpenFeature providers call; GET /api/v1/sdk/ruleset for server-side local evaluation |
Beta numbers on a stable route
One analysis carries analysis_status: "beta" while staying x-stability:
stable: the sequential analysis. GET /api/v1/results/{id}/sequential serves
it, and GET /api/v1/results/{id} and GET /api/v1/experiments/{id}/results
embed the same block as sequential_testing (#922). Its shape is settled
(dashboards read it) and the fixes to its numbers (#231, #854) change values,
not fields, so the two fields were added to it optionally. A fix that changes
its numbers gets a row in the table of changed stable operations above naming
all three operations, and bumps ENGINE_VERSION
(backend/app/core/stats_engine.py): the results routes cache their answer
under a key that includes it, so without the bump the embedded copy is served
with the old numbers for up to 24 hours.
Profiles
The core snapshot is the public promise: it must not change with the set of
installed modules. That is why the statistics (Bayesian, CUPED, sequential
testing) are core and not a module — a results response whose shape depended
on an optional module could not have one stable document (see the manifest,
modules-manifest.txt, "DELIBERATELY NOT IN THIS MANIFEST"). The three routes
that exist in both profiles but answer 501 in the core profile without the
module body (/compliance/reports/{standard}, /compliance/export,
/experiments/{id}/split-url/preview) are in the core snapshot, because their
URL and shape are.