Search feedback

Reviewed searches promoted into Training pairs and pairs explicitly saved from Playground Search receive the project tag retrieval_feedback automatically. Existing tags on a reused pair are preserved; adding the tag increments that pair's revision. An exact promotion replay does not restore a tag you later removed. Tag capacity is checked atomically: 409 tag_limit_reached leaves the review and dataset unchanged. Reporting an unreviewed search does not create a training pair or tag it.

Search feedback turns searches your application already performs into embedding training examples. An embedding call contains text, not which result answered a query, so ReOpenly never saves embedding requests as training data. Instead, your server reports a query and the ranked results it showed, and the project owner reviews the whole list and keeps every useful result.

1. Availability

Search feedback belongs to an embedding project's dataset. It is available only when the project lists the dataset.search_feedback capability (GET /api/v1/projects/{owner_slug}/{project_slug} → project.capabilities). While it is unavailable, every route below answers 409 retrieval_feedback_unavailable after authorization, and nothing already stored is deleted.

2. Endpoint, authorization and media type

POST /api/v1/projects/{owner_slug}/{project_slug}/datasets/{dataset_id}/retrieval-events
POST /api/v1/projects/{owner_slug}/{project_slug}/datasets/{dataset_id}/retrieval-events/batch

• Authenticate with an account API key (Authorization: Bearer fnf_...) or a signed-in session. Only the project owner may report or review events; access to a shared or public model grants nothing here. Every way of not owning the dataset is the same 404 retrieval_dataset_not_found.
• Send Content-Type: application/json, uncompressed.
• Call this only from your server. Never put an API key in browser code.
• The dataset_id is on the Search feedback page and in GET /api/v1/projects/{owner_slug}/{project_slug}/datasets.

This is a versioned API: compatible additions evolve /api/v1; anything that removes, renames, narrows or reinterprets a field or error needs /api/v2. Ignore response fields you do not recognize.

3. Basic template and cURL example

Basic template — replace every <...> value before sending:

curl -X POST "https://reopenly.com/api/v1/projects/<owner>/<project>/datasets/<dataset-id>/retrieval-events" \
  -H "Authorization: Bearer $REOPENLY_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{"event_id":"<unique-event-id>","model":"<model-slug>","query":"<search-query>","results":[{"rank":1,"text":"<result-text>"}]}'

cURL example — example values; replace them before sending:

curl -X POST "https://reopenly.com/api/v1/projects/acme/support/datasets/42/retrieval-events" \
  -H "Authorization: Bearer $REOPENLY_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{
    "event_id": "search-2026-09-17-000123",
    "model": "citations-06b",
    "query": "How do I resume a paused fine-tune?",
    "metadata": {
      "occurred_at": "2026-09-17T14:02:11Z",
      "model_revision": 1,
      "search_target": "support-docs-production"
    },
    "results": [
      {"rank": 1, "text": "Resume continues a run rather than replacing it.",
       "metadata": {"score": 0.83, "external_id": "chunk-782"}},
      {"rank": 2, "text": "Cancel is available while a run is queued."}
    ]
  }'

4. Required fields

Field	Rule
event_id	Your identifier for this search, ^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$, case-sensitive. It is also the replay key.
model	The project's embedding model slug — the same SavedModel.slug you send as model to the embeddings API. The event records the exact model revision it was produced with.
query	The search text, 1..8,192 UTF-8 bytes, not whitespace-only.
results	1..20 items in ascending rank order, ranks exactly 1..n. Each has rank and text (1..32,768 UTF-8 bytes, not whitespace-only).

Text is stored after Unicode NFC normalization; whitespace is preserved.

Public base models also work without a hosted alias: set model to ReOpenly/Qwen3-Embedding-0.6B or ReOpenly/Qwen3-Embedding-4B. Use the same reference that generated your vectors. The event records the verified base build and its settings; base model_revision is 1 (omit it or send 1). Unknown builds or revisions return retrieval_model_not_found. Explicitly report the query and ranked results, then review relevance labels; embedding vector calls alone do not create training examples. Persistent Search indexes still require a hosted alias. Earlier Qwen/... references remain accepted.

5. Optional metadata

Optional values exist only under typed metadata objects. Omit a member you do not have — never send null. An empty metadata object is the same as omitting it. Fields the API does not know are ignored, so a client written against a newer version keeps working; a known field with a wrong value is still refused.

Event metadata:

• occurred_at — when your search ran (RFC 3339, at most 24 hours ahead and 365 days behind receipt). Display only.
• model_revision — the model revision your server used, for delayed reporting. It must exist in that model's history. Omitted, the current revision at receipt is recorded.
• search_target — an opaque label for the index, collection or namespace you searched, ^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$, never containing ://. ReOpenly never connects to it. Do not put credentials or personal data in it.

Result metadata:

• score — your search system's score, any finite number (it may be negative). Stored and shown as provided; never used to choose a label.
• external_id — your chunk or document id, at most 128 UTF-8 bytes.
• title — at most 512 UTF-8 bytes.
• source_uri — https only, at most 2,048 bytes, with no user info, query string or fragment.

6. Limits

Limit	Value
Results per event	20
Query text	8 KiB (UTF-8)
Result text	32 KiB (UTF-8)
Total text per event	128 KiB
Single request body	128 KiB
Batch request body	1 MiB, at most 50 events
Retained events per dataset	10,000
Retained text per dataset	256 MiB
Requests per dataset per UTC minute	60
New events per dataset per UTC minute	500

GET .../retrieval-settings returns the same limits for your dataset.

7. Replay, conflicts, batches and errors

event_id is the only idempotency key; there is no idempotency header and no schema-version field in the body.

• The same event_id with the same content returns the stored event with 200 and "created": false. Key order, timezone offsets, 1 versus 1.0 and an empty metadata object do not count as different content; whitespace in text does.
• The same event_id with different content is 409 idempotency_conflict.
• A batch is all-or-nothing for new events: one invalid, conflicting or over-quota item stores nothing. Exact replays may be mixed with new events. The response keeps input order and returns 201 if anything was created, 200 if every item was a replay.

{
  "object": "retrieval_event",
  "event_id": "search-2026-09-17-000123",
  "state": "unreviewed",
  "revision": 1,
  "received_at": "2026-09-17T14:02:16Z",
  "created": true
}

Errors use one envelope and never echo your content:

{
  "detail": {
    "code": "retrieval_ingest_rate_limited",
    "message": "Too many search-feedback requests for this dataset this minute.",
    "limit": 60,
    "window_seconds": 60,
    "retry_after_seconds": 44
  }
}

HTTP	code
400	invalid_retrieval_event, invalid_cursor
404	retrieval_dataset_not_found, retrieval_event_not_found, retrieval_model_not_found
409	retrieval_feedback_unavailable, idempotency_conflict, revision_conflict, retrieval_feedback_label_conflict, retrieval_feedback_positive_conflict, retrieval_event_not_promotable, embedding_example_conflict, embedding_dataset_capacity_exceeded
413	retrieval_payload_too_large
415	unsupported_retrieval_media_type
422	Field validation (type, loc, msg only)
429	retrieval_ingest_rate_limited, retrieval_dataset_quota_exceeded

Every 429 carries a Retry-After header equal to retry_after_seconds. Retrying with the same event_id is always safe.

8. Review and promotion

The owner reviews each search on the Search feedback page, or through the API:

• Mark useful ranked results Positive and optionally write one additional positive. Each becomes one training example with the query as anchor.
• Choose at most one negative: any unkept ranked result, the visible highest-ranked-unkept suggestion, written text, or No negative. Every positive from the review shares that one optional negative, never every positive-negative combination.
• Unchecked results remain search context. They are not labels. Reported score values are displayed as provenance and never choose a label.
• Choosing no negative creates pairs rather than triplets. Pairs created with No negative can train without editing the saved examples. If a pair has no batch partner, training can borrow a safe passage from its own split; see embedding training.
• Skip creates nothing, and a search nobody reviews creates nothing. A click, a rank or an abandoned review is never a label.
• If the would-be negative is already a known positive for the same query, promotion is refused with retrieval_feedback_label_conflict and only the conflicting rank; nothing is written and no lower result is substituted.
• If a result you keep is already a hard negative for the same query, promotion is refused with retrieval_feedback_positive_conflict. Correct the earlier Training pair before keeping it as a positive.

GET   .../retrieval-events?view=inbox|history&q=&model=&received_after=&received_before=&limit=&cursor=
GET   .../retrieval-events/{event_id}
PATCH .../retrieval-events/{event_id}          {"expected_revision":1,"state":"skipped"}
POST  .../retrieval-events/{event_id}/promote  {"expected_revision":1,"kept_ranks":[1,2],"negative_mode":"rank","negative_rank":3}
GET   .../retrieval-settings

negative_mode is auto (the highest-ranked unkept suggestion), rank, custom, or none. Send negative_rank only with rank and negative_text only with custom. positive_text adds one written positive; at least one kept rank or positive_text is required.

{
  "object": "retrieval_event_promotion",
  "event_id": "search-2026-09-17-000123",
  "state": "promoted",
  "revision": 2,
  "example_ids": [441, 442],
  "positive_ranks": [1, 2],
  "hard_negative_rank": 3,
  "positive_count": 2,
  "has_negative": true
}

Pages are keyset-paginated ({data, next_cursor, total_count}); a cursor only works with the filters it was issued for. A stale expected_revision is 409 revision_conflict with expected_revision and actual_revision.

9. Storage, privacy and cost

• Reported searches are kept until you ask for them to be erased. Nothing expires automatically and there is no retention setting. The only bound is the dataset's storage limit below; when it is reached, ingestion answers 429 retrieval_dataset_quota_exceeded rather than dropping older searches.
• Promotion copies the query and the chosen positive and optional negative into Training pairs. Those examples follow the dataset and are removed when you delete them there. The reported source search remains intact until requested erasure, and its promotion link records the source event and ranks.
• ReOpenly stores the query, result text and the metadata you send. It stores no vectors, never connects to your search_target or source_uri, never runs a model for search feedback, and never bills for it.
• To export or erase a dataset's search feedback, write to the privacy address in the privacy policy.

10. Correcting Training pairs

Promoted examples appear in the dataset's Training pairs editor. Use Edit to correct the anchor, positive, or optional negative, then save or cancel inline. ReOpenly preserves the example's original Search feedback source and review provenance while updating the current training content. Deleting a Training pair removes it from the live dataset but keeps the source review's content-free history and removes the row from available_example_ids.

11. Response shapes

Detail (GET .../retrieval-events/{event_id}):

{
  "object": "retrieval_event",
  "event_id": "search-2026-09-17-000123",
  "model": "citations-06b",
  "query": "How do I resume a paused fine-tune?",
  "metadata": {"occurred_at": "2026-09-17T14:02:11Z", "model_revision": 1, "search_target": "support-docs-production"},
  "results": [
    {"rank": 1, "text": "Resume continues a run rather than replacing it.", "metadata": {"score": 0.83, "external_id": "chunk-782"}},
    {"rank": 2, "text": "Cancel is available while a run is queued."}
  ],
  "state": "promoted",
  "revision": 2,
  "received_at": "2026-09-17T14:02:16Z",
  "decision": {"state": "promoted", "decided_at": "2026-09-17T14:05:00Z", "positive_ranks": [1], "hard_negative_rank": 2, "example_ids": [441], "available_example_ids": [441], "positive_count": 1, "has_negative": true}
}

decision is null while unreviewed and {"state":"skipped","decided_at":...} after a skip. available_example_ids drops examples you later deleted from Training pairs.

Settings (GET .../retrieval-settings):

{
  "object": "retrieval_settings",
  "revision": 1,
  "workflow": {"waiting_events": 83, "promoted_events": 38, "skipped_events": 27, "training_examples_added": 38, "training_examples": 41, "retained_events": 148},
  "storage": {"retained_text_bytes": 19293798, "max_retained_events": 10000, "max_retained_text_bytes": 268435456},
  "ingest_limits": {"requests_per_utc_minute": 60, "events_per_utc_minute": 500, "events_per_batch": 50, "results_per_event": 20}
}

Every field and response is in the API reference under Search feedback, and in the downloadable OpenAPI document.

If promotion would exceed the Training pairs limit, the whole promotion returns 409 embedding_dataset_capacity_exceeded. No partial examples or review decision are saved. Remove unneeded Training pairs before retrying.