Training jobs

A training job fine-tunes a base model on examples you select, and produces a model you can host, call and download. Text-generation and embedding projects train through the same API: the project's type decides which request body POST .../training-jobs accepts, and every other route (list, detail, attempts, logs, pause, resume, finish, cancel, retry, download) is shared and returns the same job shape.

Training runs in the background. You start a job, follow it while it runs, and use its model once model_available is true. Only measured training time is billed; a run that never begins training is free.

GET  /api/v1/projects/{owner_slug}/{project_slug}/training-jobs
POST /api/v1/projects/{owner_slug}/{project_slug}/training-jobs
GET  /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/estimate-messages   text
GET  /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/profiles            embedding
POST /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/preview             embedding
POST /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/estimate            embedding
GET  /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}
POST /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/pause
POST /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/resume
POST /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/finish
POST /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/cancel
POST /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/retry
GET  /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/retry-estimate
GET  /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/attempts
GET  /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/attempts/{attempt_id}/logs
GET  /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/dataset
GET  /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/download

Every field of every request and response is in the API reference. POST .../resume and POST .../retry require an Idempotency-Key header: repeating a request with the same key returns the first result instead of starting a second run. Creation has no idempotency guarantee; after an uncertain response, check the job list before submitting again.

The list always returns {data, next_cursor, total_count} with an optional limit (25 by default, at most 100). Pass cursor=<next_cursor> unchanged for the next page. No pagination mode is required. A malformed cursor or one from another project answers 400.

Start a text-generation run

Select chats by tag and choose a base model:

{
  "dataset_tags": ["support-clean"],
  "model_name": "<catalog-model-id>",
  "max_seq_length": 2048,
  "num_train_epochs": 3
}

dataset_tags must name existing tags exactly. Several tags select the union of their chats, duplicates are ignored, and an empty list selects every chat in the project. An unknown or blank tag answers training_dataset_selection_invalid; an empty selection training_dataset_empty; a selection with no trainable text training_dataset_unusable. These are 422 on create and 409 on retry, and nothing is created or charged.

num_train_epochs is how many complete passes the run makes over your selected examples, and it accepts fractions (0.5 is half a pass). The server converts it into training steps from the selection size and the batch shape, and freezes the result on the job as planned_steps:

effective_batch_size = per_device_train_batch_size × gradient_accumulation_steps
steps_per_epoch      = ceil(train_examples / effective_batch_size)
planned_steps        = max(1, ceil(steps_per_epoch × num_train_epochs))

For example, 152 training examples at the default effective batch of 8 take ceil(152 / 8) = 19 steps per epoch, or 57 planned steps for 3 epochs. A final partial batch still counts as one step. max_steps (1 to 100) is the alternative: a fixed number of steps. Send one or the other, never both.

Other settings you may send: system_prompt, learning_rate, lora_rank, lora_alpha, per_device_train_batch_size, gradient_accumulation_steps, loss_mask (assistant_only or full_sequence), base_load_precision, warmup_steps, early_stopping_patience and the evaluation settings below. Every setting you omit is filled in from the base model's training profile and frozen with the job, so a job read back later reports exactly what it ran with. Read the allowed models and each setting's min, max, default and whether it accepts fractions (integer) from GET /api/v1/base-models rather than copying them. A refused value answers with a code and the field it is about.

system_prompt is one instruction for the whole job, added as the first system message of every example. It never affects inference: a hosted model answers exactly the messages you send it. If a selected example already has its own system message, creating the job with a system_prompt answers 422 and says how many conflict, rather than merging two sets of instructions.

Estimate before you start

GET /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/estimate-messages?tags=support-clean&model_name=<catalog-model-id>&num_train_epochs=3

tags is a comma-separated selection, and the estimate accepts the same workload choices as create: model_name, num_train_epochs or max_steps, max_seq_length, per_device_train_batch_size, gradient_accumulation_steps, base_load_precision, warmup_steps (including 0; omit it for automatic warmup), and evaluation_cadence_kind with its matching evaluation_events or evaluation_interval_steps.

It returns the selection counts (selected_chats, selected_messages, trainable_chats, nonempty_text_messages, and is_trainable: false for an empty or unusable selection), the exact train/evaluation split, the planned steps and evaluation checks, the estimated duration and credits, and two previews:

{
  "configuration_preview": {
    "base_load_precision": "4bit",
    "warmup_steps": 2,
    "warmup_source": "automatic"
  },
  "evaluation_preview": {
    "holdout_percent": 5,
    "minimum_validation_examples": 8,
    "selection_examples_required": 160
  }
}

warmup_source is custom when you sent warmup_steps and automatic otherwise. The estimate uses the same arithmetic as create, but it is advisory: create still performs its own final check.

Start an embedding run

An embedding run trains on the anchor, positive and optional negative examples of one embedding dataset. Four calls, in order:

GET  .../training-jobs/profiles    base models and the bounds of every setting
POST .../training-jobs/preview     the exact selection, split and steps
POST .../training-jobs/estimate    the credits needed to start
POST .../training-jobs             start the run

Choose the dataset, a base model from the profiles, optional tag IDs to narrow the selection, and a whole-epoch max_steps. The preview returns a selection_sha256, the exact train/validation split, explicit-negative counts, the pairs and triplets scheduled per epoch, and planned_steps_per_epoch. Create requires that selection_sha256, so a run always trains on the selection you previewed. A one-step preview is a way to discover the batch count; estimate and create require a whole number of epochs.

A pair (an example with no negative) learns from the other passages in its batch, and two examples of the same query never share a batch, since each positive is a correct answer for the other. When a query has several positives and no negatives, as a search where every result was relevant produces, some of its pairs would have no batch partner. Each of those pairs instead receives, as its negative, a positive from another query in the same training split (never a passage known to answer its own query) and trains with the triplets. The preview's borrowed_negative_count reports how many training pairs were converted; the explicit-negative counts still describe your saved examples. Your data is unchanged. Validation pairs use the same fallback within the validation split. If a stranded pair has no safe passage to borrow in its own split, the selection is still refused with embedding_batches_infeasible.

The estimate is provisional: estimated_credits is null, and admission_ceiling_credits is the amount that must be available to start the run, not its expected charge. The job's cost_credits reports what was actually charged.

Preview, estimate and create also accept optional execution_limit_seconds, an integer of at least 5 seconds that cannot exceed the platform ceiling. The selected training service must support that limit; an incompatible service is refused rather than extending it. Omitting it keeps the platform ceiling. Send the same limit with all three requests. The estimate returns the chosen admission_ceiling_seconds and its credit authorization; create freezes that ceiling for the job. Retry and resume keep it, including after the platform limit is raised. A platform limit lowered below the frozen value refuses retry/resume rather than changing the job. The shared GET .../retry-estimate confirmation is available for embedding jobs and quotes authorization from that frozen ceiling at the current rate, rather than a text-training workload estimate. It is not a predicted charge. The deadline applies separately to each attempt, including model and data preparation; time in the queue is separate. It is not an aggregate spending limit across retries/resumes, and only measured training time is billed. A short limit can expire before training starts or before the requested checkpoint is reached. This API option is separate from settings; the app currently uses the default.

Embedding settings

Preview, estimate and create accept an optional settings object; GET .../profiles publishes each setting's range and default as setting_bounds, and the current choices for each base model. A refused value answers 422 with detail.field.

Setting	What it does
learning_rate, warmup_steps	The optimizer's rate and its warmup.
per_device_train_batch_size	Examples per batch. Every other example in a batch serves as a negative, so larger batches give harder training signal.
gradient_accumulation_steps	Averages gradients over several batches. It adds no negatives.
loss_scale	The temperature of the contrastive loss.
per_device_eval_batch_size	0 picks automatically from the training batch and the profile's memory_tokens.
evaluation_checks	How many validation checks the run makes (1 to 10).
early_stopping_patience	Stop after this many checks without improvement, keeping the best checkpoint.
max_examples_per_query	The most examples one query may contribute.
lora_rank, lora_alpha, lora_dropout	The adapter's shape. A higher rank makes a larger adapter; it is applied at alpha ÷ rank; dropout applies during training only.
query_instruction	One line (up to 1,024 characters) put before every query in training, validation, the API and Search. Documents never get it.
output_dimensions	The vector size the model is trained at and returns.
max_seq_length	Tokens per training example. A model trained above 512 accepts inputs up to its training length; one trained at 512 or less accepts 512.
training_mode	How the base model is held while the adapter trains: qlora_4bit uses the least memory, lora_16bit trains against the same weights the hosted model uses.
normalize_embeddings	true by default. false returns raw vectors whose length carries meaning, compared by dot product; it requires loss_scale 1.

With accumulation, max_steps counts optimizer steps; the preview's planned_steps_per_epoch gives the number per epoch.

Training settings

Embedding projects use the same New training workflow: select a dataset and base, choose epochs, inspect the split, and adjust the available Advanced settings. Optimizer steps are calculated from the planned batches and chosen epochs. Reusable training presets are planned; there is no customer recipe selector.

Validation and the kept checkpoint

About 10% of the selection is held out for validation, always as whole groups of examples that share any text, so nothing held out is also trained on. Large groups stay in training when smaller groups can provide validation. The actual fraction can differ from 10% because groups are indivisible; check its counts in the preview. Existing frozen versions keep their original split. The run checks the validation loss up to ten times (the last at its final step): the same contrastive loss it trains on, computed on the held-out examples at the vector size and similarity the model is served with (cosine for normalized vectors, dot product for raw ones). It keeps the checkpoint with the lowest loss, the earliest on a tie. With at least four checks, two checks in a row without a new best halve the learning rate; with fewer, the rate decays linearly. Runs started before this rule keep their own: they kept the checkpoint with the best validation nDCG@10, and their job page still shows it.

That loss only chooses a checkpoint: it is measured during training, which can hold the base model at a lower precision than the hosted model uses, so it is not a measure of retrieval quality. Separate evaluation jobs for text and embeddings are planned for a later release; this release has no Retrieval quality or Retrieval evaluation panel. A successful model can be hosted without them.

Reading an embedding run. The job page draws the validation loss at each check (lower is better) and marks the checkpoint the run kept.

• Still falling at the last check: more epochs may help.
• Falling, then flat: more epochs will not help; the run already kept its best check.
• Lowest, then rising: the model is fitting the training pairs more than the held-out examples. The kept checkpoint is the lowest point; next time try fewer epochs, a lower learning rate, or early stopping.
• Flat from the first check: try a higher learning rate or more data.

Evaluation and early stopping

For a text run, a small share of a large enough selection is held out for evaluation and never trained on. Which examples are held out is a property of your dataset: every run over an unchanged dataset uses the same ones, which makes runs comparable. The policy assigns 5% to evaluation, needs at least 8 evaluation examples, and caps the set at 512:

candidate_eval = floor(usable_examples × 0.05)
eval_examples  = 0 if candidate_eval < 8 else min(candidate_eval, 512)
train_examples = usable_examples - eval_examples

So evaluation starts at 160 usable examples; a smaller selection trains on all of its examples and reports no evaluation loss.

You can choose when evaluation runs:

{"evaluation_cadence": {"kind": "events", "events": 10}}

or {"evaluation_cadence": {"kind": "fixed_interval", "interval_steps": 300}}. An event budget is 1 to 100, and a fixed interval may resolve to at most 100 checks; both include the final step exactly once. Without a cadence the run schedules about ten checks per epoch, at least two and at most fifty, and the job shows it as Managed.

The served model is the checkpoint with the lowest validation loss across the run, not necessarily the last one; on a tie the earlier checkpoint is kept. Validation loss measures next-token prediction on held-out rows that are also reused to select the checkpoint, so it is a selection signal, not task accuracy. Keep an untouched test set outside the training selection and compare the tuned model with the base model on it before relying on it.

Early stopping is off unless you ask for it. Send early_stopping_patience (1 or more) to stop once evaluation loss has not improved for that many checks. A run that stops early still ends succeeded, before planned_steps. The estimate returns early_stopping_patience_max, two fewer than the scheduled checks (the first check sets the baseline and the final one cannot shorten the run), so a schedule with fewer than three checks allows only 0.

Three settings control how an evaluation pass runs. They change neither which rows are evaluated nor when, so they cannot change what the run learns; they exist because evaluation is where a long run is most likely to run out of memory.

Setting	Default	What it does
per_device_eval_batch_size	0	Evaluation batch size; 0 means the training batch size. Lowering it is the first fix for an evaluation out-of-memory failure.
fp16_full_eval	false	Half-precision evaluation uses less memory but changes the numerics of the loss that selects the model.
eval_accumulation_steps	1	Evaluation output buffering; see below.

Evaluation output buffering

eval_accumulation_steps is the Evaluation output buffering control. The value 1 means evaluation output is collected at the end of each check, shown in the app as End of check (default); it is not a one-step interval. Custom intervals run from 2 to 32. Smaller intervals move output more often and can reduce buffer memory, at some cost in speed. Buffering does not split evaluation batches: for an evaluation out-of-memory failure, lower the evaluation batch size first, then consider half-precision evaluation, and only then change buffering.

Reading the loss charts

• Falling training loss means the model fits the training examples more closely; it does not by itself show that answers improved.
• Validation loss measures prediction on examples that did not update the model; lower is better when comparing checkpoints of one run.
• Training loss falling while validation loss rises suggests overfitting: try fewer epochs or a lower learning rate and compare on the same dataset.
• The learning-rate and gradient-norm charts are diagnostics, not scores: read their shape within a run rather than any absolute value.

A finished job that evaluated has an evaluation block: the lowest loss and its step, the last loss and step, how many checks ran, and which checkpoint is served. promoted_before_last_eval: true says the served checkpoint is earlier than the last check. A job that never evaluated reports evaluation: null.

Following a run

The job detail reports status, progress, the frozen configuration, metrics, cost_credits (the actual charge; 0 when nothing was charged) and model_available. Each job has a number, counted from 1 within its project and shown in the app, and an id used by the API. To read a job by its number:

GET /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{number}?identifier=number

Jobs are never deleted, so numbers never shift; hosted-model and billing responses show the same value as training_job_number.

Waiting for compute. An embedding job can be accepted while compute is busy with another run. It waits for available capacity, and queue time is not charged. Each account still has one active training run.

Preparation. A new job first copies its selected examples, exactly as they are, into a frozen record: editing or deleting them later cannot change what the job trains on, and a retry trains on the same data. preparation moves through queued, snapshotting, then verified, failed or cancelled; training starts only after verified. A preparation failure is an input failure (failure_category: "input") and is not charged. A job created before this step existed reports preparation: null.

Timing. timing breaks the run into queued_seconds, preparing_seconds, training_seconds, post_processing_seconds and finalizing_seconds, with total_seconds (active time), elapsed_seconds (wall clock) and idle_seconds (the difference, such as the time before a retry). A phase without reliable timestamps is omitted rather than guessed. These are not the billed interval; see Billing.

Why it ended. A finished job carries outcome.stop_reason: completed, max_steps, early_stopping, cancelled or execution_deadline. outcome.downgrades lists settings the run could not apply as sent, as setting:action:cause tokens (for example early_stopping:disabled:no_eval_set when early stopping was asked for without an evaluation set). outcome: null means the run reported nothing, as for jobs that never reached training.

Environment. The Run card records the hardware the run used (GPU model and count, memory, driver and framework versions) and the frozen per-minute rate. These explain why similar runs can differ in price or behavior; they are not quality scores.

The data it trained on. A job's detail includes the first 50 frozen examples; the rest are paged, up to 200 at a time:

GET .../training-jobs/{job_id}/dataset?offset=0&limit=25
GET .../training-jobs/{job_id}/dataset?offset=0&limit=25&search=invoice&split=eval
GET .../training-jobs/{job_id}/dataset/{chat_id}

Each item says whether it was train or eval input. search matches frozen text, and a searched page returns total: null. These read the frozen record, not your current data.

Attempts and logs

A job runs as one or more attempts (a retry or a resume adds one). The attempts route returns each attempt's status, timestamps, failure code and log metadata; the logs route streams the attempt's log as newline-delimited JSON, with offset and limit in bytes.

Attempt status	Meaning
leased	Capacity was reserved for the attempt.
accepted	The run was accepted for execution.
started	The run is executing.
finalizing	Training ended; the model is being verified and published.
succeeded	The attempt completed.
failed	The attempt ended with a failure code.
cancelled	The attempt was cancelled before training began.
expired	Contact with the run was lost and the attempt was closed.
submission_uncertain	Whether the run was accepted could not be confirmed; no duplicate run is started.

log_complete: true (and the header X-Log-Complete: true) marks a complete log. After an abrupt stop the log is the last saved copy: integrity-checked but possibly missing the final output, marked log_complete: false and log_recovered: true.

Job states

Status	Meaning
pending	Accepted and waiting to start.
running	Preparing, training, or publishing the model.
cancel_requested	Cancelled before training began; the run is being stopped.
pause_requested	Pausing: the run trains on to its next checkpoint, and those steps are billed.
paused	Stopped at a checkpoint with everything kept. Resume it or finish it. A paused job does not hold your active-training slot.
succeeded	The model was verified and published.
failed	The run ended without a model.
cancelled	Cancelled before training began: no model, no charge.

Use model_available, not the status, to decide whether a model can be hosted or downloaded: some older stopped jobs are cancelled and still have a model.

Pause, resume, finish and cancel

Which control applies depends only on whether training has begun:

Phase	Control	Result
Queued or preparing	POST .../cancel	The job settles cancelled. Nothing was learned, so there is no model and no charge.
Training	POST .../pause (Stop in the app)	The run trains on to its next checkpoint and settles paused.
Publishing	neither	Training is over and the model is moments away.

The server enforces the boundary, because a client only sees progress after the fact: cancel answers 409 training_job_already_training once training has begun, and pause answers 409 training_job_not_training_yet before it.

A pause is not immediate, and the wait is billed. It is acknowledged at once, but the run continues to its next checkpoint, the only point it can later resume from. pause.steps_until_pause and pause.next_boundary_step on the job say how far that is, and the app shows them before you confirm. Checkpoints are at most 25 steps apart, so a pause trains at most 25 more steps. Pausing before the first checkpoint still works and can take longer. When no checkpoint remains both values are null (the pause is then effectively the end of the run); a job without a pause block cannot be paused. A pause that arrives after the last checkpoint lets the run finish normally, and it settles succeeded with its model.

A paused job offers two operations:

Operation	Result
POST .../resume	Continues from the checkpoint; charts and the selected checkpoint carry on where they stopped. Takes no body and requires an Idempotency-Key.
POST .../finish	Publishes the model the run reached and settles succeeded. Free: the training time was charged when the pause settled.

Embedding runs have the same three controls. An embedding run writes a checkpoint at every validation check, so pause.next_boundary_step is the next check before the final step, and the distance is the run's evaluation cadence rather than the 25-step bound above. Finish publishes the checkpoint with the best held-out score so far, the one the run would have kept had it ended there, which can be earlier than the checkpoint the pause stopped at. Both Qwen3-Embedding-0.6B and Qwen3-Embedding-4B support stopping at a checkpoint and resuming from it. Resume preserves the selected checkpoint even when it was written by an earlier attempt; missing or unverifiable checkpoint files are refused. A failed run can also be retried from the start.

Embedding resumes keep the job's frozen learning-rate schedule. Older profiles that did not record a schedule retain linear decay. The checkpoint must still verify, and the frozen base must remain eligible for training. When early stopping is enabled, the count of consecutive evaluations without improvement also continues from the checkpoint. An incomplete evaluation history or a checkpoint that already reached early stopping cannot restart training. If an embedding run reaches early stopping at the requested pause checkpoint, it finishes with early_stopping rather than entering paused.

There is no cancel for a paused job (409): it has already trained and been charged, so discarding its model would help no one. Finish publishes the best verified checkpoint, or, for a run with no evaluation set, the last checkpoint: the same model the run would have produced by ending there. If publishing cannot verify the model, the job settles failed with no further charge and can be retried. Resume and finish refuse each other with training_job_finalizing.

When a run reaches the platform's execution limit it stops itself, keeps the checkpoint it would have selected, settles succeeded and reports outcome.stop_reason: "execution_deadline". It is charged for the time it trained and can be resumed for the remaining steps.

Retry

A failed or cancelled job can be retried. A failed job never retries itself: the platform keeps the evidence and any verified checkpoint, and you decide whether to run again.

curl -X POST "https://reopenly.com/api/v1/projects/<owner>/<project>/training-jobs/42/retry" \
  -H "Authorization: Bearer $REOPENLY_API_KEY" \
  -H "Idempotency-Key: 8f8d8ac4-8de5-4c63-a8f4-f0a39db1af4d" \
  -H "Content-Type: application/json" \
  -d '{"resume_from_checkpoint": false}'

A retry reuses the job's frozen settings and frozen data; it never re-reads the dataset, and a job whose frozen data no longer exists cannot be retried. resume_from_checkpoint: true continues from a verified checkpoint of the same frozen input when one exists. GET .../retry-estimate says whether a retry is possible and what it would cost; it starts nothing.

Billing and limits

Only the training loop is billed: for each attempt, from its first training step to the moment training ends. Queueing, preparation (loading the model, reading and tokenizing your data) and publishing the model are not charged, and a run that never begins training is free. Failed attempts are not charged. cost_credits is the actual charge, from the credit ledger.

The estimate is calibrated on measured runs at an effective batch of 8 (per_device_train_batch_size 2 × gradient_accumulation_steps 4) and 8-bit loading. Other shapes are priced with a safety margin, and a batch below 8 is priced as 8. To enlarge the effective batch, raise gradient_accumulation_steps rather than per_device_train_batch_size: a larger per-device batch needs more memory per step and, near the limit of the training hardware, can slow every step far more than its size suggests. Sending num_train_epochs rather than max_steps keeps the length of a run tied to your data when you change the batch.

A run's maximum exposure must fit the platform's execution limit, six hours by default. Read the current value from duration_components.execution_limit_seconds on POST /api/v1/pricing/estimate rather than hard-coding it. A run that does not fit is refused before it starts with training_duration_exceeds_limit (422 on create, 409 on retry): reduce num_train_epochs, max_seq_length or the selection.

Failure codes

Failure codes are stable, and none carries raw system output.

Code	Meaning
training_exceeds_provider_limit	The run is longer than the platform can execute in one go. Nothing started; reduce epochs, sequence length or the selection.
worker_disk_exhausted	The run ran out of storage space.
provider_failed	The run failed while executing.
provider_execution_timeout	The run exceeded its execution deadline.
provider_ttl_expired	The run waited or ran longer than allowed and was closed.
provider_callback_missing	The run stopped reporting progress and was closed.
provider_terminal_callback_missing	The run ended without delivering a verified model.
provider_result_unavailable	The run's result could not be retrieved.
provider_status_unavailable	The run's state could not be confirmed after repeated checks.
provider_cancelled	The run was stopped by the platform without a request from you.

Download

When model_available is true, download the trained adapter:

GET /api/v1/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/download?format=lora

Embedding adapters include manifest.json with SHA256 checksums and byte sizes for the files in the archive, including the adapter config's public metadata. You can use it to verify the downloaded files.

It streams a gzip-compressed archive. GET .../training-jobs/{job_id}/download-metadata lists its files and total size first, without downloading. To call the model instead, give it a name under Hosted models.

Text training estimates include training_plan, the schedule used to calculate the quote. It describes training and evaluation work for the selected settings.