Training jobs and attempts

Training is asynchronous GPU work. The API persists a job, dispatches an immutable input snapshot, and records one or more durable compute attempts. The job status is the product-level state; the attempt status explains what the worker/control plane is doing.

Job endpoints

GET  /api/projects/{owner_slug}/{project_slug}/training-jobs
POST /api/projects/{owner_slug}/{project_slug}/training-jobs
GET  /api/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}
POST /api/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/cancel
POST /api/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/retry
GET  /api/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/attempts
GET  /api/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/attempts/{attempt_id}/logs

The list keeps its legacy array response for callers that do not opt in. New screens should request cursor mode:

GET /api/projects/{owner_slug}/{project_slug}/training-jobs?pagination=cursor&limit=25
GET /api/projects/{owner_slug}/{project_slug}/training-jobs?pagination=cursor&limit=25&cursor=<next_cursor>

Cursor pages return {data, next_cursor, total_count}. The default limit is 25 and the maximum is 100; authorization is applied before the exact project total and seek page are calculated. Cursors are opaque and project-job cursors must be sent back unchanged. A cursor or limit without pagination=cursor returns 400, as does a malformed cursor.

Create a job with dataset tags and the catalog model:

{
  "dataset_tags": ["support-clean"],
  "model_name": "unsloth/Qwen3.5-9B",
  "max_seq_length": 2048,
  "num_train_epochs": 3
}

num_train_epochs is how long the run trains: how many complete passes it makes over your selected examples. The server derives the optimizer steps that costs from the selection size and the batch shape, and freezes them on the job as planned_steps. Every optional setting you omit is expanded from the base model's fine-tune profile and frozen with it, so a job read back later reports exactly what it ran with.

In the app, the estimate spells this conversion out as planned steps, steps per epoch and epochs. Epochs are not another name for steps: one epoch is one full pass over the training examples, while one optimizer step processes one effective batch.

The exact conversion is:

effective_batch_size = per_device_train_batch_size × gradient_accumulation_steps
steps_per_epoch      = ceil(train_examples / effective_batch_size)
planned_steps        = steps_per_epoch × num_train_epochs

For example, 152 training examples with the default effective batch of 8 cost ceil(152 / 8) = 19 optimizer steps per epoch, or 57 planned steps for 3 epochs. A final partial batch still counts as one optimizer step.

Other settings you may send: system_prompt, learning_rate, lora_rank, lora_alpha, per_device_train_batch_size, gradient_accumulation_steps, loss_mask (assistant_only or full_sequence), base_load_precision (4bit or 8bit), warmup_steps, and early_stopping_patience. A rejected value returns 400 with a code and the field it is about.

system_prompt is one instruction for the whole job, stored once and added as the first system message when each example is formatted. It never affects inference — a hosted model still answers exactly the messages you send it. If any selected example already carries its own system message, creating the job with a system_prompt returns 422 and tells you how many conflict, rather than merging two sets of instructions.

max_steps still works and still means what it always did: a fixed ceiling of optimizer steps, between 1 and 100. Send it or num_train_epochs, never both — a request carrying both is rejected.

Use GET /api/base-models for the allowed model list, the default limits, and the per-model profiles with their bounds.

Evaluation and early stopping

If your selection is large enough, a small share becomes evaluation examples and is never trained on. The run evaluates against those examples on the same cadence it checkpoints, and the model you get is the checkpoint that scored best on them rather than the one from the last step.

Which examples are used for evaluation is a property of your dataset, not of the job: the same evaluation examples are used by every job you run over an unchanged dataset. That is what makes two runs comparable — a lower evaluation loss after you raise the LoRA rank means the change helped, rather than meaning the second job sat an easier exam. Examples you add later join whichever side their own identity puts them on; nothing already assigned moves.

The current policy attempts to assign 5% to evaluation, requires at least 8 evaluation examples, and caps that set at 512:

candidate_eval = floor(selected_examples × 0.05)
eval_examples  = 0 if candidate_eval < 8 else min(candidate_eval, 512)
train_examples = selected_examples - eval_examples

That is why evaluation starts at exactly 160 selected examples: 5% of 159 rounds down to 7, while 5% of 160 is 8. A smaller selection trains on all of its examples and simply has no evaluation loss to report. The 5% / 8 / 512 bounds are conservative product policy, not a proven statistical optimum; the remaining profile benchmark task must measure them before they are treated as calibrated defaults.

Early stopping is off unless you ask for it. Send early_stopping_patience (1 or more) to have the run stop once evaluation loss has failed to improve for that many evaluations. It is off by default because stopping early gives you a shorter run than the one you asked for, and on a small evaluation set the loss is noisy enough that the call should be yours. A job that stops early still ends with status: "succeeded", before planned_steps — that is finished, not failed.

A finished job that evaluated reports an evaluation block: the best loss and the step it came from, the last loss and step, how many evaluations ran, and which checkpoint is being served. When the served checkpoint is not the most recently evaluated one, promoted_before_last_eval is true — so a 74-step job serving the checkpoint from step 50 says so. A job that never evaluated reports evaluation: null, not zeros.

The job detail shows those values and an evaluation-loss chart. With fewer than 160 selected examples there is no evaluation chart because no examples were assigned to evaluation.

Why a run ended

A finished job's API response can carry an outcome block with two fields.

stop_reason is one of completed, max_steps, early_stopping or cancelled. It is what distinguishes a run that stopped early from one you stopped yourself — both end before planned_steps, and the step count alone cannot tell them apart.

downgrades is the worker's operational audit of settings the execution plan could not use, as setting:action:cause tokens — for example early_stopping:disabled:no_eval_set, when you asked for early stopping over a selection too small to produce evaluation examples. It can also contain an adaptation of a profile-owned setting on an older job. The app's Outcome card is therefore an exception surface: it shows early stopping, cancellation, changes to fields you actually sent, and unknown tokens; it does not repeat a routine max_steps completion or present an internal profile fallback as your changed request. The raw API record remains available for audit. outcome: null means the run reported nothing — every job that finished before this field existed, and any job that never reached the training loop.

What a job trained on

A job's detail carries the first 50 examples it trained on, with chat_count and message_count giving the real totals and chats_truncated saying when the list is a preview. The rest is paged:

GET /api/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/dataset?offset=0&limit=25
GET /api/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/dataset?offset=0&limit=25&search=invoice&split=eval

It returns items (chat id, name, message count and frozen train/eval membership), total, offset and limit, plus has_more, next_offset, search_applied and split, up to 200 per page. search matches the frozen title or message text; split is all, train or eval and uses the membership frozen for that run. A searched page returns total: null so the server does not scan every immutable shard twice just to draw a page count. One example's messages, exactly as the job used them, come from:

GET /api/projects/{owner_slug}/{project_slug}/training-jobs/{job_id}/dataset/{chat_id}

Both answer from the immutable record the job trained from, not from your current chats. If you have since edited or deleted one of those conversations, these still show what the run actually used, which is the point of freezing it.

Preparation

A created job does not start training immediately. Its selected examples are first copied, exactly as they are at that moment, into an immutable record that the run reads from — so editing or deleting those chats afterwards cannot change what a job trained on, and retrying a job re-runs the same data rather than whatever the data has since become.

The job reports this as preparation, which moves through queued, snapshotting, and then one of verified, failed or cancelled. Training begins only after verified.

Preparation failures are input failures: no GPU was allocated, so the job ends with failure_category: "input" and is not charged. The same is true of cancelling while a job is still preparing — nothing has run, so nothing is billed.

A job created before this feature reports preparation: null, which means the step did not apply to it rather than that it failed.

Job numbers and API IDs

Each job response has two identifiers. number is the one-based chronological number inside its project and is what the ReOpenly UI displays. id is the database-wide operational identifier retained for API mutations, artifacts, and related resources. To retrieve a job by the displayed number, call:

GET /api/projects/{owner_slug}/{project_slug}/training-jobs/{number}?identifier=number

Training jobs are immutable records and cannot be deleted, so project-local numbers do not shift. Hosted-model and billing responses expose the same value as training_job_number when they refer to a job.

Duration admission and pricing

Fine-tune admission uses the operational projection training_workload_seconds + 1,800 seconds of GPU-side post-processing allowance. ReOpenly rejects a projection above the platform's execution ceiling before creating the job or dispatching provider work. The ceiling is 21,600 seconds (six hours) by default and is a platform setting rather than a fixed constant, so read it from duration_components.execution_limit_seconds on POST /api/pricing/estimate rather than hardcoding it. This is ReOpenly's product limit, independent of any infrastructure-provider limit. At the default ceiling, 100 steps projects to 5,250 seconds at sequence 512 and 14,700 seconds at 2,048; sequence 4,096 accepts 76 steps (21,540 seconds) and rejects 77 (21,780 seconds).

Create returns 422 and explicit retry returns 409 with the typed code training_duration_exceeds_limit. The public POST /api/pricing/estimate response reports fine-tune expected from projected active seconds, authorization_ceiling from the separately padded credit check, and duration_components for the workload, post-processing allowance, projected active seconds, and execution limit. Queue/cold-start time and central finalization are excluded from customer GPU billing.

Dataset admission

dataset_tags must contain exact, already-existing tag names. Blank, non-string, or unknown values return a typed training_dataset_selection_invalid error. Duplicate names are removed in first-seen order, and multiple tags select the union of their chats. An empty tag list selects all project chats.

The immutable snapshot keeps every selected message, but the trainer keeps only messages with nonblank text. Create rejects an empty selection with training_dataset_empty, or a selection with no trainable text with training_dataset_unusable, before creating a job or charging credits. These responses are 422 on create and 409 on retry; retry does not mutate the job or create an attempt. The estimate endpoint reports selected_chats, selected_messages, trainable_chats, and nonempty_text_messages, returning is_trainable: false for empty or unusable selections.

List and detail responses include cost_credits, the positive actual spend aggregated from training rows in the credit ledger. A pending, failed, or legacy job with no charge row returns 0; the API never estimates historical spend from today's rate.

The detail response also includes a timing projection when the job has recorded lifecycle timestamps:

{
  "total_seconds": 540,
  "queued_seconds": 18,
  "preparing_seconds": 211,
  "training_seconds": 154.2,
  "post_processing_seconds": 42.0,
  "finalizing_seconds": 156.8,
  "attempt_count": 1
}

total_seconds is the wall clock from creation to terminal completion. The timestamp-backed values cover queueing, preparation, training, worker-side post-processing, and central finalization. While a job is active, the server uses the current time after the latest durable boundary so the UI can show the phase that is accumulating instead of omitting it. For a retry, queue time starts at that retry attempt's dispatch creation (or the retry request when the attempt timestamp is unavailable). Time between a terminal attempt and a later explicit retry remains in total_seconds, but is not mislabeled as queue time. training_seconds is the trainer's own loop. Its runtime is persisted per attempt, so a fresh retry from zero still sums the earlier attempt. A phase is omitted or null when its timestamps do not support an honest measurement, which is expected for older attempts before training_started_at was added. These elapsed times are not the billable interval: billing uses each attempt's GPU-active started_at to finished_at interval.

Job states

State	Meaning
pending	Accepted and waiting for dispatch/capacity. If it has no active attempt yet, a stop is terminalized locally without contacting a provider.
running	A fine-tune attempt is executing, GPU-side post-processing is in progress, or central finalization is in progress.
cancel_requested	The owner requested cooperative cancellation while work is active.
succeeded	The final artifact was verified and promoted.
failed	The current generation ended without a promoted artifact.
cancelled	The owner stopped the run, or no work remained for the generation. A run that had already trained keeps the adapter it produced; see Cancellation.

There is no public evaluation, deployment, or timer-driven stage. An artifact is ready for download or hosting when model_available is true, which covers a succeeded job and a stopped job that had trained. The job status is not the signal: model_output_path is assigned when the job is created, so every job carries one. A stopped job can therefore be downloadable and hostable when model_available is true even though its terminal status is cancelled.

Attempt states

The project-scoped attempts endpoint returns id, attempt_number, status, timestamps, an error/failure code, and bounded log metadata. Infrastructure identifiers and diagnostics stay on the operator surface.

State	Meaning
leased	The control plane reserved a worker/provider slot for this attempt.
accepted	A fine-tune worker accepted the execution and returned its execution identity.
started	The worker began the actual operation.
finalizing	A fine-tune handed off its immutable manifest for central verification/promotion.
succeeded	The attempt completed successfully.
failed	The attempt ended with a failure code/category.
cancelled	Cooperative cancellation completed. If the run had trained, its artifact is still verified and promoted.
expired	A lease or worker heartbeat expired; reconciliation fences the attempt.
submission_uncertain	The control plane could not prove whether a remote submission was accepted; it does not create a blind duplicate.

Inference attempts normally move leased → started → succeeded or failed. Fine-tune attempts can move leased → accepted → started → finalizing → succeeded; while the attempt remains started, its dispatch projection can briefly be post_processing before the manifest handoff. A worker loss takes a terminal path without promoting an artifact. A cancellation that produced an adapter reaches finalizing like any other run and promotes what it trained. A cancellation that happened before any adapter was produced remains terminal without an artifact.

Cancellation

POST .../cancel is cooperative and is accepted from the moment an attempt is leased until it finalizes: the job is pending or running, the fine-tune attempt is leased, accepted or started, and progress is below 100. That covers the whole of preparation — model download, dataset fetch, tokenization — which on a cold worker is the longest part of a run and used to be uncancellable. A pending job with no attempt at all is an idempotent local terminal cancellation (there is no provider work to stop). post_processing and finalizing return 409 (training_job_not_in_training; finalizing retains the more specific training_job_finalizing), because by then the GPU is released and the adapter is already uploaded. Once accepted, the job becomes cancel_requested, the worker is asked to stop, and the control plane records the provider cancel intent after the request commits. Cancellation never creates a replacement attempt.

Once the worker emits post_processing, it completes checkpointing, adapter serialization, upload, and manifest handoff even if a late cancellation desire arrives. A cancellation requested during Training can still produce the adapter described below; post-processing itself is never cancellable.

Stopping keeps what the run had learned, and is charged for it. The trainer finishes its current step, saves, and uploads a complete adapter, which is verified and promoted like any other. The job stays cancelled — the status records that you stopped it, not that nothing came of it — and the GPU time it used is billed at the usual rate. A run stopped before it trained anything produces no artifact and is not charged.

The same model_available contract controls direct training-job#<id> inference and hosted serving. A stopped job is valid when its fine-tune attempt is terminal cancelled and its adapter was promoted; cancelled without a promoted artifact is not serveable.

At central finalizing, the GPU is released and the artifact is uploaded, so there is nothing left to stop; verification and promotion continue.

Provider reconciliation and failure reasons

For a queue-based managed fine-tune, the trusted control plane—not the browser and not the isolated worker—queries the provider's /status/{id} and, after a Training stop request, /cancel/{id}. The reconciler runs on the normal 30-second dispatch cadence, uses a 10-second request timeout, at most three concurrent calls, and bounded retries for transport, 429, and 5xx responses. It accepts only IN_QUEUE, IN_PROGRESS, RUNNING, COMPLETED, FAILED, CANCELLED, and TIMED_OUT. Provider output is never an artifact or a customer-facing error; only a signed ReOpenly lifecycle callback carrying a verified manifest can enter finalization. Provider results are retained for up to 30 minutes for asynchronous reconciliation.

An active provider observation without a current signed callback gets one 120-second callback grace window. The first 404 or transient status/auth failure gets one bounded 60-second observation window; a second failure or an expired provider TTL ends the attempt. Reconciliation never creates an automatic retry or a replacement attempt. The customer can start a fresh manual retry only after a terminal job is eligible.

The public failure codes are intentionally stable and contain no provider traceback or raw output:

Code	Meaning
provider_failed	The compute provider reported a failed execution.
provider_execution_timeout	An active execution exceeded its execution deadline.
provider_ttl_expired	The provider queue/execution TTL expired.
provider_callback_missing	Active provider work was not corroborated by a signed callback.
provider_terminal_callback_missing	The compute provider completed, but no signed result manifest arrived.
provider_result_unavailable	The provider result was missing twice or past its retention/TTL boundary.
provider_status_unavailable	Repeated status/cancel control-plane checks could not establish state.
provider_cancelled	The provider cancelled an execution without a ReOpenly stop request.

Explicit retry

A failed or cancelled job can be retried, and retries are always owner-initiated. Send a unique Idempotency-Key:

curl -X POST "${REOPENLY_API_ROOT}/training-jobs/42/retry" \
  -H "Authorization: Bearer ${REOPENLY_API_KEY}" \
  -H "Idempotency-Key: 8f8d8ac4-8de5-4c63-a8f4-f0a39db1af4d" \
  -H "Content-Type: application/json" \
  -d '{"resume_from_checkpoint": false}'

The retry reuses the immutable training arguments, dataset snapshot, and canonical output root. If resume_from_checkpoint is true, the control plane requires a compatible verified checkpoint. Automatic fine-tune retries are not part of the contract.