Checkpointing

Tinker supports two types of checkpoints and provides utilities for managing them during training.

Reference

Read these for details:

• tinker_cookbook/checkpoint_utils.py — CheckpointRecord, save/load helpers
• docs/save-load.mdx — Checkpointing guide (save_weights_for_sampler vs save_state)

Two checkpoint types

Type	Method	Purpose	Contains
State	save_state()	Resume training	Weights + optimizer state
Sampler	save_weights_for_sampler()	Sampling / export	Weights only

# Save full state (for resumption)
tc.save_state(name="step_100", ttl_seconds=None)

# Save sampler weights (for sampling/export)
tc.save_weights_for_sampler(name="step_100_sampler", ttl_seconds=None)

# Save both + get a SamplingClient
sc = tc.save_weights_and_get_sampling_client(name="step_100")

ttl_seconds=None means indefinite retention. Set a TTL for intermediate checkpoints to avoid storage bloat.

CheckpointRecord

Typed dataclass for checkpoint bookkeeping:

from tinker_cookbook.checkpoint_utils import CheckpointRecord

record = CheckpointRecord(
    name="step_100",
    batch=100,
    epoch=1,
    final=False,
    state_path="tinker://...",
    sampler_path="tinker://...",
    extra={"eval_loss": 0.5},  # User metadata
)

# Serialize
d = record.to_dict()

# Deserialize
record = CheckpointRecord.from_dict(d)

# Check if a field is set
record.has("state_path")  # True

Save/load helpers

from tinker_cookbook import checkpoint_utils

# Save checkpoint (async)
paths = await checkpoint_utils.save_checkpoint_async(
    training_client=tc,
    name="step_100",
    log_path="/tmp/my_run",
    loop_state={"batch": 100, "epoch": 1},
    kind="both",           # "state", "sampler", or "both"
    ttl_seconds=None,
)
# paths = {"state_path": "tinker://...", "sampler_path": "tinker://..."}

# Load checkpoint list
records = checkpoint_utils.load_checkpoints_file("/tmp/my_run")

# Get last checkpoint
record = checkpoint_utils.get_last_checkpoint(
    "/tmp/my_run",
    required_key="state_path",  # Only return records with this field
)

Resuming training

The standard pattern (used by supervised/train.py and rl/train.py):

# In CLIConfig
behavior_if_log_dir_exists: cli_utils.LogdirBehavior = "ask"  # "ask", "delete", "resume"

# In training loop
if config.load_checkpoint_path:
    tc.load_state_with_optimizer(config.load_checkpoint_path)

Set behavior_if_log_dir_exists=resume to continue from the last checkpoint in an existing log directory.

Managing checkpoints (REST API / CLI)

Beyond saving and loading during training, you can manage checkpoints via the REST API or CLI. See /tinker-sdk for RestClient details and /tinker-cli for CLI commands.

from tinker import ServiceClient
rest = ServiceClient().create_rest_client()

# List all your checkpoints
checkpoints = rest.list_user_checkpoints(limit=100)

# Publish a checkpoint (make it publicly accessible)
rest.publish_checkpoint_from_tinker_path("tinker://...")

# Set TTL (auto-delete after N seconds)
rest.set_checkpoint_ttl_from_tinker_path("tinker://...", ttl_seconds=86400)

# Delete a checkpoint
rest.delete_checkpoint_from_tinker_path("tinker://...")

Or via CLI:

tinker checkpoint list
tinker checkpoint publish <TINKER_PATH>
tinker checkpoint set-ttl <TINKER_PATH> --ttl 86400
tinker checkpoint delete <TINKER_PATH>

Common pitfalls

• Use save_state for resumable checkpoints, save_weights_for_sampler for sampling/export
• get_last_checkpoint() returns None if no matching checkpoint exists — always check
• Checkpoint paths start with tinker:// — they reference remote storage, not local files
• Set ttl_seconds on intermediate checkpoints to avoid accumulating old weights
• For RLHF pipelines, the SFT stage saves state_path (for RL init) and the RM stage saves sampler_path (for reward scoring)
• delete is permanent — there is no undo