Resume a computation after interruption

Run a small computation that saves each completed case, deliberately fail it, and resume without recomputing finished cases. You will verify the saved results rather than infer success from a still-open terminal.

This is the executable companion to Research Runs That Survive. A persistent project keeps files available; it cannot preserve process memory through a project restart, host failure, or every connection outage.

Prepare a disposable run

In a project with Python 3, create a new research-workflow folder and upload sweep.py and summarize_run.py from the complete example directory. Open a CoCalc terminal file in that folder. Record its filename so you can return to the same terminal; opening a new terminal starts a different session.

cd /home/user/research-workflow
mkdir -p runs

The scripts use only the Python standard library. Use a fresh run directory for each changed experiment and only one worker per directory. The sample validates existing checkpoint contents before skipping them; it is not a general job scheduler or a solution for multiple simultaneous workers.

Deliberately fail after three completed cases

The example computes squares for cases 0 through 5. The first command below is expected to fail, after saving cases 0, 1, and 2:

python3 -u sweep.py --out runs/failure-demo --stop 6 --fail-at 3 > runs/failure-demo.log 2>&1
run_status=$?
printf 'exit=%s\n' "$run_status"
cat runs/failure-demo.log

Expected status: exit=1. The log begins with:

completed case=0
completed case=1
completed case=2

It ends with an intentional failure at case 3. If run from a script using set -e, handle this expected failure explicitly so the wrapper does not stop before recording the status. Logs are redirected directly here; piping through tee requires care to preserve the Python command's exit status.

Inspect the completed checkpoints:

ls runs/failure-demo/case-*.json
python3 summarize_run.py runs/failure-demo --stop 6

The summary should fail with an incomplete-cases error. That failure is useful: three result files are not a completed six-case experiment.

Resume and verify

Run the same experiment without the deliberate failure flag:

python3 -u sweep.py --out runs/failure-demo --stop 6 >> runs/failure-demo.log 2>&1
run_status=$?
printf 'exit=%s\n' "$run_status"
tail -n 6 runs/failure-demo.log
python3 summarize_run.py runs/failure-demo --stop 6

Expected final log lines:

skip case=0
skip case=1
skip case=2
completed case=3
completed case=4
completed case=5

Expected status is exit=0, and the summary prints:

completed=6 sum_of_squares=55

The sample writes each result to a temporary file and renames it after writing. After interruption, an unfinished .tmp file is not a completed case and can be recomputed. This reduces partial-result confusion; it is not a guarantee of survival through storage failure. Keep configured backups for important work.

Try reconnecting to a running terminal

Use a different run directory and a longer per-case delay:

python3 -u sweep.py --out runs/reconnect-demo --stop 6 --delay 5 > runs/reconnect-demo.log 2>&1

Close only the browser tab while the command is running, then reopen the project and the same terminal file. Do not stop or restart the project. Inspect the log from another terminal if needed:

tail -n 10 /home/user/research-workflow/runs/reconnect-demo.log

When execution is finished, run the summary on runs/reconnect-demo. If the session or process ended, first check whether a worker is still running before starting another. Resume using the same output directory only after the prior worker has stopped. Do not assume a missing browser response means the command never ran.

Choose the right recovery action

For notebooks, the CLI notebook workflow explains how to retain a detached run ID and inspect execution. Detaching the request does not make an experiment checkpoint itself. For a remote SSH kernel, files written by code live on the remote machine and need their own recovery plan.

Hand off the run

Keep the scripts, run directory, log, input or parameter description, and final summary together. Record the exit status and last verified completed case in the handoff note. A collaborator should be able to run the summary without rerunning the experiment first. For a real workload, include the code revision, environment, input hashes, and a restart command.

Once the example is reviewed, remove only its two run directories if you no longer need them. Keep the scripts for reuse, and choose a new directory when changing their computation.