Run and verify a bounded parallel CPU sweep

Calculate a small family of numerical integrals with one and two worker processes. A single coordinator writes the results; independent serial and analytic checks test the calculation. This is a bounded process-pool example, not a speedup benchmark.

Validation scope: on September 13, 2026, the current files and all 19 command units across these two guides ran in a CoCalc Basic 1.7 project on Linux x86_64 with Python 3.14.4. The CLI executed each project-shell command; 17 succeeded and both intended invalid/incomplete-data checks failed. The companion suite passed all 14 tests, including output-file preservation, bounded artifact verification, real interruption, and stalled-worker cleanup. Independent comparisons verified all expected scientific results. The files also passed local checks on macOS with Python 3.9.6 and 3.12.14. These checks validate the small examples, not a performance or capacity guarantee.

Prepare a project terminal

Use a project you can edit and an existing Python 3.9 or later interpreter on Linux or macOS. This example uses only the standard library. Its output directory must support hard links; unsupported filesystems fail rather than replace existing files. A working project terminal does not require a Jupyter kernel. Check your compute allocation before running up to two workers plus their coordinator and possible Python resource-tracker process. A worker setting does not reserve CPU cores or establish a memory limit. Avoid competing work in the same project during the comparison.

Create a new folder named scientific-cpu-demo in your project's home directory. Download scientific-prototype-v2-workflows.py and scientific-prototype-v2-tests.py from the scientific example directory and upload both into that folder using Files. For CLI transfers, follow the remote research guide.

Run these commands in a project terminal:

cd "$HOME/scientific-cpu-demo"
pwd
python3 -c 'import sys; print(sys.executable); print(sys.version)'
ls scientific-prototype-v2-workflows.py scientific-prototype-v2-tests.py

Stop if the folder or interpreter is missing. Record the interpreter path and version. The script uses a guarded command-line entry point and Python's spawn process start method; these instructions do not create a process pool inside a notebook cell.

Define eight cases

For each rate, calculate the integral of exp(-rate*x) from 0 to 1 with 20,000 midpoint intervals. Create the exact small fixture:

python3 - <<'PY'
import json
from pathlib import Path
with Path("rates.json").open("x") as stream:
    json.dump([0, 0.125, 0.5, 1, 2, 4, 8, 16], stream)
print("Created eight distinct rates")
PY

The example accepts a JSON file of at most 4,096 bytes containing 1–32 unique, finite numeric rates from 0 to 100. Its argument limits are 1–4 workers and 10–500,000 intervals. The reference checks use only one and two workers; keep those small settings for this exercise.

Compare one worker with two

Use separate fresh output prefixes:

python3 scientific-prototype-v2-workflows.py parallel --rates rates.json --out cpu-one-worker --workers 1 --intervals 20000
python3 scientific-prototype-v2-workflows.py parallel --rates rates.json --out cpu-two-workers --workers 2 --intervals 20000
python3 scientific-prototype-v2-workflows.py verify --out cpu-one-worker
python3 scientific-prototype-v2-workflows.py verify --out cpu-two-workers

Each computation prints started bounded CPU sweep, eight completed rate-... lines, and finally verified complete: 8 results. Completion order can vary. Each separate verification command should also print verified complete: 8 results and exit successfully.

Each prefix has three completed-run files:

The coordinator keeps at most one pending task per worker. Workers return records; only the coordinator writes these artifacts. A .partial.jsonl file without a completion manifest is incomplete even if it contains plausible numbers.

Check against independent calculations

This checker uses a reverse-order serial sum and the closed-form integral. It also compares all saved records between the two worker counts:

python3 - <<'PY'
import hashlib
import json
import math
import unittest
from pathlib import Path

check = unittest.TestCase()
rates = json.loads(Path("rates.json").read_text())
check.assertEqual(rates, [0, 0.125, 0.5, 1, 2, 4, 8, 16])
intervals = 20000
outputs = []
for prefix in ("cpu-one-worker", "cpu-two-workers"):
    records = [json.loads(line) for line in Path(prefix + ".results.jsonl").read_text().splitlines()]
    check.assertEqual(len(records), 8)
    actual = {record["id"]: record for record in records}
    check.assertEqual(set(actual), {f"rate-{index:03d}" for index in range(8)})
    for index, rate in enumerate(rates):
        record = actual[f"rate-{index:03d}"]
        check.assertEqual(record["rate"], rate)
        check.assertEqual(record["intervals"], intervals)
        total = 0.0
        for cell in reversed(range(intervals)):
            total += math.exp(-rate * (cell + 0.5) / intervals)
        check.assertAlmostEqual(record["integral"], total / intervals, places=12)
        exact = 1.0 if rate == 0 else -math.expm1(-rate) / rate
        bound = rate * rate / (24 * intervals * intervals) + 1e-14
        check.assertLessEqual(abs(record["integral"] - exact), bound)
    manifest = json.loads(Path(prefix + ".manifest.json").read_text())
    check.assertEqual(manifest["specification"]["parameters"]["input_sha256"], hashlib.sha256(Path("rates.json").read_bytes()).hexdigest())
    outputs.append(actual)
check.assertEqual(outputs[0], outputs[1])
print("PASS: eight cases, independent serial and analytic checks, equal worker results")
PY

The rate-zero result is 1.0; the rate-one result is approximately 0.632120558763. The error bound applies to this integrand and interval, not to arbitrary numerical workloads. A failed assertion requires investigation before using the example as a research result.

Handle failures and interruption

For a deliberately resumable example, use the checkpointed computation guide. The pool's context manager may wait for running tasks during shutdown; do not assume an interrupted command has already stopped every process.

For a broader regression check, run the companion suite from this new folder, with both files beside one another:

python3 -B scientific-prototype-v2-tests.py

The recorded hosted result was Ran 14 tests in 12.472s followed by OK. The suite writes PID-prefixed synthetic fixtures beside itself, checks malformed inputs and 42 checksum-consistent corruptions, simulates a worker failure, and interrupts a real child run. It includes a two-worker interruption case with 16 rates and 500,000 intervals. Per-process CPU limits and subprocess timeouts bound that test; they do not impose a total RAM limit. Its resource module requires a compatible Unix-like runtime. Your runtime and elapsed time can differ.

verify checks schema, expected cases, counts, and checksums; it does not independently recompute the science or authenticate the artifacts. The interruption test does not establish recovery from a project stop, host failure, out-of-memory kill, or network reconnection.

Hand off and clean up

Keep rates.json, both scripts, and all three files for each completed prefix. Add a README with the exact commands, interpreter path/version, compute image when applicable, and the independent comparison result. Review environment details before publishing; see the research handoff guide.

After the commands have exited and you have retained the results, remove only the new scientific-cpu-demo folder through Files. Do not stop another person's processes or a shared project. These completion files do not replace backups.

For large row-oriented inputs, see streaming group statistics.