Analyze a CSV with streaming group statistics

Summarize a CSV one row at a time, then check the saved count, mean, and sample variance against an independent calculation. This example keeps four statistical state values per group for three known groups.

Validation scope: on September 13, 2026, the current files and all 19 command units across these two guides ran in a CoCalc Basic 1.7 project on Linux x86_64 with Python 3.14.4. The CLI executed each project-shell command; 17 succeeded and both intended invalid/incomplete-data checks failed. The companion suite passed all 14 tests, including output-file preservation, bounded artifact verification, real interruption, and stalled-worker cleanup. Independent comparisons verified all expected scientific results. The files also passed local checks on macOS with Python 3.9.6 and 3.12.14. These checks validate the small examples, not a performance or capacity guarantee.

Prepare a project terminal

Use a project you can edit, Python 3.9 or later on Linux or macOS, and available CPU, memory, and storage. See choose compute, Python environments, and the terminal. This example uses only Python's standard library. Its output directory must support hard links; unsupported filesystems fail rather than replace existing files. A working project terminal does not require a Jupyter kernel.

Create a new folder named scientific-stream-demo in your project's home directory. Download scientific-prototype-v2-workflows.py and scientific-prototype-v2-tests.py from the scientific example directory and upload both files into that folder using Files. For CLI transfers, follow the remote research guide.

Run all commands below in a project terminal, from that new folder:

cd "$HOME/scientific-stream-demo"
pwd
python3 -c 'import sys; print(sys.executable); print(sys.version)'
ls scientific-prototype-v2-workflows.py scientific-prototype-v2-tests.py

Stop if the folder or interpreter is missing. Record the printed interpreter path and version. These commands run in the project, rather than in your laptop's terminal or a notebook cell.

Create a small reference dataset

This fixture contains 12,000 measurements, divided equally among control, low, and high. Its three-column format differs from the one-column measurements.csv used in the introductory analysis guide.

python3 - <<'PY'
import csv
from pathlib import Path

groups = ("control", "low", "high")
with Path("grouped-measurements.csv").open("x", newline="") as stream:
    writer = csv.writer(stream)
    writer.writerow(["sample_id", "group", "value"])
    for index in range(12000):
        group_index = index % 3
        value = group_index * 5 + ((index * 17) % 97 - 48) / 8
        writer.writerow([index, groups[group_index], value])
print("Created 12000 synthetic measurements")
PY

The input must have the exact header sample_id,group,value, sequential sample IDs starting at 0, and only those three group names. Every group needs at least two observations. Each physical CSV record must fit within 512 bytes; multiline CSV records are unsupported. Values must be finite, with absolute value at most 1e100. These bounds define this example's accepted input, not a promise of numerical accuracy for every dataset within them.

Calculate and verify the saved run

Use a fresh output prefix and leave the input unchanged during the run:

python3 scientific-prototype-v2-workflows.py stream --input grouped-measurements.csv --out streaming-run
python3 scientific-prototype-v2-workflows.py verify --out streaming-run

Each command should exit successfully and print:

verified complete: 3 results

The completed run consists of:

• streaming-run.started.json: expected groups, algorithm parameters, script hash, and Python version.
• streaming-run.results.jsonl: three result records with id, n, mean, and sample_variance.
• streaming-run.manifest.json: completion marker, input row count and hash, result count and hash.

Expected values, rounded for display:

• control: count 4000; mean -0.00109375; sample variance 12.257309224767.
• low: count 4000; mean 4.999; sample variance 12.251343085771.
• high: count 4000; mean 10.002125; sample variance 12.253730792073.

Now compare the saved results to Python's independent statistics functions. This checker deliberately loads the small reference fixture into memory; it is not the streaming approach to use for a large production input.

python3 - <<'PY'
import csv
import hashlib
import json
import statistics
import unittest
from pathlib import Path

check = unittest.TestCase()
groups = {name: [] for name in ("control", "low", "high")}
source = Path("grouped-measurements.csv")
with source.open(newline="") as stream:
    for row in csv.DictReader(stream):
        groups[row["group"]].append(float(row["value"]))
records = [json.loads(line) for line in Path("streaming-run.results.jsonl").read_text().splitlines()]
check.assertEqual(len(records), 3)
actual = {record["id"]: record for record in records}
check.assertEqual(set(actual), set(groups))
for group, values in groups.items():
    check.assertEqual(len(values), 4000)
    check.assertEqual(actual[group]["n"], len(values))
    check.assertAlmostEqual(actual[group]["mean"], statistics.mean(values), places=12)
    check.assertAlmostEqual(actual[group]["sample_variance"], statistics.variance(values), places=11)
evidence = json.loads(Path("streaming-run.manifest.json").read_text())["evidence"]
check.assertEqual(evidence["input_rows"], 12000)
check.assertEqual(evidence["input_sha256"], hashlib.sha256(source.read_bytes()).hexdigest())
print("PASS: counts, independent statistics, and input hash")
PY

An assertion failure means the saved run failed this reference comparison. Keep the files and investigate before adapting the example to research data.

Recognize rejected and incomplete runs

Create a separate invalid input and a separate output prefix:

python3 - <<'PY'
from pathlib import Path
with Path("bad-measurements.csv").open("x") as stream:
    stream.write("sample_id,group,value\n0,control,nan\n")
PY
python3 scientific-prototype-v2-workflows.py stream --input bad-measurements.csv --out rejected-run
python3 scientific-prototype-v2-workflows.py verify --out rejected-run

The last two commands should each exit with status 1: the first rejects the non-finite value; the second reports incomplete run: no completion manifest. A .started.json or .partial.jsonl file alone is not a completed result.

The streaming calculation holds four state values per group. The hosted test measured peak traced Python allocations of 285,221 and 285,326 bytes for 6,000 and 60,000 rows with the same three groups; that does not measure total process memory or establish a CoCalc memory quota. The input hash identifies bytes consumed by the reader, not an atomic snapshot of a concurrently edited file.

verify checks the saved schema, expected IDs, counts, and checksums. It does not independently recalculate the science or authenticate the artifacts. Retain independent numerical checks when changing the data or algorithm.

Hand off and clean up

Keep the input, both example files, and all three completed-run files together. Add a short README with the commands, interpreter path/version, compute image when applicable, and the independent check result. Review environment details before publishing. Follow the research handoff guide for collaborator instructions.

After saving any results you need, remove only the new scientific-stream-demo folder through Files. Do not stop a shared project or remove another person's work. Completion files are not a backup or proof of recovery after a project or host failure.

Continue with a bounded parallel CPU sweep to compare worker counts without changing the scientific result.