# Run a public development check

WSPBench evaluates whether a model can read or write numerical neural network weights. This v0.2.1 development release includes a Python evaluator and 28 public examples. Twelve examples use the current Boolean candidate families. The other examples are diagnostic controls.

The first check needs Python 3.11 to 3.14 and a CPU. Python 3.13 is the tested version. It needs no API key, GPU, or Docker. Installing dependencies requires internet access.

## Install and check the evaluator

Download the source and development examples from wspbench.com. Keep both ZIP files in the same directory. These commands use a macOS or Linux shell.

```sh
unzip wspbench-source-v0.2.1.zip
cd wspbench-v0.2.1
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
unzip ../wspbench-development-v0.2.1.zip -d examples-data
python examples/check_submission.py --check-references
```

On Windows, extract the archives in File Explorer and activate the environment with `.venv\Scripts\Activate.ps1` in PowerShell. Use `py -3.13 -m venv .venv` if `python3` is unavailable.

The final command checks the supplied reference answers for all 12 Boolean candidate examples. It prints JSON with `checked: 12` and `exact: 12`. This verifies the evaluator and archive. It does not measure an AI model.

The package metadata and task schema still use benchmark version 0.2.0. The composed-circuit design and these downloads are revision v0.2.1.

## Give a task to your model

List the supported public examples.

```sh
python examples/check_submission.py --list
```

Choose an ID from that list. For example, the following public Write task starts with zero weights and specifies a complete truth table.

```sh
python examples/check_submission.py --task examples-data/0ccf05ea24942ed4 --prompt > prompt.json
```

Send `prompt.json` to your solver with this instruction: complete the task and return one JSON object matching its answer schema. The export uses the application's A/full solver payload and removes evaluator-only task identifiers and family labels. The task's published grammar, circuit layout, and specification remain available.

For an A trial, disable tools, network execution, and feedback. Give the solver one attempt. Do not show it `reference.json`, evaluator source, or previous answers. For a provider API, your own script sends the prompt and writes the final answer to `answer.json`. The public checker makes no provider calls and does not enforce these solver restrictions.

## Check an answer

```sh
python examples/check_submission.py --task examples-data/0ccf05ea24942ed4 --submission answer.json
```

The checker evaluates all 1,024 public Boolean inputs. It reports agreement and exact correctness, and exits with code 0 for an exact answer, 1 for an incorrect answer, or 2 for a malformed submission or input error. Rule recovery accepts a Boolean program or a complete output table. Edit effects accepts before/after programs or before/after output tables. Write accepts a numeric patch. The prompt contains each task's schema and examples.

For example, a valid but generally incorrect Write submission is:

```json
{"patch": []}
```

It makes no edits to the zero starting weights. Use it to confirm that an ordinary wrong answer is graded rather than rejected. Write output imbalance can make this answer look accurate, so always report exact correctness alongside accuracy.

These are exhaustive public development checks. Their agreement values are not interchangeable with the separate final-case metrics in the published model comparison. Development examples and reference answers are public, so they cannot provide an uncontaminated held-out score.

## Report your experiment

Keep the model name and immutable snapshot if available, reasoning setting, access condition, weight view, task IDs, attempt count, final answer JSON, and evaluator output. Report invalid answers, errors, elapsed time, and API usage separately. Never publish API keys. Identify these results as public development results.

Official submissions and a held-out evaluation service are not open yet. The development comparison does not establish an official ranking. Conditions B and C require the separate application's metered tool adapters and trace review. A local checker cannot certify access-condition compliance. No online upload form or automatic leaderboard acceptance is available in this release.

## License

The source bundle includes the Apache-2.0 license and NOTICE. It contains the Python application and the public checker. It excludes API credentials, private tasks, provider traces, and local billing records.