Troubleshooting

Scope

This guide covers the public Docker workflow exposed by poc.sh. It does not describe direct internal-script execution, artifact repair, or production operations. A failed exact obligation is diagnostic evidence, not a result to normalize, bypass, or route through a fallback.

Quick diagnostic sequence

Record the environment and exercise the public gates in this order:

docker version
docker compose version
uname -srm
git rev-parse HEAD
git status --short

./poc.sh test
mkdir -p .runs
./poc.sh run --run-id review-01 \
  >.runs/review-01.output.json \
  2>.runs/review-01.progress.jsonl
./poc.sh check --run-id review-01

jq, gzip, sha256sum, and rg appear in optional inspection examples. They are not prerequisites for the public PoC commands.

Public command behavior

Command	Successful effect
./poc.sh build	Builds the pinned image and creates .runs/ if absent, but does not create a run.
./poc.sh test	Runs the focused contract modules, one temporary real canary, artifact-tampering probes, and test-only semantic mutants. The canary is removed afterward.
./poc.sh run	Uses poc mode, creates a new run, reconstructs it twice, and prints one aquila-fusion-poc-output/v1 document.
./poc.sh run --mode MODE	Runs canary, representative, pilot, or poc mode with 1, 4, 28, or 4 units, respectively. Every mode also exercises four official negative cases.
./poc.sh check --run-id ID	Reconstructs and compares an existing completed run without repairing or rewriting it.

An omitted run ID is generated as the mode, a UTC timestamp, and eight lowercase hexadecimal characters. A caller-provided ID must begin with a lowercase letter or digit, contain only lowercase letters, digits, dots, underscores, or hyphens, and contain at most 64 characters.

Streams and exit behavior

• Application progress is compact JSON Lines on standard error.
• A successful run, check, or test writes one formatted JSON document to standard output.
• Exit status 0 means the public command completed its exact contract.
• Exit status 2 reports launcher validation, argument parsing, a controlled application error, a malformed artifact, or a failed exact obligation.
• Docker build or runtime failures may propagate another nonzero Docker or Compose status.
• An internal verifier status of 1 is expected for each of the four official negative cases. It is not a public-command failure when the intended mutation is rejected for the required reason.

Common symptoms

Symptom	Meaning and response
Docker is required or the launcher reports Docker Compose v2 is required	Install or start the missing prerequisite, verify both version commands, and rerun the same public command. The second message names the required plugin-style docker compose interface; no minimum version number beginning with 2 has been established.
AQUILA_UID/GID must be a canonical positive decimal	Run as a non-root user, or provide matching positive numeric IDs that can own .runs/. Zero, leading-zero, nonnumeric, and out-of-range values are rejected.
AQUILA_SOURCE_REVISION differs from the Git revision	Remove the conflicting override or provide the exact observed 40-character commit. Do not label an unrelated revision as the source.
runtime posture does not satisfy the container contract	The effective process posture differs from the required Compose posture. Use the supplied launcher and inspect local Compose overrides; do not relax the controls to make the gate pass.
run id must contain...	Correct the identifier. Paths, uppercase letters, and values longer than 64 characters are invalid.
run directory already exists or refusing overwrite	Runs and retained artifacts are create-only. Check the existing completed run or choose a new run ID; there is no resume operation.
Permission denied below /output	The selected UID and GID cannot write the host .runs bind mount. Restore appropriate ownership for the non-root host user; do not run the container as root or make the directory world-writable.
A progress record has "status":"failed"	Inspect the named phase's process receipt and matching standard-error log. Preserve the partial run until diagnosis is complete.
terminal_status begins with cap_triggered:	Inspect cap_trigger, timed_out, peaks, and cleanup fields in the process receipt. A capped stage is not a successful measurement.
semantic oracle failed	Inspect the unit's verification.json and exact failures labels. Do not edit the oracle or result to reconcile the failure.
A negative mutant survived or failed for the wrong reason	One of the four official falsification gates failed. Preserve the run and diagnose the named case.
A raw or result manifest mismatch, or an exact-file-set mismatch	Artifact bytes, membership, or paths changed. Do not regenerate hashes or repair the run. Produce a new run after identifying the cause.
Public output, analysis, or reproduction differs from reconstruction	A derived result no longer matches retained evidence. Preserve the complete run for comparison.
The plan differs from the current fixed definition	The checker is using a different source context. Use the recorded revision and matching repository inputs.
The runtime manifest differs from current executable identities	The current image or toolchain is not the one recorded for the run. Recheck with the matching pinned environment.

Image pulls require registry access during build; the runtime network restriction does not make the Docker build offline. Diagnose registry, proxy, DNS, or daemon failures outside the PoC rather than weakening the runtime network contract.

Inspect a failed stage

List retained logs and select the receipt named by the failed progress record:

run=.runs/review-01

rg --files "$run/logs"

jq '{
  terminal_status,
  returncode,
  timed_out,
  cap_trigger,
  reaped,
  containment_domain_empty,
  residual_processes,
  peaks
}' "$run/raw/<unit-id>/<stage>-process.json"

jq '{status, failures, checked_obligations}' \
  "$run/raw/<unit-id>/verification.json"

The outer stage logs are operational copies. Progress JSONL is not retained as a run artifact unless the operator redirects standard error, as shown in the quick diagnostic sequence.

Partial runs

The runner first creates raw/, results/, work/, and logs/. A stage failure can leave a partial create-only directory:

1. raw.sha256 and results/run-summary.json appear only after all requested units and the four official negative cases complete.
2. The analyzer creates the remaining result files and results.sha256 only after it accepts the retained raw evidence.
3. check requires a complete passing run. It is a read-only verification command, not recovery or resume machinery.

Safe cleanup

There is no project cleanup command. Preserve any run needed for diagnosis or independent review. To remove one expendable run, validate one literal run ID, inspect the exact target, and require confirmation:

run_id=review-01
[[ "$run_id" =~ ^[a-z0-9][a-z0-9._-]{0,63}$ ]] || exit 2

target=".runs/$run_id"
[ -d "$target" ] && [ ! -L "$target" ] || exit 2

rg --files "$target"
read -r -p "Type the run ID to delete: " confirmation
[ "$confirmation" = "$run_id" ] || exit 2

rm -r -- "$target"

The deletion is irreversible unless the run was copied elsewhere. Never replace the exact target with .runs/*, a repository-wide path, or an unresolved variable. Do not use docker system prune as PoC cleanup; run deletion does not require removing Docker images or caches.

Report a failure

Follow CONTRIBUTING.md. Include the exact command and status, commit, Docker and Compose versions, host operating system and architecture, public output when present, input inventory, runtime manifest, and the smallest relevant standard-error log. Do not upload a complete run unless requested, and review every attachment for sensitive content first.

See poc/ARTIFACTS.md for the retained-file reference.