DocsLog in

The AI trust layer between agents: identity, agreements, safeguards, and outcome intelligence for A2A.

Commit 6df07fcd

6df07fcdd05cMerged

aryaamin <[email protected]> · 2 months ago · 9 files · +1 added · ~8 modified

Browse files at this commit

Description

fix: stabilize shield benchmark execution

Details

Commit
6df07fcdd05c
Parent
4e1e50d12753

benchmarks/tau3/README.mdmodifiedView file

@@ -50,11 +50,13 @@
5050 uv run python ../openclasp/benchmarks/tau3/run_experiment.py \
5151 --mode baseline --domain airline --task-ids 0 --num-trials 1 \
5252 --agent-llm openai/gpt-4.1 --user-llm openai/gpt-4.1 \
53+ --agent-llm-args '{}' --user-llm-args '{}' \
5354 --save-to oc_baseline_smoke --summary-file oc_baseline_smoke.json
5455
5556 uv run python ../openclasp/benchmarks/tau3/run_experiment.py \
5657 --mode shield --domain airline --task-ids 0 --num-trials 1 \
5758 --agent-llm openai/gpt-4.1 --user-llm openai/gpt-4.1 \
59+ --agent-llm-args '{}' --user-llm-args '{}' \
5860 --save-to oc_shield_smoke --summary-file oc_shield_smoke.json
5961 ```
6062
@@ -62,15 +64,16 @@
6264
6365 ## 4. Run a controlled sample
6466
65-Use identical domain, models, task IDs, trials, seed, temperature, and concurrency for all three
67+Use identical domain, models, model arguments, task IDs, trials, seed, and concurrency for all three
6668 runs. Start with 20 airline tasks and four trials:
6769
6870 ```bash
6971 for mode in baseline generic-review shield; do
7072 uv run python ../openclasp/benchmarks/tau3/run_experiment.py \
7173 --mode "$mode" --domain airline --num-tasks 20 --num-trials 4 \
72- --seed 300 --temperature 0 --max-concurrency 1 \
74+ --seed 300 --max-concurrency 1 \
7375 --agent-llm openai/gpt-4.1 --user-llm openai/gpt-4.1 \
76+ --agent-llm-args '{}' --user-llm-args '{}' \
7477 --save-to "oc_${mode}_airline_20x4" \
7578 --summary-file "oc_${mode}_airline_20x4.json"
7679 done
@@ -100,3 +103,6 @@
100103 transport debugging.
101104 - The adapter deliberately uses `workers=0`; thread concurrency is supported, process workers are not
102105 because post-run outcomes must map back to their Shield cases.
106+- Per-task τ³ logs are enabled automatically because the current custom-agent interface exposes the
107+ simulation identifier through that context. This identifier is used locally for outcome joins and
108+ is not sent to Shield.