Research Protocol v0.1
This is how Testrian Lab decides whether an idea is real. AI agents generate and run the research; this protocol is the judge. It is designed to kill ideas cheaply, count every test, and make our live record checkable without exposing the strategies that produce it.
Contents
01Principles
- Consensus is never evidence. Language models share training data, so agreement between them is correlated opinion. Only data the models have never seen can support an idea.
- Every test is counted. Every idea, variant and run that touches data enters the trial count used to judge significance. Failures are counted with the same care as successes.
- Freeze before testing. Hypotheses, data, metrics and kill criteria are fixed and hashed before any test runs.
- Commit before trading. Every order is committed with a salted hash before it is sent.
- The judge can't be bribed. Validation code is locked. The AI agents that do research can read it, never change it.
- Methodology public, live edge never. See section 09.
02The AI council
Research ideas come from a council of frontier models from several vendors. Each role is played by a different model or service, so no model ever grades its own homework.
| Role | Implementation | Rule |
|---|---|---|
| Proposers | Frontier models from three different vendors | Blinded, abstract prompts; one source field per batch; exact model ID, temperature and prompt-template hash pinned on every idea card |
| Skeptic | Rotating model, always from a different vendor than the card's proposer | Attacks prior art, the math, look-ahead bias, costs and economic plausibility |
| Verifier | Tool-using agent, from another vendor where possible | Fetches every cited source and must extract a quoted span confirmed by a deterministic string match; checks derivations symbolically; any failure rejects the card |
| Clerk | Deterministic software, no model | IDs, timestamps, salted SHA-256 hashes and ledger writes |
Field quotas
Every month the council must draw ideas from several source fields, including statistical physics, information theory, control and filtering, ecology and epidemiology, and market microstructure. This keeps the search broad instead of drifting toward what has worked before.
Novelty gate
A new card must have embedding similarity below 0.85 to every past card, or it is closed as a duplicate.
Decoys
Every month, a few decoy cards with known flaws (look-ahead, survivorship bias, impossible costs) are mixed in blind. The skeptic and verifier together must kill at least 90% of them. The decoy kill rate is published.
03The pipeline
Stage 0: the idea card, before any data
Every idea starts as a card that records: the source field and method; the proposed mechanism; a falsifiable prediction with direction and magnitude; the expected trade frequency and expected gross edge per round trip; the data needed; the parameter space; and prior art.
Each card carries a frozen model cutoff: the latest training cutoff of any model that worked on it, or the model's release date when the cutoff is undisclosed.
Tier S: automated screening
- Development data only, with a small declared parameter budget.
- Every run and every card counts toward the effective trial count.
- A cost-aware pre-filter kills any idea whose expected gross edge per round trip is below twice the full round-trip cost, before any compute is spent.
- Runs under a hard monthly compute budget.
Tier P: the promotion track
A limited number of strategy families per month enter promotion. Each passes, in order:
- Family registration, salted, hashed and committed.
- Full development runs.
- Probability of Backtest Overfitting (PBO) via combinatorially symmetric cross-validation (CSCV).
- Deflated Sharpe Ratio (DSR) using the effective trial count (section 05).
- N-version re-implementation by a model from another vendor.
- A sealed holdout, run once.
- Paper trading against pre-registered expectations.
- Approval by the lab.
- Live trading at a small, fixed size.
Contamination rule
History from before a card's frozen model cutoff is only pseudo-out-of-sample: the models may have seen it. It can kill an idea but can never, on its own, justify capital. Paper and live trading are the true test.
Magnitude discipline
Published effect sizes are discounted. New effects must clear a t-statistic of about 3 on top of the Deflated Sharpe test.
04The validation stack
All of these are automated. They run on compute, not on calendar time.
| Test | What it does | Pass threshold |
|---|---|---|
| Null battery | At least 300 null strategy families per pipeline version, on structure-preserving surrogates of real data (stationary block bootstrap, GARCH-fitted simulations, phase-randomized surrogates). The pipeline must find nothing. | 95% upper confidence bound on the false-positive rate ≤ 2% |
| Planted-signal curve | Signals planted at net-of-cost strength, with the same data length as real development data. The pipeline must recover them. | ≥ 80% recovery at annualized Sharpe 1.5; ≥ 50% at 1.0; full curve published |
| Known-answer tests | Published worked examples of PBO, DSR, MinTRL and CSCV, plus unit tests on every statistic | Exact match within tolerance, on every commit |
| Leakage tests | Features shifted forward in time; point-in-time proof for every feature | Performance must collapse under the shift |
| Data integrity | Price data cross-checked across two independent sources; delisted instruments included; timestamps checked against a reference clock | ≥ 99.9% agreement |
| N-version re-implementation | A model from another vendor implements the strategy from the written spec alone | ≥ 99% of bars agree on the signal |
| Adverse-selection fill model | A passive order only counts as filled if price trades through it; queue position approximated | Paper fill rate within ±10 percentage points of the model |
| Cost stress | Rerun at 1.5× and 2× the full round-trip cost | Positive net at 1.5×; result reported at 2× |
| Standard-fee stress | Rerun at standard fees, without discounts or promotions | Reported |
| Adversarial review | A skeptic from another vendor, with full code access | No unresolved high-severity finding |
| Reproducibility | Clean container rebuild from the commit hash | Results within tolerance |
| Development-data decay | Logs which data each trial touched; a window used by more than half of trials in 90 days is marked exhausted and replaced | Automatic |
| Market judge | Paper, then live, against pre-registered distributions and kill rules | Continuous (section 07) |
Validation health
The null false-positive bound, planted-signal recovery, test status, decoy kill rate, burned holdouts and exhausted data windows are reported weekly. Any red metric stops all promotions automatically. Strategies already live keep running under their own kill rules.
05Counting trials honestly
A strategy that survives 1,000 attempts means much less than one that survives 10. We cluster trials by the correlation of their returns (ONC clustering) and report raw and effective trial counts side by side. The count is kept within a data vintage, meaning all trials that touched the same development window, and resets only when a fresh, never-touched window enters. Discarded cards that reached data still count.
Across all promotion-track families, a lab-wide Benjamini–Yekutieli false-discovery rate is controlled at q = 0.10.
06Locked validation code
The most likely failure of an AI-run lab is agents "fixing" the tests that block them. So the validation code, thresholds, cost and fill models, and the null and planted-signal generators live in a separate, protected repository:
- signed commits only, and only the lab can approve changes;
- research and execution agents have read-only access;
- automated checks block any validation version that the lab has not signed.
Every pre-registration records the SHA-256 of the validation configuration it was judged under. Any change to validation code requires approval and a full rerun of the null battery, the planted-signal curve and the known-answer suite before it can judge anything.
07The live record
Pre-registration
Before a strategy trades, a record is frozen containing: the card ID, the code commit, the hash of the validation configuration, sizing and risk limits, the cost and fill model, the expected distributions of results per trade and per day, and the kill and downgrade rules. It is salted with a 256-bit random salt, hashed with SHA-256, committed to a public append-only log mirrored on GitHub, and anchored with OpenTimestamps. The timestamp anchor is the root of trust, since git history alone can be rewritten.
Commitments
Before each order is sent, an automated process publishes the salted hash of the intended orders and positions. This starts with the first live trade on 1 December 2026. The salt means no one can read the orders from the hash.
Reveals
- Public: commitment hashes, and aggregated daily results rounded and published on a delay of 60 to 90 days.
- Verifiers under NDA: full order files and salts, which let them check each hash against what was actually traded.
- Public at retirement: full order files and salts of a strategy once it is retired.
Integrity of the whole record
- Single-account policy: a hashed list of every trading account is pre-registered and re-committed monthly, so results can't be cherry-picked across accounts.
- Flows: all deposits and withdrawals are reconciled against on-chain transactions.
- Returns: a GIPS-aligned method from day one: time-weighted, valued daily, net of all costs and adjusted for cash flows.
- Archives: raw exchange records are archived daily, hashed, signed and stored in three locations.
Reconciliation and kill rules
Live trading is reconciled against the research daily and monthly: signal fidelity, realized versus modelled cost, slippage, fill rate and book integrity, with every mismatch root-caused. Each strategy is downgraded or killed by pre-registered rules: results in the tail of its own pre-registered distribution halve it or stop it, and a hard drawdown limit on total capital stops everything until reviewed. Re-entry requires a new pre-registration.
08Verification levels
| Level | What | When |
|---|---|---|
| L1 cryptographic | Salted commitments with private reveals; survivorship-free strategy and idea logs; single-account commitments | From 1 Dec 2026 |
| L2 exchange-side | Read-only exchange access and hashed statements, shared privately with verifiers | From 1 Dec 2026 |
| L3 third-party | Independent verification of the record by an outside firm | At a later stage |
Qualified researchers and allocators can request verification access at research@testrian.com. Testrian Lab does not manage outside money and does not sell signals.
09The edge boundary
| Publish freely | Publish on delay or at retirement | Never publish |
|---|---|---|
| This protocol and its annexes; how the council works (roles, rotation, blinding, generic example prompts); the validation stack and its health numbers; cost-model principles and structure; open-source validation tooling; failure notes on ideas killed before live; headline funnel counts; commitment hashes | Per-field funnel detail (1 month); aggregated daily results (60–90 days); full autopsies and code of retired strategies, only when no related strategy is still running; order files and salts (verifiers under NDA, or public after retirement) | Parameters, universe, signals, timing, sizing or features of live or in-pipeline strategies; their idea cards; exact council prompts; the venue and pair mix of live strategies; account identifiers; read-only keys |
Every outgoing piece is checked before publication by an automated edge-boundary review against the registry of live strategies, a family rule that blocks autopsies while a related strategy runs, and a deterministic scan for banned claims and for return figures without verification links. A suite of at least 50 planted leaks is re-run monthly; the review must catch at least 95% of them, and the result is published.
10Versioning
This protocol is versioned. Material changes are announced in the lab notes and logged here with their date. Annexes with worked examples and the open-source validation tooling follow from November 2026.
- v0.1, 28 October 2026: first public version.
Research contact
Questions, corrections or a flaw you found in this protocol: research@testrian.com. Credible criticism is logged and answered in the lab notes.