How Our Guardrails & Memory Engine Compare to Standard AI Coding Across 63 Benchmark Tests

Why Prompt Instructions Alone Fail—and Why External Software Checks Succeed

Independent university studies—including BaxBench (ETH Zurich, 2025) and CWEval (2025)—show that while modern AI models write code that passes basic feature tests, over 60% of that working code contains serious security holes (like SQL injection, file path escapes, or leaked keys). Trying to fix this by pasting long security rules into the chat prompt slows the AI down and still fails when the AI edits multiple files.

Instead of relying on the AI to police itself, Thunderbear Code Guard™ (v2.3) and Resonant Cloud™ (v2.0) enforce three automated checks outside the AI model:

• 1. Pre-Save Security & Real-Time Check (<50ms): Inspects every proposed code change before it is written to disk. If a change introduces a security hole, leaks a secret, or puts a slow memory allocation inside a real-time audio loop, the save is blocked and the tool hands the AI a 1-line fix instruction so it repairs the issue immediately.
• 2. Multi-File Caller Map (<45ms): Maps the exact line numbers of a function and every file that calls it across Python, TypeScript/JS, C++20, Rust, and C#—so the AI reads 93.76% less text and never breaks connected files.
• 3. Time-Aware Memory & Prompt Caching (5.93ms): Tracks when project decisions are replaced, automatically suppresses outdated rules (plus their 1-hop and 2-hop follow-on assumptions), and locks the prompt layout so >85% of input tokens hit Google Gemini’s cache.

Comparing Regular AI Baseline vs. Mastery v2.1 vs. Mastery v2.2/v2.3

All three arms were evaluated on the exact same 20 benchmark tasks (12 Track 1 security-critical tasks + 8 Track 2 multi-file repository resolution tasks) under deterministic functional and live adversarial exploit test suites:

Dual Verification: Functional Correctness + Live Adversarial Exploit Payloads

Each Track 1 task is scored by two independent test suites: a functional suite verifying normal application behavior and an exploit suite executing live attack payloads (union SQL injection, ../../etc/passwd traversal, cloud metadata SSRF http://169.254.169.254/, reflected XSS, and tampered JWT payloads). A task passes SecPass@1 if and only if both suites exit 0:

Eliminating Cross-Caller Breakage and Full-File Context Reads

In multi-file repositories (350+ line core modules with multiple downstream caller modules), unconstrained agents exhibit two costly habits: (1) dumping entire files into context to locate a 10-line function, and (2) changing a method signature or return contract without updating all callers. With our call-graph slice indexer, exact symbol line ranges and all caller sites are resolved in <45ms:

Track 3: Multi-SWE-bench Polyglot & Compiled Real-Time Audio DSP (12 Tasks, p = 0.000488)

Evaluated across Python, TypeScript/JS, C++20 (g++ -O2 -std=c++20 with global operator new telemetry), Rust (rustc -O with custom #[global_allocator] counter), and C# across symbol localization, cross-file call-site verification, and real-time audio callback safety (AUDIO-RT-01, 1,000 blocks × 512 frames @ 48 kHz):

Track 4: AgentDojo / CaMeL Adversarial Security & Universal LLM-Agnostic Schema (10 Tasks, p = 0.001953)

Evaluated against multi-step tool-chaining data exfiltration (CaMeL SEC-EXFIL-01), indirect prompt injection via untrusted tool outputs, AST aliased-import evasion, and 4 heterogeneous model tool-call schemas (Standard PascalCase, OpenAI/Anthropic snake_case, MCP nested input/edits, and JSON-serialized arguments):

Track 5: ICLR 2025 LongMemEval & ICML 2025 HippoRAG 2 Continual Memory (15 Tasks, p = 0.000244)

Evaluated across knowledge-update contradiction resolution (1-hop & 2-hop transitive cascades), stopword-heavy natural-language Personalized PageRank (PPR) seed extraction, 3-store Tri-Modal Reciprocal Rank Fusion (relationship graph + full-text BM25 + pinned core memory), epistemic abstention, and KV-cache prefix immutability (comparing Flat RAG at 20.0% / 3/15 and Resonant v1.0 at 13.33% / 2/15 against Resonant v2.0 at 100.0% / 15/15):

Multi-Iteration Hill-Climbing Convergence & Unseen Blind Holdout

All Track 3–5 improvements were developed inside a strict Benchmark → Research → Develop → Re-Benchmark closed loop until marginal utility gain reached ROI saturation, then cryptographically frozen and evaluated once on an unseen 6-task Out-of-Distribution (OOD) holdout suite (note: in the composite utility table below, Track 5's strict latency-penalized sub-metric starts at 6.67% in Iteration 0 while raw task recall starts at 13.33% in v1.0 / 20.0% in Flat RAG):

Scope Conditions, Statistical Power & Threats to Validity

To ensure this report stands up to rigorous peer review and investor technical due diligence, we state both what these results prove and their exact experimental boundaries:

• Why 100.0% Is Achievable on Bounded Invariant Families: Achieving 100.0% across our 57 core tasks and 6 blind holdout tasks does not claim that AI models become infallible on arbitrary open-ended software bugs. Rather, it proves that for formally specified vulnerability and systems-engineering classes (the 10 MITRE CWE families in Track 1, explicit symbol/caller arity contracts in Tracks 2–3, session taint flows in Track 4, and time-stamped fact supersession in Track 5), a deterministic pre-write parser and graph memory layer can enforce 100% compliance before code touches disk.
• Sample Size & Exact Non-Parametric Significance: Because sample sizes per track (`8` to `15` tasks per track, `63` tasks total) are designed for fast, deterministic execution, we report exact two-tailed paired McNemar test p-values (p = 0.000977 on Track 1, p = 0.007812 on Track 2, p = 0.000488 on Track 3, p = 0.001953 on Track 4, and p = 0.000244 on Track 5) alongside non-parametric bootstrap confidence intervals rather than relying on asymptotic Gaussian approximations.
• Defense Against Overfitting (Post-Freeze Blind Holdout): Iteratively improving a guardrail against a fixed test suite carries a known risk of rule overfitting. To test generalization, we froze all guardrail rules and memory parameters after Iteration 3 (recording SHA-256 digests) and evaluated once on 6 newly authored Out-of-Distribution (OOD) tasks (e.g., C# Unity Update() allocations, Base64-encoded shell exfiltration, and 3-hop contradiction chains with distractors), passing 6/6 (100.0%) on the first execution.
• Credentialed Replication Access: While our full benchmark harness repository is kept private to prevent automated scraping of our proprietary guardrail implementation, credentialed cloud grant reviewers, institutional investors, and academic partners may verify artifact hashes via benchmark-manifest.json and request full replication access via [email protected].