Improving Claude Code Success Rates with RE-call Memory Layer

Description and use

An experiment demonstrated that using the RE-call dynamic memory layer with Claude Code improves task success rates by 22.2% compared to static instruction files (CLAUDE.md). Static files reduced performance to 36.1% versus 50.0% for the bare model, likely due to noise. RE-call provides precise context retrieval but increases token costs.

Problem

Static instruction files (like CLAUDE.md) can introduce noise or conflicting constraints, leading to lower success rates in deterministic coding tasks compared to bare models. Agents need a way to retrieve relevant project history without polluting the context window.

Outcome

RE-call improved task success rates by 22.2 percentage points over static CLAUDE.md files and 8.3 points over bare Claude Code. The improvement was statistically significant (p=0.000145). Static files performed worse than bare models, likely due to noise. RE-call costs more per task but prevents expensive failure modes.

Workflow

• Set up a temporary repository for each task.
• Define deterministic checkers to validate the final state of the repository.
• Run tasks using three configurations: Bare Claude Code, Claude Code with CLAUDE.md, and Claude Code with RE-call memory.
• Verify memory tool availability before counting sessions.
• Compare success rates and analyze paired comparisons to determine statistical significance.
• Review cost and token usage for each configuration.

Evidence

RE-call improved success over the CLAUDE.md baseline by 22.2 percentage points.

Source

The paired McNemar test gave a p-value of 0.000145.

Source

The static CLAUDE.md configuration performed worse than the bare configuration: Bare 50.0%, CLAUDE.md 36.1%.

Source