# Does asking KarmaDue first save tokens? Four real tasks, measured

Measured: 2026-11-12, 07:06 UTC (2:06 AM ET). Tokenizer: `o200k_base` (tiktoken 0.14.0). The raw files and the script are listed under Method. Numbers change as registries and advisories change.

The question: an agent hits a task that may already be solved. It can ask KarmaDue once, or it can fetch the sources itself and work out an answer. How many tokens of tool output land in its context each way?

## Results

| Task | KarmaDue (call + answer text) | Do-it-yourself, smallest path | Do-it-yourself, typical path | Was KarmaDue's answer right? |
|---|---|---|---|---|
| T1 Is `requests==2.31.0` safe, and what fixes it? | **196** | 12,234 | 37,227 | Yes. 3 advisories, fixed in 2.33.0; matches OSV after removing duplicate IDs |
| T2 Is `@modelcontextprotocol/server-filesystem` safe to run with npx? | **261** | 2,514 | 12,452 | **Partly.** It warned that 2.3.0 is under 7 days old, but its suggested pin (1.0.0) is covered by repo security advisories that OSV doesn't list yet. See Limits |
| T3 Fix `AttributeError: module 'PIL.Image' has no attribute 'ANTIALIAS'` | **354** | 9,621 | 12,440 | Yes. Replace with `Image.Resampling.LANCZOS` (founder-operated record) |
| T4 Is `tensorflow-gpu==2.12.0` safe, and what instead? | **327** | 5,271 | 269,246 | Yes. 1 advisory; the named fix 2.12.1 doesn't exist; use `tensorflow[and-cuda]` |

Ratios (smallest path ÷ KarmaDue): T1 62×, T2 9.6×, T3 27×, T4 16×. Typical path: 190×, 48×, 35×, 823×.

### What each column counts
- **KarmaDue:** tokens of the `tools/call` params (the call the model writes) plus the `content[0].text` returned. T1, T2 and T4 used `check_install`; T3 used `search_recipes`.
- **Smallest path:** the fewest raw sources that answer the question.
  - T1: the OSV query for that version (9,644) plus the PyPI JSON for the fix release (2,590).
  - T2: OSV for the package (1,286) plus the npm `latest` record (1,228).
  - T3: Pillow's deprecations page as Markdown (9,621).
  - T4: OSV for that version (3,956), the 2.12.0 version record (1,304) and a 404 check that 2.12.1 doesn't exist (11).
- **Typical path:** smallest path plus the pages an agent usually reads to decide.
  - T1: two GitHub advisory pages and the PyPI project page.
  - T2: the server README and the repo's security-advisories page.
  - T3: the Pillow 10.0.0 release notes.
  - T4: the full PyPI package record (251k tokens, one entry per release file) and the project page. That is where an agent learns the package was replaced.
- HTML pages were converted to Markdown, with scripts, navigation and footers removed, before counting. That favors the do-it-yourself side.

### Fixed one-time cost
`tools/list` is read once per session: 897 tokens for the default profile (3 tools) and 1,391 for `/safety` (5 tools), measured before `ask` was added. The `ask` listing adds 211.

### Full MCP response vs compact mode
The full JSON-RPC response carries the answer twice: in `content` text and in `structuredContent`. Compact mode (`?compact=1`) drops `structuredContent`; the text is byte-for-byte identical. Response bodies in tokens:

| Task | Full response | Compact response |
|---|---|---|
| T1 | 371 | 203 |
| T2 | 505 | 270 |
| T3 | 690 | 367 |
| T4 | 637 | 341 |

The compact numbers were measured on the fix97 build before it went live (local harness against the same live data sources). Compact mode went live on 2026-11-12 with mcp v103.

## Limits (read these)
- **Four tasks are a worked example, not a benchmark.** The tasks were picked because KarmaDue has records for them. On a miss, KarmaDue costs one short reply (an `unknown` answer or "Nothing in KD yet."), and the agent then does the do-it-yourself path anyway. So a miss costs slightly more than going straight to the sources.
- **Only tool output is counted.** The model's own reasoning, extra searches and retries are left out. They would mostly add to the do-it-yourself side, but they weren't measured.
- **Fewer tokens is not the same as a better answer.** T2 shows the risk. KarmaDue's suggested pin is covered by GitHub repository advisories (GHSA-hhwq-wxfx-cj7g, GHSA-p9cv-8g75-4xcg, GHSA-7m4x-rv66-mxhh) that OSV, KarmaDue's advisory source, doesn't have yet. An agent that read the repo's advisories page (the typical path) would have seen them. Since 2026-11-12 (fix99) KarmaDue reads repository advisories daily, so `check_install` on `@modelcontextprotocol/server-filesystem@1.0.0` now warns with those advisories. The launch line in the record still pins 1.0.0; updating it is next.
- Seed records are a **founder-operated assessment, not independent**. `no_known_issues` or `unknown` is not a safety clearance. KarmaDue doesn't check transitive dependencies or run your code.

## Method (rerun it)
- Script: `measure.py` (Python 3.12: tiktoken, requests, beautifulsoup4, markdownify). For every source it saves the raw body it counted.
- KarmaDue calls: `POST https://ogogoizwsfaduzehkshb.supabase.co/functions/v1/mcp` with `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_install","arguments":{"target":"requests==2.31.0","ecosystem":"pypi"}}}` (T2: `{"target":"@modelcontextprotocol/server-filesystem","ecosystem":"mcp"}`; T3: `search_recipes {"error_signature":"AttributeError: module 'PIL.Image' has no attribute 'ANTIALIAS'"}`; T4: `{"target":"tensorflow-gpu==2.12.0","ecosystem":"pypi"}`).
- Do-it-yourself sources: `POST https://api.osv.dev/v1/query`, `https://pypi.org/pypi/<pkg>/<ver>/json`, `https://registry.npmjs.org/<pkg>/latest`, plus the pages named above.
- Count: `len(tiktoken.get_encoding("o200k_base").encode(body))`.

Related: [token-cost.md](https://karmadue.expo.app/docs/token-cost.md) (three installs, full vs text-only), [ask.md](https://karmadue.expo.app/docs/ask.md).