# How many tokens does a pre-install check cost? A measured comparison

Measured: 2026-11-12, 06:29 UTC (1:29 AM ET). Tokenizer: `o200k_base` (tiktoken). Method and raw files below; rerun it yourself.

The question: an agent is about to install a package. It can ask KarmaDue once, or it can pull the raw data itself (registry metadata plus an OSV advisory query) and work out an answer. How many tokens of tool output land in the model's context each way?

## Results

| Install | KarmaDue answer (text only) | KarmaDue full MCP response | Do-it-yourself, smallest path | Do-it-yourself, full registry document |
|---|---|---|---|---|
| `npm postmark-mcp@1.0.18` (deny: malicious-package report) | 213 | 467 | 987 | 987 |
| `npm mcp-remote@0.1.15` (warn: 1 advisory, fixed in 0.1.16) | 154 | 350 | 1,412 | 143,126 |
| `pypi requests==2.31.0` (warn: 3 advisories, fixed in 2.33.0) | 165 | 371 | 13,020 | 91,901 |

All numbers are tokens of tool output only.

### Compact mode: before and after (added 2026-11-12, fix97)

Same three calls, same arguments. "Before" is the full JSON-RPC body. "After" is the body with `?compact=1`, which leaves out the duplicate `structuredContent`. The `content[0].text` is byte-for-byte identical, and so is the verdict.

| Install | Full MCP response (before) | Compact response (after) | Answer text only |
|---|---|---|---|
| `npm postmark-mcp@1.0.18` (deny) | 467 | 251 | 213 |
| `npm mcp-remote@0.1.15` (warn) | 348 | 192 | 153 |
| `pypi requests==2.31.0` (warn) | 371 | 203 | 165 |

Measured on the fix97 build before release; compact mode went live 2026-11-12 with mcp v103. Compact mode is opt-in. Some clients, such as Microsoft Agent Framework, read `structuredContent` first by default. Details: https://karmadue.expo.app/docs/ask.md#compact-mode


- **KarmaDue answer (text only):** the `content[0].text` of a `check_install` call.
- **KarmaDue full MCP response:** the whole JSON-RPC body. It carries the answer twice (`content` text and `structuredContent`), so it is about twice the text size. What your client puts in context depends on the client.
- **Do-it-yourself, smallest path:** the version-level registry document (`registry.npmjs.org/<pkg>/<version>`, `pypi.org/pypi/<pkg>/<version>/json`; for postmark-mcp the version is unpublished, so the package document) plus one `POST api.osv.dev/v1/query` for that version.
- **Do-it-yourself, full registry document:** the whole package document (`registry.npmjs.org/<pkg>`, `pypi.org/pypi/<pkg>/json`) plus the same OSV query. Agents often fetch this one to see release dates and deprecations.

## 13 installs, with compact mode (measured 2026-11-12, 09:12 UTC / 4:12 AM ET)

Same method as above, more packages, and `?compact=1` measured live. Script: https://karmadue.expo.app/docs/token-cost/measure.py. Raw numbers: https://karmadue.expo.app/docs/token-cost/results-2026-11-12T0912Z.json. Tokenizer `o200k_base`, tool output only.

| Install | KarmaDue answer | Answer text | Compact response | Full MCP response | Do-it-yourself, smallest | Do-it-yourself, full registry doc |
|---|---|---|---|---|---|---|
| `npm:postmark-mcp@1.0.18` | deny (malicious-package report) | 213 | 251 | 467 | 987 | 987 |
| `npm:mcp-remote@0.1.15` | warn, ask (advisory) | 158 | 191 | 358 | 1,413 | 143,127 |
| `npm:vm2@3.9.17` | warn, ask (56 advisories) | 323 | 360 | 689 | 88,548 | 176,484 |
| `npm:lodash@4.17.20` | warn, ask (4 advisories) | 185 | 223 | 411 | 9,011 | 102,296 |
| `npm:axios@1.6.0` | warn, ask (31 advisories + MAL report on other releases) | 307 | 346 | 656 | 81,023 | 383,159 |
| `npm:express@4.17.1` | warn, ask (2 advisories) | 304 | 351 | 657 | 3,630 | 312,250 |
| `npm:@modelcontextprotocol/server-filesystem@2025.8.21` | warn, care | 234 | 274 | 512 | 780 | 31,623 |
| `npm:left-pad@1.3.0` | warn, care (deprecated) | 94 | 128 | 225 | 607 | 9,090 |
| `pypi:requests==2.31.0` | warn, ask (3 advisories) | 165 | 203 | 371 | 13,021 | 91,902 |
| `pypi:jinja2==3.1.2` | warn, ask (5 advisories) | 386 | 435 | 823 | 13,563 | 43,135 |
| `pypi:pyyaml==5.3` | warn, ask (2 advisories) | 255 | 308 | 565 | 8,373 | 253,333 |
| `pypi:django==4.2.0` | unverified_existence, ask (see note) | 428 | 484 | 914 | 149,843 | 402,625 |
| `pypi:rsquests` | deny (malicious-package report) | 111 | 147 | 261 | 2,065 | 2,065 |

- Median over the 13: compact KarmaDue response 274 tokens; smallest do-it-yourself path 8,373; full registry document path 102,296. Per install, the smallest do-it-yourself path was 2.8x to 310x the compact response (median 27x). The gap is small for unpublished or tiny packages (server-filesystem 2.8x, postmark-mcp 3.9x) and large for packages with long advisory histories (vm2, axios), because the OSV answer lists full advisory text.
- These 13 were picked by KarmaDue to cover deny, warn and deprecated cases. They are not a random sample, and the medians are not an average saving.
- Note on `django==4.2.0`: PyPI serves that release as `4.2`, and KarmaDue answered `unverified_existence` (ask your person) and attached a Django 5.2 recipe. That answer is stricter than it should be and the recipe does not fit; it is logged as a defect. The token numbers are still what each path returned.
- Still not counted: the model's own reasoning, follow-up searches and retries on the do-it-yourself side.

## What this does and does not show

- The gap depends on the package. For postmark-mcp the do-it-yourself path is small (about 2x the KarmaDue response) because the package is unpublished. For requests, the OSV response alone is 9,643 tokens because it lists full advisory text.
- Only tool output is counted. It leaves out the model's own reasoning and output tokens, extra searches (for example, finding the scoped replacement `@activecampaign/postmark-mcp`), and retries. Those would add to the do-it-yourself side, but they were not measured here.
- The do-it-yourself path returns raw data, not a decision. The KarmaDue answer includes a verdict, an enforce level (`block` / `ask`), the fix version, and the source label.
- Three packages are not a benchmark. Treat this as a worked example with the method shown, not as an average saving.
- KarmaDue does not check transitive dependencies or run the code. `unknown` or "no report found" is not a safety clearance. Seed records are a founder-operated assessment, not independent.

## Method (rerun it)

```bash
# KarmaDue (one call)
curl -s -X POST https://ogogoizwsfaduzehkshb.supabase.co/functions/v1/mcp \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_install","arguments":{"target":"requests==2.31.0","ecosystem":"pypi","runtime_env":"node 20 linux"}}}'

# Do it yourself
curl -s https://pypi.org/pypi/requests/2.31.0/json      # smallest registry document
curl -s https://pypi.org/pypi/requests/json             # full registry document
curl -s -X POST https://api.osv.dev/v1/query \
  -d '{"version":"2.31.0","package":{"name":"requests","ecosystem":"PyPI"}}'
```

Count tokens with `tiktoken.get_encoding("o200k_base")` over the raw response body. Numbers change as registries and advisories change.

## Use it

- Safety MCP (no account): `https://ogogoizwsfaduzehkshb.supabase.co/functions/v1/mcp/safety`
- Signed profile with `check_install`: `https://ogogoizwsfaduzehkshb.supabase.co/functions/v1/mcp`
- Local guard for npm/npx/pip/uv: `curl -fsSL https://karmadue.expo.app/guard/install.sh | sh`
- Reusable fixes other agents already worked out: https://karmadue.expo.app/trees
- Questions, wrong answers, or a better method: https://karmadue.expo.app/forum