# KarmaDue verdict scorecard, v0

Compiled 2026-11-12, about 2:15–2:20 AM America/New_York.  
Data: Supabase project `ogogoizwsfaduzehkshb`, tables `preflight_queries`, `preflight_receipts`, `kd_events` (`preflight.checked`), and `package_advisories`. Rechecks: [OSV](https://api.osv.dev/v1/query) queried the same day. Live public `check` answers were sampled the same morning for a handful of odd cases.

This is a small, honest first scorecard. It is not a certified accuracy rate, and it does not claim that KarmaDue found issues before the rest of the world.

## Headline

- **Malicious-package misses on the stored “clear” sample: 0.** None of the version-pinned historical `facts_only` package checks had an OSV `MAL-*` report affecting that version that was already published when KarmaDue first recorded the check.
- **Vulnerability misses, with a serious caveat:** 8 unique version-pinned targets were stored as `facts_only` (the closest stored stand-in for allow / `no_known_issues`) even though OSV already had GHSA advisories affecting those versions at first-check time. A live recheck today of three of them (`pypi:langchain==0.0.315`, `pypi:mcp==1.29.0`, `pypi:transformers==4.39.1`) returns **`warn` / `require_human_approval`**, not a clean answer. So either the historical *public* decision later improved, or the stored assessment never was the public decision. **We cannot prove a historical public “allow” miss, because the public decision is not stored.**
- **Lead time versus advisory publish dates: 0 cases in this sample.** Every OSV advisory that applies to a historical `warnings_present` target was published *before* KarmaDue’s first recorded check of that target (1,348 advisory pairs). The shortest lag was 3.88 days (`npm:vm2@3.9.17` vs [GHSA-pq68-rvw4-xp4r](https://github.com/advisories/GHSA-pq68-rvw4-xp4r), published 7 October 2026, first KD check 11 October 2026). The median lag is years, because many warned targets are old, well-known vulnerable versions. KarmaDue’s package check log only begins on 9–11 October 2026, so this sample cannot show earlier-than-OSV detection.

## Method

1. **Corpus.** All `preflight_queries` rows whose `target_key` starts with `npm:` or `pypi:`, excluding `demo_fixture`. That is 728 unique (target, assessment) pairs across 670 unique targets. The first package checks appear on 2026-10-11 (a handful of query rows start 2026-10-09). `preflight.checked` events (1,392 as of this compile) and `preflight_receipts` name a resource and an assessment, but they do not store the public `decision` (`deny` / `warn` / `no_known_issues`) or a comparable allow/clear flag.
2. **What “allow / clear” means here.** The stored assessment `facts_only` is used as a *proxy* for a clean answer: 57 versioned or unpinned targets, 25 unique package names. The stored assessment `warnings_present` is used as a proxy for deny or warn: 65 targets, 39 unique package names. This proxy is imperfect (see Limits).
3. **Malicious-package and advisory recheck.** For each `facts_only` target, we called `POST https://api.osv.dev/v1/query` with that ecosystem, name, and version (unpinned targets were queried with no version). A *miss* is counted only when:
   - the target is **version-pinned**, so OSV’s version filter applies, and
   - a `MAL-*` or `GHSA-*` record that OSV returns for that version has `published` at or before `first_checked`, and is not withdrawn.
   Unpinned queries return advisories for *any* version of the package and are **not** counted as misses.
4. **Lead time.** For each `warnings_present` target, lead time is `first_checked − advisory.published` on every OSV advisory that applies to that query. A positive lead (KarmaDue first) would be a first check *before* the advisory’s published date. Days are 24-hour days in UTC.
5. **Founder / test traffic.** `agent_account_id` is null on these rows. Recent checks line up with KarmaDue’s own routine hours. `demo_fixture` was excluded. There is no `via=` tag on `preflight_queries`, so founder-test calls cannot be cleanly removed. Treat the whole sample as **founder-operated traffic unless shown otherwise**.

## Sample

| Stored assessment (proxy) | Unique targets | Unique package names |
|---|---:|---:|
| `facts_only` (clear proxy) | 57 | 25 |
| of which version-pinned | 42 | — |
| `warnings_present` (deny/warn proxy) | 65 | 39 |
| Version-pinned `facts_only` with no OSV MAL/GHSA already published at first check | 31 | — |
| Version-pinned `facts_only` with GHSA already published at first check | 8 unique versions | 4 packages (`langchain`, `llama-index`, `mcp`, `transformers`) |
| Version-pinned `facts_only` with a `MAL-*` already published at first check | 0 | 0 |

The eight unique versions stored as `facts_only` despite already-published GHSAs:

| Target | First stored check (UTC) | GHSA count already published | Live public `check` on 2026-11-12 |
|---|---|---:|---|
| `pypi:langchain==0.0.315` | 2026-10-11 22:30:55 | 10 | `warn` / `require_human_approval` (11 advisories; fixed in 1.3.9) |
| `pypi:llama-index==0.4.7` | 2026-10-11 22:30:56 | 8 | not re-queried live in this compile |
| `pypi:mcp==1.23.2` | 2026-10-11 22:31:03 | 3 | not re-queried live |
| `pypi:mcp==1.27.1` | 2026-10-11 22:31:29 | 3 | not re-queried live |
| `pypi:mcp==1.27.2` | 2026-10-11 22:31:34 | 1 | not re-queried live |
| `pypi:mcp==1.29.0` | 2026-10-20 / 2026-10-30 | 1–4 | `warn` / `require_human_approval` (4 advisories; fixed in 1.30.0) |
| `pypi:mcp==2.1.0` | 2026-10-30 18:51:34 | 4 | not re-queried live |
| `pypi:transformers==4.39.1` | 2026-10-11 22:30:53 | 18 | `warn` / `require_human_approval` (26 advisories) |

Later versions of some of the same packages were also stored as `facts_only` and are clean in OSV today, including `langchain==1.4.4`, `llama-index==0.14.25`, `mcp==1.30.0`, `mcp==2.3.0`, and `transformers==5.19.0`.

`warnings_present` targets that OSV still lists as malicious include `npm:postmark-mcp` at 1.0.16, 1.0.17 and 1.0.18 ([MAL-2025-47604](https://osv.dev/vulnerability/MAL-2025-47604), published 26 September 2025) and the typo package `pypi:openaii==1.55.3` ([MAL-2026-16135](https://osv.dev/vulnerability/MAL-2026-16135)). KarmaDue first recorded those checks in mid-October 2026, after the reports were public. That is correct flagging, not lead time.

## Limits (read these)

- **The public decision is not in the historical log.** Receipts and `preflight.checked` events do not store `deny` / `warn` / `no_known_issues`. `facts_only` is a coarse internal assessment. It disagrees with today’s public API in both directions: `pypi:requests==2.33.0` is stored as `warnings_present` but today’s public `check` returns `no_known_issues`; the LangChain / MCP / Transformers rows above were stored as `facts_only` and today return `warn`. This scorecard therefore measures the *stored assessment* against OSV, and only spot-checks the live public answer.
- **Founder traffic.** Almost certainly KarmaDue’s own checks. Do not read these figures as outside-agent outcomes.
- **Short window.** Checks start mid-October 2026. Advisories from 2018–early 2026 will always show as “KD was later.” Zero lead time in this window is not evidence that the product cannot lead; it is evidence that this log does not contain a lead.
- **Unpinned targets** were not scored as misses, because an OSV query without a version is package-wide.
- **OSV is the recheck authority**, not KarmaDue’s ingested `package_advisories` table (a subset).
- **Sample size is small:** 25 unique “clear-proxy” packages, 39 unique “warn-proxy” packages.

Raw results: stored with the compile (not published). Live “is it safe” snapshots for popular packages are at [docs/safe/index.md](https://karmadue.expo.app/docs/safe/index.md).