🧹 ModelBrew Dataset Cleaner

90+ quality checks for fine-tuning datasets: PII with real checksum validation, exact/near duplicates, prompt-injection & jailbreak patterns, label errors, truncated responses, and more. Free and open source.

Clean your dataset in the browser → GitHub

Or in your pipeline

pip install modelbrew-cleaner

from modelbrew_cleaner import clean_file, issue_summary, export_clean
rows = clean_file("train.jsonl")     # .jsonl, .json, or .csv
print(issue_summary(rows))           # {'critical': 4, 'warning': 7, 'suggestion': 6}
cleaned = export_clean(rows)         # critical rows dropped

Real output — the bundled dirty sample (12 rows)

This report is generated by actually running the cleaner on sample.jsonl at build time — 4 critical / 7 warnings / 6 suggestions; 4 rows dropped from the cleaned export.

row	severity	check	message	auto-fix
3	critical	`pii_ssn`	SSN detected	✔
4	critical	`incomplete_pair`	Output row has no instruction
5	critical	`incomplete_pair`	Instruction row has no output
7	critical	`prompt_injection`	Row contains prompt injection patterns — dangerous for training
0	warning	`duplicate_boilerplate`	Response shares a common opening with >30% of rows
1	warning	`duplicate_exact`	Exact duplicate row	✔
1	warning	`duplicate_boilerplate`	Response shares a common opening with >30% of rows
2	warning	`duplicate_near`	Near-duplicate of row 0 (93.06930693069306% similar on both prompt and response)	✔
3	warning	`pii_email`	Email detected	✔
7	warning	`jailbreak_pattern`	Jailbreak / red-team pattern detected (instruction_override) — training on this teaches the model to comply with bypass prompts	✔
10	warning	`unfinished_response`	Response appears to end mid-sentence	✔
0	suggestion	`modelbrew_too_few_rows`	Only 12 rows — ModelBrew works best with 20+ training examples
3	suggestion	`pii_name_heuristic`	Possible personal name (1) — heuristic, not NER-grade
4	suggestion	`tiny_row`	Row has very few tokens (15) — may be low signal
5	suggestion	`tiny_row`	Row has very few tokens (10) — may be low signal
7	suggestion	`tiny_row`	Row has very few tokens (16) — may be low signal
8	suggestion	`pii_name_heuristic`	Possible personal name (1) — heuristic, not NER-grade

Why we built it

We work on fine-tuning without catastrophic forgetting (patent-pending CRMA adapters). Measuring forgetting honestly forced us to fix our data first — in our measurements, dataset confounds alone accounted for a 96.9-percentage-point swing in measured forgetting. Read: Your forgetting benchmark is lying to you · Browse the forgetting leaderboard.