v0.1 update — recommended-size classifier¶

Quick update on what's happened since last week. We tried four things, shipped one, and grew the hand-labelled set from 65 to 88 fonts.

tl;dr¶

  • 5-bucket exact accuracy: 0.394 → 0.452 (+5.8)
  • 5-bucket macro-F1: 0.327 → 0.402 (+7.5)
  • Caption, Text, and Subhead recall all up
  • What we're shipping: 8 geometric features + Google Fonts family tags, RandomForest, family-level cross-validation
  • Two ideas didn't pan out (OCR, auto-labelling from tags). Keeping the code so we don't redo them by accident.

1. OCR — tried it, didn't really help¶

Thought I'd try Apple's Vision framework as a direct legibility test: render a pangram in each font at a bunch of pixel sizes (via hb-view), OCR each image, and use the size where recognition falls apart as the bucket signal. I turned off Vision's language correction so it couldn't quietly fix misreads behind my back.

Ran it on 10 fonts spanning all five buckets. A few things showed up:

  • Vision is too good at its job. Above ~12px it reads everything fine, so all the discriminating signal is squeezed into a tight 6–12px window.
  • The most useful single number is accuracy at 8px: text faces sit around 90%, display faces drop to 0%, Subhead in the middle. So it's really a text-vs-display detector, not a 5-bucket classifier.
  • Pixel and novelty fonts confuse it. PressStart2P (a Display+ pixel font) reads at 81% at 8px because crisp pixels OCR well regardless of the font's intended size.

Parked it. Code's at ocr_probe.py if we ever want to look at it again.

2. Google Fonts family tags — shipped¶

The GF tagging CSV at ~/Type/fonts/tags/all/families.csv has roughly 74 perceptual tags per family. Things like /Expressive/Loud, /Expressive/Calm, /Purpose/Easy Reading, /Theme/Stencil. It lines up with every one of our measured families once you normalise the name.

The reason this works is that tags capture something the geometry can't see — designer or perceptual intent. The geometry knows how thick the strokes are; the tags know whether a font is meant to feel loud, or whether it's optimised for easy reading. They're complementary rather than redundant.

One thing worth keeping in mind: tags are per family, not per cut. Every opsz instance of Inter has the same tag vector, so tags can't separate an Inter caption from an Inter display. They help us tell families apart from each other; they don't help inside a family.

Curated vs all 74¶

I compared three feature sets, holding out one family at a time:

  • geo — just the 8 fontquant features (the v0 baseline)
  • geo + curated — geo plus 11 tags I picked by hand: Loud / Fancy / Artistic / Vintage / Playful on the display end, Business / Calm / Competent / Sincere / Easy Reading on the text end, plus a binary flag for any /Theme or /Seasonal tag
  • geo + all — geo plus every one of the 74 tag columns

RandomForest numbers on the current 88-static training set:

feature set 5-bucket exact 5-bucket F1 3-bucket exact 3-bucket F1
geo 0.402 0.337 0.538 0.455
geo+curated 0.424 0.369 0.576 0.480
geo+all 0.452 0.402 0.586 0.487

Both tag sets beat geo. train.py automatically picks the simpler one if it's within 0.02 macro-F1 of the best — right now geo+all wins by enough to get selected. We could override to geo+curated if it matters that the CLI's "top signals" output is human-readable (11 named tags vs 74), at the cost of about 3 F1 points.

3. Auto-labelling from tags — also didn't help¶

Idea: if hand-labels are the bottleneck, use those same tag rules to auto-label hundreds more families and train on the bigger set.

To be fair to the test: the weak labels only went into the training fold, never into test. I scored only the real (human + opsz-VF) labels. I also tried it on a tag-blind geometry model so the label source (tags) wasn't also a feature — otherwise it's trivially circular.

model weak labels 5b exact 5b F1 3b exact 3b F1
geo (tag-blind) out 0.394 0.327 0.547 0.469
geo (tag-blind) in 0.416 0.360 0.563 0.487
geo+curated (what we ship) out 0.426 0.358 0.606 0.493
geo+curated (what we ship) in 0.416 0.360 0.571 0.457

The interesting bit: for the tag-blind model, weak labels do help a bit. That's just "more data, even noisy, is good." But for the model we actually ship (tags as features), weak labels are flat on 5-bucket and noticeably hurt 3-bucket. You can't double-dip the tag signal — when tags are already features, families labelled by the tag rule don't add new information, just noise. And because the weak labels only landed in Text / Display / Display+, the 3-bucket Subhead class got starved.

Parked. measurements_weak.csv is sitting on disk but isn't wired into train.py. Main takeaway: redundant signals don't compound, and what we actually needed was more real labels.

4. More hand-labels (rounds 3 + 4)¶

The v0 confusion matrix made it pretty clear where the model was struggling: Subhead at 21% recall, Display at 17%, Caption at 30%. Added 23 fonts focused there:

bucket v0 now Δ examples added
Caption 1 3 +2 PT Sans Caption, PT Serif Caption
Text 38 38 0 (already over-represented)
Subhead 8 17 +9 Orbitron, Quantico, Tektur, Michroma, Righteous, Syne Mono, ...
Display 6 10 +4 Fruktur, Matemasie, Kavoon, Bungee Spice
Display+ 12 20 +8 Kablammo, Sankofa Display, Tilt Prism, Oi, Honk, ...
total 65 88 +23

One snag: four of those labels couldn't be measured. RubikMicrobe and BitcountSingle hang fontquant indefinitely (gnarly outlines, lots of axes); Nabla and Honk error out (chromatic COLR fonts). Patched measure_static.py to run each measurement in its own 45-second subprocess so one bad font can't take down the whole batch.

5. Where we are now¶

Currently shipping 5bucket / geo+all / RandomForest. 8 geometric features + 74 tag columns + a "has any decorative tag" flag. Trained on 88 hand-labelled statics and 308 opsz-VF samples across 115 families, holding out one family at a time.

5-bucket confusion (truth on rows, predictions on columns):

Caption Text Subhead Display Display+ recall
Caption 12 10 3 1 3 41% ↑
Text 3 43 5 9 8 63% ↑
Subhead 5 32 18 6 10 25% ↑
Display 5 28 12 14 23 17% =
Display+ 4 23 9 18 92 63% =

Caption, Text, and Subhead all moved up. Display didn't budge — mostly because two of the four planned Display additions (RubikMicrobe, BitcountSingle) got dropped by the measurement pipeline, so we only really added 4 Display labels, not 6. That's the next thing to fix.

The 3-bucket scheme actually slid back a bit this round (geo+curated went 0.606 → 0.576). That's because every new label landed in Subhead / Display / Display+ and nothing in Text-or-smaller, which threw off the class balance for the coarser scheme.

In [1]:
import matplotlib.pyplot as plt
import numpy as np

buckets    = ["Caption", "Text", "Subhead", "Display", "Display+"]
recall_v0  = [30, 56, 21, 17, 64]   # v0  (65 statics, geo+curated)
recall_v01 = [41, 63, 25, 17, 63]   # v0.1 (88 statics, geo+all)

x = np.arange(len(buckets)); w = 0.38
fig, ax = plt.subplots(figsize=(8, 4.2))
ax.bar(x - w/2, recall_v0,  w, label="v0  (65 statics, geo+curated)", color="#9CA3AF")
ax.bar(x + w/2, recall_v01, w, label="v0.1 (88 statics, geo+all)",     color="#2563EB")
for i, (a, b) in enumerate(zip(recall_v0, recall_v01)):
    ax.text(i - w/2, a + 1.5, str(a), ha="center", fontsize=9, color="#374151")
    ax.text(i + w/2, b + 1.5, str(b), ha="center", fontsize=9, color="#1E3A8A")
ax.set_xticks(x); ax.set_xticklabels(buckets)
ax.set_ylabel("Recall (%)"); ax.set_ylim(0, 100)
ax.set_title("Per-class recall: v0 → v0.1")
ax.legend(loc="upper left"); ax.grid(axis="y", alpha=0.3)
plt.tight_layout(); plt.show()
No description has been provided for this image

6. What's next¶

Model's better but still nowhere near great (~0.45 exact, ~0.40 F1). Three things I want to try, roughly in order of how much I think they'll move the needle:

  1. More Display labels. Display recall is flat and it's the worst readable class. Going to grab more actual designed headline typefaces — high-contrast display serifs, heavy display sans. Maybe 8–10 more.
  2. Caption fonts from opsz instantiation. GF doesn't have many standalone caption fonts, but several of our opsz VFs do have caption masters (Source Serif 4, Newsreader, Literata, Hedvig Letters Serif, Inter). Instantiating those at their caption size gives clean Caption examples. Need to cap per family so I'm not just relearning those families.
  3. Second pair of eyes on the middle. Subhead is the most subjective bucket. Having someone else label the Text/Subhead and Subhead/Display borderline cases (accepting off-by-one) would tighten the ground truth before we expect the model to do better.

Also looked at the category: DISPLAY field in METADATA.pb as another possible signal. Turns out only about 24% of optically-display fonts actually carry it — most are filed under SERIF / SANS_SERIF (Lobster is officially SERIF, which surprised me). So it's a decent shortlist for finding display fonts to label, but not useful as a model feature or for auto-labelling.

Files from this round¶

  • classifier/train.py — runs the feature-set comparison and saves the chosen model
  • classifier/classify.py — the CLI; now looks up family tags at inference and shows tag-derived "top signals" in the output
  • classifier/build_labels.py — source of truth for hand-labels (edit this, not the CSV)
  • tag_diagnostic.py — tag-only baseline (parked)
  • ocr_probe.py — the Vision OCR spike (parked)
  • classifier/weak_label.py, classifier/weak_eval.py — the auto-labelling experiment (parked)
In [2]:
# Live numbers from the currently shipped model
import json
from pathlib import Path

meta = json.loads(Path("classifier/model_metadata.json").read_text())
print(f"Shipped: {meta['winner']}")
print(f"Trained on {meta['n_rows']} rows across {meta['n_families']} families")
if 'tag_join_coverage' in meta:
    print(f"Tag join coverage: {meta['tag_join_coverage']:.0%}")
print("\nAll feature-set / model combinations:")
for k, v in meta['all_results'].items():
    print(f"  {k:40s}  exact={v['acc']:.3f}  off1={v['off_by_one']:.3f}  F1={v['f1']:.3f}")

Shipped: 5bucket/geo+all/RandomForest
Trained on 396 rows across 115 families
Tag join coverage: 100%

All feature-set / model combinations:
  5bucket/geo/OrdinalLogReg                 exact=0.343  off1=0.747  F1=0.264
  5bucket/geo/RandomForest                  exact=0.402  off1=0.717  F1=0.337
  5bucket/geo+curated/OrdinalLogReg         exact=0.346  off1=0.730  F1=0.264
  5bucket/geo+curated/RandomForest          exact=0.424  off1=0.722  F1=0.369
  5bucket/geo+all/OrdinalLogReg             exact=0.376  off1=0.705  F1=0.310
  5bucket/geo+all/RandomForest              exact=0.452  off1=0.727  F1=0.402
  3bucket/geo/OrdinalLogReg                 exact=0.508  off1=0.854  F1=0.381
  3bucket/geo/RandomForest                  exact=0.538  off1=0.803  F1=0.455
  3bucket/geo+curated/OrdinalLogReg         exact=0.530  off1=0.871  F1=0.416
  3bucket/geo+curated/RandomForest          exact=0.576  off1=0.806  F1=0.480
  3bucket/geo+all/OrdinalLogReg             exact=0.535  off1=0.838  F1=0.416
  3bucket/geo+all/RandomForest              exact=0.586  off1=0.811  F1=0.487