v0.2 update — switching to three buckets¶

Quick one. Since v0.1 I've dropped the model from 5 buckets down to 3. Reads better, matches what designers actually want to know, and the numbers are a bit nicer too.

tl;dr¶

  • Five buckets out (Caption / Text / Subhead / Display / Display+), three buckets in: Text (5–24pt) / Display (24–48pt) / Display+ (48pt+).
  • It's basically the three questions a designer cares about: is this for body, for a headline, or for one big feature.
  • All the numbers nudge up. Exact 0.598 → 0.600, off-by-one 0.816 → 0.859, macro-F1 0.511 → 0.534.
  • Text recall jumps from 64% to 80%. Big body/UI win.
  • CLI shows proper ranges now ("recommended 5–24pt") instead of just "5pt+".

Why three buckets¶

5-bucket got stuck around 0.43 exact accuracy and we were running out of useful labels to add. Most of the trouble was on two of the five boundaries:

  • Caption vs Text. There are basically three or four real caption fonts in GF. Tinos (a body serif) and PT Serif Caption measure almost identically on the things we look at, so the model couldn't tell them apart no matter how many labels we threw at it.
  • Text vs Subhead. Subhead is the most subjective bucket in the whole rubric. Two of us labelling the same set of fonts would disagree about a quarter of the time. The model can't pick up a line we can't agree on ourselves.

The Display vs Display+ split, on the other hand, is one the model can actually learn. Decorative and script fonts carry /Theme and /Seasonal tags, and the curated tags (Loud, Fancy, Artistic) separate ornate faces from real headline typefaces pretty cleanly.

So the new scheme keeps the boundary that works and drops the two that don't. Three buckets is enough for what users care about anyway — most font-picking decisions come down to "is this for body, for a headline, or for one big use".

The new scheme¶

bucket size range what's in it
Text 5–24pt body, UI, subheads — running text at all the usual sizes
Display 24–48pt designed headline typefaces (Playfair, Bodoni, Anton, Oswald)
Display+ 48pt+ decorative, scripts, novelty — single-use feature lockups

24pt is where we used to split Subhead from Display, so the old Subhead bucket just folds into Text. The 48pt line between Display and Display+ is unchanged.

How it scores¶

Same setup as v0.1: RandomForest, hold out one family at a time, geometry features + 11 curated GF tags. Only the bucket boundaries changed.

metric old 3-bucket new 3-bucket Δ
exact accuracy 0.598 0.600 +0.002
off-by-one 0.816 0.859 +0.043
macro-F1 0.511 0.534 +0.023

Exact accuracy is basically flat, which is fine. The real wins are off-by-one (errors are now between neighbouring buckets, not wild misses) and macro-F1 (classes are more evenly weighted).

In [1]:
import matplotlib.pyplot as plt
import numpy as np

metrics = ["exact", "off-by-1", "macro-F1"]
old_3b  = [0.598, 0.816, 0.511]
new_3b  = [0.600, 0.859, 0.534]

x = np.arange(len(metrics)); w = 0.38
fig, ax = plt.subplots(figsize=(7.5, 4.2))
ax.bar(x - w/2, old_3b, w, label="old 3-bucket (Text-or-smaller / Subhead / Display-or-larger)",
       color="#9CA3AF")
ax.bar(x + w/2, new_3b, w, label="new 3-bucket (Text / Display / Display+)",
       color="#2563EB")
for i, (a, b) in enumerate(zip(old_3b, new_3b)):
    ax.text(i - w/2, a + 0.008, f"{a:.3f}", ha="center", fontsize=9, color="#374151")
    ax.text(i + w/2, b + 0.008, f"{b:.3f}", ha="center", fontsize=9, color="#1E3A8A")
ax.set_xticks(x); ax.set_xticklabels(metrics)
ax.set_ylim(0, 1.0); ax.set_ylabel("score")
ax.set_title("3-bucket headline metrics: old scheme vs new")
ax.legend(loc="lower right", fontsize=9); ax.grid(axis="y", alpha=0.3)
plt.tight_layout(); plt.show()
No description has been provided for this image

One thing worth flagging¶

Per-class recall on the new scheme:

bucket recall
Text 80%
Display 21%
Display+ 61%

Display on its own only catches 21% of true Display fonts, which looks rough. It's not really a problem in practice though. When the model gets a Display font wrong, it almost always picks the next-door bucket (Display+ or Text) rather than something wild. That's why off-by-one is 86%.

In the CLI this shows up as probabilities like "Display 43%, Display+ 48%" on borderline fonts. Playfair Display is the obvious example — it genuinely is on that line, and any working typographer would tell you the same. The split tells the user "this is display-y, sitting between headline and feature face," which is more useful than committing hard to one or the other.

Bottom line: this is a real ceiling in the data, not a bug.

What the CLI looks like now¶

Same command as before:

python ~/Type/opsz_research/classifier/classify.py path/to/Font.ttf

Here's what running it on five fonts actually looks like.

In [2]:
import subprocess, sys
from pathlib import Path

OFL = Path.home() / "Type/fonts/ofl"
HERE = Path.home() / "Type/opsz_research"
fonts = [
    "roboto/Roboto[wdth,wght].ttf",                # body sans -> Text
    "playfairdisplay/PlayfairDisplay[wght].ttf",   # the borderline case
    "anton/Anton-Regular.ttf",                     # designed headline -> Display
    "lobster/Lobster-Regular.ttf",                 # script novelty -> Display+
    "pacifico/Pacifico-Regular.ttf",               # casual script
]
cmd = [sys.executable, str(HERE / "classifier/classify.py")] + [str(OFL / f) for f in fonts]
r = subprocess.run(cmd, capture_output=True, text=True)
print(r.stdout)

Roboto[wdth,wght].ttf
  Bucket:     Text (recommended 5-24pt)
  Confidence: 73%   family='roboto'
  Probabilities: Text 73%, Display 23%, Display+ 5%
  Top signals:
    - high stroke_contrast_ratio (0.82, z=+0.92)
    - high Calm (85.00, z=+1.60)
    - low xtra_to_xh (0.35, z=-1.22)
    - low Vintage (10.00, z=-1.69)

PlayfairDisplay[wght].ttf
  Bucket:     Display+ (recommended 48pt+)
  Confidence: 48%   family='playfairdisplay'
  Probabilities: Text 9%, Display 43%, Display+ 48%
  Top signals:
    - low stroke_contrast_ratio (0.21, z=-1.17)
    - low stem_to_xh (0.04, z=-0.60)
    - low Calm (0.00, z=-0.81)
    - high Sincere (90.00, z=+1.77)

Anton-Regular.ttf
  Bucket:     Display (recommended 24-48pt)
  Confidence: 62%   family='anton'
  Probabilities: Text 19%, Display 62%, Display+ 19%
  Top signals:
    - high weight (0.62, z=+4.44)
    - low xtra_to_xh (0.11, z=-2.33)
    - high xh_to_cap (0.85, z=+1.56)
    - low width (0.45, z=-1.17)

Lobster-Regular.ttf
  Bucket:     Display+ (recommended 48pt+)
  Confidence: 68%   family='lobster'
  Probabilities: Text 8%, Display 24%, Display+ 68%
  Top signals:
    - low stroke_contrast_ratio (0.03, z=-1.78)
    - high weight (0.38, z=+1.64)
    - low xtra_to_xh (0.26, z=-1.63)
    - high stem_to_xh (0.31, z=+0.98)

Pacifico-Regular.ttf
  Bucket:     Display (recommended 24-48pt)
  Confidence: 67%   family='pacifico'
  Probabilities: Text 15%, Display 67%, Display+ 18%
  Top signals:
    - high stem_to_xh (0.83, z=+3.89)
    - high weight (0.44, z=+2.34)
    - low xh_to_cap (0.54, z=-2.12)
    - high stroke_contrast_ratio (0.85, z=+1.03)

What's next¶

A few obvious next steps, roughly in order:

  1. Run it on more real fonts and check the output by eye. The 21% Display recall is the bit I want to sanity-check the most. Off-by-one says we're close, but I want to see actual outputs before believing it.
  2. Another round of hand-labels aimed at the Display vs Display+ line specifically. That's the only boundary left to push on, and unlike v0.1, more labels here should stack — the tag features already know what "decorative" looks like, we'd just be giving them more borderline cases.
  3. Actually use it. If the output looks good, this can replace whatever rule of thumb we're using right now to bucket fonts in the GF catalogue.

Beyond that I think we're close to what this data can do. Bigger gains would need either a different kind of feature (a real legibility test from human ratings) or a much bigger labelled set.