Four scalar quality classifiers tested separate authored from commercial documents within 0.07 of each other, and the generative annotator tested separates them at three times the effect size

What was done

The seven instruments below share one property: each scores a document from its text alone, taking no link, vote, view count, or other record of how the document was received. Instruments of this kind are built to filter pretraining corpora at web scale, where reception data is unavailable for most documents and unreliable for the rest. What they disagree about is what a text-only score should measure, and the questions they are asked range from educational value to writing quality to depth of logical analysis.

This evaluation records how far seven of them agree when run over the same documents, what each one’s score tracks, and what each costs relative to the others.

The corpus is 1,214,507 web documents with a successful text extraction, retrieved from URLs submitted to a public link aggregator. It is used for scale and for format composition, not as a sample of the web.

Separation is measured against a reference set of 50 documents whose character is independently established by reading them: 32 authored technical articles, 12 commercial pages, and 6 thin pages. The reference set is a check on the instruments rather than a sample of anything, and its authored documents are technical, which bounds what its figures cover.

Separation is reported as area under the ROC curve for authored against commercial documents, with Cohen’s d, so instruments producing continuous scores and instruments producing ordinal labels are placed on one scale. Threshold-crossing rates are reported separately where a published threshold exists.

Throughput is reported in relative terms. Absolute rates depend on the accelerator; the ratios below were measured on identical hardware within each comparison.

Every figure below describes these seven instruments on this corpus and this reference set.

The scalar quality classifiers

Four instruments emit a single score per document. FineWeb-Edu scores educational value from document text.1 The nvidia quality-classifier-deberta asks for writing quality instead.2 Nemotron-4-edu is a deliberate near-duplicate of FineWeb-Edu, sharing its source samples, architecture, embedding base and regression head, differing only in which model produced the training labels.3 The stack-edu classifier scores code-bearing documents.4

What they did. All four separated authored from commercial reference documents, across a spread of 0.07 in Cohen’s d.

What moved their scores. Swapping the annotating model moved scores far less than changing what was asked. FineWeb-Edu and Nemotron-4-edu, differing only in label-producing model, correlate at r = 0.883 across the corpus. FineWeb-Edu against the writing-quality classifier correlates at r = 0.336.

Where they were weakest. At their published thresholds each admitted a small fraction of the authored reference set: 14.5% for FineWeb-Edu at its retention threshold of 3.0, 12.7% for the writing-quality classifier’s High class. FineWeb-Edu at 3.0 retains 7.96% of the corpus.

The two disagreed sharply about which formats deserve a high score, agreeing on one format only.

The writing-quality classifier’s highest-scoring format was professionally edited wire copy; FineWeb-Edu’s was reference material. A quality floor on the writing-quality classifier removed code-bearing prose: its High rate fell from 41.4% to 6.9% between documents without and with substantial code, a factor of six.

Input form mattered. Supplying markdown rather than stripped text raised reference-set discrimination from 0.24 to 0.45.

The format and topic classifiers

Two instruments assign one of 24 labels.5 They were not scored for separation; they were used to sort documents and to filter them.

Where they worked. Format classification was more robust as a blocklist than as a whitelist. Naming unacceptable formats retained 21 of 32 authored reference documents; naming acceptable formats retained 10, and the whitelist’s losses included documents classified “Q&A Forum” and “Customer Support” that were neither.

Where they failed. The format classifier degrades silently above transformers 4.46.3. Its custom rotary-embedding path misconfigures under 5.x and the model returns confident-looking output rather than raising. On identical documents, confidence on one product page was 0.999 under 4.46.3 and between 0.09 and 0.27 under 5.15.0, against a chance level of 0.042 across 24 classes. Nothing in the output shape distinguished the two runs.

The generative annotator

One instrument emits eighteen properties as structured JSON in a single generative pass, among them a five-level ordinal for depth of logical analysis.6

What it did. It produced the largest separation measured here, roughly three times that of the scalar instruments, and rated no commercial reference document at the top level.

Both variants rated every thin reference document at the lowest level and agreed with each other on 84% of depth-of-analysis labels, 82% of commercial-bias labels, 74% of content-quality labels and 66% of audience-level labels, with disagreements falling on single-step adjacencies.

Its properties correlate weakly with the scalar classifier they most resemble. Against FineWeb-Edu on 285 documents: educational value r = 0.436, depth of analysis r = 0.312, information density r = 0.239, content quality r = 0.121.

Its commercial-bias property recovered 54% of documents the format classifier labeled Product Page, and its content-type property marked 69% of them transactional or boilerplate. It assigned heavy commercial bias to 2% of Personal Blog and 1% of Nonfiction Writing documents.

Its coverage held across topics. On 285 documents stratified at 12 per topic, one output failed to parse (0.35%), one-sentence descriptions averaged 106 to 132 characters in every topic, and it operated across all 24 categories. The rate of documents rated analytical ranged from 33% in Politics, Literature and Health to 8% in Software, Games, Travel and Art & Design.

What it cost. It ran one to two orders of magnitude slower than the encoder classifiers, which completed the corpus in hours. Each variant emits approximately 700 tokens of JSON per document, and generation length rather than parameter count dominated: the 0.6B ran at roughly 2.8 times the throughput of the 4B while separating slightly better.

A 0.6B-parameter model exhausted accelerator memory at batch 32 because the key-value cache scales with batch size and sequence length rather than with parameter count. Left padding was required, since with right padding the generated continuation begins after the pad tokens, and sorting documents by length before batching kept each batch padded to a similar width.

Batching cost determinism as well as parse reliability. On 273 documents scored both ways with the same model and sampling disabled, raw JSON output was identical for 54.9%, while labels agreed far more closely at 98.5% for content quality, 97.1% for commercial bias and 96.0% for depth of analysis. Left padding alters position indices and attention over pad positions. The rise in parse failures came from length-sorted batching placing the longest documents together, where a shared prompt-token ceiling truncated them at once.

One packaged variant, propella-1-1.7b, ships without a generation configuration file where the 0.6B and 4B variants include one, and ran at a small fraction of the 4B’s throughput. The cause was not isolated.