Ticking more than one box

A text classifier is a rule that reads a passage and assigns it to one or more categories chosen in advance. The passage can be a tweet, a complaint, a court headnote, a parliamentary question or a news report. The categories can be anything a person could be trained to recognise: the policy topic of a speech, the offence a judgment concerns, the grievance a complaint raises. Everything else in this notebook follows from two choices made before any classifier is built: what the categories are, and whether a text may belong to more than one of them.

This notebook explains those choices and what follows them: how text is turned into numbers, the three families of classifier in use, the difference between choosing one category and ticking several, how labels are checked against human judgement, and why a classifier's accuracy is not a correct count. By the end you should be able to read a classification result in a report and know which questions to ask of it. The figures draw on one corpus throughout: 3,477 complaints about television made to PEMRA, the broadcast regulator, between January 2020 and September 2024, each in the complainant's own words, which Adaad labelled with eight grounds of complaint and one target. The labels behind Figures 1 to 3 are a language model's, checked so far only against the complainants' own tick-boxes. The hand-coded check the notebook describes is still being done, so the shares are provisional.

Three shapes of the problem

The simplest classifier answers yes or no. Is this message spam? Does this speech mention the budget? That is binary classification, and it is the building block for everything else.

Multi-class classification assigns each text to exactly one of several categories. A parliamentary question is about health, or education, or defence, never two at once, because the codebook says so. The categories are mutually exclusive and together they cover every case, which is why such codebooks always carry a residual category such as "other". The Comparative Agendas Project codebook used to code legislative activity across countries is a multi-class scheme with about twenty major topics and more than two hundred sub-topics, and every bill, question or speech receives one code.

Multi-label classification lets a text carry any number of categories, including none. A drama complained of as "vulgar and against our religion" raises two grounds. A news story about a flood can be about disaster relief, agriculture and public finance at once. The categories are not exclusive, and the question for each one is separate: does this text raise it, or not?

The choice between the two is a decision about the material, not about the classifier. Forcing one label on a text that carries two does not merely lose information. It miscounts both categories, because the second label vanishes from the count and the first is credited with a case it shares. The number of labels a text carries on average, its label cardinality, is the first thing to measure in any multi-label scheme. An average close to one does not settle it, because half the texts carrying no label and half carrying two also average one, so the share of texts with two or more labels has to be read alongside it. At 1.5 or more, a single label per text would have thrown away a third of the labels.

In the PEMRA register, 34 per cent of complaints raise two or more of the eight grounds, and label cardinality is 1.3. Figure 1 shows which grounds travel together. Religion and morality appear in the same complaint 426 times, one complaint in eight, and false reporting pairs with political bias in 215. A one-label design would have had to choose between them every time.

Classification uses categories fixed in advance. Its unsupervised cousins, clustering and topic modelling, find groupings in the text without being told what to look for. They are useful for discovering what a corpus contains and poor for counting, because the groups they find are not the categories anyone asked about.

Turning words into numbers

A classifier cannot read. It works on numbers, so each text is represented as a list of them. The oldest representation is the bag of words: count how often each word appears and ignore the order. "The anchor insulted the minister" and "the minister insulted the anchor" become the same bag. Pairs of adjacent words, bigrams, recover some order at the cost of a far larger vocabulary. Runs of three to five letters, character n-grams, cope with spelling variation and mixed scripts, and are the usual choice for short, informal or multilingual text.

Raw counts give too much weight to common words. The standard correction, term frequency times inverse document frequency (TF-IDF), scales each word's count down by how many documents contain it, so a word that appears everywhere counts for almost nothing and a rare, specific word counts for a lot. The result is a long, mostly empty list of weights per text, which is what the statistical models below are fitted to.

The newer representation is the embedding, a short, dense list of numbers produced by a language model, in which texts with similar meaning sit close together whatever words they use. Embeddings carry negation and paraphrase that a bag of words cannot, which is why a language model reads "not at all vulgar" correctly when a word count sees "vulgar". They are also opaque. Nobody can say which entry in the list means what.

Three ways to build a classifier

The first is a dictionary. For each category, a person writes a list of words, stems or patterns, and a text carries the category if any of them appears. The list is the whole model, so it can be printed and argued with, and it needs no labelled examples. Its weakness is recall. People describe the same thing in more ways than any list anticipates, a list written in one language or script misses the others, and a dictionary fails silently on negation.

The second is a supervised statistical model. It is given labelled examples, texts with the right categories already attached, and learns which features predict which labels. Logistic regression, naive Bayes and support vector machines are the common choices, and on bag-of-words features they are fast to fit and easy to inspect. Each word gets a weight, and the words with the largest weights for a category are its learned dictionary. The model learns only what the examples contain, so it needs enough of them per category, drawn from the same kind of text it will be asked to label.

Figure 2 prints that learned dictionary for the PEMRA model, trained on the language model's labels for 2,342 distinct complaints. Vulgarity's heaviest words are "vulgar", "vulgarity" and "indecent", as a person would have written them. The fourth is "our", learned from "our culture" and "our values", and a bag of words cannot tell that it is the noun doing the work. The lists also show the model learning Urdu and English words for the same ground side by side, because both appeared in the examples.

The third is a language model given the codebook as instructions. It reads the category definitions and the text and returns the categories, with no training examples (zero-shot) or a handful (few-shot). It reads negation, conditions and reported speech, handles several languages, and can be asked to quote the passage that justifies its decision and to abstain when the text does not support one. Its costs are different in kind. It cannot be inspected the way a weight can, its decisions shift with the wording of the instructions and the version of the model, and its agreement with itself is not evidence that it is right. A language model's labels are checked the same way as anyone else's, against a person's. A common fourth route has the language model label a large corpus and a small statistical model learn from those labels, so the labelling can be repeated cheaply and inspected. The check is still against human labels.

The worked example below runs all three on two complaints from the register. On the first, a complaint about an advertisement, they agree. On the second, a warning that action against a news channel would cause sectarian riots, the dictionary sees three religious words and stops, the model gives religion a score of 0.55 and hate 0.15, and the language model returns religion and hate together with the clause that justifies them. The second complaint is also a case for the codebook rather than for any classifier. A warning that punishing a channel would cause sectarian riots may object to incitement in a broadcast, or may be a threat about the consequences of regulating it. The language model's reading is one answer to that question, not the settled one.

One decision or several

A multi-class classifier makes one decision among all the categories at once. A statistical model gives each category a score, converts the scores into probabilities that add up to one, and chooses the highest. A language model is told to return exactly one. Either way the categories compete, and the winning category's strength is relative to the others.

A multi-label classifier makes a separate yes-or-no decision for every category, with no competition between them. The usual construction, binary relevance or one-versus-rest, fits one binary classifier per category, so a scheme with eight categories is eight classifiers, each with its own score and threshold. A text is labelled with every category whose score clears its threshold, which may be none.

The two designs can sit side by side on the same text. Each PEMRA complaint carries one target, decided multi-class, and any number of grounds, decided multi-label. Figure 3 crosses them. Complaints about dramas raise morality in 74 per cent of cases and gender in 30 per cent; complaints about news raise political bias in 41 per cent and false reporting in 33 per cent, and morality in 8 per cent. The target is one decision and the grounds are eight, and the chart only exists because they were kept apart.

The threshold is where judgement enters. A score of 0.5 is the default and is rarely right for every category, because a category that is rare in the training data receives low scores even for its true cases. Each category's threshold is set from its own precision and recall on held-out labelled data and reported, because two results built on different thresholds are not comparable.

Binary relevance treats the categories as independent, which they are not. A complaint raising religion is more likely to raise morality as well. The simpler construction is adequate for most reporting, provided the co-occurrence is measured and shown.

Where labels come from

Every classifier is judged against labels a person gave, so the quality of the judgement is the quality of those labels. Three things make them worth trusting.

The first is a written codebook. Each category has a definition and, more usefully, rules for the borderline cases, decided before anyone starts coding. A codebook that says what to do with a love affair in a drama (morality, not gender), an insult to a cleric (religion, not politics) and an unreadable text (other, with a note) produces consistent labels. One that lists category names does not.

The second is agreement between coders. Two people code the same sample independently and the agreement is measured, usually with Cohen's kappa, which discounts the agreement two coders would reach by chance. Kappa above about 0.7 is the usual bar for a reliably coded category. Below about 0.4 the definition needs rewriting, not the classifier. One coder's labels, which is what most projects have, are reported as such.

The third is blindness. The person coding the gold set must not see the classifier's output or anything else that could anchor the decision. A gold set coded with the model's suggestions in view measures agreement with the model, not the truth.

The register shows what labels look like without any of the three. PEMRA's web form offers tick-boxes from its code of conduct, and 169 complainants used them, ticking 1.6 boxes each on average. Figure 4 sets the ticks against a reading of the same texts. One complaint reads, in full, "Please take action", with five boxes ticked, among them racialism and sectarianism. Another objects to Indian advertisements on a wildlife channel and ticks contempt of court. The reading agrees with the ticks on religion in 55 of 65 cases and on hatred in 29 of 61. The ticks are a record of what a complainant clicked, not a judgement against a codebook, and they are not a gold set.

The gold set has to cover what the report will claim. If shares are reported by year and by language, the sample is stratified by year and language, so each stratum has enough labelled cases to measure the classifier's error there. Categories rare in the corpus are rare in the sample too, and the method note says their error rates are uncertain rather than reporting a precision of 0.80 estimated from five cases.

The labelled data must also be kept apart from the classifier being judged. A statistical model is trained on one part and tested on another, and near-duplicate texts stay together on one side of the split, because a model that has seen one copy of a form letter will score the second from memory. The gold set is never shown to a language model as examples either.

Measuring a classifier

For one category, four counts describe everything. True positives are the texts the classifier labelled with the category that really carry it. False positives are the texts it labelled that do not. False negatives are the ones it missed. True negatives are the rest. Laid out as a two-by-two table they are the confusion matrix, and every metric is a ratio of its cells.

Precision is true positives divided by all the positives the classifier produced, so of the labels it gave, how many were right. Recall is true positives divided by all the true cases, so of the cases that exist, how many it found. The two trade against each other through the threshold. A classifier that labels everything has perfect recall and a precision equal to the category's share of the texts, and one that labels only the certain cases has high precision and low recall. F1, the harmonic mean of the two, is the single number usually reported, and it hides which of the two is weak.

Figure 5 scores the three PEMRA classifiers this way, ground by ground, against the only human labels the register had at press time: the complainant's tick-boxes. Read it with Figure 4 in mind. Where the reference labels are noisy, the measured precision and recall can be too low or too high, and a disagreement does not say which side is wrong. A correct reading of a text the complainant mis-ticked counts as a false positive, and a wrong reading that matches a wrong tick counts as a hit. The figure is a demonstration of the measurement, not a verdict on the classifiers; a hand-coded gold set replaces the ticks in the published data when it is done.

Accuracy, the share of all decisions that were right, is the metric most often quoted and the least informative, because categories are usually rare. If 5 per cent of texts carry a category, a classifier that never assigns it is 95 per cent accurate. Precision and recall for the positive class are the honest pair.

For multi-class problems, the confusion matrix has one row and column per category, and the question is how to average across them. Micro-averaging pools all the decisions and is dominated by the large categories. Macro-averaging takes the mean of each category's score and gives a category with ten cases the same weight as one with a thousand. The two can disagree sharply, and a report gives both.

For multi-label problems, two more measures matter. Hamming loss is the share of all label decisions, one per category per text, that were wrong. It is lenient, because most decisions are correct negatives. Exact match, or subset accuracy, is the share of texts whose entire set of labels was right, and it is severe, since one wrong decision among eight fails the text. A count of texts carrying each category rests on the per-category metrics. A claim about which categories occur together rests on exact match.

From a label to a count

A classifier is usually built to answer a question about a population: what share of the complaints raised religion, how many questions concerned health. Its labels do not answer that question directly, and the difference is the most common error in published text analysis.

Suppose a category is really present in 30 per cent of texts, and the classifier finds 80 per cent of the true cases (its recall, or true positive rate) while wrongly labelling 5 per cent of the texts that do not carry it (its false positive rate). The classifier reports 24 per cent found plus 3.5 per cent false, or 27.5 per cent. The error is small here. If the true share were 5 per cent, the classifier would report 4 plus 4.75, or 8.75 per cent, nearly double the truth, because the false positives come from the much larger group of texts that do not carry the category.

The gold set supplies both rates, and they correct the count. The adjusted share is the observed share less the false positive rate, divided by the difference between the true positive rate and the false positive rate: (27.5 − 5) / (80 − 5) returns 30. The correction holds when the error rates in the gold set match those in the corpus, which is why the gold set is stratified like the report. Each subgroup reported needs its own error rates or an argument that they are the same.

Two warnings follow. The correction is undefined when the two rates are equal, which is a classifier doing no better than chance, and unstable when they are close. And an adjusted share that moves when the threshold moves is a diagnostic. It says the gold set is too small for that category, or that the classifier's errors differ between the gold set and the corpus.

The first exercise below has no data in it. You set the true share and the two error rates, and the arithmetic shows what the classifier would report, what the correction recovers, and how far a gold set of a given size can miss.

The second exercise lets you move the threshold on the PEMRA model for one ground at a time and watch precision, recall and the share of the register that gets labelled respond. It scores the model against the complainants' tick-boxes, so it shows the trade-off and not the correction, which has to wait for the gold set.

What classification cannot tell you

A classifier reads what the text says. It measures how often a category is raised in the texts that exist, not how often the thing behind the category happened. A rise in complaints raising vulgarity is a rise in complaints, and whether broadcasts changed, or complainants did, is a question the labels cannot settle. The selection of texts into the corpus is the part of the analysis no classifier improves.

Figure 6 is the PEMRA register month by month. Of its 3,477 complaints, 931 sit in 47 campaigns of five or more near-identical texts, counting each campaign's first filing, and the largest is one letter against a talk show filed 247 times in April 2020. The register then collapses to 191 complaints in 2022 and to 21 in 2024. Any share computed from these labels is a share of who wrote in, when, and how many times. The labels are as good as the gold set says; the denominator is a different question.

The same labels give a different count under a different denominator. Counted over every filed complaint, vulgarity and morality is raised in 38 per cent of the register. Counted once per distinct text, it is raised in 47 per cent. Removing repeated filings therefore raises its share of the distinct texts. Political bias moves the other way, from 14 to 9 per cent. Neither figure is wrong. One answers how often the regulator was written to about a ground, the other how many different complaints were made about it. The exercise below lets you switch between them.

The categories themselves drift. A codebook written for one period applies less well to another as the language and the subject change, and a language model's labels drift with the model. A series built from classified text carries a date on its codebook and its model, and earlier periods are re-labelled with the current method before they are compared.

The recipe, and what it needs

Write the codebook first, with the borderline cases decided and the choice between one label and several made on the material. Represent the text in a way suited to it. Code a gold set blind, stratified by whatever the report will break the results down by, with two coders where possible. Keep near-duplicates together in every split.

Run the cheapest classifier that could work and then the others, and report precision and recall for every category against the gold set, with micro and macro averages and, for multi-label work, Hamming loss and exact match. Choose and report a threshold per category. Correct the published shares with the error rates from the gold set and say where the correction is unstable. Publish the codebook, the gold labels, the per-text labels and the metrics alongside the counts, so that the next person can disagree with a label rather than with a number.

What the method needs is not computing power but attention: a codebook written with care, a few hundred texts read by a person, and the discipline to measure the classifier against them before believing anything it counts.