Pre-reviewing student reports with local AI, part 1: meeting real documents
Part 1 of 2: building the checker and tuning it on real submissions (phases 1 and 2). Part 2 will cover the automated pipeline and packaging it so the professor can host it (phases 3 and 4).
The project in one paragraph
A university professor receives project reports from student teams in a statistics / design-of-experiments course. Before reading them, they want a quick pre-review of each one: spelling and grammar error counts, whether the report follows the course template, and whether the references follow APA or IEEE. The output is one spreadsheet row per submission plus a detailed Markdown report.
The hard constraint: everything runs locally. Student work is not uploaded to any cloud API. That rules out the obvious “send it to a big hosted model” approach and shapes every decision below (the one narrow, opt-in exception I ended up building is near the end).
The stack:
- Python package (
revisor) that does the analysis, with a CLI. - pdfplumber to read the PDFs (text, bold, position of each line).
- LanguageTool in a Docker container for Spanish spelling and grammar.
- Ollama running natively (for GPU access) with a small model to comment on references
(
gemma4:e4bby the end of this post). - n8n later, as the visible pipeline the professor can adjust without touching code.
- Target machine: Linux, 32 GB RAM, a 6 GB VRAM GPU.
Phase 1 built the package and tested it against PDFs generated from the template. Phase 2, most of this post, was the first contact with real submissions — four reports from actual teams.
Setup: the easy part
LanguageTool came up with a single docker compose up -d languagetool and answered in Spanish on the first
try. A quick sanity check:
curl -s -d "language=es" -d "text=La investigacion fue buena. Los resultado es claro." \
http://localhost:8010/v2/check
returned a misspelling (MORFOLOGIK_RULE_ES, category TYPOS) and an agreement error
(AGREEMENT_NOUNS). Ollama 0.40 was already installed with qwen2.5:7b (a 7.6B model, 4-bit), which
supports JSON-schema structured outputs.
Then I ran the tool on the four real PDFs, in dry-run mode, and it all went sideways.
First run: numbers that couldn’t be right
| Team extracted | Spelling | Grammar | References flagged | |
|---|---|---|---|---|
| Submission A | ✅ | 20 | 17 | 0 of 6, but “mixes APA and IEEE” |
| Submission B | ❌ empty | 40 | 5 | 4 of 4 (all labelled APA — they were IEEE) |
| Submission C | ⚠️ team name + the professor’s name glued on | 247 | 19 | 6 of 7 |
| Submission D | ❌ empty | 121 | 5 | 4 of 14 (it has 7 references) |
247 spelling errors in a university report is not impossible, but it smelled wrong. It was. Almost every oddity traced back to a bug rather than a setting.
Five bugs that only real documents could reveal
1. The flagged word didn’t match its context
The report showed things like "A e" flagged where the word was GPA, and "i Sharm" for Urvashi.
The offsets were drifting further the deeper into the document you went.
The cause: LanguageTool is written in Java, and Java strings count UTF-16 code units. Python counts
code points. They agree for almost all text — until a character outside the Basic Multilingual Plane shows
up. Word’s equation editor produces exactly those: the italic math letters in 𝐻₀ and 𝐼𝑄𝑅 are
U+1D400-range characters that count as two units in Java and one in Python. Every equation shifted
every later offset by one more character.
The nasty part is the knock-on effect: the shifted “words” ("A e" instead of GPA) also slipped past
the acronym filter and the custom dictionary, so the bug inflated the counts and garbled the report.
The fix is a small offset map from UTF-16 units to Python indices before slicing.
2. “Grupo” is both a label and a team name
The cover page is a two-column table: label, value. The tool finds the label “Número y Nombre del Equipo” and takes the text up to the next known label. One of the other labels on the cover is “Grupo” (the course section). Two teams wrote their team as “Grupo 3 …” and “Grupo 04. …” — so the value was cut off at its very first word and came back empty.
The fix: a label occurrence right at the start of the value, followed by a number, is part of the value.
A related surprise: three of the four covers said “Profesor(a)” where the template says “Docente”.
Students had used an older version of the cover. That produced a false “cover is incomplete” in three
reports and let the professor’s name leak into a team name. Profiles now support label aliases
(Docente: [Profesor(a), Profesor, Profesora]).
3. The LLM overrode a fact
The design had the local model’s verdict on each reference as final, “because it handles cases the rules
cannot.” On real data, the model relabelled every [1], [2]… entry as APA. A reference that starts with
[n] is IEEE — that’s not a judgment call. The relabelling then triggered a bogus “the list mixes APA
and IEEE” warning.
The fix: the rules decide style and validity. The model’s output is shown as “suggestions” only.
4. First-line indents vs. hanging indents
Reference lists come in two layouts. APA usually has a hanging indent (first line flush left,
continuation lines indented). One team’s IEEE list did the opposite: the [n] line was indented and the
continuation lines were flush left. The splitter saw “flush left” and started a new reference on every
continuation line — 7 references became 14 fragments.
The fix: when a list uses [n] markers, the indentation is ignored and entries start only at [n] (or at
a line that clearly looks like a new APA entry).
5. Tables leaking into “prose”
Statistics reports are full of tables. Extracted line by line, a table row becomes word salad:
Dis 27 1.1 0.8 9.7 Se tancia 100 15131 rechaza la normalidad. The prose filter was letting some of these
through, and LanguageTool dutifully flagged Dis, tancia, Brillant, 0.9 0.9 (repeated word!) and
so on.
pdfplumber can detect table areas on a page, so lines inside a detected table are now excluded from the language check. I verified this didn’t drop real prose: everything removed was table cells, statistics or two-column rows interleaved by the extractor.
Tuning: separating real errors from domain vocabulary
With the bugs fixed, the remaining false positives were mostly legitimate words the speller doesn’t know. Each fix was checked by diffing the flagged words before and after, to make sure no real error was silenced.
- Domain dictionary. Statistical terms like homocedasticidad, intercuartil, Tukey, Lilliefors, plus tool names (Colab, dataframes). One team’s invented-but-reasonable term bailabilidad (danceability) alone accounted for about 100 of their 247 hits.
- The document’s own vocabulary. Column and variable names (
occ,flow,valence,danceability) dominated two reports. They’re different for every team, so a dictionary can’t keep up. Instead: any word that appears inside the document’s tables or as part of asnake_caseidentifier is accepted for that document. Submission D dropped from 122 to 37. - The reference list. Cited authors (Miroudot, Bramich, Almujibah) are accepted if they appear in the document’s own reference list.
- Single letters, with a nuance.
p(as in “valor p”),k(from “k-pop”) andb.(a list item) are noise. But LanguageTool also has a rule that flags a loneathat should beha(“a comido” → “ha comido”) — a real and common error. So single letters are ignored only for the two rules that flag them as unknown words, not across the board. - Punctuation category. Mostly optional commas (“…ya que”, “…pues”) and brackets broken by formulas. The professor chose to ignore it.
LanguageTool’s real Spanish category ids, which I’d only seen in mocks before: TYPOS, TYPOGRAPHY,
PUNCTUATION, CASING, AGREEMENT_NOUNS, GRAMMAR, CONFUSIONS, MISC, CAMBIOS_NORMA.
AGREEMENT_NOUNS produced the best true positives of the whole exercise.
Where the numbers landed
| Spelling (first → now) | Grammar (first → now) | Team | |
|---|---|---|---|
| Submission A | 20 → 3 | 17 → 6 | ✅ |
| Submission B | 40 → 16 | 5 → 3 | ✅ (was empty) |
| Submission C | 247 → 25 | 19 → 9 | ✅ (was polluted) |
| Submission D | 121 → 30 | 5 → 3 | ✅ (was empty) |
What’s left is mostly real: envíada, la cuál, carácteres, cumpliendose, myor, segúnla, Por estás razones, Dado a que, las desviación. The remaining noise is a handful of English words and column names that appear only in running text.
Product decisions, not code decisions
Some “false positives” turned out to be questions for the professor rather than bugs:
- Missing sections in early stages. The template has 16 sections; an early proposal naturally lacks Results or Conclusions. I proposed per-stage section lists. The answer: no — all sections are required, the professor knows which ones a stage can skip and decides during manual review. The tool reports everything; the human filters.
- Section names. One team wrote “7. Variables” instead of “Variables del Estudio”. I proposed aliases. The answer: section names must match the template, period. Using the template’s headings is part of following the template.
- Typos in headings are still tolerated (numbering, case, accents and a ≥ 90% fuzzy match), because that’s rendering, not structure.
Lesson: when a tool’s output looks wrong, sometimes it’s wrong, and sometimes it’s correctly reporting something the user wants to judge themselves. Ask before “fixing”.
The hard part: getting a small local model to comment on references usefully
The rules can say whether a reference is APA or IEEE and catch structural problems. The hope was that a local LLM could add human-like comments (“the date should be year-first”, “the URL is missing”).
Attempt 0: the original prompt
A generic instruction (“decide if each reference follows APA 7 or IEEE, list the problems”) with all
references in one request. qwen2.5:7b produced comments like:
- “Falta el año de publicación” — on a reference with the year right there.
- “El año debe estar entre paréntesis” — on
Guo, X. (2023). - “El autor no está en formato apellidos, inicial” — APA’s author rule applied to an IEEE reference.
- “Falta el volumen” / “Falta la página” — on web pages, which have neither.
Attempt 1: teach it the formats
Small models don’t know APA 7 and IEEE in detail, so the prompt now includes a compact style guide: for each format, the template per type of work (journal article, book, report, web page, repository, dataset), plus explicit review rules:
- Judge against the style the rules detected (passed in with each reference).
- Only require what that type of work has — a web page has no volume or pages.
- Before saying “X is missing”, look for X in the text.
- “Enlace”/“Link” in place of an address means the URL wasn’t copied.
- Ignore italics, fonts and odd spacing (PDF extraction artifacts).
- If it’s fine, return an empty list. Better to say nothing than something doubtful.
Plus six worked examples. Two traps along the way:
- Silent context truncation. The guide is ~2.5k tokens. Ollama’s default context window can be
smaller than that and it truncates without an error — the model would simply never see half the
instructions. Setting
num_ctx: 8192explicitly costs ~0.5 GB of VRAM on a 7B model. - Test-set contamination. My first draft of the examples used references taken from the test PDFs. That would have made the model look better on exactly those files. I replaced them with invented references of the same shapes.
Result: better on some things (it started catching “Enlace” and the date order), but a new failure appeared — the model copied an example’s list of problems verbatim onto references it didn’t fit. Four correct IEEE journal articles were told their authors were in APA format and their titles lacked quotes.
Attempt 2: one reference per request
With the whole list in one answer, examples and neighbouring entries bleed into each other. Reviewing one reference per request fixed most of the copying, and costs little because Ollama reuses the cached system prompt between requests. Five correct references in one report went from four bogus comments each to none.
Still, qwen2.5:7b ended up with roughly 8 correct comments and 20 wrong across the 24 references.
Good prompt engineering got it from useless to “mostly noise”. Time to look at the model.
Model comparison on a 6 GB GPU
Same prompt, same 4 PDFs, every comment checked by hand:
| Model | Correct | Wrong | Time per report | Fits in 6 GB VRAM? |
|---|---|---|---|---|
| qwen2.5:7b | ~8 | ~20 | 15–35 s | yes |
| qwen3:8b, thinking off | ~15 | ~29 | 26–49 s | mostly (4.0 of 6.3 GB at 8K context) |
| qwen3:8b, thinking on | ~18 | ~9 | 227–722 s | mostly; one request timed out at 300 s |
| gemma4:e4b | ~16 | ~6 | 124–204 s | no (2.9 of 8.3 GB; rest on CPU) |
Observations:
- qwen3:8b without thinking is fast but chattier than qwen2.5, and wrong more often: it asked web pages
for DOIs and told URLs starting with
https://to start withhttps://. - qwen3:8b with thinking is close to gemma in quality — it even caught a Markdown-style URL
[https://…](https://…)pasted into a reference, which nothing else noticed — but writes ~600 tokens of reasoning per reference. A minute per reference is too slow, and it timed out once. - gemma4:e4b made the fewest wrong comments, copied nothing from the examples, and left correct references alone. It flagged the missing URLs, the year-first date format, “et al.” not being allowed in an APA list, and missing years on web pages.
- Bigger isn’t an option: qwen2.5:14b (≈9 GB at 4-bit) would run mostly on the CPU, and the disk didn’t even have room for it. llama3.1:8b is the same generation as qwen2.5 and unlikely to do better at following detailed instructions.
Since the pipeline runs on a schedule (n8n), not while someone waits, 2–3 minutes per report is acceptable. A batch of 15 submissions is about 45 minutes overnight. And because the rules — not the model — decide the verdict, a slow or wrong model never changes the spreadsheet; it only affects the quality of the suggestions in the report.
Lessons so far
- Synthetic test documents hide the interesting bugs. None of the five bugs showed up in phase 1’s generated samples. Real students use old cover versions, paste equations, indent lists their own way and put tables everywhere.
- Unicode is never “just text”. Mixing a Java service and Python code means two definitions of “character”. Equations made it visible.
- Diff the flagged words, not just the counts. Every tuning step was validated by listing exactly which words stopped being flagged, so no real error was silenced by accident.
- Deterministic rules for facts, LLMs for commentary. “Starts with
[n]” is a fact; don’t let a 7B model overrule it. - Small models need the rules spelled out, one task at a time. A written style guide, one reference per request, and explicit permission to say “nothing’s wrong” mattered more than any wording tweak.
- Watch for silent failures in the plumbing: context truncation, timeouts, a model that doesn’t fit in VRAM and quietly runs on the CPU.
- Some “bugs” are product decisions. Ask the user before engineering them away.
Beyond switching models: other options on the table
Swapping one small model for another has limits, so I looked at two other directions. First, a clarification that shapes both: in this tool the LLM only comments on references. Spelling and grammar come from LanguageTool, which is rules and dictionaries, not a model.
Option 1: a dedicated Spanish spelling and grammar model
It can be distilled: a large “teacher” model labels lots of Spanish text, and a small “student” model is trained on those labels. A realistic recipe:
- Data. Public Spanish academic text (theses, Wikipedia, open journals) with automatically injected errors: accents, agreement, haber/a, porque/por qué, cuál as a relative. Add the COWS-L2H learner corpus (real errors annotated by humans) and corrections from a large cloud model on that same public text. Never student text.
- Student model. A 1.5–3B model fine-tuned with LoRA (fits a 6 GB GPU with QLoRA) to output a list of edits (“span → correction, spelling or grammar”), not a rewritten text. Counting errors needs the edits.
- Evaluation. A hand-labelled set from real reports, labelled locally. This is the step that decides whether it works, and the easiest one to skimp on.
Effort: roughly 4–8 weeks part-time: 1–2 weeks of data, a few days of training, 1–2 weeks of evaluation and iteration.
The risks: after tuning, LanguageTool’s remaining hits are mostly real errors, so the headroom is smaller than it looked on day one. A 3B model on this GPU would need minutes per report to go through all the prose. And some API providers restrict using their outputs to train other models, so the terms need checking before generating training data.
A cheaper variant with better payoff: a false-positive filter. Keep LanguageTool and train a compact Spanish classifier (BETO / RoBERTa-es size, ~110M parameters, runs fine locally) that answers one question per flag: is this a real error? The labels come for free over time, from the professor accepting or rejecting flags during manual review. Days of work instead of weeks, and it gets better the more the tool is used.
Option 2: anonymized text to a cloud LLM
Technically easy, maybe 1–2 days: a local step strips the cover page, replaces the names the tool already parses from it (students, team, professor) with placeholders, runs a local name detector (spaCy or Microsoft Presidio for Spanish) to catch names elsewhere, and sends only the selected sections. A large hosted model would handle references well, and probably grammar better than LanguageTool.
The hard part isn’t the code:
- It breaks the founding requirement that everything runs locally and student documents are not uploaded anywhere. Changing that is the professor’s call, and likely the university’s: student work is the students’ intellectual property, and Costa Rica’s data-protection law (Ley 8968) may apply.
- Removing names isn’t full anonymization. A report about a unique project (one team’s traffic dataset, another’s ultrasonic-sensor experiment) can be identified by its content alone.
- Provider terms matter. Data retention and whether API inputs are used for training have to be checked and documented, not assumed. Many APIs don’t train on API inputs by default, and some offer zero-retention agreements.
The middle ground: send only the reference list. References are public bibliographic data (titles,
authors and journals of published works), with no student prose and no student names. That’s exactly
where the local model struggles most, and it’s only 1–2k tokens per report. It still needs the professor’s
sign-off, since it’s a cloud call, but it’s a much easier case to make than sending student writing. It
is a per-assignment setting, off by default (references_cloud: false).
Trying it: references only, in the cloud
I built the middle ground and ran it on the same four PDFs. The design:
- What leaves the machine: only the extracted reference entries, each tagged with the style the rules detected. No cover, no names, no prose, no PDF.
- One request per document, not per reference: a large model doesn’t copy problems across entries the way the small ones did, so batching is safe again. The same written APA/IEEE guide goes in as the system prompt.
- Structured output: a JSON schema (
n, style, valid, problems) so the answer maps straight back onto the entries. - Rules still decide. The model’s comments stay labelled as suggestions and never change the spreadsheet. If the API refuses, the key is missing or the network is down, the report says so and the rule-based verdict stands.
- The API key comes from an environment variable, never from the config file the professor edits.
Results, every comment checked by hand like before:
| Model | Correct | Wrong | Time per report | Cost |
|---|---|---|---|---|
| gemma4:e4b (best local) | ~16 | ~6 | 124–204 s | free (electricity) |
| qwen3:8b, thinking on | ~18 | ~9 | 227–722 s | free |
| claude-opus-5-5 (cloud) | 25 | 0 | 10–17 s | ≈ US$0.03 per report |
What it caught that the rules don’t:
- The word “Enlace” (“Link”) left where the URL or DOI should be: the hyperlink text survived the export to PDF, the address didn’t. The rules missed all seven.
- APA dates written as “(25 de junio del 2026)” instead of year first: “(2026, 25 de junio)”.
- “et al.” in an APA reference list, where APA 7 lists the authors.
- IEEE web references missing “Available:” before the URL, and a comma that belongs inside the closing quotes.
Just as important, it left correct references alone and didn’t invent problems: no “missing years” that were there, no style confusion, and the Spanish IEEE form “[En línea]. Disponible:” was accepted. What it missed: some APA article titles not in sentence case, and a stray “?.” in a title. One comment (add the month to an IEEE journal reference) was right but nitpicky.
Cost turned out lower than I estimated: about 3,500 input and 500–900 output tokens per report, so a batch of 15 submissions is under half a dollar. Speed matters less here, since the pipeline runs overnight, but it removes the “will the GPU keep up” question entirely.
Where I landed
- References:
gemma4:e4blocally by default. The “references only” cloud option is built and tested, but off (references_cloud: false) until the professor decides, now with numbers to decide on: no wrong comments, seconds per report, cents per batch. - Spelling and grammar: keep LanguageTool, and add the false-positive filter once real use has produced feedback data.
- Full distilled grammar model: not now. Weeks of work for a gain that may be small after tuning, but a good candidate for a later post.
Closing phase 2
A few last items before calling the phase done:
- Switched the local model from
qwen2.5:7btogemma4:e4b. A spot check on one report took 2 min 21 s and gave the same kind of comments as in the comparison. - Turned a model finding into a rule. The cloud model kept pointing out “Enlace” where a URL should be, which the rules had missed. That’s a pattern, not a judgement call, so it became a rule: a reference that ends in “Enlace”, “Link” or “Aquí” and has no URL or DOI is invalid. On the test set it flags exactly the seven entries the model found, and nothing else. A pattern worth repeating: use the big model to discover what the rules are missing, then encode what you can as rules — they’re free, instant and deterministic.
- Ran the real flow end to end. The four PDFs went into the inbox folder, and one scan produced four spreadsheet rows, four reports, and moved each file to the processed folder. A second scan found nothing to do, and the JSON output that n8n will read was clean.
- Tested the wrong-format path. A
.docxdropped in the inbox got an INVÁLIDO row with the team read from its cover, no report, and was moved out of the inbox. - A small process lesson: the test PDFs were moved into the inbox, so after the scan the test folder was empty. Nothing was lost, but the test set is now treated as read-only: copy files into the inbox, never move them.
Coming in part 2
- Phase 3: n8n on a schedule (not a file watcher, which misses events) calling the checker through a small HTTP wrapper, all in Docker Compose with LanguageTool.
- Phase 4: package it with a Spanish README so the professor can host it themselves.
- And the professor’s answer on the “references only” cloud option.