ZIDAN MUBARAKBANDA ACEH, ID
Loading portfolio
Loading
ZIDAN MUBARAK
ENID
All work

01 / 06

Case study

Jawi OCR

Fine-tuning Qwen2.5-VL-3B to extract Jawi script and transliterate it to Rumi, with the two stages evaluated separately.

Model (opens in a new tab)Demo (opens in a new tab)
Role
Researcher and ML engineer (solo)
Year
2026
Stack
Qwen2.5-VL-3B · QLoRA · Unsloth · PyTorch · Hugging Face · Gradio
Reading time
6 min
Team
Undergraduate thesis, UIN Ar-Raniry
Bar chart comparing the base Qwen2.5-VL model with the fine-tuned adapter on CER Jawi, CER Rumi and WER Rumi.
Fig. 01 · ResultsThe results chart published with the model card: base model against the fine-tuned adapter on the held-out split.

Result

fewer character errors

CER Jawi: 82.51% → 17.35% on 790 held-out images

01 · Context

A script with no good defaults

Jawi is Malay written in Arabic script. It is the script of a large body of Acehnese, Malay and Bruneian documents, and it is not what general-purpose OCR was built for: the letters join, dots carry meaning, and vowel marks are often left out entirely.

The task has two stages. First extraction: read the Jawi characters off an image. Then transliteration: convert that Jawi into Latin Rumi following the Pedoman Ejaan Jawi Yang Disempurnakan (PEJYD). Translation is a third, separate problem and deliberately out of scope.

The thesis asked whether a small vision-language model could be adapted to both stages with parameter-efficient fine-tuning, and, more usefully, which training data actually made the difference.

02 · Setup

Thesis
Implementasi Multimodal Large Language Model untuk Ekstraksi dan Transliterasi Printed Jawi Script
Base model
Qwen2.5-VL-3B-Instruct
Method
QLoRA, 4-bit NF4, bfloat16
Training stack
Unsloth, PyTorch, PEFT
Hardware
NVIDIA T4 / A100 on Google Colab
Adapter
314 MiB safetensors, Apache 2.0
Serving
Hugging Face adapter + Gradio demo

03 · Data

Two kinds of samples, three ways to mix them

Synthetic data gives clean letterforms; real documents give noise, skew and age. The ablation exists to find out which one the model actually needed.

Synthetic

Jawi text rendered across several Arabic and Jawi typefaces, including Jawi Uthman Taha and Arabic Typesetting.

Authentic

Scanned Jawi samples, printed documents and handwriting, split so that the evaluation set stays unseen.

Held-out test

10% of the authentic set, never used in training. Every number below comes from this split.

04 · Approach

Five moves, in order

  1. 01

    Curate both datasets

    Render synthetic Jawi across fonts, collect authentic scans, and keep a held-out slice that never appears in training.

  2. 02

    Fix the instruction format

    A system prompt makes the order non-negotiable: extract first, then transliterate per PEJYD, and answer only in `Jawi:` and `Rumi:` lines so the output can be scored mechanically.

  3. 03

    Fine-tune with QLoRA

    Quantise the 3B base model to 4-bit NF4, train adapters with bfloat16 mixed precision, an 8-bit paged AdamW and gradient checkpointing via Unsloth.

  4. 04

    Run the ablation

    Train the same recipe on synthetic only, authentic only, and a combination. Compare all three on the same held-out images.

  5. 05

    Evaluate per stage

    Score Jawi extraction with CER, transliteration with CER Rumi and WER Rumi. Watch validation loss per epoch and pick the checkpoint that generalises.

05 · Results

What changed on unseen data

Zero-shot Qwen2.5-VL against the final adapter, 790 authentic images that neither model saw in training. Lower is better on every row.

CER Jawi

−79% error

Character error rate when extracting Arabic-script Jawi.

82.51%17.35%

CER Rumi

−82% error

Character error rate of the Latin transliteration.

90.59%16.48%

WER Rumi

−63% error

Word error rate: whole words read correctly.

107.08%39.36%

WER can exceed 100% when insertions outnumber correct words.

Accent bar = error remaining after fine-tuning, relative to the baseline.

06 · Reading the result

Good, not finished

The headline is a 79% drop in extraction errors, and the transliteration moved by a similar margin. The ablation pointed at the combination: synthetic fonts alone taught the letterforms but not the mess, and authentic documents alone did not cover enough variation.

The honest number is 39.36% WER. Roughly two words in five still need a human correction, which is why the model card recommends human-in-the-loop review before anything is published. Epoch 4 started to overfit, so epoch 3 is the artifact that shipped.

The Jawi transliteration Gradio Space with an image upload area, extraction and Rumi panels, and two example crops.
Fig. 02 · DemoThe Gradio demo that ships with the adapter. The two example crops are the ones in the Space repository.
A one-line printed Jawi crop used as an example in the demo.
A two-line printed Jawi crop used as an example in the demo.
Fig. 03 · Demo inputsThe two example crops shipped with the Space, one line and two lines.

What I took from it

  1. 01

    The combined scenario won: fonts alone did not cover real documents, and documents alone did not cover enough variation.

  2. 02

    Score each stage separately; a decent Rumi score can hide a bad extraction underneath it.

  3. 03

    Choose the checkpoint that generalises, not the one with the lowest training loss.

  4. 04

    Ship the demo next to the metric; the side-by-side comparison is what made the improvement legible.

Where it still fails

  • 39.36% word error rate on transliteration means a human still reviews the output.
  • The model card warns that blur, cropping and heavy background noise push CER up.
  • Cursive, heavily aged handwriting is harder than printed typesetting; the thesis scope is printed Jawi.
  • PEJYD has exceptions for Arabic loanwords that the model can still get wrong.
  • 790 test images is small by OCR standards; read the numbers as an estimate.

Sources