OpenAI's Whisper models support Cantonese, but spoken, colloquial Cantonese is a hard case for a general multilingual model. The community has published Cantonese fine-tunes on Hugging Face that are trained specifically for it. For Gongje, our on-device speech-to- text app, we needed those models in a form Apple hardware can run efficiently: CoreML, ideally on the Neural Engine.
We did this work back in February while building Gongje. This post walks through how it went, including the attempts that did not work.
The Tool: whisperkittools
Gongje uses WhisperKit for transcription, and Argmax, its authors, publish whisperkittools to convert Whisper checkpoints into the format WhisperKit loads. One command does the whole job:
whisperkit-generate-model \
--model-version <hugging-face-model-id> \
--output-dir ./cantonese-coreml 2>&1 | tee conversion.logIt converts the three pieces of a Whisper model separately — the mel-spectrogram front end, the audio encoder and the text decoder — targets the CPU and Neural Engine, then runs each converted model against the original PyTorch one and reports how closely they match. We used the defaults, which means no quantization: the converted models keep full precision.
Reading the Log
For the model we ended up shipping as the default on larger Macs, a large-v3 turbo fine-tune trained on Cantonese and English, the log recorded:
- Audio encoder: all 22,247 operations scheduled on the Neural Engine, with a PSNR of 52.5 dB against PyTorch. First load took 98 seconds while CoreML compiled it for the device.
- Text decoder: 203 of 205 operations on the Neural Engine, PSNR 41.1 dB, first load in 1.5 seconds.
- Mel spectrogram: 18 of 25 operations on the Neural Engine, PSNR 67.3 dB.
PSNR here measures how far the converted model's outputs are from the original's; higher is closer. The numbers that mattered most to us were the operation counts. A layer that falls back to the CPU forces data back and forth between processors, and for a model that runs continuously while you speak, that costs both latency and battery.
The first-load time is worth planning for. CoreML compiles a model for the specific device the first time it loads, so a 98-second wait happens once per Mac, right after download — exactly when a new user is deciding whether the app works.
Not Every Model Converted
We tried more fine-tunes than we shipped. One large-v3 Cantonese model failed on the first attempt and only converted when rerun into its own output directory. A Whisper small fine-tune produced an encoder but no text decoder. We did not record why, and we moved on to models that converted cleanly. If you try this yourself: give every model its own output directory, keep the full log, and check that all three components exist before uploading anything.
What Shipped
Two Cantonese models made it into Gongje:
- a distilled Whisper small fine-tune — around 300 MB on disk, the default for Macs with less than 16 GB of memory;
- the large-v3 turbo Cantonese and English fine-tune — about 1.5 GB, the default for everyone else.
Both are published on Hugging Face under our account, and the app keeps a reference to the original fine-tune each one was converted from. One practical wrinkle: WhisperKit's own downloader expects Argmax's repository layout, with models in subfolders. Our repositories keep the files at the root, so Gongje downloads them with a plain Hugging Face snapshot instead.
The Same Idea for the Language Model
Gongje can optionally clean up transcripts with a small local LLM. We applied the same approach there, converting a Cantonese-tuned Qwen2 7B model to Apple's MLX format with 4-bit quantization (group size 64), which brings it to about 4.3 GB.
Takeaways
- Community fine-tunes are often the best starting point for a specific language.
- Check where each layer will run, not just whether the conversion succeeded.
- Plan for first-load compilation in the user experience.
- Keep conversion logs; they are the only record of why a model works.