[Python] Performance of fine-tuned Apple Foundation Model (.fmadapter) drops significantly...

I'm facing a challenging issue with a fine-tuned Apple Foundation Model and would appreciate any insights.

The Goal I have fine-tuned an Apple Foundation Model on my custom dataset. The goal is to integrate this model into an iOS application for on-device inference.

✅ What Works: Training & Testing The fine-tuning process went very well.

Model: Apple Foundation Model Version: 26.0.0

Fine-tuning Data: A custom dataset of 37 question-answer pairs in JSONL format

Environment: Python With PyTorch (Kaggle: Linux TPU Machine )

Performance: During training and testing in my Python environment, the model's performance was excellent. It gave accurate and relevant answers on my validation set.

❌ The Problem: Poor Performance in iOS App After the successful testing phase, I exported the model to an .fmadapter file for use with Core ML. I loaded this model into my iOS project using Swift.

However, when I run inference inside the app, the model's performance is extremely poor. It gives nonsensical, irrelevant, or completely wrong answers, even to the same prompts that worked perfectly during testing.

What I've Already Checked I suspect there's a mismatch between my training environment and the on-device execution. Here’s what I've investigated so far:

Prompt Formatting: I tried to ensure the input prompt format in my Swift code is identical to the one used during training.

Inference Parameters: I've set parameters like temperature to a low value in Swift to get more deterministic results, but the output is still bad.

Quantization: I understand the .fmadapter conversion involves quantization, which might affect performance, but the drop seems too severe to be caused by this alone.

❓ My Question What could be causing this dramatic difference in performance between my testing environment and the on-device Core ML execution? Are there common mistakes related to prompt formatting, character encoding, or the .fmadapter conversion that I might be missing?

Any help or suggestions for debugging this would be greatly appreciated!

Continue reading...