saving model as GGUF file is not working

[aykutcayir34]

Export to GGUF format

model.save_pretrained_gguf(
"model",
tokenizer,
quantization_method="q4_k_m"
)

print("Save GGUF: model.save_pretrained_gguf('model', tokenizer, quantization_method='q4_k_m')")
print("✓ Same API as Unsloth!")

When I execute this block of code, I'm getting "FileNotFoundError: [Errno 2] No such file or directory: 'model/config.json'" error. I couldn't solve the problem.

[shankarrajus]

Hi,
Thanks for this package.
I am also facing the similar issue. Any idea on how to resolve it ?

Using unsloth-mlx==0.3.2 and unsloth==2026.1.2

[ARahim3]

Hi, @aykutcayir34 and @shankarrajus ,
I am extremely sorry for the delayed response. My day job kept me too busy to devote enough time to this project.
Thanks for your patience.

Can you try the latest release and give me some feedback?

[shankarrajus]

Thanks.
I tried with new version, unfortunately error still remains. I am attaching the error for reference.

==========
model.save_pretrained_gguf("model", tokenizer, quantization_method="q4_k_m")
LoRA adapters will be fused from: adapters
Exporting to GGUF format...
Exporting model to GGUF format...
Model: mlx-community/Llama-3.2-1B-Instruct-4bit
Output: model/model.gguf
Adapters: adapters

Running: mlx_lm.fuse --model mlx-community/Llama-3.2-1B-Instruct-4bit --export-gguf --gguf-path model/model.gguf --adapter-path adapters
Error during GGUF export:
Fetching 6 files: 0%| | 0/6 [00:00<?, ?it/s]
Fetching 6 files: 100%|██████████| 6/6 [00:00<00:00, 83055.52it/s]
Traceback (most recent call last):
File "/Users/shankar/Documents/study/aiml/agent/personal_projects/unsloth-projects/.venv/bin/mlx_lm.fuse", line 10, in
sys.exit(main())
~~~~^^
File "/Users/shankar/Documents/study/aiml/agent/personal_projects/unsloth-projects/.venv/lib/python3.13/site-packages/mlx_lm/fuse.py", line 64, in main
model, tokenizer, config = load(
~~~~^
args.model, adapter_path=args.adapter_path, return_config=True
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/Users/shankar/Documents/study/aiml/agent/personal_projects/unsloth-projects/.venv/lib/python3.13/site-packages/mlx_lm/utils.py", line 324, in load
model = load_adapters(model, adapter_path)
File "/Users/shankar/Documents/study/aiml/agent/personal_projects/unsloth-projects/.venv/lib/python3.13/site-packages/mlx_lm/utils.py", line 259, in load_adapters
return _load_adapters(model, adapter_path)
File "/Users/shankar/Documents/study/aiml/agent/personal_projects/unsloth-projects/.venv/lib/python3.13/site-packages/mlx_lm/tuner/utils.py", line 127, in load_adapters
with open(adapter_path / "adapter_config.json", "r") as fid:
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'adapters/adapter_config.json'

⚠️ Model config not found. This usually means:

1. The model path is incorrect
2. The model hasn't been downloaded yet

Try loading the model first with mlx_lm:
python -c "from mlx_lm import load; load('mlx-community/Llama-3.2-1B-Instruct-4bit')"

Trying alternative export method...
Alternative method also failed: Command '['mlx_lm.convert', '--hf-path', 'mlx-community/Llama-3.2-1B-Instruct-4bit', '-q', '--export-gguf']' returned non-zero exit status 2.

Manual export command:
mlx_lm.fuse --model mlx-community/Llama-3.2-1B-Instruct-4bit --export-gguf --gguf-path model/model.gguf

================

[ljhust]

same question and the AI give me the answer "mlx_lm.fuse does not support direct conversion of quantized models (here it's the 4bit quantized version 'mlx-community/Llama-3.2-1B-Instruct-4bit'), causing a NotImplementedError."

[ARahim3]

Hi @shankarrajus @ljhust ,

Thanks for your patience and for testing!

The adapter_config.json error is now fixed in v0.3.4. Our trainers now correctly save both adapters.safetensors and adapter_config.json after training.

However, there’s an issue that I should've caught and mentioned earlier: GGUF export from quantized (e.g., 4-bit) models is not supported by mlx_lm itself.

From mlx_lm source code (gguf.py:271):

if quantization:
    raise NotImplementedError("Conversion of quantized models is not yet supported.")

There are some workarounds regarding this, and I’ve updated the README with a Known Limitations section documenting this, please check.

Again, my apologies.