Add Tokenizer Comparison Script

[pandora-s-git]

This script compares the basic .encode tokenization between Hugging Face and Mistral Common tokenizers across multiple datasets.

[juliendenize]

Thanks a bunch for these scripts, left few comments. For general feedback and discussions:

• Can you run pre-commit to silence the tests and get rid of linter/formatter issues ?
• As we plan to also host chat templates, wdyt about creating a folder to host scripts, files dedicated to integrations in libraries ?

[pandora-s-git]

As we plan to also host chat templates, wdyt about creating a folder to host scripts, files dedicated to integrations in libraries ?

ive currently added a scripts folder, would you like to have subfolders for each integration?

also, should I include in this script more test cases like basic instruct datasets, function calling, etc?

[juliendenize]

ive currently added a scripts folder, would you like to have subfolders for each integration?

Yeah actually I was thinking the other way around having a parent folder named integration or external ? (if you come up with better naming don't hesitate) that would contains scripts as a subfolder with chat_templates.

also, should I include in this script more test cases like basic instruct datasets, function calling, etc?

Yes it would be nice to have multimodal (audio, image), function calling, instruct and reasoning. If you need help lmk :)