Test: Cydonia vs. Mistral

In this test I wanted to see if a finetune or the base model convinces me more. As always this is my personal opinion and highly subjective.

As disclaimer I had a JB active, so that the AI pushed all her warnings and constraints regarding the scenarios out at the beginning of it's reply, like telling me to never try that in real life and that this is a fictional scene meant for adults and all agreed upon activities given by the plot are consensual by definition in this story.

Testgroup:

  • mistral small 24B Instruct 2501 Q6_K 

  • Cydonia 24B v2.1 Q6_K (cydonia is based on 2501) 

  • mistral small 24B Instruct 2503 Q6_K

Setup: 


  • KoboldCPP 1.86.2 on cuda (c12) 

  • 16K context size 

  • Split between VRAM and RAM 

  • ContextShift and FlashAttention active 

  • Using ST with Mistral V7 for Context and Instruct Template
    • Temp: 1.17 

    • TopK: 50 

    • TopP: 0.5 

    • MinP: 0.075 

    • RepPen: 1.1

Personal Results: 


  • 2501: Just brutal, nailed the storylines, kept to my scene ideas, when asked for it invented crazy scenarios and described them in detail, sometimes added a surprising twist that matched the scenario and wasn't distracting but beneficial to the story 

  • Cydonia: wrote more "visual and haptic", altered some practice that mistral did described (but not in a refusal way, it was more like misunderstanding purposefully to circumvent certain thing in a clever way), used direct speech much more dynamically throughout the stories while base mistral used a narrator third person approach 

  • 2503: Followed instructions extremely good and when giving a line of objectives for the story handled the transitions the best, even allowed two things the other two models refused to write about

Testset: 


Conclusion: 
 From this small test set I can assume, that in case of Mistral the base model vs this specific fine-tune: 
 
 - Cydonia has more colorful descriptions
 - Mistral followed goals better and created good transitions between the goals 


I have no recommondation here, because they are really close and it basically depends on your requirements which suits you better. Personally I will stick to Cydonia until TheDrummer comes around with something new in the 12-32B range.