doc: README.md

README.md

@@ -1,22 +1,69 @@
 ---
+license: llama2
 language:
+- hu
 - en
-license: apache-2.0
 tags:
+- puli
 - text-generation-inference
 - transformers
 - unsloth
 - llama
 - trl
+- finetuned
 base_model: NYTK/PULI-LlumiX-32K
+datasets:
+- boapps/szurkemarha
+pipeline_tag: text-generation
 ---
 
-# Uploaded  model
+# PULI LlumiX 32K instruct (6.74B billion parameter)
 
-- **Developed by:** ariel-ml
-- **License:** apache-2.0
-- **Finetuned from model :** NYTK/PULI-LlumiX-32K
+Intruct finetuned version of NYTK/PULI-LlumiX-32K.
 
-This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
+## Training platform
+[Runpod](https://runpod.io) RTX 4090 GPU
 
-[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
+## Hyper parameters
+
+- Epoch: 3
+- LoRA rank (r): 16
+- LoRA alpha: 16
+- Lr: 2e-4
+- Lr scheduler: cosine
+- Optimizer: adamw_8bit
+- Weight decay: 0.01
+
+## Dataset
+
+boapps/szurkemarha
+
+Only Hungarian instructions were selected: ~53000 prompts.
+
+## Prompt template: ChatML
+```
+<|im_start|>system
+Egy segítőkész mesterséges intelligencia asszisztens vagy. Válaszold meg a kérdést legjobb tudásod szerint!<|im_end|>
+<|im_start|>user
+Ki a legerősebb szuperhős?<|im_end|>
+<|im_start|>assistant
+A legerősebb szuperhős a Marvel univerzumában Hulk.<|im_end|>
+```
+
+## Base model
+
+- Trained with OpenChatKit [github](https://github.com/togethercomputer/OpenChatKit)
+- The [LLaMA-2-7B-32K](https://huggingface.co/togethercomputer/LLaMA-2-7B-32K) model were continuously pretrained on Hungarian dataset
+- The model has been extended to a context length of 32K with position interpolation
+- Checkpoint: 100 000 steps
+
+## Base model dataset for continued pretraining
+
+- Hungarian: 7.9 billion words, documents (763K) that exceed 5000 words in length
+- English: Long Context QA (2 billion words), BookSum (78 million words)
+
+## Limitations
+
+- max_seq_length = 32 768
+- float16
+- vocab size: 32 000