Upload LoRA adapter folder

.gitattributes

@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+checkpoint-1434/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-1912/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-2390/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-2868/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-3346/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-3824/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-4302/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-4305/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-478/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-956/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+final/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,209 @@
+---
+base_model: Salesforce/Llama-xLAM-2-8b-fc-r
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:Salesforce/Llama-xLAM-2-8b-fc-r
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Salesforce/Llama-xLAM-2-8b-fc-r",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "o_proj",
+    "down_proj",
+    "k_proj",
+    "up_proj",
+    "v_proj",
+    "gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c6008e1c411944d60a2071b0e6d0b690a8c2a1e05987e988f1e4e5734424128
+size 671149168

checkpoint-1434/README.md

@@ -0,0 +1,209 @@
+---
+base_model: Salesforce/Llama-xLAM-2-8b-fc-r
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:Salesforce/Llama-xLAM-2-8b-fc-r
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

checkpoint-1434/adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Salesforce/Llama-xLAM-2-8b-fc-r",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "o_proj",
+    "down_proj",
+    "k_proj",
+    "up_proj",
+    "v_proj",
+    "gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-1434/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46afe0209d02a435adc4b628228c72683504c7d141f1aa1ccec1726b3afcde89
+size 671149168

checkpoint-1434/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4614214155c667f00d9c097d1259286474e1968780f10eca63e08af1841a8ad6
+size 1342555602

checkpoint-1434/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d96c98f1c4a1d4739ec65fce3a9e15bd4c6cacdc12cb5f504e466ed2d3ff9005
+size 14244

checkpoint-1434/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d95869f7c3821e99bd35ab74d4aa35ac511220b477bfdfeaceb13fe5c8e9e4d
+size 1064

checkpoint-1434/special_tokens_map.json

@@ -0,0 +1,27 @@
+{
+  "additional_special_tokens": [
+    "<|eot_id|>",
+    "<|eom_id|>"
+  ],
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-1434/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a55c1a4c5e3af7f2fb2bc0cd245a09dabd742dc24e7cb3741db1e11c7fe1a52
+size 17210019

checkpoint-1434/tokenizer_config.json

@@ -0,0 +1,2070 @@
+{
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|finetune_right_pad_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|eom_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|python_tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|eot_id|>",
+    "<|eom_id|>"
+  ],
+  "bos_token": "<|begin_of_text|>",
+  "chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n    {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n    {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not tools is defined %}\n    {%- set tools = none %}\n{%- endif %}\n\n{#- Extract system message #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if messages[0]['role'] == 'system' %}\n    {%- set system_message = messages[0]['content'] | trim %}\n    {%- set messages = messages[1:] %}\n    {{- system_message + \"\\n\" }}\n{%- else %}\n    {%- set system_message = \"You are a helpful assistant that can use tools. You are developed by Salesforce xLAM team.\" %}\n    {% set format_instruction %}You have access to a set of tools. When using tools, make calls in a single JSON array: \n\n[{\"name\": \"tool_call_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}}, ... (additional parallel tool calls as needed)]\n\nIf no tool is suitable, state that explicitly. If the user's input lacks required parameters, ask for clarification. Do not interpret or respond until tool results are returned. Once they are available, process them or make additional calls if needed. For tasks that don't require tools, such as casual conversation or general advice, respond directly in plain text. The available tools are:{% endset %}\n    {{- system_message + \"\\n\" }}\n    {%- if tools is not none %}\n        {{- format_instruction + \"\\n\\n\" }}\n    {%- endif %}\n{%- endif %}\n\n\n{%- if tools is not none %}\n    {%- for t in tools %}\n        {{- t | tojson(indent=4) }}\n        {{- \"\\n\\n\" }}\n    {%- endfor %}\n{%- endif %}\n{{- \"<|eot_id|>\" }}\n\n{%- for message in messages %}\n    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n    {%- elif 'tool_calls' in message %}\n        {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n        {%- if message['tool_calls'] %}\n            {{- \"[\" }}\n            {%- for tool_call_function in message.tool_calls %}\n                {%- set tool_call = tool_call_function.function %}\n                {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n                {{- '\"arguments\": ' }}\n                {{- tool_call.arguments | tojson }}\n                {{- \"}\" }}\n                {%- if not loop.last %}\n                    {{- \", \" }}\n                {%- endif %}\n            {%- endfor %}\n            {{- \"]\" }}\n            {{- \"<|eot_id|>\" }}\n        {%- elif message['content'] %}\n            {{- message['content'] | trim + '<|eot_id|>' }}\n        {%- else %}\n            {{- \"[]\\n\" + '<|eot_id|>' }}\n        {%- endif %}\n    {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n        {{- \"<|start_header_id|>\" + \"ipython\" + \"<|end_header_id|>\\n\\n\" }}\n        {%- set content = message[\"content\"] %}\n        {%- if content is mapping or (content is iterable and content is not string) %}\n            {{- content | tojson }}\n        {%- else %}\n            {{- content }}\n        {%- endif %}\n        {{- \"<|eot_id|>\" }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 16384,
+  "pad_token": "<|eot_id|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}

checkpoint-1434/trainer_state.json

@@ -0,0 +1,1058 @@
+{
+  "best_metric": 0.21637588739395142,
+  "best_model_checkpoint": "./xlam_Qlora_out_eval/checkpoint-1434",
+  "epoch": 0.9986941760250718,
+  "eval_steps": 478,
+  "global_step": 1434,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.006964394532950292,
+      "grad_norm": 0.29273906350135803,
+      "learning_rate": 1.5384615384615387e-06,
+      "loss": 1.8834,
+      "step": 10
+    },
+    {
+      "epoch": 0.013928789065900584,
+      "grad_norm": 0.3107926845550537,
+      "learning_rate": 3.0769230769230774e-06,
+      "loss": 1.876,
+      "step": 20
+    },
+    {
+      "epoch": 0.020893183598850874,
+      "grad_norm": 0.3098163902759552,
+      "learning_rate": 4.615384615384616e-06,
+      "loss": 1.9099,
+      "step": 30
+    },
+    {
+      "epoch": 0.027857578131801168,
+      "grad_norm": 0.36715859174728394,
+      "learning_rate": 6.153846153846155e-06,
+      "loss": 1.8142,
+      "step": 40
+    },
+    {
+      "epoch": 0.034821972664751455,
+      "grad_norm": 0.3511175215244293,
+      "learning_rate": 7.692307692307694e-06,
+      "loss": 1.7271,
+      "step": 50
+    },
+    {
+      "epoch": 0.04178636719770175,
+      "grad_norm": 0.3458055853843689,
+      "learning_rate": 9.230769230769232e-06,
+      "loss": 1.6073,
+      "step": 60
+    },
+    {
+      "epoch": 0.04875076173065204,
+      "grad_norm": 0.37719297409057617,
+      "learning_rate": 1.076923076923077e-05,
+      "loss": 1.4997,
+      "step": 70
+    },
+    {
+      "epoch": 0.055715156263602336,
+      "grad_norm": 0.30273741483688354,
+      "learning_rate": 1.230769230769231e-05,
+      "loss": 1.2914,
+      "step": 80
+    },
+    {
+      "epoch": 0.06267955079655263,
+      "grad_norm": 0.3690374195575714,
+      "learning_rate": 1.3846153846153847e-05,
+      "loss": 1.0086,
+      "step": 90
+    },
+    {
+      "epoch": 0.06964394532950291,
+      "grad_norm": 0.3111755847930908,
+      "learning_rate": 1.5384615384615387e-05,
+      "loss": 0.7469,
+      "step": 100
+    },
+    {
+      "epoch": 0.0766083398624532,
+      "grad_norm": 0.13778822124004364,
+      "learning_rate": 1.6923076923076924e-05,
+      "loss": 0.5848,
+      "step": 110
+    },
+    {
+      "epoch": 0.0835727343954035,
+      "grad_norm": 0.10629754513502121,
+      "learning_rate": 1.8461538461538465e-05,
+      "loss": 0.5366,
+      "step": 120
+    },
+    {
+      "epoch": 0.09053712892835379,
+      "grad_norm": 0.09681010991334915,
+      "learning_rate": 2e-05,
+      "loss": 0.5038,
+      "step": 130
+    },
+    {
+      "epoch": 0.09750152346130408,
+      "grad_norm": 0.10756956040859222,
+      "learning_rate": 1.9952095808383235e-05,
+      "loss": 0.5002,
+      "step": 140
+    },
+    {
+      "epoch": 0.10446591799425438,
+      "grad_norm": 0.10488647222518921,
+      "learning_rate": 1.9904191616766468e-05,
+      "loss": 0.5107,
+      "step": 150
+    },
+    {
+      "epoch": 0.11143031252720467,
+      "grad_norm": 0.09359394758939743,
+      "learning_rate": 1.98562874251497e-05,
+      "loss": 0.4604,
+      "step": 160
+    },
+    {
+      "epoch": 0.11839470706015495,
+      "grad_norm": 0.10392932593822479,
+      "learning_rate": 1.9808383233532935e-05,
+      "loss": 0.484,
+      "step": 170
+    },
+    {
+      "epoch": 0.12535910159310526,
+      "grad_norm": 0.12986360490322113,
+      "learning_rate": 1.9760479041916168e-05,
+      "loss": 0.4527,
+      "step": 180
+    },
+    {
+      "epoch": 0.13232349612605554,
+      "grad_norm": 0.14294420182704926,
+      "learning_rate": 1.97125748502994e-05,
+      "loss": 0.4661,
+      "step": 190
+    },
+    {
+      "epoch": 0.13928789065900582,
+      "grad_norm": 0.18483737111091614,
+      "learning_rate": 1.9664670658682635e-05,
+      "loss": 0.4395,
+      "step": 200
+    },
+    {
+      "epoch": 0.14625228519195613,
+      "grad_norm": 0.21114414930343628,
+      "learning_rate": 1.961676646706587e-05,
+      "loss": 0.4071,
+      "step": 210
+    },
+    {
+      "epoch": 0.1532166797249064,
+      "grad_norm": 0.21779198944568634,
+      "learning_rate": 1.95688622754491e-05,
+      "loss": 0.3846,
+      "step": 220
+    },
+    {
+      "epoch": 0.16018107425785671,
+      "grad_norm": 0.1294960230588913,
+      "learning_rate": 1.9520958083832338e-05,
+      "loss": 0.3952,
+      "step": 230
+    },
+    {
+      "epoch": 0.167145468790807,
+      "grad_norm": 0.0927898958325386,
+      "learning_rate": 1.9473053892215568e-05,
+      "loss": 0.4112,
+      "step": 240
+    },
+    {
+      "epoch": 0.1741098633237573,
+      "grad_norm": 0.10358591377735138,
+      "learning_rate": 1.9425149700598805e-05,
+      "loss": 0.406,
+      "step": 250
+    },
+    {
+      "epoch": 0.18107425785670758,
+      "grad_norm": 0.13144026696681976,
+      "learning_rate": 1.9377245508982038e-05,
+      "loss": 0.4058,
+      "step": 260
+    },
+    {
+      "epoch": 0.18803865238965786,
+      "grad_norm": 0.12745942175388336,
+      "learning_rate": 1.932934131736527e-05,
+      "loss": 0.3938,
+      "step": 270
+    },
+    {
+      "epoch": 0.19500304692260817,
+      "grad_norm": 0.09832354635000229,
+      "learning_rate": 1.9281437125748505e-05,
+      "loss": 0.3819,
+      "step": 280
+    },
+    {
+      "epoch": 0.20196744145555845,
+      "grad_norm": 0.0927552729845047,
+      "learning_rate": 1.9233532934131738e-05,
+      "loss": 0.3944,
+      "step": 290
+    },
+    {
+      "epoch": 0.20893183598850876,
+      "grad_norm": 0.09627388417720795,
+      "learning_rate": 1.918562874251497e-05,
+      "loss": 0.348,
+      "step": 300
+    },
+    {
+      "epoch": 0.21589623052145904,
+      "grad_norm": 0.09034324437379837,
+      "learning_rate": 1.9137724550898205e-05,
+      "loss": 0.3521,
+      "step": 310
+    },
+    {
+      "epoch": 0.22286062505440934,
+      "grad_norm": 0.09156423062086105,
+      "learning_rate": 1.9089820359281438e-05,
+      "loss": 0.3663,
+      "step": 320
+    },
+    {
+      "epoch": 0.22982501958735962,
+      "grad_norm": 0.0996803268790245,
+      "learning_rate": 1.904191616766467e-05,
+      "loss": 0.3347,
+      "step": 330
+    },
+    {
+      "epoch": 0.2367894141203099,
+      "grad_norm": 0.09962292015552521,
+      "learning_rate": 1.8994011976047905e-05,
+      "loss": 0.373,
+      "step": 340
+    },
+    {
+      "epoch": 0.2437538086532602,
+      "grad_norm": 0.08211114257574081,
+      "learning_rate": 1.894610778443114e-05,
+      "loss": 0.3641,
+      "step": 350
+    },
+    {
+      "epoch": 0.2507182031862105,
+      "grad_norm": 0.10449781268835068,
+      "learning_rate": 1.889820359281437e-05,
+      "loss": 0.3358,
+      "step": 360
+    },
+    {
+      "epoch": 0.2576825977191608,
+      "grad_norm": 0.11216866970062256,
+      "learning_rate": 1.8850299401197608e-05,
+      "loss": 0.372,
+      "step": 370
+    },
+    {
+      "epoch": 0.2646469922521111,
+      "grad_norm": 0.11125332862138748,
+      "learning_rate": 1.8802395209580838e-05,
+      "loss": 0.3521,
+      "step": 380
+    },
+    {
+      "epoch": 0.27161138678506136,
+      "grad_norm": 0.1620815545320511,
+      "learning_rate": 1.8754491017964075e-05,
+      "loss": 0.3726,
+      "step": 390
+    },
+    {
+      "epoch": 0.27857578131801164,
+      "grad_norm": 0.11439554393291473,
+      "learning_rate": 1.8706586826347305e-05,
+      "loss": 0.3287,
+      "step": 400
+    },
+    {
+      "epoch": 0.285540175850962,
+      "grad_norm": 0.10733096301555634,
+      "learning_rate": 1.865868263473054e-05,
+      "loss": 0.3459,
+      "step": 410
+    },
+    {
+      "epoch": 0.29250457038391225,
+      "grad_norm": 0.12426561117172241,
+      "learning_rate": 1.8610778443113775e-05,
+      "loss": 0.3705,
+      "step": 420
+    },
+    {
+      "epoch": 0.29946896491686253,
+      "grad_norm": 0.1148887649178505,
+      "learning_rate": 1.8562874251497008e-05,
+      "loss": 0.3216,
+      "step": 430
+    },
+    {
+      "epoch": 0.3064333594498128,
+      "grad_norm": 0.10842527449131012,
+      "learning_rate": 1.851497005988024e-05,
+      "loss": 0.3502,
+      "step": 440
+    },
+    {
+      "epoch": 0.31339775398276315,
+      "grad_norm": 0.11736515164375305,
+      "learning_rate": 1.8467065868263475e-05,
+      "loss": 0.3394,
+      "step": 450
+    },
+    {
+      "epoch": 0.32036214851571343,
+      "grad_norm": 0.1381833553314209,
+      "learning_rate": 1.8419161676646708e-05,
+      "loss": 0.3166,
+      "step": 460
+    },
+    {
+      "epoch": 0.3273265430486637,
+      "grad_norm": 0.10973569750785828,
+      "learning_rate": 1.837125748502994e-05,
+      "loss": 0.3245,
+      "step": 470
+    },
+    {
+      "epoch": 0.3328980586750239,
+      "eval_loss": 0.3075329661369324,
+      "eval_runtime": 356.0276,
+      "eval_samples_per_second": 3.396,
+      "eval_steps_per_second": 1.699,
+      "step": 478
+    },
+    {
+      "epoch": 0.334290937581614,
+      "grad_norm": 0.13474580645561218,
+      "learning_rate": 1.8323353293413175e-05,
+      "loss": 0.2914,
+      "step": 480
+    },
+    {
+      "epoch": 0.34125533211456427,
+      "grad_norm": 0.12812362611293793,
+      "learning_rate": 1.827544910179641e-05,
+      "loss": 0.3327,
+      "step": 490
+    },
+    {
+      "epoch": 0.3482197266475146,
+      "grad_norm": 0.11661785840988159,
+      "learning_rate": 1.822754491017964e-05,
+      "loss": 0.3468,
+      "step": 500
+    },
+    {
+      "epoch": 0.3551841211804649,
+      "grad_norm": 0.12990142405033112,
+      "learning_rate": 1.8179640718562878e-05,
+      "loss": 0.3318,
+      "step": 510
+    },
+    {
+      "epoch": 0.36214851571341516,
+      "grad_norm": 0.1309327632188797,
+      "learning_rate": 1.8131736526946108e-05,
+      "loss": 0.2942,
+      "step": 520
+    },
+    {
+      "epoch": 0.36911291024636544,
+      "grad_norm": 0.12246479839086533,
+      "learning_rate": 1.8083832335329345e-05,
+      "loss": 0.3751,
+      "step": 530
+    },
+    {
+      "epoch": 0.3760773047793157,
+      "grad_norm": 0.14134123921394348,
+      "learning_rate": 1.8035928143712575e-05,
+      "loss": 0.3248,
+      "step": 540
+    },
+    {
+      "epoch": 0.38304169931226606,
+      "grad_norm": 0.1433170884847641,
+      "learning_rate": 1.798802395209581e-05,
+      "loss": 0.3344,
+      "step": 550
+    },
+    {
+      "epoch": 0.39000609384521634,
+      "grad_norm": 0.13364745676517487,
+      "learning_rate": 1.7940119760479045e-05,
+      "loss": 0.2928,
+      "step": 560
+    },
+    {
+      "epoch": 0.3969704883781666,
+      "grad_norm": 0.13446496427059174,
+      "learning_rate": 1.7892215568862278e-05,
+      "loss": 0.3421,
+      "step": 570
+    },
+    {
+      "epoch": 0.4039348829111169,
+      "grad_norm": 0.1586250364780426,
+      "learning_rate": 1.784431137724551e-05,
+      "loss": 0.3099,
+      "step": 580
+    },
+    {
+      "epoch": 0.4108992774440672,
+      "grad_norm": 0.1283930391073227,
+      "learning_rate": 1.7796407185628745e-05,
+      "loss": 0.3123,
+      "step": 590
+    },
+    {
+      "epoch": 0.4178636719770175,
+      "grad_norm": 0.14261852204799652,
+      "learning_rate": 1.7748502994011978e-05,
+      "loss": 0.2886,
+      "step": 600
+    },
+    {
+      "epoch": 0.4248280665099678,
+      "grad_norm": 0.13329675793647766,
+      "learning_rate": 1.770059880239521e-05,
+      "loss": 0.3255,
+      "step": 610
+    },
+    {
+      "epoch": 0.4317924610429181,
+      "grad_norm": 0.13880160450935364,
+      "learning_rate": 1.7652694610778445e-05,
+      "loss": 0.326,
+      "step": 620
+    },
+    {
+      "epoch": 0.43875685557586835,
+      "grad_norm": 0.14388258755207062,
+      "learning_rate": 1.7604790419161678e-05,
+      "loss": 0.3269,
+      "step": 630
+    },
+    {
+      "epoch": 0.4457212501088187,
+      "grad_norm": 0.1571163535118103,
+      "learning_rate": 1.755688622754491e-05,
+      "loss": 0.2919,
+      "step": 640
+    },
+    {
+      "epoch": 0.45268564464176897,
+      "grad_norm": 0.15307635068893433,
+      "learning_rate": 1.7508982035928145e-05,
+      "loss": 0.2773,
+      "step": 650
+    },
+    {
+      "epoch": 0.45965003917471925,
+      "grad_norm": 0.1438165158033371,
+      "learning_rate": 1.7461077844311378e-05,
+      "loss": 0.2847,
+      "step": 660
+    },
+    {
+      "epoch": 0.46661443370766953,
+      "grad_norm": 0.15620216727256775,
+      "learning_rate": 1.741317365269461e-05,
+      "loss": 0.29,
+      "step": 670
+    },
+    {
+      "epoch": 0.4735788282406198,
+      "grad_norm": 0.1414933055639267,
+      "learning_rate": 1.7365269461077845e-05,
+      "loss": 0.2754,
+      "step": 680
+    },
+    {
+      "epoch": 0.48054322277357014,
+      "grad_norm": 0.15853798389434814,
+      "learning_rate": 1.7317365269461078e-05,
+      "loss": 0.3062,
+      "step": 690
+    },
+    {
+      "epoch": 0.4875076173065204,
+      "grad_norm": 0.17159980535507202,
+      "learning_rate": 1.726946107784431e-05,
+      "loss": 0.2953,
+      "step": 700
+    },
+    {
+      "epoch": 0.4944720118394707,
+      "grad_norm": 0.16937299072742462,
+      "learning_rate": 1.7221556886227548e-05,
+      "loss": 0.2895,
+      "step": 710
+    },
+    {
+      "epoch": 0.501436406372421,
+      "grad_norm": 0.1455833464860916,
+      "learning_rate": 1.717365269461078e-05,
+      "loss": 0.2912,
+      "step": 720
+    },
+    {
+      "epoch": 0.5084008009053713,
+      "grad_norm": 0.15824872255325317,
+      "learning_rate": 1.7125748502994015e-05,
+      "loss": 0.3012,
+      "step": 730
+    },
+    {
+      "epoch": 0.5153651954383216,
+      "grad_norm": 0.1499851644039154,
+      "learning_rate": 1.7077844311377248e-05,
+      "loss": 0.263,
+      "step": 740
+    },
+    {
+      "epoch": 0.5223295899712719,
+      "grad_norm": 0.1611323356628418,
+      "learning_rate": 1.702994011976048e-05,
+      "loss": 0.2595,
+      "step": 750
+    },
+    {
+      "epoch": 0.5292939845042222,
+      "grad_norm": 0.16144199669361115,
+      "learning_rate": 1.6982035928143714e-05,
+      "loss": 0.2939,
+      "step": 760
+    },
+    {
+      "epoch": 0.5362583790371724,
+      "grad_norm": 0.13433364033699036,
+      "learning_rate": 1.6934131736526948e-05,
+      "loss": 0.2817,
+      "step": 770
+    },
+    {
+      "epoch": 0.5432227735701227,
+      "grad_norm": 0.15188777446746826,
+      "learning_rate": 1.688622754491018e-05,
+      "loss": 0.2804,
+      "step": 780
+    },
+    {
+      "epoch": 0.550187168103073,
+      "grad_norm": 0.1530751883983612,
+      "learning_rate": 1.6838323353293414e-05,
+      "loss": 0.298,
+      "step": 790
+    },
+    {
+      "epoch": 0.5571515626360233,
+      "grad_norm": 0.14582635462284088,
+      "learning_rate": 1.6790419161676648e-05,
+      "loss": 0.3177,
+      "step": 800
+    },
+    {
+      "epoch": 0.5641159571689737,
+      "grad_norm": 0.174430251121521,
+      "learning_rate": 1.674251497005988e-05,
+      "loss": 0.2624,
+      "step": 810
+    },
+    {
+      "epoch": 0.571080351701924,
+      "grad_norm": 0.1558169573545456,
+      "learning_rate": 1.6694610778443114e-05,
+      "loss": 0.2631,
+      "step": 820
+    },
+    {
+      "epoch": 0.5780447462348742,
+      "grad_norm": 0.17108450829982758,
+      "learning_rate": 1.6646706586826348e-05,
+      "loss": 0.2903,
+      "step": 830
+    },
+    {
+      "epoch": 0.5850091407678245,
+      "grad_norm": 0.16796821355819702,
+      "learning_rate": 1.659880239520958e-05,
+      "loss": 0.2928,
+      "step": 840
+    },
+    {
+      "epoch": 0.5919735353007748,
+      "grad_norm": 0.16556940972805023,
+      "learning_rate": 1.6550898203592814e-05,
+      "loss": 0.2882,
+      "step": 850
+    },
+    {
+      "epoch": 0.5989379298337251,
+      "grad_norm": 0.1716109961271286,
+      "learning_rate": 1.650299401197605e-05,
+      "loss": 0.2814,
+      "step": 860
+    },
+    {
+      "epoch": 0.6059023243666753,
+      "grad_norm": 0.1695253700017929,
+      "learning_rate": 1.645508982035928e-05,
+      "loss": 0.2909,
+      "step": 870
+    },
+    {
+      "epoch": 0.6128667188996256,
+      "grad_norm": 0.1896178424358368,
+      "learning_rate": 1.6407185628742518e-05,
+      "loss": 0.2909,
+      "step": 880
+    },
+    {
+      "epoch": 0.6198311134325759,
+      "grad_norm": 0.1624131202697754,
+      "learning_rate": 1.6359281437125748e-05,
+      "loss": 0.2672,
+      "step": 890
+    },
+    {
+      "epoch": 0.6267955079655263,
+      "grad_norm": 0.16821731626987457,
+      "learning_rate": 1.6311377245508984e-05,
+      "loss": 0.2626,
+      "step": 900
+    },
+    {
+      "epoch": 0.6337599024984766,
+      "grad_norm": 0.17026785016059875,
+      "learning_rate": 1.6263473053892214e-05,
+      "loss": 0.2735,
+      "step": 910
+    },
+    {
+      "epoch": 0.6407242970314269,
+      "grad_norm": 0.19429761171340942,
+      "learning_rate": 1.621556886227545e-05,
+      "loss": 0.2605,
+      "step": 920
+    },
+    {
+      "epoch": 0.6476886915643771,
+      "grad_norm": 0.16931040585041046,
+      "learning_rate": 1.6167664670658684e-05,
+      "loss": 0.2722,
+      "step": 930
+    },
+    {
+      "epoch": 0.6546530860973274,
+      "grad_norm": 0.20588335394859314,
+      "learning_rate": 1.6119760479041918e-05,
+      "loss": 0.301,
+      "step": 940
+    },
+    {
+      "epoch": 0.6616174806302777,
+      "grad_norm": 0.16699948906898499,
+      "learning_rate": 1.607185628742515e-05,
+      "loss": 0.2646,
+      "step": 950
+    },
+    {
+      "epoch": 0.6657961173500478,
+      "eval_loss": 0.2465677410364151,
+      "eval_runtime": 356.1016,
+      "eval_samples_per_second": 3.395,
+      "eval_steps_per_second": 1.699,
+      "step": 956
+    },
+    {
+      "epoch": 0.668581875163228,
+      "grad_norm": 0.20734217762947083,
+      "learning_rate": 1.6023952095808384e-05,
+      "loss": 0.2476,
+      "step": 960
+    },
+    {
+      "epoch": 0.6755462696961783,
+      "grad_norm": 0.21317210793495178,
+      "learning_rate": 1.5976047904191618e-05,
+      "loss": 0.2434,
+      "step": 970
+    },
+    {
+      "epoch": 0.6825106642291285,
+      "grad_norm": 0.19125515222549438,
+      "learning_rate": 1.592814371257485e-05,
+      "loss": 0.2823,
+      "step": 980
+    },
+    {
+      "epoch": 0.6894750587620788,
+      "grad_norm": 0.1651381254196167,
+      "learning_rate": 1.5880239520958084e-05,
+      "loss": 0.2769,
+      "step": 990
+    },
+    {
+      "epoch": 0.6964394532950292,
+      "grad_norm": 0.18061324954032898,
+      "learning_rate": 1.583233532934132e-05,
+      "loss": 0.2788,
+      "step": 1000
+    },
+    {
+      "epoch": 0.7034038478279795,
+      "grad_norm": 0.17788180708885193,
+      "learning_rate": 1.578443113772455e-05,
+      "loss": 0.2596,
+      "step": 1010
+    },
+    {
+      "epoch": 0.7103682423609298,
+      "grad_norm": 0.19635842740535736,
+      "learning_rate": 1.5736526946107788e-05,
+      "loss": 0.2391,
+      "step": 1020
+    },
+    {
+      "epoch": 0.71733263689388,
+      "grad_norm": 0.18000468611717224,
+      "learning_rate": 1.5688622754491018e-05,
+      "loss": 0.2589,
+      "step": 1030
+    },
+    {
+      "epoch": 0.7242970314268303,
+      "grad_norm": 0.18016283214092255,
+      "learning_rate": 1.5640718562874254e-05,
+      "loss": 0.2673,
+      "step": 1040
+    },
+    {
+      "epoch": 0.7312614259597806,
+      "grad_norm": 0.2085445076227188,
+      "learning_rate": 1.5592814371257484e-05,
+      "loss": 0.295,
+      "step": 1050
+    },
+    {
+      "epoch": 0.7382258204927309,
+      "grad_norm": 0.1937682181596756,
+      "learning_rate": 1.554491017964072e-05,
+      "loss": 0.2741,
+      "step": 1060
+    },
+    {
+      "epoch": 0.7451902150256812,
+      "grad_norm": 0.18496188521385193,
+      "learning_rate": 1.5497005988023954e-05,
+      "loss": 0.2485,
+      "step": 1070
+    },
+    {
+      "epoch": 0.7521546095586314,
+      "grad_norm": 0.21024686098098755,
+      "learning_rate": 1.5449101796407188e-05,
+      "loss": 0.2569,
+      "step": 1080
+    },
+    {
+      "epoch": 0.7591190040915818,
+      "grad_norm": 0.1960270255804062,
+      "learning_rate": 1.540119760479042e-05,
+      "loss": 0.2335,
+      "step": 1090
+    },
+    {
+      "epoch": 0.7660833986245321,
+      "grad_norm": 0.19557876884937286,
+      "learning_rate": 1.5353293413173654e-05,
+      "loss": 0.279,
+      "step": 1100
+    },
+    {
+      "epoch": 0.7730477931574824,
+      "grad_norm": 0.17300210893154144,
+      "learning_rate": 1.5305389221556888e-05,
+      "loss": 0.249,
+      "step": 1110
+    },
+    {
+      "epoch": 0.7800121876904327,
+      "grad_norm": 0.1768539994955063,
+      "learning_rate": 1.5257485029940121e-05,
+      "loss": 0.2415,
+      "step": 1120
+    },
+    {
+      "epoch": 0.786976582223383,
+      "grad_norm": 0.20632915198802948,
+      "learning_rate": 1.5209580838323354e-05,
+      "loss": 0.2312,
+      "step": 1130
+    },
+    {
+      "epoch": 0.7939409767563332,
+      "grad_norm": 0.1948348730802536,
+      "learning_rate": 1.516167664670659e-05,
+      "loss": 0.2415,
+      "step": 1140
+    },
+    {
+      "epoch": 0.8009053712892835,
+      "grad_norm": 0.16288821399211884,
+      "learning_rate": 1.511377245508982e-05,
+      "loss": 0.2571,
+      "step": 1150
+    },
+    {
+      "epoch": 0.8078697658222338,
+      "grad_norm": 0.21365194022655487,
+      "learning_rate": 1.5065868263473056e-05,
+      "loss": 0.2594,
+      "step": 1160
+    },
+    {
+      "epoch": 0.8148341603551841,
+      "grad_norm": 0.20960865914821625,
+      "learning_rate": 1.5017964071856287e-05,
+      "loss": 0.2537,
+      "step": 1170
+    },
+    {
+      "epoch": 0.8217985548881344,
+      "grad_norm": 0.20458345115184784,
+      "learning_rate": 1.4970059880239522e-05,
+      "loss": 0.2675,
+      "step": 1180
+    },
+    {
+      "epoch": 0.8287629494210847,
+      "grad_norm": 0.22388549149036407,
+      "learning_rate": 1.4922155688622754e-05,
+      "loss": 0.2525,
+      "step": 1190
+    },
+    {
+      "epoch": 0.835727343954035,
+      "grad_norm": 0.2139691859483719,
+      "learning_rate": 1.4874251497005989e-05,
+      "loss": 0.251,
+      "step": 1200
+    },
+    {
+      "epoch": 0.8426917384869853,
+      "grad_norm": 0.18761980533599854,
+      "learning_rate": 1.4826347305389224e-05,
+      "loss": 0.2328,
+      "step": 1210
+    },
+    {
+      "epoch": 0.8496561330199356,
+      "grad_norm": 0.183973029255867,
+      "learning_rate": 1.4778443113772456e-05,
+      "loss": 0.2239,
+      "step": 1220
+    },
+    {
+      "epoch": 0.8566205275528859,
+      "grad_norm": 0.23718492686748505,
+      "learning_rate": 1.473053892215569e-05,
+      "loss": 0.2423,
+      "step": 1230
+    },
+    {
+      "epoch": 0.8635849220858361,
+      "grad_norm": 0.23875045776367188,
+      "learning_rate": 1.4682634730538922e-05,
+      "loss": 0.2365,
+      "step": 1240
+    },
+    {
+      "epoch": 0.8705493166187864,
+      "grad_norm": 0.21570812165737152,
+      "learning_rate": 1.4634730538922157e-05,
+      "loss": 0.268,
+      "step": 1250
+    },
+    {
+      "epoch": 0.8775137111517367,
+      "grad_norm": 0.20839928090572357,
+      "learning_rate": 1.4586826347305389e-05,
+      "loss": 0.2502,
+      "step": 1260
+    },
+    {
+      "epoch": 0.884478105684687,
+      "grad_norm": 0.21605758368968964,
+      "learning_rate": 1.4538922155688624e-05,
+      "loss": 0.2365,
+      "step": 1270
+    },
+    {
+      "epoch": 0.8914425002176374,
+      "grad_norm": 0.23300880193710327,
+      "learning_rate": 1.4491017964071859e-05,
+      "loss": 0.2404,
+      "step": 1280
+    },
+    {
+      "epoch": 0.8984068947505877,
+      "grad_norm": 0.21796733140945435,
+      "learning_rate": 1.444311377245509e-05,
+      "loss": 0.2704,
+      "step": 1290
+    },
+    {
+      "epoch": 0.9053712892835379,
+      "grad_norm": 0.25555598735809326,
+      "learning_rate": 1.4395209580838326e-05,
+      "loss": 0.2674,
+      "step": 1300
+    },
+    {
+      "epoch": 0.9123356838164882,
+      "grad_norm": 0.22284413874149323,
+      "learning_rate": 1.4347305389221557e-05,
+      "loss": 0.2471,
+      "step": 1310
+    },
+    {
+      "epoch": 0.9193000783494385,
+      "grad_norm": 0.22234566509723663,
+      "learning_rate": 1.4299401197604792e-05,
+      "loss": 0.2529,
+      "step": 1320
+    },
+    {
+      "epoch": 0.9262644728823888,
+      "grad_norm": 0.221883624792099,
+      "learning_rate": 1.4251497005988024e-05,
+      "loss": 0.1932,
+      "step": 1330
+    },
+    {
+      "epoch": 0.9332288674153391,
+      "grad_norm": 0.22873520851135254,
+      "learning_rate": 1.4203592814371259e-05,
+      "loss": 0.2412,
+      "step": 1340
+    },
+    {
+      "epoch": 0.9401932619482893,
+      "grad_norm": 0.21307863295078278,
+      "learning_rate": 1.4155688622754492e-05,
+      "loss": 0.2384,
+      "step": 1350
+    },
+    {
+      "epoch": 0.9471576564812396,
+      "grad_norm": 0.21296364068984985,
+      "learning_rate": 1.4107784431137726e-05,
+      "loss": 0.2318,
+      "step": 1360
+    },
+    {
+      "epoch": 0.95412205101419,
+      "grad_norm": 0.21060685813426971,
+      "learning_rate": 1.405988023952096e-05,
+      "loss": 0.2323,
+      "step": 1370
+    },
+    {
+      "epoch": 0.9610864455471403,
+      "grad_norm": 0.23930418491363525,
+      "learning_rate": 1.4011976047904192e-05,
+      "loss": 0.2398,
+      "step": 1380
+    },
+    {
+      "epoch": 0.9680508400800906,
+      "grad_norm": 0.22837185859680176,
+      "learning_rate": 1.3964071856287427e-05,
+      "loss": 0.233,
+      "step": 1390
+    },
+    {
+      "epoch": 0.9750152346130408,
+      "grad_norm": 0.21798169612884521,
+      "learning_rate": 1.3916167664670659e-05,
+      "loss": 0.2161,
+      "step": 1400
+    },
+    {
+      "epoch": 0.9819796291459911,
+      "grad_norm": 0.2846814692020416,
+      "learning_rate": 1.3868263473053894e-05,
+      "loss": 0.2301,
+      "step": 1410
+    },
+    {
+      "epoch": 0.9889440236789414,
+      "grad_norm": 0.22030217945575714,
+      "learning_rate": 1.3820359281437127e-05,
+      "loss": 0.2017,
+      "step": 1420
+    },
+    {
+      "epoch": 0.9959084182118917,
+      "grad_norm": 0.25554022192955017,
+      "learning_rate": 1.377245508982036e-05,
+      "loss": 0.2296,
+      "step": 1430
+    },
+    {
+      "epoch": 0.9986941760250718,
+      "eval_loss": 0.21637588739395142,
+      "eval_runtime": 356.0721,
+      "eval_samples_per_second": 3.395,
+      "eval_steps_per_second": 1.699,
+      "step": 1434
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 4305,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 478,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.68888676426793e+17,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1434/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:956bd3fe04334201160cff0d2e3faf62e32c6c8f709339d63e01a7d4b73b24d0
+size 5560

checkpoint-1912/README.md

@@ -0,0 +1,209 @@
+---
+base_model: Salesforce/Llama-xLAM-2-8b-fc-r
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:Salesforce/Llama-xLAM-2-8b-fc-r
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

checkpoint-1912/adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Salesforce/Llama-xLAM-2-8b-fc-r",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "o_proj",
+    "down_proj",
+    "k_proj",
+    "up_proj",
+    "v_proj",
+    "gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-1912/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6ea3d29d25191471cf841402ee92d3a3e6e8703263d92a49b5cf251f1843ee8
+size 671149168

checkpoint-1912/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:600dfa6960d41b89b6a1d41f9b3bd5b791aba4b0e6554bb90e731934413cf6d7
+size 1342555602

checkpoint-1912/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f2b1a8f992a3995f905827a704361cb8085a3ed3a353f94109da49c5641e67c
+size 14244

checkpoint-1912/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd3c490c908becf27d1f2471f166f418ed21eeda9029b219116d716d1a232247
+size 1064

checkpoint-1912/special_tokens_map.json

@@ -0,0 +1,27 @@
+{
+  "additional_special_tokens": [
+    "<|eot_id|>",
+    "<|eom_id|>"
+  ],
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-1912/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a55c1a4c5e3af7f2fb2bc0cd245a09dabd742dc24e7cb3741db1e11c7fe1a52
+size 17210019

checkpoint-1912/tokenizer_config.json

@@ -0,0 +1,2070 @@
+{
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|finetune_right_pad_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|eom_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|python_tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|eot_id|>",
+    "<|eom_id|>"
+  ],
+  "bos_token": "<|begin_of_text|>",
+  "chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n    {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n    {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not tools is defined %}\n    {%- set tools = none %}\n{%- endif %}\n\n{#- Extract system message #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if messages[0]['role'] == 'system' %}\n    {%- set system_message = messages[0]['content'] | trim %}\n    {%- set messages = messages[1:] %}\n    {{- system_message + \"\\n\" }}\n{%- else %}\n    {%- set system_message = \"You are a helpful assistant that can use tools. You are developed by Salesforce xLAM team.\" %}\n    {% set format_instruction %}You have access to a set of tools. When using tools, make calls in a single JSON array: \n\n[{\"name\": \"tool_call_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}}, ... (additional parallel tool calls as needed)]\n\nIf no tool is suitable, state that explicitly. If the user's input lacks required parameters, ask for clarification. Do not interpret or respond until tool results are returned. Once they are available, process them or make additional calls if needed. For tasks that don't require tools, such as casual conversation or general advice, respond directly in plain text. The available tools are:{% endset %}\n    {{- system_message + \"\\n\" }}\n    {%- if tools is not none %}\n        {{- format_instruction + \"\\n\\n\" }}\n    {%- endif %}\n{%- endif %}\n\n\n{%- if tools is not none %}\n    {%- for t in tools %}\n        {{- t | tojson(indent=4) }}\n        {{- \"\\n\\n\" }}\n    {%- endfor %}\n{%- endif %}\n{{- \"<|eot_id|>\" }}\n\n{%- for message in messages %}\n    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n    {%- elif 'tool_calls' in message %}\n        {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n        {%- if message['tool_calls'] %}\n            {{- \"[\" }}\n            {%- for tool_call_function in message.tool_calls %}\n                {%- set tool_call = tool_call_function.function %}\n                {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n                {{- '\"arguments\": ' }}\n                {{- tool_call.arguments | tojson }}\n                {{- \"}\" }}\n                {%- if not loop.last %}\n                    {{- \", \" }}\n                {%- endif %}\n            {%- endfor %}\n            {{- \"]\" }}\n            {{- \"<|eot_id|>\" }}\n        {%- elif message['content'] %}\n            {{- message['content'] | trim + '<|eot_id|>' }}\n        {%- else %}\n            {{- \"[]\\n\" + '<|eot_id|>' }}\n        {%- endif %}\n    {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n        {{- \"<|start_header_id|>\" + \"ipython\" + \"<|end_header_id|>\\n\\n\" }}\n        {%- set content = message[\"content\"] %}\n        {%- if content is mapping or (content is iterable and content is not string) %}\n            {{- content | tojson }}\n        {%- else %}\n            {{- content }}\n        {%- endif %}\n        {{- \"<|eot_id|>\" }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 16384,
+  "pad_token": "<|eot_id|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}

checkpoint-1912/trainer_state.json

@@ -0,0 +1,1402 @@
+{
+  "best_metric": 0.19295775890350342,
+  "best_model_checkpoint": "./xlam_Qlora_out_eval/checkpoint-1912",
+  "epoch": 1.3317663445634196,
+  "eval_steps": 478,
+  "global_step": 1912,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.006964394532950292,
+      "grad_norm": 0.29273906350135803,
+      "learning_rate": 1.5384615384615387e-06,
+      "loss": 1.8834,
+      "step": 10
+    },
+    {
+      "epoch": 0.013928789065900584,
+      "grad_norm": 0.3107926845550537,
+      "learning_rate": 3.0769230769230774e-06,
+      "loss": 1.876,
+      "step": 20
+    },
+    {
+      "epoch": 0.020893183598850874,
+      "grad_norm": 0.3098163902759552,
+      "learning_rate": 4.615384615384616e-06,
+      "loss": 1.9099,
+      "step": 30
+    },
+    {
+      "epoch": 0.027857578131801168,
+      "grad_norm": 0.36715859174728394,
+      "learning_rate": 6.153846153846155e-06,
+      "loss": 1.8142,
+      "step": 40
+    },
+    {
+      "epoch": 0.034821972664751455,
+      "grad_norm": 0.3511175215244293,
+      "learning_rate": 7.692307692307694e-06,
+      "loss": 1.7271,
+      "step": 50
+    },
+    {
+      "epoch": 0.04178636719770175,
+      "grad_norm": 0.3458055853843689,
+      "learning_rate": 9.230769230769232e-06,
+      "loss": 1.6073,
+      "step": 60
+    },
+    {
+      "epoch": 0.04875076173065204,
+      "grad_norm": 0.37719297409057617,
+      "learning_rate": 1.076923076923077e-05,
+      "loss": 1.4997,
+      "step": 70
+    },
+    {
+      "epoch": 0.055715156263602336,
+      "grad_norm": 0.30273741483688354,
+      "learning_rate": 1.230769230769231e-05,
+      "loss": 1.2914,
+      "step": 80
+    },
+    {
+      "epoch": 0.06267955079655263,
+      "grad_norm": 0.3690374195575714,
+      "learning_rate": 1.3846153846153847e-05,
+      "loss": 1.0086,
+      "step": 90
+    },
+    {
+      "epoch": 0.06964394532950291,
+      "grad_norm": 0.3111755847930908,
+      "learning_rate": 1.5384615384615387e-05,
+      "loss": 0.7469,
+      "step": 100
+    },
+    {
+      "epoch": 0.0766083398624532,
+      "grad_norm": 0.13778822124004364,
+      "learning_rate": 1.6923076923076924e-05,
+      "loss": 0.5848,
+      "step": 110
+    },
+    {
+      "epoch": 0.0835727343954035,
+      "grad_norm": 0.10629754513502121,
+      "learning_rate": 1.8461538461538465e-05,
+      "loss": 0.5366,
+      "step": 120
+    },
+    {
+      "epoch": 0.09053712892835379,
+      "grad_norm": 0.09681010991334915,
+      "learning_rate": 2e-05,
+      "loss": 0.5038,
+      "step": 130
+    },
+    {
+      "epoch": 0.09750152346130408,
+      "grad_norm": 0.10756956040859222,
+      "learning_rate": 1.9952095808383235e-05,
+      "loss": 0.5002,
+      "step": 140
+    },
+    {
+      "epoch": 0.10446591799425438,
+      "grad_norm": 0.10488647222518921,
+      "learning_rate": 1.9904191616766468e-05,
+      "loss": 0.5107,
+      "step": 150
+    },
+    {
+      "epoch": 0.11143031252720467,
+      "grad_norm": 0.09359394758939743,
+      "learning_rate": 1.98562874251497e-05,
+      "loss": 0.4604,
+      "step": 160
+    },
+    {
+      "epoch": 0.11839470706015495,
+      "grad_norm": 0.10392932593822479,
+      "learning_rate": 1.9808383233532935e-05,
+      "loss": 0.484,
+      "step": 170
+    },
+    {
+      "epoch": 0.12535910159310526,
+      "grad_norm": 0.12986360490322113,
+      "learning_rate": 1.9760479041916168e-05,
+      "loss": 0.4527,
+      "step": 180
+    },
+    {
+      "epoch": 0.13232349612605554,
+      "grad_norm": 0.14294420182704926,
+      "learning_rate": 1.97125748502994e-05,
+      "loss": 0.4661,
+      "step": 190
+    },
+    {
+      "epoch": 0.13928789065900582,
+      "grad_norm": 0.18483737111091614,
+      "learning_rate": 1.9664670658682635e-05,
+      "loss": 0.4395,
+      "step": 200
+    },
+    {
+      "epoch": 0.14625228519195613,
+      "grad_norm": 0.21114414930343628,
+      "learning_rate": 1.961676646706587e-05,
+      "loss": 0.4071,
+      "step": 210
+    },
+    {
+      "epoch": 0.1532166797249064,
+      "grad_norm": 0.21779198944568634,
+      "learning_rate": 1.95688622754491e-05,
+      "loss": 0.3846,
+      "step": 220
+    },
+    {
+      "epoch": 0.16018107425785671,
+      "grad_norm": 0.1294960230588913,
+      "learning_rate": 1.9520958083832338e-05,
+      "loss": 0.3952,
+      "step": 230
+    },
+    {
+      "epoch": 0.167145468790807,
+      "grad_norm": 0.0927898958325386,
+      "learning_rate": 1.9473053892215568e-05,
+      "loss": 0.4112,
+      "step": 240
+    },
+    {
+      "epoch": 0.1741098633237573,
+      "grad_norm": 0.10358591377735138,
+      "learning_rate": 1.9425149700598805e-05,
+      "loss": 0.406,
+      "step": 250
+    },
+    {
+      "epoch": 0.18107425785670758,
+      "grad_norm": 0.13144026696681976,
+      "learning_rate": 1.9377245508982038e-05,
+      "loss": 0.4058,
+      "step": 260
+    },
+    {
+      "epoch": 0.18803865238965786,
+      "grad_norm": 0.12745942175388336,
+      "learning_rate": 1.932934131736527e-05,
+      "loss": 0.3938,
+      "step": 270
+    },
+    {
+      "epoch": 0.19500304692260817,
+      "grad_norm": 0.09832354635000229,
+      "learning_rate": 1.9281437125748505e-05,
+      "loss": 0.3819,
+      "step": 280
+    },
+    {
+      "epoch": 0.20196744145555845,
+      "grad_norm": 0.0927552729845047,
+      "learning_rate": 1.9233532934131738e-05,
+      "loss": 0.3944,
+      "step": 290
+    },
+    {
+      "epoch": 0.20893183598850876,
+      "grad_norm": 0.09627388417720795,
+      "learning_rate": 1.918562874251497e-05,
+      "loss": 0.348,
+      "step": 300
+    },
+    {
+      "epoch": 0.21589623052145904,
+      "grad_norm": 0.09034324437379837,
+      "learning_rate": 1.9137724550898205e-05,
+      "loss": 0.3521,
+      "step": 310
+    },
+    {
+      "epoch": 0.22286062505440934,
+      "grad_norm": 0.09156423062086105,
+      "learning_rate": 1.9089820359281438e-05,
+      "loss": 0.3663,
+      "step": 320
+    },
+    {
+      "epoch": 0.22982501958735962,
+      "grad_norm": 0.0996803268790245,
+      "learning_rate": 1.904191616766467e-05,
+      "loss": 0.3347,
+      "step": 330
+    },
+    {
+      "epoch": 0.2367894141203099,
+      "grad_norm": 0.09962292015552521,
+      "learning_rate": 1.8994011976047905e-05,
+      "loss": 0.373,
+      "step": 340
+    },
+    {
+      "epoch": 0.2437538086532602,
+      "grad_norm": 0.08211114257574081,
+      "learning_rate": 1.894610778443114e-05,
+      "loss": 0.3641,
+      "step": 350
+    },
+    {
+      "epoch": 0.2507182031862105,
+      "grad_norm": 0.10449781268835068,
+      "learning_rate": 1.889820359281437e-05,
+      "loss": 0.3358,
+      "step": 360
+    },
+    {
+      "epoch": 0.2576825977191608,
+      "grad_norm": 0.11216866970062256,
+      "learning_rate": 1.8850299401197608e-05,
+      "loss": 0.372,
+      "step": 370
+    },
+    {
+      "epoch": 0.2646469922521111,
+      "grad_norm": 0.11125332862138748,
+      "learning_rate": 1.8802395209580838e-05,
+      "loss": 0.3521,
+      "step": 380
+    },
+    {
+      "epoch": 0.27161138678506136,
+      "grad_norm": 0.1620815545320511,
+      "learning_rate": 1.8754491017964075e-05,
+      "loss": 0.3726,
+      "step": 390
+    },
+    {
+      "epoch": 0.27857578131801164,
+      "grad_norm": 0.11439554393291473,
+      "learning_rate": 1.8706586826347305e-05,
+      "loss": 0.3287,
+      "step": 400
+    },
+    {
+      "epoch": 0.285540175850962,
+      "grad_norm": 0.10733096301555634,
+      "learning_rate": 1.865868263473054e-05,
+      "loss": 0.3459,
+      "step": 410
+    },
+    {
+      "epoch": 0.29250457038391225,
+      "grad_norm": 0.12426561117172241,
+      "learning_rate": 1.8610778443113775e-05,
+      "loss": 0.3705,
+      "step": 420
+    },
+    {
+      "epoch": 0.29946896491686253,
+      "grad_norm": 0.1148887649178505,
+      "learning_rate": 1.8562874251497008e-05,
+      "loss": 0.3216,
+      "step": 430
+    },
+    {
+      "epoch": 0.3064333594498128,
+      "grad_norm": 0.10842527449131012,
+      "learning_rate": 1.851497005988024e-05,
+      "loss": 0.3502,
+      "step": 440
+    },
+    {
+      "epoch": 0.31339775398276315,
+      "grad_norm": 0.11736515164375305,
+      "learning_rate": 1.8467065868263475e-05,
+      "loss": 0.3394,
+      "step": 450
+    },
+    {
+      "epoch": 0.32036214851571343,
+      "grad_norm": 0.1381833553314209,
+      "learning_rate": 1.8419161676646708e-05,
+      "loss": 0.3166,
+      "step": 460
+    },
+    {
+      "epoch": 0.3273265430486637,
+      "grad_norm": 0.10973569750785828,
+      "learning_rate": 1.837125748502994e-05,
+      "loss": 0.3245,
+      "step": 470
+    },
+    {
+      "epoch": 0.3328980586750239,
+      "eval_loss": 0.3075329661369324,
+      "eval_runtime": 356.0276,
+      "eval_samples_per_second": 3.396,
+      "eval_steps_per_second": 1.699,
+      "step": 478
+    },
+    {
+      "epoch": 0.334290937581614,
+      "grad_norm": 0.13474580645561218,
+      "learning_rate": 1.8323353293413175e-05,
+      "loss": 0.2914,
+      "step": 480
+    },
+    {
+      "epoch": 0.34125533211456427,
+      "grad_norm": 0.12812362611293793,
+      "learning_rate": 1.827544910179641e-05,
+      "loss": 0.3327,
+      "step": 490
+    },
+    {
+      "epoch": 0.3482197266475146,
+      "grad_norm": 0.11661785840988159,
+      "learning_rate": 1.822754491017964e-05,
+      "loss": 0.3468,
+      "step": 500
+    },
+    {
+      "epoch": 0.3551841211804649,
+      "grad_norm": 0.12990142405033112,
+      "learning_rate": 1.8179640718562878e-05,
+      "loss": 0.3318,
+      "step": 510
+    },
+    {
+      "epoch": 0.36214851571341516,
+      "grad_norm": 0.1309327632188797,
+      "learning_rate": 1.8131736526946108e-05,
+      "loss": 0.2942,
+      "step": 520
+    },
+    {
+      "epoch": 0.36911291024636544,
+      "grad_norm": 0.12246479839086533,
+      "learning_rate": 1.8083832335329345e-05,
+      "loss": 0.3751,
+      "step": 530
+    },
+    {
+      "epoch": 0.3760773047793157,
+      "grad_norm": 0.14134123921394348,
+      "learning_rate": 1.8035928143712575e-05,
+      "loss": 0.3248,
+      "step": 540
+    },
+    {
+      "epoch": 0.38304169931226606,
+      "grad_norm": 0.1433170884847641,
+      "learning_rate": 1.798802395209581e-05,
+      "loss": 0.3344,
+      "step": 550
+    },
+    {
+      "epoch": 0.39000609384521634,
+      "grad_norm": 0.13364745676517487,
+      "learning_rate": 1.7940119760479045e-05,
+      "loss": 0.2928,
+      "step": 560
+    },
+    {
+      "epoch": 0.3969704883781666,
+      "grad_norm": 0.13446496427059174,
+      "learning_rate": 1.7892215568862278e-05,
+      "loss": 0.3421,
+      "step": 570
+    },
+    {
+      "epoch": 0.4039348829111169,
+      "grad_norm": 0.1586250364780426,
+      "learning_rate": 1.784431137724551e-05,
+      "loss": 0.3099,
+      "step": 580
+    },
+    {
+      "epoch": 0.4108992774440672,
+      "grad_norm": 0.1283930391073227,
+      "learning_rate": 1.7796407185628745e-05,
+      "loss": 0.3123,
+      "step": 590
+    },
+    {
+      "epoch": 0.4178636719770175,
+      "grad_norm": 0.14261852204799652,
+      "learning_rate": 1.7748502994011978e-05,
+      "loss": 0.2886,
+      "step": 600
+    },
+    {
+      "epoch": 0.4248280665099678,
+      "grad_norm": 0.13329675793647766,
+      "learning_rate": 1.770059880239521e-05,
+      "loss": 0.3255,
+      "step": 610
+    },
+    {
+      "epoch": 0.4317924610429181,
+      "grad_norm": 0.13880160450935364,
+      "learning_rate": 1.7652694610778445e-05,
+      "loss": 0.326,
+      "step": 620
+    },
+    {
+      "epoch": 0.43875685557586835,
+      "grad_norm": 0.14388258755207062,
+      "learning_rate": 1.7604790419161678e-05,
+      "loss": 0.3269,
+      "step": 630
+    },
+    {
+      "epoch": 0.4457212501088187,
+      "grad_norm": 0.1571163535118103,
+      "learning_rate": 1.755688622754491e-05,
+      "loss": 0.2919,
+      "step": 640
+    },
+    {
+      "epoch": 0.45268564464176897,
+      "grad_norm": 0.15307635068893433,
+      "learning_rate": 1.7508982035928145e-05,
+      "loss": 0.2773,
+      "step": 650
+    },
+    {
+      "epoch": 0.45965003917471925,
+      "grad_norm": 0.1438165158033371,
+      "learning_rate": 1.7461077844311378e-05,
+      "loss": 0.2847,
+      "step": 660
+    },
+    {
+      "epoch": 0.46661443370766953,
+      "grad_norm": 0.15620216727256775,
+      "learning_rate": 1.741317365269461e-05,
+      "loss": 0.29,
+      "step": 670
+    },
+    {
+      "epoch": 0.4735788282406198,
+      "grad_norm": 0.1414933055639267,
+      "learning_rate": 1.7365269461077845e-05,
+      "loss": 0.2754,
+      "step": 680
+    },
+    {
+      "epoch": 0.48054322277357014,
+      "grad_norm": 0.15853798389434814,
+      "learning_rate": 1.7317365269461078e-05,
+      "loss": 0.3062,
+      "step": 690
+    },
+    {
+      "epoch": 0.4875076173065204,
+      "grad_norm": 0.17159980535507202,
+      "learning_rate": 1.726946107784431e-05,
+      "loss": 0.2953,
+      "step": 700
+    },
+    {
+      "epoch": 0.4944720118394707,
+      "grad_norm": 0.16937299072742462,
+      "learning_rate": 1.7221556886227548e-05,
+      "loss": 0.2895,
+      "step": 710
+    },
+    {
+      "epoch": 0.501436406372421,
+      "grad_norm": 0.1455833464860916,
+      "learning_rate": 1.717365269461078e-05,
+      "loss": 0.2912,
+      "step": 720
+    },
+    {
+      "epoch": 0.5084008009053713,
+      "grad_norm": 0.15824872255325317,
+      "learning_rate": 1.7125748502994015e-05,
+      "loss": 0.3012,
+      "step": 730
+    },
+    {
+      "epoch": 0.5153651954383216,
+      "grad_norm": 0.1499851644039154,
+      "learning_rate": 1.7077844311377248e-05,
+      "loss": 0.263,
+      "step": 740
+    },
+    {
+      "epoch": 0.5223295899712719,
+      "grad_norm": 0.1611323356628418,
+      "learning_rate": 1.702994011976048e-05,
+      "loss": 0.2595,
+      "step": 750
+    },
+    {
+      "epoch": 0.5292939845042222,
+      "grad_norm": 0.16144199669361115,
+      "learning_rate": 1.6982035928143714e-05,
+      "loss": 0.2939,
+      "step": 760
+    },
+    {
+      "epoch": 0.5362583790371724,
+      "grad_norm": 0.13433364033699036,
+      "learning_rate": 1.6934131736526948e-05,
+      "loss": 0.2817,
+      "step": 770
+    },
+    {
+      "epoch": 0.5432227735701227,
+      "grad_norm": 0.15188777446746826,
+      "learning_rate": 1.688622754491018e-05,
+      "loss": 0.2804,
+      "step": 780
+    },
+    {
+      "epoch": 0.550187168103073,
+      "grad_norm": 0.1530751883983612,
+      "learning_rate": 1.6838323353293414e-05,
+      "loss": 0.298,
+      "step": 790
+    },
+    {
+      "epoch": 0.5571515626360233,
+      "grad_norm": 0.14582635462284088,
+      "learning_rate": 1.6790419161676648e-05,
+      "loss": 0.3177,
+      "step": 800
+    },
+    {
+      "epoch": 0.5641159571689737,
+      "grad_norm": 0.174430251121521,
+      "learning_rate": 1.674251497005988e-05,
+      "loss": 0.2624,
+      "step": 810
+    },
+    {
+      "epoch": 0.571080351701924,
+      "grad_norm": 0.1558169573545456,
+      "learning_rate": 1.6694610778443114e-05,
+      "loss": 0.2631,
+      "step": 820
+    },
+    {
+      "epoch": 0.5780447462348742,
+      "grad_norm": 0.17108450829982758,
+      "learning_rate": 1.6646706586826348e-05,
+      "loss": 0.2903,
+      "step": 830
+    },
+    {
+      "epoch": 0.5850091407678245,
+      "grad_norm": 0.16796821355819702,
+      "learning_rate": 1.659880239520958e-05,
+      "loss": 0.2928,
+      "step": 840
+    },
+    {
+      "epoch": 0.5919735353007748,
+      "grad_norm": 0.16556940972805023,
+      "learning_rate": 1.6550898203592814e-05,
+      "loss": 0.2882,
+      "step": 850
+    },
+    {
+      "epoch": 0.5989379298337251,
+      "grad_norm": 0.1716109961271286,
+      "learning_rate": 1.650299401197605e-05,
+      "loss": 0.2814,
+      "step": 860
+    },
+    {
+      "epoch": 0.6059023243666753,
+      "grad_norm": 0.1695253700017929,
+      "learning_rate": 1.645508982035928e-05,
+      "loss": 0.2909,
+      "step": 870
+    },
+    {
+      "epoch": 0.6128667188996256,
+      "grad_norm": 0.1896178424358368,
+      "learning_rate": 1.6407185628742518e-05,
+      "loss": 0.2909,
+      "step": 880
+    },
+    {
+      "epoch": 0.6198311134325759,
+      "grad_norm": 0.1624131202697754,
+      "learning_rate": 1.6359281437125748e-05,
+      "loss": 0.2672,
+      "step": 890
+    },
+    {
+      "epoch": 0.6267955079655263,
+      "grad_norm": 0.16821731626987457,
+      "learning_rate": 1.6311377245508984e-05,
+      "loss": 0.2626,
+      "step": 900
+    },
+    {
+      "epoch": 0.6337599024984766,
+      "grad_norm": 0.17026785016059875,
+      "learning_rate": 1.6263473053892214e-05,
+      "loss": 0.2735,
+      "step": 910
+    },
+    {
+      "epoch": 0.6407242970314269,
+      "grad_norm": 0.19429761171340942,
+      "learning_rate": 1.621556886227545e-05,
+      "loss": 0.2605,
+      "step": 920
+    },
+    {
+      "epoch": 0.6476886915643771,
+      "grad_norm": 0.16931040585041046,
+      "learning_rate": 1.6167664670658684e-05,
+      "loss": 0.2722,
+      "step": 930
+    },
+    {
+      "epoch": 0.6546530860973274,
+      "grad_norm": 0.20588335394859314,
+      "learning_rate": 1.6119760479041918e-05,
+      "loss": 0.301,
+      "step": 940
+    },
+    {
+      "epoch": 0.6616174806302777,
+      "grad_norm": 0.16699948906898499,
+      "learning_rate": 1.607185628742515e-05,
+      "loss": 0.2646,
+      "step": 950
+    },
+    {
+      "epoch": 0.6657961173500478,
+      "eval_loss": 0.2465677410364151,
+      "eval_runtime": 356.1016,
+      "eval_samples_per_second": 3.395,
+      "eval_steps_per_second": 1.699,
+      "step": 956
+    },
+    {
+      "epoch": 0.668581875163228,
+      "grad_norm": 0.20734217762947083,
+      "learning_rate": 1.6023952095808384e-05,
+      "loss": 0.2476,
+      "step": 960
+    },
+    {
+      "epoch": 0.6755462696961783,
+      "grad_norm": 0.21317210793495178,
+      "learning_rate": 1.5976047904191618e-05,
+      "loss": 0.2434,
+      "step": 970
+    },
+    {
+      "epoch": 0.6825106642291285,
+      "grad_norm": 0.19125515222549438,
+      "learning_rate": 1.592814371257485e-05,
+      "loss": 0.2823,
+      "step": 980
+    },
+    {
+      "epoch": 0.6894750587620788,
+      "grad_norm": 0.1651381254196167,
+      "learning_rate": 1.5880239520958084e-05,
+      "loss": 0.2769,
+      "step": 990
+    },
+    {
+      "epoch": 0.6964394532950292,
+      "grad_norm": 0.18061324954032898,
+      "learning_rate": 1.583233532934132e-05,
+      "loss": 0.2788,
+      "step": 1000
+    },
+    {
+      "epoch": 0.7034038478279795,
+      "grad_norm": 0.17788180708885193,
+      "learning_rate": 1.578443113772455e-05,
+      "loss": 0.2596,
+      "step": 1010
+    },
+    {
+      "epoch": 0.7103682423609298,
+      "grad_norm": 0.19635842740535736,
+      "learning_rate": 1.5736526946107788e-05,
+      "loss": 0.2391,
+      "step": 1020
+    },
+    {
+      "epoch": 0.71733263689388,
+      "grad_norm": 0.18000468611717224,
+      "learning_rate": 1.5688622754491018e-05,
+      "loss": 0.2589,
+      "step": 1030
+    },
+    {
+      "epoch": 0.7242970314268303,
+      "grad_norm": 0.18016283214092255,
+      "learning_rate": 1.5640718562874254e-05,
+      "loss": 0.2673,
+      "step": 1040
+    },
+    {
+      "epoch": 0.7312614259597806,
+      "grad_norm": 0.2085445076227188,
+      "learning_rate": 1.5592814371257484e-05,
+      "loss": 0.295,
+      "step": 1050
+    },
+    {
+      "epoch": 0.7382258204927309,
+      "grad_norm": 0.1937682181596756,
+      "learning_rate": 1.554491017964072e-05,
+      "loss": 0.2741,
+      "step": 1060
+    },
+    {
+      "epoch": 0.7451902150256812,
+      "grad_norm": 0.18496188521385193,
+      "learning_rate": 1.5497005988023954e-05,
+      "loss": 0.2485,
+      "step": 1070
+    },
+    {
+      "epoch": 0.7521546095586314,
+      "grad_norm": 0.21024686098098755,
+      "learning_rate": 1.5449101796407188e-05,
+      "loss": 0.2569,
+      "step": 1080
+    },
+    {
+      "epoch": 0.7591190040915818,
+      "grad_norm": 0.1960270255804062,
+      "learning_rate": 1.540119760479042e-05,
+      "loss": 0.2335,
+      "step": 1090
+    },
+    {
+      "epoch": 0.7660833986245321,
+      "grad_norm": 0.19557876884937286,
+      "learning_rate": 1.5353293413173654e-05,
+      "loss": 0.279,
+      "step": 1100
+    },
+    {
+      "epoch": 0.7730477931574824,
+      "grad_norm": 0.17300210893154144,
+      "learning_rate": 1.5305389221556888e-05,
+      "loss": 0.249,
+      "step": 1110
+    },
+    {
+      "epoch": 0.7800121876904327,
+      "grad_norm": 0.1768539994955063,
+      "learning_rate": 1.5257485029940121e-05,
+      "loss": 0.2415,
+      "step": 1120
+    },
+    {
+      "epoch": 0.786976582223383,
+      "grad_norm": 0.20632915198802948,
+      "learning_rate": 1.5209580838323354e-05,
+      "loss": 0.2312,
+      "step": 1130
+    },
+    {
+      "epoch": 0.7939409767563332,
+      "grad_norm": 0.1948348730802536,
+      "learning_rate": 1.516167664670659e-05,
+      "loss": 0.2415,
+      "step": 1140
+    },
+    {
+      "epoch": 0.8009053712892835,
+      "grad_norm": 0.16288821399211884,
+      "learning_rate": 1.511377245508982e-05,
+      "loss": 0.2571,
+      "step": 1150
+    },
+    {
+      "epoch": 0.8078697658222338,
+      "grad_norm": 0.21365194022655487,
+      "learning_rate": 1.5065868263473056e-05,
+      "loss": 0.2594,
+      "step": 1160
+    },
+    {
+      "epoch": 0.8148341603551841,
+      "grad_norm": 0.20960865914821625,
+      "learning_rate": 1.5017964071856287e-05,
+      "loss": 0.2537,
+      "step": 1170
+    },
+    {
+      "epoch": 0.8217985548881344,
+      "grad_norm": 0.20458345115184784,
+      "learning_rate": 1.4970059880239522e-05,
+      "loss": 0.2675,
+      "step": 1180
+    },
+    {
+      "epoch": 0.8287629494210847,
+      "grad_norm": 0.22388549149036407,
+      "learning_rate": 1.4922155688622754e-05,
+      "loss": 0.2525,
+      "step": 1190
+    },
+    {
+      "epoch": 0.835727343954035,
+      "grad_norm": 0.2139691859483719,
+      "learning_rate": 1.4874251497005989e-05,
+      "loss": 0.251,
+      "step": 1200
+    },
+    {
+      "epoch": 0.8426917384869853,
+      "grad_norm": 0.18761980533599854,
+      "learning_rate": 1.4826347305389224e-05,
+      "loss": 0.2328,
+      "step": 1210
+    },
+    {
+      "epoch": 0.8496561330199356,
+      "grad_norm": 0.183973029255867,
+      "learning_rate": 1.4778443113772456e-05,
+      "loss": 0.2239,
+      "step": 1220
+    },
+    {
+      "epoch": 0.8566205275528859,
+      "grad_norm": 0.23718492686748505,
+      "learning_rate": 1.473053892215569e-05,
+      "loss": 0.2423,
+      "step": 1230
+    },
+    {
+      "epoch": 0.8635849220858361,
+      "grad_norm": 0.23875045776367188,
+      "learning_rate": 1.4682634730538922e-05,
+      "loss": 0.2365,
+      "step": 1240
+    },
+    {
+      "epoch": 0.8705493166187864,
+      "grad_norm": 0.21570812165737152,
+      "learning_rate": 1.4634730538922157e-05,
+      "loss": 0.268,
+      "step": 1250
+    },
+    {
+      "epoch": 0.8775137111517367,
+      "grad_norm": 0.20839928090572357,
+      "learning_rate": 1.4586826347305389e-05,
+      "loss": 0.2502,
+      "step": 1260
+    },
+    {
+      "epoch": 0.884478105684687,
+      "grad_norm": 0.21605758368968964,
+      "learning_rate": 1.4538922155688624e-05,
+      "loss": 0.2365,
+      "step": 1270
+    },
+    {
+      "epoch": 0.8914425002176374,
+      "grad_norm": 0.23300880193710327,
+      "learning_rate": 1.4491017964071859e-05,
+      "loss": 0.2404,
+      "step": 1280
+    },
+    {
+      "epoch": 0.8984068947505877,
+      "grad_norm": 0.21796733140945435,
+      "learning_rate": 1.444311377245509e-05,
+      "loss": 0.2704,
+      "step": 1290
+    },
+    {
+      "epoch": 0.9053712892835379,
+      "grad_norm": 0.25555598735809326,
+      "learning_rate": 1.4395209580838326e-05,
+      "loss": 0.2674,
+      "step": 1300
+    },
+    {
+      "epoch": 0.9123356838164882,
+      "grad_norm": 0.22284413874149323,
+      "learning_rate": 1.4347305389221557e-05,
+      "loss": 0.2471,
+      "step": 1310
+    },
+    {
+      "epoch": 0.9193000783494385,
+      "grad_norm": 0.22234566509723663,
+      "learning_rate": 1.4299401197604792e-05,
+      "loss": 0.2529,
+      "step": 1320
+    },
+    {
+      "epoch": 0.9262644728823888,
+      "grad_norm": 0.221883624792099,
+      "learning_rate": 1.4251497005988024e-05,
+      "loss": 0.1932,
+      "step": 1330
+    },
+    {
+      "epoch": 0.9332288674153391,
+      "grad_norm": 0.22873520851135254,
+      "learning_rate": 1.4203592814371259e-05,
+      "loss": 0.2412,
+      "step": 1340
+    },
+    {
+      "epoch": 0.9401932619482893,
+      "grad_norm": 0.21307863295078278,
+      "learning_rate": 1.4155688622754492e-05,
+      "loss": 0.2384,
+      "step": 1350
+    },
+    {
+      "epoch": 0.9471576564812396,
+      "grad_norm": 0.21296364068984985,
+      "learning_rate": 1.4107784431137726e-05,
+      "loss": 0.2318,
+      "step": 1360
+    },
+    {
+      "epoch": 0.95412205101419,
+      "grad_norm": 0.21060685813426971,
+      "learning_rate": 1.405988023952096e-05,
+      "loss": 0.2323,
+      "step": 1370
+    },
+    {
+      "epoch": 0.9610864455471403,
+      "grad_norm": 0.23930418491363525,
+      "learning_rate": 1.4011976047904192e-05,
+      "loss": 0.2398,
+      "step": 1380
+    },
+    {
+      "epoch": 0.9680508400800906,
+      "grad_norm": 0.22837185859680176,
+      "learning_rate": 1.3964071856287427e-05,
+      "loss": 0.233,
+      "step": 1390
+    },
+    {
+      "epoch": 0.9750152346130408,
+      "grad_norm": 0.21798169612884521,
+      "learning_rate": 1.3916167664670659e-05,
+      "loss": 0.2161,
+      "step": 1400
+    },
+    {
+      "epoch": 0.9819796291459911,
+      "grad_norm": 0.2846814692020416,
+      "learning_rate": 1.3868263473053894e-05,
+      "loss": 0.2301,
+      "step": 1410
+    },
+    {
+      "epoch": 0.9889440236789414,
+      "grad_norm": 0.22030217945575714,
+      "learning_rate": 1.3820359281437127e-05,
+      "loss": 0.2017,
+      "step": 1420
+    },
+    {
+      "epoch": 0.9959084182118917,
+      "grad_norm": 0.25554022192955017,
+      "learning_rate": 1.377245508982036e-05,
+      "loss": 0.2296,
+      "step": 1430
+    },
+    {
+      "epoch": 0.9986941760250718,
+      "eval_loss": 0.21637588739395142,
+      "eval_runtime": 356.0721,
+      "eval_samples_per_second": 3.395,
+      "eval_steps_per_second": 1.699,
+      "step": 1434
+    },
+    {
+      "epoch": 1.0030469226081657,
+      "grad_norm": 0.2862723469734192,
+      "learning_rate": 1.3724550898203594e-05,
+      "loss": 0.2538,
+      "step": 1440
+    },
+    {
+      "epoch": 1.010011317141116,
+      "grad_norm": 0.2589089870452881,
+      "learning_rate": 1.3676646706586827e-05,
+      "loss": 0.2196,
+      "step": 1450
+    },
+    {
+      "epoch": 1.0169757116740663,
+      "grad_norm": 0.27893784642219543,
+      "learning_rate": 1.362874251497006e-05,
+      "loss": 0.1935,
+      "step": 1460
+    },
+    {
+      "epoch": 1.0239401062070166,
+      "grad_norm": 0.2530602514743805,
+      "learning_rate": 1.3580838323353294e-05,
+      "loss": 0.2248,
+      "step": 1470
+    },
+    {
+      "epoch": 1.0309045007399669,
+      "grad_norm": 0.24902446568012238,
+      "learning_rate": 1.3532934131736529e-05,
+      "loss": 0.224,
+      "step": 1480
+    },
+    {
+      "epoch": 1.0378688952729171,
+      "grad_norm": 0.27132269740104675,
+      "learning_rate": 1.3485029940119762e-05,
+      "loss": 0.2208,
+      "step": 1490
+    },
+    {
+      "epoch": 1.0448332898058674,
+      "grad_norm": 0.2570977807044983,
+      "learning_rate": 1.3437125748502996e-05,
+      "loss": 0.2103,
+      "step": 1500
+    },
+    {
+      "epoch": 1.0517976843388177,
+      "grad_norm": 0.2994561493396759,
+      "learning_rate": 1.3389221556886229e-05,
+      "loss": 0.2243,
+      "step": 1510
+    },
+    {
+      "epoch": 1.058762078871768,
+      "grad_norm": 0.22994251549243927,
+      "learning_rate": 1.3341317365269462e-05,
+      "loss": 0.2217,
+      "step": 1520
+    },
+    {
+      "epoch": 1.0657264734047183,
+      "grad_norm": 0.22706159949302673,
+      "learning_rate": 1.3293413173652696e-05,
+      "loss": 0.2219,
+      "step": 1530
+    },
+    {
+      "epoch": 1.0726908679376688,
+      "grad_norm": 0.26584964990615845,
+      "learning_rate": 1.3245508982035929e-05,
+      "loss": 0.2329,
+      "step": 1540
+    },
+    {
+      "epoch": 1.079655262470619,
+      "grad_norm": 0.2549090087413788,
+      "learning_rate": 1.3197604790419162e-05,
+      "loss": 0.2235,
+      "step": 1550
+    },
+    {
+      "epoch": 1.0866196570035693,
+      "grad_norm": 0.27169135212898254,
+      "learning_rate": 1.3149700598802397e-05,
+      "loss": 0.2332,
+      "step": 1560
+    },
+    {
+      "epoch": 1.0935840515365196,
+      "grad_norm": 0.26733773946762085,
+      "learning_rate": 1.3101796407185629e-05,
+      "loss": 0.2062,
+      "step": 1570
+    },
+    {
+      "epoch": 1.1005484460694699,
+      "grad_norm": 0.3222082853317261,
+      "learning_rate": 1.3053892215568864e-05,
+      "loss": 0.2111,
+      "step": 1580
+    },
+    {
+      "epoch": 1.1075128406024202,
+      "grad_norm": 0.20403257012367249,
+      "learning_rate": 1.3005988023952097e-05,
+      "loss": 0.226,
+      "step": 1590
+    },
+    {
+      "epoch": 1.1144772351353704,
+      "grad_norm": 0.2658451199531555,
+      "learning_rate": 1.295808383233533e-05,
+      "loss": 0.2013,
+      "step": 1600
+    },
+    {
+      "epoch": 1.1214416296683207,
+      "grad_norm": 0.2719573676586151,
+      "learning_rate": 1.2910179640718564e-05,
+      "loss": 0.2036,
+      "step": 1610
+    },
+    {
+      "epoch": 1.128406024201271,
+      "grad_norm": 0.29076331853866577,
+      "learning_rate": 1.2862275449101797e-05,
+      "loss": 0.2162,
+      "step": 1620
+    },
+    {
+      "epoch": 1.1353704187342213,
+      "grad_norm": 0.2324078530073166,
+      "learning_rate": 1.2814371257485032e-05,
+      "loss": 0.2002,
+      "step": 1630
+    },
+    {
+      "epoch": 1.1423348132671716,
+      "grad_norm": 0.27219468355178833,
+      "learning_rate": 1.2766467065868264e-05,
+      "loss": 0.2238,
+      "step": 1640
+    },
+    {
+      "epoch": 1.1492992078001218,
+      "grad_norm": 0.22101271152496338,
+      "learning_rate": 1.2718562874251499e-05,
+      "loss": 0.1845,
+      "step": 1650
+    },
+    {
+      "epoch": 1.1562636023330721,
+      "grad_norm": 0.25101369619369507,
+      "learning_rate": 1.267065868263473e-05,
+      "loss": 0.2296,
+      "step": 1660
+    },
+    {
+      "epoch": 1.1632279968660224,
+      "grad_norm": 0.25895142555236816,
+      "learning_rate": 1.2622754491017965e-05,
+      "loss": 0.2271,
+      "step": 1670
+    },
+    {
+      "epoch": 1.1701923913989727,
+      "grad_norm": 0.26405826210975647,
+      "learning_rate": 1.2574850299401197e-05,
+      "loss": 0.2159,
+      "step": 1680
+    },
+    {
+      "epoch": 1.177156785931923,
+      "grad_norm": 0.23231615126132965,
+      "learning_rate": 1.2526946107784432e-05,
+      "loss": 0.1916,
+      "step": 1690
+    },
+    {
+      "epoch": 1.1841211804648732,
+      "grad_norm": 0.3211207985877991,
+      "learning_rate": 1.2479041916167665e-05,
+      "loss": 0.2154,
+      "step": 1700
+    },
+    {
+      "epoch": 1.1910855749978237,
+      "grad_norm": 0.2948794662952423,
+      "learning_rate": 1.2431137724550899e-05,
+      "loss": 0.1999,
+      "step": 1710
+    },
+    {
+      "epoch": 1.198049969530774,
+      "grad_norm": 0.28081706166267395,
+      "learning_rate": 1.2383233532934134e-05,
+      "loss": 0.2301,
+      "step": 1720
+    },
+    {
+      "epoch": 1.2050143640637243,
+      "grad_norm": 0.296283096075058,
+      "learning_rate": 1.2335329341317365e-05,
+      "loss": 0.2164,
+      "step": 1730
+    },
+    {
+      "epoch": 1.2119787585966746,
+      "grad_norm": 0.29658278822898865,
+      "learning_rate": 1.22874251497006e-05,
+      "loss": 0.1916,
+      "step": 1740
+    },
+    {
+      "epoch": 1.2189431531296249,
+      "grad_norm": 0.2917250990867615,
+      "learning_rate": 1.2239520958083832e-05,
+      "loss": 0.2045,
+      "step": 1750
+    },
+    {
+      "epoch": 1.2259075476625751,
+      "grad_norm": 0.35840919613838196,
+      "learning_rate": 1.2191616766467067e-05,
+      "loss": 0.1965,
+      "step": 1760
+    },
+    {
+      "epoch": 1.2328719421955254,
+      "grad_norm": 0.2819244861602783,
+      "learning_rate": 1.2143712574850299e-05,
+      "loss": 0.2032,
+      "step": 1770
+    },
+    {
+      "epoch": 1.2398363367284757,
+      "grad_norm": 0.3518809974193573,
+      "learning_rate": 1.2095808383233534e-05,
+      "loss": 0.2043,
+      "step": 1780
+    },
+    {
+      "epoch": 1.246800731261426,
+      "grad_norm": 0.3232385516166687,
+      "learning_rate": 1.2047904191616769e-05,
+      "loss": 0.23,
+      "step": 1790
+    },
+    {
+      "epoch": 1.2537651257943763,
+      "grad_norm": 0.23131519556045532,
+      "learning_rate": 1.2e-05,
+      "loss": 0.2106,
+      "step": 1800
+    },
+    {
+      "epoch": 1.2607295203273265,
+      "grad_norm": 0.30877017974853516,
+      "learning_rate": 1.1952095808383235e-05,
+      "loss": 0.2088,
+      "step": 1810
+    },
+    {
+      "epoch": 1.2676939148602768,
+      "grad_norm": 0.36684754490852356,
+      "learning_rate": 1.1904191616766467e-05,
+      "loss": 0.2024,
+      "step": 1820
+    },
+    {
+      "epoch": 1.274658309393227,
+      "grad_norm": 0.21124888956546783,
+      "learning_rate": 1.1856287425149702e-05,
+      "loss": 0.1929,
+      "step": 1830
+    },
+    {
+      "epoch": 1.2816227039261774,
+      "grad_norm": 0.2815339267253876,
+      "learning_rate": 1.1808383233532934e-05,
+      "loss": 0.1973,
+      "step": 1840
+    },
+    {
+      "epoch": 1.2885870984591277,
+      "grad_norm": 0.3017849922180176,
+      "learning_rate": 1.1760479041916169e-05,
+      "loss": 0.233,
+      "step": 1850
+    },
+    {
+      "epoch": 1.295551492992078,
+      "grad_norm": 0.3456547260284424,
+      "learning_rate": 1.1712574850299404e-05,
+      "loss": 0.1769,
+      "step": 1860
+    },
+    {
+      "epoch": 1.3025158875250282,
+      "grad_norm": 0.28210344910621643,
+      "learning_rate": 1.1664670658682635e-05,
+      "loss": 0.204,
+      "step": 1870
+    },
+    {
+      "epoch": 1.3094802820579785,
+      "grad_norm": 0.3701432943344116,
+      "learning_rate": 1.161676646706587e-05,
+      "loss": 0.1923,
+      "step": 1880
+    },
+    {
+      "epoch": 1.3164446765909288,
+      "grad_norm": 0.3656926453113556,
+      "learning_rate": 1.1568862275449102e-05,
+      "loss": 0.2136,
+      "step": 1890
+    },
+    {
+      "epoch": 1.323409071123879,
+      "grad_norm": 0.29142090678215027,
+      "learning_rate": 1.1520958083832337e-05,
+      "loss": 0.1957,
+      "step": 1900
+    },
+    {
+      "epoch": 1.3303734656568293,
+      "grad_norm": 0.2799660563468933,
+      "learning_rate": 1.1473053892215569e-05,
+      "loss": 0.2009,
+      "step": 1910
+    },
+    {
+      "epoch": 1.3317663445634196,
+      "eval_loss": 0.19295775890350342,
+      "eval_runtime": 356.0153,
+      "eval_samples_per_second": 3.396,
+      "eval_steps_per_second": 1.699,
+      "step": 1912
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 4305,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 478,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 8.929647234254438e+17,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1912/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:956bd3fe04334201160cff0d2e3faf62e32c6c8f709339d63e01a7d4b73b24d0
+size 5560

checkpoint-2390/README.md

@@ -0,0 +1,209 @@
+---
+base_model: Salesforce/Llama-xLAM-2-8b-fc-r
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:Salesforce/Llama-xLAM-2-8b-fc-r
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

checkpoint-2390/adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Salesforce/Llama-xLAM-2-8b-fc-r",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "o_proj",
+    "down_proj",
+    "k_proj",
+    "up_proj",
+    "v_proj",
+    "gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-2390/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c25fe76b45b61c738c5af50c849eb346dfaeeb3102c0a7abd8c16d6d6142a277
+size 671149168

checkpoint-2390/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68b30748271c159d0ef28ff2f39abf3a369e9ae422ab05f0facb7717c6f1212b
+size 1342555602

checkpoint-2390/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e45c7e2a85335ad5036342a2571df9086fcca4257cbf7e27bd3cb7490778b40e
+size 14244

checkpoint-2390/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32caa7f4861b8b3bb4853ddaed10fc51e3a5729a353a3fa812005040b9a0cd89
+size 1064

checkpoint-2390/special_tokens_map.json

@@ -0,0 +1,27 @@
+{
+  "additional_special_tokens": [
+    "<|eot_id|>",
+    "<|eom_id|>"
+  ],
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-2390/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a55c1a4c5e3af7f2fb2bc0cd245a09dabd742dc24e7cb3741db1e11c7fe1a52
+size 17210019

checkpoint-2390/tokenizer_config.json

@@ -0,0 +1,2070 @@
+{
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|finetune_right_pad_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|eom_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|python_tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|eot_id|>",
+    "<|eom_id|>"
+  ],
+  "bos_token": "<|begin_of_text|>",
+  "chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n    {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n    {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not tools is defined %}\n    {%- set tools = none %}\n{%- endif %}\n\n{#- Extract system message #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if messages[0]['role'] == 'system' %}\n    {%- set system_message = messages[0]['content'] | trim %}\n    {%- set messages = messages[1:] %}\n    {{- system_message + \"\\n\" }}\n{%- else %}\n    {%- set system_message = \"You are a helpful assistant that can use tools. You are developed by Salesforce xLAM team.\" %}\n    {% set format_instruction %}You have access to a set of tools. When using tools, make calls in a single JSON array: \n\n[{\"name\": \"tool_call_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}}, ... (additional parallel tool calls as needed)]\n\nIf no tool is suitable, state that explicitly. If the user's input lacks required parameters, ask for clarification. Do not interpret or respond until tool results are returned. Once they are available, process them or make additional calls if needed. For tasks that don't require tools, such as casual conversation or general advice, respond directly in plain text. The available tools are:{% endset %}\n    {{- system_message + \"\\n\" }}\n    {%- if tools is not none %}\n        {{- format_instruction + \"\\n\\n\" }}\n    {%- endif %}\n{%- endif %}\n\n\n{%- if tools is not none %}\n    {%- for t in tools %}\n        {{- t | tojson(indent=4) }}\n        {{- \"\\n\\n\" }}\n    {%- endfor %}\n{%- endif %}\n{{- \"<|eot_id|>\" }}\n\n{%- for message in messages %}\n    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n    {%- elif 'tool_calls' in message %}\n        {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n        {%- if message['tool_calls'] %}\n            {{- \"[\" }}\n            {%- for tool_call_function in message.tool_calls %}\n                {%- set tool_call = tool_call_function.function %}\n                {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n                {{- '\"arguments\": ' }}\n                {{- tool_call.arguments | tojson }}\n                {{- \"}\" }}\n                {%- if not loop.last %}\n                    {{- \", \" }}\n                {%- endif %}\n            {%- endfor %}\n            {{- \"]\" }}\n            {{- \"<|eot_id|>\" }}\n        {%- elif message['content'] %}\n            {{- message['content'] | trim + '<|eot_id|>' }}\n        {%- else %}\n            {{- \"[]\\n\" + '<|eot_id|>' }}\n        {%- endif %}\n    {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n        {{- \"<|start_header_id|>\" + \"ipython\" + \"<|end_header_id|>\\n\\n\" }}\n        {%- set content = message[\"content\"] %}\n        {%- if content is mapping or (content is iterable and content is not string) %}\n            {{- content | tojson }}\n        {%- else %}\n            {{- content }}\n        {%- endif %}\n        {{- \"<|eot_id|>\" }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 16384,
+  "pad_token": "<|eot_id|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}

checkpoint-2390/trainer_state.json

@@ -0,0 +1,1746 @@
+{
+  "best_metric": 0.17429304122924805,
+  "best_model_checkpoint": "./xlam_Qlora_out_eval/checkpoint-2390",
+  "epoch": 1.6646644032384434,
+  "eval_steps": 478,
+  "global_step": 2390,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.006964394532950292,
+      "grad_norm": 0.29273906350135803,
+      "learning_rate": 1.5384615384615387e-06,
+      "loss": 1.8834,
+      "step": 10
+    },
+    {
+      "epoch": 0.013928789065900584,
+      "grad_norm": 0.3107926845550537,
+      "learning_rate": 3.0769230769230774e-06,
+      "loss": 1.876,
+      "step": 20
+    },
+    {
+      "epoch": 0.020893183598850874,
+      "grad_norm": 0.3098163902759552,
+      "learning_rate": 4.615384615384616e-06,
+      "loss": 1.9099,
+      "step": 30
+    },
+    {
+      "epoch": 0.027857578131801168,
+      "grad_norm": 0.36715859174728394,
+      "learning_rate": 6.153846153846155e-06,
+      "loss": 1.8142,
+      "step": 40
+    },
+    {
+      "epoch": 0.034821972664751455,
+      "grad_norm": 0.3511175215244293,
+      "learning_rate": 7.692307692307694e-06,
+      "loss": 1.7271,
+      "step": 50
+    },
+    {
+      "epoch": 0.04178636719770175,
+      "grad_norm": 0.3458055853843689,
+      "learning_rate": 9.230769230769232e-06,
+      "loss": 1.6073,
+      "step": 60
+    },
+    {
+      "epoch": 0.04875076173065204,
+      "grad_norm": 0.37719297409057617,
+      "learning_rate": 1.076923076923077e-05,
+      "loss": 1.4997,
+      "step": 70
+    },
+    {
+      "epoch": 0.055715156263602336,
+      "grad_norm": 0.30273741483688354,
+      "learning_rate": 1.230769230769231e-05,
+      "loss": 1.2914,
+      "step": 80
+    },
+    {
+      "epoch": 0.06267955079655263,
+      "grad_norm": 0.3690374195575714,
+      "learning_rate": 1.3846153846153847e-05,
+      "loss": 1.0086,
+      "step": 90
+    },
+    {
+      "epoch": 0.06964394532950291,
+      "grad_norm": 0.3111755847930908,
+      "learning_rate": 1.5384615384615387e-05,
+      "loss": 0.7469,
+      "step": 100
+    },
+    {
+      "epoch": 0.0766083398624532,
+      "grad_norm": 0.13778822124004364,
+      "learning_rate": 1.6923076923076924e-05,
+      "loss": 0.5848,
+      "step": 110
+    },
+    {
+      "epoch": 0.0835727343954035,
+      "grad_norm": 0.10629754513502121,
+      "learning_rate": 1.8461538461538465e-05,
+      "loss": 0.5366,
+      "step": 120
+    },
+    {
+      "epoch": 0.09053712892835379,
+      "grad_norm": 0.09681010991334915,
+      "learning_rate": 2e-05,
+      "loss": 0.5038,
+      "step": 130
+    },
+    {
+      "epoch": 0.09750152346130408,
+      "grad_norm": 0.10756956040859222,
+      "learning_rate": 1.9952095808383235e-05,
+      "loss": 0.5002,
+      "step": 140
+    },
+    {
+      "epoch": 0.10446591799425438,
+      "grad_norm": 0.10488647222518921,
+      "learning_rate": 1.9904191616766468e-05,
+      "loss": 0.5107,
+      "step": 150
+    },
+    {
+      "epoch": 0.11143031252720467,
+      "grad_norm": 0.09359394758939743,
+      "learning_rate": 1.98562874251497e-05,
+      "loss": 0.4604,
+      "step": 160
+    },
+    {
+      "epoch": 0.11839470706015495,
+      "grad_norm": 0.10392932593822479,
+      "learning_rate": 1.9808383233532935e-05,
+      "loss": 0.484,
+      "step": 170
+    },
+    {
+      "epoch": 0.12535910159310526,
+      "grad_norm": 0.12986360490322113,
+      "learning_rate": 1.9760479041916168e-05,
+      "loss": 0.4527,
+      "step": 180
+    },
+    {
+      "epoch": 0.13232349612605554,
+      "grad_norm": 0.14294420182704926,
+      "learning_rate": 1.97125748502994e-05,
+      "loss": 0.4661,
+      "step": 190
+    },
+    {
+      "epoch": 0.13928789065900582,
+      "grad_norm": 0.18483737111091614,
+      "learning_rate": 1.9664670658682635e-05,
+      "loss": 0.4395,
+      "step": 200
+    },
+    {
+      "epoch": 0.14625228519195613,
+      "grad_norm": 0.21114414930343628,
+      "learning_rate": 1.961676646706587e-05,
+      "loss": 0.4071,
+      "step": 210
+    },
+    {
+      "epoch": 0.1532166797249064,
+      "grad_norm": 0.21779198944568634,
+      "learning_rate": 1.95688622754491e-05,
+      "loss": 0.3846,
+      "step": 220
+    },
+    {
+      "epoch": 0.16018107425785671,
+      "grad_norm": 0.1294960230588913,
+      "learning_rate": 1.9520958083832338e-05,
+      "loss": 0.3952,
+      "step": 230
+    },
+    {
+      "epoch": 0.167145468790807,
+      "grad_norm": 0.0927898958325386,
+      "learning_rate": 1.9473053892215568e-05,
+      "loss": 0.4112,
+      "step": 240
+    },
+    {
+      "epoch": 0.1741098633237573,
+      "grad_norm": 0.10358591377735138,
+      "learning_rate": 1.9425149700598805e-05,
+      "loss": 0.406,
+      "step": 250
+    },
+    {
+      "epoch": 0.18107425785670758,
+      "grad_norm": 0.13144026696681976,
+      "learning_rate": 1.9377245508982038e-05,
+      "loss": 0.4058,
+      "step": 260
+    },
+    {
+      "epoch": 0.18803865238965786,
+      "grad_norm": 0.12745942175388336,
+      "learning_rate": 1.932934131736527e-05,
+      "loss": 0.3938,
+      "step": 270
+    },
+    {
+      "epoch": 0.19500304692260817,
+      "grad_norm": 0.09832354635000229,
+      "learning_rate": 1.9281437125748505e-05,
+      "loss": 0.3819,
+      "step": 280
+    },
+    {
+      "epoch": 0.20196744145555845,
+      "grad_norm": 0.0927552729845047,
+      "learning_rate": 1.9233532934131738e-05,
+      "loss": 0.3944,
+      "step": 290
+    },
+    {
+      "epoch": 0.20893183598850876,
+      "grad_norm": 0.09627388417720795,
+      "learning_rate": 1.918562874251497e-05,
+      "loss": 0.348,
+      "step": 300
+    },
+    {
+      "epoch": 0.21589623052145904,
+      "grad_norm": 0.09034324437379837,
+      "learning_rate": 1.9137724550898205e-05,
+      "loss": 0.3521,
+      "step": 310
+    },
+    {
+      "epoch": 0.22286062505440934,
+      "grad_norm": 0.09156423062086105,
+      "learning_rate": 1.9089820359281438e-05,
+      "loss": 0.3663,
+      "step": 320
+    },
+    {
+      "epoch": 0.22982501958735962,
+      "grad_norm": 0.0996803268790245,
+      "learning_rate": 1.904191616766467e-05,
+      "loss": 0.3347,
+      "step": 330
+    },
+    {
+      "epoch": 0.2367894141203099,
+      "grad_norm": 0.09962292015552521,
+      "learning_rate": 1.8994011976047905e-05,
+      "loss": 0.373,
+      "step": 340
+    },
+    {
+      "epoch": 0.2437538086532602,
+      "grad_norm": 0.08211114257574081,
+      "learning_rate": 1.894610778443114e-05,
+      "loss": 0.3641,
+      "step": 350
+    },
+    {
+      "epoch": 0.2507182031862105,
+      "grad_norm": 0.10449781268835068,
+      "learning_rate": 1.889820359281437e-05,
+      "loss": 0.3358,
+      "step": 360
+    },
+    {
+      "epoch": 0.2576825977191608,
+      "grad_norm": 0.11216866970062256,
+      "learning_rate": 1.8850299401197608e-05,
+      "loss": 0.372,
+      "step": 370
+    },
+    {
+      "epoch": 0.2646469922521111,
+      "grad_norm": 0.11125332862138748,
+      "learning_rate": 1.8802395209580838e-05,
+      "loss": 0.3521,
+      "step": 380
+    },
+    {
+      "epoch": 0.27161138678506136,
+      "grad_norm": 0.1620815545320511,
+      "learning_rate": 1.8754491017964075e-05,
+      "loss": 0.3726,
+      "step": 390
+    },
+    {
+      "epoch": 0.27857578131801164,
+      "grad_norm": 0.11439554393291473,
+      "learning_rate": 1.8706586826347305e-05,
+      "loss": 0.3287,
+      "step": 400
+    },
+    {
+      "epoch": 0.285540175850962,
+      "grad_norm": 0.10733096301555634,
+      "learning_rate": 1.865868263473054e-05,
+      "loss": 0.3459,
+      "step": 410
+    },
+    {
+      "epoch": 0.29250457038391225,
+      "grad_norm": 0.12426561117172241,
+      "learning_rate": 1.8610778443113775e-05,
+      "loss": 0.3705,
+      "step": 420
+    },
+    {
+      "epoch": 0.29946896491686253,
+      "grad_norm": 0.1148887649178505,
+      "learning_rate": 1.8562874251497008e-05,
+      "loss": 0.3216,
+      "step": 430
+    },
+    {
+      "epoch": 0.3064333594498128,
+      "grad_norm": 0.10842527449131012,
+      "learning_rate": 1.851497005988024e-05,
+      "loss": 0.3502,
+      "step": 440
+    },
+    {
+      "epoch": 0.31339775398276315,
+      "grad_norm": 0.11736515164375305,
+      "learning_rate": 1.8467065868263475e-05,
+      "loss": 0.3394,
+      "step": 450
+    },
+    {
+      "epoch": 0.32036214851571343,
+      "grad_norm": 0.1381833553314209,
+      "learning_rate": 1.8419161676646708e-05,
+      "loss": 0.3166,
+      "step": 460
+    },
+    {
+      "epoch": 0.3273265430486637,
+      "grad_norm": 0.10973569750785828,
+      "learning_rate": 1.837125748502994e-05,
+      "loss": 0.3245,
+      "step": 470
+    },
+    {
+      "epoch": 0.3328980586750239,
+      "eval_loss": 0.3075329661369324,
+      "eval_runtime": 356.0276,
+      "eval_samples_per_second": 3.396,
+      "eval_steps_per_second": 1.699,
+      "step": 478
+    },
+    {
+      "epoch": 0.334290937581614,
+      "grad_norm": 0.13474580645561218,
+      "learning_rate": 1.8323353293413175e-05,
+      "loss": 0.2914,
+      "step": 480
+    },
+    {
+      "epoch": 0.34125533211456427,
+      "grad_norm": 0.12812362611293793,
+      "learning_rate": 1.827544910179641e-05,
+      "loss": 0.3327,
+      "step": 490
+    },
+    {
+      "epoch": 0.3482197266475146,
+      "grad_norm": 0.11661785840988159,
+      "learning_rate": 1.822754491017964e-05,
+      "loss": 0.3468,
+      "step": 500
+    },
+    {
+      "epoch": 0.3551841211804649,
+      "grad_norm": 0.12990142405033112,
+      "learning_rate": 1.8179640718562878e-05,
+      "loss": 0.3318,
+      "step": 510
+    },
+    {
+      "epoch": 0.36214851571341516,
+      "grad_norm": 0.1309327632188797,
+      "learning_rate": 1.8131736526946108e-05,
+      "loss": 0.2942,
+      "step": 520
+    },
+    {
+      "epoch": 0.36911291024636544,
+      "grad_norm": 0.12246479839086533,
+      "learning_rate": 1.8083832335329345e-05,
+      "loss": 0.3751,
+      "step": 530
+    },
+    {
+      "epoch": 0.3760773047793157,
+      "grad_norm": 0.14134123921394348,
+      "learning_rate": 1.8035928143712575e-05,
+      "loss": 0.3248,
+      "step": 540
+    },
+    {
+      "epoch": 0.38304169931226606,
+      "grad_norm": 0.1433170884847641,
+      "learning_rate": 1.798802395209581e-05,
+      "loss": 0.3344,
+      "step": 550
+    },
+    {
+      "epoch": 0.39000609384521634,
+      "grad_norm": 0.13364745676517487,
+      "learning_rate": 1.7940119760479045e-05,
+      "loss": 0.2928,
+      "step": 560
+    },
+    {
+      "epoch": 0.3969704883781666,
+      "grad_norm": 0.13446496427059174,
+      "learning_rate": 1.7892215568862278e-05,
+      "loss": 0.3421,
+      "step": 570
+    },
+    {
+      "epoch": 0.4039348829111169,
+      "grad_norm": 0.1586250364780426,
+      "learning_rate": 1.784431137724551e-05,
+      "loss": 0.3099,
+      "step": 580
+    },
+    {
+      "epoch": 0.4108992774440672,
+      "grad_norm": 0.1283930391073227,
+      "learning_rate": 1.7796407185628745e-05,
+      "loss": 0.3123,
+      "step": 590
+    },
+    {
+      "epoch": 0.4178636719770175,
+      "grad_norm": 0.14261852204799652,
+      "learning_rate": 1.7748502994011978e-05,
+      "loss": 0.2886,
+      "step": 600
+    },
+    {
+      "epoch": 0.4248280665099678,
+      "grad_norm": 0.13329675793647766,
+      "learning_rate": 1.770059880239521e-05,
+      "loss": 0.3255,
+      "step": 610
+    },
+    {
+      "epoch": 0.4317924610429181,
+      "grad_norm": 0.13880160450935364,
+      "learning_rate": 1.7652694610778445e-05,
+      "loss": 0.326,
+      "step": 620
+    },
+    {
+      "epoch": 0.43875685557586835,
+      "grad_norm": 0.14388258755207062,
+      "learning_rate": 1.7604790419161678e-05,
+      "loss": 0.3269,
+      "step": 630
+    },
+    {
+      "epoch": 0.4457212501088187,
+      "grad_norm": 0.1571163535118103,
+      "learning_rate": 1.755688622754491e-05,
+      "loss": 0.2919,
+      "step": 640
+    },
+    {
+      "epoch": 0.45268564464176897,
+      "grad_norm": 0.15307635068893433,
+      "learning_rate": 1.7508982035928145e-05,
+      "loss": 0.2773,
+      "step": 650
+    },
+    {
+      "epoch": 0.45965003917471925,
+      "grad_norm": 0.1438165158033371,
+      "learning_rate": 1.7461077844311378e-05,
+      "loss": 0.2847,
+      "step": 660
+    },
+    {
+      "epoch": 0.46661443370766953,
+      "grad_norm": 0.15620216727256775,
+      "learning_rate": 1.741317365269461e-05,
+      "loss": 0.29,
+      "step": 670
+    },
+    {
+      "epoch": 0.4735788282406198,
+      "grad_norm": 0.1414933055639267,
+      "learning_rate": 1.7365269461077845e-05,
+      "loss": 0.2754,
+      "step": 680
+    },
+    {
+      "epoch": 0.48054322277357014,
+      "grad_norm": 0.15853798389434814,
+      "learning_rate": 1.7317365269461078e-05,
+      "loss": 0.3062,
+      "step": 690
+    },
+    {
+      "epoch": 0.4875076173065204,
+      "grad_norm": 0.17159980535507202,
+      "learning_rate": 1.726946107784431e-05,
+      "loss": 0.2953,
+      "step": 700
+    },
+    {
+      "epoch": 0.4944720118394707,
+      "grad_norm": 0.16937299072742462,
+      "learning_rate": 1.7221556886227548e-05,
+      "loss": 0.2895,
+      "step": 710
+    },
+    {
+      "epoch": 0.501436406372421,
+      "grad_norm": 0.1455833464860916,
+      "learning_rate": 1.717365269461078e-05,
+      "loss": 0.2912,
+      "step": 720
+    },
+    {
+      "epoch": 0.5084008009053713,
+      "grad_norm": 0.15824872255325317,
+      "learning_rate": 1.7125748502994015e-05,
+      "loss": 0.3012,
+      "step": 730
+    },
+    {
+      "epoch": 0.5153651954383216,
+      "grad_norm": 0.1499851644039154,
+      "learning_rate": 1.7077844311377248e-05,
+      "loss": 0.263,
+      "step": 740
+    },
+    {
+      "epoch": 0.5223295899712719,
+      "grad_norm": 0.1611323356628418,
+      "learning_rate": 1.702994011976048e-05,
+      "loss": 0.2595,
+      "step": 750
+    },
+    {
+      "epoch": 0.5292939845042222,
+      "grad_norm": 0.16144199669361115,
+      "learning_rate": 1.6982035928143714e-05,
+      "loss": 0.2939,
+      "step": 760
+    },
+    {
+      "epoch": 0.5362583790371724,
+      "grad_norm": 0.13433364033699036,
+      "learning_rate": 1.6934131736526948e-05,
+      "loss": 0.2817,
+      "step": 770
+    },
+    {
+      "epoch": 0.5432227735701227,
+      "grad_norm": 0.15188777446746826,
+      "learning_rate": 1.688622754491018e-05,
+      "loss": 0.2804,
+      "step": 780
+    },
+    {
+      "epoch": 0.550187168103073,
+      "grad_norm": 0.1530751883983612,
+      "learning_rate": 1.6838323353293414e-05,
+      "loss": 0.298,
+      "step": 790
+    },
+    {
+      "epoch": 0.5571515626360233,
+      "grad_norm": 0.14582635462284088,
+      "learning_rate": 1.6790419161676648e-05,
+      "loss": 0.3177,
+      "step": 800
+    },
+    {
+      "epoch": 0.5641159571689737,
+      "grad_norm": 0.174430251121521,
+      "learning_rate": 1.674251497005988e-05,
+      "loss": 0.2624,
+      "step": 810
+    },
+    {
+      "epoch": 0.571080351701924,
+      "grad_norm": 0.1558169573545456,
+      "learning_rate": 1.6694610778443114e-05,
+      "loss": 0.2631,
+      "step": 820
+    },
+    {
+      "epoch": 0.5780447462348742,
+      "grad_norm": 0.17108450829982758,
+      "learning_rate": 1.6646706586826348e-05,
+      "loss": 0.2903,
+      "step": 830
+    },
+    {
+      "epoch": 0.5850091407678245,
+      "grad_norm": 0.16796821355819702,
+      "learning_rate": 1.659880239520958e-05,
+      "loss": 0.2928,
+      "step": 840
+    },
+    {
+      "epoch": 0.5919735353007748,
+      "grad_norm": 0.16556940972805023,
+      "learning_rate": 1.6550898203592814e-05,
+      "loss": 0.2882,
+      "step": 850
+    },
+    {
+      "epoch": 0.5989379298337251,
+      "grad_norm": 0.1716109961271286,
+      "learning_rate": 1.650299401197605e-05,
+      "loss": 0.2814,
+      "step": 860
+    },
+    {
+      "epoch": 0.6059023243666753,
+      "grad_norm": 0.1695253700017929,
+      "learning_rate": 1.645508982035928e-05,
+      "loss": 0.2909,
+      "step": 870
+    },
+    {
+      "epoch": 0.6128667188996256,
+      "grad_norm": 0.1896178424358368,
+      "learning_rate": 1.6407185628742518e-05,
+      "loss": 0.2909,
+      "step": 880
+    },
+    {
+      "epoch": 0.6198311134325759,
+      "grad_norm": 0.1624131202697754,
+      "learning_rate": 1.6359281437125748e-05,
+      "loss": 0.2672,
+      "step": 890
+    },
+    {
+      "epoch": 0.6267955079655263,
+      "grad_norm": 0.16821731626987457,
+      "learning_rate": 1.6311377245508984e-05,
+      "loss": 0.2626,
+      "step": 900
+    },
+    {
+      "epoch": 0.6337599024984766,
+      "grad_norm": 0.17026785016059875,
+      "learning_rate": 1.6263473053892214e-05,
+      "loss": 0.2735,
+      "step": 910
+    },
+    {
+      "epoch": 0.6407242970314269,
+      "grad_norm": 0.19429761171340942,
+      "learning_rate": 1.621556886227545e-05,
+      "loss": 0.2605,
+      "step": 920
+    },
+    {
+      "epoch": 0.6476886915643771,
+      "grad_norm": 0.16931040585041046,
+      "learning_rate": 1.6167664670658684e-05,
+      "loss": 0.2722,
+      "step": 930
+    },
+    {
+      "epoch": 0.6546530860973274,
+      "grad_norm": 0.20588335394859314,
+      "learning_rate": 1.6119760479041918e-05,
+      "loss": 0.301,
+      "step": 940
+    },
+    {
+      "epoch": 0.6616174806302777,
+      "grad_norm": 0.16699948906898499,
+      "learning_rate": 1.607185628742515e-05,
+      "loss": 0.2646,
+      "step": 950
+    },
+    {
+      "epoch": 0.6657961173500478,
+      "eval_loss": 0.2465677410364151,
+      "eval_runtime": 356.1016,
+      "eval_samples_per_second": 3.395,
+      "eval_steps_per_second": 1.699,
+      "step": 956
+    },
+    {
+      "epoch": 0.668581875163228,
+      "grad_norm": 0.20734217762947083,
+      "learning_rate": 1.6023952095808384e-05,
+      "loss": 0.2476,
+      "step": 960
+    },
+    {
+      "epoch": 0.6755462696961783,
+      "grad_norm": 0.21317210793495178,
+      "learning_rate": 1.5976047904191618e-05,
+      "loss": 0.2434,
+      "step": 970
+    },
+    {
+      "epoch": 0.6825106642291285,
+      "grad_norm": 0.19125515222549438,
+      "learning_rate": 1.592814371257485e-05,
+      "loss": 0.2823,
+      "step": 980
+    },
+    {
+      "epoch": 0.6894750587620788,
+      "grad_norm": 0.1651381254196167,
+      "learning_rate": 1.5880239520958084e-05,
+      "loss": 0.2769,
+      "step": 990
+    },
+    {
+      "epoch": 0.6964394532950292,
+      "grad_norm": 0.18061324954032898,
+      "learning_rate": 1.583233532934132e-05,
+      "loss": 0.2788,
+      "step": 1000
+    },
+    {
+      "epoch": 0.7034038478279795,
+      "grad_norm": 0.17788180708885193,
+      "learning_rate": 1.578443113772455e-05,
+      "loss": 0.2596,
+      "step": 1010
+    },
+    {
+      "epoch": 0.7103682423609298,
+      "grad_norm": 0.19635842740535736,
+      "learning_rate": 1.5736526946107788e-05,
+      "loss": 0.2391,
+      "step": 1020
+    },
+    {
+      "epoch": 0.71733263689388,
+      "grad_norm": 0.18000468611717224,
+      "learning_rate": 1.5688622754491018e-05,
+      "loss": 0.2589,
+      "step": 1030
+    },
+    {
+      "epoch": 0.7242970314268303,
+      "grad_norm": 0.18016283214092255,
+      "learning_rate": 1.5640718562874254e-05,
+      "loss": 0.2673,
+      "step": 1040
+    },
+    {
+      "epoch": 0.7312614259597806,
+      "grad_norm": 0.2085445076227188,
+      "learning_rate": 1.5592814371257484e-05,
+      "loss": 0.295,
+      "step": 1050
+    },
+    {
+      "epoch": 0.7382258204927309,
+      "grad_norm": 0.1937682181596756,
+      "learning_rate": 1.554491017964072e-05,
+      "loss": 0.2741,
+      "step": 1060
+    },
+    {
+      "epoch": 0.7451902150256812,
+      "grad_norm": 0.18496188521385193,
+      "learning_rate": 1.5497005988023954e-05,
+      "loss": 0.2485,
+      "step": 1070
+    },
+    {
+      "epoch": 0.7521546095586314,
+      "grad_norm": 0.21024686098098755,
+      "learning_rate": 1.5449101796407188e-05,
+      "loss": 0.2569,
+      "step": 1080
+    },
+    {
+      "epoch": 0.7591190040915818,
+      "grad_norm": 0.1960270255804062,
+      "learning_rate": 1.540119760479042e-05,
+      "loss": 0.2335,
+      "step": 1090
+    },
+    {
+      "epoch": 0.7660833986245321,
+      "grad_norm": 0.19557876884937286,
+      "learning_rate": 1.5353293413173654e-05,
+      "loss": 0.279,
+      "step": 1100
+    },
+    {
+      "epoch": 0.7730477931574824,
+      "grad_norm": 0.17300210893154144,
+      "learning_rate": 1.5305389221556888e-05,
+      "loss": 0.249,
+      "step": 1110
+    },
+    {
+      "epoch": 0.7800121876904327,
+      "grad_norm": 0.1768539994955063,
+      "learning_rate": 1.5257485029940121e-05,
+      "loss": 0.2415,
+      "step": 1120
+    },
+    {
+      "epoch": 0.786976582223383,
+      "grad_norm": 0.20632915198802948,
+      "learning_rate": 1.5209580838323354e-05,
+      "loss": 0.2312,
+      "step": 1130
+    },
+    {
+      "epoch": 0.7939409767563332,
+      "grad_norm": 0.1948348730802536,
+      "learning_rate": 1.516167664670659e-05,
+      "loss": 0.2415,
+      "step": 1140
+    },
+    {
+      "epoch": 0.8009053712892835,
+      "grad_norm": 0.16288821399211884,
+      "learning_rate": 1.511377245508982e-05,
+      "loss": 0.2571,
+      "step": 1150
+    },
+    {
+      "epoch": 0.8078697658222338,
+      "grad_norm": 0.21365194022655487,
+      "learning_rate": 1.5065868263473056e-05,
+      "loss": 0.2594,
+      "step": 1160
+    },
+    {
+      "epoch": 0.8148341603551841,
+      "grad_norm": 0.20960865914821625,
+      "learning_rate": 1.5017964071856287e-05,
+      "loss": 0.2537,
+      "step": 1170
+    },
+    {
+      "epoch": 0.8217985548881344,
+      "grad_norm": 0.20458345115184784,
+      "learning_rate": 1.4970059880239522e-05,
+      "loss": 0.2675,
+      "step": 1180
+    },
+    {
+      "epoch": 0.8287629494210847,
+      "grad_norm": 0.22388549149036407,
+      "learning_rate": 1.4922155688622754e-05,
+      "loss": 0.2525,
+      "step": 1190
+    },
+    {
+      "epoch": 0.835727343954035,
+      "grad_norm": 0.2139691859483719,
+      "learning_rate": 1.4874251497005989e-05,
+      "loss": 0.251,
+      "step": 1200
+    },
+    {
+      "epoch": 0.8426917384869853,
+      "grad_norm": 0.18761980533599854,
+      "learning_rate": 1.4826347305389224e-05,
+      "loss": 0.2328,
+      "step": 1210
+    },
+    {
+      "epoch": 0.8496561330199356,
+      "grad_norm": 0.183973029255867,
+      "learning_rate": 1.4778443113772456e-05,
+      "loss": 0.2239,
+      "step": 1220
+    },
+    {
+      "epoch": 0.8566205275528859,
+      "grad_norm": 0.23718492686748505,
+      "learning_rate": 1.473053892215569e-05,
+      "loss": 0.2423,
+      "step": 1230
+    },
+    {
+      "epoch": 0.8635849220858361,
+      "grad_norm": 0.23875045776367188,
+      "learning_rate": 1.4682634730538922e-05,
+      "loss": 0.2365,
+      "step": 1240
+    },
+    {
+      "epoch": 0.8705493166187864,
+      "grad_norm": 0.21570812165737152,
+      "learning_rate": 1.4634730538922157e-05,
+      "loss": 0.268,
+      "step": 1250
+    },
+    {
+      "epoch": 0.8775137111517367,
+      "grad_norm": 0.20839928090572357,
+      "learning_rate": 1.4586826347305389e-05,
+      "loss": 0.2502,
+      "step": 1260
+    },
+    {
+      "epoch": 0.884478105684687,
+      "grad_norm": 0.21605758368968964,
+      "learning_rate": 1.4538922155688624e-05,
+      "loss": 0.2365,
+      "step": 1270
+    },
+    {
+      "epoch": 0.8914425002176374,
+      "grad_norm": 0.23300880193710327,
+      "learning_rate": 1.4491017964071859e-05,
+      "loss": 0.2404,
+      "step": 1280
+    },
+    {
+      "epoch": 0.8984068947505877,
+      "grad_norm": 0.21796733140945435,
+      "learning_rate": 1.444311377245509e-05,
+      "loss": 0.2704,
+      "step": 1290
+    },
+    {
+      "epoch": 0.9053712892835379,
+      "grad_norm": 0.25555598735809326,
+      "learning_rate": 1.4395209580838326e-05,
+      "loss": 0.2674,
+      "step": 1300
+    },
+    {
+      "epoch": 0.9123356838164882,
+      "grad_norm": 0.22284413874149323,
+      "learning_rate": 1.4347305389221557e-05,
+      "loss": 0.2471,
+      "step": 1310
+    },
+    {
+      "epoch": 0.9193000783494385,
+      "grad_norm": 0.22234566509723663,
+      "learning_rate": 1.4299401197604792e-05,
+      "loss": 0.2529,
+      "step": 1320
+    },
+    {
+      "epoch": 0.9262644728823888,
+      "grad_norm": 0.221883624792099,
+      "learning_rate": 1.4251497005988024e-05,
+      "loss": 0.1932,
+      "step": 1330
+    },
+    {
+      "epoch": 0.9332288674153391,
+      "grad_norm": 0.22873520851135254,
+      "learning_rate": 1.4203592814371259e-05,
+      "loss": 0.2412,
+      "step": 1340
+    },
+    {
+      "epoch": 0.9401932619482893,
+      "grad_norm": 0.21307863295078278,
+      "learning_rate": 1.4155688622754492e-05,
+      "loss": 0.2384,
+      "step": 1350
+    },
+    {
+      "epoch": 0.9471576564812396,
+      "grad_norm": 0.21296364068984985,
+      "learning_rate": 1.4107784431137726e-05,
+      "loss": 0.2318,
+      "step": 1360
+    },
+    {
+      "epoch": 0.95412205101419,
+      "grad_norm": 0.21060685813426971,
+      "learning_rate": 1.405988023952096e-05,
+      "loss": 0.2323,
+      "step": 1370
+    },
+    {
+      "epoch": 0.9610864455471403,
+      "grad_norm": 0.23930418491363525,
+      "learning_rate": 1.4011976047904192e-05,
+      "loss": 0.2398,
+      "step": 1380
+    },
+    {
+      "epoch": 0.9680508400800906,
+      "grad_norm": 0.22837185859680176,
+      "learning_rate": 1.3964071856287427e-05,
+      "loss": 0.233,
+      "step": 1390
+    },
+    {
+      "epoch": 0.9750152346130408,
+      "grad_norm": 0.21798169612884521,
+      "learning_rate": 1.3916167664670659e-05,
+      "loss": 0.2161,
+      "step": 1400
+    },
+    {
+      "epoch": 0.9819796291459911,
+      "grad_norm": 0.2846814692020416,
+      "learning_rate": 1.3868263473053894e-05,
+      "loss": 0.2301,
+      "step": 1410
+    },
+    {
+      "epoch": 0.9889440236789414,
+      "grad_norm": 0.22030217945575714,
+      "learning_rate": 1.3820359281437127e-05,
+      "loss": 0.2017,
+      "step": 1420
+    },
+    {
+      "epoch": 0.9959084182118917,
+      "grad_norm": 0.25554022192955017,
+      "learning_rate": 1.377245508982036e-05,
+      "loss": 0.2296,
+      "step": 1430
+    },
+    {
+      "epoch": 0.9986941760250718,
+      "eval_loss": 0.21637588739395142,
+      "eval_runtime": 356.0721,
+      "eval_samples_per_second": 3.395,
+      "eval_steps_per_second": 1.699,
+      "step": 1434
+    },
+    {
+      "epoch": 1.0030469226081657,
+      "grad_norm": 0.2862723469734192,
+      "learning_rate": 1.3724550898203594e-05,
+      "loss": 0.2538,
+      "step": 1440
+    },
+    {
+      "epoch": 1.010011317141116,
+      "grad_norm": 0.2589089870452881,
+      "learning_rate": 1.3676646706586827e-05,
+      "loss": 0.2196,
+      "step": 1450
+    },
+    {
+      "epoch": 1.0169757116740663,
+      "grad_norm": 0.27893784642219543,
+      "learning_rate": 1.362874251497006e-05,
+      "loss": 0.1935,
+      "step": 1460
+    },
+    {
+      "epoch": 1.0239401062070166,
+      "grad_norm": 0.2530602514743805,
+      "learning_rate": 1.3580838323353294e-05,
+      "loss": 0.2248,
+      "step": 1470
+    },
+    {
+      "epoch": 1.0309045007399669,
+      "grad_norm": 0.24902446568012238,
+      "learning_rate": 1.3532934131736529e-05,
+      "loss": 0.224,
+      "step": 1480
+    },
+    {
+      "epoch": 1.0378688952729171,
+      "grad_norm": 0.27132269740104675,
+      "learning_rate": 1.3485029940119762e-05,
+      "loss": 0.2208,
+      "step": 1490
+    },
+    {
+      "epoch": 1.0448332898058674,
+      "grad_norm": 0.2570977807044983,
+      "learning_rate": 1.3437125748502996e-05,
+      "loss": 0.2103,
+      "step": 1500
+    },
+    {
+      "epoch": 1.0517976843388177,
+      "grad_norm": 0.2994561493396759,
+      "learning_rate": 1.3389221556886229e-05,
+      "loss": 0.2243,
+      "step": 1510
+    },
+    {
+      "epoch": 1.058762078871768,
+      "grad_norm": 0.22994251549243927,
+      "learning_rate": 1.3341317365269462e-05,
+      "loss": 0.2217,
+      "step": 1520
+    },
+    {
+      "epoch": 1.0657264734047183,
+      "grad_norm": 0.22706159949302673,
+      "learning_rate": 1.3293413173652696e-05,
+      "loss": 0.2219,
+      "step": 1530
+    },
+    {
+      "epoch": 1.0726908679376688,
+      "grad_norm": 0.26584964990615845,
+      "learning_rate": 1.3245508982035929e-05,
+      "loss": 0.2329,
+      "step": 1540
+    },
+    {
+      "epoch": 1.079655262470619,
+      "grad_norm": 0.2549090087413788,
+      "learning_rate": 1.3197604790419162e-05,
+      "loss": 0.2235,
+      "step": 1550
+    },
+    {
+      "epoch": 1.0866196570035693,
+      "grad_norm": 0.27169135212898254,
+      "learning_rate": 1.3149700598802397e-05,
+      "loss": 0.2332,
+      "step": 1560
+    },
+    {
+      "epoch": 1.0935840515365196,
+      "grad_norm": 0.26733773946762085,
+      "learning_rate": 1.3101796407185629e-05,
+      "loss": 0.2062,
+      "step": 1570
+    },
+    {
+      "epoch": 1.1005484460694699,
+      "grad_norm": 0.3222082853317261,
+      "learning_rate": 1.3053892215568864e-05,
+      "loss": 0.2111,
+      "step": 1580
+    },
+    {
+      "epoch": 1.1075128406024202,
+      "grad_norm": 0.20403257012367249,
+      "learning_rate": 1.3005988023952097e-05,
+      "loss": 0.226,
+      "step": 1590
+    },
+    {
+      "epoch": 1.1144772351353704,
+      "grad_norm": 0.2658451199531555,
+      "learning_rate": 1.295808383233533e-05,
+      "loss": 0.2013,
+      "step": 1600
+    },
+    {
+      "epoch": 1.1214416296683207,
+      "grad_norm": 0.2719573676586151,
+      "learning_rate": 1.2910179640718564e-05,
+      "loss": 0.2036,
+      "step": 1610
+    },
+    {
+      "epoch": 1.128406024201271,
+      "grad_norm": 0.29076331853866577,
+      "learning_rate": 1.2862275449101797e-05,
+      "loss": 0.2162,
+      "step": 1620
+    },
+    {
+      "epoch": 1.1353704187342213,
+      "grad_norm": 0.2324078530073166,
+      "learning_rate": 1.2814371257485032e-05,
+      "loss": 0.2002,
+      "step": 1630
+    },
+    {
+      "epoch": 1.1423348132671716,
+      "grad_norm": 0.27219468355178833,
+      "learning_rate": 1.2766467065868264e-05,
+      "loss": 0.2238,
+      "step": 1640
+    },
+    {
+      "epoch": 1.1492992078001218,
+      "grad_norm": 0.22101271152496338,
+      "learning_rate": 1.2718562874251499e-05,
+      "loss": 0.1845,
+      "step": 1650
+    },
+    {
+      "epoch": 1.1562636023330721,
+      "grad_norm": 0.25101369619369507,
+      "learning_rate": 1.267065868263473e-05,
+      "loss": 0.2296,
+      "step": 1660
+    },
+    {
+      "epoch": 1.1632279968660224,
+      "grad_norm": 0.25895142555236816,
+      "learning_rate": 1.2622754491017965e-05,
+      "loss": 0.2271,
+      "step": 1670
+    },
+    {
+      "epoch": 1.1701923913989727,
+      "grad_norm": 0.26405826210975647,
+      "learning_rate": 1.2574850299401197e-05,
+      "loss": 0.2159,
+      "step": 1680
+    },
+    {
+      "epoch": 1.177156785931923,
+      "grad_norm": 0.23231615126132965,
+      "learning_rate": 1.2526946107784432e-05,
+      "loss": 0.1916,
+      "step": 1690
+    },
+    {
+      "epoch": 1.1841211804648732,
+      "grad_norm": 0.3211207985877991,
+      "learning_rate": 1.2479041916167665e-05,
+      "loss": 0.2154,
+      "step": 1700
+    },
+    {
+      "epoch": 1.1910855749978237,
+      "grad_norm": 0.2948794662952423,
+      "learning_rate": 1.2431137724550899e-05,
+      "loss": 0.1999,
+      "step": 1710
+    },
+    {
+      "epoch": 1.198049969530774,
+      "grad_norm": 0.28081706166267395,
+      "learning_rate": 1.2383233532934134e-05,
+      "loss": 0.2301,
+      "step": 1720
+    },
+    {
+      "epoch": 1.2050143640637243,
+      "grad_norm": 0.296283096075058,
+      "learning_rate": 1.2335329341317365e-05,
+      "loss": 0.2164,
+      "step": 1730
+    },
+    {
+      "epoch": 1.2119787585966746,
+      "grad_norm": 0.29658278822898865,
+      "learning_rate": 1.22874251497006e-05,
+      "loss": 0.1916,
+      "step": 1740
+    },
+    {
+      "epoch": 1.2189431531296249,
+      "grad_norm": 0.2917250990867615,
+      "learning_rate": 1.2239520958083832e-05,
+      "loss": 0.2045,
+      "step": 1750
+    },
+    {
+      "epoch": 1.2259075476625751,
+      "grad_norm": 0.35840919613838196,
+      "learning_rate": 1.2191616766467067e-05,
+      "loss": 0.1965,
+      "step": 1760
+    },
+    {
+      "epoch": 1.2328719421955254,
+      "grad_norm": 0.2819244861602783,
+      "learning_rate": 1.2143712574850299e-05,
+      "loss": 0.2032,
+      "step": 1770
+    },
+    {
+      "epoch": 1.2398363367284757,
+      "grad_norm": 0.3518809974193573,
+      "learning_rate": 1.2095808383233534e-05,
+      "loss": 0.2043,
+      "step": 1780
+    },
+    {
+      "epoch": 1.246800731261426,
+      "grad_norm": 0.3232385516166687,
+      "learning_rate": 1.2047904191616769e-05,
+      "loss": 0.23,
+      "step": 1790
+    },
+    {
+      "epoch": 1.2537651257943763,
+      "grad_norm": 0.23131519556045532,
+      "learning_rate": 1.2e-05,
+      "loss": 0.2106,
+      "step": 1800
+    },
+    {
+      "epoch": 1.2607295203273265,
+      "grad_norm": 0.30877017974853516,
+      "learning_rate": 1.1952095808383235e-05,
+      "loss": 0.2088,
+      "step": 1810
+    },
+    {
+      "epoch": 1.2676939148602768,
+      "grad_norm": 0.36684754490852356,
+      "learning_rate": 1.1904191616766467e-05,
+      "loss": 0.2024,
+      "step": 1820
+    },
+    {
+      "epoch": 1.274658309393227,
+      "grad_norm": 0.21124888956546783,
+      "learning_rate": 1.1856287425149702e-05,
+      "loss": 0.1929,
+      "step": 1830
+    },
+    {
+      "epoch": 1.2816227039261774,
+      "grad_norm": 0.2815339267253876,
+      "learning_rate": 1.1808383233532934e-05,
+      "loss": 0.1973,
+      "step": 1840
+    },
+    {
+      "epoch": 1.2885870984591277,
+      "grad_norm": 0.3017849922180176,
+      "learning_rate": 1.1760479041916169e-05,
+      "loss": 0.233,
+      "step": 1850
+    },
+    {
+      "epoch": 1.295551492992078,
+      "grad_norm": 0.3456547260284424,
+      "learning_rate": 1.1712574850299404e-05,
+      "loss": 0.1769,
+      "step": 1860
+    },
+    {
+      "epoch": 1.3025158875250282,
+      "grad_norm": 0.28210344910621643,
+      "learning_rate": 1.1664670658682635e-05,
+      "loss": 0.204,
+      "step": 1870
+    },
+    {
+      "epoch": 1.3094802820579785,
+      "grad_norm": 0.3701432943344116,
+      "learning_rate": 1.161676646706587e-05,
+      "loss": 0.1923,
+      "step": 1880
+    },
+    {
+      "epoch": 1.3164446765909288,
+      "grad_norm": 0.3656926453113556,
+      "learning_rate": 1.1568862275449102e-05,
+      "loss": 0.2136,
+      "step": 1890
+    },
+    {
+      "epoch": 1.323409071123879,
+      "grad_norm": 0.29142090678215027,
+      "learning_rate": 1.1520958083832337e-05,
+      "loss": 0.1957,
+      "step": 1900
+    },
+    {
+      "epoch": 1.3303734656568293,
+      "grad_norm": 0.2799660563468933,
+      "learning_rate": 1.1473053892215569e-05,
+      "loss": 0.2009,
+      "step": 1910
+    },
+    {
+      "epoch": 1.3317663445634196,
+      "eval_loss": 0.19295775890350342,
+      "eval_runtime": 356.0153,
+      "eval_samples_per_second": 3.396,
+      "eval_steps_per_second": 1.699,
+      "step": 1912
+    },
+    {
+      "epoch": 1.3373378601897796,
+      "grad_norm": 0.38173311948776245,
+      "learning_rate": 1.1425149700598804e-05,
+      "loss": 0.2133,
+      "step": 1920
+    },
+    {
+      "epoch": 1.34430225472273,
+      "grad_norm": 0.38059180974960327,
+      "learning_rate": 1.1377245508982037e-05,
+      "loss": 0.203,
+      "step": 1930
+    },
+    {
+      "epoch": 1.3512666492556804,
+      "grad_norm": 0.2743132710456848,
+      "learning_rate": 1.132934131736527e-05,
+      "loss": 0.196,
+      "step": 1940
+    },
+    {
+      "epoch": 1.3582310437886307,
+      "grad_norm": 0.3110567331314087,
+      "learning_rate": 1.1281437125748505e-05,
+      "loss": 0.1868,
+      "step": 1950
+    },
+    {
+      "epoch": 1.365195438321581,
+      "grad_norm": 0.4261178970336914,
+      "learning_rate": 1.1233532934131737e-05,
+      "loss": 0.1952,
+      "step": 1960
+    },
+    {
+      "epoch": 1.3721598328545312,
+      "grad_norm": 0.3890218436717987,
+      "learning_rate": 1.1185628742514972e-05,
+      "loss": 0.1964,
+      "step": 1970
+    },
+    {
+      "epoch": 1.3791242273874815,
+      "grad_norm": 0.3712151050567627,
+      "learning_rate": 1.1137724550898203e-05,
+      "loss": 0.1933,
+      "step": 1980
+    },
+    {
+      "epoch": 1.3860886219204318,
+      "grad_norm": 0.2833101153373718,
+      "learning_rate": 1.1089820359281439e-05,
+      "loss": 0.1862,
+      "step": 1990
+    },
+    {
+      "epoch": 1.393053016453382,
+      "grad_norm": 0.3696097731590271,
+      "learning_rate": 1.1041916167664672e-05,
+      "loss": 0.2142,
+      "step": 2000
+    },
+    {
+      "epoch": 1.4000174109863324,
+      "grad_norm": 0.4144258499145508,
+      "learning_rate": 1.0994011976047905e-05,
+      "loss": 0.1915,
+      "step": 2010
+    },
+    {
+      "epoch": 1.4069818055192826,
+      "grad_norm": 0.28359559178352356,
+      "learning_rate": 1.0946107784431138e-05,
+      "loss": 0.1861,
+      "step": 2020
+    },
+    {
+      "epoch": 1.413946200052233,
+      "grad_norm": 0.327510803937912,
+      "learning_rate": 1.0898203592814372e-05,
+      "loss": 0.1961,
+      "step": 2030
+    },
+    {
+      "epoch": 1.4209105945851832,
+      "grad_norm": 0.44245216250419617,
+      "learning_rate": 1.0850299401197605e-05,
+      "loss": 0.2091,
+      "step": 2040
+    },
+    {
+      "epoch": 1.4278749891181335,
+      "grad_norm": 0.3903510570526123,
+      "learning_rate": 1.0802395209580838e-05,
+      "loss": 0.2058,
+      "step": 2050
+    },
+    {
+      "epoch": 1.4348393836510838,
+      "grad_norm": 0.32461392879486084,
+      "learning_rate": 1.0754491017964073e-05,
+      "loss": 0.1793,
+      "step": 2060
+    },
+    {
+      "epoch": 1.441803778184034,
+      "grad_norm": 0.4412670433521271,
+      "learning_rate": 1.0706586826347307e-05,
+      "loss": 0.1856,
+      "step": 2070
+    },
+    {
+      "epoch": 1.4487681727169845,
+      "grad_norm": 0.2217404693365097,
+      "learning_rate": 1.065868263473054e-05,
+      "loss": 0.1628,
+      "step": 2080
+    },
+    {
+      "epoch": 1.4557325672499348,
+      "grad_norm": 0.3572748899459839,
+      "learning_rate": 1.0610778443113773e-05,
+      "loss": 0.1904,
+      "step": 2090
+    },
+    {
+      "epoch": 1.462696961782885,
+      "grad_norm": 0.23523126542568207,
+      "learning_rate": 1.0562874251497007e-05,
+      "loss": 0.1833,
+      "step": 2100
+    },
+    {
+      "epoch": 1.4696613563158354,
+      "grad_norm": 0.35970646142959595,
+      "learning_rate": 1.051497005988024e-05,
+      "loss": 0.2018,
+      "step": 2110
+    },
+    {
+      "epoch": 1.4766257508487857,
+      "grad_norm": 0.3383442759513855,
+      "learning_rate": 1.0467065868263473e-05,
+      "loss": 0.1986,
+      "step": 2120
+    },
+    {
+      "epoch": 1.483590145381736,
+      "grad_norm": 0.33182451128959656,
+      "learning_rate": 1.0419161676646707e-05,
+      "loss": 0.194,
+      "step": 2130
+    },
+    {
+      "epoch": 1.4905545399146862,
+      "grad_norm": 0.3276427090167999,
+      "learning_rate": 1.0371257485029942e-05,
+      "loss": 0.1945,
+      "step": 2140
+    },
+    {
+      "epoch": 1.4975189344476365,
+      "grad_norm": 0.30687084794044495,
+      "learning_rate": 1.0323353293413173e-05,
+      "loss": 0.1839,
+      "step": 2150
+    },
+    {
+      "epoch": 1.5044833289805868,
+      "grad_norm": 0.3758613169193268,
+      "learning_rate": 1.0275449101796408e-05,
+      "loss": 0.2035,
+      "step": 2160
+    },
+    {
+      "epoch": 1.511447723513537,
+      "grad_norm": 0.34436604380607605,
+      "learning_rate": 1.0227544910179642e-05,
+      "loss": 0.1946,
+      "step": 2170
+    },
+    {
+      "epoch": 1.5184121180464873,
+      "grad_norm": 0.31638774275779724,
+      "learning_rate": 1.0179640718562875e-05,
+      "loss": 0.1761,
+      "step": 2180
+    },
+    {
+      "epoch": 1.5253765125794376,
+      "grad_norm": 0.3935607969760895,
+      "learning_rate": 1.0131736526946108e-05,
+      "loss": 0.2008,
+      "step": 2190
+    },
+    {
+      "epoch": 1.532340907112388,
+      "grad_norm": 0.3549712598323822,
+      "learning_rate": 1.0083832335329342e-05,
+      "loss": 0.1814,
+      "step": 2200
+    },
+    {
+      "epoch": 1.5393053016453382,
+      "grad_norm": 0.32955431938171387,
+      "learning_rate": 1.0035928143712577e-05,
+      "loss": 0.1736,
+      "step": 2210
+    },
+    {
+      "epoch": 1.5462696961782885,
+      "grad_norm": 0.3109774589538574,
+      "learning_rate": 9.988023952095808e-06,
+      "loss": 0.169,
+      "step": 2220
+    },
+    {
+      "epoch": 1.5532340907112387,
+      "grad_norm": 0.4271303117275238,
+      "learning_rate": 9.940119760479042e-06,
+      "loss": 0.1761,
+      "step": 2230
+    },
+    {
+      "epoch": 1.560198485244189,
+      "grad_norm": 0.3970726728439331,
+      "learning_rate": 9.892215568862277e-06,
+      "loss": 0.1897,
+      "step": 2240
+    },
+    {
+      "epoch": 1.5671628797771393,
+      "grad_norm": 0.4251217544078827,
+      "learning_rate": 9.84431137724551e-06,
+      "loss": 0.1771,
+      "step": 2250
+    },
+    {
+      "epoch": 1.5741272743100896,
+      "grad_norm": 0.28587886691093445,
+      "learning_rate": 9.796407185628743e-06,
+      "loss": 0.1699,
+      "step": 2260
+    },
+    {
+      "epoch": 1.5810916688430399,
+      "grad_norm": 0.36768022179603577,
+      "learning_rate": 9.748502994011977e-06,
+      "loss": 0.1933,
+      "step": 2270
+    },
+    {
+      "epoch": 1.5880560633759901,
+      "grad_norm": 0.31612274050712585,
+      "learning_rate": 9.70059880239521e-06,
+      "loss": 0.1944,
+      "step": 2280
+    },
+    {
+      "epoch": 1.5950204579089404,
+      "grad_norm": 0.34787702560424805,
+      "learning_rate": 9.652694610778443e-06,
+      "loss": 0.1758,
+      "step": 2290
+    },
+    {
+      "epoch": 1.6019848524418907,
+      "grad_norm": 0.4170469045639038,
+      "learning_rate": 9.604790419161677e-06,
+      "loss": 0.1749,
+      "step": 2300
+    },
+    {
+      "epoch": 1.608949246974841,
+      "grad_norm": 0.3099391460418701,
+      "learning_rate": 9.556886227544912e-06,
+      "loss": 0.1752,
+      "step": 2310
+    },
+    {
+      "epoch": 1.6159136415077913,
+      "grad_norm": 0.3781738877296448,
+      "learning_rate": 9.508982035928145e-06,
+      "loss": 0.2032,
+      "step": 2320
+    },
+    {
+      "epoch": 1.6228780360407415,
+      "grad_norm": 0.29017606377601624,
+      "learning_rate": 9.461077844311378e-06,
+      "loss": 0.1733,
+      "step": 2330
+    },
+    {
+      "epoch": 1.629842430573692,
+      "grad_norm": 0.45764532685279846,
+      "learning_rate": 9.413173652694612e-06,
+      "loss": 0.1765,
+      "step": 2340
+    },
+    {
+      "epoch": 1.6368068251066423,
+      "grad_norm": 0.4349108040332794,
+      "learning_rate": 9.365269461077845e-06,
+      "loss": 0.178,
+      "step": 2350
+    },
+    {
+      "epoch": 1.6437712196395926,
+      "grad_norm": 0.39599794149398804,
+      "learning_rate": 9.317365269461078e-06,
+      "loss": 0.1792,
+      "step": 2360
+    },
+    {
+      "epoch": 1.6507356141725429,
+      "grad_norm": 0.1882832944393158,
+      "learning_rate": 9.269461077844312e-06,
+      "loss": 0.1871,
+      "step": 2370
+    },
+    {
+      "epoch": 1.6577000087054932,
+      "grad_norm": 0.323946475982666,
+      "learning_rate": 9.221556886227547e-06,
+      "loss": 0.1783,
+      "step": 2380
+    },
+    {
+      "epoch": 1.6646644032384434,
+      "grad_norm": 0.36734339594841003,
+      "learning_rate": 9.17365269461078e-06,
+      "loss": 0.1591,
+      "step": 2390
+    },
+    {
+      "epoch": 1.6646644032384434,
+      "eval_loss": 0.17429304122924805,
+      "eval_runtime": 356.3726,
+      "eval_samples_per_second": 3.393,
+      "eval_steps_per_second": 1.698,
+      "step": 2390
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 4305,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 478,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.1148432793606226e+18,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-2390/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:956bd3fe04334201160cff0d2e3faf62e32c6c8f709339d63e01a7d4b73b24d0
+size 5560

checkpoint-2868/README.md

@@ -0,0 +1,209 @@
+---
+base_model: Salesforce/Llama-xLAM-2-8b-fc-r
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:Salesforce/Llama-xLAM-2-8b-fc-r
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

checkpoint-2868/adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Salesforce/Llama-xLAM-2-8b-fc-r",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "o_proj",
+    "down_proj",
+    "k_proj",
+    "up_proj",
+    "v_proj",
+    "gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-2868/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ec9f0d863dc3abd8902dd4ed843c361bd23b39006098c4932637619ebc429f7
+size 671149168

checkpoint-2868/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20a1cc3180b14b945a606fce84ee452c58beac7ee876f473622cc297c6b92e9b
+size 1342555602

checkpoint-2868/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1ff15f0e116f9cdd6bd365aee046c41884d38316eafb16ee442c6cfc286e73c
+size 14244

checkpoint-2868/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2459b29e12012943fb8a735fbcda66f1d5e11ec6991497fe73fc6eb956e320d3
+size 1064

checkpoint-2868/special_tokens_map.json

@@ -0,0 +1,27 @@
+{
+  "additional_special_tokens": [
+    "<|eot_id|>",
+    "<|eom_id|>"
+  ],
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-2868/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a55c1a4c5e3af7f2fb2bc0cd245a09dabd742dc24e7cb3741db1e11c7fe1a52
+size 17210019

checkpoint-2868/tokenizer_config.json

@@ -0,0 +1,2070 @@
+{
+  "added_tokens_decoder": {
+    "128000": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128003": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "<|finetune_right_pad_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "<|start_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "<|end_header_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "<|eom_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "<|python_tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "<|reserved_special_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "<|reserved_special_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "<|reserved_special_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "<|reserved_special_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "<|reserved_special_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "<|reserved_special_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "<|reserved_special_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "<|reserved_special_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "<|reserved_special_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128105": {
+      "content": "<|reserved_special_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128106": {
+      "content": "<|reserved_special_token_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128107": {
+      "content": "<|reserved_special_token_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128108": {
+      "content": "<|reserved_special_token_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128109": {
+      "content": "<|reserved_special_token_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128110": {
+      "content": "<|reserved_special_token_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128111": {
+      "content": "<|reserved_special_token_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128112": {
+      "content": "<|reserved_special_token_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128113": {
+      "content": "<|reserved_special_token_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128114": {
+      "content": "<|reserved_special_token_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128115": {
+      "content": "<|reserved_special_token_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128116": {
+      "content": "<|reserved_special_token_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128117": {
+      "content": "<|reserved_special_token_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128118": {
+      "content": "<|reserved_special_token_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128119": {
+      "content": "<|reserved_special_token_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128120": {
+      "content": "<|reserved_special_token_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128121": {
+      "content": "<|reserved_special_token_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128122": {
+      "content": "<|reserved_special_token_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128123": {
+      "content": "<|reserved_special_token_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128124": {
+      "content": "<|reserved_special_token_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128125": {
+      "content": "<|reserved_special_token_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128126": {
+      "content": "<|reserved_special_token_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128127": {
+      "content": "<|reserved_special_token_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128128": {
+      "content": "<|reserved_special_token_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128129": {
+      "content": "<|reserved_special_token_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128130": {
+      "content": "<|reserved_special_token_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128131": {
+      "content": "<|reserved_special_token_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128132": {
+      "content": "<|reserved_special_token_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128133": {
+      "content": "<|reserved_special_token_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128134": {
+      "content": "<|reserved_special_token_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128135": {
+      "content": "<|reserved_special_token_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128136": {
+      "content": "<|reserved_special_token_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128137": {
+      "content": "<|reserved_special_token_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128138": {
+      "content": "<|reserved_special_token_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128139": {
+      "content": "<|reserved_special_token_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128140": {
+      "content": "<|reserved_special_token_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128141": {
+      "content": "<|reserved_special_token_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128142": {
+      "content": "<|reserved_special_token_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128143": {
+      "content": "<|reserved_special_token_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128144": {
+      "content": "<|reserved_special_token_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128145": {
+      "content": "<|reserved_special_token_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128146": {
+      "content": "<|reserved_special_token_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128147": {
+      "content": "<|reserved_special_token_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128148": {
+      "content": "<|reserved_special_token_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128149": {
+      "content": "<|reserved_special_token_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128150": {
+      "content": "<|reserved_special_token_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128151": {
+      "content": "<|reserved_special_token_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128152": {
+      "content": "<|reserved_special_token_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128153": {
+      "content": "<|reserved_special_token_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128154": {
+      "content": "<|reserved_special_token_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128155": {
+      "content": "<|reserved_special_token_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128156": {
+      "content": "<|reserved_special_token_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128157": {
+      "content": "<|reserved_special_token_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128158": {
+      "content": "<|reserved_special_token_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128159": {
+      "content": "<|reserved_special_token_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128160": {
+      "content": "<|reserved_special_token_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128161": {
+      "content": "<|reserved_special_token_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128162": {
+      "content": "<|reserved_special_token_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128163": {
+      "content": "<|reserved_special_token_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128164": {
+      "content": "<|reserved_special_token_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128165": {
+      "content": "<|reserved_special_token_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128166": {
+      "content": "<|reserved_special_token_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128167": {
+      "content": "<|reserved_special_token_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128168": {
+      "content": "<|reserved_special_token_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128169": {
+      "content": "<|reserved_special_token_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128170": {
+      "content": "<|reserved_special_token_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128171": {
+      "content": "<|reserved_special_token_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128172": {
+      "content": "<|reserved_special_token_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128173": {
+      "content": "<|reserved_special_token_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128174": {
+      "content": "<|reserved_special_token_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128175": {
+      "content": "<|reserved_special_token_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128176": {
+      "content": "<|reserved_special_token_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128177": {
+      "content": "<|reserved_special_token_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128178": {
+      "content": "<|reserved_special_token_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128179": {
+      "content": "<|reserved_special_token_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128180": {
+      "content": "<|reserved_special_token_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128181": {
+      "content": "<|reserved_special_token_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128182": {
+      "content": "<|reserved_special_token_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128183": {
+      "content": "<|reserved_special_token_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128184": {
+      "content": "<|reserved_special_token_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128185": {
+      "content": "<|reserved_special_token_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128186": {
+      "content": "<|reserved_special_token_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128187": {
+      "content": "<|reserved_special_token_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128188": {
+      "content": "<|reserved_special_token_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128189": {
+      "content": "<|reserved_special_token_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128190": {
+      "content": "<|reserved_special_token_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128191": {
+      "content": "<|reserved_special_token_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128192": {
+      "content": "<|reserved_special_token_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128193": {
+      "content": "<|reserved_special_token_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128194": {
+      "content": "<|reserved_special_token_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128195": {
+      "content": "<|reserved_special_token_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128196": {
+      "content": "<|reserved_special_token_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128197": {
+      "content": "<|reserved_special_token_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128198": {
+      "content": "<|reserved_special_token_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128199": {
+      "content": "<|reserved_special_token_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128200": {
+      "content": "<|reserved_special_token_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128201": {
+      "content": "<|reserved_special_token_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128202": {
+      "content": "<|reserved_special_token_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128203": {
+      "content": "<|reserved_special_token_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128204": {
+      "content": "<|reserved_special_token_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128205": {
+      "content": "<|reserved_special_token_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128206": {
+      "content": "<|reserved_special_token_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128207": {
+      "content": "<|reserved_special_token_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128208": {
+      "content": "<|reserved_special_token_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128209": {
+      "content": "<|reserved_special_token_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128210": {
+      "content": "<|reserved_special_token_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128211": {
+      "content": "<|reserved_special_token_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128212": {
+      "content": "<|reserved_special_token_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128213": {
+      "content": "<|reserved_special_token_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128214": {
+      "content": "<|reserved_special_token_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128215": {
+      "content": "<|reserved_special_token_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128216": {
+      "content": "<|reserved_special_token_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128217": {
+      "content": "<|reserved_special_token_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128218": {
+      "content": "<|reserved_special_token_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128219": {
+      "content": "<|reserved_special_token_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128220": {
+      "content": "<|reserved_special_token_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128221": {
+      "content": "<|reserved_special_token_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128222": {
+      "content": "<|reserved_special_token_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128223": {
+      "content": "<|reserved_special_token_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128224": {
+      "content": "<|reserved_special_token_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128225": {
+      "content": "<|reserved_special_token_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128226": {
+      "content": "<|reserved_special_token_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128227": {
+      "content": "<|reserved_special_token_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128228": {
+      "content": "<|reserved_special_token_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128229": {
+      "content": "<|reserved_special_token_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128230": {
+      "content": "<|reserved_special_token_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128231": {
+      "content": "<|reserved_special_token_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128232": {
+      "content": "<|reserved_special_token_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128233": {
+      "content": "<|reserved_special_token_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128234": {
+      "content": "<|reserved_special_token_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128235": {
+      "content": "<|reserved_special_token_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128236": {
+      "content": "<|reserved_special_token_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128237": {
+      "content": "<|reserved_special_token_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128238": {
+      "content": "<|reserved_special_token_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128239": {
+      "content": "<|reserved_special_token_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128240": {
+      "content": "<|reserved_special_token_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128241": {
+      "content": "<|reserved_special_token_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128242": {
+      "content": "<|reserved_special_token_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128243": {
+      "content": "<|reserved_special_token_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128244": {
+      "content": "<|reserved_special_token_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128245": {
+      "content": "<|reserved_special_token_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128246": {
+      "content": "<|reserved_special_token_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128247": {
+      "content": "<|reserved_special_token_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128248": {
+      "content": "<|reserved_special_token_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128249": {
+      "content": "<|reserved_special_token_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128250": {
+      "content": "<|reserved_special_token_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128251": {
+      "content": "<|reserved_special_token_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128252": {
+      "content": "<|reserved_special_token_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128253": {
+      "content": "<|reserved_special_token_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128254": {
+      "content": "<|reserved_special_token_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128255": {
+      "content": "<|reserved_special_token_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|eot_id|>",
+    "<|eom_id|>"
+  ],
+  "bos_token": "<|begin_of_text|>",
+  "chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n    {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n    {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not tools is defined %}\n    {%- set tools = none %}\n{%- endif %}\n\n{#- Extract system message #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if messages[0]['role'] == 'system' %}\n    {%- set system_message = messages[0]['content'] | trim %}\n    {%- set messages = messages[1:] %}\n    {{- system_message + \"\\n\" }}\n{%- else %}\n    {%- set system_message = \"You are a helpful assistant that can use tools. You are developed by Salesforce xLAM team.\" %}\n    {% set format_instruction %}You have access to a set of tools. When using tools, make calls in a single JSON array: \n\n[{\"name\": \"tool_call_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}}, ... (additional parallel tool calls as needed)]\n\nIf no tool is suitable, state that explicitly. If the user's input lacks required parameters, ask for clarification. Do not interpret or respond until tool results are returned. Once they are available, process them or make additional calls if needed. For tasks that don't require tools, such as casual conversation or general advice, respond directly in plain text. The available tools are:{% endset %}\n    {{- system_message + \"\\n\" }}\n    {%- if tools is not none %}\n        {{- format_instruction + \"\\n\\n\" }}\n    {%- endif %}\n{%- endif %}\n\n\n{%- if tools is not none %}\n    {%- for t in tools %}\n        {{- t | tojson(indent=4) }}\n        {{- \"\\n\\n\" }}\n    {%- endfor %}\n{%- endif %}\n{{- \"<|eot_id|>\" }}\n\n{%- for message in messages %}\n    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n    {%- elif 'tool_calls' in message %}\n        {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n        {%- if message['tool_calls'] %}\n            {{- \"[\" }}\n            {%- for tool_call_function in message.tool_calls %}\n                {%- set tool_call = tool_call_function.function %}\n                {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n                {{- '\"arguments\": ' }}\n                {{- tool_call.arguments | tojson }}\n                {{- \"}\" }}\n                {%- if not loop.last %}\n                    {{- \", \" }}\n                {%- endif %}\n            {%- endfor %}\n            {{- \"]\" }}\n            {{- \"<|eot_id|>\" }}\n        {%- elif message['content'] %}\n            {{- message['content'] | trim + '<|eot_id|>' }}\n        {%- else %}\n            {{- \"[]\\n\" + '<|eot_id|>' }}\n        {%- endif %}\n    {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n        {{- \"<|start_header_id|>\" + \"ipython\" + \"<|end_header_id|>\\n\\n\" }}\n        {%- set content = message[\"content\"] %}\n        {%- if content is mapping or (content is iterable and content is not string) %}\n            {{- content | tojson }}\n        {%- else %}\n            {{- content }}\n        {%- endif %}\n        {{- \"<|eot_id|>\" }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 16384,
+  "pad_token": "<|eot_id|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}

checkpoint-2868/trainer_state.json

@@ -0,0 +1,2083 @@
+{
+  "best_metric": 0.15988580882549286,
+  "best_model_checkpoint": "./xlam_Qlora_out_eval/checkpoint-2868",
+  "epoch": 1.9975624619134673,
+  "eval_steps": 478,
+  "global_step": 2868,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.006964394532950292,
+      "grad_norm": 0.29273906350135803,
+      "learning_rate": 1.5384615384615387e-06,
+      "loss": 1.8834,
+      "step": 10
+    },
+    {
+      "epoch": 0.013928789065900584,
+      "grad_norm": 0.3107926845550537,
+      "learning_rate": 3.0769230769230774e-06,
+      "loss": 1.876,
+      "step": 20
+    },
+    {
+      "epoch": 0.020893183598850874,
+      "grad_norm": 0.3098163902759552,
+      "learning_rate": 4.615384615384616e-06,
+      "loss": 1.9099,
+      "step": 30
+    },
+    {
+      "epoch": 0.027857578131801168,
+      "grad_norm": 0.36715859174728394,
+      "learning_rate": 6.153846153846155e-06,
+      "loss": 1.8142,
+      "step": 40
+    },
+    {
+      "epoch": 0.034821972664751455,
+      "grad_norm": 0.3511175215244293,
+      "learning_rate": 7.692307692307694e-06,
+      "loss": 1.7271,
+      "step": 50
+    },
+    {
+      "epoch": 0.04178636719770175,
+      "grad_norm": 0.3458055853843689,
+      "learning_rate": 9.230769230769232e-06,
+      "loss": 1.6073,
+      "step": 60
+    },
+    {
+      "epoch": 0.04875076173065204,
+      "grad_norm": 0.37719297409057617,
+      "learning_rate": 1.076923076923077e-05,
+      "loss": 1.4997,
+      "step": 70
+    },
+    {
+      "epoch": 0.055715156263602336,
+      "grad_norm": 0.30273741483688354,
+      "learning_rate": 1.230769230769231e-05,
+      "loss": 1.2914,
+      "step": 80
+    },
+    {
+      "epoch": 0.06267955079655263,
+      "grad_norm": 0.3690374195575714,
+      "learning_rate": 1.3846153846153847e-05,
+      "loss": 1.0086,
+      "step": 90
+    },
+    {
+      "epoch": 0.06964394532950291,
+      "grad_norm": 0.3111755847930908,
+      "learning_rate": 1.5384615384615387e-05,
+      "loss": 0.7469,
+      "step": 100
+    },
+    {
+      "epoch": 0.0766083398624532,
+      "grad_norm": 0.13778822124004364,
+      "learning_rate": 1.6923076923076924e-05,
+      "loss": 0.5848,
+      "step": 110
+    },
+    {
+      "epoch": 0.0835727343954035,
+      "grad_norm": 0.10629754513502121,
+      "learning_rate": 1.8461538461538465e-05,
+      "loss": 0.5366,
+      "step": 120
+    },
+    {
+      "epoch": 0.09053712892835379,
+      "grad_norm": 0.09681010991334915,
+      "learning_rate": 2e-05,
+      "loss": 0.5038,
+      "step": 130
+    },
+    {
+      "epoch": 0.09750152346130408,
+      "grad_norm": 0.10756956040859222,
+      "learning_rate": 1.9952095808383235e-05,
+      "loss": 0.5002,
+      "step": 140
+    },
+    {
+      "epoch": 0.10446591799425438,
+      "grad_norm": 0.10488647222518921,
+      "learning_rate": 1.9904191616766468e-05,
+      "loss": 0.5107,
+      "step": 150
+    },
+    {
+      "epoch": 0.11143031252720467,
+      "grad_norm": 0.09359394758939743,
+      "learning_rate": 1.98562874251497e-05,
+      "loss": 0.4604,
+      "step": 160
+    },
+    {
+      "epoch": 0.11839470706015495,
+      "grad_norm": 0.10392932593822479,
+      "learning_rate": 1.9808383233532935e-05,
+      "loss": 0.484,
+      "step": 170
+    },
+    {
+      "epoch": 0.12535910159310526,
+      "grad_norm": 0.12986360490322113,
+      "learning_rate": 1.9760479041916168e-05,
+      "loss": 0.4527,
+      "step": 180
+    },
+    {
+      "epoch": 0.13232349612605554,
+      "grad_norm": 0.14294420182704926,
+      "learning_rate": 1.97125748502994e-05,
+      "loss": 0.4661,
+      "step": 190
+    },
+    {
+      "epoch": 0.13928789065900582,
+      "grad_norm": 0.18483737111091614,
+      "learning_rate": 1.9664670658682635e-05,
+      "loss": 0.4395,
+      "step": 200
+    },
+    {
+      "epoch": 0.14625228519195613,
+      "grad_norm": 0.21114414930343628,
+      "learning_rate": 1.961676646706587e-05,
+      "loss": 0.4071,
+      "step": 210
+    },
+    {
+      "epoch": 0.1532166797249064,
+      "grad_norm": 0.21779198944568634,
+      "learning_rate": 1.95688622754491e-05,
+      "loss": 0.3846,
+      "step": 220
+    },
+    {
+      "epoch": 0.16018107425785671,
+      "grad_norm": 0.1294960230588913,
+      "learning_rate": 1.9520958083832338e-05,
+      "loss": 0.3952,
+      "step": 230
+    },
+    {
+      "epoch": 0.167145468790807,
+      "grad_norm": 0.0927898958325386,
+      "learning_rate": 1.9473053892215568e-05,
+      "loss": 0.4112,
+      "step": 240
+    },
+    {
+      "epoch": 0.1741098633237573,
+      "grad_norm": 0.10358591377735138,
+      "learning_rate": 1.9425149700598805e-05,
+      "loss": 0.406,
+      "step": 250
+    },
+    {
+      "epoch": 0.18107425785670758,
+      "grad_norm": 0.13144026696681976,
+      "learning_rate": 1.9377245508982038e-05,
+      "loss": 0.4058,
+      "step": 260
+    },
+    {
+      "epoch": 0.18803865238965786,
+      "grad_norm": 0.12745942175388336,
+      "learning_rate": 1.932934131736527e-05,
+      "loss": 0.3938,
+      "step": 270
+    },
+    {
+      "epoch": 0.19500304692260817,
+      "grad_norm": 0.09832354635000229,
+      "learning_rate": 1.9281437125748505e-05,
+      "loss": 0.3819,
+      "step": 280
+    },
+    {
+      "epoch": 0.20196744145555845,
+      "grad_norm": 0.0927552729845047,
+      "learning_rate": 1.9233532934131738e-05,
+      "loss": 0.3944,
+      "step": 290
+    },
+    {
+      "epoch": 0.20893183598850876,
+      "grad_norm": 0.09627388417720795,
+      "learning_rate": 1.918562874251497e-05,
+      "loss": 0.348,
+      "step": 300
+    },
+    {
+      "epoch": 0.21589623052145904,
+      "grad_norm": 0.09034324437379837,
+      "learning_rate": 1.9137724550898205e-05,
+      "loss": 0.3521,
+      "step": 310
+    },
+    {
+      "epoch": 0.22286062505440934,
+      "grad_norm": 0.09156423062086105,
+      "learning_rate": 1.9089820359281438e-05,
+      "loss": 0.3663,
+      "step": 320
+    },
+    {
+      "epoch": 0.22982501958735962,
+      "grad_norm": 0.0996803268790245,
+      "learning_rate": 1.904191616766467e-05,
+      "loss": 0.3347,
+      "step": 330
+    },
+    {
+      "epoch": 0.2367894141203099,
+      "grad_norm": 0.09962292015552521,
+      "learning_rate": 1.8994011976047905e-05,
+      "loss": 0.373,
+      "step": 340
+    },
+    {
+      "epoch": 0.2437538086532602,
+      "grad_norm": 0.08211114257574081,
+      "learning_rate": 1.894610778443114e-05,
+      "loss": 0.3641,
+      "step": 350
+    },
+    {
+      "epoch": 0.2507182031862105,
+      "grad_norm": 0.10449781268835068,
+      "learning_rate": 1.889820359281437e-05,
+      "loss": 0.3358,
+      "step": 360
+    },
+    {
+      "epoch": 0.2576825977191608,
+      "grad_norm": 0.11216866970062256,
+      "learning_rate": 1.8850299401197608e-05,
+      "loss": 0.372,
+      "step": 370
+    },
+    {
+      "epoch": 0.2646469922521111,
+      "grad_norm": 0.11125332862138748,
+      "learning_rate": 1.8802395209580838e-05,
+      "loss": 0.3521,
+      "step": 380
+    },
+    {
+      "epoch": 0.27161138678506136,
+      "grad_norm": 0.1620815545320511,
+      "learning_rate": 1.8754491017964075e-05,
+      "loss": 0.3726,
+      "step": 390
+    },
+    {
+      "epoch": 0.27857578131801164,
+      "grad_norm": 0.11439554393291473,
+      "learning_rate": 1.8706586826347305e-05,
+      "loss": 0.3287,
+      "step": 400
+    },
+    {
+      "epoch": 0.285540175850962,
+      "grad_norm": 0.10733096301555634,
+      "learning_rate": 1.865868263473054e-05,
+      "loss": 0.3459,
+      "step": 410
+    },
+    {
+      "epoch": 0.29250457038391225,
+      "grad_norm": 0.12426561117172241,
+      "learning_rate": 1.8610778443113775e-05,
+      "loss": 0.3705,
+      "step": 420
+    },
+    {
+      "epoch": 0.29946896491686253,
+      "grad_norm": 0.1148887649178505,
+      "learning_rate": 1.8562874251497008e-05,
+      "loss": 0.3216,
+      "step": 430
+    },
+    {
+      "epoch": 0.3064333594498128,
+      "grad_norm": 0.10842527449131012,
+      "learning_rate": 1.851497005988024e-05,
+      "loss": 0.3502,
+      "step": 440
+    },
+    {
+      "epoch": 0.31339775398276315,
+      "grad_norm": 0.11736515164375305,
+      "learning_rate": 1.8467065868263475e-05,
+      "loss": 0.3394,
+      "step": 450
+    },
+    {
+      "epoch": 0.32036214851571343,
+      "grad_norm": 0.1381833553314209,
+      "learning_rate": 1.8419161676646708e-05,
+      "loss": 0.3166,
+      "step": 460
+    },
+    {
+      "epoch": 0.3273265430486637,
+      "grad_norm": 0.10973569750785828,
+      "learning_rate": 1.837125748502994e-05,
+      "loss": 0.3245,
+      "step": 470
+    },
+    {
+      "epoch": 0.3328980586750239,
+      "eval_loss": 0.3075329661369324,
+      "eval_runtime": 356.0276,
+      "eval_samples_per_second": 3.396,
+      "eval_steps_per_second": 1.699,
+      "step": 478
+    },
+    {
+      "epoch": 0.334290937581614,
+      "grad_norm": 0.13474580645561218,
+      "learning_rate": 1.8323353293413175e-05,
+      "loss": 0.2914,
+      "step": 480
+    },
+    {
+      "epoch": 0.34125533211456427,
+      "grad_norm": 0.12812362611293793,
+      "learning_rate": 1.827544910179641e-05,
+      "loss": 0.3327,
+      "step": 490
+    },
+    {
+      "epoch": 0.3482197266475146,
+      "grad_norm": 0.11661785840988159,
+      "learning_rate": 1.822754491017964e-05,
+      "loss": 0.3468,
+      "step": 500
+    },
+    {
+      "epoch": 0.3551841211804649,
+      "grad_norm": 0.12990142405033112,
+      "learning_rate": 1.8179640718562878e-05,
+      "loss": 0.3318,
+      "step": 510
+    },
+    {
+      "epoch": 0.36214851571341516,
+      "grad_norm": 0.1309327632188797,
+      "learning_rate": 1.8131736526946108e-05,
+      "loss": 0.2942,
+      "step": 520
+    },
+    {
+      "epoch": 0.36911291024636544,
+      "grad_norm": 0.12246479839086533,
+      "learning_rate": 1.8083832335329345e-05,
+      "loss": 0.3751,
+      "step": 530
+    },
+    {
+      "epoch": 0.3760773047793157,
+      "grad_norm": 0.14134123921394348,
+      "learning_rate": 1.8035928143712575e-05,
+      "loss": 0.3248,
+      "step": 540
+    },
+    {
+      "epoch": 0.38304169931226606,
+      "grad_norm": 0.1433170884847641,
+      "learning_rate": 1.798802395209581e-05,
+      "loss": 0.3344,
+      "step": 550
+    },
+    {
+      "epoch": 0.39000609384521634,
+      "grad_norm": 0.13364745676517487,
+      "learning_rate": 1.7940119760479045e-05,
+      "loss": 0.2928,
+      "step": 560
+    },
+    {
+      "epoch": 0.3969704883781666,
+      "grad_norm": 0.13446496427059174,
+      "learning_rate": 1.7892215568862278e-05,
+      "loss": 0.3421,
+      "step": 570
+    },
+    {
+      "epoch": 0.4039348829111169,
+      "grad_norm": 0.1586250364780426,
+      "learning_rate": 1.784431137724551e-05,
+      "loss": 0.3099,
+      "step": 580
+    },
+    {
+      "epoch": 0.4108992774440672,
+      "grad_norm": 0.1283930391073227,
+      "learning_rate": 1.7796407185628745e-05,
+      "loss": 0.3123,
+      "step": 590
+    },
+    {
+      "epoch": 0.4178636719770175,
+      "grad_norm": 0.14261852204799652,
+      "learning_rate": 1.7748502994011978e-05,
+      "loss": 0.2886,
+      "step": 600
+    },
+    {
+      "epoch": 0.4248280665099678,
+      "grad_norm": 0.13329675793647766,
+      "learning_rate": 1.770059880239521e-05,
+      "loss": 0.3255,
+      "step": 610
+    },
+    {
+      "epoch": 0.4317924610429181,
+      "grad_norm": 0.13880160450935364,
+      "learning_rate": 1.7652694610778445e-05,
+      "loss": 0.326,
+      "step": 620
+    },
+    {
+      "epoch": 0.43875685557586835,
+      "grad_norm": 0.14388258755207062,
+      "learning_rate": 1.7604790419161678e-05,
+      "loss": 0.3269,
+      "step": 630
+    },
+    {
+      "epoch": 0.4457212501088187,
+      "grad_norm": 0.1571163535118103,
+      "learning_rate": 1.755688622754491e-05,
+      "loss": 0.2919,
+      "step": 640
+    },
+    {
+      "epoch": 0.45268564464176897,
+      "grad_norm": 0.15307635068893433,
+      "learning_rate": 1.7508982035928145e-05,
+      "loss": 0.2773,
+      "step": 650
+    },
+    {
+      "epoch": 0.45965003917471925,
+      "grad_norm": 0.1438165158033371,
+      "learning_rate": 1.7461077844311378e-05,
+      "loss": 0.2847,
+      "step": 660
+    },
+    {
+      "epoch": 0.46661443370766953,
+      "grad_norm": 0.15620216727256775,
+      "learning_rate": 1.741317365269461e-05,
+      "loss": 0.29,
+      "step": 670
+    },
+    {
+      "epoch": 0.4735788282406198,
+      "grad_norm": 0.1414933055639267,
+      "learning_rate": 1.7365269461077845e-05,
+      "loss": 0.2754,
+      "step": 680
+    },
+    {
+      "epoch": 0.48054322277357014,
+      "grad_norm": 0.15853798389434814,
+      "learning_rate": 1.7317365269461078e-05,
+      "loss": 0.3062,
+      "step": 690
+    },
+    {
+      "epoch": 0.4875076173065204,
+      "grad_norm": 0.17159980535507202,
+      "learning_rate": 1.726946107784431e-05,
+      "loss": 0.2953,
+      "step": 700
+    },
+    {
+      "epoch": 0.4944720118394707,
+      "grad_norm": 0.16937299072742462,
+      "learning_rate": 1.7221556886227548e-05,
+      "loss": 0.2895,
+      "step": 710
+    },
+    {
+      "epoch": 0.501436406372421,
+      "grad_norm": 0.1455833464860916,
+      "learning_rate": 1.717365269461078e-05,
+      "loss": 0.2912,
+      "step": 720
+    },
+    {
+      "epoch": 0.5084008009053713,
+      "grad_norm": 0.15824872255325317,
+      "learning_rate": 1.7125748502994015e-05,
+      "loss": 0.3012,
+      "step": 730
+    },
+    {
+      "epoch": 0.5153651954383216,
+      "grad_norm": 0.1499851644039154,
+      "learning_rate": 1.7077844311377248e-05,
+      "loss": 0.263,
+      "step": 740
+    },
+    {
+      "epoch": 0.5223295899712719,
+      "grad_norm": 0.1611323356628418,
+      "learning_rate": 1.702994011976048e-05,
+      "loss": 0.2595,
+      "step": 750
+    },
+    {
+      "epoch": 0.5292939845042222,
+      "grad_norm": 0.16144199669361115,
+      "learning_rate": 1.6982035928143714e-05,
+      "loss": 0.2939,
+      "step": 760
+    },
+    {
+      "epoch": 0.5362583790371724,
+      "grad_norm": 0.13433364033699036,
+      "learning_rate": 1.6934131736526948e-05,
+      "loss": 0.2817,
+      "step": 770
+    },
+    {
+      "epoch": 0.5432227735701227,
+      "grad_norm": 0.15188777446746826,
+      "learning_rate": 1.688622754491018e-05,
+      "loss": 0.2804,
+      "step": 780
+    },
+    {
+      "epoch": 0.550187168103073,
+      "grad_norm": 0.1530751883983612,
+      "learning_rate": 1.6838323353293414e-05,
+      "loss": 0.298,
+      "step": 790
+    },
+    {
+      "epoch": 0.5571515626360233,
+      "grad_norm": 0.14582635462284088,
+      "learning_rate": 1.6790419161676648e-05,
+      "loss": 0.3177,
+      "step": 800
+    },
+    {
+      "epoch": 0.5641159571689737,
+      "grad_norm": 0.174430251121521,
+      "learning_rate": 1.674251497005988e-05,
+      "loss": 0.2624,
+      "step": 810
+    },
+    {
+      "epoch": 0.571080351701924,
+      "grad_norm": 0.1558169573545456,
+      "learning_rate": 1.6694610778443114e-05,
+      "loss": 0.2631,
+      "step": 820
+    },
+    {
+      "epoch": 0.5780447462348742,
+      "grad_norm": 0.17108450829982758,
+      "learning_rate": 1.6646706586826348e-05,
+      "loss": 0.2903,
+      "step": 830
+    },
+    {
+      "epoch": 0.5850091407678245,
+      "grad_norm": 0.16796821355819702,
+      "learning_rate": 1.659880239520958e-05,
+      "loss": 0.2928,
+      "step": 840
+    },
+    {
+      "epoch": 0.5919735353007748,
+      "grad_norm": 0.16556940972805023,
+      "learning_rate": 1.6550898203592814e-05,
+      "loss": 0.2882,
+      "step": 850
+    },
+    {
+      "epoch": 0.5989379298337251,
+      "grad_norm": 0.1716109961271286,
+      "learning_rate": 1.650299401197605e-05,
+      "loss": 0.2814,
+      "step": 860
+    },
+    {
+      "epoch": 0.6059023243666753,
+      "grad_norm": 0.1695253700017929,
+      "learning_rate": 1.645508982035928e-05,
+      "loss": 0.2909,
+      "step": 870
+    },
+    {
+      "epoch": 0.6128667188996256,
+      "grad_norm": 0.1896178424358368,
+      "learning_rate": 1.6407185628742518e-05,
+      "loss": 0.2909,
+      "step": 880
+    },
+    {
+      "epoch": 0.6198311134325759,
+      "grad_norm": 0.1624131202697754,
+      "learning_rate": 1.6359281437125748e-05,
+      "loss": 0.2672,
+      "step": 890
+    },
+    {
+      "epoch": 0.6267955079655263,
+      "grad_norm": 0.16821731626987457,
+      "learning_rate": 1.6311377245508984e-05,
+      "loss": 0.2626,
+      "step": 900
+    },
+    {
+      "epoch": 0.6337599024984766,
+      "grad_norm": 0.17026785016059875,
+      "learning_rate": 1.6263473053892214e-05,
+      "loss": 0.2735,
+      "step": 910
+    },
+    {
+      "epoch": 0.6407242970314269,
+      "grad_norm": 0.19429761171340942,
+      "learning_rate": 1.621556886227545e-05,
+      "loss": 0.2605,
+      "step": 920
+    },
+    {
+      "epoch": 0.6476886915643771,
+      "grad_norm": 0.16931040585041046,
+      "learning_rate": 1.6167664670658684e-05,
+      "loss": 0.2722,
+      "step": 930
+    },
+    {
+      "epoch": 0.6546530860973274,
+      "grad_norm": 0.20588335394859314,
+      "learning_rate": 1.6119760479041918e-05,
+      "loss": 0.301,
+      "step": 940
+    },
+    {
+      "epoch": 0.6616174806302777,
+      "grad_norm": 0.16699948906898499,
+      "learning_rate": 1.607185628742515e-05,
+      "loss": 0.2646,
+      "step": 950
+    },
+    {
+      "epoch": 0.6657961173500478,
+      "eval_loss": 0.2465677410364151,
+      "eval_runtime": 356.1016,
+      "eval_samples_per_second": 3.395,
+      "eval_steps_per_second": 1.699,
+      "step": 956
+    },
+    {
+      "epoch": 0.668581875163228,
+      "grad_norm": 0.20734217762947083,
+      "learning_rate": 1.6023952095808384e-05,
+      "loss": 0.2476,
+      "step": 960
+    },
+    {
+      "epoch": 0.6755462696961783,
+      "grad_norm": 0.21317210793495178,
+      "learning_rate": 1.5976047904191618e-05,
+      "loss": 0.2434,
+      "step": 970
+    },
+    {
+      "epoch": 0.6825106642291285,
+      "grad_norm": 0.19125515222549438,
+      "learning_rate": 1.592814371257485e-05,
+      "loss": 0.2823,
+      "step": 980
+    },
+    {
+      "epoch": 0.6894750587620788,
+      "grad_norm": 0.1651381254196167,
+      "learning_rate": 1.5880239520958084e-05,
+      "loss": 0.2769,
+      "step": 990
+    },
+    {
+      "epoch": 0.6964394532950292,
+      "grad_norm": 0.18061324954032898,
+      "learning_rate": 1.583233532934132e-05,
+      "loss": 0.2788,
+      "step": 1000
+    },
+    {
+      "epoch": 0.7034038478279795,
+      "grad_norm": 0.17788180708885193,
+      "learning_rate": 1.578443113772455e-05,
+      "loss": 0.2596,
+      "step": 1010
+    },
+    {
+      "epoch": 0.7103682423609298,
+      "grad_norm": 0.19635842740535736,
+      "learning_rate": 1.5736526946107788e-05,
+      "loss": 0.2391,
+      "step": 1020
+    },
+    {
+      "epoch": 0.71733263689388,
+      "grad_norm": 0.18000468611717224,
+      "learning_rate": 1.5688622754491018e-05,
+      "loss": 0.2589,
+      "step": 1030
+    },
+    {
+      "epoch": 0.7242970314268303,
+      "grad_norm": 0.18016283214092255,
+      "learning_rate": 1.5640718562874254e-05,
+      "loss": 0.2673,
+      "step": 1040
+    },
+    {
+      "epoch": 0.7312614259597806,
+      "grad_norm": 0.2085445076227188,
+      "learning_rate": 1.5592814371257484e-05,
+      "loss": 0.295,
+      "step": 1050
+    },
+    {
+      "epoch": 0.7382258204927309,
+      "grad_norm": 0.1937682181596756,
+      "learning_rate": 1.554491017964072e-05,
+      "loss": 0.2741,
+      "step": 1060
+    },
+    {
+      "epoch": 0.7451902150256812,
+      "grad_norm": 0.18496188521385193,
+      "learning_rate": 1.5497005988023954e-05,
+      "loss": 0.2485,
+      "step": 1070
+    },
+    {
+      "epoch": 0.7521546095586314,
+      "grad_norm": 0.21024686098098755,
+      "learning_rate": 1.5449101796407188e-05,
+      "loss": 0.2569,
+      "step": 1080
+    },
+    {
+      "epoch": 0.7591190040915818,
+      "grad_norm": 0.1960270255804062,
+      "learning_rate": 1.540119760479042e-05,
+      "loss": 0.2335,
+      "step": 1090
+    },
+    {
+      "epoch": 0.7660833986245321,
+      "grad_norm": 0.19557876884937286,
+      "learning_rate": 1.5353293413173654e-05,
+      "loss": 0.279,
+      "step": 1100
+    },
+    {
+      "epoch": 0.7730477931574824,
+      "grad_norm": 0.17300210893154144,
+      "learning_rate": 1.5305389221556888e-05,
+      "loss": 0.249,
+      "step": 1110
+    },
+    {
+      "epoch": 0.7800121876904327,
+      "grad_norm": 0.1768539994955063,
+      "learning_rate": 1.5257485029940121e-05,
+      "loss": 0.2415,
+      "step": 1120
+    },
+    {
+      "epoch": 0.786976582223383,
+      "grad_norm": 0.20632915198802948,
+      "learning_rate": 1.5209580838323354e-05,
+      "loss": 0.2312,
+      "step": 1130
+    },
+    {
+      "epoch": 0.7939409767563332,
+      "grad_norm": 0.1948348730802536,
+      "learning_rate": 1.516167664670659e-05,
+      "loss": 0.2415,
+      "step": 1140
+    },
+    {
+      "epoch": 0.8009053712892835,
+      "grad_norm": 0.16288821399211884,
+      "learning_rate": 1.511377245508982e-05,
+      "loss": 0.2571,
+      "step": 1150
+    },
+    {
+      "epoch": 0.8078697658222338,
+      "grad_norm": 0.21365194022655487,
+      "learning_rate": 1.5065868263473056e-05,
+      "loss": 0.2594,
+      "step": 1160
+    },
+    {
+      "epoch": 0.8148341603551841,
+      "grad_norm": 0.20960865914821625,
+      "learning_rate": 1.5017964071856287e-05,
+      "loss": 0.2537,
+      "step": 1170
+    },
+    {
+      "epoch": 0.8217985548881344,
+      "grad_norm": 0.20458345115184784,
+      "learning_rate": 1.4970059880239522e-05,
+      "loss": 0.2675,
+      "step": 1180
+    },
+    {
+      "epoch": 0.8287629494210847,
+      "grad_norm": 0.22388549149036407,
+      "learning_rate": 1.4922155688622754e-05,
+      "loss": 0.2525,
+      "step": 1190
+    },
+    {
+      "epoch": 0.835727343954035,
+      "grad_norm": 0.2139691859483719,
+      "learning_rate": 1.4874251497005989e-05,
+      "loss": 0.251,
+      "step": 1200
+    },
+    {
+      "epoch": 0.8426917384869853,
+      "grad_norm": 0.18761980533599854,
+      "learning_rate": 1.4826347305389224e-05,
+      "loss": 0.2328,
+      "step": 1210
+    },
+    {
+      "epoch": 0.8496561330199356,
+      "grad_norm": 0.183973029255867,
+      "learning_rate": 1.4778443113772456e-05,
+      "loss": 0.2239,
+      "step": 1220
+    },
+    {
+      "epoch": 0.8566205275528859,
+      "grad_norm": 0.23718492686748505,
+      "learning_rate": 1.473053892215569e-05,
+      "loss": 0.2423,
+      "step": 1230
+    },
+    {
+      "epoch": 0.8635849220858361,
+      "grad_norm": 0.23875045776367188,
+      "learning_rate": 1.4682634730538922e-05,
+      "loss": 0.2365,
+      "step": 1240
+    },
+    {
+      "epoch": 0.8705493166187864,
+      "grad_norm": 0.21570812165737152,
+      "learning_rate": 1.4634730538922157e-05,
+      "loss": 0.268,
+      "step": 1250
+    },
+    {
+      "epoch": 0.8775137111517367,
+      "grad_norm": 0.20839928090572357,
+      "learning_rate": 1.4586826347305389e-05,
+      "loss": 0.2502,
+      "step": 1260
+    },
+    {
+      "epoch": 0.884478105684687,
+      "grad_norm": 0.21605758368968964,
+      "learning_rate": 1.4538922155688624e-05,
+      "loss": 0.2365,
+      "step": 1270
+    },
+    {
+      "epoch": 0.8914425002176374,
+      "grad_norm": 0.23300880193710327,
+      "learning_rate": 1.4491017964071859e-05,
+      "loss": 0.2404,
+      "step": 1280
+    },
+    {
+      "epoch": 0.8984068947505877,
+      "grad_norm": 0.21796733140945435,
+      "learning_rate": 1.444311377245509e-05,
+      "loss": 0.2704,
+      "step": 1290
+    },
+    {
+      "epoch": 0.9053712892835379,
+      "grad_norm": 0.25555598735809326,
+      "learning_rate": 1.4395209580838326e-05,
+      "loss": 0.2674,
+      "step": 1300
+    },
+    {
+      "epoch": 0.9123356838164882,
+      "grad_norm": 0.22284413874149323,
+      "learning_rate": 1.4347305389221557e-05,
+      "loss": 0.2471,
+      "step": 1310
+    },
+    {
+      "epoch": 0.9193000783494385,
+      "grad_norm": 0.22234566509723663,
+      "learning_rate": 1.4299401197604792e-05,
+      "loss": 0.2529,
+      "step": 1320
+    },
+    {
+      "epoch": 0.9262644728823888,
+      "grad_norm": 0.221883624792099,
+      "learning_rate": 1.4251497005988024e-05,
+      "loss": 0.1932,
+      "step": 1330
+    },
+    {
+      "epoch": 0.9332288674153391,
+      "grad_norm": 0.22873520851135254,
+      "learning_rate": 1.4203592814371259e-05,
+      "loss": 0.2412,
+      "step": 1340
+    },
+    {
+      "epoch": 0.9401932619482893,
+      "grad_norm": 0.21307863295078278,
+      "learning_rate": 1.4155688622754492e-05,
+      "loss": 0.2384,
+      "step": 1350
+    },
+    {
+      "epoch": 0.9471576564812396,
+      "grad_norm": 0.21296364068984985,
+      "learning_rate": 1.4107784431137726e-05,
+      "loss": 0.2318,
+      "step": 1360
+    },
+    {
+      "epoch": 0.95412205101419,
+      "grad_norm": 0.21060685813426971,
+      "learning_rate": 1.405988023952096e-05,
+      "loss": 0.2323,
+      "step": 1370
+    },
+    {
+      "epoch": 0.9610864455471403,
+      "grad_norm": 0.23930418491363525,
+      "learning_rate": 1.4011976047904192e-05,
+      "loss": 0.2398,
+      "step": 1380
+    },
+    {
+      "epoch": 0.9680508400800906,
+      "grad_norm": 0.22837185859680176,
+      "learning_rate": 1.3964071856287427e-05,
+      "loss": 0.233,
+      "step": 1390
+    },
+    {
+      "epoch": 0.9750152346130408,
+      "grad_norm": 0.21798169612884521,
+      "learning_rate": 1.3916167664670659e-05,
+      "loss": 0.2161,
+      "step": 1400
+    },
+    {
+      "epoch": 0.9819796291459911,
+      "grad_norm": 0.2846814692020416,
+      "learning_rate": 1.3868263473053894e-05,
+      "loss": 0.2301,
+      "step": 1410
+    },
+    {
+      "epoch": 0.9889440236789414,
+      "grad_norm": 0.22030217945575714,
+      "learning_rate": 1.3820359281437127e-05,
+      "loss": 0.2017,
+      "step": 1420
+    },
+    {
+      "epoch": 0.9959084182118917,
+      "grad_norm": 0.25554022192955017,
+      "learning_rate": 1.377245508982036e-05,
+      "loss": 0.2296,
+      "step": 1430
+    },
+    {
+      "epoch": 0.9986941760250718,
+      "eval_loss": 0.21637588739395142,
+      "eval_runtime": 356.0721,
+      "eval_samples_per_second": 3.395,
+      "eval_steps_per_second": 1.699,
+      "step": 1434
+    },
+    {
+      "epoch": 1.0030469226081657,
+      "grad_norm": 0.2862723469734192,
+      "learning_rate": 1.3724550898203594e-05,
+      "loss": 0.2538,
+      "step": 1440
+    },
+    {
+      "epoch": 1.010011317141116,
+      "grad_norm": 0.2589089870452881,
+      "learning_rate": 1.3676646706586827e-05,
+      "loss": 0.2196,
+      "step": 1450
+    },
+    {
+      "epoch": 1.0169757116740663,
+      "grad_norm": 0.27893784642219543,
+      "learning_rate": 1.362874251497006e-05,
+      "loss": 0.1935,
+      "step": 1460
+    },
+    {
+      "epoch": 1.0239401062070166,
+      "grad_norm": 0.2530602514743805,
+      "learning_rate": 1.3580838323353294e-05,
+      "loss": 0.2248,
+      "step": 1470
+    },
+    {
+      "epoch": 1.0309045007399669,
+      "grad_norm": 0.24902446568012238,
+      "learning_rate": 1.3532934131736529e-05,
+      "loss": 0.224,
+      "step": 1480
+    },
+    {
+      "epoch": 1.0378688952729171,
+      "grad_norm": 0.27132269740104675,
+      "learning_rate": 1.3485029940119762e-05,
+      "loss": 0.2208,
+      "step": 1490
+    },
+    {
+      "epoch": 1.0448332898058674,
+      "grad_norm": 0.2570977807044983,
+      "learning_rate": 1.3437125748502996e-05,
+      "loss": 0.2103,
+      "step": 1500
+    },
+    {
+      "epoch": 1.0517976843388177,
+      "grad_norm": 0.2994561493396759,
+      "learning_rate": 1.3389221556886229e-05,
+      "loss": 0.2243,
+      "step": 1510
+    },
+    {
+      "epoch": 1.058762078871768,
+      "grad_norm": 0.22994251549243927,
+      "learning_rate": 1.3341317365269462e-05,
+      "loss": 0.2217,
+      "step": 1520
+    },
+    {
+      "epoch": 1.0657264734047183,
+      "grad_norm": 0.22706159949302673,
+      "learning_rate": 1.3293413173652696e-05,
+      "loss": 0.2219,
+      "step": 1530
+    },
+    {
+      "epoch": 1.0726908679376688,
+      "grad_norm": 0.26584964990615845,
+      "learning_rate": 1.3245508982035929e-05,
+      "loss": 0.2329,
+      "step": 1540
+    },
+    {
+      "epoch": 1.079655262470619,
+      "grad_norm": 0.2549090087413788,
+      "learning_rate": 1.3197604790419162e-05,
+      "loss": 0.2235,
+      "step": 1550
+    },
+    {
+      "epoch": 1.0866196570035693,
+      "grad_norm": 0.27169135212898254,
+      "learning_rate": 1.3149700598802397e-05,
+      "loss": 0.2332,
+      "step": 1560
+    },
+    {
+      "epoch": 1.0935840515365196,
+      "grad_norm": 0.26733773946762085,
+      "learning_rate": 1.3101796407185629e-05,
+      "loss": 0.2062,
+      "step": 1570
+    },
+    {
+      "epoch": 1.1005484460694699,
+      "grad_norm": 0.3222082853317261,
+      "learning_rate": 1.3053892215568864e-05,
+      "loss": 0.2111,
+      "step": 1580
+    },
+    {
+      "epoch": 1.1075128406024202,
+      "grad_norm": 0.20403257012367249,
+      "learning_rate": 1.3005988023952097e-05,
+      "loss": 0.226,
+      "step": 1590
+    },
+    {
+      "epoch": 1.1144772351353704,
+      "grad_norm": 0.2658451199531555,
+      "learning_rate": 1.295808383233533e-05,
+      "loss": 0.2013,
+      "step": 1600
+    },
+    {
+      "epoch": 1.1214416296683207,
+      "grad_norm": 0.2719573676586151,
+      "learning_rate": 1.2910179640718564e-05,
+      "loss": 0.2036,
+      "step": 1610
+    },
+    {
+      "epoch": 1.128406024201271,
+      "grad_norm": 0.29076331853866577,
+      "learning_rate": 1.2862275449101797e-05,
+      "loss": 0.2162,
+      "step": 1620
+    },
+    {
+      "epoch": 1.1353704187342213,
+      "grad_norm": 0.2324078530073166,
+      "learning_rate": 1.2814371257485032e-05,
+      "loss": 0.2002,
+      "step": 1630
+    },
+    {
+      "epoch": 1.1423348132671716,
+      "grad_norm": 0.27219468355178833,
+      "learning_rate": 1.2766467065868264e-05,
+      "loss": 0.2238,
+      "step": 1640
+    },
+    {
+      "epoch": 1.1492992078001218,
+      "grad_norm": 0.22101271152496338,
+      "learning_rate": 1.2718562874251499e-05,
+      "loss": 0.1845,
+      "step": 1650
+    },
+    {
+      "epoch": 1.1562636023330721,
+      "grad_norm": 0.25101369619369507,
+      "learning_rate": 1.267065868263473e-05,
+      "loss": 0.2296,
+      "step": 1660
+    },
+    {
+      "epoch": 1.1632279968660224,
+      "grad_norm": 0.25895142555236816,
+      "learning_rate": 1.2622754491017965e-05,
+      "loss": 0.2271,
+      "step": 1670
+    },
+    {
+      "epoch": 1.1701923913989727,
+      "grad_norm": 0.26405826210975647,
+      "learning_rate": 1.2574850299401197e-05,
+      "loss": 0.2159,
+      "step": 1680
+    },
+    {
+      "epoch": 1.177156785931923,
+      "grad_norm": 0.23231615126132965,
+      "learning_rate": 1.2526946107784432e-05,
+      "loss": 0.1916,
+      "step": 1690
+    },
+    {
+      "epoch": 1.1841211804648732,
+      "grad_norm": 0.3211207985877991,
+      "learning_rate": 1.2479041916167665e-05,
+      "loss": 0.2154,
+      "step": 1700
+    },
+    {
+      "epoch": 1.1910855749978237,
+      "grad_norm": 0.2948794662952423,
+      "learning_rate": 1.2431137724550899e-05,
+      "loss": 0.1999,
+      "step": 1710
+    },
+    {
+      "epoch": 1.198049969530774,
+      "grad_norm": 0.28081706166267395,
+      "learning_rate": 1.2383233532934134e-05,
+      "loss": 0.2301,
+      "step": 1720
+    },
+    {
+      "epoch": 1.2050143640637243,
+      "grad_norm": 0.296283096075058,
+      "learning_rate": 1.2335329341317365e-05,
+      "loss": 0.2164,
+      "step": 1730
+    },
+    {
+      "epoch": 1.2119787585966746,
+      "grad_norm": 0.29658278822898865,
+      "learning_rate": 1.22874251497006e-05,
+      "loss": 0.1916,
+      "step": 1740
+    },
+    {
+      "epoch": 1.2189431531296249,
+      "grad_norm": 0.2917250990867615,
+      "learning_rate": 1.2239520958083832e-05,
+      "loss": 0.2045,
+      "step": 1750
+    },
+    {
+      "epoch": 1.2259075476625751,
+      "grad_norm": 0.35840919613838196,
+      "learning_rate": 1.2191616766467067e-05,
+      "loss": 0.1965,
+      "step": 1760
+    },
+    {
+      "epoch": 1.2328719421955254,
+      "grad_norm": 0.2819244861602783,
+      "learning_rate": 1.2143712574850299e-05,
+      "loss": 0.2032,
+      "step": 1770
+    },
+    {
+      "epoch": 1.2398363367284757,
+      "grad_norm": 0.3518809974193573,
+      "learning_rate": 1.2095808383233534e-05,
+      "loss": 0.2043,
+      "step": 1780
+    },
+    {
+      "epoch": 1.246800731261426,
+      "grad_norm": 0.3232385516166687,
+      "learning_rate": 1.2047904191616769e-05,
+      "loss": 0.23,
+      "step": 1790
+    },
+    {
+      "epoch": 1.2537651257943763,
+      "grad_norm": 0.23131519556045532,
+      "learning_rate": 1.2e-05,
+      "loss": 0.2106,
+      "step": 1800
+    },
+    {
+      "epoch": 1.2607295203273265,
+      "grad_norm": 0.30877017974853516,
+      "learning_rate": 1.1952095808383235e-05,
+      "loss": 0.2088,
+      "step": 1810
+    },
+    {
+      "epoch": 1.2676939148602768,
+      "grad_norm": 0.36684754490852356,
+      "learning_rate": 1.1904191616766467e-05,
+      "loss": 0.2024,
+      "step": 1820
+    },
+    {
+      "epoch": 1.274658309393227,
+      "grad_norm": 0.21124888956546783,
+      "learning_rate": 1.1856287425149702e-05,
+      "loss": 0.1929,
+      "step": 1830
+    },
+    {
+      "epoch": 1.2816227039261774,
+      "grad_norm": 0.2815339267253876,
+      "learning_rate": 1.1808383233532934e-05,
+      "loss": 0.1973,
+      "step": 1840
+    },
+    {
+      "epoch": 1.2885870984591277,
+      "grad_norm": 0.3017849922180176,
+      "learning_rate": 1.1760479041916169e-05,
+      "loss": 0.233,
+      "step": 1850
+    },
+    {
+      "epoch": 1.295551492992078,
+      "grad_norm": 0.3456547260284424,
+      "learning_rate": 1.1712574850299404e-05,
+      "loss": 0.1769,
+      "step": 1860
+    },
+    {
+      "epoch": 1.3025158875250282,
+      "grad_norm": 0.28210344910621643,
+      "learning_rate": 1.1664670658682635e-05,
+      "loss": 0.204,
+      "step": 1870
+    },
+    {
+      "epoch": 1.3094802820579785,
+      "grad_norm": 0.3701432943344116,
+      "learning_rate": 1.161676646706587e-05,
+      "loss": 0.1923,
+      "step": 1880
+    },
+    {
+      "epoch": 1.3164446765909288,
+      "grad_norm": 0.3656926453113556,
+      "learning_rate": 1.1568862275449102e-05,
+      "loss": 0.2136,
+      "step": 1890
+    },
+    {
+      "epoch": 1.323409071123879,
+      "grad_norm": 0.29142090678215027,
+      "learning_rate": 1.1520958083832337e-05,
+      "loss": 0.1957,
+      "step": 1900
+    },
+    {
+      "epoch": 1.3303734656568293,
+      "grad_norm": 0.2799660563468933,
+      "learning_rate": 1.1473053892215569e-05,
+      "loss": 0.2009,
+      "step": 1910
+    },
+    {
+      "epoch": 1.3317663445634196,
+      "eval_loss": 0.19295775890350342,
+      "eval_runtime": 356.0153,
+      "eval_samples_per_second": 3.396,
+      "eval_steps_per_second": 1.699,
+      "step": 1912
+    },
+    {
+      "epoch": 1.3373378601897796,
+      "grad_norm": 0.38173311948776245,
+      "learning_rate": 1.1425149700598804e-05,
+      "loss": 0.2133,
+      "step": 1920
+    },
+    {
+      "epoch": 1.34430225472273,
+      "grad_norm": 0.38059180974960327,
+      "learning_rate": 1.1377245508982037e-05,
+      "loss": 0.203,
+      "step": 1930
+    },
+    {
+      "epoch": 1.3512666492556804,
+      "grad_norm": 0.2743132710456848,
+      "learning_rate": 1.132934131736527e-05,
+      "loss": 0.196,
+      "step": 1940
+    },
+    {
+      "epoch": 1.3582310437886307,
+      "grad_norm": 0.3110567331314087,
+      "learning_rate": 1.1281437125748505e-05,
+      "loss": 0.1868,
+      "step": 1950
+    },
+    {
+      "epoch": 1.365195438321581,
+      "grad_norm": 0.4261178970336914,
+      "learning_rate": 1.1233532934131737e-05,
+      "loss": 0.1952,
+      "step": 1960
+    },
+    {
+      "epoch": 1.3721598328545312,
+      "grad_norm": 0.3890218436717987,
+      "learning_rate": 1.1185628742514972e-05,
+      "loss": 0.1964,
+      "step": 1970
+    },
+    {
+      "epoch": 1.3791242273874815,
+      "grad_norm": 0.3712151050567627,
+      "learning_rate": 1.1137724550898203e-05,
+      "loss": 0.1933,
+      "step": 1980
+    },
+    {
+      "epoch": 1.3860886219204318,
+      "grad_norm": 0.2833101153373718,
+      "learning_rate": 1.1089820359281439e-05,
+      "loss": 0.1862,
+      "step": 1990
+    },
+    {
+      "epoch": 1.393053016453382,
+      "grad_norm": 0.3696097731590271,
+      "learning_rate": 1.1041916167664672e-05,
+      "loss": 0.2142,
+      "step": 2000
+    },
+    {
+      "epoch": 1.4000174109863324,
+      "grad_norm": 0.4144258499145508,
+      "learning_rate": 1.0994011976047905e-05,
+      "loss": 0.1915,
+      "step": 2010
+    },
+    {
+      "epoch": 1.4069818055192826,
+      "grad_norm": 0.28359559178352356,
+      "learning_rate": 1.0946107784431138e-05,
+      "loss": 0.1861,
+      "step": 2020
+    },
+    {
+      "epoch": 1.413946200052233,
+      "grad_norm": 0.327510803937912,
+      "learning_rate": 1.0898203592814372e-05,
+      "loss": 0.1961,
+      "step": 2030
+    },
+    {
+      "epoch": 1.4209105945851832,
+      "grad_norm": 0.44245216250419617,
+      "learning_rate": 1.0850299401197605e-05,
+      "loss": 0.2091,
+      "step": 2040
+    },
+    {
+      "epoch": 1.4278749891181335,
+      "grad_norm": 0.3903510570526123,
+      "learning_rate": 1.0802395209580838e-05,
+      "loss": 0.2058,
+      "step": 2050
+    },
+    {
+      "epoch": 1.4348393836510838,
+      "grad_norm": 0.32461392879486084,
+      "learning_rate": 1.0754491017964073e-05,
+      "loss": 0.1793,
+      "step": 2060
+    },
+    {
+      "epoch": 1.441803778184034,
+      "grad_norm": 0.4412670433521271,
+      "learning_rate": 1.0706586826347307e-05,
+      "loss": 0.1856,
+      "step": 2070
+    },
+    {
+      "epoch": 1.4487681727169845,
+      "grad_norm": 0.2217404693365097,
+      "learning_rate": 1.065868263473054e-05,
+      "loss": 0.1628,
+      "step": 2080
+    },
+    {
+      "epoch": 1.4557325672499348,
+      "grad_norm": 0.3572748899459839,
+      "learning_rate": 1.0610778443113773e-05,
+      "loss": 0.1904,
+      "step": 2090
+    },
+    {
+      "epoch": 1.462696961782885,
+      "grad_norm": 0.23523126542568207,
+      "learning_rate": 1.0562874251497007e-05,
+      "loss": 0.1833,
+      "step": 2100
+    },
+    {
+      "epoch": 1.4696613563158354,
+      "grad_norm": 0.35970646142959595,
+      "learning_rate": 1.051497005988024e-05,
+      "loss": 0.2018,
+      "step": 2110
+    },
+    {
+      "epoch": 1.4766257508487857,
+      "grad_norm": 0.3383442759513855,
+      "learning_rate": 1.0467065868263473e-05,
+      "loss": 0.1986,
+      "step": 2120
+    },
+    {
+      "epoch": 1.483590145381736,
+      "grad_norm": 0.33182451128959656,
+      "learning_rate": 1.0419161676646707e-05,
+      "loss": 0.194,
+      "step": 2130
+    },
+    {
+      "epoch": 1.4905545399146862,
+      "grad_norm": 0.3276427090167999,
+      "learning_rate": 1.0371257485029942e-05,
+      "loss": 0.1945,
+      "step": 2140
+    },
+    {
+      "epoch": 1.4975189344476365,
+      "grad_norm": 0.30687084794044495,
+      "learning_rate": 1.0323353293413173e-05,
+      "loss": 0.1839,
+      "step": 2150
+    },
+    {
+      "epoch": 1.5044833289805868,
+      "grad_norm": 0.3758613169193268,
+      "learning_rate": 1.0275449101796408e-05,
+      "loss": 0.2035,
+      "step": 2160
+    },
+    {
+      "epoch": 1.511447723513537,
+      "grad_norm": 0.34436604380607605,
+      "learning_rate": 1.0227544910179642e-05,
+      "loss": 0.1946,
+      "step": 2170
+    },
+    {
+      "epoch": 1.5184121180464873,
+      "grad_norm": 0.31638774275779724,
+      "learning_rate": 1.0179640718562875e-05,
+      "loss": 0.1761,
+      "step": 2180
+    },
+    {
+      "epoch": 1.5253765125794376,
+      "grad_norm": 0.3935607969760895,
+      "learning_rate": 1.0131736526946108e-05,
+      "loss": 0.2008,
+      "step": 2190
+    },
+    {
+      "epoch": 1.532340907112388,
+      "grad_norm": 0.3549712598323822,
+      "learning_rate": 1.0083832335329342e-05,
+      "loss": 0.1814,
+      "step": 2200
+    },
+    {
+      "epoch": 1.5393053016453382,
+      "grad_norm": 0.32955431938171387,
+      "learning_rate": 1.0035928143712577e-05,
+      "loss": 0.1736,
+      "step": 2210
+    },
+    {
+      "epoch": 1.5462696961782885,
+      "grad_norm": 0.3109774589538574,
+      "learning_rate": 9.988023952095808e-06,
+      "loss": 0.169,
+      "step": 2220
+    },
+    {
+      "epoch": 1.5532340907112387,
+      "grad_norm": 0.4271303117275238,
+      "learning_rate": 9.940119760479042e-06,
+      "loss": 0.1761,
+      "step": 2230
+    },
+    {
+      "epoch": 1.560198485244189,
+      "grad_norm": 0.3970726728439331,
+      "learning_rate": 9.892215568862277e-06,
+      "loss": 0.1897,
+      "step": 2240
+    },
+    {
+      "epoch": 1.5671628797771393,
+      "grad_norm": 0.4251217544078827,
+      "learning_rate": 9.84431137724551e-06,
+      "loss": 0.1771,
+      "step": 2250
+    },
+    {
+      "epoch": 1.5741272743100896,
+      "grad_norm": 0.28587886691093445,
+      "learning_rate": 9.796407185628743e-06,
+      "loss": 0.1699,
+      "step": 2260
+    },
+    {
+      "epoch": 1.5810916688430399,
+      "grad_norm": 0.36768022179603577,
+      "learning_rate": 9.748502994011977e-06,
+      "loss": 0.1933,
+      "step": 2270
+    },
+    {
+      "epoch": 1.5880560633759901,
+      "grad_norm": 0.31612274050712585,
+      "learning_rate": 9.70059880239521e-06,
+      "loss": 0.1944,
+      "step": 2280
+    },
+    {
+      "epoch": 1.5950204579089404,
+      "grad_norm": 0.34787702560424805,
+      "learning_rate": 9.652694610778443e-06,
+      "loss": 0.1758,
+      "step": 2290
+    },
+    {
+      "epoch": 1.6019848524418907,
+      "grad_norm": 0.4170469045639038,
+      "learning_rate": 9.604790419161677e-06,
+      "loss": 0.1749,
+      "step": 2300
+    },
+    {
+      "epoch": 1.608949246974841,
+      "grad_norm": 0.3099391460418701,
+      "learning_rate": 9.556886227544912e-06,
+      "loss": 0.1752,
+      "step": 2310
+    },
+    {
+      "epoch": 1.6159136415077913,
+      "grad_norm": 0.3781738877296448,
+      "learning_rate": 9.508982035928145e-06,
+      "loss": 0.2032,
+      "step": 2320
+    },
+    {
+      "epoch": 1.6228780360407415,
+      "grad_norm": 0.29017606377601624,
+      "learning_rate": 9.461077844311378e-06,
+      "loss": 0.1733,
+      "step": 2330
+    },
+    {
+      "epoch": 1.629842430573692,
+      "grad_norm": 0.45764532685279846,
+      "learning_rate": 9.413173652694612e-06,
+      "loss": 0.1765,
+      "step": 2340
+    },
+    {
+      "epoch": 1.6368068251066423,
+      "grad_norm": 0.4349108040332794,
+      "learning_rate": 9.365269461077845e-06,
+      "loss": 0.178,
+      "step": 2350
+    },
+    {
+      "epoch": 1.6437712196395926,
+      "grad_norm": 0.39599794149398804,
+      "learning_rate": 9.317365269461078e-06,
+      "loss": 0.1792,
+      "step": 2360
+    },
+    {
+      "epoch": 1.6507356141725429,
+      "grad_norm": 0.1882832944393158,
+      "learning_rate": 9.269461077844312e-06,
+      "loss": 0.1871,
+      "step": 2370
+    },
+    {
+      "epoch": 1.6577000087054932,
+      "grad_norm": 0.323946475982666,
+      "learning_rate": 9.221556886227547e-06,
+      "loss": 0.1783,
+      "step": 2380
+    },
+    {
+      "epoch": 1.6646644032384434,
+      "grad_norm": 0.36734339594841003,
+      "learning_rate": 9.17365269461078e-06,
+      "loss": 0.1591,
+      "step": 2390
+    },
+    {
+      "epoch": 1.6646644032384434,
+      "eval_loss": 0.17429304122924805,
+      "eval_runtime": 356.3726,
+      "eval_samples_per_second": 3.393,
+      "eval_steps_per_second": 1.698,
+      "step": 2390
+    },
+    {
+      "epoch": 1.6716287977713937,
+      "grad_norm": 0.5177730321884155,
+      "learning_rate": 9.125748502994013e-06,
+      "loss": 0.1807,
+      "step": 2400
+    },
+    {
+      "epoch": 1.678593192304344,
+      "grad_norm": 0.36497658491134644,
+      "learning_rate": 9.077844311377247e-06,
+      "loss": 0.1898,
+      "step": 2410
+    },
+    {
+      "epoch": 1.6855575868372943,
+      "grad_norm": 0.38655078411102295,
+      "learning_rate": 9.02994011976048e-06,
+      "loss": 0.1973,
+      "step": 2420
+    },
+    {
+      "epoch": 1.6925219813702448,
+      "grad_norm": 0.264782577753067,
+      "learning_rate": 8.982035928143713e-06,
+      "loss": 0.1733,
+      "step": 2430
+    },
+    {
+      "epoch": 1.699486375903195,
+      "grad_norm": 0.38668733835220337,
+      "learning_rate": 8.934131736526946e-06,
+      "loss": 0.1716,
+      "step": 2440
+    },
+    {
+      "epoch": 1.7064507704361453,
+      "grad_norm": 0.32163408398628235,
+      "learning_rate": 8.886227544910181e-06,
+      "loss": 0.1682,
+      "step": 2450
+    },
+    {
+      "epoch": 1.7134151649690956,
+      "grad_norm": 0.41438040137290955,
+      "learning_rate": 8.838323353293415e-06,
+      "loss": 0.1839,
+      "step": 2460
+    },
+    {
+      "epoch": 1.720379559502046,
+      "grad_norm": 0.3748877942562103,
+      "learning_rate": 8.790419161676648e-06,
+      "loss": 0.1948,
+      "step": 2470
+    },
+    {
+      "epoch": 1.7273439540349962,
+      "grad_norm": 0.3265992999076843,
+      "learning_rate": 8.742514970059881e-06,
+      "loss": 0.1766,
+      "step": 2480
+    },
+    {
+      "epoch": 1.7343083485679465,
+      "grad_norm": 0.34820470213890076,
+      "learning_rate": 8.694610778443115e-06,
+      "loss": 0.1854,
+      "step": 2490
+    },
+    {
+      "epoch": 1.7412727431008967,
+      "grad_norm": 0.34207549691200256,
+      "learning_rate": 8.646706586826348e-06,
+      "loss": 0.1728,
+      "step": 2500
+    },
+    {
+      "epoch": 1.748237137633847,
+      "grad_norm": 0.24247221648693085,
+      "learning_rate": 8.598802395209581e-06,
+      "loss": 0.1747,
+      "step": 2510
+    },
+    {
+      "epoch": 1.7552015321667973,
+      "grad_norm": 0.3863085210323334,
+      "learning_rate": 8.550898203592815e-06,
+      "loss": 0.1744,
+      "step": 2520
+    },
+    {
+      "epoch": 1.7621659266997476,
+      "grad_norm": 0.38425007462501526,
+      "learning_rate": 8.50299401197605e-06,
+      "loss": 0.1675,
+      "step": 2530
+    },
+    {
+      "epoch": 1.7691303212326979,
+      "grad_norm": 0.4716668725013733,
+      "learning_rate": 8.455089820359283e-06,
+      "loss": 0.1812,
+      "step": 2540
+    },
+    {
+      "epoch": 1.7760947157656481,
+      "grad_norm": 0.4646649956703186,
+      "learning_rate": 8.407185628742516e-06,
+      "loss": 0.1628,
+      "step": 2550
+    },
+    {
+      "epoch": 1.7830591102985984,
+      "grad_norm": 0.37651118636131287,
+      "learning_rate": 8.35928143712575e-06,
+      "loss": 0.1615,
+      "step": 2560
+    },
+    {
+      "epoch": 1.7900235048315487,
+      "grad_norm": 0.4965188801288605,
+      "learning_rate": 8.311377245508983e-06,
+      "loss": 0.1807,
+      "step": 2570
+    },
+    {
+      "epoch": 1.796987899364499,
+      "grad_norm": 0.4462849497795105,
+      "learning_rate": 8.263473053892216e-06,
+      "loss": 0.1783,
+      "step": 2580
+    },
+    {
+      "epoch": 1.8039522938974493,
+      "grad_norm": 0.3630557358264923,
+      "learning_rate": 8.21556886227545e-06,
+      "loss": 0.1763,
+      "step": 2590
+    },
+    {
+      "epoch": 1.8109166884303995,
+      "grad_norm": 0.28600937128067017,
+      "learning_rate": 8.167664670658683e-06,
+      "loss": 0.168,
+      "step": 2600
+    },
+    {
+      "epoch": 1.8178810829633498,
+      "grad_norm": 0.32010987401008606,
+      "learning_rate": 8.119760479041916e-06,
+      "loss": 0.1733,
+      "step": 2610
+    },
+    {
+      "epoch": 1.8248454774963,
+      "grad_norm": 0.4362265467643738,
+      "learning_rate": 8.07185628742515e-06,
+      "loss": 0.1693,
+      "step": 2620
+    },
+    {
+      "epoch": 1.8318098720292504,
+      "grad_norm": 0.3645702600479126,
+      "learning_rate": 8.023952095808385e-06,
+      "loss": 0.1715,
+      "step": 2630
+    },
+    {
+      "epoch": 1.8387742665622007,
+      "grad_norm": 0.40849506855010986,
+      "learning_rate": 7.976047904191618e-06,
+      "loss": 0.1637,
+      "step": 2640
+    },
+    {
+      "epoch": 1.845738661095151,
+      "grad_norm": 0.4219589829444885,
+      "learning_rate": 7.928143712574851e-06,
+      "loss": 0.1679,
+      "step": 2650
+    },
+    {
+      "epoch": 1.8527030556281012,
+      "grad_norm": 0.3615240156650543,
+      "learning_rate": 7.880239520958085e-06,
+      "loss": 0.1757,
+      "step": 2660
+    },
+    {
+      "epoch": 1.8596674501610515,
+      "grad_norm": 0.4644199311733246,
+      "learning_rate": 7.832335329341318e-06,
+      "loss": 0.1819,
+      "step": 2670
+    },
+    {
+      "epoch": 1.8666318446940018,
+      "grad_norm": 0.3925911784172058,
+      "learning_rate": 7.784431137724551e-06,
+      "loss": 0.1722,
+      "step": 2680
+    },
+    {
+      "epoch": 1.873596239226952,
+      "grad_norm": 0.4104118347167969,
+      "learning_rate": 7.736526946107785e-06,
+      "loss": 0.1738,
+      "step": 2690
+    },
+    {
+      "epoch": 1.8805606337599023,
+      "grad_norm": 0.43617820739746094,
+      "learning_rate": 7.688622754491018e-06,
+      "loss": 0.1787,
+      "step": 2700
+    },
+    {
+      "epoch": 1.8875250282928526,
+      "grad_norm": 0.3555070757865906,
+      "learning_rate": 7.640718562874251e-06,
+      "loss": 0.1682,
+      "step": 2710
+    },
+    {
+      "epoch": 1.8944894228258031,
+      "grad_norm": 0.3605625629425049,
+      "learning_rate": 7.592814371257485e-06,
+      "loss": 0.172,
+      "step": 2720
+    },
+    {
+      "epoch": 1.9014538173587534,
+      "grad_norm": 0.40475067496299744,
+      "learning_rate": 7.544910179640719e-06,
+      "loss": 0.135,
+      "step": 2730
+    },
+    {
+      "epoch": 1.9084182118917037,
+      "grad_norm": 0.33572956919670105,
+      "learning_rate": 7.497005988023953e-06,
+      "loss": 0.154,
+      "step": 2740
+    },
+    {
+      "epoch": 1.915382606424654,
+      "grad_norm": 0.4499852657318115,
+      "learning_rate": 7.449101796407186e-06,
+      "loss": 0.1675,
+      "step": 2750
+    },
+    {
+      "epoch": 1.9223470009576042,
+      "grad_norm": 0.4837128221988678,
+      "learning_rate": 7.4011976047904196e-06,
+      "loss": 0.1795,
+      "step": 2760
+    },
+    {
+      "epoch": 1.9293113954905545,
+      "grad_norm": 0.36540481448173523,
+      "learning_rate": 7.353293413173654e-06,
+      "loss": 0.1758,
+      "step": 2770
+    },
+    {
+      "epoch": 1.9362757900235048,
+      "grad_norm": 0.6061664819717407,
+      "learning_rate": 7.305389221556887e-06,
+      "loss": 0.1699,
+      "step": 2780
+    },
+    {
+      "epoch": 1.943240184556455,
+      "grad_norm": 0.4293859899044037,
+      "learning_rate": 7.25748502994012e-06,
+      "loss": 0.1487,
+      "step": 2790
+    },
+    {
+      "epoch": 1.9502045790894054,
+      "grad_norm": 0.3456893563270569,
+      "learning_rate": 7.209580838323354e-06,
+      "loss": 0.167,
+      "step": 2800
+    },
+    {
+      "epoch": 1.9571689736223559,
+      "grad_norm": 0.36705002188682556,
+      "learning_rate": 7.161676646706588e-06,
+      "loss": 0.1544,
+      "step": 2810
+    },
+    {
+      "epoch": 1.9641333681553061,
+      "grad_norm": 0.4752703011035919,
+      "learning_rate": 7.113772455089821e-06,
+      "loss": 0.163,
+      "step": 2820
+    },
+    {
+      "epoch": 1.9710977626882564,
+      "grad_norm": 0.2813783288002014,
+      "learning_rate": 7.0658682634730545e-06,
+      "loss": 0.1536,
+      "step": 2830
+    },
+    {
+      "epoch": 1.9780621572212067,
+      "grad_norm": 0.39708635210990906,
+      "learning_rate": 7.017964071856288e-06,
+      "loss": 0.1596,
+      "step": 2840
+    },
+    {
+      "epoch": 1.985026551754157,
+      "grad_norm": 0.37225341796875,
+      "learning_rate": 6.970059880239521e-06,
+      "loss": 0.1551,
+      "step": 2850
+    },
+    {
+      "epoch": 1.9919909462871073,
+      "grad_norm": 0.4137498438358307,
+      "learning_rate": 6.9221556886227545e-06,
+      "loss": 0.1542,
+      "step": 2860
+    },
+    {
+      "epoch": 1.9975624619134673,
+      "eval_loss": 0.15988580882549286,
+      "eval_runtime": 355.9958,
+      "eval_samples_per_second": 3.396,
+      "eval_steps_per_second": 1.699,
+      "step": 2868
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 4305,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 478,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.338691815552811e+18,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-2868/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:956bd3fe04334201160cff0d2e3faf62e32c6c8f709339d63e01a7d4b73b24d0
+size 5560

checkpoint-3346/README.md

@@ -0,0 +1,209 @@
+---
+base_model: Salesforce/Llama-xLAM-2-8b-fc-r
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:Salesforce/Llama-xLAM-2-8b-fc-r
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

checkpoint-3346/adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Salesforce/Llama-xLAM-2-8b-fc-r",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "o_proj",
+    "down_proj",
+    "k_proj",
+    "up_proj",
+    "v_proj",
+    "gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}