Hugging Face's logo

inference-optimization
/
Llama-3.1-8B-Instruct-QKV-Cache-FP8-Per-Head

Safetensors
llama
compressed-tensors
Model card Files
xet
Community
krishnateja95 commited on
Commit
ca74a77
·
verified ·
1 Parent(s): 584824a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +106 -3
README.md CHANGED
@@ -1,3 +1,106 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model:
4
+ - meta-llama/Llama-3.1-8B-Instruct
5
+ ---
6
+
7
+
8
+
9
+ ### Accuracy
10
+ <table>
11
+ <thead>
12
+ <tr>
13
+ <th>Task</th>
14
+ <th>Context Length</th>
15
+ <th>meta-llama/<br>Llama-3.1-8B-Instruct</th>
16
+ <th>Llama-3.1-8B-Instruct-<br>FP8-dynamic-<br>QKV-Cache-FP8-<br>Per-Head</th>
17
+ <th>Llama-3.1-8B-Instruct-<br>FP8-dynamic-<br>QKV-Cache-FP8-<br>Per-Tensor</th>
18
+ <th>Llama-3.1-8B-Instruct-<br>QKV-Cache-FP8-<br>Per-Head</th>
19
+ <th>Llama-3.1-8B-Instruct-<br>QKV-Cache-FP8-<br>Per-Tensor</th>
20
+ </tr>
21
+ </thead>
22
+ <tbody>
23
+ <tr>
24
+ <td rowspan="5"><b>NIAH<br>Single 2</b></td>
25
+ <td>4096</td>
26
+ <td>100.00</td>
27
+ <td>100.00</td>
28
+ <td>100.00</td>
29
+ <td>100.00</td>
30
+ <td>100.00</td>
31
+ </tr>
32
+ <tr>
33
+ <td>16384</td>
34
+ <td>100.00</td>
35
+ <td>100.00</td>
36
+ <td>100.00</td>
37
+ <td>100.00</td>
38
+ <td>100.00</td>
39
+ </tr>
40
+ <tr>
41
+ <td>32768</td>
42
+ <td>100.00</td>
43
+ <td>100.00</td>
44
+ <td>100.00</td>
45
+ <td>100.00</td>
46
+ <td>100.00</td>
47
+ </tr>
48
+ <tr>
49
+ <td>65536</td>
50
+ <td>100.00</td>
51
+ <td>100.00</td>
52
+ <td>100.00</td>
53
+ <td>100.00</td>
54
+ <td>100.00</td>
55
+ </tr>
56
+ <tr>
57
+ <td>131072</td>
58
+ <td>100.00</td>
59
+ <td>100.00</td>
60
+ <td>100.00</td>
61
+ <td>100.00</td>
62
+ <td>100.00</td>
63
+ </tr>
64
+ <tr>
65
+ <td rowspan="5"><b>NIAH<br>Single 3</b></td>
66
+ <td>4096</td>
67
+ <td>100.00</td>
68
+ <td>100.00</td>
69
+ <td>100.00</td>
70
+ <td>100.00</td>
71
+ <td>100.00</td>
72
+ </tr>
73
+ <tr>
74
+ <td>16384</td>
75
+ <td>100.00</td>
76
+ <td>100.00</td>
77
+ <td>100.00</td>
78
+ <td>100.00</td>
79
+ <td>100.00</td>
80
+ </tr>
81
+ <tr>
82
+ <td>32768</td>
83
+ <td>100.00</td>
84
+ <td>100.00</td>
85
+ <td>100.00</td>
86
+ <td>100.00</td>
87
+ <td>100.00</td>
88
+ </tr>
89
+ <tr>
90
+ <td>65536</td>
91
+ <td>100.00</td>
92
+ <td>100.00</td>
93
+ <td>100.00</td>
94
+ <td>100.00</td>
95
+ <td>100.00</td>
96
+ </tr>
97
+ <tr>
98
+ <td>131072</td>
99
+ <td>100.00</td>
100
+ <td>100.00</td>
101
+ <td>100.00</td>
102
+ <td>100.00</td>
103
+ <td>100.00</td>
104
+ </tr>
105
+ </tbody>
106
+ </table>