google/gemma-4-31B-it
Image-Text-to-Text • 33B • Updated • 2.24M • • 1.76k
Carl Johnson
fluffyone
AI & ML interests
None yet
Recent Activity
liked a model 1 day ago
google/gemma-4-31B-it liked a model 3 months ago
HeartMuLa/HeartMuLa-oss-3B liked a model 5 months ago
zai-org/GLM-4.6Organizations
None yet
liked a
model 1 day ago
liked a
model 3 months ago
liked a
model 5 months ago
liked a
model 8 months ago
liked a
model 9 months ago
liked 2
models 12 months ago
upvoted a collection about 1 year ago
liked 2
models over 1 year ago
liked a
Space over 1 year ago
liked 5
models over 1 year ago
mradermacher/model_requests
Updated • 157
mradermacher/magnum-v2-123b-i1-GGUF
123B • Updated • 64 • 7
anthracite-org/magnum-v2-123b
Text Generation • 123B • Updated • 59 • 64
AuriAetherwiing/MN-12B-Starcannon-v3
Text Generation • Updated • 114 • 26
AuriAetherwiing/MN-12B-Starcannon-v2
Text Generation • 12B • Updated • 11 • • 25
upvoted a collection over 1 year ago
reactedto Lewdiculous's post with ❤️🚀 almost 2 years ago
Post
67177
More context for your Pascal GPU or older!
Update: Now available in the official releases of KoboldCpp!
[releases] https://github.com/LostRuins/koboldcpp/releases/latest
These are great news for all the users with GTX 10XX, P40...
Flash Attention implementation for older NVIDIA GPUs without requiring Tensor Cores has come to llama.cpp in the last few days, and should be merged in the next version of KoboldCpp, you can already try it with another fork or by building it.
[Mentioned KCPP fork] https://github.com/Nexesenex/kobold.cpp/releases/latest
[PR] https://github.com/ggerganov/llama.cpp/pull/7188
You should expect less VRAM usage for the same context, allowing you to experience higher contexts with your current GPU.
There have also been reported final tokens/second speed improvements for inference, so that's also grand!
If you have tried it, I'd like to hear your experiences with --flashattention so far, especially for this implementation and for the large number of Pascal (GTX 10XX, P40...) cards.
Discussion linked bellow, with more links to relevant information:
https://huggingface.co/LWDCLS/LLM-Discussions/discussions/11
Cheers!
Update: Now available in the official releases of KoboldCpp!
[releases] https://github.com/LostRuins/koboldcpp/releases/latest
These are great news for all the users with GTX 10XX, P40...
Flash Attention implementation for older NVIDIA GPUs without requiring Tensor Cores has come to llama.cpp in the last few days, and should be merged in the next version of KoboldCpp, you can already try it with another fork or by building it.
[Mentioned KCPP fork] https://github.com/Nexesenex/kobold.cpp/releases/latest
[PR] https://github.com/ggerganov/llama.cpp/pull/7188
You should expect less VRAM usage for the same context, allowing you to experience higher contexts with your current GPU.
There have also been reported final tokens/second speed improvements for inference, so that's also grand!
If you have tried it, I'd like to hear your experiences with --flashattention so far, especially for this implementation and for the large number of Pascal (GTX 10XX, P40...) cards.
Discussion linked bellow, with more links to relevant information:
https://huggingface.co/LWDCLS/LLM-Discussions/discussions/11
Cheers!
- 24 replies