docs: update the transformers and API codes
Browse files
README.md
CHANGED
|
@@ -26,13 +26,15 @@ Embeddings produced by `jina-embeddings-v4` serve as the backbone for neural inf
|
|
| 26 |
|
| 27 |
|
| 28 |
Built based on [Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), `jina-embeddings-v4` has the following features:
|
| 29 |
-
|
|
|
|
| 30 |
- **Multilingual support** (20+ languages) and compatibility with a wide range of domains, including technical and visually complex documents.
|
| 31 |
- **Task-specific adapters** for retrieval, text matching, and code-related tasks, which can be selected at inference time.
|
| 32 |
- **Flexible embedding size**: dense embeddings are 2048 dimensions by default but can be truncated to as low as 128 with minimal performance loss.
|
| 33 |
|
| 34 |
|
| 35 |
Summary of features:
|
|
|
|
| 36 |
| Feature | Jina Embeddings V4 |
|
| 37 |
|------------|------------|
|
| 38 |
| Base Model | Qwen2.5-VL-3B-Instruct |
|
|
@@ -42,8 +44,8 @@ Summary of features:
|
|
| 42 |
| Single-Vector Dimension | 2048 |
|
| 43 |
| Multi-Vector Dimension | 128 |
|
| 44 |
| Matryoshka dimensions | 128, 256, 512, 1024, 2048 |
|
| 45 |
-
| Attention Mechanism | FlashAttention2 |
|
| 46 |
| Pooling Strategy | Mean pooling |
|
|
|
|
| 47 |
|
| 48 |
|
| 49 |
|
|
@@ -58,6 +60,7 @@ Please refer to our [technical report of jina-embeddings-v4](https://puginarug.c
|
|
| 58 |
<summary>Requirements</a></summary>
|
| 59 |
|
| 60 |
The following Python packages are required:
|
|
|
|
| 61 |
- `transformers>=4.52.0`
|
| 62 |
- `torch>=2.6.0`
|
| 63 |
- `peft>=0.15.2`
|
|
@@ -68,25 +71,21 @@ The following Python packages are required:
|
|
| 68 |
- **flash-attention**: Installing [flash-attention](https://github.com/Dao-AILab/flash-attention) is recommended for improved inference speed and efficiency, but not mandatory.
|
| 69 |
- **sentence-transformers**: If you want to use the model via the `sentence-transformers` interface, install this package as well.
|
| 70 |
|
| 71 |
-
|
| 72 |
</details>
|
| 73 |
|
| 74 |
|
| 75 |
<details>
|
| 76 |
-
<summary>via
|
|
|
|
| 77 |
|
| 78 |
-
Needs to be adjusted for V4
|
| 79 |
```bash
|
| 80 |
curl https://api.jina.ai/v1/embeddings \
|
| 81 |
-H "Content-Type: application/json" \
|
| 82 |
-
-H "Authorization: Bearer
|
| 83 |
-d @- <<EOFEOF
|
| 84 |
{
|
| 85 |
"model": "jina-embeddings-v4",
|
| 86 |
-
"
|
| 87 |
-
"task": "retrieval.query",
|
| 88 |
-
"normalized": true,
|
| 89 |
-
"embedding_type": "float",
|
| 90 |
"input": [
|
| 91 |
{
|
| 92 |
"text": "غروب جميل على الشاطئ"
|
|
@@ -136,37 +135,41 @@ EOFEOF
|
|
| 136 |
|
| 137 |
```python
|
| 138 |
# !pip install transformers>=4.52.0 torch>=2.6.0 peft>=0.15.2 torchvision pillow
|
| 139 |
-
# !pip install
|
| 140 |
from transformers import AutoModel
|
|
|
|
| 141 |
|
| 142 |
# Initialize the model
|
| 143 |
model = AutoModel.from_pretrained("jinaai/jina-embeddings-v4", trust_remote_code=True)
|
|
|
|
|
|
|
|
|
|
| 144 |
# ========================
|
| 145 |
# 1. Retrieval Task
|
| 146 |
# ========================
|
| 147 |
# Configure truncate_dim, max_length (for texts), max_pixels (for images), vector_type, batch_size in the encode function if needed
|
| 148 |
|
| 149 |
# Encode query
|
| 150 |
-
|
| 151 |
texts=["Overview of climate change impacts on coastal cities"],
|
| 152 |
task="retrieval",
|
| 153 |
prompt_name="query",
|
| 154 |
-
)
|
| 155 |
|
| 156 |
# Encode passage (text)
|
| 157 |
-
|
| 158 |
texts=[
|
| 159 |
"Climate change has led to rising sea levels, increased frequency of extreme weather events..."
|
| 160 |
],
|
| 161 |
task="retrieval",
|
| 162 |
prompt_name="passage",
|
| 163 |
-
)
|
| 164 |
|
| 165 |
# Encode image/document
|
| 166 |
-
|
| 167 |
images=["https://i.ibb.co/nQNGqL0/beach1.jpg"],
|
| 168 |
task="retrieval",
|
| 169 |
-
)
|
| 170 |
|
| 171 |
# ========================
|
| 172 |
# 2. Text Matching Task
|
|
@@ -183,25 +186,43 @@ texts = [
|
|
| 183 |
"해변 위로 아름다운 일몰", # Korean
|
| 184 |
]
|
| 185 |
|
| 186 |
-
text_embeddings = model.
|
| 187 |
|
| 188 |
# ========================
|
| 189 |
# 3. Code Understanding Task
|
| 190 |
# ========================
|
| 191 |
|
| 192 |
# Encode query
|
| 193 |
-
query_embedding = model.
|
| 194 |
texts=["Find a function that prints a greeting message to the console"],
|
| 195 |
task="code",
|
| 196 |
prompt_name="query",
|
| 197 |
)
|
| 198 |
|
| 199 |
# Encode code
|
| 200 |
-
code_embeddings = model.
|
| 201 |
texts=["def hello_world():\n print('Hello, World!')"],
|
| 202 |
task="code",
|
| 203 |
prompt_name="passage",
|
| 204 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 205 |
```
|
| 206 |
</details>
|
| 207 |
|
|
|
|
| 26 |
|
| 27 |
|
| 28 |
Built based on [Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), `jina-embeddings-v4` has the following features:
|
| 29 |
+
|
| 30 |
+
- **Unified embeddings** for text, images, and visual documents, supporting both dense (single-vector) and late-interaction (multi-vector) retrieval.
|
| 31 |
- **Multilingual support** (20+ languages) and compatibility with a wide range of domains, including technical and visually complex documents.
|
| 32 |
- **Task-specific adapters** for retrieval, text matching, and code-related tasks, which can be selected at inference time.
|
| 33 |
- **Flexible embedding size**: dense embeddings are 2048 dimensions by default but can be truncated to as low as 128 with minimal performance loss.
|
| 34 |
|
| 35 |
|
| 36 |
Summary of features:
|
| 37 |
+
|
| 38 |
| Feature | Jina Embeddings V4 |
|
| 39 |
|------------|------------|
|
| 40 |
| Base Model | Qwen2.5-VL-3B-Instruct |
|
|
|
|
| 44 |
| Single-Vector Dimension | 2048 |
|
| 45 |
| Multi-Vector Dimension | 128 |
|
| 46 |
| Matryoshka dimensions | 128, 256, 512, 1024, 2048 |
|
|
|
|
| 47 |
| Pooling Strategy | Mean pooling |
|
| 48 |
+
| Attention Mechanism | FlashAttention2 |
|
| 49 |
|
| 50 |
|
| 51 |
|
|
|
|
| 60 |
<summary>Requirements</a></summary>
|
| 61 |
|
| 62 |
The following Python packages are required:
|
| 63 |
+
|
| 64 |
- `transformers>=4.52.0`
|
| 65 |
- `torch>=2.6.0`
|
| 66 |
- `peft>=0.15.2`
|
|
|
|
| 71 |
- **flash-attention**: Installing [flash-attention](https://github.com/Dao-AILab/flash-attention) is recommended for improved inference speed and efficiency, but not mandatory.
|
| 72 |
- **sentence-transformers**: If you want to use the model via the `sentence-transformers` interface, install this package as well.
|
| 73 |
|
|
|
|
| 74 |
</details>
|
| 75 |
|
| 76 |
|
| 77 |
<details>
|
| 78 |
+
<summary>via <a href="https://jina.ai/embeddings/">Jina AI Embeddings API</a></summary>
|
| 79 |
+
|
| 80 |
|
|
|
|
| 81 |
```bash
|
| 82 |
curl https://api.jina.ai/v1/embeddings \
|
| 83 |
-H "Content-Type: application/json" \
|
| 84 |
+
-H "Authorization: Bearer $JINA_AI_API_TOKEN" \
|
| 85 |
-d @- <<EOFEOF
|
| 86 |
{
|
| 87 |
"model": "jina-embeddings-v4",
|
| 88 |
+
"task": "text-matching",
|
|
|
|
|
|
|
|
|
|
| 89 |
"input": [
|
| 90 |
{
|
| 91 |
"text": "غروب جميل على الشاطئ"
|
|
|
|
| 135 |
|
| 136 |
```python
|
| 137 |
# !pip install transformers>=4.52.0 torch>=2.6.0 peft>=0.15.2 torchvision pillow
|
| 138 |
+
# !pip install
|
| 139 |
from transformers import AutoModel
|
| 140 |
+
import torch
|
| 141 |
|
| 142 |
# Initialize the model
|
| 143 |
model = AutoModel.from_pretrained("jinaai/jina-embeddings-v4", trust_remote_code=True)
|
| 144 |
+
|
| 145 |
+
model.to("cuda")
|
| 146 |
+
|
| 147 |
# ========================
|
| 148 |
# 1. Retrieval Task
|
| 149 |
# ========================
|
| 150 |
# Configure truncate_dim, max_length (for texts), max_pixels (for images), vector_type, batch_size in the encode function if needed
|
| 151 |
|
| 152 |
# Encode query
|
| 153 |
+
query_embeddings = model.encode_text(
|
| 154 |
texts=["Overview of climate change impacts on coastal cities"],
|
| 155 |
task="retrieval",
|
| 156 |
prompt_name="query",
|
| 157 |
+
)
|
| 158 |
|
| 159 |
# Encode passage (text)
|
| 160 |
+
passage_embeddings = model.encode_text(
|
| 161 |
texts=[
|
| 162 |
"Climate change has led to rising sea levels, increased frequency of extreme weather events..."
|
| 163 |
],
|
| 164 |
task="retrieval",
|
| 165 |
prompt_name="passage",
|
| 166 |
+
)
|
| 167 |
|
| 168 |
# Encode image/document
|
| 169 |
+
image_embeddings = model.encode_image(
|
| 170 |
images=["https://i.ibb.co/nQNGqL0/beach1.jpg"],
|
| 171 |
task="retrieval",
|
| 172 |
+
)
|
| 173 |
|
| 174 |
# ========================
|
| 175 |
# 2. Text Matching Task
|
|
|
|
| 186 |
"해변 위로 아름다운 일몰", # Korean
|
| 187 |
]
|
| 188 |
|
| 189 |
+
text_embeddings = model.encode_text(texts=texts, task="text-matching")
|
| 190 |
|
| 191 |
# ========================
|
| 192 |
# 3. Code Understanding Task
|
| 193 |
# ========================
|
| 194 |
|
| 195 |
# Encode query
|
| 196 |
+
query_embedding = model.encode_text(
|
| 197 |
texts=["Find a function that prints a greeting message to the console"],
|
| 198 |
task="code",
|
| 199 |
prompt_name="query",
|
| 200 |
)
|
| 201 |
|
| 202 |
# Encode code
|
| 203 |
+
code_embeddings = model.encode_text(
|
| 204 |
texts=["def hello_world():\n print('Hello, World!')"],
|
| 205 |
task="code",
|
| 206 |
prompt_name="passage",
|
| 207 |
)
|
| 208 |
+
|
| 209 |
+
# ========================
|
| 210 |
+
# 4. Use multivectors
|
| 211 |
+
# ========================
|
| 212 |
+
|
| 213 |
+
multivector_embeddings = model.encode_text(
|
| 214 |
+
texts=texts,
|
| 215 |
+
task="retrieval",
|
| 216 |
+
prompt_name="query",
|
| 217 |
+
return_multivector=True,
|
| 218 |
+
)
|
| 219 |
+
|
| 220 |
+
images = ["https://i.ibb.co/nQNGqL0/beach1.jpg", "https://i.ibb.co/r5w8hG8/beach2.jpg"]
|
| 221 |
+
multivector_image_embeddings = model.encode_image(
|
| 222 |
+
images=images,
|
| 223 |
+
task="retrieval",
|
| 224 |
+
return_multivector=True,
|
| 225 |
+
)
|
| 226 |
```
|
| 227 |
</details>
|
| 228 |
|