Python Embeddings

Overview

Newer models

• text-embedding-3-small
• text-embedding-3-large Lower cost, better multilingual performance, controllable dimensions

Common use cases

• Search (relevance ranking)
• Clustering (similarity groups)
• Recommendations
• Anomaly detection
• Diversity analysis
• Text classification

Basic usage

Get an embedding

from openai import OpenAI
client = OpenAI(
    base_url="https://4zapi.com/v1",
    api_key=key
)

response = client.embeddings.create(
    input="Your text string goes here",
    model="text-embedding-3-small"
)

print(response.data[0].embedding)

Response shape

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [
        -0.006929283495992422,
        -0.005336422007530928
      ]
    }
  ],
  "model": "text-embedding-3-small",
  "usage": {
    "prompt_tokens": 5,
    "total_tokens": 5
  }
}

Model comparison

Example: review data

from openai import OpenAI
client = OpenAI(
    base_url="https://4zapi.com/v1",
    api_key=key
)

def get_embedding(text, model="text-embedding-3-small"):
   text = text.replace("\n", " ")
   return client.embeddings.create(input = [text], model=model).data[0].embedding

# Process a dataframe
df['ada_embedding'] = df.combined.apply(lambda x: get_embedding(x, model='text-embedding-3-small'))
df.to_csv('output/embedded_1k_reviews.csv', index=False)

# Load saved embeddings
import pandas as pd
import numpy as np

df = pd.read_csv('output/embedded_1k_reviews.csv')
df['ada_embedding'] = df.ada_embedding.apply(eval).apply(np.array)

Details

Dimensions

• text-embedding-3-small: default 1536
• text-embedding-3-large: default 3072
• Adjust with the Dimensions parameter

Notes

• Billing is based on input tokens
• About 800 tokens per page
• All models max input is 8191 tokens