💻Encoding + Inserting

Encode and inserting with VecDB

With your dataset currently stored locally, you will want to encode them and insert them into VecDB.

First - let us encode and retrieve a vector back. To encode an image or text, you can use our API below and play around with these yourself. If you are interested in understanding more about them, we recommend finishing the rest of the starter guide and then visiting the section on Concepts About Vectors.

Encode Text (Python)

import requests
def encode_text(text, project, api_key):
    return requests.post(
        url="https://api-dev-aueast.relevance.ai/v1/services/encoders/textimage",
        headers={"Authorization": project+ ":" + api_key},
        json={
            "text": text
        }
    ).json()["vector"]

Encode Image (Python)

import requests
def encode_image(image_url, project, api_key):
    return requests.post(
        url="https://api-dev-aueast.relevance.ai/v1/services/encoders/imagetext",
        headers={"Authorization": project + ":" + api_key},
        json={
            "image": image_url 
        }
    ).json()["vector"]
# Here, we get an image url from the documents from earlier and encode them!
vector = encode_image(docs[0]['image']['coco_url'])

You have just encoded your first text/image using the API!

Now, let us encode and insert them into VecDB. To help with encoding and inserting them into VecDB, we write 2 simple helper functions:

def encode_document(doc):
    """Encode the image urls in the documents and transforms them into vectors
    """
    d['image_vector_'] = encode_image(
        d['image']['coco_url'], 
        project=PROJECT, 
        api_key=API_KEY)

def chunk(docs, chunk_size=15):
    """Chunks a set of documents and inserts them!
    """
    for i in range(int(len(docs) / chunk_size)):
        yield docs[i:(i + 1) * chunk_size]

Using these 2 functions, we then insert all of our COCO images into VecDB.

You can use the code as below (warning: this can take a while but that is okay! Since we are cloud-based, we can start searching immediately!)

for c in chunk(docs):
    [encode_document(d) for d in c]
    # Now that they are encoded, we want to insert them now
    url = f"https://api-dev-aueast.relevance.ai/v1/datasets/{dataset_id}/documents/bulk_insert"
    response = requests.post(
        url=url,
        headers={"Authorization": project + ":" + api_key},
        json={
            "documents": c
        }
    )
    # Here, we want to see which docs inserted and which docs failed
    print(response.json())

Well done! You have inserted your first few documents into VecDB! How easy was that?

Now, read on to see how we perform searching in VecDB! (P.s. You do not have to finish inserting before you begin searching! In other words - the second you insert something into VecDB, it's immediately searchable seconds after!)