{"id":774,"date":"2023-07-26T08:28:59","date_gmt":"2023-07-26T08:28:59","guid":{"rendered":"https:\/\/tbekk.com\/devstream\/?p=774"},"modified":"2023-07-26T08:30:04","modified_gmt":"2023-07-26T08:30:04","slug":"getting-started-with-vector-dbs-in-python","status":"publish","type":"post","link":"https:\/\/tbekk.com\/devstream\/2023\/07\/26\/getting-started-with-vector-dbs-in-python\/","title":{"rendered":"Getting started with Vector DBs in Python"},"content":{"rendered":"\n<hr class=\"wp-block-separator has-text-color has-light-gray-color has-alpha-channel-opacity has-light-gray-background-color has-background is-style-wide\"\/>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><em>Link: <\/em><\/strong><a href=\"https:\/\/code.dblock.org\/2023\/06\/16\/getting-started-with-vector-dbs-in-python.html#clickhouse\"><em>code.dblock.org<\/em><\/a><\/li>\n\n\n\n<li><strong><em>Source code: <\/em><\/strong><a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\"><em>github ~ vectordb-hello-world<\/em><\/a><\/li>\n\n\n\n<li><strong><em>Author: <\/em><\/strong><a href=\"https:\/\/code.dblock.org\/about\"><em>Daniel Doubrovkine<\/em><\/a><\/li>\n\n\n\n<li><em><strong>Publication date: <\/strong>July 26, 2023<\/em> <\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-text-color has-light-gray-color has-alpha-channel-opacity has-light-gray-background-color has-background is-style-wide\"\/>\n\n\n\n<p>Vector databases are all the rage today.<\/p>\n\n\n\n<p>I\u2019ve built a few iterations of vector search, beginning in 2011 at Artsy, powered by the&nbsp;<a href=\"https:\/\/en.wikipedia.org\/wiki\/The_Art_Genome_Project\">Art Genome Project<\/a>. Compared to LLM use-cases today, Artsy is a small, 1200-dimensional sparse vector and semantic search engine. The first attempt at vector search resulted in a brute-force exact k-nearest-neighbor search with data stored in MongoDB, written in Ruby. The second attempt was an approximate nearest-neighbor implementation using&nbsp;<a href=\"https:\/\/en.wikipedia.org\/wiki\/Locality-sensitive_hashing\">LSH<\/a>, and finally&nbsp;<a href=\"https:\/\/www.cs.princeton.edu\/cass\/papers\/www11.pdf\">NN-Descent<\/a>. Around 2017 we migrated to Elasticsearch, and I am speculating the team has moved to OpenSearch by now because it\u2019s open-source.<\/p>\n\n\n\n<p>Things have evolved rapidly with generative AI, so let\u2019s try to index and search some vectors in 2023 in Python, using the simplest of the libraries, usually pure HTTP when available. You can draw your own conclusions of which engines are better and\/or easier to use. Working code for this blog post is&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\">here<\/a>.<\/p>\n\n\n\n<p>In alphabetical order.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/code.dblock.org\/2023\/06\/16\/getting-started-with-vector-dbs-in-python.html#clickhouse\">ClickHouse<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/code.dblock.org\/2023\/06\/16\/getting-started-with-vector-dbs-in-python.html#myscale\">MyScale<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/code.dblock.org\/2023\/06\/16\/getting-started-with-vector-dbs-in-python.html#opensearch\">OpenSearch<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/code.dblock.org\/2023\/06\/16\/getting-started-with-vector-dbs-in-python.html#pgvector\">pgVector<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/code.dblock.org\/2023\/06\/16\/getting-started-with-vector-dbs-in-python.html#pinecone\">Pinecone<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/code.dblock.org\/2023\/06\/16\/getting-started-with-vector-dbs-in-python.html#qdrant\">Qdrant<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/code.dblock.org\/2023\/06\/16\/getting-started-with-vector-dbs-in-python.html#redis\">Redis<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/code.dblock.org\/2023\/06\/16\/getting-started-with-vector-dbs-in-python.html#vespa\">Vespa<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/code.dblock.org\/2023\/06\/16\/getting-started-with-vector-dbs-in-python.html#weaviate\">Weaviate<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/code.dblock.org\/2023\/06\/16\/getting-started-with-vector-dbs-in-python.html#others\">Others<\/a><\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"clickhouse\">ClickHouse<\/h3>\n\n\n\n<p><a href=\"https:\/\/clickhouse.com\/\">ClickHouse<\/a>&nbsp;is a fast and resource efficient open-source database for real-time apps and analytics. You can&nbsp;<a href=\"https:\/\/clickhouse.com\/#getting_started\">download a free version<\/a>&nbsp;or use&nbsp;<a href=\"https:\/\/clickhouse.com\/\">ClickHouse Cloud<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker run -p 9000:9000 -p 9009:9009 -p 8123:8123 --platform linux\/amd64 --ulimit nofile<strong>=<\/strong>262144:262144 clickhouse\/clickhouse-server<\/code><\/pre>\n\n\n\n<p>ClickHouse offers an HTTP interface.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>endpoint <strong>=<\/strong> os.getenv(\"ENDPOINT\", \"http:\/\/localhost:8123\")\nclient <strong>=<\/strong> Client()\n<strong>print<\/strong>(client.get(endpoint).text)<\/code><\/pre>\n\n\n\n<p>Create a table with a k-nn index. Note&nbsp;<code>allow_experimental_annoy_index=1<\/code>&nbsp;in the query string that turns on the&nbsp;<a href=\"https:\/\/clickhouse.com\/docs\/en\/engines\/table-engines\/mergetree-family\/annindexes\">approximate nearest neighbor<\/a>&nbsp;index feature.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>client.post(endpoint, params<strong>=<\/strong>\"allow_experimental_annoy_index=1\", data<strong>=<\/strong>\n    \"CREATE TABLE IF NOT EXISTS default.vectors (\" \\\n        \"id String,\" \\\n        \"values Array(Float32),\" \\\n        \"metadata Map(String, String),\" \\\n        \"CONSTRAINT check_length CHECK length(values) = 3,\" \\\n        \"INDEX values_index values TYPE annoy GRANULARITY 100\" \\\n    \") \" \\\n    \"ENGINE = MergeTree \" \\\n    \"ORDER BY id\"\n)<\/code><\/pre>\n\n\n\n<p>Insert some vectors.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vectors <strong>=<\/strong> &#91;\n    {\n        \"id\": \"vec1\",\n        \"values\": &#91;0.1, 0.2, 0.3],\n        \"metadata\": {\"genre\": \"drama\"},\n    },\n    {\n        \"id\": \"vec2\",\n        \"values\": &#91;0.2, 0.3, 0.4],\n        \"metadata\": {\"genre\": \"action\"},\n    },\n]\n\n<strong>for<\/strong> vector <strong>in<\/strong> vectors:\n    client.post(endpoint, data<strong>=<\/strong>\n        f\"INSERT INTO default.vectors (id, values, metadata) \" \\\n        f\"VALUES (\\'{vector&#91;'id']}\\', {vector&#91;'values']}, {vector&#91;'metadata']})\"\n    )<\/code><\/pre>\n\n\n\n<p>Search.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>results <strong>=<\/strong> client.post(endpoint, data<strong>=<\/strong>\n    \"SELECT * \" \\\n    \"FROM default.vectors \" \\\n    \"WHERE metadata&#91;'genre']='action' \" \\\n    \"ORDER BY L2Distance(values, &#91;0.2, 0.3, 0.4])\"\n)\n\n<strong>print<\/strong>(results.text)  <\/code><\/pre>\n\n\n\n<p>You can see and run a&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\/blob\/main\/src\/click_house\/hello.py\">working sample from here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>poetry run .\/hello.py\n\n<strong>&gt;<\/strong> POST http:\/\/localhost:8123?allow_experimental_annoy_index<strong>=<\/strong>1\n  CREATE TABLE IF NOT EXISTS default.vectors <strong>(<\/strong>id String,values Array<strong>(<\/strong>Float32<strong>)<\/strong>,metadata Map<strong>(<\/strong>String, String<strong>)<\/strong>,CONSTRAINT check_length CHECK length<strong>(<\/strong>values<strong>)<\/strong> <strong>=<\/strong> 3,INDEX values_index values TYPE annoy GRANULARITY 100<strong>)<\/strong> ENGINE <strong>=<\/strong> MergeTree ORDER BY id\n&lt; POST http:\/\/localhost:8123?allow_experimental_annoy_index<strong>=<\/strong>1 - 200\n<strong>&gt;<\/strong> POST http:\/\/localhost:8123\n  INSERT INTO default.vectors <strong>(<\/strong>id, values, metadata<strong>)<\/strong> VALUES <strong>(<\/strong>'vec1', <strong>&#91;<\/strong>0.1, 0.2, 0.3], <strong>{<\/strong>'genre': 'drama'<strong>})<\/strong>\n&lt; POST http:\/\/localhost:8123 - 200\n<strong>&gt;<\/strong> POST http:\/\/localhost:8123\n  INSERT INTO default.vectors <strong>(<\/strong>id, values, metadata<strong>)<\/strong> VALUES <strong>(<\/strong>'vec2', <strong>&#91;<\/strong>0.2, 0.3, 0.4], <strong>{<\/strong>'genre': 'action'<strong>})<\/strong>\n&lt; POST http:\/\/localhost:8123 - 200\n<strong>&gt;<\/strong> POST http:\/\/localhost:8123\n  SELECT <strong>*<\/strong> FROM default.vectors WHERE metadata&#91;'genre'<strong>]=<\/strong>'action' ORDER BY L2Distance<strong>(<\/strong>values, <strong>&#91;<\/strong>0.2, 0.3, 0.4]<strong>)<\/strong>\n&lt; POST http:\/\/localhost:8123 - 200\nvec2\t<strong>&#91;<\/strong>0.2,0.3,0.4]\t<strong>{<\/strong>'genre':'action'<strong>}<\/strong>\n\n<strong>&gt;<\/strong> POST http:\/\/localhost:8123\n  DROP TABLE default.vectors\n&lt; POST http:\/\/localhost:8123 - 200<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"myscale\">MyScale<\/h3>\n\n\n\n<p><a href=\"https:\/\/myscale.com\/\">MyScale<\/a>&nbsp;performs vector search in SQL, and&nbsp;<a href=\"https:\/\/blog.myscale.com\/2023\/05\/17\/myscale-outperform-special-vectordb\/\">claims<\/a>&nbsp;to outperform other solutions by using a proprietary algorithm called&nbsp;<code>MSTG<\/code>. MyScale is built on the open-source ClickHouse, so the code is almost identical, except that one uses&nbsp;<code>VECTOR INDEX values_index values TYPE MSTG<\/code>.<\/p>\n\n\n\n<p>Sign up&nbsp;<a href=\"https:\/\/myscale.com\/\">on their website<\/a>&nbsp;for a test cluster, note the username and password. You can see and run a&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\/blob\/main\/src\/my_scale\/hello.py\">working sample from here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>USERNAME<strong>=<\/strong>... PASSWORD<strong>=<\/strong>... ENDPOINT<strong>=<\/strong>https:\/\/...aws.myscale.com:443 poetry run .\/hello.py\n\n<strong>&gt;<\/strong> POST https:\/\/...aws.myscale.com\n  CREATE TABLE IF NOT EXISTS default.vectors <strong>(<\/strong>id String,values Array<strong>(<\/strong>Float32<strong>)<\/strong>,metadata Map<strong>(<\/strong>String, String<strong>)<\/strong>,CONSTRAINT check_length CHECK length<strong>(<\/strong>values<strong>)<\/strong> <strong>=<\/strong> 3,VECTOR INDEX values_index values TYPE MSTG<strong>)<\/strong> ENGINE <strong>=<\/strong> MergeTree ORDER BY id\n&lt; POST https:\/\/...aws.myscale.com - 200\n<strong>&gt;<\/strong> POST https:\/\/...aws.myscale.com\n  INSERT INTO default.vectors <strong>(<\/strong>id, values, metadata<strong>)<\/strong> VALUES <strong>(<\/strong>'vec1', <strong>&#91;<\/strong>0.1, 0.2, 0.3], <strong>{<\/strong>'genre': 'drama'<strong>})<\/strong>\n&lt; POST https:\/\/...aws.myscale.com - 200\n<strong>&gt;<\/strong> POST https:\/\/...aws.myscale.com\n  INSERT INTO default.vectors <strong>(<\/strong>id, values, metadata<strong>)<\/strong> VALUES <strong>(<\/strong>'vec2', <strong>&#91;<\/strong>0.2, 0.3, 0.4], <strong>{<\/strong>'genre': 'action'<strong>})<\/strong>\n&lt; POST https:\/\/...aws.myscale.com - 200\n<strong>&gt;<\/strong> POST https:\/\/...aws.myscale.com\n  SELECT <strong>*<\/strong> FROM default.vectors WHERE metadata&#91;'genre'<strong>]=<\/strong>'action' ORDER BY L2Distance<strong>(<\/strong>values, <strong>&#91;<\/strong>0.2, 0.3, 0.4]<strong>)<\/strong>\n&lt; POST https:\/\/...aws.myscale.com - 200\nvec2\t<strong>&#91;<\/strong>0.2,0.3,0.4]\t<strong>{<\/strong>'genre':'action'<strong>}<\/strong>\n\n<strong>&gt;<\/strong> POST https:\/\/...aws.myscale.com\n  DROP TABLE default.vectors\n&lt; POST https:\/\/...aws.myscale.com - 200<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"opensearch\">OpenSearch<\/h3>\n\n\n\n<p><a href=\"https:\/\/opensearch.org\/\">OpenSearch<\/a>&nbsp;is a scalable, flexible, and extensible open-source software suite for search, analytics, and observability applications licensed under Apache 2.0. You can use a managed service, such as&nbsp;<a href=\"https:\/\/aws.amazon.com\/opensearch-service\/\">Amazon OpenSearch<\/a>, or download and install it locally. I usually do the latter, mostly because it\u2019s trivial, and I can work offline.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker pull opensearchproject\/opensearch:latest\ndocker run -d -p 9200:9200 -p 9600:9600 -e \"discovery.type=single-node\" opensearchproject\/opensearch:latest<\/code><\/pre>\n\n\n\n<p>Whichever option you choose you get a single endpoint (e.g. \u201chttps:\/\/localhost:9200\u201d). Locally it uses basic auth and has self-signed SSL certificates, therefore needs&nbsp;<code>verify=False<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>endpoint <strong>=<\/strong> \"https:\/\/localhost:9200\"\nusername <strong>=<\/strong> \"admin\"\npassword <strong>=<\/strong> \"admin\"\nauth <strong>=<\/strong> BasicAuth(username<strong>=<\/strong>username, password<strong>=<\/strong>password)\nclient <strong>=<\/strong> Client(verify<strong>=<\/strong>False, auth<strong>=<\/strong>auth)\nheaders <strong>=<\/strong> {\n    \"Accept\": \"application\/json; charset=utf-8\",\n    \"Content-Type\": \"application\/json; charset=utf-8\",\n}<\/code><\/pre>\n\n\n\n<p>We can get a list of existing indexes. This is a data structure with a ton of useful information, but we\u2019ll make a dictionary out of it, and use it to check whether an index exists.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>indices <strong>=<\/strong> { x&#91;\"index\"]: x <strong>for<\/strong> x <strong>in<\/strong>\n    client.get(\n        urljoin(endpoint, \"\/_cat\/indices\"), \n        headers<strong>=<\/strong>headers\n    ).json()\n}<\/code><\/pre>\n\n\n\n<p>If an index doesn\u2019t exist, we can create one. The syntax enables k-nn vector search, and include so-called property mappings. It will also need to have a fixed number of dimensions for our vectors.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>client.put(\n    urljoin(endpoint, f\"\/{index_name}\"),\n    headers<strong>=<\/strong>headers,\n    json<strong>=<\/strong>{\n        \"settings\": {\"index.knn\": True},\n        \"mappings\": {\n            \"properties\": {\n                \"values\": {\n                    \"type\": \"knn_vector\",\n                    \"dimension\": 3\n                }\n            }\n        }\n    }\n)<\/code><\/pre>\n\n\n\n<p>Indexing data can be done document-by-document or via the bulk API, which requires newline-delimited JSON. We start with some data.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vectors <strong>=<\/strong> &#91;\n    {\n        \"id\": \"vec1\",\n        \"values\": &#91;0.1, 0.2, 0.3],\n        \"metadata\": {\"genre\": \"drama\"},\n    },\n    {\n        \"id\": \"vec2\",\n        \"values\": &#91;0.2, 0.3, 0.4],\n        \"metadata\": {\"genre\": \"action\"},\n    },\n]<\/code><\/pre>\n\n\n\n<p>You can insert document-by-document.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>for<\/strong> vector <strong>in<\/strong> vectors:\n    client.post(\n        urljoin(endpoint, f\"\/{index_name}\/_doc\/{vector&#91;'id']}\"),\n                headers<strong>=<\/strong>headers,\n                json<strong>=<\/strong>vector\n    )<\/code><\/pre>\n\n\n\n<p>Or bulk insert, which asks to separate document IDs from document data, so I purposely wrote it in a way that starts with combined vector documents that include IDs, and generates JSON that the bulk API accepts as a transform.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>data <strong>=<\/strong> \"\"\n<strong>for<\/strong> vector <strong>in<\/strong> vectors:\n    data <strong>+=<\/strong> json.dumps({ \"index\": {\"_index\": index_name, \"_id\": vector&#91;\"id\"]} }) <strong>+<\/strong> \"\\n\"\n    data <strong>+=<\/strong> json.dumps({i: vector&#91;i] <strong>for<\/strong> i <strong>in<\/strong> vector <strong>if<\/strong> i <strong>!=<\/strong> \"id\"}) <strong>+<\/strong> \"\\n\"\n\nclient.post(urljoin(endpoint, \"\/_bulk\"), headers<strong>=<\/strong>headers, data<strong>=<\/strong>data)<\/code><\/pre>\n\n\n\n<p>Search for data.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>query <strong>=<\/strong> {\n    \"query\": {\n        \"knn\": {\n            \"values\": {\n                \"vector\": &#91;0.1, 0.2, 0.3],\n                \"k\": 1\n            }\n        }\n    }\n}\n\nresults <strong>=<\/strong> client.post(\n    urljoin(endpoint, f\"\/{index_name}\/_search\"),\n        headers<strong>=<\/strong>headers,\n        json<strong>=<\/strong>query\n).json()<\/code><\/pre>\n\n\n\n<p>You can see and run a&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\/blob\/main\/src\/open_search\/hello.py\">working sample from here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>USERNAME<strong>=<\/strong>admin PASSWORD<strong>=<\/strong>admin ENDPOINT<strong>=<\/strong>https:\/\/localhost:9200 poetry run src\/open_search\/hello.py\n\n<strong>&gt;<\/strong> GET https:\/\/localhost:9200\/_cat\/indices\n&lt; GET https:\/\/localhost:9200\/_cat\/indices - 200\n<strong>&gt;<\/strong> PUT https:\/\/localhost:9200\/my-index\n&lt; PUT https:\/\/localhost:9200\/my-index - 200\n<strong>&gt;<\/strong> POST https:\/\/localhost:9200\/_bulk\n&lt; POST https:\/\/localhost:9200\/_bulk - 200\n<strong>&gt;<\/strong> POST https:\/\/localhost:9200\/my-index\/_search\n&lt; POST https:\/\/localhost:9200\/my-index\/_search - 200\n<strong>{<\/strong>'total': <strong>{<\/strong>'value': 1, 'relation': 'eq'<strong>}<\/strong>, 'max_score': 0.97087383, 'hits': <strong>&#91;{<\/strong>'_index': 'my-index', '_id': 'vec1', '_score': 0.97087383, '_source': <strong>{<\/strong>'index': <strong>{<\/strong>'_index': 'my-index', '_id': 'vec2'<strong>}<\/strong>, 'values': <strong>&#91;<\/strong>0.2, 0.3, 0.4], 'metadata': <strong>{<\/strong>'genre': 'action'<strong>}}}]}<\/strong><\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"pgvector\">pgVector<\/h3>\n\n\n\n<p><a href=\"https:\/\/github.com\/pgvector\/pgvector\">pgVector<\/a>&nbsp;adds vector similarity search to open-source Postgres. You can use a local docker installation from&nbsp;<a href=\"https:\/\/hub.docker.com\/r\/ankane\/pgvector\">ankane\/pgvector<\/a>, or a&nbsp;<a href=\"https:\/\/github.com\/pgvector\/pgvector#hosted-postgres\">managed service<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker pull ankane\/pgvector or https:\/\/github.com\/pgvector\/pgvector\/issues\/54 <strong>for <\/strong>cloud providers\ndocker run -e POSTGRES_PASSWORD<strong>=<\/strong>password -p 5433:5432 ankane\/pgvector<\/code><\/pre>\n\n\n\n<p>PostgreSQL speaks its own message-based protocol, and queries are made in SQL, which is not HTTP, hence we\u2019re going to use&nbsp;<a href=\"https:\/\/github.com\/MagicStack\/asyncpg\">asyncpg<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>database <strong>=<\/strong> \"vectors\"\nconn <strong>=<\/strong> <strong>await<\/strong> asyncpg.connect(database<strong>=<\/strong>\"template1\")\nonn.execute(f\"CREATE DATABASE \\\"{database}\\\"\")<\/code><\/pre>\n\n\n\n<p>Enable vector extensions on the index.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>await<\/strong> conn.execute(f\"CREATE EXTENSION vector\")\n<strong>await<\/strong> pgvector.asyncpg.register_vector(conn)<\/code><\/pre>\n\n\n\n<p>Create a schema with a custom primary key, a 3-dimensional vector, and some JSON metadata.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>await<\/strong> conn.execute(\n    f\"CREATE TABLE vectors (id text PRIMARY KEY, values vector(3), metadata JSONB)\"\n)<\/code><\/pre>\n\n\n\n<p>Insert vectors.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vectors <strong>=<\/strong> &#91;\n    {\n        \"id\": \"vec1\",\n        \"values\": &#91;0.1, 0.2, 0.3],\n        \"metadata\": {\"genre\": \"drama\"},\n    },\n    {\n        \"id\": \"vec2\",\n        \"values\": &#91;0.2, 0.3, 0.4],\n        \"metadata\": {\"genre\": \"action\"},\n    },\n]\n\n<strong>for<\/strong> vector <strong>in<\/strong> vectors:\n    q <strong>=<\/strong> f\"INSERT INTO vectors (id, values, metadata) VALUES($1, $2, $3)\"\n    <strong>await<\/strong> conn.execute(q,\n        vector&#91;'id'],\n        vector&#91;'values'],\n        json.dumps(vector&#91;'metadata'])\n    )<\/code><\/pre>\n\n\n\n<p>Search. In the example below we filter by&nbsp;<code>genre<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>q <strong>=<\/strong> \"SELECT * FROM vectors WHERE metadata-&gt;&gt;'genre'='action' ORDER BY values &lt;-&gt; '&#91;0.2,0.1,0.5]'\"\nresults <strong>=<\/strong> <strong>await<\/strong> conn.fetch(q)\n<strong>for<\/strong> result <strong>in<\/strong> results:\n    <strong>print<\/strong>(f\"{result} ({json.loads(result&#91;'metadata'])&#91;'genre']})\")<\/code><\/pre>\n\n\n\n<p>Finally, drop this database.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>await<\/strong> conn.execute(f\"DROP DATABASE \\\"{database}\\\"\")<\/code><\/pre>\n\n\n\n<p>You can see and run a&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\/blob\/main\/src\/pg_vector\/hello.py\">working sample from here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>PGPORT<strong>=<\/strong>5433 PGUSER<strong>=<\/strong>postgres PGPASSWORD<strong>=<\/strong>password poetry run .\/hello.py\n\n&lt;Record id<strong>=<\/strong>'vec2' values<strong>=<\/strong>array<strong>(&#91;<\/strong>0.2, 0.3, 0.4], dtype<strong>=<\/strong>float32<strong>)<\/strong> metadata<strong>=<\/strong>'{\"genre\": \"action\"}'<strong>&gt;<\/strong> <strong>(<\/strong>action<strong>)<\/strong><\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"pinecone\">Pinecone<\/h3>\n\n\n\n<p>The&nbsp;<a href=\"https:\/\/www.pinecone.io\/\">Pinecone vector database<\/a>&nbsp;is easy to build high-performance vector search applications with, developer-friendly, fully managed, and scalable without infrastructure hassles.<\/p>\n\n\n\n<p>Conceptually it has indexes (which are really databases, and were probably originally called as such as the API has&nbsp;<code>\/databases<\/code>&nbsp;in it). After signing up to Pinecone you get a regional endpoint and a project ID. These form a controller URI (e.g.&nbsp;<code>https:\/\/controller.us-west4-gcp-free.pinecone.io\/<\/code>) for database operations. After you create an index, that gets its own URI that combines the index name (e.g. \u201cmy-index\u201d) and a project ID (e.g.&nbsp;<code>https:\/\/my-index-c7556fa.svc.us-west4-gcp-free.pinecone.io<\/code>). It\u2019s not quite serverless, as you do have to reason about&nbsp;<a href=\"https:\/\/docs.pinecone.io\/docs\/indexes\">pods<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>from<\/strong> urllib.parse <strong>import<\/strong> urljoin, urlparse\n\nendpoint <strong>=<\/strong> urlparse(\"https:\/\/us-west4-gcp-free.pinecone.io\")\nproject_id <strong>=<\/strong> os.environ&#91;\"PROJECT_ID\"]\n\ncontroller_endpoint <strong>=<\/strong> endpoint._replace(netloc<strong>=<\/strong>f\"controller.{endpoint.netloc}\").geturl()\n\nservice_endpoint <strong>=<\/strong> endpoint._replace(netloc<strong>=<\/strong>f'my-index-{project_id}.svc.{endpoint.netloc}').geturl()<\/code><\/pre>\n\n\n\n<p>Authentication is performed using a required API key.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>from<\/strong> httpx <strong>import<\/strong> Client\n\napi_key <strong>=<\/strong> os.environ&#91;\"API_KEY\"]\n\nclient <strong>=<\/strong> Client()\n\nheaders <strong>=<\/strong> {\n    \"Accept\": \"application\/json; charset=utf-8\",\n    \"Content-Type\": \"application\/json; charset=utf-8\",\n    \"Api-Key\": api_key,\n}<\/code><\/pre>\n\n\n\n<p>We can get a list of existing indexes. This is just a list of names, useful to check whether an index exists.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>indices <strong>=<\/strong> client.get(\n    urljoin(controller_endpoint, \"\/databases\"),\n        headers<strong>=<\/strong>headers\n).json()<\/code><\/pre>\n\n\n\n<p>If an index doesn\u2019t exist, we can create one. It will need to have a fixed number of dimensions for our vectors.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>client.post(\n    urljoin(controller_endpoint, \"\/databases\"),\n    headers<strong>=<\/strong>headers,\n    json<strong>=<\/strong>{\"name\": index_name, \"dimension\": 3},\n)<\/code><\/pre>\n\n\n\n<p>Index data.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vectors <strong>=<\/strong> &#91;\n    {\n        \"id\": \"vec1\",\n        \"values\": &#91;0.1, 0.2, 0.3],\n        \"metadata\": {\"genre\": \"drama\"},\n    },\n    {\n        \"id\": \"vec2\",\n        \"values\": &#91;0.2, 0.3, 0.4],\n        \"metadata\": {\"genre\": \"action\"},\n    },\n]\n\nclient.post(\n    urljoin(service_endpoint, \"\/vectors\/upsert\"),\n    headers<strong>=<\/strong>headers,\n    json<strong>=<\/strong>{\"vectors\": vectors, \"namespace\": \"namespace\"},\n)<\/code><\/pre>\n\n\n\n<p>Search for this vector data.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>results <strong>=<\/strong> client.post(\n    urljoin(service_endpoint, \"\/query\"),\n    headers<strong>=<\/strong>headers,\n    json<strong>=<\/strong>{\n        \"vector\": &#91;0.1, 0.2, 0.3],\n        \"top_k\": 1,\n        \"namespace\": \"namespace\",\n        \"includeMetadata\": True,\n    },\n).json()<\/code><\/pre>\n\n\n\n<p>You can see and run a&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\/blob\/main\/src\/pinecone\/hello.py\">working sample from here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>API_KEY<strong>=<\/strong>... PROJECT_ID<strong>=<\/strong>... ENDPOINT<strong>=<\/strong>https:\/\/us-west4-gcp-free.pinecone.io poetry run src\/pinecone\/hello.py\n\n<strong>&gt;<\/strong> GET https:\/\/controller.us-west4-gcp-free.pinecone.io\/databases\n&lt; GET https:\/\/controller.us-west4-gcp-free.pinecone.io\/databases - 200\n<strong>&gt;<\/strong> POST https:\/\/my-index-c7556fa.svc.us-west4-gcp-free.pinecone.io\/vectors\/upsert\n&lt; POST https:\/\/my-index-c7556fa.svc.us-west4-gcp-free.pinecone.io\/vectors\/upsert - 200\n<strong>&gt;<\/strong> POST https:\/\/my-index-c7556fa.svc.us-west4-gcp-free.pinecone.io\/query\n&lt; POST https:\/\/my-index-c7556fa.svc.us-west4-gcp-free.pinecone.io\/query - 200\n<strong>{<\/strong>'results': <strong>&#91;]<\/strong>, 'matches': <strong>&#91;{<\/strong>'id': 'vec1', 'score': 0.999999881, 'values': <strong>&#91;]<\/strong>, 'metadata': <strong>{<\/strong>'genre': 'drama'<strong>}}]<\/strong>, 'namespace': 'namespace'<strong>}<\/strong><\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"qdrant\">Qdrant<\/h3>\n\n\n\n<p><a href=\"https:\/\/qdrant.tech\/\">Qdrant<\/a>&nbsp;is a similarity vector search engine designed for a wide range of applications, including recommendation systems, image search, and natural language processing. It is scalable and allows dynamic updates to the index. It is particularly suitable for scenarios where the vector data is constantly evolving and vectors may be modified without interrupting the search functionality. Qdrant is licensed under Apache 2.0.<\/p>\n\n\n\n<p>Qdrant is built upon a concept of indexes, where vectors are organized and stored in \u201ccollections\u201d for quick retrieval. Currently, it only supports HNSW (Hierarchical Navigable Small World) as vector index.<\/p>\n\n\n\n<p>After you sign up at Qdrant Cloud Services, create a new free tier Qdrant Cluster with authentication. Note your cluster URL and API key. The endpoint will have the following format&nbsp;<code>https:\/\/my-cluster.cloud.qdrant.io:6333\/<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>client <strong>=<\/strong> Client()\n\nendpoint <strong>=<\/strong> os.environ&#91;\"ENDPOINT\"]\napi_key <strong>=<\/strong> os.environ&#91;\"API_KEY\"]\n\nheaders <strong>=<\/strong> {\n    \"Accept\": \"application\/json; charset=utf-8\",\n    \"Content-Type\": \"application\/json; charset=utf-8\",\n    \"api-key\": api_key\n}<\/code><\/pre>\n\n\n\n<p>We can create an index in collections.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>index_name <strong>=<\/strong> \"my-index\"\n\nindex <strong>=<\/strong> {\n    \"vectors\": {\n        \"size\": 3,\n        \"distance\": \"Cosine\"\n    }\n}\n\nvectors <strong>=<\/strong> &#91;\n    {\n        \"id\": 1,\n        \"vector\": &#91;0.1, 0.2, 0.3],\n        \"payload\": {\n            \"genre\": \"drama\"\n        }\n    },\n    {\n        \"id\": 2,\n        \"vector\": &#91;0.2, 0.3, 0.4],\n        \"payload\": {\n            \"genre\": \"action\"\n        }\n    }\n]\n\npayload <strong>=<\/strong> {\"points\": vectors}\n\nclient.get(urljoin(endpoint, \"collections\"), headers<strong>=<\/strong>headers),\n\nclient.put(\n    urljoin(endpoint, f\"\/collections\/{index_name}\"),\n        json<strong>=<\/strong>index,\n        headers<strong>=<\/strong>headers\n)<\/code><\/pre>\n\n\n\n<p>Upload some vectors.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>client.put(\n    urljoin(endpoint, f\"\/collections\/{index_name}\/points?wait=true\"),\n    data<strong>=<\/strong>dumps(payload), headers<strong>=<\/strong>headers)<\/code><\/pre>\n\n\n\n<p>Search.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>query <strong>=<\/strong> '{\"vector\": &#91;0.1,0.2,0.3], \"limit\": 1}'\nresponse <strong>=<\/strong> client.post(\n    urljoin(endpoint, f\"\/collections\/{index_name}\/points\/search\"),\n    data<strong>=<\/strong>query, headers<strong>=<\/strong>headers)\n<strong>print<\/strong>(response.json())<\/code><\/pre>\n\n\n\n<p>It is easy to delete all vectors in an index.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>client.delete(\n    urljoin(endpoint, f\"\/collections\/{index_name}\"),\n        headers<strong>=<\/strong>headers\n)<\/code><\/pre>\n\n\n\n<p>You can see and run a&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\/blob\/main\/src\/qdrant\/hello.py\">working sample from here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>API_KEY<strong>=<\/strong>... ENDPOINT<strong>=<\/strong>https:\/\/my-cluster.cloud.qdrant.io:6333 poetry run src\/qdrant\/hello.py\n\n<strong>&gt;<\/strong> GET https:\/\/my-cluster.cloud.qdrant.io:6333\/collections\n&lt; GET https:\/\/my-cluster.cloud.qdrant.io:6333\/collections - 200\n<strong>&gt;<\/strong> PUT https:\/\/my-cluster.cloud.qdrant.io:6333\/collections\/my-index\n&lt; PUT https:\/\/my-cluster.cloud.qdrant.io:6333\/collections\/my-index - 200\n<strong>&gt;<\/strong> PUT https:\/\/my-cluster.cloud.qdrant.io:6333\/collections\/my-index\/points?wait<strong>=<\/strong>true\n&lt; PUT https:\/\/my-cluster.cloud.qdrant.io:6333\/collections\/my-index\/points?wait<strong>=<\/strong>true - 200\n<strong>&gt;<\/strong> POST https:\/\/my-cluster.cloud.qdrant.io:6333\/collections\/my-index\/points\/search\n&lt; POST https:\/\/my-cluster.cloud.qdrant.io:6333\/collections\/my-index\/points\/search - 200\n<strong>{<\/strong>'result': <strong>&#91;{<\/strong>'id': 1, 'version': 0, 'score': 0.9999998, 'payload': None, 'vector': None<strong>}]<\/strong>, 'status': 'ok', 'time': 0.000117235<strong>}<\/strong>\n<strong>&gt;<\/strong> DELETE https:\/\/my-cluster.cloud.qdrant.io:6333\/collections\/my-index\n&lt; DELETE https:\/\/my-cluster.cloud.qdrant.io:6333\/collections\/my-index - 200<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"redis\">Redis<\/h3>\n\n\n\n<p><a href=\"https:\/\/redis.io\/\">Redis<\/a>&nbsp;is a fast, opinionated, open-source database. Its&nbsp;<a href=\"https:\/\/redis.io\/docs\/interact\/search-and-query\/search\/vectors\/\">similarity vector search<\/a>&nbsp;comes with&nbsp;<code>FLAT<\/code>&nbsp;and&nbsp;<code>HNSW<\/code>&nbsp;indexing methods (field types). Redis is licensed under BSD.<\/p>\n\n\n\n<p>I prefer to run Redis locally in Docker with&nbsp;<code>docker run -p 6379:6379 redislabs\/redisearch:latest<\/code>, but managed service options with free tiers also&nbsp;<a href=\"https:\/\/redis.com\/\">exist<\/a>.<\/p>\n\n\n\n<p>Redis speaks&nbsp;<a href=\"https:\/\/redis.io\/docs\/reference\/protocol-spec\/\">RESP<\/a>, which is not HTTP, hence we\u2019re going to use&nbsp;<a href=\"https:\/\/github.com\/redis\/redis-py\">redis-py<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>r <strong>=<\/strong> Redis(host<strong>=<\/strong>'localhost', port<strong>=<\/strong>6379, decode_responses<strong>=<\/strong>True)<\/code><\/pre>\n\n\n\n<p>We create an&nbsp;<code>HNSW<\/code>&nbsp;index called&nbsp;<code>vectors<\/code>&nbsp;of documents with a given&nbsp;<code>doc:<\/code>&nbsp;prefix. This is unlike other databases where you write docs into an index.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>index_name <strong>=<\/strong> \"vectors\"\ndoc_prefix <strong>=<\/strong> \"doc:\"\n\nschema <strong>=<\/strong> (\n    TagField(\"genre\"),\n    VectorField(\"values\",\n        \"HNSW\", {\n            \"TYPE\": \"FLOAT32\",\n            \"DIM\": 3,\n            \"DISTANCE_METRIC\": \"COSINE\"\n        }\n    )\n)\n\ndefinition <strong>=<\/strong> IndexDefinition(\n    prefix<strong>=<\/strong>&#91;doc_prefix],\n    index_type<strong>=<\/strong>IndexType.HASH\n)\n\nr.ft(index_name).create_index(fields<strong>=<\/strong>schema, definition<strong>=<\/strong>definition)<\/code><\/pre>\n\n\n\n<p>Insert some vectors. Note that redis doesn\u2019t support a deep dictionary for metadata, so we will index and filter by&nbsp;<code>genre<\/code>&nbsp;in search.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pipe <strong>=<\/strong> r.ft(index_name).pipeline()\n\nvectors <strong>=<\/strong> &#91;\n    {\n        \"id\": 1,\n        \"values\": &#91;0.1, 0.2, 0.3],\n        \"metadata\": {\"genre\": \"drama\"},\n    },\n    {\n        \"id\": 2,\n        \"values\": &#91;0.2, 0.3, 0.4],\n        \"metadata\": {\"genre\": \"action\"},\n    },\n]\n\n<strong>for<\/strong> vector <strong>in<\/strong> vectors:\n    key <strong>=<\/strong> f\"{doc_prefix}{vector&#91;'id']}\"\n    value <strong>=<\/strong> {\n        \"genre\": vector&#91;\"metadata\"]&#91;\"genre\"],\n        \"values\": np.array(vector&#91;\"values\"]).astype(np.float32).tobytes()\n    }\n    pipe.hset(key, mapping<strong>=<\/strong>value)\n\npipe.execute()<\/code><\/pre>\n\n\n\n<p>Search. We filter by&nbsp;<code>genre<\/code>&nbsp;with&nbsp;<code>@genre:{ action })<\/code>. Use&nbsp;<code>**<\/code>&nbsp;instead if you don\u2019t want filtering.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>query <strong>=<\/strong> (\n    Query(\"(@genre:{ action })=&gt;&#91;KNN 2 @values $vector as score]\")\n    .sort_by(\"score\")\n    .return_fields(\"id\", \"score\", \"genre\")\n    .dialect(2)\n)\n\nquery_params <strong>=<\/strong> {\n    \"vector\": np.array(&#91;0.1, 0.2, 0.3]).astype(np.float32).tobytes()\n}\n\nresults <strong>=<\/strong> r.ft(index_name).search(query, query_params).docs\n<strong>for<\/strong> result <strong>in<\/strong> results:\n    <strong>print<\/strong>(result)<\/code><\/pre>\n\n\n\n<p>Finally, delete the index with its vectors.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>r.ft(index_name).dropindex(True)<\/code><\/pre>\n\n\n\n<p>You can see and run a&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\/blob\/main\/src\/redis\/hello.py\">working sample from here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>poetry run .\/hello.py\n\nDocument <strong>{<\/strong>'id': 'doc:2', 'payload': None, 'score': '0.00741678476334', 'genre': 'action'<strong>}<\/strong><\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"vespa\">Vespa<\/h3>\n\n\n\n<p><a href=\"https:\/\/vespa.ai\/\">Vespa<\/a>&nbsp;is a fully featured search engine and vector database. It supports approximate nearest neighbor search, lexical search, and search in structured data, all in the same query. Vespa is Apache 2.0 licensed, and can be run in a variety of ways, including Docker and as a managed&nbsp;<a href=\"https:\/\/cloud.vespa.ai\/\">cloud service<\/a>.<\/p>\n\n\n\n<p>Let\u2019s use their Docker container for this example. Make sure you&nbsp;<a href=\"https:\/\/docs.docker.com\/desktop\/settings\/mac\/#resources\">configure Docker with at least 4GB RAM<\/a>&nbsp;(check with&nbsp;<code>docker info | grep \"Total Memory\"<\/code>).<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>docker pull vespaengine\/vespa\ndocker run --detach --name vespa --hostname vespa-container \\\n  --publish 8080:8080 --publish 19071:19071 \\\n  vespaengine\/vespa<\/code><\/pre>\n\n\n\n<p>This container listens on port&nbsp;<code>8080<\/code>&nbsp;for search and ingestion APIs, and on&nbsp;<code>19071<\/code>&nbsp;for configuration APIs.<\/p>\n\n\n\n<p>Vespa encapsulates the concept of a schema\/index in an application that needs to be defined and deployed, so it is not as straightforward as the previous example.<\/p>\n\n\n\n<p>To create a new application with a sample vector schema we need to create a&nbsp;<code>settings.xml<\/code>&nbsp;file with the overall application properties, and a&nbsp;<code>schema.md<\/code>&nbsp;file with the definition of our schema. For this example, let\u2019s create the following directory structure.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vector-app\/\n\u251c\u2500\u2500 schemas\/\n\u2502   \u2514\u2500\u2500 vector.sd\n\u2514\u2500\u2500 services.xml<\/code><\/pre>\n\n\n\n<p><code>services.xml<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong><em>&lt;?xml version=\"1.0\" encoding=\"utf-8\" ?&gt;<\/em><\/strong>\n&lt;services version=\"1.0\" xmlns:deploy=\"vespa\" xmlns:preprocess=\"properties\"&gt;\n    &lt;container id=\"default\" version=\"1.0\"&gt;\n        &lt;document-api\/&gt;\n        &lt;search\/&gt;\n        &lt;nodes&gt;\n            &lt;node hostalias=\"node1\" \/&gt;\n        &lt;\/nodes&gt;\n    &lt;\/container&gt;\n    &lt;content id=\"vector\" version=\"1.0\"&gt;\n        &lt;redundancy&gt;2&lt;\/redundancy&gt;\n        &lt;documents&gt;\n            &lt;document type=\"vector\" mode=\"index\" \/&gt;\n        &lt;\/documents&gt;\n        &lt;nodes&gt;\n            &lt;node hostalias=\"node1\" distribution-key=\"0\" \/&gt;\n        &lt;\/nodes&gt;\n    &lt;\/content&gt;\n&lt;\/services&gt;<\/code><\/pre>\n\n\n\n<p><code>vector.sd<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>schema vector {\n    document vector {\n        field id type string {\n            indexing: summary | attribute\n        }\n        field values type tensor&lt;float&gt;(x&#91;3]) {\n            indexing: summary | attribute\n            attribute {\n                distance-metric: angular\n            }\n        }\n        struct metadatatype {\n            field genre type string {}\n        }\n        field metadata type metadatatype {\n            indexing: summary\n        }\n    }\n    rank-profile vector_similarity {\n        inputs {\n            query(vector_query_embedding) tensor&lt;float&gt;(x&#91;3])\n        }\n        first-phase {\n            expression: closeness(field, values)\n        }\n    }<\/code><\/pre>\n\n\n\n<p>Deploy using the configuration API.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>(<\/strong>cd vector-app <strong>&amp;&amp;<\/strong> zip -r - .<strong>)<\/strong> | \\\n  curl --header Content-Type:application\/zip --data-binary @- \\\n  localhost:19071\/application\/v2\/tenant\/default\/prepareandactivate\n\ncurl \\\n    --header Content-Type:application\/zip \\\n    -XPOST localhost:19071\/application\/v2\/tenant\/default\/session<\/code><\/pre>\n\n\n\n<p>Setup the client.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>endpoint <strong>=<\/strong> \"https:\/\/localhost:8080\"\nclient <strong>=<\/strong> Client(verify<strong>=<\/strong>False)\nheaders <strong>=<\/strong> {\n    \"Accept\": \"application\/json; charset=utf-8\",\n    \"Content-Type\": \"application\/json; charset=utf-8\",\n}<\/code><\/pre>\n\n\n\n<p>Ingest some vectors.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vectors <strong>=<\/strong> &#91;\n    {\n        \"id\": \"vec1\",\n        \"values\": &#91;0.1, 0.2, 0.3],\n        \"metadata\": {\"genre\": \"drama\"},\n    },\n    {\n        \"id\": \"vec2\",\n        \"values\": &#91;0.2, 0.3, 0.4],\n        \"metadata\": {\"genre\": \"comedy\"},\n    },\n]\n\n<strong>for<\/strong> vector <strong>in<\/strong> vectors:\n    data <strong>=<\/strong> json.dumps({\"fields\": vector})\n    client.post(\n        urljoin(endpoint, \"\/document\/v1\/vector\/vector\/docid\/\" <strong>+<\/strong> vector&#91;\"id\"]),\n        headers<strong>=<\/strong>headers,\n        data<strong>=<\/strong>data\n    )<\/code><\/pre>\n\n\n\n<p>Search.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>query <strong>=<\/strong> \"yql=select * from sources * where {targetHits: 1} nearestNeighbor(values,vector_query_embedding)\" \\\n    \"&amp;ranking.profile=vector_similarity\" \\\n    \"&amp;hits=1\" \\\n    \"&amp;input.query(vector_query_embedding)=&#91;0.1,0.2,0.3]\"\n\nresults <strong>=<\/strong> client.get(\n    urljoin(endpoint, \"\/search\/\"),\n        headers<strong>=<\/strong>headers,\n        params<strong>=<\/strong>query\n).json()\n\n<strong>print<\/strong>(results&#91;\"root\"]&#91;\"children\"]&#91;0]&#91;\"fields\"])<\/code><\/pre>\n\n\n\n<p>You can see and run a&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\/blob\/main\/src\/vespa\/hello.py\">working sample from here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ENDPOINT<strong>=<\/strong>https:\/\/localhost:8080 CONFIG_ENDPOINT<strong>=<\/strong>https:\/\/localhost:19071 poetry run src\/vespa\/hello.py\n\n<strong>&gt;<\/strong> POST https:\/\/localhost:8080\/document\/v1\/vector\/vector\/docid\/vec1\n&lt; POST https:\/\/localhost:8080\/document\/v1\/vector\/vector\/docid\/vec1 - 200\n<strong>&gt;<\/strong> POST https:\/\/localhost:8080\/document\/v1\/vector\/vector\/docid\/vec2\n&lt; POST https:\/\/localhost:8080\/document\/v1\/vector\/vector\/docid\/vec2 - 200\n<strong>&gt;<\/strong> GET https:\/\/localhost:8080\/search\/?yql<strong>=<\/strong><strong>select<\/strong>%20%2A%20from%20sources%20%2A%20where%20%7BtargetHits%3A%201%7DnearestNeighbor%28values%2Cvector_query_embedding%29&amp;ranking.profile<strong>=<\/strong>vector_similarity&amp;hits<strong>=<\/strong>1&amp;input.query%28vector_query_embedding%29<strong>=<\/strong>%5B0.1%2C0.2%2C0.3%5D\n&lt; GET https:\/\/localhost:8080\/search\/?yql<strong>=<\/strong><strong>select<\/strong>%20%2A%20from%20sources%20%2A%20where%20%7BtargetHits%3A%201%7DnearestNeighbor%28values%2Cvector_query_embedding%29&amp;ranking.profile<strong>=<\/strong>vector_similarity&amp;hits<strong>=<\/strong>1&amp;input.query%28vector_query_embedding%29<strong>=<\/strong>%5B0.1%2C0.2%2C0.3%5D - 200\n<strong>{<\/strong>'sddocname': 'vector', 'documentid': 'id:vector:vector::vec1', 'id': 'vec1', 'values': <strong>{<\/strong>'type': 'tensor&lt;float&gt;(x&#91;3])', 'values': <strong>&#91;<\/strong>0.10000000149011612, 0.20000000298023224, 0.30000001192092896]<strong>}<\/strong>, 'metadata': <strong>{<\/strong>'genre': 'drama'<strong>}}<\/strong>\n<strong>&gt;<\/strong> DELETE https:\/\/localhost:19071\/application\/v2\/tenant\/default\/application\/default\n&lt; DELETE https:\/\/localhost:19071\/application\/v2\/tenant\/default\/application\/default - 200<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"weaviate\">Weaviate<\/h3>\n\n\n\n<p><a href=\"https:\/\/weaviate.io\/\">Weaviate<\/a>&nbsp;is a vector search engine specifically designed for natural language numerical data. It uses contextualized embeddings in data objects to understand semantic similarity. Currently, it supports only Hierarchical Navigable Small World (HNSW) indexing, and is more costly on building data to indexes. However, it has a fast query time and high scalability. Weaviate is open-source, easy to use, flexible, extensible, and has a Contributor License Agreement.<\/p>\n\n\n\n<p>After you sign up at Weaviate Cloud Services WCS, create a new free tier Weaviate Cluster with authentication. Note your cluster URL and API key (optional). The endpoint will have the following format https:\/\/myindex.weaviate.network.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>client <strong>=<\/strong> Client()\n\nendpoint <strong>=<\/strong> os.environ&#91;\"ENDPOINT\"]\napi_key <strong>=<\/strong> os.getenv(\"API_KEY\") <em># optional\n<\/em>\nheaders <strong>=<\/strong> {\n    \"Accept\": \"application\/json; charset=utf-8\",\n    \"Content-Type\": \"application\/json; charset=utf-8\"\n}\n\n<strong>if<\/strong> <strong>not<\/strong> api_key <strong>is<\/strong> None:\n    headers&#91;\"Authorization\"] <strong>=<\/strong> f\"Bearer {api_key}\"<\/code><\/pre>\n\n\n\n<p>It is easy to create some objects with vectors.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vectors <strong>=<\/strong> &#91;\n    {\n        \"id\": \"vec1\",\n        \"values\": &#91;0.1, 0.2, 0.3],\n        \"properties\": {\n            \"genre\": \"drama\"\n        }\n    },\n    {\n        \"id\": \"vec2\",\n        \"values\": &#91;0.2, 0.3, 0.4],\n        \"properties\": {\n            \"genre\": \"action\"\n        }\n    }\n]\n\nobjects <strong>=<\/strong> &#91;]\n<strong>for<\/strong> vector <strong>in<\/strong> vectors:\n    obj <strong>=<\/strong> {\n        \"class\": \"Vectors\",\n        \"properties\": {\n            \"vector\": vector&#91;\"values\"]\n        }\n    }\n    objects.append(obj)\n\nclient.post(\n    urljoin(endpoint, \"\/v1\/batch\/objects\"),\n        json<strong>=<\/strong>{\"objects\": objects},\n        headers<strong>=<\/strong>headers\n)<\/code><\/pre>\n\n\n\n<p>The search is pretty straightforward. Weaviate also has a GraphQL interface.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>query <strong>=<\/strong> {\n    \"fields\": \"vector\",\n    \"nearVector\": {\n        \"vector\": &#91;0.1],\n        \"certainty\": 0.9\n    }\n}\n\nresponse <strong>=<\/strong> client.get(\n    urljoin(endpoint, \"\/v1\/objects\"),\n        params<strong>=<\/strong>query,\n        headers<strong>=<\/strong>headers\n).json()\n\n<strong>for<\/strong> obj <strong>in<\/strong> response&#91;\"objects\"]:\n    <strong>print<\/strong>(obj)<\/code><\/pre>\n\n\n\n<p>Deleting objects of the same class is straightforward.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>client.delete(\n    urljoin(endpoint, f\"\/v1\/schema\/Vectors\"),\n        headers<strong>=<\/strong>headers\n)<\/code><\/pre>\n\n\n\n<p>You can see and run a&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\/blob\/main\/src\/weaviate\/hello.py\">working sample from here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>API_KEY<strong>=<\/strong>... ENDPOINT<strong>=<\/strong>https:\/\/my-cluster.weaviate.network poetry run src\/weaviate\/hello.py\n\n<strong>&gt;<\/strong> POST https:\/\/myindex.weaviate.network\/v1\/batch\/objects\n&lt; POST https:\/\/myindex.weaviate.network\/v1\/batch\/objects - 200\n<strong>&gt;<\/strong> GET https:\/\/myindex.weaviate.network\/v1\/objects?fields<strong>=<\/strong>vector&amp;nearVector<strong>=<\/strong>%7B%27vector%27%3A%20%5B0.1%5D%2C%20%27certainty%27%3A%200.9%7D\n&lt; GET https:\/\/myindex.weaviate.network\/v1\/objects?fields<strong>=<\/strong>vector&amp;nearVector<strong>=<\/strong>%7B%27vector%27%3A%20%5B0.1%5D%2C%20%27certainty%27%3A%200.9%7D - 200\n<strong>{<\/strong>'class': 'Vectors', 'creationTimeUnix': 1688914857307, 'id': '46e40d05-d550-4415-aa2c-7c004fcdd037', 'lastUpdateTimeUnix': 1688914857307, 'properties': <strong>{<\/strong>'vector': <strong>&#91;<\/strong>0.1, 0.2, 0.3]<strong>}<\/strong>, 'vectorWeights': None<strong>}<\/strong>\n<strong>{<\/strong>'class': 'Vectors', 'creationTimeUnix': 1688914857307, 'id': 'c14bd5b1-8b81-44a4-8051-3b9b8c52cde4', 'lastUpdateTimeUnix': 1688914857307, 'properties': <strong>{<\/strong>'vector': <strong>&#91;<\/strong>0.2, 0.3, 0.4]<strong>}<\/strong>, 'vectorWeights': None<strong>}<\/strong>\n<strong>&gt;<\/strong> DELETE https:\/\/myindex.weaviate.network\/v1\/schema\/Vectors\n&lt; DELETE https:\/\/myindex.weaviate.network\/v1\/schema\/Vectors - 200<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"others\">Others<\/h3>\n\n\n\n<p>This blog post and&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-hello-world\/\">its code<\/a>&nbsp;could use your help for more examples for&nbsp;<a href=\"https:\/\/github.com\/milvus-io\/milvus\">Milvus<\/a>,&nbsp;<a href=\"https:\/\/github.com\/vector-ai\/vectorai\">Vector.ai<\/a>,&nbsp;<a href=\"https:\/\/github.com\/nuclia\/nucliadb\">NucliaDB<\/a>,&nbsp;<a href=\"https:\/\/vald.vdaas.org\/\">Vald<\/a>, etc.<\/p>\n\n\n\n<p>I also wonder whether we need a generic client that\u2019s agnostic to which vector DB is being used to help make code portable? I&nbsp;<a href=\"https:\/\/github.com\/dblock\/vectordb-client\">took a stab at a very simple prototype<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Vector databases are all the rage today. I\u2019ve built a few iterations of vector search, beginning in 2011 at Artsy, powered by the&nbsp;Art Genome Project. Compared to LLM use-cases today,&#8230; <a class=\"read-more-link\" href=\"https:\/\/tbekk.com\/devstream\/2023\/07\/26\/getting-started-with-vector-dbs-in-python\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[207,208],"tags":[215,74,222,223,224],"class_list":["post-774","post","type-post","status-publish","format-standard","hentry","category-db","category-vector-db","tag-llm","tag-py","tag-vector-databases","tag-vector-dbs","tag-vector-search"],"_links":{"self":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/774","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/comments?post=774"}],"version-history":[{"count":1,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/774\/revisions"}],"predecessor-version":[{"id":775,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/774\/revisions\/775"}],"wp:attachment":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/media?parent=774"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/categories?post=774"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/tags?post=774"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}