An Architect's vision

  Vector Databases for Vector Search Use Cases: Typically used to power vector search use cases like visual, semantic, and multimodal search. Increasingly paired with generative artificial intelligence (AI) text models for conversational search experiences. Pairing with Generative AI Text Models: Vector databases are now paired with generative AI text models to create intelligent agents. Development Process: Starts with building an embedding model designed to encode a corpus (e.g., product images) into vectors. Data import process is referred to as data hydration. Application Usage: Application developers use the database to search for similar products by encoding a product image. The vector is then used to query for similar images. k-Nearest Neighbor (k-NN) Indexes: Within the model, k-nearest neighbor (k-NN) indexes provide efficient retrieval of vectors. A distance function like cosine is applied to rank results by similarity.

With a vector database, we can add advanced features to our AIs, like semantic information retrieval, long-term memory, and more. The diagram below gives us a better understanding of the role of vector databases in this type of application: Embeddings  are generated by AI models (such as Large Language Models) and have many attributes or features, making their representation challenging to manage. In the context of AI and machine learning, these features represent different dimensions of the data that are essential for understanding patterns, relationships, and underlying structures. Steps:  First, we use the  embedding model  to create  vector embeddings  for the  content  we want to index. The  vector embedding  is inserted into the  vector database , with some reference to the original  content  the embedding was created from. When the  application  issues a query, we use the same  embedding model  to create embeddings for the query and use those embeddings to query the  database  f

  Indexing Vectors into Vector Databases: Developers can index vectors generated by embeddings into a vector database. This enables finding similar assets by querying for neighboring vectors. Operationalizing Embedding Models: Vector databases provide a method to operationalize embedding models. Application development benefits from database capabilities like resource management, security controls, scalability, fault tolerance, and efficient information retrieval through sophisticated query languages. Empowering Developers: Vector databases empower developers to create unique application experiences. Users could snap photographs on their smartphones to search for similar images. Metadata Extraction and Hybrid Search: Developers can use other machine learning models to automate metadata extraction from content like images and scanned documents. They can index metadata alongside vectors to enable hybrid search on both keywords and vectors. Fusion of semantic understanding into relevancy

A vector database is a type of database that indexes and stores vector embeddings for fast retrieval and similarity search, with capabilities like CRUD operations, metadata filtering, and horizontal scaling. Information comes in many forms: Some information is unstructured—like text documents, rich media, and audio. Some information is structured—like application logs, tables, and graphs. Innovations in artificial intelligence and machine learning (AI/ML): Have allowed us to create a type of ML model—embedding models. Embeddings encode all types of data into vectors that capture the meaning and context of an asset. This allows us to find similar assets by searching for neighboring data points. Vector search methods: Allow unique experiences like taking a photograph with your smartphone and searching for similar images. Vector databases provide: The ability to store and retrieve vectors as high-dimensional points. Additional capabilities for efficient and fast lookup of nearest-neighbor

Vector databases are designed to efficiently handle vector data, which is prevalent in scenarios like embeddings generated by neural networks, feature vectors representing data points, and more. These databases often employ techniques such as indexing and search algorithms that are specifically tailored to work well with vector spaces.  Here are some notable ones: Faiss: Description: Faiss (Facebook AI Similarity Search) is an open-source library for efficient similarity search and clustering of dense vectors. Link: Faiss GitHub Repository Milvus: Description: Milvus is an open-source vector database designed for handling large-scale vector data. It is often used for similarity search and recommendation systems. Link: Milvus GitHub Repository Annoy: Description: Annoy is a C++ library with Python bindings for approximate nearest neighbors in high-dimensional spaces. Link: Annoy GitHub Repository Elasticsearch with Vector Similarity Plugin: Description: Elasticsearch is a widely used

They’re usually trained offline, making the model agnostic to the latest information (for example, a chatbot trained from 2011–2018 has no information about COVID-19). They make predictions by only looking up information stored in its parameters, leading to inferior interpretability. They’re mostly trained on general domain corpora, making them less effective on domain-specific tasks. There are scenarios when you want models to generate text based on specific data rather than generic data.  For example, a health insurance company may want their question answering bot to answer questions using the latest information stored in their enterprise document repository or database, so the answers are accurate and reflect their unique business rules.