An Architect's vision

  Vector Databases Usage: Typically used for vector search use cases such as visual, semantic, and multimodal search. More recently, they are paired with generative AI text models for conversational search experiences. Development Process: Begins with building an embedding model designed to encode a corpus (e.g., product images) into vectors. The data import process is referred to as data hydration. Application Development: Application developers utilize the database to search for similar products. This involves encoding a product image and using the vector to query for similar images. k-Nearest Neighbor (k-NN) Indexes: Within the model, k-nearest neighbor (k-NN) indexes facilitate efficient retrieval of vectors. A distance function like cosine is applied to rank results by similarity.

  Vector Databases for Vector Search Use Cases: Typically used to power vector search use cases like visual, semantic, and multimodal search. Increasingly paired with generative artificial intelligence (AI) text models for conversational search experiences. Pairing with Generative AI Text Models: Vector databases are now paired with generative AI text models to create intelligent agents. Development Process: Starts with building an embedding model designed to encode a corpus (e.g., product images) into vectors. Data import process is referred to as data hydration. Application Usage: Application developers use the database to search for similar products by encoding a product image. The vector is then used to query for similar images. k-Nearest Neighbor (k-NN) Indexes: Within the model, k-nearest neighbor (k-NN) indexes provide efficient retrieval of vectors. A distance function like cosine is applied to rank results by similarity.

With a vector database, we can add advanced features to our AIs, like semantic information retrieval, long-term memory, and more. The diagram below gives us a better understanding of the role of vector databases in this type of application: Embeddings  are generated by AI models (such as Large Language Models) and have many attributes or features, making their representation challenging to manage. In the context of AI and machine learning, these features represent different dimensions of the data that are essential for understanding patterns, relationships, and underlying structures. Steps:  First, we use the  embedding model  to create  vector embeddings  for the  content  we want to index. The  vector embedding  is inserted into the  vector database , with some reference to the original  content  the embedding was created from. When the  application  issues a query, we use the same  embedding model  t...

  Indexing Vectors into Vector Databases: Developers can index vectors generated by embeddings into a vector database. This enables finding similar assets by querying for neighboring vectors. Operationalizing Embedding Models: Vector databases provide a method to operationalize embedding models. Application development benefits from database capabilities like resource management, security controls, scalability, fault tolerance, and efficient information retrieval through sophisticated query languages. Empowering Developers: Vector databases empower developers to create unique application experiences. Users could snap photographs on their smartphones to search for similar images. Metadata Extraction and Hybrid Search: Developers can use other machine learning models to automate metadata extraction from content like images and scanned documents. They can index metadata alongside vectors to enable hybrid search on both keywords and vectors. Fusion of semantic understanding into releva...

A vector database is a type of database that indexes and stores vector embeddings for fast retrieval and similarity search, with capabilities like CRUD operations, metadata filtering, and horizontal scaling. Information comes in many forms: Some information is unstructured—like text documents, rich media, and audio. Some information is structured—like application logs, tables, and graphs. Innovations in artificial intelligence and machine learning (AI/ML): Have allowed us to create a type of ML model—embedding models. Embeddings encode all types of data into vectors that capture the meaning and context of an asset. This allows us to find similar assets by searching for neighboring data points. Vector search methods: Allow unique experiences like taking a photograph with your smartphone and searching for similar images. Vector databases provide: The ability to store and retrieve vectors as high-dimensional points. Additional capabilities for efficient and fast lookup of nearest-neighbor...

Vector databases are designed to efficiently handle vector data, which is prevalent in scenarios like embeddings generated by neural networks, feature vectors representing data points, and more. These databases often employ techniques such as indexing and search algorithms that are specifically tailored to work well with vector spaces.  Here are some notable ones: Faiss: Description: Faiss (Facebook AI Similarity Search) is an open-source library for efficient similarity search and clustering of dense vectors. Link: Faiss GitHub Repository Milvus: Description: Milvus is an open-source vector database designed for handling large-scale vector data. It is often used for similarity search and recommendation systems. Link: Milvus GitHub Repository Annoy: Description: Annoy is a C++ library with Python bindings for approximate nearest neighbors in high-dimensional spaces. Link: Annoy GitHub Repository Elasticsearch with Vector Similarity Plugin: Description: Elasticsearch is a widely u...

They’re usually trained offline, making the model agnostic to the latest information (for example, a chatbot trained from 2011–2018 has no information about COVID-19). They make predictions by only looking up information stored in its parameters, leading to inferior interpretability. They’re mostly trained on general domain corpora, making them less effective on domain-specific tasks. There are scenarios when you want models to generate text based on specific data rather than generic data.  For example, a health insurance company may want their question answering bot to answer questions using the latest information stored in their enterprise document repository or database, so the answers are accurate and reflect their unique business rules.

  Top-k Sampling: It's a sampling technique used to control the randomness of text generation. Top-k limits the model's choices to the k tokens with the highest probability. For example, setting k to 3 restricts the model to choosing from the top three most probable tokens. The model selects from these options using probability weighting. This method adds some randomness while preventing highly improbable completions. Result: Text generation is more likely to sound reasonable and coherent. Top-p Sampling: Another sampling technique to limit randomness in text generation. Top-p restricts the model to predictions whose combined probabilities do not exceed p. For example, setting p to 0.3 includes tokens with probabilities adding up to 0.3 (e.g., cake and donut with probabilities of 0.2 and 0.1). The model uses random probability weighting to choose from these tokens. Result: Allows control over text generation by specifying the total probability the model should choose from. Thes...

The "Top-k" value, often referred to as the "nucleus" or "top-p" value, is a hyperparameter used during the tuning of generative AI Large Language Models (LLMs). It is used to control the diversity and quality of the generated text. The specific use of the Top-k value depends on the decoding strategy employed when using LLMs, and it serves several purposes: Controlling Text Generation Diversity: The Top-k value determines the number of most likely next tokens to consider during text generation. A lower Top-k value will restrict the model to a smaller set of tokens, leading to more focused and deterministic output. A higher value will allow the model to consider a larger pool of tokens, increasing diversity in generated text. Improving Text Coherence: By using a smaller Top-k value, you can ensure that the generated text is more coherent and contextually relevant. This can be particularly useful when you want to generate text that aligns closely with the ...

Tokens in natural language processing are not provided by specific vendors; rather, they are an integral part of the field's methodology and techniques. Tokenization is a fundamental step in text processing and is performed by libraries, frameworks, and software tools that are designed for natural language processing and machine learning. These tools are typically open-source or provided by established organizations and research communities. Some common libraries and frameworks for tokenization in natural language processing include: Hugging Face Transformers: Hugging Face provides a popular open-source library for transformer-based models, including pre-trained models and tokenizers. Their Transformers library offers tokenizers for a wide range of languages and models. NLTK (Natural Language Toolkit): NLTK is a widely used Python library for natural language processing tasks, including tokenization. It provides tools for various tokenization methods spaCy: spaCy is a popular op...