Data Velocity

The choice of data processing velocity is pivotal in determining the kind of data retrieval and vector compute tasks you can perform. Different velocities offer distinct advantages and make different use cases possible. Here's a breakdown of the three primary velocity categories:

Most systems deployed in production at scale combine stream and batch processing. This enables you to leverage the immediacy of real-time updates and the depth of historical data. But reconciling stream and batch processes in a single system introduces trade-off decisions – trade-off decisions you must make to keep your data consistent across systems and across time.

When choosing and configuring the data sources for a vector retrieval system, it’s important to consider tradeoffs between data velocity, complexity, and model size.

Different data sources impact how the data is then ingested and processed by your data & ML infrastructure. For a deep dive on how your choice of data sources affects the downstream systems in your information retrieval stack, see our article on Vector Compute. For the purposes of this discussion, you can think about data source choice in terms of a tradeoff between data velocity, retrieval quality, and engineering complexity.

• Streaming data: This option allows for real-time vector computations but, due to latency concerns, comes with constraints on model complexity. Streaming data may require simpler embeddings or lookups.
• Batch data: Unlike streaming data, batch data enables complex models, including large transformers. But with batch data, updates are less frequent, making it suitable for asynchronous needs.
• Hybrid Approaches: These merge streaming filters or lookups with batch retraining, offering responsiveness alongside in-depth analysis.

How you configure your data sources to balance velocity and complexity ultimately manifests in the types of machine learning model architectures you use in your vector search system.

Example Architectures

Within a vector search system, there are three basic architecture types. These architectures balance velocity and complexity in different ways:

• Streaming: Think of architectures like Recurrent Neural Networks (RNNs), shallow neural networks, and indexing/retrieval models.
• Batch: Large pre-trained transformers, custom deep learning models, and architecture search models.
• Hybrid: Combine indexing with periodic retraining or deploying a two-stage filtering and analysis setup.

Choosing the right architecture depends on your goals:

Key Considerations

• Synchronicity Needs: Determine whether real-time updates to the vector embeddings are essential or if daily or weekly updates suffice.
• Model Accuracy Requirements: Consider whether approximate real-time performance meets your needs. For example, speed is key in preventing further loss in fraud detection. Therefore, real-time data is important. When detecting fraud, the cost of a false positive is often less than the risk posed by a false negative, so it makes sense to prioritize velocity. In a recommendation system, on the other hand, having true real-time user interaction data may be less important; the customer takes time to evaluate the current options. Therefore, you can get away with near real-time performance.

The trade-off between velocity and complexity is not the only important consideration in building your vector retrieval system. You also need to balance model size with response time.

• Streaming: Typically involves models under 100MB in size, such as shallow networks, distillation models, and compact neural networks like MobileNet.
• Batch: Here, you're looking at models exceeding 1GB in size, including giant transformers and custom deep networks.
• Hybrid: This is the sweet spot, combining smaller streaming models with larger batch models to find an optimal balance between size and speed.

As with velocity and complexity, balancing size with response time is consequential for the types of architectures you use in your vector search system.

Example Architectures

• Streaming: You might come across RNNs, CNNs with fewer than 10 layers, and efficient models like MiniLM or TinyML. Even simple dense networks or XGBoost can fit the bill.
• Batch Architectures: Think of big players like BERT, GPT-3/4, T5, EfficientNet, and custom transformers. Also, don't forget about neural architecture search models.
• Ensemble and Stacked Models: In some cases, ensemble and stacked models can provide the best of streaming and batch architectures, combining model sizes and optimizing for various needs.

Besides the velocity-complexity and model size-response time tradeoffs above, there are some additional considerations pivotal to ensuring that you meet your vector retrieval system’s objectives:

• Response Latency Constraints: Always monitor response latency constraints to ensure your models can meet real-time demands.
• Model Accuracy Requirements: Consider your precise model accuracy requirements and whether your infrastructure can effectively handle the chosen model sizes.
• Data Value: Recognize the value of processed data, as high-value data holds critical information that can lead to valuable insights and informed decision-making.

In short, keeping these key considerations in mind while you balance velocity with complexity, and model size with response times, will help ensure that you configure your specific vector retrieval system optimally.

In the context of real-time data processing, the choice between Kappa and Lambda architectures involves evaluating trade-offs between velocity, complexity, and data value:

• Kappa Architecture: Emphasizes real-time processing. Data is ingested as a continuous stream and processed in real-time to support immediate decisions. This approach is well-suited to use high-velocity cases, such as fraud detection systems in financial institutions.

• Lambda Architecture: Combines batch and real-time processing. Data is ingested both as a stream and in batch mode. This approach offers the advantage of capturing high-velocity data while maintaining the value of historical data. It's suitable when there's a need for comprehensive insights that consider both real-time and historical data.

---

• Daniel Svonava
• Paolo Perrone
• Robert Turner, editor