About
Our client is the forerunner in the world of education material for entrepreneurship. The company shares resources that the current and future leaders need to innovate, manage and build robust businesses.
Project Highlights
The client wanted to develop a web application that takes user input and responds to a query with quoted references from books on entrepreneurship uploaded to Google Bucket. The application would allow users to upload various types of documents about entrepreneurship, extract relevant metadata and generate embeddings for accurate and faster retrieval of relevant book quotes and references. To develop this application, Brainvire developed a scalable and easy-to-use web application for uploading files, extracting text from files, generating embeddings using OpenAI and storing embeddings and associated metadata in a Pinecone index to execute a similarity search.
The Challenges
- Scalable Pub/Sub Architecture:The client desired the application to handle high message volume and concurrency efficiently
- Indexing and Query Optimization:There was a need for maintaining the efficiency of indexing and querying to distribute the load and increase speed
- System resilience and fault tolerance:The system needed effective error handling and retry mechanisms
- Efficient Embedding Generation:The client wanted to reduce the API calls to create embeddings for the text chunks and maintaining API limits
Tech Stack
CSS 3
Django
Google Cloud Platform
HTML
JS
Pinecone
Python
Result
Faster Metadata Updates
We introduced parallel processing capability for worker instances to reduce time for metadata updation in the Pinecone indexHigh availability of the system
The distributed architecture and the load balancer utilization ensures uninterrupted service by redirecting requests to other available instances or workerReal-time updates
Brainvire implemented asynchronous processing and parallel execution for close to real-time updates of Pinecone metadata for enhanced user experience without delayScalable architecture
Brainvire designed a scalable architecture for consistent, smooth performances by adding more worker instances and distribute the load with increased metadata update requests