What's the optimal chunking and indexing strategy for academic papers with mixed metadata (dates, authors, abstracts)?

Combine title and abstract into a single text field for embedding rather than treating them separately - this captures the paper's complete semantic context in one vector. Use chunk_size=200, chunk_overlap=50 for the combined text to handle long abstracts without losing context at boundaries. For in-memory indexing with <10K papers, batch load in groups of 10-100 to prevent memory spikes while showing progress.

Critical production tip: Don't index all columns - select only what you'll query and return (entry_id, published, text, title, summary). Store author and category data separately if needed for filtering but not for semantic search. Use schema mapping to rename columns at ingestion: paper.text: "text" ensures consistency regardless of source format. This approach keeps your index lean (30-40% smaller) while maintaining full search capability.