Research Paper RAG / Cloud infrastructure / Retrieval
ImplementedFrom an uploaded paper to a checkable answer.
A research assistant that processes papers asynchronously, keeps each reader's retrieval scoped to their library, and returns answers with source citations.
My contribution
I designed and built the system end to end: the ingestion path, the storage boundaries between S3, DynamoDB, and the vector index, and the asynchronous processing that keeps the upload request short.
Architecture walkthrough
Follow the request.
Choose a path, then select a component to inspect its role, failure modes, and source code.
System / Research Paper RAGResearch Paper RAGIngest
The upload does the smallest durable thing and hands the rest to a queue. Watch the packet split at that handoff; a receipt goes back up to the reader, the work carries on below the line.
Interactive architecture illustration. No live requests, model calls, or calendar writes occur here.
Engineering decisions
Where the trade-offs live.
01
Separate accepting a paper from processing it.
SQS separates the upload interaction from extraction, chunking, embedding, and indexing. Document state records the difference between accepted, processing, and ready. The reader can leave while the processor works.
The trade-off
Three stores can disagree after a partial failure: an original in S3, a processing record in DynamoDB, and incomplete vectors in Pinecone. Recovery and duplicate delivery must be designed explicitly.
02
Carry the reader's identity into retrieval.
Cognito provides identity, while the chat handler scopes vector search to the user's namespace and the selected session's papers. Returned citations let the reader inspect the source behind an answer.
The trade-off
Every read path needs consistent identity checks. A scoped query does not prove every endpoint is isolated, and a citation does not prove that the generated claim is supported.
03
Keep failed work visible.
The processing queue has a dead-letter queue and CloudWatch alarms. In the reviewed template, the visibility window is 900 seconds and the redrive policy moves repeatedly failing work after three receives. Replaying a poisoned document is an operator decision after inspection.
The trade-off
A dead-letter queue preserves evidence; it does not repair a malformed PDF or a throttled model dependency. Embedding quotas and vector storage cost also need attention as the document collection grows.
Evidence & limitations
What the work establishes.
The repository implements asynchronous processing, a dead-letter queue, user-scoped vector retrieval, and source citations. Retrieval quality, ingestion throughput, and cost per document still need a published evaluation. Citations make an answer inspectable; they do not establish its correctness.
What I would validate next
- 01Build a labeled question set and score retrieval relevance and citation support.
- 02Test missing identity, cross-user requests, and duplicate processing.
- 03Measure processing latency and cost on a documented mix of paper sizes.