The problem
Opio is a fintech start-up that had just raised $4M. Its platform helps transaction-services teams run financial due diligence: AI workflows read a company’s accounting data and deal documents and produce structured insights, each linked to its source.
On the baseline use case, the retrieval system behind those workflows answered only half of the client’s question set correctly. Opio needed it to be right, and as their CTO was busy working for an important client they needed an engineer who could also work across the rest of the product.
What I built
An eval system that runs on its own. Opio’s CEO had written a 60-question test set, but it was never run automatically. I built the eval system that runs it and scores the answers, so every change to the retrieval system is measured.
A RAG system rewritten from scratch. Chunking, embeddings, the retriever and the metadata system, all rebuilt to best practices.
Document ingestion that keeps up. I worked on the data engineering workflows so the document ingestion pipeline - which was needed for the retrieval system to work - runs incrementally and fast.
A feature that helped win a deal. I contributed to the full-stack app, including a critical feature that helped Opio land an important lead: it automatically maps French FEC accounting files (the standard export of a company’s general ledger) to business categories, so the cash flow can be read category by category.
Safe, automated deployments. I put industry-standard DevOps practices in place, so deployments became automated, fast and safe.
How we knew it worked
The rewrite was measured against Opio’s baseline use case: the CEO’s 60-question set, run by the eval system I built. Accuracy went from 50% to 100%.
A perfect score on 60 questions is a start, not a proof. The next step is a larger test set and new eval sets, to show the system holds on questions it hasn’t seen.
Results
- Accuracy on the baseline eval went from 50% to 100%, and the system is in production.
- The 60-question set runs automatically, so every change is measured.
- Document ingestion is incremental and fast.
- Deployments are automated, fast and safe.
Who runs it now
Opio had just raised its round, and I was the first engineer to help the CTO build the product. Once Opio’s internal team was ready, I handed everything over to them.
