Introducing s-1: Sievly’s new document reading model
All posts
Guides
Guide: how to evaluate document accuracy on your own files
Benchmarks are a starting point. This guide shows how to build a small, honest evaluation set from your own documents.
Lena Varga
·
Research Engineer
sievly
Evaluating accuracy
Public benchmarks rarely look like your documents. A small evaluation set built from your own files will tell you more in an afternoon.
Pick the hard pages
Choose the documents that cause problems today: scans, handwriting, nested tables, long contracts. Easy pages make every tool look the same.
Write down the truth
For each page, record the values a careful reviewer would expect. Keep it in a simple spreadsheet.
Compare field by field
Run each tool and compare values field by field, not by overall text similarity. Look at where answers came from, not only whether they match.
Repeat as you change
Keep the set and rerun it whenever you change models or schemas.
More from the blog
sievly
Introducing s-1
Announcements
Introducing s-1, our new document reading model
s-1 reads PDFs, scans and spreadsheets the way a careful analyst does, and returns structured data with a citation for every value. It is available in preview today.
sievly
Reading order
Research
Research note: reading order in complex layouts
Multi-column pages, sidebars and callouts confuse simple reading order. We share what we learned teaching s-1 to follow the page the way a person would.
sievly
Quarryline × Sievly
Case Studies
How Quarryline turns research notes into a searchable knowledge base
Quarryline’s data team uses Parse and Extract to turn thousands of analyst notes, filings and spreadsheets into clean, cited records their AI assistant can trust.