One public document-conversion quality comparison
This page preserves the full comparison published on our homepage — results, method, limitations, and the bar fixed before the run — in one shareable place.
Published 2026-07-21. The re-run executed on 2026-07-21 reproduced every figure byte-identically.
The one head-to-head we've run — and the bar we missed
Before running it we fixed a pass mark: we would only claim to be better if we clearly won at least 70% of the documents. We won 33%. So we have never made that claim, and this page is not making it now.
| Document | Format | LessTokens | MarkItDown | Result |
|---|---|---|---|---|
| Income statement | XLSX | 5/5 | 5/5 | Tie |
| Holdings table | XLSX | 5/5 | 5/5 | Tie |
| Spreadsheet edge cases | XLSX | 5/5 | 4/5 | We win |
| Service agreement | DOCX | 5/5 | 5/5 | Tie |
| Quarterly deck | PPTX | 5/5 | 5/5 | Tie |
| Operations report | 5/5 | 3/5 | We win | |
| Two-column report | 5/5 | 3/5 | We win | |
| Financial CSV | CSV | 5/5 | 5/5 | Tie |
| HTML report | HTML | 5/5 | 5/5 | Tie |
45/45 versus 40/45 — three wins, six ties, no losses.
MarkItDown did well on this test
It scored 40 out of 45, and handled six of the nine documents perfectly — exactly as we did. It is a capable free tool and we are not going to pretend otherwise. The difference showed up in three places, not everywhere.
Why that bar could not be reached
On six of the nine documents both tools scored full marks. A document both tools handle perfectly is a tie however good either one is — so 33% is the highest score any product could reach on this test, including a flawless one. The bar was badly designed. We are telling you that rather than quietly moving it.
What we are willing to say
On these nine documents we scored full marks on every one, and we never scored lower than the other tool. The three documents that actually separated the two — spreadsheet edge cases, and both PDFs including a two-column layout — went to us.
What we are not saying
That we are the best tool available. There was one other tool in this test. We also have not yet run the comparison many people care about most — handing the original file straight to the AI — so we cannot tell you how we compare against that.
Everything that limits this result
- LessTokens 16.343.0 against MarkItDown 0.1.6 — the measured LessTokens pipeline under the current versioning rule, and MarkItDown's current release at the time — re-run 2026-07-21 on a freshly generated corpus, reproducing every figure exactly. Both versions are stated so you can reproduce it.
- Nine documents. That is a small sample and we describe it as one.
- The documents are synthetic, generated by standard open-source libraries rather than by our own test tooling — so the test is not written in a dialect that happens to suit us. They are still not real-world files.
- One competitor only.
- We scored full marks on everything, which means this test never found our limit. We know we won it. We do not know by how much.
How the test was built
- The nine documents are generated by standard open-source libraries — openpyxl, python-docx, python-pptx, reportlab and pypdf — not by our own test tooling. That distinction matters: fixtures written by our own builders would measure how well another tool speaks our dialect, not how good it is.
- Each document is scored on five things: reading order, headings, tables, numbers, and whether anything went missing. Anything a heuristic could legitimately disagree about is escalated to a human rather than quietly passed.
- Output from both tools passes through the same normalizer, which takes no parameter identifying which tool produced the text. It cannot favour either side.
- If a tool cannot open a file at all, that is recorded as a compatibility problem and kept out of the quality score — counting a competitor's crash as a quality win is how a lopsided result gets manufactured. It did not arise here: both tools produced output for all nine documents.
Think this is wrong?
Tell us what you would change and we will re-run it and publish the new numbers — including if they come out worse for us. Write to support@lesstokens.ai.
MarkItDown is an open-source project released by Microsoft under the MIT licence. LessTokens is an independent tool and is not affiliated with, endorsed by, or sponsored by Microsoft. The comparison was run locally against the published package; no service was called.