We test against files Microsoft Office actually wrote
A document engine tested against files its own authors made is tested against their understanding of the format. So our fixtures are written by Excel, Word and PowerPoint themselves — and the first run found four things we had wrong.
· 3 min read
There is a trap every document engine falls into. You write the reader, you write some test files, and the tests pass — because the files were written by the same people, with the same understanding of the format, as the code reading them. The tests confirm that you agree with yourself.
The files people actually open were written by Microsoft Office. So that is what Rutba Office is now tested against.
Fixtures written by Office itself
A script drives Excel, Word and PowerPoint — invisibly, touching nothing on the desktop and nothing on the network — to write a workbook, a document and a deck using what people really put in files. Each is saved twice, as OOXML and as OpenDocument, from the same content, so both formats come from one source of truth.
- A workbook: five sheets, cross-sheet formulas, named ranges, every number format, conditional formats, four kinds of chart, sixteen shapes, a picture, and a table with a total row.
- A document: a table of contents, lists, tables, floating shapes, a footnote and an endnote, a comment, fields, tracked changes, an equation, a landscape section and a watermark.
- A deck: a gradient, bullets, a shape gallery, a table, two charts, WordArt, animations, speaker notes and sections.
The test opens each file through the same document service the application uses and holds the engine to Excel’s own cached results — the values Excel wrote into the file when it saved. If our formula engine and Excel disagree, the test fails, and Excel is right.
What the first run found
| What was wrong | What a user would have seen |
|---|---|
| SUBTOTAL was not a function | Every Excel table’s total row reading #NAME? |
| An .ods lost its named ranges, merged cells and number formats | A spreadsheet that opened, and was quietly a different spreadsheet |
| Built-in date format 14 used the specification’s American order | Dates in the wrong order on any machine outside the US |
| A chart on a slide was an unsupported graphic | A deck with six chart slides showing six empty frames |
The date one is the instructive one. The specification says format 14 is month/day/year. Excel, in practice, shows the machine’s own short date. A reader written from the specification is correct and wrong at the same time — which is precisely the class of bug that only files written by the real application can catch.
Tests against files you wrote yourself confirm that you agree with yourself.
And a pile of files from the wild
Alongside the fixtures, a corpus run opens a collection of real downloaded files in the application. The last one showed 84 of 87 and refused three: two truncated archives, rightly, with a sentence saying so — and an encrypted presentation described as "a document, not a presentation".
That last one was a real bug. A password-protected .docx, .xlsx or .pptx is a compound file wrapping an encrypted package, and the reader had been failing to recognise it — so an encrypted Word document opened as an empty page. It is now refused as a password-protected file, whatever its extension, with what to do about it.
Why your .docx looks different in another program
The long answer to the question this testing exists to solve, on the Rutba Office site.
Read the article
