Spec: M3 — zixl reads installer packages #18

New issue

Closed

opened 2026-10-14 09:50:38 -05:00 by jeff · 1 comment

jeff commented

2026-10-14 09:50:38 -05:00

Owner

Problem Statement

zixl writes installer packages, but it cannot read one. Looking inside a package, its own or anyone else's, takes msitools' msiinfo and msidump. zixl's own tests depend on them, so they test zixl against a tool that has its own bugs. And a Windows user has no msiinfo at all.

Solution

The library reads an installer package back into an installer database: its tables and rows, with strings decoded from the database's code page, the contents of binary cells, the summary information, and every other stream. Two programs sit on top of it. zixl-info takes msiinfo's subcommands: tables, streams, suminfo, export [-s] and extract. zixl-dump takes msidump's options, writing tables as text archive (.idt) files and streams as files. zixl's tests then read packages with zixl's own tools. msitools stays as a cross-check, comparing what each side reads out of the same packages.

User Stories

As a packager, I want to list the tables in an installer package, so that I can see what it holds.
As a packager, I want to export a table as text, so that I can read its rows.
As a packager, I want a table exported as SQL, so that I can recreate it elsewhere.
As a packager, I want to see the summary information, so that I can check the package code, platform and installer version.
As a packager, I want to list and extract streams, so that I can get at embedded cabinets and binary data.
As a packager, I want a whole package dumped to a directory as .idt files, so that I can diff two packages.
As a Windows user, I want all of this without msitools.
As a maintainer, I want packages written by wixl and other tools read too, not just zixl's own.
As a maintainer, I want text written as Windows Installer's text archive format documents it, even where msitools does not.
As a library user, I want read to return the same installer database type that write takes, so that a package can be read, changed and written again.
As a maintainer, I want any bytes at all either read or refused with a reason, never a crash.

Implementation Decisions

Library: zixl.read takes an installer package's bytes and returns its installer database (Database) plus the package's streams. It is the reverse of writing:
- the compound file (versions 3 and 4, mini streams, the DIFAT);
- stream names;
- the string pool (short and long references, long strings, saturated duplicates);
- _Tables and _Columns, the table streams, and the summary information.
Strings: decoded to UTF-8 from the database's code page, and the summary information's strings from its own. Bytes a code page cannot decode become U+FFFD rather than an error.
A package that cannot be read is refused through Diagnostics, naming what is wrong.
Not followed into: sub-storages (transforms, _Storages) and digital signatures. They are listed but not read.
Two programs, zixl-info and zixl-dump, keep the command lines of msiinfo and msidump. Where msitools departs from Microsoft's documentation, zixl follows the documentation:
- the text archive format's escapes for tab, carriage return and line feed;
- _ForceCodepage.idt ending its line;
- no stray NUL bytes.
Times are printed in UTC. Pure Zig has no local time zone.

Testing Decisions

Round trip: a seam-1 test builds databases from WiX sources, writes them, reads them back, and checks every table, row and summary field.
Library: unit tests cover the compound-file reader, stream names and the string pool, including the long-string and long-reference cases zixl never writes.
Fuzzing: a fuzz target hands read arbitrary bytes.
Against msitools: nix flake check reads every fixture's packages, zixl's and wixl's, with both zixl's tools and msitools, and compares the output, normalizing only the documented differences.
The e2e tests switch over: they use zixl-info and zixl-dump where they now use msiinfo and msidump.

Out of Scope

Changing a package in place, and SQL queries beyond export -s.
Transforms, patches, merge modules and signature checking (msidump -S).
Extracting files from cabinets.
WiX v4 sources (M4).

Further Notes

docs/msi-layout.md already describes the layout from the writer's side. [MS-CFB] and [MS-OLEPS] cover the containers.

Tickets

## Problem Statement zixl writes installer packages, but it cannot read one. Looking inside a package, its own or anyone else's, takes msitools' `msiinfo` and `msidump`. zixl's own tests depend on them, so they test zixl against a tool that has its own bugs. And a Windows user has no `msiinfo` at all. ## Solution The library reads an installer package back into an installer database: its tables and rows, with strings decoded from the database's code page, the contents of binary cells, the summary information, and every other stream. Two programs sit on top of it. `zixl-info` takes `msiinfo`'s subcommands: `tables`, `streams`, `suminfo`, `export [-s]` and `extract`. `zixl-dump` takes `msidump`'s options, writing tables as text archive (`.idt`) files and streams as files. zixl's tests then read packages with zixl's own tools. msitools stays as a cross-check, comparing what each side reads out of the same packages. ## User Stories 1. As a packager, I want to list the tables in an installer package, so that I can see what it holds. 2. As a packager, I want to export a table as text, so that I can read its rows. 3. As a packager, I want a table exported as SQL, so that I can recreate it elsewhere. 4. As a packager, I want to see the summary information, so that I can check the package code, platform and installer version. 5. As a packager, I want to list and extract streams, so that I can get at embedded cabinets and binary data. 6. As a packager, I want a whole package dumped to a directory as `.idt` files, so that I can diff two packages. 7. As a Windows user, I want all of this without msitools. 8. As a maintainer, I want packages written by wixl and other tools read too, not just zixl's own. 9. As a maintainer, I want text written as Windows Installer's text archive format documents it, even where msitools does not. 10. As a library user, I want `read` to return the same installer database type that `write` takes, so that a package can be read, changed and written again. 11. As a maintainer, I want any bytes at all either read or refused with a reason, never a crash. ## Implementation Decisions - **Library:** `zixl.read` takes an installer package's bytes and returns its installer database (`Database`) plus the package's streams. It is the reverse of writing: - the compound file (versions 3 and 4, mini streams, the DIFAT); - stream names; - the string pool (short and long references, long strings, saturated duplicates); - `_Tables` and `_Columns`, the table streams, and the summary information. - **Strings:** decoded to UTF-8 from the database's code page, and the summary information's strings from its own. Bytes a code page cannot decode become U+FFFD rather than an error. - **A package that cannot be read** is refused through `Diagnostics`, naming what is wrong. - **Not followed into:** sub-storages (transforms, `_Storages`) and digital signatures. They are listed but not read. - **Two programs, `zixl-info` and `zixl-dump`,** keep the command lines of `msiinfo` and `msidump`. Where msitools departs from Microsoft's documentation, zixl follows the documentation: - the text archive format's escapes for tab, carriage return and line feed; - `_ForceCodepage.idt` ending its line; - no stray NUL bytes. - **Times are printed in UTC.** Pure Zig has no local time zone. ## Testing Decisions - **Round trip:** a seam-1 test builds databases from WiX sources, writes them, reads them back, and checks every table, row and summary field. - **Library:** unit tests cover the compound-file reader, stream names and the string pool, including the long-string and long-reference cases zixl never writes. - **Fuzzing:** a fuzz target hands `read` arbitrary bytes. - **Against msitools:** `nix flake check` reads every fixture's packages, zixl's and wixl's, with both zixl's tools and msitools, and compares the output, normalizing only the documented differences. - **The e2e tests switch over:** they use `zixl-info` and `zixl-dump` where they now use `msiinfo` and `msidump`. ## Out of Scope - Changing a package in place, and SQL queries beyond `export -s`. - Transforms, patches, merge modules and signature checking (`msidump -S`). - Extracting files from cabinets. - WiX v4 sources (M4). ## Further Notes `docs/msi-layout.md` already describes the layout from the writer's side. [MS-CFB] and [MS-OLEPS] cover the containers. ## Tickets - [ ] #19 - [ ] #20 - [ ] #21 - [ ] #22

jeff added the

ready-for-agent

label

2026-10-14 09:50:38 -05:00

jeff referenced this issue

2026-10-14 09:50:52 -05:00

zixl-info lists the tables and summary information of any installer package #19

jeff referenced this issue

2026-10-14 09:50:53 -05:00

zixl-info exports tables, and lists and extracts streams #20

jeff referenced this issue

2026-10-14 09:50:54 -05:00

zixl-dump writes a package out as .idt files and streams #21

jeff referenced this issue

2026-10-14 09:50:55 -05:00

The end-to-end tests read packages with zixl's own tools #22

jeff commented

2026-10-16 13:07:59 -05:00

Author

Owner

Every ticket is done: #19 (7bb90da), #20 (f549911), #21 (5cecf36), #22 (b9e6042). zixl.read reads packages back into a Database, zixl-info and zixl-dump take msiinfo's and msidump's command lines, a fuzz target hands read arbitrary bytes, and compare.sh cross-checks the reader against msitools on every fixture.

Every ticket is done: #19 (7bb90da), #20 (f549911), #21 (5cecf36), #22 (b9e6042). `zixl.read` reads packages back into a `Database`, zixl-info and zixl-dump take msiinfo's and msidump's command lines, a fuzz target hands `read` arbitrary bytes, and compare.sh cross-checks the reader against msitools on every fixture.

jeff closed this issue

2026-10-16 13:08:00 -05:00