Sort text by the Unicode Collation Algorithm, in a zig-collate of its own #1

New issue

Open

opened 2026-09-12 13:45:31 -05:00 by jeff · 0 comments

jeff commented

2026-09-12 13:45:31 -05:00

Owner

xsl:sort compares text by code point. That is not merely unlocalized, it is
wrong in a way users notice: Mueller sorts before Märtens, because u is
U+0075 and ä is U+00E4.

Fixing it properly means the Unicode Collation Algorithm, UTS #10, and
that belongs in a library of its own — zig-collate — for the same reason the
calendar is zig-datetime's. A collation is the kind of thing
that is wrong in the same way in every project that reimplements one, and
nothing about it is XSLT's.

The four pieces

Normalization to NFD — canonical decomposition, then a sort by
canonical combining class.
DUCET, the default collation element table. That is allkeys.txt, some
twenty-nine thousand entries, which wants fetching and generating at build
time the way zig-datetime handles tzdata rather than being checked in by
hand.
Sort-key construction — the primary, secondary and tertiary weight
levels concatenated, so that a comparison becomes one memcmp once the key
for each node has been built.
CLDR tailorings, optionally and last. This is the only part that is
per-language, and so the only part xsl:sort lang= actually needs.

What already exists

uucode exposes canonical_combining_class and
decomposition_type/decomposition_mapping — exactly what step 1 runs on —
along with case folding, general category and script, and it is configurable
at build time so a caller pays only for the fields it names. It has none of
the other three: no DUCET, no sort keys, no tailorings. So it is the right
dependency for the data layer and is not itself a collation.

Two things worth knowing before starting

REC/test-10-3 needs none of step 4. Both glibc's de_DE and UCA root
give ä, ö and ü the same primary weights as a, o and u, and that alone orders
Märtens < Möller < Mueller. Strip lang="de" from that stylesheet and libxslt
falls back to exactly the code-point order zxsl already produces, so the
attribute carries the whole difference.

Passing that case is not the reason to do this. libxslt has no collation of
its own: libxslt/xsltlocale.c calls newlocale() and strxfrm_l(), so its
answer comes from the host C library's locale data, and the recorded output is
a property of the machine it was recorded on. Matching libxslt would mean
matching glibc, which is ISO 14651-derived and disagrees with CLDR in places.

The reason to do it is that sorting by code point is a poor default, and a
correct root collation is a better one for every stylesheet whether or not it
says lang.

Scope note

libxslt's case-order argument is ATTRIBUTE_UNUSED — it accepts and ignores
that attribute exactly as zxsl does, and no case in its corpus exercises it. So
case-order is at parity today and is not part of this.

`xsl:sort` compares text by code point. That is not merely unlocalized, it is wrong in a way users notice: `Mueller` sorts before `Märtens`, because `u` is `U+0075` and `ä` is `U+00E4`. Fixing it properly means the Unicode Collation Algorithm, [UTS #10][uca], and that belongs in a library of its own — `zig-collate` — for the same reason the calendar is [zig-datetime][zig-datetime]'s. A collation is the kind of thing that is wrong in the same way in every project that reimplements one, and nothing about it is XSLT's. ## The four pieces 1. **Normalization to NFD** — canonical decomposition, then a sort by canonical combining class. 2. **DUCET**, the default collation element table. That is `allkeys.txt`, some twenty-nine thousand entries, which wants fetching and generating at build time the way zig-datetime handles tzdata rather than being checked in by hand. 3. **Sort-key construction** — the primary, secondary and tertiary weight levels concatenated, so that a comparison becomes one `memcmp` once the key for each node has been built. 4. **CLDR tailorings**, optionally and last. This is the only part that is per-language, and so the only part `xsl:sort lang=` actually needs. ## What already exists [uucode][uucode] exposes `canonical_combining_class` and `decomposition_type`/`decomposition_mapping` — exactly what step 1 runs on — along with case folding, general category and script, and it is configurable at build time so a caller pays only for the fields it names. It has none of the other three: no DUCET, no sort keys, no tailorings. So it is the right dependency for the data layer and is not itself a collation. ## Two things worth knowing before starting **`REC/test-10-3` needs none of step 4.** Both glibc's `de_DE` and UCA root give ä, ö and ü the same primary weights as a, o and u, and that alone orders Märtens < Möller < Mueller. Strip `lang="de"` from that stylesheet and libxslt falls back to exactly the code-point order zxsl already produces, so the attribute carries the whole difference. **Passing that case is not the reason to do this.** libxslt has no collation of its own: `libxslt/xsltlocale.c` calls `newlocale()` and `strxfrm_l()`, so its answer comes from the host C library's locale data, and the recorded output is a property of the machine it was recorded on. Matching libxslt would mean matching glibc, which is ISO 14651-derived and disagrees with CLDR in places. The reason to do it is that sorting by code point is a poor default, and a correct root collation is a better one for every stylesheet whether or not it says `lang`. ## Scope note libxslt's `case-order` argument is `ATTRIBUTE_UNUSED` — it accepts and ignores that attribute exactly as zxsl does, and no case in its corpus exercises it. So `case-order` is at parity today and is not part of this. [uca]: https://www.unicode.org/reports/tr10/ [uucode]: https://github.com/jacobsandlund/uucode [zig-datetime]: https://git.jcollie.dev/jeff/zig-datetime

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

jeff/zxsl#1

No description provided.

Rows
Columns