znewznab

A decoder for newznab API responses, for Zig 0.17.

The decoder issues no requests and follows no links. It turns a response body into values; fetching it is the caller's business, as is retrieving the NZB each result points at — for which see znzb.

Installation

$ zig fetch --save git+https://git.jcollie.dev/jeff/znewznab.git

Its one dependency is zxml, a small XML pull parser — Zig's standard library has none — which in turn uses zig-charset to decode documents that are not UTF-8. zig fetch resolves it over git+https; note that git+ssh is not a scheme Zig understands, so the dependency URL is the https form even where the push remote is ssh.

Zig 0.16

For Zig 0.16.0, use the zig-0.16 branch, which holds the last of znewznab to build with it:

$ git clone -b zig-0.16 https://git.jcollie.dev/jeff/znewznab.git

or, as a dependency:

$ zig fetch --save https://git.jcollie.dev/jeff/znewznab/archive/zig-0.16.tar.gz

The v0.1.0 tag also builds with 0.16.0.

The problem it solves

A newznab indexer answers every request over the same endpoint, and which of three unrelated documents comes back is only discoverable from the root element. The error document is the sharp edge: it arrives with an HTTP 200 status for bad credentials just as readily as for a bad parameter, so a caller that trusts the transport will happily parse a failure as an empty result set. parse sniffs the root and returns a tagged union, so that ambiguity is handled once:

• <rss> — a search result feed (t=search, t=tvsearch, t=movie, …). RSS 2.0 with per-item metadata in <newznab:attr name= value=> elements.
• <caps> — the capability document (t=caps): server identification, result limits, which searches are supported, and the category tree.
• <error> — a failure, with a newznab code and description.

Design

• One arena per response. Document owns an arena holding every string it hands out, so the whole result is released with one deinit and nothing borrows from the input buffer after parse returns.
• Liberal about what it accepts. Unknown elements, namespace prefixes, and extension attributes are skipped rather than rejected — indexers add their own freely. Missing or malformed numbers come back null rather than failing the parse. An attr whose value is in the element text instead of the value attribute is read anyway.
• newznab:attr is not a closed set, so it is exposed as an ordered list rather than a fixed struct, with attr, attrUnsigned, and attrIterator over it. category repeats, once per matching category, which is why the iterator exists and why categoryId returns the last one.
• No external entities. The document type declaration is handed over as an opaque token: parse uses zxml's plain Reader.init rather than the initDtd that reads the internal subset, so entity resolution is the five predefined entities and numeric character references and nothing else, and no attribute arrives that the document did not write. zxml fetches nothing external under either reader — a SYSTEM or PUBLIC identifier is recorded and never dereferenced — so a hostile indexer cannot point it at a file or a URL.
• Fuzz target included. src/newznab.zig carries a std.testing.fuzz target that runs the parser under the testing allocator, so zig build test replays the corpus on every run, and zig build test --fuzz fuzzes it with coverage feedback; the XML parser has its own in zxml.

The XML parser is zxml's, kept as an implementation detail rather than re-exported: code that needs the parser should depend on zxml directly, so that this package's public API stays its own. The dependency is pinned to a commit, so a build is reproducible even if zxml's main moves on.

Usage

const std = @import("std");
const znewznab = @import("znewznab");

var doc = try znewznab.parse(gpa, body, .{});
defer doc.deinit();

switch (doc.value) {
    .feed => |feed| for (feed.items) |item| {
        std.debug.print("{s} — {?d} bytes — {s}\n", .{
            item.title, item.size(), item.nzbUrl(),
        });
    },
    .caps => |caps| {
        if (!caps.supportsSearch("tv-search")) return error.NoTvSearch;
    },
    .api_error => |err| {
        // These arrive with a 200 status, so nothing upstream will have
        // noticed the request failed.
        if (err.isRateLimit()) return error.BackOff;
        if (err.isAuthFailure()) return error.BadApiKey;
        return error.IndexerRejectedRequest;
    },
}

When results are the only acceptable answer, parseFeed unwraps the union and turns an <error> document into error.IndexerReturnedError:

var doc = try znewznab.parseFeed(gpa, body, .{});
defer doc.deinit();
for (doc.value.feed.items) |item| { ... }

Reading straight from a stream instead of a slice:

var doc = try znewznab.parseReader(gpa, reader, .limited(16 * 1024 * 1024), .{});
defer doc.deinit();

Paging

<newznab:response offset= total=> is what tells you a query matched more than came back. Feed.nextOffset folds the arithmetic into one call and returns null at the end, distinguishing that from a feed whose indexer sent no <newznab:response> at all — for which hasMore returns null rather than false:

var offset: u32 = 0;
while (true) {
    const body = try fetch(indexer, query, offset);
    var doc = try znewznab.parseFeed(gpa, body, .{});
    defer doc.deinit();
    for (doc.value.feed.items) |item| try collect(item);
    offset = doc.value.feed.nextOffset() orelse break;
}

Categories

Subcategories are their parent plus an offset, so topLevelCategory recovers the parent by integer division, and znewznab.category names the eight standard roots:

const id = item.categoryId() orelse continue;
if (znewznab.newznab.topLevelCategory(id) != znewznab.category.tv) continue;

For an indexer's own tree — the names it gives 5040 and friends — read <caps> and ask Caps.categoryName.

Command line

The package also builds a znewznab binary, which decodes any of the three responses and summarizes it. It is a worked example of the API as much as a tool, and it is handy for looking at what an indexer actually returned.

$ curl -s "$INDEXER/api?t=caps&apikey=$KEY" | zig build run -- -l -
-: newznab capabilities
  server: Example Indexer
  retention: 1100 days
  max results: 100
  search: yes (q)
  tv-search: yes (q,rid,season,ep)
  5000 TV
    5040 HD

An <error> response is printed and exits non-zero, so it does not pass silently in a pipeline.

Testing

$ zig build test

The fuzz targets run their corpus as part of that. To fuzz continuously, with a web interface, or for a bounded number of runs followed by a report:

$ zig build test --fuzz
$ zig build test --fuzz=1M

The test binaries are always compiled with LLVM, since Zig's self-hosted backend emits no coverage instrumentation and the fuzzer would otherwise run blind.

Standards

Standard	Title	Support in znewznab
newznab API	Indexer query API	Search feeds, the capability document, and the error document. Namespace http://www.newznab.com/DTD/2010/feeds/attributes/. There is no maintained specification; the behaviour here is what indexers actually serve
RSS 2.0	Really Simple Syndication	The search feed is an extension of it: channel, item, enclosure, guid, pubDate
RFC 822	Date and time specification	pubDate and usenetdate, parsed by parseRfc822Date
RFC 2822	Internet Message Format	The obsolete two-digit years and alphabetic zones that same parser accepts
XML 1.0 (5th ed.)	Extensible Markup Language	Parsing is zxml's; this library only walks the result

Where this lives

The repository's home is my Forgejo instance at git.jcollie.dev/jeff/znewznab, which is where CI runs. It is mirrored on Tangled at tangled.org/jcollie.dev/znewznab, and also published on Radicle, a peer-to-peer forge built on git, where the copy needs no account and no server anyone has to keep running.

$ git clone https://git.jcollie.dev/jeff/znewznab.git

$ git clone https://tangled.org/jcollie.dev/znewznab

$ rad clone rad:z2NeKJswLGoBDUYAMcPbbYHqW4YkV

Any of them is the whole project, on the main branch, with the same history.

rad clone finds seeds through your local node's routing table rather than through a known host, so the node has to be running before it can find anything:

$ rad node start

If you already have the repository and only want to help host it, seeding it tells your node to carry a copy for others:

$ rad seed rad:z2NeKJswLGoBDUYAMcPbbYHqW4YkV

License

MIT.