znewznab

A decoder for newznab API responses, for Zig 0.16.

The decoder issues no requests and follows no links. It turns a response body into values; fetching it is the caller's business, as is retrieving the NZB each result points at — for which see znzb.

Installation

$ zig fetch --save git+https://git.ocjtech.us/jeff/znewznab.git

Its one dependency is zxml, a small XML pull parser with no dependencies of its own — Zig's standard library has none. zig fetch resolves it over git+https; note that git+ssh is not a scheme Zig understands, so the dependency URL is the https form even where the push remote is ssh.

The problem it solves

A newznab indexer answers every request over the same endpoint, and which of three unrelated documents comes back is only discoverable from the root element. The error document is the sharp edge: it arrives with an HTTP 200 status for bad credentials just as readily as for a bad parameter, so a caller that trusts the transport will happily parse a failure as an empty result set. parse sniffs the root and returns a tagged union, so that ambiguity is handled once:

• <rss> — a search result feed (t=search, t=tvsearch, t=movie, …). RSS 2.0 with per-item metadata in <newznab:attr name= value=> elements.
• <caps> — the capability document (t=caps): server identification, result limits, which searches are supported, and the category tree.
• <error> — a failure, with a newznab code and description.

Design

• One arena per response. Document owns an arena holding every string it hands out, so the whole result is released with one deinit and nothing borrows from the input buffer after parse returns.
• Liberal about what it accepts. Unknown elements, namespace prefixes, and extension attributes are skipped rather than rejected — indexers add their own freely. Missing or malformed numbers come back null rather than failing the parse. An attr whose value is in the element text instead of the value attribute is read anyway.
• newznab:attr is not a closed set, so it is exposed as an ordered list rather than a fixed struct, with attr, attrUnsigned, and attrIterator over it. category repeats, once per matching category, which is why the iterator exists and why categoryId returns the last one.
• No external entities. The parser has no DTD subset evaluation and no entity resolution beyond the five predefined ones and numeric character references, so a hostile indexer cannot point it at a file or a URL.
• Fuzz target included. src/newznab.zig carries a std.testing.fuzz target that runs the parser under the testing allocator, so zig build test replays the corpus on every run; the XML parser has its own in zxml. Note that zig build test --fuzz does not currently work with Zig 0.16.0: fuzz mode fails to build the compiler's own test runner, and a trivial one-test file reproduces it. The targets are there for the toolchain that fixes it.

The XML parser is zxml's, kept as an implementation detail rather than re-exported: code that needs the parser should depend on zxml directly, so that this package's public API stays its own. The dependency is pinned to a commit, so a build is reproducible even if zxml's main moves on.

Usage

const std = @import("std");
const znewznab = @import("znewznab");

var doc = try znewznab.parse(gpa, body, .{});
defer doc.deinit();

switch (doc.value) {
    .feed => |feed| for (feed.items) |item| {
        std.debug.print("{s} — {?d} bytes — {s}\n", .{
            item.title, item.size(), item.nzbUrl(),
        });
    },
    .caps => |caps| {
        if (!caps.supportsSearch("tv-search")) return error.NoTvSearch;
    },
    .api_error => |err| {
        // These arrive with a 200 status, so nothing upstream will have
        // noticed the request failed.
        if (err.isRateLimit()) return error.BackOff;
        if (err.isAuthFailure()) return error.BadApiKey;
        return error.IndexerRejectedRequest;
    },
}

When results are the only acceptable answer, parseFeed unwraps the union and turns an <error> document into error.IndexerReturnedError:

var doc = try znewznab.parseFeed(gpa, body, .{});
defer doc.deinit();
for (doc.value.feed.items) |item| { ... }

Reading straight from a stream instead of a slice:

var doc = try znewznab.parseReader(gpa, reader, .limited(16 * 1024 * 1024), .{});
defer doc.deinit();

Paging

<newznab:response offset= total=> is what tells you a query matched more than came back. Feed.nextOffset folds the arithmetic into one call and returns null at the end, distinguishing that from a feed whose indexer sent no <newznab:response> at all — for which hasMore returns null rather than false:

var offset: u32 = 0;
while (true) {
    const body = try fetch(indexer, query, offset);
    var doc = try znewznab.parseFeed(gpa, body, .{});
    defer doc.deinit();
    for (doc.value.feed.items) |item| try collect(item);
    offset = doc.value.feed.nextOffset() orelse break;
}

Categories

Subcategories are their parent plus an offset, so topLevelCategory recovers the parent by integer division, and znewznab.category names the eight standard roots:

const id = item.categoryId() orelse continue;
if (znewznab.newznab.topLevelCategory(id) != znewznab.category.tv) continue;

For an indexer's own tree — the names it gives 5040 and friends — read <caps> and ask Caps.categoryName.

Command line

The package also builds a znewznab binary, which decodes any of the three responses and summarizes it. It is a worked example of the API as much as a tool, and it is handy for looking at what an indexer actually returned.

$ curl -s "$INDEXER/api?t=caps&apikey=$KEY" | zig build run -- -l -
-: newznab capabilities
  server: Example Indexer
  retention: 1100 days
  max results: 100
  search: yes (q)
  tv-search: yes (q,rid,season,ep)
  5000 TV
    5040 HD

An <error> response is printed and exits non-zero, so it does not pass silently in a pipeline.

Testing

$ zig build test

The fuzz targets run their corpus as part of that. --fuzz for continuous fuzzing is blocked by the Zig 0.16.0 issue noted above.

Standards

Standard	Title	Support in znewznab
newznab API	Indexer query API	Search feeds, the capability document, and the error document. Namespace http://www.newznab.com/DTD/2010/feeds/attributes/. There is no maintained specification; the behaviour here is what indexers actually serve
RSS 2.0	Really Simple Syndication	The search feed is an extension of it: channel, item, enclosure, guid, pubDate
RFC 822	Date and time specification	pubDate and usenetdate, parsed by parseRfc822Date
RFC 2822	Internet Message Format	The obsolete two-digit years and alphabetic zones that same parser accepts
XML 1.0 (5th ed.)	Extensible Markup Language	Parsing is zxml's; this library only walks the result

License

MIT.