zig-ftp

FTP for Zig 0.16, both ends of it: a client, a server, and the wire format underneath them as a layer you can use on its own.

$ zig build
$ ./zig-out/bin/zig-ftp serve --root /srv/pub &
$ ./zig-out/bin/zig-ftp ls 127.0.0.1:2121
-rw-r--r--   1 ftp      ftp                19 Sep  6 17:27 readme.txt
drwxr-xr-x   1 ftp      ftp                18 Sep  6 17:27 pub

Three layers, and you can stop at any of them:

ftp.protocol	the wire — replies, commands, listings, addresses, pathnames. Pure functions over bytes; no I/O, no allocation you did not ask for
ftp.Client	a client, over a control connection you supply or one it opens
ftp.server	a server: an accept loop, a session, and interfaces for what is served and who may see it
zig-ftp	a command-line client and server, which exist to show what all that looks like from outside

The whole thing is built on std.Io, so it runs on whatever Io implementation the calling program chose — threads, fibers, io_uring — and none of it spawns a thread of its own.

The API documentation is generated from the doc comments, which are most of the explanation of why FTP is the shape it is.

Adding it to a project

$ zig fetch --save git+https://git.ocjtech.us/jeff/zig-ftp.git

const ftp = b.dependency("zig_ftp", .{ .target = target }).module("ftp");
exe.root_module.addImport("ftp", ftp);

The client

const ftp = @import("ftp");

var connection: ftp.Client.Connection = undefined;
try connection.connect(io, gpa, address, .{});
defer connection.close();
const client = &connection.client;

try client.login("anonymous", "nobody@example.com", null);
defer client.quit() catch {};

var listing = try client.list(gpa, "/pub");
defer listing.deinit(gpa);
for (listing.entries) |entry| {
    std.debug.print("{s} {d} bytes\n", .{ entry.name, entry.size orelse 0 });
}

_ = try client.retrieve("/pub/readme.txt", destination, .{});

login reads the server's feature list on the way in, so everything after it picks the better command without a round trip to find out: MLSD where the server has it and LIST where it does not, EPSV before PASV, EPRT before PORT. The fallbacks are there because servers that need them still exist, and Client.Options turns each preference off for testing against one.

A Connection owns a plain TCP socket and its buffers. Client.init takes an Io.Reader and an Io.Writer instead, which is how something else — a TLS layer, a transcript recorder, a pair of pipes in a test — gets underneath it.

The address of a passive data connection is taken from the control connection, never from the server's 227 reply. A server behind a NAT reports an address that cannot be reached from outside it, a server that has been configured reports the one you already have, and a hostile server reports somebody else's. Only the port is read from the reply — which is all a 229 reply carries in the first place, and one of several reasons EPSV is better.

The server

var backing: ftp.server.DirFilesystem = .init(root_dir, .{});
var credentials: ftp.server.Auth.Simple = undefined;

var server: ftp.server.Server = try .listen(io, gpa, address, .{
    .filesystem = backing.filesystem(),
    .auth = .single(&credentials, "jeff", "hunter2", .all),
    .greeting = "welcome",
}, .{});
defer server.deinit();

try server.serve();

serve runs each session as a task of its own; serveOne accepts one and runs it inline, which is what a test or an inetd-style program wants. A program that accepts connections some other way can skip Server entirely and drive ftp.server.Session over any reader and writer.

What is served

Filesystem is a vtable with eight operations — stat, list, retrieve, store, create and remove a directory, remove a file, rename. DirFilesystem implements it over an Io.Dir and is what you want if you are serving a directory. Anything else — an object store, a build cache, a tree that exists only in a test — is those eight functions.

The operations are streaming rather than handle-based: retrieve is handed the data connection's writer and store its reader, so an implementation never has to own a file handle, close it on every path out, or find a buffer for it.

Who may see it

Auth answers USER, PASS and ACCT with "yes", "no", or "not yet, send the next one" — the three-way answer FTP's login actually has. A successful answer carries an Access, which says what that session may do, and Session checks it before the file system is asked. Auth.anonymous, Auth.single, Auth.open and Auth.closed cover the usual cases.

The password arrives in clear text, because RFC 959 has no other way to send one. Auth.Context.protected reports whether the control connection is encrypted so that an implementation can refuse to accept one over a connection that is not — see TLS, below.

Containment

A client's pathname is resolved into the virtual name space before anything else happens: . and empty components are dropped, .. removes the component before it, and a .. at the root is a no-op rather than an escape. So ../../../etc/shadow is /etc/shadow and stays inside the tree, and a backend gets that guarantee whether or not it is a file system at all. DirFilesystem then opens everything relative to the root directory handle, with resolve_beneath where the platform supports it, as a second layer — and does not follow symbolic links, so one inside the tree that points out of it fails to open rather than serving what it points at.

Active-mode data connections are restricted to the control connection's own host, and to ports above 1023. Without that a server is an open relay for port scans — the FTP bounce attack — for anyone who can log in. Config can turn active mode off entirely.

The protocol layer

Everything under ftp.protocol is a pure function of bytes. It is what the client and the server share, and it is worth having on its own: to build something neither of them does, to read a captured session, to check a pathname without opening a connection.

Reply	the framing, including the multi-line form whose only terminator is the code repeated with a space after it — and the padding that keeps a line of somebody's error message from ending the reply early
Command	the verb table, and the TELNET escapes ABOR still arrives wrapped in
Code	the three-digit codes, and what their digits mean when the code is one nobody has named
address	PORT, the 227 reply to PASV, EPRT, the 229 reply to EPSV
listing	MLSD facts, and heuristics for the ls -l and MS-DOS layouts LIST sends
transfer	TYPE, STRU, MODE, and the ASCII line-ending conversion as a reader and a writer adapter
path	virtual pathname resolution, and the quoting a 257 reply uses
time	the MDTM timestamp format, and a proleptic Gregorian calendar that works before 1970
Features	the reply to FEAT

The command-line program

$ zig-ftp ls    <host>[:port] [path]
$ zig-ftp get   <host>[:port] <path> [out]
$ zig-ftp put   <host>[:port] <path> [in]
$ zig-ftp serve [--root DIR] [--bind ADDR] [--port N] [--write]

get and put use stdout and stdin when no local file is named. -a opens data connections actively, -t transfers in ASCII mode, and -o makes the client behave like one written before 1997 — PASV and LIST rather than EPSV and MLSD — which is how the fallback paths get exercised by hand.

The server listens on 2121 by default and on 127.0.0.1, because a demonstration server should not want privilege and should not be reachable from the next machine along by accident.

What is implemented

RFC 959, and the extensions that made it usable:

RFC 959	the protocol: access control, transfer parameters, the service commands, the reply codes
RFC 1123 §4.1	the corrections — STOU reporting the name it chose, PASV per transfer, and the rest
RFC 2389	FEAT and OPTS
RFC 2428	EPRT and EPSV, and so IPv6; EPSV ALL is honoured
RFC 2640	UTF-8 pathnames, which here means pathnames are opaque bytes
RFC 3659	MLST, MLSD, SIZE, MDTM, REST STREAM, TVFS

STRU and MODE accept only F and S and answer 504 to the rest, which is what every server does: the block and compressed transfer modes and the record and page structures have no implementations left to interoperate with. RFC 1639's LPRT and LPSV are named so they can be refused precisely, and not implemented — they were the interim answer to IPv6 before RFC 2428 and are Historic.

TLS

Not implemented, and honestly rather than quietly. AUTH, PBSZ, PROT and CCC are answered with a 502, and Features.auth_tls reports whether a server offered TLS so that a client which cares can refuse to send a password to one that would have protected it and did not.

The seam is real, though, and is why the client and the session both hold the control connection as an Io.Reader and an Io.Writer rather than as a socket: a TLS layer goes between them and the network without either of them changing. Zig 0.16 ships std.crypto.tls.Client and no server, so the client half of explicit FTPS could be built on the standard library today and the server half could not.

Until something does that, treat a password sent over this as public.

What is not here

Besides TLS:

• No idle timeout in the server. A client that connects and says nothing is waited on forever. std.Io has no way to put a deadline on a read through an Io.Reader, so the timeout belongs to whatever runs the session — run it as a task and cancel that task, which is what Server.serve already gives you an Io.Group for. Anything listening on a public network wants one.
• Session.Config.listing_max_entries bounds what is sent, not what is allocated. The Filesystem builds the whole list before the session can count it, so a backend that may be asked for an unboundedly large directory has to bound itself.
• ABOR cannot interrupt a transfer in progress, because a session reads its control connection between commands and not during one. It is answered correctly when there is nothing to abort. Client.abort sends the in-band half of the TELNET Synch and not the urgent-data half, which std.Io has no way to send.
• CI does not run the virtual machine tests. They need KVM and the runners cannot nest virtualisation, so every interoperability check — six machines, four other servers and five other clients — happens only when somebody runs nix flake check by hand.
• A server bound to the unspecified address cannot answer PASV unless Config.passive_address is set, and says so with a 502 rather than advertising an address it cannot vouch for. There is no portable way to ask a socket which of the machine's addresses a particular client reached it on. EPSV, which sends only a port, is unaffected.

Tests

In the tree

$ zig build test --summary all

174 of them. Most are unit tests against what the RFCs say, and the interesting ones are not:

• tests/e2e.zig runs a real client against a real server over a loopback socket, which is the only way to catch the mistakes that are self-consistent: a reply written in a shape only its own parser accepts, a transfer whose end-of-file the two ends disagree about, a data connection opened in an order that works only when nothing is really in flight.

• tests/fuzz.zig states properties rather than examples — that a parser terminates on any input, and that anything it claims to have understood survives being written back out and read again. path.resolve gets the strongest one, because it is a security boundary: no input may produce a path outside the root, and the answer must be a fixed point.

  Two of the eleven targets are whole conversations rather than single lines. session runs a complete server session over arbitrary command bytes and asserts that everything it says back parses as a sequence of replies with nothing left over — which is what would catch a pathname smuggling a CRLF into the control connection. client is the mirror: every call that reads a reply and parses something out of it, against whatever a server sends. Both run on std.Io.failing, so neither opens a socket.

  $ zig build fuzz-run                                  # a minute of each
  $ zig build fuzz-run -- --seconds 300 --target listings
  $ zig build fuzz-run -- --input fuzz-findings/x.bin --target listings
  

  zig build fuzz --fuzz is the obvious thing to run and does not work on this toolchain: the devshell's patched Zig gets it to compile, and it then stops with pcs_len was zero because nothing in 0.16.0 fills the coverage table. tools/fuzz.zig is the loop written down honestly instead — no coverage feedback, and a corpus of real replies, commands and listing lines to mutate in its place. It says what it costs, and it found the three defects below.

  The properties also run as ordinary tests under zig build test, over the same seeds, so a shape that has been interesting once stays checked.

Against other implementations

$ nix flake check          # all six
$ nix build .#checks.x86_64-linux.pyftpdlib

Six NixOS virtual machine tests, because the interesting parts of talking to a real FTP server cannot be reached from a test binary: a real server chroots and drops privileges, and a real login is checked against a real user database.

These are not run by CI. Each boots a NixOS guest and so needs KVM, which the runners cannot provide, so they are run by hand — which means an interoperability regression will sit undetected until somebody does. Run them before a release, and after anything that touches the wire. Each covers an anonymous login and an authenticated one, and each checks that the wrong name and the wrong password are both refused. The server test goes further and checks they are refused identically, since a server whose two refusals differ hands out a list of which accounts exist.

proftpd	this client against the server with every extension: MLSD, MLST, SIZE, MDTM, REST, TVFS — and a login chrooted to its own home. Authenticates from a file
ftpd	this client against inetutils' BSD daemon, which has no MLSD at all, so it is what puts the LIST parser in front of a real ls -l. Authenticates from the shadow database
vsftpd	this client against the server most Linux machines run: a second, unrelated producer of the LIST format, authenticating through PAM, and the only one here reachable over IPv6
rclone	both ends against rclone: its serve ftp is the Go server goftp.io/server, its :ftp: backend the Go client jlaffaye/ftp, and neither shares a line with anything else here
pyftpdlib	both ends in Python: pyftpdlib serving, and aioftp — asynchronous, MLSD-insisting — as the client. pyftpdlib is the only server here whose perm facts differ per account, so it is where a client's reading of them is checked against a server that varies them
server	this server against ftplib, the BSD ftp client, curl and lftp, which agree with each other about very little

Between them they exercise the four ways of opening a data connection (EPSV, PASV, EPRT, PORT), binary and ASCII transfers, mget/mput, resuming with REST, and containment — a client must not be able to name anything outside what it was given, however it spells the path.

The IPv6 half of the vsftpd test is where the client's handling of a 227 reply earns itself. PASV has six decimal bytes for an address and cannot express an IPv6 one; asked on an IPv6 connection anyway, vsftpd answers with a real port and an address of 0.0.0.0, which is the only thing the format can hold. A client that connected to the address it was given would go to 0.0.0.0 and fail. This one takes the host from the control connection and only the port from the reply, so the transfer works over a command that cannot describe the connection it is opening.

Three things were found this way and are fixed:

• NLST on a file answered "not a directory". That is what a client's mget sends to expand each of its arguments, so mget did not work at all.
• RETR on a missing file announced 150 before discovering it was missing, which leaves a client holding the empty local file it had already created. The 550 now arrives instead of the 150 rather than after it.
• MKD failed whenever the server's 257 reply did not quote the path it made. RFC 959 §5.3.2 asks for the quoting and rclone's server does not do it — the directory was there and the call reported failure anyway. PWD still insists, because there the quoted path is the entire answer and there is nothing to fall back on.

rclone also taught the client's caller something worth repeating: a FEAT list says what a server promises, never what it refuses. rclone's server does not advertise SIZE and answers it perfectly well. Nothing here gates a command on FEAT for that reason — it is used to prefer MLSD over LIST, not to rule anything out.

Fuzzing found three more, two of them crashes:

• A malformed modify= fact in an MLSD reply panicked the client. parseTimeval narrowed each field into the four and five bit integers a date is stored in before checking its range, so a month of 99 was an integer cast that could not succeed. Any server, hostile or merely broken, could crash a client listing a directory. Ranges are now checked first.
• DateTime.fromUnixSeconds panicked at the extremes of i64 — the year did not fit an i32, and the seconds-into-the-day was computed with a multiplication that overflows one day below the bottom of the range. It is reachable from a file's modification time, so a single absurd mtime would have crashed the server listing the directory it is in. The calendar is now total: every i64 names a date, and a date no second count can express saturates rather than wrapping.
• Reading a reply took one trailing carriage return off a line where writing one took them all off, so 211 End\r\r\n did not survive a round trip. Not a crash, but a codec that is lossy in one direction is a codec that will quietly corrupt something eventually.

And one thing was found about the environment rather than the code, written down in tests/nixos/proftpd.nix because it costs an afternoon otherwise: nixpkgs builds proftpd without shadow support. There is no getspnam in the binary, so mod_auth_unix only ever sees the x that /etc/passwd holds in place of the hash, and every login fails with crypt(3) failed: Invalid argument and a log line that says only "Incorrect password". The test authenticates from an AuthUserFile instead, which is an ordinary way to run proftpd and still a real crypt(3) check against a real hash.

The API documentation

https://jeff.ocj.page/zig-ftp/, rebuilt from main on every push. Zig writes it out of the doc comments, which in this project carry most of the explanation — why a multi-line reply can only be ended by looking for its own code again, why EPSV is better than PASV, what TYPE A actually converts.

$ zig build docs-serve            # then open http://127.0.0.1:8000/
$ zig build docs                  # or just build it, into zig-out/docs

It has to be served rather than opened. What Zig emits is not a page but a program: a WebAssembly viewer that fetches the source of everything it shows out of a tar file beside it, and a browser refuses to fetch anything from a file:// page. That is the same reason zig std runs a server rather than opening a file. tools/docs_server.zig is that server — one directory, one person, the loopback interface, and nothing else.

Licence

MIT. See LICENSES/MIT.txt.