Skip to content

Running in production

This page covers what you need to run etemenanki-app and katana unattended on a Linux server: the command line, what each exit status means, how the processes react to signals, where their logs go and how to control them, a hardened systemd unit for each program, and an upgrade procedure you can repeat without surprises.

Both programs share the same command-line shape, the same signal handling and the same logging stack, so most of the page applies to both. Where they differ, a table says so. The katana section of this guide covers panel-specific operation; this page covers only what katana has in common with etemenanki-app.

etemenanki-app katana
Config file -c path, default config.toml in the working directory same
--test success line Configuration OK. on standard output Configuration OK on standard output
--test failure line configuration invalid: …, written through the logger to standard output configuration error: …, written to standard error
Graceful stop SIGINT, SIGTERM SIGINT, SIGTERM
Reload automatic, when the config file changes automatic, when the config file changes
Reload signal none; SIGHUP terminates the process none; SIGHUP terminates the process
Logs standard output standard output
[log].level changed in a running process ignored until the next start applied at once

Both binaries take the same four flags and nothing else. There are no subcommands and no positional arguments.

Flag Argument Default What it does
-c, --config path config.toml The TOML file to load. A relative path is resolved against the process’s working directory, not against the binary’s location.
--test none off Parse the file and build everything it describes, print the result and exit. Binds no port and opens no connection.
-h, --help none Print usage and exit with status 0.
-V, --version none Print the name and version, for example etemenanki-app 2.0.0 or katana 3.0.1, and exit with status 0. The version is the program’s own package version, so a kernel release that changes only a library, such as etemenanki 2.0.1 or 2.0.2, still prints etemenanki-app 2.0.0.

An unknown flag, or -c without a value, is a usage error: the program prints, for example, error: unexpected argument '--bogus' found to standard error and exits with status 2.

--test runs the same code a start runs, up to the point where the program would open sockets.

  • etemenanki-app parses the file, then builds the DNS resolver, every outbound, every balancer, the routing table (loading geodata) and every inbound (reading TLS certificates and keys). A start does exactly the same, then binds the listeners.
  • katana parses the file, builds the outbound pool, then builds each [[node]]’s panel client and routing table, and checks the local settings of Hysteria 2 nodes. It does not contact the panel, so everything the panel supplies, such as a node’s protocol, port and transport, is checked only when the node starts.

Neither program checks, in --test, anything that depends on the running system:

  • that a port is free or that the process may bind it;
  • that the process may create a TUN device;
  • that upstream servers, DNS servers or the panel are reachable.

Those show up only at start. For example, a TUN inbound tagged tun-in with the device name etm0 passes --test as an unprivileged user, then fails to start with failed to start: inbound tun-in bind tun etm0 failed: Operation not permitted (os error 1).

Status Meaning
0 --test passed; -h or -V printed; or the process stopped after SIGINT or SIGTERM.
1 --test failed, or the start failed. etemenanki-app fails to start when the file cannot be read or is invalid, or when any inbound cannot bind. katana fails to start when the file cannot be read or is invalid, when the outbound pool cannot be built, when the file has no [[node]], or when not a single node’s panel client and routing table could be built.
2 Command-line usage error.
killed by SIGHUP The process has no SIGHUP handler, so the kernel terminates it. A shell reports status 129; systemd reports code=killed, status=1/HUP.

At start, etemenanki-app logs the reason as failed to start: …. katana logs one of failed to load config: …, failed to build outbounds: …, config defines no [[node]] entries or no nodes could be started.

Every relative path is resolved against the working directory of the process. That covers the -c path and every path inside the file, such as cert_file, key_file, ca_file, geoip, geosite and rule_list_path. (A Unix socket listen path is always absolute: etemenanki-app treats listen as a path only when it starts with /.) A relative path is not resolved against the directory that holds the config file.

/etc/etemenanki/config.toml (fragment)
[inbound.stream.tls]
cert_file = "tls/fullchain.pem" # read from <working directory>/tls/fullchain.pem
key_file = "tls/privkey.pem"

Checked from /etc/etemenanki, this file passes. Checked from any other directory with -c /etc/etemenanki/config.toml, it fails:

ERROR etemenanki_app: configuration invalid: No such file or directory (os error 2)

The message does not name the file that is missing. When you see it, check -c first, then every relative path in the config.

To avoid the problem, either use absolute paths throughout, or always run the program with the config directory as its working directory. The systemd units below do the second with WorkingDirectory=.

Both programs watch the directory that contains the config file, not the file itself, so that editors which save by writing a new file and renaming it are noticed. Any file event in that directory, including a file being created, written, renamed, deleted or merely opened, makes the program read the config file again. etemenanki-app then compares the bytes with the last version it read, and katana compares the parsed settings, so unrelated files cause no reload. Keep that directory for configuration only, not for logs or other busy files.

The two comparisons differ in one way that matters. etemenanki-app reloads on any byte change, so editing only a comment or whitespace still rebuilds everything and closes every connection; the reload line then reads config reload: no changes. katana applies nothing when the parsed settings are unchanged.

Neither program has a reload signal. A reload happens when the config file changes on disk; see Hot reload.

flowchart TB
  S["start"] --> L["read and parse the config"]
  L -- "error" --> E1["exit 1"]
  L --> B["build outbounds, routing, inbounds"]
  B -- "error" --> E1
  B -->|"--test"| E0["exit 0"]
  B --> N["bind listeners"]
  N -- "error" --> E1
  N --> R["running"]
  R -- "config file changed" --> R
  R -- "SIGINT or SIGTERM" --> D["close listeners and connections"]
  D --> E0
  R -- "SIGHUP" --> K["killed by the kernel"]

The diagram shows etemenanki-app. katana follows the same shape, except that it builds and starts each node separately: a node that fails to build is logged and skipped, and the process exits with 1 only when no node could be built.

Signal etemenanki-app katana
SIGINT (Ctrl-C) Graceful stop, exit 0 Graceful stop, exit 0
SIGTERM Graceful stop, exit 0 Graceful stop, exit 0
SIGHUP Terminates the process at once, no shutting down line same; traffic not yet reported to the panel is lost
SIGKILL Terminates the process at once same; traffic not yet reported to the panel is lost

A graceful stop logs shutting down and then:

  • etemenanki-app stops accepting, closes every listener and closes every open connection at once. It does not wait for connections to finish. A Unix socket listener removes its socket file. A Hysteria 2 inbound closes its QUIC connections and waits up to about 6 seconds for its UDP port to come free; every other inbound stops immediately.
  • katana stops every node: each node closes its listeners and connections, then reports its outstanding traffic to its panel and, on SSPanel, any pending audit results. Each request can take up to the node’s api.timeout, which is 5 seconds when unset, so a stop can take roughly twice that.

After the first SIGINT or SIGTERM, further SIGINT and SIGTERM signals are absorbed. If a stop hangs, only SIGKILL ends it. systemd sends SIGKILL itself once TimeoutStopSec= (90 seconds by default) has passed.

Both programs write their logs to standard output, one line per event. Under systemd, the journal collects them. Each line carries a UTC timestamp, the level, the module that logged it and the message:

2026-09-24T20:23:44.726638Z INFO etemenanki_app::instance: inbound socks-in listening on 127.0.0.1:1080
2026-09-24T20:23:45.424572Z INFO etemenanki_app: shutting down

The lines contain ANSI colour codes even when standard output is not a terminal, which shows up in the journal and in log files as sequences like [2m. Set the environment variable NO_COLOR=1 to turn the colours off. The units below do.

The filter comes from the first of these that is set:

  1. The RUST_LOG environment variable, if it is set and parses. A RUST_LOG that fails to parse is ignored as a whole. An empty RUST_LOG counts as set and enables nothing.
  2. [log].level in the config file. If the file cannot be read or parsed, this step is skipped, so the parse error is still logged at info.
  3. info.

Both take the same directive syntax: a comma-separated list, where a bare level sets the default and target=level overrides it for one module and everything below it. When several directives match, the most specific target wins, whatever its position in the list.

Value Effect
info Startup, listener and reload lines, warnings and errors. The default.
warn Warnings and errors only.
debug Adds a line for every connection that ends with an error, and more. Verbose on a busy server.
info,etemenanki_protocols=debug info everywhere, debug for the protocol implementations.
warn,etemenanki_app::instance=info Warnings everywhere, plus the listener and reload lines.
info,katana=debug katana’s own modules at debug, the kernel at info.
off Nothing at all.

The levels are error, warn, info, debug and trace, plus off. The targets are the crate names with - written as _: etemenanki_app, katana, etemenanki_protocols, etemenanki_environment and etemenanki_concepts, optionally followed by ::module.

config.toml
[log]
level = "info,etemenanki_protocols=debug"
Terminal window
RUST_LOG=debug etemenanki-app -c config.toml # overrides [log].level for this run
etemenanki-app katana
[log].level changed while running Not applied. The level stays what it was at start. Applied at once, without restarting any node.
Side effect of the change Counts as a config change: the reload logs config reload: log changed, restarts every listener and closes every connection. None beyond the new level.
[log].level removed while running Not applied The level becomes info.
New value cannot be parsed Not applied Logs invalid log level "…": … and keeps the current level.
Started with RUST_LOG RUST_LOG stays in force for the life of the process. The first change to [log].level replaces the RUST_LOG filter.

The unit below runs etemenanki-app as an unprivileged user, gives it only the capabilities it needs, and checks the config before every start. It assumes this layout:

  • Directory/usr/local/bin/
    • etemenanki-app
  • Directory/etc/etemenanki/ owner root, group etemenanki, mode 0750
    • config.toml mode 0640
    • Directorytls/
      • fullchain.pem
      • privkey.pem mode 0640
  1. Create a system user with no login shell and no home directory:

    Terminal window
    sudo useradd --system --no-create-home --shell /usr/sbin/nologin etemenanki
  2. Install the binary:

    Terminal window
    sudo install -m 0755 etemenanki-app /usr/local/bin/etemenanki-app
  3. Install the configuration so that root owns it and the service user can only read it. The config holds passwords and keys, so it must not be world-readable:

    Terminal window
    sudo install -d -o root -g etemenanki -m 0750 /etc/etemenanki
    sudo install -o root -g etemenanki -m 0640 config.toml /etc/etemenanki/config.toml

    Private keys under /etc/etemenanki/tls/ need the same root:etemenanki ownership and mode 0640.

  4. Check it as the service user, from the working directory the service will use:

    Terminal window
    sudo -u etemenanki sh -c 'cd /etc/etemenanki && exec /usr/local/bin/etemenanki-app --test -c config.toml'
    Configuration OK.

Save one of these as /etc/systemd/system/etemenanki.service. The second variant is for configs with a TUN inbound: the highlighted lines differ, and it leaves out PrivateDevices=yes.

/etc/systemd/system/etemenanki.service
[Unit]
Description=etemenanki-app proxy
After=network-online.target
Wants=network-online.target
[Service]
Type=exec
User=etemenanki
Group=etemenanki
WorkingDirectory=/etc/etemenanki
Environment=NO_COLOR=1
ExecStartPre=/usr/local/bin/etemenanki-app --test -c /etc/etemenanki/config.toml
ExecStart=/usr/local/bin/etemenanki-app -c /etc/etemenanki/config.toml
Restart=on-failure
RestartSec=5s
LimitNOFILE=1048576
# Bind ports below 1024 without running as root.
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
RestrictNamespaces=yes
LockPersonality=yes
SystemCallArchitectures=native
[Install]
WantedBy=multi-user.target

Then load and start it:

Terminal window
sudo systemctl daemon-reload
sudo systemctl enable --now etemenanki
journalctl -u etemenanki -f

The journal shows one inbound … listening on … line per inbound once the service is up.

Setting Why
WorkingDirectory=/etc/etemenanki Relative paths in the config resolve against it, both for the start and for ExecStartPre=.
ExecStartPre=… --test … Refuses to start on a broken config, and records the error as a failed pre-start step, separate from a bind error. For etemenanki-app this repeats what the start checks anyway. It does not protect a running service: systemctl restart stops the old process before the check runs. Run --test yourself before a restart.
Restart=on-failure Restarts after an exit with status 1 or a crash, but not after systemctl stop, SIGTERM, SIGINT or SIGHUP. When the config is broken, each attempt fails the same way. With RestartSec=5s the default start limit (5 starts within 10 seconds) is never reached, so systemd keeps retrying every 5 seconds until the file is fixed.
LimitNOFILE=1048576 Every client connection holds a descriptor, and usually a second one for its outbound connection. systemd’s default soft limit of 1024 is reached quickly; etemenanki-app then logs accept error, backing off 100ms: Too many open files (os error 24) and retries every 100 ms until descriptors free up. Each TCP or Unix socket inbound holds at most 65,536 live connections; see Limits.
CAP_NET_BIND_SERVICE Needed only for inbound ports below 1024. Without it, the start fails with Permission denied (os error 13). Remove both capability lines if every port is 1024 or above.
CAP_NET_ADMIN Needed only for a TUN inbound, to create the device, assign its addresses and add its routes. Nothing else in etemenanki-app uses it; the WireGuard outbound runs in user space and needs no privileges.
DeviceAllow=/dev/net/tun rw, no PrivateDevices= PrivateDevices=yes hides /dev/net/tun. The TUN variant drops it and allows only that device.
AF_NETLINK The TUN inbound installs its routes over netlink. glibc’s getaddrinfo, which the default system DNS backend calls, also opens netlink sockets to read the host’s addresses.
ProtectSystem=strict The whole file system is read-only for the service. etemenanki-app writes nothing, except Unix socket files for inbounds that listen on a path. For those, add RuntimeDirectory=etemenanki and put the sockets under /run/etemenanki/.
NO_COLOR=1 Keeps colour codes out of the journal.

There is no ExecReload=. systemctl reload etemenanki therefore fails with an error and changes nothing, which is intended: the program reloads by itself when the file changes.

  1. Put the edited file next to the live one, with the same ownership and mode, so the service can read it once it is in place:

    Terminal window
    sudo install -o root -g etemenanki -m 0640 config.toml /etc/etemenanki/config.toml.new
  2. Check it as the service user:

    Terminal window
    sudo -u etemenanki sh -c 'cd /etc/etemenanki && exec /usr/local/bin/etemenanki-app --test -c config.toml.new'
  3. Move it into place. The running service notices the change, logs config reload: …, and switches to the new configuration. A reload closes every open connection:

    Terminal window
    sudo mv /etc/etemenanki/config.toml.new /etc/etemenanki/config.toml

If the new file turns out to be invalid after all, the running service logs reload: parse failed, keeping current config: … or reload: build failed, keeping current config: … and carries on with the old one. The next start, however, will fail. Hot reload describes what a reload rebuilds and what happens when a port cannot be bound.

One katana process can serve several nodes: repeat [[node]] in one file. Separate processes are useful when nodes should restart independently, for example one process per panel. A systemd template unit runs one process per config file:

  • Directory/etc/katana/ owner root, group katana, mode 0750
    • xboard.toml mode 0640
    • sspanel.toml mode 0640
  • Directory/etc/systemd/system/
/etc/systemd/system/katana@.service
[Unit]
Description=katana node agent (%i)
After=network-online.target
Wants=network-online.target
[Service]
Type=exec
User=katana
Group=katana
WorkingDirectory=/etc/katana
Environment=NO_COLOR=1
ExecStartPre=/usr/local/bin/katana --test -c /etc/katana/%i.toml
ExecStart=/usr/local/bin/katana -c /etc/katana/%i.toml
Restart=on-failure
RestartSec=5s
LimitNOFILE=1048576
# Only if a panel assigns a node port below 1024.
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
RestrictNamespaces=yes
LockPersonality=yes
SystemCallArchitectures=native
[Install]
WantedBy=multi-user.target

The instance name after @ picks the file: katana@xboard runs /etc/katana/xboard.toml.

Terminal window
sudo useradd --system --no-create-home --shell /usr/sbin/nologin katana
sudo systemctl daemon-reload
sudo systemctl enable --now katana@xboard katana@sspanel
journalctl -u 'katana@*' -f

What differs from etemenanki-app:

  • --test is stricter than a start. A start skips a node it cannot build; ExecStartPre= refuses the whole file instead, so a typo in one node cannot leave that node silently down.
  • The panel decides the ports. katana binds whatever port the panel assigns to each node, so you may not know in advance whether CAP_NET_BIND_SERVICE is needed. katana needs no other capability and writes no files.
  • Stopping takes a few seconds. Each node reports its last traffic to the panel on the way out. Leave TimeoutStopSec= at its default, or at least a few times api.timeout, so that systemd does not kill the process before the report is sent.
  • Instances share a directory. Each instance watches /etc/katana, so editing one file makes every instance re-read its own. An instance whose file did not change applies nothing.
  • Restart=on-failure covers the process, not each node. A node that was built but cannot come up does not end the process; it retries by itself while the other nodes keep running. Each failed attempt logs one line, such as node 1: node_info failed: …; retrying in 1s (its first panel request failed) or node 1: initial start failed: …; retrying in 1s (for example, its port could not be bound). The wait starts at 1 second and doubles after each failure, up to 60 seconds or the node’s update_periodic, whichever is shorter. Saving an edit to that node’s [[node]] entry retries at once. systemd still sees a running process, so watch the journal for retrying in. Once the cause is fixed, for example the panel answers again or the port is free, the node comes up on its next attempt without a restart. Only a fix to the unit itself, such as a missing capability, needs systemctl restart. The katana section covers how a running node behaves when its panel becomes unreachable later.

The same procedure works for both programs. It checks the new binary against the live configuration before anything is touched, keeps the old binary for a rollback, and swaps the file by renaming it. The commands show etemenanki-app. For katana, use /usr/local/bin/katana, the katana user and /etc/katana, run the check once per instance file (for example -c xboard.toml), and expect Configuration OK without a full stop.

  1. Put the new binary next to the old one under a different name:

    Terminal window
    sudo install -m 0755 etemenanki-app /usr/local/bin/etemenanki-app.new
    /usr/local/bin/etemenanki-app.new --version
  2. Check the live configuration with the new binary, as the service user and from the service’s working directory. A new version may reject a key the old one accepted, and this is where you find out:

    Terminal window
    sudo -u etemenanki sh -c 'cd /etc/etemenanki && exec /usr/local/bin/etemenanki-app.new --test -c config.toml'

    Stop here if it does not print Configuration OK. The running service is untouched.

  3. Keep the current binary for a rollback:

    Terminal window
    sudo cp -p /usr/local/bin/etemenanki-app /usr/local/bin/etemenanki-app.prev
  4. Swap the files by renaming, not by copying over the running file:

    Terminal window
    sudo mv -f /usr/local/bin/etemenanki-app.new /usr/local/bin/etemenanki-app

    Linux refuses to open a running executable for writing, so cp onto it fails with Text file busy. A rename replaces only the directory entry: the running process keeps the old file, and the next start uses the new one.

  5. Restart the service. This closes every open connection; clients reconnect on their own:

    Terminal window
    sudo systemctl restart etemenanki

    For katana instances: sudo systemctl restart 'katana@*'.

  6. Confirm that the service is running and that each inbound is listening again:

    Terminal window
    systemctl status etemenanki
    journalctl -u etemenanki -n 20

To roll back, rename the old binary back into place and restart:

Terminal window
sudo mv -f /usr/local/bin/etemenanki-app.prev /usr/local/bin/etemenanki-app
sudo systemctl restart etemenanki
Symptom Cause Fix
No such file or directory (os error 2) from --test or at start The -c path, or a relative path inside the config, does not exist from the working directory. Check WorkingDirectory=, or use absolute paths.
Permission denied (os error 13) from --test or at start, without the word bind The service user cannot read the config or a file it names. Give the files group etemenanki (or katana) and mode 0640.
failed to start: inbound … bind 0.0.0.0:443 failed: Permission denied (os error 13) Port below 1024 without CAP_NET_BIND_SERVICE. Add the capability, or use a higher port.
node 1: initial start failed: Permission denied (os error 13); retrying in … (katana) The panel assigned a port below 1024 and the unit lacks CAP_NET_BIND_SERVICE. The node keeps retrying the bind, and every attempt fails the same way. Add the capability to the unit and restart the instance. The process cannot gain a capability while it runs.
failed to start: inbound … bind … failed: Address already in use (os error 98) Another process holds the port. --test cannot detect this. Stop the other process, or change the port.
failed to start: inbound … bind tun … failed: Operation not permitted (os error 1) No CAP_NET_ADMIN. Use the TUN variant of the unit.
accept error, backing off 100ms: Too many open files (os error 24) The descriptor limit is too low. Raise LimitNOFILE=.
No log lines at all [log].level is not a level name, or is off, or RUST_LOG is set to off or to an empty string. Set [log].level = "info" and check the unit’s Environment=.
Sequences like [2m and [0m in the journal Colour codes. Set Environment=NO_COLOR=1.
The service stopped and systemd did not restart it It received SIGHUP, or was stopped on purpose. Remove any ExecReload= that sends SIGHUP.
config hot-reload disabled: … (etemenanki-app) or config watcher disabled (no live reload): … (katana) The file watcher could not start, often because the inotify limits are exhausted. The service keeps running without reloads. Raise fs.inotify.max_user_instances or fs.inotify.max_user_watches, then restart.
A changed [log].level has no effect in etemenanki-app The level is read only at start. Restart the service.