Operations

For whoever keeps a NotusBase server running: install it, check it, update it, back it up, restore it and find out why it is slow. This page covers the Linux package with systemd. First server is the first-install walkthrough, so the install steps are not repeated here.

The package carries install.sh, update.sh, doctor.sh, run.sh, notusbase.service, notusbase.env.example, Caddyfile.example, alerts.yml and alerts_test.yml next to the binary, plus this guide in guide/ and CHANGES.md, which says what each version changes. Every command and every environment variable is in the reference.

Where things are

The service runs as the system user notusbase, not as root. The file system is read-only everywhere except /var/lib/notusbase, and the open-files limit is 1,048,576. The unit restarts after a crash and gives up after 10 crashes in 5 minutes, so the alert can call someone.

The unit is hardened with systemd options (no new privileges, private /tmp, no kernel tunables, a system-call filter). MemoryDenyWriteExecute is left out on purpose, because V8 generates code at run time. For the same reason the filter allows pkey_*; without it the process dies with SIGSYS on startup.

PB_WAL_SYNC_MS is 1000 by default: the WAL goes to disk every 1 s, and startup logs the value. With 0 the WAL only reaches the disk at the checkpoint, and the promise of losing at most about 1 s does not hold.

After editing notusbase.env and starting the service, run the check.

Starting and stopping

sudo systemctl start notusbase
sudo systemctl stop notusbase
journalctl -u notusbase -f
curl -s http://127.0.0.1:8090/api/health

The package's notusbase.env.example sets two ports. NOTUSBASE_HTTP serves your systems and sits behind the proxy. NOTUSBASE_ADMIN_HTTP serves the dashboard, the deploy and the metrics, and stays on loopback. Without the second variable a single port serves everything, which does not work behind a proxy: with --behind-proxy the server does not start, and without it every request that came through the proxy gets 403.

Stopping sends SIGTERM. The server finishes copying the WAL to the backup before it exits, within 90 s, and then systemd kills the process. A kill loses no confirmed write: the database recovers from the WAL on startup.

Two keys, both derived from .notusbase_secret:

sudo -u notusbase /opt/notusbase/current/notusbase deploy-key --dir /var/lib/notusbase/pb_data
sudo -u notusbase /opt/notusbase/current/notusbase metrics-key --dir /var/lib/notusbase/pb_data

The deploy key opens everything on the admin port. The metrics key opens only the metrics route. Treat both as server passwords.

Commands with the service's environment

A command that reads the bucket or the setup, such as backup-s3-status, restore-s3 or superuser upsert, needs what the server has: notusbase.env, which only root and systemd read, and setup.env, where the setup page saved the S3, which the binary reads itself. sudo -u notusbase gives it neither. run.sh gives it both:

sudo /opt/notusbase/run.sh backup-s3-status
sudo /opt/notusbase/run.sh superuser upsert [email protected] --dir /var/lib/notusbase/pb_data

It runs current/notusbase with the arguments through systemd-run, with the user, group, folder, umask, environment files and Environment= of the notusbase unit, so a hand-edited unit is followed too. In a terminal the command can ask for input; in a script, input and output pass straight through. The exit code is the command's. With another unit name or a user unit, NOTUSBASE_UNIT and NOTUSBASE_SYSTEMCTL="systemctl --user", as for update.sh.

Checking the machine

Run it after installing, after every update and whenever something looks odd:

sudo /opt/notusbase/doctor.sh

install.sh puts it in /opt/notusbase, and update.sh replaces it with each version's. It only reads. The value of a secret variable never appears: only whether it is set.

The last line is All good, with N warning(s). or N problem(s) and M warning(s). The exit code is 1 if there is any ✗, and 0 otherwise. A wrong argument exits with 2.

The fsync measure writes 200 synchronous 4 KB blocks next to the database. It is a hint, not proof. Under 0.05 ms per write, the disk probably answers before it writes, and the promise of losing at most 1 s in a power failure depends on its cache having a battery. Over 20 ms, commits will be slow. The only real proof is cutting the power in the middle of a write.

What it does not check: the port as seen from outside the network (the script looks at the listen address; test from outside with curl, expecting 404 or no connection on the admin routes), whether an alert reaches a person, and the standby.

No S3 on purpose. Without a bucket, doctor fails, because losing the machine loses the backup too. To go without S3 deliberately, write NOTUSBASE_S3=off in notusbase.env. Doctor then warns and does not fail. The server does not read that line.

Non-default setups, the same as in update.sh: NOTUSBASE_CONF (path of the notusbase.env), NOTUSBASE_UNIT (unit name) and NOTUSBASE_SYSTEMCTL (for example systemctl --user).

Updating

The binary. Clients drop for up to about 1.5 s and reconnect by themselves. A mutation in flight is resent by the client and is not applied twice.

tar xzf notusbase-VERSION-linux-x86_64.tar.gz
sudo /opt/notusbase/update.sh "$PWD/notusbase-VERSION-linux-x86_64"

update.sh refuses, without touching anything, when:

• the version is already running (is already running);
• the new binary does not even open;
• the server answers but its metrics do not. With NOTUSBASE_ADMIN_HOSTS set, the list needs localhost;
• the backup is on and late (backup behind by N s), or on with no chain in sight (backup on with no chain at all), as with a folder without permission. The message ends with fix it before updating, or pass NOTUSBASE_FORCE=1.

NOTUSBASE_FORCE=1 skips the backup refusal. It goes after sudo, because sudo clears the environment:

sudo NOTUSBASE_FORCE=1 /opt/notusbase/update.sh "$PWD/notusbase-VERSION-linux-x86_64"

That works with an ALL rule in sudoers. With a rule only for the script, sudo refuses the variable unless the rule has the SETENV tag.

Then the script copies the binary into releases/, swaps the current link, restarts the service, confirms the version through notusbase_build_info and watches the process for 15 s: no restart and /api/health answering. It also replaces doctor.sh, run.sh, the variable list, the guide and CHANGES.md, and ends with went from <old> to <new>. What changes for you: /opt/notusbase/current/CHANGES.md.

Automatic rollback. If the new version does not stay up, the script prints <new> did not stay up: going back to <old>, goes back and exits with an error. If the old one does not come up either, it says the service is down and prints the instant to use with restore --until, taken from before the attempt.

Going back by hand:

sudo /opt/notusbase/update.sh --rollback

It follows the same rule: if the previous version does not stay up, the script returns to the one that was running.

A version from before the setup page does not read setup.env: while it runs, what the page saved there (S3, login collection, signup, origins) is not set. The script says which names before the swap, as a warning. To keep them, copy those lines to notusbase.env for as long as the old version runs.

When going back is safe. Only if the previous binary accepts the database the new one left. PocketBase and NotusBase change the database on startup, and an old binary does not undo that.

• New columns and new tables: safe. The old version ignores them. One exception: notusbase copy --mask of a binary older than the table refuses a database with a table it does not know, such as _fleet (from the version with Fleet). Run the copy with the new binary, or take every server out of Fleet and drop the empty table first.
• A change of layout, like a renamed internal table: no way back with data. The old binary does not undo it, and may create its own tables beside the new ones, and then the new one stops starting too. Go back with restore --until, using an instant from before the update.

CHANGES.md says, under each version, whether it changes the database. Read it before updating: it is in the new package, and after the update in /opt/notusbase/current/CHANGES.md.

Before updating to a version that migrates, run a backup check (below). To go back after a migration, restore.

The unit and the other scripts. update.sh swaps the binary, doctor.sh, run.sh and the guide, and nothing else. When CHANGES.md says the unit or update.sh changed, run sudo ./install.sh from the new package before update.sh: it rewrites both and does not switch the running version. In notusbase.env it only adds a line the new binary needs and the file lacks, such as NOTUSBASE_DIR or NOTUSBASE_ENV, with a comment saying so. A hand-edited unit loses its edits.

Environments, on a server from before them. A binary with environments started without NOTUSBASE_ENV runs as development: email captured, fetch locked. So the first update to it, whichever script runs first, adds NOTUSBASE_ENV=production to a notusbase.env that has no such line, and says added NOTUSBASE_ENV=production. A notusbase.env that already names an environment is left as it is. A server started some other way, by hand or with Docker, needs the line or --env production before the new binary starts. If it was forgotten, the log says WARNING: started without --env at every start, and doctor.sh warns. See Environments.

Contract mismatch. The published bundle carries the runtime. If the new binary needs another contract, the log says bundle with contract X, and this binary requires Y: scheduled jobs stay pending until you deploy again. Crons and scheduled functions wait, with no loss, until you deploy again. update.sh warns at the end, notusbase_runtime_contract_mismatch goes to 1, and the NotusBaseRuntimeContractMismatch alert fires.

App code needs no restart and drops no client:

NOTUSBASE_DEPLOY_KEY=... notusbase deploy --url http://localhost:8091

Run it from a project, through the tunnel in Dashboard and second factor. A mutation caught in the swap runs again on the new code. There is no deploy history: to go back, publish again from the previous commit.

Backup

NOTUSBASE_BACKUP turns on a continuous backup: the log is the WAL itself, copied in chains, with the files in storage alongside. It lets you go back in time. It does not protect against losing the disk, which is what S3 is for. The server keeps only the two newest chains.

PocketBase's own backup is off: /api/backups answers 410, the dashboard's backups screen says so, and a cron set there fails on every run. Use NOTUSBASE_BACKUP and the commands on this page.

Checking the backup

cd / && sudo -u notusbase /opt/notusbase/current/notusbase backup-check --dir /var/lib/notusbase/pb_data --url http://localhost:8091

It restores the backup beside the database and compares it with the live one, row by row. It exits with an error if the check did not pass or gave up half way. backup-check --last only shows the last result.

Use localhost, not 127.0.0.1. The check and the deploy go through the admin port, and NOTUSBASE_ADMIN_HOSTS lists the names it accepts: localhost passes when the list has it, and 127.0.0.1 gets a 421 if the list does not.

The server also checks by itself every NOTUSBASE_BACKUP_CHECK_HOURS (24 in notusbase.env.example; 0 turns it off). The check copies the whole database, so it needs free space equal to the database size. NOTUSBASE_BACKUP_CHECK_DIR moves the copy to another disk, and NOTUSBASE_BACKUP_CHECK_MBS sets its pace.

A partial check (notusbase_backup_check_partial) is not a failure. It gave up half way to protect the WAL, and the next one resumes where it stopped. If it keeps happening, the database is too big for the write rate at that hour: use NOTUSBASE_BACKUP_CHECK_DIR or adjust NOTUSBASE_BACKUP_CHECK_WAL_MB.

A check that fails is an incident if the data differ. Do not delete any chain. With no space left it fails too.

Proving the backup restores

cd / && sudo -u notusbase /opt/notusbase/current/notusbase restore-check --dir /var/lib/notusbase/pb_data --url http://localhost:8091

backup-check rebuilds the chain by its own steps. restore-check runs the restore itself, end to end, the way you would after losing the server: restore of the local chain in use and, with S3, restore-s3 downloading for real, each into a new folder; the same comparison with the live database; the _storage files by size and SHA-256; and the copy started in isolated mode until it answers. The folder is deleted at the end, pass or fail. It exits with an error if the check did not pass or ended without a verdict; restore-check --last shows the last attempt and when one last passed.

The server runs it by itself once a week (NOTUSBASE_RESTORE_CHECK_HOURS, 0 turns the schedule off), at low priority, and the Backup block of Home → Technical details shows how long ago one passed, with a Run now button. It needs free disk for a copy of the database, twice that with S3, and leaves the disk at least 15% free; without it, it does not run and says so. A run gives up after 6 hours without a verdict, and so does a bucket or a copy that does not answer: only a wrong restore fails it. A failure raises NotusBaseRestoreCheckFailed and goes out by webhook and e-mail; no pass in two intervals and a day raises NotusBaseRestoreCheckStale.

Backup to S3

With NOTUSBASE_BACKUP and NOTUSBASE_S3_BUCKET set, a separate loop uploads the base, the closed segments and the storage folder. A slow or dead S3 only delays the remote copy: it never touches the write path, and the log says backup: S3 DOWN: ....

A round runs every 30 seconds, the open segment closes after 15 minutes at most, and the bucket keeps the last 2 chains. These three are fixed. A chain is named by the time it was born, and the names only grow: with the clock behind the newest chain on the disk or in the bucket, the new one is named a millisecond after it, and the log says the clock ... is behind the newest chain. The bucket never deletes a chain still on the local disk, the one being written among them.

Generate the key with openssl rand -hex 32. Everything is encrypted with AES-256-GCM before it leaves the machine, so losing NOTUSBASE_S3_KEY loses the bucket copy, and changing it loses access to what is already uploaded. Keep the key outside the machine and outside the bucket.

notusbase backup-s3-status
notusbase restore-s3 /new/folder --until 2026-11-06T12:00:00Z

Both commands read the same NOTUSBASE_S3_* variables and need no running server. backup-s3-status shows what can be restored from the bucket. restore-s3 downloads the chain beside the target (<folder>.s3.tmp), so it needs twice the space while it runs. On a new machine or after losing the disk, install the package, set the variables and run it.

The most you can lose is about 15 minutes plus one round: the open segment does not upload until it closes.

Limits worth knowing:

• Whoever controls the bucket can delete objects or serve an old manifest.json, and then you restore something old, valid and silent. Versioning or object lock on the bucket reduces the risk. This was not tested with NotusBase.
• Object names and sizes are visible, file ids included.
• auxiliary.db is not uploaded. .notusbase_secret and the bundle are, encrypted, and come back on restore. The secret derives the deploy key, so NOTUSBASE_S3_KEY opens more than the data.
• A file deleted from storage is not deleted from the bucket.
• A database of about 58 GiB or more does not upload (the part limit of 10,000 at 6 MiB), and the log only shows S3 DOWN.
• If the newest chain's manifest does not open, restore-s3 stops with an error instead of trying the previous one.
• It ran against a real provider once, with a small volume.

The metrics are notusbase_backup_s3_lag_seconds, notusbase_backup_s3_failing, notusbase_backup_s3_failures_total, notusbase_backup_s3_last_upload_timestamp_seconds, notusbase_backup_s3_last_success_timestamp_seconds and rate(notusbase_backup_s3_sent_bytes_total[5m]). They exist only with S3 configured. The lag rises until the segment closes (15 minutes at most) without there being a fault.

Backup stopped

When the backup folder refuses a write, the log shows backup: new chain: ... and backup: base: .... Writes to the database go on as normal, and the server tries a new base every minute without restarting.

While there is no chain there is no checkpoint, and the WAL grows: at 300 mutations/s, about 10 GB an hour. At 8 GiB the checkpoint returns without a copy so the data disk does not fill. The log then shows backup: WAL at N MB without a chain: checkpoint without copy until the next base, and notusbase_backup_wal_released goes to 1. From there:

• Mutations go on, with short stops each time the WAL wraps. With the WAL above 2 GiB, the write brake turns on while the pass catches up with the writer.
• The ceiling is where the checkpoint starts, not a hard limit: with no load limit the WAL passes it before the pass catches up. Leave a few GiB of room on the data disk.
• What entered the database after the old chain is only backed up at the next base. Losing the data disk before then loses it, ceiling or not.
• The 8 GiB is fixed (item 203).

What to do:

1. See the error with journalctl -u notusbase | grep backup: and fix the cause: permission, full disk or unmounted disk. Check notusbase_wal_bytes, notusbase_backup_wal_released and notusbase_disk_free_bytes{dir="data"}.
2. Wait. Within 1 minute the next attempt passes, and the base is copied at 64 MB/s. Then the new chain rereads the whole WAL (14 GB took 1 minute on a USB SSD).
3. Confirm the log line backup: chain <name>. notusbase_backup_lag_seconds drops to near 0, the WAL goes back under 256 MiB, and the Backup is behind warning leaves the status screen within 2 minutes. notusbase_backup_wal_released returns to 0.

Do not restart. It does nothing until the cause is gone. After a restart the old chain only continues if the WAL still follows its last segment and no checkpoint ran without a copy. If one did, the file backup.released, next to the database in pb_data, makes startup begin another chain, logging backup: new chain: there was a checkpoint without a copy. Do not delete that file by hand: it goes away with the new chain, and without it startup could join a chain with a hole in it.

The old chain stays on disk and ends where the copy stopped. restore --until with an instant inside the gap returns the end of that chain. If a base is refused at the end, the next attempt deletes what is left of it. An old chain that cannot be deleted stays in the folder with the log line backup: old chain stays: ..., and the new chain is valid anyway.

The NotusBaseBackupStalled alert watches notusbase_backup_lag_seconds. It stays at 0 when the database is idle, however old the last copy, and counts the time since there has been a write without a copy.

Restoring

On the same machine, with the database lost or corrupt and the rest of pb_data intact:

# In a root shell (sudo -i): /var/lib/notusbase opens only for root and for the notusbase user.
systemctl stop notusbase
cd /var/lib/notusbase
sudo -u notusbase /opt/notusbase/current/notusbase restore backup pb_data.new
# For an exact instant: restore --until 2026-11-06T12:00:00Z backup pb_data.new
for f in .notusbase_secret .notusbase_bundle.js auxiliary.db notusbase.log notusbase.log.1; do [ ! -e pb_data/$f ] || sudo -u notusbase cp -a pb_data/$f pb_data.new/; done
sudo -u notusbase mv pb_data pb_data.old && sudo -u notusbase mv pb_data.new pb_data
systemctl start notusbase

restore rebuilds data.db and gives back the storage folder with the files the chain holds, including ones deleted after the instant you ask for. The loop copies the rest, which has no local backup. restore also brings the published code and .notusbase_secret, the secret the deploy key comes from. Into a folder that has no secret, it brings the deploy key of the server the backup came from, with a warning: right for the same server coming back, wrong for a copy, since a production backup restored into staging would carry the production key. For a copy, --new-secret gives the folder a key of its own and leaves the backup's out; notusbase copy restores this way. --with-secret matters only when the folder already has a secret: without it that secret stays, with it the one in the backup replaces it. restore-s3 always overwrites the secret, unless --new-secret asks for a new one. A backup with no secret leaves a new deploy key to be born, and with no bundle you must deploy again.

A chain from before the files were copied prints the chain has no copy of the files: it is from before the local-backup copy. In that case copy pb_data/storage too, or bring the files from S3. A system that never received a file prints the same line, and there is nothing to copy.

On startup the server begins a new chain (backup: chain <name> in the log), because the restored database does not continue the old one. Then run backup-check, and only after that delete pb_data.old.

The code of --until. The backup keeps only the newest published code, so --until brings the data of that instant with the code of the last deploy, and says so with a warning that names the code's date: if the code changed since, deploy the commit of that time, or start the copy with --isolated first.

Precision of --until. The open segment keeps a point for every copy, 200 ms apart. A point is stamped right before its copy reads the WAL, so it holds every write confirmed before its instant: --until brings back all that was confirmed up to the newest point at or before the instant you ask for. It may also hold writes confirmed a few milliseconds after that instant (the time between the stamp and the read, longer under heavy load). To undo a mistake, pick an instant with a second of margin before it. A closed segment is compacted: only the last version of each page per 60 s window is kept, gzipped. An instant inside a compacted segment returns to the end of the previous window, up to 1 minute before the one asked for. An instant in the first window of a chain returns to the end of the previous chain, or fails with nothing in the backup before <instant> if it is the oldest. With S3, the bucket keeps the raw segment with the 200 ms points: local compaction only starts after the upload.

The server keeps only the 2 newest chains. The chain you restored from disappears at the second new chain, which may be the next startup. To keep it, copy the backup folder before starting. A chain from a server that started and stopped without writing has only the base, and is good for the instant it was born.

For a new machine or a lost disk, install the package and use notusbase restore-s3 (Backup to S3).

Copies of production

notusbase copy development|staging|isolated, run in the project folder, restores production's backup (--backup, the folder of the chains, or --s3) into a new folder in one step and leaves it fit for one use. Before it writes anything, it states the space it needs and stops if the disk does not have it. It refuses production's outlets in the environment of whoever runs it (NOTUSBASE_ENV=production, SMTP, webhooks, alerts), and the copy starts with a clean environment.

# develop with production's data masked, from the bucket (NOTUSBASE_S3_* set)
notusbase copy development --s3
# the function that failed, with what the customer sent, as that customer, on a masked copy
notusbase copy isolated --mask --function tasks:create --args @args.json --as k17abc
# rehearse the deploy of this folder on a copy of the backup, deleted at the end
notusbase copy isolated --rehearse --backup /var/backups/notusbase
# the homologation on the production server, renewed over the old one
notusbase copy staging --backup /var/backups/notusbase --folder /srv/notusbase-staging --replace

The mark. The copy gets a .notusbase_copy file saying its use and whether it is masked. serve --env production refuses a folder with it, so a copy never becomes production by mistake; restoring production goes through notusbase restore. A copy with real data also refuses development (notusbase dev, or a start with no environment), where the code under work runs with fetch open, and a staging copy is the only real-data copy that starts as staging; --isolated starts any of them. A masked copy starts in any environment but production.

The mask. Emails, phones, names, CPF and CNPJ are replaced by the field's name (email, phone, name, cpf, cnpj…), by the PocketBase email type and by the mask key of notusbase.json. The fakes reach nobody: addresses end in @example.test, phones start with 10, and a CPF or CNPJ gets valid check digits so that the code that checks them still works. The same value always gets the same fake in one run, so joins and searches still match. Then a sweep reads every other text (free text, JSON leaves, job arguments, component tables) once, for email addresses, phones written as phones, CPF and CNPJ with their dots, and the values it already replaced in a field (a value longer than 12 words is found by its first 12 and then compared whole, so a long text marked for the mask does not slow the sweep), and swaps them for the same fakes. Ids, dates, amounts and timestamps do not change. Credentials, sessions, _env and the saved results of _requests_* are emptied: those results are sealed with a session key of production, so a copy could not read them anyway. The app's own credentials are emptied too: every row of the Convex Auth tables, and any field, at the top or inside JSON, whose name ends in token, secret, password or refresh. A name that ends in code or hash is emptied only when it is just that word or comes after a sign-in word (verificationCode, resetCode, otpCode, passwordHash, pinCode), so statusCode, currencyCode, zipCode or fileHash keep their values. A key is a credential after a word that says whose secret it is (privateKey, apiKey, signingKeyPem, accessKey), while primaryKey, sortKey, publicKey and keyCode stay. The signing keys of a component's auth (the jwks table of Better Auth) are emptied whole: the component draws new ones. A credential emptied in its field also goes from every other text of the copy (a log line, a JSON note, another table): one of 12 characters or more wherever it appears, a shorter code of 6 or more only as a word of its own. Another field that holds a credential goes in the mask key as "secret"; a field that holds data but whose name says otherwise goes in "ignore". The vectors of a vector index are replaced by random ones, because a vector is derived from the real text and some of that text can be read back from it; the search on the copy then finds nothing meaningful. Storage files do not come along, and a file field is emptied, since a file's name often carries the person (laudo_ana_silva.pdf); "ignore" keeps one. A table the mask does not know stops the copy, by name.

The output says three things apart: what was replaced by field name, what the sweep replaced, and what it cannot see: a name typed in free text that no name field holds stays. That last list names the text fields and, inside JSON fields and job arguments, the path of each free-text value (profile.visits[].room; a key that is data itself shows as *). Mark such a field as "text" in notusbase.json to replace it whole:

{ "mask": { "fields": { "tickets.notes": "text", "orders.taxId": "document" }, "ignore": ["products.name"] } }

Isolated. --function module:function runs the function after the start, with --args (JSON, @file or -) and --as (the id of the account, the same as in production); --exit ends there instead of leaving the server up. The copy deploys the code of this folder (--no-deploy keeps the backup's). Without --mask, it loses exactly what staging loses (below): the production rows of _env, the client certificates, the file and backup S3, SMTP and the OAuth2 client secrets, and the token secrets are drawn again. A copy kept with --keep may later start outside isolated mode, so no production key stays on it.

Rehearsing a deploy. Before a deploy that changes the schema, or once a week, --rehearse runs deploy --plan and the deploy with this folder's code against the copy, then the project's smoke test, prints a report and deletes the copy, also when something fails. A deploy that reads a whole table to validate, or a wrong index, shows up here and not in production, and the restore is rehearsed too. On the server, which has no Node for tsc, add --skip-typecheck; the typecheck stays on your machine, where notusbase deploy runs it. The project folder is the one from your machine after npm install, with convex/ and node_modules/: without the convex package the deploy stops at the convex package is not installed. With --s3 it runs on a laptop without touching the server.

The report comes at the end: the plan, the deploy and the smoke test with the time of each, and whether the copy's log had a panic. When it fails, the tail of the copy's log comes first. The last line is result: passed or result: failed. The chain it restored from does not change: the copy makes no backup.

Staging. The homologation goes without the production rows of _env, the client certificates (A1), the file and backup S3, SMTP and the OAuth2 client secrets, and gets new token secrets, so a token issued on one server is not valid on the other. The shared rows of _env stay, and the output lists them by name: check that none of them is a production key. OAuth2 login needs a test client of its own on the staging server. --replace builds the new copy beside the old one and swaps them only when it is ready, so a failed renewal leaves the old one; it refuses a folder that is not a copy, and a copy that is running. The gaps of staging are in Environments.

Smoke test. If the project folder has a rehearse.mjs, --rehearse runs it against the copy with NOTUSBASE, NOTUSBASE_REHEARSAL_URL and NOTUSBASE_REHEARSAL_DATA in the environment. Use serverAt from the project's kit.mjs: setup would start another, empty server. In the copy nothing scheduled runs, so waitJobs and runCron wait until their deadline and fail.

import assert from "node:assert/strict";
import { serverAt } from "./kit.mjs";

const s = serverAt({ url: process.env.NOTUSBASE_REHEARSAL_URL, dataDir: process.env.NOTUSBASE_REHEARSAL_DATA });
assert.ok(Array.isArray(await s.run("tasks:listAll", {}))); // one read for each important screen
console.log("smoke ok");

Isolated mode

notusbase serve --isolated, or NOTUSBASE_ISOLATED=1, starts a copy that has no effect outside the machine. The restored _env carries the real keys, and a cron that woke up in the copy would charge a customer or send an email. So the cut lives in the binary, not in the script. notusbase copy isolated always uses it, and so should any import of a copy of a live system.

The startup log says isolated: no crons, no jobs, no network, no _env, no backup and no alerts (--isolated). notusbase copy stops if the line does not appear, because a binary without the mode would start as production.

What still works: queries, mutations, actions without network, the dashboard, the deploy, password login and writes to the copy's database.

Environments

Each server is one environment: production, staging or development. It is named with --env or NOTUSBASE_ENV; a name that is not one of the three stops the start.

Forgetting the name falls on the safe side: the server runs as development, and the log says WARNING: started without --env, so as development at every start. A server that already has variables in _env or SMTP on does not start without the name, outside notusbase dev: it is most likely a production server swapped by hand, and as development it would hold the superuser's OTP. A name that is not one of the three stops every command, not only serve. The first install.sh refuses without --env (or NOTUSBASE_ENV) and writes it to notusbase.env; on a server from before environments, the first install.sh or update.sh of this version adds NOTUSBASE_ENV=production (Updating). The metric notusbase_environment carries the name, whether it was stated and whether the exit is locked, and doctor.sh warns about anything but production, stated.

This is not isolated mode, which stays the most closed: no crons, no jobs, no _env and no network at all. A staging server works as a server; it only does not reach customers through email and fetch. What the environment does not lock:

• Backup. NOTUSBASE_BACKUP and NOTUSBASE_S3_* work in every environment. A copy started with the production notusbase.env writes its backup to the production folder or bucket, beside the production chains. Give a copy its own notusbase.env, with its own backup destination or none.
• Alerts by webhook (NOTUSBASE_ALERT_*) go out; alerts by email are captured like any email.
• OAuth2 login and the JWKS of auth.config.ts reach their providers. A copy made with notusbase copy staging has no OAuth2 client secret, so OAuth2 login there needs a test client of its own; the JWKS of auth.config.ts keeps working.
• Files in S3. The file storage setting lives in the database (dashboard, Settings, Files storage), so a copy of production restored for staging keeps pointing at the production bucket: it reads the files there, and deleting a record on the copy deletes the file in production. Switching the setting off, or to a bucket of its own, is the first thing to do on a copy: until then, a cron or a call that deletes a record with a file deletes the file in production. notusbase copy staging switches it off. Isolated mode ignores the file S3 while it runs and leaves the stored setting as it is. Files kept on the local disk are not affected: restore gives the copy its own.
• Login tokens. The secrets that sign login tokens are in the database too, and --new-secret replaces only the deploy key. A token issued by the copy is valid on production for the same user, and the other way round. notusbase copy staging draws them again; a copy made by hand with restore does not, so do not expose it with real users to people who should not log in to production.

For a copy that must have no effect outside the machine at all, use isolated mode.

Variables per environment. Each _env row belongs to one environment, or to all of them. notusbase env set writes to the environment of the server it talks to, and --env picks another one, or all for a shared row. When a name has both, the row of the server's environment wins.

notusbase env set --env all APP_NAME Shop               # every environment
notusbase env set --env staging STRIPE_KEY < test.txt   # staging only
notusbase env list                                       # name and environment of each row
notusbase env list --env production
notusbase env remove --env all APP_NAME

A server keeps the rows of every environment, so one database can be copied from production to staging and the staging server sees only its own keys and the shared ones. The dashboard's variable page shows the environment of each row and hides the values of the other environments. The dashboard header shows the environment next to the superuser, in production too, with the whole warning in its hint, and a click opens the variable page. The color says how careful to be: production red (the data and the emails are real), staging amber, development green, and an isolated copy grey.

Rows from before environments are shared: they keep applying to every environment until you separate them, by setting the same name with --env production and --env staging and then removing the shared one with --env all.

Metrics and alerts

GET /api/notusbase/metrics, in the Prometheus format, exists only on the admin port. Scrape it on the machine itself, through a tunnel or through the dashboard's domain. Use the key from notusbase metrics-key: it starts with notusbasem_, opens only this route and the Fleet summary (fleet/health), and is derived from .notusbase_secret, so there is no revoking one without changing the secret, and then the deploy key changes too. The deploy key also opens the metrics, but it opens the whole server and must not sit in Prometheus.

scrape_configs:
  - job_name: notusbase
    metrics_path: /api/notusbase/metrics
    authorization: { credentials_file: /etc/prometheus/notusbase.key }
    static_configs: [{ targets: ["localhost:8091"] }]

notusbase_http_refused_total{reason="login"} rising without stopping is someone trying passwords. The IP is not stored: look at the proxy log.

Alert rules are in alerts.yml in the package, for Prometheus, and each alert has a test in alerts_test.yml that makes it fire. Each alert's runbook_url annotation is its section in Runbooks, the step-by-step fix. It is a path on the server (/opt/notusbase/current/guide/07-runbooks.md#<section>), not a web link: open it there, or the same section on the dashboard at /_/#/notusbase/runbooks/<section>; the tables below only say where to start. After changing a rule, check both with promtool (it comes with Prometheus):

promtool check rules alerts.yml
promtool test rules alerts_test.yml

These wake someone (severity: page):

These only become tasks (severity: ticket):

Alerts without Prometheus

The binary itself sends the red warnings of the dashboard's status screen to a webhook and, with SMTP set in the dashboard, by email. Prometheus and Alertmanager stay optional. With neither NOTUSBASE_ALERT_URL nor NOTUSBASE_ALERT_EMAIL, nothing goes out.

What it warns about. The reds of the screen, with the same limits and waits as the tables above: backup late or without a chain, S3 upload failing or late, data or backup disk under 15% free, a writer that lost a batch, the brake, a growing WAL, stalled reactivity, the runtime contract and a failed backup check. Plus these, which are not red:

Their metrics are notusbase_unclean_start, notusbase_scheduled_failing_functions and notusbase_error_groups_new_total. The failure count and the error groups live in memory: groups reset when the server restarts, and an error that keeps happening is announced again once. A deploy does not restart the process and keeps them. In Prometheus they are NotusBaseUncleanRestart, NotusBaseScheduledFailing and NotusBaseNewError.

How it arrives. One message gathers everything that changed under NEW, STILL and RESOLVED, each with the title, the action, the runbook (Runbook: guide/07-runbooks.md#..., except under RESOLVED) and since when. At least 5 minutes pass between two messages, so a problem that flaps does not send one a minute. What stays open repeats every hour. An event warning, like the restart, leaves the screen by itself and sends no RESOLVED. If nothing gets out (webhook down, SMTP off), the log shows warning: not sent, retrying every round: ... once and the server tries again every minute. One working channel is enough.

No secrets. The message carries only the server name and the fixed texts of the rules: no variable, key, address or data. The webhook URL may hold a token (Telegram), so it appears neither in the log nor in a connection error.

Limits. The process sends the warning: if it hangs or is down, it says nothing. The standby covers that (see Standby). Restarting resets the interval and the repeat, so a problem still open warns again after the start. A server that crashes in series before 1 minute up does not manage to send the restart warning; the standby sees it if it stays down.

The server appears in the subject and the message by the application name set in the dashboard (Settings, Application name) and the machine, as in agenda on server1. While the name is still the one the server gives itself (NotusBase), it is the machine and the HTTP port instead, as in server1:8090. Name each server in the dashboard when several share one machine. A rename shows in the next message, without a restart; the standby, which has no dashboard, uses the machine's name. Two messages are at least 5 minutes apart, and what stays open repeats every hour. The standby probes the primary every 10 seconds, and down means 6 probes in a row without an answer. These are fixed. An email the SMTP server has not taken within 10 seconds, from the connection to the end, counts as failed: the connection is closed, and the next message goes out normally.

Example with ntfy, in notusbase.env:

NOTUSBASE_ALERT_URL=https://ntfy.sh/<long-random-topic>

Anyone who knows a public ntfy topic can read it: use a long random name, or your own ntfy server. The message holds no data, but it says the server exists and what failed.

Dashboard and second factor

The dashboard (/_/) lives on the admin port. Whether it is on the internet is a choice of each install. In the package, notusbase.env.example binds the admin port to 127.0.0.1:8091, which keeps it off the internet; you reach it through a tunnel.

The risk. A superuser's password does not open only a screen. It publishes code, reads and changes the project's variables, runs SQL and exports the whole database. Whoever guesses it has the server. So, if the dashboard is on the internet:

• Use a long, unique password kept in a password manager.
• Turn on the second factor, below. Without it, notusbase_superusers_password_only stays above 0.
• List the dashboard's domain in NOTUSBASE_ADMIN_HOSTS. An unknown host does not even reach the login.

The login limit (2 attempts every 3 seconds, fixed) only stops brute force from one address. At that rate one IP still makes about 57,000 attempts a day, and a network of machines goes around it. What protects the dashboard on the internet is the long password and the second factor.

Second factor (OTP and MFA). It is optional and native to PocketBase. It needs SMTP, because the code arrives by email.

1. In the dashboard: Settings, Email server (SMTP). With the email module in use, it is reached from Settings, Email, in the section of that name. Configure the SMTP and the sender.
2. Open Settings, Superusers, and click Email code (OTP) (or the link in the password-only notice on Home → Overview): it opens the Options of _superusers on OTP. Turn on OTP and check that MFA is on.
3. From then on a password alone does not get in: login also asks for the 8-digit code, valid for 3 minutes. This applies to every superuser, because _superusers is a system collection and takes no rule about who needs it.

With email down, whoever has access to the machine can get a code right away:

sudo -u notusbase /opt/notusbase/current/notusbase superuser otp [email protected] --dir /var/lib/notusbase/pb_data

The metrics tell you where you stand:

The deploy key and the metrics key do not go through a password or a second factor. Treat both as the server's password.

Dashboard off the internet: the SSH tunnel. With the admin port on loopback, as in the package:

ssh -L 8091:127.0.0.1:8091 server

Then http://localhost:8091/_/, /api/notusbase/metrics and notusbase deploy --url http://localhost:8091 work from your machine. Use localhost, not 127.0.0.1: if NOTUSBASE_ADMIN_HOSTS is set, it must include localhost, and any other name gets 421. To serve the dashboard under a name of its own instead, see The dashboard under a name. Doctor warns when it does not, and update.sh stops before touching anything.

Home. In the dashboard at /_/#/notusbase/home, the first tab and where the login lands, with a superuser login (through the tunnel as well). The sidebar has Overview, Limits, Servers and Technical details. Each page refreshes every 8 s from /api/notusbase/metrics, /api/notusbase/status and /api/notusbase/clients; the metrics key opens only the first. Overview keeps a 60-minute history, one point a minute, in memory only: restarting clears it. For more than an hour, or history across restarts, use Prometheus.

Overview opens with one sentence and its sign: "All good.", "2 things need attention.", or, on a server with no code yet, what to do next. Warnings show once, right under it, computed from the same limits as the alert tables: one line each with what happened, in the language of the dashboard, and a button to the fix. On the other screens a dot on the Home tab says there is one. Then four cards (calls now, failures of the last hour, what is scheduled, the backup), with color only when something is wrong, and one chart of calls and failures. Capacity and About this server are folded to one line each, and the dashboard remembers the ones you open (Capacity opens by itself when something is tight); the forecast and the tables by size are on Limits. A server with no code published shows First steps instead, with the command to copy. Before production, a server with no backup and superusers with a password only are steps of Before going to production there, not warnings; in production they are warnings. Each warning has a Step by step button to its runbook, which the dashboard shows at /_/#/notusbase/runbooks/<section> from the copy of Runbooks inside the binary; /_/#/notusbase/runbooks shows them all. Red is what wakes someone and orange is what becomes a task. NotusBaseDown and NotusBaseRestarting are not on the screen, because the server cannot see them from inside, and neither is NotusBaseDiskFilling, which needs 6 hours. Orange never goes by webhook or email. Clients lists connection, session (first 8 characters), age and subscriptions: the 200 oldest, and no IP, which only the proxy knows.

Limits and problems

One page answers "what can break, how close am I, and what do I do": Home → Limits, at /_/#/notusbase/home/limits, with a superuser login. It is read only and refreshes every 10 s while the tab is visible, from GET /api/notusbase/limits (superuser or deploy key; 401 without, 404 on the public port with --admin-http). The server builds one answer for every open tab at most every 5 s.

• Problems: the warnings that hold now, the same as in the band of Overview and in the alerts, each with what to do, in the language of the dashboard.
• Forecast: the database against NOTUSBASE_DB_LIMIT_GB (0 sets no size: nothing is enforced, and there is no date and no size warning), its growth per day and the date it reaches the limit; the data disk, free space and the date it fills at the database's pace alone (the WAL, the backup and the verification copy on the same disk grow with it, so it may come sooner). Dates need the growth from Capacity: NOTUSBASE_BACKUP on and a verification at least 20 hours old. Less than a week of history is marked as a weak estimate.
• Tables by size: rows, data, growth per day and the size in 90 days at that pace, plus the next step of the ladder when one is due. With no backup verification only rows show, counted at most every 5 minutes.
• Near the limit: every limit that is yellow or red, open, red first, then nearest.
• The other limits, in four groups: per call, concurrency, storage, transport, each folded to one line with how many it holds. A group opens into one line per limit, and a limit opens into its details. /_/#/notusbase/home/limits?limit=<id> opens one limit by name (the Fleet row of this server links there). Each shows its ceiling as the server applies it, how close the app is, what happens when it gets there, how to stay clear, and the variable that changes it (or "Fixed"). Red first, then nearest. Where a variable at 0 does not mean "no ceiling", the row says what it does instead: 0 refuses everything for the time and document limits per call, NOTUSBASE_VMS, NOTUSBASE_JOB_WORKERS, NOTUSBASE_BACKUP_CHECK_WAL_MB and NOTUSBASE_UPLOAD_MB, and those rows turn red; elsewhere it leaves V8's own heap limit, removes the grace on stop, or keeps no log. Rows where 0 is "no ceiling" say "off". A limit the server does not measure says so and only shows its ceiling; one that applies only with the backup or with S3 says so when they are off. A test fails when the server reads a NOTUSBASE_* variable that is neither on this page nor on its short list of variables that are not limits (addresses, paths, secrets, intervals, copy speeds, caches, switches, allow lists).

What each group measures:

• Per call: the highest value in one call since the server started, and the deployed function that reached it. A function removed or renamed by a deploy leaves the list. JavaScript and ArrayBuffer memory come apart for query and mutation isolates and for action isolates; it is the isolate that holds the most, and an idle isolate shows the figure of its last call.
• Concurrency: the busy shares (writer, isolates, scheduled action slots, live queries, CPU, memory) over the last minute; the counts (sessions, open files, subscriptions, action isolates, database connections) of now.
• Storage and transport: now.

Colors use the same ruler as the warnings, so a row turns yellow when Problems says so:

Logs and histories that drop the oldest when full never turn yellow: full is how they work.

The per-call peaks reset on restart, like the functions screen's. The JSON is for scripts too: each row has id, group, limit (0 means no ceiling or off), unit (count, bytes, ms, ratio), used (null when not measured), pct, level, peak (always present), where, variable, count (isolates or slots behind a ratio) and off (backup or s3 when the limit does not apply now).

Fleet

Several NotusBase servers (on one machine, or one per customer) are seen from one dashboard: Home → Servers, at /_/#/notusbase/home/servers, with a superuser login (the deploy key does not open these routes). Each server is registered on that page, under Add a server, with a name, the address of its admin port and its metrics key (notusbase metrics-key --dir <its pb_data> on that server). The metrics key only reads: it opens that server's metrics and its fleet summary, never the deploy, the data or the variables. A deploy key is refused.

# on each server to register: its metrics key
sudo -u notusbase /opt/notusbase/current/notusbase metrics-key --dir /var/lib/notusbase/pb_data

One row per server, this one first, then red, with warnings, without an answer, fine:

• Server: name, address (as text: it is the admin port, often 127.0.0.1, which your browser cannot open) and the last 4 characters of the saved key. The key is never shown again; to change it, take the server out and add it again.
• Running: version and up since; without an answer, since when and why: no answer in 3 s, connection refused, name not found, TLS certificate not valid, address refused (cloud metadata or link-local), key refused, redirected, not a NotusBase answer, or an HTTP status. A first failure is yellow, "Failed just now, trying again"; after about 5 minutes without an answer (5 to 7: it is asked once a minute, and the warning reads the minute before) it is red, "No answer", and both link to the runbook. The last answer stays on the row, marked "As last seen at HH:MM", and its ages keep counting from then. Since when a server is silent, and its last answer, are kept in _fleet (written when it starts and stops failing), so a restart or a deploy does not reset the warning.
• Warnings: the worst level and how many, and each title with a link to its runbook on this dashboard (the runbooks are the same in every binary).
• Backup: the last copy to the chain, the last check and whether it passed, and the last restore check that passed. "Backup off" when NOTUSBASE_BACKUP is not set there.
• Nearest limit: the highest percentage of that server's Limits page, and which limit.

Test asks a server now, saved or not, and says what came back. Take out deletes the row and its key.

This server asks, never the browser, which has no keys: every server at once with 3 s each (one that does not answer holds none of the others), at most 50 servers, an answer of at most 256 KiB. One answer serves every open tab for 10 s; the page asks every 10 s while it is visible. The server also asks once a minute with no page open (a server just added is asked within 2 minutes), and a server without an answer for about 5 minutes raises One of the servers is not answering (notusbase_fleet_unreachable; the alert NotusBaseFleetUnreachable waits 2 minutes more, since after a restart the gauge comes back at once). A red warning of another server does not raise one here: that server sends its own by webhook and email (Alerts without Prometheus).

The address. https://, or http:// only for localhost (which must resolve to loopback), private or loopback IPs and Tailscale IPs (100.64.0.0/10, which WireGuard already encrypts), where the key does not cross the internet in clear; the host is taken in lower case without a final dot; no user, password, query or #; https://x:443 is https://x. The route is on the admin port, like the metrics: with --admin-http on loopback, servers on the same machine are registered as http://127.0.0.1:<port>, and a server on another machine needs its admin port reachable from this one (a private network, or a proxy with HTTPS and an allow list), as for Prometheus. Redirects are not followed, not even to the same host, and no proxy from HTTPS_PROXY is used: register the address that answers. Loopback and private addresses are allowed on purpose, since that is where the servers usually are; the functions' fetch refuses them, and that is why only a superuser registers or tests an address. The address dialed, after DNS and on every connection, is never where a cloud serves metadata and credentials: link-local (169.254.0.0/16, fe80::/10), 100.100.100.0/24 (Alibaba Cloud, Tailscale's own resolver), 168.63.129.16 (Azure), fd00:ec2::/32 (AWS) and fd20:ce::/64 (Google Cloud), also inside NAT64 (64:ff9b::/96 and 64:ff9b:1::/48) and with a zone (fe80::1%eth0; an address with a zone is not registered). Those are refused, with http or https, whatever NOTUSBASE_FETCH_ALLOW says. Tailscale addresses work. An isolated copy calls no one and registers no one, and an environment with the outbound lock calls only the hosts in NOTUSBASE_FETCH_HOSTS. A server this one refused to call shows "Not called" and why; it is not counted as silent and its clocks do not move, since nothing is known of it. What protects other services on the network is that nothing of an answer reaches the page as it came: only the fields above, cut to size, or the short reason.

Where the keys are. In the _fleet table of this server's database, in plain text, so also in data.db and in every backup of it: whoever has a backup has the metrics keys of the fleet. Reached only by the superuser routes, and the activity log keeps no request bodies. The SQL console does not show the key column, and notusbase copy, staging and isolated mode delete the table, so a copy never calls the fleet. A restored backup keeps it, since it is the same server.

The JSON of GET /api/notusbase/fleet/health (metrics key) is for scripts too: version, started, at (seconds), notices (level, title, runbook, since), backup (enabled, lastWrite, checkAt, checkOK, restorePassed, restoreChecked) and headroom (id, pct, or null).

Why is it slow

Follow these in order. Each step rules out a family of causes.

1. The disk: iostat -xz 1. With %util near 100 and notusbase_commit_seconds rising, the disk is the limit. With the backup check running (notusbase_backup_check_running) it is expected: lower NOTUSBASE_BACKUP_CHECK_MBS or use another disk.
2. The brake: notusbase_brake_active at 1 means the WAL passed 2 GiB and writes yield the disk to the checkpoint.
3. A WAL growing with no brake: a stuck reader holds the checkpoint. Look for a long query, a tool left open on data.db and a backup check in progress. With no chain, see Backup stopped.
4. The CPU: top -H -p $(systemctl show -p MainPID --value notusbase). First look at the kernel clock source in /sys/devices/system/clocksource/clocksource0/current_clocksource. With hpet or acpi_pm, every time read goes to the kernel and NotusBase loses 20% to 27% of its throughput. If dmesg says the TSC was marked unstable and the machine is a single one (not several sockets), put tsc=reliable on the kernel command line and reboot. With notusbase_function_seconds high on query, the cause is an expensive function. Open the calls in notusbase logs or on the dashboard's Functions screen, at /_/#/notusbase/functions, through the tunnel.
5. Reactivity: many notusbase_live_reruns_total per second with a high notusbase_epoch_seconds point to a table written by many people and subscribed to by many people. The writer stays free, but the screens lag.
6. Conflicts: notusbase_commit_conflicts_total rising along with kind="mutation" means mutations fighting over the same keys. Each conflict reruns the function. Spread the key.
7. Memory: process_resident_memory_bytes rising without stopping is a leak. Compare it with notusbase_subscriptions, and see in notusbase_v8_heap_total_bytes and notusbase_sqlite_memory_bytes whose share is growing.
8. Clients: notusbase_sessions falling while notusbase_session_connects_total rises is a mass reconnect: the proxy, the network, or a server that keeps restarting.

The function logs (Function logs) have the functions' side, and journalctl has the server's.

Capacity

This answers when to change architecture. NOTUSBASE_DB_LIMIT_GB is the agreed database size in GB (30 by default). Failover to a standby takes about 15 minutes at that size. Home → Limits and the capacity alerts count against this number, so set it from your own estimate: tables times rows per day times bytes per row, summed per year, against the limit.

The forecast and the tables by size are on Home → Limits (/_/#/notusbase/home/limits), and Overview says in one sentence, under Capacity, whether anything is tight. The whole view comes out in the terminal:

notusbase capacity --url http://localhost:8091          # --json returns the same numbers

What it shows. The database against the agreed size, the space deleted rows left in it to give back (and whether it is being given back now), growth per day, the date it reaches the limit at that pace (only if within 10 years), the machine's RAM, the busy writer (now and the last hour's peak), sessions against the 5,000 measured, and a table per database table with rows, data and growth per day. At the bottom, what to do, with the step from the ladder below. With no pressure it says "Nothing to change".

Where it comes from. Growth comes from the backup checks of the last month, which already read every row: each writes the size of data.db and, per table, the rows and bytes it read to check.log. So the forecast needs NOTUSBASE_BACKUP on and two checks at least 20 h apart. Bytes per table are the size of the row values, without indexes or empty pages, so the sum is below the file. A partial check brings only the tables it read. The writer and the sessions come from the status screen's 60-minute history.

Warnings are orange, on Home → Overview: database above 80% of the limit, limit reached in under 90 days, database larger than RAM, writer busy more than half the time, sessions above 4,000, late scheduler, refused calls, conflicts, slow epoch and a function that refuses most of its calls. They do not go by webhook or email; for notification use the Prometheus alerts. One exception: a function whose calls the server itself refused in more than half of the cases (arguments that do not match) also goes by webhook and email, because after a deploy that changes the arguments the app in use has every call refused and no failure is counted.

The ladder. Move up one step only when the previous one is used up. Each step costs more to operate.

Function logs

The console.log lines and errors of functions, from calls that worked or not, are kept in three places:

• Live: notusbase logs follows until Ctrl-C, filterable with --function, and keeps nothing.
• The dashboard's Logs screen, at /_/#/notusbase/logs: each execution with duration, result and lines, in memory only, 12 hours at most and sized by NOTUSBASE_LOG_MEMORY_MB. It is gone on restart.
• On disk: pb_data/notusbase.log, read by notusbase logs --since and by the same screen, in the "On disk" section, for the period memory does not cover. It survives restarts and crashes.

notusbase logs --since 1h                       # the last hour, then exit
notusbase logs --since 2d --function tasks      # a whole module; tasks:create for one function
notusbase logs --since 2026-11-07T10:00:00Z --limit 50   # the last 50 since the given time

Each entry is a line with a UTC time, the function and the text. An error's stack continues on indented lines below, so grep, less and tail -f work directly on the file, even with the server stopped:

2026-11-07 10:00:01Z tasks:create [LOG] 'saving 3 items'
2026-11-07 10:00:02Z tasks:delete [ERROR] Uncaught Error: not allowed
    at handler (convex/tasks.ts:41:11)

Size and age. 64 MB, fixed, caps both files together: the current one, notusbase.log, becomes notusbase.log.1 when it reaches half, and the previous one goes. No line outlives the dashboard's log days (Settings, Logs, 5 by default), the same as PocketBase's _logs. With the days at 0, nothing is written. notusbase.log is not part of the backup.

Cost. The function only puts the line on a queue and moves on (about 50 ns per line, measured on a Mac). One goroutine writes, and the file gets what gathered every second, so a crash loses up to 1 s of lines. With the disk behind and the queue full (8,192 lines or 16 MiB), the line is dropped and the file gets notusbase [WARN] N log lines dropped: the disk could not keep up. notusbase logs --since waits for the queue to reach the file before reading.

Errors, grouped. The Functions screen also lists the errors since the server started, one row per function and message, with how many times, the first and the last time, and a link to the latest run that failed (GET /api/notusbase/errors). The message is the one the function threw, without the trace, with ids, numbers (but an HTTP status from 100 to 599 after HTTP, HTTP/1.1, HTTP/2, status or status code), emails and quoted text replaced, and only the first 4 KB read, so the same error falls in the same row; the arguments never go in. That is a net, not a filter: a name or a short code written in the message without quotes stays, and so may an apostrophe inside single or typographic quotes, a backslash before the closing quote, a random word of 12 letters whose case changes little (about one in nine), or a key of two pieces like X7K2-P9QL; and only superusers and the deploy key read it, here and in notusbase logs; it does not go to the metrics, the webhook, the e-mail or the backup. Only real failures count: a ConvexError is normal flow, a conflict is retried, and a refused call has its own warning. It lives in memory with the run log (off with NOTUSBASE_LOG_MEMORY_MB=0) and keeps at most 20 groups per function. The message shows only while the run log keeps the group's latest run, as the run's own detail does; after that the row keeps its counts and says the message left with the log, and the next failure brings it back. The status screen does the same. Groups reset when the server restarts, and an error that keeps happening is announced again once; the screen says since when they count. A deploy does not restart the process and keeps them. When a function has its 20, a new error takes the place of its group not seen for the longest, if that one is older than 24 hours, and is announced; otherwise it counts in that function's "other errors" row, which raises no warning, and the screen names the full functions. The list shows the 20 latest groups and always the "other errors" rows. The link finds the run while the run log keeps it; after that, notusbase logs --since still has its line. A new group raises New error in <function> (see Alerts without Prometheus).

The four results of a call. A run ends as ok, planned, refused or failed (status in GET /api/notusbase/runs), and the dashboard shows them as ok, refused and failed. planned is the code refusing with a ConvexError; refused is the server refusing before the code ran, because the arguments do not match the function's args or the name is not a public function. Neither is a failure: they are not in the error groups, raise no New error and are grey on the screens. They have their own groups, one row per function and reason, in Logs → Refused and GET /api/notusbase/refusals. A write that met another write on the same data is redone by the server and is one run, with retries saying how many times; it is a failure only when the server gives up redoing it (write conflict: gave up after N attempts). The one refusal that warns is a function turning down more than half of its calls for 10 minutes: see A function refuses most of its calls.

What is not here. Refused requests and login refusals by rate stay in _logs, in auxiliary.db, on the dashboard's Logs → Refused.

"use node"

Code with "use node" runs in a separate Node process, a single one, as the same user as NotusBase. Before Node starts, the process fences itself in. The goal is to contain mistakes and a compromised npm package, not hostile third-party code. Do not accept code from customers: whoever publishes already has the whole server, because ctx.runMutation calls any function, internal ones included.

What you need:

• Node 22.13 or newer, outside /home, because the unit has ProtectHome. With the wrong version the deploy fails and says why.
• NOTUSBASE_NODE with the path of the binary, for example /opt/node/bin/node.
• NOTUSBASE_NODE_PATH with the node_modules folder for external packages, also outside /home.
• Linux with Landlock on (kernel 5.13 or newer; check with grep -c landlock /sys/kernel/security/lsm). Without it Node does not start and the message says why. Ports need 6.7, and blocking signals to the server needs 6.12.
• The package's unit, as install.sh writes it: its system-call filter allows landlock_*. With a unit that lacks it, a project with "use node" does not start.

What the fence covers:

What it does not cover:

• The project's functions: ctx.runMutation and ctx.runQuery stay open to any function.
• The project's secrets: the variables are in process.env, and a compromised package can send them out through port 443.
• Outbound internet on 443, and the internal network on 443 for native code: the kernel filters ports, not IPs. Native packages get past the JS filter. To close this, use IPAddressDeny in the unit (below).
• File metadata: Landlock does not cover chmod, chown and stat. Node does not read pb_data, but native code can change the mode of a file there.
• Signals to the server on kernels below 6.12, and ports below 6.7.
• Disk: .notusbase_node/tmp is on the database disk, with no quota.
• Mac: only --permission and the JS filter, no Landlock.
• Not proven: tested with Node 24 and a 7.0 kernel, directly and in a container. A 6.8 kernel (Ubuntu 24.04) was not tested.

--permission alone does not hold malicious code, and Node itself says so. On Linux what holds is Landlock.

Variables:

The process reads only the project folder and NOTUSBASE_NODE_PATH, and runs no external program. Its ceilings are fixed: 1 GiB of RSS, with the heap at 3/4 of it, and 60 seconds without an answer from the event loop before it is killed. They are per process: whoever goes over takes down all its actions, and the next call starts another process. RSS is read once a second, so a jump within that time can slip by.

IPAddressDeny, optional. For the kernel to refuse the internal network, add this per machine with sudo systemctl edit notusbase, which survives install.sh:

[Service]
IPAddressDeny=10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 169.254.0.0/16 100.64.0.0/10

It applies to the whole unit, the server included. It breaks internal DNS, internal S3, a proxy on another machine and a standby on a private network: allow those addresses with IPAddressAllow=. It is not the default for that reason, and it was not tested.

Cores and GOMAXPROCS

On a machine with many cores, limit each NotusBase's Go threads with GOMAXPROCS: use 8 to 12, in notusbase.env. Without the limit, writes fall: on a 44-thread host they came out 34% to 78% below what 12 threads did, and scheduling was slower too. It is not the disk but contention on the single writer. A container cpuset limits the same way, because Go counts only the CPUs it may use.

The limit has a cost: reads use every core and lose with it. Heavy logic does not lose, because it runs on V8 threads outside Go. Measured on that 44-thread host, one pass, in transactions per second:

What the scripts do. On a first install, install.sh writes GOMAXPROCS=12 at the end of notusbase.env, with a comment, when the machine has more than 16 cores; on a server already installed it leaves the file alone. doctor.sh warns, without failing, when the service has more than 16 cores for Go and no GOMAXPROCS in notusbase.env or in the unit. It counts the cores the way Go does: the CPUs the process may run on, capped by CPUQuota= in the unit or the cgroup CPU limit of a container, which Go reads by itself. So a unit or container already limited to 16 or fewer gets no warning.

Several NotusBase on one machine. This is the production plan: each instance gets its own 8 to 12 Go threads, and together they fill the machine instead of one instance fighting over a single writer. Set GOMAXPROCS in each instance's notusbase.env; with AllowedCPUs= in each unit, or a cpuset per container, they also stop sharing cores.

Still to confirm with the limit only on the server: in that measurement the load generator had it too.

Standby

A standby is a copy of the primary in a folder of its own, kept by the bucket and, when it can reach the primary, by the primary itself. It serves nothing, and no cron or job runs on it. It needs the same NOTUSBASE_S3_* variables as the primary, NOTUSBASE_S3_KEY included.

notusbase replica /folder/standby
notusbase replica-status /folder/standby
notusbase promote /folder/standby -- --http 0.0.0.0:8090

• replica follows the bucket, with a pass every 30 seconds. The bucket gives it the base of each chain and the closed segments.
• With NOTUSBASE_ALERT_PRIMARY set as well, it also receives directly from the primary: it asks the primary's public address for what the local chain has past its point, and the primary answers as soon as there is a new point, or after 10 s with nothing (the heartbeat). There is nothing else to turn on. The answers are sealed with NOTUSBASE_S3_KEY, and the requests are signed with it, so the primary's address can be the public one, behind Cloudflare or not. Only one standby receives directly from a primary; another takes its place after 20 s of silence. If the network fails, the standby keeps following the bucket and catches up when it comes back.
• replica-status does not talk to the bucket, so it works with the bucket down. It shows the age of the last applied point, of the last point seen in the bucket, and of the last talk with it. The first two use the primary's clock. With the direct copy on, it also says whether the direct copy was in step with the primary at the primary's last answer: in step when what it had reached the primary's newest point (within 2 s), otherwise a warning with how far behind it was. The primary's Standby tile and notusbase_standby_lag_seconds show the same thing from the primary's side.
• promote turns the standby into the primary. It checks first, and refuses when the primary still answers on NOTUSBASE_ALERT_PRIMARY (or --primary), when neither says where the primary is, when the standby's own check found a defect (at the start, or between the start and the stop of the replica), when what came directly from the old primary does not apply (promoting would lose it), or when another server writes to the bucket; --force promotes anyway. It also refuses when a stored file a row points to is not on the standby, and only --accept-missing-files goes on (see What is lost). For that last check the primary writes a small beat to the bucket every 10 s (about 260 thousand writes a month), whether or not it has new points. Promote reads it right after stopping the replica and watches it for the old primary's whole write lease plus 2 s (about 32 s), counted on promote's own clock: a beat that moves means the configured primary is not the one writing (it was replaced, and promoting would take the replacement down). So every promote takes at least about 32 s, however long ago the primary died: no timestamp of the primary's, whose clock may be wrong, shortens it. --force skips the watch, not the wait for the lease. A bucket without a beat (a primary from before this version) is not watched. Two promotions at once cannot both win: the epoch is created in the bucket only if absent (If-None-Match, which R2 and S3 honor), and the second one refuses. A provider that refuses the header (400 or 501) cannot promote: without it two promotions at the same instant could both go on, so promote refuses. Nothing promotes without the bucket: the new epoch written there is what stops the old primary, and no option skips it. Then it stops the replica, watches the beat, writes the epoch, asks the old primary to check the epoch now (in case it is alive and reachable), makes a last pass over the bucket, applies what came directly from the old primary, prints whether the direct copy was in step with the old primary at its last answer (a warning with how far behind when not: what the old primary acknowledged in that gap comes only from the bucket, or is lost; promote goes on), waits for the old primary's write lease to run out (see The write lease), checks the database, restores .notusbase_secret and the bundle, and starts notusbase serve. Each step prints its time. With --no-serve it only prepares the folder, for those who start the service through systemd. It can be repeated if it fails half way. It needs NOTUSBASE_BACKUP, because the new primary starts its own chain.

What is lost. With the direct copy working, a write the primary acknowledged reaches the standby within 200 ms plus one round trip. Without it, what had not reached the bucket is lost: the open segment's 15 minutes plus a 30-second round plus the upload time. The direct copy also stops once a day, while the standby downloads the base of the primary's new chain from the bucket (minutes on a big database): notusbase_standby_waiting_base is 1 during that time. A single point past 256 MiB (a big import, an index built over a big table) does not go by the direct copy, which could not carry it within one request on a slow link: that segment reaches the standby through the bucket, and the direct copy goes on from the next one once the bucket has it closed (the next upload round, within 30 s, plus its upload time); notusbase_standby_bucket_only_bytes and the Standby tile show it. The direct copy carries the WAL only: stored files (_storage) reach the standby through the bucket alone, one upload round (30 s) or more after they are written; each round sends the files before the segments, so a row in the bucket finds its file there. A row that came directly can still point to a file the standby does not have, and so can an old one whose upload failed. promote checks every row of _storage, before taking the epoch and again after the last pass, and refuses when a file is missing (N stored files the rows point to are not on the standby), naming the first ones: wait for them to reach the bucket, or copy them from the old primary's storage/_storage if it still has them, and repeat; --accept-missing-files promotes without them, and the rows then answer 404. The standby's daily check counts such a file as late, not missing ("files newer than the bucket, not here yet").

The old primary goes on after promote until it reads the new epoch in the bucket, with its next beat, within 10 s. Until then it keeps taking writes from clients that still reach it, and confirms them; those writes are lost, since the new primary never sees them. Moving the route (NOTUSBASE_PROMOTE_HOOK) shortens the time clients reach it, and does not stop it.

The write lease. A primary cut off from the bucket cannot read the new epoch. So, while it has a standby, the primary confirms a write only within 30 s of its last renewal, counted on its own monotonic clock:

• Each beat renews it, when the read of the epoch right after the beat finds it unchanged. The lease runs from just before the beat was written.
• The direct copy renews it too. Each request hands back the token of an earlier answer, and the lease runs from when the primary gave that answer, not from when the request arrived. A request held up in the network adds nothing.

Either one is enough: with only the bucket down, or only the standby, nothing stops. With both down for 30 s, the primary refuses every write before running it, with Writes are stopped: ... (HTTP 503 with Retry-After: 5, a transient error the client may retry). Reads go on. The first renewal lets writes through again by itself, or the beat finds the new epoch and the server stops. A write that committed while the lease ran out before its answer (the process was frozen in between) answers Write outcome unknown, never OK. The function routes check the lease once more as the answer leaves, after the commit's notifications. PocketBase's own API (records, collections, settings, the dashboard) is checked before the handler runs, and its answer is held until the handler ends: if the lease ran out meanwhile, the client gets Write outcome unknown (HTTP 503) instead. Under all of that, a hook on the database refuses any other commit without the lease: jobs, cron, hooks. On the dashboard and in the alerts it is Writes are stopped: no lease; notusbase_lease_required, notusbase_lease_held and notusbase_lease_refused_total show it.

A primary has a standby, and the lease applies, when a bucket is configured and a standby has asked for the direct copy. The first ask creates .notusbase_standby in the data folder, so a primary restarted in the middle of a cut does not take writes freely; a file that cannot be read counts as a standby too. The primary forgets the standby only when its beat reaches the bucket and finds its own epoch, after 24 hours of its own running time without an ask: cut off from everything, it never forgets, and no jump of the wall clock makes it. A restart counts the 24 hours again. A standby turned off on purpose stops nothing while the bucket answers: the beat keeps renewing the lease, and 24 hours later the lease no longer applies. To end it sooner, stop the server, delete .notusbase_standby and start it again. Without the direct copy (no NOTUSBASE_ALERT_PRIMARY on the standby) there is no lease, and a primary cut off from the bucket never stops by itself: shut it down by hand, or cut it off from the clients, before or right after promote.

promote waits for the old primary's lease to run out before the new one takes a write, on its own monotonic clock: one lease plus 2 s from the later of the replica's stop (no ask of this standby renews the lease after it) and the moment it first saw the newest beat (a renewal by the bucket starts before the beat lands). It rereads the beat after writing the epoch, in case one was on its way. With the watch of the beat done before the epoch, this waits nothing more, and the step prints that the lease ran out; with --force, or a resumed promote, the whole wait comes here, and the step prints how long.

What it does not cover, without a fence outside NotusBase (the provider's API turning the old machine off, or a firewall rule):

• A virtual machine paused in a way that also stops its monotonic clock wakes up thinking its lease still holds. Its next beat reads the new epoch and the server stops; only what it confirms between waking and that read escapes.
• A process frozen after the last check, while the answer is on its way to the socket: the client may get OK for a write the promoted standby lacks. The checks narrow that window to the time of a write to the socket; only a fence closes it.
• A PocketBase handler that streams its answer (it flushes as it goes) is checked only before it runs.
• Two standbys on one primary: a second standby takes the direct copy after 20 s of silence and renews the old primary's lease, even after the first one promoted. Keep one standby per primary.

A write a client received as an error is not a loss even when the old primary has it: SQLite may keep a commit whose sync failed, and after a kill -9 the old primary's file can hold a write the standby never received and no client saw confirmed.

How clients find the new primary. Through the route: a DNS record, a floating IP or a proxy in front. Clients of the convex package reconnect by themselves (1 s, doubling up to 16 s), and the server takes the reconnect as it takes any other. Set NOTUSBASE_PROMOTE_HOOK on the standby to a command that moves the route; promote runs it once the new server listens, with NOTUSBASE_PROMOTED_EPOCH and NOTUSBASE_PROMOTED_DIR in its environment. For the recommended setup, a domain on Cloudflare with the proxy on, the package has route-cloudflare.sh:

# in the standby's environment file
NOTUSBASE_PROMOTE_HOOK=/opt/notusbase/current/route-cloudflare.sh
# a token with Zone.DNS edit on the domain's zone, and nothing else
CF_API_TOKEN=...
CF_RECORD_NAME=app.example.com

It points the record at the machine's public address (or at CF_RECORD_CONTENT), with the proxy on, so clients follow in seconds, with no DNS cache in their way. The record must exist already. If the hook fails, the new server stays up, the alert Route hook failed after promotion goes out, and running notusbase promote again on the same folder only runs the hook again.

Planned switchover. To move the primary on purpose (a new machine, maintenance) without losing a write: stop the primary cleanly (systemctl stop, docker compose stop; not kill -9). Its stop lets the running calls finish and refuses new ones, copies the last point, and sends to the bucket the open segment, the stored files and the vault, then marks the chain's manifest as stopped, with the run of the process that stopped, within 40 s after the drain. Its log says backup: last point went up to the bucket as final, or why it did not. Then run notusbase promote --planned /folder/standby -- <serve options>: before taking the epoch it makes a pass over the bucket and refuses unless the mark is there, so nothing changes when the stop could not send it. After the last pass it checks that the standby reached that point, segment and byte. A start of the old primary takes the mark off at its first upload round, as does any later upload; before that round, its beat already carries another run, and --planned refuses (the primary started again after its clean stop). Without --planned, a promote after a clean stop also gets the final point, and still checks it reached it.

To switch:

1. Look at replica-status. If the alert Primary is not answering came, the primary is down for the standby too.
2. Run notusbase promote /folder/standby -- <the serve options of the old primary>. If it refuses because the primary answers, decide: shut it down first, or pass --force knowing that the old primary keeps taking client writes until it reads the new epoch, and that those are lost; cut off from the bucket, it stops only when its write lease runs out, and without a lease never. If it refuses because it cannot check the primary, pass --primary <its address>, or --force if you know it is down.
3. With NOTUSBASE_PROMOTE_HOOK, the route moves by itself; without it, point DNS, the IP or the proxy at the new machine now. Clients reconnect by themselves.
4. Later, run notusbase replica in an empty folder on another machine to have a standby again.

After promote the old primary stops once it reads the newer epoch in the bucket, and refuses to start again; until then it takes writes that are lost (see What is lost). Promoting by mistake takes down a healthy primary, and there is no undo.

promote runs integrity_check before serving by default, which takes minutes on a big database (on a 19.6 GB one, several). --check quick runs quick_check before serving instead, and integrity_check right after the server starts, on a read-only connection; if that one fails, the alert Integrity check failed after promotion goes out. The default stays full until the times on a big database are measured.

On the primary, the direct copy shows in the metrics: notusbase_standby_connected, notusbase_standby_lag_seconds (the time of the newest point minus that of the last point the standby synced; 0 when it has everything, and also 0 when the primary has no new writes), notusbase_standby_lag_bytes, notusbase_standby_applied_lag_seconds, notusbase_standby_waiting_base and notusbase_standby_base_wait_seconds. The alert Standby is not receiving goes out after 5 minutes with the lag above 60 s; waiting for the base of a new chain does not count, unless it takes over an hour, and the dashboard's Standby tile shows it as its own state. The primary keeps the raw closed segments a standby has not received yet, up to notusbase_standby_raw_cap_bytes (4 GiB); past that they are compacted, and the standby catches up through the bucket. A standby silent for 10 minutes no longer holds raw segments (it comes back through the bucket), and its alert keeps firing. After 24 hours of silence the primary forgets it: its metrics go, and with them the alert and the tile, and the write lease no longer applies. A day keeps paging for a standby that died, and lets one taken out of service clear by the next day; restarting the primary forgets its metrics at once, but not the lease (see The write lease).

The standby also watches the primary from outside. With NOTUSBASE_ALERT_PRIMARY and NOTUSBASE_ALERT_URL, it probes the primary's /api/health every 10 seconds and, after 6 probes with no 2xx answer, sends Primary is not answering; on the first good answer it sends RESOLVED. A deliberate stop counts as down. With NOTUSBASE_ALERT_URL it also sends Standby cannot follow the bucket after 10 minutes in a row of not following. The standby has no dashboard or SMTP, only the webhook. The log shows replica: the primary has not answered since ... and replica: the primary answered again.

The standby can also run the backup check, so the primary does not: set NOTUSBASE_BACKUP_CHECK_HOURS=0 on the primary and NOTUSBASE_BACKUP_CHECK_HOURS=24 on the standby. The primary always sends a 256 MiB window of each closed segment for the standby to compare, through the bucket and through the direct copy. The result reaches the primary through the bucket, and notusbase_backup_check_replica goes to 1.