{
 "from": "v12.4.1",
 "to": "v12.5.0_a032",
 "arch": "x86_64",
 "size": 1830110742,
 "md5": "dc76e49ad68f52d12a6fde0aabaa9c2b",
 "built": "2026-11-05T13:18:28Z",
 "changelog": "- cluster geolocation now asks NodeWeaver's own servers for the cluster's public address (`http://pipes.nodeweaver.io/myip`, then `http://pipes.nodeweaver.eu/myip`: plain HTTP, port 80), and `api.ipify.org`, `icanhazip.com` and `checkip.amazonaws.com` only when neither gives an address that the location database knows; before, only `api.ipify.org` was asked, and a site whose DNS blocks it got no location\n- the weekly ZFS scrub and trim now run on fixed nights, so a reboot or a change of Leader no longer moves them: the trim on the night into Wednesday (the Leader trims one node at a time), the scrub on the night into Saturday or into Sunday, chosen for each node from its hostname, 3 or more days from the trim. Before, the week counted from the node's last boot: a node scrubbed on the first night after each boot, and a new Leader brought its own trim day. After the update, the first scrub or trim can come less than a week after the previous one\n- `NETMODE` `lacp` and `lacp2`, also in a `custom` value, now set the LACP hash policy that the documentation names (1 = layer3+4, 2 = layer2+3). Before, the network configuration had the key without its `bond-` prefix, so it was ignored and every LACP bond ran the default `layer2` hash: all traffic between two nodes used one link. An existing cluster keeps its old network configuration until `NETMODE` is applied again (a changed value) and the nodes reboot\n- a disk that stops answering no longer stalls the virtual machines whose data has a copy on it for as long as it hangs: the storage service treats a disk call that has not returned in 60 s (`HDD_IO_TIMEOUT` in the chunkserver configuration, 0 = off) as an I/O error, reports that copy damaged and stops using the disk, so writes go on with the other copies after about 60-75 s. A disk that timed out is used again after a restart or reload of its storage service; when every copy of a chunk timed out at the same time, those copies come back only with that restart. A node runs the new storage service after its next chunkserver restart or reboot\n- a node whose storage pool stops committing writes (`MD6`) is now hard reset only when every data disk of the node is stuck (on a single-disk node: its only disk, as before); a node with a data disk that still works keeps running with its virtual machines, and the log says once why it is not reset and which disk to check or replace. When the Leader could not read the node's whole list of pools that round, it does not reset it either. While such a disk stays stuck, the periodic jobs no longer wait on it for ever: the metadata safe copy of the backup (it held the lock of every backup mode, the ONE database dumps included), the nightly SMART report, the scrub, the trim, the ZFS snapshots and the disk fence skip, and log, a pool whose ZFS transaction group does not move while its disks write nothing; a scrub or trim already running on it stops waiting once the stall is confirmed\n- when the disk that holds a Follower's copy of the storage metadata hangs (its ZFS transaction group stops) and the node has another usable disk, the node now points its metadata folders to that disk and downloads the metadata from the Leader again (`MD1`), without reading the hung disk and without marking it broken; until that copy is complete, the node takes no part in a Leader election. Before, the node kept a dead metadata copy until a hard reset. The metadata cleanup no longer writes to a hung disk, and a broken-disk list that cannot be read no longer lets the metadata folder selection pick a disk\n- an update from a030 or a031 that is killed while the a32 post-upgrade step runs (for example by the out-of-memory killer) no longer stops all maintenance until a reboot: the older update script marks the update without naming who runs it, and the a32 post-upgrade step now names itself and that script at its start, so the mark goes once neither runs\n- new cluster attribute `K3S_INPLACE_UPGRADE` (default `NO`): with `YES` the Leader brings the k3s VMs to the nwk3s version this release was tested with, without redeploying them: Workers first, then the Master, one VM at a time, a reboot only where the upgrade needs one. The next VM waits until the previous one is back (its node Ready, every Deployment, StatefulSet and DaemonSet that was ready before ready again); anything not proven stops the upgrade at that VM, syslog (`K3SINPLACEUPGRADE`) says why, and it is tried again an hour later. The VM fetches the nwk3s version from its own git origin and runs it only when it is the exact commit this release names\n- an update now restarts the node watchdog (`nodewatchdog`) and the bond failover announcer on the new code, and starts them where they do not run; before, both kept the previous version's code until the next reboot. The watchdog is disarmed during its restart, so the restart never resets the node; a watchdog or announcer that does not run 15 s later marks the update incomplete, and it is applied again on the next round. `NODE_WATCHDOG` is not changed: a cluster without it is still only watched, not reset\n- a node set up with a static IP, and `nw networktest`, find the cluster at an IP from the Leader adverts still active on the network, compared to the whole IP: a Leader whose advert was published again in the second before the search, or seen on two interfaces, no longer reads as \"NOT a Leader node\" (setup stops) or \"Multicast query did NOT work properly\"; the IP 10.0.0.5 no longer matches a Leader at 110.0.0.5, and a DHCP-mode Leader whose label contains \"cluster\" no longer stops the join\n- the failoverability panel (`FAILOVERABILITY`) no longer hides missing chunks or missing metadata backups when the search for the Leader adverts (mDNS) does not complete: that CRIT now stays CRIT and names both findings; before, it showed UNKNOWN for that round, and the alert mail came again each time the search worked again. A single-node, endangered-chunk or other result without the search stays UNKNOWN, because a split brain cannot be excluded, but the message now also names that result\n- the node card shows CPU power/electrical limiting also when it is not a warning: an amber line when the power gap to the configured PL1 could not be measured (the power or PL1 could not be read), and a neutral line when the measured gap is below 5%; before, the card showed nothing of that limiting\n- a node set up in DHCP mode now finds its cluster when the cluster LABEL has a space or a non-ASCII character (`Sede Città`): the label adverts on the network carry the label with each byte outside `A-Z`, `a-z`, `0-9`, `.`, `-` and `_` made `_` (first 26 bytes), and the new node compares that form of the label it was given. The dashboard and the parameters keep the label as it was typed (a label changed in Sunstone is still stored in that form, as before). The new node also no longer joins a cluster whose label only ends with the given one or matches it as a pattern (`lab` and `mylab`, `a.b` and `aXb`), nor a cluster whose label advert was withdrawn during the search, and it stops with a CRITICAL message when the label matches more than one cluster. A node installed from an earlier release does not find a cluster updated to this release whose label has another character outside that set (`Site#1`): set up such a node from this release or later\n- the Leader election no longer counts a metadata version or Leader advert that its network search also reports as withdrawn: a peer that changed its version, or a Leader that stopped advertising, in the second before the search no longer counts with its old advert (an advert withdrawn while the search runs still counts, as before)\n- more from the free-text review (#2712): a diagnostics password that could not be stored is no longer stored, in the next parameter round, as the text \"Could not be stored, try again.\"; when not every node gets a new diagnostics password, its field says so; when 7z fails, the boot logpack keeps the log in `/upload/logs` and no longer says it is available, and the boot logpack is tried once per boot whatever its outcome (7z failed, no diagnostics password, not enough space on the USB, no log made), not every minute, and each failure keeps its log in /upload/logs and says so; the alert mails pass each address as it is, never as a file name pattern; a failed sync from the USB SIDELOAD partition reports its own log, and a failed sideload sync at seed shows its log in the boot warning\n- `/var/log/mail.log`, where the system log writes mail messages, is now rotated every night with the other system logs (30 kept); before, nothing rotated it. On an updated node its first rotation is on the second night after the update\n- the full-DFS harm reduction (EF1 undeploys every VM and stops the scheduler; the dobby removal; the lone-node all-in-one switch) reads used and total space from one chunkserver list: two lists could count different chunkservers, and one that left between them read as a false fill. A list that cannot be read or holds no usable numbers is now an unknown fill: no step runs and the last status stays, with a WARNING in the log (nw/opt#2720). A mono-media cluster no longer logs a WARNING for its missing HDD label every five minutes.\n- an update that was killed before its end (for example by the out-of-memory killer) no longer stops all maintenance and every later update until a reboot: the update marker now records the update script and the step it runs, and it counts only while one of them runs; the services that the update restarts (the orchestrator, Sunstone) do not count. A marker whose update has no process left is removed and logged; an update that runs long still holds it, and after 6 hours its marker is logged once an hour with the processes that hold it. A marker left by an update applied by an older version still counts, as before. `nw unlockupdates` now also unlocks when the marker was left by such an update\n- log rotation (the dmesg dump, logrotate, the rsyslog restart) runs from its own nightly cron entry at 00:00, `rotatelogs`, no longer as a step of `every1night`: it waits for no update or monitoring round, so a hung update no longer stops it (one cluster had rotated nothing for 105 days and filled the Leader's root filesystem). The Leader rotates the logs on the DFS last, in the background, with their own state file, and only while the DFS takes writes, so a rotation that blocks on the DFS stops nothing else; it then empties the setgoal logs, only after that rotation kept `setgoal.log`. After the update, the logs on the DFS skip one night: the first night records them in the new state file\n- free text (an e-mail address, a password, a file name, an error message) no longer goes through `xargs`, which stopped at an apostrophe or a quote with no pair and took quotes out (#2712): after a second change of the diagnostics password, the log packs are encrypted, and a rebuild from the resiliency copy sets the web login, with the current password (before: all passwords joined, or the oldest one); the alert mails go out when an address in `ALERT_EMAILS` holds an apostrophe; monitoring output whose first word holds an apostrophe no longer reads as empty, which suppressed the alert mails; the DFS repairs and the debug-shell repair aliases take each file name whole; the hardware reports list every controller; a failed USB sideload sync writes its whole log into `USB_SIDELOAD_SYNC_STATUS` as one valid value; the `CRITICAL NP0` and `NP1` boot messages keep the text they quote\n- the nightly maintenance (`every1night`) now runs one instance at a time, as the 1- and 5-minute maintenance does: when the previous night's run still waits for an update or a monitoring round to end, the new one logs \"a previous run is still going\" and exits. Before, each night started one more waiting run: on a leader whose update hung, 105 of them piled up, all to go on together when the update ended\n\n- the update no longer waits without limit for the periodic ZFS snapshotter to stop before it removes its leftover snapshots: after 5 minutes it logs what is still running, skips the removal (the next node start does it) and goes on; before, a snapshotter blocked on a pool whose disk had gone held the update, and the node's nightly maintenance, for good. A pool whose status cannot be read now counts as not healthy for the snapshotter, its rollback at node start and that removal\n- the nightly ZFS pool clear (`zfsautoclear`) no longer runs `zpool clear` on a pool none of whose data devices is present, such as a pool whose only disk vanished: a clear cannot bring it back and, on a suspended pool, can block. It logs warning `ZA4` with the missing device instead, and `ZA5` when it cannot read the pool's device list\n- removing leftover ZFS snapshots of the periodic snapshotter (`ZFSSNAPSHOTTER` disabled, the default), which runs at every node start and after an update, now skips a disk pool that is not healthy (not ONLINE, or with read, write or checksum errors) and logs it by name; before, a pool whose disk had gone could block that step for good, and with it the node's start or the update\n- a node now leaves its first-boot state once its metalogger holds a copy of the metadata: a later boot without metadata (a failed metadata disk) waits 30 minutes before it bids for the Leader role (`WARNING LF3`), as designed, instead of the 60 seconds of a first boot. Before, every node set up since 2023-09 kept the first-boot marker for ever, and also started its host sync in the background at every boot (nw/opt#2716)\n- new dashboard panel **Kubernetes:** (cluster probe `K8S_PLATFORM_HEALTH`): a WARN or CRIT of the platform checks that each k3s master publishes about itself (k3s service, datastore and Litestream, API stalls and latency, admission webhooks, certificates, CSI and PVCs, node pressure, registry cache, image pull and conversion, worker scaler) now shows as a warning that names the VM and the check, and a running master that has not published for 15 minutes shows as UNKNOWN; before, only EdgeInsight and the MCP tool `nw_k3s_health` showed them (a k3s cluster ran 35 hours without its registry cache, and no dashboard showed it). The application checks (pod restarts, Helm releases, API call counts, CPU and memory) stay in EdgeInsight and MCP. It needs nwk3s `cb0e04c` or later on the masters: an older master always reports \"K8S Service Health: Service k3s-agent is NOT active\", a false CRIT that holds the panel yellow until the master runs a newer nwk3s\n- `VM_IO_ERRORS` now keeps reporting a VM from the first disk I/O error its running qemu process logs until the VM runs a new qemu process (a cold boot: `onevm poweroff --hard` + `onevm resume`); before, it reported the VM only in the 45-second probe round that read the error, so a VM whose guest stopped its disk I/O after the error (a filesystem that aborted) was almost always shown as OK. An error written in the same round as a log rotation is now found too; a state file that cannot be read or written reports UNKNOWN, not OK, and a full `/tmp` no longer replaces the saved state with an empty file\n- a factory reset now also removes the metadata high-water mark of the erased cluster (and a pending rollback override): a new cluster seeded on a reset node no longer waits 5 hours (`HW1`) before its first Leader starts\n- an election held below the metadata high-water mark (`HW1`) logs its critical line when the hold starts and then every 30 minutes, not every few seconds; a damaged hold timer now starts the wait again, where it ended the wait at once\n- the critical state text (alert mail, REST `critical_state_cleartext`, dashboard, cluster attribute `CRITICAL_TEXT`) now gives, for each critical code, its first and its last log line and the number of lines between, at most 16 KiB (`CRITICAL_TEXT`: 4 KiB); a cluster attribute longer than 131071 bytes is no longer passed to the node's scripts (a warning names it). Before, a long critical text in `CRITICAL_TEXT` made every script on the node fail to start any command. An apostrophe in the critical text no longer cuts `CRITICAL_TEXT` short or leaves it empty, and a double quote no longer stops its update. A critical text that cannot be read no longer empties `CRITICAL_TEXT` (it keeps its last value), and a failed update of it is logged as a warning\n- a follow-side metadata rollback hold (`MR1`) logs its critical line when the hold starts and then every 30 minutes, not every 10 seconds\n- the metadata high-water mark now records the cluster it was seen in: a node seeded, joined or rebuilt into another cluster drops the old cluster's mark (warning `HW3`) instead of holding its first election for 5 hours (`HW1`). A mark recorded before this version is kept as before\n- a factory reset now also removes the node's leader and follower history, its operator election hold, a pending removal of the single-node support chunkserver, and the copies a USB parameter override keeps of the cluster's parameters: the override copy put the erased cluster's parameters back at the next boot, and the election hold kept the node from ever taking the Leader role in its new cluster\n- a factory reset now also removes the erased cluster's credentials and licence state: the MCP emergency token hashes, the vTPM state key sealed into the node's TPM (a cluster seeded on the node no longer takes over the old cluster's key), the licence lock (a reset node no longer refuses every VM deploy for ever after a licence lock), the first-boot marker and the ARC guest-memory cache. The node's licence code is kept\n- a file whose name holds an apostrophe, a quote or a backslash in the list of files with missing chunks no longer stops the repair of the files after it (`FORCE_CHUNK_REPAIR`, the automatic DFS repair, the debug shell `dfsrepair`, the rebuild from resiliency), and a `*` or `?` in such a name no longer selects other files; the log lines that print the list or a chunk check result keep the whole text. A file whose path holds a space is now repaired too: the list kept only the path up to its first space (also in the repair of the system files when a Leader starts)\n- `ONE_COMMAND` writes the whole command output back: an apostrophe or a quote no longer cuts it, quotes are kept, and backslashes are left out\n- a `\"` or a backslash in a value set from Sunstone or `onecluster update` no longer breaks the feedback that `IPCONF`, `FACTORY_RESET`, `REBOOT`, `LOGO`, `IMAGE_EXPORT`, `CUSTOM_GOPROXY`, `CUSTOM_SSL` and `DEBUGSHELL_COMMAND` write into the template: the update of that round failed, and every other parameter's feedback in it was lost\n- a cluster parameter value is now passed to the node's scripts as it is: a `*` or `?` in it no longer becomes the names of the files in the script's working directory, and a run of spaces or a tab in it is kept (before, it became one space)\n- a USB parameter override keeps the runs of spaces in a value (`LOGIN_BANNER`, `SNMP_LOCATION`) in the cluster parameters and in its log line; before, they became one space\n- the stale chunkserver removal logs the whole reason when a removal fails (an apostrophe in it emptied the warning)\n\n- a failed or empty host-inventory read no longer erases the Kubernetes worker-eviction history and permits another eviction in the same incident; worker selection also holds when its host-state or capacity read fails, even if that read printed plausible data\n\n- automatic Kubernetes service recovery now shares three attempts across the cleanup and scaling watchdogs, then warns and waits for an operator or confirmed healthy recovery. Replacing failed VM IDs or passing through failure/cooldown states does not restart that allowance. A reboot or leader change clears this local RAM state. Invalid scaling grace values no longer abort the PRIORITY hold log with division by zero\n- deleting a Worker during rollout now preserves the desired Worker count, including through deployment completion and cooldown; it no longer reports success and silently reduces the target. Cleanup waits for confirmed deletion before recovery, and valid quoted names of unrelated VMs no longer block the harvester\n\n- new `BLOCK_DEVICES_TIMEOUT` host attribute (on the cluster, the default for every host that does not set its own): the disks that hold the storage data now wait 30 seconds, not 240, for a command before their connection is reset, so a disk that stops answering pauses the storage writes of the whole cluster for about 30 seconds instead of 4 minutes; the update applies it without a reboot\n- a Leader whose metadata storage pool wedges now steps aside even when its own storage service stops answering first; before, it kept running and the whole cluster stayed down until someone reset it by hand. It steps aside only when, a few minutes before, no chunk was missing and another node shows directly that it can take the Leader role; as after a power cut of the Leader, the metadata changes of the last moment can be lost (`MD2`, else `MD3` naming what is missing)\n- after a power cut during writes, far fewer chunks come back with checksums that no longer match their data: the storage service now records each block's checksum as it writes the block, so at most the block being written at that moment can disagree (before, every block written since the file was opened could), and the automatic repair handles that block\n- the storage mount on the hosts retries for about 10 minutes (was about 2) before a VM gets an I/O error, so a write waits through a storage service restart; the k3s VMs mount the storage with the same value\n- new `STOP_ON_IOERROR` cluster attribute, default `NO`: set to `YES`, a disk read or write error pauses the VM instead of reaching its operating system; when the VM's disks are readable again, its host powers it off and starts it again (a cold restart), at most three times in 24 hours, after which it stays paused for an operator; the VM ping watchdog (`WATCHDOG_PING`) does not act on a paused VM\n- a VM held for a disk swap (a `WATCHDOG_PING` snapshot revert, `nw ttrestore`) is no longer started, recovered or powered off while that hold cannot be read: an I/O error of the cluster storage used to read as \"no hold\"; the VM now waits, with a log line, until the hold can be read\n- single-node clusters with copies of a chunk on several disks (`CHUNKSERVER_MODE` `single` or `multi`): the chunk check at boot (`BOOTCHECK`) now compares the copies before it repairs one; before, it repaired every copy on its own, so a copy damaged by a power cut could be made valid next to a good copy with different data\n- monitoring no longer stops while a host is unreachable: a check that does not finish within 2 minutes is stopped and reported UNKNOWN, and the other checks still run; before, one check that waited on a host that was off held all monitoring for hours, and the status shown stayed as it was\n- base OS updated to Alpine Linux 3.24: Linux kernel 6.18 LTS (was 5.15), OpenSSH 10.3, Open vSwitch 3.7, PostgreSQL 17\n- the update from 12.4 needs no reboot: until the node restarts on the new kernel it can still load kernel modules and change its firewall rules, and the restart then moves the firewall to nftables; installation packages and update files that earlier releases left behind are removed during the update, unless they were changed on the node\n- VM high availability no longer starts a second copy of an AUTOSTART VM. An AUTOSTART VM shown UNKNOWN or powered off is started again only on its own host, and only when that host is monitored and answers that the VM does not run there. A VM whose host is not monitored, cannot be reached or cannot say whether the VM runs is held, with a warning in the log every minute: nothing proves that the host has stopped it (there is no fencing), and a host that came back after a reboot or an update while its monitoring had not yet recovered got a second copy of its VMs elsewhere, for which a Kubernetes Worker was destroyed to make room. **After a node dies, its AUTOSTART VMs start again only when that node is monitored again.** To start them on another node before that, first make sure the dead node is powered off, then run `onevm resched <id>` for each VM. A failed AUTOSTART VM is recovered (`onevm recover --retry`) only when every host it ran on answers that the VM does not run there; when one of those hosts is gone, recover it by hand once you know the VM does not run there\n- the VM high availability run, and the k3s `AUTOSCALE=fill` Worker removal, now run one at a time: a run that outlasts its minute no longer gets a second one beside it acting on the same VMs and Workers\n- VM high availability now also restarts the AUTOSTART VMs of a host that dies while OpenNebula already shows it in error: those VMs stayed shown RUNNING, so nothing restarted them. Once the host has been in error and unreachable for 10 minutes, with no storage mount from it active (checked only when another live node shows that its own mounts can be found by its address) and its VMs' disks on shared storage, the host is disabled and enabled again; OpenNebula then marks its VMs UNKNOWN, and they are held as described above until the host is monitored again. A host left disabled by an interrupted attempt is enabled again only while it is still unreachable; one that answers stays disabled for the operator. On a healthy cluster this check does not touch the shared storage\n- k3s `AUTOSCALE=fill`: a Worker that is undeployed, stopped, powered off, suspended or on the failed host no longer stops the removal of a Worker that makes room for a PRIORITY VM, the k3s master (in one VM group with its Workers) now counts as such a VM, and the Worker removed is the smallest that makes room for all of them, else for the largest one, one not on the Leader first; before, it was the largest Worker. A Worker is removed only for a PRIORITY VM that high availability has rescheduled: a VM shown UNKNOWN for a moment while its host turns to error, and still running, no longer costs a Worker; and a later outage of the same VMs is no longer left without room because a Worker was removed for them in an earlier one\n- AUTOSTART: a priority tier alone enables high availability (`AUTOSTART=\"PRIO1\"` is `AUTOSTART=\"PRIO1 YES\"`, `AUTOSTART=\"PRIORITY\"` is `AUTOSTART=\"YES PRIORITY\"`), and the tier is kept whatever separates the words (`YES:PRIO1`, a value written over several lines); an explicit `NO`, `FALSE`, `OFF`, `DISABLED` or `0` still disables it, also for the deploy of pending priority VMs, the fast boot of a single-node cluster (which now starts `PRIO1` and `PRIO2` VMs first too) and the Kubernetes worker removal that makes room for a priority VM\n- a hard stop or delete of a VM (`--hard` terminate, poweroff and undeploy, `onevm recover --delete` and `--recreate`) now checks that the VM's process is really gone, and kills it when the virtualization service could not stop it; when even that fails, deleting or recreating the VM still takes it off its network, so a leftover copy cannot clash with the recreated one\n- a suspend, stop or cold migration whose save fails, and a snapshot revert that fails, are now reported as failed and the VM stays RUNNING; before, a realtime VM that cannot be saved was shown SUSPENDED or STOPPED while it still ran (see [Soft Realtime](realtime.md#dedicated-cores-cpu_pin))\n- a failed start of a VM that is already running no longer removes its network configuration (switch port, firewall and security group rules), with every network driver that runs on the node (bridge, 802.1Q, VXLAN, Open vSwitch, firewall, ebtables, node port)\n- a reboot leaves the root filesystem clean (it used to come up after a journal replay every time), and after a power cut its full check now runs in the same boot, before the system starts\n- updated libvirt 12.3\n- license expiry is now enforced: trial and node licenses really expire (trials used to renew themselves) and a node without a valid license has its VMs undeployed and is taken offline until it is licensed again; expiry shows on the dashboard and in alert emails, with a 30-day warning for maintenance/support\n- MCP server: the Leader serves an MCP (Model Context Protocol) endpoint at `https://<cluster_url>/mcp` with about 150 tools to inspect and operate the cluster from an AI assistant, using the REST API token and acting as that user; writes need `MCP_ALLOW_WRITE`, which a new cluster starts with set to `YES` and a cluster upgraded from 12.4 starts without, so it stays read-only until a person sets it (`MCP_ALLOW_DESTRUCTIVE`, off on every cluster, gates destructive operations), `DISABLE_MCP` turns it off, every request is logged as `___MCP`, and an emergency mode answers when the orchestrator is down\n- MCP emergency mode: the `mcp` token is accepted only while the cluster has confirmed it within the last 96 hours; the emergency path closes on a node whose copy of the cluster settings cannot be read or is empty; emergency writes are refused while `MCP_EMERGENCY_WRITE` cannot be read; with emergency writes off, the Open vSwitch state and a VM's guest-agent information can still be read\n- REST call `nwnodemetrics` and MCP tool `nw_node_metrics`: the measurements an EdgeInsight server collects from each node (load, CPU time, memory, disk and network counters, file systems, temperatures, BMC sensors with their limits), read as JSON without an EdgeInsight server; a node that cannot be read is reported as unknown, never as zeros\n- MCP server: a password, token, key or secret parameter is withheld from every answer (it shows as `[withheld]`); a background job (deploying or converting a persistent VM, saving or restoring a state, a TimeTurner restore) answers with a job ID, and `nw_job_status` says how it ended; a change to a template, image, virtual network, VM group or service template can be previewed with `dry_run` and undone with `nw_undo_change`; each user may make 30 changes a minute, set by `MCP_RATE_LIMIT`; new tools `vm_resize`, `rename_object`, `chown_object`, `vnet_address_range` and `one_read` (any object, or a list narrowed by words of the names), and `manage_vm` holds and releases VMs\n- `nw setparam` (and the REST call `nwsetparam`) sets a cluster or host parameter and applies it immediately; what can be set, and with which values, is defined by a settable-parameters allowlist that DNSOps records and the USB stick (`nw_config.txt`, `nw_config_override.txt`) are now checked against too (a refused line is skipped and logged)\n- `DNSOPS_OWNED` lists the attributes DNSOps controls, Sunstone edits the allowlist would refuse raise a dashboard warning, and parameter changes that fail or are rejected are rolled back and shown on the dashboard\n- vTPM: a VM's virtual TPM now survives poweroff, undeploy, recreate and node failover instead of being silently replaced by a new, empty one; its state is kept on the DFS and follows the VM\n- vTPM: new opt-in `VTPM_SEAL` cluster attribute encrypts the vTPM state with a key sealed in the nodes' physical TPMs, and the new `nw vtpmca` command enrols the cluster with the NodeWeaver vTPM EK certificate authority so the origin of a vTPM can be verified offline\n- realtime VMs (`REALTIME` 2, 3 or 4) get low-latency libvirt settings automatically at deploy and one dedicated physical core per vCPU on any host (with `CPU_PARTITION=FORCE` also isolated from the host's own tasks); the node card warns when a realtime VM has no dedicated cores\n- new `CPU_PIN = DEDICATED` VM attribute: one dedicated core per vCPU without a realtime priority, chosen among the free cores of the host, with the memory on their NUMA node; VMs with and without dedicated cores now share a host (the others are kept off the dedicated cores), `CPU_PIN_SHORTAGE = FAIL` refuses a start when too few cores are free, a suspended, stopped or cold-migrated VM gets cores picked again when it resumes, every host reports `CPU_PIN_FREE_CORES` for scheduling requirements, a VM with dedicated cores is not live migrated, and a VM with floating vCPUs is live migrated onto the shared cores of the new host; on a `FORCE` host the emulator and I/O threads of every VM are kept off the dedicated cores. See [Soft Realtime](realtime.md#dedicated-cores-cpu_pin)\n- every VM now gets one IOThread: its VirtIO disks do their I/O in a host thread of their own, outside the emulator thread; a running VM gets it when it is powered off and resumed. A VM with dedicated cores (`REALTIME` 2, 3 or 4, or `CPU_PIN = DEDICATED`) gets none unless its template sets `IOTHREADS`, and the IOThreads of the other VMs are kept off its cores like their vCPUs (see [Virtio-SCSI queues and IOthreads](linuxbestpractice.md#virtio-scsi-queues-and-iothreads))\n- `CPU_PARTITION` now accepts `NONE` (no partitioning, the default) and the default number of system cores reserved when partitioning (`SYSTEM_CORENUMBER`) is 6 instead of 4\n- `CPU_PARTITION` changes apply within about 10 minutes without a reboot, in either direction: a step down to `SYSTEM` or `NONE` gives the host back the cores, the interrupts and the power settings the higher setting took (`nw_realtime_status` shows `CONVERGING` until it is done), and running VMs keep their CPU placement until they are next started; `CPU_PARTITION_RELEASE=NO` keeps the earlier behaviour, where a step down waits for the next reboot. See [Soft Realtime](realtime.md)\n- a VM with `TOPOLOGY` pinning (`PIN_POLICY`) now starts on a `CPU_PARTITION=FORCE` host: its CPU pins are replaced by the partition's own placement, its memory placement and hugepages stay, and the VM log has a warning; a VM without CPU pins runs there too, and its log says so\n- a host that cannot read its parameters no longer falls back to an older copy of them: no VM is deployed on it until it can read them again (the VM log says which parameter), and its `CPU_SYSTEM_POOL` shows `UNKNOWN`. See [Soft Realtime](realtime.md)\n- Windows VMs: the missing Hyper-V features and the Hyper-V clock are added when the VM starts, a `<cpu>` element in `RAW` is now used, and APIC virtualization (APICv) is on by default on Intel hosts, with the hypervisor that Windows runs inside the VM (VBS, WSL2) kept off it; see [Managing Microsoft Windows VMs](mswindows.md)\n- GPU passthrough: IDs added to `VFIO_PCI_IDS` bind live with no reboot, the default GPU list grows from 145 to over 500 IDs (audio functions follow their GPU), IOMMU groups now split properly, GPUs that cannot find their VBIOS are given one, and the new `nw gpus` command shows which GPUs can be passed through\n- a VM with PCI devices (GPU, accelerator, a NIC passed through) holds all its memory from the start, so the node no longer starts one that it cannot hold together with the VMs already running there, nor another VM on a node that runs one: the VM log gives the numbers, a VM with high availability starts once the room is there, and `PASSTHROUGH_MEMORY_CHECK = \"NO\"` in the VM template starts it anyway. A running VM on such a node no longer grows past the memory its next start can have: `AUTOSCALE` grows it only that far, and a larger memory resize is refused, with the numbers in the VM log. See [PCI Passthrough](template_management.md#pci-passthrough)\n- self-hosted infrastructure: `UPDATE_REPO` now really works (a cluster that cannot reach NodeWeaver syncs updates from your own mirror), new `CUSTOM_GOPROXY`, `CUSTOM_GOPROXY_ID` and `CUSTOM_GOPROXY_PSK` serve the cluster through a management proxy you host, and opt-in `USB_AUTOUPDATE` now actually keeps the installer image and USB installer sticks current; the connectivity check now names the unreachable service (updates or remote access) and never goes critical\n- new unauthenticated read-only status API for monitoring systems and on-site technicians: `https://<cluster_url>/clusterinfo/status.json` (also `cmd=publicstatus` on the REST API) with cluster and per-node health, without node addresses or VM names\n- `REST_ALLOWHOSTSVIEW` cluster parameter (off by default): lets the Portkey appliance and the Kubernetes master read the REST `hosts` and `cluster` calls without a token\n- storage self-healing: metadata on a failed disk moves to a healthy one (or the node steps aside), a node whose storage pool stops committing writes is hard-reset by the Leader once no chunk is missing (`AUTO_REMEDIATE_STUCK_NODES=NO` only reports it), and `nw recommissiondisk` or `DISK_RECOMMISSION` returns a broken disk to service\n- DFS write-stall protection: stalled chunkservers are detected and the node to power-cycle is named (`CW1`/`CW2`), the chunkserver of a dead disk is fenced so guests stop getting I/O errors, and stale disconnected chunkservers are removed every minute\n- storage self-healing no longer acts on a storage status query that failed after it printed its counts: such a query was read as \"no chunk missing\" or \"no chunk endangered\", so the checksum repair could rewrite a chunk, a disconnected chunkserver could be removed from the storage master, and the stuck-node reset could free its one-node-at-a-time slot; and a chunkserver is restarted after a write stall only when its node reports every storage pool OK, not when a pool is unknown\n- more storage actions no longer act on a storage status query that failed, printed nothing or stopped short: the removal of the single-node safety copies (Dobby), the automatic removal of a pulled disk, the boot cleanup of chunks written before a restart, `nw removedisk`, and the wait between chunkserver restarts of the MooseFS 3 to 4 upgrade. A missing-chunk count that cannot be read now counts as missing chunks: VM high availability restarts only the VMs whose own disks are whole and defers new deployments and k3s scaling, the dashboard keeps the last missing-chunk alert, and a starting Leader waits for the other nodes. A VM disk whose chunk list stops short now counts as damaged: VM high availability and the fast boot of a single-node cluster do not start that VM (before, the chunks listed until the failure counted as the whole disk)\n- the remaining storage actions and checks that read a storage status query no longer take a query that failed, printed nothing or stopped short for an answer: the removal of the single-node safety copies (Dobby) and the chunkserver mode switch of a node left alone, the write-stall guardian (it keeps an open stall episode instead of reporting every chunkserver healthy, and reports a stall CRITICAL when the missing-chunk count cannot be read), the release of the remediation slot after a node reset (it also needs every storage pool of the node reported OK), the rebalance speed, the chunk version repair after a chunkserver registers, the `/upload` quota and the projected occupancy after a disk is fenced. The chunk and failover checks report UNKNOWN instead of a no-replica warning, \"full HA\" or \"some chunks are missing\", the rebalancing status keeps its last state, and the resiliency snapshot waits. The automatic disk removal now tells this node from a node whose name or address starts the same way (242.32.0.1 and 242.32.0.10), and a starting Leader stops waiting for the other nodes as soon as no chunk is missing.\n- `STATUS_CHUNKSERVER_WRITE_STALL` (the DFS write-stall alert) showed OK on the Leader also when the write-stall detection had stopped running, had not checked anything yet, or its result could not be read; it now shows UNKNOWN, never OK, when the Leader has no result from the last 210 seconds, and for the first 10 minutes after a Leader change, a reboot or a restart of the storage master, while the detection waits and checks nothing. On a single-node cluster, where the detection does not run, it shows OK with \"not evaluated\"\n- `STATUS_CHUNKSERVER_OSCILLATION` (bursts of storage disconnects that point to an unstable network) now reads the system log it could not read before, so it reported OK on every node; when the log cannot be read it now reports UNKNOWN, never OK\n- new `STATUS_CHUNKSERVER_SLOW_WRITES` warning: it names a chunkserver whose writes take 50 ms or more in 3 of the last 5 minutes, with the time of each minute; such a chunkserver slows every VM whose data it holds while it still shows as healthy. It reports UNKNOWN, never OK, when the storage statistics cannot be read\n- the node alerts for a kernel oops, unreadable disk sectors, out of memory, a critical disk target error, a locked update and a `bond0` loss never fired: the node could not read the system log they come from; it now reads it, and when it cannot, the alerts keep their last value and the new `NODE_ALERTS_UNKNOWN` warning names them\n- the `UPDATES_SYNCING` warning no longer appears on a new Leader after a failover: a node that is not the Leader downloaded nothing but still recorded a failed download, and it now records none\n- ZFS: `sync=standard` is now the default only on NVMe pools (SATA and rotational pools stay `sync=disabled` unless `ENABLESYNC=yes`; the last ~2 seconds of writes can be lost on a site-wide power cut), TRIM runs one node at a time, and memory-pressure hardening addresses the pool deadlock seen on nodes whose VMs use nearly all RAM\n- disk health: the nightly verdict now combines SMART with ZFS errors and slow I/O (`DISKxxxx_HEALTH`, a 0-100 risk score), and disks with I/O errors now show on the dashboard\n- VM and image health: images are checked for qcow2 corruption at deploy (warning only), and the new `VM_IO_ERRORS` check finds VMs left with stale disk handles after a storage repair, and reports UNKNOWN, not OK, for a VM it cannot check\n- an unreachable external datastore no longer freezes host and VM monitoring (new `DS_UNREACHABLE` warning) and external NFS datastores are mounted `soft`\n- `/upload` is capped at 80% of the DFS by a hard quota, orphaned image files are reclaimed only when a person runs `nw orphanimagecleaner --cleanup`, never automatically (the `DISABLE_ORPHAN_AUTOCLEAN` parameter is gone), and `nw forceimagecopy` overwrites an image or `/upload` file with a snapshot of another\n- image importer fixes: files uploaded over SMB are imported only after the client has finished writing them, and a finished import folder is no longer re-imported every few minutes\n- Kubernetes persistent volumes (CSI) now follow the default replica goal (`DFS_CSI_GOAL` overrides) instead of the larger system goal\n- the forced repair of damaged Kubernetes persistent volume data after an unclean powerdown (`AUTOREPAIR_K3SPVS`) runs only while the storage of every node is online and every disk serves; until then it waits, with a warning in the logs\n- a Kubernetes service no longer stays in SCALING when one of its VMs is deleted while the service scales (a pending Worker removed to make room for a PRIORITY VM, or a VM deleted by hand): once the service has been in SCALING for 3 minutes (`K3SFLOW_SCALING_GRACE_SECS`), the leader runs `oneflow recover` for it, but not while a PRIORITY VM is UNKNOWN or waits to start (or that cannot be read), so the Worker that OneFlow then deploys does not take the room freed for that VM; before, it stayed in SCALING and the k3s Worker scaler skipped every round until someone ran `oneflow recover` by hand\n- after an unclean powerdown, data whose checksums no longer match it on any copy is no longer turned into zeros or deleted by the automatic repairs: a cluster of two or more nodes compares the copies, keeps the one proven right, and rewrites its checksums, after a backup of every copy (`AUTOREPAIR_DFSCRC`, default on)\n- single-node clusters now default to `BOOTCHECK=ACTIVECHUNKS` (only chunks written just before the shutdown are checked) instead of a full CRC scan that could delay the Leader for hours\n- the boot-time chunk CRC check (`BOOTCHECK`) now runs only on single-node clusters, unless it is set to `FORCE`: a cluster of two or more nodes skips it for every other value, so the storage replaces a damaged copy of a chunk with the good copy from another node instead of the check making the damaged copy look valid\n- memory: the host reclaims unused memory from VMs through their balloons under pressure (`MEMORY_BALLOON`: automatic when VMs are assigned more than 50% of host RAM, `YES`/`NO` to force it); a VM with a PCI passthrough device (GPU, SR-IOV, mediated) is never ballooned, because the device pins all of its memory and the host would gain nothing, and one ballooned before this release gets its memory back at once\n- `TCC_OFFSET_C` (CPU throttles earlier), `COOLING_PERF_MODE` (Lenovo BIOS fan profile) and `RAPL_THERMAL_BACKOFF_PCT` (hold the power limit down while hot) host/cluster attributes, all opt-in; the node card now shows when a node is power- or thermally capped\n- raising a BIOS CPU power limit that is lower than the processor's own (Lenovo ThinkStation/ThinkCentre \"T\" models) is now opt-in with the new `ENABLE_RAPLLIFT` host/cluster attribute, which replaces `DISABLE_RAPLLIFT` (now ignored): without it the lift does not run and a lower BIOS power limit is no longer reported as a critical alert; a node that had its limit raised before the update keeps it until its next reboot; `RAPL_THERMAL_BACKOFF_PCT` works with or without it\n- dynamic CPU power cap on Intel nodes, on by default (`RAPL_POWER_CONTROL`, host/cluster; `NO` or `DISABLED` turns it off and restores the BIOS power limits): the package power limits are raised to the processor's turbo limit on the processor and platform side, then held every 2 seconds just under the temperature at which the CPU throttles itself, so a node runs as fast as its cooling allows without thermal or power-limit warnings; the node card names the cap when it is below 75% of the turbo limit, and limiting at that cap is not reported as a power slowdown\n- split-brain protection hardened (nodes no longer elect a second Leader just because ARP replies are filtered, two Leaders are raised as CRITICAL, an operator election hold survives a reboot); new `FORCE_FOLLOWER` host attribute pins a node as follower, and opt-in `NODE_WATCHDOG` (seconds) resets a node that still answers ping but has stopped running anything\n- self-healing: a hung Orchestrator is restarted automatically (at most once an hour), and libvirtd, syslog and cron are restarted when they stop working\n- before a new Leader starts the orchestrator database, it stops the database servers still running on other nodes; a node where one survives is hard-reset after about 5 minutes, and the database starts only once that node answers again with none running, so a node that does not come back keeps it stopped (`PF2`)\n- `PANIC_ON_OOPS` cluster parameter (`report` by default, `hardreset` reboots a node after a kernel oops), `PROXY_FOR` to publish internal HTTP(S) services through the cluster web frontend, cluster-level `LLDP` and `MEMORY_ADAPTIVE` now apply immediately without a reboot (a host's own value wins), and `VIRTIO_NET_MIGRATION_SAFE` (on by default) keeps VMs live-migratable between 12.4 and 12.5 nodes during a rolling upgrade\n- LLDP is now on by default on every node: set the `LLDP` cluster or host parameter to `NO` to turn it off\n- new node card warnings: a bond negotiating below 1 Gb/s and an upstream switch that binds only one IPv4 address per MAC; the dashboard names nodes visible over multicast but unreachable on the internal network as UNKNOWN, because their cluster membership is not verified; the local node proxy page now shows the node's role and status\n- new debug shell commands: `nw precheck` (pre-flight check for configuration mistakes that only fail later), `nw timeline` (what changed, in order, across orchestrator, kernel and BMC), `nw why-pending` and `nw why-not-migrate` (per-host reasons a VM is not placed or will not migrate), `nw ttrestore` (restore a VM from TimeTurner backups, in place or as a new VM), `nw restoredbfromdump` and `nw saveasformatfixer`\n- `nw tcpdump <interface> [seconds] [tcpdump arguments]` now captures into a file under `/upload`; its second argument is now the duration (it used to be a VLAN tag)\n- new REST calls `nwsetparam`, `nwprecheck`, `nwtimeline`, `nwwhyplacement`, `nwrecommissiondisk`, `nwforceimagecopy`, `nwrestartorchestrator` and `nwrestoredbfromdump`, equivalent to the debug shell commands\n- the web terminal (`TTYD`) now accepts root logins\n- security: SSH on cluster nodes offers a post-quantum hybrid key exchange, SSL/TLS certificates are also refreshed nightly independently of the update package (custom certificates are left untouched), and the REST API is hardened\n- alert and notification emails are now sent from the `nodeweaver.io` domain (was `nodeweaver.eu`)\n- updated DeployVM appliance to 1.3.7\n- a VM that runs on a node but is not in the NodeWeaver VM list is stopped only when its name has the form `one-<number>` (a VM NodeWeaver started); a VM with any other name is left running and reported as a warning (`ZOMBIE_VMS_STATUS`)\n- an IPMI read that fails now shows as unknown, never as \"no alerts\" or \"IPMI is not present\"\n- parameter values are read more strictly: yes/no values are judged as whole words (`none`, `only` or `10` no longer read as yes), `ENABLE_EMMC_DATAPART=no` is honoured, `NODE_FACTORY_RESET` entries must equal a whole node identifier, and the power button only powers a node off when `ACPI_SHUTDOWN` is enabled\n- DNSOps: a node takes its record only from its own name servers, which are authoritative for DNSOps (the public 1.1.1.1 and 8.8.8.8 are no longer asked for it); when they give no answer for one of the node's identifiers, the record of a later identifier is not applied and the settings DNSOps last supplied stay until they answer\n- a node no longer resets itself on high load by default: `LOAD_RESPONSE` (cluster or host) is `report` unless it is set to `hardreset`, which resets only a follower node whose 15-minute load per CPU socket is above `LOAD_THRESHOLD`, never the Leader; both can now be set with `nw setparam`; `LOAD_RESPONSE=kill_vms` is no longer supported and only reports\n- the kernel of every node now records pressure stall information (CPU, memory and I/O), from the first reboot after the update\n- orchestrator database rebuild: an image is moved to `/upload` as unlisted only when the image list of the rebuilt database was read and kept whole; otherwise the images stay in their datastore and a warning is logged\n- cluster geolocation (`GEO_DISCOVERY`) no longer uses an API key: the cluster's location comes from the [IP Geolocation by DB-IP](https://db-ip.com) database shipped with the firmware; it is on unless set to `NO` (removing the setting turns it back on), a failed lookup keeps the last location, and `GEO_TIMEZONE` is now empty\n- REST call `wol`: sends a Wake-on-LAN magic packet to a MAC address from a chosen node and network interface. Only for users in the oneadmin group, one packet a second, and each call is logged\n- REST call `nwvmguestagent`: the new action `exec_status` reads the result of a program that `exec` reported still running\n- REST call `nwredobootstrap`: only for users in the oneadmin group, as the bootstrap VM runs the cluster's `BOOTSTRAP_SCRIPT` as root; `nw redobootstrap` no longer writes the `BOOTSTRAP_SCRIPT` value to the log\n- automatic updates: with `AUTO_UPGRADE=NO` the cluster still downloads its update packages, but it no longer applies them by itself (a downloaded package was applied anyway); `nw upgradenow`, the REST call `nwupgradenow` (now only for users in the oneadmin group) or the Fleet Manager apply it. A package copied to `/upload/updates` is applied also with automatic updates off, as documented (it was not). The cluster downloads only the packages for the versions its nodes run, not every package of the repository; a testing package no longer lands among the stable ones; a download that failed now raises the warning that the updates could not be synchronised. Each node reports whether automatic updates are on (`FW_AUTOUPGRADE`: `on`, `testing`, `off`, or `unknown` when the node cannot see the shared storage)\n- Blitz image copy: an interrupted transfer continues from where it stopped, so an image that needs more than one attempt now arrives; an attempt ends after 5 minutes without data, or after 24 hours (before: after 30 minutes, whatever its progress); while an image is sent, its `BLITZ_PROGRESS` attribute shows the bytes sent, the size and the rate; one copy of an image to a cluster runs at a time\n- `BLITZ_SECRET`: `no`, `off`, `false`, `0` and `disabled` turn Blitz off, as documented (they were refused); a value of fewer than 8 characters, or with a space or a control character, no longer starts Blitz, wherever it was set\n- a parameter value that the node writes back after it applies a change is now kept whole: a value with `=` in it was cut at that `=`, refused at the next monitoring round, and the parameter was then emptied\n- after a cold boot of the whole cluster, a node announces its storage metadata version before its boot waits: a node that had been a follower announced it only after them, so a node with older metadata could see no fresher peer, become Leader, and lose the newer changes; a node also announces its version again when the process that announces it has stopped\n- the console warning for an upstream switch that accepts one IPv4 address per MAC (`Configured IP(s) ... get no ARP reply`) now works: the check ran an `arping` that refused its options, so every node read as unknown; the Leader's own addresses are now tested from another node\n- node watchdog (`NODE_WATCHDOG`, off until set): it now also resets a node whose storage metadata disk has hung, or returned I/O errors, for 5 minutes (or for `NODE_WATCHDOG` seconds, if longer) - except a Follower with another storage disk, which logs it once and keeps running while its metadata moves to the other disk; it is reset if it becomes Leader while the write is still stuck; it arms the kernel software watchdog, which the service now loads, on every node, together with the chipset watchdog when the board has one (Intel iTCO, AMD SP5100, ACPI WDAT), because on some boards the chipset watchdog arms but never resets; it never uses the Intel AMT watchdog, which does not reset the host; a disk probe that takes 10 seconds or more is logged; the start at boot no longer logs an error\n- boot: `cronwatch` and `bondannounce` are no longer in the default runlevel, which started them before `/opt` was mounted and logged `does not exist` and `failed to start` at every boot; like the node watchdog, they start after `/opt` is mounted\n- boot: `bondannounce` no longer warns at every boot that virsh could not list the running VMs; the NIC tuning bounces the active link seconds after it starts, and the same link coming back is no failover - only a move to another link is announced\n- boot: the follower start no longer logs `WARNING: nodewatchdog has already been started` (local.d has started it)\n- leader election: a node that may not become Leader (election held from the setup menu, or refused for a cluster UUID clash) no longer stops every other node from becoming Leader when it has the most recent storage metadata; the other nodes still wait for the rollback hold (`LFO_ROLLBACK_HOLD`) before one of them becomes Leader with older metadata\n- minor fixes"
}