Development Update — July 24

Today’s work is a rescue. For days the fleet’s reward uptime had silently collapsed — hundreds of visors that were plainly online were recording almost no uptime and earning nothing for it — and the outage stayed invisible because every layer that could have shouted about it was logging at Debug or discarding the error entirely. The investigation ran the failure down to two independent causes, both fixed: the reward-critical TPD heartbeat had been quietly disabled when the standalone uptime tracker was decommissioned, and the visors that were heartbeating were losing an httpauth nonce race that 401’d most of their requests. Around those two fixes sits a ring of hardening — the residual TPD memory leaks closed, latent unbounded-timeout and unbounded-map bugs guarded, and the silent-failure logging that hid all of it turned up to Warn. The bottom line for operators: current-fleet visors must update to v1.3.89 to earn rewards correctly — the heartbeat fix is visor-side, and the release raises the auto-incrementing version floor. Also today, the .dmsg/.skynet LAN gateway got a config-safe path so one board can serve a whole home-router LAN.

Skywire: The Heartbeat That Stopped

3563 fix(visor): TPD uptime heartbeat must run without the standalone tracker is the core fix. initUptimeTracker returned early whenever uptime_tracker was unconfigured — and that config is now empty fleet-wide, because the standalone uptime tracker was decommissioned — so the early return silently disabled the TPD /v4/update heartbeat along with it. Post-decommission, that heartbeat is the reward-critical presence signal: with it dead, visors reported uptime only via transport re-registration (whose auth is contended for high-transport visors — see below) and transportless visors reported nothing at all. It stayed invisible because the TPD heartbeat failure was logged at Debug, while the adjacent standalone-tracker failure correctly used Warn plus health flags. The fix decouples the two clients — build the standalone-tracker client only when its addr is set, build the TPD heartbeat client whenever a transport-discovery addr is present (always, in practice), and run whichever exist — and surfaces failure by raising the heartbeat error from Debug to Warn and flipping the isServicesHealthy/isUptimeTrackerHealthy flags, so a persistent 401 or connectivity failure is now visible in the visor’s own logs and /health. 3567 test(visor): regression-guard the reward-uptime decouple in CI extracts the URL-decision logic into a pure resolveUptimeTargets helper and unit-tests the exact invariant that broke: when uptime_tracker is nil, the TPD heartbeat must still be targeted. 3564 fix(tpd): log reward-heartbeat store failures instead of dropping them silently closes the same silent-failure class on the store side — RecordHeartbeat and RecordTransportHeartbeat are the writes that persist the whole fleet’s reward-uptime data, and both returned pipe.Exec errors with no logging while every caller discarded the error on the stated assumption that “the store layer logs.” It didn’t. Now both log at Warn on error, making that comment true and giving one central, visible signal across every path.

Skywire: The Nonce Storm

3570 fix(httpauth): share the nonce per (addr, PK) to stop concurrent-request 401s is the second, independent cause. httpauth’s server keeps a single monotonic nonce per PK, advanced on each success — but a visor process legitimately holds several httpauth clients to the same TPD as the same identity: its utclient (uptime heartbeat) and its tpdclient (transport registration) both authenticate there. Each client had its own nonce counter and its own request mutex, so two of them could pass verifyAuth at nonce N concurrently, the server would INCR twice to N+2 while each client advanced only to N+1, and every subsequent request 401’d until a resync that kept losing under load. That is the reward-uptime suppression: high-transport visors, whose registration traffic overlaps the heartbeat, thrashed worst, so their /v4/update heartbeats mostly 401’d and their recorded uptime collapsed to near-zero despite being online — roughly 290 of 517 online visors observed in that state. The fix moves the nonce counter and request serialization into a nonceState shared by all clients for the same (sanitized addr, PK) via a small package registry, so a process serializes its own requests to a server and never races its own nonce. It’s contained to pkg/httpauthclient — no server change, no interface change — and ships with a -race regression test that fires many concurrent requests from two clients sharing an identity and asserts zero nonce mismatches.

Skywire: The Ring of Hardening

With the two live causes fixed, the investigation swept up the latent bugs of the same shape. 3562 fix(tpd): cap aggregator fill time to release hung-fill memory finishes the TPD memory story from #3561: the residual leak was in-flight fills, not superseded Roots — when a peer flaps mid-fill the fill hangs, pinning all its wanted objects in the cache where the LRU can’t evict them, and CXO’s default MaxFillingTime is 10 minutes, so with hundreds of unstable peers that window is a large standing pool of hung-fill memory. Capping the aggregator’s fill time at 90s uses the existing tested timeout path and is safe because the aggregator is best-effort — a fill still running after 90s is a broken peer, not a slow one. 3571 fix(cxo): treat non-positive fill/response timeouts as default, not disabled guards the mechanism itself: two CXO timeouts silently disabled themselves at <= 0 instead of falling back to a default, each pinning a per-peer resource until the connection closed — a footgun, not a feature — so a non-positive value now uses the package default. 3565 fix(visor): bound the remaining unbounded stun.ready waits bounds two <-v.stun.ready receives that had no timeout sibling (one now on the reward-heartbeat client’s init path, where a stuck STUN would wedge heartbeat-client construction) at 20s, matching the existing pattern. 3566 fix(visor/stats): prune day-start bandwidth baselines closes an unbounded map of the same category as the CXO leak — a baselines entry was added per transport UUID and never deleted, growing without bound under transport churn — now pruned in runRetention by dropping any baseline whose day isn’t today.

Skywire: The .dmsg / .skynet LAN Gateway

A second gateway path landed alongside the transparent mesh gateway: use the existing resolving proxy, but let one board serve a whole LAN through an ordinary home router. 3572 feat(resolver): configurable SOCKS5 bind host so a board can serve the LAN adds a proxy_addr knob to the embedded dmsgweb/skynetweb proxies (skywire cli resolver), which had always bound 127.0.0.1, so only the board itself could use them — with --dmsgweb-addr 0.0.0.0 the SOCKS5 listener binds a LAN address and any device on the network reaches dmsg-only deployment services (the reward system, TPD, and the rest) with no per-device skywire install; loopback stays the default and a non-loopback bind is opt-in and documented as trusted-network-only. 3576 feat(autoconfig): DMSGWEBADDR/SKYNETWEBADDR + rework LAN-gateway guide completes the chain by adding those env knobs to skywire.conf/autoconfig, so the LAN bind is set the autoconfig-safe way that survives a restart and future skywire autoconfig runs — unlike a direct skywire.json edit, which autoconfig overwrites on Linux. 3574 docs(guides): .dmsg gateway for a home router (OpenWRT / DD-WRT) documents the whole setup — binding the resolver to the LAN, per-device SOCKS5 on both router firmwares, browser WPAD/PAC auto-config, and the trusted-network caveat.

Skywire: Release

3575 chore(vendor): bump skycoin to develop HEAD vendors the current skycoin for the release — only embedded explorer web files change, no Go code. 3577 chore(changelog): v1.3.89 cuts the release: reward uptime restored via the visor-side heartbeat decouple and the nonce fix, the TPD memory leaks closed, and the LAN .dmsg/.skynet gateway — 15 PRs since v1.3.88. Because the heartbeat fix lives in the visor and the release raises the auto-incrementing version floor, the current fleet needs to update to v1.3.89 to record reward uptime correctly — a visor left on an older build will keep failing quietly, which is exactly the outage this release ends.