* fix: Fail fast on Redis for cache and rate limits, lock migrations in Postgres
* fix: Use the fail-fast Redis client for locks
* fix: Require a database pool of at least two connections
* chore: Rename cache Redis client to ephemeralClient
* chore: Vendorize throng as ClusterManager
* fix: Force kill workers after grace period, skip fork if shutting down
* fix: Give workers their full shutdown timeout before the primary exits
* chore: Use a named export for ShutdownHelper
* fix: batch large deletes in teamPermanentDeleter to shorten lock time
Delete events, collections, and documents in batches of ids outside the
final transaction so row locks are held briefly instead of for the whole
team deletion, which blocked the hourly old-events cleanup for hours.
* fix: type batched destroy helper structurally
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: guard whole team deletion with a non-blocking mutex
Move the concurrency guard from the final transaction to the whole
routine so a second worker skips before running the batched deletes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix: Make count-based limit checks reliable under concurrency
Limits enforced by counting rows before an insert could be exceeded by
requests that ran at the same time, as each one read the same count
before any of them had written. Add a LockHelper that takes a Postgres
advisory lock for the remainder of the transaction, and use it to
serialize the check for webhook subscriptions, team domains, share
subscriptions and pins. The counts now also run on the transaction so
they include rows written earlier in the same request.
* Remove excessive tests
* Also guard simultaneous row removal
* fix: Unhandled rejection when a Redlock release fails
Releasing a lock is best-effort, a lock that has already expired or been
taken over by another process cannot be released and that is not an error
for the caller. Previously the ExecutionError thrown from release replaced
the successful return value of CacheHelper.populate.
Also catch failures when joining rooms after websocket authentication,
which was a floating promise and so crashed the process.
* fix: Coerce caught values to Error before logging
* perf: Bound MutexLock retries and de-dupe same-key cache misses
MutexLock configured redlock with retryCount: 120, retryDelay: 1000 — a
2-minute retry window for locks with a 4s default TTL. Under cache-miss
storms (e.g. websocket reconnects all missing user.collectionIds) N
contenders piled onto one lock key, retaining ~K²/2 error objects plus
pending timers/promises. Bound retries to ~the lock TTL (retryCount: 15,
retryDelay: 250) and add a per-process in-flight de-dupe in
CacheHelper.getDataOrSet so N same-key misses share one lock acquisition
and callback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: Return falsy cached values instead of treating them as misses
getDataOrSet checked `if (cache)`, so a cached null/false/0/"" was treated
as a miss and re-populated on every read (e.g. Collection.getCachedDocument
Structure caches NavigationNode[] | null via the AfterSave hook). Check for
undefined — the only value getData returns on a genuine miss — so falsy
cached values are returned without an extra lock and DB query.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* deslop
* fix: Scope bounded lock retries to cache locks only
The tightened retry window (15×250ms) was set on the shared Redlock
instance, so it also affected long-lived callers — notably the 10-minute
"migrations" lock in checkMigrations, which fatal-exits on acquire failure
and would crash concurrent boots. Restore the generous default on the
shared instance and pass a bounded retry override (MutexLock.cacheRetry
Settings) only from the cache path via a new per-acquire retry option.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: Cache falsy callback results in getDataOrSet
populate only cached truthy callback results, so a valid null (e.g. from
Collection.getCachedDocumentStructure) was never stored and every miss
re-ran the lock + DB query. Skip caching only when the result is undefined,
and guard the CacheResult detection against null (typeof null === "object").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
ResourceLockedError is emitted on every retry attempt during lock
contention but was not handled, causing it to be logged as an
unexpected error and reported to Sentry (OUTLINE-CLOUD-CAW).
Fixes OUTLINE-CLOUD-CAW
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>