16 Commits
Author SHA1 Message Date
Tom Moor eff1a43d9b fix: Keep serving requests when Redis is unavailable (#14457)
* fix: Fail fast on Redis for cache and rate limits, lock migrations in Postgres

* fix: Use the fail-fast Redis client for locks

* fix: Require a database pool of at least two connections

* chore: Rename cache Redis client to ephemeralClient
2026-11-03 08:45:56 -05:00
Tom Moor fc8ccd80fd chore: Replace redlock with a single-node Redis lock (#14212)
* chore: Replace redlock with a single-node Redis lock

* fix: Handle lost SET replies, late replies and stalled extensions in MutexLock
2026-10-25 22:58:59 -04:00
Tom Moor 2679093af3 chore: Vendorize throng as ClusterManager (#14198)
* chore: Vendorize throng as ClusterManager

* fix: Force kill workers after grace period, skip fork if shutting down

* fix: Give workers their full shutdown timeout before the primary exits

* chore: Use a named export for ShutdownHelper
2026-10-21 18:20:03 -04:00
Tom MoorandClaude Fable 5.1 b2a1e10e48 Batch large table deletes in teamPermanentDeleter (#13610)
* fix: batch large deletes in teamPermanentDeleter to shorten lock time

Delete events, collections, and documents in batches of ids outside the
final transaction so row locks are held briefly instead of for the whole
team deletion, which blocked the hourly old-events cleanup for hours.

* fix: type batched destroy helper structurally

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix: guard whole team deletion with a non-blocking mutex

Move the concurrency guard from the final transaction to the whole
routine so a second worker skips before running the batched deletes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-02 06:12:12 -04:00
Tom Moor aab83e8378 fix: Reliability of count-based limit checks (#13542)
* fix: Make count-based limit checks reliable under concurrency

Limits enforced by counting rows before an insert could be exceeded by
requests that ran at the same time, as each one read the same count
before any of them had written. Add a LockHelper that takes a Postgres
advisory lock for the remainder of the transaction, and use it to
serialize the check for webhook subscriptions, team domains, share
subscriptions and pins. The counts now also run on the transaction so
they include rows written earlier in the same request.

* Remove excessive tests

* Also guard simultaneous row removal
2026-08-24 19:14:55 -04:00
Tom Moor 11ef69e381 fix: Unhandled rejection when a Redlock lock release fails (#13316)
* fix: Unhandled rejection when a Redlock release fails

Releasing a lock is best-effort, a lock that has already expired or been
taken over by another process cannot be released and that is not an error
for the caller. Previously the ExecutionError thrown from release replaced
the successful return value of CacheHelper.populate.

Also catch failures when joining rooms after websocket authentication,
which was a floating promise and so crashed the process.

* fix: Coerce caught values to Error before logging
2026-08-06 00:52:26 +00:00
Tom MoorandClaude Opus 4.8 c8d4e8c0e7 perf: Bound MutexLock retries and de-dupe same-key cache misses (#13110)
* perf: Bound MutexLock retries and de-dupe same-key cache misses

MutexLock configured redlock with retryCount: 120, retryDelay: 1000 — a
2-minute retry window for locks with a 4s default TTL. Under cache-miss
storms (e.g. websocket reconnects all missing user.collectionIds) N
contenders piled onto one lock key, retaining ~K²/2 error objects plus
pending timers/promises. Bound retries to ~the lock TTL (retryCount: 15,
retryDelay: 250) and add a per-process in-flight de-dupe in
CacheHelper.getDataOrSet so N same-key misses share one lock acquisition
and callback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: Return falsy cached values instead of treating them as misses

getDataOrSet checked `if (cache)`, so a cached null/false/0/"" was treated
as a miss and re-populated on every read (e.g. Collection.getCachedDocument
Structure caches NavigationNode[] | null via the AfterSave hook). Check for
undefined — the only value getData returns on a genuine miss — so falsy
cached values are returned without an extra lock and DB query.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* deslop

* fix: Scope bounded lock retries to cache locks only

The tightened retry window (15×250ms) was set on the shared Redlock
instance, so it also affected long-lived callers — notably the 10-minute
"migrations" lock in checkMigrations, which fatal-exits on acquire failure
and would crash concurrent boots. Restore the generous default on the
shared instance and pass a bounded retry override (MutexLock.cacheRetry
Settings) only from the cache path via a new per-acquire retry option.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: Cache falsy callback results in getDataOrSet

populate only cached truthy callback results, so a valid null (e.g. from
Collection.getCachedDocumentStructure) was never stored and every miss
re-ran the lock + DB query. Skip caching only when the result is undefined,
and guard the CacheResult detection against null (typeof null === "object").

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 20:00:54 -04:00
Tom MoorandClaude Fable 5 a1d0520c9f Add missing JSDoc to exported members in app and server utils (#12940)
* Add missing JSDoc to exported members in app and server utils

Documents all exported functions, classes, enums, and constants in
app/utils and server/utils that were missing JSDoc, per the project
documentation guidelines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AsEQYA8zN813thxu7GjjzU

* Use @returns tag to match codebase convention

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AsEQYA8zN813thxu7GjjzU

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-08 08:30:11 -04:00
Tom MoorandClaude Opus 4.6 b494f64c4e fix: Silence expected ResourceLockedError during Redlock retries (#12049)
ResourceLockedError is emitted on every retry attempt during lock
contention but was not handled, causing it to be logged as an
unexpected error and reported to Sentry (OUTLINE-CLOUD-CAW).

Fixes OUTLINE-CLOUD-CAW

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-15 08:04:31 -04:00
Tom Moor 51ba02715f chore: Add missing error handler on MutexLock (#12021)
* chore: Add missing error handler on MutexLock

* PR feedback
2026-04-13 18:12:21 -04:00
Tom Moor 0f45778d79 perf: Protect against ValidateSSOAccessTask thundering herd (#11532)
* perf: Protect against many tabs reloading at once

* PR feedback
2026-02-23 21:05:01 -05:00
Tom Moor fbd4ded5b4 feat: Add authentication provider management (#10997)
* Gemini first-pass

* Prevent post-connect login

* stash

* stash

* Add OIDC logo

* Separate security page

* test

* Update icon

* test

* ui

* Add extra guards for disabling auth provider

* refactor

* test
2025-12-24 09:09:24 -05:00
Tom MoorandCopilot 621409ae0b fix: Ensure shutdown with db migration lock correctly releases (#10817)
* fix: Ensure unsafe shutdown with db migration lock correctly releases

* Update server/utils/MutexLock.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-06 13:43:00 -05:00
Tom Moor 89425ccdab chore: Add a mutex lock around migrations to ensure in multi-instance deployments multiple machines don't attempt to run migrations at once (#10560) 2025-11-04 17:01:44 -05:00
Tom Moor 8a3a3453e7 fix: The operation was unable to achieve a quorum during its retry window 2024-11-27 23:04:37 -05:00
Tom Moor fd5391cbb6 Cache diff generation for email notifications (#7987)
* Cache diff generation, closes #7982

* Handle cannot acquire lock

* Refactor to guard
2024-11-20 14:45:12 -08:00