Files
outline/server/utils/MutexLock.ts
T
Tom MoorandClaude Opus 4.8 c8d4e8c0e7 perf: Bound MutexLock retries and de-dupe same-key cache misses (#13110)
* perf: Bound MutexLock retries and de-dupe same-key cache misses

MutexLock configured redlock with retryCount: 120, retryDelay: 1000 — a
2-minute retry window for locks with a 4s default TTL. Under cache-miss
storms (e.g. websocket reconnects all missing user.collectionIds) N
contenders piled onto one lock key, retaining ~K²/2 error objects plus
pending timers/promises. Bound retries to ~the lock TTL (retryCount: 15,
retryDelay: 250) and add a per-process in-flight de-dupe in
CacheHelper.getDataOrSet so N same-key misses share one lock acquisition
and callback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: Return falsy cached values instead of treating them as misses

getDataOrSet checked `if (cache)`, so a cached null/false/0/"" was treated
as a miss and re-populated on every read (e.g. Collection.getCachedDocument
Structure caches NavigationNode[] | null via the AfterSave hook). Check for
undefined — the only value getData returns on a genuine miss — so falsy
cached values are returned without an extra lock and DB query.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* deslop

* fix: Scope bounded lock retries to cache locks only

The tightened retry window (15×250ms) was set on the shared Redlock
instance, so it also affected long-lived callers — notably the 10-minute
"migrations" lock in checkMigrations, which fatal-exits on acquire failure
and would crash concurrent boots. Restore the generous default on the
shared instance and pass a bounded retry override (MutexLock.cacheRetry
Settings) only from the cache path via a new per-acquire retry option.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: Cache falsy callback results in getDataOrSet

populate only cached truthy callback results, so a valid null (e.g. from
Collection.getCachedDocumentStructure) was never stored and every miss
re-ran the lock + DB query. Skip caching only when the result is undefined,
and guard the CacheResult detection against null (typeof null === "object").

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 20:00:54 -04:00

127 lines
3.7 KiB
TypeScript

import Redlock, {
ExecutionError,
ResourceLockedError,
type Lock,
type RedlockAbortSignal,
type Settings,
} from "redlock";
import Redis from "@server/storage/redis";
import Logger from "@server/logging/Logger";
import ShutdownHelper, { ShutdownOrder } from "./ShutdownHelper";
type AcquireOptions = {
/** Whether a lock should be automatically released on server shutdown */
releaseOnShutdown?: boolean;
/** Overrides the default retry settings for this acquisition. */
retry?: Partial<Settings>;
};
/**
* A distributed mutex lock backed by Redis, for coordinating exclusive access
* to resources across processes.
*/
export class MutexLock {
/** Default expiry time for acquiring lock in milliseconds. */
public static defaultLockTimeout = 4000;
/**
* Retry settings for short-lived cache locks, bounded to roughly the lock
* TTL. Contenders re-check the cache after acquiring, so a longer window only
* adds latency and retains memory during cache-miss storms.
*/
public static cacheRetrySettings: Partial<Settings> = {
retryCount: 15,
retryDelay: 250,
};
/**
* Returns the redlock instance
*/
public static get lock(): Redlock {
if (!this.redlock) {
this.redlock = new Redlock([Redis.defaultClient], {
retryJitter: 100,
retryCount: 120,
retryDelay: 1000,
});
this.redlock.on("error", (err) => {
if (err instanceof ResourceLockedError) {
// Expected during lock contention retries, not an error.
return;
} else if (err instanceof ExecutionError) {
Logger.warn("Failed to extend Redlock lock", {
message: err.message,
});
} else {
Logger.error("Unexpected Redlock error", err);
}
});
}
return this.redlock;
}
/**
* Acquire a Mutex lock
*
* @param resource The resource to lock
* @param timeout The duration to acquire the lock for if not released in milliseconds
* @returns A promise that resolves a to a Lock
*/
public static async acquire(
resource: string,
timeout: number,
options?: AcquireOptions
) {
const lock = await this.lock.acquire([resource], timeout, options?.retry);
if (options?.releaseOnShutdown) {
const key = `lock:${resource}`;
// @ts-expect-error Attach resource for use in shutdown
lock._key = key;
ShutdownHelper.add(key, ShutdownOrder.last, lock.release.bind(lock));
}
return lock;
}
/**
* Execute a routine in the context of an auto-extending lock. The lock is
* automatically acquired before the routine runs and released when it
* completes. If the lock cannot be extended, the provided AbortSignal will
* be triggered so the routine can bail out.
*
* @param resource The resource to lock.
* @param timeout The initial lock duration in milliseconds (auto-extended while running).
* @param routine The async routine to execute while holding the lock.
* @returns A promise that resolves with the routine's return value.
*/
public static async using<T>(
resource: string,
timeout: number,
routine: (signal: RedlockAbortSignal) => Promise<T>
): Promise<T> {
return this.lock.using([resource], timeout, routine);
}
/**
* Safely release a lock
*
* @param lock The lock to release
*/
public static release(lock: Lock) {
try {
if (lock && lock.expiration > new Date().getTime()) {
return lock.release();
}
return false;
} finally {
// @ts-expect-error Attach resource for use in shutdown
const key = lock._key;
if (key) {
ShutdownHelper.remove(key);
}
}
}
private static redlock: Redlock;
}