Skip to content

Commit a1a0a85

Browse files
committed
revert(mollifier): use standard REDIS_* fallback and fail loud on misconfig
Two prior changes are reverted: 1. MOLLIFIER_REDIS_HOST (plus _PORT/_USERNAME/_PASSWORD/_TLS_DISABLED) regain their `.transform((v) => v ?? process.env.REDIS_*)` fallback to the main Redis cluster, matching the convention used elsewhere in the codebase for dedicated-cluster env vars. Operators who don't set a dedicated mollifier Redis fall back to the main one — that's the accepted default. 2. getMollifierBuffer() no longer degrades to disabled with a warn log when MOLLIFIER_ENABLED=1 but MOLLIFIER_REDIS_HOST is unset. The buffer initialises normally (falling back to the main Redis if configured), and if that fails the pod crashes loudly. Same for the drainer: initializeMollifierDrainer() throws "env vars inconsistent" if the buffer comes back null, surfacing the misconfig immediately rather than silently leaving entries un-drained. Operationally: silent degradation hides config errors from operators and produces "why are no triggers being mollified?" debugging sessions. Loud failure surfaces the same misconfig at deploy time via the pod's health checks.
1 parent 5610099 commit a1a0a85

3 files changed

Lines changed: 27 additions & 35 deletions

File tree

apps/webapp/app/env.server.ts

Lines changed: 19 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1032,16 +1032,25 @@ const EnvironmentSchema = z
10321032

10331033
MOLLIFIER_ENABLED: z.string().default("0"),
10341034
MOLLIFIER_SHADOW_MODE: z.string().default("0"),
1035-
// No fallback to the main `REDIS_*` cluster. The mollifier writes to a
1036-
// dedicated Redis to keep burst traffic off the engine's primary queue.
1037-
// If `MOLLIFIER_ENABLED=1` but `MOLLIFIER_REDIS_HOST` is unset, the
1038-
// buffer degrades to disabled (with a warn log) rather than silently
1039-
// colocating with the main Redis. See `mollifierBuffer.server.ts`.
1040-
MOLLIFIER_REDIS_HOST: z.string().optional(),
1041-
MOLLIFIER_REDIS_PORT: z.coerce.number().optional(),
1042-
MOLLIFIER_REDIS_USERNAME: z.string().optional(),
1043-
MOLLIFIER_REDIS_PASSWORD: z.string().optional(),
1044-
MOLLIFIER_REDIS_TLS_DISABLED: z.string().default("false"),
1035+
MOLLIFIER_REDIS_HOST: z
1036+
.string()
1037+
.optional()
1038+
.transform((v) => v ?? process.env.REDIS_HOST),
1039+
MOLLIFIER_REDIS_PORT: z.coerce
1040+
.number()
1041+
.optional()
1042+
.transform(
1043+
(v) => v ?? (process.env.REDIS_PORT ? parseInt(process.env.REDIS_PORT) : undefined),
1044+
),
1045+
MOLLIFIER_REDIS_USERNAME: z
1046+
.string()
1047+
.optional()
1048+
.transform((v) => v ?? process.env.REDIS_USERNAME),
1049+
MOLLIFIER_REDIS_PASSWORD: z
1050+
.string()
1051+
.optional()
1052+
.transform((v) => v ?? process.env.REDIS_PASSWORD),
1053+
MOLLIFIER_REDIS_TLS_DISABLED: z.string().default(process.env.REDIS_TLS_DISABLED ?? "false"),
10451054
MOLLIFIER_TRIP_WINDOW_MS: z.coerce.number().int().positive().default(200),
10461055
MOLLIFIER_TRIP_THRESHOLD: z.coerce.number().int().positive().default(100),
10471056
MOLLIFIER_HOLD_MS: z.coerce.number().int().positive().default(500),

apps/webapp/app/v3/mollifier/mollifierBuffer.server.ts

Lines changed: 0 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -26,25 +26,7 @@ function initializeMollifierBuffer(): MollifierBuffer {
2626
});
2727
}
2828

29-
// Latch so we log the degraded-config warning exactly once per process
30-
// instead of on every `getMollifierBuffer()` call (which is per-trigger).
31-
let degradedConfigLogged = false;
32-
3329
export function getMollifierBuffer(): MollifierBuffer | null {
3430
if (env.MOLLIFIER_ENABLED !== "1") return null;
35-
// Fail safe, not loud: if MOLLIFIER_ENABLED was flipped on without
36-
// setting `MOLLIFIER_REDIS_HOST`, degrade the mollifier to disabled
37-
// rather than crash-looping the pod (or — worse — sharing the main
38-
// engine Redis). One warn log per process is enough for operators to
39-
// spot the misconfig without drowning logs in repeats.
40-
if (!env.MOLLIFIER_REDIS_HOST) {
41-
if (!degradedConfigLogged) {
42-
logger.warn(
43-
"mollifier.degraded_config: MOLLIFIER_ENABLED=1 but MOLLIFIER_REDIS_HOST is unset — treating as disabled until configured",
44-
);
45-
degradedConfigLogged = true;
46-
}
47-
return null;
48-
}
4931
return singleton("mollifierBuffer", initializeMollifierBuffer);
5032
}

apps/webapp/app/v3/mollifier/mollifierDrainer.server.ts

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,15 +6,16 @@ import { singleton } from "~/utils/singleton";
66
import { getMollifierBuffer } from "./mollifierBuffer.server";
77
import type { BufferedTriggerPayload } from "./bufferedTriggerPayload.server";
88

9-
function initializeMollifierDrainer(): MollifierDrainer<BufferedTriggerPayload> | null {
9+
function initializeMollifierDrainer(): MollifierDrainer<BufferedTriggerPayload> {
1010
const buffer = getMollifierBuffer();
1111
if (!buffer) {
12-
// Buffer degraded to disabled (e.g. MOLLIFIER_ENABLED=1 but
13-
// MOLLIFIER_REDIS_HOST unset). Don't crash the pod — return null and
14-
// let the worker shutdown registration short-circuit. The degraded
15-
// config is logged once by `getMollifierBuffer()`; we don't double
16-
// log here.
17-
return null;
12+
// Unreachable in normal config: getMollifierDrainer() gates on the
13+
// same env flag as getMollifierBuffer(). If we hit this, fail loud
14+
// — the operator has set MOLLIFIER_ENABLED=1 on a worker pod but
15+
// the buffer can't initialise (e.g. MOLLIFIER_REDIS_HOST resolves
16+
// to nothing). Crashing surfaces the misconfig immediately rather
17+
// than silently leaving entries un-drained.
18+
throw new Error("MollifierDrainer initialised without a buffer — env vars inconsistent");
1819
}
1920

2021
logger.debug("Initializing mollifier drainer", {

0 commit comments

Comments
 (0)