diff --git a/README.md b/README.md index fccae36..157d276 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,14 @@ http://АДРЕС-GATEWAY:3456 Приватные и локальные адреса не отправляются в VPN, поэтому устройства сохраняют доступ к домашней сети. Общий прокси по умолчанию принимает подключения только из приватных сетей. +### Резервный канал Gateway + +После добавления подписок откройте «Резерв» — вторую кнопку в правой панели. Выберите основной и резервный серверы (они могут быть из одной или разных подписок), сервисы для проверки и отдельный таймаут каждого сервиса. Там же настраиваются длительность сбоя и восстановления, порог активного трафика, период тишины и защита от частых переключений. + +При включении Harbor заранее проверяет dual-конфигурацию. Если VPN остановлен, она начнёт работать только после следующего обычного нажатия питания; сохранение само VPN не включает. Переключение меняет маршрут только для новых соединений — уже открытые соединения не закрываются. Если через VPN идёт активный трафик или его активность нельзя надёжно определить, Harbor ждёт и показывает скорость, число передающих соединений и безопасные подписи основных блокирующих потоков. + +Выключенный резерв полностью пассивен: Harbor не запускает проверки, таймер выбора и отдельный подсчёт активности. Если dual-конфигурация уже загружена, отключение не перезапускает VPN и не меняет текущий маршрут; обычный stop и следующий запуск вернут single-channel config. Последние важные события — включение VPN, обновления подписок, переключения и ошибки — доступны в последней кнопке «Журнал» и хранятся 30 дней без ссылок подписок и сырых диагностических ответов. + ### Устройства Gateway Откройте «Устройства» в правой панели Gateway — подписка для просмотра списка не требуется. Harbor раз в 15 секунд читает локальную таблицу соседей и показывает каждое устройство одной компактной строкой: заданное название, hostname или IP, последний контакт, выбранный график трафика и иконку применённого маршрута. По умолчанию график показывает приблизительный выход `VPN`/`Direct`; переключатель `Вход` возвращает накопленную разбивку `Gateway`/`Прокси`. Наведите курсор на имя или переведите на него фокус, чтобы открыть IP, MAC и доступный hostname; нажатие на значение копирует его. Hostname определяется через локальное обратное разрешение имён и может отсутствовать, если сеть его не публикует. Технические interface и manufacturer продолжают храниться для идентификации, но не занимают место в строке. Список разделён на «Закреплённые», «Остальные» и «Фоновые»: последняя группа сохраняется между перезапусками, показывает только identity/presence и кнопку возврата без графика, traffic и route controls. Название, закрепление, фоновое положение и накопленные totals сохраняются в volume Gateway, пока устройство остаётся в inventory. diff --git a/docs/product/application-state.md b/docs/product/application-state.md index 8665a2b..072d982 100644 --- a/docs/product/application-state.md +++ b/docs/product/application-state.md @@ -1,6 +1,6 @@ # Harbor application state v1 -`GET /api/state` is the canonical Harbor domain snapshot. Successful mutations return the same snapshot as `state`. The persisted owner is `state.json` schema v6; React keeps only drafts, disclosure, focus, animation and transport freshness. +`GET /api/state` is the canonical Harbor domain snapshot. Successful mutations return the same snapshot as `state`. The persisted owner is `state.json` schema v8; React keeps only drafts, disclosure, focus, animation and transport freshness. An abbreviated snapshot: @@ -67,6 +67,8 @@ The frontend accepts only newer snapshots. Equal revisions preserve object ident Browser boot/offline/stale state remains a transport envelope beside the domain snapshot. A transport failure retains the last accepted domain state. +Failover health and traffic observations are transient: they do not write `state.json` or increase the domain `revision` every few seconds. Each control-process lifetime publishes a new `observationEpoch` and increasing `observationSequence`. At the same domain revision the browser accepts only a newer sequence from the active epoch; after accepting a new epoch it retires the old one so a late response cannot restore stale health. + ## Desired and applied identity `desiredProfileId` and each profile's `desiredServerId` record the next local choice. `appliedProfileId`, `appliedServerId` and `appliedServerSnapshot` describe the runtime that actually owns traffic. There is no third `activeProfileId`. @@ -97,12 +99,22 @@ The route-rules mutation uses the whole-array `PUT /api/route-rules/v2` with `ru In Connect `gateway-direct`, local user rules are intentionally omitted and the snapshot reports no active or pending local rules. Gateway device policy `Напрямую` bypasses sing-box before these rules; policy `VPN` and an ordinary local/Gateway VPN pipeline evaluate them. +## Gateway failover and activity journal + +`failoverPolicy` is the desired Gateway-only policy: master enable, primary/reserve profile and server, service checks with individual timeouts, health windows, active-traffic guard and flap protection. `failoverRuntimeState` stores switch history, hold and quarantine deadlines separately, so a runtime decision is not mistaken for a desired configuration change. `appliedFailoverPolicy` stores only the two loaded targets and safe configuration fingerprints. + +Enabling failover while VPN is stopped validates a temporary dual-channel candidate but does not start VPN. The dual config is loaded only by the next explicit power-on. Enabling it over a running single-channel config remains pending until a later stop and power-on. Disabling automation stops its timer, probes and activity collector immediately, but does not restart sing-box or change the selected route; the already loaded dual config is reported as `passive-loaded` until the ordinary stop lifecycle clears it. + +The dual config keeps one stable inbound and a sing-box selector with `interrupt_exist_connections: false`. A switch changes the outbound for new connections only. Before an automatic switch, the existing `/connections` observer measures VPN byte deltas over a bounded 10-second window. Active or unknown traffic blocks the switch; the public snapshot contains only aggregate speed, connection count and at most three safe device/service labels. + +Failover mutations use `PUT /api/failover`, `POST /api/failover/pause` and `POST /api/failover/switch`. Important user events are stored separately in `activity-journal.json` and read through `GET /api/activity-journal`. The journal is not a second state owner, contains no provider URLs or raw diagnostics, uses stable ID cursors and prunes entries after 30 days. + ## Compatibility and migration Schema v5 migrates the legacy singleton and `subscription-cache.json` into one profile named `Основной`. Stable endpoint identity preserves unambiguous desired/applied selection, including transport variants whose normalized IDs differ from old labels. An explicitly stopped legacy state does not resurrect an old applied target. -Schema v6 adds the routing-rule outbound. Rules read from schemas v0-v5 migrate to `outbound: "direct"` in their existing order and both desired/applied arrays are normalized together. A schema-v6 rule without a valid outbound is rejected rather than silently rewritten. +Schema v6 adds the routing-rule outbound. Rules read from schemas v0-v5 migrate to `outbound: "direct"` in their existing order and both desired/applied arrays are normalized together. A schema-v6 rule without a valid outbound is rejected rather than silently rewritten. Schema v7 adds canonical connectivity-diagnostics settings. Schema v8 adds a disabled failover policy, empty runtime history and no applied dual config, so upgrading does not start monitoring or change traffic. Migration atomically backs up the previous `state.json`. After the embedded profile is committed, Harbor also backs up and removes the legacy subscription cache so there is one persisted owner. Invalid legacy cache/config returns to a truthful stopped first-run state instead of starting stale generated config. -The old HTTP projection remains bounded for one release. Schema v6 persistence is not downgrade-compatible: stop Harbor and restore the `state.json.backup-v-*` matching the rollback binary instead of deploying old code over v6 data. Rolling back before profiles still also requires the matching legacy subscription-cache backup. +The old HTTP projection remains bounded for one release. Schema v8 persistence is not downgrade-compatible: stop Harbor and restore the `state.json.backup-v-*` matching the rollback binary instead of deploying old code over v8 data. Rolling back before profiles still also requires the matching legacy subscription-cache backup. diff --git a/docs/recovery/state-recovery.md b/docs/recovery/state-recovery.md index a5c75ff..9f5bf07 100644 --- a/docs/recovery/state-recovery.md +++ b/docs/recovery/state-recovery.md @@ -1,6 +1,6 @@ # Harbor state recovery -Harbor keeps the existing data directory and `state.json` path. The current persisted format is `schemaVersion: 6`: schema v2 introduced local route rules, v3 added rule enabled state, v4 added stable server IDs, v5 embeds the canonical `profiles[]` collection with desired/applied profile identity, and v6 adds an explicit `vpn` or `direct` outbound to every route rule. +Harbor keeps the existing data directory and `state.json` path. The current persisted format is `schemaVersion: 8`: schema v2 introduced local route rules, v3 added rule enabled state, v4 added stable server IDs, v5 embeds the canonical `profiles[]` collection with desired/applied profile identity, v6 adds an explicit `vpn` or `direct` outbound to every route rule, v7 stores connectivity-diagnostics settings, and v8 adds Gateway failover state. ## Atomic writes @@ -36,6 +36,12 @@ state.json.backup-v5-2026-08-17T12-00-00-000Z After migration, malformed schema-v6 rules are rejected; Harbor does not reinterpret a missing or unknown outbound as direct. +## Migration to failover + +Schemas v0-v7 migrate to v8 with failover disabled, empty switch history and no applied dual config. Migration does not start probes, enable traffic accounting or change the single-channel runtime. The original state is preserved as `state.json.backup-v-*` before the atomic replacement. + +The separate `activity-journal.json` is created on the first important event. It uses the same atomic write and corrupt-file isolation mechanism as state, retains at most 30 days, and can be removed while Harbor is stopped without affecting subscriptions, routing or VPN startup. + ## Corrupt JSON If `state.json` cannot be parsed, Harbor renames the exact damaged bytes to: @@ -55,4 +61,4 @@ Perform recovery while Harbor is stopped: 3. Restore only matching state/cache backups to their original filenames. 4. Start Harbor and verify `GET /api/state` before applying a profile. -A pre-v6 binary cannot interpret the explicit ordered VPN/Direct rule contract. Restore `state.json.backup-v-*` matching the rollback binary; deploying old code over schema v6 is not safe. A rollback to pre-v5 additionally requires the matching state and subscription-cache backups because that binary cannot interpret canonical profiles. +A pre-v8 binary cannot interpret failover state. Restore `state.json.backup-v-*` matching the rollback binary; deploying old code over schema v8 is not safe. A rollback to pre-v5 additionally requires the matching state and subscription-cache backups because that binary cannot interpret canonical profiles. diff --git a/src/server/config.ts b/src/server/config.ts index 2068571..2cc78a0 100644 --- a/src/server/config.ts +++ b/src/server/config.ts @@ -15,6 +15,8 @@ export const settings = { port: parsePort(process.env.PORT, 3456), proxyPort, diagnosticsProxyPort: parsePort(process.env.DIAGNOSTICS_PROXY_PORT, 18080), + failoverPrimaryProxyPort: parsePort(process.env.FAILOVER_PRIMARY_PROXY_PORT, 18081), + failoverReserveProxyPort: parsePort(process.env.FAILOVER_RESERVE_PROXY_PORT, 18082), singboxApiPort: parsePort(process.env.SING_BOX_API_PORT, 19090), tproxyPort: parsePort(process.env.TPROXY_PORT, 7895), tproxyMark: process.env.TPROXY_MARK || "1", @@ -40,6 +42,7 @@ export const settings = { cachePath: process.env.SING_BOX_CACHE || "/var/lib/sing-box/cache.db", statePath: path.join(dataDir, "state.json"), deviceStatePath: path.join(dataDir, "devices.json"), + activityJournalPath: path.join(dataDir, "activity-journal.json"), subscriptionCachePath: path.join(dataDir, "subscription-cache.json"), sharedProxyHost: process.env.SHARED_PROXY_HOST || "", hostNetworkStatePath: diff --git a/src/server/dataplane.ts b/src/server/dataplane.ts index 26f41c1..f7f3d43 100644 --- a/src/server/dataplane.ts +++ b/src/server/dataplane.ts @@ -13,6 +13,7 @@ import { createDomainTrafficService, readSingboxConnections, } from './services/domainTrafficService.js'; +import { createSingboxSelectorService } from './services/singboxSelectorService.js'; const socketPath = settings.dataplaneSocket; const runtime = createSingboxRuntime({ @@ -40,6 +41,11 @@ const devicePolicy = createDevicePolicyService({ const connectivityDiagnostics = createConnectivityDiagnosticsService({ proxyPort: settings.diagnosticsProxyPort, }); +const failoverDiagnostics = { + primary: createConnectivityDiagnosticsService({ proxyPort: settings.failoverPrimaryProxyPort }), + reserve: createConnectivityDiagnosticsService({ proxyPort: settings.failoverReserveProxyPort }), +}; +const selector = createSingboxSelectorService({ port: settings.singboxApiPort }); const domainTraffic = createDomainTrafficService({ observe: () => readSingboxConnections(settings.singboxApiPort), devices: () => traffic.snapshot().devices, @@ -126,6 +132,33 @@ const server = http.createServer(async (req: IncomingMessage, res: ServerRespons target, })); } + if (req.method === 'POST' && req.url === '/failover/probe') { + const { role, services = [], target = null, timeoutMs = 6_000 } = record(await readJson(req)); + if (role !== 'primary' && role !== 'reserve') throw new Error('Неизвестная failover role'); + return sendJson(res, 200, await failoverDiagnostics[role].runVpn({ services, target, timeoutMs: Number(timeoutMs) })); + } + if (req.method === 'GET' && req.url === '/failover/selector') { + return sendJson(res, 200, await selector.read()); + } + if (req.method === 'PUT' && req.url === '/failover/selector') { + const { role } = record(await readJson(req)); + if (role !== 'primary' && role !== 'reserve') throw new Error('Неизвестная failover role'); + return sendJson(res, 200, await selector.select(role)); + } + if (req.method === 'PUT' && req.url === '/failover/activity') { + const { enabled } = record(await readJson(req)); + if (enabled === true) domainTraffic.enableActivity(); + else domainTraffic.disableActivity(); + return sendJson(res, 200, { enabled: enabled === true }); + } + if (req.method === 'POST' && req.url === '/failover/activity/read') { + const { thresholdBytesPerSecond = 0 } = record(await readJson(req)); + return sendJson(res, 200, { activity: domainTraffic.activitySnapshot(thresholdBytesPerSecond) }); + } + if (req.method === 'POST' && req.url === '/config/check') { + const { config } = record(await readJson(req)); + return sendJson(res, 200, runtime.checkConfig(config)); + } if (req.method === 'POST' && req.url === '/apply') { return sendJson(res, 200, await runtime.apply()); } diff --git a/src/server/dataplaneClient.ts b/src/server/dataplaneClient.ts index 402e771..a06e8ad 100644 --- a/src/server/dataplaneClient.ts +++ b/src/server/dataplaneClient.ts @@ -82,6 +82,16 @@ export function createDataplaneClient(socketPath: string, send: SendDataplaneReq throw new HarborError('DIAGNOSTICS_FAILED', { cause }); } }, + checkConfig: (config: unknown) => send(socketPath, '/config/check', 'POST', { config }, 15_000), + runFailoverProbe: (role: 'primary' | 'reserve', services: unknown, target: unknown, timeoutMs: number) => ( + send(socketPath, '/failover/probe', 'POST', { role, services, target, timeoutMs }, timeoutMs + 10_000) + ), + readFailoverSelector: () => send(socketPath, '/failover/selector', 'GET'), + selectFailoverRole: (role: 'primary' | 'reserve') => send(socketPath, '/failover/selector', 'PUT', { role }), + setFailoverActivityEnabled: (enabled: boolean) => send(socketPath, '/failover/activity', 'PUT', { enabled }), + readFailoverActivity: (thresholdBytesPerSecond: number) => ( + send(socketPath, '/failover/activity/read', 'POST', { thresholdBytesPerSecond }) + ), apply: () => update('/apply', 'POST'), restart: () => update('/restart', 'POST'), stop: () => update('/stop', 'POST'), diff --git a/src/server/features/connection/connectionService.ts b/src/server/features/connection/connectionService.ts index afed66b..7923c21 100644 --- a/src/server/features/connection/connectionService.ts +++ b/src/server/features/connection/connectionService.ts @@ -4,7 +4,9 @@ import { type StoredState, } from '../../../shared/contracts/state.js'; import { HarborError } from '../../../shared/errors.js'; -import { finishRollback } from '../../services/rollback.js'; +import { finishRollback, type RollbackStep } from '../../services/rollback.js'; +import type { AppliedFailoverPolicy } from '../../../shared/failover.js'; +import type { ActivityJournalEventInput } from '../../../shared/activityJournal.js'; interface ConnectionServiceDependencies { state: { @@ -26,6 +28,18 @@ interface ConnectionServiceDependencies { restartCommand(): Promise; }; route?: { isGatewayDirect(): boolean }; + failover?: { + build(state: StoredState, source?: 'desired' | 'applied'): { + config: unknown; + applied: AppliedFailoverPolicy; + primaryProfile: StoredProfile; + primaryServer: StoredProfile['servers'][number]; + }; + prepareActivation(role: 'primary' | 'reserve'): Promise; + restoreAppliedActivation(state: StoredState): Promise; + reconcile(): Promise; + }; + onEvent?: (event: ActivityJournalEventInput) => void; serialize(operation: () => Promise): Promise; now(): Date; } @@ -72,6 +86,33 @@ function withDesiredServer(state: StoredState, profile: StoredProfile, serverId: } export function createConnectionService(dependencies: ConnectionServiceDependencies) { + const activationTarget = ( + state: StoredState, + applied: AppliedFailoverPolicy, + role: 'primary' | 'reserve', + ) => { + const target = applied[role]; + const profile = profileById(state, target.profileId); + const server = profile?.servers.find(({ id }) => id === target.serverId); + if (!profile || !server) throw new HarborError('SERVER_NOT_FOUND'); + return { profile, server }; + }; + const finishConnectionRollback = async (error: unknown, steps: RollbackStep[], message: string) => { + try { + await finishRollback(error, steps, message); + } catch (cause) { + const code = cause && typeof cause === 'object' && 'code' in cause + && /^[A-Z0-9_]{1,50}$/.test(String(cause.code)) ? String(cause.code) : 'UNKNOWN'; + dependencies.onEvent?.({ + type: 'connection.failed', + severity: 'error', + source: 'connection', + dedupeKey: `connection.failed:${dependencies.state.read().revision}:${code}`, + data: { errorCode: code }, + }); + throw cause; + } + }; const applyWithinQueue = async ( previousState: StoredState, profile: StoredProfile, @@ -88,18 +129,26 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc if (!selectedServer) throw new HarborError('SERVER_NOT_FOUND'); if (!profile.subscriptionConfig) throw new HarborError('CONFIG_INVALID'); + const wasRunning = await dependencies.runtime.isRunning(); + if (wasRunning && previousState.failoverPolicy?.enabled) { + dependencies.state.update((state) => withDesiredServer(state, profile, selectedServer.id)); + return { profileId: profile.id, serverId: selectedServer.id, selectedTag: selectedServer.label }; + } + if (dependencies.route?.isGatewayDirect()) { dependencies.state.update((state) => withDesiredServer(state, profile, selectedServer.id)); return { profileId: profile.id, serverId: selectedServer.id, selectedTag: selectedServer.label }; } - const nextConfig = dependencies.config.build( + const failoverCandidate = previousState.failoverPolicy?.enabled + ? dependencies.failover?.build(previousState, wasRunning ? 'applied' : 'desired') + : null; + const nextConfig = failoverCandidate?.config || dependencies.config.build( profile.subscriptionConfig, selectedServer.id, previousState.routeRules, ); const previousConfig = dependencies.config.read(); - const wasRunning = await dependencies.runtime.isRunning(); let configMutationStarted = false; let runtimeMutationStarted = false; let stateCommitStarted = false; @@ -109,18 +158,20 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc dependencies.config.write(nextConfig); runtimeMutationStarted = true; await dependencies.runtime.start(); + if (failoverCandidate) await dependencies.failover!.prepareActivation('primary'); stateCommitStarted = true; dependencies.state.update((state) => ({ ...withDesiredServer(state, profile, selectedServer.id), connectionDesired: 'running', - appliedProfileId: profile.id, - appliedServerId: selectedServer.id, - appliedServerSnapshot: selectedServer, + appliedProfileId: failoverCandidate?.primaryProfile.id || profile.id, + appliedServerId: failoverCandidate?.primaryServer.id || selectedServer.id, + appliedServerSnapshot: failoverCandidate?.primaryServer || selectedServer, + appliedFailoverPolicy: failoverCandidate?.applied || null, appliedAt: dependencies.now().toISOString(), appliedRouteRules: state.routeRules, })); } catch (error) { - await finishRollback(error, [ + await finishConnectionRollback(error, [ ...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []), ...(configMutationStarted ? [{ run: () => previousConfig === null @@ -128,12 +179,28 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc : dependencies.config.restore(previousConfig), }] : []), ...(runtimeMutationStarted ? [{ - run: () => wasRunning ? dependencies.runtime.start() : dependencies.runtime.stop(), + run: async () => { + if (!wasRunning) return dependencies.runtime.stop(); + await dependencies.runtime.start(); + await dependencies.failover?.restoreAppliedActivation(previousState); + }, runtime: true, }] : []), ], 'Connection rollback failed'); } + await dependencies.failover?.reconcile(); + dependencies.onEvent?.({ + type: 'connection.started', + severity: 'info', + source: 'connection', + dedupeKey: `connection.started:${dependencies.state.read().revision}`, + data: { + profileLabel: failoverCandidate?.primaryProfile.label || profile.label, + serverLabel: failoverCandidate?.primaryServer.label || selectedServer.label, + }, + }); + return { profileId: profile.id, serverId: selectedServer.id, selectedTag: selectedServer.label }; }; @@ -186,16 +253,29 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc appliedProfileId: '', appliedServerId: '', appliedServerSnapshot: null, + appliedFailoverPolicy: null, })); } catch (error) { - await finishRollback(error, [ + await finishConnectionRollback(error, [ ...(runtimeMutationStarted && wasRunning !== null ? [{ - run: () => wasRunning ? dependencies.runtime.start() : dependencies.runtime.stop(), + run: async () => { + if (!wasRunning) return dependencies.runtime.stop(); + await dependencies.runtime.start(); + await dependencies.failover?.restoreAppliedActivation(previousState); + }, runtime: true, }] : []), ...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []), ], 'Connection rollback failed'); } + await dependencies.failover?.reconcile(); + dependencies.onEvent?.({ + type: 'connection.stopped', + severity: 'info', + source: 'connection', + dedupeKey: `connection.stopped:${dependencies.state.read().revision}`, + data: {}, + }); }); const restart = () => dependencies.serialize(async () => { @@ -213,7 +293,18 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc ? previousState.appliedServerSnapshot : null); if (!server || !profile.subscriptionConfig) throw new HarborError('CONFIG_INVALID'); - const candidateConfig = dependencies.config.build( + const failoverCandidate = previousState.failoverPolicy?.enabled + ? dependencies.failover?.build(previousState, wasRunning ? 'applied' : 'desired') + : null; + const activationRole = failoverCandidate && wasRunning + && previousState.appliedProfileId === failoverCandidate.applied.reserve.profileId + && previousState.appliedServerId === failoverCandidate.applied.reserve.serverId + ? 'reserve' as const + : 'primary' as const; + const failoverTarget = failoverCandidate + ? activationTarget(previousState, failoverCandidate.applied, activationRole) + : null; + const candidateConfig = failoverCandidate?.config || dependencies.config.build( profile.subscriptionConfig, server.id, previousState.routeRules, @@ -229,30 +320,47 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc const command = await dependencies.runtime.restartCommand(); runtimeMutationStarted = command.mutationStarted; if (!command.ok) throw command.error; + if (failoverCandidate) await dependencies.failover!.prepareActivation(activationRole); stateCommitStarted = true; dependencies.state.update((state) => ({ ...state, desiredProfileId: wasRunning ? state.desiredProfileId : profile.id, - appliedProfileId: profile.id, - appliedServerId: server.id, - appliedServerSnapshot: server, + appliedProfileId: failoverTarget?.profile.id || profile.id, + appliedServerId: failoverTarget?.server.id || server.id, + appliedServerSnapshot: failoverTarget?.server || server, + appliedFailoverPolicy: failoverCandidate?.applied || null, connectionDesired: 'running', appliedRouteRules: dependencies.route?.isGatewayDirect() ? [] : state.routeRules, })); } catch (error) { - await finishRollback(error, [ + await finishConnectionRollback(error, [ ...(configMutationStarted ? [{ run: () => previousConfig === null ? dependencies.config.remove() : dependencies.config.restore(previousConfig), }] : []), ...(runtimeMutationStarted ? [{ - run: () => wasRunning ? dependencies.runtime.start() : dependencies.runtime.stop(), + run: async () => { + if (!wasRunning) return dependencies.runtime.stop(); + await dependencies.runtime.start(); + await dependencies.failover?.restoreAppliedActivation(previousState); + }, runtime: true, }] : []), ...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []), ], 'Connection rollback failed'); } + await dependencies.failover?.reconcile(); + dependencies.onEvent?.({ + type: 'connection.started', + severity: 'info', + source: 'connection', + dedupeKey: `connection.started:${dependencies.state.read().revision}`, + data: { + profileLabel: failoverTarget?.profile.label || profile.label, + serverLabel: failoverTarget?.server.label || server.label, + }, + }); }); return { apply, activate, stop, restart }; diff --git a/src/server/features/failover/failoverService.ts b/src/server/features/failover/failoverService.ts new file mode 100644 index 0000000..218328a --- /dev/null +++ b/src/server/features/failover/failoverService.ts @@ -0,0 +1,671 @@ +import crypto from 'node:crypto'; +import { + createIdleFailoverSnapshot, + isFailoverConfigured, + nextFailoverDecision, + normalizeFailoverPolicy, + type AppliedFailoverPolicy, + type FailoverDecisionMemory, + type FailoverHealth, + type FailoverPolicy, + type FailoverRole, + type FailoverSnapshot, +} from '../../../shared/failover.js'; +import type { HarborServer, StoredState } from '../../../shared/contracts/state.js'; +import type { ActivityJournalEventInput } from '../../../shared/activityJournal.js'; +import { HarborError } from '../../../shared/errors.js'; + +interface Candidate { + config: unknown; + applied: AppliedFailoverPolicy; +} + +interface FailoverServiceDependencies { + state: { + read(): StoredState; + update(mutator: (state: StoredState) => Record): StoredState; + }; + runtime: { isRunning(): Promise }; + dataplane: { + checkConfig(config: unknown): Promise; + runFailoverProbe(role: FailoverRole, services: unknown, target: string, timeoutMs: number): Promise; + readFailoverSelector(): Promise; + selectFailoverRole(role: FailoverRole): Promise; + setFailoverActivityEnabled(enabled: boolean): Promise; + readFailoverActivity(thresholdBytesPerSecond: number): Promise; + }; + buildCandidate(state: StoredState): Candidate; + serialize(operation: () => Promise): Promise; + scheduler?: { + setTimeout(callback: () => void, intervalMs: number): NodeJS.Timeout; + clearTimeout(timer: NodeJS.Timeout): void; + }; + now?: () => Date; + onWarning?: (error: unknown) => void; + onSwitch?: (from: FailoverRole, to: FailoverRole, reason: string) => void; + onEvent?: (event: ActivityJournalEventInput) => void; +} + +const record = (value: unknown): Record => ( + value && typeof value === 'object' && !Array.isArray(value) ? value as Record : {} +); + +function targetServer(state: StoredState, role: FailoverRole): HarborServer | null { + const target = state.appliedFailoverPolicy?.[role] || state.failoverPolicy[role]; + return state.profiles.find(({ id }) => id === target.profileId) + ?.servers.find(({ id }) => id === target.serverId) || null; +} + +function currentRole(state: StoredState): FailoverRole | null { + const applied = state.appliedFailoverPolicy; + if (!applied) return null; + for (const role of ['primary', 'reserve'] as const) { + if ( + state.appliedProfileId === applied[role].profileId + && state.appliedServerId === applied[role].serverId + ) return role; + } + return null; +} + +function probeHealth(value: unknown): boolean { + const vpn = record(record(value).vpn); + const sites = Array.isArray(vpn.sites) ? vpn.sites.map(record) : []; + return sites.length === 1 && sites[0].status === 'available'; +} + +function safeErrorCode(error: unknown) { + const code = error && typeof error === 'object' && 'code' in error ? String(error.code) : ''; + return /^[A-Z0-9_]{1,50}$/.test(code) ? code : 'UNKNOWN'; +} + +export function createFailoverService(dependencies: FailoverServiceDependencies) { + const scheduler = dependencies.scheduler || { + setTimeout: (callback: () => void, intervalMs: number) => setTimeout(callback, intervalMs), + clearTimeout: (timer: NodeJS.Timeout) => clearTimeout(timer), + }; + const now = dependencies.now || (() => new Date()); + const epoch = crypto.randomUUID(); + let sequence = 0; + let generation = 0; + let timer: NodeJS.Timeout | null = null; + let collectorEnabled: boolean | null = null; + let roundPromise: Promise | null = null; + let decisionMemory: FailoverDecisionMemory | undefined; + let snapshot = createIdleFailoverSnapshot(dependencies.state.read().failoverPolicy, epoch, sequence); + + function appliedMatchesDesired(state: StoredState) { + if (!state.appliedFailoverPolicy) return false; + try { + return JSON.stringify(dependencies.buildCandidate(state).applied) + === JSON.stringify(state.appliedFailoverPolicy); + } catch { + return false; + } + } + + function publish(next: FailoverSnapshot) { + sequence += 1; + snapshot = { ...next, observationEpoch: epoch, observationSequence: sequence }; + } + + function clearTimer() { + if (timer) scheduler.clearTimeout(timer); + timer = null; + } + + function schedule(delay: number) { + clearTimer(); + timer = scheduler.setTimeout(() => { + timer = null; + void runRound().catch(dependencies.onWarning); + }, delay); + timer.unref?.(); + } + + async function disableCollector() { + if (collectorEnabled === false) return; + await dependencies.dataplane.setFailoverActivityEnabled(false); + collectorEnabled = false; + } + + async function deactivate(policy: FailoverPolicy, passiveRole: FailoverRole | null = null) { + generation += 1; + clearTimer(); + decisionMemory = undefined; + await disableCollector(); + const idle = createIdleFailoverSnapshot(policy, epoch, sequence); + if (passiveRole) { + idle.activation = 'passive-loaded'; + idle.currentRole = passiveRole; + idle.reason = 'disabled'; + } + publish(idle); + } + + function activeSnapshot(state: StoredState, status: FailoverSnapshot['status'] = 'observing'): FailoverSnapshot { + const role = state.failoverRuntimeState.reasonCode === 'selector-unknown' ? null : currentRole(state); + const channel = (target: typeof state.failoverPolicy.primary) => ({ + target, + health: 'unknown' as FailoverHealth, + failingServiceIds: [], + checkedAt: null, + stateSince: null, + }); + return { + observationEpoch: epoch, + observationSequence: sequence, + configured: isFailoverConfigured(state.failoverPolicy), + enabled: state.failoverPolicy.enabled, + paused: state.failoverPolicy.paused, + activation: appliedMatchesDesired(state) ? 'active' : 'pending', + currentRole: role || 'other', + status, + primary: channel(state.appliedFailoverPolicy?.primary || state.failoverPolicy.primary), + reserve: channel(state.appliedFailoverPolicy?.reserve || state.failoverPolicy.reserve), + nextDecisionAt: null, + reason: null, + trafficActivity: null, + policy: state.failoverPolicy, + }; + } + + async function reconcile() { + let state = dependencies.state.read(); + const policy = state.failoverPolicy; + if (!policy.enabled) { + const role = currentRole(state); + const passiveRole = role && await dependencies.runtime.isRunning() ? role : null; + return deactivate(policy, passiveRole); + } + const running = await dependencies.runtime.isRunning(); + if (!running || !state.appliedFailoverPolicy || !currentRole(state)) { + generation += 1; + clearTimer(); + await disableCollector(); + const pending = activeSnapshot(state, 'idle'); + pending.activation = running ? 'pending' : 'inactive'; + pending.reason = running ? 'pending-activation' : 'vpn-stopped'; + publish(pending); + return; + } + const role = currentRole(state)!; + const selected = record(await dependencies.dataplane.readFailoverSelector()); + if (selected.role !== role) await dependencies.dataplane.selectFailoverRole(role); + if (state.failoverRuntimeState.reasonCode === 'selector-unknown') { + state = dependencies.state.update((current) => ({ + ...current, + failoverRuntimeState: { ...current.failoverRuntimeState, reasonCode: null }, + })); + } + if (collectorEnabled !== true) { + await dependencies.dataplane.setFailoverActivityEnabled(true); + collectorEnabled = true; + } + generation += 1; + const active = activeSnapshot(state); + if (active.activation === 'pending') active.reason = 'pending-activation'; + publish(active); + schedule(0); + } + + async function reconcileAfterCommit() { + try { + await reconcile(); + } catch (error) { + dependencies.onWarning?.(error); + const failed = activeSnapshot(dependencies.state.read(), 'error'); + failed.reason = 'reconcile-failed'; + publish(failed); + } + } + + async function assessRole(role: FailoverRole, policy: FailoverPolicy) { + const custom = dependencies.state.read().diagnostics.customServices; + const results = await Promise.all(policy.checks.map(async (check) => { + try { + return { + id: check.serviceId, + ok: probeHealth(await dependencies.dataplane.runFailoverProbe( + role, + custom, + `site:${check.serviceId}`, + check.timeoutMs, + )), + }; + } catch { + return { id: check.serviceId, ok: null }; + } + })); + return { + health: results.some(({ ok }) => ok === null) + ? 'unknown' as const + : results.every(({ ok }) => ok) ? 'healthy' as const : 'unhealthy' as const, + failingServiceIds: results.filter(({ ok }) => ok === false).map(({ id }) => id), + }; + } + + async function switchWithinQueue(role: FailoverRole, reason: string) { + const before = dependencies.state.read(); + const from = currentRole(before); + if (!from || from === role) return; + try { + await dependencies.dataplane.selectFailoverRole(role); + const server = targetServer(before, role); + if (!server) throw new HarborError('SERVER_NOT_FOUND'); + const target = before.appliedFailoverPolicy![role]; + const switchedAt = now().toISOString(); + const cutoff = now().getTime() - before.failoverPolicy.flapProtection.windowMs; + const history = role === 'reserve' + ? [...before.failoverRuntimeState.failoverHistory.filter((value) => Date.parse(value) >= cutoff), switchedAt] + : before.failoverRuntimeState.failoverHistory.filter((value) => Date.parse(value) >= cutoff); + const quarantine = history.length >= before.failoverPolicy.flapProtection.count + ? new Date(now().getTime() + before.failoverPolicy.flapProtection.quarantineMs).toISOString() + : before.failoverRuntimeState.primaryQuarantineUntil; + dependencies.state.update((state) => ({ + ...state, + appliedProfileId: target.profileId, + appliedServerId: target.serverId, + appliedServerSnapshot: server, + failoverPolicy: reason === 'manual' + ? { ...state.failoverPolicy, paused: true } + : role === 'primary' ? { ...state.failoverPolicy, paused: false } : state.failoverPolicy, + failoverRuntimeState: { + ...state.failoverRuntimeState, + lastSwitchAt: switchedAt, + holdUntil: role === 'reserve' + ? new Date(now().getTime() + state.failoverPolicy.minimumReserveMs).toISOString() + : null, + primaryQuarantineUntil: quarantine, + failoverHistory: history, + reasonCode: reason, + }, + })); + } catch (error) { + try { + await dependencies.dataplane.selectFailoverRole(from); + } catch (rollback) { + generation += 1; + clearTimer(); + decisionMemory = undefined; + dependencies.state.update((state) => ({ + ...state, + failoverPolicy: { ...state.failoverPolicy, paused: true }, + failoverRuntimeState: { ...state.failoverRuntimeState, reasonCode: 'selector-unknown' }, + })); + const failed = activeSnapshot(dependencies.state.read(), 'error'); + failed.reason = 'selector-unknown'; + publish(failed); + throw new AggregateError([error, rollback], 'Failover selector rollback failed'); + } + throw error; + } + dependencies.onSwitch?.(from, role, reason); + dependencies.onEvent?.({ + type: 'failover.switched', + severity: 'info', + source: 'failover', + dedupeKey: `failover.switched:${dependencies.state.read().revision}`, + data: { + fromRole: from, + toRole: role, + primaryLabel: targetServer(dependencies.state.read(), 'primary')?.label || 'Primary', + reserveLabel: targetServer(dependencies.state.read(), 'reserve')?.label || 'Reserve', + reason, + manual: reason === 'manual', + }, + }); + } + + async function performRound(capturedGeneration: number) { + const prepared = await dependencies.serialize(async () => { + const state = dependencies.state.read(); + const role = currentRole(state); + if ( + capturedGeneration !== generation + || !state.failoverPolicy.enabled + || !state.appliedFailoverPolicy + || !role + ) return null; + const selected = record(await dependencies.dataplane.readFailoverSelector()); + if (selected.role !== role) await dependencies.dataplane.selectFailoverRole(role); + await dependencies.dataplane.setFailoverActivityEnabled(true); + collectorEnabled = true; + const latest = dependencies.state.read(); + return capturedGeneration === generation + && latest.failoverPolicy.enabled + && currentRole(latest) === role + ? { state: latest, policy: latest.failoverPolicy, role } + : null; + }); + if (!prepared) return; + const { state, policy, role } = prepared; + const checkedAt = now().toISOString(); + let primary; + let reserve; + try { + [primary, reserve] = await Promise.all([ + assessRole('primary', policy), + assessRole('reserve', policy), + ]); + } catch { + primary = { health: 'unknown' as const, failingServiceIds: [] }; + reserve = { health: 'unknown' as const, failingServiceIds: [] }; + } + if (capturedGeneration !== generation || !dependencies.state.read().failoverPolicy.enabled) return; + const activityResponse = record(await dependencies.dataplane.readFailoverActivity( + policy.trafficGuard.thresholdBytesPerSecond, + )); + if (capturedGeneration !== generation || !dependencies.state.read().failoverPolicy.enabled) return; + const activity = record(activityResponse.activity); + const observedAt = typeof activity.observedAt === 'string' ? Date.parse(activity.observedAt) : NaN; + const activityState = Number.isFinite(observedAt) && now().getTime() - observedAt <= 4_000 + && (activity.state === 'active' || activity.state === 'quiet') + ? activity.state + : 'unknown'; + const decision = nextFailoverDecision({ + now: now().getTime(), + policy, + currentRole: role, + primaryHealth: primary.health, + reserveHealth: reserve.health, + activity: activityState, + holdUntil: Date.parse(state.failoverRuntimeState.holdUntil || '') || null, + primaryQuarantineUntil: Date.parse(state.failoverRuntimeState.primaryQuarantineUntil || '') || null, + memory: decisionMemory, + }); + decisionMemory = decision.memory; + const previousSnapshot = snapshot; + const next = activeSnapshot(state, decision.status); + next.currentRole = role; + next.primary = { + ...next.primary, + ...primary, + checkedAt, + stateSince: previousSnapshot.primary.health === primary.health + ? previousSnapshot.primary.stateSince || checkedAt + : checkedAt, + }; + next.reserve = { + ...next.reserve, + ...reserve, + checkedAt, + stateSince: previousSnapshot.reserve.health === reserve.health + ? previousSnapshot.reserve.stateSince || checkedAt + : checkedAt, + }; + next.reason = decision.reason; + next.nextDecisionAt = decision.nextDecisionAt ? new Date(decision.nextDecisionAt).toISOString() : null; + next.trafficActivity = activityState === 'unknown' ? { + state: 'unknown', + observedAt: Number.isFinite(observedAt) ? new Date(observedAt).toISOString() : checkedAt, + windowMs: 10_000, + thresholdBytesPerSecond: policy.trafficGuard.thresholdBytesPerSecond, + totalBytesPerSecond: 0, + transmittingConnections: 0, + quietSince: null, + switchTarget: decision.switchTo, + blockers: [], + } : { + state: activityState, + observedAt: String(activity.observedAt), + windowMs: Number(activity.windowMs) || 10_000, + thresholdBytesPerSecond: policy.trafficGuard.thresholdBytesPerSecond, + totalBytesPerSecond: Number(activity.totalBytesPerSecond) || 0, + transmittingConnections: Number(activity.transmittingConnections) || 0, + quietSince: typeof activity.quietSince === 'string' ? activity.quietSince : null, + switchTarget: decision.switchTo, + blockers: (Array.isArray(activity.blockers) ? activity.blockers : []).slice(0, 3) as FailoverSnapshot['trafficActivity'] extends infer T ? T extends { blockers: infer B } ? B : never : never, + }; + publish(next); + if (decision.status === 'waiting-for-idle' && previousSnapshot.status !== 'waiting-for-idle') { + dependencies.onEvent?.({ + type: 'failover.waiting_for_idle', + severity: 'info', + source: 'failover', + dedupeKey: `failover.waiting_for_idle:${state.revision}:${role}:${decision.reason}`, + data: { fromRole: role, toRole: decision.switchTo || (role === 'primary' ? 'reserve' : 'primary'), reason: decision.reason }, + }); + } + if (decision.reason === 'both-unhealthy' && previousSnapshot.reason !== 'both-unhealthy') { + dependencies.onEvent?.({ + type: 'failover.both_unhealthy', + severity: 'warning', + source: 'failover', + dedupeKey: `failover.both_unhealthy:${state.revision}`, + data: { reason: decision.reason }, + }); + } + if (primary.health === 'healthy' && previousSnapshot.primary.health === 'unhealthy') { + dependencies.onEvent?.({ + type: 'failover.recovered', + severity: 'info', + source: 'failover', + dedupeKey: `failover.recovered:${state.revision}:primary`, + data: { role: 'primary', reason: decision.reason }, + }); + } + if (decision.switchTo) { + try { + const switched = await dependencies.serialize(async () => { + const current = dependencies.state.read(); + if ( + capturedGeneration !== generation + || !current.failoverPolicy.enabled + || current.failoverPolicy.paused + || currentRole(current) !== role + || !current.appliedFailoverPolicy + ) return false; + let freshPrimary; + let freshReserve; + try { + [freshPrimary, freshReserve] = await Promise.all([ + assessRole('primary', current.failoverPolicy), + assessRole('reserve', current.failoverPolicy), + ]); + } catch { + return false; + } + const healthStillAllowsSwitch = decision.switchTo === 'reserve' + ? freshPrimary.health === 'unhealthy' && freshReserve.health === 'healthy' + : freshPrimary.health === 'healthy'; + if (!healthStillAllowsSwitch || capturedGeneration !== generation) return false; + if (current.failoverPolicy.trafficGuard.enabled) { + const freshResponse = record(await dependencies.dataplane.readFailoverActivity( + current.failoverPolicy.trafficGuard.thresholdBytesPerSecond, + )); + const freshActivity = record(freshResponse.activity); + const freshObservedAt = typeof freshActivity.observedAt === 'string' + ? Date.parse(freshActivity.observedAt) + : NaN; + const freshQuietSince = typeof freshActivity.quietSince === 'string' + ? Date.parse(freshActivity.quietSince) + : NaN; + if ( + capturedGeneration !== generation + || freshActivity.state !== 'quiet' + || !Number.isFinite(freshObservedAt) + || now().getTime() - freshObservedAt > 4_000 + || !Number.isFinite(freshQuietSince) + || now().getTime() - freshQuietSince < current.failoverPolicy.trafficGuard.quietWindowMs + ) return false; + } + await switchWithinQueue(decision.switchTo!, decision.reason); + return true; + }); + if (!switched) { + const cancelled = activeSnapshot(dependencies.state.read(), 'observing'); + cancelled.reason = 'revalidation-required'; + publish(cancelled); + decisionMemory = undefined; + return; + } + } catch (error) { + const failed = activeSnapshot(dependencies.state.read(), 'error'); + failed.reason = dependencies.state.read().failoverRuntimeState.reasonCode === 'selector-unknown' + ? 'selector-unknown' + : 'switch-failed'; + publish(failed); + dependencies.onEvent?.({ + type: 'failover.switch_failed', + severity: 'error', + source: 'failover', + dedupeKey: `failover.switch_failed:${state.revision}:${role}:${decision.switchTo}`, + data: { fromRole: role, toRole: decision.switchTo, reason: decision.reason, errorCode: safeErrorCode(error) }, + }); + throw error; + } + if (capturedGeneration !== generation) return; + publish(activeSnapshot(dependencies.state.read(), decision.switchTo === 'reserve' ? 'reserve' : 'primary')); + decisionMemory = undefined; + } + } + + async function runRound() { + if (roundPromise) return roundPromise; + const capturedGeneration = generation; + roundPromise = performRound(capturedGeneration).finally(() => { + roundPromise = null; + const policy = dependencies.state.read().failoverPolicy; + if (capturedGeneration === generation && policy.enabled && dependencies.state.read().appliedFailoverPolicy) { + const decisionAt = snapshot.nextDecisionAt ? Date.parse(snapshot.nextDecisionAt) : NaN; + const decisionDelay = Number.isFinite(decisionAt) + ? Math.max(250, decisionAt - now().getTime()) + : policy.intervalMs; + schedule(Math.min(policy.intervalMs, decisionDelay)); + } + }); + return roundPromise; + } + + function save(value: unknown) { + return dependencies.serialize(async () => { + let policy: FailoverPolicy; + try { + policy = normalizeFailoverPolicy(value, { strict: true }); + } catch (cause) { + throw new HarborError('REQUEST_INVALID', { cause }); + } + if (policy.enabled && !isFailoverConfigured(policy)) throw new HarborError('REQUEST_INVALID'); + const before = dependencies.state.read(); + const candidateState = { ...before, failoverPolicy: policy }; + if (policy.enabled) { + const candidate = dependencies.buildCandidate(candidateState); + await dependencies.dataplane.checkConfig(candidate.config); + } + dependencies.state.update((state) => { + const role = before.failoverPolicy.enabled && !policy.enabled ? currentRole(state) : null; + const target = role ? state.appliedFailoverPolicy?.[role] : null; + return { + ...state, + failoverPolicy: policy, + ...(target ? { + desiredProfileId: target.profileId, + profiles: state.profiles.map((profile) => profile.id === target.profileId + ? { ...profile, desiredServerId: target.serverId } + : profile), + } : {}), + }; + }); + decisionMemory = undefined; + if (before.failoverPolicy.enabled !== policy.enabled) { + dependencies.onEvent?.({ + type: policy.enabled ? 'failover.enabled' : 'failover.disabled', + severity: 'info', + source: 'failover', + dedupeKey: `failover.${policy.enabled ? 'enabled' : 'disabled'}:${dependencies.state.read().revision}`, + data: {}, + }); + } + await reconcileAfterCommit(); + }); + } + + function pause(paused: boolean) { + return dependencies.serialize(async () => { + const before = dependencies.state.read(); + if (!paused && ( + !before.appliedFailoverPolicy + || !targetServer(before, 'primary') + || !targetServer(before, 'reserve') + )) throw new HarborError('REQUEST_INVALID'); + dependencies.state.update((state) => ({ + ...state, + failoverPolicy: { ...state.failoverPolicy, paused }, + })); + decisionMemory = undefined; + dependencies.onEvent?.({ + type: paused ? 'failover.paused' : 'failover.resumed', + severity: 'info', + source: 'failover', + dedupeKey: `failover.${paused ? 'paused' : 'resumed'}:${dependencies.state.read().revision}`, + data: {}, + }); + await reconcileAfterCommit(); + }); + } + + async function manualSwitch(role: FailoverRole) { + await dependencies.serialize(async () => { + const state = dependencies.state.read(); + if (!state.failoverPolicy.enabled || !state.appliedFailoverPolicy || !currentRole(state)) { + throw new HarborError('REQUEST_INVALID'); + } + await switchWithinQueue(role, 'manual'); + }); + await reconcileAfterCommit(); + } + + const prepareActivation = (role: FailoverRole) => dependencies.dataplane.selectFailoverRole(role); + + async function restoreAppliedActivation(state: StoredState) { + if (!state.appliedFailoverPolicy) return; + const role = currentRole(state); + if (!role) throw new HarborError('CONFIG_INVALID'); + await prepareActivation(role); + } + + function checkNow() { + return dependencies.serialize(async () => { + const state = dependencies.state.read(); + if (!state.failoverPolicy.enabled || !state.appliedFailoverPolicy || !currentRole(state)) { + throw new HarborError('REQUEST_INVALID'); + } + generation += 1; + clearTimer(); + const checkedAt = now().toISOString(); + let primary; + let reserve; + try { + [primary, reserve] = await Promise.all([ + assessRole('primary', state.failoverPolicy), + assessRole('reserve', state.failoverPolicy), + ]); + } catch { + primary = { health: 'unknown' as const, failingServiceIds: [] }; + reserve = { health: 'unknown' as const, failingServiceIds: [] }; + } + const next = activeSnapshot(dependencies.state.read(), 'observing'); + next.primary = { ...next.primary, ...primary, checkedAt, stateSince: checkedAt }; + next.reserve = { ...next.reserve, ...reserve, checkedAt, stateSince: checkedAt }; + next.reason = 'manual-check'; + publish(next); + schedule(state.failoverPolicy.intervalMs); + }); + } + + return { + snapshot: () => snapshot, + save, + pause, + manualSwitch, + checkNow, + prepareActivation, + restoreAppliedActivation, + reconcile, + runRound, + shutdown: () => deactivate(dependencies.state.read().failoverPolicy), + }; +} + +export type FailoverService = ReturnType; diff --git a/src/server/features/routing/routeRulesService.ts b/src/server/features/routing/routeRulesService.ts index e771914..9a8d5b5 100644 --- a/src/server/features/routing/routeRulesService.ts +++ b/src/server/features/routing/routeRulesService.ts @@ -35,6 +35,8 @@ interface RouteRulesDependencies { route?: { isGatewayDirect(): boolean }; serialize(operation: () => Promise): Promise; runOperation(operation: () => Promise): Promise; + afterApply?: () => Promise; + restoreAppliedActivation?: (state: StoredState) => Promise; } export function createRouteRulesService(dependencies: RouteRulesDependencies) { @@ -96,6 +98,7 @@ export function createRouteRulesService(dependencies: RouteRulesDependencies) { ...(wasRunning ? { appliedRouteRules: routeRules } : {}), routeRulesRevision: state.routeRulesRevision + 1, })); + await dependencies.afterApply?.(); } catch (error) { await finishRollback(error, [ ...(configMutationStarted ? [{ @@ -104,7 +107,10 @@ export function createRouteRulesService(dependencies: RouteRulesDependencies) { : dependencies.config.restore(previousConfig), }] : []), ...(wasRunning && runtimeMutationStarted ? [{ - run: () => dependencies.runtime.restoreRunning(), + run: async () => { + await dependencies.runtime.restoreRunning(); + await dependencies.restoreAppliedActivation?.(previousState); + }, runtime: true, }] : []), ...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []), diff --git a/src/server/features/state/stateService.ts b/src/server/features/state/stateService.ts index 22cb285..cd74e46 100644 --- a/src/server/features/state/stateService.ts +++ b/src/server/features/state/stateService.ts @@ -6,6 +6,7 @@ import { type StateSnapshot, type StoredState, } from '../../../shared/contracts/state.js'; +import type { FailoverSnapshot } from '../../../shared/failover.js'; interface RuntimeState { running?: boolean; @@ -26,6 +27,7 @@ interface StateServiceDependencies { getGatewayAutoState: () => GatewayAutoState; getOperationState: () => OperationState; configExists: () => boolean; + getFailoverSnapshot?: () => FailoverSnapshot; } function subscriptionHost(value: unknown) { @@ -51,6 +53,7 @@ export function createStateService(dependencies: StateServiceDependencies) { configExists, subscriptionHost: subscriptionHost(storedState.subscriptionUrl), operation: dependencies.getOperationState(), + failoverSnapshot: dependencies.getFailoverSnapshot?.(), }); return { snapshot, storedState, gatewayAuto, configExists }; }, diff --git a/src/server/features/subscription/subscriptionService.ts b/src/server/features/subscription/subscriptionService.ts index 967a453..8fe48c7 100644 --- a/src/server/features/subscription/subscriptionService.ts +++ b/src/server/features/subscription/subscriptionService.ts @@ -8,6 +8,7 @@ import { type StoredState, } from '../../../shared/contracts/state.js'; import { HarborError } from '../../../shared/errors.js'; +import type { ActivityJournalEventInput } from '../../../shared/activityJournal.js'; import { finishRollback } from '../../services/rollback.js'; interface ParsedSubscription { @@ -55,6 +56,11 @@ interface SubscriptionServiceDependencies { clearInterval(handle: TimerHandle): void; }; onRefreshError(error: unknown): void; + onEvent?: (event: ActivityJournalEventInput) => void; + failover?: { + reconcile(): Promise; + restoreAppliedActivation(state: StoredState): Promise; + }; now?: () => Date; } @@ -106,6 +112,10 @@ function mutationResult(profile: Pick): ProfileMu return { success: true, profileId: profile.id, label: profile.label }; } +function publicHost(url: string) { + try { return new URL(url).hostname; } catch { return ''; } +} + export function createSubscriptionService(dependencies: SubscriptionServiceDependencies) { const refreshPromises = new Map>(); let refreshTimer: TimerHandle | null = null; @@ -166,6 +176,13 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen profiles: [...current.profiles, profile], desiredProfileId: current.profiles.length ? current.desiredProfileId : profile.id, })); + dependencies.onEvent?.({ + type: 'subscription.added', + severity: 'info', + source: 'subscription', + dedupeKey: `subscription.added:${dependencies.state.read().revision}`, + data: { profileId: profile.id, profileLabel: profile.label, host: publicHost(profile.subscriptionUrl), serverCount: profile.servers.length }, + }); return mutationResult(profile); }); }; @@ -217,6 +234,7 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen profileId: string, subscriptionUrl: string, error: unknown, + origin: 'manual' | 'scheduled', ) => dependencies.serialize(async () => { const state = dependencies.state.read(); const profile = requireProfile(state, profileId); @@ -230,12 +248,27 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen ...current, profiles: replaceProfile(current, failed), })); + dependencies.onEvent?.({ + type: 'subscription.refresh_failed', + severity: 'warning', + source: 'subscription', + dedupeKey: origin === 'scheduled' + ? `subscription.refresh_failed:${failed.id}:${failed.fetchedAt || 'never'}:${failed.lastRefreshErrorCode}` + : `subscription.refresh_failed:${dependencies.state.read().revision}`, + data: { + profileId: failed.id, + profileLabel: failed.label, + host: publicHost(failed.subscriptionUrl), + errorCode: failed.lastRefreshErrorCode || 'UNKNOWN', + }, + }); }); const commitRefresh = ( profileId: string, subscriptionUrl: string, parsed: ParsedSubscription, + origin: 'manual' | 'scheduled', ) => dependencies.serialize(async () => { // Re-read the profile after provider I/O and guard its owner instead of rejecting background-only revisions. const previousState = dependencies.state.read(); @@ -257,8 +290,46 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen lastRefreshAttemptAt: parsed.fetchedAt, lastRefreshErrorCode: null, }; + const contentChanged = JSON.stringify({ + subscriptionConfig: previousProfile.subscriptionConfig, + servers: previousProfile.servers, + userInfo: previousProfile.userInfo, + }) !== JSON.stringify({ + subscriptionConfig: refreshedProfile.subscriptionConfig, + servers: refreshedProfile.servers, + userInfo: refreshedProfile.userInfo, + }); + const appendRefreshEvent = () => { + if (origin === 'scheduled' && !contentChanged && !previousProfile.lastRefreshErrorCode) return; + dependencies.onEvent?.({ + type: 'subscription.refreshed', + severity: 'info', + source: 'subscription', + dedupeKey: `subscription.refreshed:${dependencies.state.read().revision}`, + data: { + profileId: refreshedProfile.id, + profileLabel: refreshedProfile.label, + host: publicHost(refreshedProfile.subscriptionUrl), + serverCount: refreshedProfile.servers.length, + added: refreshedProfile.servers.filter(({ id }) => !previousProfile.servers.some((server) => server.id === id)).length, + removed: previousProfile.servers.filter(({ id }) => !refreshedProfile.servers.some((server) => server.id === id)).length, + }, + }); + }; + const loadedTargets = previousState.appliedFailoverPolicy + ? [previousState.appliedFailoverPolicy.primary, previousState.appliedFailoverPolicy.reserve] + : []; + const missingLoadedTarget = loadedTargets.some((target) => ( + target.profileId === profileId + && !refreshedProfile.servers.some(({ id }) => id === target.serverId) + )); + const pauseFailover = previousState.failoverPolicy?.enabled + && !previousState.failoverPolicy.paused + && missingLoadedTarget; const running = await dependencies.runtime.isRunning(); - const refreshesApplied = running && previousState.appliedProfileId === profileId; + const refreshesApplied = running + && previousState.appliedProfileId === profileId + && !previousState.appliedFailoverPolicy; const nextAppliedServerId = refreshesApplied ? dependencies.provider.selectRefreshedServer( previousState.appliedServerId, @@ -271,7 +342,17 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen dependencies.state.update((current) => ({ ...current, profiles: replaceProfile(current, refreshedProfile), + ...(pauseFailover ? { failoverPolicy: { ...current.failoverPolicy, paused: true } } : {}), })); + if (pauseFailover) dependencies.onEvent?.({ + type: 'failover.paused', + severity: 'warning', + source: 'failover', + dedupeKey: `failover.paused:missing-target:${profileId}:${dependencies.state.read().revision}`, + data: {}, + }); + await dependencies.failover?.reconcile(); + appendRefreshEvent(); return mutationResult(refreshedProfile); } @@ -305,13 +386,25 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen await finishRollback(error, [ ...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []), ...(configMutationStarted ? [{ run: () => restoreConfig(previousConfig) }] : []), - ...(runtimeMutationStarted ? [{ run: () => dependencies.runtime.start(), runtime: true }] : []), + ...(runtimeMutationStarted ? [{ + run: async () => { + await dependencies.runtime.start(); + await dependencies.failover?.restoreAppliedActivation(previousState); + }, + runtime: true, + }] : []), ], 'Subscription refresh rollback failed'); } + await dependencies.failover?.reconcile(); + appendRefreshEvent(); return mutationResult(refreshedProfile); }); - const refreshProfile = (profileIdValue: unknown, expectedRevision?: unknown) => { + const refreshProfile = ( + profileIdValue: unknown, + expectedRevision?: unknown, + origin: 'manual' | 'scheduled' = 'manual', + ) => { const profileId = String(profileIdValue || '').trim(); const existing = refreshPromises.get(profileId); if (existing) return existing; @@ -328,6 +421,7 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen profileId, initialProfile.subscriptionUrl, error, + origin, ); } throw error; @@ -336,6 +430,7 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen profileId, initialProfile.subscriptionUrl, parsed, + origin, ); })().finally(() => refreshPromises.delete(profileId)); refreshPromises.set(profileId, operation); @@ -353,12 +448,18 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen const mode = String(modeValue || 'delete'); if (!['delete', 'stop-and-delete'].includes(mode)) throw new HarborError('REQUEST_INVALID'); const running = await dependencies.runtime.isRunning(); - const applied = running && previousState.appliedProfileId === profile.id; + const failoverReferencesProfile = Boolean(previousState.appliedFailoverPolicy && ( + previousState.appliedFailoverPolicy.primary.profileId === profile.id + || previousState.appliedFailoverPolicy.reserve.profileId === profile.id + )); + const desiredFailoverReferencesProfile = previousState.failoverPolicy?.primary.profileId === profile.id + || previousState.failoverPolicy?.reserve.profileId === profile.id; + const applied = running && (previousState.appliedProfileId === profile.id || failoverReferencesProfile); if (applied && mode !== 'stop-and-delete') throw new HarborError('PROFILE_IN_USE'); const previousConfig = dependencies.config.read(); const previousGatewayAuto = dependencies.gatewayAuto.read(); - const removesAppliedTarget = previousState.appliedProfileId === profile.id; + const removesAppliedTarget = previousState.appliedProfileId === profile.id || failoverReferencesProfile; let runtimeMutationStarted = false; let configMutationStarted = false; let gatewayMutationStarted = false; @@ -390,16 +491,49 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen appliedServerSnapshot: current.appliedProfileId === profile.id ? null : current.appliedServerSnapshot, - ...(current.appliedProfileId === profile.id ? { connectionDesired: 'stopped' } : {}), + ...(removesAppliedTarget ? { + connectionDesired: 'stopped', + appliedProfileId: '', + appliedServerId: '', + appliedServerSnapshot: null, + appliedFailoverPolicy: null, + } : {}), + ...(failoverReferencesProfile || desiredFailoverReferencesProfile ? { + failoverPolicy: { + ...current.failoverPolicy, + enabled: false, + paused: false, + primary: current.failoverPolicy.primary.profileId === profile.id + ? { profileId: '', serverId: '' } + : current.failoverPolicy.primary, + reserve: current.failoverPolicy.reserve.profileId === profile.id + ? { profileId: '', serverId: '' } + : current.failoverPolicy.reserve, + }, + } : {}), })); } catch (error) { await finishRollback(error, [ ...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []), ...(gatewayMutationStarted ? [{ run: () => dependencies.gatewayAuto.set(previousGatewayAuto) }] : []), ...(configMutationStarted ? [{ run: () => restoreConfig(previousConfig) }] : []), - ...(runtimeMutationStarted ? [{ run: () => dependencies.runtime.start(), runtime: true }] : []), + ...(runtimeMutationStarted ? [{ + run: async () => { + await dependencies.runtime.start(); + await dependencies.failover?.restoreAppliedActivation(previousState); + }, + runtime: true, + }] : []), ], 'Subscription delete rollback failed'); } + await dependencies.failover?.reconcile(); + dependencies.onEvent?.({ + type: 'subscription.deleted', + severity: 'info', + source: 'subscription', + dedupeKey: `subscription.deleted:${dependencies.state.read().revision}`, + data: { profileId: profile.id, profileLabel: profile.label }, + }); return mutationResult(profile); }); @@ -436,7 +570,7 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen void (async () => { for (const { id } of dependencies.state.read().profiles) { try { - await refreshProfile(id); + await refreshProfile(id, undefined, 'scheduled'); } catch (error) { dependencies.onRefreshError(error); } diff --git a/src/server/http/routes/activityJournalRoute.ts b/src/server/http/routes/activityJournalRoute.ts new file mode 100644 index 0000000..e913117 --- /dev/null +++ b/src/server/http/routes/activityJournalRoute.ts @@ -0,0 +1,17 @@ +import type { IncomingMessage, ServerResponse } from 'node:http'; +import type { ActivityJournalService } from '../../services/activityJournalService.js'; +import { sendJson } from '../response.js'; + +export function createActivityJournalRoute({ journal }: { journal: ActivityJournalService }) { + return { + async handle(req: IncomingMessage, res: ServerResponse) { + const url = new URL(req.url || '/', 'http://localhost'); + if (url.pathname !== '/api/activity-journal' || req.method !== 'GET') return false; + sendJson(res, 200, journal.page( + Number(url.searchParams.get('limit')) || 50, + url.searchParams.get('cursor'), + )); + return true; + }, + }; +} diff --git a/src/server/http/routes/failoverRoute.ts b/src/server/http/routes/failoverRoute.ts new file mode 100644 index 0000000..cae58c5 --- /dev/null +++ b/src/server/http/routes/failoverRoute.ts @@ -0,0 +1,49 @@ +import type { IncomingMessage, ServerResponse } from 'node:http'; +import type { FailoverService } from '../../features/failover/failoverService.js'; +import { HarborError } from '../../../shared/errors.js'; + +interface FailoverRouteDependencies { + appMode: string; + failover: Pick; + readBody(req: IncomingMessage): Promise>; + withOperation(kind: string, operation: () => Promise, options?: { expectedRevision?: unknown }): Promise; + sendState(res: ServerResponse): Promise; +} + +export function createFailoverRoute(dependencies: FailoverRouteDependencies) { + return { + async handle(req: IncomingMessage, res: ServerResponse) { + const pathname = new URL(req.url || '/', 'http://localhost').pathname; + if (!pathname.startsWith('/api/failover')) return false; + if (dependencies.appMode !== 'gateway') throw new HarborError('ENDPOINT_NOT_FOUND'); + const body = await dependencies.readBody(req); + if (pathname === '/api/failover' && req.method === 'PUT') { + await dependencies.withOperation( + 'failover-save', + () => dependencies.failover.save(body.policy), + { expectedRevision: body.expectedRevision }, + ); + } else if (pathname === '/api/failover/pause' && req.method === 'POST') { + if (typeof body.paused !== 'boolean') throw new HarborError('REQUEST_INVALID'); + await dependencies.withOperation( + body.paused ? 'failover-pause' : 'failover-resume', + () => dependencies.failover.pause(body.paused as boolean), + { expectedRevision: body.expectedRevision }, + ); + } else if (pathname === '/api/failover/switch' && req.method === 'POST') { + if (body.role !== 'primary' && body.role !== 'reserve') throw new HarborError('REQUEST_INVALID'); + await dependencies.withOperation( + 'failover-switch', + () => dependencies.failover.manualSwitch(body.role as 'primary' | 'reserve'), + { expectedRevision: body.expectedRevision }, + ); + } else if (pathname === '/api/failover/check' && req.method === 'POST') { + await dependencies.failover.checkNow(); + } else { + throw new HarborError('ENDPOINT_NOT_FOUND'); + } + await dependencies.sendState(res); + return true; + }, + }; +} diff --git a/src/server/index.ts b/src/server/index.ts index 16d1ae8..90adc6d 100644 --- a/src/server/index.ts +++ b/src/server/index.ts @@ -16,7 +16,10 @@ import { import { createSingboxRuntime } from './singboxRuntime.js'; import { tcpPing } from './ping.js'; import { + buildDualChannelGatewayConfig, buildGatewayConfig, + dualChannelConfigMatchesApplied, + fingerprintSelectedOutbound, removeSingboxConfig, restoreSingboxConfig, writeSingboxConfig, @@ -81,6 +84,13 @@ import { createConnectivityDiagnosticsRoute } from './http/routes/connectivityDi import { createGatewayPresenceRoute } from './http/routes/gatewayPresenceRoute.js'; import { createSharedProxyRoute } from './http/routes/sharedProxyRoute.js'; import { createVersionRoute } from './http/routes/versionRoute.js'; +import { createSingboxSelectorService } from './services/singboxSelectorService.js'; +import { createFailoverService } from './features/failover/failoverService.js'; +import { createFailoverRoute } from './http/routes/failoverRoute.js'; +import { createActivityJournalService } from './services/activityJournalService.js'; +import { createActivityJournalRoute } from './http/routes/activityJournalRoute.js'; +import { createDomainTrafficService, readSingboxConnections } from './services/domainTrafficService.js'; +import type { ActivityJournalEventInput } from '../shared/activityJournal.js'; const MAX_BODY_BYTES = 1_000_000; const SUBSCRIPTION_REFRESH_INTERVAL_MS = 15 * 60 * 1000; @@ -142,6 +152,14 @@ if (legacyCacheOwnerMismatch) { } } const stateStore = createStateStore(settings.statePath, { legacySubscriptionCache }); +const activityJournal = createActivityJournalService({ filePath: settings.activityJournalPath }); +const appendJournal = (event: ActivityJournalEventInput) => { + try { + activityJournal.append(event); + } catch (error) { + console.warn(`[journal] событие не сохранено: ${errorMessage(error)}`); + } +}; const deviceStore = createJsonStore({ filePath: settings.deviceStatePath, defaultValue: migrateDeviceInventoryState({}), @@ -283,6 +301,37 @@ const deviceInventory = settings.appMode === 'gateway' const localConnectivityDiagnostics = !remoteDataplane ? createConnectivityDiagnosticsService({ proxyPort: settings.diagnosticsProxyPort }) : null; +const localFailoverDiagnostics = !remoteDataplane ? { + primary: createConnectivityDiagnosticsService({ proxyPort: settings.failoverPrimaryProxyPort }), + reserve: createConnectivityDiagnosticsService({ proxyPort: settings.failoverReserveProxyPort }), +} : null; +const localSelector = !remoteDataplane && settings.appMode === 'gateway' + ? createSingboxSelectorService({ port: settings.singboxApiPort }) + : null; +const localFailoverTraffic = !remoteDataplane && settings.appMode === 'gateway' + ? createDomainTrafficService({ + observe: () => readSingboxConnections(settings.singboxApiPort), + devices: () => record(deviceInventory?.snapshot()).devices, + }) + : null; +let localFailoverTrafficTimer: NodeJS.Timeout | null = null; + +function setLocalFailoverActivityEnabled(enabled: boolean) { + if (!localFailoverTraffic) throw new Error('Failover activity недоступна'); + if (!enabled) { + if (localFailoverTrafficTimer) clearInterval(localFailoverTrafficTimer); + localFailoverTrafficTimer = null; + localFailoverTraffic.disableActivity(); + return; + } + localFailoverTraffic.enableActivity(); + if (localFailoverTrafficTimer) return; + const refresh = () => localFailoverTraffic.refresh() + .catch((error: unknown) => console.warn(`[control] failover activity: ${errorMessage(error)}`)); + void refresh(); + localFailoverTrafficTimer = setInterval(refresh, 2_000); + localFailoverTrafficTimer.unref(); +} function requireLocalConnectivityDiagnostics() { if (!localConnectivityDiagnostics) throw new Error('Harbor local diagnostics are not configured'); @@ -355,6 +404,48 @@ const gatewayAutoService = createGatewayAutoService({ onDiscoveryWarning: (reason) => console.warn(`[control] Gateway не используется: ${reason}`), onTimerError: (error) => console.warn(`[control] Gateway detection failed: ${errorMessage(error)}`), }); +const failoverDataplane = remoteDataplane ? { + checkConfig: (config: unknown) => requireRemoteRuntime().checkConfig(config), + runFailoverProbe: (role: 'primary' | 'reserve', services: unknown, target: unknown, timeoutMs: number) => ( + requireRemoteRuntime().runFailoverProbe(role, services, target, timeoutMs) + ), + readFailoverSelector: () => requireRemoteRuntime().readFailoverSelector(), + selectFailoverRole: (role: 'primary' | 'reserve') => requireRemoteRuntime().selectFailoverRole(role), + setFailoverActivityEnabled: (enabled: boolean) => requireRemoteRuntime().setFailoverActivityEnabled(enabled), + readFailoverActivity: (threshold: number) => requireRemoteRuntime().readFailoverActivity(threshold), +} : { + checkConfig: async (config: unknown) => singboxRuntime.checkConfig(config), + runFailoverProbe: async (role: 'primary' | 'reserve', services: unknown, target: unknown, timeoutMs: number) => { + if (!localFailoverDiagnostics) throw new Error('Failover diagnostics недоступна'); + return localFailoverDiagnostics[role].runVpn({ services, target, timeoutMs }); + }, + readFailoverSelector: async () => { + if (!localSelector) throw new Error('Failover selector недоступен'); + return localSelector.read(); + }, + selectFailoverRole: async (role: 'primary' | 'reserve') => { + if (!localSelector) throw new Error('Failover selector недоступен'); + return localSelector.select(role); + }, + setFailoverActivityEnabled: async (enabled: boolean) => setLocalFailoverActivityEnabled(enabled), + readFailoverActivity: async (threshold: number) => ({ + activity: localFailoverTraffic?.activitySnapshot(threshold) || null, + }), +}; +const failoverService = createFailoverService({ + state: { + read: () => normalizeStoredState(stateStore.read()), + update: updateStoredState, + }, + runtime: { isRunning: async () => Boolean((await singboxRuntime.refresh()).running) }, + dataplane: failoverDataplane, + buildCandidate: buildFailoverCandidate, + serialize: serializeControl, + onWarning: (error) => console.warn(`[control] failover: ${errorMessage(error)}`), + onSwitch: (from, to, reason) => console.log(`[control] failover ${from} -> ${to}: ${reason}`), + onEvent: appendJournal, +}); +const gatewayFailover = settings.appMode === 'gateway' ? failoverService : null; const stateService = createStateService({ appMode: settings.appMode, readStoredState: () => stateStore.read(), @@ -362,6 +453,7 @@ const stateService = createStateService({ getGatewayAutoState: gatewayAutoService.read, getOperationState: () => operationState, configExists: () => fs.existsSync(settings.configPath), + getFailoverSnapshot: failoverService.snapshot, }); const stateRoute = createStateRoute({ stateService, @@ -375,6 +467,14 @@ const gatewayAutoRoute = createGatewayAutoRoute({ withOperation, readStatePayload: stateRoute.readPayload, }); +const failoverRoute = createFailoverRoute({ + appMode: settings.appMode, + failover: failoverService, + readBody, + withOperation, + sendState: (res) => stateRoute.send(res), +}); +const activityJournalRoute = createActivityJournalRoute({ journal: activityJournal }); const deviceInventoryRoute = createDeviceInventoryRoute({ deviceInventory, readBody, @@ -427,9 +527,12 @@ const subscriptionService = createSubscriptionService({ update: updateStoredState, }, config: { - build: (subscriptionConfig, selectedServerId, routeRules) => ( - buildActiveConfig(subscriptionConfig, selectedServerId, routeRules) - ), + build: (subscriptionConfig, selectedServerId, routeRules) => { + const state = normalizeStoredState(stateStore.read()); + return state.appliedFailoverPolicy + ? buildFailoverCandidate({ ...state, routeRules }, 'applied').config + : buildActiveConfig(subscriptionConfig, selectedServerId, routeRules); + }, read: () => fs.existsSync(settings.configPath) ? fs.readFileSync(settings.configPath, 'utf8') : null, @@ -453,6 +556,12 @@ const subscriptionService = createSubscriptionService({ clearInterval: (timer) => clearInterval(timer), }, onRefreshError: (error) => console.warn(`[control] подписка не обновлена: ${errorMessage(error)}`), + onEvent: appendJournal, + failover: gatewayFailover ? { + reconcile: () => gatewayFailover.reconcile() + .catch((error) => console.warn(`[control] failover reconcile: ${errorMessage(error)}`)), + restoreAppliedActivation: gatewayFailover.restoreAppliedActivation, + } : undefined, now: () => new Date(), }); const serverHealthRoute = createServerHealthRoute({ @@ -484,6 +593,14 @@ const connectionService = createConnectionService({ isGatewayDirect: () => settings.appMode === 'client' && gatewayAutoService.read().mode === 'gateway-direct', }, + failover: gatewayFailover ? { + build: buildFailoverCandidate, + prepareActivation: gatewayFailover.prepareActivation, + restoreAppliedActivation: gatewayFailover.restoreAppliedActivation, + reconcile: () => gatewayFailover.reconcile() + .catch((error) => console.warn(`[control] failover reconcile: ${errorMessage(error)}`)), + } : undefined, + onEvent: appendJournal, runtime: { isRunning: async () => Boolean((await singboxRuntime.refresh()).running), start: () => startSingbox(), @@ -524,9 +641,12 @@ const routeRulesService = createRouteRulesService({ readConfig: (profileId) => readProfileConfig(profileId), }, config: { - build: (subscriptionConfig, selectedServerId, routeRules) => ( - buildActiveConfig(subscriptionConfig, selectedServerId, routeRules) - ), + build: (subscriptionConfig, selectedServerId, routeRules) => { + const state = normalizeStoredState(stateStore.read()); + return state.appliedFailoverPolicy + ? buildFailoverCandidate({ ...state, routeRules }, 'applied').config + : buildActiveConfig(subscriptionConfig, selectedServerId, routeRules); + }, read: () => fs.existsSync(settings.configPath) ? fs.readFileSync(settings.configPath, 'utf8') : null, @@ -548,6 +668,9 @@ const routeRulesService = createRouteRulesService({ }, serialize: serializeControl, runOperation: (operation) => withOperation('route-rules', operation), + afterApply: gatewayFailover ? () => gatewayFailover.reconcile() + .catch((error) => console.warn(`[control] failover reconcile: ${errorMessage(error)}`)) : undefined, + restoreAppliedActivation: gatewayFailover?.restoreAppliedActivation, }); const routeRulesRoute = createRouteRulesRoute({ routeRules: routeRulesService, @@ -676,6 +799,47 @@ function buildActiveConfig( }); } +function buildFailoverCandidate(state: StoredState, source: 'desired' | 'applied' = 'desired') { + const policy = source === 'applied' ? state.appliedFailoverPolicy : state.failoverPolicy; + if (!policy) throw new HarborError('CONFIG_INVALID'); + if ( + policy.primary.profileId === policy.reserve.profileId + && policy.primary.serverId === policy.reserve.serverId + ) throw new HarborError('REQUEST_INVALID'); + const channel = (role: 'primary' | 'reserve') => { + const target = policy[role]; + const profile = state.profiles.find(({ id }) => id === target.profileId); + const server = profile?.servers.find(({ id }) => id === target.serverId); + if (!profile || !server || !profile.subscriptionConfig) throw new HarborError('SERVER_NOT_FOUND'); + return { profile, server }; + }; + const primary = channel('primary'); + const reserve = channel('reserve'); + const applied = { + primary: policy.primary, + reserve: policy.reserve, + primaryConfigFingerprint: fingerprintSelectedOutbound(primary.profile.subscriptionConfig, primary.server.id), + reserveConfigFingerprint: fingerprintSelectedOutbound(reserve.profile.subscriptionConfig, reserve.server.id), + }; + if (source === 'applied' && JSON.stringify(applied) !== JSON.stringify(state.appliedFailoverPolicy)) { + throw new HarborError('CONFIG_INVALID'); + } + const defaultRole = source === 'applied' + && state.appliedProfileId === policy.reserve.profileId + && state.appliedServerId === policy.reserve.serverId + ? 'reserve' + : 'primary'; + return { + config: buildDualChannelGatewayConfig({ + primary: { subscriptionConfig: primary.profile.subscriptionConfig, selectedServerId: primary.server.id }, + reserve: { subscriptionConfig: reserve.profile.subscriptionConfig, selectedServerId: reserve.server.id }, + }, { routeRules: state.routeRules, defaultRole }), + applied, + primaryProfile: primary.profile, + primaryServer: primary.server, + }; +} + const stopSingbox = () => singboxRuntime.stop(); const startSingbox = () => singboxRuntime.apply(); @@ -689,7 +853,13 @@ function writeCurrentConfig() { const server = profile?.servers.find((candidate) => candidate.id === serverId); const subscriptionConfig = profile ? readProfileConfig(profile.id) : null; if (!profile || !server || !subscriptionConfig) return null; - const activeConfig = buildActiveConfig(subscriptionConfig, server.id); + let activeConfig: unknown; + if (state.appliedFailoverPolicy) { + const candidate = buildFailoverCandidate(state, 'applied'); + activeConfig = candidate.config; + } else { + activeConfig = buildActiveConfig(subscriptionConfig, server.id); + } const previousConfig = fs.existsSync(settings.configPath) ? fs.readFileSync(settings.configPath, 'utf8') : null; @@ -704,7 +874,7 @@ function writeCurrentConfig() { } throw error; } - return { profile, server }; + return { profile, server, failoverApplied: state.appliedFailoverPolicy }; } const CONFIG_PROXY_TYPES = new Set(['vless', 'vmess', 'trojan', 'shadowsocks', 'hysteria2']); @@ -717,6 +887,13 @@ function currentConfigMatchesAppliedTarget(state: StoredState) { } catch { return false; } + if (state.appliedFailoverPolicy) { + const expectedRole = state.appliedProfileId === state.appliedFailoverPolicy.reserve.profileId + && state.appliedServerId === state.appliedFailoverPolicy.reserve.serverId + ? 'reserve' + : 'primary'; + return dualChannelConfigMatchesApplied(config, state.appliedFailoverPolicy, expectedRole); + } const proxyOutbounds = (Array.isArray(config.outbounds) ? config.outbounds : []) .map(record) .filter((outbound) => CONFIG_PROXY_TYPES.has(String(outbound.type || ''))); @@ -750,6 +927,7 @@ async function reconcileStoppedBoot({ removeConfig = false } = {}) { || state.appliedProfileId || state.appliedServerId || state.appliedServerSnapshot + || state.appliedFailoverPolicy ) { updateStoredState((current) => ({ ...current, @@ -757,6 +935,7 @@ async function reconcileStoppedBoot({ removeConfig = false } = {}) { appliedProfileId: '', appliedServerId: '', appliedServerSnapshot: null, + appliedFailoverPolicy: null, })); } } @@ -770,6 +949,8 @@ async function handleApi(req: IncomingMessage, res: ServerResponse) { if (await connectionRuntimeRoute.handle(req, res)) return; if (await routeRulesRoute.handle(req, res)) return; if (await gatewayAutoRoute.handle(req, res)) return; + if (await failoverRoute.handle(req, res)) return; + if (await activityJournalRoute.handle(req, res)) return; if (await connectivityDiagnosticsRoute.handle(req, res)) return; if (await versionRoute.handle(req, res)) return; @@ -822,6 +1003,7 @@ async function shutdown() { subscriptionService.stopAutoRefresh(); gatewayAutoService.stopDiscovery(); if (deviceDiscoveryTimer) clearInterval(deviceDiscoveryTimer); + await gatewayFailover?.shutdown().catch((error) => console.warn(`[control] failover shutdown: ${errorMessage(error)}`)); await serializeControl(() => singboxRuntime.shutdown()); process.exit(0); } @@ -846,8 +1028,14 @@ if (bootWantsRunning) { && currentConfigMatchesAppliedTarget(normalizeStoredState(stateStore.read())); if (target || canReuseCurrentConfig) { await startSingbox() - .then(() => { + .then(async () => { const current = normalizeStoredState(stateStore.read()); + const bootRole = current.appliedFailoverPolicy + && current.appliedProfileId === current.appliedFailoverPolicy.reserve.profileId + && current.appliedServerId === current.appliedFailoverPolicy.reserve.serverId + ? 'reserve' + : current.appliedFailoverPolicy ? 'primary' : null; + if (bootRole) await failoverDataplane.selectFailoverRole(bootRole); const appliedProfile = target?.profile || current.profiles.find((profile) => profile.id === current.appliedProfileId); const appliedServer = target?.server @@ -866,7 +1054,10 @@ if (bootWantsRunning) { })); } }) - .catch((error: unknown) => console.warn(`[control] sing-box не запущен: ${errorMessage(error)}`)); + .catch(async (error: unknown) => { + console.warn(`[control] sing-box не запущен: ${errorMessage(error)}`); + await reconcileStoppedBoot(); + }); } else { await reconcileStoppedBoot({ removeConfig: true }); } @@ -874,6 +1065,9 @@ if (bootWantsRunning) { await reconcileStoppedBoot(); } +await gatewayFailover?.reconcile() + .catch((error) => console.warn(`[control] failover reconcile: ${errorMessage(error)}`)); + if (deviceInventory) { await deviceInventory.reconcilePolicies() .catch((error: unknown) => console.warn(`[control] device policy не применена: ${errorMessage(error)}`)); diff --git a/src/server/services/activityJournalService.ts b/src/server/services/activityJournalService.ts new file mode 100644 index 0000000..c4aa689 --- /dev/null +++ b/src/server/services/activityJournalService.ts @@ -0,0 +1,152 @@ +import crypto from 'node:crypto'; +import fs from 'node:fs'; +import { + ACTIVITY_JOURNAL_MAX_EVENTS, + ACTIVITY_JOURNAL_RETENTION_DAYS, + normalizeActivityEventInput, + normalizeStoredActivityEvent, + type ActivityJournalEvent, + type ActivityJournalEventInput, + type ActivityJournalPage, +} from '../../shared/activityJournal.js'; +import { createJsonStore } from './stateStore.js'; + +interface JournalState { + schemaVersion: 1; + events: ActivityJournalEvent[]; +} + +const migrateJournal = (value: unknown): JournalState => { + const candidate = value && typeof value === 'object' && !Array.isArray(value) + ? value as Record + : {}; + return { + schemaVersion: 1, + events: (Array.isArray(candidate.events) ? candidate.events : []) + .map(normalizeStoredActivityEvent) + .filter((event): event is ActivityJournalEvent => Boolean(event)), + }; +}; + +export function createActivityJournalService({ + filePath, + now = () => new Date(), +}: { + filePath: string; + now?: () => Date; +}) { + const store = createJsonStore({ + filePath, + defaultValue: { schemaVersion: 1, events: [] }, + migrate: migrateJournal, + }); + let recoveryRecorded = false; + let writeFailed = false; + + function retained(events: ActivityJournalEvent[]) { + const cutoff = now().getTime() - ACTIVITY_JOURNAL_RETENTION_DAYS * 86_400_000; + return events + .filter(({ occurredAt }) => Date.parse(occurredAt) >= cutoff) + .slice(-ACTIVITY_JOURNAL_MAX_EVENTS); + } + + function append(value: ActivityJournalEventInput) { + const input = normalizeActivityEventInput(value); + const storedInput = input.dedupeKey ? { + ...input, + dedupeKey: `${input.type}:sha256:${crypto.createHash('sha256').update(input.dedupeKey).digest('hex')}`, + } : input; + let appended: ActivityJournalEvent | null = null; + try { + store.update((state) => { + const events = retained(state.events); + if (storedInput.dedupeKey && events.some(({ dedupeKey }) => dedupeKey === storedInput.dedupeKey)) { + return { schemaVersion: 1, events }; + } + appended = { + id: crypto.randomUUID(), + occurredAt: now().toISOString(), + ...storedInput, + }; + return { schemaVersion: 1, events: retained([...events, appended]) }; + }); + writeFailed = false; + } catch (error) { + writeFailed = true; + throw error; + } + return appended; + } + + function ensureRecoveryEvent() { + if (!store.recovery || recoveryRecorded) return; + append({ + type: 'journal.recovered', + severity: 'warning', + source: 'storage', + dedupeKey: `journal.recovered:${store.recovery.recoveredAt}`, + data: {}, + }); + recoveryRecorded = true; + } + + function page(limitValue: unknown = 50, cursorValue: unknown = null): ActivityJournalPage { + try { + let state = store.read(); + ensureRecoveryEvent(); + if (store.recovery) state = store.read(); + const retainedEvents = retained(state.events); + if (retainedEvents.length !== state.events.length) { + state = store.update(() => ({ schemaVersion: 1, events: retainedEvents })); + } + const events = [...state.events].reverse(); + const limit = Math.min(100, Math.max(1, Number.isSafeInteger(limitValue) ? Number(limitValue) : 50)); + const cursor = typeof cursorValue === 'string' ? cursorValue : ''; + const cursorIndex = cursor ? events.findIndex(({ id }) => id === cursor) : -1; + if (cursor && cursorIndex < 0) return { + events: [], + nextCursor: null, + retentionDays: 30, + generatedAt: now().toISOString(), + storage: writeFailed + ? { status: 'error', errorCode: 'JOURNAL_UNAVAILABLE' } + : { status: 'ready', errorCode: null }, + }; + const safeStart = cursorIndex + 1; + const selected = events.slice(safeStart, safeStart + limit); + return { + events: selected.map((event) => ({ ...event, dedupeKey: null })), + nextCursor: safeStart + selected.length < events.length ? selected.at(-1)?.id || null : null, + retentionDays: 30, + generatedAt: now().toISOString(), + storage: writeFailed + ? { status: 'error', errorCode: 'JOURNAL_UNAVAILABLE' } + : { status: 'ready', errorCode: null }, + }; + } catch { + return { + events: [], + nextCursor: null, + retentionDays: 30, + generatedAt: now().toISOString(), + storage: { status: 'error', errorCode: 'JOURNAL_UNAVAILABLE' }, + }; + } + } + + if (fs.existsSync(filePath)) { + try { + const state = store.read(); + const retainedEvents = retained(state.events); + if (retainedEvents.length !== state.events.length) { + store.update(() => ({ schemaVersion: 1, events: retainedEvents })); + } + } catch { + writeFailed = true; + } + } + + return { append, page }; +} + +export type ActivityJournalService = ReturnType; diff --git a/src/server/services/connectivityDiagnosticsService.ts b/src/server/services/connectivityDiagnosticsService.ts index e8ad9da..a8f8534 100644 --- a/src/server/services/connectivityDiagnosticsService.ts +++ b/src/server/services/connectivityDiagnosticsService.ts @@ -44,6 +44,7 @@ interface RequestOptions { ipv4?: boolean; follow?: boolean; resolve?: string | null; + timeoutMs?: number; } interface RequestResult { @@ -179,7 +180,9 @@ async function request(probe: BaseProbe, path: PathKind, proxyPort: number, exec ipv4 = false, follow = true, resolve = null, + timeoutMs = 6_000, }: RequestOptions = {}): Promise { + const boundedTimeoutMs = Math.min(30_000, Math.max(1_000, Math.round(timeoutMs))); const args = [ '--silent', '--show-error', @@ -189,9 +192,9 @@ async function request(probe: BaseProbe, path: PathKind, proxyPort: number, exec '--proto-redir', '=https', '--connect-timeout', - '3', + String(Math.min(3_000, boundedTimeoutMs) / 1_000), '--max-time', - '6', + String(boundedTimeoutMs / 1_000), '--user-agent', 'Harbor-Diagnostics/1', '--output', @@ -379,6 +382,8 @@ async function siteProbe( proxyPort: number, execute: CurlExecutor, sampleCount = 1, + timeoutMs = 6_000, + retryFailure = true, ): Promise { if (probe.validationError) return { id: probe.id, @@ -391,12 +396,12 @@ async function siteProbe( stage: 'validation', error: probe.validationError, }; - const options = { follow: probe.follow !== false, resolve: probe.resolve }; + const options = { follow: probe.follow !== false, resolve: probe.resolve, timeoutMs }; const samples = []; for (let attempt = 0; attempt < sampleCount; attempt += 1) { samples.push(await request(probe, path, proxyPort, execute, options)); } - if (sampleCount === 1 && samples[0] && !samples[0].ok) { + if (retryFailure && sampleCount === 1 && samples[0] && !samples[0].ok) { samples.push(await request(probe, path, proxyPort, execute, options)); } const status = mostCommon(samples.map(siteStatus)) || 'unavailable'; @@ -469,15 +474,18 @@ async function probeTarget( path: PathKind, proxyPort: number, execute: CurlExecutor, + timeoutMs = 6_000, + sampleCount = TARGET_SAMPLE_COUNT, + retryFailure = true, ): Promise { const network = target.kind === 'network' - ? await networkProbe(path, proxyPort, execute, TARGET_SAMPLE_COUNT) + ? await networkProbe(path, proxyPort, execute, sampleCount) : null; const ip = target.kind === 'ip' - ? await ipProbe(target.probe, path, proxyPort, execute, TARGET_SAMPLE_COUNT) + ? await ipProbe(target.probe, path, proxyPort, execute, sampleCount) : null; const site = target.kind === 'site' - ? await siteProbe(target.probe, path, proxyPort, execute, TARGET_SAMPLE_COUNT) + ? await siteProbe(target.probe, path, proxyPort, execute, sampleCount, timeoutMs, retryFailure) : null; const ipv4Sources = ip?.family === 4 ? [ip] : []; const ipv6Source = ip?.family === 6 ? ip : null; @@ -537,7 +545,25 @@ export function createConnectivityDiagnosticsService({ assessment: assessConnectivity(direct, vpn), }; } + async function runVpn({ services = [], target: targetId = null, timeoutMs = 6_000 }: { + services?: unknown; + target?: unknown; + timeoutMs?: number; + }) { + const requestedServices = typeof targetId === 'string' && targetId.startsWith('site:custom-') + ? (Array.isArray(services) ? services : []).filter((service) => `site:${String(record(service).id || '')}` === targetId) + : []; + const customProbes = await prepareCustomProbes(requestedServices, lookup); + const siteProbes = [...SITE_PROBES, ...customProbes]; + const target = resolveTarget(targetId, siteProbes); + if (!target || target.kind !== 'site') throw new Error('Failover check ожидает site target'); + return { + checkedAt: now(), + vpn: await probeTarget(target, 'vpn', proxyPort, execute, timeoutMs, TARGET_SAMPLE_COUNT, false), + }; + } return { run: runOnce, + runVpn, }; } diff --git a/src/server/services/domainTrafficService.ts b/src/server/services/domainTrafficService.ts index 45a3702..8e4e9cd 100644 --- a/src/server/services/domainTrafficService.ts +++ b/src/server/services/domainTrafficService.ts @@ -84,12 +84,33 @@ interface DomainTrafficSnapshot { }>; } +interface ActivityEntry { + device: string; + service: string; + upload: bigint; + download: bigint; +} + +interface ActivitySample { + at: number; + entries: ActivityEntry[]; +} + function record(value: unknown): Record { return value && typeof value === 'object' && !Array.isArray(value) ? value as Record : {}; } +function publicDeviceLabel(value: unknown) { + const device = record(value); + for (const candidate of [device.alias, device.hostname]) { + const label = typeof candidate === 'string' ? candidate.trim() : ''; + if (label && label.length <= 64 && !/[\/?#@\\]/.test(label) && !net.isIP(label)) return label; + } + return 'Устройство'; +} + const matchesDomain = (domain: string, suffix: string) => domain === suffix || domain.endsWith(`.${suffix}`); export function classifyDomain(value: unknown): { domain: string; service: string } | null { @@ -209,6 +230,10 @@ export function createDomainTrafficService({ unsupported_source: 0n, }; let refreshPromise: Promise | null = null; + let activityEnabled = false; + let activityStartedAt = 0; + let activitySamples: ActivitySample[] = []; + let quietSince: string | null = null; let current: DomainTrafficSnapshot = { epoch, observedAt: null, @@ -270,6 +295,7 @@ export function createDomainTrafficService({ const response = record(await observe()); if (!Array.isArray(response.connections)) throw new Error('Sing-box не вернул connections array'); const devicesByIp = new Map(); + const deviceLabels = new Map(); const observedDevices = devices(); for (const value of Array.isArray(observedDevices) ? observedDevices : []) { const device = record(value); @@ -277,9 +303,11 @@ export function createDomainTrafficService({ const id = typeof device.mac === 'string' ? deviceId(device.mac.toLowerCase()) : null; if (!net.isIPv4(ip) || !id) continue; devicesByIp.set(ip, devicesByIp.has(ip) ? null : id); + deviceLabels.set(id, publicDeviceLabel(device)); } const connections = response.connections.map((connection) => parseConnection(connection, devicesByIp)); const activeConnections = new Map(); + const activityEntries: ActivityEntry[] = []; for (const connection of connections) { const previous = previousConnections.get(connection.id); if (connection.outcome !== 'classified' && previous?.outcome !== connection.outcome) { @@ -302,6 +330,16 @@ export function createDomainTrafficService({ tracked.uploadBytes += uploadDelta; tracked.downloadBytes += downloadDelta; trackedTotals.set(trackedKey, tracked); + if (activityEnabled && connection.outbound === 'vpn' && uploadDelta + downloadDelta > 0n) { + activityEntries.push({ + device: 'deviceId' in connection + ? deviceLabels.get(connection.deviceId) || 'Устройство' + : 'Неизвестное устройство', + service: 'service' in connection ? connection.service : 'Не распознано', + upload: uploadDelta, + download: downloadDelta, + }); + } } if (connection.outcome === 'unknown_device' || connection.outcome === 'unsupported_source') { activeConnections.set(connection.id, { @@ -372,7 +410,12 @@ export function createDomainTrafficService({ }); } previousConnections = activeConnections; - current = { ...current, observedAt: now().toISOString() }; + const observed = now(); + if (activityEnabled) { + activitySamples.push({ at: observed.getTime(), entries: activityEntries }); + activitySamples = activitySamples.filter(({ at }) => at >= observed.getTime() - 10_000); + } + current = { ...current, observedAt: observed.toISOString() }; current = buildSnapshot(); return current; } catch (error) { @@ -390,5 +433,64 @@ export function createDomainTrafficService({ return refreshPromise; } - return { snapshot: () => current, refresh }; + function enableActivity() { + if (activityEnabled) return; + activityEnabled = true; + activityStartedAt = now().getTime(); + activitySamples = []; + quietSince = null; + } + + function disableActivity() { + activityEnabled = false; + activityStartedAt = 0; + activitySamples = []; + quietSince = null; + } + + function activitySnapshot(thresholdBytesPerSecond: unknown = 0) { + if (!activityEnabled || !current.observedAt) return null; + const observedAt = Date.parse(current.observedAt); + const threshold = Math.max(0, Number(thresholdBytesPerSecond) || 0); + const divisorMs = Math.max(1_000, Math.min(10_000, observedAt - activityStartedAt || 1_000)); + const totals = new Map(); + let bytes = 0n; + for (const sample of activitySamples) { + for (const entry of sample.entries) { + bytes += entry.upload + entry.download; + const key = `${entry.device}\0${entry.service}`; + const total = totals.get(key) || { ...entry, upload: 0n, download: 0n }; + total.upload += entry.upload; + total.download += entry.download; + totals.set(key, total); + } + } + const totalBytesPerSecond = Number(bytes * 1_000n / BigInt(divisorMs)); + const active = totalBytesPerSecond > threshold; + quietSince = active ? null : quietSince || current.observedAt; + const latest = activitySamples.at(-1); + return { + state: active ? 'active' : 'quiet', + observedAt: current.observedAt, + windowMs: 10_000, + thresholdBytesPerSecond: threshold, + totalBytesPerSecond, + transmittingConnections: latest?.entries.length || 0, + quietSince, + blockers: [...totals.values()] + .map((entry) => ({ + device: entry.device, + service: entry.service, + uploadBytesPerSecond: Number(entry.upload * 1_000n / BigInt(divisorMs)), + downloadBytesPerSecond: Number(entry.download * 1_000n / BigInt(divisorMs)), + })) + .sort((left, right) => ( + right.uploadBytesPerSecond + right.downloadBytesPerSecond + - left.uploadBytesPerSecond - left.downloadBytesPerSecond + )) + .slice(0, 3), + }; + } + + return { snapshot: () => current, refresh, enableActivity, disableActivity, activitySnapshot }; } diff --git a/src/server/services/singboxSelectorService.ts b/src/server/services/singboxSelectorService.ts new file mode 100644 index 0000000..7a00b23 --- /dev/null +++ b/src/server/services/singboxSelectorService.ts @@ -0,0 +1,66 @@ +import http from 'node:http'; +import { + FAILOVER_PRIMARY_TAG, + FAILOVER_RESERVE_TAG, + FAILOVER_SELECTOR_TAG, +} from '../singbox.js'; + +type Role = 'primary' | 'reserve'; + +function request(port: number, method: string, body?: unknown): Promise { + return new Promise((resolve, reject) => { + const encoded = body === undefined ? null : JSON.stringify(body); + const req = http.request({ + host: '127.0.0.1', + port, + path: `/proxies/${encodeURIComponent(FAILOVER_SELECTOR_TAG)}`, + method, + headers: encoded ? { + 'content-type': 'application/json', + 'content-length': Buffer.byteLength(encoded), + } : {}, + }, (res) => { + const chunks: Buffer[] = []; + res.on('data', (chunk: Buffer) => chunks.push(chunk)); + res.on('end', () => { + if ((res.statusCode || 500) >= 400) return reject(new Error(`Sing-box selector HTTP ${res.statusCode}`)); + if (!chunks.length) return resolve({}); + try { + resolve(JSON.parse(Buffer.concat(chunks).toString('utf8'))); + } catch (cause) { + reject(new Error('Sing-box selector вернул невалидный JSON', { cause })); + } + }); + }); + req.setTimeout(2_000, () => req.destroy(new Error('Sing-box selector timeout'))); + req.on('error', reject); + req.end(encoded); + }); +} + +const tagFor = (role: Role) => role === 'primary' ? FAILOVER_PRIMARY_TAG : FAILOVER_RESERVE_TAG; +const roleFor = (tag: unknown): Role | null => ( + tag === FAILOVER_PRIMARY_TAG ? 'primary' : tag === FAILOVER_RESERVE_TAG ? 'reserve' : null +); + +export function createSingboxSelectorService({ + port, + send = (method: string, body?: unknown) => request(port, method, body), +}: { + port: number; + send?: (method: string, body?: unknown) => Promise; +}) { + async function read() { + const value = await send('GET') as Record; + const role = roleFor(value.now); + if (!role) throw new Error('Sing-box selector вернул неизвестный outbound'); + return { role }; + } + async function select(role: Role) { + await send('PUT', { name: tagFor(role) }); + const selected = await read(); + if (selected.role !== role) throw new Error('Sing-box selector не подтвердил переключение'); + return selected; + } + return { read, select }; +} diff --git a/src/server/services/stateStore.ts b/src/server/services/stateStore.ts index 7f64335..621d893 100644 --- a/src/server/services/stateStore.ts +++ b/src/server/services/stateStore.ts @@ -13,7 +13,7 @@ import { type NormalizedServer, } from '../../shared/serverIdentity.js'; -export const STATE_SCHEMA_VERSION = 7; +export const STATE_SCHEMA_VERSION = 8; export interface AtomicWriteOptions { beforeRename?: (temporaryPath: string, filePath: string) => void; diff --git a/src/server/singbox.ts b/src/server/singbox.ts index 16ce94a..9ee2bc0 100644 --- a/src/server/singbox.ts +++ b/src/server/singbox.ts @@ -1,13 +1,20 @@ import fs from 'node:fs'; +import crypto from 'node:crypto'; import { settings } from './config.js'; import { HarborError } from '../shared/errors.js'; import { normalizeRouteRules } from '../shared/routingRules.js'; +import type { AppliedFailoverPolicy } from '../shared/failover.js'; import { atomicWriteFile, atomicWriteJson } from './services/stateStore.js'; const PROXY_TYPES = new Set(['vless', 'vmess', 'trojan', 'shadowsocks', 'hysteria2']); const MIXED_INBOUND = 'mixed-in'; const TPROXY_INBOUND = 'tproxy-in'; const DIAGNOSTICS_INBOUND = 'diagnostics-vpn-in'; +const DIAGNOSTICS_PRIMARY_INBOUND = 'diagnostics-primary-in'; +const DIAGNOSTICS_RESERVE_INBOUND = 'diagnostics-reserve-in'; +export const FAILOVER_SELECTOR_TAG = 'channel-selector'; +export const FAILOVER_PRIMARY_TAG = 'channel-primary'; +export const FAILOVER_RESERVE_TAG = 'channel-reserve'; const SNIFF_TIMEOUT = '1s'; const SNIFFERS = ['http', 'tls', 'quic']; @@ -34,18 +41,66 @@ function findOutbound(subscriptionConfig: unknown, selectedTag: unknown): ProxyO )); } +function selectedOutbound(subscriptionConfig: unknown, selectedTag: unknown, tag?: string) { + const outbound = structuredClone(findOutbound(subscriptionConfig, selectedTag)); + if (!outbound) throw new HarborError('SERVER_NOT_FOUND'); + if (tag) outbound.tag = tag; + else if (!outbound.tag) outbound.tag = 'vpn-out'; + if (outbound.type === 'vless' && !outbound.packet_encoding) outbound.packet_encoding = 'xudp'; + return outbound; +} + +export interface DualChannelConfig { + primary: { subscriptionConfig: unknown; selectedServerId: string }; + reserve: { subscriptionConfig: unknown; selectedServerId: string }; +} + +export function fingerprintConfiguredOutbound(value: unknown, selectedServerId: string) { + const outbound = structuredClone(record(value)) as ProxyOutbound; + if (!PROXY_TYPES.has(String(outbound.type || ''))) throw new HarborError('CONFIG_INVALID'); + outbound.tag = selectedServerId; + if (outbound.type === 'vless' && !outbound.packet_encoding) outbound.packet_encoding = 'xudp'; + return crypto.createHash('sha256').update(JSON.stringify(outbound)).digest('hex'); +} + +export function fingerprintSelectedOutbound(subscriptionConfig: unknown, selectedServerId: string) { + const outbound = findOutbound(subscriptionConfig, selectedServerId); + if (!outbound) throw new HarborError('SERVER_NOT_FOUND'); + return fingerprintConfiguredOutbound(outbound, selectedServerId); +} + +export function dualChannelConfigMatchesApplied( + configValue: unknown, + applied: AppliedFailoverPolicy, + expectedRole: 'primary' | 'reserve', +) { + const config = record(configValue); + const outbounds = (Array.isArray(config.outbounds) ? config.outbounds : []).map(record); + const primary = outbounds.filter(({ tag }) => tag === FAILOVER_PRIMARY_TAG); + const reserve = outbounds.filter(({ tag }) => tag === FAILOVER_RESERVE_TAG); + const selector = outbounds.find(({ tag }) => tag === FAILOVER_SELECTOR_TAG); + try { + return primary.length === 1 + && reserve.length === 1 + && fingerprintConfiguredOutbound(primary[0], applied.primary.serverId) === applied.primaryConfigFingerprint + && fingerprintConfiguredOutbound(reserve[0], applied.reserve.serverId) === applied.reserveConfigFingerprint + && selector?.type === 'selector' + && JSON.stringify(selector.outbounds) === JSON.stringify([FAILOVER_PRIMARY_TAG, FAILOVER_RESERVE_TAG]) + && selector.default === (expectedRole === 'reserve' ? FAILOVER_RESERVE_TAG : FAILOVER_PRIMARY_TAG) + && selector.interrupt_exist_connections === false + && record(config.route).final === FAILOVER_SELECTOR_TAG; + } catch { + return false; + } +} + export function buildGatewayConfig(subscriptionConfig: unknown, selectedTag: unknown, { clientDirect = false, routeRules = [], }: { clientDirect?: boolean; routeRules?: unknown } = {}) { const clientMode = settings.appMode === 'client'; const directClient = clientMode && clientDirect; - const vpnOutbound = structuredClone(findOutbound(subscriptionConfig, selectedTag)); - if (!vpnOutbound) throw new HarborError('SERVER_NOT_FOUND'); - if (!vpnOutbound.tag) vpnOutbound.tag = 'vpn-out'; - if (vpnOutbound.type === 'vless' && !vpnOutbound.packet_encoding) { - vpnOutbound.packet_encoding = 'xudp'; - } + const vpnOutbound = selectedOutbound(subscriptionConfig, selectedTag); const outboundTag = directClient ? 'direct' : vpnOutbound.tag; const inbounds = [ @@ -124,6 +179,76 @@ export function buildGatewayConfig(subscriptionConfig: unknown, selectedTag: unk }; } + +export function buildDualChannelGatewayConfig( + channels: DualChannelConfig, + { routeRules = [], defaultRole = 'primary' }: { routeRules?: unknown; defaultRole?: 'primary' | 'reserve' } = {}, +) { + if (settings.appMode === 'client') throw new Error('Dual-channel config доступен только Gateway'); + const primary = selectedOutbound( + channels.primary.subscriptionConfig, + channels.primary.selectedServerId, + FAILOVER_PRIMARY_TAG, + ); + const reserve = selectedOutbound( + channels.reserve.subscriptionConfig, + channels.reserve.selectedServerId, + FAILOVER_RESERVE_TAG, + ); + const userRules = normalizeRouteRules(routeRules) + .filter((rule) => rule.enabled) + .map((rule) => ({ + [rule.type]: [rule.value], + outbound: rule.outbound === 'vpn' ? FAILOVER_SELECTOR_TAG : 'direct', + })); + const userInbounds = [TPROXY_INBOUND, MIXED_INBOUND]; + return { + log: { level: settings.logLevel, timestamp: true }, + experimental: { + cache_file: { enabled: true, path: settings.cachePath }, + clash_api: { external_controller: `127.0.0.1:${settings.singboxApiPort}` }, + }, + dns: { independent_cache: true }, + inbounds: [ + { type: 'tproxy', tag: TPROXY_INBOUND, listen: '::', listen_port: settings.tproxyPort }, + { type: 'mixed', tag: MIXED_INBOUND, listen: settings.bindIp, listen_port: settings.proxyPort, set_system_proxy: false }, + { type: 'mixed', tag: DIAGNOSTICS_INBOUND, listen: '127.0.0.1', listen_port: settings.diagnosticsProxyPort, set_system_proxy: false }, + { type: 'mixed', tag: DIAGNOSTICS_PRIMARY_INBOUND, listen: '127.0.0.1', listen_port: settings.failoverPrimaryProxyPort, set_system_proxy: false }, + { type: 'mixed', tag: DIAGNOSTICS_RESERVE_INBOUND, listen: '127.0.0.1', listen_port: settings.failoverReserveProxyPort, set_system_proxy: false }, + ], + outbounds: [ + primary, + reserve, + { + type: 'selector', + tag: FAILOVER_SELECTOR_TAG, + outbounds: [FAILOVER_PRIMARY_TAG, FAILOVER_RESERVE_TAG], + default: defaultRole === 'reserve' ? FAILOVER_RESERVE_TAG : FAILOVER_PRIMARY_TAG, + interrupt_exist_connections: false, + }, + { type: 'direct', tag: 'direct' }, + ], + route: { + rule_set: [], + rules: [ + { + inbound: [TPROXY_INBOUND, MIXED_INBOUND, DIAGNOSTICS_INBOUND, DIAGNOSTICS_PRIMARY_INBOUND, DIAGNOSTICS_RESERVE_INBOUND], + action: 'sniff', + sniffer: SNIFFERS, + timeout: SNIFF_TIMEOUT, + }, + { inbound: [DIAGNOSTICS_PRIMARY_INBOUND], outbound: FAILOVER_PRIMARY_TAG }, + { inbound: [DIAGNOSTICS_RESERVE_INBOUND], outbound: FAILOVER_RESERVE_TAG }, + { inbound: [DIAGNOSTICS_INBOUND], outbound: FAILOVER_SELECTOR_TAG }, + ...userRules, + { inbound: userInbounds, outbound: FAILOVER_SELECTOR_TAG }, + ], + final: FAILOVER_SELECTOR_TAG, + auto_detect_interface: true, + }, + }; +} + export function writeSingboxConfig(config: unknown) { atomicWriteJson(settings.configPath, config); } diff --git a/src/server/singboxRuntime.ts b/src/server/singboxRuntime.ts index d9bf59a..f1723c4 100644 --- a/src/server/singboxRuntime.ts +++ b/src/server/singboxRuntime.ts @@ -19,6 +19,23 @@ export function createSingboxRuntime({ const state = () => ({ running: Boolean(child), startedAt }); + function checkConfig(config: unknown) { + const directory = fs.mkdtempSync(`${configPath}.check-`); + const candidatePath = `${directory}/config.json`; + try { + fs.writeFileSync(candidatePath, JSON.stringify(config)); + const check = spawnSync('sing-box', ['check', '-c', candidatePath], { encoding: 'utf8' }); + if (check.status !== 0) { + throw new HarborError('CONFIG_INVALID', { + cause: new Error((check.stderr || check.stdout || check.error?.message || 'sing-box check failed').trim()), + }); + } + return { valid: true }; + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + } + async function stop() { if (gateway) setGatewayInterception(false, tproxyChain); if (!child) { @@ -100,6 +117,7 @@ export function createSingboxRuntime({ get running() { return Boolean(child); }, get startedAt() { return startedAt; }, refresh: async () => state(), + checkConfig, apply, restart: () => apply({ force: true }), stop, diff --git a/src/shared/activityJournal.ts b/src/shared/activityJournal.ts new file mode 100644 index 0000000..deb2c4b --- /dev/null +++ b/src/shared/activityJournal.ts @@ -0,0 +1,156 @@ +export const ACTIVITY_JOURNAL_RETENTION_DAYS = 30; +export const ACTIVITY_JOURNAL_MAX_EVENTS = 10_000; + +export const ACTIVITY_EVENT_TYPES = [ + 'connection.started', 'connection.stopped', 'connection.failed', + 'subscription.added', 'subscription.refreshed', 'subscription.refresh_failed', 'subscription.deleted', + 'failover.enabled', 'failover.disabled', 'failover.paused', 'failover.resumed', + 'failover.waiting_for_idle', 'failover.switched', 'failover.switch_failed', + 'failover.both_unhealthy', 'failover.recovered', 'journal.recovered', +] as const; + +export type ActivityEventType = typeof ACTIVITY_EVENT_TYPES[number]; +export type ActivityEventSeverity = 'info' | 'warning' | 'error'; +export type ActivityEventSource = 'connection' | 'subscription' | 'failover' | 'storage'; + +export interface ActivityJournalEvent { + id: string; + occurredAt: string; + type: ActivityEventType | 'unknown'; + severity: ActivityEventSeverity; + source: ActivityEventSource; + dedupeKey: string | null; + data: Record; +} + +export type ActivityJournalEventInput = Omit; + +export interface ActivityJournalPage { + events: ActivityJournalEvent[]; + nextCursor: string | null; + retentionDays: 30; + generatedAt: string; + storage: { status: 'ready' | 'error'; errorCode: string | null }; +} + +const ALLOWED_DATA_KEYS: Record = { + 'connection.started': ['profileLabel', 'serverLabel'], + 'connection.stopped': [], + 'connection.failed': ['errorCode'], + 'subscription.added': ['profileId', 'profileLabel', 'host', 'serverCount'], + 'subscription.refreshed': ['profileId', 'profileLabel', 'host', 'serverCount', 'added', 'removed'], + 'subscription.refresh_failed': ['profileId', 'profileLabel', 'host', 'errorCode'], + 'subscription.deleted': ['profileId', 'profileLabel'], + 'failover.enabled': ['primaryLabel', 'reserveLabel'], + 'failover.disabled': [], + 'failover.paused': [], + 'failover.resumed': [], + 'failover.waiting_for_idle': ['fromRole', 'toRole', 'reason'], + 'failover.switched': ['fromRole', 'toRole', 'primaryLabel', 'reserveLabel', 'reason', 'manual'], + 'failover.switch_failed': ['fromRole', 'toRole', 'reason', 'errorCode'], + 'failover.both_unhealthy': ['reason'], + 'failover.recovered': ['role', 'reason'], + 'journal.recovered': [], +}; + +const typeSet = new Set(ACTIVITY_EVENT_TYPES); +const severitySet = new Set(['info', 'warning', 'error']); +const sourceSet = new Set(['connection', 'subscription', 'failover', 'storage']); +const record = (value: unknown): Record => ( + value && typeof value === 'object' && !Array.isArray(value) ? value as Record : {} +); + +const LABEL_KEYS = new Set(['profileLabel', 'serverLabel', 'primaryLabel', 'reserveLabel']); +const ROLE_KEYS = new Set(['fromRole', 'toRole', 'role']); +const SAFE_LABEL_FALLBACKS: Record = { + profileLabel: 'Подписка', + serverLabel: 'Сервер', + primaryLabel: 'Основной канал', + reserveLabel: 'Резервный канал', +}; + +function safeScalar(key: string, value: unknown) { + if (value === null || typeof value === 'boolean') return value; + if (typeof value === 'number' && Number.isSafeInteger(value) && value >= 0) return value; + if (typeof value !== 'string' || value.length > 120 || /[\r\n]/.test(value)) { + throw new TypeError('Unsafe journal value'); + } + if (LABEL_KEYS.has(key) && ( + !value.trim() + || /(?:[a-z][a-z0-9+.-]*:\/\/)|[\/?#@\\]/i.test(value) + || /(?:^|\D)(?:\d{1,3}\.){3}\d{1,3}(?:\D|$)/.test(value) + || /(?:^|[^0-9a-f])(?:[0-9a-f]{0,4}:){2,}[0-9a-f]{0,4}(?:[^0-9a-f]|$)/i.test(value) + )) return SAFE_LABEL_FALLBACKS[key]; + if (ROLE_KEYS.has(key) && !['primary', 'reserve'].includes(value)) throw new TypeError('Unsafe journal role'); + if (key === 'errorCode' && !/^[A-Z0-9_]{1,50}$/.test(value)) throw new TypeError('Unsafe journal error code'); + if (key === 'reason' && !/^[a-z0-9-]{1,80}$/.test(value)) throw new TypeError('Unsafe journal reason'); + if (key === 'profileId' && !/^[a-zA-Z0-9_-]{1,80}$/.test(value)) throw new TypeError('Unsafe journal profile id'); + if (key === 'host' && ( + !/^[a-z0-9.-]{1,120}$/i.test(value) + || /^(?:\d{1,3}\.){3}\d{1,3}$/.test(value) + || value.includes('..') + )) return 'Провайдер'; + return value; +} + +export function normalizeActivityEventInput(value: unknown): ActivityJournalEventInput { + const candidate = record(value); + const type = String(candidate.type || '') as ActivityEventType; + const severity = String(candidate.severity || '') as ActivityEventSeverity; + const source = String(candidate.source || '') as ActivityEventSource; + if (!typeSet.has(type) || !severitySet.has(severity) || !sourceSet.has(source)) { + throw new TypeError('Unknown journal event'); + } + const inputData = record(candidate.data); + const allowed = new Set(ALLOWED_DATA_KEYS[type]); + if (Object.keys(inputData).some((key) => !allowed.has(key))) throw new TypeError('Unsafe journal data key'); + const data = Object.fromEntries(Object.entries(inputData).map(([key, item]) => [key, safeScalar(key, item)])); + const dedupeKey = candidate.dedupeKey == null ? null : String(candidate.dedupeKey).trim(); + if (dedupeKey !== null && ( + !dedupeKey.startsWith(`${type}:`) + || !/^[a-zA-Z0-9_.:-]{1,160}$/.test(dedupeKey) + )) { + throw new TypeError('Invalid journal dedupe key'); + } + return { type, severity, source, dedupeKey, data }; +} + +export function normalizeStoredActivityEvent(value: unknown): ActivityJournalEvent | null { + const candidate = record(value); + const id = typeof candidate.id === 'string' && /^[a-f0-9-]{20,50}$/i.test(candidate.id) ? candidate.id : ''; + const occurredAt = typeof candidate.occurredAt === 'string' && Number.isFinite(Date.parse(candidate.occurredAt)) + ? candidate.occurredAt + : ''; + if (!id || !occurredAt) return null; + try { + return { id, occurredAt, ...normalizeActivityEventInput(candidate) }; + } catch { + const severity = String(candidate.severity || '') as ActivityEventSeverity; + const source = String(candidate.source || '') as ActivityEventSource; + return severitySet.has(severity) && sourceSet.has(source) && typeof candidate.type === 'string' + && /^[a-z][a-z0-9_.-]{0,79}$/.test(candidate.type) + ? { id, occurredAt, type: 'unknown', severity, source, dedupeKey: null, data: {} } + : null; + } +} + +export function assertActivityJournalPage(value: unknown): ActivityJournalPage { + const candidate = record(value); + const events = Array.isArray(candidate.events) ? candidate.events.map(normalizeStoredActivityEvent) : []; + const storage = record(candidate.storage); + if ( + !Array.isArray(candidate.events) || events.some((event) => event === null) + || !(candidate.nextCursor === null || typeof candidate.nextCursor === 'string') + || candidate.retentionDays !== ACTIVITY_JOURNAL_RETENTION_DAYS + || typeof candidate.generatedAt !== 'string' || !Number.isFinite(Date.parse(candidate.generatedAt)) + || !['ready', 'error'].includes(String(storage.status || '')) + || !(storage.errorCode === null || typeof storage.errorCode === 'string') + ) throw new TypeError('Invalid activity journal page'); + return { + events: events as ActivityJournalEvent[], + nextCursor: candidate.nextCursor as string | null, + retentionDays: 30, + generatedAt: candidate.generatedAt, + storage: { status: storage.status as 'ready' | 'error', errorCode: storage.errorCode as string | null }, + }; +} diff --git a/src/shared/contracts/state.ts b/src/shared/contracts/state.ts index 070c2ae..704726d 100644 --- a/src/shared/contracts/state.ts +++ b/src/shared/contracts/state.ts @@ -8,6 +8,16 @@ import { normalizeDiagnosticSettings, type DiagnosticSettings, } from '../connectivityDiagnostics.js'; +import { + createIdleFailoverSnapshot, + normalizeAppliedFailoverPolicy, + normalizeFailoverPolicy, + normalizeFailoverRuntimeState, + type AppliedFailoverPolicy, + type FailoverPolicy, + type FailoverRuntimeState, + type FailoverSnapshot, +} from '../failover.js'; export type HarborMode = 'client' | 'gateway'; export type ConnectionState = 'running' | 'stopped'; @@ -83,6 +93,7 @@ export interface StateSnapshot { lastError: string | null; }; diagnostics: DiagnosticSettings; + failover: FailoverSnapshot; route: { rulesContractVersion?: typeof ROUTE_RULES_CONTRACT_VERSION; mode: string; @@ -122,6 +133,9 @@ export interface PersistedState extends Record { connectionDesired?: ConnectionState; gatewayAutoEnabled?: boolean; diagnostics: DiagnosticSettings; + failoverPolicy: FailoverPolicy; + failoverRuntimeState: FailoverRuntimeState; + appliedFailoverPolicy: AppliedFailoverPolicy | null; } // Legacy fields are derived in memory for bounded callers during the v5 cutover. @@ -293,6 +307,9 @@ export function normalizeStoredState(value: unknown): StoredState { ? state.routeRulesRevision : 0, diagnostics: normalizeDiagnosticSettings(state.diagnostics), + failoverPolicy: normalizeFailoverPolicy(state.failoverPolicy), + failoverRuntimeState: normalizeFailoverRuntimeState(state.failoverRuntimeState), + appliedFailoverPolicy: normalizeAppliedFailoverPolicy(state.appliedFailoverPolicy), subscriptionUrl: selectedProfile?.subscriptionUrl || '', selectedServerId, selectedTag: selectedServer?.label || '', @@ -327,6 +344,7 @@ export function createStateSnapshot({ appMode, configExists, operation = { kind: null, status: 'idle', startedAt: null, error: null }, + failoverSnapshot, now = new Date(), }: { storedState: unknown; @@ -336,6 +354,7 @@ export function createStateSnapshot({ configExists: boolean; subscriptionHost?: string; operation?: OperationState; + failoverSnapshot?: FailoverSnapshot | null; now?: Date; }): StateSnapshot { const stored = normalizeStoredState(storedState); @@ -391,6 +410,7 @@ export function createStateSnapshot({ lastError: null, }, diagnostics: stored.diagnostics, + failover: failoverSnapshot || createIdleFailoverSnapshot(stored.failoverPolicy), route: { rulesContractVersion: ROUTE_RULES_CONTRACT_VERSION, mode: routeMode, @@ -429,18 +449,24 @@ export function assertStateSnapshot(snapshot: unknown): StateSnapshot { const candidateWithDiagnostics = rawCandidate && rawCandidate.diagnostics === undefined ? { ...rawCandidate, diagnostics: normalizeDiagnosticSettings(null) } : rawCandidate; - const candidate = candidateWithDiagnostics?.route?.rulesContractVersion === undefined - && Array.isArray(candidateWithDiagnostics?.route?.localRules) - && Array.isArray(candidateWithDiagnostics?.route?.activeLocalRules) + const candidateWithFailover = candidateWithDiagnostics && candidateWithDiagnostics.failover === undefined ? { ...candidateWithDiagnostics, - route: { - ...candidateWithDiagnostics.route, - localRules: candidateWithDiagnostics.route.localRules.map(legacyRule), - activeLocalRules: candidateWithDiagnostics.route.activeLocalRules.map(legacyRule), - }, + failover: createIdleFailoverSnapshot(normalizeFailoverPolicy(null)), } : candidateWithDiagnostics; + const candidate = candidateWithFailover?.route?.rulesContractVersion === undefined + && Array.isArray(candidateWithFailover?.route?.localRules) + && Array.isArray(candidateWithFailover?.route?.activeLocalRules) + ? { + ...candidateWithFailover, + route: { + ...candidateWithFailover.route, + localRules: candidateWithFailover.route.localRules.map(legacyRule), + activeLocalRules: candidateWithFailover.route.activeLocalRules.map(legacyRule), + }, + } + : candidateWithFailover; const validDate = (value: unknown) => typeof value === 'string' && Number.isFinite(Date.parse(value)); const nullableDate = (value: unknown) => value === null || validDate(value); const nullableString = (value: unknown) => value === null || typeof value === 'string'; @@ -486,6 +512,37 @@ export function assertStateSnapshot(snapshot: unknown): StateSnapshot { return false; } }; + const validFailover = (value: FailoverSnapshot) => { + const channel = (item: FailoverSnapshot['primary']) => ( + item && typeof item.target?.profileId === 'string' && typeof item.target?.serverId === 'string' + && ['healthy', 'unhealthy', 'unknown', 'not-monitoring'].includes(item.health) + && Array.isArray(item.failingServiceIds) && item.failingServiceIds.every((id) => typeof id === 'string') + && nullableDate(item.checkedAt) && nullableDate(item.stateSince) + ); + const activity = value.trafficActivity; + try { + normalizeFailoverPolicy(value.policy, { strict: true }); + } catch { + return false; + } + return channel(value.primary) && channel(value.reserve) + && nullableDate(value.nextDecisionAt) && nullableString(value.reason) + && (activity === null || ( + ['active', 'quiet', 'unknown'].includes(activity.state) + && validDate(activity.observedAt) + && [activity.windowMs, activity.thresholdBytesPerSecond, activity.totalBytesPerSecond, activity.transmittingConnections] + .every((number) => Number.isFinite(number) && number >= 0) + && nullableDate(activity.quietSince) + && (activity.switchTarget === null || ['primary', 'reserve'].includes(activity.switchTarget)) + && Array.isArray(activity.blockers) + && activity.blockers.length <= 3 + && activity.blockers.every((blocker) => ( + typeof blocker.device === 'string' && typeof blocker.service === 'string' + && Number.isFinite(blocker.uploadBytesPerSecond) && blocker.uploadBytesPerSecond >= 0 + && Number.isFinite(blocker.downloadBytesPerSecond) && blocker.downloadBytesPerSecond >= 0 + )) + )); + }; if ( !snapshot || @@ -516,6 +573,18 @@ export function assertStateSnapshot(snapshot: unknown): StateSnapshot { !nullableDate(candidate.connection.startedAt) || !nullableString(candidate.connection.lastError) || !validDiagnostics(candidate.diagnostics) || + !candidate.failover || + typeof candidate.failover.observationEpoch !== 'string' || + !Number.isSafeInteger(candidate.failover.observationSequence) || + candidate.failover.observationSequence < 0 || + typeof candidate.failover.enabled !== 'boolean' || + typeof candidate.failover.paused !== 'boolean' || + typeof candidate.failover.configured !== 'boolean' || + !candidate.failover.policy || + !['inactive', 'active', 'pending', 'passive-loaded'].includes(candidate.failover.activation) || + !['primary', 'reserve', 'other', 'none'].includes(candidate.failover.currentRole) || + !['idle', 'observing', 'primary', 'reserve', 'waiting-for-idle', 'blocked', 'switching', 'error'].includes(candidate.failover.status) || + !validFailover(candidate.failover) || !candidate.route || ![undefined, ROUTE_RULES_CONTRACT_VERSION].includes(candidate.route.rulesContractVersion) || typeof candidate.route.mode !== 'string' || diff --git a/src/shared/failover.ts b/src/shared/failover.ts new file mode 100644 index 0000000..65c9aaa --- /dev/null +++ b/src/shared/failover.ts @@ -0,0 +1,355 @@ +export type FailoverRole = 'primary' | 'reserve'; +export type FailoverHealth = 'healthy' | 'unhealthy' | 'unknown' | 'not-monitoring'; + +export interface FailoverTarget { + profileId: string; + serverId: string; +} + +export interface FailoverCheck { + serviceId: string; + timeoutMs: number; +} + +export interface FailoverPolicy { + version: 1; + enabled: boolean; + paused: boolean; + primary: FailoverTarget; + reserve: FailoverTarget; + checks: FailoverCheck[]; + intervalMs: number; + failureWindowMs: number; + recoveryWindowMs: number; + trafficGuard: { + enabled: boolean; + thresholdBytesPerSecond: number; + quietWindowMs: number; + }; + minimumReserveMs: number; + flapProtection: { + count: number; + windowMs: number; + quarantineMs: number; + }; +} + +export interface FailoverRuntimeState { + lastSwitchAt: string | null; + holdUntil: string | null; + primaryQuarantineUntil: string | null; + failoverHistory: string[]; + reasonCode: string | null; +} + +export interface AppliedFailoverPolicy { + primary: FailoverTarget; + reserve: FailoverTarget; + primaryConfigFingerprint: string; + reserveConfigFingerprint: string; +} + +export interface FailoverActivity { + state: 'active' | 'quiet' | 'unknown'; + observedAt: string; + windowMs: number; + thresholdBytesPerSecond: number; + totalBytesPerSecond: number; + transmittingConnections: number; + quietSince: string | null; + switchTarget: FailoverRole | null; + blockers: Array<{ + device: string; + service: string; + uploadBytesPerSecond: number; + downloadBytesPerSecond: number; + }>; +} + +export interface FailoverSnapshot { + observationEpoch: string; + observationSequence: number; + configured: boolean; + enabled: boolean; + paused: boolean; + activation: 'inactive' | 'active' | 'pending' | 'passive-loaded'; + currentRole: FailoverRole | 'other' | 'none'; + status: 'idle' | 'observing' | 'primary' | 'reserve' | 'waiting-for-idle' | 'blocked' | 'switching' | 'error'; + primary: { target: FailoverTarget; health: FailoverHealth; failingServiceIds: string[]; checkedAt: string | null; stateSince: string | null }; + reserve: { target: FailoverTarget; health: FailoverHealth; failingServiceIds: string[]; checkedAt: string | null; stateSince: string | null }; + nextDecisionAt: string | null; + reason: string | null; + trafficActivity: FailoverActivity | null; + policy: FailoverPolicy; +} + +export interface FailoverDecisionMemory { + primaryFailedSince: number | null; + primaryRecoveredSince: number | null; + quietSince: number | null; +} + +export interface FailoverDecisionInput { + now: number; + policy: FailoverPolicy; + currentRole: FailoverRole; + primaryHealth: Exclude; + reserveHealth: Exclude; + activity: 'active' | 'quiet' | 'unknown'; + holdUntil?: number | null; + primaryQuarantineUntil?: number | null; + memory?: FailoverDecisionMemory; +} + +export interface FailoverDecision { + status: FailoverSnapshot['status']; + reason: string; + switchTo: FailoverRole | null; + memory: FailoverDecisionMemory; + nextDecisionAt: number | null; +} + +const text = (value: unknown) => typeof value === 'string' ? value.trim() : ''; +const record = (value: unknown): Record => ( + value && typeof value === 'object' && !Array.isArray(value) ? value as Record : {} +); +const bounded = (value: unknown, fallback: number, minimum: number, maximum: number) => { + const candidate = Number(value); + return Number.isFinite(candidate) ? Math.min(maximum, Math.max(minimum, Math.round(candidate))) : fallback; +}; +const equivalent = (left: unknown, right: unknown): boolean => { + if (Array.isArray(left) || Array.isArray(right)) { + return Array.isArray(left) && Array.isArray(right) + && left.length === right.length + && left.every((value, index) => equivalent(value, right[index])); + } + if (left && right && typeof left === 'object' && typeof right === 'object') { + const leftRecord = left as Record; + const rightRecord = right as Record; + const leftKeys = Object.keys(leftRecord).sort(); + const rightKeys = Object.keys(rightRecord).sort(); + return leftKeys.length === rightKeys.length + && leftKeys.every((key, index) => key === rightKeys[index] && equivalent(leftRecord[key], rightRecord[key])); + } + return Object.is(left, right); +}; + +export const DEFAULT_FAILOVER_POLICY: FailoverPolicy = Object.freeze({ + version: 1, + enabled: false, + paused: false, + primary: { profileId: '', serverId: '' }, + reserve: { profileId: '', serverId: '' }, + checks: [{ serviceId: 'youtube', timeoutMs: 6_000 }, { serviceId: 'google', timeoutMs: 6_000 }], + intervalMs: 60_000, + failureWindowMs: 120_000, + recoveryWindowMs: 900_000, + trafficGuard: { enabled: true, thresholdBytesPerSecond: 32 * 1024, quietWindowMs: 30_000 }, + minimumReserveMs: 600_000, + flapProtection: { count: 3, windowMs: 86_400_000, quarantineMs: 21_600_000 }, +}); + +export const DEFAULT_FAILOVER_RUNTIME_STATE: FailoverRuntimeState = Object.freeze({ + lastSwitchAt: null, + holdUntil: null, + primaryQuarantineUntil: null, + failoverHistory: [], + reasonCode: null, +}); + +function target(value: unknown): FailoverTarget { + const candidate = record(value); + return { profileId: text(candidate.profileId), serverId: text(candidate.serverId) }; +} + +export function normalizeFailoverPolicy(value: unknown, { strict = false } = {}): FailoverPolicy { + const candidate = record(value); + const requestedChecks = Array.isArray(candidate.checks) ? candidate.checks : DEFAULT_FAILOVER_POLICY.checks; + const checks = requestedChecks.map((value) => { + const check = record(value); + return { + serviceId: text(check.serviceId).slice(0, 100), + timeoutMs: bounded(check.timeoutMs, 6_000, 2_000, 30_000), + }; + }).filter(({ serviceId }, index, all) => serviceId && all.findIndex((item) => item.serviceId === serviceId) === index).slice(0, 10); + const trafficGuard = record(candidate.trafficGuard); + const flapProtection = record(candidate.flapProtection); + const policy: FailoverPolicy = { + version: 1, + enabled: candidate.enabled === true, + paused: candidate.paused === true, + primary: target(candidate.primary), + reserve: target(candidate.reserve), + checks, + intervalMs: bounded(candidate.intervalMs, 60_000, 15_000, 900_000), + failureWindowMs: 0, + recoveryWindowMs: bounded(candidate.recoveryWindowMs, 900_000, 60_000, 86_400_000), + trafficGuard: { + enabled: trafficGuard.enabled !== false, + thresholdBytesPerSecond: bounded(trafficGuard.thresholdBytesPerSecond, 32 * 1024, 1024, 100 * 1024 * 1024), + quietWindowMs: bounded(trafficGuard.quietWindowMs, 30_000, 5_000, 600_000), + }, + minimumReserveMs: bounded(candidate.minimumReserveMs, 600_000, 60_000, 86_400_000), + flapProtection: { + count: bounded(flapProtection.count, 3, 2, 10), + windowMs: bounded(flapProtection.windowMs, 86_400_000, 3_600_000, 72 * 3_600_000), + quarantineMs: bounded(flapProtection.quarantineMs, 21_600_000, 600_000, 7 * 86_400_000), + }, + }; + policy.failureWindowMs = bounded( + candidate.failureWindowMs, + 120_000, + policy.intervalMs * 2, + 1_800_000, + ); + if (strict && !equivalent(policy, value)) { + throw new TypeError('Invalid failover policy'); + } + return policy; +} + +export function normalizeFailoverRuntimeState(value: unknown): FailoverRuntimeState { + const candidate = record(value); + const date = (value: unknown) => typeof value === 'string' && Number.isFinite(Date.parse(value)) ? value : null; + return { + lastSwitchAt: date(candidate.lastSwitchAt), + holdUntil: date(candidate.holdUntil), + primaryQuarantineUntil: date(candidate.primaryQuarantineUntil), + failoverHistory: (Array.isArray(candidate.failoverHistory) ? candidate.failoverHistory : []) + .map(date).filter((item): item is string => Boolean(item)).slice(-20), + reasonCode: text(candidate.reasonCode) || null, + }; +} + +export function normalizeAppliedFailoverPolicy(value: unknown): AppliedFailoverPolicy | null { + if (!value) return null; + const candidate = record(value); + const primary = target(candidate.primary); + const reserve = target(candidate.reserve); + const primaryConfigFingerprint = text(candidate.primaryConfigFingerprint); + const reserveConfigFingerprint = text(candidate.reserveConfigFingerprint); + return primary.profileId && primary.serverId && reserve.profileId && reserve.serverId + && /^[a-f0-9]{64}$/.test(primaryConfigFingerprint) + && /^[a-f0-9]{64}$/.test(reserveConfigFingerprint) + ? { primary, reserve, primaryConfigFingerprint, reserveConfigFingerprint } + : null; +} + +export function isFailoverConfigured(policy: FailoverPolicy) { + return Boolean( + policy.primary.profileId && policy.primary.serverId + && policy.reserve.profileId && policy.reserve.serverId + && policy.checks.length + && (policy.primary.profileId !== policy.reserve.profileId + || policy.primary.serverId !== policy.reserve.serverId) + ); +} + +export function createIdleFailoverSnapshot( + policy: FailoverPolicy, + observationEpoch = '', + observationSequence = 0, +): FailoverSnapshot { + const channel = (target: FailoverTarget) => ({ + target, + health: 'not-monitoring' as const, + failingServiceIds: [], + checkedAt: null, + stateSince: null, + }); + return { + observationEpoch, + observationSequence, + configured: isFailoverConfigured(policy), + enabled: policy.enabled, + paused: policy.paused, + activation: 'inactive', + currentRole: 'none', + status: 'idle', + primary: channel(policy.primary), + reserve: channel(policy.reserve), + nextDecisionAt: null, + reason: null, + trafficActivity: null, + policy, + }; +} + +export function nextFailoverDecision(input: FailoverDecisionInput): FailoverDecision { + const memory = input.memory || { + primaryFailedSince: null, + primaryRecoveredSince: null, + quietSince: null, + }; + const next = { ...memory }; + if (!input.policy.enabled || input.policy.paused) { + return { status: 'idle', reason: input.policy.paused ? 'paused' : 'disabled', switchTo: null, memory: next, nextDecisionAt: null }; + } + if (input.primaryHealth === 'unhealthy' && input.reserveHealth === 'unhealthy') { + next.quietSince = null; + return { status: 'blocked', reason: 'both-unhealthy', switchTo: null, memory: next, nextDecisionAt: null }; + } + + let target: FailoverRole | null = null; + let readyAt: number | null = null; + if (input.currentRole === 'primary') { + next.primaryRecoveredSince = null; + if (input.primaryHealth !== 'unhealthy') { + next.primaryFailedSince = null; + next.quietSince = null; + return { status: 'primary', reason: input.primaryHealth === 'healthy' ? 'primary-healthy' : 'health-unknown', switchTo: null, memory: next, nextDecisionAt: null }; + } + next.primaryFailedSince ??= input.now; + readyAt = next.primaryFailedSince + input.policy.failureWindowMs; + if (input.now < readyAt || input.reserveHealth !== 'healthy') { + next.quietSince = null; + return { status: 'observing', reason: input.reserveHealth === 'healthy' ? 'failure-window' : 'reserve-not-healthy', switchTo: null, memory: next, nextDecisionAt: readyAt }; + } + target = 'reserve'; + } else { + next.primaryFailedSince = null; + if (input.primaryHealth !== 'healthy') { + next.primaryRecoveredSince = null; + next.quietSince = null; + return { status: 'reserve', reason: 'primary-not-recovered', switchTo: null, memory: next, nextDecisionAt: null }; + } + if ((input.primaryQuarantineUntil || 0) > input.now) { + next.primaryRecoveredSince = null; + next.quietSince = null; + return { + status: 'reserve', + reason: 'recovery-hold', + switchTo: null, + memory: next, + nextDecisionAt: input.primaryQuarantineUntil || null, + }; + } + next.primaryRecoveredSince ??= input.now; + const recoveredAt = next.primaryRecoveredSince + input.policy.recoveryWindowMs; + readyAt = input.reserveHealth === 'unhealthy' + ? recoveredAt + : Math.max(recoveredAt, input.holdUntil || 0, input.primaryQuarantineUntil || 0); + if (input.now < readyAt) { + next.quietSince = null; + return { status: 'reserve', reason: 'recovery-hold', switchTo: null, memory: next, nextDecisionAt: readyAt }; + } + target = 'primary'; + } + + if (input.policy.trafficGuard.enabled) { + if (input.activity === 'unknown') { + next.quietSince = null; + return { status: 'blocked', reason: 'activity-unknown', switchTo: null, memory: next, nextDecisionAt: null }; + } + if (input.activity === 'active') { + next.quietSince = null; + return { status: 'waiting-for-idle', reason: 'active-traffic', switchTo: null, memory: next, nextDecisionAt: null }; + } + next.quietSince ??= input.now; + readyAt = next.quietSince + input.policy.trafficGuard.quietWindowMs; + if (input.now < readyAt) { + return { status: 'waiting-for-idle', reason: 'quiet-window', switchTo: null, memory: next, nextDecisionAt: readyAt }; + } + } + return { status: 'switching', reason: target === 'reserve' ? 'primary-failed' : 'primary-recovered', switchTo: target, memory: next, nextDecisionAt: null }; +} diff --git a/src/shared/versions.ts b/src/shared/versions.ts index 8e80f7e..66a176e 100644 --- a/src/shared/versions.ts +++ b/src/shared/versions.ts @@ -1,7 +1,7 @@ export const HARBOR_VERSIONS = Object.freeze({ - macClient: '0.27.0', - gatewayClient: '0.28.0', - gatewayBackend: '0.28.0', + macClient: '0.28.0', + gatewayClient: '0.29.0', + gatewayBackend: '0.29.0', }); export interface ParsedVersion { diff --git a/src/web/App.tsx b/src/web/App.tsx index 7e28ab8..ab4bf6f 100644 --- a/src/web/App.tsx +++ b/src/web/App.tsx @@ -25,6 +25,7 @@ const componentActions = { setDevicePolicy: api.devices.setPolicy, pingServers: api.servers.ping, runConnectivityDiagnostics: api.diagnostics.connectivity, + loadActivityJournal: api.activityJournal.page, }; interface UiError { @@ -51,6 +52,10 @@ const operationErrorContext: Record = { 'subscription-refresh': 'subscription', 'subscription-forget': 'subscription', 'route-rules': 'routing', + 'failover-save': 'failover', + 'failover-pause': 'failover', + 'failover-resume': 'failover', + 'failover-switch': 'failover', }; function asHarborApiError(error: unknown) { @@ -304,6 +309,26 @@ export function App() { () => api.diagnostics.updateSettings(settings, revisionRef.current), 'diagnostics', )} + onSaveFailover={(policy: unknown) => run( + 'failover', + () => api.failover.save(policy, revisionRef.current), + 'failover', + )} + onPauseFailover={(paused: boolean) => run( + 'failover', + () => api.failover.pause(paused, revisionRef.current), + 'failover', + )} + onSwitchFailover={(role: 'primary' | 'reserve') => run( + 'failover', + () => api.failover.switch(role, revisionRef.current), + 'failover', + )} + onCheckFailover={() => run( + 'failover', + () => api.failover.check(), + 'failover', + )} onDismissError={() => { setError(null); setDismissedCanonicalError(canonicalErrorId); diff --git a/src/web/api/harborClient.ts b/src/web/api/harborClient.ts index 3d451ea..be56381 100644 --- a/src/web/api/harborClient.ts +++ b/src/web/api/harborClient.ts @@ -192,6 +192,26 @@ export const api = { }, ), }, + failover: { + save: (policy: unknown, expectedRevision: number) => request('/api/failover', { + method: 'PUT', + body: JSON.stringify({ policy, expectedRevision }), + }), + pause: (paused: boolean, expectedRevision: number) => request('/api/failover/pause', { + method: 'POST', + body: JSON.stringify({ paused, expectedRevision }), + }), + switch: (role: 'primary' | 'reserve', expectedRevision: number) => request('/api/failover/switch', { + method: 'POST', + body: JSON.stringify({ role, expectedRevision }), + }), + check: () => request('/api/failover/check', { + method: 'POST', + }), + }, + activityJournal: { + page: (cursor: string | null = null) => request(`/api/activity-journal?limit=50${cursor ? `&cursor=${encodeURIComponent(cursor)}` : ''}`), + }, singbox: { stop: () => request('/api/singbox/stop', { method: 'POST' }), restart: () => request('/api/singbox/restart', { method: 'POST' }), @@ -237,6 +257,7 @@ export function parseHarborState(value: unknown): HarborClientState { selection: snapshot.selection, connection: snapshot.connection, diagnostics: snapshot.diagnostics, + failover: snapshot.failover, route: snapshot.route, operation: snapshot.operation, servers: snapshot.servers, diff --git a/src/web/components/ClientOverviewPage.tsx b/src/web/components/ClientOverviewPage.tsx index fe2ffac..d153400 100644 --- a/src/web/components/ClientOverviewPage.tsx +++ b/src/web/components/ClientOverviewPage.tsx @@ -47,6 +47,13 @@ import { InstructionsToggle, useInstructionsFeature, } from '../features/instructions/index.js'; +import { FailoverPanel, FailoverToggle, useFailoverFeature } from '../features/failover/index.js'; +import { + ActivityJournalPanel, + ActivityJournalToggle, + useActivityJournalFeature, +} from '../features/activity-journal/index.js'; +import type { FailoverPolicy } from '../../shared/failover.js'; import { HARBOR_VERSIONS, parseVersion, @@ -65,9 +72,33 @@ const VERSION_PARTS = [ ] as const; const DRAWER_SWITCH_MS = 620; -const DRAWER_ORDER = ['subscription', 'instructions', 'devices', 'diagnostics', 'routing'] as const; +const DRAWER_ORDER = ['subscription', 'failover', 'instructions', 'devices', 'diagnostics', 'routing', 'journal'] as const; type DrawerKey = typeof DRAWER_ORDER[number]; +const failoverReasonLabel = (reason: string | null) => ({ + 'primary-healthy': 'основной работает', + 'health-unknown': 'ожидаем проверку', + 'failure-window': 'подтверждаем сбой', + 'reserve-not-healthy': 'резерв не подтверждён', + 'both-unhealthy': 'оба канала недоступны', + 'primary-not-recovered': 'основной восстанавливается', + 'recovery-hold': 'проверяем стабильность', + 'activity-unknown': 'активность неизвестна', + 'active-traffic': 'ждём завершения работы', + 'quiet-window': 'проверяем тишину', + 'primary-failed': 'основной недоступен', + 'primary-recovered': 'основной восстановился', + 'pending-activation': 'изменения ожидают запуска', + 'vpn-stopped': 'VPN выключен', + paused: 'автоматика на паузе', + disabled: 'резерв выключен', + 'switch-failed': 'не удалось переключить', + 'selector-unknown': 'текущий канал неизвестен', + 'reconcile-failed': 'мониторинг временно недоступен', + 'revalidation-required': 'условия проверяются заново', + 'manual-check': 'оба канала проверены', +}[reason || ''] || 'наблюдение'); + interface UiError { context?: string; profileId?: string; @@ -93,6 +124,7 @@ interface ComponentActions { setDevicePolicy: (id: string, mode: 'vpn' | 'direct', expectedRevision: number) => Promise; pingServers: (profileId: string, ids: string[]) => Promise; runConnectivityDiagnostics: (target?: unknown) => Promise; + loadActivityJournal: (cursor?: string | null) => Promise; } interface ClientViewState extends StateSnapshot { @@ -119,6 +151,10 @@ interface ClientOverviewPageProps { onSetGatewayAuto: (enabled: boolean) => Promise; onSaveRouteRules: (rules: RouteRule[], expectedRevision: number) => Promise; onUpdateDiagnosticsSettings: (settings: unknown) => Promise; + onSaveFailover: (policy: FailoverPolicy) => Promise; + onPauseFailover: (paused: boolean) => Promise; + onSwitchFailover: (role: 'primary' | 'reserve') => Promise; + onCheckFailover: () => Promise; onDismissError: () => void; } @@ -252,6 +288,7 @@ const operationProgress: Partial = { @@ -266,6 +303,10 @@ const canonicalOperationKeys: Record = { 'profile-delete': 'profileDelete', 'gateway-auto': 'gatewayAuto', 'route-rules': 'routeRules', + 'failover-save': 'failover', + 'failover-pause': 'failover', + 'failover-resume': 'failover', + 'failover-switch': 'failover', 'subscription-import': 'profileAdd', 'subscription-refresh': 'profileRefresh', 'subscription-forget': 'profileDelete', @@ -425,6 +466,10 @@ export function ClientOverviewPage({ onSetGatewayAuto, onSaveRouteRules, onUpdateDiagnosticsSettings, + onSaveFailover, + onPauseFailover, + onSwitchFailover, + onCheckFailover, onDismissError, }: ClientOverviewPageProps) { const isGateway = state?.mode === 'gateway'; @@ -533,6 +578,8 @@ export function ClientOverviewPage({ port: state?.clientRuntime?.proxyPort || (isGateway ? 8080 : 8082), controlHost, }); + const failoverFeature = useFailoverFeature(); + const activityJournalFeature = useActivityJournalFeature(); const diagnosticsAvailable = hasSubscription; const drawerControls = { subscription: { @@ -541,6 +588,12 @@ export function ClientOverviewPage({ show: subscriptionFeature.toggle, close: subscriptionFeature.close, }, + failover: { + isOpen: failoverFeature.isOpen, + panelRef: failoverFeature.panelRef, + show: failoverFeature.toggle, + close: failoverFeature.close, + }, instructions: { isOpen: instructionsFeature.isOpen, panelRef: instructionsFeature.panelRef, @@ -565,10 +618,16 @@ export function ClientOverviewPage({ show: routingFeature.open, close: routingFeature.forceClose, }, + journal: { + isOpen: activityJournalFeature.isOpen, + panelRef: activityJournalFeature.panelRef, + show: activityJournalFeature.toggle, + close: activityJournalFeature.close, + }, }; const drawerOrder = isGateway ? DRAWER_ORDER - : DRAWER_ORDER.filter((drawer) => drawer !== 'devices'); + : DRAWER_ORDER.filter((drawer) => !['devices', 'failover', 'journal'].includes(drawer)); const activeRailDrawer = drawerSwitchTarget && drawerControls[drawerSwitchTarget].isOpen ? drawerSwitchTarget : drawerOrder.find((drawer) => drawerControls[drawer].isOpen) || null; @@ -587,6 +646,8 @@ export function ClientOverviewPage({ instructionsFeature.close(); devicesFeature.close(); diagnosticsFeature.close(); + failoverFeature.close(); + activityJournalFeature.close(); } }, [hasSubscription, isGateway]); @@ -656,6 +717,7 @@ export function ClientOverviewPage({ routingFeature.requestClose(); return; } + if (current === 'failover' && !failoverFeature.beforeCloseRef.current()) return; if (!current) { drawerControls[target].show(); return; @@ -733,6 +795,13 @@ export function ClientOverviewPage({ : switchingServer && operationProfile && operationServer ? `Переключаем на ${subscriptionDomain(operationProfile.subscription.host)} · ${operationServer.label}` : ''; + const failoverIdentity = isGateway && state.failover.enabled + ? `${state.failover.currentRole === 'reserve' + ? 'Резервный канал' + : state.failover.currentRole === 'primary' + ? 'Основной канал' + : 'Текущий канал вне резервной пары'} · ${failoverReasonLabel(state.failover.reason)}` + : ''; return (
switchDrawer('subscription')} /> + {isGateway && switchDrawer('failover')} + />} switchDrawer('routing')} /> + {isGateway && switchDrawer('journal')} + />} }
} - serverSlot={} + serverSlot={} statusSlot={<> @@ -857,6 +936,22 @@ export function ClientOverviewPage({ } />} + {isGateway && hasSubscription && } + {isGateway && hasSubscription && }
diff --git a/src/web/features/activity-journal/ActivityJournalFeature.tsx b/src/web/features/activity-journal/ActivityJournalFeature.tsx new file mode 100644 index 0000000..fc6137e --- /dev/null +++ b/src/web/features/activity-journal/ActivityJournalFeature.tsx @@ -0,0 +1,167 @@ +import { useEffect, useRef, useState } from 'react'; +import { assertActivityJournalPage, type ActivityJournalEvent } from '../../../shared/activityJournal.js'; +import { Drawer } from '../../ui/Drawer.js'; +import { RailAction } from '../../ui/RailAction.js'; + +export function useActivityJournalFeature() { + const [isOpen, setIsOpen] = useState(false); + const panelRef = useRef(null); + const toggleRef = useRef(null); + const closeRef = useRef(null); + useEffect(() => { + if (!isOpen) return undefined; + const frame = requestAnimationFrame(() => closeRef.current?.focus()); + const close = (event: PointerEvent | KeyboardEvent) => { + if (event.type === 'keydown' && (event as KeyboardEvent).key !== 'Escape') return; + if (event.type !== 'keydown' && ( + panelRef.current?.contains(event.target as Node) || toggleRef.current?.contains(event.target as Node) + )) return; + setIsOpen(false); + }; + document.addEventListener('pointerdown', close); + document.addEventListener('keydown', close); + return () => { + cancelAnimationFrame(frame); + document.removeEventListener('pointerdown', close); + document.removeEventListener('keydown', close); + requestAnimationFrame(() => { + if (panelRef.current?.contains(document.activeElement)) toggleRef.current?.focus(); + }); + }; + }, [isOpen]); + return { isOpen, panelRef, toggleRef, closeRef, close: () => setIsOpen(false), toggle: () => setIsOpen((value) => !value) }; +} + +export type ActivityJournalFeature = ReturnType; + +export function ActivityJournalToggle({ feature, open, onToggle }: { + feature: ActivityJournalFeature; + open: boolean; + onToggle: () => void; +}) { + return + + ; +} + +function eventCopy(event: ActivityJournalEvent) { + const value = event.data; + const copies: Record = { + 'connection.started': ['VPN включён', [value.profileLabel, value.serverLabel].filter(Boolean).join(' · ')], + 'connection.stopped': ['VPN выключен', 'Остановлен пользователем'], + 'connection.failed': ['VPN не запущен', String(value.errorCode || '')], + 'subscription.added': ['Подписка добавлена', `${value.profileLabel || ''} · серверов: ${value.serverCount || 0}`], + 'subscription.refreshed': ['Подписка обновлена', `${value.profileLabel || ''} · серверов: ${value.serverCount || 0} · +${value.added || 0} / −${value.removed || 0}`], + 'subscription.refresh_failed': ['Подписка не обновлена', `${value.profileLabel || ''} · ${value.errorCode || ''}`], + 'subscription.deleted': ['Подписка удалена', String(value.profileLabel || '')], + 'failover.enabled': ['Резервный канал включён', 'Мониторинг начнётся после активации dual-config'], + 'failover.disabled': ['Резервный канал выключен', 'Автоматика полностью остановлена'], + 'failover.paused': ['Автопереключение на паузе', 'Проверки продолжаются'], + 'failover.resumed': ['Автопереключение возобновлено', ''], + 'failover.waiting_for_idle': ['Переключение отложено', 'Обнаружен активный трафик'], + 'failover.switched': ['Новые соединения переключены', `${value.fromRole || ''} → ${value.toRole || ''} · ${value.reason || ''}`], + 'failover.switch_failed': ['Переключение не выполнено', String(value.errorCode || '')], + 'failover.both_unhealthy': ['Оба канала недоступны', 'Текущий маршрут сохранён'], + 'failover.recovered': ['Основной канал восстановлен', String(value.reason || '')], + 'journal.recovered': ['Журнал восстановлен', 'Повреждённый файл сохранён отдельно'], + }; + return copies[event.type] || ['Системное событие', '']; +} + +function dayLabel(value: string) { + const date = new Date(value); + const today = new Date(); + const startDate = new Date(today.getFullYear(), today.getMonth(), today.getDate()); + const start = startDate.getTime(); + const yesterday = new Date(startDate); + yesterday.setDate(yesterday.getDate() - 1); + const day = new Date(date.getFullYear(), date.getMonth(), date.getDate()).getTime(); + if (day === start) return 'Сегодня'; + if (day === yesterday.getTime()) return 'Вчера'; + return new Intl.DateTimeFormat('ru-RU', { day: 'numeric', month: 'long' }).format(date); +} + +export function ActivityJournalPanel({ feature, loadPage }: { + feature: ActivityJournalFeature; + loadPage: (cursor?: string | null) => Promise; +}) { + const [events, setEvents] = useState([]); + const [nextCursor, setNextCursor] = useState(null); + const [status, setStatus] = useState<'idle' | 'loading' | 'ready' | 'refreshing' | 'older' | 'error'>('idle'); + const [announcement, setAnnouncement] = useState(''); + + async function load(cursor: string | null = null, refresh = false) { + setStatus(cursor ? 'older' : refresh ? 'refreshing' : 'loading'); + try { + const page = assertActivityJournalPage(await loadPage(cursor)); + if (page.storage.status === 'error') throw new Error('journal unavailable'); + if (refresh) { + const known = new Set(events.map(({ id }) => id)); + setAnnouncement(`Журнал обновлён, новых событий: ${page.events.filter(({ id }) => !known.has(id)).length}`); + } + setEvents((current) => cursor ? [...current, ...page.events] : page.events); + setNextCursor(page.nextCursor); + setStatus('ready'); + } catch { + setStatus('error'); + } + } + + useEffect(() => { + if (feature.isOpen) void load(null, status !== 'idle'); + }, [feature.isOpen]); + + const groups = events.reduce>((result, event) => { + const label = dayLabel(event.occurredAt); + const group = result.at(-1); + if (group?.label === label) group.events.push(event); + else result.push({ label, events: [event] }); + return result; + }, []); + + return +
+ Важные события хранятся 30 дней +

Журнал

+
+ {announcement} + {status === 'error' &&
Журнал временно недоступен
} + {status === 'loading' && !events.length ?
{[0, 1, 2, 3].map((value) => )}
+ : !events.length && status === 'ready' ?

За последние 30 дней важных событий пока нет

+ :
{groups.map((group) =>
+

{group.label}

+
    {group.events.map((event) => { + const [title, details] = eventCopy(event); + return
  1. +
    {event.source}
    +
    {title}{details && {details}}{event.severity !== 'info' && {event.severity === 'error' ? 'Ошибка' : 'Внимание'}}
    +
    2. ; + })}
+
)}
} +
+ {nextCursor ? : {events.length ? 'Это вся история за последние 30 дней' : 'Храним события 30 дней'}} +
+ ; +} diff --git a/src/web/features/activity-journal/index.ts b/src/web/features/activity-journal/index.ts new file mode 100644 index 0000000..58767d8 --- /dev/null +++ b/src/web/features/activity-journal/index.ts @@ -0,0 +1 @@ +export { ActivityJournalPanel, ActivityJournalToggle, useActivityJournalFeature } from './ActivityJournalFeature.js'; diff --git a/src/web/features/diagnostics/ConnectivityDiagnosticsPanel.tsx b/src/web/features/diagnostics/ConnectivityDiagnosticsPanel.tsx index 02b474c..bfcf31d 100644 --- a/src/web/features/diagnostics/ConnectivityDiagnosticsPanel.tsx +++ b/src/web/features/diagnostics/ConnectivityDiagnosticsPanel.tsx @@ -23,6 +23,7 @@ import { type DiagnosticSiteResult, } from './connectivityResult.js'; import type { DiagnosticsFeature } from './DiagnosticsFeature.js'; +import { saveCustomDiagnosticService } from './customServiceAction.js'; const CUSTOM_SERVICES_KEY = 'harbor-diagnostic-services'; const HIDDEN_SERVICES_KEY = 'harbor-hidden-diagnostic-services'; @@ -334,19 +335,15 @@ export function ConnectivityDiagnosticsPanel({ async function addService(event: FormEvent) { event.preventDefault(); try { - if (customServices.length >= MAX_CUSTOM_DIAGNOSTIC_SERVICES) return; - const parsed = new URL(serviceUrl.trim()); - if (parsed.protocol !== 'https:') throw new Error('Нужен публичный HTTPS-адрес.'); setSettingsSaving(true); - const saved = await updateSettings({ - customServices: [...customServices, { - id: `custom-${globalThis.crypto?.randomUUID?.() || Date.now()}`, - label: serviceName.trim() || parsed.hostname, - url: parsed.href, - }], + const saved = await saveCustomDiagnosticService({ + name: serviceName, + url: serviceUrl, + customServices, hiddenServiceIds, + updateSettings, }); - if (saved === false) throw new Error('Не удалось сохранить сервис.'); + if (!saved) return; setServiceName(''); setServiceUrl(''); setFormError(''); diff --git a/src/web/features/diagnostics/customServiceAction.ts b/src/web/features/diagnostics/customServiceAction.ts new file mode 100644 index 0000000..c8cd920 --- /dev/null +++ b/src/web/features/diagnostics/customServiceAction.ts @@ -0,0 +1,33 @@ +import { + MAX_CUSTOM_DIAGNOSTIC_SERVICES, + type DiagnosticService, + type DiagnosticSettings, +} from '../../../shared/connectivityDiagnostics.js'; + +export async function saveCustomDiagnosticService({ + name, + url, + customServices, + hiddenServiceIds, + updateSettings, +}: { + name: string; + url: string; + customServices: DiagnosticService[]; + hiddenServiceIds: string[]; + updateSettings: (settings: Pick) => Promise; +}) { + if (customServices.length >= MAX_CUSTOM_DIAGNOSTIC_SERVICES) return false; + const parsed = new URL(url.trim()); + if (parsed.protocol !== 'https:') throw new Error('Нужен публичный HTTPS-адрес.'); + const saved = await updateSettings({ + customServices: [...customServices, { + id: `custom-${globalThis.crypto?.randomUUID?.() || Date.now()}`, + label: name.trim() || parsed.hostname, + url: parsed.href, + }], + hiddenServiceIds, + }); + if (saved === false) throw new Error('Не удалось сохранить сервис.'); + return true; +} diff --git a/src/web/features/diagnostics/index.ts b/src/web/features/diagnostics/index.ts index c3bcce0..74a2489 100644 --- a/src/web/features/diagnostics/index.ts +++ b/src/web/features/diagnostics/index.ts @@ -4,3 +4,4 @@ export { useDiagnosticsFeature, type DiagnosticsFeature, } from './DiagnosticsFeature.js'; +export { saveCustomDiagnosticService } from './customServiceAction.js'; diff --git a/src/web/features/failover/FailoverFeature.tsx b/src/web/features/failover/FailoverFeature.tsx new file mode 100644 index 0000000..b1eb969 --- /dev/null +++ b/src/web/features/failover/FailoverFeature.tsx @@ -0,0 +1,331 @@ +import { useEffect, useMemo, useRef, useState } from 'react'; +import { CONNECTIVITY_SITES, MAX_CUSTOM_DIAGNOSTIC_SERVICES, type DiagnosticSettings } from '../../../shared/connectivityDiagnostics.js'; +import { normalizeFailoverPolicy, type FailoverPolicy, type FailoverSnapshot } from '../../../shared/failover.js'; +import type { ProfileSnapshot } from '../../../shared/contracts/state.js'; +import { Drawer } from '../../ui/Drawer.js'; +import { RailAction } from '../../ui/RailAction.js'; +import { saveCustomDiagnosticService } from '../diagnostics/index.js'; + +export function useFailoverFeature() { + const [isOpen, setIsOpen] = useState(false); + const panelRef = useRef(null); + const toggleRef = useRef(null); + const closeRef = useRef(null); + const beforeCloseRef = useRef<() => boolean>(() => true); + const close = () => { + if (beforeCloseRef.current()) setIsOpen(false); + }; + useEffect(() => { + if (!isOpen) return undefined; + const frame = requestAnimationFrame(() => closeRef.current?.focus()); + const handleClose = (event: PointerEvent | KeyboardEvent) => { + if (event.type === 'keydown' && (event as KeyboardEvent).key !== 'Escape') return; + if (event.type !== 'keydown' && ( + panelRef.current?.contains(event.target as Node) || toggleRef.current?.contains(event.target as Node) + )) return; + close(); + }; + document.addEventListener('pointerdown', handleClose); + document.addEventListener('keydown', handleClose); + return () => { + cancelAnimationFrame(frame); + document.removeEventListener('pointerdown', handleClose); + document.removeEventListener('keydown', handleClose); + requestAnimationFrame(() => { + if (panelRef.current?.contains(document.activeElement)) toggleRef.current?.focus(); + }); + }; + }, [isOpen]); + return { + isOpen, panelRef, toggleRef, closeRef, beforeCloseRef, close, + toggle: () => isOpen ? close() : setIsOpen(true), + }; +} + +export type FailoverFeature = ReturnType; + +export function FailoverToggle({ feature, open, onToggle }: { + feature: FailoverFeature; + open: boolean; + onToggle: () => void; +}) { + return + + ; +} + +const seconds = (milliseconds: number) => Math.round(milliseconds / 1000); +const milliseconds = (value: string, fallback: number) => { + const parsed = Number(value); + return Number.isFinite(parsed) ? Math.round(parsed * 1000) : fallback; +}; + +const reasonLabel = (reason: string | null) => ({ + 'primary-healthy': 'Основной канал работает', + 'health-unknown': 'Ожидаем результаты проверки', + 'failure-window': 'Подтверждаем сбой основного канала', + 'reserve-not-healthy': 'Резервный канал ещё не подтверждён', + 'both-unhealthy': 'Оба канала недоступны', + 'primary-not-recovered': 'Основной канал восстанавливается', + 'recovery-hold': 'Проверяем стабильность основного канала', + 'activity-unknown': 'Не удалось определить активность', + 'active-traffic': 'Ждём завершения активной работы', + 'quiet-window': 'Проверяем тишину перед переключением', + 'primary-failed': 'Основной канал недоступен', + 'primary-recovered': 'Основной канал восстановился', + 'pending-activation': 'Изменения ожидают следующего запуска VPN', + 'vpn-stopped': 'VPN выключен', + paused: 'Автоматика на паузе', + disabled: 'Резерв выключен', + 'switch-failed': 'Не удалось переключить канал', + 'selector-unknown': 'Не удалось подтвердить текущий канал', + 'reconcile-failed': 'Настройки сохранены, мониторинг временно недоступен', + 'revalidation-required': 'Условия переключения проверяются заново', + 'manual-check': 'Оба канала проверены', +}[reason || ''] || 'Наблюдаем за каналами'); + +function targetLabel(profiles: ProfileSnapshot[], target: { profileId: string; serverId: string }) { + const profile = profiles.find(({ id }) => id === target.profileId); + const server = profile?.servers.find(({ id }) => id === target.serverId); + return profile && server ? `${profile.label} · ${server.label}` : 'Не выбран'; +} + +export function FailoverPanel({ + feature, + snapshot, + profiles, + diagnostics, + blocked, + onSave, + onPause, + onSwitch, + onCheck, + onUpdateDiagnostics, +}: { + feature: FailoverFeature; + snapshot: FailoverSnapshot; + profiles: ProfileSnapshot[]; + diagnostics: DiagnosticSettings; + blocked: boolean; + onSave: (policy: FailoverPolicy) => Promise; + onPause: (paused: boolean) => Promise; + onSwitch: (role: 'primary' | 'reserve') => Promise; + onCheck: () => Promise; + onUpdateDiagnostics: (settings: unknown) => Promise; +}) { + const [draft, setDraft] = useState(() => snapshot.policy); + const [dirty, setDirty] = useState(false); + const [confirmDiscard, setConfirmDiscard] = useState(false); + const [addingService, setAddingService] = useState(false); + const [serviceName, setServiceName] = useState(''); + const [serviceUrl, setServiceUrl] = useState(''); + const [serviceError, setServiceError] = useState(''); + useEffect(() => { + if (!dirty) setDraft(snapshot.policy); + }, [snapshot.policy, dirty]); + useEffect(() => { + feature.beforeCloseRef.current = () => { + if (!dirty) return true; + setConfirmDiscard(true); + return false; + }; + const beforeUnload = (event: BeforeUnloadEvent) => { + if (!dirty) return; + event.preventDefault(); + }; + window.addEventListener('beforeunload', beforeUnload); + return () => { + feature.beforeCloseRef.current = () => true; + window.removeEventListener('beforeunload', beforeUnload); + }; + }, [dirty, feature.beforeCloseRef]); + const services = useMemo(() => [ + ...CONNECTIVITY_SITES, + ...diagnostics.customServices, + ], [diagnostics.customServices]); + const update = (value: Partial) => { + setDraft((current) => normalizeFailoverPolicy({ ...current, ...value })); + setDirty(true); + }; + const updateTarget = (role: 'primary' | 'reserve', patch: Partial) => { + const next = { ...draft[role], ...patch }; + const profile = profiles.find(({ id }) => id === next.profileId); + if (patch.profileId !== undefined) next.serverId = profile?.desiredServerId || profile?.servers[0]?.id || ''; + update({ [role]: next } as Partial); + }; + const role = snapshot.currentRole === 'primary' || snapshot.currentRole === 'reserve' + ? snapshot.currentRole + : null; + const switchRole = role === 'primary' ? 'reserve' : role === 'reserve' ? 'primary' : null; + const activity = snapshot.trafficActivity; + const targetExists = (target: FailoverPolicy['primary']) => profiles + .find(({ id }) => id === target.profileId)?.servers.some(({ id }) => id === target.serverId); + const targetsValid = targetExists(draft.primary) && targetExists(draft.reserve) + && (draft.primary.profileId !== draft.reserve.profileId || draft.primary.serverId !== draft.reserve.serverId); + const draftValid = Boolean(targetsValid && draft.checks.length); + const quietElapsed = activity?.quietSince + ? Math.max(0, Date.now() - Date.parse(activity.quietSince)) + : 0; + const addService = async () => { + try { + const saved = await saveCustomDiagnosticService({ + name: serviceName, + url: serviceUrl, + customServices: diagnostics.customServices, + hiddenServiceIds: diagnostics.hiddenServiceIds, + updateSettings: onUpdateDiagnostics, + }); + if (!saved) return; + setServiceName(''); + setServiceUrl(''); + setServiceError(''); + setAddingService(false); + } catch (error) { + setServiceError(error instanceof Error ? error.message : 'Проверьте адрес.'); + } + }; + + return +
+ Gateway +

Резервный канал

+

Переключает только новые соединения. Уже открытые соединения Harbor не закрывает.

+
+ +
{ + event.preventDefault(); + void onSave(draft).then((result) => { if (result !== false) setDirty(false); }); + }}> + + + {(['primary', 'reserve'] as const).map((channel) => { + const profile = profiles.find(({ id }) => id === draft[channel].profileId); + const health = snapshot[channel].health; + const missing = Boolean(snapshot[channel].target.profileId) && !targetExists(snapshot[channel].target); + const healthLabel = missing + ? 'Цель недоступна' + : channel === 'primary' && snapshot.reason === 'failure-window' + ? `Нестабилен · ${seconds(Math.max(0, Date.now() - Date.parse(snapshot.primary.stateSince || new Date().toISOString())))} из ${seconds(draft.failureWindowMs)} с` + : channel === 'primary' && snapshot.currentRole === 'reserve' && health === 'healthy' && snapshot.reason === 'recovery-hold' + ? 'Восстанавливается' + : health === 'healthy' ? 'Работает' : health === 'unhealthy' ? 'Недоступен' : health === 'not-monitoring' ? 'Не проверяется' : 'Нет данных'; + return
+
+ {channel === 'primary' ? 'Основной' : 'Резервный'} + {healthLabel} +
+ + +
; + })} + +
+ Что проверять + {services.map((service) => { + const check = draft.checks.find(({ serviceId }) => serviceId === service.id); + return
+ + {check && } +
; + })} + {!addingService && } + {addingService &&
+ setServiceName(event.target.value)} /> + setServiceUrl(event.target.value)} /> + {serviceError && {serviceError}} + + +
} +
+ +
+ + + + + + +
+ +
Защита от повторных сбоев + + + + +
+ +
+ {blocked ? 'Harbor выполняет действие…' : snapshot.activation === 'pending' ? 'Включится при следующем запуске VPN' : reasonLabel(snapshot.reason)} + Новые соединения: {role ? targetLabel(profiles, snapshot[role].target) : 'текущий канал вне настроенной пары'} + {snapshot.nextDecisionAt && Следующее решение не раньше чем через {seconds(Math.max(0, Date.parse(snapshot.nextDecisionAt) - Date.now()))} с} + {snapshot.currentRole === 'other' && Чтобы запустить автоматику, выключите VPN и включите снова на основном канале.} + {activity && {activity.state === 'active' ? `Активный трафик · ${Math.round(activity.totalBytesPerSecond / 1024)} КБ/с · соединений: ${activity.transmittingConnections}` : activity.state === 'quiet' ? `Проверяем тишину · ${Math.min(seconds(quietElapsed), seconds(draft.trafficGuard.quietWindowMs))} из ${seconds(draft.trafficGuard.quietWindowMs)} с` : 'Не удалось определить активность · автоматическое переключение остановлено'}} + {activity?.blockers.slice(0, 2).map((blocker) => {blocker.device} · {blocker.service} · {Math.round((blocker.uploadBytesPerSecond + blocker.downloadBytesPerSecond) / 1024)} КБ/с)} + {activity && activity.blockers.length > 2 && Ещё {activity.blockers.length - 2}} +
+ + {!draftValid &&

Выберите два разных сервера. Они могут быть из одной подписки.

} + {confirmDiscard &&
+ Отменить несохранённые изменения? + + +
} + +
+ + + {snapshot.enabled && } + {switchRole && snapshot.enabled && snapshot.activation === 'active' && } + {(snapshot.currentRole === 'primary' || snapshot.currentRole === 'reserve') && Текущие соединения Harbor не закроет} +
+
+ ; +} diff --git a/src/web/features/failover/index.ts b/src/web/features/failover/index.ts new file mode 100644 index 0000000..ae1b506 --- /dev/null +++ b/src/web/features/failover/index.ts @@ -0,0 +1 @@ +export { FailoverPanel, FailoverToggle, useFailoverFeature } from './FailoverFeature.js'; diff --git a/src/web/state/harborReducer.ts b/src/web/state/harborReducer.ts index 46b520d..0531350 100644 --- a/src/web/state/harborReducer.ts +++ b/src/web/state/harborReducer.ts @@ -4,6 +4,10 @@ export type SyncErrorKind = 'incompatible-api' | 'control-unreachable' | 'fatal' export interface HarborReducerState { snapshot: HarborClientState | null; + failoverTransport: { + activeEpoch: string; + retiredEpochs: string[]; + }; transport: { bootStatus: 'loading' | 'ready' | SyncErrorKind; lastSuccessfulSyncAt: string | null; @@ -20,6 +24,7 @@ export type HarborAction = export const initialHarborState: HarborReducerState = { snapshot: null, + failoverTransport: { activeEpoch: '', retiredEpochs: [] }, transport: { bootStatus: 'loading', lastSuccessfulSyncAt: null, @@ -31,6 +36,14 @@ export const initialHarborState: HarborReducerState = { export const STALE_FAILURE_THRESHOLD = 3; +function failoverEpoch(snapshot: HarborClientState | null) { + return snapshot?.failover?.observationEpoch || ''; +} + +function failoverSequence(snapshot: HarborClientState | null) { + return snapshot?.failover?.observationSequence || 0; +} + export function classifySyncError(error: unknown): SyncErrorKind { const candidate = error && typeof error === 'object' ? error as Record : {}; const status = Number(candidate.status) || 0; @@ -71,9 +84,29 @@ export function harborReducer(current: HarborReducerState, action: HarborAction) const snapshot = action.snapshot; const newer = !current.snapshot || snapshot.revision > current.snapshot.revision; + const equal = Boolean(current.snapshot) && snapshot.revision === current.snapshot?.revision; + const incomingEpoch = failoverEpoch(snapshot); + const activeEpoch = current.failoverTransport.activeEpoch || failoverEpoch(current.snapshot); + const unseenEpoch = Boolean(incomingEpoch) + && incomingEpoch !== activeEpoch + && !current.failoverTransport.retiredEpochs.includes(incomingEpoch); + const newerObservation = equal && Boolean(snapshot.failover) && ( + unseenEpoch + || (incomingEpoch === activeEpoch && failoverSequence(snapshot) > failoverSequence(current.snapshot)) + ); + const nextSnapshot = newer + ? snapshot + : newerObservation && current.snapshot + ? { ...current.snapshot, failover: snapshot.failover } + : current.snapshot; + const nextActiveEpoch = newer || unseenEpoch ? incomingEpoch : activeEpoch; + const retiredEpochs = unseenEpoch && activeEpoch + ? [...current.failoverTransport.retiredEpochs, activeEpoch] + : current.failoverTransport.retiredEpochs; return { - snapshot: newer ? snapshot : current.snapshot, + snapshot: nextSnapshot, + failoverTransport: { activeEpoch: nextActiveEpoch, retiredEpochs }, transport: { bootStatus: 'ready', lastSuccessfulSyncAt: action.receivedAt, diff --git a/src/web/state/operations.ts b/src/web/state/operations.ts index 4ad0d57..17ace3b 100644 --- a/src/web/state/operations.ts +++ b/src/web/state/operations.ts @@ -1,6 +1,6 @@ export type OperationKey = 'connection' | 'serverApply' | 'profileAdd' | 'profileRename' | 'profileSelect' | 'profileActivate' | 'profileRefresh' | 'profileDelete' - | 'gatewayAuto' | 'routeRules' | 'diagnosticsSettings'; + | 'gatewayAuto' | 'routeRules' | 'diagnosticsSettings' | 'failover'; export interface OperationState { status: 'running'; @@ -22,6 +22,7 @@ const OPERATION_KEYS: readonly OperationKey[] = [ 'gatewayAuto', 'routeRules', 'diagnosticsSettings', + 'failover', ]; // The backend has one canonical revision and one mutation queue, so the UI mirrors that lock. diff --git a/src/web/styles/features/activity-journal.css b/src/web/styles/features/activity-journal.css new file mode 100644 index 0000000..e24a8fd --- /dev/null +++ b/src/web/styles/features/activity-journal.css @@ -0,0 +1,27 @@ +.client-journal-header { display: grid; gap: 7px; margin: 0 8px 30px; } +.client-journal-header > span, .client-journal-groups h3 { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); } +.client-journal-header > div { display: flex; align-items: center; gap: 12px; } +.client-journal-header h2 { margin: 0; font: var(--type-drawer-title); letter-spacing: var(--type-drawer-title-tracking); text-transform: var(--type-drawer-title-transform); } +.client-journal-header button { width: 34px; height: 34px; display: grid; place-items: center; border: 0; background: transparent; color: var(--client-muted); cursor: pointer; } +.client-journal-header svg { width: 17px; fill: none; stroke: currentColor; stroke-width: 1.7; } +.client-journal-header button:hover, .client-journal-header button:focus-visible { color: var(--client-accent); outline: 0; filter: drop-shadow(0 0 7px var(--client-accent)); } +.client-journal-header button:disabled svg { animation: client-spin 900ms linear infinite; } +.client-journal-groups { display: grid; gap: 26px; margin: 0 8px; } +.client-journal-groups section { display: grid; gap: 7px; } +.client-journal-groups h3 { margin: 0; } +.client-journal-groups ol { display: grid; margin: 0; padding: 0; } +.client-journal-groups li { min-height: 58px; display: grid; grid-template-columns: 52px minmax(0, 1fr); gap: 14px; align-items: start; padding: 10px 0; border-top: 1px solid color-mix(in oklch, var(--client-border) 48%, transparent); } +.client-journal-time, .client-journal-groups li > div:last-child { min-width: 0; display: grid; gap: 3px; } +.client-journal-time time { color: var(--client-text); font: var(--type-data); font-variant-numeric: var(--numeric-tabular); letter-spacing: var(--type-data-tracking); text-transform: var(--type-data-transform); } +.client-journal-time span, .client-journal-groups li span { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); overflow-wrap: anywhere; } +.client-journal-time span { text-transform: var(--type-label-transform); } +.client-journal-groups li strong { font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); } +.client-journal-groups li em { width: fit-content; color: oklch(0.68 0.15 28); font: var(--type-label); font-style: normal; letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); } +.client-journal-error, .client-journal-empty { min-height: 80px; margin: 0 8px; color: var(--client-muted); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); } +.client-journal-error button, .client-journal-footer button { border: 0; background: transparent; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; } +.client-journal-skeleton { display: grid; gap: 14px; margin: 0 8px; } +.client-journal-skeleton span { height: 48px; background: color-mix(in oklch, var(--client-border) 24%, transparent); opacity: .55; } +.client-journal-footer { min-height: 72px; display: grid; place-items: center; margin: 12px 8px 0; color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); text-align: center; } +.client-journal-footer button:focus-visible, .client-journal-error button:focus-visible { outline: 2px solid var(--client-accent); outline-offset: 2px; } +@media (max-width: 560px) { .client-journal-groups li { grid-template-columns: 1fr; gap: 5px; } .client-journal-time { display: flex; gap: 8px; } } +@media (prefers-reduced-motion: reduce) { .client-journal-header button:disabled svg { animation: none; } } diff --git a/src/web/styles/features/failover.css b/src/web/styles/features/failover.css new file mode 100644 index 0000000..13c909d --- /dev/null +++ b/src/web/styles/features/failover.css @@ -0,0 +1,45 @@ +.client-failover-header { display: grid; gap: 8px; margin: 0 8px 30px; } +.client-failover-header > span, .client-failover-channel-title > span, .client-failover-services legend { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); } +.client-failover-header h2 { margin: 0; font: var(--type-drawer-title); letter-spacing: var(--type-drawer-title-tracking); text-transform: var(--type-drawer-title-transform); } +.client-failover-header p { margin: 0; color: var(--client-muted); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); } +.client-failover-form { display: grid; gap: 24px; margin: 0 8px; } +.client-failover-master { min-height: 54px; display: flex; align-items: center; justify-content: space-between; gap: 20px; border-bottom: 1px solid color-mix(in oklch, var(--client-border) 48%, transparent); } +.client-failover-master > span { display: grid; gap: 3px; } +.client-failover-master strong { font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); } +.client-failover-master small { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); } +.client-failover-master input { width: 42px; height: 22px; } +.client-failover-channel { display: grid; gap: 10px; } +.client-failover-channel-title { min-height: 28px; display: flex; align-items: center; justify-content: space-between; gap: 12px; } +.client-failover-channel-title strong { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); transition: color 600ms ease, filter 600ms ease; } +.client-failover-channel-title strong.is-healthy { color: var(--client-accent); } +.client-failover-channel-title strong.is-unhealthy { color: oklch(0.68 0.15 28); } +.client-failover-channel label, .client-failover-timing label, .client-failover-advanced label { display: grid; grid-template-columns: minmax(0, 1fr) minmax(150px, 48%); align-items: center; gap: 16px; min-height: 38px; color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); } +.client-failover-channel select, .client-failover-timing input, .client-failover-advanced input { min-width: 0; min-height: 34px; border: 0; border-bottom: 1px solid var(--client-border); background: transparent; color: var(--client-text); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); } +.client-failover-services { display: grid; gap: 9px; margin: 0; padding: 0; border: 0; } +.client-failover-services legend { margin-bottom: 10px; } +.client-failover-service { display: grid; grid-template-columns: minmax(0, 1fr) auto; align-items: center; gap: 12px; min-height: 34px; } +.client-failover-service > label { display: flex; align-items: center; gap: 10px; font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); } +.client-failover-service > .client-failover-service-timeout { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); } +.client-failover-service-timeout input { width: 52px; min-height: 30px; border: 0; border-bottom: 1px solid var(--client-border); background: transparent; color: var(--client-text); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); } +.client-failover-add-service { justify-self: start; padding: 7px 0; border: 0; background: transparent; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; } +.client-failover-add-service:disabled { color: var(--client-muted); cursor: default; opacity: .45; } +.client-failover-service-editor { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1.6fr); gap: 8px 12px; padding: 8px 0; } +.client-failover-service-editor input { min-width: 0; min-height: 34px; border: 0; border-bottom: 1px solid var(--client-border); background: transparent; color: var(--client-text); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); } +.client-failover-service-editor span { grid-column: 1 / -1; color: oklch(0.68 0.15 28); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); } +.client-failover-service-editor button { justify-self: start; padding: 4px 0; border: 0; background: transparent; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; } +.client-failover-timing { display: grid; gap: 5px; } +.client-failover-advanced summary { min-height: 38px; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; } +.client-failover-runtime { min-height: 132px; display: grid; align-content: center; gap: 5px; padding: 14px 0; border-top: 1px solid color-mix(in oklch, var(--client-border) 48%, transparent); border-bottom: 1px solid color-mix(in oklch, var(--client-border) 48%, transparent); font-variant-numeric: var(--numeric-tabular); } +.client-failover-runtime strong { font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); } +.client-failover-runtime span, .client-failover-runtime small { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); } +.client-failover-validation { margin: -12px 0 0; color: oklch(0.68 0.15 28); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); } +.client-failover-discard { min-height: 74px; display: flex; flex-wrap: wrap; align-items: center; gap: 8px 16px; padding: 10px 0; border-top: 1px solid var(--client-border); border-bottom: 1px solid var(--client-border); } +.client-failover-discard span { flex-basis: 100%; color: var(--client-text); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); } +.client-failover-discard button { padding: 4px 0; border: 0; background: transparent; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; } +.client-failover-actions { min-height: 42px; display: flex; flex-wrap: wrap; gap: 18px; } +.client-failover-actions button { padding: 7px 0; border: 0; background: transparent; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; } +.client-failover-actions small { flex-basis: 100%; margin-top: -14px; color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); } +.client-failover-actions button:disabled { color: var(--client-muted); cursor: default; opacity: .45; } +.client-failover-actions button:focus-visible, .client-failover-form input:focus-visible, .client-failover-form select:focus-visible, .client-failover-advanced summary:focus-visible, .client-failover-add-service:focus-visible, .client-failover-service-editor button:focus-visible, .client-failover-discard button:focus-visible { outline: 2px solid var(--client-accent); outline-offset: 2px; } +@media (max-width: 560px) { .client-failover-channel label, .client-failover-timing label, .client-failover-advanced label { grid-template-columns: 1fr; gap: 4px; } .client-failover-service, .client-failover-service-editor { grid-template-columns: 1fr; gap: 2px; } } +@media (prefers-reduced-motion: reduce) { .client-failover-channel-title strong { transition: none; } } diff --git a/src/web/styles/index.css b/src/web/styles/index.css index 0b10ac4..5da1609 100644 --- a/src/web/styles/index.css +++ b/src/web/styles/index.css @@ -8,5 +8,7 @@ @import './features/servers.css'; @import './primitives.css'; @import './features/diagnostics.css'; +@import './features/failover.css'; +@import './features/activity-journal.css'; @import './layout.css'; @import './themes.css'; diff --git a/test/server/activity-journal-route.test.js b/test/server/activity-journal-route.test.js new file mode 100644 index 0000000..5d30b59 --- /dev/null +++ b/test/server/activity-journal-route.test.js @@ -0,0 +1,25 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; + +import { createActivityJournalRoute } from '../../dist/server/http/routes/activityJournalRoute.js'; + +test('journal route is read-only and forwards bounded cursor pagination', async () => { + const calls = []; + const route = createActivityJournalRoute({ + journal: { page: (...args) => { + calls.push(args); + return { events: [], nextCursor: null, retentionDays: 30, generatedAt: 'now', storage: { status: 'ready', errorCode: null } }; + } }, + }); + let status; + let payload; + const response = { + writeHead: (value) => { status = value; }, + end: (value) => { payload = JSON.parse(value); }, + }; + assert.equal(await route.handle({ method: 'GET', url: '/api/activity-journal?limit=25&cursor=evt' }, response), true); + assert.equal(status, 200); + assert.equal(payload.retentionDays, 30); + assert.deepEqual(calls, [[25, 'evt']]); + assert.equal(await route.handle({ method: 'POST', url: '/api/activity-journal' }, response), false); +}); diff --git a/test/server/activity-journal-service.test.js b/test/server/activity-journal-service.test.js new file mode 100644 index 0000000..5bc696e --- /dev/null +++ b/test/server/activity-journal-service.test.js @@ -0,0 +1,145 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; + +import { createActivityJournalService } from '../../dist/server/services/activityJournalService.js'; +import { assertActivityJournalPage } from '../../dist/shared/activityJournal.js'; + +function fixture(t) { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'harbor-journal-')); + t.after(() => fs.rmSync(directory, { recursive: true, force: true })); + return path.join(directory, 'activity-journal.json'); +} + +const event = (dedupeKey, profileLabel = 'Home') => ({ + type: 'subscription.refreshed', + severity: 'info', + source: 'subscription', + dedupeKey: `subscription.refreshed:${dedupeKey}`, + data: { profileId: 'profile-1', profileLabel, host: 'provider.example', serverCount: 12, added: 2, removed: 1 }, +}); + +test('journal appends typed events, deduplicates and keeps stable newest-first cursors', (t) => { + let clock = new Date('2026-08-19T10:00:00.000Z'); + const filePath = fixture(t); + const service = createActivityJournalService({ filePath, now: () => clock }); + service.append(event('refresh:1', 'One')); + clock = new Date('2026-08-19T10:01:00.000Z'); + service.append(event('refresh:2', 'Two')); + service.append(event('refresh:2', 'Duplicate')); + const first = service.page(1); + assert.equal(first.events[0].data.profileLabel, 'Two'); + assert.ok(first.nextCursor); + + clock = new Date('2026-08-19T10:02:00.000Z'); + service.append(event('refresh:3', 'Three')); + const older = service.page(10, first.nextCursor); + assert.deepEqual(older.events.map(({ data }) => data.profileLabel), ['One']); + assert.equal(first.events[0].dedupeKey, null); + const inode = fs.statSync(filePath).ino; + assert.equal(service.page(10).events.length, 3); + assert.equal(fs.statSync(filePath).ino, inode); + assert.deepEqual(service.page(10, 'expired-cursor').events, []); +}); + +test('journal prunes events older than 30 days and rejects unsafe payloads', (t) => { + let clock = new Date('2026-07-01T00:00:00.000Z'); + const filePath = fixture(t); + const service = createActivityJournalService({ filePath, now: () => clock }); + service.append(event('old')); + clock = new Date('2026-08-19T00:00:00.000Z'); + assert.deepEqual(service.page().events, []); + assert.deepEqual(JSON.parse(fs.readFileSync(filePath, 'utf8')).events, []); + service.append(event('new')); + assert.throws(() => service.append({ ...event('unsafe'), data: { rawUrl: 'https://secret' } }), /Unsafe/); + service.append({ ...event('ip'), data: { ...event('ip').data, host: '192.168.1.1' } }); + service.append(event('credential-label', 'https://user:pass@example.test/private?token=secret')); + service.append(event('path-label', '192.168.1.7/private')); + service.append(event('ipv6-label', '2001:db8::1')); + const page = service.page(); + const labels = page.events.slice(0, 3).map(({ data }) => data.profileLabel); + assert.deepEqual(labels, ['Подписка', 'Подписка', 'Подписка']); + assert.equal(page.events.find(({ data }) => data.host === 'Провайдер')?.data.host, 'Провайдер'); + for (const dedupeKey of [ + 'subscription.refreshed:192.168.1.1', + 'subscription.refreshed:2001:db8::1', + 'subscription.refreshed:user:pass', + ]) service.append({ ...event('safe'), dedupeKey }); + service.append({ ...event('safe'), dedupeKey: 'subscription.refreshed:user:pass' }); + const persisted = fs.readFileSync(filePath, 'utf8'); + assert.doesNotMatch(persisted, /user:pass|192\.168\.1\.1|192\.168\.1\.7|2001:db8|token=secret/); + assert.match(persisted, /subscription\.refreshed:sha256:[a-f0-9]{64}/); + assert.throws(() => service.append({ ...event('safe'), dedupeKey: 'https://user:pass@example.test/private?token=x' }), /dedupe/i); + assert.throws(() => service.append({ ...event('safe'), dedupeKey: 'subscription.refreshed:private/path' }), /dedupe/i); +}); + +test('journal persists the 10,000 event cap when opening an oversized store', (t) => { + const filePath = fixture(t); + const occurredAt = '2026-08-19T00:00:00.000Z'; + const events = Array.from({ length: 10_001 }, (_, index) => ({ + id: `00000000-0000-4000-8000-${String(index).padStart(12, '0')}`, + occurredAt, + ...event(`event:${index}`), + })); + fs.writeFileSync(filePath, JSON.stringify({ schemaVersion: 1, events })); + const service = createActivityJournalService({ + filePath, + now: () => new Date('2026-08-19T01:00:00.000Z'), + }); + assert.equal(JSON.parse(fs.readFileSync(filePath, 'utf8')).events.length, 10_000); + assert.equal(service.page(1).events.length, 1); +}); + +test('corrupt journal is isolated and recovery becomes a safe event', (t) => { + const filePath = fixture(t); + fs.writeFileSync(filePath, '{broken'); + const service = createActivityJournalService({ + filePath, + now: () => new Date('2026-08-19T12:00:00.000Z'), + }); + const page = service.page(); + assert.equal(page.storage.status, 'ready'); + assert.equal(page.events[0].type, 'journal.recovered'); + assert.ok(fs.readdirSync(path.dirname(filePath)).some((name) => name.includes('.corrupt-'))); +}); + +test('journal exposes a latched write failure until a later append succeeds', (t) => { + const filePath = fixture(t); + const directory = path.dirname(filePath); + const service = createActivityJournalService({ filePath }); + service.append(event('before-error')); + fs.chmodSync(directory, 0o555); + try { + assert.throws(() => service.append(event('lost'))); + const failed = service.page(); + assert.equal(failed.storage.status, 'error'); + assert.equal(failed.storage.errorCode, 'JOURNAL_UNAVAILABLE'); + assert.equal(failed.events.length, 1); + assert.equal(failed.events[0].dedupeKey, null); + } finally { + fs.chmodSync(directory, 0o755); + } + service.append(event('recovered')); + assert.equal(service.page().storage.status, 'ready'); +}); + +test('journal page parser rejects malformed wire data and strips unknown event fields', (t) => { + const service = createActivityJournalService({ filePath: fixture(t) }); + service.append(event('wire')); + const page = service.page(); + const parsed = assertActivityJournalPage({ + ...page, + events: page.events.map((item) => ({ ...item, ignored: 'value' })), + }); + assert.equal(Object.hasOwn(parsed.events[0], 'ignored'), false); + assert.throws(() => assertActivityJournalPage({ ...page, retentionDays: 31 }), TypeError); + assert.throws(() => assertActivityJournalPage({ ...page, events: [{ broken: true }] }), TypeError); + const future = assertActivityJournalPage({ + ...page, + events: [{ ...page.events[0], type: 'future.safe_event', data: { raw: 'not exposed' } }], + }); + assert.equal(future.events[0].type, 'unknown'); + assert.deepEqual(future.events[0].data, {}); +}); diff --git a/test/server/connection-service.test.js b/test/server/connection-service.test.js index ecabbd3..3efb372 100644 --- a/test/server/connection-service.test.js +++ b/test/server/connection-service.test.js @@ -52,6 +52,7 @@ function createHarness(overrides = {}) { let stateUpdates = 0; let startFailure = overrides.failStart || null; const events = []; + const journalEvents = []; const serialize = (operation) => { const run = async () => { @@ -119,13 +120,16 @@ function createHarness(overrides = {}) { restartCommand: () => captureRuntimeCommand(restart, { preMutationErrorCodes: ['CONFIG_INVALID'] }), }, route: { isGatewayDirect: () => overrides.gatewayDirect === true }, + failover: overrides.failover, serialize, now: () => new Date('2026-08-08T12:00:00.000Z'), + onEvent: (event) => journalEvents.push(event), }); return { service, events, + journalEvents, serialize, snapshot: () => structuredClone({ state, config, running, peak }), }; @@ -167,6 +171,8 @@ test('failed candidate config or runtime restores the old profile, config and ru const before = harness.snapshot(); await assert.rejects(harness.service.apply('work', 'b')); assert.deepEqual(domain(harness.snapshot()), domain(before)); + assert.equal(harness.journalEvents.at(-1).type, 'connection.failed'); + assert.match(harness.journalEvents.at(-1).data.errorCode, /^[A-Z0-9_]+$/); } }); @@ -256,6 +262,193 @@ test('running restart preserves a pending desired profile while restoring the ap assert.equal(harness.snapshot().state.appliedProfileId, 'personal'); }); +test('running dual restart rebuilds only the applied failover pair', async () => { + const state = initialState(); + state.failoverPolicy = { enabled: true }; + state.appliedFailoverPolicy = { + primary: { profileId: 'personal', serverId: 'a' }, + reserve: { profileId: 'work', serverId: 'b' }, + }; + const sources = []; + const prepared = []; + const harness = createHarness({ + state, + failover: { + build: (_state, source) => { + sources.push(source); + return { + config: { source }, + applied: state.appliedFailoverPolicy, + primaryProfile: state.profiles[0], + primaryServer: serverA, + }; + }, + prepareActivation: async (role) => { prepared.push(role); harness.events.push(`selector.${role}`); }, + restoreAppliedActivation: async () => {}, + reconcile: async () => {}, + }, + }); + await harness.service.restart(); + assert.deepEqual(sources, ['applied']); + assert.deepEqual(prepared, ['primary']); + assert.deepEqual(harness.events.slice(0, 4), ['config.write', 'runtime.restart', 'selector.primary', 'state.update']); + assert.match(harness.snapshot().config, /"source":"applied"/); +}); + +test('running single-channel keeps server edits pending after failover is enabled', async () => { + const state = initialState(); + state.failoverPolicy = { enabled: true }; + state.appliedFailoverPolicy = null; + const harness = createHarness({ state }); + await harness.service.apply('work', 'b'); + assert.equal(harness.snapshot().state.desiredProfileId, 'work'); + assert.equal(harness.snapshot().state.appliedProfileId, 'personal'); + assert.equal(harness.events.includes('config.write'), false); + assert.equal(harness.events.includes('runtime.start'), false); +}); + +test('running dual restart preserves reserve and commits only after selector read-back', async () => { + const state = initialState(); + state.failoverPolicy = { enabled: true }; + state.appliedFailoverPolicy = { + primary: { profileId: 'personal', serverId: 'a' }, + reserve: { profileId: 'work', serverId: 'b' }, + }; + state.appliedProfileId = 'work'; + state.appliedServerId = 'b'; + state.appliedServerSnapshot = serverB; + const harness = createHarness({ + state, + failover: { + build: () => ({ + config: { dual: true }, + applied: state.appliedFailoverPolicy, + primaryProfile: state.profiles[0], + primaryServer: serverA, + }), + prepareActivation: async (role) => { harness.events.push(`selector.${role}`); }, + restoreAppliedActivation: async () => { harness.events.push('selector.primary'); }, + reconcile: async () => {}, + }, + }); + await harness.service.restart(); + assert.equal(harness.snapshot().state.appliedProfileId, 'work'); + assert.equal(harness.snapshot().state.appliedServerId, 'b'); + assert.deepEqual(harness.events.slice(0, 4), ['config.write', 'runtime.restart', 'selector.reserve', 'state.update']); +}); + +test('selector activation failure rolls runtime and config back before applied truth changes', async () => { + const state = initialState(); + state.failoverPolicy = { enabled: true }; + state.appliedFailoverPolicy = { + primary: { profileId: 'personal', serverId: 'a' }, + reserve: { profileId: 'work', serverId: 'b' }, + }; + const harness = createHarness({ + state, + failover: { + build: () => ({ + config: { dual: true }, + applied: state.appliedFailoverPolicy, + primaryProfile: state.profiles[0], + primaryServer: serverA, + }), + prepareActivation: async (role) => { + harness.events.push(`selector.${role}`); + if (harness.events.filter((event) => event.startsWith('selector.')).length === 1) { + throw new Error('selector failed'); + } + }, + restoreAppliedActivation: async () => { harness.events.push('selector.primary'); }, + reconcile: async () => {}, + }, + }); + const before = harness.snapshot(); + await assert.rejects(harness.service.restart(), /selector failed/); + assert.deepEqual(domain(harness.snapshot()), domain(before)); + assert.equal(harness.events.includes('state.update'), false); + assert.deepEqual(harness.events.filter((event) => event.startsWith('selector.')), ['selector.primary', 'selector.primary']); +}); + +test('state commit rollback restores the previously selected reserve role', async () => { + const state = initialState(); + state.failoverPolicy = { enabled: true }; + state.appliedFailoverPolicy = { + primary: { profileId: 'personal', serverId: 'a' }, + reserve: { profileId: 'work', serverId: 'b' }, + }; + state.appliedProfileId = 'work'; + state.appliedServerId = 'b'; + state.appliedServerSnapshot = serverB; + const harness = createHarness({ + state, + failStateAt: 1, + failover: { + build: () => ({ + config: { dual: true }, + applied: state.appliedFailoverPolicy, + primaryProfile: state.profiles[0], + primaryServer: serverA, + }), + prepareActivation: async (role) => { harness.events.push(`selector.${role}`); }, + restoreAppliedActivation: async () => { harness.events.push('selector.reserve'); }, + reconcile: async () => {}, + }, + }); + const before = harness.snapshot(); + await assert.rejects(harness.service.restart(), /state failed/); + assert.deepEqual(domain(harness.snapshot()), domain(before)); + assert.deepEqual(harness.events.filter((event) => event.startsWith('selector.')), ['selector.reserve', 'selector.reserve']); +}); + +test('failed stop restores the selected reserve role before restoring state', async () => { + const state = initialState(); + state.failoverPolicy = { enabled: true }; + state.appliedFailoverPolicy = { + primary: { profileId: 'personal', serverId: 'a' }, + reserve: { profileId: 'work', serverId: 'b' }, + }; + state.appliedProfileId = 'work'; + state.appliedServerId = 'b'; + state.appliedServerSnapshot = serverB; + const harness = createHarness({ + state, + failStateAt: 1, + failover: { + restoreAppliedActivation: async () => { harness.events.push('selector.reserve'); }, + reconcile: async () => {}, + }, + }); + const before = harness.snapshot(); + await assert.rejects(harness.service.stop(), /state failed/); + assert.deepEqual(domain(harness.snapshot()), domain(before)); + assert.deepEqual(harness.events.slice(0, 4), ['runtime.stop', 'state.update', 'runtime.start', 'selector.reserve']); +}); + +test('disabled passive dual rollback restores the selected reserve role', async () => { + const state = initialState(); + state.failoverPolicy = { enabled: false }; + state.appliedFailoverPolicy = { + primary: { profileId: 'personal', serverId: 'a' }, + reserve: { profileId: 'work', serverId: 'b' }, + }; + state.appliedProfileId = 'work'; + state.appliedServerId = 'b'; + state.appliedServerSnapshot = serverB; + const harness = createHarness({ + state, + failStateAt: 1, + failover: { + restoreAppliedActivation: async () => { harness.events.push('selector.reserve'); }, + reconcile: async () => {}, + }, + }); + const before = harness.snapshot(); + await assert.rejects(harness.service.restart(), /state failed/); + assert.deepEqual(domain(harness.snapshot()), domain(before)); + assert.equal(harness.events.includes('selector.reserve'), true); +}); + test('restart fails closed when runtime status is unknown', async () => { const state = initialState(); state.desiredProfileId = 'work'; diff --git a/test/server/connectivity-diagnostics.test.js b/test/server/connectivity-diagnostics.test.js index dfb7471..aa0cc0d 100644 --- a/test/server/connectivity-diagnostics.test.js +++ b/test/server/connectivity-diagnostics.test.js @@ -506,3 +506,18 @@ test('a targeted custom row validates and samples only that service', async () = assert.deepEqual(calls, ['https://second.example/', 'https://second.example/', 'https://second.example/']); assert.equal(result.direct.sites[0].latencyMs, 120); }); + +test('failover VPN checks apply their per-service timeout to curl', async () => { + const calls = []; + await createConnectivityDiagnosticsService({ + proxyPort: 18080, + execute: async (args) => { calls.push(args); return response(); }, + }).runVpn({ target: 'site:youtube', timeoutMs: 9_000 }); + + assert.equal(calls.length, 3); + for (const args of calls) { + assert.equal(args[args.indexOf('--max-time') + 1], '9'); + assert.equal(args[args.indexOf('--connect-timeout') + 1], '3'); + assert.ok(args.includes('http://127.0.0.1:18080')); + } +}); diff --git a/test/server/dataplane-client.test.js b/test/server/dataplane-client.test.js index 6c9c5b3..d65bd52 100644 --- a/test/server/dataplane-client.test.js +++ b/test/server/dataplane-client.test.js @@ -32,6 +32,12 @@ test('control uses the dataplane socket protocol', async () => { [{ id: 'custom-test', url: 'https://example.com' }], 'site:custom-test', ); + await client.checkConfig({ outbounds: [] }); + await client.runFailoverProbe('primary', [], 'site:youtube', 9_000); + await client.readFailoverSelector(); + await client.selectFailoverRole('reserve'); + await client.setFailoverActivityEnabled(true); + await client.readFailoverActivity(1024); assert.equal(client.running, true); await client.restart(); assert.equal((await client.stop()).running, false); @@ -44,6 +50,12 @@ test('control uses the dataplane socket protocol', async () => { 'GET /device-policy /run/dataplane.sock', 'PUT /device-policy /run/dataplane.sock', 'POST /diagnostics/connectivity /run/dataplane.sock', + 'POST /config/check /run/dataplane.sock', + 'POST /failover/probe /run/dataplane.sock', + 'GET /failover/selector /run/dataplane.sock', + 'PUT /failover/selector /run/dataplane.sock', + 'PUT /failover/activity /run/dataplane.sock', + 'POST /failover/activity/read /run/dataplane.sock', 'POST /restart /run/dataplane.sock', 'POST /stop /run/dataplane.sock', ]); @@ -53,6 +65,10 @@ test('control uses the dataplane socket protocol', async () => { target: 'site:custom-test', }); assert.equal(requests[7].timeoutMs, 25_000); + assert.deepEqual(requests[8].body, { config: { outbounds: [] } }); + assert.deepEqual(requests[9].body, { role: 'primary', services: [], target: 'site:youtube', timeoutMs: 9_000 }); + assert.equal(requests[9].timeoutMs, 19_000); + assert.deepEqual(requests[11].body, { role: 'reserve' }); }); test('connectivity diagnostics expose a retryable domain error', async () => { diff --git a/test/server/domain-traffic.test.js b/test/server/domain-traffic.test.js index 7c5c810..daf7a6e 100644 --- a/test/server/domain-traffic.test.js +++ b/test/server/domain-traffic.test.js @@ -8,7 +8,7 @@ import { deviceId } from '../../dist/server/services/deviceInventoryService.js'; const mac = '00:11:22:33:44:55'; const id = deviceId(mac); -const device = { ip: '192.168.50.7', mac }; +const device = { ip: '192.168.50.7', mac, alias: 'MacBook' }; const connection = (connectionId, type, host, upload, download, sourceIP = device.ip, chains = ['vpn-out']) => ({ id: connectionId, metadata: { type, host, sourceIP }, @@ -205,3 +205,36 @@ test('domain classification normalizes known services and rejects IP or malforme assert.equal(classifyDomain('192.0.2.1'), null); assert.equal(classifyDomain('broken_label.example'), null); }); + +test('failover activity is zero-work while disabled and uses the existing connection poll', async () => { + let observedAt = new Date('2026-08-19T10:00:00.000Z'); + let response = { connections: [ + connection('work', 'tproxy/tproxy-in', 'r1.googlevideo.com', 0, 0, device.ip, ['channel-primary', 'channel-selector']), + connection('probe', 'mixed/diagnostics-primary-in', 'youtube.com', 0, 10_000, device.ip, ['channel-primary']), + ] }; + const service = createDomainTrafficService({ + observe: async () => response, + devices: () => [device], + now: () => observedAt, + }); + + await service.refresh(); + assert.equal(service.activitySnapshot(100), null); + service.enableActivity(); + observedAt = new Date('2026-08-19T10:00:02.000Z'); + response.connections[0].download = 2_000; + response.connections[1].download = 99_000; + await service.refresh(); + const active = service.activitySnapshot(500); + assert.equal(active.state, 'active'); + assert.equal(active.totalBytesPerSecond, 1_000); + assert.equal(active.transmittingConnections, 1); + assert.equal(active.blockers[0].device, 'MacBook'); + assert.equal(active.blockers[0].service, 'YouTube'); + + observedAt = new Date('2026-08-19T10:00:14.000Z'); + await service.refresh(); + assert.equal(service.activitySnapshot(500).state, 'quiet'); + service.disableActivity(); + assert.equal(service.activitySnapshot(500), null); +}); diff --git a/test/server/failover-route.test.js b/test/server/failover-route.test.js new file mode 100644 index 0000000..5a65124 --- /dev/null +++ b/test/server/failover-route.test.js @@ -0,0 +1,54 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; + +import { createFailoverRoute } from '../../dist/server/http/routes/failoverRoute.js'; + +function request(method, url = '/api/failover') { + return { method, url }; +} + +test('Gateway failover route serializes save, pause, checks and manual switch mutations', async () => { + const calls = []; + let body = { policy: { enabled: true }, expectedRevision: 7 }; + const route = createFailoverRoute({ + appMode: 'gateway', + failover: { + save: async (value) => calls.push(['save', value]), + pause: async (value) => calls.push(['pause', value]), + manualSwitch: async (value) => calls.push(['switch', value]), + checkNow: async () => calls.push(['check']), + }, + readBody: async () => body, + withOperation: async (kind, operation, options) => { + calls.push(['operation', kind, options]); + return operation(); + }, + sendState: async () => calls.push(['state']), + }); + + assert.equal(await route.handle(request('PUT'), {}), true); + body = { paused: true, expectedRevision: 8 }; + assert.equal(await route.handle(request('POST', '/api/failover/pause'), {}), true); + body = { role: 'reserve', expectedRevision: 9 }; + assert.equal(await route.handle(request('POST', '/api/failover/switch'), {}), true); + body = { expectedRevision: 10 }; + assert.equal(await route.handle(request('POST', '/api/failover/check'), {}), true); + assert.deepEqual(calls, [ + ['operation', 'failover-save', { expectedRevision: 7 }], ['save', { enabled: true }], ['state'], + ['operation', 'failover-pause', { expectedRevision: 8 }], ['pause', true], ['state'], + ['operation', 'failover-switch', { expectedRevision: 9 }], ['switch', 'reserve'], ['state'], + ['check'], ['state'], + ]); +}); + +test('failover mutations stay Gateway-only and reject unknown roles', async () => { + const connectRoute = createFailoverRoute({ + appMode: 'client', failover: {}, readBody: async () => ({}), withOperation: async () => {}, sendState: async () => {}, + }); + await assert.rejects(connectRoute.handle(request('PUT'), {}), { code: 'ENDPOINT_NOT_FOUND' }); + + const gatewayRoute = createFailoverRoute({ + appMode: 'gateway', failover: {}, readBody: async () => ({ role: 'other' }), withOperation: async () => {}, sendState: async () => {}, + }); + await assert.rejects(gatewayRoute.handle(request('POST', '/api/failover/switch'), {}), { code: 'REQUEST_INVALID' }); +}); diff --git a/test/server/failover-service.test.js b/test/server/failover-service.test.js new file mode 100644 index 0000000..2c0e6ca --- /dev/null +++ b/test/server/failover-service.test.js @@ -0,0 +1,635 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import test from 'node:test'; + +import { + DEFAULT_FAILOVER_POLICY, + isFailoverConfigured, + nextFailoverDecision, + normalizeFailoverPolicy, +} from '../../dist/shared/failover.js'; +import { createFailoverService } from '../../dist/server/features/failover/failoverService.js'; + +const enabled = normalizeFailoverPolicy({ + ...DEFAULT_FAILOVER_POLICY, + enabled: true, + primary: { profileId: 'profile-1', serverId: 'server-1' }, + reserve: { profileId: 'profile-2', serverId: 'server-2' }, + intervalMs: 15_000, + failureWindowMs: 30_000, + recoveryWindowMs: 60_000, + trafficGuard: { enabled: true, thresholdBytesPerSecond: 1024, quietWindowMs: 5_000 }, + minimumReserveMs: 60_000, +}); + +test('failover policy defaults to strict disabled zero-work state', () => { + const policy = normalizeFailoverPolicy(null); + const decision = nextFailoverDecision({ + now: 0, + policy, + currentRole: 'primary', + primaryHealth: 'unhealthy', + reserveHealth: 'healthy', + activity: 'quiet', + }); + assert.equal(policy.enabled, false); + assert.equal(decision.status, 'idle'); + assert.equal(decision.switchTo, null); +}); + +test('failover save rejects policy values that would otherwise be silently clamped', async () => { + const testHarness = harness(serviceState()); + await assert.rejects(testHarness.service.save({ ...enabled, intervalMs: -1 }), { code: 'REQUEST_INVALID' }); +}); + +test('strict failover validation accepts semantically identical reordered JSON keys', () => { + const reordered = { + flapProtection: enabled.flapProtection, + minimumReserveMs: enabled.minimumReserveMs, + trafficGuard: enabled.trafficGuard, + recoveryWindowMs: enabled.recoveryWindowMs, + failureWindowMs: enabled.failureWindowMs, + intervalMs: enabled.intervalMs, + checks: enabled.checks, + reserve: enabled.reserve, + primary: enabled.primary, + paused: enabled.paused, + enabled: enabled.enabled, + version: enabled.version, + }; + assert.deepEqual(normalizeFailoverPolicy(reordered, { strict: true }), enabled); +}); + +test('enabled failover requires two distinct existing targets', async () => { + const sameTarget = normalizeFailoverPolicy({ ...enabled, reserve: enabled.primary }); + assert.equal(isFailoverConfigured(sameTarget), false); + await assert.rejects(harness(serviceState()).service.save(sameTarget), { code: 'REQUEST_INVALID' }); + + const state = serviceState({ ...enabled, paused: true }); + state.profiles[1].servers = []; + const testHarness = harness(state); + await assert.rejects(testHarness.service.pause(false), { code: 'REQUEST_INVALID' }); + assert.equal(testHarness.read().failoverPolicy.paused, true); +}); + +test('local Gateway activity uses the existing domain traffic collector only while enabled', () => { + const source = fs.readFileSync(new URL('../../src/server/index.ts', import.meta.url), 'utf8'); + assert.match(source, /const localFailoverTraffic = !remoteDataplane && settings\.appMode === 'gateway'[\s\S]*createDomainTrafficService/); + assert.match(source, /setLocalFailoverActivityEnabled[\s\S]*enableActivity\(\)[\s\S]*setInterval\(refresh, 2_000\)/); + assert.match(source, /if \(!enabled\)[\s\S]*clearInterval\(localFailoverTrafficTimer\)[\s\S]*disableActivity\(\)/); + assert.doesNotMatch(source, /setFailoverActivityEnabled: async \(\) => \(\{\}\)/); +}); + +test('failure window and traffic guard delay reserve switch until continuous quiet', () => { + let decision = nextFailoverDecision({ + now: 0, + policy: enabled, + currentRole: 'primary', + primaryHealth: 'unhealthy', + reserveHealth: 'healthy', + activity: 'active', + }); + decision = nextFailoverDecision({ + now: 30_000, + policy: enabled, + currentRole: 'primary', + primaryHealth: 'unhealthy', + reserveHealth: 'healthy', + activity: 'active', + memory: decision.memory, + }); + assert.equal(decision.status, 'waiting-for-idle'); + decision = nextFailoverDecision({ + now: 31_000, + policy: enabled, + currentRole: 'primary', + primaryHealth: 'unhealthy', + reserveHealth: 'healthy', + activity: 'quiet', + memory: decision.memory, + }); + assert.equal(decision.switchTo, null); + decision = nextFailoverDecision({ + now: 36_000, + policy: enabled, + currentRole: 'primary', + primaryHealth: 'unhealthy', + reserveHealth: 'healthy', + activity: 'quiet', + memory: decision.memory, + }); + assert.equal(decision.switchTo, 'reserve'); +}); + +test('both unhealthy and unknown activity never cause a switch', () => { + const both = nextFailoverDecision({ + now: 30_000, + policy: enabled, + currentRole: 'primary', + primaryHealth: 'unhealthy', + reserveHealth: 'unhealthy', + activity: 'quiet', + }); + assert.equal(both.reason, 'both-unhealthy'); + assert.equal(both.switchTo, null); + + const unknown = nextFailoverDecision({ + now: 30_000, + policy: enabled, + currentRole: 'primary', + primaryHealth: 'unhealthy', + reserveHealth: 'healthy', + activity: 'unknown', + memory: { primaryFailedSince: 0, primaryRecoveredSince: null, quietSince: null }, + }); + assert.equal(unknown.reason, 'activity-unknown'); + assert.equal(unknown.switchTo, null); +}); + +test('failback waits for recovery, hold and quarantine deadlines', () => { + let decision = nextFailoverDecision({ + now: 10_000, + policy: enabled, + currentRole: 'reserve', + primaryHealth: 'healthy', + reserveHealth: 'healthy', + activity: 'quiet', + holdUntil: 50_000, + primaryQuarantineUntil: 70_000, + }); + assert.equal(decision.nextDecisionAt, 70_000); + decision = nextFailoverDecision({ + now: 70_000, + policy: enabled, + currentRole: 'reserve', + primaryHealth: 'healthy', + reserveHealth: 'healthy', + activity: 'quiet', + holdUntil: 50_000, + primaryQuarantineUntil: 70_000, + memory: decision.memory, + }); + assert.equal(decision.switchTo, null); + assert.equal(decision.nextDecisionAt, 130_000); + decision = nextFailoverDecision({ + now: 130_000, + policy: enabled, + currentRole: 'reserve', + primaryHealth: 'healthy', + reserveHealth: 'healthy', + activity: 'quiet', + holdUntil: 50_000, + primaryQuarantineUntil: 70_000, + memory: { ...decision.memory, quietSince: 125_000 }, + }); + assert.equal(decision.switchTo, 'primary'); + + const deadReserve = nextFailoverDecision({ + now: 70_000, + policy: enabled, + currentRole: 'reserve', + primaryHealth: 'healthy', + reserveHealth: 'unhealthy', + activity: 'quiet', + holdUntil: 500_000, + primaryQuarantineUntil: 700_000, + memory: { primaryFailedSince: null, primaryRecoveredSince: 10_000, quietSince: 65_000 }, + }); + assert.equal(deadReserve.switchTo, null); + assert.equal(deadReserve.nextDecisionAt, 700_000); +}); + +function serviceState(policy = enabled) { + return { + revision: 1, + failoverPolicy: policy, + failoverRuntimeState: { lastSwitchAt: null, holdUntil: null, primaryQuarantineUntil: null, failoverHistory: [], reasonCode: null }, + appliedFailoverPolicy: { + primary: { profileId: 'profile-1', serverId: 'server-1' }, + reserve: { profileId: 'profile-2', serverId: 'server-2' }, + primaryConfigFingerprint: 'a'.repeat(64), + reserveConfigFingerprint: 'b'.repeat(64), + }, + appliedProfileId: 'profile-1', + appliedServerId: 'server-1', + profiles: [ + { id: 'profile-1', servers: [{ id: 'server-1', label: 'Primary' }] }, + { id: 'profile-2', servers: [{ id: 'server-2', label: 'Reserve' }] }, + ], + diagnostics: { customServices: [] }, + }; +} + +function harness(initial = serviceState(), overrides = {}) { + let state = structuredClone(initial); + const calls = []; + const dependencies = { + state: { + read: () => state, + update: (mutator) => { + state = { ...mutator(state), revision: state.revision + 1 }; + calls.push('state:update'); + return state; + }, + }, + runtime: { isRunning: async () => true }, + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async () => ({ vpn: { sites: [{ status: 'available' }] } }), + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async (role) => { calls.push(`select:${role}`); return { role }; }, + setFailoverActivityEnabled: async (value) => { calls.push(`activity:${value}`); }, + readFailoverActivity: async () => ({ activity: { state: 'quiet', observedAt: new Date().toISOString() } }), + }, + buildCandidate: () => ({ config: {}, applied: initial.appliedFailoverPolicy }), + serialize: (operation) => operation(), + scheduler: { + setTimeout: () => ({ unref() {} }), + clearTimeout: () => {}, + }, + ...overrides, + }; + return { service: createFailoverService(dependencies), calls, read: () => state }; +} + +test('disabled failover performs one restart-safe collector cleanup and no steady-state work', async () => { + const state = serviceState(normalizeFailoverPolicy(null)); + state.appliedFailoverPolicy = null; + const testHarness = harness(state); + await testHarness.service.reconcile(); + assert.deepEqual(testHarness.calls, ['activity:false']); + await testHarness.service.reconcile(); + assert.deepEqual(testHarness.calls, ['activity:false']); + assert.equal(testHarness.service.snapshot().activation, 'inactive'); +}); + +test('disabled failover reports an already loaded dual config without managing it', async () => { + const testHarness = harness(serviceState(normalizeFailoverPolicy(null))); + await testHarness.service.reconcile(); + assert.deepEqual(testHarness.calls, ['activity:false']); + assert.equal(testHarness.service.snapshot().activation, 'passive-loaded'); + assert.equal(testHarness.service.snapshot().currentRole, 'primary'); +}); + +test('runtime rollback restores the role from applied truth even while disabled', async () => { + const state = serviceState(normalizeFailoverPolicy(null)); + state.appliedProfileId = state.appliedFailoverPolicy.reserve.profileId; + state.appliedServerId = state.appliedFailoverPolicy.reserve.serverId; + const testHarness = harness(state); + await testHarness.service.restoreAppliedActivation(state); + assert.deepEqual(testHarness.calls, ['select:reserve']); +}); + +test('an in-flight observation is discarded after failover is disabled', async () => { + const pending = []; + const testHarness = harness(serviceState(), { + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: (role) => new Promise((resolve) => pending.push({ role, resolve })), + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async (role) => { testHarness.calls.push(`select:${role}`); return { role }; }, + setFailoverActivityEnabled: async (value) => { testHarness.calls.push(`activity:${value}`); }, + readFailoverActivity: async () => ({ activity: { state: 'quiet', observedAt: new Date().toISOString() } }), + }, + }); + await testHarness.service.reconcile(); + const round = testHarness.service.runRound(); + await new Promise((resolve) => setImmediate(resolve)); + await testHarness.service.save({ ...enabled, enabled: false }); + for (const probe of pending) probe.resolve({ vpn: { sites: [{ status: probe.role === 'reserve' ? 'available' : 'unavailable' }] } }); + await round; + assert.equal(testHarness.read().failoverPolicy.enabled, false); + assert.equal(testHarness.calls.some((call) => call.startsWith('select:reserve')), false); + assert.deepEqual(testHarness.calls.filter((call) => call.startsWith('activity:')), ['activity:true', 'activity:true', 'activity:false']); +}); + +test('one channel probe failure does not erase the other channel health', async () => { + const testHarness = harness(serviceState(), { + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async (role) => { + if (role === 'primary') throw new Error('primary probe transport failed'); + return { vpn: { sites: [{ status: 'available' }] } }; + }, + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async (role) => ({ role }), + setFailoverActivityEnabled: async () => {}, + readFailoverActivity: async () => ({ activity: { state: 'active', observedAt: new Date().toISOString() } }), + }, + }); + await testHarness.service.checkNow(); + assert.equal(testHarness.service.snapshot().primary.health, 'unknown'); + assert.equal(testHarness.service.snapshot().reserve.health, 'healthy'); +}); + +test('monitoring wakes at an earlier decision deadline instead of waiting a full interval', async () => { + let clock = 0; + const delays = []; + const testHarness = harness(serviceState(), { + now: () => new Date(clock), + scheduler: { + setTimeout: (_callback, delay) => { delays.push(delay); return { unref() {} }; }, + clearTimeout: () => {}, + }, + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async (role) => ({ vpn: { sites: [{ status: role === 'primary' ? 'unavailable' : 'available' }] } }), + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async (role) => ({ role }), + setFailoverActivityEnabled: async () => {}, + readFailoverActivity: async () => ({ activity: { state: 'quiet', observedAt: new Date(clock).toISOString() } }), + }, + }); + await testHarness.service.reconcile(); + await testHarness.service.runRound(); + clock = 30_000; + await testHarness.service.runRound(); + assert.equal(delays.at(-1), enabled.trafficGuard.quietWindowMs); +}); + +test('a late activity response cannot switch after failover is disabled', async () => { + let resolveActivity; + const activity = new Promise((resolve) => { resolveActivity = resolve; }); + const testHarness = harness(serviceState(), { + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async (role) => ({ vpn: { sites: [{ status: role === 'primary' ? 'unavailable' : 'available' }] } }), + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async (role) => { testHarness.calls.push(`select:${role}`); return { role }; }, + setFailoverActivityEnabled: async (value) => { testHarness.calls.push(`activity:${value}`); }, + readFailoverActivity: async () => activity, + }, + }); + await testHarness.service.reconcile(); + const round = testHarness.service.runRound(); + await new Promise(setImmediate); + await testHarness.service.save({ ...enabled, enabled: false }); + resolveActivity({ activity: { state: 'quiet', observedAt: new Date().toISOString() } }); + await round; + assert.equal(testHarness.calls.some((value) => value.startsWith('select:')), false); +}); + +test('automatic switch revalidates activity immediately before selector mutation', async () => { + let clock = 0; + let activityReads = 0; + const testHarness = harness(serviceState(), { + now: () => new Date(clock), + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async (role) => ({ vpn: { sites: [{ status: role === 'primary' ? 'unavailable' : 'available' }] } }), + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async (role) => { testHarness.calls.push(`select:${role}`); return { role }; }, + setFailoverActivityEnabled: async () => {}, + readFailoverActivity: async () => ({ + activity: { + state: ++activityReads >= 4 ? 'active' : 'quiet', + observedAt: new Date(clock).toISOString(), + }, + }), + }, + }); + await testHarness.service.reconcile(); + await testHarness.service.runRound(); + clock = 30_000; + await testHarness.service.runRound(); + clock = 35_000; + await testHarness.service.runRound(); + assert.equal(testHarness.calls.some((value) => value.startsWith('select:')), false); + assert.equal(testHarness.service.snapshot().reason, 'revalidation-required'); +}); + +test('selector read-back failure rolls an uncertain switch back to the canonical role', async () => { + const testHarness = harness(serviceState(), { + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async () => ({}), + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async (role) => { + testHarness.calls.push(`select:${role}`); + if (role === 'reserve') throw new Error('read-back failed'); + return { role }; + }, + setFailoverActivityEnabled: async () => {}, + readFailoverActivity: async () => ({ activity: null }), + }, + }); + await assert.rejects(testHarness.service.manualSwitch('reserve'), /read-back failed/); + assert.deepEqual(testHarness.calls, ['select:reserve', 'select:primary']); + assert.equal(testHarness.read().appliedServerId, 'server-1'); +}); + +test('failed collector disable is retried because local state remains unknown', async () => { + let attempts = 0; + const testHarness = harness(serviceState(normalizeFailoverPolicy(null)), { + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async () => ({}), + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async () => ({}), + setFailoverActivityEnabled: async () => { + attempts += 1; + if (attempts === 1) throw new Error('dataplane unavailable'); + }, + readFailoverActivity: async () => ({ activity: null }), + }, + }); + await assert.rejects(testHarness.service.reconcile(), /dataplane unavailable/); + await testHarness.service.reconcile(); + assert.equal(attempts, 2); +}); + +test('each monitoring round repairs selector and collector state after a dataplane restart', async () => { + const state = serviceState(); + state.appliedProfileId = 'profile-2'; + state.appliedServerId = 'server-2'; + let selected = 'reserve'; + let enables = 0; + const testHarness = harness(state, { + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async () => ({ vpn: { sites: [{ status: 'available' }] } }), + readFailoverSelector: async () => ({ role: selected }), + selectFailoverRole: async (role) => { selected = role; return { role }; }, + setFailoverActivityEnabled: async (value) => { if (value) enables += 1; }, + readFailoverActivity: async () => ({ activity: { state: 'quiet', observedAt: new Date().toISOString() } }), + }, + }); + await testHarness.service.reconcile(); + selected = 'primary'; + await testHarness.service.runRound(); + assert.equal(selected, 'reserve'); + assert.equal(enables, 2); +}); + +test('selector acknowledgement precedes state commit and a failed commit rolls selector back', async () => { + const state = serviceState(); + const calls = []; + const testHarness = harness(state, { + state: { + read: () => state, + update: (mutator) => { + const candidate = mutator(state); + calls.push('state:update'); + if (candidate.appliedServerId === 'server-2') throw new Error('write failed'); + Object.assign(state, candidate); + return state; + }, + }, + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async () => ({}), + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async (role) => { calls.push(`select:${role}`); return { role }; }, + setFailoverActivityEnabled: async () => {}, + readFailoverActivity: async () => ({ activity: null }), + }, + }); + await assert.rejects(testHarness.service.manualSwitch('reserve'), /write failed/); + assert.deepEqual(calls, ['select:reserve', 'state:update', 'select:primary']); + assert.equal(state.appliedServerId, 'server-1'); +}); + +test('manual selector and pause publish in one canonical commit', async () => { + const testHarness = harness(serviceState()); + await testHarness.service.manualSwitch('reserve'); + assert.equal(testHarness.read().appliedServerId, 'server-2'); + assert.equal(testHarness.read().failoverPolicy.paused, true); + assert.equal(testHarness.calls.filter((call) => call === 'state:update').length, 1); +}); + +test('post-commit monitoring failure does not report a committed pause as failed', async () => { + const warnings = []; + const testHarness = harness(serviceState(), { + runtime: { isRunning: async () => { throw new Error('runtime unavailable'); } }, + onWarning: (error) => warnings.push(error.message), + }); + await testHarness.service.pause(true); + assert.equal(testHarness.read().failoverPolicy.paused, true); + assert.equal(testHarness.service.snapshot().status, 'error'); + assert.deepEqual(warnings, ['runtime unavailable']); +}); + +test('pause and resume discard stale failure-window evidence', async () => { + let clock = 1_000; + const testHarness = harness(serviceState(), { + now: () => new Date(clock), + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async (role) => ({ + vpn: { sites: [{ status: role === 'primary' ? 'unavailable' : 'available' }] }, + }), + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async (role) => { testHarness.calls.push(`select:${role}`); return { role }; }, + setFailoverActivityEnabled: async (value) => { testHarness.calls.push(`activity:${value}`); }, + readFailoverActivity: async () => ({ + activity: { state: 'quiet', observedAt: new Date(clock).toISOString(), quietSince: new Date(clock - 10_000).toISOString() }, + }), + }, + }); + await testHarness.service.runRound(); + await testHarness.service.pause(true); + clock = 3_600_000; + await testHarness.service.pause(false); + await testHarness.service.runRound(); + assert.equal(testHarness.calls.includes('select:reserve'), false); + assert.equal(testHarness.service.snapshot().reason, 'failure-window'); +}); + +test('an unconfirmed selector rollback durably pauses automation', async () => { + const testHarness = harness(serviceState(), { + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async () => ({}), + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async () => { throw new Error('selector unavailable'); }, + setFailoverActivityEnabled: async () => {}, + readFailoverActivity: async () => ({ activity: null }), + }, + }); + await assert.rejects(testHarness.service.manualSwitch('reserve'), AggregateError); + assert.equal(testHarness.read().failoverPolicy.paused, true); + assert.equal(testHarness.read().failoverRuntimeState.reasonCode, 'selector-unknown'); + assert.equal(testHarness.service.snapshot().reason, 'selector-unknown'); + assert.equal(testHarness.service.snapshot().currentRole, 'other'); +}); + +test('successful selector reconciliation clears an unknown-role latch', async () => { + const state = serviceState(); + state.failoverRuntimeState.reasonCode = 'selector-unknown'; + const testHarness = harness(state); + await testHarness.service.reconcile(); + assert.equal(testHarness.read().failoverRuntimeState.reasonCode, null); + assert.equal(testHarness.service.snapshot().currentRole, 'primary'); +}); + +test('disabling on reserve preserves it as the next ordinary single-channel target', async () => { + const state = serviceState(); + state.appliedProfileId = 'profile-2'; + state.appliedServerId = 'server-2'; + const testHarness = harness(state); + await testHarness.service.save({ ...enabled, enabled: false }); + assert.equal(testHarness.read().desiredProfileId, 'profile-2'); + assert.equal(testHarness.read().profiles[1].desiredServerId, 'server-2'); + assert.equal(testHarness.service.snapshot().activation, 'passive-loaded'); +}); + +test('significant health states append once without logging every observation', async () => { + let clock = 0; + const health = { primary: 'unavailable', reserve: 'available' }; + const events = []; + const testHarness = harness(serviceState(), { + now: () => new Date(clock), + onEvent: (event) => events.push(event), + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async (role) => ({ vpn: { sites: [{ status: health[role] }] } }), + readFailoverSelector: async () => ({ role: 'primary' }), + selectFailoverRole: async (role) => ({ role }), + setFailoverActivityEnabled: async () => {}, + readFailoverActivity: async () => ({ activity: { state: 'quiet', observedAt: new Date(clock).toISOString() } }), + }, + }); + await testHarness.service.reconcile(); + await testHarness.service.runRound(); + clock = 30_000; + await testHarness.service.runRound(); + health.reserve = 'unavailable'; + clock = 31_000; + await testHarness.service.runRound(); + clock = 32_000; + await testHarness.service.runRound(); + health.primary = 'available'; + clock = 33_000; + await testHarness.service.runRound(); + + assert.deepEqual(events.map(({ type }) => type), [ + 'failover.waiting_for_idle', + 'failover.both_unhealthy', + 'failover.recovered', + ]); +}); + +test('pending target edits keep selector commits bound to the loaded channel pair', async () => { + const state = serviceState(); + state.failoverPolicy = normalizeFailoverPolicy({ + ...state.failoverPolicy, + reserve: { profileId: 'profile-3', serverId: 'server-3' }, + }); + state.profiles.push({ id: 'profile-3', servers: [{ id: 'server-3', label: 'Desired later' }] }); + const testHarness = harness(state, { + buildCandidate: () => ({ + config: {}, + applied: { + ...state.appliedFailoverPolicy, + reserve: { profileId: 'profile-3', serverId: 'server-3' }, + reserveConfigFingerprint: 'c'.repeat(64), + }, + }), + }); + + await testHarness.service.manualSwitch('reserve'); + assert.equal(testHarness.read().appliedProfileId, 'profile-2'); + assert.equal(testHarness.read().appliedServerId, 'server-2'); + assert.equal(testHarness.service.snapshot().activation, 'pending'); + assert.deepEqual(testHarness.service.snapshot().reserve.target, { profileId: 'profile-2', serverId: 'server-2' }); +}); diff --git a/test/server/route-rules-service.test.js b/test/server/route-rules-service.test.js index bfe0afb..07756e5 100644 --- a/test/server/route-rules-service.test.js +++ b/test/server/route-rules-service.test.js @@ -109,6 +109,9 @@ function createHarness(overrides = {}) { events.push('operation'); return operation(); }, + restoreAppliedActivation: overrides.restoreAppliedActivation + ? async (previousState) => overrides.restoreAppliedActivation(previousState, events) + : undefined, }); return { @@ -246,6 +249,24 @@ test('route rules rollback continues and classifies runtime restore failure', as assert.ok(broken.events.filter((event) => event === 'state.update').length >= 1); }); +test('route rules rollback restores the previous reserve selector', async () => { + const state = canonicalState(); + state.appliedFailoverPolicy = { + primary: { profileId: 'primary', serverId: 'other' }, + reserve: { profileId: 'primary', serverId: 'server' }, + }; + const harness = createHarness({ + state, + failures: { stateUpdates: [{ after: new Error('state') }] }, + restoreAppliedActivation: async (previousState, events) => { + assert.equal(previousState.appliedServerId, 'server'); + events.push('selector.reserve'); + }, + }); + await assert.rejects(harness.service.update(newRules, 2, 2), /state/); + assert.ok(harness.events.indexOf('selector.reserve') > harness.events.indexOf('runtime.restore')); +}); + test('route rules route preserves one adapter and state-only response', async () => { const calls = []; const route = createRouteRulesRoute({ @@ -260,5 +281,6 @@ test('route rules route preserves one adapter and state-only response', async () const source = readFileSync(new URL('../../src/server/index.ts', import.meta.url), 'utf8'); assert.match(source, /createRouteRulesRoute\(\{/); + assert.match(source, /state\.appliedFailoverPolicy[\s\S]*buildFailoverCandidate\(\{ \.\.\.state, routeRules \}, 'applied'\)\.config/); assert.doesNotMatch(source, /function applyRouteRules|req\.url === ['"]\/api\/route-rules['"]/); }); diff --git a/test/server/singbox-gateway-mode.test.js b/test/server/singbox-gateway-mode.test.js index ec74996..705a583 100644 --- a/test/server/singbox-gateway-mode.test.js +++ b/test/server/singbox-gateway-mode.test.js @@ -8,7 +8,13 @@ process.env.APP_MODE = 'gateway'; process.env.DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'vpn-proxy-gateway-test-')); process.env.SING_BOX_CACHE = path.join(process.env.DATA_DIR, 'cache.db'); -const { buildGatewayConfig } = await import(`../../dist/server/singbox.js?gateway=${Date.now()}`); +const { + buildDualChannelGatewayConfig, + buildGatewayConfig, + dualChannelConfigMatchesApplied, + fingerprintConfiguredOutbound, + fingerprintSelectedOutbound, +} = await import(`../../dist/server/singbox.js?gateway=${Date.now()}`); const subscriptionConfig = { outbounds: [{ @@ -59,3 +65,62 @@ test('gateway preserves mixed user-rule order and dynamic VPN target', () => { assert.deepEqual(config.experimental.clash_api, { external_controller: '127.0.0.1:19090' }); assert.equal(config.route.final, 'test-vpn'); }); + +test('gateway dual-channel config fixes probes to role tags and keeps inbound connections', () => { + const reserveConfig = structuredClone(subscriptionConfig); + reserveConfig.outbounds[0].tag = 'same-provider-tag'; + const primaryConfig = structuredClone(subscriptionConfig); + primaryConfig.outbounds[0].tag = 'same-provider-tag'; + const config = buildDualChannelGatewayConfig({ + primary: { subscriptionConfig: primaryConfig, selectedServerId: 'same-provider-tag' }, + reserve: { subscriptionConfig: reserveConfig, selectedServerId: 'same-provider-tag' }, + }, { + routeRules: [{ type: 'domain', value: 'api.example.com', enabled: true, outbound: 'vpn' }], + }); + + assert.deepEqual(config.outbounds.map(({ tag }) => tag), [ + 'channel-primary', 'channel-reserve', 'channel-selector', 'direct', + ]); + assert.deepEqual(config.outbounds[2], { + type: 'selector', + tag: 'channel-selector', + outbounds: ['channel-primary', 'channel-reserve'], + default: 'channel-primary', + interrupt_exist_connections: false, + }); + assert.deepEqual(config.route.rules.slice(1, 5), [ + { inbound: ['diagnostics-primary-in'], outbound: 'channel-primary' }, + { inbound: ['diagnostics-reserve-in'], outbound: 'channel-reserve' }, + { inbound: ['diagnostics-vpn-in'], outbound: 'channel-selector' }, + { domain: ['api.example.com'], outbound: 'channel-selector' }, + ]); + assert.equal(config.route.final, 'channel-selector'); + const restoredReserve = buildDualChannelGatewayConfig({ + primary: { subscriptionConfig: primaryConfig, selectedServerId: 'same-provider-tag' }, + reserve: { subscriptionConfig: reserveConfig, selectedServerId: 'same-provider-tag' }, + }, { defaultRole: 'reserve' }); + assert.equal(restoredReserve.outbounds[2].default, 'channel-reserve'); +}); + +test('cached dual-channel outbounds must match the applied provider fingerprints', () => { + const config = buildDualChannelGatewayConfig({ + primary: { subscriptionConfig, selectedServerId: 'test-vpn' }, + reserve: { subscriptionConfig, selectedServerId: 'test-vpn' }, + }); + const expected = fingerprintSelectedOutbound(subscriptionConfig, 'test-vpn'); + const applied = { + primary: { profileId: 'primary', serverId: 'test-vpn' }, + reserve: { profileId: 'reserve', serverId: 'test-vpn' }, + primaryConfigFingerprint: expected, + reserveConfigFingerprint: expected, + }; + assert.equal(dualChannelConfigMatchesApplied(config, applied, 'primary'), true); + assert.equal(dualChannelConfigMatchesApplied(config, applied, 'reserve'), false); + assert.equal(fingerprintConfiguredOutbound(config.outbounds[0], 'test-vpn'), expected); + config.outbounds[0].uuid = '11111111-1111-4111-8111-111111111111'; + assert.notEqual(fingerprintConfiguredOutbound(config.outbounds[0], 'test-vpn'), expected); + assert.equal(dualChannelConfigMatchesApplied(config, applied, 'primary'), false); + config.outbounds[0].uuid = subscriptionConfig.outbounds[0].uuid; + config.outbounds[2].outbounds = ['channel-primary']; + assert.equal(dualChannelConfigMatchesApplied(config, applied, 'primary'), false); +}); diff --git a/test/server/singbox-selector-capability.test.js b/test/server/singbox-selector-capability.test.js new file mode 100644 index 0000000..77f2e93 --- /dev/null +++ b/test/server/singbox-selector-capability.test.js @@ -0,0 +1,361 @@ +import assert from 'node:assert/strict'; +import { spawn, spawnSync } from 'node:child_process'; +import dgram from 'node:dgram'; +import fs from 'node:fs'; +import net from 'node:net'; +import os from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; + +import { createFailoverService } from '../../dist/server/features/failover/failoverService.js'; +import { createDomainTrafficService } from '../../dist/server/services/domainTrafficService.js'; +import { DEFAULT_FAILOVER_POLICY, normalizeFailoverPolicy } from '../../dist/shared/failover.js'; + +const image = process.env.HARBOR_SINGBOX_IMAGE; + +function listen(server, host = '127.0.0.1') { + return new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(0, host, () => resolve(server.address().port)); + }); +} + +function listenUdp(socket, host = '127.0.0.1') { + return new Promise((resolve, reject) => { + socket.once('error', reject); + socket.bind(0, host, () => resolve(socket.address().port)); + }); +} + +function readExactly(socket, size) { + return new Promise((resolve, reject) => { + let value = Buffer.alloc(0); + const onData = (chunk) => { + value = Buffer.concat([value, chunk]); + if (value.length < size) return; + cleanup(); + if (value.length > size) { + socket.pause(); + socket.unshift(value.subarray(size)); + } + resolve(value.subarray(0, size)); + }; + const cleanup = () => { + socket.off('data', onData); + socket.off('error', reject); + socket.off('end', onEnd); + }; + const onEnd = () => { + cleanup(); + reject(new Error('Socket ended early')); + }; + socket.on('data', onData); + socket.once('error', reject); + socket.once('end', onEnd); + socket.resume(); + }); +} + +async function openSocksConnection(proxyPort, targetPort, diagnostic = () => {}) { + const socket = net.connect(proxyPort, '127.0.0.1'); + socket.setTimeout(3_000, () => socket.destroy(new Error('SOCKS fixture timed out'))); + await new Promise((resolve, reject) => { + socket.once('connect', resolve); + socket.once('error', reject); + }); + diagnostic('tcp connected'); + socket.write(Buffer.from([5, 1, 0])); + assert.deepEqual(await readExactly(socket, 2), Buffer.from([5, 0])); + diagnostic('socks greeting accepted'); + const host = Buffer.from('host.docker.internal'); + socket.write(Buffer.from([5, 1, 0, 3, host.length, ...host, targetPort >> 8, targetPort & 255])); + const response = await readExactly(socket, 4); + diagnostic(`socks connect response ${response.toString('hex')}`); + assert.equal(response[1], 0); + const addressLength = response[3] === 0 + ? 0 + : response[3] === 1 + ? 4 + : response[3] === 4 + ? 16 + : (await readExactly(socket, 1))[0]; + await readExactly(socket, addressLength + 2); + return socket; +} + +async function echo(socket, value) { + socket.write(value); + assert.equal((await readExactly(socket, Buffer.byteLength(value))).toString(), value); +} + +async function openSocksUdpAssociation(proxyPort, targetPort, resolveRelayPort) { + const control = net.connect(proxyPort, '127.0.0.1'); + control.setTimeout(3_000, () => control.destroy(new Error('SOCKS UDP fixture timed out'))); + await new Promise((resolve, reject) => { + control.once('connect', resolve); + control.once('error', reject); + }); + control.write(Buffer.from([5, 1, 0])); + assert.deepEqual(await readExactly(control, 2), Buffer.from([5, 0])); + control.write(Buffer.from([5, 3, 0, 1, 0, 0, 0, 0, 0, 0])); + const response = await readExactly(control, 4); + assert.equal(response[1], 0); + const addressLength = response[3] === 1 ? 4 : response[3] === 4 ? 16 : (await readExactly(control, 1))[0]; + const addressAndPort = await readExactly(control, addressLength + 2); + const boundPort = addressAndPort.readUInt16BE(addressAndPort.length - 2); + const relayPort = resolveRelayPort(boundPort); + + const socket = dgram.createSocket('udp4'); + await listenUdp(socket); + return { + close() { socket.close(); control.destroy(); }, + async echo(value) { + const host = Buffer.from('host.docker.internal'); + const payload = Buffer.from(value); + const packet = Buffer.from([0, 0, 0, 3, host.length, ...host, targetPort >> 8, targetPort & 255, ...payload]); + const reply = new Promise((resolve, reject) => { + const timeout = setTimeout(() => reject(new Error(`SOCKS UDP echo timed out; relay ${boundPort}`)), 3_000); + socket.once('message', (message) => { + clearTimeout(timeout); + const headerLength = message[3] === 1 ? 10 : message[3] === 4 ? 22 : 7 + message[4]; + resolve(message.subarray(headerLength).toString()); + }); + }); + socket.send(packet, relayPort, '127.0.0.1'); + assert.equal(await reply, value); + }, + }; +} + +async function waitForApi(port) { + for (let attempt = 0; attempt < 80; attempt += 1) { + try { + const response = await fetch(`http://127.0.0.1:${port}/proxies/channel-selector`); + if (response.ok) return; + } catch {} + await new Promise((resolve) => setTimeout(resolve, 100)); + } + throw new Error('Clash API did not start'); +} + +test('sing-box 1.13 selector preserves inbound TCP and UDP connections and routes new connections', { + skip: image ? false : 'Set HARBOR_SINGBOX_IMAGE to run the local Docker capability proof', + timeout: 30_000, +}, async (t) => { + const echoServer = net.createServer((socket) => socket.pipe(socket)); + const echoPort = await listen(echoServer, '0.0.0.0'); + const udpEchoServer = dgram.createSocket('udp4'); + udpEchoServer.on('message', (message, remote) => udpEchoServer.send(message, remote.port, remote.address)); + const udpEchoPort = await listenUdp(udpEchoServer, '0.0.0.0'); + const proxyReservation = net.createServer(); + const proxyPort = await listen(proxyReservation); + await new Promise((resolve) => proxyReservation.close(resolve)); + const apiReservation = net.createServer(); + const apiPort = await listen(apiReservation); + await new Promise((resolve) => apiReservation.close(resolve)); + const fixtureDir = fs.mkdtempSync(path.join(os.tmpdir(), 'harbor-selector-')); + const configPath = path.join(fixtureDir, 'config.json'); + const config = { + log: { level: 'error' }, + experimental: { + cache_file: { enabled: true, path: '/config/cache.db' }, + clash_api: { external_controller: '0.0.0.0:19091' }, + }, + inbounds: [ + { type: 'mixed', tag: 'mixed-in', listen: '0.0.0.0', listen_port: 18081 }, + { type: 'mixed', tag: 'diagnostics-primary-in', listen: '0.0.0.0', listen_port: 18082 }, + { type: 'mixed', tag: 'diagnostics-reserve-in', listen: '0.0.0.0', listen_port: 18083 }, + ], + outbounds: [ + { type: 'direct', tag: 'channel-primary' }, + { type: 'direct', tag: 'channel-reserve' }, + { + type: 'selector', + tag: 'channel-selector', + outbounds: ['channel-primary', 'channel-reserve'], + default: 'channel-primary', + interrupt_exist_connections: false, + }, + ], + route: { + rules: [ + { inbound: ['diagnostics-primary-in'], outbound: 'channel-primary' }, + { inbound: ['diagnostics-reserve-in'], outbound: 'channel-reserve' }, + { inbound: ['mixed-in'], outbound: 'channel-selector' }, + ], + final: 'channel-selector', + }, + }; + fs.writeFileSync(configPath, JSON.stringify(config)); + + const mount = `${fixtureDir}:/config`; + const check = spawnSync('docker', [ + 'run', '--rm', '-v', mount, '--entrypoint', 'sing-box', image, + 'check', '-c', '/config/config.json', + ], { encoding: 'utf8' }); + assert.equal(check.status, 0, check.stderr || check.stdout); + t.diagnostic('config accepted'); + + const containerName = `harbor-selector-${process.pid}-${Date.now()}`; + const udpRelayContainerPorts = Array.from({ length: 32 }, (_, index) => 18084 + index); + const runtime = spawn('docker', [ + 'run', '--rm', '--name', containerName, + '--sysctl', `net.ipv4.ip_local_port_range=${udpRelayContainerPorts[0]} ${udpRelayContainerPorts.at(-1)}`, + '-p', `${proxyPort}:18081/tcp`, + ...udpRelayContainerPorts.flatMap((port) => ['-p', `127.0.0.1::${port}/udp`]), + '-p', `${apiPort}:19091`, + '-v', mount, '--entrypoint', 'sing-box', image, + 'run', '-c', '/config/config.json', + ], { stdio: ['ignore', 'pipe', 'pipe'] }); + let runtimeOutput = ''; + runtime.stdout.on('data', (chunk) => { runtimeOutput += chunk; }); + runtime.stderr.on('data', (chunk) => { runtimeOutput += chunk; }); + t.after(async () => { + if (runtime.exitCode == null) { + runtime.kill('SIGTERM'); + await new Promise((resolve) => runtime.once('close', resolve)); + } + echoServer.close(); + udpEchoServer.close(); + fs.rmSync(fixtureDir, { recursive: true, force: true }); + }); + + try { + await waitForApi(apiPort); + const runtimePid = runtime.pid; + t.diagnostic('api ready'); + const resolveUdpRelayPort = (containerPort) => { + const mapping = spawnSync('docker', ['port', containerName, `${containerPort}/udp`], { encoding: 'utf8' }); + assert.equal(mapping.status, 0, mapping.stderr || mapping.stdout); + const port = Number(mapping.stdout.trim().split(':').at(-1)); + assert.ok(Number.isSafeInteger(port) && port > 0, mapping.stdout); + return port; + }; + const before = await (await fetch(`http://127.0.0.1:${apiPort}/proxies/channel-selector`)).json(); + assert.equal(before.now, 'channel-primary'); + + const existing = await openSocksConnection(proxyPort, echoPort, (message) => t.diagnostic(message)); + t.diagnostic('primary connection open'); + t.after(() => existing.destroy()); + await echo(existing, 'before-switch'); + const existingUdp = await openSocksUdpAssociation(proxyPort, udpEchoPort, resolveUdpRelayPort); + t.after(() => existingUdp.close()); + await existingUdp.echo('before-switch-udp'); + + const policy = normalizeFailoverPolicy({ + ...DEFAULT_FAILOVER_POLICY, + enabled: true, + primary: { profileId: 'primary-profile', serverId: 'primary-server' }, + reserve: { profileId: 'reserve-profile', serverId: 'reserve-server' }, + intervalMs: 15_000, + failureWindowMs: 30_000, + recoveryWindowMs: 60_000, + trafficGuard: { enabled: true, thresholdBytesPerSecond: 1024, quietWindowMs: 5_000 }, + minimumReserveMs: 60_000, + }); + const applied = { + primary: policy.primary, + reserve: policy.reserve, + primaryConfigFingerprint: 'a'.repeat(64), + reserveConfigFingerprint: 'b'.repeat(64), + }; + let clock = 0; + let state = { + revision: 1, + failoverPolicy: policy, + failoverRuntimeState: { lastSwitchAt: null, holdUntil: null, primaryQuarantineUntil: null, failoverHistory: [], reasonCode: null }, + appliedFailoverPolicy: applied, + appliedProfileId: policy.primary.profileId, + appliedServerId: policy.primary.serverId, + appliedServerSnapshot: { id: policy.primary.serverId, label: 'Primary' }, + profiles: [ + { id: policy.primary.profileId, servers: [{ id: policy.primary.serverId, label: 'Primary' }] }, + { id: policy.reserve.profileId, servers: [{ id: policy.reserve.serverId, label: 'Reserve' }] }, + ], + diagnostics: { customServices: [] }, + }; + const readSelector = async () => { + const value = await (await fetch(`http://127.0.0.1:${apiPort}/proxies/channel-selector`)).json(); + return { role: value.now === 'channel-primary' ? 'primary' : value.now === 'channel-reserve' ? 'reserve' : 'other' }; + }; + const selectRole = async (role) => { + const response = await fetch(`http://127.0.0.1:${apiPort}/proxies/channel-selector`, { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ name: role === 'primary' ? 'channel-primary' : 'channel-reserve' }), + }); + assert.equal(response.status, 204); + const selected = await readSelector(); + assert.equal(selected.role, role); + return selected; + }; + const traffic = createDomainTrafficService({ + observe: async () => (await fetch(`http://127.0.0.1:${apiPort}/connections`)).json(), + devices: () => [], + now: () => new Date(clock), + }); + await traffic.refresh(); + const failover = createFailoverService({ + state: { + read: () => state, + update: (mutator) => { + state = { ...mutator(state), revision: state.revision + 1 }; + return state; + }, + }, + runtime: { isRunning: async () => true }, + dataplane: { + checkConfig: async () => ({}), + runFailoverProbe: async (role) => ({ vpn: { sites: [{ status: role === 'primary' ? 'unavailable' : 'available' }] } }), + readFailoverSelector: readSelector, + selectFailoverRole: selectRole, + setFailoverActivityEnabled: async (value) => value ? traffic.enableActivity() : traffic.disableActivity(), + readFailoverActivity: async (threshold) => { + await traffic.refresh(); + return { activity: traffic.activitySnapshot(threshold) }; + }, + }, + buildCandidate: () => ({ config: {}, applied }), + serialize: (operation) => operation(), + scheduler: { setTimeout: () => ({ unref() {} }), clearTimeout: () => {} }, + now: () => new Date(clock), + }); + t.after(() => failover.shutdown()); + await failover.reconcile(); + for (clock of [0, 30_000, 35_000]) { + await echo(existing, `active-${clock}-${'x'.repeat(64 * 1024)}`); + await failover.runRound(); + } + assert.equal((await readSelector()).role, 'primary'); + assert.equal(failover.snapshot().status, 'waiting-for-idle'); + t.diagnostic('active traffic delayed selector switch'); + + clock = 46_000; + await failover.runRound(); + assert.equal((await readSelector()).role, 'primary'); + clock = 51_000; + await failover.runRound(); + const after = await (await fetch(`http://127.0.0.1:${apiPort}/proxies/channel-selector`)).json(); + assert.equal(after.now, 'channel-reserve'); + t.diagnostic('quiet window elapsed; selector switched and read back'); + + await echo(existing, 'after-switch-existing'); + await existingUdp.echo('after-switch-existing-udp'); + const createdAfterSwitch = await openSocksConnection(proxyPort, echoPort, (message) => t.diagnostic(message)); + t.diagnostic('reserve connection open'); + t.after(() => createdAfterSwitch.destroy()); + await echo(createdAfterSwitch, 'after-switch-new'); + const udpCreatedAfterSwitch = await openSocksUdpAssociation(proxyPort, udpEchoPort, resolveUdpRelayPort); + t.after(() => udpCreatedAfterSwitch.close()); + await udpCreatedAfterSwitch.echo('after-switch-new-udp'); + + const connections = await (await fetch(`http://127.0.0.1:${apiPort}/connections`)).json(); + const chains = connections.connections.map((connection) => connection.chains); + assert.ok(chains.some((chain) => chain.includes('channel-primary')), JSON.stringify(chains)); + assert.ok(chains.some((chain) => chain.includes('channel-reserve')), JSON.stringify(chains)); + assert.equal(runtime.pid, runtimePid); + assert.equal(runtime.exitCode, null); + } catch (cause) { + assert.fail(`${cause.stack || cause}\n${runtimeOutput}`); + } +}); diff --git a/test/server/state-contract.test.js b/test/server/state-contract.test.js index b4c3ae2..572cd93 100644 --- a/test/server/state-contract.test.js +++ b/test/server/state-contract.test.js @@ -320,6 +320,7 @@ setInterval(() => {}, 60_000); 'configExists', 'connection', 'diagnostics', + 'failover', 'fetchedAt', 'gatewayAuto', 'generatedAt', diff --git a/test/server/state-store.test.js b/test/server/state-store.test.js index 97b395a..defef48 100644 --- a/test/server/state-store.test.js +++ b/test/server/state-store.test.js @@ -115,6 +115,39 @@ test('schema v5 migrates rules and diagnostics settings with an exact backup', ( assert.equal(fs.readFileSync(store.migration.backupPath, 'utf8'), bytes); }); +test('schema v7 migrates failover disabled without losing canonical state', (t) => { + const filePath = fixture(t); + const legacy = { + schemaVersion: 7, + revision: 19, + profiles: [], + desiredProfileId: '', + appliedProfileId: '', + appliedServerId: '', + appliedServerSnapshot: null, + routeRules: [], + appliedRouteRules: [], + routeRulesRevision: 4, + diagnostics: { configured: true, customServices: [], hiddenServiceIds: ['google'] }, + }; + fs.writeFileSync(filePath, JSON.stringify(legacy)); + + const store = createStateStore(filePath, { + now: () => new Date('2026-08-19T12:00:00.000Z'), + }); + const migrated = store.read(); + + assert.equal(migrated.schemaVersion, 8); + assert.equal(migrated.revision, 19); + assert.equal(migrated.routeRulesRevision, 4); + assert.equal(migrated.failoverPolicy.enabled, false); + assert.equal(migrated.failoverRuntimeState.lastSwitchAt, null); + assert.equal(migrated.appliedFailoverPolicy, null); + assert.deepEqual(migrated.diagnostics.hiddenServiceIds, ['google']); + assert.equal(store.migration.fromVersion, 7); + assert.deepEqual(JSON.parse(fs.readFileSync(store.migration.backupPath, 'utf8')), legacy); +}); + test('schema v6 rejects missing or unknown outbound without rewriting source bytes', (t) => { for (const [name, rule] of [ ['missing', { type: 'domain', value: 'example.com', enabled: true }], diff --git a/test/server/subscription-mutation.test.js b/test/server/subscription-mutation.test.js index 20c0633..8bdebf2 100644 --- a/test/server/subscription-mutation.test.js +++ b/test/server/subscription-mutation.test.js @@ -124,6 +124,8 @@ function createHarness(overrides = {}) { clearInterval: (timer) => { timer.clearCalls += 1; }, }, onRefreshError: overrides.onRefreshError || (() => {}), + onEvent: overrides.onEvent, + failover: overrides.failover, now: () => new Date('2026-08-08T12:30:00.000Z'), }); @@ -305,6 +307,36 @@ test('inactive delete is state-only; applied delete requires one stop-and-delete assert.equal(after.config, null); }); +test('failed active delete restores the previous reserve selector', async () => { + const state = defaultState(); + state.failoverPolicy = { + enabled: true, + paused: false, + primary: { profileId: 'work', serverId: 'shared' }, + reserve: { profileId: 'personal', serverId: 'shared' }, + }; + state.appliedFailoverPolicy = { + primary: state.failoverPolicy.primary, + reserve: state.failoverPolicy.reserve, + }; + const harness = createHarness({ + state, + failStateAt: 1, + failover: { + reconcile: async () => {}, + restoreAppliedActivation: async (previousState) => { + assert.equal(previousState.appliedProfileId, 'personal'); + harness.calls.push('selector.reserve'); + }, + }, + }); + await assert.rejects(harness.service.deleteProfile('personal', 'stop-and-delete'), /state failed/); + assert.equal(harness.snapshot().running, true); + assert.equal(harness.snapshot().config, 'old-config'); + assert.equal(harness.snapshot().state.appliedProfileId, 'personal'); + assert.ok(harness.calls.indexOf('selector.reserve') > harness.calls.indexOf('runtime.start')); +}); + test('deleting a pending desired profile leaves the running applied route untouched', async () => { const state = defaultState(); state.desiredProfileId = 'work'; @@ -317,6 +349,90 @@ test('deleting a pending desired profile leaves the running applied route untouc assert.equal(harness.calls.includes('gateway.set'), false); }); +test('deleting a desired failover target disables only the pending policy', async () => { + const state = defaultState(); + state.failoverPolicy = { + enabled: true, + paused: false, + primary: { profileId: 'work', serverId: 'shared' }, + reserve: { profileId: 'personal', serverId: 'shared' }, + }; + const harness = createHarness({ state }); + await harness.service.deleteProfile('work', 'delete', 3); + const after = harness.snapshot(); + assert.equal(after.running, true); + assert.equal(after.config, 'old-config'); + assert.equal(after.state.failoverPolicy.enabled, false); + assert.deepEqual(after.state.failoverPolicy.primary, { profileId: '', serverId: '' }); + assert.deepEqual(after.state.failoverPolicy.reserve, { profileId: 'personal', serverId: 'shared' }); +}); + +test('refresh pauses failover when a loaded channel target disappears', async () => { + const state = defaultState(); + state.failoverPolicy = { + enabled: true, + paused: false, + primary: { profileId: 'personal', serverId: 'shared' }, + reserve: { profileId: 'work', serverId: 'shared' }, + }; + state.appliedFailoverPolicy = { + primary: state.failoverPolicy.primary, + reserve: state.failoverPolicy.reserve, + primaryConfigFingerprint: 'a'.repeat(64), + reserveConfigFingerprint: 'b'.repeat(64), + }; + const events = []; + const harness = createHarness({ state, onEvent: (event) => events.push(event) }); + await harness.service.refreshProfile('personal'); + assert.equal(harness.snapshot().state.failoverPolicy.paused, true); + assert.equal(harness.snapshot().running, true); + assert.equal(harness.snapshot().config, 'old-config'); + assert.ok(events.some(({ type }) => type === 'failover.paused')); +}); + +test('scheduled refresh failures use one stable key for the same failure streak', async () => { + const failure = Object.assign(new Error('down'), { code: 'PROVIDER_UNAVAILABLE' }); + const events = []; + const harness = createHarness({ + fetchSubscription: async () => { throw failure; }, + onEvent: (event) => events.push(event), + }); + await assert.rejects(harness.service.refreshProfile('personal', undefined, 'scheduled')); + await assert.rejects(harness.service.refreshProfile('personal', undefined, 'scheduled')); + const keys = events.filter(({ type }) => type === 'subscription.refresh_failed').map(({ dedupeKey }) => dedupeKey); + assert.equal(keys.length, 2); + assert.equal(keys[0], keys[1]); +}); + +test('scheduled success logs only content changes or recovery while manual refresh always logs', async () => { + const unchanged = (id) => ({ + config: { profile: id }, + servers: [oldServer], + userInfo: { total: 100 }, + fetchedAt: '2026-08-08T12:00:00.000Z', + }); + const events = []; + const harness = createHarness({ + fetchSubscription: async () => unchanged('work'), + onEvent: (event) => events.push(event), + }); + await harness.service.refreshProfile('work', undefined, 'scheduled'); + assert.equal(events.length, 0); + await harness.service.refreshProfile('work'); + assert.deepEqual(events.map(({ type }) => type), ['subscription.refreshed']); + + const recoveryState = defaultState(); + recoveryState.profiles[1].lastRefreshErrorCode = 'PROVIDER_UNAVAILABLE'; + const recoveryEvents = []; + const recovery = createHarness({ + state: recoveryState, + fetchSubscription: async () => unchanged('work'), + onEvent: (event) => recoveryEvents.push(event), + }); + await recovery.service.refreshProfile('work', undefined, 'scheduled'); + assert.deepEqual(recoveryEvents.map(({ type }) => type), ['subscription.refreshed']); +}); + test('auto refresh iterates profiles once, reports scoped failures, and stops idempotently', async () => { const errors = []; const failure = Object.assign(new Error('down'), { code: 'PROVIDER_UNAVAILABLE' }); diff --git a/test/web/activity-journal-feature-contract.test.js b/test/web/activity-journal-feature-contract.test.js new file mode 100644 index 0000000..b38d76c --- /dev/null +++ b/test/web/activity-journal-feature-contract.test.js @@ -0,0 +1,27 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import path from 'node:path'; +import test from 'node:test'; + +const root = path.resolve(import.meta.dirname, '../..'); +const page = fs.readFileSync(path.join(root, 'src/web/components/ClientOverviewPage.tsx'), 'utf8'); +const feature = fs.readFileSync(path.join(root, 'src/web/features/activity-journal/ActivityJournalFeature.tsx'), 'utf8'); +const styles = fs.readFileSync(path.join(root, 'src/web/styles/features/activity-journal.css'), 'utf8'); + +test('journal is the last Gateway-only drawer and refreshes whenever it opens', () => { + assert.match(page, / { + assert.match(feature, /Важные события хранятся 30 дней/); + assert.match(feature, /Сегодня[\s\S]*Вчера/); + assert.match(feature, /Обновить журнал/); + assert.match(feature, /M5 8V4m0 4h4/); + assert.match(feature, /Показать ещё/); + assert.match(feature, /Журнал временно недоступен[\s\S]*Повторить/); + assert.match(styles, /\.client-journal-skeleton/); + assert.match(styles, /@media \(prefers-reduced-motion: reduce\)/); +}); diff --git a/test/web/connection-panel-contract.test.js b/test/web/connection-panel-contract.test.js index 1f2fb42..742f437 100644 --- a/test/web/connection-panel-contract.test.js +++ b/test/web/connection-panel-contract.test.js @@ -34,7 +34,7 @@ test('connection feature preserves actions, local preference and opaque neighbor assert.match(panel, /client-state-detail[\s\S]*\{routingSlot\}[\s\S]*client-state-copy[\s\S]*\{serverSlot\}[\s\S]*client-proxies[\s\S]*\{statusSlot\}/); assert.match(page, /routingSlot=\{\}/); + assert.match(page, /serverSlot=\{\}/); assert.match(page, /mainIdentity[\s\S]*appliedProfile[\s\S]*appliedServer/); assert.match(page, /statusSlot=\{<>[\s\S]*InlineError[\s\S]*InlineProgress/); }); diff --git a/test/web/device-inventory-contract.test.js b/test/web/device-inventory-contract.test.js index afc60d5..8a5315b 100644 --- a/test/web/device-inventory-contract.test.js +++ b/test/web/device-inventory-contract.test.js @@ -233,7 +233,7 @@ test('Gateway Home reuses the canonical device snapshot for applied route and gl assert.match(overview, /const appliedServer = appliedProfile\?\.servers\.find[\s\S]*state\.selection\.appliedServerSnapshot/); assert.match(overview, /const mainIdentity = gatewayDirect[\s\S]*: connected[\s\S]*appliedProfile && appliedServer[\s\S]*`\$\{subscriptionDomain\(appliedProfile\.subscription\.host\)\} · \$\{appliedServer\.label\}`/); assert.match(overview, /const switchIdentity = gatewayDirect[\s\S]*switchingServer && operationProfile && operationServer[\s\S]*`Переключаем на \$\{subscriptionDomain\(operationProfile\.subscription\.host\)\} · \$\{operationServer\.label\}`/); - assert.match(overview, /serverSlot=\{\}/); + assert.match(overview, /serverSlot=\{\}/); assert.match(feature, /formatByteString\(globalTraffic\?\.totalBytes \|\| '0'\)/); assert.match(feature, /const history = globalTraffic\?\.history \|\| \[\][\s\S]*samples=\{history\}[\s\S]*routeLabel="Gateway"[\s\S]*series="speed"/); assert.match(connection, /
)/); }); -test('serial probes and editor stay panel-owned while backend state owns the service set', () => { +test('serial probes stay panel-owned while custom service validation is shared by both editors', () => { assert.match(panel, /const targets = onlyTarget \? \[onlyTarget\] : \[[\s\S]*CONNECTIVITY_NETWORK_SOURCE\.id[\s\S]*CONNECTIVITY_IP_SOURCES\.map[\s\S]*sites\.map/); assert.match(panel, /for \(const target of targets\) \{[\s\S]*setActiveTarget\(target\)[\s\S]*await runConnectivityDiagnostics\(target\)[\s\S]*if \(legacyFullResult\) break/); assert.match(panel, /CUSTOM_SERVICES_KEY = 'harbor-diagnostic-services'/); @@ -92,7 +93,8 @@ test('serial probes and editor stay panel-owned while backend state owns the ser assert.doesNotMatch(panel, /localStorage\.setItem/); assert.doesNotMatch(panel, /useState\(read(?:Custom|Hidden)Services\)/); assert.match(panel, /slice\(0, MAX_CUSTOM_DIAGNOSTIC_SERVICES\)/); - assert.match(panel, /parsed\.protocol !== 'https:'/); + assert.match(panel, /saveCustomDiagnosticService/); + assert.match(customServiceAction, /parsed\.protocol !== 'https:'/); assert.match(panel, /document\.startViewTransition\(update\)/); assert.match(panel, /retryable: Boolean\(Reflect\.get\(value, 'retryable'\)\)/); assert.match(panel, /requestDetails\(error\)[\s\S]*requestError\.retryable/); diff --git a/test/web/failover-feature-contract.test.js b/test/web/failover-feature-contract.test.js new file mode 100644 index 0000000..4523374 --- /dev/null +++ b/test/web/failover-feature-contract.test.js @@ -0,0 +1,41 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import path from 'node:path'; +import test from 'node:test'; + +const root = path.resolve(import.meta.dirname, '../..'); +const page = fs.readFileSync(path.join(root, 'src/web/components/ClientOverviewPage.tsx'), 'utf8'); +const feature = fs.readFileSync(path.join(root, 'src/web/features/failover/FailoverFeature.tsx'), 'utf8'); +const styles = fs.readFileSync(path.join(root, 'src/web/styles/features/failover.css'), 'utf8'); + +test('reserve is a Gateway-only drawer immediately after subscriptions', () => { + assert.match(page, / { + assert.match(feature, /Использовать резерв[\s\S]*Полностью пассивен/); + assert.match(feature, /\['primary', 'reserve'\][\s\S]*Подписка[\s\S]*Сервер/); + assert.match(feature, /Что проверять/); + assert.match(feature, /min="2"[\s\S]*max="30"[\s\S]*Таймаут проверки:/); + assert.match(feature, /Проверять каждые, сек[\s\S]*Сбой должен длиться, сек[\s\S]*Восстановление, сек/); + assert.match(feature, /Не переключать во время работы[\s\S]*Тишина перед переключением[\s\S]*Активный трафик, КБ\/с/); + assert.match(feature, /Защита от повторных сбоев[\s\S]*Окно повторных сбоев[\s\S]*Карантин основного/); + assert.match(feature, /Добавить HTTPS-сервис/); + assert.match(feature, /Проверить оба канала/); + assert.match(feature, /beforeunload/); +}); + +test('runtime status keeps fixed geometry and explains active traffic without motion dependence', () => { + assert.match(feature, /Ждём завершения активной работы/); + assert.match(feature, /Активный трафик ·[\s\S]*transmittingConnections/); + assert.match(feature, /activity\?\.blockers\.slice\(0, 2\)\.map/); + assert.match(feature, /Ещё \{activity\.blockers\.length - 2\}/); + assert.match(feature, /Нестабилен ·[\s\S]*Следующее решение не раньше чем через/); + assert.match(feature, /switchRole && snapshot\.enabled && snapshot\.activation === 'active'/); + assert.match(styles, /\.client-failover-runtime \{[^}]*min-height:\s*132px/); + assert.match(styles, /@media \(prefers-reduced-motion: reduce\)[\s\S]*transition:\s*none/); +}); diff --git a/test/web/harbor-state.test.js b/test/web/harbor-state.test.js index 94c62de..948aade 100644 --- a/test/web/harbor-state.test.js +++ b/test/web/harbor-state.test.js @@ -56,6 +56,10 @@ test('typed Harbor client validates unknown state and isolates wire compatibilit gatewayAvailable: true, }); assert.equal(Object.hasOwn(parsed, 'proxyPort'), false); + assert.throws(() => parseHarborState({ + ...snapshot, + failover: { ...snapshot.failover, policy: { ...snapshot.failover.policy, intervalMs: 'fast' } }, + }), { code: 'INCOMPATIBLE_API' }); let incompatible; try { parseHarborState({ ...snapshot, revision: -1 }); @@ -116,6 +120,21 @@ test('an equal revision keeps the current snapshot identity', () => { assert.equal(next.snapshot.selection.desiredServerId, 'one'); }); +test('equal revision accepts only newer failover observations and retires old epochs', () => { + const value = (epoch, sequence, status) => ({ + ...snapshot(4, 'one'), + failover: { observationEpoch: epoch, observationSequence: sequence, status }, + }); + let state = receive(initialHarborState, value('epoch-a', 1, 'primary')); + state = receive(state, value('epoch-a', 2, 'waiting-for-idle')); + assert.equal(state.snapshot.failover.status, 'waiting-for-idle'); + state = receive(state, value('epoch-b', 1, 'reserve')); + assert.equal(state.snapshot.failover.status, 'reserve'); + state = receive(state, value('epoch-a', 99, 'primary')); + assert.equal(state.snapshot.failover.status, 'reserve'); + assert.deepEqual(state.failoverTransport.retiredEpochs, ['epoch-a']); +}); + test('selection has no client-side shadow and follows canonical profile snapshots', () => { const state = receive(initialHarborState, snapshot(2, 'two')); assert.equal(Object.hasOwn(state, 'pendingServerId'), false); diff --git a/test/web/responsive-layout-contract.test.js b/test/web/responsive-layout-contract.test.js index b177481..fd57892 100644 --- a/test/web/responsive-layout-contract.test.js +++ b/test/web/responsive-layout-contract.test.js @@ -166,13 +166,13 @@ test('secondary menus share one right rail and switch equal drawers as a vertica assert.match(instructions, /\(null\)/); assert.match(component, /const activeRailDrawer = drawerSwitchTarget && drawerControls\[drawerSwitchTarget\]\.isOpen[\s\S]*drawerControls\[drawer\]\.isOpen/); assert.match(component, /setDrawerSwitchTarget\(target\);[\s\S]*flushSync\(\(\) => toControl\.show\(\)\)/); assert.match(component, /fromControl\.close\(\);[\s\S]*setDrawerSwitchTarget\(null\)/); - assert.match(component, / \{[\s\S]*from\.inert = false;[\s\S]*from\.removeAttribute\('aria-hidden'\)/); assert.match(component, /DRAWER_SWITCH_MS = 620[\s\S]*prefers-reduced-motion: reduce[\s\S]*flushSync/); assert.match(component, /current === 'routing' && routingFeature\.dirty[\s\S]*routingFeature\.requestClose\(\)/); diff --git a/test/web/style-boundaries.test.js b/test/web/style-boundaries.test.js index 8eabd4d..59d8614 100644 --- a/test/web/style-boundaries.test.js +++ b/test/web/style-boundaries.test.js @@ -31,36 +31,38 @@ const expectedImports = [ './features/servers.css', './primitives.css', './features/diagnostics.css', + './features/failover.css', + './features/activity-journal.css', './layout.css', './themes.css', ]; const sha256 = (value) => crypto.createHash('sha256').update(value).digest('hex'); const acceptedLedger = { counts: { - cascadeEdges: 954, + cascadeEdges: 984, customProperties: 106, - declarations: 3487, + declarations: 3805, important: 0, keyframes: 48, - media: 13, - rules: 971, - variableReferences: 837, + media: 17, + rules: 1045, + variableReferences: 977, }, hashes: { - cascadeEdges: 'd53f6a2236717d6bc27d486fc19f0f80a33169fa7f68a6df2a19536fd30be3bb', + cascadeEdges: '423d29f846b45c3522fdc101b5aa1399264f2abe5953b9687b380d5fb865bd07', customProperties: 'fd6f16069f9fe3526f09d4d574431ddae67150d8e7a8a40e04c9fd2d179ec7ac', - declarations: 'e1401e80ed2ebc5d3601d4d61a3b44c2adbf91db451907ef95a0998170ac5de5', + declarations: '52b5bcd795dbea19950afee0259c62cb0fb1aaca094d553cfaaae046a5f747f9', duplicateKeyframes: '4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945', duplicateSelectors: '8982145dba05b33bf1c93b2d54cfbe76305b276ff3c657aa020144016ada5848', keyframes: 'a0d69c5b3e5f235d3a76ed04bdd64f67a73fafc9c59fe2635ffda603709c0ad9', - ruleDeclarationSequences: '89f5f3b302f3a10df0ba90dfbbc63a7ccbc1d8a0c784be26a475321f44dfa8e8', - selectors: 'dc1a2687ec872d1922a39e8258977949a9b775168764a59119ae6ea93a9f88a1', - variableReferences: '843a5f71b0fb74691ed623dbc7ebb7ddddc34e5736e1f419f17cf54e22d43073', - witnesses: '81676a3fdb6f3bbd63e86b275b6cb6d40e930651b9d5cbde208408f3163ab2b4', + ruleDeclarationSequences: 'bd8d10966c15df58eb76d67fc5a21acaeec524a64805797d5fb33af3b942292e', + selectors: '76d2d373aa9ecfd5cac3f95a47ed0fa2c44a659ac98f02f6bfd2f4505de23226', + variableReferences: 'bea2b082cf2366041936c2da22ac3e0ab83b3e5cead1326a0f65a015daedbb3b', + witnesses: '605555d8bae9b8d64b1c960c398b14cf910e624d547b72e4fcfced85eec30991', }, }; -test('public stylesheet exposes exactly twelve flat semantic owners', () => { +test('public stylesheet exposes exactly fourteen flat semantic owners', () => { const imports = Array.from(index.matchAll(/^@import ['"](\.\/[^'"]+\.css)['"];$/gm), ([, file]) => file); assert.deepEqual(imports, expectedImports); assert.equal(index, `${expectedImports.map((file) => `@import '${file}';`).join('\n')}\n`); @@ -209,7 +211,7 @@ test('client typography uses the shared semantic scale outside the token owner', test('accepted stylesheet has pinned declaration, selector, keyframe, variable, and cascade ledgers', () => { const witnesses = readStyleWitnesses(root); - assert.equal(witnesses.length, 884); + assert.equal(witnesses.length, 1021); assert.equal(witnesses.filter((witness) => witness.unknown || witness.ancestorUnknown).length, 0); const ledger = createStyleLedger(readStyleSource(root), { witnesses }); assert.deepEqual(ledger.counts, acceptedLedger.counts); @@ -405,8 +407,8 @@ test('main owns one public stylesheet and the regrouped production CSS is determ assert.equal((main.match(/import ['"][^'"]+\.css['"]/g) || []).length, 1); const assets = fs.readdirSync(path.join(root, 'dist/assets')).filter((file) => file.endsWith('.css')); - assert.deepEqual(assets, ['index-CdbGat4L.css']); + assert.deepEqual(assets, ['index-DaioW5qf.css']); const built = fs.readFileSync(path.join(root, 'dist/assets', assets[0])); - assert.equal(built.byteLength, 133670); - assert.equal(sha256(built), '1aaa222692046eb166e5bfb8a195e53cf0ef61692a82ae53ca0119b4c8f878a2'); + assert.equal(built.byteLength, 144728); + assert.equal(sha256(built), '7d175321821b38be6a947a76457e310cf519e2af4b9ee95fbb0097933ca70a28'); }); diff --git a/test/web/ui-foundations-contract.test.js b/test/web/ui-foundations-contract.test.js index 041f11d..7c19fe6 100644 --- a/test/web/ui-foundations-contract.test.js +++ b/test/web/ui-foundations-contract.test.js @@ -34,7 +34,7 @@ test('all repeated client primitive consumers use the shared owners', () => { assert.equal((production.match(/