Update Harbor client and gateway integration workflows
This commit is contained in:
@@ -75,6 +75,14 @@ http://АДРЕС-GATEWAY:3456
|
||||
|
||||
Приватные и локальные адреса не отправляются в VPN, поэтому устройства сохраняют доступ к домашней сети. Общий прокси по умолчанию принимает подключения только из приватных сетей.
|
||||
|
||||
### Резервный канал Gateway
|
||||
|
||||
После добавления подписок откройте «Резерв» — вторую кнопку в правой панели. Выберите основной и резервный серверы (они могут быть из одной или разных подписок), сервисы для проверки и отдельный таймаут каждого сервиса. Там же настраиваются длительность сбоя и восстановления, порог активного трафика, период тишины и защита от частых переключений.
|
||||
|
||||
При включении Harbor заранее проверяет dual-конфигурацию. Если VPN остановлен, она начнёт работать только после следующего обычного нажатия питания; сохранение само VPN не включает. Переключение меняет маршрут только для новых соединений — уже открытые соединения не закрываются. Если через VPN идёт активный трафик или его активность нельзя надёжно определить, Harbor ждёт и показывает скорость, число передающих соединений и безопасные подписи основных блокирующих потоков.
|
||||
|
||||
Выключенный резерв полностью пассивен: Harbor не запускает проверки, таймер выбора и отдельный подсчёт активности. Если dual-конфигурация уже загружена, отключение не перезапускает VPN и не меняет текущий маршрут; обычный stop и следующий запуск вернут single-channel config. Последние важные события — включение VPN, обновления подписок, переключения и ошибки — доступны в последней кнопке «Журнал» и хранятся 30 дней без ссылок подписок и сырых диагностических ответов.
|
||||
|
||||
### Устройства Gateway
|
||||
|
||||
Откройте «Устройства» в правой панели Gateway — подписка для просмотра списка не требуется. Harbor раз в 15 секунд читает локальную таблицу соседей и показывает каждое устройство одной компактной строкой: заданное название, hostname или IP, последний контакт, выбранный график трафика и иконку применённого маршрута. По умолчанию график показывает приблизительный выход `VPN`/`Direct`; переключатель `Вход` возвращает накопленную разбивку `Gateway`/`Прокси`. Наведите курсор на имя или переведите на него фокус, чтобы открыть IP, MAC и доступный hostname; нажатие на значение копирует его. Hostname определяется через локальное обратное разрешение имён и может отсутствовать, если сеть его не публикует. Технические interface и manufacturer продолжают храниться для идентификации, но не занимают место в строке. Список разделён на «Закреплённые», «Остальные» и «Фоновые»: последняя группа сохраняется между перезапусками, показывает только identity/presence и кнопку возврата без графика, traffic и route controls. Название, закрепление, фоновое положение и накопленные totals сохраняются в volume Gateway, пока устройство остаётся в inventory.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Harbor application state v1
|
||||
|
||||
`GET /api/state` is the canonical Harbor domain snapshot. Successful mutations return the same snapshot as `state`. The persisted owner is `state.json` schema v6; React keeps only drafts, disclosure, focus, animation and transport freshness.
|
||||
`GET /api/state` is the canonical Harbor domain snapshot. Successful mutations return the same snapshot as `state`. The persisted owner is `state.json` schema v8; React keeps only drafts, disclosure, focus, animation and transport freshness.
|
||||
|
||||
An abbreviated snapshot:
|
||||
|
||||
@@ -67,6 +67,8 @@ The frontend accepts only newer snapshots. Equal revisions preserve object ident
|
||||
|
||||
Browser boot/offline/stale state remains a transport envelope beside the domain snapshot. A transport failure retains the last accepted domain state.
|
||||
|
||||
Failover health and traffic observations are transient: they do not write `state.json` or increase the domain `revision` every few seconds. Each control-process lifetime publishes a new `observationEpoch` and increasing `observationSequence`. At the same domain revision the browser accepts only a newer sequence from the active epoch; after accepting a new epoch it retires the old one so a late response cannot restore stale health.
|
||||
|
||||
## Desired and applied identity
|
||||
|
||||
`desiredProfileId` and each profile's `desiredServerId` record the next local choice. `appliedProfileId`, `appliedServerId` and `appliedServerSnapshot` describe the runtime that actually owns traffic. There is no third `activeProfileId`.
|
||||
@@ -97,12 +99,22 @@ The route-rules mutation uses the whole-array `PUT /api/route-rules/v2` with `ru
|
||||
|
||||
In Connect `gateway-direct`, local user rules are intentionally omitted and the snapshot reports no active or pending local rules. Gateway device policy `Напрямую` bypasses sing-box before these rules; policy `VPN` and an ordinary local/Gateway VPN pipeline evaluate them.
|
||||
|
||||
## Gateway failover and activity journal
|
||||
|
||||
`failoverPolicy` is the desired Gateway-only policy: master enable, primary/reserve profile and server, service checks with individual timeouts, health windows, active-traffic guard and flap protection. `failoverRuntimeState` stores switch history, hold and quarantine deadlines separately, so a runtime decision is not mistaken for a desired configuration change. `appliedFailoverPolicy` stores only the two loaded targets and safe configuration fingerprints.
|
||||
|
||||
Enabling failover while VPN is stopped validates a temporary dual-channel candidate but does not start VPN. The dual config is loaded only by the next explicit power-on. Enabling it over a running single-channel config remains pending until a later stop and power-on. Disabling automation stops its timer, probes and activity collector immediately, but does not restart sing-box or change the selected route; the already loaded dual config is reported as `passive-loaded` until the ordinary stop lifecycle clears it.
|
||||
|
||||
The dual config keeps one stable inbound and a sing-box selector with `interrupt_exist_connections: false`. A switch changes the outbound for new connections only. Before an automatic switch, the existing `/connections` observer measures VPN byte deltas over a bounded 10-second window. Active or unknown traffic blocks the switch; the public snapshot contains only aggregate speed, connection count and at most three safe device/service labels.
|
||||
|
||||
Failover mutations use `PUT /api/failover`, `POST /api/failover/pause` and `POST /api/failover/switch`. Important user events are stored separately in `activity-journal.json` and read through `GET /api/activity-journal`. The journal is not a second state owner, contains no provider URLs or raw diagnostics, uses stable ID cursors and prunes entries after 30 days.
|
||||
|
||||
## Compatibility and migration
|
||||
|
||||
Schema v5 migrates the legacy singleton and `subscription-cache.json` into one profile named `Основной`. Stable endpoint identity preserves unambiguous desired/applied selection, including transport variants whose normalized IDs differ from old labels. An explicitly stopped legacy state does not resurrect an old applied target.
|
||||
|
||||
Schema v6 adds the routing-rule outbound. Rules read from schemas v0-v5 migrate to `outbound: "direct"` in their existing order and both desired/applied arrays are normalized together. A schema-v6 rule without a valid outbound is rejected rather than silently rewritten.
|
||||
Schema v6 adds the routing-rule outbound. Rules read from schemas v0-v5 migrate to `outbound: "direct"` in their existing order and both desired/applied arrays are normalized together. A schema-v6 rule without a valid outbound is rejected rather than silently rewritten. Schema v7 adds canonical connectivity-diagnostics settings. Schema v8 adds a disabled failover policy, empty runtime history and no applied dual config, so upgrading does not start monitoring or change traffic.
|
||||
|
||||
Migration atomically backs up the previous `state.json`. After the embedded profile is committed, Harbor also backs up and removes the legacy subscription cache so there is one persisted owner. Invalid legacy cache/config returns to a truthful stopped first-run state instead of starting stale generated config.
|
||||
|
||||
The old HTTP projection remains bounded for one release. Schema v6 persistence is not downgrade-compatible: stop Harbor and restore the `state.json.backup-v<fromVersion>-*` matching the rollback binary instead of deploying old code over v6 data. Rolling back before profiles still also requires the matching legacy subscription-cache backup.
|
||||
The old HTTP projection remains bounded for one release. Schema v8 persistence is not downgrade-compatible: stop Harbor and restore the `state.json.backup-v<fromVersion>-*` matching the rollback binary instead of deploying old code over v8 data. Rolling back before profiles still also requires the matching legacy subscription-cache backup.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Harbor state recovery
|
||||
|
||||
Harbor keeps the existing data directory and `state.json` path. The current persisted format is `schemaVersion: 6`: schema v2 introduced local route rules, v3 added rule enabled state, v4 added stable server IDs, v5 embeds the canonical `profiles[]` collection with desired/applied profile identity, and v6 adds an explicit `vpn` or `direct` outbound to every route rule.
|
||||
Harbor keeps the existing data directory and `state.json` path. The current persisted format is `schemaVersion: 8`: schema v2 introduced local route rules, v3 added rule enabled state, v4 added stable server IDs, v5 embeds the canonical `profiles[]` collection with desired/applied profile identity, v6 adds an explicit `vpn` or `direct` outbound to every route rule, v7 stores connectivity-diagnostics settings, and v8 adds Gateway failover state.
|
||||
|
||||
## Atomic writes
|
||||
|
||||
@@ -36,6 +36,12 @@ state.json.backup-v5-2026-08-17T12-00-00-000Z
|
||||
|
||||
After migration, malformed schema-v6 rules are rejected; Harbor does not reinterpret a missing or unknown outbound as direct.
|
||||
|
||||
## Migration to failover
|
||||
|
||||
Schemas v0-v7 migrate to v8 with failover disabled, empty switch history and no applied dual config. Migration does not start probes, enable traffic accounting or change the single-channel runtime. The original state is preserved as `state.json.backup-v<fromVersion>-*` before the atomic replacement.
|
||||
|
||||
The separate `activity-journal.json` is created on the first important event. It uses the same atomic write and corrupt-file isolation mechanism as state, retains at most 30 days, and can be removed while Harbor is stopped without affecting subscriptions, routing or VPN startup.
|
||||
|
||||
## Corrupt JSON
|
||||
|
||||
If `state.json` cannot be parsed, Harbor renames the exact damaged bytes to:
|
||||
@@ -55,4 +61,4 @@ Perform recovery while Harbor is stopped:
|
||||
3. Restore only matching state/cache backups to their original filenames.
|
||||
4. Start Harbor and verify `GET /api/state` before applying a profile.
|
||||
|
||||
A pre-v6 binary cannot interpret the explicit ordered VPN/Direct rule contract. Restore `state.json.backup-v<fromVersion>-*` matching the rollback binary; deploying old code over schema v6 is not safe. A rollback to pre-v5 additionally requires the matching state and subscription-cache backups because that binary cannot interpret canonical profiles.
|
||||
A pre-v8 binary cannot interpret failover state. Restore `state.json.backup-v<fromVersion>-*` matching the rollback binary; deploying old code over schema v8 is not safe. A rollback to pre-v5 additionally requires the matching state and subscription-cache backups because that binary cannot interpret canonical profiles.
|
||||
|
||||
@@ -15,6 +15,8 @@ export const settings = {
|
||||
port: parsePort(process.env.PORT, 3456),
|
||||
proxyPort,
|
||||
diagnosticsProxyPort: parsePort(process.env.DIAGNOSTICS_PROXY_PORT, 18080),
|
||||
failoverPrimaryProxyPort: parsePort(process.env.FAILOVER_PRIMARY_PROXY_PORT, 18081),
|
||||
failoverReserveProxyPort: parsePort(process.env.FAILOVER_RESERVE_PROXY_PORT, 18082),
|
||||
singboxApiPort: parsePort(process.env.SING_BOX_API_PORT, 19090),
|
||||
tproxyPort: parsePort(process.env.TPROXY_PORT, 7895),
|
||||
tproxyMark: process.env.TPROXY_MARK || "1",
|
||||
@@ -40,6 +42,7 @@ export const settings = {
|
||||
cachePath: process.env.SING_BOX_CACHE || "/var/lib/sing-box/cache.db",
|
||||
statePath: path.join(dataDir, "state.json"),
|
||||
deviceStatePath: path.join(dataDir, "devices.json"),
|
||||
activityJournalPath: path.join(dataDir, "activity-journal.json"),
|
||||
subscriptionCachePath: path.join(dataDir, "subscription-cache.json"),
|
||||
sharedProxyHost: process.env.SHARED_PROXY_HOST || "",
|
||||
hostNetworkStatePath:
|
||||
|
||||
@@ -13,6 +13,7 @@ import {
|
||||
createDomainTrafficService,
|
||||
readSingboxConnections,
|
||||
} from './services/domainTrafficService.js';
|
||||
import { createSingboxSelectorService } from './services/singboxSelectorService.js';
|
||||
|
||||
const socketPath = settings.dataplaneSocket;
|
||||
const runtime = createSingboxRuntime({
|
||||
@@ -40,6 +41,11 @@ const devicePolicy = createDevicePolicyService({
|
||||
const connectivityDiagnostics = createConnectivityDiagnosticsService({
|
||||
proxyPort: settings.diagnosticsProxyPort,
|
||||
});
|
||||
const failoverDiagnostics = {
|
||||
primary: createConnectivityDiagnosticsService({ proxyPort: settings.failoverPrimaryProxyPort }),
|
||||
reserve: createConnectivityDiagnosticsService({ proxyPort: settings.failoverReserveProxyPort }),
|
||||
};
|
||||
const selector = createSingboxSelectorService({ port: settings.singboxApiPort });
|
||||
const domainTraffic = createDomainTrafficService({
|
||||
observe: () => readSingboxConnections(settings.singboxApiPort),
|
||||
devices: () => traffic.snapshot().devices,
|
||||
@@ -126,6 +132,33 @@ const server = http.createServer(async (req: IncomingMessage, res: ServerRespons
|
||||
target,
|
||||
}));
|
||||
}
|
||||
if (req.method === 'POST' && req.url === '/failover/probe') {
|
||||
const { role, services = [], target = null, timeoutMs = 6_000 } = record(await readJson(req));
|
||||
if (role !== 'primary' && role !== 'reserve') throw new Error('Неизвестная failover role');
|
||||
return sendJson(res, 200, await failoverDiagnostics[role].runVpn({ services, target, timeoutMs: Number(timeoutMs) }));
|
||||
}
|
||||
if (req.method === 'GET' && req.url === '/failover/selector') {
|
||||
return sendJson(res, 200, await selector.read());
|
||||
}
|
||||
if (req.method === 'PUT' && req.url === '/failover/selector') {
|
||||
const { role } = record(await readJson(req));
|
||||
if (role !== 'primary' && role !== 'reserve') throw new Error('Неизвестная failover role');
|
||||
return sendJson(res, 200, await selector.select(role));
|
||||
}
|
||||
if (req.method === 'PUT' && req.url === '/failover/activity') {
|
||||
const { enabled } = record(await readJson(req));
|
||||
if (enabled === true) domainTraffic.enableActivity();
|
||||
else domainTraffic.disableActivity();
|
||||
return sendJson(res, 200, { enabled: enabled === true });
|
||||
}
|
||||
if (req.method === 'POST' && req.url === '/failover/activity/read') {
|
||||
const { thresholdBytesPerSecond = 0 } = record(await readJson(req));
|
||||
return sendJson(res, 200, { activity: domainTraffic.activitySnapshot(thresholdBytesPerSecond) });
|
||||
}
|
||||
if (req.method === 'POST' && req.url === '/config/check') {
|
||||
const { config } = record(await readJson(req));
|
||||
return sendJson(res, 200, runtime.checkConfig(config));
|
||||
}
|
||||
if (req.method === 'POST' && req.url === '/apply') {
|
||||
return sendJson(res, 200, await runtime.apply());
|
||||
}
|
||||
|
||||
@@ -82,6 +82,16 @@ export function createDataplaneClient(socketPath: string, send: SendDataplaneReq
|
||||
throw new HarborError('DIAGNOSTICS_FAILED', { cause });
|
||||
}
|
||||
},
|
||||
checkConfig: (config: unknown) => send(socketPath, '/config/check', 'POST', { config }, 15_000),
|
||||
runFailoverProbe: (role: 'primary' | 'reserve', services: unknown, target: unknown, timeoutMs: number) => (
|
||||
send(socketPath, '/failover/probe', 'POST', { role, services, target, timeoutMs }, timeoutMs + 10_000)
|
||||
),
|
||||
readFailoverSelector: () => send(socketPath, '/failover/selector', 'GET'),
|
||||
selectFailoverRole: (role: 'primary' | 'reserve') => send(socketPath, '/failover/selector', 'PUT', { role }),
|
||||
setFailoverActivityEnabled: (enabled: boolean) => send(socketPath, '/failover/activity', 'PUT', { enabled }),
|
||||
readFailoverActivity: (thresholdBytesPerSecond: number) => (
|
||||
send(socketPath, '/failover/activity/read', 'POST', { thresholdBytesPerSecond })
|
||||
),
|
||||
apply: () => update('/apply', 'POST'),
|
||||
restart: () => update('/restart', 'POST'),
|
||||
stop: () => update('/stop', 'POST'),
|
||||
|
||||
@@ -4,7 +4,9 @@ import {
|
||||
type StoredState,
|
||||
} from '../../../shared/contracts/state.js';
|
||||
import { HarborError } from '../../../shared/errors.js';
|
||||
import { finishRollback } from '../../services/rollback.js';
|
||||
import { finishRollback, type RollbackStep } from '../../services/rollback.js';
|
||||
import type { AppliedFailoverPolicy } from '../../../shared/failover.js';
|
||||
import type { ActivityJournalEventInput } from '../../../shared/activityJournal.js';
|
||||
|
||||
interface ConnectionServiceDependencies {
|
||||
state: {
|
||||
@@ -26,6 +28,18 @@ interface ConnectionServiceDependencies {
|
||||
restartCommand(): Promise<RuntimeCommandResult>;
|
||||
};
|
||||
route?: { isGatewayDirect(): boolean };
|
||||
failover?: {
|
||||
build(state: StoredState, source?: 'desired' | 'applied'): {
|
||||
config: unknown;
|
||||
applied: AppliedFailoverPolicy;
|
||||
primaryProfile: StoredProfile;
|
||||
primaryServer: StoredProfile['servers'][number];
|
||||
};
|
||||
prepareActivation(role: 'primary' | 'reserve'): Promise<unknown>;
|
||||
restoreAppliedActivation(state: StoredState): Promise<unknown>;
|
||||
reconcile(): Promise<unknown>;
|
||||
};
|
||||
onEvent?: (event: ActivityJournalEventInput) => void;
|
||||
serialize<T>(operation: () => Promise<T>): Promise<T>;
|
||||
now(): Date;
|
||||
}
|
||||
@@ -72,6 +86,33 @@ function withDesiredServer(state: StoredState, profile: StoredProfile, serverId:
|
||||
}
|
||||
|
||||
export function createConnectionService(dependencies: ConnectionServiceDependencies) {
|
||||
const activationTarget = (
|
||||
state: StoredState,
|
||||
applied: AppliedFailoverPolicy,
|
||||
role: 'primary' | 'reserve',
|
||||
) => {
|
||||
const target = applied[role];
|
||||
const profile = profileById(state, target.profileId);
|
||||
const server = profile?.servers.find(({ id }) => id === target.serverId);
|
||||
if (!profile || !server) throw new HarborError('SERVER_NOT_FOUND');
|
||||
return { profile, server };
|
||||
};
|
||||
const finishConnectionRollback = async (error: unknown, steps: RollbackStep[], message: string) => {
|
||||
try {
|
||||
await finishRollback(error, steps, message);
|
||||
} catch (cause) {
|
||||
const code = cause && typeof cause === 'object' && 'code' in cause
|
||||
&& /^[A-Z0-9_]{1,50}$/.test(String(cause.code)) ? String(cause.code) : 'UNKNOWN';
|
||||
dependencies.onEvent?.({
|
||||
type: 'connection.failed',
|
||||
severity: 'error',
|
||||
source: 'connection',
|
||||
dedupeKey: `connection.failed:${dependencies.state.read().revision}:${code}`,
|
||||
data: { errorCode: code },
|
||||
});
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
const applyWithinQueue = async (
|
||||
previousState: StoredState,
|
||||
profile: StoredProfile,
|
||||
@@ -88,18 +129,26 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc
|
||||
if (!selectedServer) throw new HarborError('SERVER_NOT_FOUND');
|
||||
if (!profile.subscriptionConfig) throw new HarborError('CONFIG_INVALID');
|
||||
|
||||
const wasRunning = await dependencies.runtime.isRunning();
|
||||
if (wasRunning && previousState.failoverPolicy?.enabled) {
|
||||
dependencies.state.update((state) => withDesiredServer(state, profile, selectedServer.id));
|
||||
return { profileId: profile.id, serverId: selectedServer.id, selectedTag: selectedServer.label };
|
||||
}
|
||||
|
||||
if (dependencies.route?.isGatewayDirect()) {
|
||||
dependencies.state.update((state) => withDesiredServer(state, profile, selectedServer.id));
|
||||
return { profileId: profile.id, serverId: selectedServer.id, selectedTag: selectedServer.label };
|
||||
}
|
||||
|
||||
const nextConfig = dependencies.config.build(
|
||||
const failoverCandidate = previousState.failoverPolicy?.enabled
|
||||
? dependencies.failover?.build(previousState, wasRunning ? 'applied' : 'desired')
|
||||
: null;
|
||||
const nextConfig = failoverCandidate?.config || dependencies.config.build(
|
||||
profile.subscriptionConfig,
|
||||
selectedServer.id,
|
||||
previousState.routeRules,
|
||||
);
|
||||
const previousConfig = dependencies.config.read();
|
||||
const wasRunning = await dependencies.runtime.isRunning();
|
||||
let configMutationStarted = false;
|
||||
let runtimeMutationStarted = false;
|
||||
let stateCommitStarted = false;
|
||||
@@ -109,18 +158,20 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc
|
||||
dependencies.config.write(nextConfig);
|
||||
runtimeMutationStarted = true;
|
||||
await dependencies.runtime.start();
|
||||
if (failoverCandidate) await dependencies.failover!.prepareActivation('primary');
|
||||
stateCommitStarted = true;
|
||||
dependencies.state.update((state) => ({
|
||||
...withDesiredServer(state, profile, selectedServer.id),
|
||||
connectionDesired: 'running',
|
||||
appliedProfileId: profile.id,
|
||||
appliedServerId: selectedServer.id,
|
||||
appliedServerSnapshot: selectedServer,
|
||||
appliedProfileId: failoverCandidate?.primaryProfile.id || profile.id,
|
||||
appliedServerId: failoverCandidate?.primaryServer.id || selectedServer.id,
|
||||
appliedServerSnapshot: failoverCandidate?.primaryServer || selectedServer,
|
||||
appliedFailoverPolicy: failoverCandidate?.applied || null,
|
||||
appliedAt: dependencies.now().toISOString(),
|
||||
appliedRouteRules: state.routeRules,
|
||||
}));
|
||||
} catch (error) {
|
||||
await finishRollback(error, [
|
||||
await finishConnectionRollback(error, [
|
||||
...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []),
|
||||
...(configMutationStarted ? [{
|
||||
run: () => previousConfig === null
|
||||
@@ -128,12 +179,28 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc
|
||||
: dependencies.config.restore(previousConfig),
|
||||
}] : []),
|
||||
...(runtimeMutationStarted ? [{
|
||||
run: () => wasRunning ? dependencies.runtime.start() : dependencies.runtime.stop(),
|
||||
run: async () => {
|
||||
if (!wasRunning) return dependencies.runtime.stop();
|
||||
await dependencies.runtime.start();
|
||||
await dependencies.failover?.restoreAppliedActivation(previousState);
|
||||
},
|
||||
runtime: true,
|
||||
}] : []),
|
||||
], 'Connection rollback failed');
|
||||
}
|
||||
|
||||
await dependencies.failover?.reconcile();
|
||||
dependencies.onEvent?.({
|
||||
type: 'connection.started',
|
||||
severity: 'info',
|
||||
source: 'connection',
|
||||
dedupeKey: `connection.started:${dependencies.state.read().revision}`,
|
||||
data: {
|
||||
profileLabel: failoverCandidate?.primaryProfile.label || profile.label,
|
||||
serverLabel: failoverCandidate?.primaryServer.label || selectedServer.label,
|
||||
},
|
||||
});
|
||||
|
||||
return { profileId: profile.id, serverId: selectedServer.id, selectedTag: selectedServer.label };
|
||||
};
|
||||
|
||||
@@ -186,16 +253,29 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc
|
||||
appliedProfileId: '',
|
||||
appliedServerId: '',
|
||||
appliedServerSnapshot: null,
|
||||
appliedFailoverPolicy: null,
|
||||
}));
|
||||
} catch (error) {
|
||||
await finishRollback(error, [
|
||||
await finishConnectionRollback(error, [
|
||||
...(runtimeMutationStarted && wasRunning !== null ? [{
|
||||
run: () => wasRunning ? dependencies.runtime.start() : dependencies.runtime.stop(),
|
||||
run: async () => {
|
||||
if (!wasRunning) return dependencies.runtime.stop();
|
||||
await dependencies.runtime.start();
|
||||
await dependencies.failover?.restoreAppliedActivation(previousState);
|
||||
},
|
||||
runtime: true,
|
||||
}] : []),
|
||||
...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []),
|
||||
], 'Connection rollback failed');
|
||||
}
|
||||
await dependencies.failover?.reconcile();
|
||||
dependencies.onEvent?.({
|
||||
type: 'connection.stopped',
|
||||
severity: 'info',
|
||||
source: 'connection',
|
||||
dedupeKey: `connection.stopped:${dependencies.state.read().revision}`,
|
||||
data: {},
|
||||
});
|
||||
});
|
||||
|
||||
const restart = () => dependencies.serialize(async () => {
|
||||
@@ -213,7 +293,18 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc
|
||||
? previousState.appliedServerSnapshot
|
||||
: null);
|
||||
if (!server || !profile.subscriptionConfig) throw new HarborError('CONFIG_INVALID');
|
||||
const candidateConfig = dependencies.config.build(
|
||||
const failoverCandidate = previousState.failoverPolicy?.enabled
|
||||
? dependencies.failover?.build(previousState, wasRunning ? 'applied' : 'desired')
|
||||
: null;
|
||||
const activationRole = failoverCandidate && wasRunning
|
||||
&& previousState.appliedProfileId === failoverCandidate.applied.reserve.profileId
|
||||
&& previousState.appliedServerId === failoverCandidate.applied.reserve.serverId
|
||||
? 'reserve' as const
|
||||
: 'primary' as const;
|
||||
const failoverTarget = failoverCandidate
|
||||
? activationTarget(previousState, failoverCandidate.applied, activationRole)
|
||||
: null;
|
||||
const candidateConfig = failoverCandidate?.config || dependencies.config.build(
|
||||
profile.subscriptionConfig,
|
||||
server.id,
|
||||
previousState.routeRules,
|
||||
@@ -229,30 +320,47 @@ export function createConnectionService(dependencies: ConnectionServiceDependenc
|
||||
const command = await dependencies.runtime.restartCommand();
|
||||
runtimeMutationStarted = command.mutationStarted;
|
||||
if (!command.ok) throw command.error;
|
||||
if (failoverCandidate) await dependencies.failover!.prepareActivation(activationRole);
|
||||
stateCommitStarted = true;
|
||||
dependencies.state.update((state) => ({
|
||||
...state,
|
||||
desiredProfileId: wasRunning ? state.desiredProfileId : profile.id,
|
||||
appliedProfileId: profile.id,
|
||||
appliedServerId: server.id,
|
||||
appliedServerSnapshot: server,
|
||||
appliedProfileId: failoverTarget?.profile.id || profile.id,
|
||||
appliedServerId: failoverTarget?.server.id || server.id,
|
||||
appliedServerSnapshot: failoverTarget?.server || server,
|
||||
appliedFailoverPolicy: failoverCandidate?.applied || null,
|
||||
connectionDesired: 'running',
|
||||
appliedRouteRules: dependencies.route?.isGatewayDirect() ? [] : state.routeRules,
|
||||
}));
|
||||
} catch (error) {
|
||||
await finishRollback(error, [
|
||||
await finishConnectionRollback(error, [
|
||||
...(configMutationStarted ? [{
|
||||
run: () => previousConfig === null
|
||||
? dependencies.config.remove()
|
||||
: dependencies.config.restore(previousConfig),
|
||||
}] : []),
|
||||
...(runtimeMutationStarted ? [{
|
||||
run: () => wasRunning ? dependencies.runtime.start() : dependencies.runtime.stop(),
|
||||
run: async () => {
|
||||
if (!wasRunning) return dependencies.runtime.stop();
|
||||
await dependencies.runtime.start();
|
||||
await dependencies.failover?.restoreAppliedActivation(previousState);
|
||||
},
|
||||
runtime: true,
|
||||
}] : []),
|
||||
...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []),
|
||||
], 'Connection rollback failed');
|
||||
}
|
||||
await dependencies.failover?.reconcile();
|
||||
dependencies.onEvent?.({
|
||||
type: 'connection.started',
|
||||
severity: 'info',
|
||||
source: 'connection',
|
||||
dedupeKey: `connection.started:${dependencies.state.read().revision}`,
|
||||
data: {
|
||||
profileLabel: failoverTarget?.profile.label || profile.label,
|
||||
serverLabel: failoverTarget?.server.label || server.label,
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
return { apply, activate, stop, restart };
|
||||
|
||||
@@ -0,0 +1,671 @@
|
||||
import crypto from 'node:crypto';
|
||||
import {
|
||||
createIdleFailoverSnapshot,
|
||||
isFailoverConfigured,
|
||||
nextFailoverDecision,
|
||||
normalizeFailoverPolicy,
|
||||
type AppliedFailoverPolicy,
|
||||
type FailoverDecisionMemory,
|
||||
type FailoverHealth,
|
||||
type FailoverPolicy,
|
||||
type FailoverRole,
|
||||
type FailoverSnapshot,
|
||||
} from '../../../shared/failover.js';
|
||||
import type { HarborServer, StoredState } from '../../../shared/contracts/state.js';
|
||||
import type { ActivityJournalEventInput } from '../../../shared/activityJournal.js';
|
||||
import { HarborError } from '../../../shared/errors.js';
|
||||
|
||||
interface Candidate {
|
||||
config: unknown;
|
||||
applied: AppliedFailoverPolicy;
|
||||
}
|
||||
|
||||
interface FailoverServiceDependencies {
|
||||
state: {
|
||||
read(): StoredState;
|
||||
update(mutator: (state: StoredState) => Record<string, unknown>): StoredState;
|
||||
};
|
||||
runtime: { isRunning(): Promise<boolean> };
|
||||
dataplane: {
|
||||
checkConfig(config: unknown): Promise<unknown>;
|
||||
runFailoverProbe(role: FailoverRole, services: unknown, target: string, timeoutMs: number): Promise<unknown>;
|
||||
readFailoverSelector(): Promise<unknown>;
|
||||
selectFailoverRole(role: FailoverRole): Promise<unknown>;
|
||||
setFailoverActivityEnabled(enabled: boolean): Promise<unknown>;
|
||||
readFailoverActivity(thresholdBytesPerSecond: number): Promise<unknown>;
|
||||
};
|
||||
buildCandidate(state: StoredState): Candidate;
|
||||
serialize<T>(operation: () => Promise<T>): Promise<T>;
|
||||
scheduler?: {
|
||||
setTimeout(callback: () => void, intervalMs: number): NodeJS.Timeout;
|
||||
clearTimeout(timer: NodeJS.Timeout): void;
|
||||
};
|
||||
now?: () => Date;
|
||||
onWarning?: (error: unknown) => void;
|
||||
onSwitch?: (from: FailoverRole, to: FailoverRole, reason: string) => void;
|
||||
onEvent?: (event: ActivityJournalEventInput) => void;
|
||||
}
|
||||
|
||||
const record = (value: unknown): Record<string, unknown> => (
|
||||
value && typeof value === 'object' && !Array.isArray(value) ? value as Record<string, unknown> : {}
|
||||
);
|
||||
|
||||
function targetServer(state: StoredState, role: FailoverRole): HarborServer | null {
|
||||
const target = state.appliedFailoverPolicy?.[role] || state.failoverPolicy[role];
|
||||
return state.profiles.find(({ id }) => id === target.profileId)
|
||||
?.servers.find(({ id }) => id === target.serverId) || null;
|
||||
}
|
||||
|
||||
function currentRole(state: StoredState): FailoverRole | null {
|
||||
const applied = state.appliedFailoverPolicy;
|
||||
if (!applied) return null;
|
||||
for (const role of ['primary', 'reserve'] as const) {
|
||||
if (
|
||||
state.appliedProfileId === applied[role].profileId
|
||||
&& state.appliedServerId === applied[role].serverId
|
||||
) return role;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function probeHealth(value: unknown): boolean {
|
||||
const vpn = record(record(value).vpn);
|
||||
const sites = Array.isArray(vpn.sites) ? vpn.sites.map(record) : [];
|
||||
return sites.length === 1 && sites[0].status === 'available';
|
||||
}
|
||||
|
||||
function safeErrorCode(error: unknown) {
|
||||
const code = error && typeof error === 'object' && 'code' in error ? String(error.code) : '';
|
||||
return /^[A-Z0-9_]{1,50}$/.test(code) ? code : 'UNKNOWN';
|
||||
}
|
||||
|
||||
export function createFailoverService(dependencies: FailoverServiceDependencies) {
|
||||
const scheduler = dependencies.scheduler || {
|
||||
setTimeout: (callback: () => void, intervalMs: number) => setTimeout(callback, intervalMs),
|
||||
clearTimeout: (timer: NodeJS.Timeout) => clearTimeout(timer),
|
||||
};
|
||||
const now = dependencies.now || (() => new Date());
|
||||
const epoch = crypto.randomUUID();
|
||||
let sequence = 0;
|
||||
let generation = 0;
|
||||
let timer: NodeJS.Timeout | null = null;
|
||||
let collectorEnabled: boolean | null = null;
|
||||
let roundPromise: Promise<void> | null = null;
|
||||
let decisionMemory: FailoverDecisionMemory | undefined;
|
||||
let snapshot = createIdleFailoverSnapshot(dependencies.state.read().failoverPolicy, epoch, sequence);
|
||||
|
||||
function appliedMatchesDesired(state: StoredState) {
|
||||
if (!state.appliedFailoverPolicy) return false;
|
||||
try {
|
||||
return JSON.stringify(dependencies.buildCandidate(state).applied)
|
||||
=== JSON.stringify(state.appliedFailoverPolicy);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function publish(next: FailoverSnapshot) {
|
||||
sequence += 1;
|
||||
snapshot = { ...next, observationEpoch: epoch, observationSequence: sequence };
|
||||
}
|
||||
|
||||
function clearTimer() {
|
||||
if (timer) scheduler.clearTimeout(timer);
|
||||
timer = null;
|
||||
}
|
||||
|
||||
function schedule(delay: number) {
|
||||
clearTimer();
|
||||
timer = scheduler.setTimeout(() => {
|
||||
timer = null;
|
||||
void runRound().catch(dependencies.onWarning);
|
||||
}, delay);
|
||||
timer.unref?.();
|
||||
}
|
||||
|
||||
async function disableCollector() {
|
||||
if (collectorEnabled === false) return;
|
||||
await dependencies.dataplane.setFailoverActivityEnabled(false);
|
||||
collectorEnabled = false;
|
||||
}
|
||||
|
||||
async function deactivate(policy: FailoverPolicy, passiveRole: FailoverRole | null = null) {
|
||||
generation += 1;
|
||||
clearTimer();
|
||||
decisionMemory = undefined;
|
||||
await disableCollector();
|
||||
const idle = createIdleFailoverSnapshot(policy, epoch, sequence);
|
||||
if (passiveRole) {
|
||||
idle.activation = 'passive-loaded';
|
||||
idle.currentRole = passiveRole;
|
||||
idle.reason = 'disabled';
|
||||
}
|
||||
publish(idle);
|
||||
}
|
||||
|
||||
function activeSnapshot(state: StoredState, status: FailoverSnapshot['status'] = 'observing'): FailoverSnapshot {
|
||||
const role = state.failoverRuntimeState.reasonCode === 'selector-unknown' ? null : currentRole(state);
|
||||
const channel = (target: typeof state.failoverPolicy.primary) => ({
|
||||
target,
|
||||
health: 'unknown' as FailoverHealth,
|
||||
failingServiceIds: [],
|
||||
checkedAt: null,
|
||||
stateSince: null,
|
||||
});
|
||||
return {
|
||||
observationEpoch: epoch,
|
||||
observationSequence: sequence,
|
||||
configured: isFailoverConfigured(state.failoverPolicy),
|
||||
enabled: state.failoverPolicy.enabled,
|
||||
paused: state.failoverPolicy.paused,
|
||||
activation: appliedMatchesDesired(state) ? 'active' : 'pending',
|
||||
currentRole: role || 'other',
|
||||
status,
|
||||
primary: channel(state.appliedFailoverPolicy?.primary || state.failoverPolicy.primary),
|
||||
reserve: channel(state.appliedFailoverPolicy?.reserve || state.failoverPolicy.reserve),
|
||||
nextDecisionAt: null,
|
||||
reason: null,
|
||||
trafficActivity: null,
|
||||
policy: state.failoverPolicy,
|
||||
};
|
||||
}
|
||||
|
||||
async function reconcile() {
|
||||
let state = dependencies.state.read();
|
||||
const policy = state.failoverPolicy;
|
||||
if (!policy.enabled) {
|
||||
const role = currentRole(state);
|
||||
const passiveRole = role && await dependencies.runtime.isRunning() ? role : null;
|
||||
return deactivate(policy, passiveRole);
|
||||
}
|
||||
const running = await dependencies.runtime.isRunning();
|
||||
if (!running || !state.appliedFailoverPolicy || !currentRole(state)) {
|
||||
generation += 1;
|
||||
clearTimer();
|
||||
await disableCollector();
|
||||
const pending = activeSnapshot(state, 'idle');
|
||||
pending.activation = running ? 'pending' : 'inactive';
|
||||
pending.reason = running ? 'pending-activation' : 'vpn-stopped';
|
||||
publish(pending);
|
||||
return;
|
||||
}
|
||||
const role = currentRole(state)!;
|
||||
const selected = record(await dependencies.dataplane.readFailoverSelector());
|
||||
if (selected.role !== role) await dependencies.dataplane.selectFailoverRole(role);
|
||||
if (state.failoverRuntimeState.reasonCode === 'selector-unknown') {
|
||||
state = dependencies.state.update((current) => ({
|
||||
...current,
|
||||
failoverRuntimeState: { ...current.failoverRuntimeState, reasonCode: null },
|
||||
}));
|
||||
}
|
||||
if (collectorEnabled !== true) {
|
||||
await dependencies.dataplane.setFailoverActivityEnabled(true);
|
||||
collectorEnabled = true;
|
||||
}
|
||||
generation += 1;
|
||||
const active = activeSnapshot(state);
|
||||
if (active.activation === 'pending') active.reason = 'pending-activation';
|
||||
publish(active);
|
||||
schedule(0);
|
||||
}
|
||||
|
||||
async function reconcileAfterCommit() {
|
||||
try {
|
||||
await reconcile();
|
||||
} catch (error) {
|
||||
dependencies.onWarning?.(error);
|
||||
const failed = activeSnapshot(dependencies.state.read(), 'error');
|
||||
failed.reason = 'reconcile-failed';
|
||||
publish(failed);
|
||||
}
|
||||
}
|
||||
|
||||
async function assessRole(role: FailoverRole, policy: FailoverPolicy) {
|
||||
const custom = dependencies.state.read().diagnostics.customServices;
|
||||
const results = await Promise.all(policy.checks.map(async (check) => {
|
||||
try {
|
||||
return {
|
||||
id: check.serviceId,
|
||||
ok: probeHealth(await dependencies.dataplane.runFailoverProbe(
|
||||
role,
|
||||
custom,
|
||||
`site:${check.serviceId}`,
|
||||
check.timeoutMs,
|
||||
)),
|
||||
};
|
||||
} catch {
|
||||
return { id: check.serviceId, ok: null };
|
||||
}
|
||||
}));
|
||||
return {
|
||||
health: results.some(({ ok }) => ok === null)
|
||||
? 'unknown' as const
|
||||
: results.every(({ ok }) => ok) ? 'healthy' as const : 'unhealthy' as const,
|
||||
failingServiceIds: results.filter(({ ok }) => ok === false).map(({ id }) => id),
|
||||
};
|
||||
}
|
||||
|
||||
async function switchWithinQueue(role: FailoverRole, reason: string) {
|
||||
const before = dependencies.state.read();
|
||||
const from = currentRole(before);
|
||||
if (!from || from === role) return;
|
||||
try {
|
||||
await dependencies.dataplane.selectFailoverRole(role);
|
||||
const server = targetServer(before, role);
|
||||
if (!server) throw new HarborError('SERVER_NOT_FOUND');
|
||||
const target = before.appliedFailoverPolicy![role];
|
||||
const switchedAt = now().toISOString();
|
||||
const cutoff = now().getTime() - before.failoverPolicy.flapProtection.windowMs;
|
||||
const history = role === 'reserve'
|
||||
? [...before.failoverRuntimeState.failoverHistory.filter((value) => Date.parse(value) >= cutoff), switchedAt]
|
||||
: before.failoverRuntimeState.failoverHistory.filter((value) => Date.parse(value) >= cutoff);
|
||||
const quarantine = history.length >= before.failoverPolicy.flapProtection.count
|
||||
? new Date(now().getTime() + before.failoverPolicy.flapProtection.quarantineMs).toISOString()
|
||||
: before.failoverRuntimeState.primaryQuarantineUntil;
|
||||
dependencies.state.update((state) => ({
|
||||
...state,
|
||||
appliedProfileId: target.profileId,
|
||||
appliedServerId: target.serverId,
|
||||
appliedServerSnapshot: server,
|
||||
failoverPolicy: reason === 'manual'
|
||||
? { ...state.failoverPolicy, paused: true }
|
||||
: role === 'primary' ? { ...state.failoverPolicy, paused: false } : state.failoverPolicy,
|
||||
failoverRuntimeState: {
|
||||
...state.failoverRuntimeState,
|
||||
lastSwitchAt: switchedAt,
|
||||
holdUntil: role === 'reserve'
|
||||
? new Date(now().getTime() + state.failoverPolicy.minimumReserveMs).toISOString()
|
||||
: null,
|
||||
primaryQuarantineUntil: quarantine,
|
||||
failoverHistory: history,
|
||||
reasonCode: reason,
|
||||
},
|
||||
}));
|
||||
} catch (error) {
|
||||
try {
|
||||
await dependencies.dataplane.selectFailoverRole(from);
|
||||
} catch (rollback) {
|
||||
generation += 1;
|
||||
clearTimer();
|
||||
decisionMemory = undefined;
|
||||
dependencies.state.update((state) => ({
|
||||
...state,
|
||||
failoverPolicy: { ...state.failoverPolicy, paused: true },
|
||||
failoverRuntimeState: { ...state.failoverRuntimeState, reasonCode: 'selector-unknown' },
|
||||
}));
|
||||
const failed = activeSnapshot(dependencies.state.read(), 'error');
|
||||
failed.reason = 'selector-unknown';
|
||||
publish(failed);
|
||||
throw new AggregateError([error, rollback], 'Failover selector rollback failed');
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
dependencies.onSwitch?.(from, role, reason);
|
||||
dependencies.onEvent?.({
|
||||
type: 'failover.switched',
|
||||
severity: 'info',
|
||||
source: 'failover',
|
||||
dedupeKey: `failover.switched:${dependencies.state.read().revision}`,
|
||||
data: {
|
||||
fromRole: from,
|
||||
toRole: role,
|
||||
primaryLabel: targetServer(dependencies.state.read(), 'primary')?.label || 'Primary',
|
||||
reserveLabel: targetServer(dependencies.state.read(), 'reserve')?.label || 'Reserve',
|
||||
reason,
|
||||
manual: reason === 'manual',
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async function performRound(capturedGeneration: number) {
|
||||
const prepared = await dependencies.serialize(async () => {
|
||||
const state = dependencies.state.read();
|
||||
const role = currentRole(state);
|
||||
if (
|
||||
capturedGeneration !== generation
|
||||
|| !state.failoverPolicy.enabled
|
||||
|| !state.appliedFailoverPolicy
|
||||
|| !role
|
||||
) return null;
|
||||
const selected = record(await dependencies.dataplane.readFailoverSelector());
|
||||
if (selected.role !== role) await dependencies.dataplane.selectFailoverRole(role);
|
||||
await dependencies.dataplane.setFailoverActivityEnabled(true);
|
||||
collectorEnabled = true;
|
||||
const latest = dependencies.state.read();
|
||||
return capturedGeneration === generation
|
||||
&& latest.failoverPolicy.enabled
|
||||
&& currentRole(latest) === role
|
||||
? { state: latest, policy: latest.failoverPolicy, role }
|
||||
: null;
|
||||
});
|
||||
if (!prepared) return;
|
||||
const { state, policy, role } = prepared;
|
||||
const checkedAt = now().toISOString();
|
||||
let primary;
|
||||
let reserve;
|
||||
try {
|
||||
[primary, reserve] = await Promise.all([
|
||||
assessRole('primary', policy),
|
||||
assessRole('reserve', policy),
|
||||
]);
|
||||
} catch {
|
||||
primary = { health: 'unknown' as const, failingServiceIds: [] };
|
||||
reserve = { health: 'unknown' as const, failingServiceIds: [] };
|
||||
}
|
||||
if (capturedGeneration !== generation || !dependencies.state.read().failoverPolicy.enabled) return;
|
||||
const activityResponse = record(await dependencies.dataplane.readFailoverActivity(
|
||||
policy.trafficGuard.thresholdBytesPerSecond,
|
||||
));
|
||||
if (capturedGeneration !== generation || !dependencies.state.read().failoverPolicy.enabled) return;
|
||||
const activity = record(activityResponse.activity);
|
||||
const observedAt = typeof activity.observedAt === 'string' ? Date.parse(activity.observedAt) : NaN;
|
||||
const activityState = Number.isFinite(observedAt) && now().getTime() - observedAt <= 4_000
|
||||
&& (activity.state === 'active' || activity.state === 'quiet')
|
||||
? activity.state
|
||||
: 'unknown';
|
||||
const decision = nextFailoverDecision({
|
||||
now: now().getTime(),
|
||||
policy,
|
||||
currentRole: role,
|
||||
primaryHealth: primary.health,
|
||||
reserveHealth: reserve.health,
|
||||
activity: activityState,
|
||||
holdUntil: Date.parse(state.failoverRuntimeState.holdUntil || '') || null,
|
||||
primaryQuarantineUntil: Date.parse(state.failoverRuntimeState.primaryQuarantineUntil || '') || null,
|
||||
memory: decisionMemory,
|
||||
});
|
||||
decisionMemory = decision.memory;
|
||||
const previousSnapshot = snapshot;
|
||||
const next = activeSnapshot(state, decision.status);
|
||||
next.currentRole = role;
|
||||
next.primary = {
|
||||
...next.primary,
|
||||
...primary,
|
||||
checkedAt,
|
||||
stateSince: previousSnapshot.primary.health === primary.health
|
||||
? previousSnapshot.primary.stateSince || checkedAt
|
||||
: checkedAt,
|
||||
};
|
||||
next.reserve = {
|
||||
...next.reserve,
|
||||
...reserve,
|
||||
checkedAt,
|
||||
stateSince: previousSnapshot.reserve.health === reserve.health
|
||||
? previousSnapshot.reserve.stateSince || checkedAt
|
||||
: checkedAt,
|
||||
};
|
||||
next.reason = decision.reason;
|
||||
next.nextDecisionAt = decision.nextDecisionAt ? new Date(decision.nextDecisionAt).toISOString() : null;
|
||||
next.trafficActivity = activityState === 'unknown' ? {
|
||||
state: 'unknown',
|
||||
observedAt: Number.isFinite(observedAt) ? new Date(observedAt).toISOString() : checkedAt,
|
||||
windowMs: 10_000,
|
||||
thresholdBytesPerSecond: policy.trafficGuard.thresholdBytesPerSecond,
|
||||
totalBytesPerSecond: 0,
|
||||
transmittingConnections: 0,
|
||||
quietSince: null,
|
||||
switchTarget: decision.switchTo,
|
||||
blockers: [],
|
||||
} : {
|
||||
state: activityState,
|
||||
observedAt: String(activity.observedAt),
|
||||
windowMs: Number(activity.windowMs) || 10_000,
|
||||
thresholdBytesPerSecond: policy.trafficGuard.thresholdBytesPerSecond,
|
||||
totalBytesPerSecond: Number(activity.totalBytesPerSecond) || 0,
|
||||
transmittingConnections: Number(activity.transmittingConnections) || 0,
|
||||
quietSince: typeof activity.quietSince === 'string' ? activity.quietSince : null,
|
||||
switchTarget: decision.switchTo,
|
||||
blockers: (Array.isArray(activity.blockers) ? activity.blockers : []).slice(0, 3) as FailoverSnapshot['trafficActivity'] extends infer T ? T extends { blockers: infer B } ? B : never : never,
|
||||
};
|
||||
publish(next);
|
||||
if (decision.status === 'waiting-for-idle' && previousSnapshot.status !== 'waiting-for-idle') {
|
||||
dependencies.onEvent?.({
|
||||
type: 'failover.waiting_for_idle',
|
||||
severity: 'info',
|
||||
source: 'failover',
|
||||
dedupeKey: `failover.waiting_for_idle:${state.revision}:${role}:${decision.reason}`,
|
||||
data: { fromRole: role, toRole: decision.switchTo || (role === 'primary' ? 'reserve' : 'primary'), reason: decision.reason },
|
||||
});
|
||||
}
|
||||
if (decision.reason === 'both-unhealthy' && previousSnapshot.reason !== 'both-unhealthy') {
|
||||
dependencies.onEvent?.({
|
||||
type: 'failover.both_unhealthy',
|
||||
severity: 'warning',
|
||||
source: 'failover',
|
||||
dedupeKey: `failover.both_unhealthy:${state.revision}`,
|
||||
data: { reason: decision.reason },
|
||||
});
|
||||
}
|
||||
if (primary.health === 'healthy' && previousSnapshot.primary.health === 'unhealthy') {
|
||||
dependencies.onEvent?.({
|
||||
type: 'failover.recovered',
|
||||
severity: 'info',
|
||||
source: 'failover',
|
||||
dedupeKey: `failover.recovered:${state.revision}:primary`,
|
||||
data: { role: 'primary', reason: decision.reason },
|
||||
});
|
||||
}
|
||||
if (decision.switchTo) {
|
||||
try {
|
||||
const switched = await dependencies.serialize(async () => {
|
||||
const current = dependencies.state.read();
|
||||
if (
|
||||
capturedGeneration !== generation
|
||||
|| !current.failoverPolicy.enabled
|
||||
|| current.failoverPolicy.paused
|
||||
|| currentRole(current) !== role
|
||||
|| !current.appliedFailoverPolicy
|
||||
) return false;
|
||||
let freshPrimary;
|
||||
let freshReserve;
|
||||
try {
|
||||
[freshPrimary, freshReserve] = await Promise.all([
|
||||
assessRole('primary', current.failoverPolicy),
|
||||
assessRole('reserve', current.failoverPolicy),
|
||||
]);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
const healthStillAllowsSwitch = decision.switchTo === 'reserve'
|
||||
? freshPrimary.health === 'unhealthy' && freshReserve.health === 'healthy'
|
||||
: freshPrimary.health === 'healthy';
|
||||
if (!healthStillAllowsSwitch || capturedGeneration !== generation) return false;
|
||||
if (current.failoverPolicy.trafficGuard.enabled) {
|
||||
const freshResponse = record(await dependencies.dataplane.readFailoverActivity(
|
||||
current.failoverPolicy.trafficGuard.thresholdBytesPerSecond,
|
||||
));
|
||||
const freshActivity = record(freshResponse.activity);
|
||||
const freshObservedAt = typeof freshActivity.observedAt === 'string'
|
||||
? Date.parse(freshActivity.observedAt)
|
||||
: NaN;
|
||||
const freshQuietSince = typeof freshActivity.quietSince === 'string'
|
||||
? Date.parse(freshActivity.quietSince)
|
||||
: NaN;
|
||||
if (
|
||||
capturedGeneration !== generation
|
||||
|| freshActivity.state !== 'quiet'
|
||||
|| !Number.isFinite(freshObservedAt)
|
||||
|| now().getTime() - freshObservedAt > 4_000
|
||||
|| !Number.isFinite(freshQuietSince)
|
||||
|| now().getTime() - freshQuietSince < current.failoverPolicy.trafficGuard.quietWindowMs
|
||||
) return false;
|
||||
}
|
||||
await switchWithinQueue(decision.switchTo!, decision.reason);
|
||||
return true;
|
||||
});
|
||||
if (!switched) {
|
||||
const cancelled = activeSnapshot(dependencies.state.read(), 'observing');
|
||||
cancelled.reason = 'revalidation-required';
|
||||
publish(cancelled);
|
||||
decisionMemory = undefined;
|
||||
return;
|
||||
}
|
||||
} catch (error) {
|
||||
const failed = activeSnapshot(dependencies.state.read(), 'error');
|
||||
failed.reason = dependencies.state.read().failoverRuntimeState.reasonCode === 'selector-unknown'
|
||||
? 'selector-unknown'
|
||||
: 'switch-failed';
|
||||
publish(failed);
|
||||
dependencies.onEvent?.({
|
||||
type: 'failover.switch_failed',
|
||||
severity: 'error',
|
||||
source: 'failover',
|
||||
dedupeKey: `failover.switch_failed:${state.revision}:${role}:${decision.switchTo}`,
|
||||
data: { fromRole: role, toRole: decision.switchTo, reason: decision.reason, errorCode: safeErrorCode(error) },
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
if (capturedGeneration !== generation) return;
|
||||
publish(activeSnapshot(dependencies.state.read(), decision.switchTo === 'reserve' ? 'reserve' : 'primary'));
|
||||
decisionMemory = undefined;
|
||||
}
|
||||
}
|
||||
|
||||
async function runRound() {
|
||||
if (roundPromise) return roundPromise;
|
||||
const capturedGeneration = generation;
|
||||
roundPromise = performRound(capturedGeneration).finally(() => {
|
||||
roundPromise = null;
|
||||
const policy = dependencies.state.read().failoverPolicy;
|
||||
if (capturedGeneration === generation && policy.enabled && dependencies.state.read().appliedFailoverPolicy) {
|
||||
const decisionAt = snapshot.nextDecisionAt ? Date.parse(snapshot.nextDecisionAt) : NaN;
|
||||
const decisionDelay = Number.isFinite(decisionAt)
|
||||
? Math.max(250, decisionAt - now().getTime())
|
||||
: policy.intervalMs;
|
||||
schedule(Math.min(policy.intervalMs, decisionDelay));
|
||||
}
|
||||
});
|
||||
return roundPromise;
|
||||
}
|
||||
|
||||
function save(value: unknown) {
|
||||
return dependencies.serialize(async () => {
|
||||
let policy: FailoverPolicy;
|
||||
try {
|
||||
policy = normalizeFailoverPolicy(value, { strict: true });
|
||||
} catch (cause) {
|
||||
throw new HarborError('REQUEST_INVALID', { cause });
|
||||
}
|
||||
if (policy.enabled && !isFailoverConfigured(policy)) throw new HarborError('REQUEST_INVALID');
|
||||
const before = dependencies.state.read();
|
||||
const candidateState = { ...before, failoverPolicy: policy };
|
||||
if (policy.enabled) {
|
||||
const candidate = dependencies.buildCandidate(candidateState);
|
||||
await dependencies.dataplane.checkConfig(candidate.config);
|
||||
}
|
||||
dependencies.state.update((state) => {
|
||||
const role = before.failoverPolicy.enabled && !policy.enabled ? currentRole(state) : null;
|
||||
const target = role ? state.appliedFailoverPolicy?.[role] : null;
|
||||
return {
|
||||
...state,
|
||||
failoverPolicy: policy,
|
||||
...(target ? {
|
||||
desiredProfileId: target.profileId,
|
||||
profiles: state.profiles.map((profile) => profile.id === target.profileId
|
||||
? { ...profile, desiredServerId: target.serverId }
|
||||
: profile),
|
||||
} : {}),
|
||||
};
|
||||
});
|
||||
decisionMemory = undefined;
|
||||
if (before.failoverPolicy.enabled !== policy.enabled) {
|
||||
dependencies.onEvent?.({
|
||||
type: policy.enabled ? 'failover.enabled' : 'failover.disabled',
|
||||
severity: 'info',
|
||||
source: 'failover',
|
||||
dedupeKey: `failover.${policy.enabled ? 'enabled' : 'disabled'}:${dependencies.state.read().revision}`,
|
||||
data: {},
|
||||
});
|
||||
}
|
||||
await reconcileAfterCommit();
|
||||
});
|
||||
}
|
||||
|
||||
function pause(paused: boolean) {
|
||||
return dependencies.serialize(async () => {
|
||||
const before = dependencies.state.read();
|
||||
if (!paused && (
|
||||
!before.appliedFailoverPolicy
|
||||
|| !targetServer(before, 'primary')
|
||||
|| !targetServer(before, 'reserve')
|
||||
)) throw new HarborError('REQUEST_INVALID');
|
||||
dependencies.state.update((state) => ({
|
||||
...state,
|
||||
failoverPolicy: { ...state.failoverPolicy, paused },
|
||||
}));
|
||||
decisionMemory = undefined;
|
||||
dependencies.onEvent?.({
|
||||
type: paused ? 'failover.paused' : 'failover.resumed',
|
||||
severity: 'info',
|
||||
source: 'failover',
|
||||
dedupeKey: `failover.${paused ? 'paused' : 'resumed'}:${dependencies.state.read().revision}`,
|
||||
data: {},
|
||||
});
|
||||
await reconcileAfterCommit();
|
||||
});
|
||||
}
|
||||
|
||||
async function manualSwitch(role: FailoverRole) {
|
||||
await dependencies.serialize(async () => {
|
||||
const state = dependencies.state.read();
|
||||
if (!state.failoverPolicy.enabled || !state.appliedFailoverPolicy || !currentRole(state)) {
|
||||
throw new HarborError('REQUEST_INVALID');
|
||||
}
|
||||
await switchWithinQueue(role, 'manual');
|
||||
});
|
||||
await reconcileAfterCommit();
|
||||
}
|
||||
|
||||
const prepareActivation = (role: FailoverRole) => dependencies.dataplane.selectFailoverRole(role);
|
||||
|
||||
async function restoreAppliedActivation(state: StoredState) {
|
||||
if (!state.appliedFailoverPolicy) return;
|
||||
const role = currentRole(state);
|
||||
if (!role) throw new HarborError('CONFIG_INVALID');
|
||||
await prepareActivation(role);
|
||||
}
|
||||
|
||||
function checkNow() {
|
||||
return dependencies.serialize(async () => {
|
||||
const state = dependencies.state.read();
|
||||
if (!state.failoverPolicy.enabled || !state.appliedFailoverPolicy || !currentRole(state)) {
|
||||
throw new HarborError('REQUEST_INVALID');
|
||||
}
|
||||
generation += 1;
|
||||
clearTimer();
|
||||
const checkedAt = now().toISOString();
|
||||
let primary;
|
||||
let reserve;
|
||||
try {
|
||||
[primary, reserve] = await Promise.all([
|
||||
assessRole('primary', state.failoverPolicy),
|
||||
assessRole('reserve', state.failoverPolicy),
|
||||
]);
|
||||
} catch {
|
||||
primary = { health: 'unknown' as const, failingServiceIds: [] };
|
||||
reserve = { health: 'unknown' as const, failingServiceIds: [] };
|
||||
}
|
||||
const next = activeSnapshot(dependencies.state.read(), 'observing');
|
||||
next.primary = { ...next.primary, ...primary, checkedAt, stateSince: checkedAt };
|
||||
next.reserve = { ...next.reserve, ...reserve, checkedAt, stateSince: checkedAt };
|
||||
next.reason = 'manual-check';
|
||||
publish(next);
|
||||
schedule(state.failoverPolicy.intervalMs);
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
snapshot: () => snapshot,
|
||||
save,
|
||||
pause,
|
||||
manualSwitch,
|
||||
checkNow,
|
||||
prepareActivation,
|
||||
restoreAppliedActivation,
|
||||
reconcile,
|
||||
runRound,
|
||||
shutdown: () => deactivate(dependencies.state.read().failoverPolicy),
|
||||
};
|
||||
}
|
||||
|
||||
export type FailoverService = ReturnType<typeof createFailoverService>;
|
||||
@@ -35,6 +35,8 @@ interface RouteRulesDependencies {
|
||||
route?: { isGatewayDirect(): boolean };
|
||||
serialize<T>(operation: () => Promise<T>): Promise<T>;
|
||||
runOperation<T>(operation: () => Promise<T>): Promise<T>;
|
||||
afterApply?: () => Promise<unknown>;
|
||||
restoreAppliedActivation?: (state: StoredState) => Promise<unknown>;
|
||||
}
|
||||
|
||||
export function createRouteRulesService(dependencies: RouteRulesDependencies) {
|
||||
@@ -96,6 +98,7 @@ export function createRouteRulesService(dependencies: RouteRulesDependencies) {
|
||||
...(wasRunning ? { appliedRouteRules: routeRules } : {}),
|
||||
routeRulesRevision: state.routeRulesRevision + 1,
|
||||
}));
|
||||
await dependencies.afterApply?.();
|
||||
} catch (error) {
|
||||
await finishRollback(error, [
|
||||
...(configMutationStarted ? [{
|
||||
@@ -104,7 +107,10 @@ export function createRouteRulesService(dependencies: RouteRulesDependencies) {
|
||||
: dependencies.config.restore(previousConfig),
|
||||
}] : []),
|
||||
...(wasRunning && runtimeMutationStarted ? [{
|
||||
run: () => dependencies.runtime.restoreRunning(),
|
||||
run: async () => {
|
||||
await dependencies.runtime.restoreRunning();
|
||||
await dependencies.restoreAppliedActivation?.(previousState);
|
||||
},
|
||||
runtime: true,
|
||||
}] : []),
|
||||
...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []),
|
||||
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
type StateSnapshot,
|
||||
type StoredState,
|
||||
} from '../../../shared/contracts/state.js';
|
||||
import type { FailoverSnapshot } from '../../../shared/failover.js';
|
||||
|
||||
interface RuntimeState {
|
||||
running?: boolean;
|
||||
@@ -26,6 +27,7 @@ interface StateServiceDependencies {
|
||||
getGatewayAutoState: () => GatewayAutoState;
|
||||
getOperationState: () => OperationState;
|
||||
configExists: () => boolean;
|
||||
getFailoverSnapshot?: () => FailoverSnapshot;
|
||||
}
|
||||
|
||||
function subscriptionHost(value: unknown) {
|
||||
@@ -51,6 +53,7 @@ export function createStateService(dependencies: StateServiceDependencies) {
|
||||
configExists,
|
||||
subscriptionHost: subscriptionHost(storedState.subscriptionUrl),
|
||||
operation: dependencies.getOperationState(),
|
||||
failoverSnapshot: dependencies.getFailoverSnapshot?.(),
|
||||
});
|
||||
return { snapshot, storedState, gatewayAuto, configExists };
|
||||
},
|
||||
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
type StoredState,
|
||||
} from '../../../shared/contracts/state.js';
|
||||
import { HarborError } from '../../../shared/errors.js';
|
||||
import type { ActivityJournalEventInput } from '../../../shared/activityJournal.js';
|
||||
import { finishRollback } from '../../services/rollback.js';
|
||||
|
||||
interface ParsedSubscription {
|
||||
@@ -55,6 +56,11 @@ interface SubscriptionServiceDependencies {
|
||||
clearInterval(handle: TimerHandle): void;
|
||||
};
|
||||
onRefreshError(error: unknown): void;
|
||||
onEvent?: (event: ActivityJournalEventInput) => void;
|
||||
failover?: {
|
||||
reconcile(): Promise<unknown>;
|
||||
restoreAppliedActivation(state: StoredState): Promise<unknown>;
|
||||
};
|
||||
now?: () => Date;
|
||||
}
|
||||
|
||||
@@ -106,6 +112,10 @@ function mutationResult(profile: Pick<StoredProfile, 'id' | 'label'>): ProfileMu
|
||||
return { success: true, profileId: profile.id, label: profile.label };
|
||||
}
|
||||
|
||||
function publicHost(url: string) {
|
||||
try { return new URL(url).hostname; } catch { return ''; }
|
||||
}
|
||||
|
||||
export function createSubscriptionService(dependencies: SubscriptionServiceDependencies) {
|
||||
const refreshPromises = new Map<string, Promise<ProfileMutationResult>>();
|
||||
let refreshTimer: TimerHandle | null = null;
|
||||
@@ -166,6 +176,13 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen
|
||||
profiles: [...current.profiles, profile],
|
||||
desiredProfileId: current.profiles.length ? current.desiredProfileId : profile.id,
|
||||
}));
|
||||
dependencies.onEvent?.({
|
||||
type: 'subscription.added',
|
||||
severity: 'info',
|
||||
source: 'subscription',
|
||||
dedupeKey: `subscription.added:${dependencies.state.read().revision}`,
|
||||
data: { profileId: profile.id, profileLabel: profile.label, host: publicHost(profile.subscriptionUrl), serverCount: profile.servers.length },
|
||||
});
|
||||
return mutationResult(profile);
|
||||
});
|
||||
};
|
||||
@@ -217,6 +234,7 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen
|
||||
profileId: string,
|
||||
subscriptionUrl: string,
|
||||
error: unknown,
|
||||
origin: 'manual' | 'scheduled',
|
||||
) => dependencies.serialize(async () => {
|
||||
const state = dependencies.state.read();
|
||||
const profile = requireProfile(state, profileId);
|
||||
@@ -230,12 +248,27 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen
|
||||
...current,
|
||||
profiles: replaceProfile(current, failed),
|
||||
}));
|
||||
dependencies.onEvent?.({
|
||||
type: 'subscription.refresh_failed',
|
||||
severity: 'warning',
|
||||
source: 'subscription',
|
||||
dedupeKey: origin === 'scheduled'
|
||||
? `subscription.refresh_failed:${failed.id}:${failed.fetchedAt || 'never'}:${failed.lastRefreshErrorCode}`
|
||||
: `subscription.refresh_failed:${dependencies.state.read().revision}`,
|
||||
data: {
|
||||
profileId: failed.id,
|
||||
profileLabel: failed.label,
|
||||
host: publicHost(failed.subscriptionUrl),
|
||||
errorCode: failed.lastRefreshErrorCode || 'UNKNOWN',
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
const commitRefresh = (
|
||||
profileId: string,
|
||||
subscriptionUrl: string,
|
||||
parsed: ParsedSubscription,
|
||||
origin: 'manual' | 'scheduled',
|
||||
) => dependencies.serialize(async () => {
|
||||
// Re-read the profile after provider I/O and guard its owner instead of rejecting background-only revisions.
|
||||
const previousState = dependencies.state.read();
|
||||
@@ -257,8 +290,46 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen
|
||||
lastRefreshAttemptAt: parsed.fetchedAt,
|
||||
lastRefreshErrorCode: null,
|
||||
};
|
||||
const contentChanged = JSON.stringify({
|
||||
subscriptionConfig: previousProfile.subscriptionConfig,
|
||||
servers: previousProfile.servers,
|
||||
userInfo: previousProfile.userInfo,
|
||||
}) !== JSON.stringify({
|
||||
subscriptionConfig: refreshedProfile.subscriptionConfig,
|
||||
servers: refreshedProfile.servers,
|
||||
userInfo: refreshedProfile.userInfo,
|
||||
});
|
||||
const appendRefreshEvent = () => {
|
||||
if (origin === 'scheduled' && !contentChanged && !previousProfile.lastRefreshErrorCode) return;
|
||||
dependencies.onEvent?.({
|
||||
type: 'subscription.refreshed',
|
||||
severity: 'info',
|
||||
source: 'subscription',
|
||||
dedupeKey: `subscription.refreshed:${dependencies.state.read().revision}`,
|
||||
data: {
|
||||
profileId: refreshedProfile.id,
|
||||
profileLabel: refreshedProfile.label,
|
||||
host: publicHost(refreshedProfile.subscriptionUrl),
|
||||
serverCount: refreshedProfile.servers.length,
|
||||
added: refreshedProfile.servers.filter(({ id }) => !previousProfile.servers.some((server) => server.id === id)).length,
|
||||
removed: previousProfile.servers.filter(({ id }) => !refreshedProfile.servers.some((server) => server.id === id)).length,
|
||||
},
|
||||
});
|
||||
};
|
||||
const loadedTargets = previousState.appliedFailoverPolicy
|
||||
? [previousState.appliedFailoverPolicy.primary, previousState.appliedFailoverPolicy.reserve]
|
||||
: [];
|
||||
const missingLoadedTarget = loadedTargets.some((target) => (
|
||||
target.profileId === profileId
|
||||
&& !refreshedProfile.servers.some(({ id }) => id === target.serverId)
|
||||
));
|
||||
const pauseFailover = previousState.failoverPolicy?.enabled
|
||||
&& !previousState.failoverPolicy.paused
|
||||
&& missingLoadedTarget;
|
||||
const running = await dependencies.runtime.isRunning();
|
||||
const refreshesApplied = running && previousState.appliedProfileId === profileId;
|
||||
const refreshesApplied = running
|
||||
&& previousState.appliedProfileId === profileId
|
||||
&& !previousState.appliedFailoverPolicy;
|
||||
const nextAppliedServerId = refreshesApplied
|
||||
? dependencies.provider.selectRefreshedServer(
|
||||
previousState.appliedServerId,
|
||||
@@ -271,7 +342,17 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen
|
||||
dependencies.state.update((current) => ({
|
||||
...current,
|
||||
profiles: replaceProfile(current, refreshedProfile),
|
||||
...(pauseFailover ? { failoverPolicy: { ...current.failoverPolicy, paused: true } } : {}),
|
||||
}));
|
||||
if (pauseFailover) dependencies.onEvent?.({
|
||||
type: 'failover.paused',
|
||||
severity: 'warning',
|
||||
source: 'failover',
|
||||
dedupeKey: `failover.paused:missing-target:${profileId}:${dependencies.state.read().revision}`,
|
||||
data: {},
|
||||
});
|
||||
await dependencies.failover?.reconcile();
|
||||
appendRefreshEvent();
|
||||
return mutationResult(refreshedProfile);
|
||||
}
|
||||
|
||||
@@ -305,13 +386,25 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen
|
||||
await finishRollback(error, [
|
||||
...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []),
|
||||
...(configMutationStarted ? [{ run: () => restoreConfig(previousConfig) }] : []),
|
||||
...(runtimeMutationStarted ? [{ run: () => dependencies.runtime.start(), runtime: true }] : []),
|
||||
...(runtimeMutationStarted ? [{
|
||||
run: async () => {
|
||||
await dependencies.runtime.start();
|
||||
await dependencies.failover?.restoreAppliedActivation(previousState);
|
||||
},
|
||||
runtime: true,
|
||||
}] : []),
|
||||
], 'Subscription refresh rollback failed');
|
||||
}
|
||||
await dependencies.failover?.reconcile();
|
||||
appendRefreshEvent();
|
||||
return mutationResult(refreshedProfile);
|
||||
});
|
||||
|
||||
const refreshProfile = (profileIdValue: unknown, expectedRevision?: unknown) => {
|
||||
const refreshProfile = (
|
||||
profileIdValue: unknown,
|
||||
expectedRevision?: unknown,
|
||||
origin: 'manual' | 'scheduled' = 'manual',
|
||||
) => {
|
||||
const profileId = String(profileIdValue || '').trim();
|
||||
const existing = refreshPromises.get(profileId);
|
||||
if (existing) return existing;
|
||||
@@ -328,6 +421,7 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen
|
||||
profileId,
|
||||
initialProfile.subscriptionUrl,
|
||||
error,
|
||||
origin,
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
@@ -336,6 +430,7 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen
|
||||
profileId,
|
||||
initialProfile.subscriptionUrl,
|
||||
parsed,
|
||||
origin,
|
||||
);
|
||||
})().finally(() => refreshPromises.delete(profileId));
|
||||
refreshPromises.set(profileId, operation);
|
||||
@@ -353,12 +448,18 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen
|
||||
const mode = String(modeValue || 'delete');
|
||||
if (!['delete', 'stop-and-delete'].includes(mode)) throw new HarborError('REQUEST_INVALID');
|
||||
const running = await dependencies.runtime.isRunning();
|
||||
const applied = running && previousState.appliedProfileId === profile.id;
|
||||
const failoverReferencesProfile = Boolean(previousState.appliedFailoverPolicy && (
|
||||
previousState.appliedFailoverPolicy.primary.profileId === profile.id
|
||||
|| previousState.appliedFailoverPolicy.reserve.profileId === profile.id
|
||||
));
|
||||
const desiredFailoverReferencesProfile = previousState.failoverPolicy?.primary.profileId === profile.id
|
||||
|| previousState.failoverPolicy?.reserve.profileId === profile.id;
|
||||
const applied = running && (previousState.appliedProfileId === profile.id || failoverReferencesProfile);
|
||||
if (applied && mode !== 'stop-and-delete') throw new HarborError('PROFILE_IN_USE');
|
||||
|
||||
const previousConfig = dependencies.config.read();
|
||||
const previousGatewayAuto = dependencies.gatewayAuto.read();
|
||||
const removesAppliedTarget = previousState.appliedProfileId === profile.id;
|
||||
const removesAppliedTarget = previousState.appliedProfileId === profile.id || failoverReferencesProfile;
|
||||
let runtimeMutationStarted = false;
|
||||
let configMutationStarted = false;
|
||||
let gatewayMutationStarted = false;
|
||||
@@ -390,16 +491,49 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen
|
||||
appliedServerSnapshot: current.appliedProfileId === profile.id
|
||||
? null
|
||||
: current.appliedServerSnapshot,
|
||||
...(current.appliedProfileId === profile.id ? { connectionDesired: 'stopped' } : {}),
|
||||
...(removesAppliedTarget ? {
|
||||
connectionDesired: 'stopped',
|
||||
appliedProfileId: '',
|
||||
appliedServerId: '',
|
||||
appliedServerSnapshot: null,
|
||||
appliedFailoverPolicy: null,
|
||||
} : {}),
|
||||
...(failoverReferencesProfile || desiredFailoverReferencesProfile ? {
|
||||
failoverPolicy: {
|
||||
...current.failoverPolicy,
|
||||
enabled: false,
|
||||
paused: false,
|
||||
primary: current.failoverPolicy.primary.profileId === profile.id
|
||||
? { profileId: '', serverId: '' }
|
||||
: current.failoverPolicy.primary,
|
||||
reserve: current.failoverPolicy.reserve.profileId === profile.id
|
||||
? { profileId: '', serverId: '' }
|
||||
: current.failoverPolicy.reserve,
|
||||
},
|
||||
} : {}),
|
||||
}));
|
||||
} catch (error) {
|
||||
await finishRollback(error, [
|
||||
...(stateCommitStarted ? [{ run: () => dependencies.state.update(() => previousState) }] : []),
|
||||
...(gatewayMutationStarted ? [{ run: () => dependencies.gatewayAuto.set(previousGatewayAuto) }] : []),
|
||||
...(configMutationStarted ? [{ run: () => restoreConfig(previousConfig) }] : []),
|
||||
...(runtimeMutationStarted ? [{ run: () => dependencies.runtime.start(), runtime: true }] : []),
|
||||
...(runtimeMutationStarted ? [{
|
||||
run: async () => {
|
||||
await dependencies.runtime.start();
|
||||
await dependencies.failover?.restoreAppliedActivation(previousState);
|
||||
},
|
||||
runtime: true,
|
||||
}] : []),
|
||||
], 'Subscription delete rollback failed');
|
||||
}
|
||||
await dependencies.failover?.reconcile();
|
||||
dependencies.onEvent?.({
|
||||
type: 'subscription.deleted',
|
||||
severity: 'info',
|
||||
source: 'subscription',
|
||||
dedupeKey: `subscription.deleted:${dependencies.state.read().revision}`,
|
||||
data: { profileId: profile.id, profileLabel: profile.label },
|
||||
});
|
||||
return mutationResult(profile);
|
||||
});
|
||||
|
||||
@@ -436,7 +570,7 @@ export function createSubscriptionService(dependencies: SubscriptionServiceDepen
|
||||
void (async () => {
|
||||
for (const { id } of dependencies.state.read().profiles) {
|
||||
try {
|
||||
await refreshProfile(id);
|
||||
await refreshProfile(id, undefined, 'scheduled');
|
||||
} catch (error) {
|
||||
dependencies.onRefreshError(error);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
import type { IncomingMessage, ServerResponse } from 'node:http';
|
||||
import type { ActivityJournalService } from '../../services/activityJournalService.js';
|
||||
import { sendJson } from '../response.js';
|
||||
|
||||
export function createActivityJournalRoute({ journal }: { journal: ActivityJournalService }) {
|
||||
return {
|
||||
async handle(req: IncomingMessage, res: ServerResponse) {
|
||||
const url = new URL(req.url || '/', 'http://localhost');
|
||||
if (url.pathname !== '/api/activity-journal' || req.method !== 'GET') return false;
|
||||
sendJson(res, 200, journal.page(
|
||||
Number(url.searchParams.get('limit')) || 50,
|
||||
url.searchParams.get('cursor'),
|
||||
));
|
||||
return true;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
import type { IncomingMessage, ServerResponse } from 'node:http';
|
||||
import type { FailoverService } from '../../features/failover/failoverService.js';
|
||||
import { HarborError } from '../../../shared/errors.js';
|
||||
|
||||
interface FailoverRouteDependencies {
|
||||
appMode: string;
|
||||
failover: Pick<FailoverService, 'save' | 'pause' | 'manualSwitch' | 'checkNow'>;
|
||||
readBody(req: IncomingMessage): Promise<Record<string, unknown>>;
|
||||
withOperation<T>(kind: string, operation: () => Promise<T>, options?: { expectedRevision?: unknown }): Promise<T>;
|
||||
sendState(res: ServerResponse): Promise<void>;
|
||||
}
|
||||
|
||||
export function createFailoverRoute(dependencies: FailoverRouteDependencies) {
|
||||
return {
|
||||
async handle(req: IncomingMessage, res: ServerResponse) {
|
||||
const pathname = new URL(req.url || '/', 'http://localhost').pathname;
|
||||
if (!pathname.startsWith('/api/failover')) return false;
|
||||
if (dependencies.appMode !== 'gateway') throw new HarborError('ENDPOINT_NOT_FOUND');
|
||||
const body = await dependencies.readBody(req);
|
||||
if (pathname === '/api/failover' && req.method === 'PUT') {
|
||||
await dependencies.withOperation(
|
||||
'failover-save',
|
||||
() => dependencies.failover.save(body.policy),
|
||||
{ expectedRevision: body.expectedRevision },
|
||||
);
|
||||
} else if (pathname === '/api/failover/pause' && req.method === 'POST') {
|
||||
if (typeof body.paused !== 'boolean') throw new HarborError('REQUEST_INVALID');
|
||||
await dependencies.withOperation(
|
||||
body.paused ? 'failover-pause' : 'failover-resume',
|
||||
() => dependencies.failover.pause(body.paused as boolean),
|
||||
{ expectedRevision: body.expectedRevision },
|
||||
);
|
||||
} else if (pathname === '/api/failover/switch' && req.method === 'POST') {
|
||||
if (body.role !== 'primary' && body.role !== 'reserve') throw new HarborError('REQUEST_INVALID');
|
||||
await dependencies.withOperation(
|
||||
'failover-switch',
|
||||
() => dependencies.failover.manualSwitch(body.role as 'primary' | 'reserve'),
|
||||
{ expectedRevision: body.expectedRevision },
|
||||
);
|
||||
} else if (pathname === '/api/failover/check' && req.method === 'POST') {
|
||||
await dependencies.failover.checkNow();
|
||||
} else {
|
||||
throw new HarborError('ENDPOINT_NOT_FOUND');
|
||||
}
|
||||
await dependencies.sendState(res);
|
||||
return true;
|
||||
},
|
||||
};
|
||||
}
|
||||
+204
-10
@@ -16,7 +16,10 @@ import {
|
||||
import { createSingboxRuntime } from './singboxRuntime.js';
|
||||
import { tcpPing } from './ping.js';
|
||||
import {
|
||||
buildDualChannelGatewayConfig,
|
||||
buildGatewayConfig,
|
||||
dualChannelConfigMatchesApplied,
|
||||
fingerprintSelectedOutbound,
|
||||
removeSingboxConfig,
|
||||
restoreSingboxConfig,
|
||||
writeSingboxConfig,
|
||||
@@ -81,6 +84,13 @@ import { createConnectivityDiagnosticsRoute } from './http/routes/connectivityDi
|
||||
import { createGatewayPresenceRoute } from './http/routes/gatewayPresenceRoute.js';
|
||||
import { createSharedProxyRoute } from './http/routes/sharedProxyRoute.js';
|
||||
import { createVersionRoute } from './http/routes/versionRoute.js';
|
||||
import { createSingboxSelectorService } from './services/singboxSelectorService.js';
|
||||
import { createFailoverService } from './features/failover/failoverService.js';
|
||||
import { createFailoverRoute } from './http/routes/failoverRoute.js';
|
||||
import { createActivityJournalService } from './services/activityJournalService.js';
|
||||
import { createActivityJournalRoute } from './http/routes/activityJournalRoute.js';
|
||||
import { createDomainTrafficService, readSingboxConnections } from './services/domainTrafficService.js';
|
||||
import type { ActivityJournalEventInput } from '../shared/activityJournal.js';
|
||||
|
||||
const MAX_BODY_BYTES = 1_000_000;
|
||||
const SUBSCRIPTION_REFRESH_INTERVAL_MS = 15 * 60 * 1000;
|
||||
@@ -142,6 +152,14 @@ if (legacyCacheOwnerMismatch) {
|
||||
}
|
||||
}
|
||||
const stateStore = createStateStore(settings.statePath, { legacySubscriptionCache });
|
||||
const activityJournal = createActivityJournalService({ filePath: settings.activityJournalPath });
|
||||
const appendJournal = (event: ActivityJournalEventInput) => {
|
||||
try {
|
||||
activityJournal.append(event);
|
||||
} catch (error) {
|
||||
console.warn(`[journal] событие не сохранено: ${errorMessage(error)}`);
|
||||
}
|
||||
};
|
||||
const deviceStore = createJsonStore<InventoryState>({
|
||||
filePath: settings.deviceStatePath,
|
||||
defaultValue: migrateDeviceInventoryState({}),
|
||||
@@ -283,6 +301,37 @@ const deviceInventory = settings.appMode === 'gateway'
|
||||
const localConnectivityDiagnostics = !remoteDataplane
|
||||
? createConnectivityDiagnosticsService({ proxyPort: settings.diagnosticsProxyPort })
|
||||
: null;
|
||||
const localFailoverDiagnostics = !remoteDataplane ? {
|
||||
primary: createConnectivityDiagnosticsService({ proxyPort: settings.failoverPrimaryProxyPort }),
|
||||
reserve: createConnectivityDiagnosticsService({ proxyPort: settings.failoverReserveProxyPort }),
|
||||
} : null;
|
||||
const localSelector = !remoteDataplane && settings.appMode === 'gateway'
|
||||
? createSingboxSelectorService({ port: settings.singboxApiPort })
|
||||
: null;
|
||||
const localFailoverTraffic = !remoteDataplane && settings.appMode === 'gateway'
|
||||
? createDomainTrafficService({
|
||||
observe: () => readSingboxConnections(settings.singboxApiPort),
|
||||
devices: () => record(deviceInventory?.snapshot()).devices,
|
||||
})
|
||||
: null;
|
||||
let localFailoverTrafficTimer: NodeJS.Timeout | null = null;
|
||||
|
||||
function setLocalFailoverActivityEnabled(enabled: boolean) {
|
||||
if (!localFailoverTraffic) throw new Error('Failover activity недоступна');
|
||||
if (!enabled) {
|
||||
if (localFailoverTrafficTimer) clearInterval(localFailoverTrafficTimer);
|
||||
localFailoverTrafficTimer = null;
|
||||
localFailoverTraffic.disableActivity();
|
||||
return;
|
||||
}
|
||||
localFailoverTraffic.enableActivity();
|
||||
if (localFailoverTrafficTimer) return;
|
||||
const refresh = () => localFailoverTraffic.refresh()
|
||||
.catch((error: unknown) => console.warn(`[control] failover activity: ${errorMessage(error)}`));
|
||||
void refresh();
|
||||
localFailoverTrafficTimer = setInterval(refresh, 2_000);
|
||||
localFailoverTrafficTimer.unref();
|
||||
}
|
||||
|
||||
function requireLocalConnectivityDiagnostics() {
|
||||
if (!localConnectivityDiagnostics) throw new Error('Harbor local diagnostics are not configured');
|
||||
@@ -355,6 +404,48 @@ const gatewayAutoService = createGatewayAutoService({
|
||||
onDiscoveryWarning: (reason) => console.warn(`[control] Gateway не используется: ${reason}`),
|
||||
onTimerError: (error) => console.warn(`[control] Gateway detection failed: ${errorMessage(error)}`),
|
||||
});
|
||||
const failoverDataplane = remoteDataplane ? {
|
||||
checkConfig: (config: unknown) => requireRemoteRuntime().checkConfig(config),
|
||||
runFailoverProbe: (role: 'primary' | 'reserve', services: unknown, target: unknown, timeoutMs: number) => (
|
||||
requireRemoteRuntime().runFailoverProbe(role, services, target, timeoutMs)
|
||||
),
|
||||
readFailoverSelector: () => requireRemoteRuntime().readFailoverSelector(),
|
||||
selectFailoverRole: (role: 'primary' | 'reserve') => requireRemoteRuntime().selectFailoverRole(role),
|
||||
setFailoverActivityEnabled: (enabled: boolean) => requireRemoteRuntime().setFailoverActivityEnabled(enabled),
|
||||
readFailoverActivity: (threshold: number) => requireRemoteRuntime().readFailoverActivity(threshold),
|
||||
} : {
|
||||
checkConfig: async (config: unknown) => singboxRuntime.checkConfig(config),
|
||||
runFailoverProbe: async (role: 'primary' | 'reserve', services: unknown, target: unknown, timeoutMs: number) => {
|
||||
if (!localFailoverDiagnostics) throw new Error('Failover diagnostics недоступна');
|
||||
return localFailoverDiagnostics[role].runVpn({ services, target, timeoutMs });
|
||||
},
|
||||
readFailoverSelector: async () => {
|
||||
if (!localSelector) throw new Error('Failover selector недоступен');
|
||||
return localSelector.read();
|
||||
},
|
||||
selectFailoverRole: async (role: 'primary' | 'reserve') => {
|
||||
if (!localSelector) throw new Error('Failover selector недоступен');
|
||||
return localSelector.select(role);
|
||||
},
|
||||
setFailoverActivityEnabled: async (enabled: boolean) => setLocalFailoverActivityEnabled(enabled),
|
||||
readFailoverActivity: async (threshold: number) => ({
|
||||
activity: localFailoverTraffic?.activitySnapshot(threshold) || null,
|
||||
}),
|
||||
};
|
||||
const failoverService = createFailoverService({
|
||||
state: {
|
||||
read: () => normalizeStoredState(stateStore.read()),
|
||||
update: updateStoredState,
|
||||
},
|
||||
runtime: { isRunning: async () => Boolean((await singboxRuntime.refresh()).running) },
|
||||
dataplane: failoverDataplane,
|
||||
buildCandidate: buildFailoverCandidate,
|
||||
serialize: serializeControl,
|
||||
onWarning: (error) => console.warn(`[control] failover: ${errorMessage(error)}`),
|
||||
onSwitch: (from, to, reason) => console.log(`[control] failover ${from} -> ${to}: ${reason}`),
|
||||
onEvent: appendJournal,
|
||||
});
|
||||
const gatewayFailover = settings.appMode === 'gateway' ? failoverService : null;
|
||||
const stateService = createStateService({
|
||||
appMode: settings.appMode,
|
||||
readStoredState: () => stateStore.read(),
|
||||
@@ -362,6 +453,7 @@ const stateService = createStateService({
|
||||
getGatewayAutoState: gatewayAutoService.read,
|
||||
getOperationState: () => operationState,
|
||||
configExists: () => fs.existsSync(settings.configPath),
|
||||
getFailoverSnapshot: failoverService.snapshot,
|
||||
});
|
||||
const stateRoute = createStateRoute({
|
||||
stateService,
|
||||
@@ -375,6 +467,14 @@ const gatewayAutoRoute = createGatewayAutoRoute({
|
||||
withOperation,
|
||||
readStatePayload: stateRoute.readPayload,
|
||||
});
|
||||
const failoverRoute = createFailoverRoute({
|
||||
appMode: settings.appMode,
|
||||
failover: failoverService,
|
||||
readBody,
|
||||
withOperation,
|
||||
sendState: (res) => stateRoute.send(res),
|
||||
});
|
||||
const activityJournalRoute = createActivityJournalRoute({ journal: activityJournal });
|
||||
const deviceInventoryRoute = createDeviceInventoryRoute({
|
||||
deviceInventory,
|
||||
readBody,
|
||||
@@ -427,9 +527,12 @@ const subscriptionService = createSubscriptionService({
|
||||
update: updateStoredState,
|
||||
},
|
||||
config: {
|
||||
build: (subscriptionConfig, selectedServerId, routeRules) => (
|
||||
buildActiveConfig(subscriptionConfig, selectedServerId, routeRules)
|
||||
),
|
||||
build: (subscriptionConfig, selectedServerId, routeRules) => {
|
||||
const state = normalizeStoredState(stateStore.read());
|
||||
return state.appliedFailoverPolicy
|
||||
? buildFailoverCandidate({ ...state, routeRules }, 'applied').config
|
||||
: buildActiveConfig(subscriptionConfig, selectedServerId, routeRules);
|
||||
},
|
||||
read: () => fs.existsSync(settings.configPath)
|
||||
? fs.readFileSync(settings.configPath, 'utf8')
|
||||
: null,
|
||||
@@ -453,6 +556,12 @@ const subscriptionService = createSubscriptionService({
|
||||
clearInterval: (timer) => clearInterval(timer),
|
||||
},
|
||||
onRefreshError: (error) => console.warn(`[control] подписка не обновлена: ${errorMessage(error)}`),
|
||||
onEvent: appendJournal,
|
||||
failover: gatewayFailover ? {
|
||||
reconcile: () => gatewayFailover.reconcile()
|
||||
.catch((error) => console.warn(`[control] failover reconcile: ${errorMessage(error)}`)),
|
||||
restoreAppliedActivation: gatewayFailover.restoreAppliedActivation,
|
||||
} : undefined,
|
||||
now: () => new Date(),
|
||||
});
|
||||
const serverHealthRoute = createServerHealthRoute({
|
||||
@@ -484,6 +593,14 @@ const connectionService = createConnectionService({
|
||||
isGatewayDirect: () => settings.appMode === 'client'
|
||||
&& gatewayAutoService.read().mode === 'gateway-direct',
|
||||
},
|
||||
failover: gatewayFailover ? {
|
||||
build: buildFailoverCandidate,
|
||||
prepareActivation: gatewayFailover.prepareActivation,
|
||||
restoreAppliedActivation: gatewayFailover.restoreAppliedActivation,
|
||||
reconcile: () => gatewayFailover.reconcile()
|
||||
.catch((error) => console.warn(`[control] failover reconcile: ${errorMessage(error)}`)),
|
||||
} : undefined,
|
||||
onEvent: appendJournal,
|
||||
runtime: {
|
||||
isRunning: async () => Boolean((await singboxRuntime.refresh()).running),
|
||||
start: () => startSingbox(),
|
||||
@@ -524,9 +641,12 @@ const routeRulesService = createRouteRulesService({
|
||||
readConfig: (profileId) => readProfileConfig(profileId),
|
||||
},
|
||||
config: {
|
||||
build: (subscriptionConfig, selectedServerId, routeRules) => (
|
||||
buildActiveConfig(subscriptionConfig, selectedServerId, routeRules)
|
||||
),
|
||||
build: (subscriptionConfig, selectedServerId, routeRules) => {
|
||||
const state = normalizeStoredState(stateStore.read());
|
||||
return state.appliedFailoverPolicy
|
||||
? buildFailoverCandidate({ ...state, routeRules }, 'applied').config
|
||||
: buildActiveConfig(subscriptionConfig, selectedServerId, routeRules);
|
||||
},
|
||||
read: () => fs.existsSync(settings.configPath)
|
||||
? fs.readFileSync(settings.configPath, 'utf8')
|
||||
: null,
|
||||
@@ -548,6 +668,9 @@ const routeRulesService = createRouteRulesService({
|
||||
},
|
||||
serialize: serializeControl,
|
||||
runOperation: (operation) => withOperation('route-rules', operation),
|
||||
afterApply: gatewayFailover ? () => gatewayFailover.reconcile()
|
||||
.catch((error) => console.warn(`[control] failover reconcile: ${errorMessage(error)}`)) : undefined,
|
||||
restoreAppliedActivation: gatewayFailover?.restoreAppliedActivation,
|
||||
});
|
||||
const routeRulesRoute = createRouteRulesRoute({
|
||||
routeRules: routeRulesService,
|
||||
@@ -676,6 +799,47 @@ function buildActiveConfig(
|
||||
});
|
||||
}
|
||||
|
||||
function buildFailoverCandidate(state: StoredState, source: 'desired' | 'applied' = 'desired') {
|
||||
const policy = source === 'applied' ? state.appliedFailoverPolicy : state.failoverPolicy;
|
||||
if (!policy) throw new HarborError('CONFIG_INVALID');
|
||||
if (
|
||||
policy.primary.profileId === policy.reserve.profileId
|
||||
&& policy.primary.serverId === policy.reserve.serverId
|
||||
) throw new HarborError('REQUEST_INVALID');
|
||||
const channel = (role: 'primary' | 'reserve') => {
|
||||
const target = policy[role];
|
||||
const profile = state.profiles.find(({ id }) => id === target.profileId);
|
||||
const server = profile?.servers.find(({ id }) => id === target.serverId);
|
||||
if (!profile || !server || !profile.subscriptionConfig) throw new HarborError('SERVER_NOT_FOUND');
|
||||
return { profile, server };
|
||||
};
|
||||
const primary = channel('primary');
|
||||
const reserve = channel('reserve');
|
||||
const applied = {
|
||||
primary: policy.primary,
|
||||
reserve: policy.reserve,
|
||||
primaryConfigFingerprint: fingerprintSelectedOutbound(primary.profile.subscriptionConfig, primary.server.id),
|
||||
reserveConfigFingerprint: fingerprintSelectedOutbound(reserve.profile.subscriptionConfig, reserve.server.id),
|
||||
};
|
||||
if (source === 'applied' && JSON.stringify(applied) !== JSON.stringify(state.appliedFailoverPolicy)) {
|
||||
throw new HarborError('CONFIG_INVALID');
|
||||
}
|
||||
const defaultRole = source === 'applied'
|
||||
&& state.appliedProfileId === policy.reserve.profileId
|
||||
&& state.appliedServerId === policy.reserve.serverId
|
||||
? 'reserve'
|
||||
: 'primary';
|
||||
return {
|
||||
config: buildDualChannelGatewayConfig({
|
||||
primary: { subscriptionConfig: primary.profile.subscriptionConfig, selectedServerId: primary.server.id },
|
||||
reserve: { subscriptionConfig: reserve.profile.subscriptionConfig, selectedServerId: reserve.server.id },
|
||||
}, { routeRules: state.routeRules, defaultRole }),
|
||||
applied,
|
||||
primaryProfile: primary.profile,
|
||||
primaryServer: primary.server,
|
||||
};
|
||||
}
|
||||
|
||||
const stopSingbox = () => singboxRuntime.stop();
|
||||
const startSingbox = () => singboxRuntime.apply();
|
||||
|
||||
@@ -689,7 +853,13 @@ function writeCurrentConfig() {
|
||||
const server = profile?.servers.find((candidate) => candidate.id === serverId);
|
||||
const subscriptionConfig = profile ? readProfileConfig(profile.id) : null;
|
||||
if (!profile || !server || !subscriptionConfig) return null;
|
||||
const activeConfig = buildActiveConfig(subscriptionConfig, server.id);
|
||||
let activeConfig: unknown;
|
||||
if (state.appliedFailoverPolicy) {
|
||||
const candidate = buildFailoverCandidate(state, 'applied');
|
||||
activeConfig = candidate.config;
|
||||
} else {
|
||||
activeConfig = buildActiveConfig(subscriptionConfig, server.id);
|
||||
}
|
||||
const previousConfig = fs.existsSync(settings.configPath)
|
||||
? fs.readFileSync(settings.configPath, 'utf8')
|
||||
: null;
|
||||
@@ -704,7 +874,7 @@ function writeCurrentConfig() {
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
return { profile, server };
|
||||
return { profile, server, failoverApplied: state.appliedFailoverPolicy };
|
||||
}
|
||||
|
||||
const CONFIG_PROXY_TYPES = new Set(['vless', 'vmess', 'trojan', 'shadowsocks', 'hysteria2']);
|
||||
@@ -717,6 +887,13 @@ function currentConfigMatchesAppliedTarget(state: StoredState) {
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
if (state.appliedFailoverPolicy) {
|
||||
const expectedRole = state.appliedProfileId === state.appliedFailoverPolicy.reserve.profileId
|
||||
&& state.appliedServerId === state.appliedFailoverPolicy.reserve.serverId
|
||||
? 'reserve'
|
||||
: 'primary';
|
||||
return dualChannelConfigMatchesApplied(config, state.appliedFailoverPolicy, expectedRole);
|
||||
}
|
||||
const proxyOutbounds = (Array.isArray(config.outbounds) ? config.outbounds : [])
|
||||
.map(record)
|
||||
.filter((outbound) => CONFIG_PROXY_TYPES.has(String(outbound.type || '')));
|
||||
@@ -750,6 +927,7 @@ async function reconcileStoppedBoot({ removeConfig = false } = {}) {
|
||||
|| state.appliedProfileId
|
||||
|| state.appliedServerId
|
||||
|| state.appliedServerSnapshot
|
||||
|| state.appliedFailoverPolicy
|
||||
) {
|
||||
updateStoredState((current) => ({
|
||||
...current,
|
||||
@@ -757,6 +935,7 @@ async function reconcileStoppedBoot({ removeConfig = false } = {}) {
|
||||
appliedProfileId: '',
|
||||
appliedServerId: '',
|
||||
appliedServerSnapshot: null,
|
||||
appliedFailoverPolicy: null,
|
||||
}));
|
||||
}
|
||||
}
|
||||
@@ -770,6 +949,8 @@ async function handleApi(req: IncomingMessage, res: ServerResponse) {
|
||||
if (await connectionRuntimeRoute.handle(req, res)) return;
|
||||
if (await routeRulesRoute.handle(req, res)) return;
|
||||
if (await gatewayAutoRoute.handle(req, res)) return;
|
||||
if (await failoverRoute.handle(req, res)) return;
|
||||
if (await activityJournalRoute.handle(req, res)) return;
|
||||
if (await connectivityDiagnosticsRoute.handle(req, res)) return;
|
||||
|
||||
if (await versionRoute.handle(req, res)) return;
|
||||
@@ -822,6 +1003,7 @@ async function shutdown() {
|
||||
subscriptionService.stopAutoRefresh();
|
||||
gatewayAutoService.stopDiscovery();
|
||||
if (deviceDiscoveryTimer) clearInterval(deviceDiscoveryTimer);
|
||||
await gatewayFailover?.shutdown().catch((error) => console.warn(`[control] failover shutdown: ${errorMessage(error)}`));
|
||||
await serializeControl(() => singboxRuntime.shutdown());
|
||||
process.exit(0);
|
||||
}
|
||||
@@ -846,8 +1028,14 @@ if (bootWantsRunning) {
|
||||
&& currentConfigMatchesAppliedTarget(normalizeStoredState(stateStore.read()));
|
||||
if (target || canReuseCurrentConfig) {
|
||||
await startSingbox()
|
||||
.then(() => {
|
||||
.then(async () => {
|
||||
const current = normalizeStoredState(stateStore.read());
|
||||
const bootRole = current.appliedFailoverPolicy
|
||||
&& current.appliedProfileId === current.appliedFailoverPolicy.reserve.profileId
|
||||
&& current.appliedServerId === current.appliedFailoverPolicy.reserve.serverId
|
||||
? 'reserve'
|
||||
: current.appliedFailoverPolicy ? 'primary' : null;
|
||||
if (bootRole) await failoverDataplane.selectFailoverRole(bootRole);
|
||||
const appliedProfile = target?.profile
|
||||
|| current.profiles.find((profile) => profile.id === current.appliedProfileId);
|
||||
const appliedServer = target?.server
|
||||
@@ -866,7 +1054,10 @@ if (bootWantsRunning) {
|
||||
}));
|
||||
}
|
||||
})
|
||||
.catch((error: unknown) => console.warn(`[control] sing-box не запущен: ${errorMessage(error)}`));
|
||||
.catch(async (error: unknown) => {
|
||||
console.warn(`[control] sing-box не запущен: ${errorMessage(error)}`);
|
||||
await reconcileStoppedBoot();
|
||||
});
|
||||
} else {
|
||||
await reconcileStoppedBoot({ removeConfig: true });
|
||||
}
|
||||
@@ -874,6 +1065,9 @@ if (bootWantsRunning) {
|
||||
await reconcileStoppedBoot();
|
||||
}
|
||||
|
||||
await gatewayFailover?.reconcile()
|
||||
.catch((error) => console.warn(`[control] failover reconcile: ${errorMessage(error)}`));
|
||||
|
||||
if (deviceInventory) {
|
||||
await deviceInventory.reconcilePolicies()
|
||||
.catch((error: unknown) => console.warn(`[control] device policy не применена: ${errorMessage(error)}`));
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
import crypto from 'node:crypto';
|
||||
import fs from 'node:fs';
|
||||
import {
|
||||
ACTIVITY_JOURNAL_MAX_EVENTS,
|
||||
ACTIVITY_JOURNAL_RETENTION_DAYS,
|
||||
normalizeActivityEventInput,
|
||||
normalizeStoredActivityEvent,
|
||||
type ActivityJournalEvent,
|
||||
type ActivityJournalEventInput,
|
||||
type ActivityJournalPage,
|
||||
} from '../../shared/activityJournal.js';
|
||||
import { createJsonStore } from './stateStore.js';
|
||||
|
||||
interface JournalState {
|
||||
schemaVersion: 1;
|
||||
events: ActivityJournalEvent[];
|
||||
}
|
||||
|
||||
const migrateJournal = (value: unknown): JournalState => {
|
||||
const candidate = value && typeof value === 'object' && !Array.isArray(value)
|
||||
? value as Record<string, unknown>
|
||||
: {};
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
events: (Array.isArray(candidate.events) ? candidate.events : [])
|
||||
.map(normalizeStoredActivityEvent)
|
||||
.filter((event): event is ActivityJournalEvent => Boolean(event)),
|
||||
};
|
||||
};
|
||||
|
||||
export function createActivityJournalService({
|
||||
filePath,
|
||||
now = () => new Date(),
|
||||
}: {
|
||||
filePath: string;
|
||||
now?: () => Date;
|
||||
}) {
|
||||
const store = createJsonStore<JournalState>({
|
||||
filePath,
|
||||
defaultValue: { schemaVersion: 1, events: [] },
|
||||
migrate: migrateJournal,
|
||||
});
|
||||
let recoveryRecorded = false;
|
||||
let writeFailed = false;
|
||||
|
||||
function retained(events: ActivityJournalEvent[]) {
|
||||
const cutoff = now().getTime() - ACTIVITY_JOURNAL_RETENTION_DAYS * 86_400_000;
|
||||
return events
|
||||
.filter(({ occurredAt }) => Date.parse(occurredAt) >= cutoff)
|
||||
.slice(-ACTIVITY_JOURNAL_MAX_EVENTS);
|
||||
}
|
||||
|
||||
function append(value: ActivityJournalEventInput) {
|
||||
const input = normalizeActivityEventInput(value);
|
||||
const storedInput = input.dedupeKey ? {
|
||||
...input,
|
||||
dedupeKey: `${input.type}:sha256:${crypto.createHash('sha256').update(input.dedupeKey).digest('hex')}`,
|
||||
} : input;
|
||||
let appended: ActivityJournalEvent | null = null;
|
||||
try {
|
||||
store.update((state) => {
|
||||
const events = retained(state.events);
|
||||
if (storedInput.dedupeKey && events.some(({ dedupeKey }) => dedupeKey === storedInput.dedupeKey)) {
|
||||
return { schemaVersion: 1, events };
|
||||
}
|
||||
appended = {
|
||||
id: crypto.randomUUID(),
|
||||
occurredAt: now().toISOString(),
|
||||
...storedInput,
|
||||
};
|
||||
return { schemaVersion: 1, events: retained([...events, appended]) };
|
||||
});
|
||||
writeFailed = false;
|
||||
} catch (error) {
|
||||
writeFailed = true;
|
||||
throw error;
|
||||
}
|
||||
return appended;
|
||||
}
|
||||
|
||||
function ensureRecoveryEvent() {
|
||||
if (!store.recovery || recoveryRecorded) return;
|
||||
append({
|
||||
type: 'journal.recovered',
|
||||
severity: 'warning',
|
||||
source: 'storage',
|
||||
dedupeKey: `journal.recovered:${store.recovery.recoveredAt}`,
|
||||
data: {},
|
||||
});
|
||||
recoveryRecorded = true;
|
||||
}
|
||||
|
||||
function page(limitValue: unknown = 50, cursorValue: unknown = null): ActivityJournalPage {
|
||||
try {
|
||||
let state = store.read();
|
||||
ensureRecoveryEvent();
|
||||
if (store.recovery) state = store.read();
|
||||
const retainedEvents = retained(state.events);
|
||||
if (retainedEvents.length !== state.events.length) {
|
||||
state = store.update(() => ({ schemaVersion: 1, events: retainedEvents }));
|
||||
}
|
||||
const events = [...state.events].reverse();
|
||||
const limit = Math.min(100, Math.max(1, Number.isSafeInteger(limitValue) ? Number(limitValue) : 50));
|
||||
const cursor = typeof cursorValue === 'string' ? cursorValue : '';
|
||||
const cursorIndex = cursor ? events.findIndex(({ id }) => id === cursor) : -1;
|
||||
if (cursor && cursorIndex < 0) return {
|
||||
events: [],
|
||||
nextCursor: null,
|
||||
retentionDays: 30,
|
||||
generatedAt: now().toISOString(),
|
||||
storage: writeFailed
|
||||
? { status: 'error', errorCode: 'JOURNAL_UNAVAILABLE' }
|
||||
: { status: 'ready', errorCode: null },
|
||||
};
|
||||
const safeStart = cursorIndex + 1;
|
||||
const selected = events.slice(safeStart, safeStart + limit);
|
||||
return {
|
||||
events: selected.map((event) => ({ ...event, dedupeKey: null })),
|
||||
nextCursor: safeStart + selected.length < events.length ? selected.at(-1)?.id || null : null,
|
||||
retentionDays: 30,
|
||||
generatedAt: now().toISOString(),
|
||||
storage: writeFailed
|
||||
? { status: 'error', errorCode: 'JOURNAL_UNAVAILABLE' }
|
||||
: { status: 'ready', errorCode: null },
|
||||
};
|
||||
} catch {
|
||||
return {
|
||||
events: [],
|
||||
nextCursor: null,
|
||||
retentionDays: 30,
|
||||
generatedAt: now().toISOString(),
|
||||
storage: { status: 'error', errorCode: 'JOURNAL_UNAVAILABLE' },
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
if (fs.existsSync(filePath)) {
|
||||
try {
|
||||
const state = store.read();
|
||||
const retainedEvents = retained(state.events);
|
||||
if (retainedEvents.length !== state.events.length) {
|
||||
store.update(() => ({ schemaVersion: 1, events: retainedEvents }));
|
||||
}
|
||||
} catch {
|
||||
writeFailed = true;
|
||||
}
|
||||
}
|
||||
|
||||
return { append, page };
|
||||
}
|
||||
|
||||
export type ActivityJournalService = ReturnType<typeof createActivityJournalService>;
|
||||
@@ -44,6 +44,7 @@ interface RequestOptions {
|
||||
ipv4?: boolean;
|
||||
follow?: boolean;
|
||||
resolve?: string | null;
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
interface RequestResult {
|
||||
@@ -179,7 +180,9 @@ async function request(probe: BaseProbe, path: PathKind, proxyPort: number, exec
|
||||
ipv4 = false,
|
||||
follow = true,
|
||||
resolve = null,
|
||||
timeoutMs = 6_000,
|
||||
}: RequestOptions = {}): Promise<RequestResult> {
|
||||
const boundedTimeoutMs = Math.min(30_000, Math.max(1_000, Math.round(timeoutMs)));
|
||||
const args = [
|
||||
'--silent',
|
||||
'--show-error',
|
||||
@@ -189,9 +192,9 @@ async function request(probe: BaseProbe, path: PathKind, proxyPort: number, exec
|
||||
'--proto-redir',
|
||||
'=https',
|
||||
'--connect-timeout',
|
||||
'3',
|
||||
String(Math.min(3_000, boundedTimeoutMs) / 1_000),
|
||||
'--max-time',
|
||||
'6',
|
||||
String(boundedTimeoutMs / 1_000),
|
||||
'--user-agent',
|
||||
'Harbor-Diagnostics/1',
|
||||
'--output',
|
||||
@@ -379,6 +382,8 @@ async function siteProbe(
|
||||
proxyPort: number,
|
||||
execute: CurlExecutor,
|
||||
sampleCount = 1,
|
||||
timeoutMs = 6_000,
|
||||
retryFailure = true,
|
||||
): Promise<SiteProbeResult> {
|
||||
if (probe.validationError) return {
|
||||
id: probe.id,
|
||||
@@ -391,12 +396,12 @@ async function siteProbe(
|
||||
stage: 'validation',
|
||||
error: probe.validationError,
|
||||
};
|
||||
const options = { follow: probe.follow !== false, resolve: probe.resolve };
|
||||
const options = { follow: probe.follow !== false, resolve: probe.resolve, timeoutMs };
|
||||
const samples = [];
|
||||
for (let attempt = 0; attempt < sampleCount; attempt += 1) {
|
||||
samples.push(await request(probe, path, proxyPort, execute, options));
|
||||
}
|
||||
if (sampleCount === 1 && samples[0] && !samples[0].ok) {
|
||||
if (retryFailure && sampleCount === 1 && samples[0] && !samples[0].ok) {
|
||||
samples.push(await request(probe, path, proxyPort, execute, options));
|
||||
}
|
||||
const status = mostCommon(samples.map(siteStatus)) || 'unavailable';
|
||||
@@ -469,15 +474,18 @@ async function probeTarget(
|
||||
path: PathKind,
|
||||
proxyPort: number,
|
||||
execute: CurlExecutor,
|
||||
timeoutMs = 6_000,
|
||||
sampleCount = TARGET_SAMPLE_COUNT,
|
||||
retryFailure = true,
|
||||
): Promise<ConnectivityPathResult> {
|
||||
const network = target.kind === 'network'
|
||||
? await networkProbe(path, proxyPort, execute, TARGET_SAMPLE_COUNT)
|
||||
? await networkProbe(path, proxyPort, execute, sampleCount)
|
||||
: null;
|
||||
const ip = target.kind === 'ip'
|
||||
? await ipProbe(target.probe, path, proxyPort, execute, TARGET_SAMPLE_COUNT)
|
||||
? await ipProbe(target.probe, path, proxyPort, execute, sampleCount)
|
||||
: null;
|
||||
const site = target.kind === 'site'
|
||||
? await siteProbe(target.probe, path, proxyPort, execute, TARGET_SAMPLE_COUNT)
|
||||
? await siteProbe(target.probe, path, proxyPort, execute, sampleCount, timeoutMs, retryFailure)
|
||||
: null;
|
||||
const ipv4Sources = ip?.family === 4 ? [ip] : [];
|
||||
const ipv6Source = ip?.family === 6 ? ip : null;
|
||||
@@ -537,7 +545,25 @@ export function createConnectivityDiagnosticsService({
|
||||
assessment: assessConnectivity(direct, vpn),
|
||||
};
|
||||
}
|
||||
async function runVpn({ services = [], target: targetId = null, timeoutMs = 6_000 }: {
|
||||
services?: unknown;
|
||||
target?: unknown;
|
||||
timeoutMs?: number;
|
||||
}) {
|
||||
const requestedServices = typeof targetId === 'string' && targetId.startsWith('site:custom-')
|
||||
? (Array.isArray(services) ? services : []).filter((service) => `site:${String(record(service).id || '')}` === targetId)
|
||||
: [];
|
||||
const customProbes = await prepareCustomProbes(requestedServices, lookup);
|
||||
const siteProbes = [...SITE_PROBES, ...customProbes];
|
||||
const target = resolveTarget(targetId, siteProbes);
|
||||
if (!target || target.kind !== 'site') throw new Error('Failover check ожидает site target');
|
||||
return {
|
||||
checkedAt: now(),
|
||||
vpn: await probeTarget(target, 'vpn', proxyPort, execute, timeoutMs, TARGET_SAMPLE_COUNT, false),
|
||||
};
|
||||
}
|
||||
return {
|
||||
run: runOnce,
|
||||
runVpn,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -84,12 +84,33 @@ interface DomainTrafficSnapshot {
|
||||
}>;
|
||||
}
|
||||
|
||||
interface ActivityEntry {
|
||||
device: string;
|
||||
service: string;
|
||||
upload: bigint;
|
||||
download: bigint;
|
||||
}
|
||||
|
||||
interface ActivitySample {
|
||||
at: number;
|
||||
entries: ActivityEntry[];
|
||||
}
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return value && typeof value === 'object' && !Array.isArray(value)
|
||||
? value as Record<string, unknown>
|
||||
: {};
|
||||
}
|
||||
|
||||
function publicDeviceLabel(value: unknown) {
|
||||
const device = record(value);
|
||||
for (const candidate of [device.alias, device.hostname]) {
|
||||
const label = typeof candidate === 'string' ? candidate.trim() : '';
|
||||
if (label && label.length <= 64 && !/[\/?#@\\]/.test(label) && !net.isIP(label)) return label;
|
||||
}
|
||||
return 'Устройство';
|
||||
}
|
||||
|
||||
const matchesDomain = (domain: string, suffix: string) => domain === suffix || domain.endsWith(`.${suffix}`);
|
||||
|
||||
export function classifyDomain(value: unknown): { domain: string; service: string } | null {
|
||||
@@ -209,6 +230,10 @@ export function createDomainTrafficService({
|
||||
unsupported_source: 0n,
|
||||
};
|
||||
let refreshPromise: Promise<DomainTrafficSnapshot> | null = null;
|
||||
let activityEnabled = false;
|
||||
let activityStartedAt = 0;
|
||||
let activitySamples: ActivitySample[] = [];
|
||||
let quietSince: string | null = null;
|
||||
let current: DomainTrafficSnapshot = {
|
||||
epoch,
|
||||
observedAt: null,
|
||||
@@ -270,6 +295,7 @@ export function createDomainTrafficService({
|
||||
const response = record(await observe());
|
||||
if (!Array.isArray(response.connections)) throw new Error('Sing-box не вернул connections array');
|
||||
const devicesByIp = new Map<string, string | null>();
|
||||
const deviceLabels = new Map<string, string>();
|
||||
const observedDevices = devices();
|
||||
for (const value of Array.isArray(observedDevices) ? observedDevices : []) {
|
||||
const device = record(value);
|
||||
@@ -277,9 +303,11 @@ export function createDomainTrafficService({
|
||||
const id = typeof device.mac === 'string' ? deviceId(device.mac.toLowerCase()) : null;
|
||||
if (!net.isIPv4(ip) || !id) continue;
|
||||
devicesByIp.set(ip, devicesByIp.has(ip) ? null : id);
|
||||
deviceLabels.set(id, publicDeviceLabel(device));
|
||||
}
|
||||
const connections = response.connections.map((connection) => parseConnection(connection, devicesByIp));
|
||||
const activeConnections = new Map<string, PreviousConnection>();
|
||||
const activityEntries: ActivityEntry[] = [];
|
||||
for (const connection of connections) {
|
||||
const previous = previousConnections.get(connection.id);
|
||||
if (connection.outcome !== 'classified' && previous?.outcome !== connection.outcome) {
|
||||
@@ -302,6 +330,16 @@ export function createDomainTrafficService({
|
||||
tracked.uploadBytes += uploadDelta;
|
||||
tracked.downloadBytes += downloadDelta;
|
||||
trackedTotals.set(trackedKey, tracked);
|
||||
if (activityEnabled && connection.outbound === 'vpn' && uploadDelta + downloadDelta > 0n) {
|
||||
activityEntries.push({
|
||||
device: 'deviceId' in connection
|
||||
? deviceLabels.get(connection.deviceId) || 'Устройство'
|
||||
: 'Неизвестное устройство',
|
||||
service: 'service' in connection ? connection.service : 'Не распознано',
|
||||
upload: uploadDelta,
|
||||
download: downloadDelta,
|
||||
});
|
||||
}
|
||||
}
|
||||
if (connection.outcome === 'unknown_device' || connection.outcome === 'unsupported_source') {
|
||||
activeConnections.set(connection.id, {
|
||||
@@ -372,7 +410,12 @@ export function createDomainTrafficService({
|
||||
});
|
||||
}
|
||||
previousConnections = activeConnections;
|
||||
current = { ...current, observedAt: now().toISOString() };
|
||||
const observed = now();
|
||||
if (activityEnabled) {
|
||||
activitySamples.push({ at: observed.getTime(), entries: activityEntries });
|
||||
activitySamples = activitySamples.filter(({ at }) => at >= observed.getTime() - 10_000);
|
||||
}
|
||||
current = { ...current, observedAt: observed.toISOString() };
|
||||
current = buildSnapshot();
|
||||
return current;
|
||||
} catch (error) {
|
||||
@@ -390,5 +433,64 @@ export function createDomainTrafficService({
|
||||
return refreshPromise;
|
||||
}
|
||||
|
||||
return { snapshot: () => current, refresh };
|
||||
function enableActivity() {
|
||||
if (activityEnabled) return;
|
||||
activityEnabled = true;
|
||||
activityStartedAt = now().getTime();
|
||||
activitySamples = [];
|
||||
quietSince = null;
|
||||
}
|
||||
|
||||
function disableActivity() {
|
||||
activityEnabled = false;
|
||||
activityStartedAt = 0;
|
||||
activitySamples = [];
|
||||
quietSince = null;
|
||||
}
|
||||
|
||||
function activitySnapshot(thresholdBytesPerSecond: unknown = 0) {
|
||||
if (!activityEnabled || !current.observedAt) return null;
|
||||
const observedAt = Date.parse(current.observedAt);
|
||||
const threshold = Math.max(0, Number(thresholdBytesPerSecond) || 0);
|
||||
const divisorMs = Math.max(1_000, Math.min(10_000, observedAt - activityStartedAt || 1_000));
|
||||
const totals = new Map<string, ActivityEntry>();
|
||||
let bytes = 0n;
|
||||
for (const sample of activitySamples) {
|
||||
for (const entry of sample.entries) {
|
||||
bytes += entry.upload + entry.download;
|
||||
const key = `${entry.device}\0${entry.service}`;
|
||||
const total = totals.get(key) || { ...entry, upload: 0n, download: 0n };
|
||||
total.upload += entry.upload;
|
||||
total.download += entry.download;
|
||||
totals.set(key, total);
|
||||
}
|
||||
}
|
||||
const totalBytesPerSecond = Number(bytes * 1_000n / BigInt(divisorMs));
|
||||
const active = totalBytesPerSecond > threshold;
|
||||
quietSince = active ? null : quietSince || current.observedAt;
|
||||
const latest = activitySamples.at(-1);
|
||||
return {
|
||||
state: active ? 'active' : 'quiet',
|
||||
observedAt: current.observedAt,
|
||||
windowMs: 10_000,
|
||||
thresholdBytesPerSecond: threshold,
|
||||
totalBytesPerSecond,
|
||||
transmittingConnections: latest?.entries.length || 0,
|
||||
quietSince,
|
||||
blockers: [...totals.values()]
|
||||
.map((entry) => ({
|
||||
device: entry.device,
|
||||
service: entry.service,
|
||||
uploadBytesPerSecond: Number(entry.upload * 1_000n / BigInt(divisorMs)),
|
||||
downloadBytesPerSecond: Number(entry.download * 1_000n / BigInt(divisorMs)),
|
||||
}))
|
||||
.sort((left, right) => (
|
||||
right.uploadBytesPerSecond + right.downloadBytesPerSecond
|
||||
- left.uploadBytesPerSecond - left.downloadBytesPerSecond
|
||||
))
|
||||
.slice(0, 3),
|
||||
};
|
||||
}
|
||||
|
||||
return { snapshot: () => current, refresh, enableActivity, disableActivity, activitySnapshot };
|
||||
}
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
import http from 'node:http';
|
||||
import {
|
||||
FAILOVER_PRIMARY_TAG,
|
||||
FAILOVER_RESERVE_TAG,
|
||||
FAILOVER_SELECTOR_TAG,
|
||||
} from '../singbox.js';
|
||||
|
||||
type Role = 'primary' | 'reserve';
|
||||
|
||||
function request(port: number, method: string, body?: unknown): Promise<unknown> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const encoded = body === undefined ? null : JSON.stringify(body);
|
||||
const req = http.request({
|
||||
host: '127.0.0.1',
|
||||
port,
|
||||
path: `/proxies/${encodeURIComponent(FAILOVER_SELECTOR_TAG)}`,
|
||||
method,
|
||||
headers: encoded ? {
|
||||
'content-type': 'application/json',
|
||||
'content-length': Buffer.byteLength(encoded),
|
||||
} : {},
|
||||
}, (res) => {
|
||||
const chunks: Buffer[] = [];
|
||||
res.on('data', (chunk: Buffer) => chunks.push(chunk));
|
||||
res.on('end', () => {
|
||||
if ((res.statusCode || 500) >= 400) return reject(new Error(`Sing-box selector HTTP ${res.statusCode}`));
|
||||
if (!chunks.length) return resolve({});
|
||||
try {
|
||||
resolve(JSON.parse(Buffer.concat(chunks).toString('utf8')));
|
||||
} catch (cause) {
|
||||
reject(new Error('Sing-box selector вернул невалидный JSON', { cause }));
|
||||
}
|
||||
});
|
||||
});
|
||||
req.setTimeout(2_000, () => req.destroy(new Error('Sing-box selector timeout')));
|
||||
req.on('error', reject);
|
||||
req.end(encoded);
|
||||
});
|
||||
}
|
||||
|
||||
const tagFor = (role: Role) => role === 'primary' ? FAILOVER_PRIMARY_TAG : FAILOVER_RESERVE_TAG;
|
||||
const roleFor = (tag: unknown): Role | null => (
|
||||
tag === FAILOVER_PRIMARY_TAG ? 'primary' : tag === FAILOVER_RESERVE_TAG ? 'reserve' : null
|
||||
);
|
||||
|
||||
export function createSingboxSelectorService({
|
||||
port,
|
||||
send = (method: string, body?: unknown) => request(port, method, body),
|
||||
}: {
|
||||
port: number;
|
||||
send?: (method: string, body?: unknown) => Promise<unknown>;
|
||||
}) {
|
||||
async function read() {
|
||||
const value = await send('GET') as Record<string, unknown>;
|
||||
const role = roleFor(value.now);
|
||||
if (!role) throw new Error('Sing-box selector вернул неизвестный outbound');
|
||||
return { role };
|
||||
}
|
||||
async function select(role: Role) {
|
||||
await send('PUT', { name: tagFor(role) });
|
||||
const selected = await read();
|
||||
if (selected.role !== role) throw new Error('Sing-box selector не подтвердил переключение');
|
||||
return selected;
|
||||
}
|
||||
return { read, select };
|
||||
}
|
||||
@@ -13,7 +13,7 @@ import {
|
||||
type NormalizedServer,
|
||||
} from '../../shared/serverIdentity.js';
|
||||
|
||||
export const STATE_SCHEMA_VERSION = 7;
|
||||
export const STATE_SCHEMA_VERSION = 8;
|
||||
|
||||
export interface AtomicWriteOptions {
|
||||
beforeRename?: (temporaryPath: string, filePath: string) => void;
|
||||
|
||||
+131
-6
@@ -1,13 +1,20 @@
|
||||
import fs from 'node:fs';
|
||||
import crypto from 'node:crypto';
|
||||
import { settings } from './config.js';
|
||||
import { HarborError } from '../shared/errors.js';
|
||||
import { normalizeRouteRules } from '../shared/routingRules.js';
|
||||
import type { AppliedFailoverPolicy } from '../shared/failover.js';
|
||||
import { atomicWriteFile, atomicWriteJson } from './services/stateStore.js';
|
||||
|
||||
const PROXY_TYPES = new Set(['vless', 'vmess', 'trojan', 'shadowsocks', 'hysteria2']);
|
||||
const MIXED_INBOUND = 'mixed-in';
|
||||
const TPROXY_INBOUND = 'tproxy-in';
|
||||
const DIAGNOSTICS_INBOUND = 'diagnostics-vpn-in';
|
||||
const DIAGNOSTICS_PRIMARY_INBOUND = 'diagnostics-primary-in';
|
||||
const DIAGNOSTICS_RESERVE_INBOUND = 'diagnostics-reserve-in';
|
||||
export const FAILOVER_SELECTOR_TAG = 'channel-selector';
|
||||
export const FAILOVER_PRIMARY_TAG = 'channel-primary';
|
||||
export const FAILOVER_RESERVE_TAG = 'channel-reserve';
|
||||
const SNIFF_TIMEOUT = '1s';
|
||||
const SNIFFERS = ['http', 'tls', 'quic'];
|
||||
|
||||
@@ -34,18 +41,66 @@ function findOutbound(subscriptionConfig: unknown, selectedTag: unknown): ProxyO
|
||||
));
|
||||
}
|
||||
|
||||
function selectedOutbound(subscriptionConfig: unknown, selectedTag: unknown, tag?: string) {
|
||||
const outbound = structuredClone(findOutbound(subscriptionConfig, selectedTag));
|
||||
if (!outbound) throw new HarborError('SERVER_NOT_FOUND');
|
||||
if (tag) outbound.tag = tag;
|
||||
else if (!outbound.tag) outbound.tag = 'vpn-out';
|
||||
if (outbound.type === 'vless' && !outbound.packet_encoding) outbound.packet_encoding = 'xudp';
|
||||
return outbound;
|
||||
}
|
||||
|
||||
export interface DualChannelConfig {
|
||||
primary: { subscriptionConfig: unknown; selectedServerId: string };
|
||||
reserve: { subscriptionConfig: unknown; selectedServerId: string };
|
||||
}
|
||||
|
||||
export function fingerprintConfiguredOutbound(value: unknown, selectedServerId: string) {
|
||||
const outbound = structuredClone(record(value)) as ProxyOutbound;
|
||||
if (!PROXY_TYPES.has(String(outbound.type || ''))) throw new HarborError('CONFIG_INVALID');
|
||||
outbound.tag = selectedServerId;
|
||||
if (outbound.type === 'vless' && !outbound.packet_encoding) outbound.packet_encoding = 'xudp';
|
||||
return crypto.createHash('sha256').update(JSON.stringify(outbound)).digest('hex');
|
||||
}
|
||||
|
||||
export function fingerprintSelectedOutbound(subscriptionConfig: unknown, selectedServerId: string) {
|
||||
const outbound = findOutbound(subscriptionConfig, selectedServerId);
|
||||
if (!outbound) throw new HarborError('SERVER_NOT_FOUND');
|
||||
return fingerprintConfiguredOutbound(outbound, selectedServerId);
|
||||
}
|
||||
|
||||
export function dualChannelConfigMatchesApplied(
|
||||
configValue: unknown,
|
||||
applied: AppliedFailoverPolicy,
|
||||
expectedRole: 'primary' | 'reserve',
|
||||
) {
|
||||
const config = record(configValue);
|
||||
const outbounds = (Array.isArray(config.outbounds) ? config.outbounds : []).map(record);
|
||||
const primary = outbounds.filter(({ tag }) => tag === FAILOVER_PRIMARY_TAG);
|
||||
const reserve = outbounds.filter(({ tag }) => tag === FAILOVER_RESERVE_TAG);
|
||||
const selector = outbounds.find(({ tag }) => tag === FAILOVER_SELECTOR_TAG);
|
||||
try {
|
||||
return primary.length === 1
|
||||
&& reserve.length === 1
|
||||
&& fingerprintConfiguredOutbound(primary[0], applied.primary.serverId) === applied.primaryConfigFingerprint
|
||||
&& fingerprintConfiguredOutbound(reserve[0], applied.reserve.serverId) === applied.reserveConfigFingerprint
|
||||
&& selector?.type === 'selector'
|
||||
&& JSON.stringify(selector.outbounds) === JSON.stringify([FAILOVER_PRIMARY_TAG, FAILOVER_RESERVE_TAG])
|
||||
&& selector.default === (expectedRole === 'reserve' ? FAILOVER_RESERVE_TAG : FAILOVER_PRIMARY_TAG)
|
||||
&& selector.interrupt_exist_connections === false
|
||||
&& record(config.route).final === FAILOVER_SELECTOR_TAG;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export function buildGatewayConfig(subscriptionConfig: unknown, selectedTag: unknown, {
|
||||
clientDirect = false,
|
||||
routeRules = [],
|
||||
}: { clientDirect?: boolean; routeRules?: unknown } = {}) {
|
||||
const clientMode = settings.appMode === 'client';
|
||||
const directClient = clientMode && clientDirect;
|
||||
const vpnOutbound = structuredClone(findOutbound(subscriptionConfig, selectedTag));
|
||||
if (!vpnOutbound) throw new HarborError('SERVER_NOT_FOUND');
|
||||
if (!vpnOutbound.tag) vpnOutbound.tag = 'vpn-out';
|
||||
if (vpnOutbound.type === 'vless' && !vpnOutbound.packet_encoding) {
|
||||
vpnOutbound.packet_encoding = 'xudp';
|
||||
}
|
||||
const vpnOutbound = selectedOutbound(subscriptionConfig, selectedTag);
|
||||
const outboundTag = directClient ? 'direct' : vpnOutbound.tag;
|
||||
|
||||
const inbounds = [
|
||||
@@ -124,6 +179,76 @@ export function buildGatewayConfig(subscriptionConfig: unknown, selectedTag: unk
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
export function buildDualChannelGatewayConfig(
|
||||
channels: DualChannelConfig,
|
||||
{ routeRules = [], defaultRole = 'primary' }: { routeRules?: unknown; defaultRole?: 'primary' | 'reserve' } = {},
|
||||
) {
|
||||
if (settings.appMode === 'client') throw new Error('Dual-channel config доступен только Gateway');
|
||||
const primary = selectedOutbound(
|
||||
channels.primary.subscriptionConfig,
|
||||
channels.primary.selectedServerId,
|
||||
FAILOVER_PRIMARY_TAG,
|
||||
);
|
||||
const reserve = selectedOutbound(
|
||||
channels.reserve.subscriptionConfig,
|
||||
channels.reserve.selectedServerId,
|
||||
FAILOVER_RESERVE_TAG,
|
||||
);
|
||||
const userRules = normalizeRouteRules(routeRules)
|
||||
.filter((rule) => rule.enabled)
|
||||
.map((rule) => ({
|
||||
[rule.type]: [rule.value],
|
||||
outbound: rule.outbound === 'vpn' ? FAILOVER_SELECTOR_TAG : 'direct',
|
||||
}));
|
||||
const userInbounds = [TPROXY_INBOUND, MIXED_INBOUND];
|
||||
return {
|
||||
log: { level: settings.logLevel, timestamp: true },
|
||||
experimental: {
|
||||
cache_file: { enabled: true, path: settings.cachePath },
|
||||
clash_api: { external_controller: `127.0.0.1:${settings.singboxApiPort}` },
|
||||
},
|
||||
dns: { independent_cache: true },
|
||||
inbounds: [
|
||||
{ type: 'tproxy', tag: TPROXY_INBOUND, listen: '::', listen_port: settings.tproxyPort },
|
||||
{ type: 'mixed', tag: MIXED_INBOUND, listen: settings.bindIp, listen_port: settings.proxyPort, set_system_proxy: false },
|
||||
{ type: 'mixed', tag: DIAGNOSTICS_INBOUND, listen: '127.0.0.1', listen_port: settings.diagnosticsProxyPort, set_system_proxy: false },
|
||||
{ type: 'mixed', tag: DIAGNOSTICS_PRIMARY_INBOUND, listen: '127.0.0.1', listen_port: settings.failoverPrimaryProxyPort, set_system_proxy: false },
|
||||
{ type: 'mixed', tag: DIAGNOSTICS_RESERVE_INBOUND, listen: '127.0.0.1', listen_port: settings.failoverReserveProxyPort, set_system_proxy: false },
|
||||
],
|
||||
outbounds: [
|
||||
primary,
|
||||
reserve,
|
||||
{
|
||||
type: 'selector',
|
||||
tag: FAILOVER_SELECTOR_TAG,
|
||||
outbounds: [FAILOVER_PRIMARY_TAG, FAILOVER_RESERVE_TAG],
|
||||
default: defaultRole === 'reserve' ? FAILOVER_RESERVE_TAG : FAILOVER_PRIMARY_TAG,
|
||||
interrupt_exist_connections: false,
|
||||
},
|
||||
{ type: 'direct', tag: 'direct' },
|
||||
],
|
||||
route: {
|
||||
rule_set: [],
|
||||
rules: [
|
||||
{
|
||||
inbound: [TPROXY_INBOUND, MIXED_INBOUND, DIAGNOSTICS_INBOUND, DIAGNOSTICS_PRIMARY_INBOUND, DIAGNOSTICS_RESERVE_INBOUND],
|
||||
action: 'sniff',
|
||||
sniffer: SNIFFERS,
|
||||
timeout: SNIFF_TIMEOUT,
|
||||
},
|
||||
{ inbound: [DIAGNOSTICS_PRIMARY_INBOUND], outbound: FAILOVER_PRIMARY_TAG },
|
||||
{ inbound: [DIAGNOSTICS_RESERVE_INBOUND], outbound: FAILOVER_RESERVE_TAG },
|
||||
{ inbound: [DIAGNOSTICS_INBOUND], outbound: FAILOVER_SELECTOR_TAG },
|
||||
...userRules,
|
||||
{ inbound: userInbounds, outbound: FAILOVER_SELECTOR_TAG },
|
||||
],
|
||||
final: FAILOVER_SELECTOR_TAG,
|
||||
auto_detect_interface: true,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function writeSingboxConfig(config: unknown) {
|
||||
atomicWriteJson(settings.configPath, config);
|
||||
}
|
||||
|
||||
@@ -19,6 +19,23 @@ export function createSingboxRuntime({
|
||||
|
||||
const state = () => ({ running: Boolean(child), startedAt });
|
||||
|
||||
function checkConfig(config: unknown) {
|
||||
const directory = fs.mkdtempSync(`${configPath}.check-`);
|
||||
const candidatePath = `${directory}/config.json`;
|
||||
try {
|
||||
fs.writeFileSync(candidatePath, JSON.stringify(config));
|
||||
const check = spawnSync('sing-box', ['check', '-c', candidatePath], { encoding: 'utf8' });
|
||||
if (check.status !== 0) {
|
||||
throw new HarborError('CONFIG_INVALID', {
|
||||
cause: new Error((check.stderr || check.stdout || check.error?.message || 'sing-box check failed').trim()),
|
||||
});
|
||||
}
|
||||
return { valid: true };
|
||||
} finally {
|
||||
fs.rmSync(directory, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function stop() {
|
||||
if (gateway) setGatewayInterception(false, tproxyChain);
|
||||
if (!child) {
|
||||
@@ -100,6 +117,7 @@ export function createSingboxRuntime({
|
||||
get running() { return Boolean(child); },
|
||||
get startedAt() { return startedAt; },
|
||||
refresh: async () => state(),
|
||||
checkConfig,
|
||||
apply,
|
||||
restart: () => apply({ force: true }),
|
||||
stop,
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
export const ACTIVITY_JOURNAL_RETENTION_DAYS = 30;
|
||||
export const ACTIVITY_JOURNAL_MAX_EVENTS = 10_000;
|
||||
|
||||
export const ACTIVITY_EVENT_TYPES = [
|
||||
'connection.started', 'connection.stopped', 'connection.failed',
|
||||
'subscription.added', 'subscription.refreshed', 'subscription.refresh_failed', 'subscription.deleted',
|
||||
'failover.enabled', 'failover.disabled', 'failover.paused', 'failover.resumed',
|
||||
'failover.waiting_for_idle', 'failover.switched', 'failover.switch_failed',
|
||||
'failover.both_unhealthy', 'failover.recovered', 'journal.recovered',
|
||||
] as const;
|
||||
|
||||
export type ActivityEventType = typeof ACTIVITY_EVENT_TYPES[number];
|
||||
export type ActivityEventSeverity = 'info' | 'warning' | 'error';
|
||||
export type ActivityEventSource = 'connection' | 'subscription' | 'failover' | 'storage';
|
||||
|
||||
export interface ActivityJournalEvent {
|
||||
id: string;
|
||||
occurredAt: string;
|
||||
type: ActivityEventType | 'unknown';
|
||||
severity: ActivityEventSeverity;
|
||||
source: ActivityEventSource;
|
||||
dedupeKey: string | null;
|
||||
data: Record<string, string | number | boolean | null>;
|
||||
}
|
||||
|
||||
export type ActivityJournalEventInput = Omit<ActivityJournalEvent, 'id' | 'occurredAt'>;
|
||||
|
||||
export interface ActivityJournalPage {
|
||||
events: ActivityJournalEvent[];
|
||||
nextCursor: string | null;
|
||||
retentionDays: 30;
|
||||
generatedAt: string;
|
||||
storage: { status: 'ready' | 'error'; errorCode: string | null };
|
||||
}
|
||||
|
||||
const ALLOWED_DATA_KEYS: Record<ActivityEventType, readonly string[]> = {
|
||||
'connection.started': ['profileLabel', 'serverLabel'],
|
||||
'connection.stopped': [],
|
||||
'connection.failed': ['errorCode'],
|
||||
'subscription.added': ['profileId', 'profileLabel', 'host', 'serverCount'],
|
||||
'subscription.refreshed': ['profileId', 'profileLabel', 'host', 'serverCount', 'added', 'removed'],
|
||||
'subscription.refresh_failed': ['profileId', 'profileLabel', 'host', 'errorCode'],
|
||||
'subscription.deleted': ['profileId', 'profileLabel'],
|
||||
'failover.enabled': ['primaryLabel', 'reserveLabel'],
|
||||
'failover.disabled': [],
|
||||
'failover.paused': [],
|
||||
'failover.resumed': [],
|
||||
'failover.waiting_for_idle': ['fromRole', 'toRole', 'reason'],
|
||||
'failover.switched': ['fromRole', 'toRole', 'primaryLabel', 'reserveLabel', 'reason', 'manual'],
|
||||
'failover.switch_failed': ['fromRole', 'toRole', 'reason', 'errorCode'],
|
||||
'failover.both_unhealthy': ['reason'],
|
||||
'failover.recovered': ['role', 'reason'],
|
||||
'journal.recovered': [],
|
||||
};
|
||||
|
||||
const typeSet = new Set<string>(ACTIVITY_EVENT_TYPES);
|
||||
const severitySet = new Set(['info', 'warning', 'error']);
|
||||
const sourceSet = new Set(['connection', 'subscription', 'failover', 'storage']);
|
||||
const record = (value: unknown): Record<string, unknown> => (
|
||||
value && typeof value === 'object' && !Array.isArray(value) ? value as Record<string, unknown> : {}
|
||||
);
|
||||
|
||||
const LABEL_KEYS = new Set(['profileLabel', 'serverLabel', 'primaryLabel', 'reserveLabel']);
|
||||
const ROLE_KEYS = new Set(['fromRole', 'toRole', 'role']);
|
||||
const SAFE_LABEL_FALLBACKS: Record<string, string> = {
|
||||
profileLabel: 'Подписка',
|
||||
serverLabel: 'Сервер',
|
||||
primaryLabel: 'Основной канал',
|
||||
reserveLabel: 'Резервный канал',
|
||||
};
|
||||
|
||||
function safeScalar(key: string, value: unknown) {
|
||||
if (value === null || typeof value === 'boolean') return value;
|
||||
if (typeof value === 'number' && Number.isSafeInteger(value) && value >= 0) return value;
|
||||
if (typeof value !== 'string' || value.length > 120 || /[\r\n]/.test(value)) {
|
||||
throw new TypeError('Unsafe journal value');
|
||||
}
|
||||
if (LABEL_KEYS.has(key) && (
|
||||
!value.trim()
|
||||
|| /(?:[a-z][a-z0-9+.-]*:\/\/)|[\/?#@\\]/i.test(value)
|
||||
|| /(?:^|\D)(?:\d{1,3}\.){3}\d{1,3}(?:\D|$)/.test(value)
|
||||
|| /(?:^|[^0-9a-f])(?:[0-9a-f]{0,4}:){2,}[0-9a-f]{0,4}(?:[^0-9a-f]|$)/i.test(value)
|
||||
)) return SAFE_LABEL_FALLBACKS[key];
|
||||
if (ROLE_KEYS.has(key) && !['primary', 'reserve'].includes(value)) throw new TypeError('Unsafe journal role');
|
||||
if (key === 'errorCode' && !/^[A-Z0-9_]{1,50}$/.test(value)) throw new TypeError('Unsafe journal error code');
|
||||
if (key === 'reason' && !/^[a-z0-9-]{1,80}$/.test(value)) throw new TypeError('Unsafe journal reason');
|
||||
if (key === 'profileId' && !/^[a-zA-Z0-9_-]{1,80}$/.test(value)) throw new TypeError('Unsafe journal profile id');
|
||||
if (key === 'host' && (
|
||||
!/^[a-z0-9.-]{1,120}$/i.test(value)
|
||||
|| /^(?:\d{1,3}\.){3}\d{1,3}$/.test(value)
|
||||
|| value.includes('..')
|
||||
)) return 'Провайдер';
|
||||
return value;
|
||||
}
|
||||
|
||||
export function normalizeActivityEventInput(value: unknown): ActivityJournalEventInput {
|
||||
const candidate = record(value);
|
||||
const type = String(candidate.type || '') as ActivityEventType;
|
||||
const severity = String(candidate.severity || '') as ActivityEventSeverity;
|
||||
const source = String(candidate.source || '') as ActivityEventSource;
|
||||
if (!typeSet.has(type) || !severitySet.has(severity) || !sourceSet.has(source)) {
|
||||
throw new TypeError('Unknown journal event');
|
||||
}
|
||||
const inputData = record(candidate.data);
|
||||
const allowed = new Set(ALLOWED_DATA_KEYS[type]);
|
||||
if (Object.keys(inputData).some((key) => !allowed.has(key))) throw new TypeError('Unsafe journal data key');
|
||||
const data = Object.fromEntries(Object.entries(inputData).map(([key, item]) => [key, safeScalar(key, item)]));
|
||||
const dedupeKey = candidate.dedupeKey == null ? null : String(candidate.dedupeKey).trim();
|
||||
if (dedupeKey !== null && (
|
||||
!dedupeKey.startsWith(`${type}:`)
|
||||
|| !/^[a-zA-Z0-9_.:-]{1,160}$/.test(dedupeKey)
|
||||
)) {
|
||||
throw new TypeError('Invalid journal dedupe key');
|
||||
}
|
||||
return { type, severity, source, dedupeKey, data };
|
||||
}
|
||||
|
||||
export function normalizeStoredActivityEvent(value: unknown): ActivityJournalEvent | null {
|
||||
const candidate = record(value);
|
||||
const id = typeof candidate.id === 'string' && /^[a-f0-9-]{20,50}$/i.test(candidate.id) ? candidate.id : '';
|
||||
const occurredAt = typeof candidate.occurredAt === 'string' && Number.isFinite(Date.parse(candidate.occurredAt))
|
||||
? candidate.occurredAt
|
||||
: '';
|
||||
if (!id || !occurredAt) return null;
|
||||
try {
|
||||
return { id, occurredAt, ...normalizeActivityEventInput(candidate) };
|
||||
} catch {
|
||||
const severity = String(candidate.severity || '') as ActivityEventSeverity;
|
||||
const source = String(candidate.source || '') as ActivityEventSource;
|
||||
return severitySet.has(severity) && sourceSet.has(source) && typeof candidate.type === 'string'
|
||||
&& /^[a-z][a-z0-9_.-]{0,79}$/.test(candidate.type)
|
||||
? { id, occurredAt, type: 'unknown', severity, source, dedupeKey: null, data: {} }
|
||||
: null;
|
||||
}
|
||||
}
|
||||
|
||||
export function assertActivityJournalPage(value: unknown): ActivityJournalPage {
|
||||
const candidate = record(value);
|
||||
const events = Array.isArray(candidate.events) ? candidate.events.map(normalizeStoredActivityEvent) : [];
|
||||
const storage = record(candidate.storage);
|
||||
if (
|
||||
!Array.isArray(candidate.events) || events.some((event) => event === null)
|
||||
|| !(candidate.nextCursor === null || typeof candidate.nextCursor === 'string')
|
||||
|| candidate.retentionDays !== ACTIVITY_JOURNAL_RETENTION_DAYS
|
||||
|| typeof candidate.generatedAt !== 'string' || !Number.isFinite(Date.parse(candidate.generatedAt))
|
||||
|| !['ready', 'error'].includes(String(storage.status || ''))
|
||||
|| !(storage.errorCode === null || typeof storage.errorCode === 'string')
|
||||
) throw new TypeError('Invalid activity journal page');
|
||||
return {
|
||||
events: events as ActivityJournalEvent[],
|
||||
nextCursor: candidate.nextCursor as string | null,
|
||||
retentionDays: 30,
|
||||
generatedAt: candidate.generatedAt,
|
||||
storage: { status: storage.status as 'ready' | 'error', errorCode: storage.errorCode as string | null },
|
||||
};
|
||||
}
|
||||
@@ -8,6 +8,16 @@ import {
|
||||
normalizeDiagnosticSettings,
|
||||
type DiagnosticSettings,
|
||||
} from '../connectivityDiagnostics.js';
|
||||
import {
|
||||
createIdleFailoverSnapshot,
|
||||
normalizeAppliedFailoverPolicy,
|
||||
normalizeFailoverPolicy,
|
||||
normalizeFailoverRuntimeState,
|
||||
type AppliedFailoverPolicy,
|
||||
type FailoverPolicy,
|
||||
type FailoverRuntimeState,
|
||||
type FailoverSnapshot,
|
||||
} from '../failover.js';
|
||||
|
||||
export type HarborMode = 'client' | 'gateway';
|
||||
export type ConnectionState = 'running' | 'stopped';
|
||||
@@ -83,6 +93,7 @@ export interface StateSnapshot {
|
||||
lastError: string | null;
|
||||
};
|
||||
diagnostics: DiagnosticSettings;
|
||||
failover: FailoverSnapshot;
|
||||
route: {
|
||||
rulesContractVersion?: typeof ROUTE_RULES_CONTRACT_VERSION;
|
||||
mode: string;
|
||||
@@ -122,6 +133,9 @@ export interface PersistedState extends Record<string, unknown> {
|
||||
connectionDesired?: ConnectionState;
|
||||
gatewayAutoEnabled?: boolean;
|
||||
diagnostics: DiagnosticSettings;
|
||||
failoverPolicy: FailoverPolicy;
|
||||
failoverRuntimeState: FailoverRuntimeState;
|
||||
appliedFailoverPolicy: AppliedFailoverPolicy | null;
|
||||
}
|
||||
|
||||
// Legacy fields are derived in memory for bounded callers during the v5 cutover.
|
||||
@@ -293,6 +307,9 @@ export function normalizeStoredState(value: unknown): StoredState {
|
||||
? state.routeRulesRevision
|
||||
: 0,
|
||||
diagnostics: normalizeDiagnosticSettings(state.diagnostics),
|
||||
failoverPolicy: normalizeFailoverPolicy(state.failoverPolicy),
|
||||
failoverRuntimeState: normalizeFailoverRuntimeState(state.failoverRuntimeState),
|
||||
appliedFailoverPolicy: normalizeAppliedFailoverPolicy(state.appliedFailoverPolicy),
|
||||
subscriptionUrl: selectedProfile?.subscriptionUrl || '',
|
||||
selectedServerId,
|
||||
selectedTag: selectedServer?.label || '',
|
||||
@@ -327,6 +344,7 @@ export function createStateSnapshot({
|
||||
appMode,
|
||||
configExists,
|
||||
operation = { kind: null, status: 'idle', startedAt: null, error: null },
|
||||
failoverSnapshot,
|
||||
now = new Date(),
|
||||
}: {
|
||||
storedState: unknown;
|
||||
@@ -336,6 +354,7 @@ export function createStateSnapshot({
|
||||
configExists: boolean;
|
||||
subscriptionHost?: string;
|
||||
operation?: OperationState;
|
||||
failoverSnapshot?: FailoverSnapshot | null;
|
||||
now?: Date;
|
||||
}): StateSnapshot {
|
||||
const stored = normalizeStoredState(storedState);
|
||||
@@ -391,6 +410,7 @@ export function createStateSnapshot({
|
||||
lastError: null,
|
||||
},
|
||||
diagnostics: stored.diagnostics,
|
||||
failover: failoverSnapshot || createIdleFailoverSnapshot(stored.failoverPolicy),
|
||||
route: {
|
||||
rulesContractVersion: ROUTE_RULES_CONTRACT_VERSION,
|
||||
mode: routeMode,
|
||||
@@ -429,18 +449,24 @@ export function assertStateSnapshot(snapshot: unknown): StateSnapshot {
|
||||
const candidateWithDiagnostics = rawCandidate && rawCandidate.diagnostics === undefined
|
||||
? { ...rawCandidate, diagnostics: normalizeDiagnosticSettings(null) }
|
||||
: rawCandidate;
|
||||
const candidate = candidateWithDiagnostics?.route?.rulesContractVersion === undefined
|
||||
&& Array.isArray(candidateWithDiagnostics?.route?.localRules)
|
||||
&& Array.isArray(candidateWithDiagnostics?.route?.activeLocalRules)
|
||||
const candidateWithFailover = candidateWithDiagnostics && candidateWithDiagnostics.failover === undefined
|
||||
? {
|
||||
...candidateWithDiagnostics,
|
||||
route: {
|
||||
...candidateWithDiagnostics.route,
|
||||
localRules: candidateWithDiagnostics.route.localRules.map(legacyRule),
|
||||
activeLocalRules: candidateWithDiagnostics.route.activeLocalRules.map(legacyRule),
|
||||
},
|
||||
failover: createIdleFailoverSnapshot(normalizeFailoverPolicy(null)),
|
||||
}
|
||||
: candidateWithDiagnostics;
|
||||
const candidate = candidateWithFailover?.route?.rulesContractVersion === undefined
|
||||
&& Array.isArray(candidateWithFailover?.route?.localRules)
|
||||
&& Array.isArray(candidateWithFailover?.route?.activeLocalRules)
|
||||
? {
|
||||
...candidateWithFailover,
|
||||
route: {
|
||||
...candidateWithFailover.route,
|
||||
localRules: candidateWithFailover.route.localRules.map(legacyRule),
|
||||
activeLocalRules: candidateWithFailover.route.activeLocalRules.map(legacyRule),
|
||||
},
|
||||
}
|
||||
: candidateWithFailover;
|
||||
const validDate = (value: unknown) => typeof value === 'string' && Number.isFinite(Date.parse(value));
|
||||
const nullableDate = (value: unknown) => value === null || validDate(value);
|
||||
const nullableString = (value: unknown) => value === null || typeof value === 'string';
|
||||
@@ -486,6 +512,37 @@ export function assertStateSnapshot(snapshot: unknown): StateSnapshot {
|
||||
return false;
|
||||
}
|
||||
};
|
||||
const validFailover = (value: FailoverSnapshot) => {
|
||||
const channel = (item: FailoverSnapshot['primary']) => (
|
||||
item && typeof item.target?.profileId === 'string' && typeof item.target?.serverId === 'string'
|
||||
&& ['healthy', 'unhealthy', 'unknown', 'not-monitoring'].includes(item.health)
|
||||
&& Array.isArray(item.failingServiceIds) && item.failingServiceIds.every((id) => typeof id === 'string')
|
||||
&& nullableDate(item.checkedAt) && nullableDate(item.stateSince)
|
||||
);
|
||||
const activity = value.trafficActivity;
|
||||
try {
|
||||
normalizeFailoverPolicy(value.policy, { strict: true });
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
return channel(value.primary) && channel(value.reserve)
|
||||
&& nullableDate(value.nextDecisionAt) && nullableString(value.reason)
|
||||
&& (activity === null || (
|
||||
['active', 'quiet', 'unknown'].includes(activity.state)
|
||||
&& validDate(activity.observedAt)
|
||||
&& [activity.windowMs, activity.thresholdBytesPerSecond, activity.totalBytesPerSecond, activity.transmittingConnections]
|
||||
.every((number) => Number.isFinite(number) && number >= 0)
|
||||
&& nullableDate(activity.quietSince)
|
||||
&& (activity.switchTarget === null || ['primary', 'reserve'].includes(activity.switchTarget))
|
||||
&& Array.isArray(activity.blockers)
|
||||
&& activity.blockers.length <= 3
|
||||
&& activity.blockers.every((blocker) => (
|
||||
typeof blocker.device === 'string' && typeof blocker.service === 'string'
|
||||
&& Number.isFinite(blocker.uploadBytesPerSecond) && blocker.uploadBytesPerSecond >= 0
|
||||
&& Number.isFinite(blocker.downloadBytesPerSecond) && blocker.downloadBytesPerSecond >= 0
|
||||
))
|
||||
));
|
||||
};
|
||||
|
||||
if (
|
||||
!snapshot ||
|
||||
@@ -516,6 +573,18 @@ export function assertStateSnapshot(snapshot: unknown): StateSnapshot {
|
||||
!nullableDate(candidate.connection.startedAt) ||
|
||||
!nullableString(candidate.connection.lastError) ||
|
||||
!validDiagnostics(candidate.diagnostics) ||
|
||||
!candidate.failover ||
|
||||
typeof candidate.failover.observationEpoch !== 'string' ||
|
||||
!Number.isSafeInteger(candidate.failover.observationSequence) ||
|
||||
candidate.failover.observationSequence < 0 ||
|
||||
typeof candidate.failover.enabled !== 'boolean' ||
|
||||
typeof candidate.failover.paused !== 'boolean' ||
|
||||
typeof candidate.failover.configured !== 'boolean' ||
|
||||
!candidate.failover.policy ||
|
||||
!['inactive', 'active', 'pending', 'passive-loaded'].includes(candidate.failover.activation) ||
|
||||
!['primary', 'reserve', 'other', 'none'].includes(candidate.failover.currentRole) ||
|
||||
!['idle', 'observing', 'primary', 'reserve', 'waiting-for-idle', 'blocked', 'switching', 'error'].includes(candidate.failover.status) ||
|
||||
!validFailover(candidate.failover) ||
|
||||
!candidate.route ||
|
||||
![undefined, ROUTE_RULES_CONTRACT_VERSION].includes(candidate.route.rulesContractVersion) ||
|
||||
typeof candidate.route.mode !== 'string' ||
|
||||
|
||||
@@ -0,0 +1,355 @@
|
||||
export type FailoverRole = 'primary' | 'reserve';
|
||||
export type FailoverHealth = 'healthy' | 'unhealthy' | 'unknown' | 'not-monitoring';
|
||||
|
||||
export interface FailoverTarget {
|
||||
profileId: string;
|
||||
serverId: string;
|
||||
}
|
||||
|
||||
export interface FailoverCheck {
|
||||
serviceId: string;
|
||||
timeoutMs: number;
|
||||
}
|
||||
|
||||
export interface FailoverPolicy {
|
||||
version: 1;
|
||||
enabled: boolean;
|
||||
paused: boolean;
|
||||
primary: FailoverTarget;
|
||||
reserve: FailoverTarget;
|
||||
checks: FailoverCheck[];
|
||||
intervalMs: number;
|
||||
failureWindowMs: number;
|
||||
recoveryWindowMs: number;
|
||||
trafficGuard: {
|
||||
enabled: boolean;
|
||||
thresholdBytesPerSecond: number;
|
||||
quietWindowMs: number;
|
||||
};
|
||||
minimumReserveMs: number;
|
||||
flapProtection: {
|
||||
count: number;
|
||||
windowMs: number;
|
||||
quarantineMs: number;
|
||||
};
|
||||
}
|
||||
|
||||
export interface FailoverRuntimeState {
|
||||
lastSwitchAt: string | null;
|
||||
holdUntil: string | null;
|
||||
primaryQuarantineUntil: string | null;
|
||||
failoverHistory: string[];
|
||||
reasonCode: string | null;
|
||||
}
|
||||
|
||||
export interface AppliedFailoverPolicy {
|
||||
primary: FailoverTarget;
|
||||
reserve: FailoverTarget;
|
||||
primaryConfigFingerprint: string;
|
||||
reserveConfigFingerprint: string;
|
||||
}
|
||||
|
||||
export interface FailoverActivity {
|
||||
state: 'active' | 'quiet' | 'unknown';
|
||||
observedAt: string;
|
||||
windowMs: number;
|
||||
thresholdBytesPerSecond: number;
|
||||
totalBytesPerSecond: number;
|
||||
transmittingConnections: number;
|
||||
quietSince: string | null;
|
||||
switchTarget: FailoverRole | null;
|
||||
blockers: Array<{
|
||||
device: string;
|
||||
service: string;
|
||||
uploadBytesPerSecond: number;
|
||||
downloadBytesPerSecond: number;
|
||||
}>;
|
||||
}
|
||||
|
||||
export interface FailoverSnapshot {
|
||||
observationEpoch: string;
|
||||
observationSequence: number;
|
||||
configured: boolean;
|
||||
enabled: boolean;
|
||||
paused: boolean;
|
||||
activation: 'inactive' | 'active' | 'pending' | 'passive-loaded';
|
||||
currentRole: FailoverRole | 'other' | 'none';
|
||||
status: 'idle' | 'observing' | 'primary' | 'reserve' | 'waiting-for-idle' | 'blocked' | 'switching' | 'error';
|
||||
primary: { target: FailoverTarget; health: FailoverHealth; failingServiceIds: string[]; checkedAt: string | null; stateSince: string | null };
|
||||
reserve: { target: FailoverTarget; health: FailoverHealth; failingServiceIds: string[]; checkedAt: string | null; stateSince: string | null };
|
||||
nextDecisionAt: string | null;
|
||||
reason: string | null;
|
||||
trafficActivity: FailoverActivity | null;
|
||||
policy: FailoverPolicy;
|
||||
}
|
||||
|
||||
export interface FailoverDecisionMemory {
|
||||
primaryFailedSince: number | null;
|
||||
primaryRecoveredSince: number | null;
|
||||
quietSince: number | null;
|
||||
}
|
||||
|
||||
export interface FailoverDecisionInput {
|
||||
now: number;
|
||||
policy: FailoverPolicy;
|
||||
currentRole: FailoverRole;
|
||||
primaryHealth: Exclude<FailoverHealth, 'not-monitoring'>;
|
||||
reserveHealth: Exclude<FailoverHealth, 'not-monitoring'>;
|
||||
activity: 'active' | 'quiet' | 'unknown';
|
||||
holdUntil?: number | null;
|
||||
primaryQuarantineUntil?: number | null;
|
||||
memory?: FailoverDecisionMemory;
|
||||
}
|
||||
|
||||
export interface FailoverDecision {
|
||||
status: FailoverSnapshot['status'];
|
||||
reason: string;
|
||||
switchTo: FailoverRole | null;
|
||||
memory: FailoverDecisionMemory;
|
||||
nextDecisionAt: number | null;
|
||||
}
|
||||
|
||||
const text = (value: unknown) => typeof value === 'string' ? value.trim() : '';
|
||||
const record = (value: unknown): Record<string, unknown> => (
|
||||
value && typeof value === 'object' && !Array.isArray(value) ? value as Record<string, unknown> : {}
|
||||
);
|
||||
const bounded = (value: unknown, fallback: number, minimum: number, maximum: number) => {
|
||||
const candidate = Number(value);
|
||||
return Number.isFinite(candidate) ? Math.min(maximum, Math.max(minimum, Math.round(candidate))) : fallback;
|
||||
};
|
||||
const equivalent = (left: unknown, right: unknown): boolean => {
|
||||
if (Array.isArray(left) || Array.isArray(right)) {
|
||||
return Array.isArray(left) && Array.isArray(right)
|
||||
&& left.length === right.length
|
||||
&& left.every((value, index) => equivalent(value, right[index]));
|
||||
}
|
||||
if (left && right && typeof left === 'object' && typeof right === 'object') {
|
||||
const leftRecord = left as Record<string, unknown>;
|
||||
const rightRecord = right as Record<string, unknown>;
|
||||
const leftKeys = Object.keys(leftRecord).sort();
|
||||
const rightKeys = Object.keys(rightRecord).sort();
|
||||
return leftKeys.length === rightKeys.length
|
||||
&& leftKeys.every((key, index) => key === rightKeys[index] && equivalent(leftRecord[key], rightRecord[key]));
|
||||
}
|
||||
return Object.is(left, right);
|
||||
};
|
||||
|
||||
export const DEFAULT_FAILOVER_POLICY: FailoverPolicy = Object.freeze({
|
||||
version: 1,
|
||||
enabled: false,
|
||||
paused: false,
|
||||
primary: { profileId: '', serverId: '' },
|
||||
reserve: { profileId: '', serverId: '' },
|
||||
checks: [{ serviceId: 'youtube', timeoutMs: 6_000 }, { serviceId: 'google', timeoutMs: 6_000 }],
|
||||
intervalMs: 60_000,
|
||||
failureWindowMs: 120_000,
|
||||
recoveryWindowMs: 900_000,
|
||||
trafficGuard: { enabled: true, thresholdBytesPerSecond: 32 * 1024, quietWindowMs: 30_000 },
|
||||
minimumReserveMs: 600_000,
|
||||
flapProtection: { count: 3, windowMs: 86_400_000, quarantineMs: 21_600_000 },
|
||||
});
|
||||
|
||||
export const DEFAULT_FAILOVER_RUNTIME_STATE: FailoverRuntimeState = Object.freeze({
|
||||
lastSwitchAt: null,
|
||||
holdUntil: null,
|
||||
primaryQuarantineUntil: null,
|
||||
failoverHistory: [],
|
||||
reasonCode: null,
|
||||
});
|
||||
|
||||
function target(value: unknown): FailoverTarget {
|
||||
const candidate = record(value);
|
||||
return { profileId: text(candidate.profileId), serverId: text(candidate.serverId) };
|
||||
}
|
||||
|
||||
export function normalizeFailoverPolicy(value: unknown, { strict = false } = {}): FailoverPolicy {
|
||||
const candidate = record(value);
|
||||
const requestedChecks = Array.isArray(candidate.checks) ? candidate.checks : DEFAULT_FAILOVER_POLICY.checks;
|
||||
const checks = requestedChecks.map((value) => {
|
||||
const check = record(value);
|
||||
return {
|
||||
serviceId: text(check.serviceId).slice(0, 100),
|
||||
timeoutMs: bounded(check.timeoutMs, 6_000, 2_000, 30_000),
|
||||
};
|
||||
}).filter(({ serviceId }, index, all) => serviceId && all.findIndex((item) => item.serviceId === serviceId) === index).slice(0, 10);
|
||||
const trafficGuard = record(candidate.trafficGuard);
|
||||
const flapProtection = record(candidate.flapProtection);
|
||||
const policy: FailoverPolicy = {
|
||||
version: 1,
|
||||
enabled: candidate.enabled === true,
|
||||
paused: candidate.paused === true,
|
||||
primary: target(candidate.primary),
|
||||
reserve: target(candidate.reserve),
|
||||
checks,
|
||||
intervalMs: bounded(candidate.intervalMs, 60_000, 15_000, 900_000),
|
||||
failureWindowMs: 0,
|
||||
recoveryWindowMs: bounded(candidate.recoveryWindowMs, 900_000, 60_000, 86_400_000),
|
||||
trafficGuard: {
|
||||
enabled: trafficGuard.enabled !== false,
|
||||
thresholdBytesPerSecond: bounded(trafficGuard.thresholdBytesPerSecond, 32 * 1024, 1024, 100 * 1024 * 1024),
|
||||
quietWindowMs: bounded(trafficGuard.quietWindowMs, 30_000, 5_000, 600_000),
|
||||
},
|
||||
minimumReserveMs: bounded(candidate.minimumReserveMs, 600_000, 60_000, 86_400_000),
|
||||
flapProtection: {
|
||||
count: bounded(flapProtection.count, 3, 2, 10),
|
||||
windowMs: bounded(flapProtection.windowMs, 86_400_000, 3_600_000, 72 * 3_600_000),
|
||||
quarantineMs: bounded(flapProtection.quarantineMs, 21_600_000, 600_000, 7 * 86_400_000),
|
||||
},
|
||||
};
|
||||
policy.failureWindowMs = bounded(
|
||||
candidate.failureWindowMs,
|
||||
120_000,
|
||||
policy.intervalMs * 2,
|
||||
1_800_000,
|
||||
);
|
||||
if (strict && !equivalent(policy, value)) {
|
||||
throw new TypeError('Invalid failover policy');
|
||||
}
|
||||
return policy;
|
||||
}
|
||||
|
||||
export function normalizeFailoverRuntimeState(value: unknown): FailoverRuntimeState {
|
||||
const candidate = record(value);
|
||||
const date = (value: unknown) => typeof value === 'string' && Number.isFinite(Date.parse(value)) ? value : null;
|
||||
return {
|
||||
lastSwitchAt: date(candidate.lastSwitchAt),
|
||||
holdUntil: date(candidate.holdUntil),
|
||||
primaryQuarantineUntil: date(candidate.primaryQuarantineUntil),
|
||||
failoverHistory: (Array.isArray(candidate.failoverHistory) ? candidate.failoverHistory : [])
|
||||
.map(date).filter((item): item is string => Boolean(item)).slice(-20),
|
||||
reasonCode: text(candidate.reasonCode) || null,
|
||||
};
|
||||
}
|
||||
|
||||
export function normalizeAppliedFailoverPolicy(value: unknown): AppliedFailoverPolicy | null {
|
||||
if (!value) return null;
|
||||
const candidate = record(value);
|
||||
const primary = target(candidate.primary);
|
||||
const reserve = target(candidate.reserve);
|
||||
const primaryConfigFingerprint = text(candidate.primaryConfigFingerprint);
|
||||
const reserveConfigFingerprint = text(candidate.reserveConfigFingerprint);
|
||||
return primary.profileId && primary.serverId && reserve.profileId && reserve.serverId
|
||||
&& /^[a-f0-9]{64}$/.test(primaryConfigFingerprint)
|
||||
&& /^[a-f0-9]{64}$/.test(reserveConfigFingerprint)
|
||||
? { primary, reserve, primaryConfigFingerprint, reserveConfigFingerprint }
|
||||
: null;
|
||||
}
|
||||
|
||||
export function isFailoverConfigured(policy: FailoverPolicy) {
|
||||
return Boolean(
|
||||
policy.primary.profileId && policy.primary.serverId
|
||||
&& policy.reserve.profileId && policy.reserve.serverId
|
||||
&& policy.checks.length
|
||||
&& (policy.primary.profileId !== policy.reserve.profileId
|
||||
|| policy.primary.serverId !== policy.reserve.serverId)
|
||||
);
|
||||
}
|
||||
|
||||
export function createIdleFailoverSnapshot(
|
||||
policy: FailoverPolicy,
|
||||
observationEpoch = '',
|
||||
observationSequence = 0,
|
||||
): FailoverSnapshot {
|
||||
const channel = (target: FailoverTarget) => ({
|
||||
target,
|
||||
health: 'not-monitoring' as const,
|
||||
failingServiceIds: [],
|
||||
checkedAt: null,
|
||||
stateSince: null,
|
||||
});
|
||||
return {
|
||||
observationEpoch,
|
||||
observationSequence,
|
||||
configured: isFailoverConfigured(policy),
|
||||
enabled: policy.enabled,
|
||||
paused: policy.paused,
|
||||
activation: 'inactive',
|
||||
currentRole: 'none',
|
||||
status: 'idle',
|
||||
primary: channel(policy.primary),
|
||||
reserve: channel(policy.reserve),
|
||||
nextDecisionAt: null,
|
||||
reason: null,
|
||||
trafficActivity: null,
|
||||
policy,
|
||||
};
|
||||
}
|
||||
|
||||
export function nextFailoverDecision(input: FailoverDecisionInput): FailoverDecision {
|
||||
const memory = input.memory || {
|
||||
primaryFailedSince: null,
|
||||
primaryRecoveredSince: null,
|
||||
quietSince: null,
|
||||
};
|
||||
const next = { ...memory };
|
||||
if (!input.policy.enabled || input.policy.paused) {
|
||||
return { status: 'idle', reason: input.policy.paused ? 'paused' : 'disabled', switchTo: null, memory: next, nextDecisionAt: null };
|
||||
}
|
||||
if (input.primaryHealth === 'unhealthy' && input.reserveHealth === 'unhealthy') {
|
||||
next.quietSince = null;
|
||||
return { status: 'blocked', reason: 'both-unhealthy', switchTo: null, memory: next, nextDecisionAt: null };
|
||||
}
|
||||
|
||||
let target: FailoverRole | null = null;
|
||||
let readyAt: number | null = null;
|
||||
if (input.currentRole === 'primary') {
|
||||
next.primaryRecoveredSince = null;
|
||||
if (input.primaryHealth !== 'unhealthy') {
|
||||
next.primaryFailedSince = null;
|
||||
next.quietSince = null;
|
||||
return { status: 'primary', reason: input.primaryHealth === 'healthy' ? 'primary-healthy' : 'health-unknown', switchTo: null, memory: next, nextDecisionAt: null };
|
||||
}
|
||||
next.primaryFailedSince ??= input.now;
|
||||
readyAt = next.primaryFailedSince + input.policy.failureWindowMs;
|
||||
if (input.now < readyAt || input.reserveHealth !== 'healthy') {
|
||||
next.quietSince = null;
|
||||
return { status: 'observing', reason: input.reserveHealth === 'healthy' ? 'failure-window' : 'reserve-not-healthy', switchTo: null, memory: next, nextDecisionAt: readyAt };
|
||||
}
|
||||
target = 'reserve';
|
||||
} else {
|
||||
next.primaryFailedSince = null;
|
||||
if (input.primaryHealth !== 'healthy') {
|
||||
next.primaryRecoveredSince = null;
|
||||
next.quietSince = null;
|
||||
return { status: 'reserve', reason: 'primary-not-recovered', switchTo: null, memory: next, nextDecisionAt: null };
|
||||
}
|
||||
if ((input.primaryQuarantineUntil || 0) > input.now) {
|
||||
next.primaryRecoveredSince = null;
|
||||
next.quietSince = null;
|
||||
return {
|
||||
status: 'reserve',
|
||||
reason: 'recovery-hold',
|
||||
switchTo: null,
|
||||
memory: next,
|
||||
nextDecisionAt: input.primaryQuarantineUntil || null,
|
||||
};
|
||||
}
|
||||
next.primaryRecoveredSince ??= input.now;
|
||||
const recoveredAt = next.primaryRecoveredSince + input.policy.recoveryWindowMs;
|
||||
readyAt = input.reserveHealth === 'unhealthy'
|
||||
? recoveredAt
|
||||
: Math.max(recoveredAt, input.holdUntil || 0, input.primaryQuarantineUntil || 0);
|
||||
if (input.now < readyAt) {
|
||||
next.quietSince = null;
|
||||
return { status: 'reserve', reason: 'recovery-hold', switchTo: null, memory: next, nextDecisionAt: readyAt };
|
||||
}
|
||||
target = 'primary';
|
||||
}
|
||||
|
||||
if (input.policy.trafficGuard.enabled) {
|
||||
if (input.activity === 'unknown') {
|
||||
next.quietSince = null;
|
||||
return { status: 'blocked', reason: 'activity-unknown', switchTo: null, memory: next, nextDecisionAt: null };
|
||||
}
|
||||
if (input.activity === 'active') {
|
||||
next.quietSince = null;
|
||||
return { status: 'waiting-for-idle', reason: 'active-traffic', switchTo: null, memory: next, nextDecisionAt: null };
|
||||
}
|
||||
next.quietSince ??= input.now;
|
||||
readyAt = next.quietSince + input.policy.trafficGuard.quietWindowMs;
|
||||
if (input.now < readyAt) {
|
||||
return { status: 'waiting-for-idle', reason: 'quiet-window', switchTo: null, memory: next, nextDecisionAt: readyAt };
|
||||
}
|
||||
}
|
||||
return { status: 'switching', reason: target === 'reserve' ? 'primary-failed' : 'primary-recovered', switchTo: target, memory: next, nextDecisionAt: null };
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
export const HARBOR_VERSIONS = Object.freeze({
|
||||
macClient: '0.27.0',
|
||||
gatewayClient: '0.28.0',
|
||||
gatewayBackend: '0.28.0',
|
||||
macClient: '0.28.0',
|
||||
gatewayClient: '0.29.0',
|
||||
gatewayBackend: '0.29.0',
|
||||
});
|
||||
|
||||
export interface ParsedVersion {
|
||||
|
||||
@@ -25,6 +25,7 @@ const componentActions = {
|
||||
setDevicePolicy: api.devices.setPolicy,
|
||||
pingServers: api.servers.ping,
|
||||
runConnectivityDiagnostics: api.diagnostics.connectivity,
|
||||
loadActivityJournal: api.activityJournal.page,
|
||||
};
|
||||
|
||||
interface UiError {
|
||||
@@ -51,6 +52,10 @@ const operationErrorContext: Record<string, string> = {
|
||||
'subscription-refresh': 'subscription',
|
||||
'subscription-forget': 'subscription',
|
||||
'route-rules': 'routing',
|
||||
'failover-save': 'failover',
|
||||
'failover-pause': 'failover',
|
||||
'failover-resume': 'failover',
|
||||
'failover-switch': 'failover',
|
||||
};
|
||||
|
||||
function asHarborApiError(error: unknown) {
|
||||
@@ -304,6 +309,26 @@ export function App() {
|
||||
() => api.diagnostics.updateSettings(settings, revisionRef.current),
|
||||
'diagnostics',
|
||||
)}
|
||||
onSaveFailover={(policy: unknown) => run(
|
||||
'failover',
|
||||
() => api.failover.save(policy, revisionRef.current),
|
||||
'failover',
|
||||
)}
|
||||
onPauseFailover={(paused: boolean) => run(
|
||||
'failover',
|
||||
() => api.failover.pause(paused, revisionRef.current),
|
||||
'failover',
|
||||
)}
|
||||
onSwitchFailover={(role: 'primary' | 'reserve') => run(
|
||||
'failover',
|
||||
() => api.failover.switch(role, revisionRef.current),
|
||||
'failover',
|
||||
)}
|
||||
onCheckFailover={() => run(
|
||||
'failover',
|
||||
() => api.failover.check(),
|
||||
'failover',
|
||||
)}
|
||||
onDismissError={() => {
|
||||
setError(null);
|
||||
setDismissedCanonicalError(canonicalErrorId);
|
||||
|
||||
@@ -192,6 +192,26 @@ export const api = {
|
||||
},
|
||||
),
|
||||
},
|
||||
failover: {
|
||||
save: (policy: unknown, expectedRevision: number) => request('/api/failover', {
|
||||
method: 'PUT',
|
||||
body: JSON.stringify({ policy, expectedRevision }),
|
||||
}),
|
||||
pause: (paused: boolean, expectedRevision: number) => request('/api/failover/pause', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ paused, expectedRevision }),
|
||||
}),
|
||||
switch: (role: 'primary' | 'reserve', expectedRevision: number) => request('/api/failover/switch', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ role, expectedRevision }),
|
||||
}),
|
||||
check: () => request('/api/failover/check', {
|
||||
method: 'POST',
|
||||
}),
|
||||
},
|
||||
activityJournal: {
|
||||
page: (cursor: string | null = null) => request(`/api/activity-journal?limit=50${cursor ? `&cursor=${encodeURIComponent(cursor)}` : ''}`),
|
||||
},
|
||||
singbox: {
|
||||
stop: () => request('/api/singbox/stop', { method: 'POST' }),
|
||||
restart: () => request('/api/singbox/restart', { method: 'POST' }),
|
||||
@@ -237,6 +257,7 @@ export function parseHarborState(value: unknown): HarborClientState {
|
||||
selection: snapshot.selection,
|
||||
connection: snapshot.connection,
|
||||
diagnostics: snapshot.diagnostics,
|
||||
failover: snapshot.failover,
|
||||
route: snapshot.route,
|
||||
operation: snapshot.operation,
|
||||
servers: snapshot.servers,
|
||||
|
||||
@@ -47,6 +47,13 @@ import {
|
||||
InstructionsToggle,
|
||||
useInstructionsFeature,
|
||||
} from '../features/instructions/index.js';
|
||||
import { FailoverPanel, FailoverToggle, useFailoverFeature } from '../features/failover/index.js';
|
||||
import {
|
||||
ActivityJournalPanel,
|
||||
ActivityJournalToggle,
|
||||
useActivityJournalFeature,
|
||||
} from '../features/activity-journal/index.js';
|
||||
import type { FailoverPolicy } from '../../shared/failover.js';
|
||||
import {
|
||||
HARBOR_VERSIONS,
|
||||
parseVersion,
|
||||
@@ -65,9 +72,33 @@ const VERSION_PARTS = [
|
||||
] as const;
|
||||
|
||||
const DRAWER_SWITCH_MS = 620;
|
||||
const DRAWER_ORDER = ['subscription', 'instructions', 'devices', 'diagnostics', 'routing'] as const;
|
||||
const DRAWER_ORDER = ['subscription', 'failover', 'instructions', 'devices', 'diagnostics', 'routing', 'journal'] as const;
|
||||
type DrawerKey = typeof DRAWER_ORDER[number];
|
||||
|
||||
const failoverReasonLabel = (reason: string | null) => ({
|
||||
'primary-healthy': 'основной работает',
|
||||
'health-unknown': 'ожидаем проверку',
|
||||
'failure-window': 'подтверждаем сбой',
|
||||
'reserve-not-healthy': 'резерв не подтверждён',
|
||||
'both-unhealthy': 'оба канала недоступны',
|
||||
'primary-not-recovered': 'основной восстанавливается',
|
||||
'recovery-hold': 'проверяем стабильность',
|
||||
'activity-unknown': 'активность неизвестна',
|
||||
'active-traffic': 'ждём завершения работы',
|
||||
'quiet-window': 'проверяем тишину',
|
||||
'primary-failed': 'основной недоступен',
|
||||
'primary-recovered': 'основной восстановился',
|
||||
'pending-activation': 'изменения ожидают запуска',
|
||||
'vpn-stopped': 'VPN выключен',
|
||||
paused: 'автоматика на паузе',
|
||||
disabled: 'резерв выключен',
|
||||
'switch-failed': 'не удалось переключить',
|
||||
'selector-unknown': 'текущий канал неизвестен',
|
||||
'reconcile-failed': 'мониторинг временно недоступен',
|
||||
'revalidation-required': 'условия проверяются заново',
|
||||
'manual-check': 'оба канала проверены',
|
||||
}[reason || ''] || 'наблюдение');
|
||||
|
||||
interface UiError {
|
||||
context?: string;
|
||||
profileId?: string;
|
||||
@@ -93,6 +124,7 @@ interface ComponentActions {
|
||||
setDevicePolicy: (id: string, mode: 'vpn' | 'direct', expectedRevision: number) => Promise<unknown>;
|
||||
pingServers: (profileId: string, ids: string[]) => Promise<unknown>;
|
||||
runConnectivityDiagnostics: (target?: unknown) => Promise<unknown>;
|
||||
loadActivityJournal: (cursor?: string | null) => Promise<unknown>;
|
||||
}
|
||||
|
||||
interface ClientViewState extends StateSnapshot {
|
||||
@@ -119,6 +151,10 @@ interface ClientOverviewPageProps {
|
||||
onSetGatewayAuto: (enabled: boolean) => Promise<unknown>;
|
||||
onSaveRouteRules: (rules: RouteRule[], expectedRevision: number) => Promise<unknown>;
|
||||
onUpdateDiagnosticsSettings: (settings: unknown) => Promise<unknown>;
|
||||
onSaveFailover: (policy: FailoverPolicy) => Promise<unknown>;
|
||||
onPauseFailover: (paused: boolean) => Promise<unknown>;
|
||||
onSwitchFailover: (role: 'primary' | 'reserve') => Promise<unknown>;
|
||||
onCheckFailover: () => Promise<unknown>;
|
||||
onDismissError: () => void;
|
||||
}
|
||||
|
||||
@@ -252,6 +288,7 @@ const operationProgress: Partial<Record<keyof OperationRegistrySnapshot, readonl
|
||||
profileRefresh: ['subscription', 'Обновляем подписку…'],
|
||||
profileDelete: ['subscription', 'Удаляем подписку…'],
|
||||
routeRules: ['routing', 'Применяем локальные правила…'],
|
||||
failover: ['failover', 'Применяем настройки резерва…'],
|
||||
};
|
||||
|
||||
const canonicalOperationKeys: Record<string, OperationKey> = {
|
||||
@@ -266,6 +303,10 @@ const canonicalOperationKeys: Record<string, OperationKey> = {
|
||||
'profile-delete': 'profileDelete',
|
||||
'gateway-auto': 'gatewayAuto',
|
||||
'route-rules': 'routeRules',
|
||||
'failover-save': 'failover',
|
||||
'failover-pause': 'failover',
|
||||
'failover-resume': 'failover',
|
||||
'failover-switch': 'failover',
|
||||
'subscription-import': 'profileAdd',
|
||||
'subscription-refresh': 'profileRefresh',
|
||||
'subscription-forget': 'profileDelete',
|
||||
@@ -425,6 +466,10 @@ export function ClientOverviewPage({
|
||||
onSetGatewayAuto,
|
||||
onSaveRouteRules,
|
||||
onUpdateDiagnosticsSettings,
|
||||
onSaveFailover,
|
||||
onPauseFailover,
|
||||
onSwitchFailover,
|
||||
onCheckFailover,
|
||||
onDismissError,
|
||||
}: ClientOverviewPageProps) {
|
||||
const isGateway = state?.mode === 'gateway';
|
||||
@@ -533,6 +578,8 @@ export function ClientOverviewPage({
|
||||
port: state?.clientRuntime?.proxyPort || (isGateway ? 8080 : 8082),
|
||||
controlHost,
|
||||
});
|
||||
const failoverFeature = useFailoverFeature();
|
||||
const activityJournalFeature = useActivityJournalFeature();
|
||||
const diagnosticsAvailable = hasSubscription;
|
||||
const drawerControls = {
|
||||
subscription: {
|
||||
@@ -541,6 +588,12 @@ export function ClientOverviewPage({
|
||||
show: subscriptionFeature.toggle,
|
||||
close: subscriptionFeature.close,
|
||||
},
|
||||
failover: {
|
||||
isOpen: failoverFeature.isOpen,
|
||||
panelRef: failoverFeature.panelRef,
|
||||
show: failoverFeature.toggle,
|
||||
close: failoverFeature.close,
|
||||
},
|
||||
instructions: {
|
||||
isOpen: instructionsFeature.isOpen,
|
||||
panelRef: instructionsFeature.panelRef,
|
||||
@@ -565,10 +618,16 @@ export function ClientOverviewPage({
|
||||
show: routingFeature.open,
|
||||
close: routingFeature.forceClose,
|
||||
},
|
||||
journal: {
|
||||
isOpen: activityJournalFeature.isOpen,
|
||||
panelRef: activityJournalFeature.panelRef,
|
||||
show: activityJournalFeature.toggle,
|
||||
close: activityJournalFeature.close,
|
||||
},
|
||||
};
|
||||
const drawerOrder = isGateway
|
||||
? DRAWER_ORDER
|
||||
: DRAWER_ORDER.filter((drawer) => drawer !== 'devices');
|
||||
: DRAWER_ORDER.filter((drawer) => !['devices', 'failover', 'journal'].includes(drawer));
|
||||
const activeRailDrawer = drawerSwitchTarget && drawerControls[drawerSwitchTarget].isOpen
|
||||
? drawerSwitchTarget
|
||||
: drawerOrder.find((drawer) => drawerControls[drawer].isOpen) || null;
|
||||
@@ -587,6 +646,8 @@ export function ClientOverviewPage({
|
||||
instructionsFeature.close();
|
||||
devicesFeature.close();
|
||||
diagnosticsFeature.close();
|
||||
failoverFeature.close();
|
||||
activityJournalFeature.close();
|
||||
}
|
||||
}, [hasSubscription, isGateway]);
|
||||
|
||||
@@ -656,6 +717,7 @@ export function ClientOverviewPage({
|
||||
routingFeature.requestClose();
|
||||
return;
|
||||
}
|
||||
if (current === 'failover' && !failoverFeature.beforeCloseRef.current()) return;
|
||||
if (!current) {
|
||||
drawerControls[target].show();
|
||||
return;
|
||||
@@ -733,6 +795,13 @@ export function ClientOverviewPage({
|
||||
: switchingServer && operationProfile && operationServer
|
||||
? `Переключаем на ${subscriptionDomain(operationProfile.subscription.host)} · ${operationServer.label}`
|
||||
: '';
|
||||
const failoverIdentity = isGateway && state.failover.enabled
|
||||
? `${state.failover.currentRole === 'reserve'
|
||||
? 'Резервный канал'
|
||||
: state.failover.currentRole === 'primary'
|
||||
? 'Основной канал'
|
||||
: 'Текущий канал вне резервной пары'} · ${failoverReasonLabel(state.failover.reason)}`
|
||||
: '';
|
||||
|
||||
return (
|
||||
<div
|
||||
@@ -752,6 +821,11 @@ export function ClientOverviewPage({
|
||||
open={activeRailDrawer === 'subscription'}
|
||||
onToggle={() => switchDrawer('subscription')}
|
||||
/>
|
||||
{isGateway && <FailoverToggle
|
||||
feature={failoverFeature}
|
||||
open={activeRailDrawer === 'failover'}
|
||||
onToggle={() => switchDrawer('failover')}
|
||||
/>}
|
||||
<InstructionsToggle
|
||||
feature={instructionsFeature}
|
||||
open={activeRailDrawer === 'instructions'}
|
||||
@@ -775,6 +849,11 @@ export function ClientOverviewPage({
|
||||
hasSubscription={hasSubscription}
|
||||
onOpen={() => switchDrawer('routing')}
|
||||
/>
|
||||
{isGateway && <ActivityJournalToggle
|
||||
feature={activityJournalFeature}
|
||||
open={activeRailDrawer === 'journal'}
|
||||
onToggle={() => switchDrawer('journal')}
|
||||
/>}
|
||||
</nav>}
|
||||
<main className={`client-panel${showPower ? '' : ' is-setup'}${!isGateway && hasSubscription ? ' has-subscription' : ''}${isGateway && hasSubscription ? ' is-gateway-home' : ''}`}>
|
||||
<ConnectionPanel
|
||||
@@ -807,7 +886,7 @@ export function ClientOverviewPage({
|
||||
blocked={connectionBlocked}
|
||||
onRestart={onRestart}
|
||||
/>}
|
||||
serverSlot={<AppliedIdentity identity={mainIdentity} operation={switchIdentity} />}
|
||||
serverSlot={<AppliedIdentity identity={mainIdentity} operation={switchIdentity || failoverIdentity} />}
|
||||
statusSlot={<>
|
||||
<InlineError error={error} context="connection" />
|
||||
<InlineProgress operations={visibleOperations} context="connection" />
|
||||
@@ -857,6 +936,22 @@ export function ClientOverviewPage({
|
||||
<InlineProgress operations={visibleOperations} context="routing" />
|
||||
</>}
|
||||
/>}
|
||||
{isGateway && hasSubscription && <FailoverPanel
|
||||
feature={failoverFeature}
|
||||
snapshot={state.failover}
|
||||
profiles={profiles}
|
||||
diagnostics={state.diagnostics}
|
||||
blocked={operationBlocked(visibleOperations, 'failover')}
|
||||
onSave={onSaveFailover}
|
||||
onPause={onPauseFailover}
|
||||
onSwitch={onSwitchFailover}
|
||||
onCheck={onCheckFailover}
|
||||
onUpdateDiagnostics={onUpdateDiagnosticsSettings}
|
||||
/>}
|
||||
{isGateway && hasSubscription && <ActivityJournalPanel
|
||||
feature={activityJournalFeature}
|
||||
loadPage={actions.loadActivityJournal}
|
||||
/>}
|
||||
<RoutingDiscardDialog feature={routingFeature} />
|
||||
<SubscriptionDeleteDialog feature={subscriptionFeature} />
|
||||
</div>
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import { assertActivityJournalPage, type ActivityJournalEvent } from '../../../shared/activityJournal.js';
|
||||
import { Drawer } from '../../ui/Drawer.js';
|
||||
import { RailAction } from '../../ui/RailAction.js';
|
||||
|
||||
export function useActivityJournalFeature() {
|
||||
const [isOpen, setIsOpen] = useState(false);
|
||||
const panelRef = useRef<HTMLElement>(null);
|
||||
const toggleRef = useRef<HTMLButtonElement>(null);
|
||||
const closeRef = useRef<HTMLButtonElement>(null);
|
||||
useEffect(() => {
|
||||
if (!isOpen) return undefined;
|
||||
const frame = requestAnimationFrame(() => closeRef.current?.focus());
|
||||
const close = (event: PointerEvent | KeyboardEvent) => {
|
||||
if (event.type === 'keydown' && (event as KeyboardEvent).key !== 'Escape') return;
|
||||
if (event.type !== 'keydown' && (
|
||||
panelRef.current?.contains(event.target as Node) || toggleRef.current?.contains(event.target as Node)
|
||||
)) return;
|
||||
setIsOpen(false);
|
||||
};
|
||||
document.addEventListener('pointerdown', close);
|
||||
document.addEventListener('keydown', close);
|
||||
return () => {
|
||||
cancelAnimationFrame(frame);
|
||||
document.removeEventListener('pointerdown', close);
|
||||
document.removeEventListener('keydown', close);
|
||||
requestAnimationFrame(() => {
|
||||
if (panelRef.current?.contains(document.activeElement)) toggleRef.current?.focus();
|
||||
});
|
||||
};
|
||||
}, [isOpen]);
|
||||
return { isOpen, panelRef, toggleRef, closeRef, close: () => setIsOpen(false), toggle: () => setIsOpen((value) => !value) };
|
||||
}
|
||||
|
||||
export type ActivityJournalFeature = ReturnType<typeof useActivityJournalFeature>;
|
||||
|
||||
export function ActivityJournalToggle({ feature, open, onToggle }: {
|
||||
feature: ActivityJournalFeature;
|
||||
open: boolean;
|
||||
onToggle: () => void;
|
||||
}) {
|
||||
return <RailAction
|
||||
buttonRef={feature.toggleRef}
|
||||
className="client-journal-toggle"
|
||||
open={open}
|
||||
controls="client-activity-journal"
|
||||
ariaLabel={open ? 'Закрыть журнал событий' : 'Открыть журнал событий'}
|
||||
label="Журнал"
|
||||
onClick={onToggle}
|
||||
>
|
||||
<svg viewBox="0 0 24 24" aria-hidden="true">
|
||||
<path d="M5 8V4m0 4h4M5.6 7.1A8 8 0 1 1 4 12M12 7.5V12l3 2" />
|
||||
<circle cx="12" cy="12" r=".8" />
|
||||
</svg>
|
||||
</RailAction>;
|
||||
}
|
||||
|
||||
function eventCopy(event: ActivityJournalEvent) {
|
||||
const value = event.data;
|
||||
const copies: Record<string, [string, string]> = {
|
||||
'connection.started': ['VPN включён', [value.profileLabel, value.serverLabel].filter(Boolean).join(' · ')],
|
||||
'connection.stopped': ['VPN выключен', 'Остановлен пользователем'],
|
||||
'connection.failed': ['VPN не запущен', String(value.errorCode || '')],
|
||||
'subscription.added': ['Подписка добавлена', `${value.profileLabel || ''} · серверов: ${value.serverCount || 0}`],
|
||||
'subscription.refreshed': ['Подписка обновлена', `${value.profileLabel || ''} · серверов: ${value.serverCount || 0} · +${value.added || 0} / −${value.removed || 0}`],
|
||||
'subscription.refresh_failed': ['Подписка не обновлена', `${value.profileLabel || ''} · ${value.errorCode || ''}`],
|
||||
'subscription.deleted': ['Подписка удалена', String(value.profileLabel || '')],
|
||||
'failover.enabled': ['Резервный канал включён', 'Мониторинг начнётся после активации dual-config'],
|
||||
'failover.disabled': ['Резервный канал выключен', 'Автоматика полностью остановлена'],
|
||||
'failover.paused': ['Автопереключение на паузе', 'Проверки продолжаются'],
|
||||
'failover.resumed': ['Автопереключение возобновлено', ''],
|
||||
'failover.waiting_for_idle': ['Переключение отложено', 'Обнаружен активный трафик'],
|
||||
'failover.switched': ['Новые соединения переключены', `${value.fromRole || ''} → ${value.toRole || ''} · ${value.reason || ''}`],
|
||||
'failover.switch_failed': ['Переключение не выполнено', String(value.errorCode || '')],
|
||||
'failover.both_unhealthy': ['Оба канала недоступны', 'Текущий маршрут сохранён'],
|
||||
'failover.recovered': ['Основной канал восстановлен', String(value.reason || '')],
|
||||
'journal.recovered': ['Журнал восстановлен', 'Повреждённый файл сохранён отдельно'],
|
||||
};
|
||||
return copies[event.type] || ['Системное событие', ''];
|
||||
}
|
||||
|
||||
function dayLabel(value: string) {
|
||||
const date = new Date(value);
|
||||
const today = new Date();
|
||||
const startDate = new Date(today.getFullYear(), today.getMonth(), today.getDate());
|
||||
const start = startDate.getTime();
|
||||
const yesterday = new Date(startDate);
|
||||
yesterday.setDate(yesterday.getDate() - 1);
|
||||
const day = new Date(date.getFullYear(), date.getMonth(), date.getDate()).getTime();
|
||||
if (day === start) return 'Сегодня';
|
||||
if (day === yesterday.getTime()) return 'Вчера';
|
||||
return new Intl.DateTimeFormat('ru-RU', { day: 'numeric', month: 'long' }).format(date);
|
||||
}
|
||||
|
||||
export function ActivityJournalPanel({ feature, loadPage }: {
|
||||
feature: ActivityJournalFeature;
|
||||
loadPage: (cursor?: string | null) => Promise<unknown>;
|
||||
}) {
|
||||
const [events, setEvents] = useState<ActivityJournalEvent[]>([]);
|
||||
const [nextCursor, setNextCursor] = useState<string | null>(null);
|
||||
const [status, setStatus] = useState<'idle' | 'loading' | 'ready' | 'refreshing' | 'older' | 'error'>('idle');
|
||||
const [announcement, setAnnouncement] = useState('');
|
||||
|
||||
async function load(cursor: string | null = null, refresh = false) {
|
||||
setStatus(cursor ? 'older' : refresh ? 'refreshing' : 'loading');
|
||||
try {
|
||||
const page = assertActivityJournalPage(await loadPage(cursor));
|
||||
if (page.storage.status === 'error') throw new Error('journal unavailable');
|
||||
if (refresh) {
|
||||
const known = new Set(events.map(({ id }) => id));
|
||||
setAnnouncement(`Журнал обновлён, новых событий: ${page.events.filter(({ id }) => !known.has(id)).length}`);
|
||||
}
|
||||
setEvents((current) => cursor ? [...current, ...page.events] : page.events);
|
||||
setNextCursor(page.nextCursor);
|
||||
setStatus('ready');
|
||||
} catch {
|
||||
setStatus('error');
|
||||
}
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
if (feature.isOpen) void load(null, status !== 'idle');
|
||||
}, [feature.isOpen]);
|
||||
|
||||
const groups = events.reduce<Array<{ label: string; events: ActivityJournalEvent[] }>>((result, event) => {
|
||||
const label = dayLabel(event.occurredAt);
|
||||
const group = result.at(-1);
|
||||
if (group?.label === label) group.events.push(event);
|
||||
else result.push({ label, events: [event] });
|
||||
return result;
|
||||
}, []);
|
||||
|
||||
return <Drawer
|
||||
panelRef={feature.panelRef}
|
||||
closeRef={feature.closeRef}
|
||||
id="client-activity-journal"
|
||||
open={feature.isOpen}
|
||||
label="Журнал Harbor"
|
||||
closeLabel="Закрыть журнал"
|
||||
onClose={feature.close}
|
||||
className="client-journal-drawer"
|
||||
>
|
||||
<header className="client-journal-header">
|
||||
<span>Важные события хранятся 30 дней</span>
|
||||
<div><h2>Журнал</h2><button type="button" aria-label="Обновить журнал" disabled={status === 'refreshing'} onClick={() => void load(null, true)}>
|
||||
<svg viewBox="0 0 24 24" aria-hidden="true"><path d="M20 11a8 8 0 1 0-2.3 6.7M20 5v6h-6" /></svg>
|
||||
</button></div>
|
||||
</header>
|
||||
<span className="client-live-region" role="status" aria-live="polite">{announcement}</span>
|
||||
{status === 'error' && <div className="client-journal-error" role="status">Журнал временно недоступен <button type="button" onClick={() => void load()}>Повторить</button></div>}
|
||||
{status === 'loading' && !events.length ? <div className="client-journal-skeleton" aria-label="Загружаем журнал">{[0, 1, 2, 3].map((value) => <span key={value} />)}</div>
|
||||
: !events.length && status === 'ready' ? <p className="client-journal-empty">За последние 30 дней важных событий пока нет</p>
|
||||
: <div className="client-journal-groups">{groups.map((group) => <section key={group.label}>
|
||||
<h3>{group.label}</h3>
|
||||
<ol>{group.events.map((event) => {
|
||||
const [title, details] = eventCopy(event);
|
||||
return <li key={event.id} className={event.severity === 'error' ? 'is-error' : event.severity === 'warning' ? 'is-warning' : ''}>
|
||||
<div className="client-journal-time"><time dateTime={event.occurredAt}>{new Intl.DateTimeFormat('ru-RU', { hour: '2-digit', minute: '2-digit' }).format(new Date(event.occurredAt))}</time><span>{event.source}</span></div>
|
||||
<div><strong>{title}</strong>{details && <span>{details}</span>}{event.severity !== 'info' && <em>{event.severity === 'error' ? 'Ошибка' : 'Внимание'}</em>}</div>
|
||||
</li>;
|
||||
})}</ol>
|
||||
</section>)}</div>}
|
||||
<footer className="client-journal-footer">
|
||||
{nextCursor ? <button type="button" disabled={status === 'older'} onClick={() => void load(nextCursor)}>{status === 'older' ? 'Загружаем…' : 'Показать ещё'}</button> : <span>{events.length ? 'Это вся история за последние 30 дней' : 'Храним события 30 дней'}</span>}
|
||||
</footer>
|
||||
</Drawer>;
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
export { ActivityJournalPanel, ActivityJournalToggle, useActivityJournalFeature } from './ActivityJournalFeature.js';
|
||||
@@ -23,6 +23,7 @@ import {
|
||||
type DiagnosticSiteResult,
|
||||
} from './connectivityResult.js';
|
||||
import type { DiagnosticsFeature } from './DiagnosticsFeature.js';
|
||||
import { saveCustomDiagnosticService } from './customServiceAction.js';
|
||||
|
||||
const CUSTOM_SERVICES_KEY = 'harbor-diagnostic-services';
|
||||
const HIDDEN_SERVICES_KEY = 'harbor-hidden-diagnostic-services';
|
||||
@@ -334,19 +335,15 @@ export function ConnectivityDiagnosticsPanel({
|
||||
async function addService(event: FormEvent<HTMLFormElement>) {
|
||||
event.preventDefault();
|
||||
try {
|
||||
if (customServices.length >= MAX_CUSTOM_DIAGNOSTIC_SERVICES) return;
|
||||
const parsed = new URL(serviceUrl.trim());
|
||||
if (parsed.protocol !== 'https:') throw new Error('Нужен публичный HTTPS-адрес.');
|
||||
setSettingsSaving(true);
|
||||
const saved = await updateSettings({
|
||||
customServices: [...customServices, {
|
||||
id: `custom-${globalThis.crypto?.randomUUID?.() || Date.now()}`,
|
||||
label: serviceName.trim() || parsed.hostname,
|
||||
url: parsed.href,
|
||||
}],
|
||||
const saved = await saveCustomDiagnosticService({
|
||||
name: serviceName,
|
||||
url: serviceUrl,
|
||||
customServices,
|
||||
hiddenServiceIds,
|
||||
updateSettings,
|
||||
});
|
||||
if (saved === false) throw new Error('Не удалось сохранить сервис.');
|
||||
if (!saved) return;
|
||||
setServiceName('');
|
||||
setServiceUrl('');
|
||||
setFormError('');
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
import {
|
||||
MAX_CUSTOM_DIAGNOSTIC_SERVICES,
|
||||
type DiagnosticService,
|
||||
type DiagnosticSettings,
|
||||
} from '../../../shared/connectivityDiagnostics.js';
|
||||
|
||||
export async function saveCustomDiagnosticService({
|
||||
name,
|
||||
url,
|
||||
customServices,
|
||||
hiddenServiceIds,
|
||||
updateSettings,
|
||||
}: {
|
||||
name: string;
|
||||
url: string;
|
||||
customServices: DiagnosticService[];
|
||||
hiddenServiceIds: string[];
|
||||
updateSettings: (settings: Pick<DiagnosticSettings, 'customServices' | 'hiddenServiceIds'>) => Promise<unknown>;
|
||||
}) {
|
||||
if (customServices.length >= MAX_CUSTOM_DIAGNOSTIC_SERVICES) return false;
|
||||
const parsed = new URL(url.trim());
|
||||
if (parsed.protocol !== 'https:') throw new Error('Нужен публичный HTTPS-адрес.');
|
||||
const saved = await updateSettings({
|
||||
customServices: [...customServices, {
|
||||
id: `custom-${globalThis.crypto?.randomUUID?.() || Date.now()}`,
|
||||
label: name.trim() || parsed.hostname,
|
||||
url: parsed.href,
|
||||
}],
|
||||
hiddenServiceIds,
|
||||
});
|
||||
if (saved === false) throw new Error('Не удалось сохранить сервис.');
|
||||
return true;
|
||||
}
|
||||
@@ -4,3 +4,4 @@ export {
|
||||
useDiagnosticsFeature,
|
||||
type DiagnosticsFeature,
|
||||
} from './DiagnosticsFeature.js';
|
||||
export { saveCustomDiagnosticService } from './customServiceAction.js';
|
||||
|
||||
@@ -0,0 +1,331 @@
|
||||
import { useEffect, useMemo, useRef, useState } from 'react';
|
||||
import { CONNECTIVITY_SITES, MAX_CUSTOM_DIAGNOSTIC_SERVICES, type DiagnosticSettings } from '../../../shared/connectivityDiagnostics.js';
|
||||
import { normalizeFailoverPolicy, type FailoverPolicy, type FailoverSnapshot } from '../../../shared/failover.js';
|
||||
import type { ProfileSnapshot } from '../../../shared/contracts/state.js';
|
||||
import { Drawer } from '../../ui/Drawer.js';
|
||||
import { RailAction } from '../../ui/RailAction.js';
|
||||
import { saveCustomDiagnosticService } from '../diagnostics/index.js';
|
||||
|
||||
export function useFailoverFeature() {
|
||||
const [isOpen, setIsOpen] = useState(false);
|
||||
const panelRef = useRef<HTMLElement>(null);
|
||||
const toggleRef = useRef<HTMLButtonElement>(null);
|
||||
const closeRef = useRef<HTMLButtonElement>(null);
|
||||
const beforeCloseRef = useRef<() => boolean>(() => true);
|
||||
const close = () => {
|
||||
if (beforeCloseRef.current()) setIsOpen(false);
|
||||
};
|
||||
useEffect(() => {
|
||||
if (!isOpen) return undefined;
|
||||
const frame = requestAnimationFrame(() => closeRef.current?.focus());
|
||||
const handleClose = (event: PointerEvent | KeyboardEvent) => {
|
||||
if (event.type === 'keydown' && (event as KeyboardEvent).key !== 'Escape') return;
|
||||
if (event.type !== 'keydown' && (
|
||||
panelRef.current?.contains(event.target as Node) || toggleRef.current?.contains(event.target as Node)
|
||||
)) return;
|
||||
close();
|
||||
};
|
||||
document.addEventListener('pointerdown', handleClose);
|
||||
document.addEventListener('keydown', handleClose);
|
||||
return () => {
|
||||
cancelAnimationFrame(frame);
|
||||
document.removeEventListener('pointerdown', handleClose);
|
||||
document.removeEventListener('keydown', handleClose);
|
||||
requestAnimationFrame(() => {
|
||||
if (panelRef.current?.contains(document.activeElement)) toggleRef.current?.focus();
|
||||
});
|
||||
};
|
||||
}, [isOpen]);
|
||||
return {
|
||||
isOpen, panelRef, toggleRef, closeRef, beforeCloseRef, close,
|
||||
toggle: () => isOpen ? close() : setIsOpen(true),
|
||||
};
|
||||
}
|
||||
|
||||
export type FailoverFeature = ReturnType<typeof useFailoverFeature>;
|
||||
|
||||
export function FailoverToggle({ feature, open, onToggle }: {
|
||||
feature: FailoverFeature;
|
||||
open: boolean;
|
||||
onToggle: () => void;
|
||||
}) {
|
||||
return <RailAction
|
||||
buttonRef={feature.toggleRef}
|
||||
className="client-failover-toggle"
|
||||
open={open}
|
||||
controls="client-failover"
|
||||
ariaLabel={open ? 'Закрыть резервный канал' : 'Настроить резервный канал'}
|
||||
label="Резерв"
|
||||
onClick={onToggle}
|
||||
>
|
||||
<svg viewBox="0 0 24 24" aria-hidden="true">
|
||||
<path d="M5 7h11m-3-3 3 3-3 3M19 17H8m3-3-3 3 3 3" />
|
||||
<circle cx="4" cy="7" r="1" /><circle cx="20" cy="17" r="1" />
|
||||
</svg>
|
||||
</RailAction>;
|
||||
}
|
||||
|
||||
const seconds = (milliseconds: number) => Math.round(milliseconds / 1000);
|
||||
const milliseconds = (value: string, fallback: number) => {
|
||||
const parsed = Number(value);
|
||||
return Number.isFinite(parsed) ? Math.round(parsed * 1000) : fallback;
|
||||
};
|
||||
|
||||
const reasonLabel = (reason: string | null) => ({
|
||||
'primary-healthy': 'Основной канал работает',
|
||||
'health-unknown': 'Ожидаем результаты проверки',
|
||||
'failure-window': 'Подтверждаем сбой основного канала',
|
||||
'reserve-not-healthy': 'Резервный канал ещё не подтверждён',
|
||||
'both-unhealthy': 'Оба канала недоступны',
|
||||
'primary-not-recovered': 'Основной канал восстанавливается',
|
||||
'recovery-hold': 'Проверяем стабильность основного канала',
|
||||
'activity-unknown': 'Не удалось определить активность',
|
||||
'active-traffic': 'Ждём завершения активной работы',
|
||||
'quiet-window': 'Проверяем тишину перед переключением',
|
||||
'primary-failed': 'Основной канал недоступен',
|
||||
'primary-recovered': 'Основной канал восстановился',
|
||||
'pending-activation': 'Изменения ожидают следующего запуска VPN',
|
||||
'vpn-stopped': 'VPN выключен',
|
||||
paused: 'Автоматика на паузе',
|
||||
disabled: 'Резерв выключен',
|
||||
'switch-failed': 'Не удалось переключить канал',
|
||||
'selector-unknown': 'Не удалось подтвердить текущий канал',
|
||||
'reconcile-failed': 'Настройки сохранены, мониторинг временно недоступен',
|
||||
'revalidation-required': 'Условия переключения проверяются заново',
|
||||
'manual-check': 'Оба канала проверены',
|
||||
}[reason || ''] || 'Наблюдаем за каналами');
|
||||
|
||||
function targetLabel(profiles: ProfileSnapshot[], target: { profileId: string; serverId: string }) {
|
||||
const profile = profiles.find(({ id }) => id === target.profileId);
|
||||
const server = profile?.servers.find(({ id }) => id === target.serverId);
|
||||
return profile && server ? `${profile.label} · ${server.label}` : 'Не выбран';
|
||||
}
|
||||
|
||||
export function FailoverPanel({
|
||||
feature,
|
||||
snapshot,
|
||||
profiles,
|
||||
diagnostics,
|
||||
blocked,
|
||||
onSave,
|
||||
onPause,
|
||||
onSwitch,
|
||||
onCheck,
|
||||
onUpdateDiagnostics,
|
||||
}: {
|
||||
feature: FailoverFeature;
|
||||
snapshot: FailoverSnapshot;
|
||||
profiles: ProfileSnapshot[];
|
||||
diagnostics: DiagnosticSettings;
|
||||
blocked: boolean;
|
||||
onSave: (policy: FailoverPolicy) => Promise<unknown>;
|
||||
onPause: (paused: boolean) => Promise<unknown>;
|
||||
onSwitch: (role: 'primary' | 'reserve') => Promise<unknown>;
|
||||
onCheck: () => Promise<unknown>;
|
||||
onUpdateDiagnostics: (settings: unknown) => Promise<unknown>;
|
||||
}) {
|
||||
const [draft, setDraft] = useState(() => snapshot.policy);
|
||||
const [dirty, setDirty] = useState(false);
|
||||
const [confirmDiscard, setConfirmDiscard] = useState(false);
|
||||
const [addingService, setAddingService] = useState(false);
|
||||
const [serviceName, setServiceName] = useState('');
|
||||
const [serviceUrl, setServiceUrl] = useState('');
|
||||
const [serviceError, setServiceError] = useState('');
|
||||
useEffect(() => {
|
||||
if (!dirty) setDraft(snapshot.policy);
|
||||
}, [snapshot.policy, dirty]);
|
||||
useEffect(() => {
|
||||
feature.beforeCloseRef.current = () => {
|
||||
if (!dirty) return true;
|
||||
setConfirmDiscard(true);
|
||||
return false;
|
||||
};
|
||||
const beforeUnload = (event: BeforeUnloadEvent) => {
|
||||
if (!dirty) return;
|
||||
event.preventDefault();
|
||||
};
|
||||
window.addEventListener('beforeunload', beforeUnload);
|
||||
return () => {
|
||||
feature.beforeCloseRef.current = () => true;
|
||||
window.removeEventListener('beforeunload', beforeUnload);
|
||||
};
|
||||
}, [dirty, feature.beforeCloseRef]);
|
||||
const services = useMemo(() => [
|
||||
...CONNECTIVITY_SITES,
|
||||
...diagnostics.customServices,
|
||||
], [diagnostics.customServices]);
|
||||
const update = (value: Partial<FailoverPolicy>) => {
|
||||
setDraft((current) => normalizeFailoverPolicy({ ...current, ...value }));
|
||||
setDirty(true);
|
||||
};
|
||||
const updateTarget = (role: 'primary' | 'reserve', patch: Partial<FailoverPolicy[typeof role]>) => {
|
||||
const next = { ...draft[role], ...patch };
|
||||
const profile = profiles.find(({ id }) => id === next.profileId);
|
||||
if (patch.profileId !== undefined) next.serverId = profile?.desiredServerId || profile?.servers[0]?.id || '';
|
||||
update({ [role]: next } as Partial<FailoverPolicy>);
|
||||
};
|
||||
const role = snapshot.currentRole === 'primary' || snapshot.currentRole === 'reserve'
|
||||
? snapshot.currentRole
|
||||
: null;
|
||||
const switchRole = role === 'primary' ? 'reserve' : role === 'reserve' ? 'primary' : null;
|
||||
const activity = snapshot.trafficActivity;
|
||||
const targetExists = (target: FailoverPolicy['primary']) => profiles
|
||||
.find(({ id }) => id === target.profileId)?.servers.some(({ id }) => id === target.serverId);
|
||||
const targetsValid = targetExists(draft.primary) && targetExists(draft.reserve)
|
||||
&& (draft.primary.profileId !== draft.reserve.profileId || draft.primary.serverId !== draft.reserve.serverId);
|
||||
const draftValid = Boolean(targetsValid && draft.checks.length);
|
||||
const quietElapsed = activity?.quietSince
|
||||
? Math.max(0, Date.now() - Date.parse(activity.quietSince))
|
||||
: 0;
|
||||
const addService = async () => {
|
||||
try {
|
||||
const saved = await saveCustomDiagnosticService({
|
||||
name: serviceName,
|
||||
url: serviceUrl,
|
||||
customServices: diagnostics.customServices,
|
||||
hiddenServiceIds: diagnostics.hiddenServiceIds,
|
||||
updateSettings: onUpdateDiagnostics,
|
||||
});
|
||||
if (!saved) return;
|
||||
setServiceName('');
|
||||
setServiceUrl('');
|
||||
setServiceError('');
|
||||
setAddingService(false);
|
||||
} catch (error) {
|
||||
setServiceError(error instanceof Error ? error.message : 'Проверьте адрес.');
|
||||
}
|
||||
};
|
||||
|
||||
return <Drawer
|
||||
panelRef={feature.panelRef}
|
||||
closeRef={feature.closeRef}
|
||||
id="client-failover"
|
||||
open={feature.isOpen}
|
||||
label="Резервный канал"
|
||||
closeLabel="Закрыть резервный канал"
|
||||
onClose={feature.close}
|
||||
className="client-failover-drawer"
|
||||
>
|
||||
<header className="client-failover-header">
|
||||
<span>Gateway</span>
|
||||
<h2>Резервный канал</h2>
|
||||
<p>Переключает только новые соединения. Уже открытые соединения Harbor не закрывает.</p>
|
||||
</header>
|
||||
|
||||
<form className="client-failover-form" onSubmit={(event) => {
|
||||
event.preventDefault();
|
||||
void onSave(draft).then((result) => { if (result !== false) setDirty(false); });
|
||||
}}>
|
||||
<label className="client-failover-master">
|
||||
<span><strong>Использовать резерв</strong><small>{draft.enabled ? 'Мониторинг включён' : 'Полностью пассивен'}</small></span>
|
||||
<input type="checkbox" checked={draft.enabled} onChange={(event) => update({ enabled: event.target.checked })} />
|
||||
</label>
|
||||
|
||||
{(['primary', 'reserve'] as const).map((channel) => {
|
||||
const profile = profiles.find(({ id }) => id === draft[channel].profileId);
|
||||
const health = snapshot[channel].health;
|
||||
const missing = Boolean(snapshot[channel].target.profileId) && !targetExists(snapshot[channel].target);
|
||||
const healthLabel = missing
|
||||
? 'Цель недоступна'
|
||||
: channel === 'primary' && snapshot.reason === 'failure-window'
|
||||
? `Нестабилен · ${seconds(Math.max(0, Date.now() - Date.parse(snapshot.primary.stateSince || new Date().toISOString())))} из ${seconds(draft.failureWindowMs)} с`
|
||||
: channel === 'primary' && snapshot.currentRole === 'reserve' && health === 'healthy' && snapshot.reason === 'recovery-hold'
|
||||
? 'Восстанавливается'
|
||||
: health === 'healthy' ? 'Работает' : health === 'unhealthy' ? 'Недоступен' : health === 'not-monitoring' ? 'Не проверяется' : 'Нет данных';
|
||||
return <section className="client-failover-channel" key={channel}>
|
||||
<div className="client-failover-channel-title">
|
||||
<span>{channel === 'primary' ? 'Основной' : 'Резервный'}</span>
|
||||
<strong className={health === 'healthy' ? 'is-healthy' : health === 'unhealthy' || missing ? 'is-unhealthy' : ''}>{healthLabel}</strong>
|
||||
</div>
|
||||
<label><span>Подписка</span><select value={draft[channel].profileId} onChange={(event) => updateTarget(channel, { profileId: event.target.value })}>
|
||||
<option value="">Не выбрана</option>
|
||||
{profiles.map((item) => <option value={item.id} key={item.id}>{item.label}</option>)}
|
||||
</select></label>
|
||||
<label><span>Сервер</span><select value={draft[channel].serverId} onChange={(event) => updateTarget(channel, { serverId: event.target.value })} disabled={!profile}>
|
||||
<option value="">Не выбран</option>
|
||||
{profile?.servers.map((server) => <option value={server.id} key={server.id}>{server.label}</option>)}
|
||||
</select></label>
|
||||
</section>;
|
||||
})}
|
||||
|
||||
<fieldset className="client-failover-services">
|
||||
<legend>Что проверять</legend>
|
||||
{services.map((service) => {
|
||||
const check = draft.checks.find(({ serviceId }) => serviceId === service.id);
|
||||
return <div className="client-failover-service" key={service.id}>
|
||||
<label>
|
||||
<input type="checkbox" checked={Boolean(check)} onChange={(event) => update({
|
||||
checks: event.target.checked
|
||||
? [...draft.checks, { serviceId: service.id, timeoutMs: 6_000 }]
|
||||
: draft.checks.filter(({ serviceId }) => serviceId !== service.id),
|
||||
})} />
|
||||
<span>{service.label}</span>
|
||||
</label>
|
||||
{check && <label className="client-failover-service-timeout">
|
||||
<span>таймаут, сек</span>
|
||||
<input
|
||||
type="number"
|
||||
min="2"
|
||||
max="30"
|
||||
aria-label={`Таймаут проверки: ${service.label}`}
|
||||
value={seconds(check.timeoutMs)}
|
||||
onChange={(event) => update({ checks: draft.checks.map((item) => item.serviceId === service.id
|
||||
? { ...item, timeoutMs: milliseconds(event.target.value, item.timeoutMs) }
|
||||
: item) })}
|
||||
/>
|
||||
</label>}
|
||||
</div>;
|
||||
})}
|
||||
{!addingService && <button type="button" className="client-failover-add-service" disabled={diagnostics.customServices.length >= MAX_CUSTOM_DIAGNOSTIC_SERVICES} onClick={() => setAddingService(true)}>+ Добавить HTTPS-сервис</button>}
|
||||
{addingService && <div className="client-failover-service-editor">
|
||||
<input aria-label="Название HTTPS-сервиса" placeholder="Название" value={serviceName} onChange={(event) => setServiceName(event.target.value)} />
|
||||
<input aria-label="HTTPS-адрес сервиса" placeholder="https://example.com/health" value={serviceUrl} onChange={(event) => setServiceUrl(event.target.value)} />
|
||||
{serviceError && <span role="alert">{serviceError}</span>}
|
||||
<button type="button" onClick={() => void addService()}>Добавить</button>
|
||||
<button type="button" onClick={() => { setAddingService(false); setServiceError(''); }}>Отмена</button>
|
||||
</div>}
|
||||
</fieldset>
|
||||
|
||||
<section className="client-failover-timing" aria-label="Пороги переключения">
|
||||
<label><span>Проверять каждые, сек</span><input type="number" min="15" max="900" value={seconds(draft.intervalMs)} onChange={(event) => update({ intervalMs: milliseconds(event.target.value, draft.intervalMs) })} /></label>
|
||||
<label><span>Сбой должен длиться, сек</span><input type="number" min={seconds(draft.intervalMs * 2)} max="1800" value={seconds(draft.failureWindowMs)} onChange={(event) => update({ failureWindowMs: milliseconds(event.target.value, draft.failureWindowMs) })} /></label>
|
||||
<label><span>Восстановление, сек</span><input type="number" min="60" max="86400" value={seconds(draft.recoveryWindowMs)} onChange={(event) => update({ recoveryWindowMs: milliseconds(event.target.value, draft.recoveryWindowMs) })} /></label>
|
||||
<label><span>Не переключать во время работы</span><input type="checkbox" checked={draft.trafficGuard.enabled} onChange={(event) => update({ trafficGuard: { ...draft.trafficGuard, enabled: event.target.checked } })} /></label>
|
||||
<label><span>Тишина перед переключением, сек</span><input type="number" min="5" max="600" value={seconds(draft.trafficGuard.quietWindowMs)} onChange={(event) => update({ trafficGuard: { ...draft.trafficGuard, quietWindowMs: milliseconds(event.target.value, draft.trafficGuard.quietWindowMs) } })} /></label>
|
||||
<label><span>Активный трафик, КБ/с</span><input type="number" min="1" max="102400" value={Math.round(draft.trafficGuard.thresholdBytesPerSecond / 1024)} onChange={(event) => update({ trafficGuard: { ...draft.trafficGuard, thresholdBytesPerSecond: Number(event.target.value) * 1024 } })} /></label>
|
||||
</section>
|
||||
|
||||
<details className="client-failover-advanced"><summary>Защита от повторных сбоев</summary>
|
||||
<label><span>Минимум на резерве, сек</span><input type="number" min="60" max="86400" value={seconds(draft.minimumReserveMs)} onChange={(event) => update({ minimumReserveMs: milliseconds(event.target.value, draft.minimumReserveMs) })} /></label>
|
||||
<label><span>Падений до карантина</span><input type="number" min="2" max="10" value={draft.flapProtection.count} onChange={(event) => update({ flapProtection: { ...draft.flapProtection, count: Number(event.target.value) } })} /></label>
|
||||
<label><span>Окно повторных сбоев, ч</span><input type="number" min="1" max="72" value={Math.round(draft.flapProtection.windowMs / 3_600_000)} onChange={(event) => update({ flapProtection: { ...draft.flapProtection, windowMs: Number(event.target.value) * 3_600_000 } })} /></label>
|
||||
<label><span>Карантин основного, мин</span><input type="number" min="10" max="10080" value={Math.round(draft.flapProtection.quarantineMs / 60_000)} onChange={(event) => update({ flapProtection: { ...draft.flapProtection, quarantineMs: Number(event.target.value) * 60_000 } })} /></label>
|
||||
</details>
|
||||
|
||||
<div className="client-failover-runtime">
|
||||
<strong role="status" aria-live="polite">{blocked ? 'Harbor выполняет действие…' : snapshot.activation === 'pending' ? 'Включится при следующем запуске VPN' : reasonLabel(snapshot.reason)}</strong>
|
||||
<span>Новые соединения: {role ? targetLabel(profiles, snapshot[role].target) : 'текущий канал вне настроенной пары'}</span>
|
||||
{snapshot.nextDecisionAt && <span>Следующее решение не раньше чем через {seconds(Math.max(0, Date.parse(snapshot.nextDecisionAt) - Date.now()))} с</span>}
|
||||
{snapshot.currentRole === 'other' && <span>Чтобы запустить автоматику, выключите VPN и включите снова на основном канале.</span>}
|
||||
{activity && <span>{activity.state === 'active' ? `Активный трафик · ${Math.round(activity.totalBytesPerSecond / 1024)} КБ/с · соединений: ${activity.transmittingConnections}` : activity.state === 'quiet' ? `Проверяем тишину · ${Math.min(seconds(quietElapsed), seconds(draft.trafficGuard.quietWindowMs))} из ${seconds(draft.trafficGuard.quietWindowMs)} с` : 'Не удалось определить активность · автоматическое переключение остановлено'}</span>}
|
||||
{activity?.blockers.slice(0, 2).map((blocker) => <small key={`${blocker.device}:${blocker.service}`}>{blocker.device} · {blocker.service} · {Math.round((blocker.uploadBytesPerSecond + blocker.downloadBytesPerSecond) / 1024)} КБ/с</small>)}
|
||||
{activity && activity.blockers.length > 2 && <small>Ещё {activity.blockers.length - 2}</small>}
|
||||
</div>
|
||||
|
||||
{!draftValid && <p className="client-failover-validation">Выберите два разных сервера. Они могут быть из одной подписки.</p>}
|
||||
{confirmDiscard && <div className="client-failover-discard" role="alert">
|
||||
<span>Отменить несохранённые изменения?</span>
|
||||
<button type="button" onClick={() => { setDraft(snapshot.policy); setDirty(false); setConfirmDiscard(false); feature.beforeCloseRef.current = () => true; feature.close(); }}>Отменить изменения</button>
|
||||
<button type="button" onClick={() => setConfirmDiscard(false)}>Продолжить настройку</button>
|
||||
</div>}
|
||||
|
||||
<div className="client-failover-actions">
|
||||
<button type="button" disabled={blocked || !snapshot.enabled || snapshot.activation !== 'active'} onClick={() => void onCheck()}>Проверить оба канала</button>
|
||||
<button type="submit" disabled={blocked || !dirty || !draftValid}>Сохранить</button>
|
||||
{snapshot.enabled && <button type="button" disabled={blocked} onClick={() => void onPause(!snapshot.paused)}>{snapshot.paused ? 'Возобновить' : 'Пауза'}</button>}
|
||||
{switchRole && snapshot.enabled && snapshot.activation === 'active' && <button type="button" disabled={blocked} aria-label={`Переключить новые соединения на ${switchRole === 'reserve' ? 'резервный' : 'основной'} сейчас`} onClick={() => void onSwitch(switchRole)}>Переключить новые сейчас</button>}
|
||||
{(snapshot.currentRole === 'primary' || snapshot.currentRole === 'reserve') && <small>Текущие соединения Harbor не закроет</small>}
|
||||
</div>
|
||||
</form>
|
||||
</Drawer>;
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
export { FailoverPanel, FailoverToggle, useFailoverFeature } from './FailoverFeature.js';
|
||||
@@ -4,6 +4,10 @@ export type SyncErrorKind = 'incompatible-api' | 'control-unreachable' | 'fatal'
|
||||
|
||||
export interface HarborReducerState {
|
||||
snapshot: HarborClientState | null;
|
||||
failoverTransport: {
|
||||
activeEpoch: string;
|
||||
retiredEpochs: string[];
|
||||
};
|
||||
transport: {
|
||||
bootStatus: 'loading' | 'ready' | SyncErrorKind;
|
||||
lastSuccessfulSyncAt: string | null;
|
||||
@@ -20,6 +24,7 @@ export type HarborAction =
|
||||
|
||||
export const initialHarborState: HarborReducerState = {
|
||||
snapshot: null,
|
||||
failoverTransport: { activeEpoch: '', retiredEpochs: [] },
|
||||
transport: {
|
||||
bootStatus: 'loading',
|
||||
lastSuccessfulSyncAt: null,
|
||||
@@ -31,6 +36,14 @@ export const initialHarborState: HarborReducerState = {
|
||||
|
||||
export const STALE_FAILURE_THRESHOLD = 3;
|
||||
|
||||
function failoverEpoch(snapshot: HarborClientState | null) {
|
||||
return snapshot?.failover?.observationEpoch || '';
|
||||
}
|
||||
|
||||
function failoverSequence(snapshot: HarborClientState | null) {
|
||||
return snapshot?.failover?.observationSequence || 0;
|
||||
}
|
||||
|
||||
export function classifySyncError(error: unknown): SyncErrorKind {
|
||||
const candidate = error && typeof error === 'object' ? error as Record<string, unknown> : {};
|
||||
const status = Number(candidate.status) || 0;
|
||||
@@ -71,9 +84,29 @@ export function harborReducer(current: HarborReducerState, action: HarborAction)
|
||||
|
||||
const snapshot = action.snapshot;
|
||||
const newer = !current.snapshot || snapshot.revision > current.snapshot.revision;
|
||||
const equal = Boolean(current.snapshot) && snapshot.revision === current.snapshot?.revision;
|
||||
const incomingEpoch = failoverEpoch(snapshot);
|
||||
const activeEpoch = current.failoverTransport.activeEpoch || failoverEpoch(current.snapshot);
|
||||
const unseenEpoch = Boolean(incomingEpoch)
|
||||
&& incomingEpoch !== activeEpoch
|
||||
&& !current.failoverTransport.retiredEpochs.includes(incomingEpoch);
|
||||
const newerObservation = equal && Boolean(snapshot.failover) && (
|
||||
unseenEpoch
|
||||
|| (incomingEpoch === activeEpoch && failoverSequence(snapshot) > failoverSequence(current.snapshot))
|
||||
);
|
||||
const nextSnapshot = newer
|
||||
? snapshot
|
||||
: newerObservation && current.snapshot
|
||||
? { ...current.snapshot, failover: snapshot.failover }
|
||||
: current.snapshot;
|
||||
const nextActiveEpoch = newer || unseenEpoch ? incomingEpoch : activeEpoch;
|
||||
const retiredEpochs = unseenEpoch && activeEpoch
|
||||
? [...current.failoverTransport.retiredEpochs, activeEpoch]
|
||||
: current.failoverTransport.retiredEpochs;
|
||||
|
||||
return {
|
||||
snapshot: newer ? snapshot : current.snapshot,
|
||||
snapshot: nextSnapshot,
|
||||
failoverTransport: { activeEpoch: nextActiveEpoch, retiredEpochs },
|
||||
transport: {
|
||||
bootStatus: 'ready',
|
||||
lastSuccessfulSyncAt: action.receivedAt,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
export type OperationKey = 'connection' | 'serverApply' | 'profileAdd' | 'profileRename'
|
||||
| 'profileSelect' | 'profileActivate' | 'profileRefresh' | 'profileDelete'
|
||||
| 'gatewayAuto' | 'routeRules' | 'diagnosticsSettings';
|
||||
| 'gatewayAuto' | 'routeRules' | 'diagnosticsSettings' | 'failover';
|
||||
|
||||
export interface OperationState {
|
||||
status: 'running';
|
||||
@@ -22,6 +22,7 @@ const OPERATION_KEYS: readonly OperationKey[] = [
|
||||
'gatewayAuto',
|
||||
'routeRules',
|
||||
'diagnosticsSettings',
|
||||
'failover',
|
||||
];
|
||||
|
||||
// The backend has one canonical revision and one mutation queue, so the UI mirrors that lock.
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
.client-journal-header { display: grid; gap: 7px; margin: 0 8px 30px; }
|
||||
.client-journal-header > span, .client-journal-groups h3 { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); }
|
||||
.client-journal-header > div { display: flex; align-items: center; gap: 12px; }
|
||||
.client-journal-header h2 { margin: 0; font: var(--type-drawer-title); letter-spacing: var(--type-drawer-title-tracking); text-transform: var(--type-drawer-title-transform); }
|
||||
.client-journal-header button { width: 34px; height: 34px; display: grid; place-items: center; border: 0; background: transparent; color: var(--client-muted); cursor: pointer; }
|
||||
.client-journal-header svg { width: 17px; fill: none; stroke: currentColor; stroke-width: 1.7; }
|
||||
.client-journal-header button:hover, .client-journal-header button:focus-visible { color: var(--client-accent); outline: 0; filter: drop-shadow(0 0 7px var(--client-accent)); }
|
||||
.client-journal-header button:disabled svg { animation: client-spin 900ms linear infinite; }
|
||||
.client-journal-groups { display: grid; gap: 26px; margin: 0 8px; }
|
||||
.client-journal-groups section { display: grid; gap: 7px; }
|
||||
.client-journal-groups h3 { margin: 0; }
|
||||
.client-journal-groups ol { display: grid; margin: 0; padding: 0; }
|
||||
.client-journal-groups li { min-height: 58px; display: grid; grid-template-columns: 52px minmax(0, 1fr); gap: 14px; align-items: start; padding: 10px 0; border-top: 1px solid color-mix(in oklch, var(--client-border) 48%, transparent); }
|
||||
.client-journal-time, .client-journal-groups li > div:last-child { min-width: 0; display: grid; gap: 3px; }
|
||||
.client-journal-time time { color: var(--client-text); font: var(--type-data); font-variant-numeric: var(--numeric-tabular); letter-spacing: var(--type-data-tracking); text-transform: var(--type-data-transform); }
|
||||
.client-journal-time span, .client-journal-groups li span { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); overflow-wrap: anywhere; }
|
||||
.client-journal-time span { text-transform: var(--type-label-transform); }
|
||||
.client-journal-groups li strong { font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); }
|
||||
.client-journal-groups li em { width: fit-content; color: oklch(0.68 0.15 28); font: var(--type-label); font-style: normal; letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); }
|
||||
.client-journal-error, .client-journal-empty { min-height: 80px; margin: 0 8px; color: var(--client-muted); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); }
|
||||
.client-journal-error button, .client-journal-footer button { border: 0; background: transparent; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; }
|
||||
.client-journal-skeleton { display: grid; gap: 14px; margin: 0 8px; }
|
||||
.client-journal-skeleton span { height: 48px; background: color-mix(in oklch, var(--client-border) 24%, transparent); opacity: .55; }
|
||||
.client-journal-footer { min-height: 72px; display: grid; place-items: center; margin: 12px 8px 0; color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); text-align: center; }
|
||||
.client-journal-footer button:focus-visible, .client-journal-error button:focus-visible { outline: 2px solid var(--client-accent); outline-offset: 2px; }
|
||||
@media (max-width: 560px) { .client-journal-groups li { grid-template-columns: 1fr; gap: 5px; } .client-journal-time { display: flex; gap: 8px; } }
|
||||
@media (prefers-reduced-motion: reduce) { .client-journal-header button:disabled svg { animation: none; } }
|
||||
@@ -0,0 +1,45 @@
|
||||
.client-failover-header { display: grid; gap: 8px; margin: 0 8px 30px; }
|
||||
.client-failover-header > span, .client-failover-channel-title > span, .client-failover-services legend { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); }
|
||||
.client-failover-header h2 { margin: 0; font: var(--type-drawer-title); letter-spacing: var(--type-drawer-title-tracking); text-transform: var(--type-drawer-title-transform); }
|
||||
.client-failover-header p { margin: 0; color: var(--client-muted); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); }
|
||||
.client-failover-form { display: grid; gap: 24px; margin: 0 8px; }
|
||||
.client-failover-master { min-height: 54px; display: flex; align-items: center; justify-content: space-between; gap: 20px; border-bottom: 1px solid color-mix(in oklch, var(--client-border) 48%, transparent); }
|
||||
.client-failover-master > span { display: grid; gap: 3px; }
|
||||
.client-failover-master strong { font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); }
|
||||
.client-failover-master small { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); }
|
||||
.client-failover-master input { width: 42px; height: 22px; }
|
||||
.client-failover-channel { display: grid; gap: 10px; }
|
||||
.client-failover-channel-title { min-height: 28px; display: flex; align-items: center; justify-content: space-between; gap: 12px; }
|
||||
.client-failover-channel-title strong { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); transition: color 600ms ease, filter 600ms ease; }
|
||||
.client-failover-channel-title strong.is-healthy { color: var(--client-accent); }
|
||||
.client-failover-channel-title strong.is-unhealthy { color: oklch(0.68 0.15 28); }
|
||||
.client-failover-channel label, .client-failover-timing label, .client-failover-advanced label { display: grid; grid-template-columns: minmax(0, 1fr) minmax(150px, 48%); align-items: center; gap: 16px; min-height: 38px; color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); }
|
||||
.client-failover-channel select, .client-failover-timing input, .client-failover-advanced input { min-width: 0; min-height: 34px; border: 0; border-bottom: 1px solid var(--client-border); background: transparent; color: var(--client-text); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); }
|
||||
.client-failover-services { display: grid; gap: 9px; margin: 0; padding: 0; border: 0; }
|
||||
.client-failover-services legend { margin-bottom: 10px; }
|
||||
.client-failover-service { display: grid; grid-template-columns: minmax(0, 1fr) auto; align-items: center; gap: 12px; min-height: 34px; }
|
||||
.client-failover-service > label { display: flex; align-items: center; gap: 10px; font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); }
|
||||
.client-failover-service > .client-failover-service-timeout { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); }
|
||||
.client-failover-service-timeout input { width: 52px; min-height: 30px; border: 0; border-bottom: 1px solid var(--client-border); background: transparent; color: var(--client-text); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); }
|
||||
.client-failover-add-service { justify-self: start; padding: 7px 0; border: 0; background: transparent; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; }
|
||||
.client-failover-add-service:disabled { color: var(--client-muted); cursor: default; opacity: .45; }
|
||||
.client-failover-service-editor { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1.6fr); gap: 8px 12px; padding: 8px 0; }
|
||||
.client-failover-service-editor input { min-width: 0; min-height: 34px; border: 0; border-bottom: 1px solid var(--client-border); background: transparent; color: var(--client-text); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); }
|
||||
.client-failover-service-editor span { grid-column: 1 / -1; color: oklch(0.68 0.15 28); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); }
|
||||
.client-failover-service-editor button { justify-self: start; padding: 4px 0; border: 0; background: transparent; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; }
|
||||
.client-failover-timing { display: grid; gap: 5px; }
|
||||
.client-failover-advanced summary { min-height: 38px; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; }
|
||||
.client-failover-runtime { min-height: 132px; display: grid; align-content: center; gap: 5px; padding: 14px 0; border-top: 1px solid color-mix(in oklch, var(--client-border) 48%, transparent); border-bottom: 1px solid color-mix(in oklch, var(--client-border) 48%, transparent); font-variant-numeric: var(--numeric-tabular); }
|
||||
.client-failover-runtime strong { font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); }
|
||||
.client-failover-runtime span, .client-failover-runtime small { color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); }
|
||||
.client-failover-validation { margin: -12px 0 0; color: oklch(0.68 0.15 28); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); }
|
||||
.client-failover-discard { min-height: 74px; display: flex; flex-wrap: wrap; align-items: center; gap: 8px 16px; padding: 10px 0; border-top: 1px solid var(--client-border); border-bottom: 1px solid var(--client-border); }
|
||||
.client-failover-discard span { flex-basis: 100%; color: var(--client-text); font: var(--type-body); letter-spacing: var(--type-body-tracking); text-transform: var(--type-body-transform); }
|
||||
.client-failover-discard button { padding: 4px 0; border: 0; background: transparent; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; }
|
||||
.client-failover-actions { min-height: 42px; display: flex; flex-wrap: wrap; gap: 18px; }
|
||||
.client-failover-actions button { padding: 7px 0; border: 0; background: transparent; color: var(--client-accent); font: var(--type-control); letter-spacing: var(--type-control-tracking); text-transform: var(--type-control-transform); cursor: pointer; }
|
||||
.client-failover-actions small { flex-basis: 100%; margin-top: -14px; color: var(--client-muted); font: var(--type-label); letter-spacing: var(--type-label-tracking); text-transform: var(--type-label-transform); }
|
||||
.client-failover-actions button:disabled { color: var(--client-muted); cursor: default; opacity: .45; }
|
||||
.client-failover-actions button:focus-visible, .client-failover-form input:focus-visible, .client-failover-form select:focus-visible, .client-failover-advanced summary:focus-visible, .client-failover-add-service:focus-visible, .client-failover-service-editor button:focus-visible, .client-failover-discard button:focus-visible { outline: 2px solid var(--client-accent); outline-offset: 2px; }
|
||||
@media (max-width: 560px) { .client-failover-channel label, .client-failover-timing label, .client-failover-advanced label { grid-template-columns: 1fr; gap: 4px; } .client-failover-service, .client-failover-service-editor { grid-template-columns: 1fr; gap: 2px; } }
|
||||
@media (prefers-reduced-motion: reduce) { .client-failover-channel-title strong { transition: none; } }
|
||||
@@ -8,5 +8,7 @@
|
||||
@import './features/servers.css';
|
||||
@import './primitives.css';
|
||||
@import './features/diagnostics.css';
|
||||
@import './features/failover.css';
|
||||
@import './features/activity-journal.css';
|
||||
@import './layout.css';
|
||||
@import './themes.css';
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
|
||||
import { createActivityJournalRoute } from '../../dist/server/http/routes/activityJournalRoute.js';
|
||||
|
||||
test('journal route is read-only and forwards bounded cursor pagination', async () => {
|
||||
const calls = [];
|
||||
const route = createActivityJournalRoute({
|
||||
journal: { page: (...args) => {
|
||||
calls.push(args);
|
||||
return { events: [], nextCursor: null, retentionDays: 30, generatedAt: 'now', storage: { status: 'ready', errorCode: null } };
|
||||
} },
|
||||
});
|
||||
let status;
|
||||
let payload;
|
||||
const response = {
|
||||
writeHead: (value) => { status = value; },
|
||||
end: (value) => { payload = JSON.parse(value); },
|
||||
};
|
||||
assert.equal(await route.handle({ method: 'GET', url: '/api/activity-journal?limit=25&cursor=evt' }, response), true);
|
||||
assert.equal(status, 200);
|
||||
assert.equal(payload.retentionDays, 30);
|
||||
assert.deepEqual(calls, [[25, 'evt']]);
|
||||
assert.equal(await route.handle({ method: 'POST', url: '/api/activity-journal' }, response), false);
|
||||
});
|
||||
@@ -0,0 +1,145 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import test from 'node:test';
|
||||
|
||||
import { createActivityJournalService } from '../../dist/server/services/activityJournalService.js';
|
||||
import { assertActivityJournalPage } from '../../dist/shared/activityJournal.js';
|
||||
|
||||
function fixture(t) {
|
||||
const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'harbor-journal-'));
|
||||
t.after(() => fs.rmSync(directory, { recursive: true, force: true }));
|
||||
return path.join(directory, 'activity-journal.json');
|
||||
}
|
||||
|
||||
const event = (dedupeKey, profileLabel = 'Home') => ({
|
||||
type: 'subscription.refreshed',
|
||||
severity: 'info',
|
||||
source: 'subscription',
|
||||
dedupeKey: `subscription.refreshed:${dedupeKey}`,
|
||||
data: { profileId: 'profile-1', profileLabel, host: 'provider.example', serverCount: 12, added: 2, removed: 1 },
|
||||
});
|
||||
|
||||
test('journal appends typed events, deduplicates and keeps stable newest-first cursors', (t) => {
|
||||
let clock = new Date('2026-08-19T10:00:00.000Z');
|
||||
const filePath = fixture(t);
|
||||
const service = createActivityJournalService({ filePath, now: () => clock });
|
||||
service.append(event('refresh:1', 'One'));
|
||||
clock = new Date('2026-08-19T10:01:00.000Z');
|
||||
service.append(event('refresh:2', 'Two'));
|
||||
service.append(event('refresh:2', 'Duplicate'));
|
||||
const first = service.page(1);
|
||||
assert.equal(first.events[0].data.profileLabel, 'Two');
|
||||
assert.ok(first.nextCursor);
|
||||
|
||||
clock = new Date('2026-08-19T10:02:00.000Z');
|
||||
service.append(event('refresh:3', 'Three'));
|
||||
const older = service.page(10, first.nextCursor);
|
||||
assert.deepEqual(older.events.map(({ data }) => data.profileLabel), ['One']);
|
||||
assert.equal(first.events[0].dedupeKey, null);
|
||||
const inode = fs.statSync(filePath).ino;
|
||||
assert.equal(service.page(10).events.length, 3);
|
||||
assert.equal(fs.statSync(filePath).ino, inode);
|
||||
assert.deepEqual(service.page(10, 'expired-cursor').events, []);
|
||||
});
|
||||
|
||||
test('journal prunes events older than 30 days and rejects unsafe payloads', (t) => {
|
||||
let clock = new Date('2026-07-01T00:00:00.000Z');
|
||||
const filePath = fixture(t);
|
||||
const service = createActivityJournalService({ filePath, now: () => clock });
|
||||
service.append(event('old'));
|
||||
clock = new Date('2026-08-19T00:00:00.000Z');
|
||||
assert.deepEqual(service.page().events, []);
|
||||
assert.deepEqual(JSON.parse(fs.readFileSync(filePath, 'utf8')).events, []);
|
||||
service.append(event('new'));
|
||||
assert.throws(() => service.append({ ...event('unsafe'), data: { rawUrl: 'https://secret' } }), /Unsafe/);
|
||||
service.append({ ...event('ip'), data: { ...event('ip').data, host: '192.168.1.1' } });
|
||||
service.append(event('credential-label', 'https://user:pass@example.test/private?token=secret'));
|
||||
service.append(event('path-label', '192.168.1.7/private'));
|
||||
service.append(event('ipv6-label', '2001:db8::1'));
|
||||
const page = service.page();
|
||||
const labels = page.events.slice(0, 3).map(({ data }) => data.profileLabel);
|
||||
assert.deepEqual(labels, ['Подписка', 'Подписка', 'Подписка']);
|
||||
assert.equal(page.events.find(({ data }) => data.host === 'Провайдер')?.data.host, 'Провайдер');
|
||||
for (const dedupeKey of [
|
||||
'subscription.refreshed:192.168.1.1',
|
||||
'subscription.refreshed:2001:db8::1',
|
||||
'subscription.refreshed:user:pass',
|
||||
]) service.append({ ...event('safe'), dedupeKey });
|
||||
service.append({ ...event('safe'), dedupeKey: 'subscription.refreshed:user:pass' });
|
||||
const persisted = fs.readFileSync(filePath, 'utf8');
|
||||
assert.doesNotMatch(persisted, /user:pass|192\.168\.1\.1|192\.168\.1\.7|2001:db8|token=secret/);
|
||||
assert.match(persisted, /subscription\.refreshed:sha256:[a-f0-9]{64}/);
|
||||
assert.throws(() => service.append({ ...event('safe'), dedupeKey: 'https://user:pass@example.test/private?token=x' }), /dedupe/i);
|
||||
assert.throws(() => service.append({ ...event('safe'), dedupeKey: 'subscription.refreshed:private/path' }), /dedupe/i);
|
||||
});
|
||||
|
||||
test('journal persists the 10,000 event cap when opening an oversized store', (t) => {
|
||||
const filePath = fixture(t);
|
||||
const occurredAt = '2026-08-19T00:00:00.000Z';
|
||||
const events = Array.from({ length: 10_001 }, (_, index) => ({
|
||||
id: `00000000-0000-4000-8000-${String(index).padStart(12, '0')}`,
|
||||
occurredAt,
|
||||
...event(`event:${index}`),
|
||||
}));
|
||||
fs.writeFileSync(filePath, JSON.stringify({ schemaVersion: 1, events }));
|
||||
const service = createActivityJournalService({
|
||||
filePath,
|
||||
now: () => new Date('2026-08-19T01:00:00.000Z'),
|
||||
});
|
||||
assert.equal(JSON.parse(fs.readFileSync(filePath, 'utf8')).events.length, 10_000);
|
||||
assert.equal(service.page(1).events.length, 1);
|
||||
});
|
||||
|
||||
test('corrupt journal is isolated and recovery becomes a safe event', (t) => {
|
||||
const filePath = fixture(t);
|
||||
fs.writeFileSync(filePath, '{broken');
|
||||
const service = createActivityJournalService({
|
||||
filePath,
|
||||
now: () => new Date('2026-08-19T12:00:00.000Z'),
|
||||
});
|
||||
const page = service.page();
|
||||
assert.equal(page.storage.status, 'ready');
|
||||
assert.equal(page.events[0].type, 'journal.recovered');
|
||||
assert.ok(fs.readdirSync(path.dirname(filePath)).some((name) => name.includes('.corrupt-')));
|
||||
});
|
||||
|
||||
test('journal exposes a latched write failure until a later append succeeds', (t) => {
|
||||
const filePath = fixture(t);
|
||||
const directory = path.dirname(filePath);
|
||||
const service = createActivityJournalService({ filePath });
|
||||
service.append(event('before-error'));
|
||||
fs.chmodSync(directory, 0o555);
|
||||
try {
|
||||
assert.throws(() => service.append(event('lost')));
|
||||
const failed = service.page();
|
||||
assert.equal(failed.storage.status, 'error');
|
||||
assert.equal(failed.storage.errorCode, 'JOURNAL_UNAVAILABLE');
|
||||
assert.equal(failed.events.length, 1);
|
||||
assert.equal(failed.events[0].dedupeKey, null);
|
||||
} finally {
|
||||
fs.chmodSync(directory, 0o755);
|
||||
}
|
||||
service.append(event('recovered'));
|
||||
assert.equal(service.page().storage.status, 'ready');
|
||||
});
|
||||
|
||||
test('journal page parser rejects malformed wire data and strips unknown event fields', (t) => {
|
||||
const service = createActivityJournalService({ filePath: fixture(t) });
|
||||
service.append(event('wire'));
|
||||
const page = service.page();
|
||||
const parsed = assertActivityJournalPage({
|
||||
...page,
|
||||
events: page.events.map((item) => ({ ...item, ignored: 'value' })),
|
||||
});
|
||||
assert.equal(Object.hasOwn(parsed.events[0], 'ignored'), false);
|
||||
assert.throws(() => assertActivityJournalPage({ ...page, retentionDays: 31 }), TypeError);
|
||||
assert.throws(() => assertActivityJournalPage({ ...page, events: [{ broken: true }] }), TypeError);
|
||||
const future = assertActivityJournalPage({
|
||||
...page,
|
||||
events: [{ ...page.events[0], type: 'future.safe_event', data: { raw: 'not exposed' } }],
|
||||
});
|
||||
assert.equal(future.events[0].type, 'unknown');
|
||||
assert.deepEqual(future.events[0].data, {});
|
||||
});
|
||||
@@ -52,6 +52,7 @@ function createHarness(overrides = {}) {
|
||||
let stateUpdates = 0;
|
||||
let startFailure = overrides.failStart || null;
|
||||
const events = [];
|
||||
const journalEvents = [];
|
||||
|
||||
const serialize = (operation) => {
|
||||
const run = async () => {
|
||||
@@ -119,13 +120,16 @@ function createHarness(overrides = {}) {
|
||||
restartCommand: () => captureRuntimeCommand(restart, { preMutationErrorCodes: ['CONFIG_INVALID'] }),
|
||||
},
|
||||
route: { isGatewayDirect: () => overrides.gatewayDirect === true },
|
||||
failover: overrides.failover,
|
||||
serialize,
|
||||
now: () => new Date('2026-08-08T12:00:00.000Z'),
|
||||
onEvent: (event) => journalEvents.push(event),
|
||||
});
|
||||
|
||||
return {
|
||||
service,
|
||||
events,
|
||||
journalEvents,
|
||||
serialize,
|
||||
snapshot: () => structuredClone({ state, config, running, peak }),
|
||||
};
|
||||
@@ -167,6 +171,8 @@ test('failed candidate config or runtime restores the old profile, config and ru
|
||||
const before = harness.snapshot();
|
||||
await assert.rejects(harness.service.apply('work', 'b'));
|
||||
assert.deepEqual(domain(harness.snapshot()), domain(before));
|
||||
assert.equal(harness.journalEvents.at(-1).type, 'connection.failed');
|
||||
assert.match(harness.journalEvents.at(-1).data.errorCode, /^[A-Z0-9_]+$/);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -256,6 +262,193 @@ test('running restart preserves a pending desired profile while restoring the ap
|
||||
assert.equal(harness.snapshot().state.appliedProfileId, 'personal');
|
||||
});
|
||||
|
||||
test('running dual restart rebuilds only the applied failover pair', async () => {
|
||||
const state = initialState();
|
||||
state.failoverPolicy = { enabled: true };
|
||||
state.appliedFailoverPolicy = {
|
||||
primary: { profileId: 'personal', serverId: 'a' },
|
||||
reserve: { profileId: 'work', serverId: 'b' },
|
||||
};
|
||||
const sources = [];
|
||||
const prepared = [];
|
||||
const harness = createHarness({
|
||||
state,
|
||||
failover: {
|
||||
build: (_state, source) => {
|
||||
sources.push(source);
|
||||
return {
|
||||
config: { source },
|
||||
applied: state.appliedFailoverPolicy,
|
||||
primaryProfile: state.profiles[0],
|
||||
primaryServer: serverA,
|
||||
};
|
||||
},
|
||||
prepareActivation: async (role) => { prepared.push(role); harness.events.push(`selector.${role}`); },
|
||||
restoreAppliedActivation: async () => {},
|
||||
reconcile: async () => {},
|
||||
},
|
||||
});
|
||||
await harness.service.restart();
|
||||
assert.deepEqual(sources, ['applied']);
|
||||
assert.deepEqual(prepared, ['primary']);
|
||||
assert.deepEqual(harness.events.slice(0, 4), ['config.write', 'runtime.restart', 'selector.primary', 'state.update']);
|
||||
assert.match(harness.snapshot().config, /"source":"applied"/);
|
||||
});
|
||||
|
||||
test('running single-channel keeps server edits pending after failover is enabled', async () => {
|
||||
const state = initialState();
|
||||
state.failoverPolicy = { enabled: true };
|
||||
state.appliedFailoverPolicy = null;
|
||||
const harness = createHarness({ state });
|
||||
await harness.service.apply('work', 'b');
|
||||
assert.equal(harness.snapshot().state.desiredProfileId, 'work');
|
||||
assert.equal(harness.snapshot().state.appliedProfileId, 'personal');
|
||||
assert.equal(harness.events.includes('config.write'), false);
|
||||
assert.equal(harness.events.includes('runtime.start'), false);
|
||||
});
|
||||
|
||||
test('running dual restart preserves reserve and commits only after selector read-back', async () => {
|
||||
const state = initialState();
|
||||
state.failoverPolicy = { enabled: true };
|
||||
state.appliedFailoverPolicy = {
|
||||
primary: { profileId: 'personal', serverId: 'a' },
|
||||
reserve: { profileId: 'work', serverId: 'b' },
|
||||
};
|
||||
state.appliedProfileId = 'work';
|
||||
state.appliedServerId = 'b';
|
||||
state.appliedServerSnapshot = serverB;
|
||||
const harness = createHarness({
|
||||
state,
|
||||
failover: {
|
||||
build: () => ({
|
||||
config: { dual: true },
|
||||
applied: state.appliedFailoverPolicy,
|
||||
primaryProfile: state.profiles[0],
|
||||
primaryServer: serverA,
|
||||
}),
|
||||
prepareActivation: async (role) => { harness.events.push(`selector.${role}`); },
|
||||
restoreAppliedActivation: async () => { harness.events.push('selector.primary'); },
|
||||
reconcile: async () => {},
|
||||
},
|
||||
});
|
||||
await harness.service.restart();
|
||||
assert.equal(harness.snapshot().state.appliedProfileId, 'work');
|
||||
assert.equal(harness.snapshot().state.appliedServerId, 'b');
|
||||
assert.deepEqual(harness.events.slice(0, 4), ['config.write', 'runtime.restart', 'selector.reserve', 'state.update']);
|
||||
});
|
||||
|
||||
test('selector activation failure rolls runtime and config back before applied truth changes', async () => {
|
||||
const state = initialState();
|
||||
state.failoverPolicy = { enabled: true };
|
||||
state.appliedFailoverPolicy = {
|
||||
primary: { profileId: 'personal', serverId: 'a' },
|
||||
reserve: { profileId: 'work', serverId: 'b' },
|
||||
};
|
||||
const harness = createHarness({
|
||||
state,
|
||||
failover: {
|
||||
build: () => ({
|
||||
config: { dual: true },
|
||||
applied: state.appliedFailoverPolicy,
|
||||
primaryProfile: state.profiles[0],
|
||||
primaryServer: serverA,
|
||||
}),
|
||||
prepareActivation: async (role) => {
|
||||
harness.events.push(`selector.${role}`);
|
||||
if (harness.events.filter((event) => event.startsWith('selector.')).length === 1) {
|
||||
throw new Error('selector failed');
|
||||
}
|
||||
},
|
||||
restoreAppliedActivation: async () => { harness.events.push('selector.primary'); },
|
||||
reconcile: async () => {},
|
||||
},
|
||||
});
|
||||
const before = harness.snapshot();
|
||||
await assert.rejects(harness.service.restart(), /selector failed/);
|
||||
assert.deepEqual(domain(harness.snapshot()), domain(before));
|
||||
assert.equal(harness.events.includes('state.update'), false);
|
||||
assert.deepEqual(harness.events.filter((event) => event.startsWith('selector.')), ['selector.primary', 'selector.primary']);
|
||||
});
|
||||
|
||||
test('state commit rollback restores the previously selected reserve role', async () => {
|
||||
const state = initialState();
|
||||
state.failoverPolicy = { enabled: true };
|
||||
state.appliedFailoverPolicy = {
|
||||
primary: { profileId: 'personal', serverId: 'a' },
|
||||
reserve: { profileId: 'work', serverId: 'b' },
|
||||
};
|
||||
state.appliedProfileId = 'work';
|
||||
state.appliedServerId = 'b';
|
||||
state.appliedServerSnapshot = serverB;
|
||||
const harness = createHarness({
|
||||
state,
|
||||
failStateAt: 1,
|
||||
failover: {
|
||||
build: () => ({
|
||||
config: { dual: true },
|
||||
applied: state.appliedFailoverPolicy,
|
||||
primaryProfile: state.profiles[0],
|
||||
primaryServer: serverA,
|
||||
}),
|
||||
prepareActivation: async (role) => { harness.events.push(`selector.${role}`); },
|
||||
restoreAppliedActivation: async () => { harness.events.push('selector.reserve'); },
|
||||
reconcile: async () => {},
|
||||
},
|
||||
});
|
||||
const before = harness.snapshot();
|
||||
await assert.rejects(harness.service.restart(), /state failed/);
|
||||
assert.deepEqual(domain(harness.snapshot()), domain(before));
|
||||
assert.deepEqual(harness.events.filter((event) => event.startsWith('selector.')), ['selector.reserve', 'selector.reserve']);
|
||||
});
|
||||
|
||||
test('failed stop restores the selected reserve role before restoring state', async () => {
|
||||
const state = initialState();
|
||||
state.failoverPolicy = { enabled: true };
|
||||
state.appliedFailoverPolicy = {
|
||||
primary: { profileId: 'personal', serverId: 'a' },
|
||||
reserve: { profileId: 'work', serverId: 'b' },
|
||||
};
|
||||
state.appliedProfileId = 'work';
|
||||
state.appliedServerId = 'b';
|
||||
state.appliedServerSnapshot = serverB;
|
||||
const harness = createHarness({
|
||||
state,
|
||||
failStateAt: 1,
|
||||
failover: {
|
||||
restoreAppliedActivation: async () => { harness.events.push('selector.reserve'); },
|
||||
reconcile: async () => {},
|
||||
},
|
||||
});
|
||||
const before = harness.snapshot();
|
||||
await assert.rejects(harness.service.stop(), /state failed/);
|
||||
assert.deepEqual(domain(harness.snapshot()), domain(before));
|
||||
assert.deepEqual(harness.events.slice(0, 4), ['runtime.stop', 'state.update', 'runtime.start', 'selector.reserve']);
|
||||
});
|
||||
|
||||
test('disabled passive dual rollback restores the selected reserve role', async () => {
|
||||
const state = initialState();
|
||||
state.failoverPolicy = { enabled: false };
|
||||
state.appliedFailoverPolicy = {
|
||||
primary: { profileId: 'personal', serverId: 'a' },
|
||||
reserve: { profileId: 'work', serverId: 'b' },
|
||||
};
|
||||
state.appliedProfileId = 'work';
|
||||
state.appliedServerId = 'b';
|
||||
state.appliedServerSnapshot = serverB;
|
||||
const harness = createHarness({
|
||||
state,
|
||||
failStateAt: 1,
|
||||
failover: {
|
||||
restoreAppliedActivation: async () => { harness.events.push('selector.reserve'); },
|
||||
reconcile: async () => {},
|
||||
},
|
||||
});
|
||||
const before = harness.snapshot();
|
||||
await assert.rejects(harness.service.restart(), /state failed/);
|
||||
assert.deepEqual(domain(harness.snapshot()), domain(before));
|
||||
assert.equal(harness.events.includes('selector.reserve'), true);
|
||||
});
|
||||
|
||||
test('restart fails closed when runtime status is unknown', async () => {
|
||||
const state = initialState();
|
||||
state.desiredProfileId = 'work';
|
||||
|
||||
@@ -506,3 +506,18 @@ test('a targeted custom row validates and samples only that service', async () =
|
||||
assert.deepEqual(calls, ['https://second.example/', 'https://second.example/', 'https://second.example/']);
|
||||
assert.equal(result.direct.sites[0].latencyMs, 120);
|
||||
});
|
||||
|
||||
test('failover VPN checks apply their per-service timeout to curl', async () => {
|
||||
const calls = [];
|
||||
await createConnectivityDiagnosticsService({
|
||||
proxyPort: 18080,
|
||||
execute: async (args) => { calls.push(args); return response(); },
|
||||
}).runVpn({ target: 'site:youtube', timeoutMs: 9_000 });
|
||||
|
||||
assert.equal(calls.length, 3);
|
||||
for (const args of calls) {
|
||||
assert.equal(args[args.indexOf('--max-time') + 1], '9');
|
||||
assert.equal(args[args.indexOf('--connect-timeout') + 1], '3');
|
||||
assert.ok(args.includes('http://127.0.0.1:18080'));
|
||||
}
|
||||
});
|
||||
|
||||
@@ -32,6 +32,12 @@ test('control uses the dataplane socket protocol', async () => {
|
||||
[{ id: 'custom-test', url: 'https://example.com' }],
|
||||
'site:custom-test',
|
||||
);
|
||||
await client.checkConfig({ outbounds: [] });
|
||||
await client.runFailoverProbe('primary', [], 'site:youtube', 9_000);
|
||||
await client.readFailoverSelector();
|
||||
await client.selectFailoverRole('reserve');
|
||||
await client.setFailoverActivityEnabled(true);
|
||||
await client.readFailoverActivity(1024);
|
||||
assert.equal(client.running, true);
|
||||
await client.restart();
|
||||
assert.equal((await client.stop()).running, false);
|
||||
@@ -44,6 +50,12 @@ test('control uses the dataplane socket protocol', async () => {
|
||||
'GET /device-policy /run/dataplane.sock',
|
||||
'PUT /device-policy /run/dataplane.sock',
|
||||
'POST /diagnostics/connectivity /run/dataplane.sock',
|
||||
'POST /config/check /run/dataplane.sock',
|
||||
'POST /failover/probe /run/dataplane.sock',
|
||||
'GET /failover/selector /run/dataplane.sock',
|
||||
'PUT /failover/selector /run/dataplane.sock',
|
||||
'PUT /failover/activity /run/dataplane.sock',
|
||||
'POST /failover/activity/read /run/dataplane.sock',
|
||||
'POST /restart /run/dataplane.sock',
|
||||
'POST /stop /run/dataplane.sock',
|
||||
]);
|
||||
@@ -53,6 +65,10 @@ test('control uses the dataplane socket protocol', async () => {
|
||||
target: 'site:custom-test',
|
||||
});
|
||||
assert.equal(requests[7].timeoutMs, 25_000);
|
||||
assert.deepEqual(requests[8].body, { config: { outbounds: [] } });
|
||||
assert.deepEqual(requests[9].body, { role: 'primary', services: [], target: 'site:youtube', timeoutMs: 9_000 });
|
||||
assert.equal(requests[9].timeoutMs, 19_000);
|
||||
assert.deepEqual(requests[11].body, { role: 'reserve' });
|
||||
});
|
||||
|
||||
test('connectivity diagnostics expose a retryable domain error', async () => {
|
||||
|
||||
@@ -8,7 +8,7 @@ import { deviceId } from '../../dist/server/services/deviceInventoryService.js';
|
||||
|
||||
const mac = '00:11:22:33:44:55';
|
||||
const id = deviceId(mac);
|
||||
const device = { ip: '192.168.50.7', mac };
|
||||
const device = { ip: '192.168.50.7', mac, alias: 'MacBook' };
|
||||
const connection = (connectionId, type, host, upload, download, sourceIP = device.ip, chains = ['vpn-out']) => ({
|
||||
id: connectionId,
|
||||
metadata: { type, host, sourceIP },
|
||||
@@ -205,3 +205,36 @@ test('domain classification normalizes known services and rejects IP or malforme
|
||||
assert.equal(classifyDomain('192.0.2.1'), null);
|
||||
assert.equal(classifyDomain('broken_label.example'), null);
|
||||
});
|
||||
|
||||
test('failover activity is zero-work while disabled and uses the existing connection poll', async () => {
|
||||
let observedAt = new Date('2026-08-19T10:00:00.000Z');
|
||||
let response = { connections: [
|
||||
connection('work', 'tproxy/tproxy-in', 'r1.googlevideo.com', 0, 0, device.ip, ['channel-primary', 'channel-selector']),
|
||||
connection('probe', 'mixed/diagnostics-primary-in', 'youtube.com', 0, 10_000, device.ip, ['channel-primary']),
|
||||
] };
|
||||
const service = createDomainTrafficService({
|
||||
observe: async () => response,
|
||||
devices: () => [device],
|
||||
now: () => observedAt,
|
||||
});
|
||||
|
||||
await service.refresh();
|
||||
assert.equal(service.activitySnapshot(100), null);
|
||||
service.enableActivity();
|
||||
observedAt = new Date('2026-08-19T10:00:02.000Z');
|
||||
response.connections[0].download = 2_000;
|
||||
response.connections[1].download = 99_000;
|
||||
await service.refresh();
|
||||
const active = service.activitySnapshot(500);
|
||||
assert.equal(active.state, 'active');
|
||||
assert.equal(active.totalBytesPerSecond, 1_000);
|
||||
assert.equal(active.transmittingConnections, 1);
|
||||
assert.equal(active.blockers[0].device, 'MacBook');
|
||||
assert.equal(active.blockers[0].service, 'YouTube');
|
||||
|
||||
observedAt = new Date('2026-08-19T10:00:14.000Z');
|
||||
await service.refresh();
|
||||
assert.equal(service.activitySnapshot(500).state, 'quiet');
|
||||
service.disableActivity();
|
||||
assert.equal(service.activitySnapshot(500), null);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
|
||||
import { createFailoverRoute } from '../../dist/server/http/routes/failoverRoute.js';
|
||||
|
||||
function request(method, url = '/api/failover') {
|
||||
return { method, url };
|
||||
}
|
||||
|
||||
test('Gateway failover route serializes save, pause, checks and manual switch mutations', async () => {
|
||||
const calls = [];
|
||||
let body = { policy: { enabled: true }, expectedRevision: 7 };
|
||||
const route = createFailoverRoute({
|
||||
appMode: 'gateway',
|
||||
failover: {
|
||||
save: async (value) => calls.push(['save', value]),
|
||||
pause: async (value) => calls.push(['pause', value]),
|
||||
manualSwitch: async (value) => calls.push(['switch', value]),
|
||||
checkNow: async () => calls.push(['check']),
|
||||
},
|
||||
readBody: async () => body,
|
||||
withOperation: async (kind, operation, options) => {
|
||||
calls.push(['operation', kind, options]);
|
||||
return operation();
|
||||
},
|
||||
sendState: async () => calls.push(['state']),
|
||||
});
|
||||
|
||||
assert.equal(await route.handle(request('PUT'), {}), true);
|
||||
body = { paused: true, expectedRevision: 8 };
|
||||
assert.equal(await route.handle(request('POST', '/api/failover/pause'), {}), true);
|
||||
body = { role: 'reserve', expectedRevision: 9 };
|
||||
assert.equal(await route.handle(request('POST', '/api/failover/switch'), {}), true);
|
||||
body = { expectedRevision: 10 };
|
||||
assert.equal(await route.handle(request('POST', '/api/failover/check'), {}), true);
|
||||
assert.deepEqual(calls, [
|
||||
['operation', 'failover-save', { expectedRevision: 7 }], ['save', { enabled: true }], ['state'],
|
||||
['operation', 'failover-pause', { expectedRevision: 8 }], ['pause', true], ['state'],
|
||||
['operation', 'failover-switch', { expectedRevision: 9 }], ['switch', 'reserve'], ['state'],
|
||||
['check'], ['state'],
|
||||
]);
|
||||
});
|
||||
|
||||
test('failover mutations stay Gateway-only and reject unknown roles', async () => {
|
||||
const connectRoute = createFailoverRoute({
|
||||
appMode: 'client', failover: {}, readBody: async () => ({}), withOperation: async () => {}, sendState: async () => {},
|
||||
});
|
||||
await assert.rejects(connectRoute.handle(request('PUT'), {}), { code: 'ENDPOINT_NOT_FOUND' });
|
||||
|
||||
const gatewayRoute = createFailoverRoute({
|
||||
appMode: 'gateway', failover: {}, readBody: async () => ({ role: 'other' }), withOperation: async () => {}, sendState: async () => {},
|
||||
});
|
||||
await assert.rejects(gatewayRoute.handle(request('POST', '/api/failover/switch'), {}), { code: 'REQUEST_INVALID' });
|
||||
});
|
||||
@@ -0,0 +1,635 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import test from 'node:test';
|
||||
|
||||
import {
|
||||
DEFAULT_FAILOVER_POLICY,
|
||||
isFailoverConfigured,
|
||||
nextFailoverDecision,
|
||||
normalizeFailoverPolicy,
|
||||
} from '../../dist/shared/failover.js';
|
||||
import { createFailoverService } from '../../dist/server/features/failover/failoverService.js';
|
||||
|
||||
const enabled = normalizeFailoverPolicy({
|
||||
...DEFAULT_FAILOVER_POLICY,
|
||||
enabled: true,
|
||||
primary: { profileId: 'profile-1', serverId: 'server-1' },
|
||||
reserve: { profileId: 'profile-2', serverId: 'server-2' },
|
||||
intervalMs: 15_000,
|
||||
failureWindowMs: 30_000,
|
||||
recoveryWindowMs: 60_000,
|
||||
trafficGuard: { enabled: true, thresholdBytesPerSecond: 1024, quietWindowMs: 5_000 },
|
||||
minimumReserveMs: 60_000,
|
||||
});
|
||||
|
||||
test('failover policy defaults to strict disabled zero-work state', () => {
|
||||
const policy = normalizeFailoverPolicy(null);
|
||||
const decision = nextFailoverDecision({
|
||||
now: 0,
|
||||
policy,
|
||||
currentRole: 'primary',
|
||||
primaryHealth: 'unhealthy',
|
||||
reserveHealth: 'healthy',
|
||||
activity: 'quiet',
|
||||
});
|
||||
assert.equal(policy.enabled, false);
|
||||
assert.equal(decision.status, 'idle');
|
||||
assert.equal(decision.switchTo, null);
|
||||
});
|
||||
|
||||
test('failover save rejects policy values that would otherwise be silently clamped', async () => {
|
||||
const testHarness = harness(serviceState());
|
||||
await assert.rejects(testHarness.service.save({ ...enabled, intervalMs: -1 }), { code: 'REQUEST_INVALID' });
|
||||
});
|
||||
|
||||
test('strict failover validation accepts semantically identical reordered JSON keys', () => {
|
||||
const reordered = {
|
||||
flapProtection: enabled.flapProtection,
|
||||
minimumReserveMs: enabled.minimumReserveMs,
|
||||
trafficGuard: enabled.trafficGuard,
|
||||
recoveryWindowMs: enabled.recoveryWindowMs,
|
||||
failureWindowMs: enabled.failureWindowMs,
|
||||
intervalMs: enabled.intervalMs,
|
||||
checks: enabled.checks,
|
||||
reserve: enabled.reserve,
|
||||
primary: enabled.primary,
|
||||
paused: enabled.paused,
|
||||
enabled: enabled.enabled,
|
||||
version: enabled.version,
|
||||
};
|
||||
assert.deepEqual(normalizeFailoverPolicy(reordered, { strict: true }), enabled);
|
||||
});
|
||||
|
||||
test('enabled failover requires two distinct existing targets', async () => {
|
||||
const sameTarget = normalizeFailoverPolicy({ ...enabled, reserve: enabled.primary });
|
||||
assert.equal(isFailoverConfigured(sameTarget), false);
|
||||
await assert.rejects(harness(serviceState()).service.save(sameTarget), { code: 'REQUEST_INVALID' });
|
||||
|
||||
const state = serviceState({ ...enabled, paused: true });
|
||||
state.profiles[1].servers = [];
|
||||
const testHarness = harness(state);
|
||||
await assert.rejects(testHarness.service.pause(false), { code: 'REQUEST_INVALID' });
|
||||
assert.equal(testHarness.read().failoverPolicy.paused, true);
|
||||
});
|
||||
|
||||
test('local Gateway activity uses the existing domain traffic collector only while enabled', () => {
|
||||
const source = fs.readFileSync(new URL('../../src/server/index.ts', import.meta.url), 'utf8');
|
||||
assert.match(source, /const localFailoverTraffic = !remoteDataplane && settings\.appMode === 'gateway'[\s\S]*createDomainTrafficService/);
|
||||
assert.match(source, /setLocalFailoverActivityEnabled[\s\S]*enableActivity\(\)[\s\S]*setInterval\(refresh, 2_000\)/);
|
||||
assert.match(source, /if \(!enabled\)[\s\S]*clearInterval\(localFailoverTrafficTimer\)[\s\S]*disableActivity\(\)/);
|
||||
assert.doesNotMatch(source, /setFailoverActivityEnabled: async \(\) => \(\{\}\)/);
|
||||
});
|
||||
|
||||
test('failure window and traffic guard delay reserve switch until continuous quiet', () => {
|
||||
let decision = nextFailoverDecision({
|
||||
now: 0,
|
||||
policy: enabled,
|
||||
currentRole: 'primary',
|
||||
primaryHealth: 'unhealthy',
|
||||
reserveHealth: 'healthy',
|
||||
activity: 'active',
|
||||
});
|
||||
decision = nextFailoverDecision({
|
||||
now: 30_000,
|
||||
policy: enabled,
|
||||
currentRole: 'primary',
|
||||
primaryHealth: 'unhealthy',
|
||||
reserveHealth: 'healthy',
|
||||
activity: 'active',
|
||||
memory: decision.memory,
|
||||
});
|
||||
assert.equal(decision.status, 'waiting-for-idle');
|
||||
decision = nextFailoverDecision({
|
||||
now: 31_000,
|
||||
policy: enabled,
|
||||
currentRole: 'primary',
|
||||
primaryHealth: 'unhealthy',
|
||||
reserveHealth: 'healthy',
|
||||
activity: 'quiet',
|
||||
memory: decision.memory,
|
||||
});
|
||||
assert.equal(decision.switchTo, null);
|
||||
decision = nextFailoverDecision({
|
||||
now: 36_000,
|
||||
policy: enabled,
|
||||
currentRole: 'primary',
|
||||
primaryHealth: 'unhealthy',
|
||||
reserveHealth: 'healthy',
|
||||
activity: 'quiet',
|
||||
memory: decision.memory,
|
||||
});
|
||||
assert.equal(decision.switchTo, 'reserve');
|
||||
});
|
||||
|
||||
test('both unhealthy and unknown activity never cause a switch', () => {
|
||||
const both = nextFailoverDecision({
|
||||
now: 30_000,
|
||||
policy: enabled,
|
||||
currentRole: 'primary',
|
||||
primaryHealth: 'unhealthy',
|
||||
reserveHealth: 'unhealthy',
|
||||
activity: 'quiet',
|
||||
});
|
||||
assert.equal(both.reason, 'both-unhealthy');
|
||||
assert.equal(both.switchTo, null);
|
||||
|
||||
const unknown = nextFailoverDecision({
|
||||
now: 30_000,
|
||||
policy: enabled,
|
||||
currentRole: 'primary',
|
||||
primaryHealth: 'unhealthy',
|
||||
reserveHealth: 'healthy',
|
||||
activity: 'unknown',
|
||||
memory: { primaryFailedSince: 0, primaryRecoveredSince: null, quietSince: null },
|
||||
});
|
||||
assert.equal(unknown.reason, 'activity-unknown');
|
||||
assert.equal(unknown.switchTo, null);
|
||||
});
|
||||
|
||||
test('failback waits for recovery, hold and quarantine deadlines', () => {
|
||||
let decision = nextFailoverDecision({
|
||||
now: 10_000,
|
||||
policy: enabled,
|
||||
currentRole: 'reserve',
|
||||
primaryHealth: 'healthy',
|
||||
reserveHealth: 'healthy',
|
||||
activity: 'quiet',
|
||||
holdUntil: 50_000,
|
||||
primaryQuarantineUntil: 70_000,
|
||||
});
|
||||
assert.equal(decision.nextDecisionAt, 70_000);
|
||||
decision = nextFailoverDecision({
|
||||
now: 70_000,
|
||||
policy: enabled,
|
||||
currentRole: 'reserve',
|
||||
primaryHealth: 'healthy',
|
||||
reserveHealth: 'healthy',
|
||||
activity: 'quiet',
|
||||
holdUntil: 50_000,
|
||||
primaryQuarantineUntil: 70_000,
|
||||
memory: decision.memory,
|
||||
});
|
||||
assert.equal(decision.switchTo, null);
|
||||
assert.equal(decision.nextDecisionAt, 130_000);
|
||||
decision = nextFailoverDecision({
|
||||
now: 130_000,
|
||||
policy: enabled,
|
||||
currentRole: 'reserve',
|
||||
primaryHealth: 'healthy',
|
||||
reserveHealth: 'healthy',
|
||||
activity: 'quiet',
|
||||
holdUntil: 50_000,
|
||||
primaryQuarantineUntil: 70_000,
|
||||
memory: { ...decision.memory, quietSince: 125_000 },
|
||||
});
|
||||
assert.equal(decision.switchTo, 'primary');
|
||||
|
||||
const deadReserve = nextFailoverDecision({
|
||||
now: 70_000,
|
||||
policy: enabled,
|
||||
currentRole: 'reserve',
|
||||
primaryHealth: 'healthy',
|
||||
reserveHealth: 'unhealthy',
|
||||
activity: 'quiet',
|
||||
holdUntil: 500_000,
|
||||
primaryQuarantineUntil: 700_000,
|
||||
memory: { primaryFailedSince: null, primaryRecoveredSince: 10_000, quietSince: 65_000 },
|
||||
});
|
||||
assert.equal(deadReserve.switchTo, null);
|
||||
assert.equal(deadReserve.nextDecisionAt, 700_000);
|
||||
});
|
||||
|
||||
function serviceState(policy = enabled) {
|
||||
return {
|
||||
revision: 1,
|
||||
failoverPolicy: policy,
|
||||
failoverRuntimeState: { lastSwitchAt: null, holdUntil: null, primaryQuarantineUntil: null, failoverHistory: [], reasonCode: null },
|
||||
appliedFailoverPolicy: {
|
||||
primary: { profileId: 'profile-1', serverId: 'server-1' },
|
||||
reserve: { profileId: 'profile-2', serverId: 'server-2' },
|
||||
primaryConfigFingerprint: 'a'.repeat(64),
|
||||
reserveConfigFingerprint: 'b'.repeat(64),
|
||||
},
|
||||
appliedProfileId: 'profile-1',
|
||||
appliedServerId: 'server-1',
|
||||
profiles: [
|
||||
{ id: 'profile-1', servers: [{ id: 'server-1', label: 'Primary' }] },
|
||||
{ id: 'profile-2', servers: [{ id: 'server-2', label: 'Reserve' }] },
|
||||
],
|
||||
diagnostics: { customServices: [] },
|
||||
};
|
||||
}
|
||||
|
||||
function harness(initial = serviceState(), overrides = {}) {
|
||||
let state = structuredClone(initial);
|
||||
const calls = [];
|
||||
const dependencies = {
|
||||
state: {
|
||||
read: () => state,
|
||||
update: (mutator) => {
|
||||
state = { ...mutator(state), revision: state.revision + 1 };
|
||||
calls.push('state:update');
|
||||
return state;
|
||||
},
|
||||
},
|
||||
runtime: { isRunning: async () => true },
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async () => ({ vpn: { sites: [{ status: 'available' }] } }),
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async (role) => { calls.push(`select:${role}`); return { role }; },
|
||||
setFailoverActivityEnabled: async (value) => { calls.push(`activity:${value}`); },
|
||||
readFailoverActivity: async () => ({ activity: { state: 'quiet', observedAt: new Date().toISOString() } }),
|
||||
},
|
||||
buildCandidate: () => ({ config: {}, applied: initial.appliedFailoverPolicy }),
|
||||
serialize: (operation) => operation(),
|
||||
scheduler: {
|
||||
setTimeout: () => ({ unref() {} }),
|
||||
clearTimeout: () => {},
|
||||
},
|
||||
...overrides,
|
||||
};
|
||||
return { service: createFailoverService(dependencies), calls, read: () => state };
|
||||
}
|
||||
|
||||
test('disabled failover performs one restart-safe collector cleanup and no steady-state work', async () => {
|
||||
const state = serviceState(normalizeFailoverPolicy(null));
|
||||
state.appliedFailoverPolicy = null;
|
||||
const testHarness = harness(state);
|
||||
await testHarness.service.reconcile();
|
||||
assert.deepEqual(testHarness.calls, ['activity:false']);
|
||||
await testHarness.service.reconcile();
|
||||
assert.deepEqual(testHarness.calls, ['activity:false']);
|
||||
assert.equal(testHarness.service.snapshot().activation, 'inactive');
|
||||
});
|
||||
|
||||
test('disabled failover reports an already loaded dual config without managing it', async () => {
|
||||
const testHarness = harness(serviceState(normalizeFailoverPolicy(null)));
|
||||
await testHarness.service.reconcile();
|
||||
assert.deepEqual(testHarness.calls, ['activity:false']);
|
||||
assert.equal(testHarness.service.snapshot().activation, 'passive-loaded');
|
||||
assert.equal(testHarness.service.snapshot().currentRole, 'primary');
|
||||
});
|
||||
|
||||
test('runtime rollback restores the role from applied truth even while disabled', async () => {
|
||||
const state = serviceState(normalizeFailoverPolicy(null));
|
||||
state.appliedProfileId = state.appliedFailoverPolicy.reserve.profileId;
|
||||
state.appliedServerId = state.appliedFailoverPolicy.reserve.serverId;
|
||||
const testHarness = harness(state);
|
||||
await testHarness.service.restoreAppliedActivation(state);
|
||||
assert.deepEqual(testHarness.calls, ['select:reserve']);
|
||||
});
|
||||
|
||||
test('an in-flight observation is discarded after failover is disabled', async () => {
|
||||
const pending = [];
|
||||
const testHarness = harness(serviceState(), {
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: (role) => new Promise((resolve) => pending.push({ role, resolve })),
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async (role) => { testHarness.calls.push(`select:${role}`); return { role }; },
|
||||
setFailoverActivityEnabled: async (value) => { testHarness.calls.push(`activity:${value}`); },
|
||||
readFailoverActivity: async () => ({ activity: { state: 'quiet', observedAt: new Date().toISOString() } }),
|
||||
},
|
||||
});
|
||||
await testHarness.service.reconcile();
|
||||
const round = testHarness.service.runRound();
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
await testHarness.service.save({ ...enabled, enabled: false });
|
||||
for (const probe of pending) probe.resolve({ vpn: { sites: [{ status: probe.role === 'reserve' ? 'available' : 'unavailable' }] } });
|
||||
await round;
|
||||
assert.equal(testHarness.read().failoverPolicy.enabled, false);
|
||||
assert.equal(testHarness.calls.some((call) => call.startsWith('select:reserve')), false);
|
||||
assert.deepEqual(testHarness.calls.filter((call) => call.startsWith('activity:')), ['activity:true', 'activity:true', 'activity:false']);
|
||||
});
|
||||
|
||||
test('one channel probe failure does not erase the other channel health', async () => {
|
||||
const testHarness = harness(serviceState(), {
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async (role) => {
|
||||
if (role === 'primary') throw new Error('primary probe transport failed');
|
||||
return { vpn: { sites: [{ status: 'available' }] } };
|
||||
},
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async (role) => ({ role }),
|
||||
setFailoverActivityEnabled: async () => {},
|
||||
readFailoverActivity: async () => ({ activity: { state: 'active', observedAt: new Date().toISOString() } }),
|
||||
},
|
||||
});
|
||||
await testHarness.service.checkNow();
|
||||
assert.equal(testHarness.service.snapshot().primary.health, 'unknown');
|
||||
assert.equal(testHarness.service.snapshot().reserve.health, 'healthy');
|
||||
});
|
||||
|
||||
test('monitoring wakes at an earlier decision deadline instead of waiting a full interval', async () => {
|
||||
let clock = 0;
|
||||
const delays = [];
|
||||
const testHarness = harness(serviceState(), {
|
||||
now: () => new Date(clock),
|
||||
scheduler: {
|
||||
setTimeout: (_callback, delay) => { delays.push(delay); return { unref() {} }; },
|
||||
clearTimeout: () => {},
|
||||
},
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async (role) => ({ vpn: { sites: [{ status: role === 'primary' ? 'unavailable' : 'available' }] } }),
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async (role) => ({ role }),
|
||||
setFailoverActivityEnabled: async () => {},
|
||||
readFailoverActivity: async () => ({ activity: { state: 'quiet', observedAt: new Date(clock).toISOString() } }),
|
||||
},
|
||||
});
|
||||
await testHarness.service.reconcile();
|
||||
await testHarness.service.runRound();
|
||||
clock = 30_000;
|
||||
await testHarness.service.runRound();
|
||||
assert.equal(delays.at(-1), enabled.trafficGuard.quietWindowMs);
|
||||
});
|
||||
|
||||
test('a late activity response cannot switch after failover is disabled', async () => {
|
||||
let resolveActivity;
|
||||
const activity = new Promise((resolve) => { resolveActivity = resolve; });
|
||||
const testHarness = harness(serviceState(), {
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async (role) => ({ vpn: { sites: [{ status: role === 'primary' ? 'unavailable' : 'available' }] } }),
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async (role) => { testHarness.calls.push(`select:${role}`); return { role }; },
|
||||
setFailoverActivityEnabled: async (value) => { testHarness.calls.push(`activity:${value}`); },
|
||||
readFailoverActivity: async () => activity,
|
||||
},
|
||||
});
|
||||
await testHarness.service.reconcile();
|
||||
const round = testHarness.service.runRound();
|
||||
await new Promise(setImmediate);
|
||||
await testHarness.service.save({ ...enabled, enabled: false });
|
||||
resolveActivity({ activity: { state: 'quiet', observedAt: new Date().toISOString() } });
|
||||
await round;
|
||||
assert.equal(testHarness.calls.some((value) => value.startsWith('select:')), false);
|
||||
});
|
||||
|
||||
test('automatic switch revalidates activity immediately before selector mutation', async () => {
|
||||
let clock = 0;
|
||||
let activityReads = 0;
|
||||
const testHarness = harness(serviceState(), {
|
||||
now: () => new Date(clock),
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async (role) => ({ vpn: { sites: [{ status: role === 'primary' ? 'unavailable' : 'available' }] } }),
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async (role) => { testHarness.calls.push(`select:${role}`); return { role }; },
|
||||
setFailoverActivityEnabled: async () => {},
|
||||
readFailoverActivity: async () => ({
|
||||
activity: {
|
||||
state: ++activityReads >= 4 ? 'active' : 'quiet',
|
||||
observedAt: new Date(clock).toISOString(),
|
||||
},
|
||||
}),
|
||||
},
|
||||
});
|
||||
await testHarness.service.reconcile();
|
||||
await testHarness.service.runRound();
|
||||
clock = 30_000;
|
||||
await testHarness.service.runRound();
|
||||
clock = 35_000;
|
||||
await testHarness.service.runRound();
|
||||
assert.equal(testHarness.calls.some((value) => value.startsWith('select:')), false);
|
||||
assert.equal(testHarness.service.snapshot().reason, 'revalidation-required');
|
||||
});
|
||||
|
||||
test('selector read-back failure rolls an uncertain switch back to the canonical role', async () => {
|
||||
const testHarness = harness(serviceState(), {
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async () => ({}),
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async (role) => {
|
||||
testHarness.calls.push(`select:${role}`);
|
||||
if (role === 'reserve') throw new Error('read-back failed');
|
||||
return { role };
|
||||
},
|
||||
setFailoverActivityEnabled: async () => {},
|
||||
readFailoverActivity: async () => ({ activity: null }),
|
||||
},
|
||||
});
|
||||
await assert.rejects(testHarness.service.manualSwitch('reserve'), /read-back failed/);
|
||||
assert.deepEqual(testHarness.calls, ['select:reserve', 'select:primary']);
|
||||
assert.equal(testHarness.read().appliedServerId, 'server-1');
|
||||
});
|
||||
|
||||
test('failed collector disable is retried because local state remains unknown', async () => {
|
||||
let attempts = 0;
|
||||
const testHarness = harness(serviceState(normalizeFailoverPolicy(null)), {
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async () => ({}),
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async () => ({}),
|
||||
setFailoverActivityEnabled: async () => {
|
||||
attempts += 1;
|
||||
if (attempts === 1) throw new Error('dataplane unavailable');
|
||||
},
|
||||
readFailoverActivity: async () => ({ activity: null }),
|
||||
},
|
||||
});
|
||||
await assert.rejects(testHarness.service.reconcile(), /dataplane unavailable/);
|
||||
await testHarness.service.reconcile();
|
||||
assert.equal(attempts, 2);
|
||||
});
|
||||
|
||||
test('each monitoring round repairs selector and collector state after a dataplane restart', async () => {
|
||||
const state = serviceState();
|
||||
state.appliedProfileId = 'profile-2';
|
||||
state.appliedServerId = 'server-2';
|
||||
let selected = 'reserve';
|
||||
let enables = 0;
|
||||
const testHarness = harness(state, {
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async () => ({ vpn: { sites: [{ status: 'available' }] } }),
|
||||
readFailoverSelector: async () => ({ role: selected }),
|
||||
selectFailoverRole: async (role) => { selected = role; return { role }; },
|
||||
setFailoverActivityEnabled: async (value) => { if (value) enables += 1; },
|
||||
readFailoverActivity: async () => ({ activity: { state: 'quiet', observedAt: new Date().toISOString() } }),
|
||||
},
|
||||
});
|
||||
await testHarness.service.reconcile();
|
||||
selected = 'primary';
|
||||
await testHarness.service.runRound();
|
||||
assert.equal(selected, 'reserve');
|
||||
assert.equal(enables, 2);
|
||||
});
|
||||
|
||||
test('selector acknowledgement precedes state commit and a failed commit rolls selector back', async () => {
|
||||
const state = serviceState();
|
||||
const calls = [];
|
||||
const testHarness = harness(state, {
|
||||
state: {
|
||||
read: () => state,
|
||||
update: (mutator) => {
|
||||
const candidate = mutator(state);
|
||||
calls.push('state:update');
|
||||
if (candidate.appliedServerId === 'server-2') throw new Error('write failed');
|
||||
Object.assign(state, candidate);
|
||||
return state;
|
||||
},
|
||||
},
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async () => ({}),
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async (role) => { calls.push(`select:${role}`); return { role }; },
|
||||
setFailoverActivityEnabled: async () => {},
|
||||
readFailoverActivity: async () => ({ activity: null }),
|
||||
},
|
||||
});
|
||||
await assert.rejects(testHarness.service.manualSwitch('reserve'), /write failed/);
|
||||
assert.deepEqual(calls, ['select:reserve', 'state:update', 'select:primary']);
|
||||
assert.equal(state.appliedServerId, 'server-1');
|
||||
});
|
||||
|
||||
test('manual selector and pause publish in one canonical commit', async () => {
|
||||
const testHarness = harness(serviceState());
|
||||
await testHarness.service.manualSwitch('reserve');
|
||||
assert.equal(testHarness.read().appliedServerId, 'server-2');
|
||||
assert.equal(testHarness.read().failoverPolicy.paused, true);
|
||||
assert.equal(testHarness.calls.filter((call) => call === 'state:update').length, 1);
|
||||
});
|
||||
|
||||
test('post-commit monitoring failure does not report a committed pause as failed', async () => {
|
||||
const warnings = [];
|
||||
const testHarness = harness(serviceState(), {
|
||||
runtime: { isRunning: async () => { throw new Error('runtime unavailable'); } },
|
||||
onWarning: (error) => warnings.push(error.message),
|
||||
});
|
||||
await testHarness.service.pause(true);
|
||||
assert.equal(testHarness.read().failoverPolicy.paused, true);
|
||||
assert.equal(testHarness.service.snapshot().status, 'error');
|
||||
assert.deepEqual(warnings, ['runtime unavailable']);
|
||||
});
|
||||
|
||||
test('pause and resume discard stale failure-window evidence', async () => {
|
||||
let clock = 1_000;
|
||||
const testHarness = harness(serviceState(), {
|
||||
now: () => new Date(clock),
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async (role) => ({
|
||||
vpn: { sites: [{ status: role === 'primary' ? 'unavailable' : 'available' }] },
|
||||
}),
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async (role) => { testHarness.calls.push(`select:${role}`); return { role }; },
|
||||
setFailoverActivityEnabled: async (value) => { testHarness.calls.push(`activity:${value}`); },
|
||||
readFailoverActivity: async () => ({
|
||||
activity: { state: 'quiet', observedAt: new Date(clock).toISOString(), quietSince: new Date(clock - 10_000).toISOString() },
|
||||
}),
|
||||
},
|
||||
});
|
||||
await testHarness.service.runRound();
|
||||
await testHarness.service.pause(true);
|
||||
clock = 3_600_000;
|
||||
await testHarness.service.pause(false);
|
||||
await testHarness.service.runRound();
|
||||
assert.equal(testHarness.calls.includes('select:reserve'), false);
|
||||
assert.equal(testHarness.service.snapshot().reason, 'failure-window');
|
||||
});
|
||||
|
||||
test('an unconfirmed selector rollback durably pauses automation', async () => {
|
||||
const testHarness = harness(serviceState(), {
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async () => ({}),
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async () => { throw new Error('selector unavailable'); },
|
||||
setFailoverActivityEnabled: async () => {},
|
||||
readFailoverActivity: async () => ({ activity: null }),
|
||||
},
|
||||
});
|
||||
await assert.rejects(testHarness.service.manualSwitch('reserve'), AggregateError);
|
||||
assert.equal(testHarness.read().failoverPolicy.paused, true);
|
||||
assert.equal(testHarness.read().failoverRuntimeState.reasonCode, 'selector-unknown');
|
||||
assert.equal(testHarness.service.snapshot().reason, 'selector-unknown');
|
||||
assert.equal(testHarness.service.snapshot().currentRole, 'other');
|
||||
});
|
||||
|
||||
test('successful selector reconciliation clears an unknown-role latch', async () => {
|
||||
const state = serviceState();
|
||||
state.failoverRuntimeState.reasonCode = 'selector-unknown';
|
||||
const testHarness = harness(state);
|
||||
await testHarness.service.reconcile();
|
||||
assert.equal(testHarness.read().failoverRuntimeState.reasonCode, null);
|
||||
assert.equal(testHarness.service.snapshot().currentRole, 'primary');
|
||||
});
|
||||
|
||||
test('disabling on reserve preserves it as the next ordinary single-channel target', async () => {
|
||||
const state = serviceState();
|
||||
state.appliedProfileId = 'profile-2';
|
||||
state.appliedServerId = 'server-2';
|
||||
const testHarness = harness(state);
|
||||
await testHarness.service.save({ ...enabled, enabled: false });
|
||||
assert.equal(testHarness.read().desiredProfileId, 'profile-2');
|
||||
assert.equal(testHarness.read().profiles[1].desiredServerId, 'server-2');
|
||||
assert.equal(testHarness.service.snapshot().activation, 'passive-loaded');
|
||||
});
|
||||
|
||||
test('significant health states append once without logging every observation', async () => {
|
||||
let clock = 0;
|
||||
const health = { primary: 'unavailable', reserve: 'available' };
|
||||
const events = [];
|
||||
const testHarness = harness(serviceState(), {
|
||||
now: () => new Date(clock),
|
||||
onEvent: (event) => events.push(event),
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async (role) => ({ vpn: { sites: [{ status: health[role] }] } }),
|
||||
readFailoverSelector: async () => ({ role: 'primary' }),
|
||||
selectFailoverRole: async (role) => ({ role }),
|
||||
setFailoverActivityEnabled: async () => {},
|
||||
readFailoverActivity: async () => ({ activity: { state: 'quiet', observedAt: new Date(clock).toISOString() } }),
|
||||
},
|
||||
});
|
||||
await testHarness.service.reconcile();
|
||||
await testHarness.service.runRound();
|
||||
clock = 30_000;
|
||||
await testHarness.service.runRound();
|
||||
health.reserve = 'unavailable';
|
||||
clock = 31_000;
|
||||
await testHarness.service.runRound();
|
||||
clock = 32_000;
|
||||
await testHarness.service.runRound();
|
||||
health.primary = 'available';
|
||||
clock = 33_000;
|
||||
await testHarness.service.runRound();
|
||||
|
||||
assert.deepEqual(events.map(({ type }) => type), [
|
||||
'failover.waiting_for_idle',
|
||||
'failover.both_unhealthy',
|
||||
'failover.recovered',
|
||||
]);
|
||||
});
|
||||
|
||||
test('pending target edits keep selector commits bound to the loaded channel pair', async () => {
|
||||
const state = serviceState();
|
||||
state.failoverPolicy = normalizeFailoverPolicy({
|
||||
...state.failoverPolicy,
|
||||
reserve: { profileId: 'profile-3', serverId: 'server-3' },
|
||||
});
|
||||
state.profiles.push({ id: 'profile-3', servers: [{ id: 'server-3', label: 'Desired later' }] });
|
||||
const testHarness = harness(state, {
|
||||
buildCandidate: () => ({
|
||||
config: {},
|
||||
applied: {
|
||||
...state.appliedFailoverPolicy,
|
||||
reserve: { profileId: 'profile-3', serverId: 'server-3' },
|
||||
reserveConfigFingerprint: 'c'.repeat(64),
|
||||
},
|
||||
}),
|
||||
});
|
||||
|
||||
await testHarness.service.manualSwitch('reserve');
|
||||
assert.equal(testHarness.read().appliedProfileId, 'profile-2');
|
||||
assert.equal(testHarness.read().appliedServerId, 'server-2');
|
||||
assert.equal(testHarness.service.snapshot().activation, 'pending');
|
||||
assert.deepEqual(testHarness.service.snapshot().reserve.target, { profileId: 'profile-2', serverId: 'server-2' });
|
||||
});
|
||||
@@ -109,6 +109,9 @@ function createHarness(overrides = {}) {
|
||||
events.push('operation');
|
||||
return operation();
|
||||
},
|
||||
restoreAppliedActivation: overrides.restoreAppliedActivation
|
||||
? async (previousState) => overrides.restoreAppliedActivation(previousState, events)
|
||||
: undefined,
|
||||
});
|
||||
|
||||
return {
|
||||
@@ -246,6 +249,24 @@ test('route rules rollback continues and classifies runtime restore failure', as
|
||||
assert.ok(broken.events.filter((event) => event === 'state.update').length >= 1);
|
||||
});
|
||||
|
||||
test('route rules rollback restores the previous reserve selector', async () => {
|
||||
const state = canonicalState();
|
||||
state.appliedFailoverPolicy = {
|
||||
primary: { profileId: 'primary', serverId: 'other' },
|
||||
reserve: { profileId: 'primary', serverId: 'server' },
|
||||
};
|
||||
const harness = createHarness({
|
||||
state,
|
||||
failures: { stateUpdates: [{ after: new Error('state') }] },
|
||||
restoreAppliedActivation: async (previousState, events) => {
|
||||
assert.equal(previousState.appliedServerId, 'server');
|
||||
events.push('selector.reserve');
|
||||
},
|
||||
});
|
||||
await assert.rejects(harness.service.update(newRules, 2, 2), /state/);
|
||||
assert.ok(harness.events.indexOf('selector.reserve') > harness.events.indexOf('runtime.restore'));
|
||||
});
|
||||
|
||||
test('route rules route preserves one adapter and state-only response', async () => {
|
||||
const calls = [];
|
||||
const route = createRouteRulesRoute({
|
||||
@@ -260,5 +281,6 @@ test('route rules route preserves one adapter and state-only response', async ()
|
||||
|
||||
const source = readFileSync(new URL('../../src/server/index.ts', import.meta.url), 'utf8');
|
||||
assert.match(source, /createRouteRulesRoute\(\{/);
|
||||
assert.match(source, /state\.appliedFailoverPolicy[\s\S]*buildFailoverCandidate\(\{ \.\.\.state, routeRules \}, 'applied'\)\.config/);
|
||||
assert.doesNotMatch(source, /function applyRouteRules|req\.url === ['"]\/api\/route-rules['"]/);
|
||||
});
|
||||
|
||||
@@ -8,7 +8,13 @@ process.env.APP_MODE = 'gateway';
|
||||
process.env.DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'vpn-proxy-gateway-test-'));
|
||||
process.env.SING_BOX_CACHE = path.join(process.env.DATA_DIR, 'cache.db');
|
||||
|
||||
const { buildGatewayConfig } = await import(`../../dist/server/singbox.js?gateway=${Date.now()}`);
|
||||
const {
|
||||
buildDualChannelGatewayConfig,
|
||||
buildGatewayConfig,
|
||||
dualChannelConfigMatchesApplied,
|
||||
fingerprintConfiguredOutbound,
|
||||
fingerprintSelectedOutbound,
|
||||
} = await import(`../../dist/server/singbox.js?gateway=${Date.now()}`);
|
||||
|
||||
const subscriptionConfig = {
|
||||
outbounds: [{
|
||||
@@ -59,3 +65,62 @@ test('gateway preserves mixed user-rule order and dynamic VPN target', () => {
|
||||
assert.deepEqual(config.experimental.clash_api, { external_controller: '127.0.0.1:19090' });
|
||||
assert.equal(config.route.final, 'test-vpn');
|
||||
});
|
||||
|
||||
test('gateway dual-channel config fixes probes to role tags and keeps inbound connections', () => {
|
||||
const reserveConfig = structuredClone(subscriptionConfig);
|
||||
reserveConfig.outbounds[0].tag = 'same-provider-tag';
|
||||
const primaryConfig = structuredClone(subscriptionConfig);
|
||||
primaryConfig.outbounds[0].tag = 'same-provider-tag';
|
||||
const config = buildDualChannelGatewayConfig({
|
||||
primary: { subscriptionConfig: primaryConfig, selectedServerId: 'same-provider-tag' },
|
||||
reserve: { subscriptionConfig: reserveConfig, selectedServerId: 'same-provider-tag' },
|
||||
}, {
|
||||
routeRules: [{ type: 'domain', value: 'api.example.com', enabled: true, outbound: 'vpn' }],
|
||||
});
|
||||
|
||||
assert.deepEqual(config.outbounds.map(({ tag }) => tag), [
|
||||
'channel-primary', 'channel-reserve', 'channel-selector', 'direct',
|
||||
]);
|
||||
assert.deepEqual(config.outbounds[2], {
|
||||
type: 'selector',
|
||||
tag: 'channel-selector',
|
||||
outbounds: ['channel-primary', 'channel-reserve'],
|
||||
default: 'channel-primary',
|
||||
interrupt_exist_connections: false,
|
||||
});
|
||||
assert.deepEqual(config.route.rules.slice(1, 5), [
|
||||
{ inbound: ['diagnostics-primary-in'], outbound: 'channel-primary' },
|
||||
{ inbound: ['diagnostics-reserve-in'], outbound: 'channel-reserve' },
|
||||
{ inbound: ['diagnostics-vpn-in'], outbound: 'channel-selector' },
|
||||
{ domain: ['api.example.com'], outbound: 'channel-selector' },
|
||||
]);
|
||||
assert.equal(config.route.final, 'channel-selector');
|
||||
const restoredReserve = buildDualChannelGatewayConfig({
|
||||
primary: { subscriptionConfig: primaryConfig, selectedServerId: 'same-provider-tag' },
|
||||
reserve: { subscriptionConfig: reserveConfig, selectedServerId: 'same-provider-tag' },
|
||||
}, { defaultRole: 'reserve' });
|
||||
assert.equal(restoredReserve.outbounds[2].default, 'channel-reserve');
|
||||
});
|
||||
|
||||
test('cached dual-channel outbounds must match the applied provider fingerprints', () => {
|
||||
const config = buildDualChannelGatewayConfig({
|
||||
primary: { subscriptionConfig, selectedServerId: 'test-vpn' },
|
||||
reserve: { subscriptionConfig, selectedServerId: 'test-vpn' },
|
||||
});
|
||||
const expected = fingerprintSelectedOutbound(subscriptionConfig, 'test-vpn');
|
||||
const applied = {
|
||||
primary: { profileId: 'primary', serverId: 'test-vpn' },
|
||||
reserve: { profileId: 'reserve', serverId: 'test-vpn' },
|
||||
primaryConfigFingerprint: expected,
|
||||
reserveConfigFingerprint: expected,
|
||||
};
|
||||
assert.equal(dualChannelConfigMatchesApplied(config, applied, 'primary'), true);
|
||||
assert.equal(dualChannelConfigMatchesApplied(config, applied, 'reserve'), false);
|
||||
assert.equal(fingerprintConfiguredOutbound(config.outbounds[0], 'test-vpn'), expected);
|
||||
config.outbounds[0].uuid = '11111111-1111-4111-8111-111111111111';
|
||||
assert.notEqual(fingerprintConfiguredOutbound(config.outbounds[0], 'test-vpn'), expected);
|
||||
assert.equal(dualChannelConfigMatchesApplied(config, applied, 'primary'), false);
|
||||
config.outbounds[0].uuid = subscriptionConfig.outbounds[0].uuid;
|
||||
config.outbounds[2].outbounds = ['channel-primary'];
|
||||
assert.equal(dualChannelConfigMatchesApplied(config, applied, 'primary'), false);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,361 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawn, spawnSync } from 'node:child_process';
|
||||
import dgram from 'node:dgram';
|
||||
import fs from 'node:fs';
|
||||
import net from 'node:net';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import test from 'node:test';
|
||||
|
||||
import { createFailoverService } from '../../dist/server/features/failover/failoverService.js';
|
||||
import { createDomainTrafficService } from '../../dist/server/services/domainTrafficService.js';
|
||||
import { DEFAULT_FAILOVER_POLICY, normalizeFailoverPolicy } from '../../dist/shared/failover.js';
|
||||
|
||||
const image = process.env.HARBOR_SINGBOX_IMAGE;
|
||||
|
||||
function listen(server, host = '127.0.0.1') {
|
||||
return new Promise((resolve, reject) => {
|
||||
server.once('error', reject);
|
||||
server.listen(0, host, () => resolve(server.address().port));
|
||||
});
|
||||
}
|
||||
|
||||
function listenUdp(socket, host = '127.0.0.1') {
|
||||
return new Promise((resolve, reject) => {
|
||||
socket.once('error', reject);
|
||||
socket.bind(0, host, () => resolve(socket.address().port));
|
||||
});
|
||||
}
|
||||
|
||||
function readExactly(socket, size) {
|
||||
return new Promise((resolve, reject) => {
|
||||
let value = Buffer.alloc(0);
|
||||
const onData = (chunk) => {
|
||||
value = Buffer.concat([value, chunk]);
|
||||
if (value.length < size) return;
|
||||
cleanup();
|
||||
if (value.length > size) {
|
||||
socket.pause();
|
||||
socket.unshift(value.subarray(size));
|
||||
}
|
||||
resolve(value.subarray(0, size));
|
||||
};
|
||||
const cleanup = () => {
|
||||
socket.off('data', onData);
|
||||
socket.off('error', reject);
|
||||
socket.off('end', onEnd);
|
||||
};
|
||||
const onEnd = () => {
|
||||
cleanup();
|
||||
reject(new Error('Socket ended early'));
|
||||
};
|
||||
socket.on('data', onData);
|
||||
socket.once('error', reject);
|
||||
socket.once('end', onEnd);
|
||||
socket.resume();
|
||||
});
|
||||
}
|
||||
|
||||
async function openSocksConnection(proxyPort, targetPort, diagnostic = () => {}) {
|
||||
const socket = net.connect(proxyPort, '127.0.0.1');
|
||||
socket.setTimeout(3_000, () => socket.destroy(new Error('SOCKS fixture timed out')));
|
||||
await new Promise((resolve, reject) => {
|
||||
socket.once('connect', resolve);
|
||||
socket.once('error', reject);
|
||||
});
|
||||
diagnostic('tcp connected');
|
||||
socket.write(Buffer.from([5, 1, 0]));
|
||||
assert.deepEqual(await readExactly(socket, 2), Buffer.from([5, 0]));
|
||||
diagnostic('socks greeting accepted');
|
||||
const host = Buffer.from('host.docker.internal');
|
||||
socket.write(Buffer.from([5, 1, 0, 3, host.length, ...host, targetPort >> 8, targetPort & 255]));
|
||||
const response = await readExactly(socket, 4);
|
||||
diagnostic(`socks connect response ${response.toString('hex')}`);
|
||||
assert.equal(response[1], 0);
|
||||
const addressLength = response[3] === 0
|
||||
? 0
|
||||
: response[3] === 1
|
||||
? 4
|
||||
: response[3] === 4
|
||||
? 16
|
||||
: (await readExactly(socket, 1))[0];
|
||||
await readExactly(socket, addressLength + 2);
|
||||
return socket;
|
||||
}
|
||||
|
||||
async function echo(socket, value) {
|
||||
socket.write(value);
|
||||
assert.equal((await readExactly(socket, Buffer.byteLength(value))).toString(), value);
|
||||
}
|
||||
|
||||
async function openSocksUdpAssociation(proxyPort, targetPort, resolveRelayPort) {
|
||||
const control = net.connect(proxyPort, '127.0.0.1');
|
||||
control.setTimeout(3_000, () => control.destroy(new Error('SOCKS UDP fixture timed out')));
|
||||
await new Promise((resolve, reject) => {
|
||||
control.once('connect', resolve);
|
||||
control.once('error', reject);
|
||||
});
|
||||
control.write(Buffer.from([5, 1, 0]));
|
||||
assert.deepEqual(await readExactly(control, 2), Buffer.from([5, 0]));
|
||||
control.write(Buffer.from([5, 3, 0, 1, 0, 0, 0, 0, 0, 0]));
|
||||
const response = await readExactly(control, 4);
|
||||
assert.equal(response[1], 0);
|
||||
const addressLength = response[3] === 1 ? 4 : response[3] === 4 ? 16 : (await readExactly(control, 1))[0];
|
||||
const addressAndPort = await readExactly(control, addressLength + 2);
|
||||
const boundPort = addressAndPort.readUInt16BE(addressAndPort.length - 2);
|
||||
const relayPort = resolveRelayPort(boundPort);
|
||||
|
||||
const socket = dgram.createSocket('udp4');
|
||||
await listenUdp(socket);
|
||||
return {
|
||||
close() { socket.close(); control.destroy(); },
|
||||
async echo(value) {
|
||||
const host = Buffer.from('host.docker.internal');
|
||||
const payload = Buffer.from(value);
|
||||
const packet = Buffer.from([0, 0, 0, 3, host.length, ...host, targetPort >> 8, targetPort & 255, ...payload]);
|
||||
const reply = new Promise((resolve, reject) => {
|
||||
const timeout = setTimeout(() => reject(new Error(`SOCKS UDP echo timed out; relay ${boundPort}`)), 3_000);
|
||||
socket.once('message', (message) => {
|
||||
clearTimeout(timeout);
|
||||
const headerLength = message[3] === 1 ? 10 : message[3] === 4 ? 22 : 7 + message[4];
|
||||
resolve(message.subarray(headerLength).toString());
|
||||
});
|
||||
});
|
||||
socket.send(packet, relayPort, '127.0.0.1');
|
||||
assert.equal(await reply, value);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function waitForApi(port) {
|
||||
for (let attempt = 0; attempt < 80; attempt += 1) {
|
||||
try {
|
||||
const response = await fetch(`http://127.0.0.1:${port}/proxies/channel-selector`);
|
||||
if (response.ok) return;
|
||||
} catch {}
|
||||
await new Promise((resolve) => setTimeout(resolve, 100));
|
||||
}
|
||||
throw new Error('Clash API did not start');
|
||||
}
|
||||
|
||||
test('sing-box 1.13 selector preserves inbound TCP and UDP connections and routes new connections', {
|
||||
skip: image ? false : 'Set HARBOR_SINGBOX_IMAGE to run the local Docker capability proof',
|
||||
timeout: 30_000,
|
||||
}, async (t) => {
|
||||
const echoServer = net.createServer((socket) => socket.pipe(socket));
|
||||
const echoPort = await listen(echoServer, '0.0.0.0');
|
||||
const udpEchoServer = dgram.createSocket('udp4');
|
||||
udpEchoServer.on('message', (message, remote) => udpEchoServer.send(message, remote.port, remote.address));
|
||||
const udpEchoPort = await listenUdp(udpEchoServer, '0.0.0.0');
|
||||
const proxyReservation = net.createServer();
|
||||
const proxyPort = await listen(proxyReservation);
|
||||
await new Promise((resolve) => proxyReservation.close(resolve));
|
||||
const apiReservation = net.createServer();
|
||||
const apiPort = await listen(apiReservation);
|
||||
await new Promise((resolve) => apiReservation.close(resolve));
|
||||
const fixtureDir = fs.mkdtempSync(path.join(os.tmpdir(), 'harbor-selector-'));
|
||||
const configPath = path.join(fixtureDir, 'config.json');
|
||||
const config = {
|
||||
log: { level: 'error' },
|
||||
experimental: {
|
||||
cache_file: { enabled: true, path: '/config/cache.db' },
|
||||
clash_api: { external_controller: '0.0.0.0:19091' },
|
||||
},
|
||||
inbounds: [
|
||||
{ type: 'mixed', tag: 'mixed-in', listen: '0.0.0.0', listen_port: 18081 },
|
||||
{ type: 'mixed', tag: 'diagnostics-primary-in', listen: '0.0.0.0', listen_port: 18082 },
|
||||
{ type: 'mixed', tag: 'diagnostics-reserve-in', listen: '0.0.0.0', listen_port: 18083 },
|
||||
],
|
||||
outbounds: [
|
||||
{ type: 'direct', tag: 'channel-primary' },
|
||||
{ type: 'direct', tag: 'channel-reserve' },
|
||||
{
|
||||
type: 'selector',
|
||||
tag: 'channel-selector',
|
||||
outbounds: ['channel-primary', 'channel-reserve'],
|
||||
default: 'channel-primary',
|
||||
interrupt_exist_connections: false,
|
||||
},
|
||||
],
|
||||
route: {
|
||||
rules: [
|
||||
{ inbound: ['diagnostics-primary-in'], outbound: 'channel-primary' },
|
||||
{ inbound: ['diagnostics-reserve-in'], outbound: 'channel-reserve' },
|
||||
{ inbound: ['mixed-in'], outbound: 'channel-selector' },
|
||||
],
|
||||
final: 'channel-selector',
|
||||
},
|
||||
};
|
||||
fs.writeFileSync(configPath, JSON.stringify(config));
|
||||
|
||||
const mount = `${fixtureDir}:/config`;
|
||||
const check = spawnSync('docker', [
|
||||
'run', '--rm', '-v', mount, '--entrypoint', 'sing-box', image,
|
||||
'check', '-c', '/config/config.json',
|
||||
], { encoding: 'utf8' });
|
||||
assert.equal(check.status, 0, check.stderr || check.stdout);
|
||||
t.diagnostic('config accepted');
|
||||
|
||||
const containerName = `harbor-selector-${process.pid}-${Date.now()}`;
|
||||
const udpRelayContainerPorts = Array.from({ length: 32 }, (_, index) => 18084 + index);
|
||||
const runtime = spawn('docker', [
|
||||
'run', '--rm', '--name', containerName,
|
||||
'--sysctl', `net.ipv4.ip_local_port_range=${udpRelayContainerPorts[0]} ${udpRelayContainerPorts.at(-1)}`,
|
||||
'-p', `${proxyPort}:18081/tcp`,
|
||||
...udpRelayContainerPorts.flatMap((port) => ['-p', `127.0.0.1::${port}/udp`]),
|
||||
'-p', `${apiPort}:19091`,
|
||||
'-v', mount, '--entrypoint', 'sing-box', image,
|
||||
'run', '-c', '/config/config.json',
|
||||
], { stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
let runtimeOutput = '';
|
||||
runtime.stdout.on('data', (chunk) => { runtimeOutput += chunk; });
|
||||
runtime.stderr.on('data', (chunk) => { runtimeOutput += chunk; });
|
||||
t.after(async () => {
|
||||
if (runtime.exitCode == null) {
|
||||
runtime.kill('SIGTERM');
|
||||
await new Promise((resolve) => runtime.once('close', resolve));
|
||||
}
|
||||
echoServer.close();
|
||||
udpEchoServer.close();
|
||||
fs.rmSync(fixtureDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
try {
|
||||
await waitForApi(apiPort);
|
||||
const runtimePid = runtime.pid;
|
||||
t.diagnostic('api ready');
|
||||
const resolveUdpRelayPort = (containerPort) => {
|
||||
const mapping = spawnSync('docker', ['port', containerName, `${containerPort}/udp`], { encoding: 'utf8' });
|
||||
assert.equal(mapping.status, 0, mapping.stderr || mapping.stdout);
|
||||
const port = Number(mapping.stdout.trim().split(':').at(-1));
|
||||
assert.ok(Number.isSafeInteger(port) && port > 0, mapping.stdout);
|
||||
return port;
|
||||
};
|
||||
const before = await (await fetch(`http://127.0.0.1:${apiPort}/proxies/channel-selector`)).json();
|
||||
assert.equal(before.now, 'channel-primary');
|
||||
|
||||
const existing = await openSocksConnection(proxyPort, echoPort, (message) => t.diagnostic(message));
|
||||
t.diagnostic('primary connection open');
|
||||
t.after(() => existing.destroy());
|
||||
await echo(existing, 'before-switch');
|
||||
const existingUdp = await openSocksUdpAssociation(proxyPort, udpEchoPort, resolveUdpRelayPort);
|
||||
t.after(() => existingUdp.close());
|
||||
await existingUdp.echo('before-switch-udp');
|
||||
|
||||
const policy = normalizeFailoverPolicy({
|
||||
...DEFAULT_FAILOVER_POLICY,
|
||||
enabled: true,
|
||||
primary: { profileId: 'primary-profile', serverId: 'primary-server' },
|
||||
reserve: { profileId: 'reserve-profile', serverId: 'reserve-server' },
|
||||
intervalMs: 15_000,
|
||||
failureWindowMs: 30_000,
|
||||
recoveryWindowMs: 60_000,
|
||||
trafficGuard: { enabled: true, thresholdBytesPerSecond: 1024, quietWindowMs: 5_000 },
|
||||
minimumReserveMs: 60_000,
|
||||
});
|
||||
const applied = {
|
||||
primary: policy.primary,
|
||||
reserve: policy.reserve,
|
||||
primaryConfigFingerprint: 'a'.repeat(64),
|
||||
reserveConfigFingerprint: 'b'.repeat(64),
|
||||
};
|
||||
let clock = 0;
|
||||
let state = {
|
||||
revision: 1,
|
||||
failoverPolicy: policy,
|
||||
failoverRuntimeState: { lastSwitchAt: null, holdUntil: null, primaryQuarantineUntil: null, failoverHistory: [], reasonCode: null },
|
||||
appliedFailoverPolicy: applied,
|
||||
appliedProfileId: policy.primary.profileId,
|
||||
appliedServerId: policy.primary.serverId,
|
||||
appliedServerSnapshot: { id: policy.primary.serverId, label: 'Primary' },
|
||||
profiles: [
|
||||
{ id: policy.primary.profileId, servers: [{ id: policy.primary.serverId, label: 'Primary' }] },
|
||||
{ id: policy.reserve.profileId, servers: [{ id: policy.reserve.serverId, label: 'Reserve' }] },
|
||||
],
|
||||
diagnostics: { customServices: [] },
|
||||
};
|
||||
const readSelector = async () => {
|
||||
const value = await (await fetch(`http://127.0.0.1:${apiPort}/proxies/channel-selector`)).json();
|
||||
return { role: value.now === 'channel-primary' ? 'primary' : value.now === 'channel-reserve' ? 'reserve' : 'other' };
|
||||
};
|
||||
const selectRole = async (role) => {
|
||||
const response = await fetch(`http://127.0.0.1:${apiPort}/proxies/channel-selector`, {
|
||||
method: 'PUT',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ name: role === 'primary' ? 'channel-primary' : 'channel-reserve' }),
|
||||
});
|
||||
assert.equal(response.status, 204);
|
||||
const selected = await readSelector();
|
||||
assert.equal(selected.role, role);
|
||||
return selected;
|
||||
};
|
||||
const traffic = createDomainTrafficService({
|
||||
observe: async () => (await fetch(`http://127.0.0.1:${apiPort}/connections`)).json(),
|
||||
devices: () => [],
|
||||
now: () => new Date(clock),
|
||||
});
|
||||
await traffic.refresh();
|
||||
const failover = createFailoverService({
|
||||
state: {
|
||||
read: () => state,
|
||||
update: (mutator) => {
|
||||
state = { ...mutator(state), revision: state.revision + 1 };
|
||||
return state;
|
||||
},
|
||||
},
|
||||
runtime: { isRunning: async () => true },
|
||||
dataplane: {
|
||||
checkConfig: async () => ({}),
|
||||
runFailoverProbe: async (role) => ({ vpn: { sites: [{ status: role === 'primary' ? 'unavailable' : 'available' }] } }),
|
||||
readFailoverSelector: readSelector,
|
||||
selectFailoverRole: selectRole,
|
||||
setFailoverActivityEnabled: async (value) => value ? traffic.enableActivity() : traffic.disableActivity(),
|
||||
readFailoverActivity: async (threshold) => {
|
||||
await traffic.refresh();
|
||||
return { activity: traffic.activitySnapshot(threshold) };
|
||||
},
|
||||
},
|
||||
buildCandidate: () => ({ config: {}, applied }),
|
||||
serialize: (operation) => operation(),
|
||||
scheduler: { setTimeout: () => ({ unref() {} }), clearTimeout: () => {} },
|
||||
now: () => new Date(clock),
|
||||
});
|
||||
t.after(() => failover.shutdown());
|
||||
await failover.reconcile();
|
||||
for (clock of [0, 30_000, 35_000]) {
|
||||
await echo(existing, `active-${clock}-${'x'.repeat(64 * 1024)}`);
|
||||
await failover.runRound();
|
||||
}
|
||||
assert.equal((await readSelector()).role, 'primary');
|
||||
assert.equal(failover.snapshot().status, 'waiting-for-idle');
|
||||
t.diagnostic('active traffic delayed selector switch');
|
||||
|
||||
clock = 46_000;
|
||||
await failover.runRound();
|
||||
assert.equal((await readSelector()).role, 'primary');
|
||||
clock = 51_000;
|
||||
await failover.runRound();
|
||||
const after = await (await fetch(`http://127.0.0.1:${apiPort}/proxies/channel-selector`)).json();
|
||||
assert.equal(after.now, 'channel-reserve');
|
||||
t.diagnostic('quiet window elapsed; selector switched and read back');
|
||||
|
||||
await echo(existing, 'after-switch-existing');
|
||||
await existingUdp.echo('after-switch-existing-udp');
|
||||
const createdAfterSwitch = await openSocksConnection(proxyPort, echoPort, (message) => t.diagnostic(message));
|
||||
t.diagnostic('reserve connection open');
|
||||
t.after(() => createdAfterSwitch.destroy());
|
||||
await echo(createdAfterSwitch, 'after-switch-new');
|
||||
const udpCreatedAfterSwitch = await openSocksUdpAssociation(proxyPort, udpEchoPort, resolveUdpRelayPort);
|
||||
t.after(() => udpCreatedAfterSwitch.close());
|
||||
await udpCreatedAfterSwitch.echo('after-switch-new-udp');
|
||||
|
||||
const connections = await (await fetch(`http://127.0.0.1:${apiPort}/connections`)).json();
|
||||
const chains = connections.connections.map((connection) => connection.chains);
|
||||
assert.ok(chains.some((chain) => chain.includes('channel-primary')), JSON.stringify(chains));
|
||||
assert.ok(chains.some((chain) => chain.includes('channel-reserve')), JSON.stringify(chains));
|
||||
assert.equal(runtime.pid, runtimePid);
|
||||
assert.equal(runtime.exitCode, null);
|
||||
} catch (cause) {
|
||||
assert.fail(`${cause.stack || cause}\n${runtimeOutput}`);
|
||||
}
|
||||
});
|
||||
@@ -320,6 +320,7 @@ setInterval(() => {}, 60_000);
|
||||
'configExists',
|
||||
'connection',
|
||||
'diagnostics',
|
||||
'failover',
|
||||
'fetchedAt',
|
||||
'gatewayAuto',
|
||||
'generatedAt',
|
||||
|
||||
@@ -115,6 +115,39 @@ test('schema v5 migrates rules and diagnostics settings with an exact backup', (
|
||||
assert.equal(fs.readFileSync(store.migration.backupPath, 'utf8'), bytes);
|
||||
});
|
||||
|
||||
test('schema v7 migrates failover disabled without losing canonical state', (t) => {
|
||||
const filePath = fixture(t);
|
||||
const legacy = {
|
||||
schemaVersion: 7,
|
||||
revision: 19,
|
||||
profiles: [],
|
||||
desiredProfileId: '',
|
||||
appliedProfileId: '',
|
||||
appliedServerId: '',
|
||||
appliedServerSnapshot: null,
|
||||
routeRules: [],
|
||||
appliedRouteRules: [],
|
||||
routeRulesRevision: 4,
|
||||
diagnostics: { configured: true, customServices: [], hiddenServiceIds: ['google'] },
|
||||
};
|
||||
fs.writeFileSync(filePath, JSON.stringify(legacy));
|
||||
|
||||
const store = createStateStore(filePath, {
|
||||
now: () => new Date('2026-08-19T12:00:00.000Z'),
|
||||
});
|
||||
const migrated = store.read();
|
||||
|
||||
assert.equal(migrated.schemaVersion, 8);
|
||||
assert.equal(migrated.revision, 19);
|
||||
assert.equal(migrated.routeRulesRevision, 4);
|
||||
assert.equal(migrated.failoverPolicy.enabled, false);
|
||||
assert.equal(migrated.failoverRuntimeState.lastSwitchAt, null);
|
||||
assert.equal(migrated.appliedFailoverPolicy, null);
|
||||
assert.deepEqual(migrated.diagnostics.hiddenServiceIds, ['google']);
|
||||
assert.equal(store.migration.fromVersion, 7);
|
||||
assert.deepEqual(JSON.parse(fs.readFileSync(store.migration.backupPath, 'utf8')), legacy);
|
||||
});
|
||||
|
||||
test('schema v6 rejects missing or unknown outbound without rewriting source bytes', (t) => {
|
||||
for (const [name, rule] of [
|
||||
['missing', { type: 'domain', value: 'example.com', enabled: true }],
|
||||
|
||||
@@ -124,6 +124,8 @@ function createHarness(overrides = {}) {
|
||||
clearInterval: (timer) => { timer.clearCalls += 1; },
|
||||
},
|
||||
onRefreshError: overrides.onRefreshError || (() => {}),
|
||||
onEvent: overrides.onEvent,
|
||||
failover: overrides.failover,
|
||||
now: () => new Date('2026-08-08T12:30:00.000Z'),
|
||||
});
|
||||
|
||||
@@ -305,6 +307,36 @@ test('inactive delete is state-only; applied delete requires one stop-and-delete
|
||||
assert.equal(after.config, null);
|
||||
});
|
||||
|
||||
test('failed active delete restores the previous reserve selector', async () => {
|
||||
const state = defaultState();
|
||||
state.failoverPolicy = {
|
||||
enabled: true,
|
||||
paused: false,
|
||||
primary: { profileId: 'work', serverId: 'shared' },
|
||||
reserve: { profileId: 'personal', serverId: 'shared' },
|
||||
};
|
||||
state.appliedFailoverPolicy = {
|
||||
primary: state.failoverPolicy.primary,
|
||||
reserve: state.failoverPolicy.reserve,
|
||||
};
|
||||
const harness = createHarness({
|
||||
state,
|
||||
failStateAt: 1,
|
||||
failover: {
|
||||
reconcile: async () => {},
|
||||
restoreAppliedActivation: async (previousState) => {
|
||||
assert.equal(previousState.appliedProfileId, 'personal');
|
||||
harness.calls.push('selector.reserve');
|
||||
},
|
||||
},
|
||||
});
|
||||
await assert.rejects(harness.service.deleteProfile('personal', 'stop-and-delete'), /state failed/);
|
||||
assert.equal(harness.snapshot().running, true);
|
||||
assert.equal(harness.snapshot().config, 'old-config');
|
||||
assert.equal(harness.snapshot().state.appliedProfileId, 'personal');
|
||||
assert.ok(harness.calls.indexOf('selector.reserve') > harness.calls.indexOf('runtime.start'));
|
||||
});
|
||||
|
||||
test('deleting a pending desired profile leaves the running applied route untouched', async () => {
|
||||
const state = defaultState();
|
||||
state.desiredProfileId = 'work';
|
||||
@@ -317,6 +349,90 @@ test('deleting a pending desired profile leaves the running applied route untouc
|
||||
assert.equal(harness.calls.includes('gateway.set'), false);
|
||||
});
|
||||
|
||||
test('deleting a desired failover target disables only the pending policy', async () => {
|
||||
const state = defaultState();
|
||||
state.failoverPolicy = {
|
||||
enabled: true,
|
||||
paused: false,
|
||||
primary: { profileId: 'work', serverId: 'shared' },
|
||||
reserve: { profileId: 'personal', serverId: 'shared' },
|
||||
};
|
||||
const harness = createHarness({ state });
|
||||
await harness.service.deleteProfile('work', 'delete', 3);
|
||||
const after = harness.snapshot();
|
||||
assert.equal(after.running, true);
|
||||
assert.equal(after.config, 'old-config');
|
||||
assert.equal(after.state.failoverPolicy.enabled, false);
|
||||
assert.deepEqual(after.state.failoverPolicy.primary, { profileId: '', serverId: '' });
|
||||
assert.deepEqual(after.state.failoverPolicy.reserve, { profileId: 'personal', serverId: 'shared' });
|
||||
});
|
||||
|
||||
test('refresh pauses failover when a loaded channel target disappears', async () => {
|
||||
const state = defaultState();
|
||||
state.failoverPolicy = {
|
||||
enabled: true,
|
||||
paused: false,
|
||||
primary: { profileId: 'personal', serverId: 'shared' },
|
||||
reserve: { profileId: 'work', serverId: 'shared' },
|
||||
};
|
||||
state.appliedFailoverPolicy = {
|
||||
primary: state.failoverPolicy.primary,
|
||||
reserve: state.failoverPolicy.reserve,
|
||||
primaryConfigFingerprint: 'a'.repeat(64),
|
||||
reserveConfigFingerprint: 'b'.repeat(64),
|
||||
};
|
||||
const events = [];
|
||||
const harness = createHarness({ state, onEvent: (event) => events.push(event) });
|
||||
await harness.service.refreshProfile('personal');
|
||||
assert.equal(harness.snapshot().state.failoverPolicy.paused, true);
|
||||
assert.equal(harness.snapshot().running, true);
|
||||
assert.equal(harness.snapshot().config, 'old-config');
|
||||
assert.ok(events.some(({ type }) => type === 'failover.paused'));
|
||||
});
|
||||
|
||||
test('scheduled refresh failures use one stable key for the same failure streak', async () => {
|
||||
const failure = Object.assign(new Error('down'), { code: 'PROVIDER_UNAVAILABLE' });
|
||||
const events = [];
|
||||
const harness = createHarness({
|
||||
fetchSubscription: async () => { throw failure; },
|
||||
onEvent: (event) => events.push(event),
|
||||
});
|
||||
await assert.rejects(harness.service.refreshProfile('personal', undefined, 'scheduled'));
|
||||
await assert.rejects(harness.service.refreshProfile('personal', undefined, 'scheduled'));
|
||||
const keys = events.filter(({ type }) => type === 'subscription.refresh_failed').map(({ dedupeKey }) => dedupeKey);
|
||||
assert.equal(keys.length, 2);
|
||||
assert.equal(keys[0], keys[1]);
|
||||
});
|
||||
|
||||
test('scheduled success logs only content changes or recovery while manual refresh always logs', async () => {
|
||||
const unchanged = (id) => ({
|
||||
config: { profile: id },
|
||||
servers: [oldServer],
|
||||
userInfo: { total: 100 },
|
||||
fetchedAt: '2026-08-08T12:00:00.000Z',
|
||||
});
|
||||
const events = [];
|
||||
const harness = createHarness({
|
||||
fetchSubscription: async () => unchanged('work'),
|
||||
onEvent: (event) => events.push(event),
|
||||
});
|
||||
await harness.service.refreshProfile('work', undefined, 'scheduled');
|
||||
assert.equal(events.length, 0);
|
||||
await harness.service.refreshProfile('work');
|
||||
assert.deepEqual(events.map(({ type }) => type), ['subscription.refreshed']);
|
||||
|
||||
const recoveryState = defaultState();
|
||||
recoveryState.profiles[1].lastRefreshErrorCode = 'PROVIDER_UNAVAILABLE';
|
||||
const recoveryEvents = [];
|
||||
const recovery = createHarness({
|
||||
state: recoveryState,
|
||||
fetchSubscription: async () => unchanged('work'),
|
||||
onEvent: (event) => recoveryEvents.push(event),
|
||||
});
|
||||
await recovery.service.refreshProfile('work', undefined, 'scheduled');
|
||||
assert.deepEqual(recoveryEvents.map(({ type }) => type), ['subscription.refreshed']);
|
||||
});
|
||||
|
||||
test('auto refresh iterates profiles once, reports scoped failures, and stops idempotently', async () => {
|
||||
const errors = [];
|
||||
const failure = Object.assign(new Error('down'), { code: 'PROVIDER_UNAVAILABLE' });
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import test from 'node:test';
|
||||
|
||||
const root = path.resolve(import.meta.dirname, '../..');
|
||||
const page = fs.readFileSync(path.join(root, 'src/web/components/ClientOverviewPage.tsx'), 'utf8');
|
||||
const feature = fs.readFileSync(path.join(root, 'src/web/features/activity-journal/ActivityJournalFeature.tsx'), 'utf8');
|
||||
const styles = fs.readFileSync(path.join(root, 'src/web/styles/features/activity-journal.css'), 'utf8');
|
||||
|
||||
test('journal is the last Gateway-only drawer and refreshes whenever it opens', () => {
|
||||
assert.match(page, /<RoutingToggle[\s\S]*\{isGateway && <ActivityJournalToggle/);
|
||||
assert.match(page, /DRAWER_ORDER = \[[^\]]*'journal'\]/);
|
||||
assert.match(feature, /if \(feature\.isOpen\) void load\(null, status !== 'idle'\)/);
|
||||
assert.match(feature, /assertActivityJournalPage\(await loadPage\(cursor\)\)/);
|
||||
});
|
||||
|
||||
test('journal presents 30-day grouped history with refresh, pagination and stable states', () => {
|
||||
assert.match(feature, /Важные события хранятся 30 дней/);
|
||||
assert.match(feature, /Сегодня[\s\S]*Вчера/);
|
||||
assert.match(feature, /Обновить журнал/);
|
||||
assert.match(feature, /M5 8V4m0 4h4/);
|
||||
assert.match(feature, /Показать ещё/);
|
||||
assert.match(feature, /Журнал временно недоступен[\s\S]*Повторить/);
|
||||
assert.match(styles, /\.client-journal-skeleton/);
|
||||
assert.match(styles, /@media \(prefers-reduced-motion: reduce\)/);
|
||||
});
|
||||
@@ -34,7 +34,7 @@ test('connection feature preserves actions, local preference and opaque neighbor
|
||||
assert.match(panel, /client-state-detail[\s\S]*\{routingSlot\}[\s\S]*client-state-copy[\s\S]*\{serverSlot\}[\s\S]*client-proxies[\s\S]*\{statusSlot\}/);
|
||||
assert.match(page, /routingSlot=\{<RoutingPendingStatus[\s\S]*blocked=\{connectionBlocked\}[\s\S]*onRestart=\{onRestart\}/);
|
||||
assert.match(routing, /client-route-rules-pending[\s\S]*Перезапустить VPN/);
|
||||
assert.match(page, /serverSlot=\{<AppliedIdentity identity=\{mainIdentity\} operation=\{switchIdentity\} \/>\}/);
|
||||
assert.match(page, /serverSlot=\{<AppliedIdentity identity=\{mainIdentity\} operation=\{switchIdentity \|\| failoverIdentity\} \/>\}/);
|
||||
assert.match(page, /mainIdentity[\s\S]*appliedProfile[\s\S]*appliedServer/);
|
||||
assert.match(page, /statusSlot=\{<>[\s\S]*InlineError[\s\S]*InlineProgress/);
|
||||
});
|
||||
|
||||
@@ -233,7 +233,7 @@ test('Gateway Home reuses the canonical device snapshot for applied route and gl
|
||||
assert.match(overview, /const appliedServer = appliedProfile\?\.servers\.find[\s\S]*state\.selection\.appliedServerSnapshot/);
|
||||
assert.match(overview, /const mainIdentity = gatewayDirect[\s\S]*: connected[\s\S]*appliedProfile && appliedServer[\s\S]*`\$\{subscriptionDomain\(appliedProfile\.subscription\.host\)\} · \$\{appliedServer\.label\}`/);
|
||||
assert.match(overview, /const switchIdentity = gatewayDirect[\s\S]*switchingServer && operationProfile && operationServer[\s\S]*`Переключаем на \$\{subscriptionDomain\(operationProfile\.subscription\.host\)\} · \$\{operationServer\.label\}`/);
|
||||
assert.match(overview, /serverSlot=\{<AppliedIdentity identity=\{mainIdentity\} operation=\{switchIdentity\} \/>\}/);
|
||||
assert.match(overview, /serverSlot=\{<AppliedIdentity identity=\{mainIdentity\} operation=\{switchIdentity \|\| failoverIdentity\} \/>\}/);
|
||||
assert.match(feature, /formatByteString\(globalTraffic\?\.totalBytes \|\| '0'\)/);
|
||||
assert.match(feature, /const history = globalTraffic\?\.history \|\| \[\][\s\S]*samples=\{history\}[\s\S]*routeLabel="Gateway"[\s\S]*series="speed"/);
|
||||
assert.match(connection, /<section className=\{`client-power-section[\s\S]*client-state-detail[\s\S]*client-connection-title[\s\S]*\{serverSlot\}[\s\S]*client-proxies/);
|
||||
|
||||
@@ -10,6 +10,7 @@ const page = fs.readFileSync(path.join(root, 'src/web/components/ClientOverviewP
|
||||
const feature = fs.readFileSync(path.join(root, 'src/web/features/diagnostics/DiagnosticsFeature.tsx'), 'utf8');
|
||||
const panel = fs.readFileSync(path.join(root, 'src/web/features/diagnostics/ConnectivityDiagnosticsPanel.tsx'), 'utf8');
|
||||
const model = fs.readFileSync(path.join(root, 'src/web/features/diagnostics/connectivityResult.ts'), 'utf8');
|
||||
const customServiceAction = fs.readFileSync(path.join(root, 'src/web/features/diagnostics/customServiceAction.ts'), 'utf8');
|
||||
const boundary = fs.readFileSync(path.join(root, 'src/web/features/diagnostics/index.ts'), 'utf8');
|
||||
|
||||
const ip = { source: 'cloudflare', address: '198.51.100.10', extra: true };
|
||||
@@ -82,7 +83,7 @@ test('unknown target and legacy-full results pass one identity-preserving parser
|
||||
assert.doesNotMatch(model, /\sas\s(?:ConnectivityResult|Record<string, unknown>)/);
|
||||
});
|
||||
|
||||
test('serial probes and editor stay panel-owned while backend state owns the service set', () => {
|
||||
test('serial probes stay panel-owned while custom service validation is shared by both editors', () => {
|
||||
assert.match(panel, /const targets = onlyTarget \? \[onlyTarget\] : \[[\s\S]*CONNECTIVITY_NETWORK_SOURCE\.id[\s\S]*CONNECTIVITY_IP_SOURCES\.map[\s\S]*sites\.map/);
|
||||
assert.match(panel, /for \(const target of targets\) \{[\s\S]*setActiveTarget\(target\)[\s\S]*await runConnectivityDiagnostics\(target\)[\s\S]*if \(legacyFullResult\) break/);
|
||||
assert.match(panel, /CUSTOM_SERVICES_KEY = 'harbor-diagnostic-services'/);
|
||||
@@ -92,7 +93,8 @@ test('serial probes and editor stay panel-owned while backend state owns the ser
|
||||
assert.doesNotMatch(panel, /localStorage\.setItem/);
|
||||
assert.doesNotMatch(panel, /useState\(read(?:Custom|Hidden)Services\)/);
|
||||
assert.match(panel, /slice\(0, MAX_CUSTOM_DIAGNOSTIC_SERVICES\)/);
|
||||
assert.match(panel, /parsed\.protocol !== 'https:'/);
|
||||
assert.match(panel, /saveCustomDiagnosticService/);
|
||||
assert.match(customServiceAction, /parsed\.protocol !== 'https:'/);
|
||||
assert.match(panel, /document\.startViewTransition\(update\)/);
|
||||
assert.match(panel, /retryable: Boolean\(Reflect\.get\(value, 'retryable'\)\)/);
|
||||
assert.match(panel, /requestDetails\(error\)[\s\S]*requestError\.retryable/);
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import test from 'node:test';
|
||||
|
||||
const root = path.resolve(import.meta.dirname, '../..');
|
||||
const page = fs.readFileSync(path.join(root, 'src/web/components/ClientOverviewPage.tsx'), 'utf8');
|
||||
const feature = fs.readFileSync(path.join(root, 'src/web/features/failover/FailoverFeature.tsx'), 'utf8');
|
||||
const styles = fs.readFileSync(path.join(root, 'src/web/styles/features/failover.css'), 'utf8');
|
||||
|
||||
test('reserve is a Gateway-only drawer immediately after subscriptions', () => {
|
||||
assert.match(page, /<SubscriptionToggle[\s\S]*\{isGateway && <FailoverToggle[\s\S]*<InstructionsToggle/);
|
||||
assert.match(page, /\{isGateway && hasSubscription && <FailoverPanel/);
|
||||
assert.match(page, /DRAWER_ORDER = \['subscription', 'failover'/);
|
||||
assert.match(feature, /label="Резерв"/);
|
||||
assert.match(feature, /Уже открытые соединения Harbor не закрывает/);
|
||||
});
|
||||
|
||||
test('failover settings expose an off switch, channel targets, checks and per-service timeouts', () => {
|
||||
assert.match(feature, /Использовать резерв[\s\S]*Полностью пассивен/);
|
||||
assert.match(feature, /\['primary', 'reserve'\][\s\S]*Подписка[\s\S]*Сервер/);
|
||||
assert.match(feature, /Что проверять/);
|
||||
assert.match(feature, /min="2"[\s\S]*max="30"[\s\S]*Таймаут проверки:/);
|
||||
assert.match(feature, /Проверять каждые, сек[\s\S]*Сбой должен длиться, сек[\s\S]*Восстановление, сек/);
|
||||
assert.match(feature, /Не переключать во время работы[\s\S]*Тишина перед переключением[\s\S]*Активный трафик, КБ\/с/);
|
||||
assert.match(feature, /Защита от повторных сбоев[\s\S]*Окно повторных сбоев[\s\S]*Карантин основного/);
|
||||
assert.match(feature, /Добавить HTTPS-сервис/);
|
||||
assert.match(feature, /Проверить оба канала/);
|
||||
assert.match(feature, /beforeunload/);
|
||||
});
|
||||
|
||||
test('runtime status keeps fixed geometry and explains active traffic without motion dependence', () => {
|
||||
assert.match(feature, /Ждём завершения активной работы/);
|
||||
assert.match(feature, /Активный трафик ·[\s\S]*transmittingConnections/);
|
||||
assert.match(feature, /activity\?\.blockers\.slice\(0, 2\)\.map/);
|
||||
assert.match(feature, /Ещё \{activity\.blockers\.length - 2\}/);
|
||||
assert.match(feature, /Нестабилен ·[\s\S]*Следующее решение не раньше чем через/);
|
||||
assert.match(feature, /switchRole && snapshot\.enabled && snapshot\.activation === 'active'/);
|
||||
assert.match(styles, /\.client-failover-runtime \{[^}]*min-height:\s*132px/);
|
||||
assert.match(styles, /@media \(prefers-reduced-motion: reduce\)[\s\S]*transition:\s*none/);
|
||||
});
|
||||
@@ -56,6 +56,10 @@ test('typed Harbor client validates unknown state and isolates wire compatibilit
|
||||
gatewayAvailable: true,
|
||||
});
|
||||
assert.equal(Object.hasOwn(parsed, 'proxyPort'), false);
|
||||
assert.throws(() => parseHarborState({
|
||||
...snapshot,
|
||||
failover: { ...snapshot.failover, policy: { ...snapshot.failover.policy, intervalMs: 'fast' } },
|
||||
}), { code: 'INCOMPATIBLE_API' });
|
||||
let incompatible;
|
||||
try {
|
||||
parseHarborState({ ...snapshot, revision: -1 });
|
||||
@@ -116,6 +120,21 @@ test('an equal revision keeps the current snapshot identity', () => {
|
||||
assert.equal(next.snapshot.selection.desiredServerId, 'one');
|
||||
});
|
||||
|
||||
test('equal revision accepts only newer failover observations and retires old epochs', () => {
|
||||
const value = (epoch, sequence, status) => ({
|
||||
...snapshot(4, 'one'),
|
||||
failover: { observationEpoch: epoch, observationSequence: sequence, status },
|
||||
});
|
||||
let state = receive(initialHarborState, value('epoch-a', 1, 'primary'));
|
||||
state = receive(state, value('epoch-a', 2, 'waiting-for-idle'));
|
||||
assert.equal(state.snapshot.failover.status, 'waiting-for-idle');
|
||||
state = receive(state, value('epoch-b', 1, 'reserve'));
|
||||
assert.equal(state.snapshot.failover.status, 'reserve');
|
||||
state = receive(state, value('epoch-a', 99, 'primary'));
|
||||
assert.equal(state.snapshot.failover.status, 'reserve');
|
||||
assert.deepEqual(state.failoverTransport.retiredEpochs, ['epoch-a']);
|
||||
});
|
||||
|
||||
test('selection has no client-side shadow and follows canonical profile snapshots', () => {
|
||||
const state = receive(initialHarborState, snapshot(2, 'two'));
|
||||
assert.equal(Object.hasOwn(state, 'pendingServerId'), false);
|
||||
|
||||
@@ -166,13 +166,13 @@ test('secondary menus share one right rail and switch equal drawers as a vertica
|
||||
assert.match(instructions, /<Drawer[\s\S]*className="client-instructions"/);
|
||||
assert.match(routing, /<Drawer[\s\S]*className="client-local-rules"/);
|
||||
assert.match(subscription, /<Drawer[\s\S]*className="client-subscription-drawer"/);
|
||||
assert.match(component, /const DRAWER_ORDER = \['subscription', 'instructions', 'devices', 'diagnostics', 'routing'\]/);
|
||||
assert.match(component, /const DRAWER_ORDER = \['subscription', 'failover', 'instructions', 'devices', 'diagnostics', 'routing', 'journal'\]/);
|
||||
assert.match(component, /function switchDrawer\(target: DrawerKey\)[\s\S]*from\.inert = true[\s\S]*translateY\(\$\{direction \* 100\}%\)[\s\S]*translateY\(\$\{-direction \* 100\}%\)/);
|
||||
assert.match(component, /const \[drawerSwitchTarget, setDrawerSwitchTarget\] = useState<DrawerKey \| null>\(null\)/);
|
||||
assert.match(component, /const activeRailDrawer = drawerSwitchTarget && drawerControls\[drawerSwitchTarget\]\.isOpen[\s\S]*drawerControls\[drawer\]\.isOpen/);
|
||||
assert.match(component, /setDrawerSwitchTarget\(target\);[\s\S]*flushSync\(\(\) => toControl\.show\(\)\)/);
|
||||
assert.match(component, /fromControl\.close\(\);[\s\S]*setDrawerSwitchTarget\(null\)/);
|
||||
assert.match(component, /<SubscriptionToggle[\s\S]*open=\{activeRailDrawer === 'subscription'\}[\s\S]*<RoutingToggle[\s\S]*open=\{activeRailDrawer === 'routing'\}/);
|
||||
assert.match(component, /<SubscriptionToggle[\s\S]*open=\{activeRailDrawer === 'subscription'\}[\s\S]*<FailoverToggle[\s\S]*<RoutingToggle[\s\S]*open=\{activeRailDrawer === 'routing'\}[\s\S]*<ActivityJournalToggle/);
|
||||
assert.match(component, /const cancel = \(\) => \{[\s\S]*from\.inert = false;[\s\S]*from\.removeAttribute\('aria-hidden'\)/);
|
||||
assert.match(component, /DRAWER_SWITCH_MS = 620[\s\S]*prefers-reduced-motion: reduce[\s\S]*flushSync/);
|
||||
assert.match(component, /current === 'routing' && routingFeature\.dirty[\s\S]*routingFeature\.requestClose\(\)/);
|
||||
|
||||
@@ -31,36 +31,38 @@ const expectedImports = [
|
||||
'./features/servers.css',
|
||||
'./primitives.css',
|
||||
'./features/diagnostics.css',
|
||||
'./features/failover.css',
|
||||
'./features/activity-journal.css',
|
||||
'./layout.css',
|
||||
'./themes.css',
|
||||
];
|
||||
const sha256 = (value) => crypto.createHash('sha256').update(value).digest('hex');
|
||||
const acceptedLedger = {
|
||||
counts: {
|
||||
cascadeEdges: 954,
|
||||
cascadeEdges: 984,
|
||||
customProperties: 106,
|
||||
declarations: 3487,
|
||||
declarations: 3805,
|
||||
important: 0,
|
||||
keyframes: 48,
|
||||
media: 13,
|
||||
rules: 971,
|
||||
variableReferences: 837,
|
||||
media: 17,
|
||||
rules: 1045,
|
||||
variableReferences: 977,
|
||||
},
|
||||
hashes: {
|
||||
cascadeEdges: 'd53f6a2236717d6bc27d486fc19f0f80a33169fa7f68a6df2a19536fd30be3bb',
|
||||
cascadeEdges: '423d29f846b45c3522fdc101b5aa1399264f2abe5953b9687b380d5fb865bd07',
|
||||
customProperties: 'fd6f16069f9fe3526f09d4d574431ddae67150d8e7a8a40e04c9fd2d179ec7ac',
|
||||
declarations: 'e1401e80ed2ebc5d3601d4d61a3b44c2adbf91db451907ef95a0998170ac5de5',
|
||||
declarations: '52b5bcd795dbea19950afee0259c62cb0fb1aaca094d553cfaaae046a5f747f9',
|
||||
duplicateKeyframes: '4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945',
|
||||
duplicateSelectors: '8982145dba05b33bf1c93b2d54cfbe76305b276ff3c657aa020144016ada5848',
|
||||
keyframes: 'a0d69c5b3e5f235d3a76ed04bdd64f67a73fafc9c59fe2635ffda603709c0ad9',
|
||||
ruleDeclarationSequences: '89f5f3b302f3a10df0ba90dfbbc63a7ccbc1d8a0c784be26a475321f44dfa8e8',
|
||||
selectors: 'dc1a2687ec872d1922a39e8258977949a9b775168764a59119ae6ea93a9f88a1',
|
||||
variableReferences: '843a5f71b0fb74691ed623dbc7ebb7ddddc34e5736e1f419f17cf54e22d43073',
|
||||
witnesses: '81676a3fdb6f3bbd63e86b275b6cb6d40e930651b9d5cbde208408f3163ab2b4',
|
||||
ruleDeclarationSequences: 'bd8d10966c15df58eb76d67fc5a21acaeec524a64805797d5fb33af3b942292e',
|
||||
selectors: '76d2d373aa9ecfd5cac3f95a47ed0fa2c44a659ac98f02f6bfd2f4505de23226',
|
||||
variableReferences: 'bea2b082cf2366041936c2da22ac3e0ab83b3e5cead1326a0f65a015daedbb3b',
|
||||
witnesses: '605555d8bae9b8d64b1c960c398b14cf910e624d547b72e4fcfced85eec30991',
|
||||
},
|
||||
};
|
||||
|
||||
test('public stylesheet exposes exactly twelve flat semantic owners', () => {
|
||||
test('public stylesheet exposes exactly fourteen flat semantic owners', () => {
|
||||
const imports = Array.from(index.matchAll(/^@import ['"](\.\/[^'"]+\.css)['"];$/gm), ([, file]) => file);
|
||||
assert.deepEqual(imports, expectedImports);
|
||||
assert.equal(index, `${expectedImports.map((file) => `@import '${file}';`).join('\n')}\n`);
|
||||
@@ -209,7 +211,7 @@ test('client typography uses the shared semantic scale outside the token owner',
|
||||
|
||||
test('accepted stylesheet has pinned declaration, selector, keyframe, variable, and cascade ledgers', () => {
|
||||
const witnesses = readStyleWitnesses(root);
|
||||
assert.equal(witnesses.length, 884);
|
||||
assert.equal(witnesses.length, 1021);
|
||||
assert.equal(witnesses.filter((witness) => witness.unknown || witness.ancestorUnknown).length, 0);
|
||||
const ledger = createStyleLedger(readStyleSource(root), { witnesses });
|
||||
assert.deepEqual(ledger.counts, acceptedLedger.counts);
|
||||
@@ -405,8 +407,8 @@ test('main owns one public stylesheet and the regrouped production CSS is determ
|
||||
assert.equal((main.match(/import ['"][^'"]+\.css['"]/g) || []).length, 1);
|
||||
|
||||
const assets = fs.readdirSync(path.join(root, 'dist/assets')).filter((file) => file.endsWith('.css'));
|
||||
assert.deepEqual(assets, ['index-CdbGat4L.css']);
|
||||
assert.deepEqual(assets, ['index-DaioW5qf.css']);
|
||||
const built = fs.readFileSync(path.join(root, 'dist/assets', assets[0]));
|
||||
assert.equal(built.byteLength, 133670);
|
||||
assert.equal(sha256(built), '1aaa222692046eb166e5bfb8a195e53cf0ef61692a82ae53ca0119b4c8f878a2');
|
||||
assert.equal(built.byteLength, 144728);
|
||||
assert.equal(sha256(built), '7d175321821b38be6a947a76457e310cf519e2af4b9ee95fbb0097933ca70a28');
|
||||
});
|
||||
|
||||
@@ -34,7 +34,7 @@ test('all repeated client primitive consumers use the shared owners', () => {
|
||||
|
||||
assert.equal((production.match(/<Tooltip\b/g) || []).length, 14);
|
||||
assert.equal((production.match(/<CopyButton\b/g) || []).length, 2);
|
||||
assert.equal((production.match(/<RailAction\b/g) || []).length, 5);
|
||||
assert.equal((production.match(/<Drawer\b/g) || []).length, 5);
|
||||
assert.equal((production.match(/<RailAction\b/g) || []).length, 7);
|
||||
assert.equal((production.match(/<Drawer\b/g) || []).length, 7);
|
||||
assert.doesNotMatch(production, /className="client-tooltip"|className="client-copy-label"|className="client-drawer-close"/);
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user