Migrate Harbor state and traffic history to SQLite
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Harbor application state v1
|
||||
|
||||
`GET /api/state` is the canonical Harbor domain snapshot. Successful mutations return the same snapshot as `state`. The persisted owner is `state.json` schema v8; React keeps only drafts, disclosure, focus, animation and transport freshness.
|
||||
`GET /api/state` is the canonical Harbor domain snapshot. Successful mutations return the same snapshot as `state`. The persisted owner is the `state` JSON document in `harbor.sqlite` (currently document schema v10). React keeps only drafts, disclosure, focus, animation and transport freshness.
|
||||
|
||||
An abbreviated snapshot:
|
||||
|
||||
@@ -107,7 +107,7 @@ Enabling failover while VPN is stopped validates a temporary dual-channel candid
|
||||
|
||||
The dual config keeps one stable inbound and a sing-box selector with `interrupt_exist_connections: false`. A switch changes the outbound for new connections only. Before an automatic switch, the existing `/connections` observer measures VPN byte deltas over a bounded 10-second window. Active or unknown traffic blocks the switch; the public snapshot contains only aggregate speed, connection count and at most three safe device/service labels.
|
||||
|
||||
Failover mutations use `PUT /api/failover`, `POST /api/failover/pause` and `POST /api/failover/switch`. Important user events are stored separately in `activity-journal.json` and read through `GET /api/activity-journal`. The journal is not a second state owner, contains no provider URLs or raw diagnostics, uses stable ID cursors and prunes entries after 30 days.
|
||||
Failover mutations use `PUT /api/failover`, `POST /api/failover/pause` and `POST /api/failover/switch`. Important user events are separate rows in the `harbor.sqlite` journal table and read through `GET /api/activity-journal`. The journal is not a second state owner, contains no provider URLs or raw diagnostics, uses stable ID cursors and retains at most 30 days and 10,000 entries.
|
||||
|
||||
## Compatibility and migration
|
||||
|
||||
@@ -115,6 +115,16 @@ Schema v5 migrates the legacy singleton and `subscription-cache.json` into one p
|
||||
|
||||
Schema v6 adds the routing-rule outbound. Rules read from schemas v0-v5 migrate to `outbound: "direct"` in their existing order and both desired/applied arrays are normalized together. A schema-v6 rule without a valid outbound is rejected rather than silently rewritten. Schema v7 adds canonical connectivity-diagnostics settings. Schema v8 adds a disabled failover policy, empty runtime history and no applied dual config, so upgrading does not start monitoring or change traffic.
|
||||
|
||||
Migration atomically backs up the previous `state.json`. After the embedded profile is committed, Harbor also backs up and removes the legacy subscription cache so there is one persisted owner. Invalid legacy cache/config returns to a truthful stopped first-run state instead of starting stale generated config.
|
||||
The initial SQLite migration imports settings, devices and journal in one transaction, including a legacy subscription cache when needed. Original JSON files remain unchanged as transition-time backups, with no parallel writes. Invalid input or a conflicting cache owner aborts migration without replacing state or starting stale configuration. Subsequent starts use only SQLite. Current document normalizers also preserve traffic-display settings and the device inventory's tag catalogue.
|
||||
|
||||
The old HTTP projection remains bounded for one release. Schema v8 persistence is not downgrade-compatible: stop Harbor and restore the `state.json.backup-v<fromVersion>-*` matching the rollback binary instead of deploying old code over v8 data. Rolling back before profiles still also requires the matching legacy subscription-cache backup.
|
||||
The existing HTTP compatibility projection is unchanged. A pre-SQLite binary cannot read current persistence: restore a complete compatible backup or explicitly export current data before downgrading. Old JSON files do not contain post-migration changes. See [state recovery](../recovery/state-recovery.md).
|
||||
|
||||
## Local traffic history
|
||||
|
||||
`GET /api/traffic/history` reads only the local collector's `traffic.sqlite`, through a worker and, on a split Gateway, the existing control/dataplane socket. It accepts `range=24h|7d|30d|90d`, `level=service|domain|hostname|ip`, parent filters `service/domain/hostname`, `originId`, `route=all|vpn|direct|other`, `search`, `offset` and an optional `until` timestamp in milliseconds. Pages contain at most 100 groups. Bytes are decimal strings, preserving integers above JavaScript's safe-number range.
|
||||
|
||||
The response reports the requested/effective period, first available observation, minute/hour boundary, current collector state, gap count and partial coverage. `query.until` is the effective end of a complete bucket; use it for matching drilldown and pagination. Current history can lag by one minute. Data older than 7 days is hourly; retention and rollup can change available granularity between requests.
|
||||
|
||||
History starts with the new collector, not with previously accumulated device counters. Full observed hostnames and IPs remain distinct; service grouping is a local presentation classification, not proof of ownership of an IP. Unknown domains remain unknown. A history storage error reports unavailable/partial data without stopping VPN or exported metrics.
|
||||
|
||||
Prometheus is optional, independent and contains only exported metrics—not a copy of this database. There is no synchronization, automatic UI fallback, scrape backfill or restoration from Prometheus. Local cleanup does not delete external history or change its retention.
|
||||
|
||||
@@ -1,64 +1,39 @@
|
||||
# Harbor state recovery
|
||||
|
||||
Harbor keeps the existing data directory and `state.json` path. The current persisted format is `schemaVersion: 8`: schema v2 introduced local route rules, v3 added rule enabled state, v4 added stable server IDs, v5 embeds the canonical `profiles[]` collection with desired/applied profile identity, v6 adds an explicit `vpn` or `direct` outbound to every route rule, v7 stores connectivity-diagnostics settings, and v8 adds Gateway failover state.
|
||||
## Storage owners
|
||||
|
||||
## Atomic writes
|
||||
Harbor uses the existing data directory. `harbor.sqlite` is the only working owner of settings, profiles, subscriptions, device inventory and accumulated device counters. Settings and devices are versioned JSON documents inside SQLite (currently state schema 10 and inventory schema 3); the journal is a separate indexed table with the existing 30-day/10,000-event limit.
|
||||
|
||||
Persistent files are written to a unique temporary file in the same directory, flushed with `fsync`, closed and atomically renamed over the target. A failure before rename leaves the previous target untouched and removes the temporary file.
|
||||
`traffic.sqlite` is separate, replaceable working history: 90 days, completed minute buckets for the latest 7 days and hourly buckets before that. Removing history does not reset settings or exported counters. Prometheus is optional and independently retains only the metrics it scrapes; it cannot restore this database.
|
||||
|
||||
Profile/server switching prepares candidate config and runtime before the final state publication. If any later step fails, Harbor restores the previous config, runtime and canonical state.
|
||||
Secrets, hardware identity, generated configuration and sing-box's own cache remain files.
|
||||
|
||||
## Migration to profiles
|
||||
## Atomic writes and migration
|
||||
|
||||
On startup, a legacy state is normalized before the process starts. Harbor creates one profile named `Основной`, moves the provider URL/config and metadata into it, and preserves unambiguous desired/applied server identity. A legacy state explicitly marked stopped clears stale applied residue.
|
||||
SQLite uses WAL, FULL synchronous commits and a five-second busy timeout. A document mutation runs its read and write in one transaction. Journal append/deduplication/pruning is transactional. Profile/server switching still prepares candidate configuration and runtime before publishing the canonical state.
|
||||
|
||||
Before replacing state Harbor saves the original beside it:
|
||||
Before the first successful SQLite startup, Harbor imports `state.json`, `devices.json`, `activity-journal.json` and, when required by a pre-profile schema, `subscription-cache.json` in one transaction. Existing normalizers preserve revision, stable IDs, ordered rules and decimal-string counters. A null optional subscription cache is valid.
|
||||
|
||||
```text
|
||||
state.json.backup-v4-2026-08-11T12-00-00-000Z
|
||||
```
|
||||
The import marker commits with all imported records. An unsupported version, invalid input or conflicting cache owner aborts the entire import. Harbor does not erase settings, rename damaged originals, start stale configuration or silently return to first-run. Correct the reported original and retry only after making a backup.
|
||||
|
||||
After a valid profile has been committed, the raw legacy cache is saved and removed as a second owner:
|
||||
After a successful import, the original JSON files remain unchanged under their original names as transition-time backups. They are never read or written as current state again. Changing them does not change Harbor. A corrupt or unsupported SQLite database does not fall back to those stale JSON files.
|
||||
|
||||
```text
|
||||
subscription-cache.json.backup-v1-2026-08-11T12-00-00-000Z
|
||||
```
|
||||
## Backup and recovery
|
||||
|
||||
An invalid legacy provider config is backed up but not started. Harbor removes stale generated config and returns to a stopped first-run state.
|
||||
Stop both control and collector processes before offline recovery. On a Gateway this means the control and dataplane components; stop the Mac backend for Mac recovery.
|
||||
|
||||
## Migration to ordered VPN/Direct rules
|
||||
1. Preserve the whole data directory, including any `-wal` and `-shm` files, before changing anything.
|
||||
2. Restore a matching backup of `harbor.sqlite` and any necessary secret/config files. Do not mix a database with another backup's WAL.
|
||||
3. Start the same compatible release and inspect `GET /api/state` before applying a profile.
|
||||
|
||||
When schemas v0-v5 are read, Harbor preserves the order of `routeRules` and `appliedRouteRules` and adds `outbound: "direct"` to legacy entries before atomically committing schema v6. The original file is preserved using its actual source version, for example:
|
||||
For online backups use SQLite's backup API; copying only a live `.sqlite` file can omit committed WAL data. Offline copies after a clean stop are simpler.
|
||||
|
||||
```text
|
||||
state.json.backup-v5-2026-08-17T12-00-00-000Z
|
||||
```
|
||||
To discard only working traffic history, stop the collector and move its `traffic.sqlite` plus any associated `traffic.sqlite-wal` and `traffic.sqlite-shm` aside together. Leave `harbor.sqlite` untouched. A new collector database starts a new coverage period; there is no automatic Prometheus backfill. Do not unlink an open database.
|
||||
|
||||
After migration, malformed schema-v6 rules are rejected; Harbor does not reinterpret a missing or unknown outbound as direct.
|
||||
Deletion/retention makes pages reusable; it does not necessarily shrink the physical file immediately. Traffic retention and compaction run in the worker, outside connection processing.
|
||||
|
||||
## Migration to failover
|
||||
## Downgrade
|
||||
|
||||
Schemas v0-v7 migrate to v8 with failover disabled, empty switch history and no applied dual config. Migration does not start probes, enable traffic accounting or change the single-channel runtime. The original state is preserved as `state.json.backup-v<fromVersion>-*` before the atomic replacement.
|
||||
A pre-SQLite binary ignores `harbor.sqlite`. Merely starting it would revive old JSON settings and lose all changes since the cutover. Automatic downgrade is unsupported.
|
||||
|
||||
The separate `activity-journal.json` is created on the first important event. It uses the same atomic write and corrupt-file isolation mechanism as state, retains at most 30 days, and can be removed while Harbor is stopped without affecting subscriptions, routing or VPN startup.
|
||||
|
||||
## Corrupt JSON
|
||||
|
||||
If `state.json` cannot be parsed, Harbor renames the exact damaged bytes to:
|
||||
|
||||
```text
|
||||
state.json.corrupt-2026-08-11T12-00-00-000Z
|
||||
```
|
||||
|
||||
It then creates a valid empty current-schema state and reports storage recovery. A corrupt legacy subscription cache is preserved with the same suffix and is never used to start stale config.
|
||||
|
||||
## Manual recovery and downgrade
|
||||
|
||||
Perform recovery while Harbor is stopped:
|
||||
|
||||
1. Copy the whole data directory.
|
||||
2. Inspect the intended backup with `jq . <backup-file>`.
|
||||
3. Restore only matching state/cache backups to their original filenames.
|
||||
4. Start Harbor and verify `GET /api/state` before applying a profile.
|
||||
|
||||
A pre-v8 binary cannot interpret failover state. Restore `state.json.backup-v<fromVersion>-*` matching the rollback binary; deploying old code over schema v8 is not safe. A rollback to pre-v5 additionally requires the matching state and subscription-cache backups because that binary cannot interpret canonical profiles.
|
||||
Either restore a complete pre-upgrade backup deliberately, accepting the loss of subsequent changes, or first export current state into the exact schema required by the older binary. No automatic export/downgrade tool is provided. Preserve the SQLite backup in either case; do not overwrite current state with stale JSON as a recovery shortcut.
|
||||
|
||||
Reference in New Issue
Block a user