Filomena
Filomena / Docs

Production operations

Release gates, backups, recovery, monitoring and the canary contract.

Availability contract

Filomena 0.1 runs one active replica. It is outside the alert delivery and remediation path. An outage delays diagnoses but cannot block alerts or mutate production systems. Persisted events recover after restart within the configured recovery window.

The chart rejects replicaCount values other than one. Planned upgrades use Recreate and briefly interrupt the API. The PodDisruptionBudget prevents voluntary eviction while the pod is healthy. Node loss and volume reattachment remain recovery events rather than transparent failovers.

Release gate

A production release requires:

  1. The complete .NET suite, including PostgreSQL and ClickHouse contracts, passes.
  2. The container builds and the NuGet and container vulnerability checks report no known high or critical vulnerabilities.
  3. Helm lint and the kind smoke, load, restart, upgrade and rollback job pass.
  4. The release image is multi-architecture, carries an SBOM and provenance, and is signed with Cosign.
  5. The production values reference the image by digest.
  6. A current database backup has passed a restore drill.
  7. Queue, processing-failure, availability, pipeline-gap and stale-backup alerts are active.
  8. Thirty consecutive canary days complete without lost events, unrecovered retryable work, schema failure, backup failure, or a security regression.

Release tags use v<semver>. MinVer derives the .NET assembly and package versions from that tag. The release workflow rejects a mismatch, passes the resolved version into the container build, and updates the Helm chart version, application version, image tag and image digest after the registry returns the published digest.

Required production values

Use separate ingest, read and admin tokens. Configure provider-native backups and verify restores. For SQLite, enable backups to a separate persistent claim. Enable the ServiceMonitor and PrometheusRule only where their CRDs exist. Configure the NetworkPolicy for the exact Grafana, Prometheus, DNS, Kubernetes API, telemetry provider, Slack and model-provider paths used by the installation.

image:
  repository: ghcr.io/bfxavier/filomena
  digest: sha256:<released digest>

config:
  stateProvider: postgresql
  slackCanaryChannelId: C0123456789

secrets:
  stateConnectionString: Host=postgres.example;Port=5432;Database=filomena;Username=filomena;Password=...

serviceMonitor:
  enabled: true

prometheusRule:
  enabled: true
  grafanaDeliveryParity: true

Keep the API private. Publish only the Grafana webhook and Slack interaction paths when an ingress is required. The metrics endpoint has no application-level authentication and must remain reachable only by the monitoring system.

Management UI access

The management UI is served at /ui/ on the existing ClusterIP service. Do not add it to a public ingress. Reach the service through a private network path such as NetBird and enter the read token when prompted. For the PX deployment the in-cluster URL is:

http://alert-triage.alert-triage-prod.svc.cluster.local:8080/ui/

Generic Helm installations use:

http://<release>-filomena.<namespace>.svc.cluster.local:8080/ui/

If private clients do not resolve Kubernetes service DNS, expose the ClusterIP through the private network's DNS or service-routing layer. Do not work around that by publishing /ui/ externally.

Backup and restore

Use the PostgreSQL service's backup and point-in-time recovery facilities for the recommended production provider.

When SQLite is selected, the application writes online backups to the configured backup directory, validates each with PRAGMA integrity_check, atomically publishes it, and enforces retention. Backup success and failure counters and the last-success timestamp are exported on /metrics.

Restore is an explicit offline operation:

FILOMENA_NAMESPACE=filomena \
FILOMENA_RELEASE=filomena \
FILOMENA_BACKUP_CLAIM=filomena-backups \
FILOMENA_RESTORE_IMAGE=ghcr.io/bfxavier/filomena@sha256:<digest> \
./deploy/restore-sqlite.sh <backup-file>

For ClickHouse state, use the ClickHouse operator's backup and restore mechanism. Apply the versioned DDL under deploy/clickhouse/init before starting an upgraded Filomena image.

Canary evidence

Set FILOMENA_CANARY_ACTIVE=true and provide FILOMENA_CANARY_INGEST_TOKEN to make the scheduled workflow send a unique firing event, wait for its run, send its resolution and fail when Slack delivery failures increase. Configure SlackCanaryChannelId to isolate these messages. Set FILOMENA_CANARY_GRAFANA_URL and FILOMENA_CANARY_GRAFANA_TOKEN to inject the firing and resolved alerts through Grafana Alertmanager instead of posting directly to Filomena. Canary events skip evidence collection and model analysis after the root notification, keeping the test deterministic while exercising ingestion, queueing, persistence, incident lifecycle and Slack.

Record the following each day for 30 days:

  • Accepted, duplicate and deferred webhook events
  • Completed, skipped, duplicate and failed runs
  • Retry exhaustion and oldest pending-event age
  • Queue depth and processing latency
  • State-store readiness
  • Backup success, backup age and restore-drill result
  • Model tokens, model cost and provider errors
  • Human feedback coverage and verdicts
  • Incident resolution time

The canary passes only when all persisted events reach a terminal state within the recovery window and every alert listed in the release gate remains clear. The scheduled workflow keeps each daily JSON report as a 90-day build artifact.