Architecture

Filomena is an asynchronous alert diagnosis service. The webhook persists alert lifecycle events before placing them on a bounded in-memory queue. Workers recover persisted, retryable events after restarts.

Processing lifecycle

1. Authenticate and parse the Grafana or Alertmanager webhook.
2. Map organization labels to canonical OpenTelemetry identity and compute the event ID and fingerprint.
3. Insert the lifecycle event into the configured state store.
4. Resolve the canonical service identity and attach the event to an incident.
5. Deduplicate completed runs inside the configured window.
6. Load human-confirmed cases for the same alert.
7. Select matching diagnostic packs.
8. Query logs, metrics, traces, topology, infrastructure, runbooks and changes concurrently with source-specific bounds.
9. Allow at most the configured number of model-planned queries through the versioned read-only connector catalog.
10. Build a valid JSON prompt with separate alert, prior-case and evidence budgets.
11. Request a strict structured diagnosis from the analysis provider.
12. Reject citations to evidence that was not collected.
13. Persist the run and update the incident before notifying Slack or application logs.
14. Retry classified transient failures with exponential backoff.

Service identity and incidents

Canonical service identity uses OpenTelemetry service, deployment and Kubernetes resource attributes with compatibility aliases for existing Grafana labels. Ownership, repository and runbook metadata travel with the identity.

An explicit incident label correlates alerts across services. The fallback correlation key is the service namespace, name, environment and cluster. An incident remains open until the latest lifecycle event for every member is resolved. Kubernetes replica alerts share a workload fingerprint while the pod and container identify the incident member. Diagnosis revisions update the incident without discarding earlier triage runs.

Operational changes are immutable events keyed by a caller-supplied change ID. The change diagnostic pack selects events for the canonical service and alert window. Trace evidence uses Tempo, and topology evidence uses OpenTelemetry service-graph metrics.

Event identity

An event is one alert lifecycle transition: its fingerprint, its status and its start and end time. A repeated firing notification for the same alert start is therefore the same event, is stored once and is diagnosed once. A new start time, or a change to or from resolved, is a new event. A fingerprint is diagnosed once per open lifecycle. Further firing events are suppressed until the incident is fully resolved. A member resolution inside an open incident does not start a new lifecycle.

An attempt that collected no evidence is retried inside the attempt cap, because evidence can arrive later than the alert that reports it. A suppressed repeat and a resolution are terminal and are never retried.

Prior cases

A prior case is a completed run for the same alert that a human rated, newest first, capped at five. A case counts only when its latest feedback teaches something: a correct or partial verdict, or a corrected root cause or resolution. A verdict the reviewer later withdraws stops being a prior case.

Retrieval reads every rated run for that one fingerprint rather than a recent window, so an older confirmed case cannot be hidden by newer runs whose feedback teaches nothing. The number of such runs is bounded by how often a person rates one alert. Every state provider push the fingerprint and the rating filter into a single query and share one selection function, so neither provider carries its own copy of the rule.

Offline evaluation

Export writes stored runs as cases, replay asks a candidate model for a fresh diagnosis of the same evidence, and score reports one summary. Compare scores both and returns a verdict: a metric with a known good direction may not move against it beyond a tolerance, and the candidate must cover every case the baseline did. The exit code carries the verdict, so the comparison can gate a change instead of only describing it.

The repository includes sanitized public cases. CI validates their grounding, read-only actions, human labels and evidence provenance. The same evaluator can compare prompt and model candidates.

Release verification

Release verification compares equalized pre-release and post-release rates for alerts, failed analyses and restarts plus p95 latency. A minimum post-release observation window prevents an early pass. The shared evaluator is exposed through the admin API and Filomena.Eval release-verify, with the verdict recorded in the audit log and optionally sent to Slack.

Safety invariants

• Alert labels, annotations, logs and prior cases are untrusted model input.
• Diagnostic queries apply resource selectors before row limits.
• Each evidence source has a quota so one adapter cannot evict all other findings.
• Every connector is read-only, versioned, allowlisted and limited to a 24-hour query window.
• HTTP connectors use configured base URLs and allowed path prefixes and only issue GET requests.
• Prompt truncation removes whole findings and low-priority fields while preserving valid JSON.
• The response schema restricts evidence citations to IDs collected for the current run.
• Recommended actions are limited to read-only investigation steps.
• Slack request signatures expire after five minutes.
• Notification failures do not change a completed diagnosis into a failed run.
• A completion the analysis provider cannot deliver in full is retried, not treated as a diagnosis. Truncated and malformed responses take the same path as an invalid one.

Extension points

Public domain models and provider contracts live in Filomena.Core. The API composes concrete providers at startup:

• Filomena.Storage.Postgres implements the recommended production state store.
• Filomena.Storage.Sqlite implements the local and lightweight state store.
• Filomena.Storage.ClickHouse implements the legacy ClickHouse state store.
• Filomena.Evidence.ClickHouse implements OpenTelemetry log and trace evidence plus Application Gateway access-log evidence.
• Filomena.Evidence.Loki implements LogQL log and Application Gateway access-log evidence.
• Filomena.Evidence.Prometheus implements instant and range metric evidence.
• Filomena.Evidence.Kubernetes implements pod, event and current or previous container-log evidence.
• Filomena.Evidence.ArgoCd implements application, condition, resource and operation evidence.
• No-op providers disable a capability without disabling unrelated diagnostic packs.

IStateStore covers event deduplication, retry recovery, prior cases, related alerts, notification delivery state, run queries and evaluation export. Its narrower inherited interfaces let consumers request only the operations they use.

Evidence contracts are split by capability: ILogEvidenceProvider, IAccessLogEvidenceProvider, IMetricsEvidenceProvider, IKubernetesEvidenceProvider and IArgoCdEvidenceProvider. The API and diagnostic packs do not reference provider clients, database contexts or source schemas.

IReadOnlyEvidenceConnector is the follow-up contract. Its descriptor declares a stable ID, version and operation allowlist. The catalog rejects unknown operations and oversized requests before a provider runs. Each returned finding carries connector, operation, time window, scope and truncation provenance.

IDiagnosticAdapter is the diagnostic-pack contract. Each adapter declares catalog metadata, decides whether it applies to an alert and returns findings with stable evidence IDs. Filomena.Packs provides the packageable base class for external pack projects.

Runtime-loaded binary plugins are not supported. Providers compile with the application so dependency and permission boundaries remain explicit.

Persistence

Every state adapter stores:

• alert_events
• triage_runs
• triage_feedback
• slack_deliveries
• operational_changes
• incidents
• incident_alerts
• run_verifications
• audit_events

PostgreSQL is the default for Compose and Helm production installations. Filomena creates and versions its tables in an existing database. The connection string is supplied through a secret.

The SQLite database must be on a durable volume. It is intended for a single Filomena instance. It runs in write-ahead logging mode, so a query never blocks the writer, and it keeps -wal and -shm files beside the database.

PostgreSQL supports multiple active workers. A durable lease per alert fingerprint prevents two replicas from diagnosing the same lifecycle concurrently. The Helm chart uses rolling updates for PostgreSQL and rejects multiple replicas for SQLite or ClickHouse. SQLite remains the lightweight single-instance option.

The ClickHouse state provider uses the schema under deploy/clickhouse/init. Alert and run history have a one-year TTL. Feedback has a two-year TTL. Its event deduplication is process-local because MergeTree does not enforce unique keys, so this provider must also run as a single Filomena instance.

When enabled, the ClickHouse evidence providers read OpenTelemetry logs from otel.otel_logs and spans from otel.otel_traces. Filomena never writes to telemetry tables. State and evidence providers are configured independently.