Open source, self-hosted

Grafana alerts, diagnosed against your own evidence

Filomena turns an alert into a bounded investigation across logs, metrics and infrastructure, then makes the model cite what it used. Bring your own model endpoint.

#incidentsFilomena

[TRIAGE] DemoApiHighErrorRate

The demo API is returning server errors.

Cause: Connection pool exhausted after the demo-api:2.4.1 deployment

Observed:

  • Database connection pool exhausted after deployment image demo-api:2.4.1

Next:

  • Check pool saturation either side of the demo-api:2.4.1 deployment

Owner: demo-team | Confidence: 65%

The message format Filomena posts to Slack. The alert, service, owner and the observed finding come from the demo fixtures in examples/; the cause and the next step stand in for model output, which is asked for as read-only investigation steps. Slack credentials and a logs provider are both off by default, so a bare docker compose up will not post this. Stored confidence is the lower of the model’s own number and 0.55 + collected findings × 0.1, and never above 0.9.

Evidence comes from what you already run

How a run works

Four stages, one recorded run

A completed run is stored with its evidence, diagnosis and economics. A failed run is stored with the stage it failed at and whether it can retry, and retries are capped.

  1. Step 01

    Ingest

    Grafana alerts become canonical events with a stable fingerprint, a resolved service identity and explicit incident membership.

  2. Step 02

    Collect

    Diagnostic packs query only the providers you enabled. Limits, time windows and label filters are pushed down to the provider.

  3. Step 03

    Diagnose

    The model is called with a schema carrying the exact evidence IDs this run collected. A citation outside that set fails the run.

  4. Step 04

    Learn

    Operators answer in Slack. Confirmed cases feed later investigations and the offline evaluation suite.

What a finished run holds

Grounding happens before the response exists

Filomena calls the model with a schema listing the IDs of the findings this run collected, so those are the only ones it can cite. A citation outside the set fails the run with “Analysis provider cited evidence that was not collected.” What lands in diagnosis.evidence is the text of the findings it cited. A run that failed or was skipped returns no diagnosis, and a run nobody has reviewed yet returns no feedback.

GET /v1/runs/{runId}keys abridged
{
  "run": {
    "runId":       string,
    "fingerprint": string,
    "serviceName": string,
    "status":      string,
    "stage":       string,
    "retryable":   bool,
    "economics":   { "inputTokens": int, "outputTokens": int,
                     "estimatedCostUsd": decimal }
  },
  "alert": {
    "status":   string,
    "ruleUid":  string,
    "startsAt": timestamp,
    "labels":   { string: string }
  },
  "service": {
    "name":        string,
    "environment": string,
    "cluster":     string,
    "owner":       string
  },
  "diagnosis": {
    "likely_cause":        string,
    "confidence":          double,
    "evidence":            [string],
    "recommended_actions": [string]
  } | null,
  "evidence": {
    "window":   string,
    "findings": [{ "id": string, "source": string, "text": string }]
  } | null,
  "feedback": {
    "verdict":   "correct" | "partial" | "incorrect",
    "rootCause": string
  } | null
}

What gets recorded

The tool measures its own cost

One run carries its alert, service, evidence, diagnosis and operator verdict, plus the bill for producing it. A retry is its own run with its own ID, and both stay joined to the alert by fingerprint.

Input tokens
Per run
Output tokens
Per run
Estimated model cost
Per run
Evidence findings and sources
Per run
Feedback verdict
Per run
Time to resolution
Per incident

Run it yourself

Your telemetry, your model endpoint, your bill

Start on PostgreSQL, add the evidence providers you already operate, and keep SQLite or ClickHouse where they fit the deployment.

Terminal
$ git clone https://github.com/bfxavier/filomena.git
$ cd filomena
$ docker compose up --build -d
Continue the quickstart