Logs and monitoring

The bot writes its logs to the container's stdout. Every line of a review carries a job tag, so the stream stays readable even when several teams share one bot. For a centralised stack (Graylog / ELK / Loki) there is a structured JSON format with the fields of the particular request.

The job tag in the logs

You read the logs as usual: docker compose logs app (or docker compose logs -f app to follow them). Every line inside a review carries the job tag [group/project!7273 @40b1db1f] — the repository path, the request number and the SHA of the head commit. When there are several teams, a grep by project or request number collects the history of one review without mixing in the lines of neighbouring jobs.

You can see what the review is waiting on

The long steps of a review are announced in the log before the model is called — as a line like → Judge claude-opus-5: waiting for a response… — and when the answer comes back the familiar line with tokens and duration follows. Without that first line, minutes of waiting looked like a halt in the log: the records could not tell a working job from a stuck one. The roles are named for what they are — generator, extra generator of the ensemble, judge, arbiter — so the log shows immediately which call is dragging.

The structured format (JSON)

For a centralised log stack, turn the structured format on: LOG_FORMAT=json in .env. The job attribution goes out as separate fields — project_path / mr_iid / head_sha / vendor / attempt — which Graylog, ELK and Loki can filter on without parsing text. The default (text) is human readable, with the same tag inside the line.

# .env — LOG_FORMAT takes two values (strictly lower case):
#   text — human readable (THE DEFAULT), with the job tag inside the line
#   json — structured: one JSON line per record, with the job fields separate
LOG_FORMAT=json

# an example json record (a single line):
# {"level":"log","pid":1,"timestamp":1751970923893,"message":"Найдено замечаний: 2",
#  "context":"ReviewProcessor","project_path":"frontend/shop","mr_iid":7273,
#  "head_sha":"40b1db1f3b4beb7833a79682acae2c738328b159"}

The review budget: a dashboard for whoever owns the limits

The spending cap (…_BUDGET_TOKENS) is visible outside the summaries too: every run writes the worst threshold it reached into the Postgres metrics (budget_barrier: approaching / exceeded / stop) along with the spend per backend as JSON (tokens_by_backend). Whoever owns the limits does not have to read pull requests — a Grafana panel over the same database is enough (or any SQL client):

-- budget thresholds over 7 days: how many runs hit one, and on which backend
SELECT date_trunc('day', created_at) AS day,
       budget_barrier,
       count(*)                      AS runs,
       -- total spend per backend: unfolding the tokens_by_backend JSON
       jsonb_object_agg_sample.backend,
       sum(jsonb_object_agg_sample.tokens::bigint) AS tokens
FROM reviewgate_review_metrics,
     LATERAL jsonb_each_text(tokens_by_backend::jsonb)
       AS jsonb_object_agg_sample(backend, tokens)
WHERE created_at > now() - interval '7 days'
  AND budget_barrier IS NOT NULL
GROUP BY 1, 2, 4
ORDER BY 1 DESC, tokens DESC;

Alerting is up to whatever you already run: on the exceeded thresholds the bot writes WARN and ERROR into its own log (journald or Graylog below), and a stop is visible in the metrics as well (budget_barrier = 'stop').

Shipping to Graylog

The bot ships nothing to external systems itself — the logs leave through Docker's own log driver (GELF) from the bot's compose file. No new outbound connection appears from the application.

services:
  app:
    logging:
      driver: gelf
      options:
        gelf-address: "udp://graylog.internal:12201"
        tag: reviewgate-bot

Unfolding the JSON into Graylog fields

Docker's GELF driver puts the stdout line into the message field as is — it is a transport, not a parser. So the whole JSON record arrives as one string inside message, with the job fields (project_path, mr_iid, head_sha, vendor) still folded inside it. The container metadata (container_name, image_name, tag) is added by Docker itself and is visible as fields straight away. Unfold the JSON on the Graylog side with one of the two methods below, not both.

Method 1 — a JSON extractor on the input (recommended: simpler, no streams)

1. System → Inputs → on your GELF input press Manage extractors.
2. Get started → Load Message — this pulls a recent record so you can pick the field.
3. Next to the message field → Select extractor type → JSON.
4. Set Key prefix to rg_ and enable Flatten structures. The prefix is mandatory: without it the inner level / message / timestamp overwrite Graylog's own service fields of the same names — message and level get clobbered, and a malformed timestamp can make the record be dropped at indexing time. With the prefix the keys become rg_message / rg_level and never collide.
5. Create extractor.

That is it — the extractor works automatically on every new message of that input; no stream and no pipeline are needed. The fields become rg_project_path, rg_mr_iid, rg_context and so on.

Method 2 — a pipeline rule (clean field names, if you have several sources)

Three objects, and all three are required — the third is the one people forget:

1. System → Pipelines → the Manage rules tab → Create Rule → paste the rule below → Save (it checks the syntax).
2. Manage pipelines → Add new pipeline → add this rule to Stage 0.
3. Pipeline connections → Edit connections → connect the pipeline to the Default Stream (All messages) → Save.

rule "reviewgate: unfold the json from message"
when
  has_field("tag") && to_string($message.tag) == "reviewgate-bot"
  && starts_with(to_string($message.message), "{")
then
  // the message line is the bot's JSON record; we take the fields we need explicitly.
  // The service fields level/message/timestamp are NOT overwritten (that would break the record time).
  let j = parse_json(to_string($message.message));
  set_fields(select_jsonpath(j, {
    "project_path": "$.project_path",
    "mr_iid":       "$.mr_iid",
    "head_sha":     "$.head_sha",
    "context":      "$.context",
    "note_id":      "$.note_id",
    "attempt":      "$.attempt",
    "rg_message":   "$.message",
    "rg_level":     "$.level"
  }));
end

Verifying it

1. Search → expand the new message → the fields project_path, mr_iid, context (method 2) or rg_project_path and so on (method 1) have appeared.
2. Method 2: System → Pipelines → Manage rules — the rule's Executed counter grows, and Failed = 0.
3. The final check: a search for project_path:"group/project" (or with the rg_ prefix) returns messages.

Review metrics in Postgres

If the bot has a DATABASE_URL, every review writes a row into the reviewgate_review_metrics table: the models, the tokens, how many findings the judge dropped, the duration. Metadata only — no code, no diff, no finding texts. Without DATABASE_URL the accounting is simply off and the review still works. Rows older than METRICS_RETENTION_DAYS are deleted on write.

The three queries you usually want:

-- who ran how many reviews in 30 days, and what it cost in tokens
SELECT user_login,
       count(*)                                  AS runs,
       sum(gen_in_tokens + val_in_tokens)        AS in_tokens,
       sum(gen_out_tokens + val_out_tokens)      AS out_tokens
  FROM reviewgate_review_metrics
 WHERE surface = 'cli' AND created_at > now() - interval '30 days'
 GROUP BY user_login
 ORDER BY out_tokens DESC;

-- comparing the surfaces: reviews of pull requests against runs in the editor
SELECT coalesce(surface, 'webhook') AS surface, count(*) AS runs,
       sum(gen_out_tokens + val_out_tokens) AS out_tokens
  FROM reviewgate_review_metrics
 WHERE created_at > now() - interval '30 days'
 GROUP BY 1;

-- what the team trips over (empty means everything goes through)
SELECT status, count(*) FROM reviewgate_review_metrics
 WHERE status <> 'reviewed' AND created_at > now() - interval '7 days'
 GROUP BY status ORDER BY 2 DESC;

There is no code in the logs