RU
team · threads

You disagree with a finding: how to argue so the bot answers

The bot left a finding on a line of code in your merge or pull request, and you have a reason it should not be there: the log is internal, the code is old, the pattern is deliberate. You can resolve the thread and move on — people often do, and that is exactly how a review turns into a formality. Or you can object in the thread and get an answer back.

Here is an example — a thread in a public pull request of our sample repository on GitHub (the Russian run is a separate merge request on GitLab). The developer objects: the logs go to a collector inside the same network, they are kept for 14 days, only the on-call team has access, and support searches by e-mail because the customer does not know the invoice id. The bot replies that it understands the operational reasoning, but the team's rule is about not keeping customer data in infrastructure governed differently from the primary database, and the identifier is enough for debugging; if support needs to search by e-mail, that belongs in a proper customer lookup, not in grepping logs. It keeps the finding at major.

The bot's answer is one model call and a few cents. Whether you agree with it is a separate matter; the point is that it is an argument you can answer, not a verdict you can only close.

Replies are off by default. They are turned on by reply.enabled: true in the .reviewgate/config.yml of the branch the merge or pull request comes from, or by REPLY_ENABLED=true on the bot itself — then for the whole installation (an explicit reply.enabled: false in a repository switches it off there). And the webhook has to deliver comment events. Both traps are covered below.

Where you can just answer, and where you have to call it

In its own threads the bot answers any reply; there is no need to call it. Its own are the threads of its findings on lines of code — and on GitLab also the thread under the summary.

In a pull request's conversation on GitHub (the Conversation tab) it is different: the summary there is an ordinary comment in the shared feed, and every reply under it is a separate conversation for the bot. There, as in someone else's thread, call the bot by name — the answer arrives as a new comment in the feed.

In anyone else's thread — a conversation between two people, or one on a line the bot never touched — you have to call it by name, mentioning the account it posts as: on GitLab the user whose token it runs with, on GitHub the app's name (ours is @reviewgate-bot). A mention works anywhere in the text, even inside a quote or a code block. Without a mention the bot has no reason to think the conversation concerns it, and a review tool that joins every discussion uninvited becomes noise within a day.

The bot recognises its own thread technically: the first comment was written by the bot's account and carries its hidden marker. A person quoting a finding does not open a thread the bot answers unprompted — otherwise every quoted review comment would start a conversation. By the same account the bot recognises its own replies and never answers them. That is why, if the bot runs with a person's personal token (a user token on GitLab, a PAT on GitHub), that person cannot talk to the bot: it takes their replies for its own.

.reviewgate/config.yml
# .reviewgate/config.yml — in the BRANCH the pull request comes from
reply:
  enabled: true              # off by default
  max_replies_per_thread: 3  # the only brake on a long argument

Three replies per thread by default (reply.max_replies_per_thread). The limit is checked before the model is called: once it is reached, the bot goes quiet. In a GitHub pull request's conversation the limit is shared by the whole pull request. Nothing else brakes an exchange between two bots: the code does not recognise other bots, so if a teammate's assistant answers in the same thread, only the limit stops the exchange.

Until the limit is spent, every reply in the bot's thread is a paid model call, even «thanks». The model decides to stay silent itself, after it has received the whole input, so silence is paid too: it leaves no trace in the thread and does not count toward the limit, and the bot's log gets a line ↩ The model stayed silent. On our runs with sonnet-5, at current prices, the first reply in a thread cost 3–4 cents and the next ones half a cent to a cent and a half: the input came from Anthropic's cache.

In resolved threads the bot never answers — reopen the thread if you want an answer. Drafts, on the other hand, do not stop it: skipping the review of a draft does not apply to conversations, and a question in a draft gets an answer — as it does in a closed or already merged request.

Editing your comment does not count. The bot reacts to new comments, not to edits. Rewriting your objection in place is invisible to it; to ask again, post a new comment.

Who answers: the first model in generators of your schema, without the judge. The reply is therefore as good as your generator, not as good as your whole ensemble. The model for replies can be changed with reply.model — the vendor stays the same; for another vendor there is reply.backend.

Why it stays silent

1. Replies are off until you turn them on. The reply.enabled key is off by default, and so is REPLY_ENABLED on the bot. When the key is present in the repository, it decides: reply.enabled: true turns replies on even with REPLY_ENABLED=false on the bot, and reply.enabled: false turns them off even with REPLY_ENABLED=true. Without the key in the repository, REPLY_ENABLED applies, and only a literal true counts there: 1 or yes will not turn replies on. A team that has never turned them on has a bot that reviews and never answers — that is not a fault, it is the product's default.

2. The config is read from the merge request's branch, not from the target. The policy for a merge request is taken from the branch it comes from. Turn replies on in main, and in open merge requests the bot stays silent until their branches pick up the change — for example until main is merged into them. Two things to keep in mind: such a merge is a new paid review run, and the bot does not go back to old comments — write the objection again. You do not need the branches if you turn replies on at the bot with REPLY_ENABLED=true: it applies at once to every merge request whose branch has no reply.enabled key.

3. The webhook does not deliver comments. A review arriving normally proves nothing about replies: they travel on other events. On GitLab the hook needs Comments (note_events) in addition to merge request events. On GitHub a reply in a finding's thread arrives as «Pull request review comment», and a reply in the pull request's conversation as «Issue comment», which needs the app permission «Issues: read». If these events are missing from the hook's settings, nobody tells you: neither GitLab or GitHub nor the bot shows an error — the comment simply never reaches the bot, and it stays silent.

4. The model decided to stay silent. A reply like «fixed» or «thanks» needs no answer, and the model may stay silent on its own. The thread shows nothing, and the bot's log has a line ↩ The model stayed silent.

how to tell which one it isWrite a comment in one of the bot's own threads and find it in the bot's logs (docker compose logs --since 10m app | grep -i reply). No Queued a reply line — the event did not arrive: it is the webhook. The line is there and is followed by ⏭ Reply skipped: — the bot got the comment and declined, and the reason is in the same line: replies are off, the thread is resolved, the limit is spent, someone else's thread without a mention. ↩ The model stayed silent — the bot asked the model, and the model chose not to answer.

The exchange, in full

The finding, on the line where the e-mail goes into the log — the whole thread on GitHub (the Russian run: GitLab):

in the thread
🟠 no-customer-data-in-logs 📐 — The log line includes command.customerEmail, which is
   customer data. Logs must carry only identifiers and outcomes so incidents can be
   investigated without leaking PII.

The objection — a real argument, not «you are wrong»:

in the thread
To be concrete: our logs go to a self-hosted collector inside the same VPC, retention is
14 days, and only the on-call team has access. Support searches by e-mail because a
customer calling about a missing invoice does not know the invoice id.

Given that, is this finding still worth acting on?

The bot's answer, two minutes later:

in the thread
I understand the operational reasoning, but the team's rule here isn't about log-shipping
risk alone — it's about not persisting customer PII into infrastructure that isn't governed
the same way as the primary datastore (access reviews, retention, right-to-erasure, etc.
rarely match your DB's policies even for internal-only collectors). VPC-only and 14-day
retention reduces exposure but doesn't remove the PII-in-logs risk category this rule
targets.

For the support workflow, the identifier is enough for on-call/dev debugging; if support
needs to search by e-mail, that should go through a proper customer lookup (e.g. a query
against the invoices/customers table or a dedicated support tool), not by grepping
application logs. I'd keep this as a major finding — please log the invoice id only.

It acknowledged the argument instead of repeating the finding. It named what the rule is actually about: not the risk of shipping logs, but keeping customer data out of infrastructure governed differently from the primary database — the distinction the objection had missed. And it answered the underlying need, offering another way to solve the support problem rather than just refusing. It also did not fold: a tool that agrees with the last person who pushed back is not a reviewer, it is a mirror.

Both conversations are public — the pull request and the merge request on our sample repository. The Russian one is a separate run, not a translation, and it arrived at the same position independently.

Give it a fact it cannot have

what the bot sees when it answersWhen it answers, the bot sees the thread, 30 lines of code on either side of the finding's line, your team's standards, and the merge request's title and description — and nothing else: no diff, no whole files (under the summary and in a GitHub conversation it sees no code at all). It does not know that your logs stay in the VPC, that this module is scheduled for deletion, that the pattern was agreed in an ADR two years ago. Those are the objections worth writing, because they are the only ones that add information. «This is a false positive» adds none. «Not important» adds none. The reply you get back will be as thin as the objection.

If the fact concerns not one line but the whole merge request or the project, put it where every run sees it: in the merge request description — reviews and replies both read it — or in dont_flag in the config.

an answer does not remove the findingWorth being clear about, because people expect otherwise: the conversation changes nothing in the summary or the gate. The finding stays, and the check stays red if it was red. A reply is an argument, not a verdict. What takes a finding off the gate is covered in The bot's threads get resolved, the code stays the same: fixing the code, resolving the thread — with caveats — or changing the rule or the config.

That is the fork at the end of every argument you win: if the bot was wrong because your rule was written without an exception, fix the rule — one sentence in the rule text stops the whole class. If the bot was right but you are not going to fix it now, that is a decision, not a dispute: say why in the thread and resolve it. On GitLab that is one action — the «Resolve thread» checkbox under the reply, and the bot will not answer; on GitHub resolving is a separate button, and the bot may answer before you press it.

The limit is a feature. A ten-message argument with a review tool is not a good use of anyone's afternoon or budget — if three exchanges have not settled it, the disagreement is about the rule, not about the line. Take it to the config.

When it goes wrong

The bot answered something generic. It usually means the objection carried no new fact. Write what the bot cannot see: a constraint, a team agreement, the plan for this code.

The bot agreed and the finding is still there. That is expected: the answer changes nothing in the summary or the gate. If the finding is wrong, resolve the thread, and if your rule produced it, fix the rule — otherwise it may come back in the next merge request.

The thread went quiet after three replies. That is the limit. Open a new thread and call the bot by name — a new thread gets its own limit (except in a GitHub pull request's conversation, where it is shared by the whole pull request). Better still, settle it in the config, where the next merge request will inherit the answer.

It answered in a draft you did not expect it to. Drafts are skipped for reviews, not for conversations. A question asked in a draft is answered, and it costs the same.

Next