Similar posts

Similarity filter

Sources often repeat each other: three news accounts post the same story, one account posts the same picture twice. The similarity filter compares every new post with the ones delivered to the chat over the last hours and drops look-alikes.

• Similarity threshold, % — how close two posts must be to count as the same, from 1 to 99 percent. The switch turns on at 80 by default.
• Check period, hours — how many hours back to compare, from 1 to 72.
• Minimum post length — shorter posts are not compared by text: letters and digits count, 50 by default, 0 compares all. A repeat of the same link is always caught.

The filter is chat-wide: it looks across all sources of the chat, so it is set only in the chat's Message Filtering — a source does not have it. A dropped clone lands in the events log with its similarity percentage and, if a chat for filtered messages is connected, there too.

How posts are compared

Each new post is checked against what the chat has already received, in three steps:

1. Exact repeat. The same link or the same text word for word is a duplicate at any threshold. A post the source edited and re-published under the same link is not.
2. Same news in other words. A language model reads the new post next to the closest delivered ones and judges whether they tell the same story — a retelling with other wording, order and details still counts. The threshold sets how sure the model must be.
3. Same picture. If the posts carry the same image, the bar for the text is lower — see Same pictures and videos below.

Steps 2 and 3 compare only with posts from other sources, so a series of similar posts from one source is left alone. Texts shorter than six words are checked only for an exact repeat. A post at least three times longer than its match, and 400 characters more, is not a duplicate: it is a fuller version of the story and is delivered.

Same pictures and videos

Besides the text, the filter compares what the posts show. For a photo it looks at the photo itself, for a video — at its preview (the cover frame). The comparison works on the overall picture rather than exact bytes, so it recognizes the same image after the typical changes other channels make:

• another size or re-compression;
• a slightly trimmed edge;
• a channel's logo or a copyright line in a corner.

A matching picture alone does not make a post a duplicate: one stock photo can illustrate different news. It lowers the bar for the text instead — the same picture with a loosely similar text is dropped as a clone, even when the text alone would not reach the threshold.

Heavily cropped images (a strip cut from the top or bottom, a zoomed-in fragment), mirrored images and collages are treated as different pictures.

Update the original from a clone

The dropped look-alike — the clone — is sometimes richer than the published original: it has photos the original lacked, or a longer text. The clone can then update the original. Under the filter settings there are two switches:

• Media from clone — if a clone brings media, update the already published post with it. Append adds the clone's media to the original's, Replace puts them in its place. The Only if the original has no media switch leaves posts that already have media as they are.
• Text from clone — replace the published post's text if the clone has noticeably more text. The Longer by, % field sets how much longer the clone's text must be than the original's.

The original in the chat is re-published with the merged media and text. Media the original already has is not added a second time — neither the same file nor the same photo or video re-uploaded or re-compressed by another source. Every such update lands in the events log.