URL Reputation and Content Fingerprinting

IP and sending-domain reputation (blocklists) judge who is sending. Content filters add two more axes that judge what is being sent: the reputation of every domain linked inside the message body, and a fuzzy fingerprint of the message itself that matches it against previously seen spam campaigns. Both operate independently of the sender's own authentication and IP reputation — a fully authenticated message from a clean IP is still filtered if it links to a blocklisted domain or fingerprints as a known campaign.

Part 1 — URL/domain reputation: SURBL

SURBL is the archetype of a URI DNSBL: instead of listing sending IPs, it lists domains (and some IPs) that appear in the bodies of unsolicited or malicious mail. URIBL.com and the Spamhaus DBL/HBL (see Blocklists & Spamhaus) work on the same model. Spam filters extract every URL from a message, reduce each to a domain, and query it against these zones; a hit typically scores heavily or blocks outright.

List zones and return values

The public data is served as one bitmasked combined zone, multi.surbl.org, aggregating these datasets:

List	Meaning	Bit value (last octet)
DM	Disposable email domains	4
PH	Phishing sites	8
MW	Malware sites	16
CT	Click-tracker domains	32
ABUSE	General abuse / spam sites	64
CR	Cracked (compromised legitimate) sites	128

• Responses are A records in 127.0.0.X form; membership in multiple lists sums the bits — e.g. 127.0.0.80 = MW + ABUSE (16 + 64).
• NXDOMAIN = not listed; an A record = listed. SURBL states the A record is "the strongly preferred response for automated use."
• TTL: 60 seconds default on the live multi zone; the data itself is updated roughly every 30–40 seconds.
• A response of 127.0.0.1 is not a listing — it signals the querier's access is blocked (excess public-mirror volume; requires signing up for SURBL's Sponsored Data Service).
• Beyond domain lists, SURBL also operates HASHBL hash-based reputation queries in categories: abuse, cracked, malware, phish, email, crypto, phone.
• Data delivery formats for filter operators: DNS (Private Query Service), rsync (recommended for high-volume mail filtering), RPZ (web filtering), REST API, CSV, and RTF (JSON real-time feeds).

Note the CT (click tracker) list exists at all: tracking/redirect domains are a first-class listing category, not an edge case.

How filters extract and query URLs (SURBL implementation guidelines)

SURBL's published guidance to filter implementers describes what production content filters actually do:

1. Extract every URI from the message, with full resolution of redirections into the final target domain — i.e., filters are told to follow redirectors and shorteners to the destination and check that domain, not just the visible link.
2. Reduce URIs to domains/subdomains. With the wildcarded multi zone, subdomains of a listed domain match automatically, so filters need not normalize to registered-domain level.
3. Perform no DNS name resolution on the extracted domains — the domain string itself is the lookup key.
4. Query by prepending the domain to the zone (domainundertest.com.multi.surbl.org) and doing an A-record lookup.
5. Numeric-IP URLs (http://10.20.30.40/) are checked with octets reversed, DNSBL-style: 40.30.20.10.multi.surbl.org (base-10 octets).
6. Maintain a local whitelist of known-good high-traffic domains (yahoo.com, w3.org, google.com) to avoid pointless queries.
7. Validate that answers fall in 127/8 — an answer outside 127.0.0.0/8 indicates a wildcarding/redirecting DNS resolver corrupting results; run a local caching nameserver.

Two explicit prohibitions from SURBL, because both cause false positives with shared hosting: do not use body-URI data to check sender IP addresses, and do not resolve listed domains to IPs and blocklist those IPs.

What this mechanism implies

• Listing one domain poisons every message that links to it, from every sender, on every IP — a URI-DNSBL listing of a shared domain is a cross-customer incident for an ESP.
• Because filters resolve redirects, hiding a bad destination behind a shortener or your own redirect domain does not evade the check — it instead exposes your redirector domain to listing when the destinations it fronts are abused (this is exactly what SURBL's CT list captures).
• Because subdomains of a listed domain match under the wildcarded zone, a listing at the registered-domain level takes down every customer subdomain beneath it.

Part 2 — Content fingerprinting: fuzzy hashing (rspamd)

Fuzzy hashing detects that a message is a near-duplicate of previously reported spam, even after the spammer mutates it. rspamd's fuzzy_check module and fuzzy_storage worker are a fully documented open implementation; commercial fingerprinting systems (and shared-hash networks like Razor/Pyzor/DCC) follow the same principles. rspamd also ships a public shared fuzzy feed that many installations query by default, so a fingerprint learned anywhere can score everywhere.

How the fingerprint is built: shingles

• Message text is tokenized into words and split into overlapping word sequences (trigrams) — "shingles."
• Each shingle is hashed with multiple hash functions (32 hashes per shingle); the per-message fingerprint is the set of these shingle hashes. (rspamd cites Broder's resemblance/shingling research as the basis.)
• Comparison is probabilistic: similarity is computed from the count and positions of matching shingle hashes between the candidate message and stored hashes, yielding a match percentage rather than exact equality.
• Text hash algorithm options: mumhash (current default, recommended), xxhash, fasthash, siphash (legacy). Changing the algorithm invalidates all stored data.
• Images and attachments are matched exactly, not fuzzily, via blake2b digests of content — one changed byte breaks the match, which is why image-spam campaigns re-render images per send, and why filters pair this with perceptual image hashing elsewhere.

Key defaults in fuzzy_check: min_bytes = 1k (minimum attachment/image size considered), min_height/min_width = 32 px for images, min_length = 0 (all text parts checked), text_multiplier = 4.0, timeout = 2s, retransmits = 1; mime_types selects which attachment types (e.g. application/*) are hashed; short_text_direct_hash exact-hashes texts too short for shingling.

HTML-structure fuzzy hashing (rspamd v3.14+)

Since v3.14.0, rspamd can also fingerprint the DOM structure independently of text: tokens of the form tagname[.class][@domain] (e.g. a.button@example.com; only the first CSS class is used, known tracking classes are filtered, link domains are normalized to eTLD+1). The combined hash weights: structure shingles 50%, call-to-action (CTA) domains 30%, all link domains 15%, feature counts (tags/links) 5%. Guards against matching generic templates: requires ≥ min_html_tags (default 10; example configs use 15), ≥ 2 links, DOM depth ≥ 3.

The CTA-domain weighting is aimed at phishing: a phish that perfectly clones a brand template gets structure similarity ~0.9, but with different CTA domains the combined similarity collapses (documented example: 0.9 structure × mismatched CTA ⇒ combined 0.45) — and conversely, a spam campaign that keeps its CTA domain while shuffling copy still matches.

Why mutations and token-stuffing don't evade it

Because the fingerprint is a large set of overlapping word-trigram hashes compared probabilistically:

• Small edits (swapped words, inserted names, reordered paragraphs) change only the shingles that overlap the edit; the bulk of shingles still match, and similarity stays above threshold.
• Token stuffing (appending random words, hidden text, hash-busters) adds shingles but does not remove the matching ones; it only dilutes the similarity ratio fractionally, while the absolute count of known-bad shingles still matches. Defeating shingle matching requires rewriting essentially the whole body — at which point HTML-structure hashing (v3.14+) still matches the unchanged template and CTA domain.
• Per-recipient merge fields, unsubscribe URLs, and tracking tokens likewise leave the shared shingle mass intact — which is exactly why one campaign, reported by early recipients, is recognized in the rest of the send.

Weights, thresholds, and gradual scoring

Stored hashes carry a weight that accumulates as sources (user reports, trap hits, honeypots) re-report the same fingerprint. Scoring is deliberately gradual, not binary:

• Each flag/rule has a max_score (hash-weight threshold). Below the threshold the symbol scores 0; the score then rises from the threshold to 2× threshold (hyperbolic-tangent curve — ≈50% of metric weight at the threshold, full score at 2×). Example: report weight 1 with threshold 20 means a fingerprint needs ≥ 20 independent complaints before it scores at all.
• Hashes are organized by numeric flags mapping categories to symbols — the conventional layout: flag 1 = confirmed spam (FUZZY_DENIED, max_score 20), flag 2 = probable spam (FUZZY_PROB, max_score 10), flag 3 = legitimate/whitelisted content (FUZZY_WHITE, max_score 2). Flag numbers must be unique across writable rules; skip_hashes whitelists specific fingerprints.
• Learning: rspamc -f <flag> -w <weight> fuzzy_add <message> (or -S FUZZY_DENIED), removal via fuzzy_del; read_only = true makes a rule query-only; an optional Lua learn-condition can gate or re-flag learning.

Operational side (fuzzy_storage worker): single-writer architecture with an update queue synced to disk every sync = 1min; backends SQLite (default) or Redis; hash expiry via expire (90d recommended initially — old fingerprints age out as campaigns die); sizing ≈ 400k hashes per 100 MB RAM (1.5M ≈ 500 MB); learning restricted to allow_update IPs; optional Curve25519 transport encryption (encrypted_only = true) and master–slave replication (TCP 11335) with per-slave flag translation.

Sender implications

The two mechanisms combine into concrete rules for anyone running an ESP or a sending program:

• Every domain in the body is a reputation carrier — link domains, image-host domains, redirect/click-tracking domains, even domains in plain text. Their reputation is evaluated separately from the From domain and the sending IP.
• Shared tracking domains are shared fate. An ESP-wide click-tracking or link-wrapping domain aggregates the reputation of every customer's destinations; one abusive customer can get it listed on a URI DNSBL (SURBL's CT category exists for exactly this), filtering all customers' mail at once. Mitigations: per-customer tracking subdomains on customer-owned domains (CNAME'd), proactive URI-DNSBL monitoring of tracking domains, and screening destination URLs at send time against SURBL/URIBL/DBL.
• Public shorteners are worse than useless: filters resolve them to the destination anyway, and the shortener domain itself carries the accumulated reputation of all its abusers — see Content & Design for Deliverability.
• Link-domain hygiene is a monitoring task, not a one-time check: query your tracking, image, and landing domains against multi.surbl.org (and Spamhaus DBL) on the same cadence as IP blocklist monitoring, and check every outbound campaign's URLs before send.
• Fingerprinting makes complaint velocity retroactive: the first few thousand recipients who report a campaign create/weight its fuzzy hash, and the remainder of the same send then matches it. Segment-and-pause sending (send to most-engaged first, watch early complaint signals, then continue) directly exploits the threshold/2×-threshold scoring window.
• Template mutations don't reset content reputation. Minor copy changes, subject rotation, or merge-field personalization leave the shingle fingerprint (and the HTML-structure/CTA fingerprint) intact. The only real reset is genuinely different content to an audience that wants it — content reputation is a symptom of list quality, not a content-engineering problem.
• Consistent CTA domains cut both ways: keeping your links on your own stable, well-reputed domains helps structure-hashing distinguish you from phishing clones of your template; scattering links across throwaway domains looks like evasion.

Related

• Blocklists & Spamhaus — IP/domain DNSBL mechanics, Spamhaus DBL/HBL (the other major domain-reputation datasets).
• Content & Design for Deliverability — practical content rules (shorteners, images, link practices) that these mechanisms explain.
• Reputation Monitoring — where URI-DNSBL checks of tracking/link domains belong in a monitoring program.