Status:  Production
Type:  SHA-1 hash

Cloud DNS:  <key>.shorthash.mail.abusix.zone.
Rsync File:  lists/shorthash.zone

Return Codes:  127.0.3.1
Test Points:  127.0.02, 127.0.3.1, *.test, d2e4345eef7b21a542ed6d7c3dd191585b344461 (abusix.ai/testpoint), f4d986915d728956d139397effd00fee0e3725e4 (abusix.ai/testpoint/hash/short)

Listing duration:  Approximately 5.2 days after last seen

Description

We developed this list to list Short URLs seen in the message body of spam sent to our “pristine” traps.

This list compliments the domain blacklist as using Short URLs has become a common way for spam to avoid domain blacklisting by hiding behind these services along with legitimate users so it’s not possible to list some Short URL domains without causing significant false-positives.  Additionally, these shortening services are usually very poor at handling abuse of their services.

Because it is not possible to represent a full URL in a DNS query, the Short URLs are normalized first, then SHA-1 hashed and the hash value is used for lookup instead of the URL.

To normalize the short URL, remove the scheme, then take only the “hostname” (lowercased) and “pathname” and then calculate the SHA-1 hash of the result:

http://BiT.do/e3s49?foo=bar&bar=baz → SHA1(bit.do/e3s49) = bb395cece75455415de5f3b6f75c13352586788c

Note

As this is a completely new type of anti-spam check, it will require support for this to be added to your chosen mail platform.  See below for example code for rspamd.

Rspamd

The following code can be added to /etc/rspamd/rspamd.local.lua to enable rspamd to query this zone.   Note that you should replace "<APIKEY>" with your API key or set the "check_shorturls_dns" variable appropriately for your DNS namespace if you are using rsync.

local rregexp = require "rspamd_regexp"
local rlogger = require "rspamd_logger"
local rhash = require "rspamd_cryptobox_hash"
local rutil = require "lua_util"

local re_short_path = rregexp.create_cached('/^(?!(?:[a-z]{3,11}|[A-Z]{3,11}|[0-9]{3,11})$)[a-zA-Z0-9]{3,11}$/')
local check_shorturls_dns = '.<APIKEY>.shorthash.mail.abusix.zone.'

local check_shorturls_cb = function (task)
    local function find_short_urls (url)
        local path = url:get_path();
        if (re_short_path:match(path)) then
   return true
        end
    end
    local shorturls = rutil.extract_specific_urls({
        task = task,
limit = 5,
        prefix = 'shorturls',
filter = find_short_urls
    });

    if (not shorturls) then return false end

    local r = task:get_resolver()

    for _, url in pairs(shorturls) do
        -- Normalize
        local surl = url:get_host():lower() .. '/' .. url:get_path()
        local surl_hash = rhash.create_specific('sha1', surl):hex()
        local lookup = surl_hash .. check_shorturls_dns
        local function dns_cb(_,_,results,err)
            if (not results) then return false end
            if (tostring(results[1]) == '127.0.3.1') then
                rlogger.errx('found URL %s (%s) in Short URL blacklist', surl, surl_hash)
                return task:insert_result('RBL_AMI_SHORTURL', 1.0, surl);
            end
        end
        r:resolve_a({ task = task, name = lookup , callback = dns_cb, forced = true })
    end
end

local check_shorturls = rspamd_config:register_symbol({
    name = "RBL_AMI_SHORTURL",
    score = 3.0,
    description = "Short URL found in Abusix Short URL blacklist",
    group = "abusix",
    type = "callback",
    callback = check_shorturls_cb
});
Did this answer your question?