dtrh.net~/blog
KBS
│KBS all posts

Bittorrent From 100 Hours To 1 000 000 Torrents

Bittorrent From 100 Hours To 1 000 000 Torrents

SOMETHING MOST OF US CAN RELATE TO

Without sounding unduly severe, it is reasonable to assume that if you have never encountered the BitTorrent protocol—or the term “torrent,” as it is used colloquially by digital pirates—you may have missed an essential chapter in the history of online file-sharing. From the pioneering days of Napster, LimeWire, and Kazaa to the broader umbrella of peer-to-peer (P2P) technologies, torrents have been an enduring fixture of internet culture. Those who were online in the late 1990s will recall the public anxiety over technology in the run-up to the new millennium—Y2K panic, media crackdowns on alleged infringers, and a series of record-setting lawsuits designed to make examples of individual downloaders.

This article recounts my recent experiment with BitTorrent’s Distributed Hash Table (DHT). I will avoid excessive technical detail so the discussion remains accessible to a general audience.


THE EXPERIMENT

Every sound study relies on data. I self-hosted a service that periodically probed the BitTorrent DHT, recording each “hit” in an SQLite database by means of magnetico. In essence, the service crawls the DHT by contacting peers—other participants in the network—then stores key metadata for each discovered torrent.

A torrent’s unique identifier is its infohash, a SHA-1 (or optionally SHA-256) hash of the underlying file set. Knowing only this hash, one can reconstruct an entire magnet link; provided at least one peer can supply the corresponding metadata, the rest of the file can be located piece-by-piece.

To maximize throughput, I deployed the crawler on a server with a 40 GbE connection and ample upload and download capacity. I configured cron jobs to ensure the service restarted automatically after reboots or unexpected failures. Below is the supervisory script:

#!/bin/sh
#
#  Binaries:    $HOME/.local/bin/magneticod
#               $HOME/.local/bin/magneticow
#  Credentials: $HOME/.config/magneticod/credentials
#  URL:         http://dht.media.dtrh.net:48080
#
#  Location:    ~/Scripts/cron/hourly
#  Schedule:    @hourly
####

# --- MAGNETICOD --------------------------------------------------------------
if ! pgrep -fx "$HOME/.local/bin/magneticod" > /dev/null; then
  screen -dmS dMAGNETICOD \
    "$HOME/.local/bin/magneticod" \
    --indexer-max-neighbors=2000 \
    --leech-max-n=25
fi

# --- MAGNETICOW --------------------------------------------------------------
if ! pgrep -fx "$HOME/.local/bin/magneticow" > /dev/null; then
  screen -dmS dMAGNETICOW \
    "$HOME/.local/bin/magneticow" \
    -c "$HOME/.config/magneticod/credentials"
fi

THE DATA

Owning an indexed torrent catalog is no more illegal than owning a firearm and being accused of robbing a bank with it. BitTorrent’s DHT is a key/value store: the key is the infohash, and the value is a list of peers (IP addresses and ports) currently offering the corresponding data. Because a cryptographic hash is unique to its source, an infohash can reliably stand in for the torrent file itself.

Diagram of DHT lookup
Figure 1 – DHT lookup in the BitTorrent protocol.


ONE MILLION TORRENTS

After approximately 100 hours of continuous crawling, the database surpassed one million unique torrents. The SQLite file occupies just over 4 GiB:

CLI output showing database size
Figure 2 – Command-line confirmation of database size.

Additional statistics are illustrated below:

Further CLI data
Figure 3 – Supplementary dataset statistics.


NOTEWORTHY FINDINGS

Roughly 80 % of torrents circulating in the DHT are pornographic—an observation that likely speaks to human proclivities more than technological constraints. Yet, with targeted queries, one can uncover genuinely valuable material. By searching for uncommon compression formats such as .zst, I routinely find multi-terabyte archives of rare content; discovery has become something of a hobby.

A particularly notable trove is available here: https://rabbit-hole.ca/db.sqlite.csv. This dataset appears to be an internal index from a major public torrent site, leaked less than a year ago. After basic sanitization (all adult content removed), the file enumerates nearly three million unique torrents—primarily television and film.


Figure 4 – Sample rows from the leaked CSV (headers trimmed for clarity).

While publishing such a catalog exceeds typical notions of “public” sharing, the information is already circulating widely; my contribution is merely to present it in a cleaner, more accessible format.

“Oh well…” meme image

Draw your own conclusions.


normal~/dtrh.net│Bittorrent From 100 Hours To 1 000 000 Torrents