[Python] Why is my DHT crawler using libtorrent extremely slow, and how can I optimize it?

I’m building a DHT crawler using libtorrent to discover torrents and their metadata. However, the crawler is extremely slow, processing a small amount of torrents. Additionally, many torrents are unresponsive or fail to fetch metadata. I’ve tried adjusting session settings, but the performance remains unsatisfactory.

Here’s the simplified version of my code:

import time
import libtorrent as lt

MAX_TORRENTS = 200 # Maximum number of simultaneous torrents

session = lt.session()

dht_bootstrap_nodes = [
"router.utorrent.com",
"router.bittorrent.com:6881",
"dht.transmissionbt.com:6881",
"dht.libtorrent.org:25401",
"router.bitcoment.com:6881",
"router.bt.ouinet.work:6881",
"dht.aelitis.com:6881",
]

# Configure session settings
settings = {
"active_downloads": MAX_TORRENTS,
"active_seeds": MAX_TORRENTS,
"active_limit": MAX_TORRENTS * 2,
"enable_dht": True,
"dht_bootstrap_nodes": ",".join(dht_bootstrap_nodes),
"alert_mask": lt.alert.category_t.dht_notification
| lt.alert.category_t.status_notification,
}
session.apply_settings(settings)

current_time = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
print(f"[{current_time}] Start...")
try:
while True:
alerts = session.pop_alerts()
for alert in alerts:
current_time = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
if isinstance(alert, lt.dht_get_peers_alert):
print(f"[{current_time}] Found info hash: {alert.info_hash}")
elif isinstance(alert, lt.metadata_received_alert):
print(
f"[{current_time}] Metadata received for info hash: {alert.info_hash}"
)
time.sleep(0.01)
except KeyboardInterrupt:
print("Stopping the DHT crawler...")

current_time = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
print(f"[{current_time}] Stopping the DHT crawler...")


Results:

[2025-03-07 08:24:21] Start...
[2025-03-07 08:24:22] Found info hash: 707de37217f78b300dea728cc84981d4df74bd79
[2025-03-07 08:24:23] Found info hash: 7168eb3468a2858449e148920820efebebaac0d3
[2025-03-07 08:24:40] Found info hash: 72b00a1d82a2432eece385c160ae0e22195bb5f9
[2025-03-07 08:24:48] Found info hash: 7365e551ff7733ac8c9196f2f63e66981fccdb14
[2025-03-07 08:24:50] Found info hash: 7365ebbaa583e61fa7a921411dca9c39e85b09ac
[2025-03-07 08:25:33] Found info hash: 7365e87e242ecf777b963852958292e95b52f1a5
[2025-03-07 08:25:45] Found info hash: 7365e5d78f817202b86b27395bd59f00340c1835
[2025-03-07 08:26:07] Found info hash: 7365e5d78f817202b86b27395bd59f00340c1835
[2025-03-07 08:26:44] Found info hash: 7365e13c29484a0304ee35a844835fe1b81bca2f
[2025-03-07 08:26:45] Found info hash: 7365e514f5b40caff0e842c92b80c62004ccb156
[2025-03-07 08:26:56] Found info hash: a440faf6639806df76e15607cd71be421a6b2728
[2025-03-07 08:27:04] Found info hash: a440f18f096e3e215f4e8eacb2e9699972fa8368
[2025-03-07 08:27:28] Found info hash: 7365e13c29484a0304ee35a844835fe1b81bca2f
[2025-03-07 08:27:32] Found info hash: a440f6f0fcaa3a62ffa50cc008db025e5dddb70b
[2025-03-07 08:27:45] Found info hash: 7365e4866e555f0e991cd4136df7d7d2381d696d
[2025-03-07 08:27:59] Found info hash: 7365e13c29484a0304ee35a844835fe1b81bca2f
[2025-03-07 08:28:00] Found info hash: 7365e4866e555f0e991cd4136df7d7d2381d696d
[2025-03-07 08:28:22] Found info hash: a440f5cb77ea154ebf7a5ec972061e05d3bb7e94
[2025-03-07 08:28:34] Found info hash: 0fe6bf6df3b4f0e9c214e5a69bd93562e5606351
[2025-03-07 08:28:34] Found info hash: 0fe6ce9f178bc8f0eb501a2342b670e4fd7be0b1
[2025-03-07 08:28:34] Found info hash: 0fe6e4ef11d16a71dc12d27d508e3ce8272d8a25
[2025-03-07 08:28:34] Found info hash: 0fe6f97701a9906dcdb58070f7e24b5f353c0066
[2025-03-07 08:28:34] Found info hash: 0fe6fccc925f09926d391695b40eeeebb0ee4c91
[2025-03-07 08:28:54] Found info hash: 7365e514f5b40caff0e842c92b80c62004ccb156
[2025-03-07 08:29:21] Found info hash: a440fecb25c736e2f6625ac238c9291ffeaa926c
^CStopping the DHT crawler...
[2025-03-07 08:29:26] Stopping the DHT crawler...


If I modify the loop to print all alerts, I see many unhandled alerts, such as lsd_error_alert and external_ip_alert, but the DHT-related alerts are infrequent:

[2025-03-07 08:32:14] Start...
[2025-03-07 08:32:14] Alert: Local Service Discovery error: Invalid argument
[2025-03-07 08:32:14] Alert type: <class 'libtorrent.lsd_error_alert'>
[2025-03-07 08:32:14] Alert: Local Service Discovery error: Invalid argument
[2025-03-07 08:32:14] Alert type: <class 'libtorrent.lsd_error_alert'>
[2025-03-07 08:32:15] Alert: external IP received: 78.26.199.15
[2025-03-07 08:32:15] Alert type: <class 'libtorrent.external_ip_alert'>
[2025-03-07 08:32:16] Alert: incoming dht get_peers: 7365e90ffc5205a0d7a781e8e27b10f28f9fc97b
[2025-03-07 08:32:16] Alert type: <class 'libtorrent.dht_get_peers_alert'>
[2025-03-07 08:32:21] Alert: incoming dht get_peers: a440f4c7ff178202be2e1a1f2020ce45e25ae38f
[2025-03-07 08:32:21] Alert type: <class 'libtorrent.dht_get_peers_alert'>
[2025-03-07 08:32:22] Alert: incoming dht get_peers: a440f49544d1e8ef1112ad8c2ef73aa627f7d6db
[2025-03-07 08:32:22] Alert type: <class 'libtorrent.dht_get_peers_alert'>
[2025-03-07 08:32:30] Alert: DHT bootstrap complete
[2025-03-07 08:32:30] Alert type: <class 'libtorrent.alert'>
[2025-03-07 08:32:39] Alert: incoming dht get_peers: a440f39d3604b0510c3024c2dfdee946ada60e78
[2025-03-07 08:32:39] Alert type: <class 'libtorrent.dht_get_peers_alert'>
[2025-03-07 08:32:39] Alert: incoming dht get_peers: a440fb5561ecada7f18dd4de87fe7644202a245e
[2025-03-07 08:32:39] Alert type: <class 'libtorrent.dht_get_peers_alert'>
[2025-03-07 08:32:48] Alert: incoming dht get_peers: a440f3d386310aad792ce271646aa0d169e63541
[2025-03-07 08:32:48] Alert type: <class 'libtorrent.dht_get_peers_alert'>
^CStopping the DHT crawler...
[2025-03-07 08:33:00] Stopping the DHT crawler...


What I’ve Tried:

1. Adjusted session settings.
2. Added DHT bootstrap nodes to improve connectivity.
3. Reduced the sleep time in the main loop to process alerts faster.

Questions:

1. Why is the crawler so slow, and why are so many torrents unresponsive?
2. Are there any specific session settings or configurations I should adjust to improve performance?
3. Are there any best practices for optimizing a DHT crawler using libtorrent?

Continue reading...