Scraping Dark Web

Installing package

Let’s try and automate interacting with sites on the Dark Web. Start by installing the tor-package by running following command:

sudo apt install -y tor

This will allow us to tunnel through the onion router and move through different relays and nodes.

There are 3 types of relays in the tor network:

  1. Entry Relays: Entry relays are the first point of contact for Tor users.
  2. Middle Relays: Middle relays receive encrypted traffic from entry relays and forward it to other relays or exit relays.
  3. Exit Relays: Exit relays are the last hop in the Tor network before the traffic exits onto the regular internet.

An important command that is bundled with the tor-package is torify. It’s a wrapper for torsocks and tor. You can put it in front of other commands you want to run and it will tunnel them through tor.

Manpage Tor


If we try to use curl and wrap it we see that we still need to configure it.

Request fail

We open the Tor configuration file:

sudo vim /etc/tor/torrc

Make sure the ‘ControlPort’ is enabled. Enabling the control port allows external applications/scripts to interact with the tor process.

/etc/tor/torrc configuration

CookieAuthentication should be set to 1. Tor will generate a cookie file containing a random value that serves as an auth-token (‘control_auth_cookie’). The control port will only accept connections from processes that provide this token.

Restart the tor-service:

sudo service tor restart

Curl request

If we want to make the curl request, we have to specify the SOCKS5 proxy to use. We do this by using the --socks5-hostname flag. Specify our current localhost and the port we want to use.

socks5-hostname flag man

curl --socks5-hostname http://lockbit7z2jwcskxpbokpemdxmltipntwlkmidcll2qirbu7ykg46eyd.onion/

Make sure tor service is running. Check by running netstat -tuln and look for ports 9050/9051.

Listing ports

Simple Python Script

Install the required python lib by running following cmd:

pip install requests_tor

Simple example:

from requests_tor import RequestsTor

requests = RequestsTor(tor_ports=(9050,), tor_cport=9051)
url = "http://lockbit7z2jwcskxpbokpemdxmltipntwlkmidcll2qirbu7ykg46eyd.onion/"
r = requests.get(url)


We now have a way to automate making requests.

This post is licensed under CC BY 4.0 by the author.