Skip to content

Installing 4CAT

Jump to bottom
Dale Wahl edited this page Jul 21, 2022 · 78 revisions

Install and run 4CAT

You can install 4CAT on your local machine or a server. This can be useful if you want to capture data from various online platforms, or analyse data you've captured previously. This page describes how you can install 4CAT and run it. Here you can review a full list of available data sources available.

Please note that at this time, scraping 4chan, 8chan, and 8kun require Sphinx for text search. You can follow these instructions to set up Sphinx. For some data sources, you will additionally need to provide keys for the API from which the data is retrieved.

Install 4CAT via Docker

The recommended method is to use Docker.

  1. Install Docker Desktop, and start it. Note that on Windows, you may need to ensure that WSL (Windows Subsystem for Linux) integration is enabled in Docker. You can find this in the Docker setting in Settings -> Resources-> WSL Integration -> Enable integration with required distros.

Simple: Use Docker Hub Image

  1. Download our docker-compose_prod.yml file located here and our .env file located here (you can use your browser's "Save page as" to download these files*).
    • Optionally edit the .env file to change database passwords, server name, or ports
    • *Sometimes downloading the .env file will be renamed env.txt which will not work. You can rename it yourself with the command mv env.txt .env or similar.
  2. Run docker-compose -f docker-compose_prod.yml up -d from the folder where you downloaded the above files
    • You may need to run this command as an administrator by either adding sudo to the beginning of the command (Mac/Linux) or right clicking your Command Prompt and using "Run as administrator" (Windows).
  3. Congrats! Open http://localhost:80 on your browser of choice

Common issues w/ Docker

  • http://localhost:80 opens an incorrect page such as the Docker welcome page (from their starter project). Try pressing F5 to reload or clearing your browser cache and then reloading the page.
  • Occasionally, on the first build, there is a bug where the frontend fails to start because the backend takes longer than normal to install. On successful installation, you should see three containers running in the Docker UI or running docker ps. If the frontend is missing, start that container again via the UI or run docker start 4cat_frontend.

Customizable: Use a local clone of 4CAT in order to customize files

Note: you can of course edit files directly in the Docker containers from the simple install, however, these will be lost if you rebuild the containers.

  1. Clone the 4CAT repository, or download the most recent release and unzip it.
  2. In a terminal/command prompt, navigate to the folder in which you just installed 4CAT (the folder that contains the docker-compose.yml file)
  3. Run the command docker-compose build to build from your local 4CAT files
  4. If this is the first time you're starting the Docker container, it will take a while for all components to be built.
  5. Run the command docker-compose up -d to start your container for the first time.
  6. Once this is done, you can access the 4CAT interface via http://localhost:80.
  7. You can now edit files locally and rebuild your 4CAT version via docker-compose down then docker-compose up --build -d (or just edit directly in the docker containers, restarting the backend via python3 4cat-daemon.py restart).

Note: if your computer/server is already using some of the same ports that Docker wishes to use, you can modify the .env file in the home directory and change the ports that Docker uses. Any modifications to configuration files will require you to rebuild the docker images with docker-compose up --build.

Install 4CAT via Docker onto a server

With Docker, 4CAT is set to host itself on localhost:80 by default. This can be modified in the .env file located in the main directory. If you would like to make 4CAT externally accessible, it is necessary to change SERVER_NAME to your domain or IP address. This will add it to the whitelists in config.py-example file. You could also redirect to 4CAT via another service such as Apache or Nginx.

# Modify SERVER_NAME and/or PUBLIC_PORT to make 4CAT available externally

SERVER_NAME = 4cat.example.com # You could also use your server's IP address
PUBLIC_PORT = 80

# This example would allow you to navigate to http://4cat.example.com (or http://4cat.example.com:80) and access your version of 4CAT

Install 4CAT manually

If you cannot or don't want to use Docker, you can run 4CAT directly from the code rather than via Docker. This requires more set-up and the manual installation of various dependencies, but can be useful if you want to develop data sources or processors for 4CAT.

Requirements

It is recommended that you run 4CAT on a UNIX-like system (e.g. Linux or MacOS). It will also run under Windows, but the instructions below are written with a UNIX-like in mind. 4CAT further requires Python 3.8 and PostgreSQL 9.5. Lower versions of either may work, but are not officially supported.

Installation

Clone the repository somewhere:

git clone https://www.github.com/digitalmethodsinitiative/4cat.git

After cloning the repository, copy config/config.ini-example to config/config.ini and edit the file to match your machine's configuration. The various options are explained in the file itself:

cd 4cat/config
cp config.ini-example config.ini
nano config.ini

Next, install the dependencies. On Linux systems that use apt, the following should suffice:

apt install python3-pip libpq-dev python3-dev postgresql-server-dev-all unzip postgresql-client

Adapt these to your own package manager (e.g. yum or brew) as necessary. From here on, we are working within Python, so it is recommended you create a virtual environment to install Python packages and run 4CAT in. There are several ways to set up a virtual environment, the link earlier in this paragraph lists the best practices.

Within your virtual environment, while in the 4CAT root folder, install the required Python packages:

pip3 install -r requirements.txt

Some of the dependencies may have their own dependencies. For instance, on Windows the pyahocorasick library needs to Microsoft Visual C++ Build Tools to be installed. If you encounter similar issues, please file an issue!

Next, you should make sure a database is available for 4CAT. 4CAT requires a PostgreSQL database to store dataset metadata, the job queue and other assorted data. You should create the database yourself, and add the database login details to config.ini. After doing so, run the following command to create the tables, indices, et cetera, required by 4CAT:

psql --user=[username] --dbname=[database name] < backend/database.sql

Replace [username] and [database name] with the relevant values. You may be prompted for a password.

Finally, to make sure everything is in working order, run the following command and follow the instructions:

python3 helper-scripts/migrate.py

You can now run 4CAT!

Running the backend

The backend is run as a daemon that can be started and stopped using the included 4cat-daemon.py script:

python3 4cat-daemon.py start

Other valid arguments are stop, restart and status. Note that if you change any configuration options, you will need to restart the daemon for the changes to take effect. For development/testing it may be helpful to run 4cat-daemon.py interactively with the -i switch (i.e., python3 4cat-daemon.py -i start). This will log output to the terminal as well.

Note: The 4CAT daemon was made to run on a UNIX-like system and the above will not work on Windows. On Windows, the 4CAT daemon will always run interactively, and can be quit by entering 'q' and pressing Enter.

4CAT logs to 4cat.log in the root folder by default.

Running the web tool

The web tool is a Flask app. It is recommended that you run the web tool as a WSGI module: see the Flask documentation for more details. For testing and development, you can run the Flask app locally from the command line. For Mac:

FLASK_APP=webtool flask run

For Windows:

set FLASK_APP=webtool
flask run

With the default configuration, you can now navigate to http://localhost:5000 where you'll find the web tool that allows you to query the database and create datasets.

Creating your first user

Finally, you will likely want to be able to create one or more users. The is a helper script to create your first user. Additional users can be create with the script or in the Control Panel on the frontend website.

python3 helper-scripts/create_user.py -u admin -e -p WHATEVERPASSWORDYOUWANT -a

  • -u username, in this case "admin"
  • -p password for user
  • -e no email for this user
  • -a makes the user an admin

You may wish to modify some 4CAT settings

Most settings are now accessible in the 4CAT database in the settings table. You can update settings via the 4CAT web interface if you are an admin user by navigating the the "4CAT Settings" tab at the top of your browser.

If you are unable to navigate to the 4CAT web interface with default settings (e.g., if you are deploying 4CAT on a server instead of localhost), you can also modify settings directly in the database.

  1. Connect to psql
    • psql --user=[username] --dbname=[database name] (changing username and database name as appropriate)
  2. View settings
    • SELECT * FROM settings;
    • or perhaps SELECT * FROM settings WHERE name LIKE '%flask%'; to view flask settings
  3. Update as needed
    • UPDATE settings SET value = '"my.server.com"' WHERE name = 'flask.server_name';
    • UPDATE settings SET value = '["localhost", "my.server.com"]' WHERE name = 'flask.autologin.hostnames';

🐈🐈🐈🐈

Clone this wiki locally