Skip to content

Latest commit

 

History

History
205 lines (148 loc) · 9.33 KB

troubleshooting.adoc

File metadata and controls

205 lines (148 loc) · 9.33 KB
Raw

Troubleshooting

The many different steps makes the GRR server installation process susceptible to errors. Here some common issues and solutions are described.

ImportError: cannot import name jobs_pb2 or similar

If you see "ImportError: cannot import name jobs_pb2" or a similar error for any other _pb2 file, you need to regenerate the protobuf files. Just run

python setup.py build
sudo python setup.py install

The upstart/init.d scripts show no output

When I run an init.d script e.g. "/etc/init.d/grr-http-server start" it does not show me any output.

Make sure that the "START" parameter in the corresponding default file, e.g. "/etc/default/grr-http-server", has been changed to "yes".

I cannot start any/some of the GRR services using the init.d scripts

When it run an init.d script e.g. "/etc/init.d/grr-http-server start" it indicates it started the service although when I check with "/etc/init.d/grr-http-server status" it says it is not running.

You can troubleshoot by running the services in the foreground, e.g. to run the HTTP Front-end server in the foreground:

sudo grr_server --start_http_server --verbose

Any/some of the GRR services are not running correctly

Check if the logs contain an indication of what is going wrong.

Troubleshoot by running the services in the foreground, e.g. to run the UI in the foreground:

sudo grr_server --verbose --start_ui

Cannot open libtsk3.so.3

error while loading shared libraries: libtsk3.so.3: cannot open shared object file: No such file or directory

The libtsk3 library cannot be found in the ld cache. Check if the path to libtsk3.so.3 is in /etc/ld.so.conf (or equivalent) and update the cache:

sudo ldconfig

Cron Job view reports an error

Delete and recreate all the cronjobs using GRR console:

aff4.FACTORY.Delete("aff4:/cron", token=data_store.default_token)
from grr.server.aff4_objects import cronjobs
cronjobs.ScheduleSystemCronFlows(token=data_store.default_token)

Protobuf 2.5.0

GRR only works with the Protobuf library v2.5.0 or newer. If your distribution comes with an older version, follow the instructions below:

Remove any version of protobuf that is older than 2.5.0.

Note that this will also remove a couple of dependencies:

sudo apt-get remove libprotobuf python-protobuf

To build Protobuf from source, install the following packages on a Debian-based system:

sudo apt-get install build-essential uuid-dev zlib1g-dev python-dev

Download protobuf version 2.5.0 or later from protobuf

Extract the package:

tar xfv protobuf-2.5.0.tar.gz
cd protobuf-2.5.0

Build and install libprotobuf:

./configure --prefix=/usr/
make
sudo make install
sudo ldconfig

Build and install the Python bindings:

cd python
python setup.py build
python setup.py test
sudo python setup.py install

Missing Rekall Profiles

If you get errors like:

Error loading profile: Could not load profile nt/GUID/ABC123...
Needed profile nt/GUID/ABC123... not found!

when using rekall, you’re missing a profile (see the Rekall FAQ and blogpost for some background about what this means).

The simplest way to get this fixed is to add it into Rekall’s list of GUIDs, which is of great benefit to the whole memory forensics community. You can do this yourself via a pull request on rekall-profiles, or simply email the GUID to [email protected]. Once it’s in the public rekall server, the GRR server will download and use it automatically next time you run a rekall flow that requires that profile. If your GRR server doesn’t have internet access you’ll need to run the GetMissingProfiles function from the GRR console on a machine that has internet access and can access the GRR database, like this:

from grr.server import rekall_profile_server
rekall_profile_server.GRRRekallProfileServer().GetMissingProfiles()

I don’t see my clients

Here are common client troubleshooting steps you can work through to diagnose client install and communications problems.

Check that the client got installed

You should have something like:

  • C:\Windows\System32\GRR\3.1.0.2\GRR.exe on windows;

  • /usr/lib/grr/grr_3.1.0.2_amd64/grrd on linux; and

  • /usr/local/lib/grr_3.1.0.2_amd64/grrd on OSX

with variations based on the GRR version and architecture you installed (32bit is i386). If you didn’t get this far, then there is a problem with your installer binary.

Check the client is running

On linux and OS X you should see two processes, something like:

$ ps aux | grep grr
root       957  0.0  0.0  11792   944 ?        Ss   01:12   0:00 /usr/sbin/grrd --config=/usr/lib/grr/grr_3.1.0.2_amd64/grrd.yaml
root      1015  0.2  2.4 750532 93532 ?        Sl   01:12   0:01 /usr/sbin/grrd --config=/usr/lib/grr/grr_3.1.0.2_amd64/grrd.yaml

On windows you should see a GRR Monitor service and a GRR.exe process in taskmanager.

If it isn’t running check the install logs and other logs in the same directory:

  • Linux/OS X: /var/log/grr_installer.txt

  • Windows: C:\Windows\System32\LogFiles\GRR_installer.txt

and then try running it interactively as below.

Check the client machine can talk to the server

The URL here should be the server address and port you picked when you set up the server and listed in Client.control_urls in the client’s config.

wget http://yourgrrserver.yourcompany.com:8080/server.pem
# Check your config settings, note that clients earlier than 3.1.0.2 used Client.control_urls
$ sudo cat /usr/lib/grr/grr_3.1.0.2_amd64/grrd.yaml | grep Client.server_urls
Client.server_urls: http://yourgrrserver.yourcompany.com:8080/

If you can’t download that server.pem, the common causes are that your server isn’t running or there are firewalls in the way.

Run the client in verbose mode

Linux: Stop the daemon version of the service and run it in verbose mode:

$ sudo service grr stop
$ sudo /usr/sbin/grrd --config=/usr/lib/grr/grr_3.1.0.2_amd64/grrd.yaml --verbose

OS X: Unload the service and run it in verbose mode:

$ sudo launchctl unload /Library/LaunchDaemons/com.google.code.grr.plist
$ sudo /usr/local/lib/grr_3.1.0.2_amd64/grrd --config=/usr/local/lib/grr_3.1.0.2_amd64/grrd.yaml --verbose

Windows: The normal installer isn’t a terminal app, so you don’t get any output if you run it interactively.

  • Install the debug dbg_GRR_3.1.0.2_(amd64|i386).exe version to make it a terminal app.

  • Stop the GRR Monitor service in task manager

Then run in a terminal as Administrator

cd C:\Windows\System32\GRR\3.1.0.2\
GRR.exe --config=GRR.exe.yaml --verbose

If this is a new client you should see some 406’s as it enrols, then they stop and are replaced with sending message lines with increasing sleeps in between. The output should look similar to this:

Starting client aff4:/C.a2be2a27a8d69c61
aff4:/C.a2be2a27a8d69c61: Could not connect to server at http://somehost:port/, status 406
Server PEM re-keyed.
sending enrollment request
aff4:/C.a2be2a27a8d69c61: Could not connect to server at http://somehost:port/, status 406
Server PEM re-keyed.
sending enrollment request
aff4:/C.a2be2a27a8d69c61: Could not connect to server at http://somehost:port/, status 406
Server PEM re-keyed.
aff4:/C.a2be2a27a8d69c61: Sending 3(1499), Received 0 messages in 0.771058797836 sec. Sleeping for 0.34980125
aff4:/C.a2be2a27a8d69c61: Sending 0(634), Received 0 messages in 0.370272874832 sec. Sleeping for 0.4022714375
aff4:/C.a2be2a27a8d69c61: Sending 0(634), Received 0 messages in 0.333703994751 sec. Sleeping for 0.462612153125
aff4:/C.a2be2a27a8d69c61: Sending 0(634), Received 0 messages in 0.345727920532 sec. Sleeping for 0.532003976094
aff4:/C.a2be2a27a8d69c61: Sending 0(634), Received 0 messages in 0.346176147461 sec. Sleeping for 0.611804572508
aff4:/C.a2be2a27a8d69c61: Sending 0(634), Received 8 messages in 0.348709106445 sec. Sleeping for 0.2

If enrolment isn’t succeeding (406s never go away), make sure you’re running a worker process and check logs in /var/log/grr-worker.log.