Skip to content

Latest commit

 

History

History
86 lines (65 loc) · 3.8 KB

debugging.md

File metadata and controls

86 lines (65 loc) · 3.8 KB
Raw

Debugging

Logs

There are a few different logs that may be useful when debugging:

  • Server log: By default this is on the server in OUTPUT_DIR/turbinia-server.log
  • Worker logs: By default this is on the workers in OUTPUT_DIR/turbinia-worker.log
  • Task logs: These are logs that are generated by a Task running in the Worker. If you have GCS enabled, then these logs are saved there, and in either case, the paths can be determined with turbiniactl -a status (note that -a is required to show all data associated with the tasks).
  • Task execution logs. These are the logs that are generated by binaries executed from a Task. For example, Plaso generates its own log file, and that gets saved by the Task. As with the Task logs, these can be stored in GCS and retrieved with gsutil.

Task Debugging

Task Failures

The following is an example of debugging a task failure:

Running turbiniactl, we see a failure of the PsortTask:

$ ./turbiniactl status -d1

Retrieved 2 Task results:
2017-12-06T15:09:15.013Z PsortTask Failed: Execution failed with status 1
2017-12-06T15:08:49.616Z PlasoTask Successful: Completed successfully in 0:00:14.178579 on aaronp.dev

We can specify -a to get all info we have about that Task, and this will include the request Id, Task Id, and the other logs associated with the task.

$ ./turbiniactl -a status -d1

Retrieved 2 Task results:
2017-12-06T15:09:15.013Z request: None task: 1f8c4b321f444614bde296b0a00bb91f PsortTask Failed: Execution failed with status 1
        gs://my-turbinia-bucket/output/1512601742-1f8c4b321f444614bde296b0a00bb91f-PsortTask/worker-log.txt
2017-12-06T15:08:49.616Z request: b94ad420cf5a483fb3fc4409d5fc6dcd task: 020fe728a2b64dc7ba1da2656e1cc454 PlasoTask Successful: Completed successfully in 0:00:14.178579 on aaronp.dev
        gs://my-turbinia-bucket/output/1512601704-020fe728a2b64dc7ba1da2656e1cc454-PlasoTask/020fe728a2b64dc7ba1da2656e1cc454.log
        gs://my-turbinia-bucket/output/1512601704-020fe728a2b64dc7ba1da2656e1cc454-PlasoTask/worker-log.txt

Note that there may be local paths that show up in the turbiniactl status output, and these are local to the Worker that executed that Task.

Using the saved paths above we can display the output with gsutil.

$ gsutil cat gs://my-turbinia-bucket/output/1512601742-1f8c4b321f444614bde296b0a00bb91f-PsortTask/worker-log.txt

Running psort as [psort.py --status_view none --logfile /var/tmp/1512601742-1f8c4b321f444614bde296b0a00bb91f-PsortTask/1f8c4b321f444614bde296b0a00bb91f.log -w /var/tmp/1512601742-1f8c4b321f444614bde296b0a00bb91f-PsortTask/1f8c4b321f444614bde296b0a00bb91f.csv /var/tmp/1512601730-b04f479ef9094954968474431aa30e8a-PsortTask/b04f479ef9094954968474431aa30e8a.csv]
Execution failed with status 1

Disappearing Tasks

If the Turbinia server is scheduling Tasks, but you're not seeing the Tasks or their output for some reason, the most common scenario is that either a second Turbinia server or a second set of Workers is running and operating on the same PubSub queues. Make sure that you have unique values per each Turbinia instance for the following config variables: PSQ_TOPIC, INSTANCE_ID, PUBSUB_TOPIC and KOMBU_CHANNEL (if you are using a local Turbinia installation type.

Common Errors

  • If you encounter a cycle of errors on the Turbinia server when decoding specific malformed pubsub messages, and the failure happens prior to the server acknowledging the pubsub message, you may need to manually clear the pubsub queue. If you are sure there aren't other outstanding requests, you can just pull and auto-ack the existing messages, otherwise you may need a variation of this command to selectively ack messages:
gcloud beta pubsub subscriptions pull --auto-ack $my-pubsub-subscription-name