add centralized logging VM

[redshiftzero]

Fixes #19, integrating the logging service @kushaldas wrote (which works beautifully based on qubesbuilder.BuildLog) using a new centralized logging VM:

• Adds a non-networked sd-log AppVM which starts on boot
• Allows RPC calls to sd-log from any sd-workstation tagged VM

Testing

1. make all should provision the sd-log VM
2. You can test this functionally using this branch from securedrop-client. When you run the client, you should see logs continue to appear in ~/.securedrop_client/logs/ as well as in the sd-log AppVM in ~/QubesIncomingLogs
3. new tests should pass (make test)

TODO after merge:

1. Add the qubes logging handler on all the Python subcomponents of the workstation
2. Go through each subcomponent/VM and audit the logging that is currently performed
   (will file more detailed followups about these things)

[redshiftzero]

(lint failure is due to a file that we're going to package up anyway so not fixing for now)

[redshiftzero]

(note that I still need to update the docs and rpm-build/SPECS/securedrop-workstation-dom0-config.spec for this)

[redshiftzero]

rebased on top of latest master and tested, all looks good except freedomofpress/securedrop-builder#107 needs addressing before this is ready for review

[redshiftzero]

@emkll expressed interest in looking at this in particular so let's hold off on merge until he gets a chance to review

[kushaldas]

The PR needs a rebase against master, behind 16 commits. That caused an endless loop of errors related to stretch apt repos for me. Was really confused why.

We should also add that sd-upgrade-templates sls dependency (which @emkll was working on) to the vm (after #372 gets merged).

[emkll]

Thanks @redshiftzero I've had a few issues setting up (see comments inline and error below).
I've built securedrop-client on the branch, but i suspect I am missing a wheel/package during the build:

• checkout centralized-logging branch, python3 setup.py dist
• checkout master is packaging repo, make securedrop-client
• the build completes without error but running the client fails:

user@sd-svs:~$ securedrop-client 
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 2f363b3d680e, init
INFO  [alembic.runtime.migration] Running upgrade 2f363b3d680e -> fecf1191b6f0, remove decryption_vs_content contraint
INFO  [alembic.runtime.migration] Running upgrade fecf1191b6f0 -> bafdcae12f97, Add files.original_filename
INFO  [alembic.runtime.migration] Running upgrade bafdcae12f97 -> 36a79ffcfbfb, add first_name, last_name, fullname, initials
INFO  [alembic.runtime.migration] Running upgrade 36a79ffcfbfb -> 86b01b6290da, add reply draft
Traceback (most recent call last):
  File "./bin/sd-client", line 5, in <module>
    from securedrop_client.app import run
  File "/opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client/app.py", line 38, in <module>
    from oqubeslogging import OQubesLog
ModuleNotFoundError: No module named 'oqubeslogging'
user@sd-svs:~$ 

Am I missing something when building the client deb?

[1] : https://www.qubes-os.org/doc/salt/
[2]: https://www.qubes-os.org/doc/resize-disk-image/

[redshiftzero]

thanks, will fix, rebase (likely on top of #372) then ping for re-review

[redshiftzero]

still testing but heads up, my expectation is that the branch above for securedrop-client is used in the staging environment in sd-svs (I haven't made that branch work with the packaged code as that handler will be (eventually) included as a dependency, but I can if using staging is too annoying)

[redshiftzero]

OK this should be ready for another spin

[emkll]

Thanks for the changes @redshiftzero , changes look good, tested as follows:

Clean install testing

• make all should provision the sd-log VM
• You can test this functionally using this branch from securedrop-client. When you run the client, you should see logs continue to appear in ~/.securedrop_client/logs/ as well as in the sd-log AppVM in ~/QubesIncomingLogs (see [1] below, but logging feature is functional 🎉 )
• new tests should pass (make test)
• make clean removes the rpc policies

Upgrade testing

• make sd-log sd-log VM
• You can test this functionally using this branch from securedrop-client. When you run the client, you should see logs continue to appear in ~/.securedrop_client/logs/ as well as in the sd-log AppVM in ~/QubesIncomingLogs (see [1] below, but logging feature is functional 🎉 )
• new tests should pass (make test)

I just realized after make clean that the dev-requirements in the client repo are out of sync [2], will open a PR over there .

Approving but not merging as @kushaldas requested changes
[1]:

user@sd-log:~/QubesIncomingLogs/sd-svs$ cat log_2019-12-16_22-07-29 
2019-12-16 22:07:29.563389 +0000 > starting log
2019-12-16 22:07:29.563875 +0000 sd-svs: Starting SecureDrop Client 0.0.9
2019-12-16 22:07:48.231374 +0000 sd-svs: Completed API call. Cleaning up and running callback.
2019-12-16 22:07:48.231457 +0000 sd-svs: user successfully logged in
2019-12-16 22:07:48.252780 +0000 sd-svs: Unrecoverable error
2019-12-16 22:07:48.252859 +0000 sd-svs: Traceback (most recent call last):
2019-12-16 22:07:48.252881 +0000 sd-svs:   File "/home/user/securedrop-client/securedrop_client/logic.py", line 259, in <lambda>
2019-12-16 22:07:48.252899 +0000 sd-svs:     lambda: self.completed_api_call(new_thread_id, success_callback))
2019-12-16 22:07:48.252917 +0000 sd-svs:   File "/home/user/securedrop-client/securedrop_client/logic.py", line 302, in completed_api_call
2019-12-16 22:07:48.252931 +0000 sd-svs:     user_callback(result_data)
2019-12-16 22:07:48.252948 +0000 sd-svs:   File "/home/user/securedrop-client/securedrop_client/logic.py", line 324, in on_authenticate_success
2019-12-16 22:07:48.252963 +0000 sd-svs:     self.api.journalist_first_name,
2019-12-16 22:07:48.252979 +0000 sd-svs: AttributeError: 'API' object has no attribute 'journalist_first_name'
2019-12-16 22:07:48.322761 +0000 > closing log

[2]https://github.com/freedomofpress/securedrop-client/blob/centralized-logging/dev-requirements.txt#L161

updated

[emkll]

LGTM, thanks!
Went through another round of local testing:

• make sd-log
• make remove-sd-log
• make sd-log
• ✔️ make test
• ✔️ test functionality with securedrop-client as described in the description