WIP fix derive logic and redundant hash calc

[Dobatymo]

This addresses #253 and #288

Counts all files (even those which might be skipped). This is consistent with the newly added behavior for file-level metadata.

BUG: if the last file is skipped, then a derive is never queued

Any idea how to solve this? Is there a way to queue a derive WITHOUT uploading a file?
The easiest way would be to just upload everything and skip whatever you like, and then after everything is completed, simple do queue a derive with a separate call.
Then we could do away with this fragile file counting.

[JustAnotherArchivist]

Is there a way to queue a derive WITHOUT uploading a file?

Related: #252

This would be nice to have in general. I wrote a script for myself last year since I didn't want to click around on the web interface all the time as I'm using --no-derive to avoid hitting those two bugs you're trying to solve as well as wanting to be sure everything's fine before initiating the derive that can take a very long time. I'm not sure if that works for non-admin accounts though, and it should most definitely not be used as a basis for an implementation here as it emulates the website interaction, so I'm not going to link it here.

[jjjake]

Yes, it's possible to queue a derive task without uploading anything:

$ ia tasks jj-test-2020-05-14 --cmd derive.php

Python:

>>> r = item.derive()

[Dobatymo]

Great, I will use this instead of the counting logic.

[Dobatymo]

I removed all counting logic and just call derive() in the end. Do you think that is an acceptable solution?
The tests don't seem to like it though. If you think we should chose this solution. I will have a look at the tests.

[jjjake]

No, I think the derive task should still be queued in the upload request. I'd like to avoid submitting an extra request to another API if possible.

[JustAnotherArchivist]

@jjjake

--cmd derive.php

Could this be added to the documentation? It only mentions make_dark.php and make_undark.php currently: https://archive.org/services/docs/api/tasks.html#supported-tasks. Maybe also an example in the ia tasks help.

[soredake]

What's the status of this?

[Dobatymo]

@soredake jjjake didn't like the approach to have a seperate API call to queue the derive. Also at the time the unittests failed for some reason. I haven't looked at this PR for the past few years and now there are merge conflicts.