Feedback wanted on v2 features and improvements

[jjjake]

I will be working towards a v2 release in the coming months.

I would love feedback/recommendations from anyone who has em'! Please feel free to create a new issue, or reply here. I will be creating new issues myself, as I start working on or considering new features and improvements.

Thanks!

[trumad]

When a file can't be uploaded (for example, an invalid zip file that the internet archive rejects), it would be better if the uploader could simply skip to the next file and keep going. The current behaviour is that it stops entirely. Then I have to delete the offending file, and start the upload all over again. It would be better if IA upload could keep a list of failed files, continue uploading any files it can upload, and then spit out a list of failed files & error messages at the end. Please let me know if you'd like more details. Thanks for your work on this!

[vxbinaca]

Requested flags:

• Manually add item to derive que after all other operations are complete.
  (I have to do this from time to time to fix Twitch rips made too early before Twitch fully processed the stream. Spacing out the rips in my cronab for them has mostly allowed me to avoid this.)

I wanted to thank staff for allowing VTT subtitles to display (no more clunky manual conversion on my end), as well as the UI enhancements. WebGL rendering of STL files was much appreciated as well due to upcoming projects I have involving physibles. Thank you!

[p8NG6L]

I would like to second the request to provide a way to queue a derive task manually. I just created #252 before noticing this issue.

[vxbinaca]

Oh gimme a way to dark items I have full admin over. Unless you can do that already.

[jjjake]

@vxbinaca Yes, I'd like to add the ability to dark/undark items. We'll need to make a few changes server-side to support this (as well as submitting derives). I hope to get it in the v2.0.0 release, but if not then ASAP. Thanks for the recommendation.

[vxbinaca]

Disregard my last ideas I just need to actually use the tool more deeply (which I've delved deeper into recently), because just running ia delete [IDENTIFIER] does what I wanted anyway.

• 'manage' for doing darks/undarks/generally managing the items, this also goes in line with things you see on the site.

ex:

ia manage [IDENTIFIER] --dark --comment "Not in line with collection subject"

I figure manage because on the web interface you see that word being used. It's up to you whether to error out of a dark or undark without comment but I try to comment my darks.

[vxbinaca]

What would it take to be able to set items to log-in only? Currently you have a collection for such items but that should be a thing we set for more sensitive items. Why can't we set this as a flag for items we have full admin over?

Ex: Youtube has a inconsistent policy on brief nudity. If I were to get a video that had that I'd like to make it log-in only. Some Source Filmmaker material (like 3D movies made in a program) I want to display is of an semi-adult nature. I figure to avoid conflict it would be easier to wall off the items behind log-in only.

[jjjake]

@vxbinaca I believe an item simply needs to be added to the loggedin collection for it to be only viewable by logged-in users (I'm not certain, but I'm pretty sure that's how it works).

ia is able to do this right now. You might run into permissions issues, but there's not much I can do about that. Please send any perms or policy questions to [email protected].

[vxbinaca]

Option to not download derived files so I don't have to rm -rf *.ogv *.sqlite etc etc

[jeffgerhard]

If you are still looking for feature requests, I would love to have a way to assert title pages when uploading text items. But I think this is impossible to do in internetarchive because it's not in the underlying item architecture. Correct me if I'm wrong!

Anyhow, to get around this, I wrote a quick thing today to handle my use case. I get around this limitation by downloading scandata.xml files, editing, then re-uploading. But it's still suboptimal since it has to run after the items have already been uploaded and gone through a derive process.

It's not critical, but it would be cool to figure out a way to do this more efficiently when uploading large numbers of documents that have a predictable structure.

[funkyfuture]

while the docs are good, they could be better. for example i found the possibility to set "REMOVE_TAG" as metadata value in the code, but not in the docs. i also have the impression that the current api.rst is rather a good guide, but a complete API documentation is missing.

two ideas for enhancements:

• accept pathlib.Path objects where appropriate
• having a proxy mapping for metadata bound to Item.metadata, very roughly like this:

class ItemMetadata(collections.abc.Mapping):
    def __init__(self, item: Item): ...
    def __delitem__(self, key: str): self.__setitem__(key, "REMOVE_TAG")
    def save(self): ...  # instead of Item.modify_metadata

imo some interfaces are unnecessarily complex, e.g. one method for file uploads is enough with this signature:

class Item:
    def upload_files(*files, …): …