Move back out of tools.wmflabs.org

[hartman]

Due to the enablement of CSP, and the stricter code loading policies that in the future will start being implemented, I suggest moving this code back into meta.wikimedia.org, as tools.wmflabs.org is not a trusted domain within the WM ecosystem.
See also https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Huge_Script_errors_in_console_log

[Pathoschild]

That's a pretty significant change. While it's trivial to copy the scripts to Meta (presumably in the MediaWiki namespace), there are countless references to the scripts across all wikis that would need to be updated.

[hartman]

Yeah, let's also be a bit careful here in terms of effort. There are some ideas of alternative solution strategies. Might be best to privately communicate with Bawolff .

[MarcoAurelioWM]

I agree this is a very important decision to make. I feel this script suite is better maintained here, but maybe to bypass CSP we could get Phabricator Diffusion (phabricator.wikimedia.org) to keep a clone of this repo and have mw.loader.loaded from there? If we want to explore that path, I can create repositories on Phabricator and try to test that.

[Pathoschild]

@MarcoAurelioWM That might be a good middle approach. That lets us keep deploying updates through Git, but WMF has the ability to freeze or remove the mirror if needed. Would Diffusion mirror the upstream repo automatically, or would someone need to do it manually each time?

[MarcoAurelioWM]

@Pathoschild Indeed you can keep the tool development here in GitHub and just ask Phabricator Diffusion to watch this GitHub repo for updates. In that case Phabricator Diffusion will simply watch from time to time, automatically, if there have been any updates upstream (this GitHub repository) and if there's any, Phabricator will import them (it polls, as it's described here)

For example, I keep the configuration files of my bot here at GitHub (https://github.com/MarcoAurelioWM/MABot-jobs) and I have a mirror of said code at Phabricator (https://phabricator.wikimedia.org/diffusion/TMAB/). Once I merge some code at my GitHub repo, Phabricator will automatically check if there has been updates and will import the code.

Like I've said, the updates happen automatically but not immediatelly. Phabricator does fetch updates (polls) from repos depending on its activity, to better spend its resources (cfr. help link above). That is, for the most active repositories (for example, MediaWiki core) Phabricator tries to fetch updates each 15 seconds. For more quiet repositories, Phabricator can take up to 6 hours to go and look if there has been any change. You can, however, order Phabricator to update the repository at any given time using a button in the repository.

Phabricator can also be used to host repositories locally as we use Gerrit or GitHub. In that case updates happen directly on push, but Phabricator is --IMHO-- currently not very user friendly when it comes to use it as a platform for code review and repository hosting, as it requires depending on several weird tools such as Arcanist for Differential to upload changes for code review and merge them (there's no way to merge commits via the UI, the CI application is still very rough, etc.).

Hope that this helps. In case of further questions, please let me know. Thanks.

Edit with regards to freezing and deleting Phabricator repositories: yes, Phabricator repositories can be marked as inactive. That means Phabricator will stop monitoring for updates. Deleting Phabricator repositories, however, is not doable via the GUI, and needs someone with access to the server to do it (currently only a handful of people IIRC).

[MarcoAurelioWM]

One problem so far, however, is that I'm not sure if Phabricator "raw" files can be used. Last time I tried to use a "raw" file the link expired after some time, rendering the link useless. If that is still happening then there's no point in trying to mirror this code there. I'll ask around.

[MarcoAurelioWM]

As far as I could see the problem still exists. File links are all temporary thus cannot be used as a solution for this I'm afraid 😿

[Pathoschild]

That's too bad. Something like Toolforge CDNjs but for community GitHub repos would be useful.

[MarcoAurelioWM]

Yes. I agree, but it seems that the static resources served via phab.wmfusercontent.org do all expire after some time. Plus it is also very possible that wmfusercontent is also CSP-blocked so we wouldn't achieve the CSP bypassing. With permission, I'm going to ping @20after4. He's the Wikimedia Phabricator expert and he will surely know better with regards to the proposed approach (#113 (comment)) and if that's possible.

[Pathoschild]

@MarcoAurelioWM Have you heard anything back from the tech team on the above question? (Also, wouldn't it make sense to just allow *.wmflabs.org in the policy, since it's a Wikimedia domain too?)

[MarcoAurelioWM]

Hello. I've not heard anything, nope. I can try to ask again if that helps.

[Pathoschild]

This was brought up again in Phabricator tickets T304151 and T304208. Seems moving the scripts to wiki pages is still the only solution, so I started a discussion on Meta about the best place to put them.

[20after4]

The wmfusercontent domain is intentionally separate from phabricator.wikimedia.org and allowing it in the csp would defeat the purpose of having a separate domain. The reason for it to exist is to avoid XSS exposure due to allowing users uploading malicious code to Phabricator.

[Pathoschild]

The discussion on Meta was archived with no clear decision on where the scripts should be now.