WebFile redirecter #437
WebFile redirecter #437
Conversation
|
This url switching is thus driven by a csv file I need to manually build after inspecting each package's chocolateyinstall.ps1 finding the url and mapping it to an alternate url? I suppose this is scriptable to generate a csv, but does this highlight a bigger issue with urls within packages? It kills package portability. Why wouldnt upstream adhere to using a url mapper too... packageid, url. Btw neat take on the approach!! I want to opensource my choco repo but do it in a way the pkgs could be used by someone w/o mods. I was originally planning to make the hostname just an env var... but this wont help upstream internet based urls plus you could have foo.com/setup.exe and bar.com/setup.exe I am interested in ppls thoughts... |
|
While I like the idea of re-directing to a different URL when required, I am wondering what the exact use case is for this. Does the recent work that @ferventcoder did to first check the download cache to see if a file is already downloaded, before accessing the URL in the nuspec file cover a lot of the use cases? How I was planning on using it was the following... On one machine that has access to the interwebs, install all the packages that you need. Mirror the Chocolatey Packages that you want to somewhere internal, i.e. ProGet, or a file source. Deploy the Cache folder from the interwebs facing machine onto all other machines. Perform all installs on other machines using -source parameter, and you have a full offline install. Granted, there is a bit of work up front to make this happen, but once you do it once, it's done. What is the use case of the URL re-direct? Is it required now that Chocolatey first checks whether the required file has already been downloaded? |
|
The use case starts with a skilled admin user installing the package which downloads the app from a hard-coded Internet URL onto their own (Internet connected) workstation. After checking it works as intended & incidentally virus scanning it, they would (yes, manually) update the CSV file - there is a hint in the console/chocolateyInstall.log for beginners. They also have to dump the orginal or redownloaded binary near the CSV as suggested or at a random other location of their choice. Now other boxes can use the unaltered package without Internet access - chocolatey will now find the url in the CSV it always checks and so fetch from the alternate url. It's not so much targeting already downloaded on this one box, as for whether it's already been downloaded & blessed in the whole organisation. Partly this is for simplicity - wanting to avoid getting in WebDAV for HTTP uploads and keeping the code and cache format simple (compared to my work based on MRJ's fork). Partly as security policy at my firm & I suspect others is that "new stuff from the Internet" must only be used via a conscious, non-automated action (no automagic 'it just does it all for you'). The final steps of uploading it to be widely available in the cache folder & updating a CSV can both be secured and audited easily - e.g. stored in git or a repository like ProGet or Artifactory that we would also use for the Nuget source. Yes, this would allow us & other companties to open source more packaging work as we'd embed Internet Urls (not imposing any url mapping on others) in the scripts on github but redirect them when used locally. We'd also be able to guarentee supply - even if the original url was broken - as it's modeled as a side-loading repository, not an opaque cache that someone might "clean out" without a backup. Finally it would allow semi-skilled admins to monkey-patch in extremis (e.g. slow package update after a security issue). Pre-populating a machine-local cache would also work, but that would mean a hell of a lot of filecopying - e.g. our SQL Server package uses a 2.5 GB download, and many others are 100's of MB - for each version. That would be pushed to every C drive of every box, want it or not. I'd much rather a pull model and prefer HTTP over SMB file shares though accept others may not. I'm prepared to believe not everyone has the security angle I do - but if so, just set up all the boxes with a web proxy server like squid/varnish/whatever & be done with it. That helps them get to the Internet in general, and you can add caching to the proxy if you worry about Internet bandwidth - simpler & more efficient than any Chocolatey caching system discussed so far. |
|
I've said something similar on the forum already: I think this is the right way to do redirection, but I'd like to see a good place to hang other redirection schemes (like a file based cache, a rewrite rule, or even a call out to another script). I think that's mainly about breaking out the CSV specific code and making Get-RedirectedWebFileUrl a 'switch' to call other methods. I don't think it would take much to get there from here, and I'm willing to do some of the work. |
|
We could rename Get-RedirectedWebFileUrl to "Map-CSVRedirectedWebFileUrl", change to a ref param for $url and return a boolean: $true if mapped. Then make the addition in Get-ChocolateyWebFile call a "Map-ChocolateyUrl" similar to your commit, which daisy chains calls to Map-CSVRedirectedWebFileUrl and other similar functions until one returns $true or runs out. (I'm assuming it would be winner takes all in a fixed order, rather than calling all mappers always.) For the smart-script based idea, I haven't tried it but was the point of the extensions functionality to allow packages ending with ".extension" to contain a psm1 which could override any existing chocolatey framework/helper method? So if I made a "mycompany-redirecter.extension" package, I could embed in it a script to override Map-ChocolateyUrl & do whatever I want. Not sure if it's possible to still call the original from the extension, but splitting Map-ChocolateyUrl from Map-CSVRedirectedWebFileUrl means you could get that functionality still if you wanted it. |
|
Your first paragraph all sounds good to me. I don't know about the extension thing, but if it works as you say then adding a 'null' mapper at the start of the chain would work nicely, e.g. Map-UserRedirectedUrl it would just return $false. |
|
I'd only suggest, instead of complicating things for the caller by using a ref parameter, to return a PSObject with two properties: [string/uri] Url and [bool] Redirected. Or, even simpler, return redirected url or $null if not redirected. |
|
@jberezanski agree, returning a PSObject, which can be inspected (and easily extended if required) definitely makes a lot of sense. |
|
Updated broadly as discussed, except the custom PSObject only has one property, newUrl, and it will overwrite the current one if non-null, rather than using a second boolean. Also, extension methods don't seem to be able to override existing helper methods, so I added a way to run an extra method if defined, e.g. in an extension. And I had to switch back to "Get-" rather than "Map-" as the latter is not a Microsoft Approved Verb. |
Allow switching URLs in packages for locally-controlled equivalents in Get-ChocolateyWebFile.
This enables packages to install without direct Internet access, and manual caching.
Enabled via the environment variable "ChocolateyWebFileRedirecterCsv", which should point to a uri or filepath of a CSV file in UTF8 (or ASCII) form containing "url,redirectedUrl" columns and one url mapping per row.