Configure without dependency to github

[wgorder]

@guybedford

I work in at a large enterprise where corporate proxies to github is a problem. Is there a way to get a successfull JSPM build without any need to talk to github? I see this as just an example:

demo-client  jspm install npm:angular-ui-bootstrap
     Looking up npm:angular-ui-bootstrap
     Updating registry cache...
     Downloading npm:[email protected]
     Looking up npm:angular
     Looking up npm:bootstrap
     Downloading npm:[email protected]

warn Main entry point not found for npm:[email protected].
     Adjust this property in the package.json or with an override, setting "main": false if this is the intention.

ok   Installed npm:bootstrap@^3.0.0 (3.3.5)
     Looking up github:jspm/nodelibs-path
     Looking up github:jspm/nodelibs-process
     Looking up github:jspm/nodelibs-fs
     Looking up npm:process
ok   Installed github:jspm/nodelibs-path@^0.1.0 (0.1.0)
     Looking up npm:path-browserify
ok   Installed npm:angular (1.4.4)
ok   Installed npm:[email protected] (0.0.0)
ok   Installed angular-ui-bootstrap as npm:angular-ui-bootstrap@^0.13.3 (0.13.3)
ok   Install tree has no forks.

ok   Install complete.

How can I configure things to make the below calls use npm instead?

ok   Installed npm:bootstrap@^3.0.0 (3.3.5)
     Looking up github:jspm/nodelibs-path
     Looking up github:jspm/nodelibs-process
     Looking up github:jspm/nodelibs-fs
     Looking up npm:process
ok   Installed github:jspm/nodelibs-path@^0.1.0 (0.1.0)

[wgorder]

Alternatively if there is a way I can make this work by pointing to an internal Stash repo instead of trying to go to github that would work too. NPM is mirrored inside via Artrifactory. If I cant make this work I probably have to rule out JSPM as an option and go another route unfortunately, so I was hoping you could help.

[guybedford]

@wgorder is the problem here explicitly the dependency on GitHub, or just getting the proxy to GitHub to behave properly?

Comprehensive proxy support is an issue at the moment I know, and has come up recently a few times.

[wgorder]

The problem is a hard dependency on github for our builds to work. We can work around the proxy issue in a couple ways but that too has been painful.

Ideally if it just a couple small registries that are required on github is there a way to clone those internally (We have Stash) and just keep them in sync with the upstream stuff on Github, and somehow tell JSMP to look there instead of Github?

The problem I am hearing is:

1. If github went away, or the network was unavailable all of our builds would fail.
2. Auditing etc. is harder when your depending on some outside resource. We basically mirror everything.

This is large enterprise shenanigans, but it is still a part of how things work for lots of us in this space.

[guybedford]

@wgorder the jspm registry itself can be cloned and mirrored fine, so the only issue would be mirroring the entire github: registry locally for jspm. This could certainly be done fairly simply, but would require some development work to set up, but basically would involve adding the feature to this registry adapter in jspm to support some type of local mirroring protocol that say first runs in a "connected mode", acting as a cached proxy to GitHub, and then enabling the ability to turn off the connectivity to GitHub all together.

In fact, the better way to design such a thing would be as a special registry adapter - mirror that could then be designed to mirror any registry in jspm. So I could run say jspm registry create github my-mirror, which would generate a new "mirror registry".

We haven't had to adapt to these sorts of scenarios yet, so such work would need to be driven by a company needing it.

[eladh]

Hi @guybedford ,@wgorder

We are also analyzing JSPM and our IT Department policy prohibits us from connecting to GitHub directly.
We are also fully integrated with NPM & Artifactory and all of our server /client modules obtained by the Artifactory (also Migrated from bower + npm 2.x -> npm 3)
In our case we don't going to ask for jspm:github packages.

I took the simple jspm-systemjs-babel-seed-master project :

And got the following output :

     Looking up npm:babel-core
     Looking up npm:babel-runtime
     Looking up npm:core-js
     Updating registry cache...
     Looking up github:jspm/nodelibs-process
ok   Installed babel-runtime as npm:babel-runtime@^5.1.11 (5.1.13)
ok   Installed babel as npm:babel-core@^5.1.11 (5.1.13)
     Looking up npm:process
ok   Installed github:jspm/nodelibs-process@^0.1.0 (0.1.1)
ok   Installed npm:process@^0.10.0 (0.10.1)
ok   Installed core-js as npm:core-js@^0.8.4 (0.8.4)
ok   Install tree has no forks.

warn [email protected] is unsupported for this version of jspm. Use jspm dl-loader --latest to update.

warn [email protected] is unsupported for this version of jspm. Use jspm dl-loader --latest to update.

warn [email protected] is unsupported for this version of jspm. Use jspm dl-loader --latest to update.
     Looking up loader files...
       system.js
       system.src.js
       system.js.map
       system-csp-production.js
       system-csp-production.js.map
       system-polyfills.js
       system-polyfills.src.js
       system-polyfills.js.map
       system-csp-production.src.js
     Using loader versions:
       [email protected]
ok   Loader files downloaded successfully
ok   Install complete.

@guybedford , as you mentioned before Comprehensive proxy support is going to be hard to implement both for JSPM and also in most large enterprise proxy won't be the way to go .

we need to find a way to decouple it , so - we want to use Artifactory & NPM

The main issues are :

1. As for the JSPM registry - I was trying to clone the repo to my local GIT Server and was looking to set
   My JSPM client to work with it - but I didn't found any configuration.
   Eventually I changed the URL repo hardcoded - but still without any luck , we still got connection errors. In the "Updating registry cache" phase.
2. Retrieving the - github:jspm/nodelibs-process , can we use NPM to retrieve ?
   will i was installing npm:bootstrap-select i got some more github depends like :

    Removing package files for npm:[email protected]
    Removing package files for github:jspm/[email protected]
    Removing package files for github:jspm/[email protected]
    Removing package files for github:systemjs/[email protected]

1. one last chance :
   Artifactory got support for Proxing & Caching remote VCS (like github) and they expose API to download the tarball (its a binary repository)
   for example : GET /api/vcs/downloadTag/github/jquery/jquery/2.0.1
   is there any way to integrate with artifactory VCS Support ?

source : https://www.jfrog.com/confluence/display/RTF/VCS+Repositories

thanks in advance,
elad :-)

[wgorder]

Yeah we did not have the time to dig in an try to make things compliant so we ended up going with webpack instead. FWIW we also use NPM with Artifactory mirror.

[eladh]

thanks @wgorder.

i really want to give it a chance and also i like the option to split between system.js + jspm and in webpack i can use just bundles even in development

@guybedford i will continue investigate the issue but for now is any way to do so.
as you mentioned before maybe custom registry adapter - mirror , is there any ref's you can give us how to develop this kind of support.

10x again,
elad.

[guybedford]

This would all be possible to do in a bit of configuration certainly. The easy way would probably be to create a custom github endpoint which redirects to custom git endpoint mirrors for the relevant repos. It does require knowledge of the registry api to set something like that up though, and would likely be an hour or so of work to do. I can't just go ahead and put that hour in though just yet unfortunately as I have other priorities right now.

[eladh]

thanks.

[eladh]

Hi @guybedford ,

Sorry to be a bother.

i looked up about your advice for a custom git endpoint and with : https://github.com/Orbs/jspm-git
when i replaced the jspm-github with jspm-git and cloned the nodeBuiltins i was able to work things out.

after installing the latest version i got the following error :

warn Error on getPackageConfig for npm:jspm/nodelibs-process
     TypeError: Cannot read property 'shasum' of undefined
         at NPMLocation.getPackageConfig (C:\Config\npm\node_modules\jspm\node_modules\jspm-npm\lib\npm.js:238:27)
         at C:\Config\npm\node_modules\jspm\lib\registry.js:117:30
         at lib$rsvp$$internal$$tryCatch (C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:493:16)
         at lib$rsvp$$internal$$invokeCallback (C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:505:17)
         at C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:1001:13
         at lib$rsvp$asap$$flush (C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:1198:9)
         at doNTCallback0 (node.js:428:9)
         at process._tickCallback (node.js:357:13)

err  Error: Error getting package config for npm:jspm/nodelibs-process.
         at C:\Config\npm\node_modules\jspm\node_modules\systemjs-builder\lib\builder.js:27:9
         at Object.lib$rsvp$events$$default.trigger (C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:245:13)
         at null._onTimeout (C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:779:47)
         at Timer.listOnTimeout (timers.js:92:15)

warn Error on download for npm:jspm/nodelibs-process
     TypeError: Cannot read property 'tarball' of undefined
         at NPMLocation.download (C:\Config\npm\node_modules\jspm\node_modules\jspm-npm\lib\npm.js:329:45)
         at C:\Config\npm\node_modules\jspm\lib\registry.js:117:30
         at lib$rsvp$$internal$$tryCatch (C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:493:16)
         at lib$rsvp$$internal$$invokeCallback (C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:505:17)
         at C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:1001:13
         at lib$rsvp$asap$$flush (C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:1198:9)
         at doNTCallback0 (node.js:428:9)
         at process._tickCallback (node.js:357:13)

err  Error: Error downloading npm:jspm/nodelibs-process.
         at C:\Config\npm\node_modules\jspm\node_modules\systemjs-builder\lib\builder.js:27:9
         at Object.lib$rsvp$events$$default.trigger (C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:245:13)
         at null._onTimeout (C:\Config\npm\node_modules\jspm\node_modules\rsvp\dist\rsvp.js:779:47)
         at Timer.listOnTimeout (timers.js:92:15)

any thoughts what i should do next ?

Best regards,
elad.

[guybedford]

Yes this is not a trivial process, and would need some involved work.

[karel1980]

For many projects I would be possible to replace all github: dependencies with npm: dependencies. The only repositories which do not have a release in npmjs are the jspm/nodelibs-* ones and systemjs/plugin-json one. @guybedford are you considering publishing those to npmjs.org (possibly with @jspm scope) eventually?

[frederikschubert]

I would also like to install the systemjs-hot-reloader from npm but cannot get it to work.

[frederikschubert]

@guybedford Would it be possible to publish the nodelibs-* packages to npm like @karel1980 suggested? Our installs are flaky because of connection problems with github and all npm packages are cached in our private npm registry.

[guybedford]

@frederikschubert yes it would be possible to switch the nodelibs- packages to npm sources. The jspm registry would still be linked to github though or is that not a problem for you?

[frederikschubert]

That would be alright, as the most problems occur when fetching the packages from github.

[guybedford]

@frederikschubert I'd be open to implementing this. Note that we'd need to have jspmNodeConversion: false in all the packages as well for this to work. Would you be interested in working on this? It would probably be advisable though to work on this for jspm 0.17 only I think due to backwards compatibility, unless we can come up with a really smooth upgrade path. These libraries are listed in https://github.com/jspm/npm/blob/0.17/lib/node-conversion.js#L13.

[frederikschubert]

Sorry for the late reply! I am not sure what steps have to be taken to implement this.

1. add jspmNodeConversion: false to all the nodelibs-* packages?
2. ?

[guybedford]

@frederikschubert

1. Add jspmNodeConversion: false to all the nodelibs-* packages
2. Publish each nodelibs-* package to npm as jspm-nodelibs-x (adding me as a collaborator please) with a 0.2.0 version.
3. Update the nodelibs path in jspm-npm to npm:nodelibs- at https://github.com/jspm/npm/blob/0.17/lib/node-conversion.js#L10. and the version to 0.2.0 or whatever we are going with here https://github.com/jspm/npm/blob/0.17/lib/node-conversion.js#L11.

[frederikschubert]

Alright. Thanks for the instructions!

[frederikschubert]

I just published jspm-nodelibs-assert and added you as a collaborator. Does this look ok to you?

[guybedford]

@frederikschubert that looks exactly right.

[frederikschubert]

Ok, then I will continue with the other packages and ping you here when I am done.

Progress

• assert
• buffer
• child_process
• cluster
• console
• constants
• crypto
• dgram
• dns
• domain
• events
• fs
• http
• https
• module
• net
• os
• path
• process
• punycode
• querystring
• readline
• repl
• stream
• string_decoder
• sys
• timers
• tls
• tty
• url
• util
• vm
• zlib

[guybedford]

👍

[frederikschubert]

@guybedford I created the PRs.

[guybedford]

@frederikschubert thanks so much, is it all working ok for you there now?

There's one more thing we do need to do as well and that is to provide an upgrade path that updates the github:jspm/nodelibs-x libraries to npm versions to ensure we get proper deduping for new npm installs. There's already code exactly like this at https://github.com/jspm/jspm-cli/blob/0.17/lib/config/index.js#L179, which can just be duplicated into a new block that always runs in the 0.17 branch.

I'm happy to look at this as well in due course so no pressure, but if you can try something I'd certainly be grateful. Thanks again for your efforts here.

[frederikschubert]

Glad I can help. :) Yes it is working. I linked the jspm-npm package in one of our projects and performed a fresh install after running jspm cache-clear to be sure.

Should also the other references to the nodelibs in this block change?

I am not quiet sure where the upgrade code should be run. Do you mean somewhere in this function https://github.com/jspm/jspm-cli/blob/0.17/lib/install.js#L75?

[guybedford]

@frederikschubert yes the upgrade path references in that block should be updated as well. It can be added as a new duplicate block in https://github.com/jspm/jspm-cli/blob/0.17/lib/config/index.js#L313 that always runs but with a flag that checks if anything was actually replaced, and if something was replaced then runs a full install like in https://github.com/jspm/jspm-cli/blob/0.17/lib/config/index.js#L278.

[frederikschubert]

Ok I will try to take a stab at it and make a PR.

[stefanlivens]

So if I understand correctly, using [email protected] should not use GitHub anymore for dependencies.
But I just tried to install some packages into a clean environment, and it tries to download systemjs from github:systemjs/systemjs?
So @frederikschubert how did you manage that?
(I'm behind a very restrictive firewall/proxy, and wil end up on an even stricter Jenkins build-server, so I cloned the jspm registry already to an internal git repository)

[ndzeller]

Bump. Has there been more progress on this? All of the jspm-nodelibs-* except jspm-nodelibs-buffer work for me after a jspm install. jspm/nodelibs-buffer#6

[openwms]

With 0.16.53 Github is used and that leads in 2018 most of the time to ratelimit issues during the builds (claphands). Really wonder how others can use this library?

// cc: @frederikschubert @guybedford

warn GitHub rate limit reached.
Would you like to set up your GitHub credentials? [yes]:ok   Installed npm:invariant@^2.2.1 (2.2.4)

Please release at lease a patch version to fix this

[guybedford]

@openwms note that the 0.17 version no longer requires this github dependency for this reason.