github fetch not working: project cannot build without GIT ; hence it currently cannot build.

[JVDptt]

Describe the bug
A clear and concise description of what the bug is.

GitHub compressed GIT stream generation is broken :
Using either :
$ git clone 'https://github.com/clasp-developers/clasp'
or
$ mkdir clasp; git init; git remote add 'https://github.com/clasp-developers/clasp'; git remote update
or
downloading and unpacking clasp's 2.5.0.tar.gz Release tarball, and then running
$ ./koga
, fails with , in the case of GIT pull / remote update, with
$ export GIT_CURL_VERBOSE=1
export GIT_TRACE=1
export GIT_TRACE_PACKET=1
settings in effect:
'remote: Compressing objects: 100% (2200/2200), done.
21:27:43.170086 pkt-line.c:86 packet: sideband< PACK ...
21:27:43.170278 run-command.c:657 trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 1595386 on jvdspc.jvds.net' --pack_header=2,174095
21:27:43.174099 git.c:463 trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 1595386 on jvdspc.jvds.net' --pack_header=2,174095
21:29:17.043461 http.c:843 == Info: HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)
21:29:17.043498 http.c:843 == Info: Connection #0 to host github.com left intact
error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)
error: 990 bytes of body are still expected
21:29:17.043529 pkt-line.c:86 packet: git> 0002
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
'

Attempts to run koga also fail to check-out 'ansi-tests' git :

' 21:17:41.710552 http.c:802 <= Recv header: x-github-request-id: B301:1129A8:318A1F0:34DDFE8:66033B75
21:17:41.710563 http.c:790 <= Recv header, 0000000002 bytes (0x00000002)
21:17:41.710570 http.c:802 <= Recv header:
21:17:41.710641 pkt-line.c:86 packet: clone< packfile
21:17:41.710900 run-command.c:657 trace: run_command: git index-pack --stdin --fix-thin '--keep=fetch-pack 1594951 on jvdspc.jvds.net' --check-self-contained-and-connected
21:17:41.714568 git.c:463 trace: built-in: git index-pack --stdin --fix-thin '--keep=fetch-pack 1594951 on jvdspc.jvds.net' --check-self-contained-and-connected
21:17:41.765111 pkt-line.c:86 packet: sideband< PACK ...
^[[B21:18:52.344265 http.c:843 == Info: Connection #0 to host github.com left intact
error: 4507 bytes of body are still expected
21:18:52.344314 pkt-line.c:86 packet: git> 0002
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
Unhandled UIOP/RUN-PROGRAM:SUBPROCESS-ERROR in thread #<SB-THREAD:THREAD tid=1594948 "main thread" RUNNING
{1000F70003}>:
Subprocess #<UIOP/LAUNCH-PROGRAM::PROCESS-INFO {1002E5B583}>
with command ("git" "clone"
"https://github.com/clasp-developers/ansi-test.git"
"dependencies/ansi-test/")
exited with error code 128

Backtrace for: #<SB-THREAD:THREAD tid=1594948 "main thread" RUNNING {1000F70003}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<UIOP/RUN-PROGRAM:SUBPROCESS-ERROR {1002E55EA3}> # :QUIT T)
1: (SB-DEBUG::RUN-HOOK INVOKE-DEBUGGER-HOOK #<UIOP/RUN-PROGRAM:SUBPROCESS-ERROR {1002E55EA3}>)
2: (INVOKE-DEBUGGER #<UIOP/RUN-PROGRAM:SUBPROCESS-ERROR {1002E55EA3}>)
'

So, clasp cannot be built from GIT checkout (no means of performing a GIT checkout can succeed) ,
or from tar file . I have tried all the workarounds mentioned at :

https://stackoverflow.com/questions/66366582/github-unexpected-disconnect-while-reading-sideband-packet

to no avail.

Please, either :
A) liase with GitHub administrators to fix this issue ;
or B) Provide another GIT Clasp + ansi-test repositories not on GitHub ;
or C) Provide a ./koga script that does not use GIT to checkout ansi-tests ;
or D) Provide a full set of tar files, including of all dependencies + ansi_tests,
that do not require GIT to build .

Expected behavior
A clear and concise description of what you expected to happen.

It should be possible to clone the Clasp and ansi-test GIT repositories from GitHub, or
obtain tar files of them that contain scripts that do not require GIT to build.

Actual behavior
A clear and concise description of what happened instead. This should include error messages and backtraces if there are any. If that would be more than a few dozen lines, please attach it in a file instead.

It is not possible to clone or fetch either the Clasp or ansi-test repositories from GitHub, and
using the './koga' script from release tar files also fails because it attempts to clone ansi-test.

Code at issue

   $ git clone 'https://github.com/clasp-developers/clasp'
   
   
If applicable, example code that can be evaluated or compiled to reproduce the problem.
Ideally, this is short and self contained.
If it is not practicable to eliminate dependencies, please at least try to reduce them.

If the code is more than a few dozen lines, please attach a source file instead.

Other steps to reproduce
If the problem cannot be reproduced from just a code sample, include other steps here.

1. Step one
2. Step two
3. Etc.

Context

• Versions or commit hashes of Clasp on which you observed the problem (if you have built from up to date source, saying so is fine, you don't need to dig around for hashes)
• If relevant, your operating system and other aspects of your computing environment
• If you believe the compiler may be at fault, relevant declarations, including optimize settings (use (clasp-cltl2:declaration-information 'optimize))
• Any other context about the problem

[yitzchak]

I am unable to reproduce your issue on Linux or Mac. Can you tell us more about your OS and setup?

Please note that this is really sounds like a network issue.

[Bike]

It sounds like the thing we've been hitting where gitlab.common-lisp.net is intermittently down, which breaks the build since koga tries to pull it. Which is a network issue in a sense, but maybe we could do something about it? I thought we had mirrors set up already, actually

[yitzchak]

We already use mirrors, so I am thinking an intermitent network issue or a configuration issue.

[JVDptt]

Thanks all for taking a look at this .

It is still happening 100% for me, either checking out main clasp repo,
https://github.com/clasp-developers/clasp ,
or the sub-module
https://github.com/clasp-developers/ansi-test.git
under dependencies/ansi-test.

Any GIT fetch seems to fail on the last expected compressed packet:

remote: Compressing objects: 100% (2209/2209), done. 13:00:12.155557 pkt-line.c:86 packet: sideband< PACK ... 13:00:12.155719 run-command.c:657 trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 1737365 on jvdspc.jvds.net' --pack_header=2,174155 13:00:12.159811 git.c:463 trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 1737365 on jvdspc.jvds.net' --pack_header=2,174155 13:04:16.240984 http.c:843 == Info: Connection #0 to host github.com left intact **error: 2139 bytes of body are still expected** 13:04:16.241027 pkt-line.c:86 packet: git> 0002 fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output

I am running GIT v2.44 on fully up-to-date Fedora 38 (FC-38) Linux x86_64 distro -
glibc-2.37-18 , gcc-13.2.1, clang + llvm v16.0.6, ninja v1.11 , openssl-3.0.9-2, kernel-6.6.22 .

It appears the remote end is not flushing the final packet before disconnecting -
I can append a tcpdump log showing this behaviour if desired - the remote end
really is hanging up before the final packet of the compressed stream is sent,
according to tcpdump.

I can 'git remote add' and 'git remote update' other huge GitHub projects, such as :
https://github.com/chromiumembedded/cef
with no problems ; unfortunately, this problem only occurs with the Clasp project.

My network is over WiFi+Cellular, I use one of two accounts, one a bit faster ( max 2MB/s download, 512kps upload,
avg download @ 800-1200kbps ), and the other a bit slower ( max @ 1.2MB/s download, 256kbps upload,
avg download @ 256-512kbps ) - one phone recharges while the other provides internet for me here at home,
, where I work from, over a WiFi Hot-Spot .

[JVDptt]

Traceroute to github.com :

1 router (192.168.37.126) 9.779 ms 9.691 ms 9.829 ms 2 100.112.0.5 (100.112.0.5) 152.032 ms 151.995 ms 152.526 ms 3 172.24.114.113 (172.24.114.113) 69.827 ms 69.997 ms 71.181 ms 4 10.160.164.183 (10.160.164.183) 71.159 ms 71.267 ms 71.089 ms 5 172.24.227.5 (172.24.227.5) 69.310 ms 69.276 ms 69.343 ms 6 172.24.227.30 (172.24.227.30) 77.155 ms 66.761 ms 66.352 ms 7 185.6.36.28 (185.6.36.28) 66.487 ms 47.324 ms 48.309 ms 8 ae20-0.icr01.dub07.ntwk.msn.net (104.44.236.61) 54.308 ms 81.537 ms ae24-0.icr01.dub08.ntwk.msn.net (104.44.238.45) 63.263 ms

Of course, then when I run it with tcpdump enabled, the git remote update command succeeds !

as root, I run:
# tcpdump -s8192 -vvv -ttt -l host 4.208.26.197 >/tmp/github.strace.log 2>&1 &

while as my userid, I run :
$ git remote update
in clasp GIT directory, where I did
$ git remote add origin https://github.com/clasp-developers/clasp

Now, for the first time, it works!

Sorry, it appears this issue is transitory .

I wish we could get to the bottom of why GitHub sometimes fails to send the last packet of compressed GIT stream,
but it appears we will not be able to do so on this occasion :-( .

[JVDptt]

I guess the "fix" is to run tcpdump in background :-) ... weird!

[JVDptt]

My ultimate goal is to build a fork of libcef with a LISP as an installed fully supported Scripting Language ,
as well as JavaScript, and a Server-Side HTML+LISP templating + server-side LISP scripting interface , and
a LISP execution Browser Extension for Chrome + Firefox .
My favorite LISP for Shell-like interpreter applications is PicoLISP, and for modular / library style compiled systems SBCL.
I have built and installed SBCL-2.4.2 . The only thing I don't like about SBCL is its huge 2GB monolithic memory image
and corefile / 'save-lisp-and-die' based executable generation strategy , and need to bootstrap from CMU/CL -
I'd much prefer it if the common lisp implementation was entirely shared library based, and could survive an
executable generation, and could build itself from scratch without needing an installed instance of SBCL / CMU/CL .
The same problems exist also for most LISPs, including PicoLISP, which also cannot build without use of
a pre-existing PicoLISP instance.
I just wanted to see how far Clasp has gotten in dealing with the above issues.
Thanks ! Sorry for the transitory network issue, which does repeatedly occur if I don't run tcpdump .