'Too many open files'

[amirmc]

I'm hitting a completely new error. Not sure what's causing this but I wonder if it's related to the previously mentioned memory issues.

$ ./main.native
# repos: 697
Create cache repo/result.commits
  *** Sys_error("Too many open files")
Fatal error: exception # opam-version    1.2.2
# os              darwin
./main.native: "open" failed on repo/result.commits: Too many open files

[mor1]

A thought: There are typically limits on Unix for the maximum number of
open files at any time. How does cosmetrics handle files?
On 3 Sep 2015 7:20 pm, "Amir Chaudhry" [email protected] wrote:

I'm hitting a completely new error. Not sure what's causing this but I
wonder if it's related to the previously mentioned memory issues.

$ ./main.native

repos: 697

Create cache repo/result.commits
*** Sys_error("Too many open files")
Fatal error: exception # opam-version 1.2.2

os darwin

./main.native: "open" failed on repo/result.commits: Too many open files

—
Reply to this email directly or view it on GitHub
#2.

[amirmc]

Using Lwt? https://github.com/ocamllabs/cosmetrics/blob/master/src/Cosmetrics.ml#L38

The odd thing is that things were working fine yesterday and I believe my local copy was up to date. Other packages have been updated though and I've rebuilt cosmetrics since then.

[avsm]

Try running ulimit -n 1024 and retrying the failing command.

[Chris00]

There are a few Lwt_list.map_p and I suspect that the one at line 427 of main.ml is causing that. Unfortunately these functions do not have a way to limit the parallelism. I should write something to do that — @avsm or @mor1, if you have something ready to use, please share! ;-)

In the meantime, replacing it with Lwt_list.map_s should solve the issue — at the expense of being (possibly much) slower.

[Drup]

Lwt_stream.create_bounded ?

[amirmc]

It worked with @avsm's suggestion. After some light web-searching, I think MacOSX's default limit is 256 and the change here won't persist after a restart (just noting it for anyone else who finds their way to this issue).

[samoht]

Lwt_pool is quite useful there, see for instance https://github.com/mirage/ocaml-git/blob/master/lib/unix/git_unix.ml#L228

[Chris00]

@samoht Thanks, that is indeed useful. Lwt should provide a global pool for disk accesses however — otherwise it is difficult for different libraries to cooperate and the system limit is easily exceeded.

[amirmc]

I know @Chris00 has pushed a fix but I still seem to have this issue.

$ ./main.native
# repos: 705
# repositories used: 594
Fatal error: exception # opam-version    1.2.2
# os              darwin
./main.native: "open" failed on mbr-format.html: Too many open files

and for info...

$ ulimit -n
256