Allow for parallel format creation #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- make forking optional - test for availability of ForkManager - add command line option to disable forking - when running with forkmanager, dump output of compile logs to $nul - if --quiet is given, also hide the compile logs
norbusan
commented
May 8, 2024
| @@ -128,6 +130,7 @@ BEGIN | |||
| "no-engine-subdir", | |||
| "no-error-if-no-engine=s", | |||
| "no-error-if-no-format", | |||
| "no-fork", | |||
There was a problem hiding this comment.
Add a command line argument to disable forking even in presence of ForkManager module.
norbusan
commented
May 8, 2024
| } else { | ||
| $USE_FORKMANAGER = 1; | ||
| } | ||
| } |
There was a problem hiding this comment.
Dynamically check for presence of the forking module
norbusan
commented
May 8, 2024
texlive-scripts/fmtutil.pl
Outdated
| # in mktexfmtMode we must redirect *all* output to stderr | ||
| $cmdline .= " >&2" if $mktexfmtMode; | ||
| if ($opts{'quiet'} || $USE_FORKMANAGER) { | ||
| $cmdline .= " >$nul 2>$nul"; |
There was a problem hiding this comment.
If we run in quiet mode, or under fork manager, we do NOT show the output of the format creation run.
This complete log is still in the log file.
|
with forkmanager |
|
without forking |
|
with forkmanager and --quiet |
|
without forkmanager and --quiet |
|
Timing for Without forkmanager: With forkmanager: |
|
As your TODO implies, |
|
Anyway, nproc is not so important. What's important to me is the (lack of) output. If I'm understanding right, I don't like the fact that forking implies --quiet. The result will be that nothing shows up in install-tl.log / tlmgr-commands.log. I.e., debugging becomes impossible. The .log files from TeX are not sufficient to figure out what's going on when it fails, in my experience. I don't have an easy solution. Sorry. |
|
Yes, there still need some work with the output, I'll try to implement per-process output buffer, like I discussed on TeX-Live/texlive-source#69. There's also the problem with |
Good point. Maybe set it to something conservative like 8 or so?
Probably better to, I don't trust forks on Windows. |
What about the following, which we are already doing in tlmgr, namely use If all went well, just drop the stdout, but if something fails, dump everything to stdout. Would that work? |
Indeed, the reason for the Maybe the best is:
and after we have returned control to the main program:
WDYT? |
|
Maybe set it to something conservative like 8 or so? Or rad
/proc/cpuinfo on linux
https://stackoverflow.com/questions/45181115 suggests
getconf _NPROCESSORS_ONLN 2>/dev/null || sysctl -n hw.ncpu
Said to be tested and works on Linux, Mac, BSD.
(Does work on my machine at home.)
If no output / failure of the above, I'm tempted to say it's safer to
simply not do any forking.
What about Windows? Just disable on Windows?
Probably better to, I don't trust forks on Windows.
Fully agreed :).
|
|
Dick Koch confirms sysctl -n hw.ncpu for Macs. |
|
I changed the code to:
|
|
I also have tried now to deal with the deferred prints and the mixed up printing. |
norbusan
commented
May 10, 2024
| @@ -203,6 +206,18 @@ sub main { | |||
| die "$0: Unexpected non-option argument(s): @ARGV\n" | |||
| . "Try \"$prg --help\" for more information.\n"; | |||
| } | |||
| # for full fmtutil, let us try to use ForkManager if available | |||
| # do not try this on Windows | |||
| if (wndws() || $opts{"no-fork"}) { | |||
There was a problem hiding this comment.
We disable forking on windows
|
I confess I didn't try to look at the current state of the patch.
- if none-success, dump stdout/stderr of the process
- dump defererred print statements from the tmp dirs
By "dump" you mean "output to a log file", not "discard", right?
As things stand, install-tl and tlmgr handle the fmtutil call
differently. I created this complication in r69653 (January 31, 2024)
and r69614 earlier, for the sake of the context cache postcommand.
For tlmgr, fmtutil is still invoked through tlmgr's do_cmd_and_check
which calls TLUtils::run_cmd, which saves stdout but not stderr, on the
theory that it's fine to show stderr on the terminal as it comes along.
For install-tl, on the other hand, fmtutil (and the other postinst
commands) are invoked through the new fn TLUtils::run_cmd_with_log,
which captures both stdout and stderr in a single string and then passes
it to a specified function.
The whole goal there was to capture the (incredibly voluminous and
useless) output of the context cache updates and write to either
install-tl.log or tlmgr-commands.
All of which is to say, maybe fmtutil could also call
TLUtils::run_cmd_with_log with an appropriate log fn that saves the
string away and writes it to the log at the end. I'm not sure if that
would be easier or harder than what you're already doing.
It occurs to me now that maybe what we want is to change
run_cmd_with_log so that the logging function called gets stdout and
stderr as separate strings, plus the exit status of the command, and
then writes the stderr string to the terminal (the stderr fd) if the
exit status is bad. Hmm.
Anyway, overall, I think the ideal is to log everything for every command.
One other question: what does fmtutil write to stderr by default? (I
mean, not the mktexfmt case.) Anything? Can't investigate further right
now, my day is just about over ... --thanks, karl.
|
|
With the current code and an error happening, we get the following output. That shows that the deferred printing is working and output is shown. |
|
On Thu, 09 May 2024, kberry wrote:
By "dump" you mean "output to a log file", not "discard", right?
To be more specific about the current state:
in mktextfm mode - no changes.
otherwise:
- formats are created one by one or in parallel (depending on fork)
- successful format builds do NOT show the stdout/stderr
- failure format builds capture stdout and stderr (2>&1 style)
and print it to stdout
- deferred prints are shown at the end of the output
As things stand, install-tl and tlmgr handle the fmtutil call
differently. I created this complication in r69653 (January 31, 2024)
and r69614 earlier, for the sake of the context cache postcommand.
I need to think about the interaction between tlmgr/install-tl/fmtutil,
since as of now we only looked at running fmtutil standalone.
All of which is to say, maybe fmtutil could also call
TLUtils::run_cmd_with_log with an appropriate log fn that saves the
string away and writes it to the log at the end. I'm not sure if that
would be easier or harder than what you're already doing.
That is what the code is now doing with run_cmd (not _with_log).
What do you mean with
and writes it to the log at the end
fmtutil itself does NOT write any log file at all.
Anyway, overall, I think the ideal is to log everything for every command.
The engine run log file is preserved.
For successful runs, do we need to preserve the stdout/stderr of the
run?
One other question: what does fmtutil write to stderr by default? (I
mean, not the mktexfmt case.) Anything? Can't investigate further right
now, my day is just about over ... --thanks, karl.
There are error/warning message:
```
sub print_warning {
print STDERR "$prg [WARNING]: ", @_ if (!$opts{'quiet'})
}
sub print_error {
print STDERR "$prg [ERROR]: ", @_;
}
```
and similar with the deferred prints.
|
|
kb> and writes it to the log at the end
np> fmtutil itself does NOT write any log file at all.
Right. I meant {install-tl,tlmgr-commands}.log.
For successful runs, do we need to preserve the stdout/stderr of the
run?
Yes, I think the full stdout/stderr should end up in
{install-tl,tlmgr-commands}.log as it does now. -k
|
|
Really nice work @norbusan! I was thinking about something like the following for the output. Store each line that would be printed in a variable and print it all at once in when the process is exiting. This would print the output of each format close together. What do you say? diff --git a/texlive-scripts/fmtutil.pl b/texlive-scripts/fmtutil.pl
index cc304a0..ce46c20 100755
--- a/texlive-scripts/fmtutil.pl
+++ b/texlive-scripts/fmtutil.pl
@@ -58,6 +58,13 @@ my $FMT_NOTAVAIL = 4;
my $nul = (wndws() ? 'nul' : '/dev/null');
my $sep = (wndws() ? ';' : ':');
+my $print_stderr = sub {
+ print STDERR @_;
+};
+my $print_stdout = sub {
+ print STDOUT @_;
+};
+
my @deferred_stderr;
my @deferred_stdout;
# $::opt_verbosity = 3; # manually enable debugging
@@ -533,11 +540,21 @@ sub callback_build_formats {
next if ($swi eq "format=engine" && $fmt ne $eng);
next if ($swi eq "format!=engine" && $fmt eq $eng);
$total++;
+ my $collected_stderr = "";
+ my $collected_stdout = "";
if ($USE_FORKMANAGER) {
$pm->start("select_and_rebuild_format($fmt, $eng, $what, $whatarg)") and next;
+ $print_stderr = sub {
+ $collected_stderr = $collected_stderr . join "", @_;
+ };
+ $print_stdout = sub {
+ $collected_stdout = $collected_stdout . join "", @_;
+ };
}
my $val = select_and_rebuild_format($fmt, $eng, $what, $whatarg);
if ($USE_FORKMANAGER) {
+ print STDERR $collected_stderr;
+ print STDOUT $collected_stdout;
my @array = ($fmt, $eng, \@deferred_stdout, \@deferred_stderr);
$pm->finish($val, \@array);
} else {
@@ -1452,23 +1469,23 @@ sub reset_root_home {
#
sub print_info {
if ($mktexfmtMode) {
- print STDERR "$prg [INFO]: ", @_ if (!$opts{'quiet'});
+ $print_stderr->("$prg [INFO]: ", @_) if (!$opts{'quiet'});
} else {
- print STDOUT "$prg [INFO]: ", @_ if (!$opts{'quiet'});
+ $print_stdout->("$prg [INFO]: ", @_) if (!$opts{'quiet'});
}
}
sub print_verbose {
if ($mktexfmtMode) {
- print STDERR "$prg: ", @_;
+ $print_stderr->("$prg: ", @_);
} else {
- print STDOUT "$prg: ", @_;
+ $print_stdout->("$prg: ", @_);
}
}
sub print_warning {
- print STDERR "$prg [WARNING]: ", @_ if (!$opts{'quiet'})
+ $print_stderr->("$prg [WARNING]: ", @_) if (!$opts{'quiet'})
}
sub print_error {
- print STDERR "$prg [ERROR]: ", @_;
+ $print_stderr->("$prg [ERROR]: ", @_);
}
#
# same with deferredWe could also collect the output into an array, and it may be more performant actually. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.