Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Output redirected to a file appears to get truncated. #95

Open
pwrightkcl opened this issue Apr 19, 2023 · 4 comments
Open

Output redirected to a file appears to get truncated. #95

pwrightkcl opened this issue Apr 19, 2023 · 4 comments
Assignees

Comments

@pwrightkcl
Copy link

pwrightkcl commented Apr 19, 2023

I am running a script that includes some Niftyreg commands and redirecting the output to a log file. After reg_aladin is complete, its output and several other lines before and after it are removed from the log. I tried monitoring the log live using tail -f and it output "file truncated" at the end of the reg_aladin output, followed by the content of the log file, starting from the first line, but missing the reg_aladin output and two lines before and after. The script includes subsequent lines using reg_aladin and reg_f3d, but these do not get truncated, and the truncation appears to occur before the subsequent lines are reached.

I do not know if this is caused by reg_aladin or how to determine whether it is but am raising the issue here in the hope someone can advise me.

I have tried running my script both with a standard redirect and prefixing it with stdbuf -oL. It is the latter that let me monitor the progress of the log file and catch the truncation taking place.

@pwrightkcl
Copy link
Author

pwrightkcl commented Apr 19, 2023

Here's what the log file looked like as it was getting truncated. You can see the two log lines prior to the first reg_aladin line are removed. There should be another three additional log lines after reg_aladin and before the line "Extracting volumes from GIF prior." that are also removed.

Note that reg_aladin runs again after "Aligning ICBM template to T1." but no further truncation occurs.

Wed Apr 19 09:57:53 UTC 2023
Creating local script:
/nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/ScriptBaMoS_sub-0_.sh
Running /nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/ScriptBaMoS_sub-0_.sh.
Scratch path: /tmp/sub-0_BaMoSCross__
Copying T1 image and converting to float.
Aligning FLAIR to T1.
[/usr/local/bin/reg_aladin] 
# snip
[reg_aladin_sym] * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
tail: sub-0/ses-v3/anat/sub-0_BaMoS.log: file truncated
Wed Apr 19 09:57:53 UTC 2023
Creating local script:
/nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/ScriptBaMoS_sub-0_.sh
Running /nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/ScriptBaMoS_sub-0_.sh.
Scratch path: /tmp/sub-0_BaMoSCross__
Extracting volumes from GIF prior.
Creating artefact mask based on GIF parcellation.
Aligning ICBM template to T1.
[/usr/local/bin/reg_aladin] 

@pwrightkcl
Copy link
Author

pwrightkcl commented Apr 19, 2023

Here is some actual code. I'm afraid it's long: there's a wrapper script containing the command with the redirection to the log file, that command generates a third script and executes it. I am trying to snip out irrelevant lines but show the parts relevant to the truncated log, including lines either side of reg_aladin.

Lines from the wrapper script:

BaMoS_log="${BaMoS_results_path}"/${id}_BaMoS.log
date > "$BaMoS_log"
stdbuf -oL "$script_path"/BaMoS_RISAPS.sh "$id" "$input_FLAIR" "$input_T1" "$GIF_results_path" "$BaMoS_results_path" >> "$BaMoS_log" 2>&1

Lines from the BaMoS_RISAPS.sh, which generates a third script:

echo "echo \"Scratch path: $PathScratch\"" >> "$NameScript"
# snip
{
  echo "cp ${PathResults}/* ${PathScratch}/."

  echo "echo \"Copying T1 image and converting to float.\""
  echo "cp ${ImageT1} ${PathScratch}/T1_${ID}.nii.gz"
  echo "${PathSeg}/seg_maths ${PathScratch}/T1_${ID}.nii.gz -odt float ${PathScratch}/T1_${ID}.nii.gz"

  echo "echo \"Aligning FLAIR to T1.\""
  echo "${PathReg}/reg_aladin -ref ${PathScratch}/T1_${ID}.nii.gz -flo ${ImageFLAIR} \
                              -aff ${PathScratch}/Aff_FLAIRtoT1.txt -res ${PathScratch}/FLAIR_${ID}.nii.gz -rigOnly"
  echo "${PathSeg}/seg_maths ${PathScratch}/FLAIR_${ID}.nii.gz -odt float ${PathScratch}/FLAIR_${ID}.nii.gz"
} >> "$NameScript"
# snip
{
  echo "echo \"Copying scratch directory to results directory.\""
  echo "cp ${PathScratch}/* ${PathResults}/."

  echo "echo \"Extracting volumes from GIF prior.\""
} >> "$NameScript"
#snip
  {
    echo "echo \"Aligning ICBM template to T1.\""
    echo "${PathReg}/reg_aladin -ref ${PathScratch}/T1_${ID}.nii.gz -flo ${PathICBM}/ICBM_Template.nii.gz \
                                -aff ${PathScratch}/${ID}_AffTransf.txt"
    echo "${PathReg}/reg_f3d -ref ${PathScratch}/T1_${ID}.nii.gz -flo ${PathICBM}/ICBM_Template.nii.gz \
                             -aff ${PathScratch}/${ID}_AffTransf.txt -cpp ${PathScratch}/${ID}_cpp.nii.gz"
    echo "cp ${PathScratch}/*cpp* ${PathResults}/."
  } >> "$NameScript"
#snip
bash "$NameScript"

Lines from the third script referenced by $NameScript. These show the three log lines that should appear between the first reg_aladin and the "Extracting volumes from GIF prior." The two lines before reg_aladin appeared but were then truncated along with the reg_aladin output. The three following lines never appeared at all:
"Copying GIF priors."
"Copying GIF results."
"Copying scratch directory to results directory."

#!/bin/bash
echo "Scratch path: /tmp/sub-0_BaMoSCross__"
mkdir -p /tmp/sub-0_BaMoSCross__ 
cp /nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/* /tmp/sub-0_BaMoSCross__/.
echo "Copying T1 image and converting to float."
cp /nfs/project/RISAPS/rawdata/sub-0/ses-v3/anat/sub-0_ses-v3_T1.nii.gz /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz
/usr/local/bin/seg_maths /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz -odt float /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz
echo "Aligning FLAIR to T1."
/usr/local/bin/reg_aladin -ref /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz -flo /nfs/project/RISAPS/rawdata/sub-0/ses-v3/anat/sub-0_ses-v3_FLAIR.nii.gz                               -aff /tmp/sub-0_BaMoSCross__/Aff_FLAIRtoT1.txt -res /tmp/sub-0_BaMoSCross__/FLAIR_sub-0.nii.gz -rigOnly
/usr/local/bin/seg_maths /tmp/sub-0_BaMoSCross__/FLAIR_sub-0.nii.gz -odt float /tmp/sub-0_BaMoSCross__/FLAIR_sub-0.nii.gz
cp /nfs/project/RISAPS/derivatives/GIF/sub-0/ses-v3/anat/sub-0_ses-v3_T1_TIV.nii.gz /tmp/sub-0_BaMoSCross__/Mask_sub-0.nii.gz
/usr/local/bin/seg_maths /tmp/sub-0_BaMoSCross__/Mask_sub-0.nii.gz -bin -odt char /tmp/sub-0_BaMoSCross__/GIF_B1_sub-0.nii.gz
echo "Copying GIF priors."
cp /nfs/project/RISAPS/derivatives/GIF/sub-0/ses-v3/anat/*prior* /tmp/sub-0_BaMoSCross__/GIF_prior_sub-0.nii.gz
echo "Copying GIF results."
cp /nfs/project/RISAPS/derivatives/GIF/sub-0/ses-v3/anat/*Segmentation* /tmp/sub-0_BaMoSCross__/GIF_Segmentation_sub-0.nii.gz
cp /nfs/project/RISAPS/derivatives/GIF/sub-0/ses-v3/anat/*prior* /tmp/sub-0_BaMoSCross__/GIF_prior_sub-0.nii.gz
cp /nfs/project/RISAPS/derivatives/GIF/sub-0/ses-v3/anat/*Parcellation* /tmp/sub-0_BaMoSCross__/GIF_Parcellation_sub-0.nii.gz
echo "Copying scratch directory to results directory."
cp /tmp/sub-0_BaMoSCross__/* /nfs/project/RISAPS/derivatives/BaMoS/sub-0/ses-v3/anat/.
echo "Extracting volumes from GIF prior."
#snip (several seg_maths lines)
echo "Creating artefact mask based on GIF parcellation."
#snip (even more seg_maths_lines)
echo "Aligning ICBM template to T1."
/usr/local/bin/reg_aladin -ref /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz -flo /nfs/project/RISAPS/sourcedata/ICBM_priors/ICBM_Template.nii.gz                                 -aff /tmp/sub-0_BaMoSCross__/sub-0_AffTransf.txt
/usr/local/bin/reg_f3d -ref /tmp/sub-0_BaMoSCross__/T1_sub-0.nii.gz -flo /nfs/project/RISAPS/sourcedata/ICBM_priors/ICBM_Template.nii.gz                              -aff /tmp/sub-0_BaMoSCross__/sub-0_AffTransf.txt -cpp /tmp/sub-0_BaMoSCross__/sub-0_cpp.nii.gz

@mmodat
Copy link
Contributor

mmodat commented Jul 10, 2023

Hi Paul,

Sorry, this has been left for a long time. Do you still have an issue? I can't think of a reason why reg_aladin would yield truncated line.

@pwrightkcl
Copy link
Author

I haven't examined it in detail since I did the testing above. I'd still prefer if this didn't happen, but like you I have no idea why it does, or even whether it is reg_aladin that causes it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants