Issue at KCWI reduction

[luisgdh]

Hello. I am trying to reduce KCWI data taken on 2024.04.03. I downloaded the data from KOA and I am running the command:

reduce_kcwi -f KR*.fits -r

I am getting several errors, and the DRPF and KCWI logs are attached. However, the most relevant error seems to be:

2024-04-16 12:58:27:KCWI:ERROR:CorrectDefects.py: Defect list not found for /home/luisgdh/anaconda3/envs/kcwidrp/lib/python3.7/site-packages/kcwidrp/data/defect_L2U2L1U1_2x2.dat

In this directory, there are several files, like defect_ALL_1x1.dat, defect_ALL_2x2.dat, defect_L2U2L1U1_1x1.dat, but no defect_L2U2L1U1_2x2.dat

Thank you very much

DRPF.log
KCWI.log

[MNBrod]

There are two separate issues here. Firstly, the defect list missing is definitely an oversight, I will work on getting that uploaded ASAP. Secondly, the pipeline currently does not support the use of KOA filenames (KR* or KB*), only "OFNAME" (kr* or kb*). There is a fix for this being testing this week, but it will not be deployed until at least next week. In the mean time, I suggest reducing data using the filenames directly from the telescope. This can be done through KOA by selecting "non-unique filename, assigned at the telescope" under option 3 on the download form above the data table.

I will update here once the defect list is added.

[luisgdh]

Ah, thanks for the quick response. I re-downloaded the data, now using the "OFNAME". However, I still cannot get it to work. I get vague error messages that don't tell me what's going on, like
2024-04-18 10:36:53:DRPF:INFO: Event failed: name bias_make_master, action MakeMasterBias, arg name kb240404_00020.fits, recurr False

Could you please take a look at the new logs and help me understanding what's going on?

Some errors that I get that could help explaining what's going on are:
2024-04-18 10:36:59:DRPF:ERROR: Exception invalid syntax (selenium_manager.py, line 44) while invoking MakeMasterBias
2024-04-18 10:37:03:DRPF:ERROR: Exception 'NoneType' object has no attribute 'meta' while invoking ExtractArcs

Thank you again
DRPF.log
KCWI.log

[MNBrod]

Just checking - between runs, are you removed the redux, plots, and logs directory, along with the kcwir.proc file?

The .proc file stores the files that have already been processed - if the file is left over between runs, it can lead to raw files not being reprocessed the second time.

[luisgdh]

Yes, I did remove those files between runs

Just to be clear, what I am doing is: I went to my project at KOA, downloaded all files from a specific night, added all of them to one directory, and ran (I am trying to reduce blue data first):

reduce_kcwi -f kb*.fits -b

Am I missing something? As far as I understood, the only thing that I needed to do is to check the configuration file, which seems to be ok

[rmcgurk]

As mentioned in the documentation here: https://kcwi-drp.readthedocs.io/en/latest/running.html
Looks like you are missing the -g flag, which groups the files according to what the pipeline needs. Please try running:
reduce_kcwi -b -f kb*.fits -g

[luisgdh]

We are making progress, but I got stuck in another problem. Now, the relevant error appears to be:

2024-04-29 17:02:10:DRPF:ERROR: Exception invalid syntax (selenium_manager.py, line 44) while invoking SolveArcs

Line 44 of selenium_manager.py reads:

if (path := os.getenv("SE_MANAGER_PATH")) is not None:

What I don't understand is: the walrus operator [DRPF.log](https://github.com/Keck-DataReductionPipelines/KCWI_DRP/files/15158267/DRPF.log) was only introduced in python 3.8.

However, we are supposed to install kcwidrp on python 3.7, right?

Again, log files (for the red channel) are attached, in case that's not what is going on

Thank you very much
KCWI.log
DRPF.log

[MNBrod]

Apologies for the late reply.

Are you getting any other plots written to the logs directory? Selenium should only be involved in plotting, so I'm surprised that the error only pops up during the wavelength solution calculation, as opposed to earlier. The logs don't indicate any errors, so I assume plotting as a whole is still working.

I am working on replicating this on my machine, to try to be of more help. Could you tell me what OS you are running this on, and also send the output of pip list? That will help me set my testing machine up as close to your machine as I can