Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate predictions on "swap" synthetic test dataset #80

Open
SteveOv opened this issue Aug 12, 2024 · 6 comments
Open

Investigate predictions on "swap" synthetic test dataset #80

SteveOv opened this issue Aug 12, 2024 · 6 comments
Assignees
Labels
question Further information is requested

Comments

@SteveOv
Copy link
Owner

SteveOv commented Aug 12, 2024

Investigate the phenomenon shown below, when we test against a synthetic test dataset where instances are swapped if the original secondary eclipse is found to be deeper.

All instances Transiting instances
predictions-nonmc-vs-labels-synthetic-mist-tess-dataset-swap10 predictions-nonmc-vs-labels-synthetic-mist-tess-dataset-swap10-transiting
@SteveOv
Copy link
Owner Author

SteveOv commented Aug 12, 2024

Added a swapped field to saved CSVs with 81a257d

@SteveOv
Copy link
Owner Author

SteveOv commented Aug 12, 2024

Can confirm that this is definitely down to the instances that have been swapped.

Swapped instances Non-swapped instances
predictions-nonmc-vs-labels-synthetic-mist-tess-dataset-swap10-swapped predictions-nonmc-vs-labels-synthetic-mist-tess-dataset-swap10-not-swapped

@SteveOv
Copy link
Owner Author

SteveOv commented Aug 12, 2024

By combining the swapped and transiting criteria we see that the majority are swapped and transiting.

Swapped & transiting instances swapped & non-transiting instances
pred-swt pred-swnt

@SteveOv
Copy link
Owner Author

SteveOv commented Aug 12, 2024

Doesn't completely resolve the issue, but I've found that I've been handling the change in bP and bS when switching components incorrectly. These need to be recalculated as they must relate to the newly assigned star A

@SteveOv
Copy link
Owner Author

SteveOv commented Aug 16, 2024

Tried training models on a datasets with swap enabled. Invariably, these improved the results with the synthetic set with swap enabled at the expense of these without it, however the net result was significantly worse predictions.

The following shows the predictions for k against both "swapped" and "non-swapped" synthetic-mist-tess-datasets with a model trained on s 100k dataset with swapped instances (without additional restrictions on k, J or qphot);

synth test dataset with swap synth test dataset without swap
swapped non-swapped

For the "swap" model (trained on 100k train/val instances without swap):

test dataset all instances transiting non-transiting
synth test dataset with swap (k<=10) 0.060 041 0.124 873 0.042 900
synth test dataset without swap 0.061 276 0.092 278 0.053 035
formal test dataset (effectively with swap) 0.064 847 0.107 058 0.0530122

For the control model (trained on 100k train/val instances without swap):

test dataset all instances transiting non-transiting
synth test dataset with swap (k<=10) 0.071 529 0.179 780 0.042 909
synth test dataset without swap 0.040 515 0.063 774 0.034 333
formal test dataset (effectively with swap) 0.050 801 0.077 074 0.043 503

@SteveOv
Copy link
Owner Author

SteveOv commented Oct 9, 2024

Returning to this with models trained with the mags feature centred on the midpoint between the eclipses and roll (agumentation) <= 512 bins.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant