Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time window for coco dataset training and inference and time window and dataloader use for GEN1 #19

Open
gwgknudayanga opened this issue Oct 30, 2024 · 10 comments

Comments

@gwgknudayanga
Copy link

gwgknudayanga commented Oct 30, 2024

  • coco.yaml is missing

  • I just run the code for coco dataset. it seems the time window is always 1. According to your mem_update logic in this case it doesn't experience any voltage membrane decay and it just send the value of the received data clamp to 4 as maximum. So what is the time window that you have used to make it behave like spiking neurons?

  • Even with that running i am getting following error when running.
    File "/home/atiye/SpikeYOLO/ultralytics/models/yolo/detect/val.py", line 146, in get_stats
    self.nt_per_class = np.bincount(stats[-1].astype(int), minlength=self.nc) # number of targets per class
    IndexError: list index out of range

  • Another question is, how many time bins/time window you have used to organize the GEN1 dvs data? Can you please provide the code dataloader code for GEN1 as then the spiking neurons shows the membrane decay based real spiking neuron behavior?

  • Also since mem_update is used for both train and validation it cannot see how binary spikes (1,0) are available during inference as mentioned in paper

@XinhaoLuo666
Copy link
Collaborator

1,3. We re-uploaded coco.yaml,you can try it again
2. you can change the time window in cfg/model/snn_yolov8.yaml [MS_GetT]
4. Time window of Gen1 is same as the paper reported. Gen1 dataset will upload a few weeks later
5. There is no function in the code to convert an integer into a pulse train, as this does not provide inference acceleration or power consumption on the GPU. But it's easy to implement, for example, you just need to turn a 3 into 3 1s

@gwgknudayanga
Copy link
Author

@XinhaoLuo666,
Regarding the GEN1 dataset evaluation, I organize each sample of the dataset to 5 time steps. each time step has 2 channels. And then i feed these to the network and trained your network . But the maximum mAP@50 that i could obtained in validation set is 36.1%. (I used horizontal,vertical as data augmentations. ). Don't know why i am getting these low values compared to the reported values in the paper.

@XinhaoLuo666
Copy link
Collaborator

For different T's, do you guarantee that their data-enhanced images are the same? Assuming that the image remains as it is in the first frame and the third image is flipped left and right, the model's ability to model the timing task will be significantly damaged

@gwgknudayanga
Copy link
Author

@XinhaoLuo666
In GEN1 dataset there are no images and it's only the events, isn't it? I feed the direct events of a sample organizing them into a tensor of dimension (5,16,2,640,640) which are (T,N,C,H,W). So any augmentation happens to the whole tensor at once and hence similar to all 5 time steps in similar manner.
I don't why the mAP@50 is 36.1% even after 200 iterations. It is better you can upload the code for GEN1 dataloading with guidelines/READme and also with sample weights file so that it is easy to reproduce the results and proced on top of it.

Thank you.

@feekyzz
Copy link

feekyzz commented Nov 9, 2024

GEN1数据集如何使用呀,有没有哥们教一教我!

@gwgknudayanga
Copy link
Author

gwgknudayanga commented Dec 8, 2024

Hi @XinhaoLuo666,

Is the class for preprocessing and loading of GEN1 event data to SpikeYOLO available to download now?
It doesn't seems yet. Also if you resized the GEN1 data spatially before feeding to your network, what was the resized resolution?

@XinhaoLuo666
Copy link
Collaborator

Hi @XinhaoLuo666,

Is the class for preprocessing and loading of GEN1 event data to SpikeYOLO available to download now? It doesn't seems yet. Also if you resized the GEN1 data spatially before feeding to your network, what was the resized resolution?

We have uploaded a new folder "SpikeYOLO_for_Gen1" that can be used for training and inference directly on the GEN1 dataset

@XinhaoLuo666
Copy link
Collaborator

GEN1数据集如何使用呀,有没有哥们教一教我!

We have uploaded a new folder "SpikeYOLO_for_Gen1" that can be used for training and inference directly on the GEN1 dataset

@gwgknudayanga
Copy link
Author

Thank you. I just had a look. It seems even for the GEN1 data , the same augmentations for image-based data are used.
For example as a part of v8_transform used in spikeyolo for GEN1, it uses RandomPerspective tranformation. Within that it can be seen opencv APIs with appending values like 114 to binary spike data. I thought to use opencv API it is just for image data. And also adding 114 values for padding is for binary spike data might cause issues. So i believe you use these same augmentations for image for events without any issue right? Or am i missing something?

@XinhaoLuo666
Copy link
Collaborator

Thank you. I just had a look. It seems even for the GEN1 data , the same augmentations for image-based data are used. For example as a part of v8_transform used in spikeyolo for GEN1, it uses RandomPerspective tranformation. Within that it can be seen opencv APIs with appending values like 114 to binary spike data. I thought to use opencv API it is just for image data. And also adding 114 values for padding is for binary spike data might cause issues. So i believe you use these same augmentations for image for events without any issue right? Or am i missing something?

No, we didn't use augmentation when working with neuromorphic data, he did execute the "RandomPerspective tranformation" function, but this function has been internally altered to remove the augmentation part

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants