Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some thing wrong with add_histogram function #96

Open
ngocgiang99 opened this issue Jun 17, 2021 · 2 comments
Open

Some thing wrong with add_histogram function #96

ngocgiang99 opened this issue Jun 17, 2021 · 2 comments

Comments

@ngocgiang99
Copy link

ngocgiang99 commented Jun 17, 2021 •
edited
Loading

I had a strange problem when I tried to implement EfficientNet model. Last week, this code worked fine. But now, when I retrain then occurs some error. I faced this error in the validation step.

This is my code: https://github.com/ngocgiang99/Paper-Implementation. Please checkout to branch feat_efficient_net.

This is error log:

Traceback (most recent call last):
  File "train.py", line 73, in <module>
    main(config)
  File "train.py", line 54, in main
    trainer.train()
  File "D:\Work\Me\Paper-Implementation\efficient_net\base\base_trainer.py", l
    result = self._train_epoch(epoch)
  File "D:\Work\Me\Paper-Implementation\efficient_net\trainer\trainer.py", lin
    val_log = self._valid_epoch(epoch)
  File "D:\Work\Me\Paper-Implementation\efficient_net\trainer\trainer.py", lin
    self.writer.add_histogram(name, p, bins='auto')
  File "D:\Work\Me\Paper-Implementation\efficient_net\logger\visualization.py"
    add_data(tag, data, self.step, *args, **kwargs)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tenso
in add_histogram
    histogram(tag, values, bins, max_bins=max_bins), global_step, walltime)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tenso in histogram
    hist = make_histogram(values.astype(float), bins, max_bins)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tenso in make_histogram
    counts, limits = np.histogram(values, bins=bins)
  File "<__array_function__ internals>", line 6, in histogram
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\lib\histogram
    bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\lib\histogrn_edges
    endpoint=True, dtype=bin_type)
  File "<__array_function__ internals>", line 6, in linspace
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\core\function_base.py", line 135, in linspace
    y = _nx.arange(0, num, dtype=dt).reshape((-1,) + (1,) * ndim(delta))
numpy.core._exceptions.MemoryError: Unable to allocate 10.3 TiB for an array with shape (1418558411252,) and data type float64

Conda environment:

# Name                    Version                   Build  Channel
absl-py                   0.12.0                   pypi_0    pypi
ca-certificates           2021.4.13            haa95532_1
cachetools                4.2.2                    pypi_0    pypi
certifi                   2020.12.5        py37haa95532_0
chardet                   4.0.0                    pypi_0    pypi
google-auth               1.30.1                   pypi_0    pypi
google-auth-oauthlib      0.4.4                    pypi_0    pypi
grpcio                    1.38.0                   pypi_0    pypi
idna                      2.10                     pypi_0    pypi
importlib-metadata        4.3.1                    pypi_0    pypi
markdown                  3.3.4                    pypi_0    pypi
numpy                     1.20.3                   pypi_0    pypi
oauthlib                  3.1.0                    pypi_0    pypi
openssl                   1.1.1k               h2bbff1b_0
pandas                    1.2.4                    pypi_0    pypi
pillow                    8.2.0                    pypi_0    pypi
pip                       21.1.1           py37haa95532_0
protobuf                  3.17.1                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
python                    3.7.10               h6244533_0
python-dateutil           2.8.1                    pypi_0    pypi
pytz                      2021.1                   pypi_0    pypi
requests                  2.25.1                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rsa                       4.7.2                    pypi_0    pypi
setuptools                52.0.0           py37haa95532_0
six                       1.16.0                   pypi_0    pypi
sqlite                    3.35.4               h2bbff1b_0
tensorboard               2.5.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
torch                     1.8.1+cu111              pypi_0    pypi
torchaudio                0.8.1                    pypi_0    pypi
torchvision               0.9.1+cu111              pypi_0    pypi
tqdm                      4.61.0                   pypi_0    pypi
typing-extensions         3.10.0.0                 pypi_0    pypi
urllib3                   1.26.5                   pypi_0    pypi
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
werkzeug                  2.0.1                    pypi_0    pypi
wheel                     0.36.2             pyhd3eb1b0_0
wincertstore              0.2                      py37_0
zipp                      3.4.1                    pypi_0

Thanks.
@MohamedA95
Copy link

Hi,
I am not an expert but just out of curiosity is normal for it to try to allocate 10.3 TiB???

@ngocgiang99
Copy link
Author

ngocgiang99 commented Jun 17, 2021 •
edited
Loading

Hi @MohamedA95 , I printed the shape of p variable. The shape is small but I don't know why it allocate very large shape.
Here is log when I printed shape of p:

histogram shape:  torch.Size([32, 3, 3, 3])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32, 32, 1, 1])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32, 1, 3, 3])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([16, 32, 1, 1])
histogram shape:  torch.Size([16])
histogram shape:  torch.Size([16])
histogram shape:  torch.Size([16])
histogram shape:  torch.Size([96, 16, 1, 1])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([96, 1, 3, 3])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([24, 96, 1, 1])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([144, 24, 1, 1])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144, 1, 3, 3])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([24, 144, 1, 1])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([144, 24, 1, 1])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144, 1, 5, 5])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([40, 144, 1, 1])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([240, 40, 1, 1])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240, 1, 5, 5])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([40, 240, 1, 1])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([240, 40, 1, 1])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240, 1, 3, 3])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([80, 240, 1, 1])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([480, 80, 1, 1])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480, 1, 3, 3])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([80, 480, 1, 1])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([480, 80, 1, 1])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480, 1, 3, 3])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([80, 480, 1, 1])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([480, 80, 1, 1])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480, 1, 5, 5])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([112, 480, 1, 1])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([672, 112, 1, 1])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672, 1, 5, 5])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([112, 672, 1, 1])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([672, 112, 1, 1])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672, 1, 5, 5])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([112, 672, 1, 1])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([672, 112, 1, 1])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672, 1, 5, 5])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([192, 672, 1, 1])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([1152, 192, 1, 1])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152, 1, 5, 5])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([192, 1152, 1, 1])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([1152, 192, 1, 1])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152, 1, 5, 5])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([192, 1152, 1, 1])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([1152, 192, 1, 1])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152, 1, 5, 5])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([192, 1152, 1, 1])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([1152, 192, 1, 1])
histogram shape:  torch.Size([1152])
Traceback (most recent call last):
  File "train.py", line 73, in <module>
    main(config)
  File "train.py", line 54, in main
    trainer.train()
  File "D:\Work\Me\Paper-Implementation\efficient_net\base\base_trainer.py", line 63, in train
    result = self._train_epoch(epoch)
  File "D:\Work\Me\Paper-Implementation\efficient_net\trainer\trainer.py", line 68, in _train_epoch
    val_log = self._valid_epoch(epoch)
  File "D:\Work\Me\Paper-Implementation\efficient_net\trainer\trainer.py", line 100, in _valid_epoch    self.writer.add_histogram(name, p, bins='auto')
  File "D:\Work\Me\Paper-Implementation\efficient_net\logger\visualization.py", line 65, in wrapper
    add_data(tag, data, self.step, *args, **kwargs)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tensorboard\writer.py", line 429, in add_histogram
    histogram(tag, values, bins, max_bins=max_bins), global_step, walltime)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tensorboard\summary.py", line 300, in histogram
    hist = make_histogram(values.astype(float), bins, max_bins)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tensorboard\summary.py", line 309, in make_histogram
    counts, limits = np.histogram(values, bins=bins)
  File "<__array_function__ internals>", line 6, in histogram
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\lib\histograms.py", line 792, in 
histogram
    bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\lib\histograms.py", line 448, in 
_get_bin_edges
    endpoint=True, dtype=bin_type)
  File "<__array_function__ internals>", line 6, in linspace
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\core\function_base.py", line 135, in linspace
    y = _nx.arange(0, num, dtype=dt).reshape((-1,) + (1,) * ndim(delta))
numpy.core._exceptions.MemoryError: Unable to allocate 10.3 TiB for an array with shape (1418558411252,) and data type float64

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@MohamedA95 @ngocgiang99