Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ValueError when running local tests: Tensor [...] must be from the same graph as Tensor [...] #11

Open
Deathn0t opened this issue Jan 2, 2020 · 4 comments
Open

ValueError when running local tests: Tensor [...] must be from the same graph as Tensor [...] #11

Deathn0t opened this issue Jan 2, 2020 · 4 comments

Comments

@Deathn0t
Copy link

Deathn0t commented Jan 2, 2020

Describe the bug
When I run local test I am having this error:

ValueError: Tensor("buffer_size:0", shape=(), dtype=int64, device=/device:CPU:0) must be from the same graph as Tensor("MapDataset_2:0", shape=(), dtype=variant).

The same error occurs when I try to use the notebook

To Reproduce

  1. Run
python run_local_test.py

Expected behavior
Logs obtained:

2020-01-02 11:22:52,457 INFO score.py: ===== Start scoring program. Version: v20191219 =====
2020-01-02 11:22:53,929 INFO ingestion.py: ************************************************
2020-01-02 11:22:53,929 INFO ingestion.py: ******** Processing dataset Miniciao ********
2020-01-02 11:22:53,929 INFO ingestion.py: ************************************************
2020-01-02 11:22:53,929 INFO ingestion.py: Reading training set and test set...
2020-01-02 11:22:56,215 INFO ingestion.py: Creating model...this process should not exceed 20min.
2020-01-02 11:22:56,231 INFO ingestion.py: Initialization success, time spent so far 0.016284942626953125 sec
2020-01-02 11:22:56,231 INFO ingestion.py: ===== Start core part of ingestion program. Version: v20191204 =====
2020-01-02 11:22:56,231 INFO ingestion.py: Begin training the model...
2020-01-02 11:22:56,232 INFO model.py: Counting number of examples on train set.
2020-01-02 11:22:56,476 INFO score.py: Detected the start of ingestion after 4 seconds. Start scoring.
2020-01-02 11:22:56,517 INFO model.py: Finished counting. There are 82 examples for training set.
2020-01-02 11:22:56,517 INFO model.py: Model already trained for 0.0000 epochs.
2020-01-02 11:22:56,517 INFO model.py: Begin training for another 10 steps...
2020-01-02 11:22:57,513 INFO api.py: Tensor shape before preprocessing: (1, 32, 32, 3)
2020-01-02 11:22:58,716 INFO api.py: Tensor shape after preprocessing: (1, 32, 32, 3)
2020-01-02 11:22:58,723 INFO ingestion.py: Failed to run ingestion.
2020-01-02 11:22:58,723 ERROR ingestion.py: Encountered exception:
Tensor("buffer_size:0", shape=(), dtype=int64, device=/device:CPU:0) must be from the same graph as Tensor("MapDataset:0", shape=(), dtype=variant).
Traceback (most recent call last):
  File "/Users/romainegele/Documents/AutoDL/auto-nlp-#1/autodl_starting_kit_stable_TF1/AutoDL_ingestion_program/ingestion.py", line 381, in <module>
    remaining_time_budget=remaining_time_budget)
  File "/Users/romainegele/Documents/AutoDL/auto-nlp-#1/autodl_starting_kit_stable_TF1/AutoDL_sample_code_submission/model.py", line 181, in train
    self.classifier.train(input_fn=train_input_fn, steps=steps_to_train)
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1188, in _train_model_default
    input_fn, ModeKeys.TRAIN))
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1025, in _get_features_and_labels_from_input_fn
    self._call_input_fn(input_fn, mode))
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1116, in _call_input_fn
    return input_fn(**kwargs)
  File "/Users/romainegele/Documents/AutoDL/auto-nlp-#1/autodl_starting_kit_stable_TF1/AutoDL_sample_code_submission/model.py", line 177, in <lambda>
    train_input_fn = lambda: self.input_function(dataset, is_training=True)
  File "/Users/romainegele/Documents/AutoDL/auto-nlp-#1/autodl_starting_kit_stable_TF1/AutoDL_sample_code_submission/model.py", line 309, in input_function
    dataset = dataset.shuffle(buffer_size=self.default_shuffle_buffer)
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 1873, in shuffle
    buffer_size, seed, reshuffle_each_iteration))
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 930, in shuffle
    return ShuffleDataset(self, buffer_size, seed, reshuffle_each_iteration)
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3108, in __init__
    **self._flat_structure)
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_dataset_ops.py", line 5607, in shuffle_dataset
    name=name)
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 367, in _apply_op_helper
    g = ops._get_graph_from_inputs(_Flatten(keywords.values()))
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 5979, in _get_graph_from_inputs
    _assert_same_graph(original_graph_element, graph_element)
  File "/Users/romainegele/opt/anaconda3/envs/auto-dl/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 5914, in _assert_same_graph
    (item, original_item))
ValueError: Tensor("buffer_size:0", shape=(), dtype=int64, device=/device:CPU:0) must be from the same graph as Tensor("MapDataset:0", shape=(), dtype=variant).
2020-01-02 11:22:58,724 INFO ingestion.py: Wrote the file end.txt marking the end of ingestion.
2020-01-02 11:22:58,724 INFO ingestion.py: [-] Done, but encountered some errors during ingestion.
2020-01-02 11:22:58,724 INFO ingestion.py: [-] Overall time spent  2.49 sec
2020-01-02 11:22:58,728 INFO ingestion.py: [Ingestion terminated]
2020-01-02 11:22:59,487 INFO score.py: Detected ingestion program had stopped running because an 'end.txt' file is written by ingestion. Stop scoring now.
2020-01-02 11:22:59,488 INFO score.py: Final area under learning curve for miniciao: 0.0000
2020-01-02 11:22:59,489 INFO score.py: Computing error bars with 10 scorings...
2020-01-02 11:22:59,489 INFO score.py:
Latest prediction NAUC:
* Mean: -1
* Standard deviation: -1
* Variance: -1
2020-01-02 11:22:59,489 INFO score.py: Computing ALC error bars with 5 curves...
2020-01-02 11:22:59,489 INFO score.py:
Area under Learning Curve:
* Mean: 0.0
* Standard deviation: 0.0
* Variance: 0.0
2020-01-02 11:22:59,491 ERROR score.py: [-] Some error occurred in ingestion program. Please see output/error log of Ingestion Step.
2020-01-02 11:22:59,491 INFO score.py: [Scoring terminated]

Desktop (please complete the following information):

  • OS: macOS Catalina version 10.15.2
  • Browser: Chrome
  • Version: Version 79.0.3945.88 (Official Build) (64-bit)

Additional context
Python 3.7.5 (Clang 4.0.1)

Python packages:

absl-py==0.9.0
appnope==0.1.0
astor==0.8.1
attrs==19.3.0
backcall==0.1.0
bleach==3.1.0
certifi==2019.11.28
cycler==0.10.0
decorator==4.4.1
defusedxml==0.6.0
entrypoints==0.3
gast==0.2.2
google-pasta==0.1.8
grpcio==1.26.0
h5py==2.10.0
importlib-metadata==1.3.0
ipykernel==5.1.3
ipython==7.11.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.15.2
Jinja2==2.10.3
joblib==0.14.1
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==5.3.4
jupyter-console==6.0.0
jupyter-core==4.6.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.2
mistune==0.8.4
more-itertools==8.0.2
nbconvert==5.6.1
nbformat==4.4.0
notebook==6.0.2
numpy==1.18.0
opt-einsum==3.1.0
pandas==0.25.3
pandocfilters==1.4.2
parso==0.5.2
pexpect==4.7.0
pickleshare==0.7.5
powerline-status==2.7
prometheus-client==0.7.1
prompt-toolkit==2.0.10
protobuf==3.11.2
psutil==5.6.7
ptyprocess==0.6.0
Pygments==2.5.2
pyparsing==2.4.6
pyrsistent==0.15.6
python-dateutil==2.8.1
pytz==2019.3
PyYAML==5.2
pyzmq==18.1.1
qtconsole==4.6.0
scikit-learn==0.22
scipy==1.4.1
Send2Trash==1.5.0
six==1.13.0
tensorboard==1.15.0
tensorflow==1.15.0
tensorflow-estimator==1.15.1
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
tornado==6.0.3
traitlets==4.3.3
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.16.0
widgetsnbextension==3.5.1
wrapt==1.11.2
zipp==0.6.0
@zhengying-liu
Copy link
Owner

The issue is due to the version of TensorFlow. You used tensorflow==1.15.0 but the starting kit is written for tensorflow==1.13.1. It seems that you are running the command in your own Python environment, but not in the Docker image that we provide: evariste/autodl:cpu-latest or evariste/autodl:gpu-latest.

To avoid all issues related to compatibility and package versions, it's strongly recommended to use Docker. Please follow the instructions here:
https://github.com/zhengying-liu/autodl_starting_kit_stable#local-development-and-testing

The Docker image maybe large (~11GB) for unstable Internet connection, so we provide an alternative way to download it. The instructions can be found on the AutoDL challenge page. We summarize it here:

cat autodl-gpu-latest.tar.part* > autodl-gpu-latest.tar
  • Load the image to Docker using:
docker load < autodl-gpu-latest.tar

If it works for you, I'll close the issue.

@cnjackhu
Copy link

I download the docker image file and run it as container, but after I open the tutorial.ipynb and run it still has error, and I print the version of tensorflow is 1.12.0. But you mention is 1.13.1

The error message is:
AttributeError: module 'tensorflow._api.v1.data.experimental' has no attribute 'AUTOTUNE'

@cnjackhu
Copy link

After I run the command

!pip install --upgrade tensorflow==1.13.1 in the ipynb,

the tensorflow version is still 1.12 and
still the same error message :
AttributeError: module 'tensorflow._api.v1.data.experimental' has no attribute 'AUTOTUNE'

error

@cnjackhu
Copy link

I Found the reason, if run container from the image tar file downloaded from baiduyun , the tensorflow version inside is 1.12.

If pull from image in dockerhub: evariste/autodl:cpu-latest, and run it as container,
The tensorflow version is 1.13.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Deathn0t @zhengying-liu @cnjackhu