Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multirun ignoring some overrides #30

Open
Entodi opened this issue Jun 7, 2023 · 4 comments
Open

Multirun ignoring some overrides #30

Entodi opened this issue Jun 7, 2023 · 4 comments

Comments

@Entodi
Copy link

Entodi commented Jun 7, 2023 •
edited
Loading

Here's an example of a script I run:

python main.py script.dataset.object.fold=${FOLD} --multirun

where FOLD is a bash variable. When it runs in a multi-run mode the sweep does not override the default value that was specified by the script.dataset.object.fold=${FOLD}. So, if my default fold was 0, but FOLD=4, it will just use the default value in the sweep.

[2023-06-07 00:38:48,645][HYDRA] 	#0 : wd=2.2e-06 lr=0.0019 epoch=5 bs=8

To make a small fix, I added a fold parameter in the config file and pointed it in script.dataset.object.fold=${fold}.

I'm not sure if this is the expected behavior to override only the first-order parameters. Because it does override script/dataset=dataset2, for example. Could you clarify this?

Thank you,
Alex
@Delaunay
Copy link
Collaborator

Delaunay commented Jun 7, 2023

Can you give me your hydra configuration ?
Is Fold an hyperparameter ?

@Entodi
Copy link
Author

Entodi commented Jun 7, 2023 •
edited
Loading

The fold is not a hyperparameter, but I want to set it using the SLURM_ARRAY_TASK.

Here is the modified yaml:

defaults:
  - dataset: cifar10
  - override hydra/sweeper: orion

hydra:
  sweep:
    dir: multirun/${now:%Y-%m-%d}/${now:%H-%M-%S}
    subdir: ${hydra.sweeper.experiment.name}/${hydra.sweeper.experiment.uuid}/${hydra.job.id}

  sweeper:
    params:
      a: "uniform(0, 1)"
      b: "uniform(0, 1)"

    experiment:
      name: 'experiment'
      version: '1'

    algorithm:
      type: random
      config:
        seed: 1

    worker:
      n_workers: -1
      max_broken: 3
      max_trials: 100

    storage:
      type: legacy
      database:
        type: pickleddb
        host: 'database.pkl'

# Default values
a: 0
b: 0

The cifar10.yaml in the directory dataset:

object:
  fold: 0

If I run:

HYDRA_FULL_ERROR=1 python main.py dataset=cifar10 dataset.object.fold=3 --multirun

I get:

<my_dir>/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
[2023-06-07 13:59:37,540][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2023-06-07 13:59:37,541][HYDRA] with parametrization {'a': 'uniform(0, 1)', 'b': 'uniform(0, 1)'}
[2023-06-07 13:59:40,153][HYDRA] algorithms is deprecated and will be removed in v0.4.0. Use algorithm instead.
 <my_dir>/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning:In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
[2023-06-07 14:00:02,842][HYDRA] Launching 128 jobs locally
{'dataset': {'object': {'fold': 0}}, 'a': 0.2878, 'b': 0.8581}
[2023-06-07 14:06:53,268][HYDRA]        #1 : dataset=cifar10 b=0.72 a=0.4934
{'dataset': {'object': {'fold': 0}}, 'a': 0.4934, 'b': 0.72}
[2023-06-07 14:06:53,392][HYDRA]        #2 : dataset=cifar10 b=0.5606 a=0.9892
{'dataset': {'object': {'fold': 0}}, 'a': 0.9892, 'b': 0.5606}
[2023-06-07 14:06:53,610][HYDRA]        #3 : dataset=cifar10 b=0.9306 a=0.06306

If I run:

HYDRA_FULL_ERROR=1 python main.py dataset.object.fold=3 --multirun

I get the error:

<my_dir>/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
[2023-06-07 14:00:40,087][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2023-06-07 14:00:40,088][HYDRA] with parametrization {'a': 'uniform(0, 1)', 'b': 'uniform(0, 1)'}
[2023-06-07 14:00:42,677][HYDRA] algorithms is deprecated and will be removed in v0.4.0. Use algorithm instead.
Traceback (most recent call last):
  File "main.py", line 12, in <module>
    main()
  File "<my_dir>/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 465, in _run_app
    run_and_report(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 466, in <lambda>
    lambda: hydra.multirun(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 162, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
  File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/orion_sweeper.py", line 79, in sweep
    return self.sweeper.sweep(arguments)
  File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 500, in sweep
    raise e
  File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 497, in sweep
    self.optimize(self.client)
  File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 517, in optimize
    returns = self.execute_trials(trials)
  File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 557, in execute_trials
    self.validate_batch_is_legal(overrides)
  File "<my_dir>/lib/python3.8/site-packages/hydra/plugins/sweeper.py", line 63, in validate_batch_is_legal
    config_loader.load_sweep_config(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 319, in load_sweep_config
    sweep_config = self.load_configuration(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
    return self._load_configuration_impl(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 244, in _load_configuration_impl
    parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 228, in _parse_overrides_and_create_caching_repo
    parsed_overrides = parser.parse_overrides(overrides=overrides)
  File "<my_dir>/lib/python3.8/site-packages/hydra/core/override_parser/overrides_parser.py", line 96, in parse_overrides
    raise OverrideParseException(
hydra.errors.OverrideParseException: no viable alternative at input '{'object''
See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details

My versions of hydra and orion packages:

(my_env) <my_dir>/orion_test$ pip show hydra-core
Name: hydra-core
Version: 1.3.2
Summary: A framework for elegantly configuring complex applications
Home-page: https://github.com/facebookresearch/hydra
Author: Omry Yadan
Author-email: [email protected]
License: MIT
Location: <my_dir>/anaconda3/envs/my_env/lib/python3.8/site-packages
Requires: antlr4-python3-runtime, importlib-resources, omegaconf, packaging
Required-by: hydra-orion-sweeper
(my_env) <my_dir>/orion_test$ pip show hydra-orion-sweeper
Name: hydra-orion-sweeper
Version: 1.6.2
Summary: Hydra Orion Sweeper plugin
Home-page: https://orion.readthedocs.io/
Author: Pierre Delaunay
Author-email: [email protected]
License: 
Location: <my_dir>/anaconda3/envs/my_env/lib/python3.8/site-packages
Requires: hydra-core, omegaconf, orion, typing-extensions
Required-by:

@Delaunay
Copy link
Collaborator

Delaunay commented Jun 7, 2023

Can you try with the latest version ?
The issue should be resolved, feel free to reopen this if this is not the case

@Delaunay Delaunay closed this as completed Jun 7, 2023
@Entodi
Copy link
Author

Entodi commented Jun 7, 2023 •
edited
Loading

It worked for the command ✅:

HYDRA_FULL_ERROR=1 python main.py dataset=cifar10 dataset.object.fold=3 --multirun

But it does not work if ❌:

HYDRA_FULL_ERROR=1 python main.py dataset.object.fold=3 --multirun

The error:

(orion) <node>:<my_dir>/orion_test$ HYDRA_FULL_ERROR=1 python main.py dataset.object.fold=3 --multirun
<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
[2023-06-07 15:17:18,714][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2023-06-07 15:17:18,714][HYDRA] with parametrization {'a': 'uniform(0, 1)', 'b': 'uniform(0, 1)'}
[2023-06-07 15:17:21,178][HYDRA] algorithms is deprecated and will be removed in v0.4.0. Use algorithm instead.
Traceback (most recent call last):
  File "main.py", line 13, in <module>
    main()
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 465, in _run_app
    run_and_report(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 466, in <lambda>
    lambda: hydra.multirun(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 162, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/orion_sweeper.py", line 79, in sweep
    return self.sweeper.sweep(arguments)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 510, in sweep
    raise e
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 507, in sweep
    self.optimize(self.client)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 527, in optimize
    returns = self.execute_trials(trials)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 567, in execute_trials
    self.validate_batch_is_legal(overrides)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/plugins/sweeper.py", line 63, in validate_batch_is_legal
    config_loader.load_sweep_config(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 319, in load_sweep_config
    sweep_config = self.load_configuration(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
    return self._load_configuration_impl(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 244, in _load_configuration_impl
    parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 228, in _parse_overrides_and_create_caching_repo
    parsed_overrides = parser.parse_overrides(overrides=overrides)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/core/override_parser/overrides_parser.py", line 96, in parse_overrides
    raise OverrideParseException(
hydra.errors.OverrideParseException: no viable alternative at input '{'object''
See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details

@Delaunay Delaunay reopened this Jun 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Entodi @Delaunay