Hyperparameter tuning using hyperopt

[pikaduck]

Was the hyperparameter module for hyperparameter tuning using hyperopt removed in the latest commit? Facing this issue with SequenceTaggerParamSelector

columns = {0 : 'text',
            2 : 'ner'}
corpus : Corpus = ColumnCorpus('/home/ubuntu/sakshi/flairNLP/data',
                                    columns,
                                    train_file = 'train_merged_without_ds.txt',
                                    dev_file = 'dev_merged_without_ds.txt',
                                    test_file = 'test_merged_without_ds.txt')
tag_dictionary = corpus.make_tag_dictionary('ner')
search_space = SearchSpace()
search_space.add(Parameter.EMBEDDINGS, hp.choice, options = [
        StackedEmbeddings([WordEmbeddings('en')]),
        StackedEmbeddings([WordEmbeddings('glove'), FlairEmbeddings('news-forward'), FlairEmbeddings('news-backward')]),
        FlairEmbeddings('news-forward'),
        StackedEmbeddings([FlairEmbeddings('news-forward'), FlairEmbeddings('news-backward')]),
    ])
    
search_space.add(Parameter.HIDDEN_SIZE, hp.choice, options = [128, 256, 512])
search_space.add(Parameter.RNN_LAYERS, hp.choice, options = [1, 2])
search_space.add(Parameter.DROPOUT, hp.uniform, low = 0.4, high = 0.6)
search_space.add(Parameter.LEARNING_RATE, hp.choice, options = [0.01, 0.1, 0.15, 0.2])
search_space.add(Parameter.MINI_BATCH_SIZE, hp.choice, options = [8, 16, 32])
    
param_selector = SequenceTaggerParamSelector(
        corpus,
        tag_type='ner',
        base_path='/home/ubuntu/sakshi/flairNLP/models/resources/merged_without_ds',
        max_epochs=100,
        training_runs=1,
        optimization_value=OptimizationValue.DEV_LOSS
    )
param_selector.optimize(search_space, max_evals=10)

With the above code, it trained 1 model and tested it and the results were also asserted very clearly, but after that, this AttributeError occurred :

- F-score (micro) 0.8181
- F-score (macro) 0.5455
- Accuracy 0.6925

By class:
              precision    recall  f1-score   support

  EntityName     0.8760    0.7828    0.8268      6929
 EntityValue     0.8454    0.7772    0.8099      6642
 EntityRange     0.0000    0.0000    0.0000        11

   micro avg     0.8608    0.7794    0.8181     13582
   macro avg     0.5738    0.5200    0.5455     13582
weighted avg     0.8603    0.7794    0.8178     13582
 samples avg     0.6925    0.6925    0.6925     13582

2021-12-03 03:02:21,145 ----------------------------------------------------------------------------------------------------
job exception: loop of ufunc does not support argument 0 of type Tensor which has no callable conjugate method

  0%|                                                                    | 0/10 [3:32:22<?, ?trial/s, best loss=?]
AttributeError: 'Tensor' object has no attribute 'conjugate'

Note : The above model whose results are shown had FlairEmbeddings('new-forward') in its Embeddings

The module may be deprecated but please help me out here, I have tried training manually with a combination of a zillion parameters and the it's becoming tiring now.

[whoisjones]

hi @pikaduck, yes this has been removed due to compatibility reasons.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[peshmerge]

Hi, I'm currently using Flair for my master's thesis project, and I wonder if there's any other way to do hyperparameter tuning since the module is removed from Flair? Thanks :)

[ijazul-haq]

I'm also stuck in the same issue. I used FlairNLP for my PhD work, but i think this library is not designed for research work.

[peshmerge]

I'm also stuck in the same issue. I used FlairNLP for my PhD work, but i think this library is not designed for research work.

I was able to perform hyperparameter optimization using Optuna. I will write a blog post regarding how to perform that. I don't know how urgent it is for you. Can you wait a bit? If you can't, I can send you an MVP!

[alanakbik]

Hello @peshmerge sounds interesting, please do share the link to the blog post! I'd be happy to try it out.

[Aariciah]

Hi @peshmerge did you end up writing the blog post?