Merge pull request #1992 from numbbo/development

merge Development into Master for 2.4 release

.github/ISSUE_TEMPLATE/bug-report-or-feature-request.md → .github/ISSUE_TEMPLATE/bug-report.md

@@ -2,15 +2,15 @@
 name: Bug report
 about: Create a report to help us improve
 title: '[Bug report] please put your title here'
-labels: ''
+labels: 'bug'
 assignees: ''
 
 ---
 
 **Describe the bug**
-Describe the bug, and paste the executed code and the resulting output. If you think it is helpful, you can also talk about the expected behaviour. If applicable, add screenshots to help explain the problem.
+Describe the bug, ideally paste the executed code and the resulting output. If you think it is helpful, also describe the expected behaviour. If applicable, add screenshots to help explain the problem.
 
-**Your System (please complete the following information):**
+**Your System (please complete the following information when it seems relevant):**
  - Operating system: [e.g. macOS 10.15.4, Windows 10, ...]
  - COCO version: [e.g. 2.2.1 (according to `cocopp.__version__`), from the `development` branch, ...]
  - Browser (if relevant): [e.g. chrome, safari]

.github/ISSUE_TEMPLATE/feature-request.md

@@ -0,0 +1,11 @@
+---
+name: Feature request
+about: Create a request to help us improve
+title: '[Feature request] please put your title here'
+labels: 'Feature-request'
+assignees: ''
+
+---
+
+**Describe the feature**
+Describe what is missing and why this is relevant. If you think it is helpful, you can also specify the expected behaviour.

.github/ISSUE_TEMPLATE/submit-a-coco-data-set.md

@@ -1,15 +1,15 @@
 ---
 name: Submit a COCO data set
 about: Describes how to submit a COCO data set to the official archive.
-title: "[COCO SUBMISSION] please put your title here"
-labels: ''
+title: "[DATA SUBMISSION] please put your title here"
+labels: 'data submission'
 assignees: brockho
 
 ---
 
 Hello COCO user!
 
-Here is the right place to submit one or several COCO data sets to the official COCO archive (https://coco.gforge.inria.fr/doku.php?id=algorithms). Please provide the information requested below and submit the issue. Before, we kindly ask you to read about the details in our [How to publish a dataset documentation](https://github.com/numbbo/coco/blob/development/howtos/publish-a-dataset-howto.md).
+Here is the right place to submit one or several COCO data sets to the official COCO archive (https://coco.gforge.inria.fr/doku.php?id=algorithms). Please provide the information requested below and submit the issue. Do not hesitate to read about the details in our [How to publish a dataset documentation](https://github.com/numbbo/coco/blob/development/howtos/publish-a-dataset-howto.md).
 
 #### Reference
 [Put the full reference (for citations) and a link to the pdf here.]
@@ -18,10 +18,7 @@ Here is the right place to submit one or several COCO data sets to the official
 [Please provide a short description for each algorithm in the data.]
 
 #### Link to Data
-[Please upload your COCO data and provide the link here. If you want to submit more than one data set at the same time, please explain which files corresponds to which algorithm.]
+[Please prepare and upload your COCO data (see [how to](https://github.com/numbbo/coco/blob/development/howtos/publish-a-dataset-howto.md)) and provide the link here. If you want to submit more than one data set at the same time, please indicate which file corresponds to which algorithm.]
 
-#### Potentially: Source Code of Experiment
-[We are happy to also share any source code of the algorithm and the COCO experiment, you were running. Please provide, if you wish, any data, link, etc. here]
-
-#### Information About the Authors
-[If you wish to let us know your name and affiliation, we are happy to share this information, too.]
+#### Optional: Source Code of Experiment
+[We are happy to also share any source code of the algorithm and the COCO experiment, you were running. Please provide, if you wish, any code, links, etc. here]

AUTHORS

@@ -13,6 +13,7 @@ Contributors to NumBBO/CoCO have been
 - Verena Heidrich-Meisner
 - Filip Matzner
 - Olaf Mersmann (3)
+- Duc Manh Nguyen 
 - Petr Posik
 - Mike Preuss
 - Raymond Ros (3)
@@ -22,6 +23,7 @@ Contributors to NumBBO/CoCO have been
 - Heike Trautmann
 - Dejan Tusar (3)
 - Tea Tusar (3)
+- Konstantinos Varelas (2)
 
 (2): main contributor
 (3): power contributor

README.md

@@ -30,7 +30,7 @@ from using the platform:
 ![General COCO Structure](coco.png)
 
 For more general information:
-- read our [benchmarking guidelines introduction](http://numbbo.github.io/coco-doc/)
+- read our [benchmarking guidelines introduction](https://arxiv.org/pdf/1603.08785.pdf)
 - read the [COCO experimental setup](http://numbbo.github.io/coco-doc/experimental-setup) description
 - see the [`bbob-biobj` and `bbob-biobj-ext` COCO multi-objective functions testbed](http://numbbo.github.io/coco-doc/bbob-biobj/functions) documentation and the [specificities of the performance assessment for the bi-objective testbeds](http://numbbo.github.io/coco-doc/bbob-biobj/perf-assessment).
 - consult the [BBOB workshops series](http://numbbo.github.io/workshops),
@@ -591,17 +591,16 @@ Citation
 --------
 You may cite this work in a scientific context as
 
-N. Hansen, A. Auger, O. Mersmann, T. Tušar, D. Brockhoff. [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](http://numbbo.github.io/coco-doc/), _ArXiv e-prints_, [arXiv:1603.08785](http://arxiv.org/abs/1603.08785), 2016.
-
+N. Hansen, A. Auger, R. Ros, O. Mersmann, T. Tušar, D. Brockhoff. [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://doi.org/10.1080/10556788.2020.1808977), Optimization Methods and Software, 2020. [[arXiv version](https://arxiv.org/pdf/1603.08785.pdf)]
 ```
-@ARTICLE{hansen2016cocoplat, 
-  author = {Hansen, N. and Auger, A. and Mersmann, O. and 
+@ARTICLE{hansen2020cocoplat, 
+  author = {Hansen, N. and Auger, A. and Ros, R. and Mersmann, O. and 
              Tu{\v s}ar, T. and Brockhoff, D.},
   title = {{COCO}: A Platform for Comparing Continuous Optimizers 
              in a Black-Box Setting},
-  journal = {ArXiv e-prints},
-  volume = {arXiv:1603.08785},
-  year = 2016
+  journal = {Optimization Methods and Software},
+  doi = {https://doi.org/10.1080/10556788.2020.1808977},
+  year = 2020
 }
 ```
 
@@ -619,7 +618,7 @@ Links and Documentation <a name="Links"></a>
   found at http://coco.gforge.inria.fr/ppdata-archive
 * Stay informed about the BBOB workshop series and releases of the NumBBO/Coco software 
   by registering at http://coco.gforge.inria.fr/register
-* Read about the basic principles behind the Coco platform at http://numbbo.github.io/coco-doc/.
+* Read about the basic principles behind the Coco platform in [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://arxiv.org/pdf/1603.08785.pdf).
 * For details on the experimental set-up to carry out the benchmarking
   please refer to http://numbbo.github.io/coco-doc/experimental-setup/.
 * More detailed documentation of the existing benchmark suites can be found here:
@@ -643,7 +642,7 @@ Links and Documentation <a name="Links"></a>
   
 Comprehensive List of Documentations <a name="Documentations"></a>
 --------------------------------------------
-* General introduction: http://numbbo.github.io/coco-doc
+* General introduction: [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://arxiv.org/pdf/1603.08785.pdf)
 * Experimental setup: http://numbbo.github.io/coco-doc/experimental-setup/
 * Testbeds
   - bbob: http://coco.gforge.inria.fr/downloads/download16.00/bbobdocfunctions.pdf

appveyor.yml

@@ -21,7 +21,8 @@ init:
 
 install:
   # - cmd: conda install -y matplotlib numpy scipy setuptools six pytest cython
-  - cmd: pip install matplotlib numpy scipy setuptools six pytest cython  # use pip because conda install numpy fails with Python != 3.6
+  - cmd: pip install matplotlib --ignore-installed # use pip because conda install numpy fails with Python != 3.6
+  - cmd: pip install numpy scipy setuptools six pytest cython 
   - cmd: python do.py run-python install-user
   - cmd: python do.py install-postprocessing
 

code-experiments/test/regression-test/test_suites.py

@@ -98,7 +98,7 @@ def regression_test_a_suite(suite_name, filename):
             data_file_path = ("data/regression_test_%ddata_for_suite_" % ndata) + name + ".py"
 
             if not os.path.exists(data_file_path):
-                remote_data_path = 'http://coco.gforge.inria.fr/regression-tests/'
+                remote_data_path = 'https://coco.gforge.inria.fr/regression-tests/'
                 # download data from remote_data_path:
                 if not os.path.exists(os.path.split(data_file_path)[0]):
                     try:

code-postprocessing/cocopp/archiving.py

@@ -83,7 +83,7 @@
 except ImportError:
     from urllib import urlretrieve as _urlretrieve
 
-coco_url = "http://coco.gforge.inria.fr"
+coco_url = "https://coco.gforge.inria.fr"
 cocopp_home = os.path.abspath(os.path.expanduser(os.path.join("~", ".cocopp")))
 cocopp_home_archives = os.path.join(cocopp_home, "data-archives")
 default_archive_location = os.path.join(cocopp_home, 'data-archives')
@@ -263,7 +263,7 @@ def _get_remote(url, target_folder=None, redownload=False):
             "or use the `update` method to re-download the remote definition file."
             % (_definition_file_to_read(target_folder), url))
         arch.remote_data_path = url
-    assert arch.remote_data_path == url  # check that url was in the definition file
+    assert arch.remote_data_path.replace('https', 'http') == url.replace('https', 'http')  # check that url was in the definition file
     return arch
 
 def get(url_or_folder=None):
@@ -1300,7 +1300,7 @@ def add_archive(self, name):
         """Allow to use a new official archive.
         
         The archive must exist as a subfolder of
-        http://coco.gforge.inria.fr/data-archive
+        https://coco.gforge.inria.fr/data-archive
         """
         self._list += [(self._base + name, None),]
         self.set_as_attributes_in()
@@ -1371,7 +1371,11 @@ def update_all(self):
         # self.set_as_attributes_in(update=True)
         for name in self.names:
             name = name.replace('-', '_')
-            getattr(self, name).update()
+            try:
+                getattr(self, name).update()
+            except AttributeError:
+                if name != 'test':
+                    raise
 
 official_archives = OfficialArchives()
 # TODO-decide: when should we (try to) update these?

code-postprocessing/cocopp/comp2/pprldistr2.py

@@ -78,8 +78,8 @@ def beautify(handles):
     toolsdivers.legend(loc='best')
 
     x = numpy.asarray(axisHandle.get_xticks())
-    axisHandle.set_xticklabels([str(int(numpy.log10(xx))) for xx in x])
     axisHandle.set_xticks(x)
+    axisHandle.set_xticklabels([str(int(numpy.log10(xx))) for xx in x])
     plt.xlim(xlim)
 
 

code-postprocessing/cocopp/dataformatsettings.py

@@ -35,7 +35,7 @@ def align_data_into_evals(self, aligner, data, dataset):
         """
         dataset.evals, maxevals, finalfunvals = aligner(data,
                             self.evaluation_idx, self.function_value_idx)
-        assert all(dataset.evals[0][1:] == 1)
+        assert all(dataset.evals[0][1:] == 1), dataset._evals[0]
         return maxevals, finalfunvals
 
 

code-postprocessing/cocopp/firstsession.py

@@ -30,7 +30,7 @@
 import cocopp
 
 # Collect and unarchive data (~20MB)
-dataurl = 'http://coco.gforge.inria.fr/data-archive/2009/BIPOP-CMA-ES_hansen_noiseless.tgz'
+dataurl = 'https://coco.gforge.inria.fr/data-archive/2009/BIPOP-CMA-ES_hansen_noiseless.tgz'
 filename, headers = urlretrieve(dataurl)
 archivefile = tarfile.open(filename)
 archivefile.extractall()

code-postprocessing/cocopp/genericsettings.py

@@ -92,6 +92,13 @@
                        instancesOfInterestBiobj2017,
                        instancesOfInterestBiobj2018]
 
+reference_algorithm_styles = {'ls': '-', 'linewidth': 4,
+                              'marker': 'D', 'markersize': 11., 'markeredgewidth': 1.5,
+                              'markerfacecolor': 'linen', #'wheat', OR: antiquewhite
+                              'markeredgecolor': 'linen', #'wheat',
+                              'color': 'linen', #'wheat'
+                             }
+
 line_styles = [  # used by ppfigs and pprlmany  
     {'marker': 'o', 'markersize': 31, 'linestyle': '-', 'color': '#000080'},  # 'NavyBlue'
     {'marker': 'd', 'markersize': 26, 'linestyle': '-', 'color': '#ff00ff'},  # 'Magenta'
@@ -106,7 +113,7 @@
     {'marker': 'p', 'markersize': 24, 'linestyle': '-', 'color': 'c'},
     #{'marker': 'H', 'markersize': 23, 'linestyle': '-', 'color': '#bebebe'},  # 'Gray'
     # {'marker': 'o', 'markersize': 23, 'linestyle': '-', 'color': '#ffff00'}, # 'Yellow'
-    {'marker': '3', 'markersize': 23, 'linestyle': '-', 'color': '#adff2f'},  # 'GreenYellow'
+    # {'marker': '3', 'markersize': 23, 'linestyle': '-', 'color': '#adff2f'},  # 'GreenYellow'
     {'marker': '1', 'markersize': 23, 'linestyle': '-', 'color': '#228b22'},  # 'ForestGreen'
     {'marker': 'D', 'markersize': 23, 'linestyle': '-', 'color': '#ffc0cb'},  # 'Lavender'
     {'marker': '<', 'markersize': 23, 'linestyle': '-', 'color': '#87ceeb'},  # 'SkyBlue' close to CornflowerBlue
@@ -135,6 +142,38 @@
     {'marker': '3', 'markersize': 23, 'linestyle': '-', 'color': '#bebebe'},  # 'Gray'
 
 ]
+line_styles_old = [  # used by ppfigs and pprlmany see https://matplotlib.org/gallery/color/named_colors.html https://i.stack.imgur.com/lFZum.png
+    {'marker': 'o', 'markersize': 25, 'linestyle': '-', 'color': 'midnightblue'},
+    {'marker': 'v', 'markersize': 30, 'linestyle': '-', 'color': 'blue'},
+    {'marker': '*', 'markersize': 31, 'linestyle': '-', 'color': 'mediumpurple'},
+    {'marker': 's', 'markersize': 20, 'linestyle': '-', 'color': 'rebeccapurple'},  # square
+    {'marker': '^', 'markersize': 27, 'linestyle': '-', 'color': 'blueviolet'},  # or darkviolet
+    {'marker': 'd', 'markersize': 26, 'linestyle': '-', 'color': 'violet'},
+    {'marker': 'h', 'markersize': 25, 'linestyle': '-', 'color': 'fuchsia'},  # like magenta
+    {'marker': 's', 'markersize': 24, 'linestyle': '-', 'color': 'cornerflowerblue'},  # deeppink seems too close to reddish
+    {'marker': 'H', 'markersize': 24, 'linestyle': '-', 'color': 'crimson'},  # dark reddish
+    {'marker': '<', 'markersize': 24, 'linestyle': '-', 'color': 'deepskyblue'}, # or lightskyblue
+    {'marker': 'D', 'markersize': 24, 'linestyle': '-', 'color': 'darkturquoise'},
+    {'marker': '1', 'markersize': 24, 'linestyle': '-', 'color': 'cyan'},
+    {'marker': '2', 'markersize': 24, 'linestyle': '-', 'color': 'lime'},
+    {'marker': '4', 'markersize': 24, 'linestyle': '-', 'color': 'green'},
+    {'marker': '3', 'markersize': 24, 'linestyle': '-', 'color': 'yellowgreen'},
+    {'marker': 'o', 'markersize': 25, 'linestyle': '-', 'color': 'r'},
+    {'marker': 'v', 'markersize': 30, 'linestyle': '-', 'color': 'b'},
+    {'marker': '*', 'markersize': 31, 'linestyle': '-', 'color': 'm'},
+    {'marker': 's', 'markersize': 20, 'linestyle': '-', 'color': 'c'},  # square
+    {'marker': '^', 'markersize': 27, 'linestyle': '-', 'color': 'y'},
+    {'marker': 'd', 'markersize': 26, 'linestyle': '-', 'color': 'k'},
+    {'marker': 'h', 'markersize': 25, 'linestyle': '-', 'color': 'b'},
+    {'marker': 's', 'markersize': 24, 'linestyle': '-', 'color': 'g'},
+    {'marker': 'H', 'markersize': 24, 'linestyle': '-', 'color': 'c'},
+    {'marker': '<', 'markersize': 24, 'linestyle': '-', 'color': 'r'},
+    {'marker': 'D', 'markersize': 24, 'linestyle': '-', 'color': 'k'},
+    {'marker': '1', 'markersize': 24, 'linestyle': '-', 'color': 'm'},
+    {'marker': '2', 'markersize': 24, 'linestyle': '-', 'color': 'g'},
+    {'marker': '4', 'markersize': 24, 'linestyle': '-', 'color': 'y'},
+    {'marker': '3', 'markersize': 24, 'linestyle': '-', 'color': 'r'}
+]
 line_styles_old = [  # used by ppfigs and pprlmany
     {'marker': 'o', 'markersize': 25, 'linestyle': '-', 'color': 'b'},
     {'marker': 'v', 'markersize': 30, 'linestyle': '-', 'color': 'r'},

code-postprocessing/cocopp/pproc.py

@@ -1176,7 +1176,7 @@ def evals_with_simulated_restarts(self,
                 res += [samplesize * [np.nan]]  # TODO: this is "many" data with little information
                 continue
             elif nsucc == len(evals) and not bootstrap:
-                res += [sorted(evals)]
+                res += [sorted(evals[randintfirst(0, len(evals), samplesize)])]
                 continue
             nindices = ~indices
             assert sum(indices) + sum(nindices) == len(evals)

code-postprocessing/cocopp/test.py

@@ -139,7 +139,7 @@ def retrieve_algorithm(data_path, folder_name, algorithm_name, file_name=None):
     replaced by `cocopp.archiving.get` or `cocopp.archiving.official_archives` or `cocopp.archives.all`)"""
     algorithm_file = join_path(data_path, file_name if file_name else algorithm_name)
     if not os.path.exists(algorithm_file):
-        data_url = 'http://coco.gforge.inria.fr/data-archive/%s/%s' % (folder_name, algorithm_name)
+        data_url = 'https://coco.gforge.inria.fr/data-archive/%s/%s' % (folder_name, algorithm_name)
         urlretrieve(data_url, algorithm_file)
 
 

code-postprocessing/helper-scripts/changeAlgIdAndComment.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
+from __future__ import print_function
 import os
 import sys
 import getopt
@@ -19,17 +20,21 @@
 written: db 28/01/2010
          db 26/06/2013 corrected documentation
          db 19/06/2017 updated docstrings, allow bbob-biobj(-ext) data sets
+         db 16/11/2020 finally moved to python 3 :-)
 
 """
 
 __all__ = ['main']
 
+
 class Usage(Exception):
     def __init__(self, msg):
         self.msg = msg
 
+
 def usage():
-    print main.__doc__
+    print(main.__doc__)
+
 
 def main(argv=None):
     """Main routine.
@@ -65,22 +70,21 @@ def main(argv=None):
 
     """
 
-
     if argv is None:
         argv = sys.argv[1:]
 
     try:
         try:
             opts, args = getopt.getopt(argv, "h",
                                        ["help"])
-        except getopt.error, msg:
-            raise Usage(msg)
+        except:
+            raise Usage(sys.exc_info()[0])
 
         if not args:
             usage()
             sys.exit()
 
-        #Process options
+        # Process options
         for o, a in opts:
             if o in ("-h", "--help"):
                 usage()
@@ -91,11 +95,11 @@ def main(argv=None):
         # check if all arguments are there and ask for them if not:
         if len(args) < 3:
             if len(args) < 2:
-                name = raw_input("You forgot to specify an algorithm name. " +
-                                 "Please enter one (algId):")
+                name = input("You forgot to specify an algorithm name. " +
+                             "Please enter one (algId):")
                 args.append(name)
-            comment = raw_input("You forgot to specify a comment. Please " +
-                                "enter one for algorithm " + args[1] + ":")
+            comment = input("You forgot to specify a comment. Please " +
+                            "enter one for algorithm " + args[1] + ":")
             args.append(comment)
         folder = args[0]
         # make sure that folder name ends with a '/' to be able to append
@@ -107,10 +111,10 @@ def main(argv=None):
         comment = args[2]
 
         if not os.path.exists(folder):
-            print "ERROR: folder " + folder + " does not exist!"
+            print("ERROR: folder " + folder + " does not exist!")
             sys.exit()
         if not os.path.isdir(folder):
-            print "ERROR: " + folder + " is not a directory"
+            print("ERROR: " + folder + " is not a directory")
             sys.exit()
 
         # get all .info files in folder:
@@ -166,9 +170,11 @@ def main(argv=None):
 
         sys.exit()
 
-    except Usage, err:
-        print >>sys.stderr, err.msg
-        print >>sys.stderr, "for help use -h or --help"
+    except IOError as e:
+        errno, strerror = e.args
+        print('>> ' + strerror + e.msg)
+        print('>> ' + errno)
+        print('for help use -h or --help')
         return 2