Merge pull request #943 from numbbo/development

Merge development into master before v1.0 release

AUTHORS

@@ -13,7 +13,7 @@ Contributors to NumBBO/CoCO have been
 - Olaf Mersmann (3)
 - Petr Posik
 - Mike Preuss
-- Raimond Ros (3)
+- Raymond Ros (3)
 - Marc Schoenauer
 - Thanh-Do Tran
 - Heike Trautmann

README.md

@@ -15,9 +15,9 @@ Contributions to link further languages (including a better
 example in `C++`) are more than welcome.
 
 For more information, 
-- [benchmarking guidlines introduction](none) (link is coming soon)
-- [COCO experimental setup](http://numbbo.github.io/coco-doc/experimental-setup) description
-- [`bbob-biobj` COCO multi-objective functions testbed](http://numbbo.github.io/coco-doc/bbob-biobj/functions) documentation
+- read our [benchmarking guidelines introduction](http://numbbo.github.io/coco-doc/)
+- read the [COCO experimental setup](http://numbbo.github.io/coco-doc/experimental-setup) description
+- see the [`bbob-biobj` COCO multi-objective functions testbed](http://numbbo.github.io/coco-doc/bbob-biobj/functions) documentation and the [specificities of the performance assessment for the bi-objective testbed](none) (link is coming soon)
 - consult the [BBOB workshops series](http://numbbo.github.io/workshops),
 - consider to [register here](http://numbbo.github.io/register) for news, 
 - see the [previous COCO home page here](http://coco.gforge.inria.fr/) and 
@@ -181,7 +181,7 @@ our issue tracker at https://github.com/numbbo/coco/issues.
    - the `exampleexperiment.???` files in the build folder are the entry points to
      understand the usage of the code (as end-user). They are supposed to
      actually be executable (in case, after compilation, which should be taken
-     care of by do.py and/or make) and run typically random search on (some of)
+     care of by `do.py` and/or `make`) and run typically random search on (some of)
      the provided benchmark suites.
 
 * documentation and examples might not be too meaningful for the time being,
@@ -190,12 +190,13 @@ our issue tracker at https://github.com/numbbo/coco/issues.
   (though the implementation in C naturally looks somewhat different). In the
   future, documentation will be contained mainly in the docs/ subfolder with the
   source code extracted automatically into pdfs in this folder and to web pages
-  under the numbbo.github.io/ domain.
+  under the numbbo.github.io/ domain, see also
+  [here](https://github.com/numbbo/coco#links-and-documentation-).
 
 * the code-experiments/src folder is where most of the important/interesting
   things happen. Many files provide comparatively decent documentation at the
   moment which are translated via doxygen into a more readable web page at
-  numbbo.github.io/coco-doc/C/. Generally:
+  http://numbbo.github.io/coco-doc/C/. Generally:
   - [coco.h](./code-experiments/src/coco.h) is the public interface, in particular
     as used in the example_experiment.c file
   - coco_internal.h provides the type definition of coco_problem_t
@@ -211,11 +212,11 @@ our issue tracker at https://github.com/numbbo/coco/issues.
     benchmark functions, transformations, benchmark suites, etc.
   - currently, three benchmark suites and corresponding logging facilities are
     implemented:
-    * bbob: standard single-objective BBOB benchmark suite with 24 noiseless,
+    * `bbob`: standard single-objective BBOB benchmark suite with 24 noiseless,
       scalable test functions
-    * bbob-biobj: a bi-objective benchmark suite, combining 10 selected
+    * `bbob-biobj`: a bi-objective benchmark suite, combining 10 selected
       functions from the bbob suite, resulting in 55 noiseless functions
-    * toy: a simple, probably easier-to-understand example for reading and testing
+    * `toy`: a simple, probably easier-to-understand example for reading and testing
 
 * code-experiments/tools are a few meta-tools, mainly the amalgamate.py to merge all
   the C code into one file
@@ -224,13 +225,16 @@ our issue tracker at https://github.com/numbbo/coco/issues.
 
 * code-postprocessing/bbob_pproc contains the postprocessing code, written in
   python, with which algorithm data sets can be read in and the performance of
-  the algorithms can be displayed in terms of data profiles, ERT vs. dimension
+  the algorithms can be displayed in terms of data profiles, aRT vs. dimension
   plots, or simple tables.
 
 * code-postprocessing/latex-templates contains LaTeX templates for displaying
-  algorithm performances in publisher-conform PDFs for the GECCO and CEC
-  conferences (for the single-objective bbob suite only, templates for the
-  bi-objective bbob-biobj suite will be provided in a later release).
+  algorithm performances in publisher-conform PDFs for the GECCO
+  conference.
+
+* code-preprocessing/archive-update/ contains internal code for combining
+  the archives of algorithms to create/update the hypervolume
+  reference values for the `bbob-biobj` test suite
 
 * docs should contain an updated version of the documentation, see above.
 
@@ -292,7 +296,7 @@ Traceback (most recent call last):
     raise child_exception
 OSError: [Errno 2] No such file or directory
 ```
-It might be because your system does not know the `matlab` command. To fix this,
+it might be because your system does not know the `matlab` command. To fix this,
 you should edit the file `/etc/paths` and add the path to the `matlab` bin file 
 (Linux/Mac) or add the path to the folder where the `matlab.exe` lies to your 
 Windows path. For instance, the `etc/paths` should look like something like this
@@ -306,7 +310,7 @@ Windows path. For instance, the `etc/paths` should look like something like this
 ```
 
 #### SMS-EMOA example does not compile under Mac 
-With the more complex SMS-EMOA example. The problem is related to the compilation
+With the more complex SMS-EMOA example, the problem is related to the compilation
 of the external C++ hypervolume calculation in `hv.cpp`. 
 
 A fix for this issue consists in adding to the files `hv.cpp` and `paretofront.c`
@@ -397,6 +401,13 @@ effect. In this case it has been successful to remove all previously installed v
 see [here](https://github.com/numbbo/coco/issues/586) for a few more details. 
 
 
+#### Too long paths for postprocessing
+It can happen that the postprocessing fails due to too long paths to the algorithm data.
+Unfortunately, the error you get in this case does not indicate directly to the problem
+but only tells that a certain file could not be read. Please try to shorten the
+folder names in such a case.
+
+
 Details
 -------
 - The C code features an object oriented implementation, where the
@@ -426,10 +437,11 @@ Details
 Links and Documentation <a name="Links"></a>
 -----------------------
 * The [_BBOB workshop series_](http://numbbo.github.io/workshops), which uses the
-  NumBBO/Coco framework extensively, can be tracked at 
+  NumBBO/Coco framework extensively, can be tracked
   [here](http://numbbo.github.io/workshops "BBOB Workshops")
 * Stay informed about the BBOB workshop series and releases of the NumBBO/Coco software 
   by registering at http://coco.gforge.inria.fr/register
+* Read about the basic principles behind the Coco platform at http://numbbo.github.io/coco-doc/.
 * Please refer to http://numbbo.github.io/coco-doc/experimental-setup/ for more details on the
   experimental set-up for black-box optimization benchmarking.
 * More detailed documentation of the existing benchmark suites can be found here:

code-experiments/build/c/example_experiment.c

@@ -7,6 +7,7 @@
 #include <stdlib.h>
 #include <stdio.h>
 #include <string.h>
+#include <time.h>
 
 #include "coco.h"
 
@@ -66,6 +67,20 @@ void my_grid_search(evaluate_function_t evaluate,
                     const double *upper_bounds,
                     const size_t max_budget);
 
+/* Structure and functions needed for timing the experiment */
+typedef struct {
+	size_t number_of_dimensions;
+	size_t current_idx;
+	char **output;
+	size_t previous_dimension;
+	size_t cumulative_evaluations;
+	time_t start_time;
+	time_t overall_start_time;
+} timing_data_t;
+static timing_data_t *timing_data_initialize(coco_suite_t *suite);
+static void timing_data_time_problem(timing_data_t *timing_data, coco_problem_t *problem);
+static void timing_data_finalize(timing_data_t *timing_data);
+
 /**
  * The main method initializes the random number generator and calls the example experiment on the
  * bi-objective suite.
@@ -94,7 +109,8 @@ int main(void) {
 }
 
 /**
- * A simple example of benchmarking random search on a suite with instances from 2016.
+ * A simple example of benchmarking random search on a suite with instances from 2016 that can serve also as
+ * a timing experiment.
  *
  * @param suite_name Name of the suite (use "bbob" for the single-objective and "bbob-biobj" for the
  * bi-objective suite).
@@ -109,6 +125,7 @@ void example_experiment(const char *suite_name,
   size_t run;
   coco_suite_t *suite;
   coco_observer_t *observer;
+  timing_data_t *timing_data;
 
   /* Set some options for the observer. See documentation for other options. */
   char *observer_options =
@@ -121,6 +138,9 @@ void example_experiment(const char *suite_name,
   observer = coco_observer(observer_name, observer_options);
   coco_free_memory(observer_options);
 
+  /* Initialize timing */
+  timing_data = timing_data_initialize(suite);
+
   /* Iterate over all problems in the suite */
   while ((PROBLEM = coco_suite_get_next_problem(suite, observer)) != NULL) {
 
@@ -147,16 +167,21 @@ void example_experiment(const char *suite_name,
 
       /* Break the loop if the algorithm performed no evaluations or an unexpected thing happened */
       if (coco_problem_get_evaluations(PROBLEM) == evaluations_done) {
-        printf("WARNING: Budget has not been exhausted (%lu/%lu evaluations done)!\n", evaluations_done,
-            dimension * BUDGET_MULTIPLIER);
+        printf("WARNING: Budget has not been exhausted (%lu/%lu evaluations done)!\n",
+        		(unsigned long) evaluations_done, (unsigned long) dimension * BUDGET_MULTIPLIER);
         break;
       }
       else if (coco_problem_get_evaluations(PROBLEM) < evaluations_done)
         coco_error("Something unexpected happened - function evaluations were decreased!");
     }
 
+    /* Keep track of time */
+    timing_data_time_problem(timing_data, PROBLEM);
   }
 
+  /* Output and finalize the timing data */
+  timing_data_finalize(timing_data);
+
   coco_observer_free(observer);
   coco_suite_free(suite);
 
@@ -282,5 +307,92 @@ void my_grid_search(evaluate_function_t evaluate,
   coco_free_memory(grid_step);
 }
 
+/**
+ * Allocates memory for the timing_data_t object and initializes it.
+ */
+static timing_data_t *timing_data_initialize(coco_suite_t *suite) {
+
+	timing_data_t *timing_data = (timing_data_t *) coco_allocate_memory(sizeof(*timing_data));
+	size_t function_idx, dimension_idx, instance_idx, i;
+
+	/* Find out the number of all dimensions */
+	coco_suite_decode_problem_index(suite, coco_suite_get_number_of_problems(suite) - 1, &function_idx,
+			&dimension_idx, &instance_idx);
+	timing_data->number_of_dimensions = dimension_idx + 1;
+	timing_data->current_idx = 0;
+	timing_data->output = (char **) coco_allocate_memory(timing_data->number_of_dimensions * sizeof(char *));
+	for (i = 0; i < timing_data->number_of_dimensions; i++) {
+		timing_data->output[i] = NULL;
+	}
+	timing_data->previous_dimension = 0;
+	timing_data->cumulative_evaluations = 0;
+	time(&timing_data->start_time);
+	time(&timing_data->overall_start_time);
+
+	return timing_data;
+}
 
+/**
+ * Keeps track of the total number of evaluations and elapsed time. Produces an output string when the
+ * current problem is of a different dimension than the previous one or when NULL.
+ */
+static void timing_data_time_problem(timing_data_t *timing_data, coco_problem_t *problem) {
+
+	double elapsed_seconds = 0;
+
+	if ((problem == NULL) || (timing_data->previous_dimension != coco_problem_get_dimension(problem))) {
+
+		/* Output existing timing information */
+		if (timing_data->cumulative_evaluations > 0) {
+			time_t now;
+			time(&now);
+			elapsed_seconds = difftime(now, timing_data->start_time) / (double) timing_data->cumulative_evaluations;
+			timing_data->output[timing_data->current_idx++] = coco_strdupf("d=%lu done in %.2e seconds/evaluation\n",
+					timing_data->previous_dimension, elapsed_seconds);
+		}
+
+		if (problem != NULL) {
+			/* Re-initialize the timing_data */
+			timing_data->previous_dimension = coco_problem_get_dimension(problem);
+			timing_data->cumulative_evaluations = coco_problem_get_evaluations(problem);
+			time(&timing_data->start_time);
+		}
+
+	} else {
+		timing_data->cumulative_evaluations += coco_problem_get_evaluations(problem);
+	}
+}
 
+/**
+ * Outputs and finalizes the given timing data.
+ */
+static void timing_data_finalize(timing_data_t *timing_data) {
+
+	/* Record the last problem */
+	timing_data_time_problem(timing_data, NULL);
+
+  if (timing_data) {
+  	size_t i;
+  	double elapsed_seconds;
+		time_t now;
+		int hours, minutes, seconds;
+
+		time(&now);
+		elapsed_seconds = difftime(now, timing_data->overall_start_time);
+
+  	printf("\n");
+  	for (i = 0; i < timing_data->number_of_dimensions; i++) {
+    	if (timing_data->output[i]) {
+				printf("%s", timing_data->output[i]);
+				coco_free_memory(timing_data->output[i]);
+    	}
+    }
+  	hours = (int) elapsed_seconds / 3600;
+  	minutes = ((int) elapsed_seconds % 3600) / 60;
+  	seconds = (int)elapsed_seconds - (hours * 3600) - (minutes * 60);
+  	printf("Total elapsed time: %dh%02dm%02ds\n", hours, minutes, seconds);
+
+    coco_free_memory(timing_data->output);
+    coco_free_memory(timing_data);
+  }
+}

code-experiments/build/java/ExampleExperiment.java

@@ -95,6 +95,9 @@ public static void exampleExperiment(String suiteName, String observerName, Rand
 			Observer observer = new Observer(observerName, observerOptions);
 			Benchmark benchmark = new Benchmark(suite, observer);
 
+			/* Initialize timing */
+			Timing timing = new Timing();
+
 			/* Iterate over all problems in the suite */
 			while ((PROBLEM = benchmark.getNextProblem()) != null) {
 
@@ -128,10 +131,15 @@ public static void exampleExperiment(String suiteName, String observerName, Rand
 						System.out.println("ERROR: Something unexpected happened - function evaluations were decreased!");
 				}
 
+				/* Keep track of time */
+				timing.timeProblem(PROBLEM);
 			}
-
+
+			/* Output the timing data */
+			timing.output();
+
 			benchmark.finalizeBenchmark();
-			
+
 		} catch (Exception e) {
 			System.err.println(e.toString());
 		}