From d27944780496050b3497499158973dc599c675d6 Mon Sep 17 00:00:00 2001 From: Yu Wu Date: Wed, 3 Jan 2024 16:52:29 +0800 Subject: [PATCH 1/9] fix upload data after generation edit document Signed-off-by: Yu Wu --- doc/fate_test.md | 9 +- doc/fate_test_command.md | 845 ++++++++++++++++++++ python/fate_test/scripts/_utils.py | 8 + python/fate_test/scripts/data_cli.py | 122 +-- python/fate_test/scripts/performance_cli.py | 6 +- 5 files changed, 870 insertions(+), 120 deletions(-) create mode 100644 doc/fate_test_command.md diff --git a/doc/fate_test.md b/doc/fate_test.md index d5cb4fd..746dfbd 100644 --- a/doc/fate_test.md +++ b/doc/fate_test.md @@ -47,14 +47,14 @@ shown in last step ## command types -- suite: used for running testsuites, +- [suite](./fate_test_command.md#testsuite): used for running testsuites, collection of FATE jobs ```bash fate_test suite -i ``` -- data: used for upload, delete, and generate dataset +- [data](./fate_test_command.md#data): used for upload, delete, and generate dataset - upload/delete data command: @@ -74,14 +74,15 @@ shown in last step fate_test data generate -i ``` -- benchmark-quality: used for comparing modeling quality between FATE +- [benchmark-quality](./fate_test_command.md#benchmark-quality): used for comparing modeling quality between FATE and other machine learning systems ```bash fate_test bq -i ``` -- benchmark-performance: used for checking FATE algorithm performance; user +- [benchmark-performance](./fate_test_command.md#benchmark-performance): used for checking FATE algorithm performance; + user should first generate and upload data before running performance testsuite ```bash diff --git a/doc/fate_test_command.md b/doc/fate_test_command.md new file mode 100644 index 0000000..e723ec6 --- /dev/null +++ b/doc/fate_test_command.md @@ -0,0 +1,845 @@ +# FATE Test + +A collection of useful tools to running FATE's test. + +## Testsuite + +Testsuite is used for running a collection of jobs in sequence. Data +used for jobs could be uploaded before jobs are submitted and, +optionally, be cleaned after jobs finish. This tool is useful for FATE's +release test. + +### command options + +```bash +fate_test suite --help +``` + +1. include: + + ```bash + fate_test suite -i + ``` + + will run testsuites in + *path1* + +2. exclude: + + ```bash + fate_test suite -i -e -e ... + ``` + + will run testsuites in *path1* but not in *path2* and *path3* + +3. glob: + + ```bash + fate_test suite -i -g "hetero*" + ``` + + will run testsuites in sub directory start with *hetero* of + *path1* + +4. replace: + + ```bash + fate_test suite -i -r '{"maxIter": 5}' + ``` + + will find all key-value pair with key "maxIter" in + data conf or + conf or + dsl and replace the value with 5 + +5. timeout: + + ```bash + fate_test suite -i -m 3600 + ``` + + will run testsuites in *path1* and timeout when job does not finish + within 3600s; if tasks need more time, use a larger threshold + +6. task-cores + + ```bash + fate_test suite -i -p 4 + ``` + + will run testsuites in *path1* with script config "task-cores" set to 4; + only effective for DSL conf + +7. update-job-parameters + + ```bash + fate_test suite -i -uj {} + ``` + + will run testsuites in *path1* with respective job parameters set to + provided values + +8. update-component-parameters + + ```bash + fate_test suite -i -uc {} + ``` + + will run testsuites in *path1* with respective component parameters + set to provided values + +9. skip-jobs: + + ```bash + fate_test suite -i --skip-pipeline-jobs + ``` + + will run testsuites in *path1* but skip all *pipeline tasks* in + testsuites. + +10. skip-data: + + ```bash + fate_test suite -i --skip-data + ``` + + will run testsuites in *path1* without uploading data specified in + *testsuite.yaml*. + +11. data only: + + ```bash + fate_test suite -i --data-only + ``` + + will only upload data specified in *testsuite.yaml* without running + jobs + +12. disable-clean-data: + + ```bash + fate_test suite -i --disable-clean-data + ``` + + will run testsuites in *path1* without removing data from storage + after tasks + finish + +13. enable-clean-data: + + ```bash + fate_test suite -i --enable-clean-data + ``` + + will remove data from storage after finishing running testsuites + +14. yes: + + ```bash + fate_test suite -i --yes + ``` + + will run testsuites in *path1* directly, skipping double check + +### testsuite configuration + +Configuration of jobs should be specified in a testsuite whose file name +ends with "\*testsuite.yaml". For testsuite examples, please +refer [here](https://github.com/FederatedAI/FATE/tree/master/examples/pipeline. + +A testsuite includes the following elements: + +- data: list of local data to be uploaded before running FATE jobs + + - file: path to original data file to be uploaded, should be + relative to testsuite or FATE installation path + - meta: meta info on table, including data type and file format + - head: whether file includes header + - partitions: number of partition for data storage + - table\_name: table name in storage + - namespace: table namespace in storage + - role: which role to upload the data, as specified in + fate\_test.config; naming format is: + "{role\_type}\_{role\_index}", index starts at 0 + + ```yaml + - file: examples/data/breast_hetero_guest.csv + meta: + delimiter: "," + dtype: float64 + input_format: dense + label_type: int64 + label_name: y + match_id_name: id + match_id_range: 0 + tag_value_delimiter: ":" + tag_with_value: false + weight_type: float64 + partitions: 4 + head: true + extend_sid: true + table_name: breast_hetero_guest + namespace: experiment + role: guest_0 + ``` + +- tasks: includes arbitrary number of pipeline jobs with + paths to corresponding python script + + - job: name of job to be run, must be unique within each group + list + + - script: path to pipeline script, should be relative to + testsuite + + ```yaml + tasks: + normal-lr: + script: test_lr.py + lr-cv: + script: test_lr_cv.py + ``` + +## Benchmark Quality + +Benchmark-quality is used for comparing modeling quality between FATE +and other machine learning systems. Benchmark produces a metrics +comparison summary for each benchmark job group. + +Benchmark can also compare metrics of different models from the same +script/PipeLine job. Please refer to the [script writing +guide](#testing-script) below for +instructions. + +```bash +fate_test benchmark-quality -i examples/benchmark_quality/hetero_linear_regression +``` + +```bash +|----------------------------------------------------------------------| +| Data Summary | +|-------+--------------------------------------------------------------| +| Data | Information | +|-------+--------------------------------------------------------------| +| train | {'guest': 'motor_hetero_guest', 'host': 'motor_hetero_host'} | +| test | {'guest': 'motor_hetero_guest', 'host': 'motor_hetero_host'} | +|-------+--------------------------------------------------------------| + + +|-------------------------------------------------------------------------------------------------------------------------------------| +| Metrics Summary | +|-------------------------------------------+-------------------------+--------------------+---------------------+--------------------| +| Model Name | root_mean_squared_error | r2_score | mean_squared_error | explained_variance | +|-------------------------------------------+-------------------------+--------------------+---------------------+--------------------| +| local-hetero_linear_regression-regression | 0.312552080517407 | 0.9040310440206087 | 0.09768880303575968 | 0.9040312584426697 | +| FATE-hetero_linear_regression-regression | 0.3139977881119483 | 0.9031411831961411 | 0.09859461093919598 | 0.903146386539082 | +|-------------------------------------------+-------------------------+--------------------+---------------------+--------------------| +|-------------------------------------| +| Match Results | +|-------------------------+-----------| +| Metric | All Match | +| root_mean_squared_error | True | +| r2_score | True | +| mean_squared_error | True | +| explained_variance | True | +|-------------------------+-----------| + + +|-------------------------------------------------------------------------------------| +| FATE Script Metrics Summary | +|--------------------+---------------------+--------------------+---------------------| +| Script Model Name | min | max | mean | +|--------------------+---------------------+--------------------+---------------------| +| linr_train-FATE | -1.5305666678748353 | 1.4968292506353484 | 0.03948016870496807 | +| linr_validate-FATE | -1.5305666678748353 | 1.4968292506353484 | 0.03948016870496807 | +|--------------------+---------------------+--------------------+---------------------| +|---------------------------------------| +| FATE Script Metrics Match Results | +|----------------+----------------------| +| Metric | All Match | +|----------------+----------------------| +| min | True | +| max | True | +| mean | True | +|----------------+----------------------| +``` + +### command options + +use the following command to show help message + +```bash +fate_test benchmark-quality --help +``` + +1. include: + + ```bash + fate_test benchmark-quality -i + ``` + + will run benchmark testsuites in + *path1* + +2. exclude: + + ```bash + fate_test benchmark-quality -i -e -e ... + ``` + + will run benchmark testsuites in *path1* but not in *path2* and + *path3* + +3. glob: + + ```bash + fate_test benchmark-quality -i -g "hetero*" + ``` + + will run benchmark testsuites in sub directory start with *hetero* + of + *path1* + +4. tol: + + ```bash + fate_test benchmark-quality -i -t 1e-3 + ``` + + will run benchmark testsuites in *path1* with absolute tolerance of + difference between metrics set to 0.001. If absolute difference + between metrics is smaller than *tol*, then metrics are considered + almost equal. Check benchmark testsuite [writing + guide](#benchmark-testsuite) on setting alternative tolerance. + +5. storage-tag + + ```bash + fate_test performance -i -s test + ``` + + will run benchmark testsuites in *path1* with performance stored + under provided tag for future comparison; note that FATE-Test + always records the most recent run for each tag; if the same tag + is used more than once, only metrics from the latest job is + kept + +6. history-tag + + ```bash + fate_test performance -i -v test1 -v test2 + ``` + + will run benchmark testsuites in *path1* with performance compared + to history jobs under provided + tag(s) + +7. skip-data: + + ```bash + fate_test benchmark-quality -i --skip-data + ``` + + will run benchmark testsuites in *path1* without uploading data + specified in + *benchmark.yaml*. + +8. disable-clean-data: + + ```bash + fate_test suite -i --disable-clean-data + ``` + + will run benchmark testsuites in *path1* without removing data from + storage after tasks + finish + +9. enable-clean-data: + + ```bash + fate_test suite -i --enable-clean-data + ``` + + will remove data from storage after finishing running benchmark + testsuites + +10. yes: + ```bash + fate_test benchmark-quality -i --yes + ``` + + will run benchmark testsuites in *path1* directly, skipping double + check + +### benchmark job configuration + +Configuration of jobs should be specified in a benchmark testsuite whose +file name ends with "\*benchmark.yaml". For benchmark testsuite example, +please refer [here](https://github.com/FederatedAI/FATE/tree/master/examples/benchmark_quality). + +A benchmark testsuite includes the following elements: + +- data: list of local data to be uploaded before running FATE jobs + + - file: path to original data file to be uploaded, should be + relative to testsuite or FATE installation path + - meta: meta info on table, including data type and file format + - head: whether file includes header + - partitions: number of partition for data storage + - table\_name: table name in storage + - namespace: table namespace in storage + - role: which role to upload the data, as specified in + fate\_test.config; naming format is: + "{role\_type}\_{role\_index}", index starts at 0 + + ```yaml + - file: examples/data/breast_hetero_guest.csv + meta: + delimiter: "," + dtype: float64 + input_format: dense + label_type: int64 + label_name: y + match_id_name: id + match_id_range: 0 + tag_value_delimiter: ":" + tag_with_value: false + weight_type: float64 + partitions: 4 + head: true + extend_sid: true + table_name: breast_hetero_guest + namespace: experiment + role: guest_0 + ``` + +- job group: each group includes arbitrary number of jobs with paths + to corresponding script and configuration + + - job: name of job to be run, must be unique within each group + list + + - script: path to [testing script](#testing-script), should be + relative to testsuite + - conf: path to job configuration file for script, should be + relative to testsuite + + ```yaml + local: + script: "./sklearn-lr-binary.py" + conf: "./breast_lr_sklearn_config.yaml" + + ``` + + - compare\_setting: additional setting for quality metrics + comparison, currently only takes `relative_tol` + + If metrics *a* and *b* satisfy *abs(a-b) \<= max(relative\_tol + \* max(abs(a), abs(b)), absolute\_tol)* (from [math + module](https://docs.python.org/3/library/math.html#math.isclose)), + they are considered almost equal. In the below example, metrics + from "local" and "FATE" jobs are considered almost equal if + their relative difference is smaller than *0.05 \* + max(abs(local\_metric), abs(pipeline\_metric)*. + + ```yaml + hetero_lr-binary-0-breast: + local: + script: "./sklearn-lr-binary.py" + conf: "./breast_lr_sklearn_config.yaml" + FATE-hetero-lr: + script: "./pipeline-lr-binary.py" + conf: "./breast_config.yaml" + FATE-hetero-sshe-lr: + script: "./pipeline-sshe-lr-binary.py" + conf: "./breast_config.yaml" + compare_setting: + relative_tol: 0.01 + ``` + +### testing script + +All job scripts need to have `Main` function as an entry point for +executing jobs; scripts should return two dictionaries: first with data +information key-value pairs: {data\_type}: {data\_name\_dictionary}; the +second contains {metric\_name}: {metric\_value} key-value pairs for +metric comparison. + +By default, the final data summary shows the output from the job named +"FATE"; if no such job exists, data information returned by the first +job is shown. For clear presentation, we suggest that user follow this +general [guideline](../../examples/data/README.md#data-set-naming-rule) +for data set naming. In the case of multi-host task, consider numbering +host as such: + + {'guest': 'default_credit_homo_guest', + 'host_1': 'default_credit_homo_host_1', + 'host_2': 'default_credit_homo_host_2'} + +Returned quality metrics of the same key are to be compared. Note that +only **real-value** metrics can be compared. + +To compare metrics of different models from the same script, metrics of +each model need to be wrapped into dictionary in the same format as the +general metric output above. + +In the returned dictionary of script, use reserved key `script_metrics` +to indicate the collection of metrics to be compared. + +- FATE script: `Main` should have three inputs: + - config: job configuration, + object loaded from "fate\_test\_config.yaml" + - param: job parameter setting, dictionary loaded from "conf" file + specified in benchmark testsuite + - namespace: namespace suffix, user-given *namespace* or generated + timestamp string when using *namespace-mangling* +- non-FATE script: `Main` should have one or two inputs: + - param: job parameter setting, dictionary loaded from "conf" file + specified in benchmark testsuite + - (optional) config: job configuration, + object loaded from "fate\_test\_config.yaml" + +Note that `Main` in FATE & non-FATE scripts can also be set to not take any +input argument. + +## Benchmark Performance + +`Performance` sub-command is used to test +efficiency of designated FATE jobs. + +Examples may be found [here](https://github.com/FederatedAI/FATE/tree/master/examples/benchmark_quality). + +### command options + +```bash +fate_test performance --help +``` + +1. job-type: + + ```bash + fate_test performance -t intersect + ``` + + will run testsuites from intersect sub-directory (set in config) in + the default performance directory; note that only one of `task` and + `include` is + needed + +2. include: + + ```bash + fate_test performance -i ; note that only one of ``task`` and ``include`` needs to be specified. + ``` + + will run testsuites in *path1*. Note that only one of `task` and + `include` needs to be specified; when both are given, path from + `include` takes + priority. + +3. replace: + + ```bash + fate_test performance -i -r '{"maxIter": 5}' + ``` + + will find all key-value pair with key "maxIter" in + data conf or + conf or + dsl and replace the value with 5 + +4. timeout: + + ```bash + fate_test performance -i -m 3600 + ``` + + will run testsuites in *path1* and timeout when job does not finish + within 3600s; if tasks need more time, use a larger threshold + +5. max-iter: + + ```bash + fate_test performance -i -e 5 + ``` + + will run testsuites in *path1* with all values to key "max\_iter" + set to 5 + +6. max-depth + + ```bash + fate_test performance -i -d 4 + ``` + + will run testsuites in *path1* with all values to key "max\_depth" + set to 4 + +7. num-trees + + ```bash + fate_test performance -i -nt 5 + ``` + + will run testsuites in *path1* with all values to key "num\_trees" + set to 5 + +8. task-cores + + ```bash + fate_test performance -i -p 4 + ``` + + will run testsuites in *path1* with "task\_cores" in script config set to 4 + +9. update-job-parameters + + ```bash + fate_test performance -i -uj {} + ``` + + will run testsuites in *path1* with respective job parameters set to + provided values + +10. update-component-parameters + + ```bash + fate_test performance -i -uc {} + ``` + + will run testsuites in *path1* with respective component parameters + set to provided values + +11. storage-tag + + ```bash + fate_test performance -i -s test + ``` + + will run testsuites in *path1* with performance time stored under + provided tag for future comparison; note that FATE-Test always + records the most recent run for each tag; if the same tag is used + more than once, only performance from the latest job is + kept + +12. history-tag + + ```bash + fate_test performance -i -v test1 -v test2 + ``` + + will run testsuites in *path1* with performance time compared to + history jobs under provided + tag(s) + +13. skip-data: + + ```bash + fate_test performance -i --skip-data + ``` + + will run testsuites in *path1* without uploading data specified in + *testsuite.yaml*. + +14. disable-clean-data: + + ```bash + fate_test performance -i --disable-clean-data + ``` + + will run testsuites in *path1* without removing data from storage + after tasks finish + +15. yes: + + ```bash + fate_test performance -i --yes + ``` + + will run testsuites in *path1* directly, skipping double check + +## data + +`Data` sub-command is used for upload, +delete, and generate +dataset. + +### data command options + +```bash +fate_test data --help +``` + +1. include: + + ```bash + fate_test data [upload|delete] -i + ``` + + will upload/delete dataset in testsuites in + *path1* + +2. exclude: + + ```bash + fate_test data [upload|delete] -i -e -e ... + ``` + + will upload/delete dataset in testsuites in *path1* but not in + *path2* and + *path3* + +3. glob: + + ```bash + fate_test data [upload|delete] -i -g "hetero*" + ``` + + will upload/delete dataset in testsuites in sub directory start with + *hetero* of + *path1* + +### generate command options + +```bash +fate_test data --help +``` + +1. include: + + ```bash + fate_test data generate -i + ``` + + will generate dataset in testsuites in *path1*; note that only one + of `type` and `include` is + needed + +2. host-data-type: + + ```bash + fate_test suite -i -ht {tag-value | dense | tag } + ``` + + will generate dataset in testsuites *path1* where host data are of + selected + format + +3. sparsity: + + ```bash + fate_test suite -i -s 0.2 + ``` + + will generate dataset in testsuites in *path1* with sparsity at 0.1; + useful for tag-formatted + data + +4. encryption-type: + + ```bash + fate_test data generate -i -p {sha256 | md5} + ``` + + will generate dataset in testsuites in *path1* with hash id using + SHA256 + method + +5. match-rate: + + ```bash + fate_test data generate -i -m 1.0 + ``` + + will generate dataset in testsuites in *path1* where generated host + and guest data have intersection rate of + 1.0 + +6. guest-data-size: + + ```bash + fate_test data generate -i -ng 10000 + ``` + + will generate dataset in testsuites *path1* where guest data each + have 10000 + entries + +7. host-data-size: + + ```bash + fate_test data generate -i -nh 10000 + ``` + + will generate dataset in testsuites *path1* where host data have + 10000 + entries + +8. guest-feature-num: + + ```bash + fate_test data generate -i -fg 20 + ``` + + will generate dataset in testsuites *path1* where guest data have 20 + features + +9. host-feature-num: + + ```bash + fate_test data generate -i -fh 200 + ``` + + will generate dataset in testsuites *path1* where host data have 200 + features + +10. output-path: + + ```bash + fate_test data generate -i -o + ``` + + will generate dataset in testsuites *path1* and write file to + *path2* + +11. force: + + ```bash + fate_test data generate -i -o --force + ``` + + will generate dataset in testsuites *path1* and write file to + *path2*; will overwrite existing file(s) if designated file name + found under + *path2* + +12. split-host: + + ```bash + fate_test data generate -i -nh 10000 --split-host + ``` + + will generate dataset in testsuites *path1*; 10000 entries will be + divided equally among all host data + sets + +13. upload-data + + ```bash + fate_test data generate -i --upload-data + ``` + + will generate dataset in testsuites *path1* and upload generated + data for all parties to + FATE + +14. remove-data + + ```bash + fate_test data generate -i --upload-data --remove-data + ``` + + (effective with `upload-data` set to True) will delete generated + data after generate and upload dataset in testsuites + *path1* + diff --git a/python/fate_test/scripts/_utils.py b/python/fate_test/scripts/_utils.py index 3a006ca..c10bca7 100644 --- a/python/fate_test/scripts/_utils.py +++ b/python/fate_test/scripts/_utils.py @@ -190,3 +190,11 @@ def _add_replace_hook(replace): DATA_LOAD_HOOK.add_replace_hook(replace) CONF_LOAD_HOOK.add_replace_hook(replace) DSL_LOAD_HOOK.add_replace_hook(replace) + + +def _update_data_path(suite, output_dir): + for data in suite.dataset: + data_name = os.path.basename(data.file) + data_file_path = os.path.join(str(output_dir), data_name) + data.file = data_file_path + data.config['file'] = data_file_path diff --git a/python/fate_test/scripts/data_cli.py b/python/fate_test/scripts/data_cli.py index b50b002..c580205 100644 --- a/python/fate_test/scripts/data_cli.py +++ b/python/fate_test/scripts/data_cli.py @@ -1,17 +1,15 @@ import os import re -import sys -import time import uuid -from datetime import timedelta import click +from ruamel import yaml + from fate_test._client import Clients from fate_test._config import Config from fate_test._io import LOGGER, echo from fate_test.scripts._options import SharedOptions -from fate_test.scripts._utils import _load_testsuites, _delete_data, _big_data_task, _upload_data -from ruamel import yaml +from fate_test.scripts._utils import _load_testsuites, _delete_data, _big_data_task, _upload_data, _update_data_path @click.group(name="data") @@ -161,11 +159,11 @@ def delete(ctx, include, exclude, glob, yes, suite_type, **kwargs): help="Generated data will be uploaded") @click.option('--remove-data', is_flag=True, default=False, help="The generated data will be deleted") -@click.option('--use-local-data', is_flag=True, default=False, - help="The existing data of the server will be uploaded, This parameter is not recommended for " - "distributed applications") +# @click.option('--use-local-data', is_flag=True, default=False, +# help="The existing data of the server will be uploaded, This parameter is not recommended for " +# "distributed applications") # @click.option('--parallelize', is_flag=True, default=False, -# help="It is directly used to upload data, and will not generate data") +# help="It is directly used to upload data, and will not generate data") @SharedOptions.get_shared_options(hidden=True) @click.pass_context def generate(ctx, include, host_data_type, encryption_type, match_rate, sparsity, guest_data_size, @@ -208,48 +206,14 @@ def generate(ctx, include, host_data_type, encryption_type, match_rate, sparsity _big_data_task(include, guest_data_size, host_data_size, guest_feature_num, host_feature_num, host_data_type, config_inst, encryption_type, match_rate, sparsity, force, split_host, output_path) if upload_data: - """if use_local_data: - _config.use_local_data = 0 - _config.data_switch = remove_data""" client = Clients(config_inst) for suite in suites: + output_dir = output_path if output_path else os.path.abspath(config_inst.cache_directory) + _update_data_path(suite, output_dir) + # echo.echo(f"data files: {[data.file for data in suite.dataset]}") _upload_data(client, suite, config_inst) -"""@data_group.command("download") -@click.option("-t", "--type", type=click.Choice(["mnist"]), default="mnist", - help="config file") -@click.option('-o', '--output-path', type=click.Path(exists=True), - help="output path of mnist data, the default path is examples/data") -@SharedOptions.get_shared_options(hidden=True) -@click.pass_context -def download_mnists(ctx, output_path, **kwargs): - ctx.obj.update(**kwargs) - ctx.obj.post_process() - namespace = ctx.obj["namespace"] - config_inst = ctx.obj["config"] - yes = ctx.obj["yes"] - echo.welcome() - echo.echo(f"testsuite namespace: {namespace}", fg='red') - - if output_path is None: - config = get_config(config_inst) - output_path = str(config.data_base_dir) + "/examples/data/" - if not yes and not click.confirm("running?"): - return - try: - download_mnist(Path(output_path), "mnist_train") - download_mnist(Path(output_path), "mnist_eval", is_train=False) - except Exception: - exception_id = uuid.uuid1() - echo.echo(f"exception_id={exception_id}") - LOGGER.exception(f"exception id: {exception_id}") - finally: - echo.stdout_newline() - echo.farewell() - echo.echo(f"testsuite namespace: {namespace}", fg='red')""" - - @data_group.command("query_schema") @click.option('-cpn', '--component-name', required=True, type=str, help="component name(task name)") @click.option('-j', '--job-id', required=True, type=str, help="job id") @@ -354,69 +318,3 @@ def download_mnist(base, name, is_train=True): with config_path.open("w") as f: yaml.safe_dump(config, f, indent=2, default_flow_style=False) - -def data_upload(clients, conf: Config, upload_config): - def _await_finish(job_id, task_name=None): - deadline = time.time() + sys.maxsize - start = time.time() - param = dict( - job_id=job_id, - role=None - ) - while True: - stdout = clients["guest_0"].flow_client("job/query", param) - status = stdout["data"][0]["f_status"] - elapse_seconds = int(time.time() - start) - date = time.strftime('%Y-%m-%d %X') - if task_name: - log_msg = f"[{date}][{task_name}]{status}, elapse: {timedelta(seconds=elapse_seconds)}" - else: - log_msg = f"[{date}]{job_id} {status}, elapse: {timedelta(seconds=elapse_seconds)}" - if (status == "running" or status == "waiting") and time.time() < deadline: - print(log_msg, end="\r") - time.sleep(1) - continue - else: - print(" " * 60, end="\r") # clean line - echo.echo(log_msg) - return status - - task_data = upload_config["data"] - for i, data in enumerate(task_data): - format_msg = f"@{data['file']} >> {data['namespace']}.{data['table_name']}" - echo.echo(f"[{time.strftime('%Y-%m-%d %X')}]uploading {format_msg}") - try: - data["file"] = str(os.path.join(conf.data_base_dir, data["file"])) - param = dict( - file=data["file"], - head=data["head"], - partition=data["partition"], - table_name=data["table_name"], - namespace=data["namespace"] - ) - stdout = clients["guest_0"].flow_client("data/upload", param, drop=1) - job_id = stdout.get('jobId', None) - echo.echo(f"[{time.strftime('%Y-%m-%d %X')}]upload done {format_msg}, job_id={job_id}\n") - if job_id is None: - echo.echo("table already exist. To upload again, Please add '-f 1' in start cmd") - continue - _await_finish(job_id) - param = dict( - table_name=data["table_name"], - namespace=data["namespace"] - ) - stdout = clients["guest_0"].flow_client("table/info", param) - - count = stdout["data"]["count"] - if count != data["count"]: - raise AssertionError("Count of upload file is not as expect, count is: {}," - "expect is: {}".format(count, data["count"])) - echo.echo(f"[{time.strftime('%Y-%m-%d %X')}] check_data_out {stdout} \n") - except Exception as e: - exception_id = uuid.uuid1() - echo.echo(f"exception in {data['file']}, exception_id={exception_id}") - LOGGER.exception(f"exception id: {exception_id}") - echo.echo(f"upload {i + 1}th data {data['table_name']} fail, exception_id: {exception_id}") - # raise RuntimeError(f"exception occur while uploading data for {data['file']}") from e - finally: - echo.stdout_newline() diff --git a/python/fate_test/scripts/performance_cli.py b/python/fate_test/scripts/performance_cli.py index b9f7ab6..b9ceaaf 100644 --- a/python/fate_test/scripts/performance_cli.py +++ b/python/fate_test/scripts/performance_cli.py @@ -51,13 +51,11 @@ help="Extract performance time consuming from history tags for comparison") @click.option("--skip-data", is_flag=True, default=False, help="skip uploading data specified in testsuite") -@click.option("--provider", type=str, - help="Select the fate version, for example: fate@1.7") @click.option("--disable-clean-data", "clean_data", flag_value=False, default=None) @SharedOptions.get_shared_options(hidden=True) @click.pass_context def run_task(ctx, job_type, include, timeout, epochs, - max_depth, num_trees, task_cores, storage_tag, history_tag, skip_data, clean_data, provider, **kwargs): + max_depth, num_trees, task_cores, storage_tag, history_tag, skip_data, clean_data, **kwargs): """ Test the performance of big data tasks, alias: bp """ @@ -90,7 +88,7 @@ def get_perf_template(conf: Config, job_type): echo.welcome() echo.echo(f"testsuite namespace: {namespace}", fg='red') echo.echo("loading testsuites:") - suites = _load_testsuites(includes=include, excludes=tuple(), glob=None, provider=provider, + suites = _load_testsuites(includes=include, excludes=tuple(), glob=None, suffix="performance.yaml", suite_type="performance") for i, suite in enumerate(suites): echo.echo(f"\tdataset({len(suite.dataset)}) pipeline jobs({len(suite.pipeline_jobs)}) {suite.path}") From 510eef6239ff31f868073eaf437a8b93292e2ecf Mon Sep 17 00:00:00 2001 From: Yu Wu Date: Wed, 3 Jan 2024 17:23:52 +0800 Subject: [PATCH 2/9] add partitions arg to fate test commands Signed-off-by: Yu Wu --- python/fate_test/scripts/_options.py | 3 +++ python/fate_test/scripts/_utils.py | 11 ++++++++++- python/fate_test/scripts/benchmark_cli.py | 3 ++- python/fate_test/scripts/data_cli.py | 4 ++-- python/fate_test/scripts/performance_cli.py | 2 +- python/fate_test/scripts/testsuite_cli.py | 2 +- 6 files changed, 19 insertions(+), 6 deletions(-) diff --git a/python/fate_test/scripts/_options.py b/python/fate_test/scripts/_options.py index d32e971..e6e35f5 100644 --- a/python/fate_test/scripts/_options.py +++ b/python/fate_test/scripts/_options.py @@ -1,6 +1,7 @@ import time import click + from fate_test._config import parse_config, default_config from fate_test.scripts._utils import _set_namespace @@ -21,6 +22,8 @@ class SharedOptions(object): "extend_sid": (('--extend-sid',), dict(type=bool, is_flag=True, help="whether to append uuid as sid when uploading data", default=None), None), + "partitions": (('--partitions', '-dp'), + dict(type=int, help="data partitions when uploading data", default=None), None) # "auto_increasing_sid": (('--auto_increasing_sid',), # dict(type=bool, is_flag=True, help="whether to generate sid value starting at 0", # default=None), None), diff --git a/python/fate_test/scripts/_utils.py b/python/fate_test/scripts/_utils.py index c10bca7..b3f60e3 100644 --- a/python/fate_test/scripts/_utils.py +++ b/python/fate_test/scripts/_utils.py @@ -95,7 +95,9 @@ def _find_testsuite_files(path): @LOGGER.catch -def _upload_data(clients: Clients, suite, config: Config, output_path=None): +def _upload_data(clients: Clients, suite, config: Config, output_path=None, **kwargs): + if kwargs.get("partitions") is not None: + _update_data_config(suite, partitions=kwargs.get("partitions")) with click.progressbar(length=len(suite.dataset), label="dataset", show_eta=False, @@ -198,3 +200,10 @@ def _update_data_path(suite, output_dir): data_file_path = os.path.join(str(output_dir), data_name) data.file = data_file_path data.config['file'] = data_file_path + + +def _update_data_config(suite, partitions=None): + if partitions is not None: + for data in suite.dataset: + data.config['partitions'] = partitions + data.partitions = partitions diff --git a/python/fate_test/scripts/benchmark_cli.py b/python/fate_test/scripts/benchmark_cli.py index 365484a..77ee84a 100644 --- a/python/fate_test/scripts/benchmark_cli.py +++ b/python/fate_test/scripts/benchmark_cli.py @@ -6,6 +6,7 @@ from inspect import signature import click + from fate_test._client import Clients from fate_test._config import Config from fate_test._io import LOGGER, echo @@ -85,7 +86,7 @@ def run_benchmark(ctx, include, exclude, glob, skip_data, tol, clean_data, stora echo.echo(f"[{i + 1}/{len(suites)}]start at {time.strftime('%Y-%m-%d %X')} {suite.path}", fg='red') if not skip_data: try: - _upload_data(client, suite, config_inst) + _upload_data(client, suite, config_inst, partitions=ctx.obj["partitions"]) except Exception as e: raise RuntimeError(f"exception occur while uploading data for {suite.path}") from e if kwargs.get("data_only"): diff --git a/python/fate_test/scripts/data_cli.py b/python/fate_test/scripts/data_cli.py index c580205..f6fa3a7 100644 --- a/python/fate_test/scripts/data_cli.py +++ b/python/fate_test/scripts/data_cli.py @@ -72,7 +72,7 @@ def upload(ctx, include, exclude, glob, suite_type, role, config_type, **kwargs) return for suite in suites: - _upload_data(client, suite, config_inst) + _upload_data(client, suite, config_inst, partitions=ctx.obj["partitions"]) else: config = get_config(config_inst) if config_type == 'min_test': @@ -211,7 +211,7 @@ def generate(ctx, include, host_data_type, encryption_type, match_rate, sparsity output_dir = output_path if output_path else os.path.abspath(config_inst.cache_directory) _update_data_path(suite, output_dir) # echo.echo(f"data files: {[data.file for data in suite.dataset]}") - _upload_data(client, suite, config_inst) + _upload_data(client, suite, config_inst, partitions=ctx.obj["partitions"]) @data_group.command("query_schema") diff --git a/python/fate_test/scripts/performance_cli.py b/python/fate_test/scripts/performance_cli.py index b9ceaaf..c265930 100644 --- a/python/fate_test/scripts/performance_cli.py +++ b/python/fate_test/scripts/performance_cli.py @@ -107,7 +107,7 @@ def get_perf_template(conf: Config, job_type): if not skip_data: try: - _upload_data(client, suite, config_inst) + _upload_data(client, suite, config_inst, partitions=ctx.obj["partitions"]) except Exception as e: raise RuntimeError(f"exception occur while uploading data for {suite.path}") from e diff --git a/python/fate_test/scripts/testsuite_cli.py b/python/fate_test/scripts/testsuite_cli.py index a846707..a39813a 100644 --- a/python/fate_test/scripts/testsuite_cli.py +++ b/python/fate_test/scripts/testsuite_cli.py @@ -108,7 +108,7 @@ def run_suite(ctx, include, exclude, glob, echo.echo(f"[{i + 1}/{len(suites)}]start at {time.strftime('%Y-%m-%d %X')} {suite.path}", fg='red') if not skip_data: try: - _upload_data(client, suite, config_inst) + _upload_data(client, suite, config_inst, partitions=ctx.obj["partitions"]) except Exception as e: raise RuntimeError(f"exception occur while uploading data for {suite.path}") from e if data_only: From 4a320ff791183dc2f6478295db4587ad2994ebfd Mon Sep 17 00:00:00 2001 From: Yu Wu Date: Tue, 9 Jan 2024 20:11:29 +0800 Subject: [PATCH 3/9] add task engine run to benchmark quality & performance, testsuite Signed-off-by: Yu Wu --- python/fate_test/scripts/_options.py | 16 +++++++++++++++- python/fate_test/scripts/benchmark_cli.py | 2 ++ python/fate_test/scripts/performance_cli.py | 2 ++ python/fate_test/scripts/testsuite_cli.py | 2 ++ 4 files changed, 21 insertions(+), 1 deletion(-) diff --git a/python/fate_test/scripts/_options.py b/python/fate_test/scripts/_options.py index e6e35f5..2bdc1d5 100644 --- a/python/fate_test/scripts/_options.py +++ b/python/fate_test/scripts/_options.py @@ -6,6 +6,16 @@ from fate_test.scripts._utils import _set_namespace +def parse_custom_type(value): + parts = value.split('=') + if len(parts) == 2 and parts[1].isdigit(): + return parts[0], int(parts[1]) + elif len(parts) == 2 and isinstance(parts[1], str): + return parts[0], parts[1] + else: + raise click.BadParameter('Invalid input format. Use "str=int" or "str=str".') + + class SharedOptions(object): _options = { "config": (('-c', '--config'), @@ -23,7 +33,11 @@ class SharedOptions(object): dict(type=bool, is_flag=True, help="whether to append uuid as sid when uploading data", default=None), None), "partitions": (('--partitions', '-dp'), - dict(type=int, help="data partitions when uploading data", default=None), None) + dict(type=int, help="data partitions when uploading data", default=None), None), + "engine_run": (('--engine-run', '-eg'), dict(type=parse_custom_type, + help="config for pipeline task `engine run`, " + "specify config params in str=int or str=str format", + multiple=True, default=[None]), None) # "auto_increasing_sid": (('--auto_increasing_sid',), # dict(type=bool, is_flag=True, help="whether to generate sid value starting at 0", # default=None), None), diff --git a/python/fate_test/scripts/benchmark_cli.py b/python/fate_test/scripts/benchmark_cli.py index 77ee84a..fa6b155 100644 --- a/python/fate_test/scripts/benchmark_cli.py +++ b/python/fate_test/scripts/benchmark_cli.py @@ -60,6 +60,8 @@ def run_benchmark(ctx, include, exclude, glob, skip_data, tol, clean_data, stora config_inst.update_conf(task_cores=task_cores) if timeout is not None: config_inst.update_conf(timeout=timeout) + if ctx.obj["engine_run"][0] is not None: + config_inst.update_conf(engine_run=dict(ctx.obj["engine_run"])) """if ctx.obj["auto_increasing_sid"] is not None: config_inst.auto_increasing_sid = ctx.obj["auto_increasing_sid"]""" diff --git a/python/fate_test/scripts/performance_cli.py b/python/fate_test/scripts/performance_cli.py index c265930..2f0d151 100644 --- a/python/fate_test/scripts/performance_cli.py +++ b/python/fate_test/scripts/performance_cli.py @@ -68,6 +68,8 @@ def run_task(ctx, job_type, include, timeout, epochs, config_inst.update_conf(task_cores=task_cores) if timeout is not None: config_inst.update_conf(timeout=timeout) + if ctx.obj["engine_run"][0] is not None: + config_inst.update_conf(engine_run=dict(ctx.obj["engine_run"])) """if ctx.obj["auto_increasing_sid"] is not None: config_inst.auto_increasing_sid = ctx.obj["auto_increasing_sid"]""" namespace = ctx.obj["namespace"] diff --git a/python/fate_test/scripts/testsuite_cli.py b/python/fate_test/scripts/testsuite_cli.py index a39813a..d3b7c7b 100644 --- a/python/fate_test/scripts/testsuite_cli.py +++ b/python/fate_test/scripts/testsuite_cli.py @@ -72,6 +72,8 @@ def run_suite(ctx, include, exclude, glob, config_inst = ctx.obj["config"] if ctx.obj["extend_sid"] is not None: config_inst.extend_sid = ctx.obj["extend_sid"] + if ctx.obj["engine_run"][0] is not None: + config_inst.update_conf(engine_run=dict(ctx.obj["engine_run"])) if task_cores is not None: config_inst.update_conf(task_cores=task_cores) if timeout is not None: From 26554d0ee88bbdad8db1d321515dbbab43b75060 Mon Sep 17 00:00:00 2001 From: Yu Wu Date: Tue, 9 Jan 2024 20:15:54 +0800 Subject: [PATCH 4/9] update doc Signed-off-by: Yu Wu --- doc/fate_test_command.md | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/doc/fate_test_command.md b/doc/fate_test_command.md index e723ec6..dabe682 100644 --- a/doc/fate_test_command.md +++ b/doc/fate_test_command.md @@ -133,7 +133,15 @@ fate_test suite --help will remove data from storage after finishing running testsuites -14. yes: +14. engine-run: + + ```bash + fate_test suite -i *path1* --engine-run cores=4 --engine-run spark.driver.memory=8G --skip-data + ``` + + will run testsuites in *path1* with engine run params set to specified values + +15. yes: ```bash fate_test suite -i --yes @@ -363,7 +371,15 @@ fate_test benchmark-quality --help will remove data from storage after finishing running benchmark testsuites -10. yes: +11. engine-run: + + ```bash + fate_test benchmark-quality -i *path1* --engine-run cores=4 --engine-run spark.driver.memory=8G --skip-data + ``` + + will run testsuites in *path1* with engine run params set to specified values + +12. yes: ```bash fate_test benchmark-quality -i --yes ``` @@ -650,7 +666,15 @@ fate_test performance --help will run testsuites in *path1* without removing data from storage after tasks finish -15. yes: +15. engine-run: + + ```bash + fate_test performance -i *path1* --engine-run cores=4 --engine-run spark.driver.memory=8G --skip-data + ``` + + will run testsuites in *path1* with engine run params set to specified values + +16. yes: ```bash fate_test performance -i --yes From 6b8637fd60afabdfdf040aa24fd7a4521c073635 Mon Sep 17 00:00:00 2001 From: Yu Wu Date: Mon, 22 Jan 2024 11:10:05 +0800 Subject: [PATCH 5/9] update fate-test Signed-off-by: Yu Wu --- python/fate_test/scripts/testsuite_cli.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/python/fate_test/scripts/testsuite_cli.py b/python/fate_test/scripts/testsuite_cli.py index d3b7c7b..82194de 100644 --- a/python/fate_test/scripts/testsuite_cli.py +++ b/python/fate_test/scripts/testsuite_cli.py @@ -172,7 +172,13 @@ def _raise(err_msg, status="failed", job_id=None, event=None, time_elapsed=None) os.environ.pop("pipeline_job_info") except Exception as e: - _raise(e) + job_info = os.environ.get("pipeline_job_info") + if job_info is None: + job_id, status, time_elapsed, event = None, 'failed', None, None + else: + job_id, status, time_elapsed, event = extract_job_status(job_info, client, guest_party_id) + _raise(e, job_id=job_id, status=status, event=event, time_elapsed=time_elapsed) + os.environ.pop("pipeline_job_info") continue else: try: @@ -187,7 +193,10 @@ def _raise(err_msg, status="failed", job_id=None, event=None, time_elapsed=None) except Exception as e: job_info = os.environ.get("pipeline_job_info") - job_id, status, time_elapsed, event = extract_job_status(job_info, client, guest_party_id) + if job_info is None: + job_id, status, time_elapsed, event = None, 'failed', None, None + else: + job_id, status, time_elapsed, event = extract_job_status(job_info, client, guest_party_id) _raise(e, job_id=job_id, status=status, event=event, time_elapsed=time_elapsed) os.environ.pop("pipeline_job_info") continue From 79dec1654dc8417b4541a7a7b2589f935146bf8c Mon Sep 17 00:00:00 2001 From: Yu Wu Date: Thu, 29 Feb 2024 10:18:12 +0800 Subject: [PATCH 6/9] add engine run to default config attribute Signed-off-by: Yu Wu --- python/fate_test/_config.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/fate_test/_config.py b/python/fate_test/_config.py index ba42bce..d3a6012 100644 --- a/python/fate_test/_config.py +++ b/python/fate_test/_config.py @@ -186,7 +186,7 @@ def __init__(self, config): self.auto_increasing_sid = None self.task_cores = None self.timeout = None - # self.work_mode = config.get("work_mode", 0) + self.engine_run = None service_id = 0 os.makedirs(os.path.dirname(self.cache_directory), exist_ok=True) From ca061615daffab77389b97b556ef10376722ebdb Mon Sep 17 00:00:00 2001 From: Yu Wu Date: Thu, 29 Feb 2024 15:31:07 +0800 Subject: [PATCH 7/9] update version number Signed-off-by: Yu Wu --- python/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/setup.py b/python/setup.py index 74dd5f4..d26a151 100644 --- a/python/setup.py +++ b/python/setup.py @@ -20,7 +20,7 @@ setup_kwargs = { "name": "fate-test", - "version": "2.0.0", + "version": "2.1.0", "description": "test tools for FATE", "long_description": 'FATE Test\n=========\n\nA collection of useful tools to running FATE\'s test.\n\n.. image:: images/tutorial.gif\n :align: center\n :alt: tutorial\n\nquick start\n-----------\n\n1. (optional) create virtual env\n\n .. code-block:: bash\n\n python -m venv venv\n source venv/bin/activate\n pip install -U pip\n\n\n2. install fate_test\n\n .. code-block:: bash\n\n pip install fate_test\n fate_test --help\n\n\n3. edit default fate_test_config.yaml\n\n .. code-block:: bash\n\n # edit priority config file with system default editor\n # filling some field according to comments\n fate_test config edit\n\n4. configure FATE-Pipeline and FATE-Flow Commandline server setting\n\n.. code-block:: bash\n\n # configure FATE-Pipeline server setting\n pipeline init --port 9380 --ip 127.0.0.1\n # configure FATE-Flow Commandline server setting\n flow init --port 9380 --ip 127.0.0.1\n\n5. run some fate_test suite\n\n .. code-block:: bash\n\n fate_test suite -i \n\n\n6. run some fate_test benchmark\n\n .. code-block:: bash\n\n fate_test benchmark-quality -i \n\n7. useful logs or exception will be saved to logs dir with namespace shown in last step\n\ndevelop install\n---------------\nIt is more convenient to use the editable mode during development: replace step 2 with flowing steps\n\n.. code-block:: bash\n\n pip install -e ${FATE}/python/fate_client && pip install -e ${FATE}/python/fate_test\n\n\n\ncommand types\n-------------\n\n- suite: used for running testsuites, collection of FATE jobs\n\n .. code-block:: bash\n\n fate_test suite -i \n\n\n- benchmark-quality used for comparing modeling quality between FATE and other machine learning systems\n\n .. code-block:: bash\n\n fate_test benchmark-quality -i \n\n\n\nconfiguration by examples\n--------------------------\n\n1. no need ssh tunnel:\n\n - 9999, service: service_a\n - 10000, service: service_b\n\n and both service_a, service_b can be requested directly:\n\n .. code-block:: yaml\n\n work_mode: 1 # 0 for standalone, 1 for cluster\n data_base_dir: \n parties:\n guest: [10000]\n host: [9999, 10000]\n arbiter: [9999]\n services:\n - flow_services:\n - {address: service_a, parties: [9999]}\n - {address: service_b, parties: [10000]}\n\n2. need ssh tunnel:\n\n - 9999, service: service_a\n - 10000, service: service_b\n\n service_a, can be requested directly while service_b don\'t,\n but you can request service_b in other node, say B:\n\n .. code-block:: yaml\n\n work_mode: 0 # 0 for standalone, 1 for cluster\n data_base_dir: \n parties:\n guest: [10000]\n host: [9999, 10000]\n arbiter: [9999]\n services:\n - flow_services:\n - {address: service_a, parties: [9999]}\n - flow_services:\n - {address: service_b, parties: [10000]}\n ssh_tunnel: # optional\n enable: true\n ssh_address: :\n ssh_username: \n ssh_password: # optional\n ssh_priv_key: "~/.ssh/id_rsa"\n\n\nTestsuite\n---------\n\nTestsuite is used for running a collection of jobs in sequence. Data used for jobs could be uploaded before jobs are\nsubmitted, and are cleaned when jobs finished. This tool is useful for FATE\'s release test.\n\ncommand options\n~~~~~~~~~~~~~~~\n\n.. code-block:: bash\n\n fate_test suite --help\n\n1. include:\n\n .. code-block:: bash\n\n fate_test suite -i \n\n will run testsuites in *path1*\n\n2. exclude:\n\n .. code-block:: bash\n\n fate_test suite -i -e -e ...\n\n will run testsuites in *path1* but not in *path2* and *path3*\n\n3. glob:\n\n .. code-block:: bash\n\n fate_test suite -i -g "hetero*"\n\n will run testsuites in sub directory start with *hetero* of *path1*\n\n4. replace:\n\n .. code-block:: bash\n\n fate_test suite -i -r \'{"maxIter": 5}\'\n\n will find all key-value pair with key "maxIter" in `data conf` or `conf` or `dsl` and replace the value with 5\n\n\n5. skip-data:\n\n .. code-block:: bash\n\n fate_test suite -i --skip-data\n\n will run testsuites in *path1* without uploading data specified in *benchmark.json*.\n\n\n6. yes:\n\n .. code-block:: bash\n\n fate_test suite -i --yes\n\n will run testsuites in *path1* directly, skipping double check\n\n7. skip-dsl-jobs:\n\n .. code-block:: bash\n\n fate_test suite -i --skip-dsl-jobs\n\n will run testsuites in *path1* but skip all *tasks* in testsuites. It\'s would be useful when only pipeline tasks needed.\n\n8. skip-pipeline-jobs:\n\n .. code-block:: bash\n\n fate_test suite -i --skip-pipeline-jobs\n\n will run testsuites in *path1* but skip all *pipeline tasks* in testsuites. It\'s would be useful when only dsl tasks needed.\n\n\nBenchmark Quality\n------------------\n\nBenchmark-quality is used for comparing modeling quality between FATE\nand other machine learning systems. Benchmark produces a metrics comparison\nsummary for each benchmark job group.\n\n.. code-block:: bash\n\n fate_test benchmark-quality -i examples/benchmark_quality/hetero_linear_regression\n\n.. code-block:: bash\n\n +-------+--------------------------------------------------------------+\n | Data | Name |\n +-------+--------------------------------------------------------------+\n | train | {\'guest\': \'motor_hetero_guest\', \'host\': \'motor_hetero_host\'} |\n | test | {\'guest\': \'motor_hetero_guest\', \'host\': \'motor_hetero_host\'} |\n +-------+--------------------------------------------------------------+\n +------------------------------------+--------------------+--------------------+-------------------------+---------------------+\n | Model Name | explained_variance | r2_score | root_mean_squared_error | mean_squared_error |\n +------------------------------------+--------------------+--------------------+-------------------------+---------------------+\n | local-linear_regression-regression | 0.9035168452250094 | 0.9035070863155368 | 0.31340413289880553 | 0.09822215051805216 |\n | FATE-linear_regression-regression | 0.903146386539082 | 0.9031411831961411 | 0.3139977881119483 | 0.09859461093919596 |\n +------------------------------------+--------------------+--------------------+-------------------------+---------------------+\n +-------------------------+-----------+\n | Metric | All Match |\n +-------------------------+-----------+\n | explained_variance | True |\n | r2_score | True |\n | root_mean_squared_error | True |\n | mean_squared_error | True |\n +-------------------------+-----------+\n\ncommand options\n~~~~~~~~~~~~~~~\n\nuse the following command to show help message\n\n.. code-block:: bash\n\n fate_test benchmark-quality --help\n\n1. include:\n\n .. code-block:: bash\n\n fate_test benchmark-quality -i \n\n will run benchmark testsuites in *path1*\n\n2. exclude:\n\n .. code-block:: bash\n\n fate_test benchmark-quality -i -e -e ...\n\n will run benchmark testsuites in *path1* but not in *path2* and *path3*\n\n3. glob:\n\n .. code-block:: bash\n\n fate_test benchmark-quality -i -g "hetero*"\n\n will run benchmark testsuites in sub directory start with *hetero* of *path1*\n\n4. tol:\n\n .. code-block:: bash\n\n fate_test benchmark-quality -i -t 1e-3\n\n will run benchmark testsuites in *path1* with absolute tolerance of difference between metrics set to 0.001.\n If absolute difference between metrics is smaller than *tol*, then metrics are considered\n almost equal. Check benchmark testsuite `writing guide <#benchmark-testsuite>`_ on setting alternative tolerance.\n\n5. skip-data:\n\n .. code-block:: bash\n\n fate_test benchmark-quality -i --skip-data\n\n will run benchmark testsuites in *path1* without uploading data specified in *benchmark.json*.\n\n\n6. yes:\n\n .. code-block:: bash\n\n fate_test benchmark-quality -i --yes\n\n will run benchmark testsuites in *path1* directly, skipping double check\n\n\nbenchmark testsuite\n~~~~~~~~~~~~~~~~~~~\n\nConfiguration of jobs should be specified in a benchmark testsuite whose file name ends\nwith "\\*benchmark.json". For benchmark testsuite example,\nplease refer `here <../../examples/benchmark_quality>`_.\n\nA benchmark testsuite includes the following elements:\n\n- data: list of local data to be uploaded before running FATE jobs\n\n - file: path to original data file to be uploaded, should be relative to testsuite or FATE installation path\n - head: whether file includes header\n - partition: number of partition for data storage\n - table_name: table name in storage\n - namespace: table namespace in storage\n - role: which role to upload the data, as specified in fate_test.config;\n naming format is: "{role_type}_{role_index}", index starts at 0\n\n .. code-block:: json\n\n "data": [\n {\n "file": "examples/data/motor_hetero_host.csv",\n "head": 1,\n "partition": 8,\n "table_name": "motor_hetero_host",\n "namespace": "experiment",\n "role": "host_0"\n }\n ]\n\n- job group: each group includes arbitrary number of jobs with paths to corresponding script and configuration\n\n - job: name of job to be run, must be unique within each group list\n\n - script: path to `testing script <#testing-script>`_, should be relative to testsuite\n - conf: path to job configuration file for script, should be relative to testsuite\n\n .. code-block:: json\n\n "local": {\n "script": "./local-linr.py",\n "conf": "./linr_config.yaml"\n }\n\n - compare_setting: additional setting for quality metrics comparison, currently only takes ``relative_tol``\n\n If metrics *a* and *b* satisfy *abs(a-b) <= max(relative_tol \\* max(abs(a), abs(b)), absolute_tol)*\n (from `math module `_),\n they are considered almost equal. In the below example, metrics from "local" and "FATE" jobs are\n considered almost equal if their relative difference is smaller than\n *0.05 \\* max(abs(local_metric), abs(pipeline_metric)*.\n\n .. code-block:: json\n\n "linear_regression-regression": {\n "local": {\n "script": "./local-linr.py",\n "conf": "./linr_config.yaml"\n },\n "FATE": {\n "script": "./fate-linr.py",\n "conf": "./linr_config.yaml"\n },\n "compare_setting": {\n "relative_tol": 0.01\n }\n }\n\n\ntesting script\n~~~~~~~~~~~~~~\n\nAll job scripts need to have ``Main`` function as an entry point for executing jobs; scripts should\nreturn two dictionaries: first with data information key-value pairs: {data_type}: {data_name_dictionary};\nthe second contains {metric_name}: {metric_value} key-value pairs for metric comparison.\n\nBy default, the final data summary shows the output from the job named "FATE"; if no such job exists,\ndata information returned by the first job is shown. For clear presentation, we suggest that user follow\nthis general `guideline <../../examples/data/README.md#data-set-naming-rule>`_ for data set naming. In the case of multi-host\ntask, consider numbering host as such:\n\n::\n\n {\'guest\': \'default_credit_homo_guest\',\n \'host_1\': \'default_credit_homo_host_1\',\n \'host_2\': \'default_credit_homo_host_2\'}\n\nReturned quality metrics of the same key are to be compared.\nNote that only **real-value** metrics can be compared.\n\n- FATE script: ``Main`` always has three inputs:\n\n - config: job configuration, `JobConfig <../fate_client/pipeline/utils/tools.py#L64>`_ object loaded from "fate_test_config.yaml"\n - param: job parameter setting, dictionary loaded from "conf" file specified in benchmark testsuite\n - namespace: namespace suffix, user-given *namespace* or generated timestamp string when using *namespace-mangling*\n\n- non-FATE script: ``Main`` always has one input:\n\n - param: job parameter setting, dictionary loaded from "conf" file specified in benchmark testsuite\n\n\ndata\n----\n\n`Data` sub-command is used for upload or delete dataset in suite\'s.\n\ncommand options\n~~~~~~~~~~~~~~~\n\n.. code-block:: bash\n\n fate_test data --help\n\n1. include:\n\n .. code-block:: bash\n\n fate_test data [upload|delete] -i \n\n will upload/delete dataset in testsuites in *path1*\n\n2. exclude:\n\n .. code-block:: bash\n\n fate_test data [upload|delete] -i -e -e ...\n\n will upload/delete dataset in testsuites in *path1* but not in *path2* and *path3*\n\n3. glob:\n\n .. code-block:: bash\n\n fate_test data [upload|delete] -i -g "hetero*"\n\n will upload/delete dataset in testsuites in sub directory start with *hetero* of *path1*\n\n\nfull command options\n---------------------\n\n.. click:: fate_test.scripts.cli:cli\n :prog: fate_test\n :show-nested:\n', "author": "FederatedAI", From 1161b02cf871a4e2387d72359abbac3b3657cec0 Mon Sep 17 00:00:00 2001 From: Yu Wu Date: Thu, 29 Feb 2024 20:57:58 +0800 Subject: [PATCH 8/9] update 2.1.0 release note Signed-off-by: Yu Wu --- RELEASE.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/RELEASE.md b/RELEASE.md index 1e578b5..265ac5a 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,4 +1,4 @@ -## Release 2.0.0 +## Release 2.1.0 ### Major Features and Improvements > Fate-Test: FATE Automated Testing Tool -* Migrated automated testing for functionality, performance, and correctness +* Add Support for Job Runtime Configuration From 8a4bf22ca7d2a2723ba0caa2b446debdaeac094e Mon Sep 17 00:00:00 2001 From: Yu Wu Date: Thu, 29 Feb 2024 21:02:45 +0800 Subject: [PATCH 9/9] update 2.1.0 release note Signed-off-by: Yu Wu --- RELEASE.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/RELEASE.md b/RELEASE.md index 265ac5a..edbd1b7 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -2,3 +2,8 @@ ### Major Features and Improvements > Fate-Test: FATE Automated Testing Tool * Add Support for Job Runtime Configuration + +## Release 2.0.0 +### Major Features and Improvements +> Fate-Test: FATE Automated Testing Tool +* Migrated automated testing for functionality, performance, and correctness \ No newline at end of file