fate_flow status/start/stop/restart
Refer to Standalone Deployment.
+Refer to Standalone Deployment.
Refer to All-in-one Deployment.
+Refer to Allinone Deployment.
FATE provides client tools including SDK, CLI, and Pipeline. If you don't have FATE Client deployed in your environment, you can download it using pip install fate_client. The following operations are based on CLI.
Once your data is prepared, you can start submitting jobs to FATE Flow:
When the job runs, Reader reads directly from Storage
Table Bind.
You can choose one of the following three deployment modes based on your requirements:
"},{"location":"quick_start/#11-pypi-package-installation","title":"1.1 Pypi Package Installation","text":"Note: This mode operates in a single-machine mode.
"},{"location":"quick_start/#111-installation","title":"1.1.1 Installation","text":"# FATE requires Python >= 3.8\nconda create -n fate_env python=3.8\nconda activate fate_env\npip install fate_client[fate,fate_flow]==2.0.0.b0\nfate_flow init --ip 127.0.0.1 --port 9380 --home $HOME_DIR\nip: The IP address where the service runs. - port: The HTTP port the service runs on. - home: The data storage directory, including data, models, logs, job configurations, and SQLite databases."},{"location":"quick_start/#113-service-startstop","title":"1.1.3 Service Start/Stop","text":"fate_flow status/start/stop/restart\nRefer to Standalone Deployment.
"},{"location":"quick_start/#13-cluster-deployment","title":"1.3 Cluster Deployment","text":"Refer to All-in-one Deployment.
"},{"location":"quick_start/#2-user-guide","title":"2. User Guide","text":"FATE provides client tools including SDK, CLI, and Pipeline. If you don't have FATE Client deployed in your environment, you can download it using pip install fate_client. The following operations are based on CLI.
In version 2.0-beta, data uploading is a two-step process:
flow data upload -c examples/upload/upload_guest.json\nflow data upload -c examples/upload/upload_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"36491bc8-3fef-11ee-be05-16b977118319\",\n\"namespace\": \"upload\"\n},\n\"job_id\": \"202308211451535620150\",\n\"message\": \"success\"\n}\nSince upload is an asynchronous operation, you need to confirm if it was successful before proceeding to the next step.
flow table query --namespace upload --name 36491bc8-3fef-11ee-be05-16b977118319\nflow data transformer -c examples/transformer/transformer_guest.json\nflow data transformer -c examples/transformer/transformer_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"breast_hetero_guest\",\n\"namespace\": \"experiment\"\n},\n\"job_id\": \"202308211557455662860\",\n\"message\": \"success\"\n}\nSince the transformer is also an asynchronous operation, you need to confirm if it was successful before proceeding.
flow table query --namespace experiment --name breast_hetero_guest\nflow table query --namespace experiment --name breast_hetero_host\nOnce your data is prepared, you can start submitting jobs to FATE Flow:
flow job submit -c examples/lr/train_lr.yaml {\n\"code\": 0,\n\"data\": {\n\"model_id\": \"202308211911505128750\",\n\"model_version\": \"0\"\n},\n\"job_id\": \"202308211911505128750\",\n\"message\": \"success\"\n}\nWhile a job is running, you can check its status using the query command:
flow job query -j $job_id\nDuring job execution, you can stop the current job using the stop command:
flow job stop -j $job_id\nIf a job fails during execution, you can rerun it using the rerun command:
flow job rerun -j $job_id\nJob outputs include data, models, and metrics.
"},{"location":"quick_start/#231-output-metrics","title":"2.3.1 Output Metrics","text":"To query output metrics, use the following command:
flow output query-metric -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-metric -j 202308211911505128750 -r arbiter -p 9998 -tn lr_0 to query metrics. The query result will look like this: {\n\"code\": 0,\n\"data\": [\n{\n\"data\": [\n{\n\"metric\": [\n0.0\n],\n\"step\": 0,\n\"timestamp\": 1692616428.253495\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n},\n{\n\"data\": [\n{\n\"metric\": [\n-0.07785049080848694\n],\n\"step\": 1,\n\"timestamp\": 1692616432.9727712\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n}\n],\n\"message\": \"success\"\n}\nTo query output models, use the following command:
flow output query-model -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-model -j 202308211911505128750 -r host -p 9998 -tn lr_0 to query models. The query result will be similar to this: {\n\"code\": 0,\n\"data\": [\n{\n\"model\": {\n\"file\": \"202308211911505128750_host_9998_lr_0\",\n\"namespace\": \"202308211911505128750_host_9998_lr_0\"\n},\n\"name\": \"HeteroLRHost_9998_0\",\n\"namespace\": \"202308211911505128750_host_9998_lr_0\",\n\"role\": \"host\",\n\"party_id\": \"9998\",\n\"work_mode\": 1\n}\n],\n\"message\": \"success\"\n}\nTo download models, use the following command:
flow output download-model -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-model -j 202308211911505128750 -r host -p 9998 -tn lr_0 -o ./ to download the model. The download result will be similar to this: {\n\"code\": 0,\n\"directory\": \"./output_model_202308211911505128750_host_9998_lr_0\",\n\"message\": \"download success, please check the path: ./output_model_202308211911505128750_host_9998_lr_0\"\n}\nTo query output data tables, use the following command:
flow output query-data-table -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-data-table -j 202308211911505128750 -r host -p 9998 -tn binning_0 to query data tables. The query result will be similar to this: {\n\"train_output_data\": [\n{\n\"name\": \"9e28049c401311ee85c716b977118319\",\n\"namespace\": \"202308211911505128750_binning_0\"\n}\n]\n}\nflow output display-data -j $job_id -r $role -p $party_id -tn $task_name\nflow output display-data -j 202308211911505128750 -r host -p 9998 -tn binning_0."},{"location":"quick_start/#2333-download-data","title":"2.3.3.3 Download Data","text":"flow output download-data -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-data -j 202308211911505128750 -r guest -p 9999 -tn lr_0 -o ./. The download result will be as follows:
{\n\"code\": 0,\n\"directory\": \"./output_data_202308211911505128750_guest_9999_lr_0\",\n\"message\": \"download success, please check the path: ./output_data_202308211911505128750_guest_9999_lr_0\"\n}\nFATE Flow uses YAML to define system configurations, and the configuration file is located at: conf/service_conf.yaml. The specific configuration contents and their meanings are as follows:
key_0:\nmodule: fate_flow.hub.encrypt.password_encrypt#pwdecrypt\nprivate_path: private_key.pem\nfate_flow/conf/."},{"location":"system_conf/#fateflow-configuration","title":"FateFlow Configuration","text":"host: 127.0.0.1\nhttp_port: 9380\ngrpc_port: 9360\nproxy_name: rollsite\nnginx:\nhost:\nhttp_port:\ngrpc_port:\nengine: sqlite\ndecrypt_key:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\nsqlite:\npath:\nfate_flow/fate_flow_sqlite.db."},{"location":"system_conf/#engine-configuration","title":"Engine Configuration","text":"default_engines:\ncomputing: standalone\nfederation: standalone\nstorage: standalone\nfateflow.env.pulsar:\nhost: 192.168.0.5\nport: 6650\nmng_port: 8080\ncluster: standalone\ntenant: fl-tenant\ntopic_ttl: 30\nroute_table:\nmode: replication\nmax_message_size: 1048576\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\nprotocol: http\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\nprotocol: http\nrollsite:\nhost: 127.0.0.1\nport: 9370\n host: 127.0.0.1\nport: 9370\n cores: 32\neggroll:\ncores: 32\nnodes: 2\neggroll:\nhome: cores: 32\n hdfs:\nname_node: hdfs://fate-cluster\nhook_module:\nclient_authentication: fate_flow.hook.flow.client_authentication\nsite_authentication: fate_flow.hook.flow.site_authentication\npermission: fate_flow.hook.flow.permission\nauthentication:\nclient: false\nsite: false\npermission: false\nmodel_store:\nengine: file\ndecrypt_key:\nfile:\npath:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\ntencent_cos:\nRegion:\nSecretId:\nSecretKey:\nBucket:\nfate_flow/model. - mysql: MySQL service configuration; if using password encryption functionality, you need to set the \"passwd\" in this configuration to ciphertext and configure the key path in Encryption Module. - tencent_cos: Tencent Cloud key configuration."},{"location":"system_conf/#zookeeper-configuration","title":"ZooKeeper Configuration","text":"zookeeper:\nhosts:\n- 127.0.0.1:2181\nuse_acl: true\nuser: fate\npassword: fate\nAt repo root, execute
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs to serve docs in http://localhost:8000
or
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs build\nto build docs to site folder.
mkdocs-material and servel plugins are needed to build this docs
Fisrt, create an python virtual environment
python3 -m venv \"fatedocs\"\nsource fatedocs/bin/activate\npip install -U pip\npip install -r doc/mkdocs/requirements.txt\nNow, use
mkdocs serve\nat repo root to serve docs or
use
mkdocs build\nat repo root to build docs to folder site
We use mkdocs-material to build our docs. Servel markdown extensions are really useful to write pretty documents such as admonitions and content-tabs.
Servel plugins are introdused to makes mkdocs-material much powerful:
mkdocstrings automatic documentation from sources code. We mostly use this to automatic generate params api for federatedml.
awesome-pages for powerful nav rule
i18n for multi-languege support
mkdocs-jupyter for jupyter format support
mkdocs-simple-hooks for simple plugin-in
This image is modified from mkdocs-meterial with some plugins embeded.
Usage
Mount the folder where your mkdocs.yml resides as a volume into /docs:
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs\ndocker run --rm -it -v ${PWD}:/docs sagewei/mkdocs build\ndocker run --rm -it -v ~/.ssh:/root/.ssh -v ${PWD}:/docs sagewei0/mkdocs gh-deploy \nMostly copied from https://github.com/cirruslabs/cirrus-ci-docs/tree/master/theme
"},{"location":"swagger/","title":"API","text":""},{"location":"swagger/#swagger-api","title":"Swagger API","text":""},{"location":"zh/fate_flow/","title":"\u6574\u4f53\u8bbe\u8ba1","text":""},{"location":"zh/fate_flow/#1","title":"1. \u903b\u8f91\u67b6\u6784","text":"\u4f5c\u4e1a\u8fd0\u884c\u65f6\uff0cReader\u76f4\u63a5\u4eceStorage\u8bfb\u53d6
Table Bind\uff1a
\u4ee5\u4e0b\u4e09\u79cd\u6a21\u5f0f\u53ef\u6839\u636e\u9700\u6c42\u81ea\u884c\u9009\u62e9\u4e00\u79cd
"},{"location":"zh/quick_start/#11-pypi","title":"1.1 Pypi\u5305\u5b89\u88c5","text":"\u8bf4\u660e\uff1a\u6b64\u65b9\u5f0f\u7684\u8fd0\u884c\u6a21\u5f0f\u4e3a\u5355\u673a\u6a21\u5f0f
"},{"location":"zh/quick_start/#111","title":"1.1.1 \u5b89\u88c5","text":"# fate\u7684\u8fd0\u884c\u73af\u5883\u4e3apython>=3.8\nconda create -n fate_env python=3.8\nconda activate fate_env\npip install fate_client[fate,fate_flow]==2.0.0.b0\nfate_flow init --ip 127.0.0.1 --port 9380 --home $HOME_DIR\nfate_flow status/start/stop/restart\n\u53c2\u8003\u5355\u673a\u7248\u90e8\u7f72
"},{"location":"zh/quick_start/#13","title":"1.3 \u96c6\u7fa4\u90e8\u7f72","text":"\u53c2\u8003allinone\u90e8\u7f72
"},{"location":"zh/quick_start/#2","title":"2. \u4f7f\u7528\u6307\u5357","text":"fate\u63d0\u4f9b\u7684\u5ba2\u6237\u7aef\u5305\u62ecSDK\u3001CLI\u548cPipeline\uff0c\u82e5\u4f60\u7684\u73af\u5883\u4e2d\u6ca1\u6709\u90e8\u7f72FATE Client,\u53ef\u4ee5\u4f7f\u7528pip install fate_client\u4e0b\u8f7d\uff0c\u4ee5\u4e0b\u7684\u4f7f\u7528\u64cd\u4f5c\u5747\u57fa\u4e8ecli\u7f16\u5199\u3002
\u57282.0-beta\u7248\u672c\u4e2d\uff0c\u6570\u636e\u4e0a\u4f20\u5206\u4e3a\u4e24\u6b65\uff1a - upload: \u5c06\u6570\u636e\u4e0a\u4f20\u5230FATE\u652f\u6301\u5b58\u50a8\u670d\u52a1\u4e2d - transformer: \u5c06\u6570\u636e\u8f6c\u5316\u6210dataframe
"},{"location":"zh/quick_start/#2111","title":"2.1.1.1 \u914d\u7f6e\u53ca\u6570\u636e","text":"flow data upload -c examples/upload/upload_guest.json\nflow data upload -c examples/upload/upload_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"36491bc8-3fef-11ee-be05-16b977118319\",\n\"namespace\": \"upload\"\n},\n\"job_id\": \"202308211451535620150\",\n\"message\": \"success\"\n}\n\u56e0\u4e3aupload\u4e3a\u5f02\u6b65\u64cd\u4f5c\uff0c\u9700\u8981\u786e\u8ba4\u662f\u5426\u4e0a\u4f20\u6210\u529f\u624d\u53ef\u8fdb\u884c\u540e\u7eed\u64cd\u4f5c\u3002
flow table query --namespace upload --name 36491bc8-3fef-11ee-be05-16b977118319\n{\n\"code\": 0,\n\"data\": {\n\"count\": 569,\n\"data_type\": \"table\",\n\"engine\": \"standalone\",\n\"meta\": {\n\"delimiter\": \",\",\n\"dtype\": \"'float32\",\n\"header\": \"extend_sid,id,x0,x1,x2,x3,x4,x5,x6,x7,x8,x9,x10,x11,x12,x13,x14,x15,x16,x17,x18,x19\",\n\"input_format\": \"dense\",\n\"label_type\": \"int\",\n\"match_id_name\": \"id\",\n\"match_id_range\": 0,\n\"sample_id_name\": \"extend_sid\",\n\"tag_value_delimiter\": \":\",\n\"tag_with_value\": false,\n\"weight_type\": \"float32\"\n},\n\"name\": \"36491bc8-3fef-11ee-be05-16b977118319\",\n\"namespace\": \"upload\",\n\"path\": \"xxx\",\n\"source\": {\n\"component\": \"upload\",\n\"output_artifact_key\": \"data\",\n\"output_index\": null,\n\"party_task_id\": \"\",\n\"task_id\": \"\",\n\"task_name\": \"upload\"\n}\n},\n\"message\": \"success\"\n}\nflow data transformer -c examples/transformer/transformer_guest.json\nflow data transformer -c examples/transformer/transformer_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"breast_hetero_guest\",\n\"namespace\": \"experiment\"\n},\n\"job_id\": \"202308211557455662860\",\n\"message\": \"success\"\n}\n\u56e0\u4e3atransformer\u4e5f\u662f\u5f02\u6b65\u64cd\u4f5c\uff0c\u9700\u8981\u786e\u8ba4\u662f\u5426\u4e0a\u4f20\u6210\u529f\u624d\u53ef\u8fdb\u884c\u540e\u7eed\u64cd\u4f5c\u3002
flow table query --namespace experiment --name breast_hetero_guest\nflow table query --namespace experiment --name breast_hetero_host\n\u5f53\u4f60\u7684\u6570\u636e\u51c6\u5907\u597d\u540e\uff0c\u53ef\u4ee5\u5f00\u59cb\u63d0\u4ea4\u4f5c\u4e1a\u7ed9FATE Flow\uff1a - \u8bad\u7ec3job\u914d\u7f6eexample\u4f4d\u4e8elr-train; - \u9884\u6d4bjob\u914d\u7f6eexample\u4f4d\u4e8elr-predict;\u9884\u6d4b\u4efb\u52a1\u9700\u8981\u4fee\u6539\"dag.conf.model_warehouse\"\u6210\u8bad\u7ec3\u4f5c\u4e1a\u7684\u8f93\u51fa\u6a21\u578b\u3002 - \u8bad\u7ec3\u548c\u9884\u6d4bjob\u914d\u7f6e\u4e2d\u7ad9\u70b9id\u4e3a\"9998\"\u548c\"9999\"\u3002\u5982\u679c\u4f60\u7684\u90e8\u7f72\u73af\u5883\u4e3a\u96c6\u7fa4\u7248\uff0c\u9700\u8981\u66ff\u6362\u6210\u771f\u5b9e\u7684\u7ad9\u70b9id\uff1b\u5355\u673a\u7248\u53ef\u4f7f\u7528\u9ed8\u8ba4\u914d\u7f6e\u3002 - \u5982\u679c\u60f3\u8981\u4f7f\u7528\u81ea\u5df1\u7684\u6570\u636e\uff0c\u53ef\u4ee5\u66f4\u6539\u914d\u7f6e\u4e2dguest\u548chost\u7684data_warehouse\u7684namespace\u548cname - \u63d0\u4ea4\u4f5c\u4e1a\u7684\u547d\u4ee4\u4e3a:
flow job submit -c examples/lr/train_lr.yaml {\n\"code\": 0,\n\"data\": {\n\"model_id\": \"202308211911505128750\",\n\"model_version\": \"0\"\n},\n\"job_id\": \"202308211911505128750\",\n\"message\": \"success\"\n}\n\u5728\u4f5c\u4e1a\u7684\u8fd0\u884c\u8fc7\u7a0b\u65f6\uff0c\u4f60\u53ef\u4ee5\u901a\u8fc7\u67e5\u8be2\u547d\u4ee4\u83b7\u53d6\u4f5c\u4e1a\u7684\u8fd0\u884c\u72b6\u6001
flow job query -j $job_id\n\u5728\u4f5c\u4e1a\u7684\u8fd0\u884c\u8fc7\u7a0b\u65f6\uff0c\u4f60\u53ef\u4ee5\u901a\u8fc7\u505c\u6b62\u4f5c\u4e1a\u547d\u4ee4\u6765\u7ec8\u6b62\u5f53\u524d\u4f5c\u4e1a
flow job stop -j $job_id\n\u5728\u4f5c\u4e1a\u7684\u8fd0\u884c\u8fc7\u7a0b\u65f6\uff0c\u5982\u679c\u8fd0\u884c\u5931\u8d25\uff0c\u4f60\u53ef\u4ee5\u901a\u8fc7\u91cd\u8dd1\u547d\u4ee4\u6765\u91cd\u8dd1\u5f53\u524d\u4f5c\u4e1a
flow job rerun -j $job_id\n\u4f5c\u4e1a\u7684\u8f93\u51fa\u5305\u62ec\u6570\u636e\u3001\u6a21\u578b\u548c\u6307\u6807
"},{"location":"zh/quick_start/#231","title":"2.3.1 \u8f93\u51fa\u6307\u6807","text":"\u67e5\u8be2\u8f93\u51fa\u6307\u6807\u547d\u4ee4\uff1a
flow output query-metric -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-metric -j 202308211911505128750 -r arbiter -p 9998 -tn lr_0\u67e5\u8be2\u3002 \u67e5\u8be2\u7ed3\u679c\u5982\u4e0b: {\n\"code\": 0,\n\"data\": [\n{\n\"data\": [\n{\n\"metric\": [\n0.0\n],\n\"step\": 0,\n\"timestamp\": 1692616428.253495\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n},\n{\n\"data\": [\n{\n\"metric\": [\n-0.07785049080848694\n],\n\"step\": 1,\n\"timestamp\": 1692616432.9727712\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n}\n],\n\"message\": \"success\"\n}\nflow output query-model -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-model -j 202308211911505128750 -r host -p 9998 -tn lr_0\u67e5\u8be2\u3002 \u67e5\u8be2\u7ed3\u679c\u5982\u4e0b\uff1a {\n\"code\": 0,\n\"data\": {\n\"output_model\": {\n\"data\": {\n\"estimator\": {\n\"end_epoch\": 10,\n\"is_converged\": false,\n\"lr_scheduler\": {\n\"lr_params\": {\n\"start_factor\": 0.7,\n\"total_iters\": 100\n},\n\"lr_scheduler\": {\n\"_get_lr_called_within_step\": false,\n\"_last_lr\": [\n0.07269999999999996\n],\n\"_step_count\": 10,\n\"base_lrs\": [\n0.1\n],\n\"end_factor\": 1.0,\n\"last_epoch\": 9,\n\"start_factor\": 0.7,\n\"total_iters\": 100,\n\"verbose\": false\n},\n\"method\": \"linear\"\n},\n\"optimizer\": {\n\"alpha\": 0.001,\n\"l1_penalty\": false,\n\"l2_penalty\": true,\n\"method\": \"sgd\",\n\"model_parameter\": [\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n]\n],\n\"model_parameter_dtype\": \"float32\",\n\"optim_param\": {\n\"lr\": 0.1\n},\n\"optimizer\": {\n\"param_groups\": [\n{\n\"dampening\": 0,\n\"differentiable\": false,\n\"foreach\": null,\n\"initial_lr\": 0.1,\n\"lr\": 0.07269999999999996,\n\"maximize\": false,\n\"momentum\": 0,\n\"nesterov\": false,\n\"params\": [\n0\n],\n\"weight_decay\": 0\n}\n],\n\"state\": {}\n}\n},\n\"param\": {\n\"coef_\": [\n[\n-0.10828543454408646\n],\n[\n-0.07341302931308746\n],\n[\n-0.10850320011377335\n],\n[\n-0.10066638141870499\n],\n[\n-0.04595951363444328\n],\n[\n-0.07001449167728424\n],\n[\n-0.08949052542448044\n],\n[\n-0.10958756506443024\n],\n[\n-0.04012322425842285\n],\n[\n0.02270071767270565\n],\n[\n-0.07198350876569748\n],\n[\n0.00548586156219244\n],\n[\n-0.06599288433790207\n],\n[\n-0.06410090625286102\n],\n[\n0.016374297440052032\n],\n[\n-0.01607361063361168\n],\n[\n-0.011447405442595482\n],\n[\n-0.04352564364671707\n],\n[\n0.013161249458789825\n],\n[\n0.013506329618394375\n]\n],\n\"dtype\": \"float32\",\n\"intercept_\": null\n}\n}\n},\n\"meta\": {\n\"batch_size\": null,\n\"epochs\": 10,\n\"init_param\": {\n\"fill_val\": 0.0,\n\"fit_intercept\": false,\n\"method\": \"zeros\",\n\"random_state\": null\n},\n\"label_count\": false,\n\"learning_rate_param\": {\n\"method\": \"linear\",\n\"scheduler_params\": {\n\"start_factor\": 0.7,\n\"total_iters\": 100\n}\n},\n\"optimizer_param\": {\n\"alpha\": 0.001,\n\"method\": \"sgd\",\n\"optimizer_params\": {\n\"lr\": 0.1\n},\n\"penalty\": \"l2\"\n},\n\"ovr\": false\n}\n}\n},\n\"message\": \"success\"\n}\nflow output download-model -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-model -j 202308211911505128750 -r host -p 9998 -tn lr_0 -o ./\u4e0b\u8f7d\u3002 \u4e0b\u8f7d\u7ed3\u679c\u5982\u4e0b\uff1a {\n\"code\": 0,\n\"directory\": \"./output_model_202308211911505128750_host_9998_lr_0\",\n\"message\": \"download success, please check the path: ./output_model_202308211911505128750_host_9998_lr_0\"\n}\nflow output query-data-table -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-data-table -j 202308211911505128750 -r host -p 9998 -tn binning_0\u67e5\u8be2\u3002 \u67e5\u8be2\u7ed3\u679c\u5982\u4e0b\uff1a {\n\"train_output_data\": [\n{\n\"name\": \"9e28049c401311ee85c716b977118319\",\n\"namespace\": \"202308211911505128750_binning_0\"\n}\n]\n}\nflow output display-data -j $job_id -r $role -p $party_id -tn $task_name\nflow output display-data -j 202308211911505128750 -r host -p 9998 -tn binning_0\u9884\u89c8\u8f93\u51fa\u6570\u636e\u3002"},{"location":"zh/quick_start/#2333","title":"2.3.3.3 \u4e0b\u8f7d\u6570\u636e","text":"flow output download-data -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-data -j 202308211911505128750 -r guest -p 9999 -tn lr_0 -o ./\u4e0b\u8f7d\u8f93\u51fa\u6570\u636e\u3002 \u4e0b\u8f7d\u7ed3\u679c\u5982\u4e0b\uff1a {\n\"code\": 0,\n\"directory\": \"./output_data_202308211911505128750_guest_9999_lr_0\",\n\"message\": \"download success, please check the path: ./output_data_202308211911505128750_guest_9999_lr_0\"\n}\nFATE Flow\u4f7f\u7528yaml\u5b9a\u4e49\u7cfb\u7edf\u914d\u7f6e\uff0c\u914d\u7f6e\u8def\u5f84\u4f4d\u4e8e: conf/service_conf.yaml, \u5177\u4f53\u914d\u7f6e\u5185\u5bb9\u53ca\u5176\u542b\u4e49\u5982\u4e0b\uff1a
\u914d\u7f6e\u9879 \u8bf4\u660e \u503c party_id \u672c\u65b9\u7ad9\u70b9id \u5982: \"9999\", \"10000 use_registry \u662f\u5426\u4f7f\u7528\u6ce8\u518c\u4e2d\u5fc3\uff0c\u5f53\u524d\u4ec5\u652f\u6301zookeeper\u6a21\u5f0f\uff0c\u9700\u8981\u4fdd\u8bc1zookeeper\u7684\u914d\u7f6e\u6b63\u786e\uff1b\u6ce8\uff1a\u82e5\u4f7f\u7528\u9ad8\u53ef\u7528\u6a21\u5f0f\uff0c\u9700\u4fdd\u8bc1\u8be5\u914d\u7f6e\u8bbe\u7f6e\u4e3atrue true/false encrypt \u52a0\u5bc6\u6a21\u5757 \u89c1\u52a0\u5bc6\u6a21\u5757 fateflow FATE Flow\u670d\u52a1\u7684\u914d\u7f6e\uff0c\u4e3b\u8981\u5305\u62ec\u7aef\u53e3\u3001\u547d\u4ee4\u901a\u9053\u670d\u52a1\u3001\u4ee3\u7406\u7b49 \u89c1FateFlow\u914d\u7f6e database \u6570\u636e\u5e93\u670d\u52a1\u7684\u914d\u7f6e\u4fe1\u606f \u89c1\u6570\u636e\u5e93\u914d\u7f6e default_engines \u7cfb\u7edf\u7684\u5f15\u64ce\u670d\u52a1\uff0c\u4e3b\u8981\u5305\u62ec\u8ba1\u7b97\u3001\u5b58\u50a8\u548c\u901a\u4fe1\u5f15\u64ce \u89c1\u5f15\u64ce\u914d\u7f6e default_provider \u7ec4\u4ef6\u7684\u6765\u6e90\u4fe1\u606f\uff0c\u4e3b\u8981\u5305\u62ec\u63d0\u4f9b\u65b9\u540d\u79f0\u3001\u7ec4\u4ef6\u7248\u672c\u548c\u8fd0\u884c\u6a21\u5f0f \u89c1\u9ed8\u8ba4\u6ce8\u518c\u7b97\u6cd5\u914d\u7f6e federation \u901a\u4fe1\u670d\u52a1\u6c60 \u89c1\u901a\u4fe1\u5f15\u64ce\u6c60 computing \u8ba1\u7b97\u670d\u52a1\u6c60 \u89c1\u8ba1\u7b97\u5f15\u64ce\u6c60 storage \u5b58\u50a8\u670d\u52a1\u6c60 \u89c1\u5b58\u50a8\u5f15\u64ce\u6c60 hook_module \u94a9\u5b50\u914d\u7f6e\uff0c\u5f53\u524d\u652f\u6301\u5ba2\u6237\u7aef\u8ba4\u8bc1\u3001\u7ad9\u70b9\u7aef\u8ba4\u8bc1\u4ee5\u53ca\u9274\u6743\u94a9\u5b50 \u89c1\u94a9\u5b50\u6a21\u5757\u914d\u7f6e authentication \u8ba4\u8bc1&&\u9274\u6743\u5f00\u5173 \u89c1\u8ba4\u8bc1\u5f00\u5173 model_store \u6a21\u578b\u5b58\u50a8\u914d\u7f6e \u89c1\u6a21\u578b\u5b58\u50a8 zookeeper zookeeper\u670d\u52a1\u7684\u914d\u7f6e \u89c1zookeeper\u914d\u7f6e"},{"location":"zh/system_conf/#_2","title":"\u52a0\u5bc6\u6a21\u5757","text":"key_0:\nmodule: fate_flow.hub.encrypt.password_encrypt#pwdecrypt\nprivate_path: private_key.pem\nhost: 127.0.0.1\nhttp_port: 9380\ngrpc_port: 9360\nproxy_name: rollsite\nnginx:\nhost:\nhttp_port:\ngrpc_port:\nengine: sqlite\ndecrypt_key:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\nsqlite:\npath:\ndefault_engines:\ncomputing: standalone\nfederation: standalone\nstorage: standalone\npulsar:\nhost: 192.168.0.5\nport: 6650\nmng_port: 8080\ncluster: standalone\ntenant: fl-tenant\ntopic_ttl: 30\n# default conf/pulsar_route_table.yaml\nroute_table:\n# mode: replication / client, default: replication\nmode: replication\nmax_message_size: 1048576\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\n# http or grpc\nprotocol: http\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\n# http or grpc\nprotocol: http\nrollsite:\nhost: 127.0.0.1\nport: 9370\n host: 127.0.0.1\nport: 9370\n cores: 32\neggroll:\ncores: 32\nnodes: 2\neggroll:\nhome: cores: 32\n hdfs:\nname_node: hdfs://fate-cluster\nhook_module:\nclient_authentication: fate_flow.hook.flow.client_authentication\nsite_authentication: fate_flow.hook.flow.site_authentication\npermission: fate_flow.hook.flow.permission\nauthentication:\nclient: false\nsite: false\npermission: false\nmodel_store:\nengine: file\ndecrypt_key:\nfile:\npath:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\ntencent_cos:\nRegion:\nSecretId:\nSecretKey:\nBucket:\nzookeeper:\nhosts:\n- 127.0.0.1:2181\nuse_acl: true\nuser: fate\npassword: fate\nWhen the job runs, Reader reads directly from Storage
Table Bind.
You can choose one of the following three deployment modes based on your requirements:
"},{"location":"quick_start/#11-pypi-package-installation","title":"1.1 Pypi Package Installation","text":"Note: This mode operates in a single-machine mode.
"},{"location":"quick_start/#111-installation","title":"1.1.1 Installation","text":"# FATE requires Python >= 3.8\nconda create -n fate_env python=3.8\nconda activate fate_env\npip install fate_client[fate,fate_flow]==2.0.0.b0\nfate_flow init --ip 127.0.0.1 --port 9380 --home $HOME_DIR\nip: The IP address where the service runs. - port: The HTTP port the service runs on. - home: The data storage directory, including data, models, logs, job configurations, and SQLite databases."},{"location":"quick_start/#113-service-startstop","title":"1.1.3 Service Start/Stop","text":"fate_flow status/start/stop/restart\nRefer to Standalone Deployment.
"},{"location":"quick_start/#13-cluster-deployment","title":"1.3 Cluster Deployment","text":"Refer to All-in-one Deployment.
"},{"location":"quick_start/#2-user-guide","title":"2. User Guide","text":"FATE provides client tools including SDK, CLI, and Pipeline. If you don't have FATE Client deployed in your environment, you can download it using pip install fate_client. The following operations are based on CLI.
In version 2.0-beta, data uploading is a two-step process:
flow data upload -c examples/upload/upload_guest.json\nflow data upload -c examples/upload/upload_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"36491bc8-3fef-11ee-be05-16b977118319\",\n\"namespace\": \"upload\"\n},\n\"job_id\": \"202308211451535620150\",\n\"message\": \"success\"\n}\nSince upload is an asynchronous operation, you need to confirm if it was successful before proceeding to the next step.
flow table query --namespace upload --name 36491bc8-3fef-11ee-be05-16b977118319\nflow data transformer -c examples/transformer/transformer_guest.json\nflow data transformer -c examples/transformer/transformer_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"breast_hetero_guest\",\n\"namespace\": \"experiment\"\n},\n\"job_id\": \"202308211557455662860\",\n\"message\": \"success\"\n}\nSince the transformer is also an asynchronous operation, you need to confirm if it was successful before proceeding.
flow table query --namespace experiment --name breast_hetero_guest\nflow table query --namespace experiment --name breast_hetero_host\nOnce your data is prepared, you can start submitting jobs to FATE Flow:
flow job submit -c examples/lr/train_lr.yaml {\n\"code\": 0,\n\"data\": {\n\"model_id\": \"202308211911505128750\",\n\"model_version\": \"0\"\n},\n\"job_id\": \"202308211911505128750\",\n\"message\": \"success\"\n}\nWhile a job is running, you can check its status using the query command:
flow job query -j $job_id\nDuring job execution, you can stop the current job using the stop command:
flow job stop -j $job_id\nIf a job fails during execution, you can rerun it using the rerun command:
flow job rerun -j $job_id\nJob outputs include data, models, and metrics.
"},{"location":"quick_start/#231-output-metrics","title":"2.3.1 Output Metrics","text":"To query output metrics, use the following command:
flow output query-metric -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-metric -j 202308211911505128750 -r arbiter -p 9998 -tn lr_0 to query metrics. The query result will look like this: {\n\"code\": 0,\n\"data\": [\n{\n\"data\": [\n{\n\"metric\": [\n0.0\n],\n\"step\": 0,\n\"timestamp\": 1692616428.253495\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n},\n{\n\"data\": [\n{\n\"metric\": [\n-0.07785049080848694\n],\n\"step\": 1,\n\"timestamp\": 1692616432.9727712\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n}\n],\n\"message\": \"success\"\n}\nTo query output models, use the following command:
flow output query-model -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-model -j 202308211911505128750 -r host -p 9998 -tn lr_0 to query models. The query result will be similar to this: {\n\"code\": 0,\n\"data\": [\n{\n\"model\": {\n\"file\": \"202308211911505128750_host_9998_lr_0\",\n\"namespace\": \"202308211911505128750_host_9998_lr_0\"\n},\n\"name\": \"HeteroLRHost_9998_0\",\n\"namespace\": \"202308211911505128750_host_9998_lr_0\",\n\"role\": \"host\",\n\"party_id\": \"9998\",\n\"work_mode\": 1\n}\n],\n\"message\": \"success\"\n}\nTo download models, use the following command:
flow output download-model -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-model -j 202308211911505128750 -r host -p 9998 -tn lr_0 -o ./ to download the model. The download result will be similar to this: {\n\"code\": 0,\n\"directory\": \"./output_model_202308211911505128750_host_9998_lr_0\",\n\"message\": \"download success, please check the path: ./output_model_202308211911505128750_host_9998_lr_0\"\n}\nTo query output data tables, use the following command:
flow output query-data-table -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-data-table -j 202308211911505128750 -r host -p 9998 -tn binning_0 to query data tables. The query result will be similar to this: {\n\"train_output_data\": [\n{\n\"name\": \"9e28049c401311ee85c716b977118319\",\n\"namespace\": \"202308211911505128750_binning_0\"\n}\n]\n}\nflow output display-data -j $job_id -r $role -p $party_id -tn $task_name\nflow output display-data -j 202308211911505128750 -r host -p 9998 -tn binning_0."},{"location":"quick_start/#2333-download-data","title":"2.3.3.3 Download Data","text":"flow output download-data -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-data -j 202308211911505128750 -r guest -p 9999 -tn lr_0 -o ./. The download result will be as follows:
{\n\"code\": 0,\n\"directory\": \"./output_data_202308211911505128750_guest_9999_lr_0\",\n\"message\": \"download success, please check the path: ./output_data_202308211911505128750_guest_9999_lr_0\"\n}\nFATE Flow uses YAML to define system configurations, and the configuration file is located at: conf/service_conf.yaml. The specific configuration contents and their meanings are as follows:
key_0:\nmodule: fate_flow.hub.encrypt.password_encrypt#pwdecrypt\nprivate_path: private_key.pem\nfate_flow/conf/."},{"location":"system_conf/#fateflow-configuration","title":"FateFlow Configuration","text":"host: 127.0.0.1\nhttp_port: 9380\ngrpc_port: 9360\nproxy_name: rollsite\nnginx:\nhost:\nhttp_port:\ngrpc_port:\nengine: sqlite\ndecrypt_key:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\nsqlite:\npath:\nfate_flow/fate_flow_sqlite.db."},{"location":"system_conf/#engine-configuration","title":"Engine Configuration","text":"default_engines:\ncomputing: standalone\nfederation: standalone\nstorage: standalone\nfateflow.env.pulsar:\nhost: 192.168.0.5\nport: 6650\nmng_port: 8080\ncluster: standalone\ntenant: fl-tenant\ntopic_ttl: 30\nroute_table:\nmode: replication\nmax_message_size: 1048576\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\nprotocol: http\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\nprotocol: http\nrollsite:\nhost: 127.0.0.1\nport: 9370\n host: 127.0.0.1\nport: 9370\n cores: 32\neggroll:\ncores: 32\nnodes: 2\neggroll:\nhome: cores: 32\n hdfs:\nname_node: hdfs://fate-cluster\nhook_module:\nclient_authentication: fate_flow.hook.flow.client_authentication\nsite_authentication: fate_flow.hook.flow.site_authentication\npermission: fate_flow.hook.flow.permission\nauthentication:\nclient: false\nsite: false\npermission: false\nmodel_store:\nengine: file\ndecrypt_key:\nfile:\npath:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\ntencent_cos:\nRegion:\nSecretId:\nSecretKey:\nBucket:\nfate_flow/model. - mysql: MySQL service configuration; if using password encryption functionality, you need to set the \"passwd\" in this configuration to ciphertext and configure the key path in Encryption Module. - tencent_cos: Tencent Cloud key configuration."},{"location":"system_conf/#zookeeper-configuration","title":"ZooKeeper Configuration","text":"zookeeper:\nhosts:\n- 127.0.0.1:2181\nuse_acl: true\nuser: fate\npassword: fate\nAt repo root, execute
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs to serve docs in http://localhost:8000
or
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs build\nto build docs to site folder.
mkdocs-material and servel plugins are needed to build this docs
Fisrt, create an python virtual environment
python3 -m venv \"fatedocs\"\nsource fatedocs/bin/activate\npip install -U pip\npip install -r doc/mkdocs/requirements.txt\nNow, use
mkdocs serve\nat repo root to serve docs or
use
mkdocs build\nat repo root to build docs to folder site
We use mkdocs-material to build our docs. Servel markdown extensions are really useful to write pretty documents such as admonitions and content-tabs.
Servel plugins are introdused to makes mkdocs-material much powerful:
mkdocstrings automatic documentation from sources code. We mostly use this to automatic generate params api for federatedml.
awesome-pages for powerful nav rule
i18n for multi-languege support
mkdocs-jupyter for jupyter format support
mkdocs-simple-hooks for simple plugin-in
This image is modified from mkdocs-meterial with some plugins embeded.
Usage
Mount the folder where your mkdocs.yml resides as a volume into /docs:
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs\ndocker run --rm -it -v ${PWD}:/docs sagewei/mkdocs build\ndocker run --rm -it -v ~/.ssh:/root/.ssh -v ${PWD}:/docs sagewei0/mkdocs gh-deploy \nMostly copied from https://github.com/cirruslabs/cirrus-ci-docs/tree/master/theme
"},{"location":"swagger/","title":"API","text":""},{"location":"swagger/#swagger-api","title":"Swagger API","text":""}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"fate_flow/","title":"Overall Design","text":""},{"location":"fate_flow/#1-logical-architecture","title":"1. Logical Architecture","text":"When the job runs, Reader reads directly from Storage
Table Bind.
You can choose one of the following three deployment modes based on your requirements:
"},{"location":"quick_start/#11-pypi-package-installation","title":"1.1 Pypi Package Installation","text":"Note: This mode operates in a single-machine mode.
"},{"location":"quick_start/#111-installation","title":"1.1.1 Installation","text":"# FATE requires Python >= 3.8\nconda create -n fate_env python=3.8\nconda activate fate_env\npip install fate_client[fate,fate_flow]==2.0.0.b0\nfate_flow init --ip 127.0.0.1 --port 9380 --home $HOME_DIR\nip: The IP address where the service runs. - port: The HTTP port the service runs on. - home: The data storage directory, including data, models, logs, job configurations, and SQLite databases."},{"location":"quick_start/#113-service-startstop","title":"1.1.3 Service Start/Stop","text":"fate_flow status/start/stop/restart\nRefer to Standalone Deployment.
"},{"location":"quick_start/#13-cluster-deployment","title":"1.3 Cluster Deployment","text":"Refer to Allinone Deployment.
"},{"location":"quick_start/#2-user-guide","title":"2. User Guide","text":"FATE provides client tools including SDK, CLI, and Pipeline. If you don't have FATE Client deployed in your environment, you can download it using pip install fate_client. The following operations are based on CLI.
In version 2.0-beta, data uploading is a two-step process:
flow data upload -c examples/upload/upload_guest.json\nflow data upload -c examples/upload/upload_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"36491bc8-3fef-11ee-be05-16b977118319\",\n\"namespace\": \"upload\"\n},\n\"job_id\": \"202308211451535620150\",\n\"message\": \"success\"\n}\nSince upload is an asynchronous operation, you need to confirm if it was successful before proceeding to the next step.
flow table query --namespace upload --name 36491bc8-3fef-11ee-be05-16b977118319\nflow data transformer -c examples/transformer/transformer_guest.json\nflow data transformer -c examples/transformer/transformer_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"breast_hetero_guest\",\n\"namespace\": \"experiment\"\n},\n\"job_id\": \"202308211557455662860\",\n\"message\": \"success\"\n}\nSince the transformer is also an asynchronous operation, you need to confirm if it was successful before proceeding.
flow table query --namespace experiment --name breast_hetero_guest\nflow table query --namespace experiment --name breast_hetero_host\nOnce your data is prepared, you can start submitting jobs to FATE Flow:
flow job submit -c examples/lr/train_lr.yaml {\n\"code\": 0,\n\"data\": {\n\"model_id\": \"202308211911505128750\",\n\"model_version\": \"0\"\n},\n\"job_id\": \"202308211911505128750\",\n\"message\": \"success\"\n}\nWhile a job is running, you can check its status using the query command:
flow job query -j $job_id\nDuring job execution, you can stop the current job using the stop command:
flow job stop -j $job_id\nIf a job fails during execution, you can rerun it using the rerun command:
flow job rerun -j $job_id\nJob outputs include data, models, and metrics.
"},{"location":"quick_start/#231-output-metrics","title":"2.3.1 Output Metrics","text":"To query output metrics, use the following command:
flow output query-metric -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-metric -j 202308211911505128750 -r arbiter -p 9998 -tn lr_0 to query metrics. The query result will look like this: {\n\"code\": 0,\n\"data\": [\n{\n\"data\": [\n{\n\"metric\": [\n0.0\n],\n\"step\": 0,\n\"timestamp\": 1692616428.253495\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n},\n{\n\"data\": [\n{\n\"metric\": [\n-0.07785049080848694\n],\n\"step\": 1,\n\"timestamp\": 1692616432.9727712\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n}\n],\n\"message\": \"success\"\n}\nTo query output models, use the following command:
flow output query-model -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-model -j 202308211911505128750 -r host -p 9998 -tn lr_0 to query models. The query result will be similar to this: {\n\"code\": 0,\n\"data\": [\n{\n\"model\": {\n\"file\": \"202308211911505128750_host_9998_lr_0\",\n\"namespace\": \"202308211911505128750_host_9998_lr_0\"\n},\n\"name\": \"HeteroLRHost_9998_0\",\n\"namespace\": \"202308211911505128750_host_9998_lr_0\",\n\"role\": \"host\",\n\"party_id\": \"9998\",\n\"work_mode\": 1\n}\n],\n\"message\": \"success\"\n}\nTo download models, use the following command:
flow output download-model -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-model -j 202308211911505128750 -r host -p 9998 -tn lr_0 -o ./ to download the model. The download result will be similar to this: {\n\"code\": 0,\n\"directory\": \"./output_model_202308211911505128750_host_9998_lr_0\",\n\"message\": \"download success, please check the path: ./output_model_202308211911505128750_host_9998_lr_0\"\n}\nTo query output data tables, use the following command:
flow output query-data-table -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-data-table -j 202308211911505128750 -r host -p 9998 -tn binning_0 to query data tables. The query result will be similar to this: {\n\"train_output_data\": [\n{\n\"name\": \"9e28049c401311ee85c716b977118319\",\n\"namespace\": \"202308211911505128750_binning_0\"\n}\n]\n}\nflow output display-data -j $job_id -r $role -p $party_id -tn $task_name\nflow output display-data -j 202308211911505128750 -r host -p 9998 -tn binning_0."},{"location":"quick_start/#2333-download-data","title":"2.3.3.3 Download Data","text":"flow output download-data -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-data -j 202308211911505128750 -r guest -p 9999 -tn lr_0 -o ./. The download result will be as follows:
{\n\"code\": 0,\n\"directory\": \"./output_data_202308211911505128750_guest_9999_lr_0\",\n\"message\": \"download success, please check the path: ./output_data_202308211911505128750_guest_9999_lr_0\"\n}\nFATE Flow uses YAML to define system configurations, and the configuration file is located at: conf/service_conf.yaml. The specific configuration contents and their meanings are as follows:
key_0:\nmodule: fate_flow.hub.encrypt.password_encrypt#pwdecrypt\nprivate_path: private_key.pem\nfate_flow/conf/."},{"location":"system_conf/#fateflow-configuration","title":"FateFlow Configuration","text":"host: 127.0.0.1\nhttp_port: 9380\ngrpc_port: 9360\nproxy_name: rollsite\nnginx:\nhost:\nhttp_port:\ngrpc_port:\nengine: sqlite\ndecrypt_key:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\nsqlite:\npath:\nfate_flow/fate_flow_sqlite.db."},{"location":"system_conf/#engine-configuration","title":"Engine Configuration","text":"default_engines:\ncomputing: standalone\nfederation: standalone\nstorage: standalone\nfateflow.env.pulsar:\nhost: 192.168.0.5\nport: 6650\nmng_port: 8080\ncluster: standalone\ntenant: fl-tenant\ntopic_ttl: 30\nroute_table:\nmode: replication\nmax_message_size: 1048576\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\nprotocol: http\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\nprotocol: http\nrollsite:\nhost: 127.0.0.1\nport: 9370\n host: 127.0.0.1\nport: 9370\n cores: 32\neggroll:\ncores: 32\nnodes: 2\neggroll:\nhome: cores: 32\n hdfs:\nname_node: hdfs://fate-cluster\nhook_module:\nclient_authentication: fate_flow.hook.flow.client_authentication\nsite_authentication: fate_flow.hook.flow.site_authentication\npermission: fate_flow.hook.flow.permission\nauthentication:\nclient: false\nsite: false\npermission: false\nmodel_store:\nengine: file\ndecrypt_key:\nfile:\npath:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\ntencent_cos:\nRegion:\nSecretId:\nSecretKey:\nBucket:\nfate_flow/model. - mysql: MySQL service configuration; if using password encryption functionality, you need to set the \"passwd\" in this configuration to ciphertext and configure the key path in Encryption Module. - tencent_cos: Tencent Cloud key configuration."},{"location":"system_conf/#zookeeper-configuration","title":"ZooKeeper Configuration","text":"zookeeper:\nhosts:\n- 127.0.0.1:2181\nuse_acl: true\nuser: fate\npassword: fate\nAt repo root, execute
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs to serve docs in http://localhost:8000
or
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs build\nto build docs to site folder.
mkdocs-material and servel plugins are needed to build this docs
Fisrt, create an python virtual environment
python3 -m venv \"fatedocs\"\nsource fatedocs/bin/activate\npip install -U pip\npip install -r doc/mkdocs/requirements.txt\nNow, use
mkdocs serve\nat repo root to serve docs or
use
mkdocs build\nat repo root to build docs to folder site
We use mkdocs-material to build our docs. Servel markdown extensions are really useful to write pretty documents such as admonitions and content-tabs.
Servel plugins are introdused to makes mkdocs-material much powerful:
mkdocstrings automatic documentation from sources code. We mostly use this to automatic generate params api for federatedml.
awesome-pages for powerful nav rule
i18n for multi-languege support
mkdocs-jupyter for jupyter format support
mkdocs-simple-hooks for simple plugin-in
This image is modified from mkdocs-meterial with some plugins embeded.
Usage
Mount the folder where your mkdocs.yml resides as a volume into /docs:
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs\ndocker run --rm -it -v ${PWD}:/docs sagewei/mkdocs build\ndocker run --rm -it -v ~/.ssh:/root/.ssh -v ${PWD}:/docs sagewei0/mkdocs gh-deploy \nMostly copied from https://github.com/cirruslabs/cirrus-ci-docs/tree/master/theme
"},{"location":"swagger/","title":"API","text":""},{"location":"swagger/#swagger-api","title":"Swagger API","text":""},{"location":"zh/fate_flow/","title":"\u6574\u4f53\u8bbe\u8ba1","text":""},{"location":"zh/fate_flow/#1","title":"1. \u903b\u8f91\u67b6\u6784","text":"\u4f5c\u4e1a\u8fd0\u884c\u65f6\uff0cReader\u76f4\u63a5\u4eceStorage\u8bfb\u53d6
Table Bind\uff1a
\u4ee5\u4e0b\u4e09\u79cd\u6a21\u5f0f\u53ef\u6839\u636e\u9700\u6c42\u81ea\u884c\u9009\u62e9\u4e00\u79cd
"},{"location":"zh/quick_start/#11-pypi","title":"1.1 Pypi\u5305\u5b89\u88c5","text":"\u8bf4\u660e\uff1a\u6b64\u65b9\u5f0f\u7684\u8fd0\u884c\u6a21\u5f0f\u4e3a\u5355\u673a\u6a21\u5f0f
"},{"location":"zh/quick_start/#111","title":"1.1.1 \u5b89\u88c5","text":"# fate\u7684\u8fd0\u884c\u73af\u5883\u4e3apython>=3.8\nconda create -n fate_env python=3.8\nconda activate fate_env\npip install fate_client[fate,fate_flow]==2.0.0.b0\nfate_flow init --ip 127.0.0.1 --port 9380 --home $HOME_DIR\nfate_flow status/start/stop/restart\n\u53c2\u8003\u5355\u673a\u7248\u90e8\u7f72
"},{"location":"zh/quick_start/#13","title":"1.3 \u96c6\u7fa4\u90e8\u7f72","text":"\u53c2\u8003allinone\u90e8\u7f72
"},{"location":"zh/quick_start/#2","title":"2. \u4f7f\u7528\u6307\u5357","text":"fate\u63d0\u4f9b\u7684\u5ba2\u6237\u7aef\u5305\u62ecSDK\u3001CLI\u548cPipeline\uff0c\u82e5\u4f60\u7684\u73af\u5883\u4e2d\u6ca1\u6709\u90e8\u7f72FATE Client,\u53ef\u4ee5\u4f7f\u7528pip install fate_client\u4e0b\u8f7d\uff0c\u4ee5\u4e0b\u7684\u4f7f\u7528\u64cd\u4f5c\u5747\u57fa\u4e8ecli\u7f16\u5199\u3002
\u57282.0-beta\u7248\u672c\u4e2d\uff0c\u6570\u636e\u4e0a\u4f20\u5206\u4e3a\u4e24\u6b65\uff1a - upload: \u5c06\u6570\u636e\u4e0a\u4f20\u5230FATE\u652f\u6301\u5b58\u50a8\u670d\u52a1\u4e2d - transformer: \u5c06\u6570\u636e\u8f6c\u5316\u6210dataframe
"},{"location":"zh/quick_start/#2111","title":"2.1.1.1 \u914d\u7f6e\u53ca\u6570\u636e","text":"flow data upload -c examples/upload/upload_guest.json\nflow data upload -c examples/upload/upload_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"36491bc8-3fef-11ee-be05-16b977118319\",\n\"namespace\": \"upload\"\n},\n\"job_id\": \"202308211451535620150\",\n\"message\": \"success\"\n}\n\u56e0\u4e3aupload\u4e3a\u5f02\u6b65\u64cd\u4f5c\uff0c\u9700\u8981\u786e\u8ba4\u662f\u5426\u4e0a\u4f20\u6210\u529f\u624d\u53ef\u8fdb\u884c\u540e\u7eed\u64cd\u4f5c\u3002
flow table query --namespace upload --name 36491bc8-3fef-11ee-be05-16b977118319\n{\n\"code\": 0,\n\"data\": {\n\"count\": 569,\n\"data_type\": \"table\",\n\"engine\": \"standalone\",\n\"meta\": {\n\"delimiter\": \",\",\n\"dtype\": \"'float32\",\n\"header\": \"extend_sid,id,x0,x1,x2,x3,x4,x5,x6,x7,x8,x9,x10,x11,x12,x13,x14,x15,x16,x17,x18,x19\",\n\"input_format\": \"dense\",\n\"label_type\": \"int\",\n\"match_id_name\": \"id\",\n\"match_id_range\": 0,\n\"sample_id_name\": \"extend_sid\",\n\"tag_value_delimiter\": \":\",\n\"tag_with_value\": false,\n\"weight_type\": \"float32\"\n},\n\"name\": \"36491bc8-3fef-11ee-be05-16b977118319\",\n\"namespace\": \"upload\",\n\"path\": \"xxx\",\n\"source\": {\n\"component\": \"upload\",\n\"output_artifact_key\": \"data\",\n\"output_index\": null,\n\"party_task_id\": \"\",\n\"task_id\": \"\",\n\"task_name\": \"upload\"\n}\n},\n\"message\": \"success\"\n}\nflow data transformer -c examples/transformer/transformer_guest.json\nflow data transformer -c examples/transformer/transformer_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"breast_hetero_guest\",\n\"namespace\": \"experiment\"\n},\n\"job_id\": \"202308211557455662860\",\n\"message\": \"success\"\n}\n\u56e0\u4e3atransformer\u4e5f\u662f\u5f02\u6b65\u64cd\u4f5c\uff0c\u9700\u8981\u786e\u8ba4\u662f\u5426\u4e0a\u4f20\u6210\u529f\u624d\u53ef\u8fdb\u884c\u540e\u7eed\u64cd\u4f5c\u3002
flow table query --namespace experiment --name breast_hetero_guest\nflow table query --namespace experiment --name breast_hetero_host\n\u5f53\u4f60\u7684\u6570\u636e\u51c6\u5907\u597d\u540e\uff0c\u53ef\u4ee5\u5f00\u59cb\u63d0\u4ea4\u4f5c\u4e1a\u7ed9FATE Flow\uff1a - \u8bad\u7ec3job\u914d\u7f6eexample\u4f4d\u4e8elr-train; - \u9884\u6d4bjob\u914d\u7f6eexample\u4f4d\u4e8elr-predict;\u9884\u6d4b\u4efb\u52a1\u9700\u8981\u4fee\u6539\"dag.conf.model_warehouse\"\u6210\u8bad\u7ec3\u4f5c\u4e1a\u7684\u8f93\u51fa\u6a21\u578b\u3002 - \u8bad\u7ec3\u548c\u9884\u6d4bjob\u914d\u7f6e\u4e2d\u7ad9\u70b9id\u4e3a\"9998\"\u548c\"9999\"\u3002\u5982\u679c\u4f60\u7684\u90e8\u7f72\u73af\u5883\u4e3a\u96c6\u7fa4\u7248\uff0c\u9700\u8981\u66ff\u6362\u6210\u771f\u5b9e\u7684\u7ad9\u70b9id\uff1b\u5355\u673a\u7248\u53ef\u4f7f\u7528\u9ed8\u8ba4\u914d\u7f6e\u3002 - \u5982\u679c\u60f3\u8981\u4f7f\u7528\u81ea\u5df1\u7684\u6570\u636e\uff0c\u53ef\u4ee5\u66f4\u6539\u914d\u7f6e\u4e2dguest\u548chost\u7684data_warehouse\u7684namespace\u548cname - \u63d0\u4ea4\u4f5c\u4e1a\u7684\u547d\u4ee4\u4e3a:
flow job submit -c examples/lr/train_lr.yaml {\n\"code\": 0,\n\"data\": {\n\"model_id\": \"202308211911505128750\",\n\"model_version\": \"0\"\n},\n\"job_id\": \"202308211911505128750\",\n\"message\": \"success\"\n}\n\u5728\u4f5c\u4e1a\u7684\u8fd0\u884c\u8fc7\u7a0b\u65f6\uff0c\u4f60\u53ef\u4ee5\u901a\u8fc7\u67e5\u8be2\u547d\u4ee4\u83b7\u53d6\u4f5c\u4e1a\u7684\u8fd0\u884c\u72b6\u6001
flow job query -j $job_id\n\u5728\u4f5c\u4e1a\u7684\u8fd0\u884c\u8fc7\u7a0b\u65f6\uff0c\u4f60\u53ef\u4ee5\u901a\u8fc7\u505c\u6b62\u4f5c\u4e1a\u547d\u4ee4\u6765\u7ec8\u6b62\u5f53\u524d\u4f5c\u4e1a
flow job stop -j $job_id\n\u5728\u4f5c\u4e1a\u7684\u8fd0\u884c\u8fc7\u7a0b\u65f6\uff0c\u5982\u679c\u8fd0\u884c\u5931\u8d25\uff0c\u4f60\u53ef\u4ee5\u901a\u8fc7\u91cd\u8dd1\u547d\u4ee4\u6765\u91cd\u8dd1\u5f53\u524d\u4f5c\u4e1a
flow job rerun -j $job_id\n\u4f5c\u4e1a\u7684\u8f93\u51fa\u5305\u62ec\u6570\u636e\u3001\u6a21\u578b\u548c\u6307\u6807
"},{"location":"zh/quick_start/#231","title":"2.3.1 \u8f93\u51fa\u6307\u6807","text":"\u67e5\u8be2\u8f93\u51fa\u6307\u6807\u547d\u4ee4\uff1a
flow output query-metric -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-metric -j 202308211911505128750 -r arbiter -p 9998 -tn lr_0\u67e5\u8be2\u3002 \u67e5\u8be2\u7ed3\u679c\u5982\u4e0b: {\n\"code\": 0,\n\"data\": [\n{\n\"data\": [\n{\n\"metric\": [\n0.0\n],\n\"step\": 0,\n\"timestamp\": 1692616428.253495\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n},\n{\n\"data\": [\n{\n\"metric\": [\n-0.07785049080848694\n],\n\"step\": 1,\n\"timestamp\": 1692616432.9727712\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n}\n],\n\"message\": \"success\"\n}\nflow output query-model -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-model -j 202308211911505128750 -r host -p 9998 -tn lr_0\u67e5\u8be2\u3002 \u67e5\u8be2\u7ed3\u679c\u5982\u4e0b\uff1a {\n\"code\": 0,\n\"data\": {\n\"output_model\": {\n\"data\": {\n\"estimator\": {\n\"end_epoch\": 10,\n\"is_converged\": false,\n\"lr_scheduler\": {\n\"lr_params\": {\n\"start_factor\": 0.7,\n\"total_iters\": 100\n},\n\"lr_scheduler\": {\n\"_get_lr_called_within_step\": false,\n\"_last_lr\": [\n0.07269999999999996\n],\n\"_step_count\": 10,\n\"base_lrs\": [\n0.1\n],\n\"end_factor\": 1.0,\n\"last_epoch\": 9,\n\"start_factor\": 0.7,\n\"total_iters\": 100,\n\"verbose\": false\n},\n\"method\": \"linear\"\n},\n\"optimizer\": {\n\"alpha\": 0.001,\n\"l1_penalty\": false,\n\"l2_penalty\": true,\n\"method\": \"sgd\",\n\"model_parameter\": [\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n],\n[\n0.0\n]\n],\n\"model_parameter_dtype\": \"float32\",\n\"optim_param\": {\n\"lr\": 0.1\n},\n\"optimizer\": {\n\"param_groups\": [\n{\n\"dampening\": 0,\n\"differentiable\": false,\n\"foreach\": null,\n\"initial_lr\": 0.1,\n\"lr\": 0.07269999999999996,\n\"maximize\": false,\n\"momentum\": 0,\n\"nesterov\": false,\n\"params\": [\n0\n],\n\"weight_decay\": 0\n}\n],\n\"state\": {}\n}\n},\n\"param\": {\n\"coef_\": [\n[\n-0.10828543454408646\n],\n[\n-0.07341302931308746\n],\n[\n-0.10850320011377335\n],\n[\n-0.10066638141870499\n],\n[\n-0.04595951363444328\n],\n[\n-0.07001449167728424\n],\n[\n-0.08949052542448044\n],\n[\n-0.10958756506443024\n],\n[\n-0.04012322425842285\n],\n[\n0.02270071767270565\n],\n[\n-0.07198350876569748\n],\n[\n0.00548586156219244\n],\n[\n-0.06599288433790207\n],\n[\n-0.06410090625286102\n],\n[\n0.016374297440052032\n],\n[\n-0.01607361063361168\n],\n[\n-0.011447405442595482\n],\n[\n-0.04352564364671707\n],\n[\n0.013161249458789825\n],\n[\n0.013506329618394375\n]\n],\n\"dtype\": \"float32\",\n\"intercept_\": null\n}\n}\n},\n\"meta\": {\n\"batch_size\": null,\n\"epochs\": 10,\n\"init_param\": {\n\"fill_val\": 0.0,\n\"fit_intercept\": false,\n\"method\": \"zeros\",\n\"random_state\": null\n},\n\"label_count\": false,\n\"learning_rate_param\": {\n\"method\": \"linear\",\n\"scheduler_params\": {\n\"start_factor\": 0.7,\n\"total_iters\": 100\n}\n},\n\"optimizer_param\": {\n\"alpha\": 0.001,\n\"method\": \"sgd\",\n\"optimizer_params\": {\n\"lr\": 0.1\n},\n\"penalty\": \"l2\"\n},\n\"ovr\": false\n}\n}\n},\n\"message\": \"success\"\n}\nflow output download-model -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-model -j 202308211911505128750 -r host -p 9998 -tn lr_0 -o ./\u4e0b\u8f7d\u3002 \u4e0b\u8f7d\u7ed3\u679c\u5982\u4e0b\uff1a {\n\"code\": 0,\n\"directory\": \"./output_model_202308211911505128750_host_9998_lr_0\",\n\"message\": \"download success, please check the path: ./output_model_202308211911505128750_host_9998_lr_0\"\n}\nflow output query-data-table -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-data-table -j 202308211911505128750 -r host -p 9998 -tn binning_0\u67e5\u8be2\u3002 \u67e5\u8be2\u7ed3\u679c\u5982\u4e0b\uff1a {\n\"train_output_data\": [\n{\n\"name\": \"9e28049c401311ee85c716b977118319\",\n\"namespace\": \"202308211911505128750_binning_0\"\n}\n]\n}\nflow output display-data -j $job_id -r $role -p $party_id -tn $task_name\nflow output display-data -j 202308211911505128750 -r host -p 9998 -tn binning_0\u9884\u89c8\u8f93\u51fa\u6570\u636e\u3002"},{"location":"zh/quick_start/#2333","title":"2.3.3.3 \u4e0b\u8f7d\u6570\u636e","text":"flow output download-data -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-data -j 202308211911505128750 -r guest -p 9999 -tn lr_0 -o ./\u4e0b\u8f7d\u8f93\u51fa\u6570\u636e\u3002 \u4e0b\u8f7d\u7ed3\u679c\u5982\u4e0b\uff1a {\n\"code\": 0,\n\"directory\": \"./output_data_202308211911505128750_guest_9999_lr_0\",\n\"message\": \"download success, please check the path: ./output_data_202308211911505128750_guest_9999_lr_0\"\n}\nFATE Flow\u4f7f\u7528yaml\u5b9a\u4e49\u7cfb\u7edf\u914d\u7f6e\uff0c\u914d\u7f6e\u8def\u5f84\u4f4d\u4e8e: conf/service_conf.yaml, \u5177\u4f53\u914d\u7f6e\u5185\u5bb9\u53ca\u5176\u542b\u4e49\u5982\u4e0b\uff1a
\u914d\u7f6e\u9879 \u8bf4\u660e \u503c party_id \u672c\u65b9\u7ad9\u70b9id \u5982: \"9999\", \"10000 use_registry \u662f\u5426\u4f7f\u7528\u6ce8\u518c\u4e2d\u5fc3\uff0c\u5f53\u524d\u4ec5\u652f\u6301zookeeper\u6a21\u5f0f\uff0c\u9700\u8981\u4fdd\u8bc1zookeeper\u7684\u914d\u7f6e\u6b63\u786e\uff1b\u6ce8\uff1a\u82e5\u4f7f\u7528\u9ad8\u53ef\u7528\u6a21\u5f0f\uff0c\u9700\u4fdd\u8bc1\u8be5\u914d\u7f6e\u8bbe\u7f6e\u4e3atrue true/false encrypt \u52a0\u5bc6\u6a21\u5757 \u89c1\u52a0\u5bc6\u6a21\u5757 fateflow FATE Flow\u670d\u52a1\u7684\u914d\u7f6e\uff0c\u4e3b\u8981\u5305\u62ec\u7aef\u53e3\u3001\u547d\u4ee4\u901a\u9053\u670d\u52a1\u3001\u4ee3\u7406\u7b49 \u89c1FateFlow\u914d\u7f6e database \u6570\u636e\u5e93\u670d\u52a1\u7684\u914d\u7f6e\u4fe1\u606f \u89c1\u6570\u636e\u5e93\u914d\u7f6e default_engines \u7cfb\u7edf\u7684\u5f15\u64ce\u670d\u52a1\uff0c\u4e3b\u8981\u5305\u62ec\u8ba1\u7b97\u3001\u5b58\u50a8\u548c\u901a\u4fe1\u5f15\u64ce \u89c1\u5f15\u64ce\u914d\u7f6e default_provider \u7ec4\u4ef6\u7684\u6765\u6e90\u4fe1\u606f\uff0c\u4e3b\u8981\u5305\u62ec\u63d0\u4f9b\u65b9\u540d\u79f0\u3001\u7ec4\u4ef6\u7248\u672c\u548c\u8fd0\u884c\u6a21\u5f0f \u89c1\u9ed8\u8ba4\u6ce8\u518c\u7b97\u6cd5\u914d\u7f6e federation \u901a\u4fe1\u670d\u52a1\u6c60 \u89c1\u901a\u4fe1\u5f15\u64ce\u6c60 computing \u8ba1\u7b97\u670d\u52a1\u6c60 \u89c1\u8ba1\u7b97\u5f15\u64ce\u6c60 storage \u5b58\u50a8\u670d\u52a1\u6c60 \u89c1\u5b58\u50a8\u5f15\u64ce\u6c60 hook_module \u94a9\u5b50\u914d\u7f6e\uff0c\u5f53\u524d\u652f\u6301\u5ba2\u6237\u7aef\u8ba4\u8bc1\u3001\u7ad9\u70b9\u7aef\u8ba4\u8bc1\u4ee5\u53ca\u9274\u6743\u94a9\u5b50 \u89c1\u94a9\u5b50\u6a21\u5757\u914d\u7f6e authentication \u8ba4\u8bc1&&\u9274\u6743\u5f00\u5173 \u89c1\u8ba4\u8bc1\u5f00\u5173 model_store \u6a21\u578b\u5b58\u50a8\u914d\u7f6e \u89c1\u6a21\u578b\u5b58\u50a8 zookeeper zookeeper\u670d\u52a1\u7684\u914d\u7f6e \u89c1zookeeper\u914d\u7f6e"},{"location":"zh/system_conf/#_2","title":"\u52a0\u5bc6\u6a21\u5757","text":"key_0:\nmodule: fate_flow.hub.encrypt.password_encrypt#pwdecrypt\nprivate_path: private_key.pem\nhost: 127.0.0.1\nhttp_port: 9380\ngrpc_port: 9360\nproxy_name: rollsite\nnginx:\nhost:\nhttp_port:\ngrpc_port:\nengine: sqlite\ndecrypt_key:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\nsqlite:\npath:\ndefault_engines:\ncomputing: standalone\nfederation: standalone\nstorage: standalone\npulsar:\nhost: 192.168.0.5\nport: 6650\nmng_port: 8080\ncluster: standalone\ntenant: fl-tenant\ntopic_ttl: 30\n# default conf/pulsar_route_table.yaml\nroute_table:\n# mode: replication / client, default: replication\nmode: replication\nmax_message_size: 1048576\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\n# http or grpc\nprotocol: http\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\n# http or grpc\nprotocol: http\nrollsite:\nhost: 127.0.0.1\nport: 9370\n host: 127.0.0.1\nport: 9370\n cores: 32\neggroll:\ncores: 32\nnodes: 2\neggroll:\nhome: cores: 32\n hdfs:\nname_node: hdfs://fate-cluster\nhook_module:\nclient_authentication: fate_flow.hook.flow.client_authentication\nsite_authentication: fate_flow.hook.flow.site_authentication\npermission: fate_flow.hook.flow.permission\nauthentication:\nclient: false\nsite: false\npermission: false\nmodel_store:\nengine: file\ndecrypt_key:\nfile:\npath:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\ntencent_cos:\nRegion:\nSecretId:\nSecretKey:\nBucket:\nzookeeper:\nhosts:\n- 127.0.0.1:2181\nuse_acl: true\nuser: fate\npassword: fate\nWhen the job runs, Reader reads directly from Storage
Table Bind.
You can choose one of the following three deployment modes based on your requirements:
"},{"location":"quick_start/#11-pypi-package-installation","title":"1.1 Pypi Package Installation","text":"Note: This mode operates in a single-machine mode.
"},{"location":"quick_start/#111-installation","title":"1.1.1 Installation","text":"# FATE requires Python >= 3.8\nconda create -n fate_env python=3.8\nconda activate fate_env\npip install fate_client[fate,fate_flow]==2.0.0.b0\nfate_flow init --ip 127.0.0.1 --port 9380 --home $HOME_DIR\nip: The IP address where the service runs. - port: The HTTP port the service runs on. - home: The data storage directory, including data, models, logs, job configurations, and SQLite databases."},{"location":"quick_start/#113-service-startstop","title":"1.1.3 Service Start/Stop","text":"fate_flow status/start/stop/restart\nRefer to Standalone Deployment.
"},{"location":"quick_start/#13-cluster-deployment","title":"1.3 Cluster Deployment","text":"Refer to Allinone Deployment.
"},{"location":"quick_start/#2-user-guide","title":"2. User Guide","text":"FATE provides client tools including SDK, CLI, and Pipeline. If you don't have FATE Client deployed in your environment, you can download it using pip install fate_client. The following operations are based on CLI.
In version 2.0-beta, data uploading is a two-step process:
flow data upload -c examples/upload/upload_guest.json\nflow data upload -c examples/upload/upload_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"36491bc8-3fef-11ee-be05-16b977118319\",\n\"namespace\": \"upload\"\n},\n\"job_id\": \"202308211451535620150\",\n\"message\": \"success\"\n}\nSince upload is an asynchronous operation, you need to confirm if it was successful before proceeding to the next step.
flow table query --namespace upload --name 36491bc8-3fef-11ee-be05-16b977118319\nflow data transformer -c examples/transformer/transformer_guest.json\nflow data transformer -c examples/transformer/transformer_host.json\n{\n\"code\": 0,\n\"data\": {\n\"name\": \"breast_hetero_guest\",\n\"namespace\": \"experiment\"\n},\n\"job_id\": \"202308211557455662860\",\n\"message\": \"success\"\n}\nSince the transformer is also an asynchronous operation, you need to confirm if it was successful before proceeding.
flow table query --namespace experiment --name breast_hetero_guest\nflow table query --namespace experiment --name breast_hetero_host\nOnce your data is prepared, you can start submitting jobs to FATE Flow:
flow job submit -c examples/lr/train_lr.yaml {\n\"code\": 0,\n\"data\": {\n\"model_id\": \"202308211911505128750\",\n\"model_version\": \"0\"\n},\n\"job_id\": \"202308211911505128750\",\n\"message\": \"success\"\n}\nWhile a job is running, you can check its status using the query command:
flow job query -j $job_id\nDuring job execution, you can stop the current job using the stop command:
flow job stop -j $job_id\nIf a job fails during execution, you can rerun it using the rerun command:
flow job rerun -j $job_id\nJob outputs include data, models, and metrics.
"},{"location":"quick_start/#231-output-metrics","title":"2.3.1 Output Metrics","text":"To query output metrics, use the following command:
flow output query-metric -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-metric -j 202308211911505128750 -r arbiter -p 9998 -tn lr_0 to query metrics. The query result will look like this: {\n\"code\": 0,\n\"data\": [\n{\n\"data\": [\n{\n\"metric\": [\n0.0\n],\n\"step\": 0,\n\"timestamp\": 1692616428.253495\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n},\n{\n\"data\": [\n{\n\"metric\": [\n-0.07785049080848694\n],\n\"step\": 1,\n\"timestamp\": 1692616432.9727712\n}\n],\n\"groups\": [\n{\n\"index\": null,\n\"name\": \"default\"\n},\n{\n\"index\": null,\n\"name\": \"train\"\n}\n],\n\"name\": \"lr_loss\",\n\"step_axis\": \"iterations\",\n\"type\": \"loss\"\n}\n],\n\"message\": \"success\"\n}\nTo query output models, use the following command:
flow output query-model -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-model -j 202308211911505128750 -r host -p 9998 -tn lr_0 to query models. The query result will be similar to this: {\n\"code\": 0,\n\"data\": [\n{\n\"model\": {\n\"file\": \"202308211911505128750_host_9998_lr_0\",\n\"namespace\": \"202308211911505128750_host_9998_lr_0\"\n},\n\"name\": \"HeteroLRHost_9998_0\",\n\"namespace\": \"202308211911505128750_host_9998_lr_0\",\n\"role\": \"host\",\n\"party_id\": \"9998\",\n\"work_mode\": 1\n}\n],\n\"message\": \"success\"\n}\nTo download models, use the following command:
flow output download-model -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-model -j 202308211911505128750 -r host -p 9998 -tn lr_0 -o ./ to download the model. The download result will be similar to this: {\n\"code\": 0,\n\"directory\": \"./output_model_202308211911505128750_host_9998_lr_0\",\n\"message\": \"download success, please check the path: ./output_model_202308211911505128750_host_9998_lr_0\"\n}\nTo query output data tables, use the following command:
flow output query-data-table -j $job_id -r $role -p $party_id -tn $task_name\nflow output query-data-table -j 202308211911505128750 -r host -p 9998 -tn binning_0 to query data tables. The query result will be similar to this: {\n\"train_output_data\": [\n{\n\"name\": \"9e28049c401311ee85c716b977118319\",\n\"namespace\": \"202308211911505128750_binning_0\"\n}\n]\n}\nflow output display-data -j $job_id -r $role -p $party_id -tn $task_name\nflow output display-data -j 202308211911505128750 -r host -p 9998 -tn binning_0."},{"location":"quick_start/#2333-download-data","title":"2.3.3.3 Download Data","text":"flow output download-data -j $job_id -r $role -p $party_id -tn $task_name -o $download_dir\nflow output download-data -j 202308211911505128750 -r guest -p 9999 -tn lr_0 -o ./. The download result will be as follows:
{\n\"code\": 0,\n\"directory\": \"./output_data_202308211911505128750_guest_9999_lr_0\",\n\"message\": \"download success, please check the path: ./output_data_202308211911505128750_guest_9999_lr_0\"\n}\nFATE Flow uses YAML to define system configurations, and the configuration file is located at: conf/service_conf.yaml. The specific configuration contents and their meanings are as follows:
key_0:\nmodule: fate_flow.hub.encrypt.password_encrypt#pwdecrypt\nprivate_path: private_key.pem\nfate_flow/conf/."},{"location":"system_conf/#fateflow-configuration","title":"FateFlow Configuration","text":"host: 127.0.0.1\nhttp_port: 9380\ngrpc_port: 9360\nproxy_name: rollsite\nnginx:\nhost:\nhttp_port:\ngrpc_port:\nengine: sqlite\ndecrypt_key:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\nsqlite:\npath:\nfate_flow/fate_flow_sqlite.db."},{"location":"system_conf/#engine-configuration","title":"Engine Configuration","text":"default_engines:\ncomputing: standalone\nfederation: standalone\nstorage: standalone\nfateflow.env.pulsar:\nhost: 192.168.0.5\nport: 6650\nmng_port: 8080\ncluster: standalone\ntenant: fl-tenant\ntopic_ttl: 30\nroute_table:\nmode: replication\nmax_message_size: 1048576\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\nprotocol: http\nnginx:\nhost: 127.0.0.1\nhttp_port: 9300\ngrpc_port: 9310\nprotocol: http\nrollsite:\nhost: 127.0.0.1\nport: 9370\n host: 127.0.0.1\nport: 9370\n cores: 32\neggroll:\ncores: 32\nnodes: 2\neggroll:\nhome: cores: 32\n hdfs:\nname_node: hdfs://fate-cluster\nhook_module:\nclient_authentication: fate_flow.hook.flow.client_authentication\nsite_authentication: fate_flow.hook.flow.site_authentication\npermission: fate_flow.hook.flow.permission\nauthentication:\nclient: false\nsite: false\npermission: false\nmodel_store:\nengine: file\ndecrypt_key:\nfile:\npath:\nmysql:\nname: fate_flow\nuser: fate\npasswd: fate\nhost: 127.0.0.1\nport: 3306\nmax_connections: 100\nstale_timeout: 30\ntencent_cos:\nRegion:\nSecretId:\nSecretKey:\nBucket:\nfate_flow/model. - mysql: MySQL service configuration; if using password encryption functionality, you need to set the \"passwd\" in this configuration to ciphertext and configure the key path in Encryption Module. - tencent_cos: Tencent Cloud key configuration."},{"location":"system_conf/#zookeeper-configuration","title":"ZooKeeper Configuration","text":"zookeeper:\nhosts:\n- 127.0.0.1:2181\nuse_acl: true\nuser: fate\npassword: fate\nAt repo root, execute
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs to serve docs in http://localhost:8000
or
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs build\nto build docs to site folder.
mkdocs-material and servel plugins are needed to build this docs
Fisrt, create an python virtual environment
python3 -m venv \"fatedocs\"\nsource fatedocs/bin/activate\npip install -U pip\npip install -r doc/mkdocs/requirements.txt\nNow, use
mkdocs serve\nat repo root to serve docs or
use
mkdocs build\nat repo root to build docs to folder site
We use mkdocs-material to build our docs. Servel markdown extensions are really useful to write pretty documents such as admonitions and content-tabs.
Servel plugins are introdused to makes mkdocs-material much powerful:
mkdocstrings automatic documentation from sources code. We mostly use this to automatic generate params api for federatedml.
awesome-pages for powerful nav rule
i18n for multi-languege support
mkdocs-jupyter for jupyter format support
mkdocs-simple-hooks for simple plugin-in
This image is modified from mkdocs-meterial with some plugins embeded.
Usage
Mount the folder where your mkdocs.yml resides as a volume into /docs:
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs sagewei0/mkdocs\ndocker run --rm -it -v ${PWD}:/docs sagewei/mkdocs build\ndocker run --rm -it -v ~/.ssh:/root/.ssh -v ${PWD}:/docs sagewei0/mkdocs gh-deploy \nMostly copied from https://github.com/cirruslabs/cirrus-ci-docs/tree/master/theme
"},{"location":"swagger/","title":"API","text":""},{"location":"swagger/#swagger-api","title":"Swagger API","text":""}]} \ No newline at end of file diff --git a/2.0.0-beta/sitemap.xml b/2.0.0-beta/sitemap.xml index 1cfca4f9d..a0788c873 100644 --- a/2.0.0-beta/sitemap.xml +++ b/2.0.0-beta/sitemap.xml @@ -2,112 +2,112 @@fate_flow status/start/stop/restart
参考单机版部署
+参考单机版部署
fate提供的客户端包括SDK、CLI和Pipeline，若你的环境中没有部署FATE Client,可以使用pip install fate_client下载，以下的使用操作均基于cli编写。
当你的数据准备好后，可以开始提交作业给FATE Flow： -- 训练job配置example位于lr-train; -- 预测job配置example位于lr-predict;预测任务需要修改"dag.conf.model_warehouse"成训练作业的输出模型。 +- 训练job配置example位于lr-train; +- 预测job配置example位于lr-predict;预测任务需要修改"dag.conf.model_warehouse"成训练作业的输出模型。 - 训练和预测job配置中站点id为"9998"和"9999"。如果你的部署环境为集群版，需要替换成真实的站点id；单机版可使用默认配置。 - 如果想要使用自己的数据，可以更改配置中guest和host的data_warehouse的namespace和name - 提交作业的命令为: @@ -1689,10 +1689,11 @@