Skip to content

Commit

Permalink
Update KubeFATE to support FATE v1.8.1 (#824) (#828)
Browse files Browse the repository at this point in the history 
Signed-off-by: Chenlong Ma <[email protected]>

Signed-off-by: Chenlong Ma <[email protected]>

Signed-off-by: Chenlong Ma <[email protected]>
  • Loading branch information
@owlet42
owlet42 authored Dec 12, 2022
1 parent d710a7d commit c7036e4
Show file tree
Hide file tree
Showing 34 changed files with 224 additions and 97 deletions.
2 changes: 1 addition & 1 deletion docker-build/.env
View file
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
#PREFIX=federatedai
#IMG_TAG=1.8.0-release
#IMG_TAG=1.8.1-release
2 changes: 1 addition & 1 deletion docker-deploy/.env
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
RegistryURI=
TAG=1.8.0-release
TAG=1.8.1-release
SERVING_TAG=2.1.5-release

# PREFIX: namespace on the registry's server.
Expand Down
12 changes: 6 additions & 6 deletions docker-deploy/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -170,12 +170,12 @@ CONTAINER ID IMAGE COMMAND
3dca43f3c9d5 federatedai/serving-admin:2.1.5-release "/bin/sh -c 'java -c…" 5 minutes ago Up 5 minutes 0.0.0.0:8350->8350/tcp, :::8350->8350/tcp serving-9999_serving-admin_1
fe924918509b federatedai/serving-proxy:2.1.5-release "/bin/sh -c 'java -D…" 5 minutes ago Up 5 minutes 0.0.0.0:8059->8059/tcp, :::8059->8059/tcp, 0.0.0.0:8869->8869/tcp, :::8869->8869/tcp, 8879/tcp serving-9999_serving-proxy_1
b62ed8ba42b7 bitnami/zookeeper:3.7.0 "/opt/bitnami/script…" 5 minutes ago Up 5 minutes 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp, 0.0.0.0:49226->2888/tcp, :::49226->2888/tcp, 0.0.0.0:49225->3888/tcp, :::49225->3888/tcp serving-9999_serving-zookeeper_1
3c643324066f federatedai/client:1.8.0-release "/bin/sh -c 'flow in…" 5 minutes ago Up 5 minutes 0.0.0.0:20000->20000/tcp, :::20000->20000/tcp confs-9999_client_1
3fe0af1ebd71 federatedai/fateboard:1.8.0-release "/bin/sh -c 'java -D…" 5 minutes ago Up 5 minutes 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp confs-9999_fateboard_1
635b7d99357e federatedai/python:1.8.0-release "container-entrypoin…" 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:9360->9360/tcp, :::9360->9360/tcp, 8080/tcp, 0.0.0.0:9380->9380/tcp, :::9380->9380/tcp confs-9999_python_1
8b515f08add3 federatedai/eggroll:1.8.0-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 8080/tcp, 0.0.0.0:9370->9370/tcp, :::9370->9370/tcp confs-9999_rollsite_1
108cc061c191 federatedai/eggroll:1.8.0-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 4670/tcp, 8080/tcp confs-9999_clustermanager_1
f10575e76899 federatedai/eggroll:1.8.0-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 4671/tcp, 8080/tcp confs-9999_nodemanager_1
3c643324066f federatedai/client:1.8.1-release "/bin/sh -c 'flow in…" 5 minutes ago Up 5 minutes 0.0.0.0:20000->20000/tcp, :::20000->20000/tcp confs-9999_client_1
3fe0af1ebd71 federatedai/fateboard:1.8.1-release "/bin/sh -c 'java -D…" 5 minutes ago Up 5 minutes 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp confs-9999_fateboard_1
635b7d99357e federatedai/python:1.8.1-release "container-entrypoin…" 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:9360->9360/tcp, :::9360->9360/tcp, 8080/tcp, 0.0.0.0:9380->9380/tcp, :::9380->9380/tcp confs-9999_python_1
8b515f08add3 federatedai/eggroll:1.8.1-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 8080/tcp, 0.0.0.0:9370->9370/tcp, :::9370->9370/tcp confs-9999_rollsite_1
108cc061c191 federatedai/eggroll:1.8.1-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 4670/tcp, 8080/tcp confs-9999_clustermanager_1
f10575e76899 federatedai/eggroll:1.8.1-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 4671/tcp, 8080/tcp confs-9999_nodemanager_1
aa0a0002de93 mysql:8.0.28 "docker-entrypoint.s…" 5 minutes ago Up 5 minutes 3306/tcp, 33060/tcp confs-9999_mysql_1
```

Expand Down
12 changes: 6 additions & 6 deletions docker-deploy/README_zh.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -189,12 +189,12 @@ CONTAINER ID IMAGE COMMAND
3dca43f3c9d5 federatedai/serving-admin:2.1.5-release "/bin/sh -c 'java -c…" 5 minutes ago Up 5 minutes 0.0.0.0:8350->8350/tcp, :::8350->8350/tcp serving-9999_serving-admin_1
fe924918509b federatedai/serving-proxy:2.1.5-release "/bin/sh -c 'java -D…" 5 minutes ago Up 5 minutes 0.0.0.0:8059->8059/tcp, :::8059->8059/tcp, 0.0.0.0:8869->8869/tcp, :::8869->8869/tcp, 8879/tcp serving-9999_serving-proxy_1
b62ed8ba42b7 bitnami/zookeeper:3.7.0 "/opt/bitnami/script…" 5 minutes ago Up 5 minutes 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp, 0.0.0.0:49226->2888/tcp, :::49226->2888/tcp, 0.0.0.0:49225->3888/tcp, :::49225->3888/tcp serving-9999_serving-zookeeper_1
3c643324066f federatedai/client:1.8.0-release "/bin/sh -c 'flow in…" 5 minutes ago Up 5 minutes 0.0.0.0:20000->20000/tcp, :::20000->20000/tcp confs-9999_client_1
3fe0af1ebd71 federatedai/fateboard:1.8.0-release "/bin/sh -c 'java -D…" 5 minutes ago Up 5 minutes 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp confs-9999_fateboard_1
635b7d99357e federatedai/python:1.8.0-release "container-entrypoin…" 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:9360->9360/tcp, :::9360->9360/tcp, 8080/tcp, 0.0.0.0:9380->9380/tcp, :::9380->9380/tcp confs-9999_python_1
8b515f08add3 federatedai/eggroll:1.8.0-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 8080/tcp, 0.0.0.0:9370->9370/tcp, :::9370->9370/tcp confs-9999_rollsite_1
108cc061c191 federatedai/eggroll:1.8.0-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 4670/tcp, 8080/tcp confs-9999_clustermanager_1
f10575e76899 federatedai/eggroll:1.8.0-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 4671/tcp, 8080/tcp confs-9999_nodemanager_1
3c643324066f federatedai/client:1.8.1-release "/bin/sh -c 'flow in…" 5 minutes ago Up 5 minutes 0.0.0.0:20000->20000/tcp, :::20000->20000/tcp confs-9999_client_1
3fe0af1ebd71 federatedai/fateboard:1.8.1-release "/bin/sh -c 'java -D…" 5 minutes ago Up 5 minutes 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp confs-9999_fateboard_1
635b7d99357e federatedai/python:1.8.1-release "container-entrypoin…" 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:9360->9360/tcp, :::9360->9360/tcp, 8080/tcp, 0.0.0.0:9380->9380/tcp, :::9380->9380/tcp confs-9999_python_1
8b515f08add3 federatedai/eggroll:1.8.1-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 8080/tcp, 0.0.0.0:9370->9370/tcp, :::9370->9370/tcp confs-9999_rollsite_1
108cc061c191 federatedai/eggroll:1.8.1-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 4670/tcp, 8080/tcp confs-9999_clustermanager_1
f10575e76899 federatedai/eggroll:1.8.1-release "/tini -- bash -c 'j…" 5 minutes ago Up 5 minutes 4671/tcp, 8080/tcp confs-9999_nodemanager_1
aa0a0002de93 mysql:8.0.28 "docker-entrypoint.s…" 5 minutes ago Up 5 minutes 3306/tcp, 33060/tcp confs-9999_mysql_1
```

Expand Down
127 changes: 127 additions & 0 deletions docker-deploy/training_template/backends/eggroll/conf/whitelist.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
{
"builtins": [
"int",
"list",
"set"
],
"collections": [
"OrderedDict",
"defaultdict"
],
"eggroll.core.transfer_model": [
"ErRollSiteHeader"
],
"eggroll.roll_pair.task.storage": [
"BSS"
],
"federatedml.cipher_compressor.compressor": [
"PackingCipherTensor",
"NormalCipherPackage",
"PackingCipherTensorPackage"
],
"federatedml.ensemble.basic_algorithms.decision_tree.tree_core.feature_histogram": [
"FeatureHistogramWeights",
"HistogramBag"
],
"federatedml.ensemble.basic_algorithms.decision_tree.tree_core.feature_importance": [
"FeatureImportance"
],
"federatedml.ensemble.basic_algorithms.decision_tree.tree_core.g_h_optim": [
"SplitInfoPackage"
],
"federatedml.ensemble.basic_algorithms.decision_tree.tree_core.node": [
"Node"
],
"federatedml.ensemble.basic_algorithms.decision_tree.tree_core.splitter": [
"SplitInfo"
],
"federatedml.evaluation.performance_recorder": [
"PerformanceRecorder"
],
"federatedml.feature.binning.bin_result": [
"BinColResults"
],
"federatedml.feature.binning.optimal_binning.bucket_info": [
"Bucket"
],
"federatedml.feature.binning.quantile_summaries": [
"QuantileSummaries",
"Stats",
"SparseQuantileSummaries"
],
"federatedml.feature.fate_element_type": [
"NoneType"
],
"federatedml.feature.homo_feature_binning.homo_binning_base": [
"SplitPointNode"
],
"federatedml.feature.instance": [
"Instance"
],
"federatedml.feature.one_hot_encoder": [
"TransferPair"
],
"federatedml.feature.sparse_vector": [
"SparseVector"
],
"federatedml.framework.weights": [
"TransferableWeights",
"DictWeights",
"NumpyWeights",
"ListWeights",
"OrderDictWeights",
"NumericWeights"
],
"federatedml.linear_model.linear_model_weight": [
"LinearModelWeights"
],
"federatedml.secureprotol.fate_paillier": [
"PaillierPublicKey",
"PaillierEncryptedNumber"
],
"federatedml.secureprotol.fixedpoint": [
"FixedPointNumber"
],
"federatedml.secureprotol.number_theory.field.integers_modulo_prime_field": [
"IntegersModuloPrimeElement"
],
"federatedml.secureprotol.number_theory.group.twisted_edwards_curve_group": [
"TwistedEdwardsCurveElement"
],
"federatedml.secureprotol.symmetric_encryption.cryptor_executor": [
"CryptoExecutor"
],
"federatedml.secureprotol.symmetric_encryption.pohlig_hellman_encryption": [
"PohligHellmanCipherKey",
"PohligHellmanCiphertext"
],
"federatedml.statistic.intersect.intersect_preprocess": [
"BitArray"
],
"federatedml.statistic.statics": [
"SummaryStatistics"
],
"gmpy2": [
"from_binary"
],
"numpy": [
"dtype",
"ndarray"
],
"numpy.core.multiarray": [
"_reconstruct",
"scalar"
],
"numpy.core.numeric": [
"_frombuffer"
],
"tensorflow.python.framework.ops": [
"convert_to_tensor"
],
"torch._utils": [
"_rebuild_tensor_v2"
],
"torch.storage": [
"_load_from_bytes"
]
}
8 changes: 4 additions & 4 deletions docs/Manage_FATE_and_FATE-Serving_Version.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,18 +30,18 @@ The chart can be downloaded in each KubeFATE release, with name `fate-{release_v

Download it and copy it to the folder to upload.
```
$ kubefate chart upload -f ./fate-v1.8.0.tgz
$ kubefate chart upload -f ./fate-v1.8.1.tgz
Upload file success
$ kubefate chart ls
UUID NAME VERSION APPVERSION
ca3f7843-749a-4f69-9f6b-4c544a7623ac fate v1.8.0 v1.8.0
ca3f7843-749a-4f69-9f6b-4c544a7623ac fate v1.8.1 v1.8.1
```

Then, we can deploy the fate cluster of v1.8.0 version. The detail of cluster.yaml please refer to: [FATE Cluster Configuration](./configurations/FATE_cluster_configuration.md)
Then, we can deploy the fate cluster of v1.8.1 version. The detail of cluster.yaml please refer to: [FATE Cluster Configuration](./configurations/FATE_cluster_configuration.md)
```
chartName: fate
chartVersion: v1.8.0
chartVersion: v1.8.1
```

We can delete the chart with:
Expand Down
24 changes: 12 additions & 12 deletions .../tutorials/Build_Two_Parties_FATE_Cluster_in_One_Linux_Machine_with_MiniKube.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ After the tutorial, the deployment architecture looks like the following diagram
[use image pull secrets](https://github.com/federatedai/KubeFATE/blob/master/docs/Use_image_pull_secrets.md).
5. Network connectivity to dockerhub or 163 Docker Image Registry, and google gcr.
6. Setup the global KubeFATE version using in the tutorial and create a folder for the whole tutorial. We use
KubeFATE v1.8.0 in this tutorial, other versions should be similar.
KubeFATE v1.8.1 in this tutorial, other versions should be similar.
```
export release_version=v1.8.0 && export kubefate_version=v1.4.4 && cd ~ && mkdir demo && cd demo
export release_version=v1.8.1 && export kubefate_version=v1.4.4 && cd ~ && mkdir demo && cd demo
```

**<font color="red">!!!Note: in this tutorial, the IP of the machine we used is 192.168.100.123. Please change it to your machine's IP in all the following commands and config files.</font></div>**
Expand Down Expand Up @@ -81,7 +81,7 @@ Till now, Kubernetes have been ready.
## Setup Kubefate
### Install KubeFATE CLI
Go to [KubeFATE Release](https://github.com/FederatedAI/KubeFATE/releases), and find the latest kubefate-k8s release
pack, which is `v1.8.0` as set to ENVs before. (replace ${release_version} with the newest version available)
pack, which is `v1.8.1` as set to ENVs before. (replace ${release_version} with the newest version available)
```
curl -LO https://github.com/FederatedAI/KubeFATE/releases/download/${release_version}/kubefate-k8s-${release_version}.tar.gz && tar -xzf ./kubefate-k8s-${release_version}.tar.gz
```
Expand Down Expand Up @@ -230,10 +230,10 @@ For `/kubefate/examples/party-9999/cluster.yaml`, modify it as following:
name: fate-9999
namespace: fate-9999
chartName: fate
chartVersion: v1.8.0
chartVersion: v1.8.1
partyId: 9999
registry: "hub.c.163.com/federatedai"
imageTag: "1.8.0-release"
imageTag: "1.8.1-release"
pullPolicy:
imagePullSecrets:
- name: myregistrykey
Expand Down Expand Up @@ -284,10 +284,10 @@ and for fate-10000:
name: fate-10000
namespace: fate-10000
chartName: fate
chartVersion: v1.8.0
chartVersion: v1.8.1
partyId: 10000
registry: "hub.c.163.com/federatedai"
imageTag: "1.8.0-release"
imageTag: "1.8.1-release"
pullPolicy:
imagePullSecrets:
- name: myregistrykey
Expand Down Expand Up @@ -354,8 +354,8 @@ or watch the clusters till their STATUS changing to `Running`:
```
kubefate@machine:~/kubefate$ watch kubefate cluster ls
UUID NAME NAMESPACE REVISION STATUS CHART ChartVERSION AGE
51476469-b473-4d41-b2d5-ea7241d5eac7 fate-9999 fate-9999 1 Running fate v1.8.0 88s
dacc0549-b9fc-463f-837a-4e7316db2537 fate-10000 fate-10000 1 Running fate v1.8.0 69s
51476469-b473-4d41-b2d5-ea7241d5eac7 fate-9999 fate-9999 1 Running fate v1.8.1 88s
dacc0549-b9fc-463f-837a-4e7316db2537 fate-10000 fate-10000 1 Running fate v1.8.1 69s
```
We have about 10G Docker images that need to be pulled, this step will take a while for the first time.
An alternative way is offline loading the images to the local environment.
Expand Down Expand Up @@ -387,16 +387,16 @@ UUID 51476469-b473-4d41-b2d5-ea7241d5eac7
Name fate-9999
NameSpace fate-9999
ChartName fate
ChartVersion v1.8.0
ChartVersion v1.8.1
Revision 1
Age 15h
Status Running
Spec backend: eggroll
chartName: fate
chartVersion: v1.8.0
chartVersion: v1.8.1
imagePullSecrets:
- name: myregistrykey
imageTag: 1.8.0-release
imageTag: 1.8.1-release
ingress:
client:
annotations:
Expand Down
Loading

0 comments on commit c7036e4

Please sign in to comment.