Merge branch 'master' of https://github.com/FederatedAI/FATE into fea…

…ture-2.0.0-rc-sshe-glm

README.md

@@ -9,7 +9,6 @@
   <img src="./doc/images/FATE_logo.png">
 </div>
 
-[DOCS](./doc) | [中文](./README_zh.md)
 
 FATE (Federated AI Technology Enabler) is the world's first industrial grade federated learning open source framework to enable enterprises and institutions to collaborate on data while protecting data security and privacy. 
 It implements secure computation protocols based on homomorphic encryption and multi-party computation (MPC). 
@@ -36,8 +35,23 @@ Deploying FATE to multiple nodes to achieve scalability, reliability and managea
 - [Cluster deployment by CLI](./deploy/cluster-deploy): Using CLI to deploy a FATE cluster.
 
 ### Quick Start
-- [Training Demo With Installing FATE AND FATE-Flow From Pypi](doc/2.0/fate/quick_start.md)
 - [Training Demo With Installing FATE Only From Pypi](doc/2.0/fate/ml)
+- [Training Demo With Installing FATE AND FATE-Flow From Pypi](doc/2.0/fate/quick_start.md)
+
+### More examples
+- [ML examples](examples/launchers)
+- [PipeLine examples](examples/pipeline)
+
+## Documentation
+
+### FATE Design
+- [Architecture](./doc/architecture/README.md): Building Unified and Standardized API for Heterogeneous Computing Engines Interconnection
+- [FATE Algorithm Components](./doc/2.0/fate/components/README.md): Building Standardized Algorithm Components for different Scheduling Engines
+- [OSX (Open Site Exchange)](./doc/2.0/osx/osx.md): Building Open Platform for Cross-Site Communication Interconnection
+- [FATE-Flow](https://github.com/FederatedAI/FATE-Flow/blob/main/doc/fate_flow.md): Building Open and Standardized Scheduling Platform for Scheduling Interconnection 
+- [PipeLine Design](https://github.com/FederatedAI/FATE-Client/blob/main/doc/pipeline.md): Building Scalable Federated DSL for Application Layer Interconnection And Providing Tools For Fast Federated Modeling
+- [RoadMap](./doc/images/roadmap.png)
+- [Paper & Conference](./doc/resources/README.md)
 
 ## Related Repositories (Projects)
 - [KubeFATE](https://github.com/FederatedAI/KubeFATE): An operational tool for the FATE platform using cloud native technologies such as containers and Kubernetes.

RELEASE.md

@@ -34,7 +34,7 @@
 > Algorithm Performance Improvements: (Comparison with FATE-v1.11.*)
 * PSI (Privacy Set Intersection): tested on a dataset of 100 million with an intersection result of 100 million, 1.8+ times of FATE-v1.11.4
 * Hetero-SSHE-LR: tested on data of guest 10w * 30 dimensions and host 10w * 300 dimensions, 4.3+ times of FATE-v1.11.4
-* Hetero-NN(Based on FedPass Protocol):  tested on data of guest 10w * 30 dimensions and host 10w * 300 dimensions, basically consistent with the plaintext performance, 56+ times of FATE-v1.11.4 
+* Hetero-NN(Based on FedPass Protocol):  tested on data of guest 10w * 30 dimensions and host 10w * 300 dimensions, basically consistent with the plaintext performance, 143+ times of FATE-v1.11.4 
 * Hetero-Coordinated-LR: tested on data of guest 10w * 30 dimensions and host 10w * 300 dimensions, 1.2+ times of FATE-v1.11.4
 * Hetero-Feature-Binning: tested on data of guest 10w * 30 dimensions and host 10w * 300 dimensions, 1.5+ times of FATE-v1.11.4
 

deploy/cluster-deploy/allinone/fate-exchange_deployment_guide.zh.md

@@ -316,7 +316,7 @@ EOF
 
 **需要连接exchange的各party的osx模块，app用户修改**
 
-修改/data/projects/fate/osx/conf/broker/route_table.json部分，默认路由信息指向部署好的exchange，不需要配置对端fateflow信息，修改后需重启osx：
+修改/data/projects/fate/osx/conf/broker/route_table.json部分，默认路由信息指向部署好的exchange，并配置本方fateflow等信息，不需要配置对端rollsite或者osx信息，修改后需重启osx：
 
 ```
  "default": {
@@ -328,6 +328,35 @@ EOF
             ]
         }
 ```
+比如party9999，即目标服务器（192.168.0.2）配置 /data/projects/fate/osx/conf/broker/route_table.json为：
+
+```
+{
+        "route_table": {
+                "default": {
+                        "default": [
+                                {
+                                        "ip": "192.168.0.1",
+                                        "port": 9370
+                                }
+                        ]
+                },
+                "9999": {
+                        "default": [{
+                                "ip": "rollsite",
+                                "port": 9370
+                        }],
+                        "fateflow": [{
+                                "ip": "fateflow",
+                                "port": 9360
+                        }]
+                }
+        },
+        "permission": {
+                "default_allow": true
+        }
+}
+```
 
 
 

deploy/standalone-deploy/README.zh.md

@@ -41,7 +41,7 @@ pip install fate_client[fate,fate_flow]==2.0.0
 #### 2.2.1.2 服务初始化
 ```shell
 fate_flow init --ip 127.0.0.1 --port 9380 --home $HOME_DIR
-pipeline --ip 127.0.0.1 --port 9380
+pipeline init --ip 127.0.0.1 --port 9380
 ```
 - `ip`：服务运行的ip
 - `port`：服务运行的 http 端口

doc/2.0/fate/ml/hetero_nn_tutorial.md

@@ -234,10 +234,13 @@ def run(ctx):
     ds, model, optimizer, loss, args = get_setting(ctx)
     trainer = train(ctx, ds, model, optimizer, loss, args)
     pred = predict(trainer, ds)
-    # compute auc here
-    from sklearn.metrics import roc_auc_score
-    print('auc is')
-    print(roc_auc_score(pred.label_ids, pred.predictions))
+    if ctx.is_on_guest:
+        # print("pred:", pred)
+        # compute auc here
+        from sklearn.metrics import roc_auc_score
+        print('auc is')
+        print(roc_auc_score(pred.label_ids, pred.predictions))
+
 
 if __name__ == '__main__':
     from fate.arch.launchers.multiprocess_launcher import launch

doc/2.0/fate/performance.md

@@ -18,6 +18,6 @@ Testing configuration:
 | ------------------------| ---------------------------------------- | --------------------------- | ------------------ |
 | PSI                     | 50m54s                                   | 1h32m20s                    |  1.8x+             |                              
 | Hetero-SSHE-LR          | 4m54s/epoch                              | 21m03s/epoch                |  4.3x+             |                              
-| Hetero-NN               | 52.5s/epoch(based on FedPass protocol)   | 2940s/epoch                 |  56x+              |                              
+| Hetero-NN               | 2.425s/epoch(based on FedPass protocol)  | 347s/epoch                  |  143x+             |                              
 | Hetero-Coordinated-LR   | 2m16s/epoch                              | 2m41s/epoch                 |  1.2x+             |                              
 | Hetero-Feature-Binning  | 1m08s                                    | 1m45s                       |  1.5x+             |                              

doc/architecture/README.md

@@ -0,0 +1,18 @@
+### FATE InterOp Goal
+![](../images/interop_goal.png)
+
+### FATE InterOp Principles
+![](../images/interop-principles.png)
+
+###  FATE Overall Architecture
+
+![](../images/fate_arch.png)
+
+###  FATE Core Architecture
+![](../images/fate-core-arch.png)
+
+
+
+
+
+

doc/resources/README.md

@@ -0,0 +1,25 @@
+# Materials
+
+[中文](./README.zh.md)
+
+## Speech and Conference
+
+- [2021. Professor Yang's Seminar Presentation: An Introducton to Federated Learning (2021)](杨强教授：2021联邦学习专题研讨会.pdf)
+- [2019. SecureBoost-ijcai2019-workshop](SecureBoost-ijcai2019-workshop.pdf)
+- [2019. GDPR_Data_Shortage_and_AI-AAAI_2019_PPT](GDPR_Data_Shortage_and_AI-AAAI_2019_PPT.pdf)
+
+
+## Paper
+1. Yang Q, Liu Y, Chen T, et al. Federated machine learning: Concept and applications[J]. ACM Transactions on Intelligent Systems and Technology (TIST), 2019, 10(2): 1-19.
+2. Liu Y, Fan T, Chen T, et al. FATE: An industrial grade platform for collaborative learning with data protection[J]. Journal of Machine Learning Research, 2021, 22(226): 1-6
+3. Cheng K, Fan T, Jin Y, et al. Secureboost: A lossless federated learning framework[J]. IEEE Intelligent Systems, 2021.
+4. Chen W, Ma G, Fan T, et al. SecureBoost+: A High Performance Gradient Boosting Tree Framework for Large Scale Vertical Federated Learning[J]. arXiv preprint arXiv:2110.10927, 2021.
+5. Zhang Q, Wang C, Wu H, et al. GELU-Net: A Globally Encrypted, Locally Unencrypted Deep Neural Network for Privacy-Preserved Learning[C]//IJCAI. 2018: 3933-3939.
+6. Zhang Y, Zhu H. Additively Homomorphical Encryption based Deep Neural Network for Asymmetrically Collaborative Machine Learning[J]. arXiv preprint arXiv:2007.06849, 2020.
+7. Yang K, Fan T, Chen T, et al. A quasi-newton method based vertical federated learning framework for logistic regression[J]. arXiv preprint arXiv:1912.00513, 2019.
+8. Hardy S, Henecka W, Ivey-Law H, et al. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption[J]. arXiv preprint arXiv:1711.10677, 2017.
+9. Chen C, Zhou J, Wang L, et al. When homomorphic encryption marries secret sharing: Secure large-scale sparse logistic regression and applications in risk control[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021: 2652-2662.
+10. Gu H, Luo J, Kang Y, et al. FedPass: Privacy-Preserving Vertical Federated Deep Learning with Adaptive Obfuscation[J]. arXiv preprint arXiv:2301.12623, 2023.
+
+
+

doc/resources/README.zh.md

@@ -0,0 +1,25 @@
+# 资料
+
+[English](./README.md)
+
+## 演讲 & 会议
+- [2021. 杨强教授：2021联邦学习专题研讨会](杨强教授：2021联邦学习专题研讨会.pdf)
+- [2019. SecureBoost-ijcai2019-workshop](SecureBoost-ijcai2019-workshop.pdf)
+- [2019. GDPR_Data_Shortage_and_AI-AAAI_2019_PPT](GDPR_Data_Shortage_and_AI-AAAI_2019_PPT.pdf)
+
+## 论文
+1. Yang Q, Liu Y, Chen T, et al. Federated machine learning: Concept and applications[J]. ACM Transactions on Intelligent Systems and Technology (TIST), 2019, 10(2): 1-19.
+2. Liu Y, Fan T, Chen T, et al. FATE: An industrial grade platform for collaborative learning with data protection[J]. Journal of Machine Learning Research, 2021, 22(226): 1-6
+3. Cheng K, Fan T, Jin Y, et al. Secureboost: A lossless federated learning framework[J]. IEEE Intelligent Systems, 2021.
+4. Chen W, Ma G, Fan T, et al. SecureBoost+: A High Performance Gradient Boosting Tree Framework for Large Scale Vertical Federated Learning[J]. arXiv preprint arXiv:2110.10927, 2021.
+5. Zhang Q, Wang C, Wu H, et al. GELU-Net: A Globally Encrypted, Locally Unencrypted Deep Neural Network for Privacy-Preserved Learning[C]//IJCAI. 2018: 3933-3939.
+6. Zhang Y, Zhu H. Additively Homomorphical Encryption based Deep Neural Network for Asymmetrically Collaborative Machine Learning[J]. arXiv preprint arXiv:2007.06849, 2020.
+7. Yang K, Fan T, Chen T, et al. A quasi-newton method based vertical federated learning framework for logistic regression[J]. arXiv preprint arXiv:1912.00513, 2019.
+8. Hardy S, Henecka W, Ivey-Law H, et al. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption[J]. arXiv preprint arXiv:1711.10677, 2017.
+9. Chen C, Zhou J, Wang L, et al. When homomorphic encryption marries secret sharing: Secure large-scale sparse logistic regression and applications in risk control[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021: 2652-2662.
+10. Gu H, Luo J, Kang Y, et al. FedPass: Privacy-Preserving Vertical Federated Deep Learning with Adaptive Obfuscation[J]. arXiv preprint arXiv:2301.12623, 2023.
+
+
+
+
+