Merge branch 'master' of https://github.com/TencentBlueKing/bk-ci int…

…o feat_10448_fork

# Conflicts:
#	src/backend/ci/core/misc/biz-plugin/src/main/kotlin/com/tencent/devops/plugin/service/ScmCheckService.kt
#	src/backend/ci/core/misc/biz-plugin/src/main/kotlin/com/tencent/devops/plugin/service/git/CodeWebhookService.kt

.github/workflows/frontend.yml

@@ -5,7 +5,7 @@ name: Frontend CI
 
 on:
   push:
-    branches: [master]
+    branches: ["*"]
     paths:
       - "src/frontend/**"
   pull_request:

README.md

@@ -59,6 +59,8 @@ bk-ci提供了流水线、代码检查、代码库、凭证管理、环境管理
 - [BK-JOB](https://github.com/Tencent/bk-job)：蓝鲸作业平台(Job)是一套运维脚本管理系统，具备海量任务并发处理能力。
 - [BK-PaaS](https://github.com/Tencent/bk-PaaS)：蓝鲸PaaS平台是一个开放式的开发平台，让开发者可以方便快捷地创建、开发、部署和管理SaaS应用。
 - [BK-SOPS](https://github.com/Tencent/bk-sops)：蓝鲸标准运维（SOPS）是通过可视化的图形界面进行任务流程编排和执行的系统，是蓝鲸体系中一款轻量级的调度编排类SaaS产品。
+- [BK-Repo](https://github.com/Tencentblueking/bk-repo)：蓝鲸制品库平台是一套为企业提供各种类型制品包存储、代理、分发、晋级、扫描、依赖包管理的持续交付平台。
+- [BK-Turbo](https://github.com/Tencentblueking/bk-turbo): 蓝鲸编译加速平台为CI场景下提供UE、C/C++等多种语言的编译加速服务能力
 
 ## Contributing
 - 关于 bk-ci 分支管理、issue 以及 pr 规范，请阅读 [Contributing](CONTRIBUTING.md)

README_EN.md

@@ -61,7 +61,9 @@ bk-ci provides seven core services, namely Process, CodeCheck, Repository, Ticke
 - [BK-CMDB](https://github.com/Tencent/bk-cmdb): BlueKing Configuration Management DataBase (BlueKing CMDB) is an enterprise level configuration management platform for assets and applications.
 - [BK-JOB](https://github.com/Tencent/bk-job): BlueKing JOB is a set of operation and maintenance script management platform with the ability to handle a large number of tasks concurrently.
 - [BK-PaaS](https://github.com/Tencent/bk-PaaS): BlueKing PaaS is an open development platform that allows developers to create, develop, deploy and manage SaaS applications quickly and easily.
+- [BK-Repo](https://github.com/Tencentblueking/bk-repo): The BlueKing Artifact Repository Platform is a continuous delivery platform that provides enterprises with various types of artifact package storage, proxy, distribution, promotion, scanning, and dependency package management.
 - [BK-SOPS](https://github.com/Tencent/bk-sops): BlueKing Standard OPS (SOPS) is a light-weighted SaaS product in the Tencent BlueKing product system designed for the orchestration and execution of tasks through a graphical interface.
+- [BK-Turbo](https://github.com/Tencentblueking/bk-turbo): The BlueKing Turbo Platform  provides compilation acceleration services for various languages, including UE and C/C++, in CI scenarios.
 ## Contributing
 
 - Please read [Contributing](CONTRIBUTING.en.md) for the branch management, issue and pr specifications of bk-ci.

docs/overview/db/data_clear.md

@@ -0,0 +1,141 @@
+# 蓝盾数据清理
+## 背景
+
+ 随着蓝盾的构建量快速增长， 蓝盾DB的数据量也越来越大，而蓝盾的DB的容量是有限的，故蓝盾需要一套DB数据清理方案来保证系统的稳定。
+
+
+
+## 清理步骤
+
+#### 一、确定表数据清理方案
+
+​         蓝盾的表可以分为流水数据表和非流水线数据表这二种类型，流水数据表很早之前的数据用户并不关心，所以流水线数库表可以通过分区表的方式来定时清理数据；非流水线数据表的数据相对比较重要，没法简单按时间维度删除数据，故我们需要开发一个依据特定条件删除数据的定时任务。
+
+
+
+#### 二、分区表数据清理
+
+#####   1、确定数据库表是否能调整为分区表
+
+**新增表：** 新表没有历史包袱，只需明确分区的字段和每个分区的大小就行，以下是按时间就行分区表的建表语句：
+
+```
+CREATE TABLE `T_REPOSITORY_COMMIT` (
+  `ID` bigint(20) NOT NULL AUTO_INCREMENT,
+  `BUILD_ID` varchar(34) DEFAULT NULL,
+  `PIPELINE_ID` varchar(34) DEFAULT NULL,
+  `REPO_ID` bigint(20) DEFAULT NULL,
+  `TYPE` smallint(6) DEFAULT NULL COMMENT '1-svn, 2-git, 3-gitlab',
+  `COMMIT` varchar(64) DEFAULT NULL,
+  `COMMITTER` varchar(64) DEFAULT NULL,
+  `COMMIT_TIME` datetime DEFAULT NULL,
+  `COMMENT` longtext /*!99104 COMPRESSED */,
+  `ELEMENT_ID` varchar(34) DEFAULT NULL,
+  `REPO_NAME` varchar(128) DEFAULT NULL,
+  `CREATE_TIME` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
+  `URL` varchar(255) DEFAULT NULL COMMENT '代码库URL',
+  PRIMARY KEY (`ID`,`CREATE_TIME`),
+  KEY `BUILD_ID_INDEX` (`BUILD_ID`),
+  KEY `IDX_PIPE_ELEMENT_REPO_TIME` (`PIPELINE_ID`,`ELEMENT_ID`,`REPO_ID`,`COMMIT_TIME`),
+  KEY `IDX_BUILD_ID_TIME` (`BUILD_ID`,`COMMIT_TIME`),
+  KEY `IDX_PIPE_ELEMENT_NAME_REPO_TIME` (`PIPELINE_ID`,`ELEMENT_ID`,`REPO_NAME`,`COMMIT_TIME`)
+) ENGINE=InnoDB AUTO_INCREMENT=3170191500 DEFAULT CHARSET=utf8mb4
+/*!50100 PARTITION BY RANGE (TO_DAYS(CREATE_TIME))
+(PARTITION p20240424 VALUES LESS THAN (739366) ENGINE = InnoDB,
+ PARTITION p20240501 VALUES LESS THAN (739373) ENGINE = InnoDB,
+ PARTITION p20240508 VALUES LESS THAN (739380) ENGINE = InnoDB,
+ PARTITION p20240515 VALUES LESS THAN (739387) ENGINE = InnoDB,
+ PARTITION p20240522 VALUES LESS THAN (739394) ENGINE = InnoDB,
+ PARTITION p20240529 VALUES LESS THAN (739401) ENGINE = InnoDB,
+ PARTITION p20240605 VALUES LESS THAN (739408) ENGINE = InnoDB,
+ PARTITION p20240612 VALUES LESS THAN (739415) ENGINE = InnoDB,
+ PARTITION p20240619 VALUES LESS THAN (739422) ENGINE = InnoDB,
+ PARTITION p20240626 VALUES LESS THAN (739429) ENGINE = InnoDB,
+ PARTITION p20240703 VALUES LESS THAN (739436) ENGINE = InnoDB,
+ PARTITION p20240710 VALUES LESS THAN (739443) ENGINE = InnoDB,
+ PARTITION p20240717 VALUES LESS THAN (739450) ENGINE = InnoDB,
+ PARTITION p20240724 VALUES LESS THAN (739457) ENGINE = InnoDB,
+ PARTITION p20240731 VALUES LESS THAN (739464) ENGINE = InnoDB,
+ PARTITION p20240807 VALUES LESS THAN (739471) ENGINE = InnoDB,
+ PARTITION p20240814 VALUES LESS THAN (739478) ENGINE = InnoDB,
+ PARTITION p20240821 VALUES LESS THAN (739485) ENGINE = InnoDB,
+ PARTITION p20240828 VALUES LESS THAN (739492) ENGINE = InnoDB) */;
+```
+
+**存量表：**  存量表要在不影响现有业务的情况下调整为分区表相对比较麻烦点，以T_REPOSITORY_COMMIT表为例，以下是具体步骤：
+
+1、创建临时表T_REPOSITORY_COMMIT_TMP(必须为分区表，分区键必须包含在主键内)
+
+2、把T_REPOSITORY_COMMIT表的数据同步至T_REPOSITORY_COMMIT_TMP表并建立同步方案
+
+3、执行 RENAME TABLE T_REPOSITORY_COMMIT TO T_REPOSITORY_COMMIT_BAK,T_REPOSITORY_COMMIT_TMP TO T_REPOSITORY_COMMIT ,T_REPOSITORY_COMMIT_BAK TO T_REPOSITORY_COMMIT_TMP; 该语句进行新旧表名互换。
+
+4、删除T_REPOSITORY_COMMIT_TMP表
+
+
+
+##### 2、为分区表创立自动创建分区和删除分区的执行计划
+
+
+
+**各微服务数据库的分区表情况：**
+
+| 数据库名称        | 表名                                | 分区字段        | 过期时间（单位：天） | 分区间隔（单位：天） |
+| ----------------- | ----------------------------------- | --------------- | -------------------- | -------------------- |
+| devops_process    | T_PIPELINE_TRIGGER_EVENT            | CREATE_TIME     | 35                   | 7                    |
+| devops_process    | T_PIPELINE_TRIGGER_DETAIL           | CREATE_TIME     | 35                   | 7                    |
+| devops_process    | T_PIPELINE_BUILD_VAR                | CREATE_TIME     | 42                   | 7                    |
+| devops_process    | T_PIPELINE_BUILD_TASK               | CREATE_TIME     | 42                   | 7                    |
+| devops_process    | T_PIPELINE_BUILD_STAGE              | CREATE_TIME     | 42                   | 7                    |
+| devops_process    | T_PIPELINE_BUILD_CONTAINER          | CREATE_TIME     | 42                   | 7                    |
+| devops_process    | T_PROJECT_PIPELINE_CALLBACK_HISTORY | CREATE_TIME     | 3                    | 3                    |
+| devops_process    | T_PIPELINE_WEBHOOK_BUILD_LOG        | CREATE_TIME     | 3                    | 3                    |
+| devops_process    | T_PIPELINE_WEBHOOK_BUILD_LOG_DETAIL | CREATE_TIME     | 3                    | 3                    |
+| devops_repository | T_REPOSITORY_COMMIT                 | CREATE_TIME     | 35                   | 7                    |
+| devops_repository | T_REPOSITORY_WEBHOOK_REQUEST        | CREATE_TIME     | 35                   | 7                    |
+| devops_notify     | T_NOTIFY_WEWORK                     | CREATED_TIME    | 8                    | 1                    |
+| devops_notify     | T_NOTIFY_EMAIL                      | CREATED_TIME    | 8                    | 1                    |
+| devops_notify     | T_NOTIFY_RTX                        | CREATED_TIME    | 8                    | 1                    |
+| devops_notify     | T_NOTIFY_SMS                        | CREATED_TIME    | 8                    | 1                    |
+| devops_notify     | T_NOTIFY_WECHAT                     | CREATED_TIME    | 8                    | 1                    |
+| devops_plugin     | T_PLUGIN_GIT_CHECK                  | CREATE_TIME     | 31                   | 1                    |
+| devops_stream     | T_GIT_REQUEST_EVENT_BUILD           | CREATE_TIME     | 35                   | 7                    |
+| devops_stream     | T_GIT_REQUEST_REPO_EVENT            | CREATE_TIME     | 35                   | 7                    |
+| devops_stream     | T_GIT_REQUEST_EVENT_NOT_BUILD       | CREATE_TIME     | 35                   | 7                    |
+| devops_stream     | T_GIT_USER_MESSAGE                  | CREATE_TIME     | 35                   | 7                    |
+| devops_stream     | T_GIT_REQUEST_EVENT                 | CREATE_TIME     | 35                   | 7                    |
+| devops_metrics    | T_PROJECT_THIRD_PLATFORM_DATA       | CREATE_TIME     | 371                  | 7                    |
+| devops_metrics    | T_PIPELINE_FAIL_DETAIL_DATA         | CREATE_TIME     | 371                  | 7                    |
+| devops_metrics    | T_PIPELINE_FAIL_SUMMARY_DATA        | CREATE_TIME     | 371                  | 7                    |
+| devops_metrics    | T_ATOM_MONITOR_DATA_DAILY           | STATISTICS_TIME | 371                  | 7                    |
+| devops_metrics    | T_ATOM_INDEX_STATISTICS_DAILY       | CREATE_TIME     | 371                  | 7                    |
+| devops_metrics    | T_ATOM_FAIL_DETAIL_DATA             | CREATE_TIME     | 210                  | 7                    |
+| devops_metrics    | T_PIPELINE_OVERVIEW_DATA            | CREATE_TIME     | 371                  | 7                    |
+| devops_metrics    | T_ATOM_FAIL_SUMMARY_DATA            | CREATE_TIME     | 371                  | 7                    |
+| devops_metrics    | T_PIPELINE_STAGE_OVERVIEW_DATA      | CREATE_TIME     | 371                  | 7                    |
+| devops_metrics    | T_ATOM_OVERVIEW_DATA                | CREATE_TIME     | 371                  | 7                    |
+| devops_metrics    | T_PROJECT_USER_DAILY                | THE_DATE        | 90                   | 15                   |
+| devops_metrics    | T_PROJECT_BUILD_SUMMARY_DAILY       | THE_DATE        | 90                   | 15                   |
+
+
+
+#### 三、非分区表数据清理
+
+非分区表采用misc服务的定时任务依据时间、构建数量等条件进行清理，具体配置如下：
+
+```
+build:
+  data:
+    clear:
+      switch: true # 是否开启自动清理构建历史数据；true:开启，false:不开启
+      maxEveryProjectHandleNum: 5 #并发清理项目数量
+      monthRange: -1 # 清理多少个月前的普通渠道（BS）流水线运行时数据，对于卡在 stage 审核的流程在清理后就无法继续审核。
+      maxKeepNum: 10000 # 普通渠道（BS）流水线最大保留多少条构建历史记录 
+      codeccDayRange: -14 # 清理多少天前的codecc渠道流水线运行时数据
+      codeccMaxKeepNum: 14 # codecc渠道流水线最大保留多少条构建历史记录 
+      otherMonthRange: -1 # 清理多少天前的其它渠道流水线运行时数据
+      otherMaxKeepNum: 500 # 其它渠道流水线最大保留多少条构建历史记录 
+      clearChannelCodes: "BS,PREBUILD,CI,CODECC" # 支持清理构建上数据的渠道类型
+      maxThreadHandleProjectNum: 5 # 开启清理线程最大数量
+```
+

docs/overview/db/devops_ci_dispatch.md

@@ -438,6 +438,7 @@
 |  18   | CONTAINER_HASH_ID |   varchar   | 128 |   0    |    Y     |  N   |       | 容器 ID 日志使用  |
 |  19   | ENV_ID |   bigint   | 20 |   0    |    Y     |  N   |       | 第三方构建所属环境  |
 |  20   | IGNORE_ENV_AGENT_IDS |   json   | 1073741824 |   0    |    Y     |  N   |       | 这次调度被排除的 agent 节点  |
+|  21   | JOB_ID |   varchar   | 128 |   0    |    Y     |  N   |       | 当前构建所属 jobid  |
 
 **表名：** <a>T_DISPATCH_THIRDPARTY_AGENT_DOCKER_DEBUG</a>
 

docs/overview/db/devops_ci_environment.md

@@ -190,6 +190,7 @@
 |  1   | ENV_ID |   bigint   | 20 |   0    |    N     |  Y   |       | 环境 ID  |
 |  2   | NODE_ID |   bigint   | 20 |   0    |    N     |  Y   |       | 节点 ID  |
 |  3   | PROJECT_ID |   varchar   | 64 |   0    |    N     |  N   |       | 项目 ID  |
+|  4   | ENABLE_NODE |   bit   | 1 |   0    |    N     |  N   |   b'1'    | 是否启用节点  |
 
 **表名：** <a>T_ENV_SHARE_PROJECT</a>
 

docs/overview/db/devops_ci_log.md

@@ -42,10 +42,12 @@
 |  3   | TAG |   varchar   | 64 |   0    |    Y     |  N   |       | 标签  |
 |  4   | SUB_TAG |   varchar   | 256 |   0    |    Y     |  N   |       | 子标签  |
 |  5   | JOB_ID |   varchar   | 64 |   0    |    Y     |  N   |       | JOBID  |
-|  6   | MODE |   varchar   | 32 |   0    |    Y     |  N   |       | LogStorageMode  |
-|  7   | EXECUTE_COUNT |   int   | 10 |   0    |    N     |  N   |       | 执行次数  |
-|  8   | FINISHED |   bit   | 1 |   0    |    N     |  N   |   b'0'    | buildisfinishedornot  |
-|  9   | CREATE_TIME |   datetime   | 23 |   0    |    N     |  N   |   CURRENT_TIMESTAMP(3)    | 创建时间  |
+|  6   | USER_JOB_ID |   varchar   | 128 |   0    |    Y     |  N   |       | 真正的 jobId，已经存在的 JOB_ID 字段其实是 containerhashid  |
+|  7   | STEP_ID |   varchar   | 64 |   0    |    Y     |  N   |       | 用户填写的插件 id  |
+|  8   | MODE |   varchar   | 32 |   0    |    Y     |  N   |       | LogStorageMode  |
+|  9   | EXECUTE_COUNT |   int   | 10 |   0    |    N     |  N   |       | 执行次数  |
+|  10   | FINISHED |   bit   | 1 |   0    |    N     |  N   |   b'0'    | buildisfinishedornot  |
+|  11   | CREATE_TIME |   datetime   | 23 |   0    |    N     |  N   |   CURRENT_TIMESTAMP(3)    | 创建时间  |
 
 **表名：** <a>T_LOG_SUBTAGS</a>
 

docs/overview/db/devops_ci_process.md

@@ -211,7 +211,7 @@
 |  4   | BUILD_NUM |   int   | 10 |   0    |    Y     |  N   |   0    | 构建次数  |
 |  5   | PROJECT_ID |   varchar   | 64 |   0    |    N     |  N   |       | 项目 ID  |
 |  6   | PIPELINE_ID |   varchar   | 34 |   0    |    N     |  N   |       | 流水线 ID  |
-|  7   | VERSION |   int   | 10 |   0    |    Y     |  N   |       | 版本号  |
+|  7   | VERSION |   int   | 10 |   0    |    Y     |  N   |       | 编排版本号  |
 |  8   | START_USER |   varchar   | 64 |   0    |    Y     |  N   |       | 启动者  |
 |  9   | TRIGGER |   varchar   | 32 |   0    |    N     |  N   |       | 触发器  |
 |  10   | START_TIME |   timestamp   | 19 |   0    |    Y     |  N   |       | 开始时间  |
@@ -241,7 +241,7 @@
 |  34   | BUILD_NUM_ALIAS |   varchar   | 256 |   0    |    Y     |  N   |       | 自定义构建号  |
 |  35   | CONCURRENCY_GROUP |   varchar   | 255 |   0    |    Y     |  N   |       | 并发时,设定的 group  |
 |  36   | UPDATE_TIME |   datetime   | 19 |   0    |    Y     |  N   |   CURRENT_TIMESTAMP    | 更新时间  |
-|  37   | VERSION_NAME |   varchar   | 64 |   0    |    Y     |  N   |       | 版本名称  |
+|  37   | VERSION_NAME |   varchar   | 64 |   0    |    Y     |  N   |       | 正式版本名称  |
 |  38   | YAML_VERSION |   varchar   | 34 |   0    |    Y     |  N   |       | YAML 的版本标记  |
 
 **表名：** <a>T_PIPELINE_BUILD_HISTORY_DEBUG</a>
@@ -258,7 +258,7 @@
 |  4   | BUILD_NUM |   int   | 10 |   0    |    Y     |  N   |   0    | 构建次数  |
 |  5   | PROJECT_ID |   varchar   | 64 |   0    |    N     |  N   |       | 项目 ID  |
 |  6   | PIPELINE_ID |   varchar   | 34 |   0    |    N     |  N   |       | 流水线 ID  |
-|  7   | VERSION |   int   | 10 |   0    |    Y     |  N   |       | 版本号  |
+|  7   | VERSION |   int   | 10 |   0    |    Y     |  N   |       | 编排版本号  |
 |  8   | START_USER |   varchar   | 64 |   0    |    Y     |  N   |       | 启动者  |
 |  9   | TRIGGER |   varchar   | 32 |   0    |    N     |  N   |       | 触发器  |
 |  10   | START_TIME |   timestamp   | 19 |   0    |    Y     |  N   |       | 开始时间  |
@@ -1265,6 +1265,7 @@
 |  9   | SECRET_TOKEN |   text   | 65535 |   0    |    Y     |  N   |       | Sendtoyourwithhttpheader:X-DEVOPS-WEBHOOK-TOKEN  |
 |  10   | ENABLE |   bit   | 1 |   0    |    N     |  N   |   b'1'    | 启用  |
 |  11   | FAILURE_TIME |   datetime   | 19 |   0    |    Y     |  N   |       | 失败时间  |
+|  12   | SECRET_PARAM |   text   | 65535 |   0    |    Y     |  N   |       | 鉴权参数  |
 
 **表名：** <a>T_PROJECT_PIPELINE_CALLBACK_HISTORY</a>
 
@@ -1399,3 +1400,4 @@
 |  12   | BUILD_NO |   text   | 65535 |   0    |    Y     |  N   |       | 构建号  |
 |  13   | PARAM |   mediumtext   | 16777215 |   0    |    Y     |  N   |       | 参数  |
 |  14   | DELETED |   bit   | 1 |   0    |    Y     |  N   |   b'0'    | 流水线已被软删除  |
+|  15   | INSTANCE_ERROR_INFO |   text   | 65535 |   0    |    Y     |  N   |       | 实例化错误信息  |