Curve Roadmap 2023

[ilixiaocui]

Update at 2023-07-24

CurveFS Feature

• Performance

progress	expected	describe	related
Doing	Q3	Rename performance optimization	pending (The design scheme has been confirmed, but there is a lack of developers to complete the function development. If you want to participate in the development of this function, please contact us)
Done	Q2	Metadata performance optimization when multiple mounts enable cto consistency(In the cto consistency scenario, the client's metadata cache is invalid, resulting in slow metadata operations)	Done in v2.6

• Function

progress	expected	describe	related
Doing	Q2	Support CurveBS as data storage backend(In terms of function, the data deletion function needs to be completed. performance and stability need to be tested)	pending (The development and basic testing of the single-client mounting mode has been completed in master branch, and the multi-client mounting mode still has data synchronization problems, If you want to participate in the development of this function, please contact us)
Doing	Q3	Support HDFS SDK(hadoop users can use CurveFS as HDFS storage)
Delay	NULL	Support hot and cold data hierarchical storage and life cycle management	Depending on the feature above
Doing	Q3	Support online migration of a cluster across two IDCs(eg. MDS/ETCD/metaservers)
	Q4	Support single replica storage backend (We can use various distributed kv cache systems to replace the S3 storage backend and implement a single copy mode to support temporary data storage services)

• Scenes

progress	expected	describe	related
Done	Q2	Support AI scene(It is necessary to study the IO model of various stages of AI and perform targeted optimization)	We have completed the docking and tuning of AI business scenarios in NetEase, and the performance and cost have achieved significant results. In the second half of the year, there will be specific output of relevant cases and optimization solutions, so stay tuned
Delay	NULL	Support hot data storage of ES\Clickhouse(Need to focus on performance optimization)	Depending on the feature Support CurveBS as data storage backend

CurveBS Feature

• Function

progress	expected	describe	related
Done	Q2	Integrate various tools of CurveBS into new tool	https://github.com/opencurve/curve/tree/master/tools-v2
Done	Q2	Compatible with librbd library interface, no need to compile QEMU and Libvirt when using Curve volume	Basically completed, the beta version is expected to be released in early August
Done	Q2	Hybrid storage engine supporting SSD/NVMe and HDD	It has been used in NetEase, the beta version is expected to be released in early August
Done	Q2	Supports single-cluster multi disk type storage pools, such as all-flash pools and HDD/SSD hybrid storage pools	It has been used in NetEase, the beta version is expected to be released in early August
Delay	NULL	Support EC	pending (The design scheme has been confirmed, but there is a lack of developers to complete the function development. If you want to participate in the development of this function, please contact us)
Delay	NULL	Support single machine scale in cluster	pending (If you want to participate in the development of this function, please contact us)
Doing	Q3	Support volume snapshot store in the CurveBS cluster	Basically completed, the beta version is expected to be released in August
Doing	Q3	Support authentication in Curve system	Basically completed, the beta version is expected to be released in August
Doing	Q3	Support online migration of a cluster across two IDCs(eg. MDS/ETCD/Chunkservers)

• Performance

progress	expected	describe	related
Doing	Q3	Brpc/Braft optimization(Reduce software layer latency. Improve the stability of the cluster running under high load, for example, the heartbeat timeout caused by high load further leads to frequent switching of the replication group leader)
Doing	Q3	Performance analysis and optimization in mixed flash scenarios(NVME+HDD)
Delay	NULL	Performance analysis tool

• Deployment

progress	expected	describe	related
Done	Q2	curveadm support the client boots up and automatically mounts	Done in v0.3

Cloud Native

progress	expected	describe	related
Done	Q2	Support CurveBS operator(Including cluster deployment, daily operation and maintenance, and visualization of monitoring indicators)	https://github.com/opencurve/curve-operator
Done	Q2	Support CurveFS operator(Including cluster deployment, daily operation and maintenance, and visualization of monitoring indicators)	https://github.com/opencurve/curve-operator

CI

progress	expected	describe	related
Done	Q2	CI Stability Optimization of CurveBS	We are using the new stability toolset now in CI
	Q4	CI visualization optimization

[DemoLiang]

我觉得可以考虑优先增加下块存储的开机自启动和自动挂载，当时的bs客户端重启后必须手动启动docker还有手动挂载块设备

[shentupenghui]

我觉得优先云原生化部署和可观测方面特性，减少客户使用运维成本

[shawn0915]

Support hot and cold data hierarchical storage and life cycle management

期待此特性Release，尤其是热数据IO性能调优提升。

[ilixiaocui]

我觉得可以考虑优先增加下块存储的开机自启动和自动挂载，当时的bs客户端重启后必须手动启动docker还有手动挂载块设备

good idea!

[ilixiaocui]

我觉得优先云原生化部署和可观测方面特性，减少客户使用运维成本

1. What exactly does it mean to optimize cloud-native deployment? Currently only csi is supported, the focus of this year in roadmap is the support for operator.

2. What exactly does optimization of observable properties mean? Currently using the form of promethues+grafana. Does it mean that the display form is not enough? Or is the indicator description not enough? or something else？

[ilixiaocui]

Support hot and cold data hierarchical storage and life cycle management

期待此特性Release，尤其是热数据IO性能调优提升。

Welcome to continue to pay attention, it would be even better if we can participate together :)

[Wine93]

Hi, guys, we will discuss CTO optimization draft at 2023-02-21 15:00-16:00.
You can click the tencent meeting link below to join us:
https://meeting.tencent.com/dm/hTU86nag6WG0

design documents

• CTO 元数据性能优化方案.pdf
• JuiceFS 元数据性能分析.pdf
• NFS 测试报告.pdf
• [翻译] nfs(5) - Linux man page.pdf

[caoxianfei1]

Hello, everyone, we will discuss CurveBS deployment solution on Kubernetes at 2023-02-24 16:00-17:00
You can click the tencent meeting link below to join us:
https://meeting.tencent.com/dm/TlpXDnFYcpqI

Design document for reference：
Curve-BS云原生部署方案.pdf

Discussion Group

[legionxiong]

我觉得可以考虑优先增加下块存储的开机自启动和自动挂载，当时的bs客户端重启后必须手动启动docker还有手动挂载块设备

已有方案并通过验证， 近期会提交到社区 review。

[legionxiong]

我觉得可以考虑优先增加下块存储的开机自启动和自动挂载，当时的bs客户端重启后必须手动启动docker还有手动挂载块设备

已有方案并通过验证， 近期会提交到社区 review。

Curve: #2285
CurveAdm: opencurve/curveadm#201

[Cyber-SiKu]

我觉得可以考虑优先增加下块存储的开机自启动和自动挂载，当时的bs客户端重启后必须手动启动docker还有手动挂载块设备

已有方案并通过验证， 近期会提交到社区 review。

Curve: #2285 CurveAdm: opencurve/curveadm#201

您是做的chunkserver自动重启挂载吗？

[lizonglingo]

今天在试用过程中有一些问题想了解下：

1. client 关机/重启后，需要 restart container 再进入手动挂载，退出容器后还需要手动再挂载到系统上。能不能改进 client 重启自动挂载的支持？
2. 关于 UI 交互界面，可能对于运维人员来说，对 pool、user、usage 等集群、存储池、用户、容量的信息最好能清晰明了方便的查看，所以后续有没有 Web 交互集群的支持（如 Longhorn UI 那种，平替 curveadm、curve 命令）？

[ilixiaocui]

今天在试用过程中有一些问题想了解下：

1. client 关机/重启后，需要 restart container 再进入手动挂载，退出容器后还需要手动再挂载到系统上。能不能改进 client 重启自动挂载的支持？
2. 关于 UI 交互界面，可能对于运维人员来说，对 pool、user、usage 等集群、存储池、用户、容量的信息最好能清晰明了方便的查看，所以后续有没有 Web 交互集群的支持（如 Longhorn UI 那种，平替 curveadm、curve 命令）？

1： The curveadm develop version has been supported, but the current client crash and manual stop need to manually unmount the mount point
2：There is already a preliminary version of the console: opencurve/curve-meetup-slides#82

[SeanHai]

Hi, guys, we will discuss authentication in Curve system at 2023-05-23 16:00-17:00.
You can click the tencent meeting link below to join us:
https://meeting.tencent.com/dm/4SH79fdWD4Y6