User QoS resource total limit #370

huerni · 2024-11-20T09:38:49Z

No description provided.

huerni · 2024-11-20T09:40:22Z

可能功能还没完整实现，可以先看看AccountMetaContainer，QosResource，UserResourceMeta这些实现形式有什么问题

RileyWen · 2024-11-21T11:15:57Z

这些操作太重了开会聊下吧

RileyWen · 2024-11-26T10:30:23Z

src/CraneCtld/AccountMetaContainer.h

理论上来说这个地方我们没有锁定全集的需求可以放松限制用带sharding的concurrent hashmap. 见https://github.com/greg7mdp/parallel-hashmap

这块AtomicHashmap全用Exclusive Locking太重了

RileyWen · 2024-11-26T10:49:49Z

src/CraneCtld/AccountManager.cpp

@@ -833,6 +834,12 @@ AccountManager::CraneExpected<void> AccountManager::ModifyQos(
  // Mongodb
  Qos qos;
  g_db_client->SelectQos("name", name, &qos);
+
+  // Modify QosResource when max_jobs_per_user or max_cpus_per_user is changed.
+  if (item == "max_jobs_per_user" || item == "max_cpus_per_user")


这个地方可以enum化吗

应该可以

RileyWen · 2024-11-26T10:50:28Z

src/CraneCtld/AccountManager.cpp

+  for (const auto& [partition, qos] :
+       res_user.account_to_attrs_map[account_name].allowed_partition_qos_map) {
+    for (const auto& qos_name : qos.second) {
+      const Qos* qos_content = GetExistedQosInfoNoLock_(qos_name);


这个地方应该是不需要查qos表的吧

RileyWen · 2024-11-26T10:52:00Z

src/CraneCtld/AccountManager.cpp

@@ -1580,6 +1587,18 @@ AccountManager::CraneExpected<void> AccountManager::AddUser_(
  }
  res_user.account_to_attrs_map[object_account].blocked = false;

+  AccountMetaContainer::QosResourceList qos_resource_list;
+  for (const auto& [partition, qos] :


qos的作用已经在任务提交的check阶段被apply了这个地方不需要也不应该查qos吧

要记录每个user的qos资源使用情况的话，还是得有个地方记录user的资源变化那可以放在CheckAndApplyQosLimitOnTask中？然后全集就不需要锁了，删除操作也不需要了，CheckAndApplyQosLimitOnTask已经判断了所以只需要malloc和free时，对qos_to_resource_map加锁就可以了

目前的实现逻辑应该就已经是这样的了任务在经过CheckAndApplyQosLimitOnTask的时候本来就应用了QOS的限制就不需要专门存了只需要记录每一个用户的被应用Qos限制后的资源总量就好了

RileyWen · 2024-11-29T06:57:17Z

src/CraneCtld/CtldPublicDefs.h

+
+struct QosResourceLimit {
+  QosResource res_total;
+  QosResource res_avail;


这个地方没必要三个吧？。。

如果作业运行中修改Qos资源的话，还是需要三个

…edQos, or ModifyQos are performed.

…eQosResourceOnUser.

huerni · 2024-12-03T08:08:17Z

现在是 checkAndApplyQos时判断用户对应Qos资源变化，对用户修改Qos不会影响队列中的作业，和之前的情况一致，然后ModifyQos的max_jobs_per_user和max_cpus_per_user时，会实时影响队列中的作业

L-Xiafeng · 2024-12-04T08:46:55Z

src/CraneCtld/CtldPublicDefs.h

@@ -703,6 +703,17 @@ inline bool CheckIfTimeLimitIsValid(absl::Duration d) {
  return CheckIfTimeLimitSecIsValid(sec);
 }

+struct QosResource {
+  uint32_t cpus_per_user;


Qos可能会有GPU资源，要不这里直接用ResPerUser这样来表示

对这个地方用ResourceView感觉可能就好了

RileyWen · 2024-12-04T15:46:32Z

src/CraneCtld/AccountManager.cpp

@@ -973,6 +981,11 @@ result::result<void, std::string> AccountManager::CheckAndApplyQosLimitOnTask(
      qos_share_ptr->max_cpus_per_user)
    return result::fail("cpus-per-task reached the user's limit.");

+  g_account_meta_container->AddQosResourceToUser(


g_account_meta_container 这个类不需要维护User的Qos呀这个只要AccountManager维护就好了这边增加一个Copy又要维护一致性还要引入很多锁就在AccountManager::CheckAndApplyQosLimitOnTask 里面去g_account_meta_container 取一次user已用资源总量和 task的resource求一次和，再和 qos的上限一对比就好了 g_account_meta_container 就是负责简单的运行时资源统计，正常来说对外暴露线程安全的加和减操作感觉就行了这样子不用多维护一致性这样AddQosResourceToUser FreeQosResource InitFromDB_ ModifyQosResourceOnUser 都可以省了而且效率更高要做的事情就是把加和减操作的并行度增大开销降低就好了

整个逻辑就简单很多了

这个逻辑的话是在CheckAndApplyQosLimitOnTask阶段就已经扣除了Qos资源，也就是默认队列中的所有任务已经使用了Qos资源我实现的是任务运行时扣除Qos资源，确实也没必要我照着你这个逻辑改一下吧

L-Xiafeng · 2024-12-11T08:36:38Z

src/CraneCtld/AccountMetaContainer.cpp

+          return;
+        }
+
+        val.resource.GetAllocatableRes().cpu_count += task.cpus_per_task;


这里应该直接加task.requested_node_res_view*node

L-Xiafeng · 2024-12-11T08:36:50Z

src/CraneCtld/AccountMetaContainer.cpp

+  user_meta_map_[username].qos_resource_in_use.modify_if(
+      task.qos, [&](std::pair<const std::string, QosResource>& pair) {
+        auto& val = pair.second;
+        val.resource.GetAllocatableRes().cpu_count -= task.cpus_per_task;


L-Xiafeng · 2024-12-11T08:43:57Z

src/CraneCtld/CtldPublicDefs.h

+  uint32_t jobs_per_user;
+};
+
+struct ResourcePerUser {


这个东西可以放AccountMetaContainer.h里面吧

可以的，我开始就是放在里面的

huerni added the enhancement New feature or request label Nov 20, 2024

huerni linked an issue Nov 20, 2024 that may be closed by this pull request

用户Qos资源总量限制 #369

Open

huerni self-assigned this Nov 20, 2024

huerni force-pushed the dev/qos_res_limit branch from e4a4a44 to a324eeb Compare November 21, 2024 07:44

RileyWen reviewed Nov 26, 2024

View reviewed changes

RileyWen reviewed Nov 29, 2024

View reviewed changes

huerni added 13 commits December 3, 2024 09:44

feat: AccountMetaContainer

ac24230

feat: InitFromDB

290de2e

fix: CheckAndApplyQosLimitOnUser

3d26835

feat: MallocQosResourceToUser

0df3171

feat: Improve AccountMetaContainer.

6ba3204

feat: add g_account_meta_container

f64b25c

feat:malloc and free qos resource

621237a

feat: Synchronize resources when AddUser, DeleteUser, ModifyUserAllow…

44180b3

…edQos, or ModifyQos are performed.

refactor

16d6d7b

fix: In the function DeleteAccountAllowedQosFromMapNoLock_, call Eras…

4a9bc01

…eQosResourceOnUser.

feat: add parallel-hashmap

a83d4d1

feat: reduce lock granularity

00c1401

refactor

475f71f

huerni force-pushed the dev/qos_res_limit branch from 7ae6410 to 475f71f Compare December 3, 2024 08:06

L-Xiafeng reviewed Dec 4, 2024

View reviewed changes

RileyWen reviewed Dec 4, 2024

View reviewed changes

refactor

e7cdbe8

huerni force-pushed the dev/qos_res_limit branch from 1b285a3 to e7cdbe8 Compare December 5, 2024 08:39

huerni requested review from RileyWen and L-Xiafeng December 9, 2024 01:42

L-Xiafeng reviewed Dec 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

User QoS resource total limit #370

User QoS resource total limit #370

huerni commented Nov 20, 2024

huerni commented Nov 20, 2024

RileyWen commented Nov 21, 2024

RileyWen Nov 26, 2024

RileyWen Nov 26, 2024

RileyWen Nov 26, 2024

huerni Nov 27, 2024

RileyWen Nov 26, 2024

RileyWen Nov 26, 2024

huerni Nov 27, 2024

RileyWen Nov 29, 2024

RileyWen Nov 29, 2024

huerni Dec 3, 2024

huerni commented Dec 3, 2024

L-Xiafeng Dec 4, 2024

RileyWen Dec 4, 2024

RileyWen Dec 4, 2024

RileyWen Dec 4, 2024

huerni Dec 5, 2024

L-Xiafeng Dec 11, 2024

huerni Dec 11, 2024

L-Xiafeng Dec 11, 2024

L-Xiafeng Dec 11, 2024

huerni Dec 11, 2024

User QoS resource total limit #370

Are you sure you want to change the base?

User QoS resource total limit #370

Conversation

huerni commented Nov 20, 2024

huerni commented Nov 20, 2024

RileyWen commented Nov 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huerni commented Dec 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment