Code

[FEAT] Ray shuffle experiment#2883

Code

Closed

jaychia wants to merge 16 commits intomainEventual-Inc/Daft:mainfrom jay/shuffle-experimentEventual-Inc/Daft:jay/shuffle-experimentCopy head branch name to clipboard

+901-38

Commits

Commits on Sep 29, 2024

• Added a Ray-based shuffle service that uses an actor-per-node, no idea if this works though

  

  Jay Chia

  committed
  2bdb0ce

• Naive forwarding to physical plan

  

  Jay Chia

  committed
  0e91bb2

• yoloswag

  

  Jay Chia

  committed
  fb1fde4

• Who knows if this works

  

  Jay Chia

  committed
  f137172

• fix

  

  Jay Chia

  committed
  b90435c

• fix

  

  Jay Chia

  committed
  0ae0d71

• only request for little bit of CPU on actor to match placement group

  

  Jay Chia

  committed
  626e952

• rename to fully materializing

  

  Jay Chia

  committed
  a9bfb4f

• Add another streaming push variant

  

  Jay Chia

  committed
  cef8843

• stub out the thing

  

  Jay Chia

  committed
  005c804

• Add simple pull-based shuffle

  

  Jay Chia

  committed
  7e0f88d

• Who knows if this works

  

  Jay Chia

  committed
  0bc057d

• naming

  

  Jay Chia

  committed
  37faaa0

• Fixes it works locally

  

  Jay Chia

  committed
  1e8b87d

• Add node affinity for reducer and merger

  

  Jay Chia

  committed
  37b1d8e

Commits on Sep 30, 2024

• Fix math for number of returns from mapper

  

  Jay Chia

  committed
  ade5e2f