Code

Add code eval#587

Code

Merged

mcarbin merged 29 commits intomainmosaicml/llm-foundry:mainfrom sam/add-coding-evalmosaicml/llm-foundry:sam/add-coding-evalCopy head branch name to clipboardSep 26, 2023

+605-7

Commits

Commits on Sep 8, 2023

• add code eval

  samhavens

  committed
  a57dde8

  Copy full SHA for a57dde8

Commits on Sep 12, 2023

• Create coding_tasks.yaml

  samhavens

  authored
  a63c4aa

  Copy full SHA for a63c4aa

• Update coding_tasks.yaml

  samhavens

  authored
  71de0bf

  Copy full SHA for 71de0bf

• Update tasks.yaml

  Show description for 85b727f
  samhavens

  authored
  85b727f

  Copy full SHA for 85b727f

• Update eval_gauntlet.yaml

  Show description for b0670e8
  samhavens

  authored
  b0670e8

  Copy full SHA for b0670e8

Commits on Sep 13, 2023

• update readme add code eval

  samhavens

  committed
  1e868d8

  Copy full SHA for 1e868d8

• torch nograd eval_model

  samhavens

  committed
  c6e54ae

  Copy full SHA for c6e54ae

Commits on Sep 14, 2023

• integrate seed

  mcarbin

  committed
  a2c877b

  Copy full SHA for a2c877b

• Merge branch 'mosaicml:sam/add-coding-eval' into sam/add-coding-eval

  mcarbin

  authored
  071ddde

  Copy full SHA for 071ddde

Commits on Sep 15, 2023

• pass_at_k threading

  mcarbin

  committed
  12c0175

  Copy full SHA for 12c0175

Commits on Sep 22, 2023

• remove extraneous print

  mcarbin

  committed
  2755e8d

  Copy full SHA for 2755e8d

• update num_beams to default of 20

  mcarbin

  committed
  06ae4fb

  Copy full SHA for 06ae4fb

• update num_beamss for coding eval to 20 as the standard for pass@1

  mcarbin

  committed
  bfbbf17

  Copy full SHA for bfbbf17

• remove C from default coding tasks because it's too small

  mcarbin

  committed
  e042eb7

  Copy full SHA for e042eb7

• update num_beams in full task list to default of 20

  mcarbin

  committed
  1fa60ec

  Copy full SHA for 1fa60ec

Commits on Sep 25, 2023

• Merge branch 'main' into sam/add-coding-eval

  mcarbin

  committed
  9e241f5

  Copy full SHA for 9e241f5

• fix break from merge and precommit

  mcarbin

  committed
  4b5f2d9

  Copy full SHA for 4b5f2d9

• yamllint

  mcarbin

  committed
  c3819de

  Copy full SHA for c3819de

• remove code finetuning

  mcarbin

  committed
  cb45e4b

  Copy full SHA for cb45e4b

• Merge branch 'main' into sam/add-coding-eval

  mcarbin

  committed
  e3d7dda

  Copy full SHA for e3d7dda

Commits on Sep 26, 2023

• Merge branch 'main' into sam/add-coding-eval

  dakinggg

  authored
  0acec3d

  Copy full SHA for 0acec3d

• Merge branch 'main' into sam/add-coding-eval

  dakinggg

  authored
  79d1d8c

  Copy full SHA for 79d1d8c

• bump version for CI

  mcarbin

  committed
  628e020

  Copy full SHA for 628e020

• Merge remote-tracking branch 'refs/remotes/mosaicml/sam/add-coding-eval' into sam/add-coding-eval

  mcarbin

  committed
  797c690

  Copy full SHA for 797c690

• update gauntlet

  mcarbin

  committed
  0860f91

  Copy full SHA for 0860f91

• update tasks

  mcarbin

  committed
  fbb7342

  Copy full SHA for fbb7342

• Merge branch 'main' into sam/add-coding-eval

  mcarbin

  authored
  b10beaf

  Copy full SHA for b10beaf

• remove torch.no_grad

  mcarbin

  committed
  63470b4

  Copy full SHA for 63470b4

• Merge branch 'main' into sam/add-coding-eval

  dakinggg

  authored
  707a2e7

  Copy full SHA for 707a2e7